JP6091092B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method for calculating a distance from a first camera to an object by a stereo method from first and second images obtained by imaging the object with first and second cameras.

  Conventionally, for example, by using two cameras to capture two images, the principle of triangulation is used to measure the distance from one camera to an object, for example (see Patent Document 1). . This is installed so that the object is centered on the main camera, and the angle of the slave camera is adjusted so that the object is projected at the center of the image, the distance between the master camera and the slave camera, The distance from the main camera to the object is calculated based on the angle of the slave camera.

  Patent Document 1 also proposes measuring the size of an object based on the number of effective pixels of the object imaged by the main camera based on the number of pixels in the horizontal direction of the reference object stored in advance. .

Japanese Patent Laid-Open No. 9-318351

  By the way, in the distance surveying method as described in Patent Document 1, it is necessary for the operator to perform mechanical adjustment such as setting the distance between the main camera and the slave camera, and particularly adjusting the angle of the slave camera. There is a problem that it takes. In addition, there is a problem that the measurement accuracy is not good because mechanical errors and work errors occur.

  Therefore, in recent years, a stereo method using an epipolar line has been used as a method for measuring the distance to an object using a plurality of cameras in the field of image processing. Specifically, first, a point corresponding to the feature point of the object in the image of the main camera is specified. Next, an epipolar plane passing through the master camera and the feature points in the image of the master camera and the slave camera is set, and an epipolar line is set in the image plane of the slave camera. Subsequently, the points corresponding to the feature points of the object in the image of the main camera are associated by searching on the epipolar line on the image plane of the slave camera. Then, the distance between the main camera and the object can be calculated from the positional relationship in the three-dimensional virtual space of the main camera, the sub camera, the feature point, and the corresponding point on the epipolar line by the principle of triangulation. it can.

  However, for example, when a plurality of objects having the same shape are captured in the image, when searching for a point corresponding to the feature point of the object on the epipolar line of the image of the slave camera, an object similar to the epipolar line There may be more than one. At this time, if another object is erroneously detected as a point corresponding to the feature point of the object to be measured, there is a problem that the calculation of the distance between the main camera and the object is also erroneously calculated.

  Therefore, the present invention prevents erroneous detection when searching for feature points of an object on the epipolar line of the second image, and can accurately calculate the distance from the first camera to the object. It is an object of the present invention to provide a processing device and an image processing method.

An image processing apparatus according to the present invention acquires a first image obtained by imaging an object with a first camera and a second image obtained by imaging the object with a second camera having a line of sight different from that of the first camera, An epipolar line is generated on the second image, a feature point of the object on the second image is specified by searching on the epipolar line, and from the first camera to the object by a stereo method. In the image processing apparatus that calculates the distance , the reference object is stored in the storage unit in advance by the first camera at the reference distance between the reference object and the first camera. the number of pixels at the time of photographing to be substantially angle full length of said object, and based on the distance at which the reference, prior to the first camera from the image primes of the object in the first image The approximate value of the distance to the object is calculated, and the search range when searching for the feature point of the object on the epipolar line of the second image is set to be narrowed according to the approximate value of the distance. A control unit is provided.

In the image processing method according to the present invention, the control unit captures a first image by capturing an object with the first camera, and the control unit has a line of sight different from that of the first camera. A second image acquisition step of acquiring a second image by imaging a target object with a second camera, and an epipolar line generation step of generating an epipolar line on at least the second image, The control unit stores the reference object at the reference camera between the reference camera and the reference camera between the reference camera and the first camera. the number of pixels at the time of photographing so that the length of the object, and based on the distance at which the reference, the image prime of the object in the first image distance between the object and the first camera Approximate distance for calculating approximate values The calculation step and the control unit are set to narrow a search range when searching for a feature point of the object on the epipolar line of the second image according to the approximate value of the calculated distance. A search range setting step, and the control unit specifies a feature point of an object on the second image by searching on the epipolar line in the set search range, and the first method is performed by a stereo method. A distance calculating step of calculating a distance from the camera to the object.

According to the present invention, at the reference distance between the reference target object and the first camera stored in the storage unit in advance, the reference target object is substantially viewed by the first camera. full and the number of pixels when photographed so, the length of the object, and based on the distance to be the reference, approximate value of the distance between the first camera and the object from the image prime object in the first image And the search range when searching for the feature point of the object on the epipolar line of the second image can be set to be narrowed according to the approximate value of the distance. Thereby, even if there are a plurality of similar objects on the epipolar line of the second image, for example, it is possible to prevent erroneous detection when searching for feature points of the object. The distance to the object can be calculated with high accuracy.

The schematic diagram which shows the robot apparatus which concerns on embodiment of this invention. The schematic diagram which shows the structure of the robot controller of FIG. The figure which shows the relationship between a main camera image, a subcamera image, and an epipolar plane. The figure which shows the relationship between the effective pixel number of a workpiece | work in a main camera image, and distance. The figure which shows the search range on the epipolar line of a subcamera image. The flowchart which shows the distance calculation control which concerns on this Embodiment.

  Hereinafter, embodiments according to the present invention will be described with reference to FIGS. 1 to 6. First, a robot apparatus 1 including an image processing apparatus (robot controller 10) according to the present invention will be described with reference to FIG.

  As shown in FIG. 1, a robot apparatus 1 according to the present embodiment is an assembly robot called a so-called robot cell that is combined with a plurality of robot apparatuses to construct a series of production lines. The robot apparatus 1 is configured such that a robot arm 2 that performs an assembly operation on a work (object) W is disposed inside a frame-shaped booth 7. A camera 3 and an illumination 5 for imaging the workpiece W from above the robot arm 2 are attached to the top of the booth 7.

  A robot controller (image processing apparatus, computer) 10 that provides a drive command to the robot arm 2 is provided inside the gantry 6 that supports the robot arm 2. The robot controller 10 calculates the distance between the workpiece W and the camera 3 from the image captured by the camera 3, recognizes the three-dimensional position from the distance, the line of sight of the camera 3, etc., and issues a drive command to the robot arm 2. Generate. The robot apparatus 1 has the basic configuration of the robot arm 2, the camera 3, and the robot controller 10. The robot apparatus 1 recognizes the work W based on the image captured by the camera 3, and grips the work W by the robot arm 2 for assembly. Work to do.

  As shown in FIG. 2, the camera 3 is composed of a stereo camera including a main camera (first camera) 3a and a slave camera (second camera) 3b whose camera parameters are calibrated. Is used to recognize the position (distance) of the workpiece W.

  The robot controller 10 includes an arithmetic device (control unit) 102, a storage device (storage unit) 103, an interface device 106, a recording disk reading device 115, a communication device 112, and the like. Is communicably connected. The bus 111 is connected to the robot arm 2 and the teaching pendant 116 described above. The teaching pendant 116 is an operation terminal for giving an operation command to the robot arm 2.

  The arithmetic unit 102 includes a CPU 102a, and the CPU 102a calculates the position (distance) of the work W from a stereo image obtained by capturing the work W by the master camera 3a and the slave camera 3b as will be described in detail later. A drive command is given to the arm 2. The arithmetic device 102 includes an image processing device 102b and a sound processing device 102c, and is configured to output a desired image and sound to the display 107 and the speaker 109.

The storage device 103 includes a RAM 103b and a ROM 103a. For example, the ROM 103a stores an image processing program IPP for performing image processing (calculation of distance) described in detail later. Further, as will be described in detail later, an image obtained by capturing a reference workpiece W _BASE in advance is temporarily stored in the RAM 103b or the like. Note that the image processing program IPP may be read from information recorded on, for example, a recording disk Disk and stored in the ROM 103a, or may be stored in the ROM 103a in a form downloaded from an external network via the communication device 112. .

  The interface device 106 is connected to the stereo camera 3 including the main camera 3a and the slave camera 3b described above via an interface 106c. Thereby, the image G1 imaged by the main camera 3a and the image G2 imaged by the slave camera 3b can be received and temporarily stored in the RAM 103b of the storage device 103 or the like. On the other hand, it is also possible to change the imaging conditions such as exposure, shutter speed, and zoom by transmitting a command from the CPU 102a or the like to the main camera 3a or the slave camera 3b. For example, a keyboard 106a or a mouse 106b for an operator to input information may be connected to the interface 106c.

  Next, the principle of distance calculation by the stereo method using epipolar lines will be described with reference to FIG. First, a first image G1 captured by the main camera 3a and a second image G2 captured by the slave camera 3b are acquired. Then, an epipolar plane passing through the main camera 3a, the slave camera 3b, and the work W (line of sight connecting the camera and the work) can be set on the three-dimensional virtual space. A line passing through the epipolar plane can be projected onto the second image G2 and set as an epipolar line EP. It can be said that the epipolar line EP of the second image G2 is a line of sight from the main camera 3a to the feature point of the workpiece W. Since the epipolar plane is obtained, the epipolar line EP can be set also in the first image G1.

  Since the feature point Pb of the workpiece W exists on the epipolar line EP in the first image G1, and the feature point Pr of the workpiece W exists on the epipolar line EP in the second image G2, the feature points Pb, Pr are Search and seek. When the feature point Pr is obtained, the viewing angle between the master camera 3a and the feature point Pr of the work W is obtained from the slave camera 3b. Therefore, the triangle is obtained from the view angle and the known distance between the master camera 3a and the slave camera 3b. Based on the principle of surveying, the distance L between the main camera 3a and the workpiece W can be calculated. Since the distance between the slave camera 3b and the workpiece W can also be calculated based on the same principle, the three-dimensional position (three-dimensional coordinates) of the workpiece W can be obtained from the relationship between these distances and viewing angles. Therefore, if the distance between the camera 3 and the workpiece W can be calculated, the three-dimensional position of the workpiece W can also be calculated.

  Next, distance calculation control according to the present embodiment will be described with reference to FIGS. As shown in FIG. 6, when the distance calculation control is started (started), first, the computing device (control unit) 102 acquires the first image G1 obtained by photographing the workpiece W from the main camera 3a (S1) ( First image acquisition step). Similarly, the arithmetic unit 102 acquires the second image G2 in which the workpiece W is photographed from the slave camera 3b (S2) (second image acquisition step).

  Subsequently, the arithmetic unit 102 calculates the epipolar line EP as described above (S3) (see FIG. 3) (epipolar line generation step), and projects it on the first image G1 and the second image G2. When the epipolar line EP is projected onto the first image G1, the computing device 102 calculates a feature point Pb in the first image G1 (main camera image) (S4).

Next, in step S5, an approximate value of the distance L (see FIG. 3) between the main camera 3a and the workpiece W is calculated. In detail, first, as shown in FIG. 4C, the workpiece W _BASE and the main camera 3a are arranged so that both ends in the horizontal direction of the reference workpiece W _BASE are captured in full within the angle of view of the main camera 3a. Adjust the distance and take a picture with the main camera 3a. At that time, the distance D from the main camera 3a to the work W _BASE is measured, and further, the number of pixels P _W of the work W _BASE is measured. Then, the storage device 103 stores the values of the known length X of the workpiece W _{BASE in} the horizontal direction, the horizontal length WID1 on the image, the number of pixels _PW, and the distance D.

As long as there is no change in the positional relationship between the workpiece and the main camera 3a, the correspondence between the length WID1 on the horizontal image of the workpiece W _{BASE serving as} the reference and the number of pixels P _W is sufficient as long as the positional relationship between the workpiece and the main camera 3a is not changed. When calculating the distance L to the workpiece W, it can be handled as a constant.

  In order to obtain the distance L to the workpiece W having a known length X in the horizontal direction on the first image G1 using each constant obtained above, a synergistic ratio as shown in FIG. Thus, it can be obtained by the following formula (1). Note that the length of the main camera 3a on the first image G1 in the horizontal direction at a position away from the main camera 3a by a distance L is “WID2”.

The length WID2, when measuring the number of pixels P _X of the work W of known length X in the horizontal direction described above, each of the horizontal length and synergistic number of pixels of the workpiece W _BASE as the workpiece W and the reference It is calculated | required by following Numerical formula (2) using ratio.

  In short, the distance L from the main camera 3a to the workpiece W can be obtained from the following equation (3) from the equations (1) and (2).

In the above calculation example, the horizontal length of the workpiece W is calculated as the feature amount of the workpiece W. However, the present invention is not limited to this, and the vertical length or area of the workpiece W is used as the feature amount for calculation. Also good. When the feature amount is the length in the vertical direction of the workpiece W, Px in Equation (3) is the number of pixels in the vertical direction of the workpiece W, X is a known length in the vertical direction of the workpiece W, and WID1 is the workpiece W _BASE. , Pw is the number of pixels in the vertical direction of the work W _BASE . When the feature amount is the area of the work W, Px in the above formula (3) is the number of pixels of the area of the work W, X is the area of the work W, WID1 is the area of the work W _BASE , and Pw is the work W _BASE. This is the number of pixels of the area.

  As described above, the arithmetic device 102 measures the pixel number Px of the work W from the first image G1 of the main camera 3a, and uses the constants stored in the storage device 103 to calculate the main camera from the above equation (3). An approximate value of the distance L from 3a to the workpiece W is calculated (S5) (approximate distance calculating step).

Next, the computing device 102 sets the search range when searching for the feature point Pr of the object on the epipolar line EP of the second image G2 in accordance with the approximate value of the calculated distance L. (S6) (search range setting step). Specifically, as shown in FIG. 5, the position A ′ moved −d _{Z to} the near side of the main camera 3 a from the position separated by the approximate value of the distance L calculated from the main camera 3 a, and + d _Z on the far side. The search range is set (limited) to the moved position B ′. Incidentally, ± d _Z is set to the maximum value but the combined size error of the image recognition error and the workpiece W in the image processing operations.

  Subsequently, the computing device 102 searches for a feature point of the work W between positions A′B ′ on the epipolar line EP in the second image G2 of the slave camera 3b, and features on the image within the search range. To calculate the feature point Pr (S7).

  Then, the distance L from the main camera 3a to the workpiece W is calculated by using the stereo method described above from the feature point Pb of the first image G1 obtained in step S4 and the feature point Pr specified in step S7. (S8) (Distance calculation step). This is the end of the distance calculation control.

That is, as shown in FIG. 5, for example, when searching at the position AB on the epipolar line EP that is full of the angle of view in the second image G2 of the slave camera 3b, on the image at the adjacent workpiece W _{+ 1} or workpiece W− _1, for example. There is a risk of misdetecting these features as feature points. However, to calculate the approximate value of the distance L to the workpiece W, since the search range is limited to a range from the vicinity of ± d _Z, for example, adjacent work W ₊₁ and work W _-1 is that can be excluded from the search range, False detection due to feature point association is prevented.

As described above, to calculate the approximate value of the distance L between the main camera 3a and the workpiece W from the number of pixels P _X feature amount of the work W in the first image G1. Then, according to the approximate value of the distance L, the search range when searching for the feature points of the workpiece W on the epipolar line EP of the second image G2 can be set to be narrowed. Thereby, even if there are a plurality of similar works, for example, on the epipolar line EP of the second image G2, it is possible to prevent erroneous detection when searching for a feature point of the work W. From the main camera 3a, The distance to the workpiece W can be calculated with high accuracy.

  Since the distance to the workpiece, that is, the three-dimensional position of the workpiece can be calculated with high accuracy, even when the robot arm 2 performs an operation on the workpiece W, a highly accurate and accurate operation can be performed.

  In the present embodiment, an example in which the image processing apparatus is mounted on the robot controller 10 has been described as an example. However, the present invention is not limited to this, and what is required to accurately measure the distance to an object includes: Anything can be used. That is, for example, it can be used for a device that is required to detect a distance to an object using an image, such as a three-dimensional shape measuring instrument or an automobile collision prevention device.

3a ... 1st camera (main camera): 3b ... 2nd camera (secondary camera): 10 ... Image processing device, computer (robot controller): 102 ... Control unit (arithmetic unit): 103 ... Storage unit (storage device): D ... distance: EP ... epipolar lines: G1 ... first image: G2 ... second image: L ... distance: Pr ... minutiae: P W _... number of pixels: P X _... number of pixels: W ... object (workpiece)

Claims (4)

A first image obtained by imaging the object with the first camera and a second image obtained by imaging the object with a second camera having a line of sight different from that of the first camera are acquired, and an epipolar line is formed on at least the second image. In the image processing apparatus, the feature point of the object on the second image is specified by searching on the epipolar line, and the distance from the first camera to the object is calculated by a stereo method. ,
At the reference distance between the reference target object and the first camera stored in the storage unit in advance , the reference target object is substantially filled with the angle of view by the first camera. the number of pixels at the time of shooting, calculates the length of the object, and based on the distance at which the reference, the approximate value of the distance from the image prime object and said first camera and said object in said first image And, according to the approximate value of the distance, provided with a control unit that is set to narrow the search range when searching for the feature points of the object on the epipolar line of the second image,
An image processing apparatus. The first and second cameras;
The image processing apparatus according to claim 1, wherein a distance from the first camera to an object is calculated;
A robot arm that performs work on the object based on the calculated distance,
A robot apparatus characterized by that. A first image acquisition step in which the control unit acquires a first image by capturing an image of an object with the first camera;
A second image acquisition step in which the control unit acquires a second image by capturing an object with a second camera having a line of sight different from that of the first camera;
The controller generates an epipolar line on at least the second image; and
At the reference distance between the reference object and the first camera stored in the storage unit in advance by the control unit , the reference object is substantially viewed by the first camera. full become as the number of pixels at the time of shooting, the length of the object, and based on the distance at which the reference distance from the image prime object to the first camera the object in the first image An approximate distance calculating step for calculating an approximate value of
A search range setting step in which the control unit sets so as to narrow a search range when searching for a feature point of the object on the epipolar line of the second image according to the approximate value of the calculated distance. When,
The control unit specifies a feature point of the object on the second image by searching on the epipolar line in the set search range, and from the first camera to the object by a stereo method. A distance calculating step for calculating a distance,
An image processing method.   An image processing program for causing a computer to execute each step according to claim 3.

Images