JPS6015780A - Robot - Google Patents

Abstract

PURPOSE:To attain a perfect visual feedback by performing the arm control of a robot by using the distance (distance picture) between each point of an object and a certain fixed point and a picture obtained on the robot arm to detect the 3-dimensional position, posture and form of the object. CONSTITUTION:A distance picture detector 19 obtains the distance picture of the object 11 and sends it to an image processor 21. The processor 21 also receives a picture viewed from the robot arm side from a picture detector 20 set on the arm 17. Based on both pictures, the form, position and posture of the object 11 are recognized respectively. The control signal is sent to a robot controller 18 to perform the visual feedback control so that the arm 17 is corrected to be at a correct position to the object 11. The processor 21 differentiates the distance picture signal to produce a jump edge picture and separates the object 11 into the areas closed by the jump edge for the recognition of the object 11. Thus the centroid and inclination of an important plane are also detected from the separated areas.

Description

[Detailed Description of the Invention] [Field of Application of the Invention] The present invention relates to the presence or absence of a target object necessary for performing various operations such as assembly, inspection, and adjustment by a robot. This is about a robot that can recognize.

[Background of the invention]

To recognize the position and orientation of an object using robot vision, an image detector 3 is placed on an IC work table 1 as shown in FIG. To detect. Then, the position and orientation of the object are recognized, and the four-bot arm 2 operates according to that information.

In this method, the plane position and orientation of an object on the table plane can be recognized, but the three-dimensional shape cannot be recognized. For example, it is not possible to approach objects piled up on a table by tilting the robot arm so as to match the grip surface of the object according to the inclination information of each object. Furthermore, unless there is a clear contrast (difference in shading) between the object and the work table, the position and orientation on the table plane cannot be determined.

On the other hand, some robots have visual devices attached to their hands to provide feedback. FIG. 2 is an example of this, in which a welding line 5 to be welded by a welding head 7 is tracked. In this case, a slit source light projector 6 is attached to the tip of the robot arm 2, and a slit light 8 is projected onto the welding line. The image is then detected from a diagonal direction with an image detector 6 attached to the tip of the robot arm, the position of the bath tangent is determined, and the robot arm is controlled so that the deviation from the target value is zero. .

However, this method can only detect the position of the weld line, that is, the position of the groove-shaped object;
Recognize posture, assemble, and inspect.

It is not possible to perform adjustments or other work.

[Purpose of the invention]

An object of the present invention is to eliminate the drawbacks of the prior art described above, and to provide a robot that can work by recognizing the shape, position, and orientation of a target object in the robot's coordinate system, regardless of the orientation and shape of the target object. It's about doing.

[Summary of the invention]

The present invention provides a detector that detects distance information, that is, a distance image, from a point in a fixed coordinate system of the robot to each point on the surface of the target object, and a distance image of the entire work target. The detected distance image is then differentiated two-dimensionally using an image processor to detect the outline of the target object, and the area separated by this outline and the distance image corresponding to that area are obtained. Discover objects placed three-dimensionally from a distance, recognize their positions, postures, and shapes, transmit the results to the robot arm control device, and move the robot arm from the three-dimensional direction of the target object. Then, the image detector mounted on the robot arm detects the position and orientation of the target object again with better accuracy than the distance image detector described above, and sends the robot arm a correction □. The ultimate goal is to enable the robot to work on a workpiece with arbitrary three-dimensional positions and orientations with high positioning accuracy.

[Embodiments of the invention]

Hereinafter, the present invention will be explained in detail with reference to the drawings.

First, the processing contents in the present invention will be explained. As mentioned above, the processing consists of the first step of detecting a distance image of the entire work object from a fixed point in the robot's coordinate system, discovering the target object, and recognizing its position and orientation; The second step consists of moving the robot arm closer to the target object IC using the results of , and then detecting the position and orientation of the target more accurately using the image of ゛ from the image detector on the robot arm. (Figure 13).

As shown in FIG. 3, the distance image 9 detected in the first stage is determined by the value Vi of each point of the 11M1 image or the distance between the corresponding point P on the surface of the object 11 and the detector 10. l! ■su to the target = 10 drinks...(]) Release: non-zero constant, C: constant It is an image with a linear relationship. Therefore, especially when K is a negative number, the square can be regarded as representing the height of each point on the surface of the object. Now, if we define a case where several objects are folded one on top of the other and captured by a distance image detector from directly above, as shown in FIG. 4, then a distance image 9 as shown in FIG. is detected.

As is clear from FIG. 5, since the distance to the detector generally changes rapidly at the boundary line between objects or between objects and the background, the value of the distance image also changes rapidly. In distance images, such boundaries are called jump edges.

When this change is detected using two-dimensional differentiation of the image,
It will look like Figure 6. Several methods are known for two-dimensional differentiation of images, but here we will use a calculation that calculates the differential value d of the point ('l)) on the image shown in Figure 7. . However, (number, )) is the orthogonal coordinate of a plane perpendicular to the distance direction between the detector and the object. When the obtained differential image is binarized using a certain fixed threshold value, a jump edge image as shown in FIG. 8 is obtained. In the figure, the black line represents the jump edge 12, which corresponds to the outline of the object as described above. Therefore, jump edge 12
The areas surrounded by correspond to each object, and by separating and extracting each area on the image, the objects can be separated on the image.

Region R corresponding to each object separated and extracted as above
, the area S and the average height Hl of the area, that is, the area that meets the criteria, is extracted as the area of the target object. S,, Sl are the upper and lower limits of the area variation range when the target object is projected onto a certain plane, and l-1+, Hz are the upper and lower limits of the height at which the target object is expected to exist. Each value is a value set in advance. The shape of the area on the image of the target object separated in this way generally does not have a constant shape. In other words, this shape is the two-dimensional shape of the target object projected onto a certain plane. For example, if the three-dimensional shape of the target object is a rectangular parallelepiped, it may be rectangular or rectangular depending on its orientation as shown in Figure 9. It becomes a hexagon. In the present invention, in order to recognize the three-dimensional shape two positions and postures of the target object,
Assuming that the object is made up of several planes, we detect the intersection lines between the planes, that is, the ridge lines. The ridge line corresponds to a point on the distance image where the rate of change in value changes, as shown in FIG. 10, that is, a point where the second-order differential value of the image has an extreme value. Such a group of lines is called a half edge 13 in a distance image. There are various methods for the second-order differential method of images, but here we use the upper integer, and the value of r is determined by the change in the inclination angle of the distance image value V° of the point of interest ('*J'). handle.

Next, by binarizing the quadratic differential image calculated using equation (5) according to a certain threshold, the roof edge 15
Detect. The operation shown in equation (5) is as follows.

It may be applied over the entire detected distance image, but
Performing calculations only on the region of the target object that was previously separated and extracted leads to a reduction in processing time.

Jump edge 12 of the target object detected through the above processing. For example, in the case of a distance image of an enlarged rectangular parallelepiped shown in FIG. 10, an image consisting of half edges 13 is
It will look like Figure 11.

Since the regions surrounded by jump edges 12 and half edges 13 correspond to individual faces that make up the object, the regions are separated on the image 7, and by extracting the region 14 with the largest area, the main parts that make up the object are extracted. One of the faces 15 can be taken out.

Next, plane approximation is performed for this region. That is, the plane equation that can be claimed is tIx + % y + ts = z ・= (61, detected from the range image of m points in the area.Zyz coordinates are (aftx, a4, hi) '1 = = 1.2・
...-, then tl Itm Itm in equation (6) is (
tk) is a matrix t whose rows are t = (ATA)-' ATS (7). Here, A is a matrix whose rows are (α#1.α!, 1.0), b is a matrix whose rows are (hrt), ' is a transposed matrix, and -8 is an inverse matrix. Also, the xyz coordinates (αkl
, α#, , ri can be expressed as the following equation (8) by performing linear transformation on the coordinates Ci, j) on the distance image and the value ゛ of the distance image.

This is a conversion process from the coordinates on the detected image to the actual coordinates of the detection system. xll eq+ z'lt *q
s is a constant determined based on the detection resolution of the detector, etc.

From the approximation equation (6) for the principal plane 15 of the target object obtained in this way, the three-dimensional posture of this object is determined by the angle ψ between the two axes in the normal direction 16 of the principal plane. normal x
Using the angle θ between the projection onto the y plane and the X axis, θ-tan-' -! -! It is expressed as tl..., -(II (Fig. 12).If the direction of the normal line is determined,
The shape of the main plane seen from this direction can be determined by simple coordinate transformation, and from the transformed shape, it is also possible to recognize the shape and orientation of the target object using conventional planar image processing technology. .

In addition, the position of the target object is the center of gravity of the main plane ("o*
yoyq) is determined by the following 0υ formula.

Here, - represents the average within the area. In addition, the average in this case is not limited to m points within the region, but can be calculated over the entire region to obtain a more accurate one.

Now, the processing result of the first stage explained above, that is, the position of the center of gravity of the main plane of the target object (S, C)
The directions ψ and θ of the normal line are based on the coordinate system of the detection system, and since this sitting image system is generally different from the coordinate system of the robot arm, the robot arm cannot be directly controlled using these values. For this purpose, the coordinates are transformed into the coordinate system of the robot arm. Depending on the control system of the robot arm, various coordinate systems such as a rectangular coordinate system, a cylindrical coordinate system, and a polar coordinate system can be considered, and conversion to these coordinate systems is well known from linear mathematical formulas.

In addition, in each process of the first stage explained above,
Although the calculation methods are limited to formulas +21, +51, +71, etc., it goes without saying that other calculation methods having similar functions may be used.

Next, the robot arm is controlled so that the image detector installed on the robot arm can detect the object from the direction normal to the center of gravity of the main plane of the target object.

Thereafter, the image is detected from above the robot arm, the positional relationship between the robot arm and the target object is detected more accurately, and the process moves to the second stage of the process, which leads to accurate work of the robot arm. The image detector 20 on the robot arm has a higher resolution than the distance image detector used in the first stage, but since the target object has been discovered in the first stage, its detection field of view can be small. . As the detected images, as in the first stage, a distance image is detected and similar processing is performed;
Various methods can be considered, such as a method that detects and processes a dimensional grayscale image, a method that processes a binary image thereof, a method that applies a light cutting method using slit light, and any of these methods may be used.

Next, a specific example of an apparatus that implements the processing contents described above will be described. FIG. 13 shows an outline of the overall configuration. As shown in the figure, the device includes a robot system consisting of a robot arm 17 and a robot control device 18, and a distance image detector 1.
9, a visual system consisting of an image detector 20 and an image processor 21 on the robot arm.

First, the robot system will be explained using FIG. 16.

17 is an industrial robot, for example, an articulated robot having five degrees of freedom. Industrial - The bot 17 has a swivel base 17α that rotates around a vertical axis with respect to a base 22, and a horizontal axis 2.
Upper arm 17b rotating around 5α, horizontal axis 23 at the end
Forearm 17C1 rotating around A, horizontal axis 23a at the end
and a wrist 17d that rotates around an axis perpendicular to the horizontal axis 25C. A hand mechanism 24 with fingers (chuck) 25 is attached to the wrist 17d. Image detector 200 mounted on the robot arm. Although it may be attached to any of these locations, it is desirable to attach it to the hand mechanism 24 in order to operate the robot and detect images from the most appropriate direction.

18 is a robot control device that controls the robot.

Next, the robot control device 18 will be explained according to FIG. The robot controller 18 includes a control unit 26 for controlling the articulated robot 17 having five degrees of freedom, and a control unit 26 for operating or moving the robot along a predetermined trajectory based on the velocity programmed from point to point. , and a teaching unit 27 for teaching pre-programmed trajectory and speed information. The control unit 26 and the robot mechanism 17 constitute a position control system. This position control system counts output pulses generated by a pulse encoder PE connected to an actuator M with a counter 28 and feeds them back to a control unit 26, and a target value predetermined by a microprocessor 29 or given from the outside. The system is configured to detect a digital signal that is different from the desired coordinate value, convert this digital signal into an analog signal by a D/A converter 30, and drive the actuator M.

The drive circuit 31 is a circuit that drives the actuator M based on a speed signal from a tacho generator TG connected to the actuator M and an analog signal from the D/A converter 30. The serial interface 32 is for connecting with the teaching unit 27. The ROM 33 is a memory that stores programs for operating the robot.

The RAM 34 inputs information from a teaching operation performed using the teaching unit 27 or operation information from the image processor 29 input via the interface 35 to the calculation unit 3.
The robot hand mechanism 24 is the result of interpolation calculation using 6.
This is to store the movement locus of the movement. 37 is a pass line.

11AM The position data of the four-bot hand mechanism 24 stored in the 34 is read out by the microprocessor 29,
The rotational displacement θ1 detected by the counter 28. θ1.・
... θ, and drives the hand mechanism 24 of the robot to a desired or target position (for example, the target object position input from the visual system).

Although the above description of the four-bot system is limited to robots with five degrees of freedom, robots with other degrees of freedom, for example, six degrees of freedom, can be configured in the same way. In the present invention, the degree of freedom is particularly limited.

It's not something you do.

Next, the visual system, which is another component of this embodiment, will be explained.

An embodiment of the distance image detector 19 is shown in FIG.

Is the detector above the object 11 as shown in the figure?

The slit light source 39 that emits the slit light 38 from the slit light source 39 and the image pickup device 40 that detects the optical image of the intersection line of the slit light 38 and the object 11, that is, the slit bright line from a diagonal direction, are placed horizontally while maintaining their positional relationship. It consists of a feeding device 41 that is driven at a constant speed in the direction, and a light cutting line extraction device 42 that separates and extracts the slit bright line detected by the imager 40 from the image and outputs its shape as a waveform signal. To explain this operation using FIGS. 16 to 20, the slit light 38
Suppose that the object 11 is in contact with the position shown in FIG. 16. Then, the image detected by the imager 40 is
For example, it becomes as shown in FIG. The change in brightness along the vertical line, for example mAB, on this image is the 18th
It will look like the figure. By sequentially moving line AB in the i direction and extracting the position of the brightest point (C in Figure 18) along this line AB, the shape of the slit bright line can be changed into a waveform as shown in Figure 19. It can be taken out as a signal.

This shape indicates the cross-sectional shape of the object 11. The separation and extraction of the waveform signals described above is performed by the optical cutting line extraction device M42 shown in FIG. A specific example of this optical cutting line extraction device 42 is disclosed in, for example, Japanese Patent Laid-Open No. 70407/1983.

That is, as shown in FIG. 26, the optical cutting line extraction device 42 receives the video signal V from the image detector, the set value ＼, and the trigger signal Hd for each scan of the image detector (in the case of a TV camera). , horizontal synchronization signal) as input, and the maximum value position detection circuit 57
, a center position detection circuit 58 , and a selection circuit 59 .

Its functions, as illustrated in FIG. 27 (α), are as follows:
If the maximum IK Vmax of the video signal is within the range between the first and second set values, the maximum value position detection circuit 7 determines the y position Y. Also, Vmax is ζ
(in the case of FIG. 27Cb)), the y position
Finally, move to the y position where V = again, and further 1' = (
Find the center position as Yt + Yt )A. During one scan, if ■ becomes equal to or greater than or equal to within the J path 58, the selection circuit 59 outputs the output of the circuit 58; on the other hand, if ■ does not become equal to or more than output as the position of the optical cutting line. Also,
If ■ never exceeds V1 during one scan,
The selection circuit 59 outputs zero or an undefined output.

FIG. 28 shows the embodiment of FIG. 26 in more detail. The contents of the operation will be explained in more detail using FIG. 28.

The maximum position detection circuit 57 detects the peak holder 60 . Comparator 61. One-shot circuit 62. AND circuit 65. Register 64. It is composed of a comparator 66 and a flip-flop 67. On the other hand, the center position detection circuit 58 is connected to the comparator 68. flip flop69. One-shot circuits 70, 71. Y, a register 72Y, a register 73, and an AND circuit 74. Also, y at each moment
It has a counter 65 to know the position. In the maximum position detection circuit 57, the video signal ■ enters a peak holder 60, which holds the peak value of V up to a certain moment during one scan. The comparator 61 compares this held peak value with V at that moment, and outputs a signal to the one-shot circuit 62 when ■ becomes larger than the held peak value by a predetermined value or more. On the other hand, ■ is compared with a pigeon in a comparator 66, and outputs when ■≧pigeon.

By ANDing this with the output of the one-shot circuit 62 in an AND circuit 63, it is possible to detect the moment when ■≧V and ■ becomes the peak. Add this to register 640
The instantaneous value of the counter 65, which is used as a fourth signal and is reset by the trigger signal ntL, is stored in the register 64. Therefore, at the end of the -scan, the video signal V holds the y-coordinate position Y where the maximum value ■mαX is shown.

In the center position detection circuit 58, the video signal V is compared with a set value by a comparator 68, and its output is input to the set side S of a flip-flop 69. Note that the flip-flop 69 is reset by the trigger signal HcL. Therefore, the one-shot circuit 70 detects the first moment when ■≧S in one scan. By using this as the fourth signal of the register 72, the register 72 holds the position X on the y coordinate where ■≧different for the first time. On the other hand, the one-shot circuit 71 detects the instant when ■≧ does not hold, and if this is used as a load signal for the register 73, the register 73 will have the following information: [, V≧ζ and ( Position Y on the y coordinate,
is held. 74 is Yr=(washiyt)/2
This is an average value circuit that calculates .

The output of the clip 70 knob 69 is when V≧V1.
It can be used to detect whether it was present during one scan. Furthermore, the flip-flop 67 receives a trigger signal n
Since d is used as a reset signal and the output of the comparator 66 is used as a set signal, it can be used to detect whether the case of ■≧ exists during one scan.

Therefore, in the selection circuit 59, these are used as condition signals, and
≧ If there is a pigeon, change the 740 hold value Y' to <v
≦Pigeon, 640 hold value Y is V≧V, and if there is no case, outputs a signal representing zero or indeterminate.

Although details of the image detectors 40 and 42 were not mentioned in this embodiment, they are a combination of a TV camera secondary solid-state image sensor, a one-dimensional solid-state image sensor, and an optical scanning method.
Any combination of a light receiver such as a photomultiplier tube and an optical scanning method may be used.

Further, although the embodiments shown in FIGS. 26 and 28 are illustrated as having dedicated hardware for achieving each means, these means can all be processed by a program using a microcomputer. I can do it.

Furthermore, the slit light 38 is sent little by little at a constant speed by the feeding device 41.
By successively extracting waveform signals in the shape of the slit bright line while moving the position and the imaging position, the second waveform signal as a whole is obtained.
As shown in FIG. 0, a distance image 9 is obtained.

The imager 40 in this embodiment may be any two-dimensional image detector, such as a TV camera or a combination of a linear sensor and a galvano mirror. can be detected.

FIG. 21 shows another embodiment of the distance image detector 19. As shown in the figure, this embodiment has a structure in which two sets of the detector imager and the optical cutting line extraction device shown in FIG. 15 are combined. That is, with respect to the plane formed by the slit light 38,
Two imagers 40α and 40h are provided at symmetrical positions, and the slit light source 39 is also moved by the feeding device 41. The method for extracting the optically cut waveform is exactly the same as in the previous example, but in this embodiment, the obtained waveform signals are synthesized so as to compensate for the cut portions of the waveforms, and a distance image is generated.

Here, the broken part of the waveform corresponds to a part 44 where the slit bright line cannot be detected by one of the imagers, as shown in FIG. 22, and becomes a kind of shadow in the distance image. In this embodiment, as described above, since waveform signals detected from two directions are combined, there is an effect that a distance image with fewer shadows can be obtained.

In addition, as a detector for the distance image, a pulsed laser is used to irradiate the corresponding point, and the opposite! 'F A method of generating a distance image by measuring the flight time of the light, applying high frequency amplitude modulation to the laser source, irradiating it to the corresponding point, and measuring the phase lag of the high frequency amplitude of the reflected light to create a distance image. There are known generation methods, and these may also be used. In these methods, the distance between each point on the object is measured using a laser spot, so the mechanism that scans the spot two-dimensionally, for example, two sets of galvanometer mirrors, is 8 hard.
On the other hand, these methods have the advantage of being able to illuminate the source from directly above and detect the reflected light from directly above, so that a distance image with no invisible parts, ie, no blind spots or shadows, can be generated.

Next, an example of the image detector 20 attached to the robot arm will be described.

Figure 23 shows the image detector 2 attached to the robot arm.
This is an example of a distance image detector with 0 and '. The TV right camera 5 detects a light cutting waveform caused by the slit light 38 projected from the slit light source 39. This detector is attached to the hand mechanism 24 of the robot, and a distance image can be generated by extracting the optical cutting line and capturing the waveform signal while moving the hand. The distance image can be processed in exactly the same way as the distance image attached to a fixed position described above. Note that details of this embodiment and other embodiments are as follows. That is, 77 is an industrial robot, for example, an articulated robot having five degrees of freedom. Industrial robot 17
is a rotating base 1 that rotates around the millet axis with respect to the base 22.
7e, an upper arm 17h1 that rotates around a horizontal axis 25a-; a forearm 17c1 that rotates around a horizontal axis 23A; a forearm 17c that rotates around a horizontal axis 25c;
The wrist 17d rotates around an axis perpendicular to 0. A hand mechanism 24 with fingers (chuck) 25 is attached to the wrist 17d. This hand mechanism 24
is equipped with a main body shape detector 76. The main body shape detector 76 is scanned by a light cutting head 77, a linear movement mechanism 79 that scans the light cutting head 77 in the y-axis direction, a motor 80 that drives the linear movement mechanism 79, and a linear movement mechanism 79. The displacement detector 82, such as a rotary encoder, detects the scanning amount of the optical cutting radar 77 from the reference position.

Although FIG. 29 only depicts a feed screw and a nut as the linear movement mechanism 79, there is actually a sliding mechanism.

83 is a motor control circuit that drives the motor 80 at a constant speed. Reference numeral 42 denotes a light section line extraction circuit which inputs the two-dimensional image signal obtained from the image detector 78 and extracts a light section line as described in Japanese Patent Application Laid-Open No. 70407/1983. Reference numeral 21 denotes an image processor that detects the position and inclination of an object that is closest to the robot and that the robot wants to work on. 18 is a robot control device that controls the robot 17.

FIG. 23 shows the optical cutting head 7 in the embodiment of FIG. 29.
7 is shown in more detail. Slit floodlight 59
are straight filament lamps 84, slits 85. It is composed of a cylindrical lens 86. The image detector 78 includes an imaging lens 87. It is composed of a TV camera 45. 88 in the diagram
shows a two-dimensional array sensor chip in a solid-state 'rv camera.

The slit projector 69 uses a linear source emitted from the slit 8♂ of the source generated by the lamp 84 to illuminate the front side as a parallel beam using a cylindrical lens 86.

38 shown in FIG. 26 indicates the surface of the emitted slit. The image detector 78 is tilted and fixed by the holding part 20 so that its original axis intersects the slit optical surface 38a obliquely (at an angle α). The trapezoidal surface in FIG. 23 s8h indicates the field of view detected by the image detector 78 in the slit light surface 58 (Z). This bright ridge image is detected by an image detector. Fig. 30 shows an example of robot work in which the position and orientation of a connector part 90 with a wire is detected, the connector part 90 with a wire is grasped, and the This shows the working situation of inserting the pin 92. Fig. 31 is a detection image of the image detector 78 shown in Fig. 23, which is detected brightly as a slit bright line image. 6
9 and the image detector 780.As is clear from the relationship between the number of songs, the top of the screen is far away and the bottom is close. Optical cutting line extraction circuit 4
In step 2, a light cutting line is extracted using the slit bright line image in FIG. 31 as the distance from the screen to the slit bright line. 32nd
The figure shows an example of the process. (For details, see Japanese Patent Application Publication No. 56-704.
It is disclosed in No. 07. ) FIG. 32 (α) is an input image input from the image detector 78 to the optical cutting line extraction circuit 42. The video signal of, for example, one scanning line xi in the vertical direction obtained from the image detector 78 is as shown in FIG. 32<b>. From this, the threshold value ~ and the video signal are compared, and the center position Zi of the intersection of is determined. If Zi is determined and outputted for each layer, an optical cutting line (waveform data) can be obtained as shown in FIG. 32(C). Note that the optical cutting extraction may be performed by detecting the peak position, in addition to centering by the threshold processing shown here. Further, in this example, the far side of the Z-axis coordinate is set as the origin, but the near side may be set as the origin. It is also easily possible to perform coordinate transformation in the Z-axis direction according to the perspective law. Further, although this processing can be performed at high speed using an electric circuit, it may also be entirely performed by software processing.

By performing the above processing while moving the optical cutting head using the motor 80, a distance image can be obtained. An example is shown in Figure 33. FIG. 33 is an example in which the optical cutting lines are thinned out and displayed. That is, when viewed as an xy plane image, the brightness Z corresponds to the distance S from the C cutting head 3. The brighter the image, the closer the image is.
=S.

sinα, 7. = 51tinα. ) Movement of the head, that is, feeding in the y-axis direction, is performed by a constant speed motor, and the optical cutting line extraction circuit 42 samples the optical cutting line at regular time intervals using a sampling signal 92 obtained from the image processor 21. While being fed by the pulse motor 80, the optical cutting line is sampled at constant pulse intervals (at every interval of movement in the Z direction by a constant distance P). Any method may be used, such as combining a DC motor 80 and an encoder 82 and sampling every time a certain amount of loaf is moved. The number of optical cutting lines to be detected may be selected from two or more depending on the work target of the robot vision. The same goes for pitch. These are controlled by the image processor 21. The processing content of the distance image input to the image processor 21 may also be selected depending on the work target of the robot vision. As an example, regarding the work object shown in FIG. 30, the obtained distance image is that shown in FIG. ) is obtained from the optical cutting line extraction circuit 42 and sent to the image memory I through the parallel interface 50h.
(This is stored in the AM 51α. The microprocessor 48 of the image processor 21, for example, as shown in FIG.
and distance 1i −1 no 1 for xi+ 1 (for 5×5 pixels), Z! +1. j-1, Zi-1, J
+1. Zi+1. Read j+1, z= ,; =l Z' 1 * 7' I Z vine +
1 #, 7' + ' l + Ii: i41. j-1
-I7i-y, -J-Yu are differentiated, and when this value is larger than a certain large reference value, there is a jump edge (boundary line between the object and the background), so the above values of Zt and Store in another area. Furthermore, the microprocessor 48 performs segmentation of the closed area surrounded by solid lines (jump edges) Ej as shown in FIG. Then, the area and height of the center of gravity are determined for each closed region, and a target object that is within the specified range and has the highest height of the center of gravity is separated and extracted.

Furthermore, for example, the roof edge Er shown by a dotted line in FIG.
A plane area surrounded by the jump edge E and the jump edge E is extracted, the separated and extracted plane area is approximated as a plane, and its normal direction is recognized as the three-dimensional posture of the part. That is, the plane equation that can be used to approximate the area of the main plane surrounded by the jump edge and the roof edge using the least squares method is t, x-4-t, y+t, = z...(6)
and xy of m points in the principal plane extracted from the distance image
Let the z coordinate be (arl, arl, hi) i = 1.
2.・=, m, then 'l t' of the expression fil! ＋'l
is calculated as i = (ATA)-' AT, 6 (7), where the matrix t has rows of (su)ノ:=1.2.5. Here A is (α71.αr2, 1.0)
b is a matrix whose rows are (bi), T is a transposed matrix, and -1 is an inverse matrix.

Note that the position of the center of gravity of the main plane area has already been determined during segmentation. The normal direction of this principal plane is expressed by the angle ψ between the normal and the Z axis, and the angle θ between the projection of the normal on the x, y plane and the X axis.

ψ and θ are t (formula (6))% formula% (
9) Furthermore, as shown in Figure 35, by rotating the parameters of the principal plane according to the values of ψ and θ and converting them into a'', y', and z', the shape of the principal plane as seen from the normal direction can be recognized. be able to.

In this manner, the microprocessor 48 can determine the location and spatial orientation of the major surfaces of the object of interest relative to the robot's hand mechanism. The size of that surface as mentioned above, whether this is the surface of the connector,
It can be verified from the shape. Furthermore, by detecting the direction in which the wire A2 is located nearby, it can be determined that the opposite side is the direction in which the pin should be inserted. The remaining two sides are the surfaces that the robot fingers should grip. Note that [)M49 of Image Processor +/'21 includes jump edge detection and segmentation, extraction of a plane area of a part by separating and extracting a target part and roof edge detection, and three-dimensional analysis of a part by the normal direction of the plane. Programs such as target posture recognition and parts recognition are stored. It/Vv15
1h is for temporarily storing data calculated by the calculation unit 55. The 9th line is the pass line.

and the interface 52 of the image processor 21
From there, data on the position, orientation, and direction of the main surface of the object are sent to the robot control device 18.

That is, since the grip point of the robot's finger 25, the position to approach, and its direction have been determined, the microprocessor 29 of the robot leg restraint device 18 adds these to the four-pot arm control data stored in the decoy 33. It is converted into the control coordinate system of the robot and sent to the drive circuit 3 via the D/A converter 30.
1. Each actuator M provided in the robot 17 is driven, and the fingers of the robot 17 grasp the connector 90 based on the above information and insert it into the pin 92 existing at the position given by the teaching in advance.

Note that in the embodiment of the present invention, a steady slit light is used as the light cutting method, but this may be replaced by strobe light emission. Further, in the embodiment of the present invention, a slit light is used, but a bright and dark straight edge, bright on one side and dark on the other side, may also be used. Alternatively, instead of the slit light pressure, a method may be adopted in which the spot light is scanned over the light cutting surface 25.

Further, in order to obtain a distance image from the optical cutting detection head 77 attached to the hand mechanism 24 of the robot 170, the detection head scanning mechanism provided at the end of the hand mechanism 24 scans linearly as in the embodiment shown in FIG. It is clear that in addition to the robot control device, the hand mechanism 24 of the robot can be moved at a constant speed in the y-axis direction, and the coordinate values can be sent to the image processor 21 and sampled at regular intervals P. It is. However, the robot's hand mechanism 24
The embodiment in which scanning is performed by moving in parallel has poorer responsiveness than the embodiment shown in FIG.

In addition, in order to obtain a distance image from the optical cutting detection head, a detection head scanning mechanism is essential, but this is explained in the second embodiment.
67 of the linear movement mechanism shown in FIG. As illustrated in the figure, the slit floodlight 3
A mechanism for swinging only 9 may be used.

In this case, the distance image is not proportional to each rotation angle and becomes a complex relational expression including trigonometric functions, so determining the inclination direction and position of the plane where the target object is located becomes extremely complicated. However, when it is not necessary to accurately determine the inclination, direction, and position of the plane on which the object is located, the above relational expression can be replaced with an approximate expression, and a swinging mechanism can be used as the scanning mechanism.

However, as a scanning mechanism attached to the hand mechanism 24, a swinging mechanism can be simpler as a mechanism than a linear scanning mechanism.

Furthermore, as described above, it is of course possible to use a method other than the light cutting method as a distance image detector.

FIG. 24 shows an embodiment of a detector using an oblique cross slit beam 46 as an image detector attached to a four-bot arm. In this embodiment, two slit light sources 69α and 39h are used.
are arranged so that the diagonal cross-shaped slit light 46 can be irradiated with them, and the TV left camera 5 is used to detect the light cutting waveform when the two slit light sources 39α39jS are emitted one by one.

゛A waveform signal is extracted from the Tv image signal using the optical cutting line extraction device 42, and the actual distance to the target object is determined using data 27 indicating the correspondence between predetermined waveform signal values and actual coordinates. , detect posture, etc. The details of this embodiment are described in Japanese Patent Application No. 57-201931.

An example of detecting the position and posture of an object using the robot vision device of the present invention will be described below.

Here, in order to make the explanation easier to understand, the device configuration shown in FIG. 69 will be explained as an example.

Incidentally, the following explanation can be applied in exactly the same manner to the apparatus configuration shown in FIG. 40.

In FIG. 59, two slit light sources 101α.

41 (α) (b)
), it forms a diagonal grid pattern. This correspondence relationship is stored in advance in the memory device using an object whose dimensions are known while changing its position from the detector. When detecting the position and orientation of the object, the actual position and inclination of the object are determined by comparing this pre-stored correspondence with the position on the image of the light cutting line detected by the imaging device 102. can be used. In the present invention, two slit lights 103α and i ash are switched and projected,
Since the distance and inclination of the object in two non-parallel planes can be determined, the three-dimensional position and orientation of the object can be uniquely determined.

In addition, in the present invention, two slit lights 10'5cL and 10
It is assumed that the detector or the object is roughly aligned in advance so that the intersection line 106 of 3b intersects with the object surface, and the relative position of the detector and the object is determined using the robot vision device of the present invention. and posture shall be accurately detected.

Next, a method of determining the correspondence between the position of the detected optical cutting line on the image and the actual position of the object will be explained using FIG. 42. Although only the case of the slit light source 101a shown in FIG. 39 will be described here, the same applies to the case of the slit light source 101'b shown in FIG. 39 and the slit light sources 101α and 101b shown in FIG. 42nd
As shown in the figure, the intersection line 105 of the two slit lights coincides with the X axis, and the slit light 10 from the slit light source 101a
Assume a coordinate system xyz such that 3α coincides with the xz plane. First, as shown in FIG. 42, a plane 106 that is perpendicular to the two axes and wider than the field of view of the imaging device 102 is erected on a scaled rail 107 so that it can be moved in parallel in the X-axis direction. By moving the plane 106 away from the detector in increments of, for example, 1 CrIL, the position of the light cutting line on the detected image is determined.
An equidistant diagram as shown in FIG. 43 is obtained. Next, the busy plane 106 is a rectangle with a constant width, and such a rectangle is prepared in several widths, and is also erected perpendicular to the two axes on the rail 107 and moved in the X-axis direction. Plane 1
If the width measured in the X-axis direction of 06 is set in steps of 2 cm, for example, and the locus of the end point of the light cutting line detected when each plane 106 is moved in the two-axis direction, the result will be as shown in Fig. 44. A contour plot (in terms of the X coordinate) is obtained. Fourth
In the equidistant line diagram shown in Figure 3 and the equispaced line diagram shown in Figure 44, if we look at the changes in distance and position in the x' force direction along a certain horizontal line hh' on the diagram, we find that, respectively, The result will be as shown in FIGS. 45 and 46. These are the distance function 2 in the vertical coordinate on the diagram and the function X of the position in the X direction.
If we express the coordinate number in the machine direction on the diagram as a variable, then z)'=f(i#a)'0+aj 11moaj2i2+
a) '5i'' -1-・ (L2x)'-4i#hjO
+bj 1 i 10h)'2 i"+hj5i" -f
---It can be approximated as Ql. The degree of approximation is determined by the required detection accuracy. For all the nodes on the diagram (for example) ° = 1 to 256), a
If jn, bjn (n = 0, 1, 2.-) are determined in advance using the above method, the two directions and I
You can set the direction. In addition, when calculating the inclination of a plane constituting the detection target, the actual coordinates (, y, z ) (z'
, z'), the angle between that plane and the X-axis can be calculated using θx=tan-''7:QJ.

Next, the means for extracting the optical cutting line from the detected image will be specifically explained. In this case as well, the case of the one-sided slit light 105α shown in FIG. 59 will be explained. The optical cutting line imaged by the imaging device 102 generally has a certain width due to the width of the slit light, the blur of the optical system, etc. Therefore, the horizontal scanning direction of the imaging device 102 is changed to the 43rd horizontal scanning direction.
The coordinates t of the peak position (i.e., the brightest point) of each horizontal scanning signal are set to the position of the light cutting line at each node - (See Figure 47). In this way, 2n3 +13
Extract the coordinates of the position of the optical cutting line to calculate.

Next, the process of detecting the three-dimensional position and orientation of an object by the robot vision device of the present invention is as follows: Detection of a cylindrical object 109 shown in FIG. 48 and a circular hole 110σ on a plane shown in FIG. This will be explained using an example.

First, a method for detecting the position and orientation of the cylindrical object 109 will be explained. Suppose that when the slit light 103α shown in FIG. 40 is projected, for example, a light cut image as shown by the solid line A in FIG. 50 is detected by the imaging device 102. In FIG. 51, when the position of the brightest point in the horizontal direction is detected as described above and the shape of the light section line is extracted, it becomes as shown in FIG. When the waveform shown in Fig. 51 is approximated by a broken line and the coordinates (Ls)') of the end points of each line segment are converted into actual coordinates (-, z) using equation 03H, which is determined in advance, Fig. 52 is obtained. It becomes as shown. In this figure, the approximate center position of the top surface of the cylindrical object 109 in the X direction can be found from the line segment A with the smallest two coordinates, that is, the midpoint of the line segment closest to the detector. Also, using equation α4, object 1 in the xz plane
09 tilt is seen. Next, by switching the slit light source and performing the same detection on the optically sectioned image (indicated by broken line B in FIG. 50) obtained for the other slit light 103h shown in FIG. 1
09 and the approximate center position of the upper surface in the y direction and the inclination of the object 109 in the yz plane. Furthermore, based on the above detection results, the distance ml from the detector to the center position of the top surface of the object 109
2 is, when the top surface of the cylindrical object 109 is almost perpendicular to the two axes, 2=50-♂-niji, h-±no・0ω2
h −yl 2 or 2=Δ”50”x −”s, begging. .

2 to 40 2. Here, xl s% (X is the X coordinate of the end point of the optically sectioned image of the xz plane (A shown in FIG.
This is the y-coordinate of the end point of the optically sectioned image of the z-plane (B shown in FIG. 50), and "+*"v m"sy"4 represents the two coordinates of the four end points, respectively. The above detection results show that the inclinations of the upper surface of the cylindrical object 109 in the xz plane and the yz plane are significantly different from 900. If it is determined that the error in distance elevation cannot be ignored, move the four-bot arm according to the inclination detection result so that the two axes, that is, the intersection line of the slit lights 3α and 3h, are almost perpendicular to the surface, Detect the center position and distance again.

Next, a method of detecting the center position of the circular hole 110 perpendicular to the plane shown in FIG. 49 and the direction in which the hole is placed will be described. Figure 53 shows the hole 110 shown in Figure 49.
It is a figure which shows an example of the optical cutting line obtained when imaged by an imaging device. In this case, as in the previous example, two sets of optically sectioned images in the r2 plane and the yz plane are independently detected by switching the slit light source. The solid line and broken line in FIG. 53 indicate two sets of optically sectioned images obtained in this manner. In the same figure, to the point.

By performing processing in which the line segment connecting - and the line segment connecting point "Iybt" respectively correspond to the line segments A and H in FIG. Direction detection can be performed.

Below, specific embodiments of the robot visual device of the present invention will be described. First, the overall structure and operation of the apparatus will be explained, and then the specific structure of each part of the detector will be described.

FIG. 24 is a diagram showing an embodiment of the overall configuration of the robot visual system of the present invention. The detector is composed of an imaging device 2 and slit light sources 59α (101α) and 39h (101), and is arranged and configured as follows.
1o'5b) is aligned with the rotation axis of the gripper 25 of the robot.

(2) Optical axis 108α of slit light sources 59α and 59b,
108b and the optical axis 108 of the imaging device 45 (102)
C intersects at approximately one point on the intersection line 105 of the slit lights 46α and 46b.

(3) Optical axis 108α of slit light sources 39α and 39b,
108h and the optical axis of the imaging device 45 (102) 10th C
are on the same plane.

(4) The angle between the optical axis 108α of the slit light source 39α and the line of intersection 105 of the slit light is equal to the angle between the optical axis 108h of the slit light source 39A and the line of intersection ins of the slit light.

(5) Optical axis 108α of slit light source 59a and imaging device R
45 (102), the optical axis 108b of the slit light source 397S, and the optical axis 108 of the imaging device 45.
It is equal to the angle formed by C.

A detector is configured on the robot arm 24 with the arrangement as described above. As another modification of the detector configuration method, the above (
There are 1 to 4 logically possible combinations among 1) to (5). Further, in FIG. 24, the detector is fixed to a portion of the robot that rotates the gripper 25, but it may be fixed on the gripper 25 and rotated together with the gripper 25.

The entire robot visual system is controlled by an image processor 47 shown in FIG. That is, the slit light switching device 113 is controlled and the slit light sources 39a, 39
h is made to emit light one by one, and each light section image is detected by the imaging device 45 (102). The detected image signal is input to the optical cutting line extraction circuit 42, and the optical cutting waveform is input to the image processor 47. The optical cutting line extraction circuit 42 compresses two-dimensional image data into one-dimensional waveform data according to the method described above, and may use the device disclosed in Japanese Patent Laid-Open No. 70407/1983. The two sets of optically sectioned line waveform data obtained in this way are transferred to the image processor (microphone computer) 4.
7, the light pressure is processed according to the procedure described. That is, after the waveform data is approximated by a broken line, E Ql -Qe is calculated to detect the three-dimensional position and orientation of the object, and the data 117 is output to the control device of the robot arm mechanism. Formula 02
Coefficients α nor, b nor, ()° necessary to calculate 0
=0.1,2,3..., rL-0,1,2,3...
) is determined in advance by the method described above and stored in the storage device 116, and is read out and used as needed.

Next, the operation of the robot visual system of this embodiment will be explained in detail.

Before detecting the target object, as already explained, the formula (1
The coefficients σ)'n, hnotL (and the coefficients of 3/ obtained in the same manner) shown in 1+21 are stored in the storage device 116 VC.
Some work is required to store it. Using the device shown in Fig. 4J42, the robosoton operation is performed and fixed so that the intersection line 105 of the slit light is parallel to the rail 107, and the equidistant line diagram shown in Fig. 46 and Fig. Draw the monospaced line diagram shown in the figure. This work is switched by the slit switching device 113, and the two slits and lights 46α (103tL) and 46A (
103A) respectively. Next, polynomial approximation is performed by the image processor 47 based on these data, and the coefficients α of the equation α13 in each part are stored in the storage device 116.

In the target object detection process, as already explained using FIGS. 48 and 49 as examples, the image processor 47 controls the slit light switching device 113 to cause the two sets of slit light sources 101α and 101h to emit light, respectively. A photosection waveform based on the photosection image of the case is obtained by the photosection extraction circuit 42, and after this is approximated by a broken line by the image processor 47, the actual position coordinates of each line segment are stored in the storage device 11.
Formula 03 is based on the coefficients αn and hjn stored in 6.
Ql (and similar formulas for yj).

The image processor 47 includes two sets of slit light sources 101
Among the line segments when α, lol, and 6 are emitted, the line segments on the target object are each 1 based on the distance z as a criterion.
(Usually, the line segment closest to the detector and near the center of the field of view) is selected one by one from each of the four endpoints.
-Qe and the midpoint of the line segment are calculated to detect the position and orientation of the target object and output them to the control device of the robot arm mechanism.

Next, the specific configuration of each part of the detector will be explained. By using a TV camera as the imaging device 45 and making the direction of i shown in FIG. 47 coincide with the horizontal scanning direction, the optical cutting line can be detected at high speed. Of course, a two-dimensional image detector other than a TV camera, such as a combination of a one-dimensional photodiode array and a galvanometer mirror, may also be used. Among TV cameras, those using two-dimensional solid-state image sensors in particular are characterized by earthquake resistance, impact resistance, (b) lifespan, (c) image distortion, and (d) small size. It is superior to those using electron tubes in five aspects: light weight and (e) power saving. FIG. 54 is a diagram showing a specific example of an imaging device using a two-dimensional solid-state image sensor 19.

In this specific example, the parts mounted on the robot arm are a two-dimensional solid-state image sensor 19 excluding the TV right camera control unit 122, and a front image signal amplifier 120, as shown in FIG. and a synchronous drive signal waveform shaping circuit 121, further reducing the size and weight. In FIG. 54, the front image signal amplifier 120 is mounted on the substrate 123a, and the two-dimensional solid-state image sensor 119 and the synchronous drive signal waveform shaping circuit 121 are mounted on the substrate 123a.
h, and is fixed on the back cover 125 with a support 124. A connector 126 is attached to the back cover 125, and the -off signal and power supply are performed through this connector 126. In addition, the back cover 125 and the outer frame 127 are made of a non-metal such as plastic with copper foil for electrostatic shielding applied to the inner surface (or metal plating applied). Furthermore, it is also possible to make the imaging lens 12B out of plastic. As described above, if the imaging device according to this embodiment is used, in addition to features such as earthquake resistance, impact resistance, long life, low image distortion, and power saving, the detector can be significantly reduced in size and weight.

Next, the specific configuration of the slit light sources 101α and 101/l will be explained. As a light source, halogen lamps, discharge tubes, light emitting diodes, etc. can be used, but (
Light-emitting diodes are the best in three respects: (a) earthquake resistance, impact resistance, (b) lifespan, and (c) compactness and light weight. FIG. 56 is a diagram showing a specific example of a slit light source using a light emitting diode. The light emitting diode 129 is of a high output type that emits infrared light. Since the maximum sensitivity wavelength of two-dimensional solid-state image sensors is usually around 800 nm, detection sensitivity can be maximized by using a light emitting diode of this wavelength.

As shown in FIG. 56, a plurality of light emitting diodes 129 are arranged in parallel along the longitudinal direction of the cylindrical lens 130, for example. In the plan view shown in FIG. 56(α), the light emitting diode 129 and the slit 131 are in an image forming relationship with respect to the cylindrical lens 160. Further, the position of the imaging plane 152 is set to the approximate position of the object, and the imaging plane 132 and the slit 131 are in an imaging relationship with respect to the imaging lens 133. By employing the above configuration, the light emitted from the light emitting diode 129 can be efficiently condensed to form a slit light, so that in addition to the above three characteristics, a bright slit light source can be realized. Furthermore, by using a lens-based light emitting diode 29 with strong directivity, an even more powerful slit light source can be realized.

FIG. 57 is a diagram showing another specific example of a slit light source using a light emitting diode. As shown in FIG. 57, a plurality of light emitting diodes 129 are arranged along the slit 131,
The slit 131 and the imaging surface 132 are in an imaging relationship with respect to the imaging lens 133. Also, cylindrical lens 130
In order to prevent unevenness in the slit light on the imaging plane 132,
It is inserted between the imaging lens 133 and the slit 1310.

In this specific example, if the imaging relationship of the imaging plane 132 is out of order, the slit light may become uneven, but this can be ignored in the present invention, which detects the position of the light cutting line. According to this specific example, as is clear from a comparison with FIG. 22, the length in the optical axis direction can be shortened, and in addition to the characteristics of the light emitting diode 129, the slit light source can be made significantly smaller. There is.

Also, like the imaging device, if the outer frame is made of non-metallic material such as plastic, the detector can be made significantly smaller.

It has the effect of being lightweight.

Note that in the specific examples of the slit light source shown in FIGS. 56 and 57, the slit light does not have a perfect flat shape. This problem can be avoided by making the diameter of the imaging lens 133 unnecessarily thick.

Other embodiments of the image detector 20 attached to the robot arm include a grayscale image detector using a combination of a TV camera, a linear sensor, or a point sensor (photodiode, photomulti, etc.) and a scanning mechanism such as a galvanometer mirror. In the present invention, before recognition processing is performed by the image detector on the four-pot arm, the image from the distance image detector at a fixed position is processed, and the robot arm approaches the target object from the normal direction of the main plane. Since the image can be detected from that direction by the image detector on the robot arm, it is possible to detect the correct planar shape of the main planes. , position, and posture can be recognized.

Further, when the background and the target object can be detected separately by binarizing a grayscale image, the shape, position, and orientation of the target object can be detected at high speed using binary image processing, such as template matching.

The position and orientation of the target object obtained by processing the image from the image detector on the robot arm as explained above is
Because it is based on the coordinate system fixed to the image detector, that is, the relative positional relationship with the robot arm, if the absolute positioning accuracy is poor due to deflection of the robot arm, it is difficult to make the robot arm approach the target object accurately due to pressure. can. Further, in this case, the four-pot control device needs to calculate the motion amount of each axis of the robot from the posture of the robot arm at the time of detection from the robot arm and the position and posture of the target object from the image processor.

Next, FIG. 25 shows an embodiment of an image processor 21, which is one of the other components of the visual system.

The image processor 21 processes the images detected by the distance image detector 19 and the image detector 20 on the robot arm according to the method described above, discovers the target object 11,
Its position and orientation are recognized and the information is transmitted to the pot control device 18. The image processor 21 is controlled by a microprocessor 48 and operates according to a program stored in a ROM 49. The optical cutting line extraction device 42 for generating a distance image and the image detector 20 on the robot arm are connected by a high-speed parallel I/Fsoα, 50h, and the detected image information is stored in the I(AM51α). In addition to being used to temporarily store image information, the RAM 51 is also used as a work area in the image processing process.The RAM 51 is also connected to the robot control device 18 via an interface 52 to transmit recognition results. do.

In the above-described embodiment of the image processor 21, a case has been described in which software processing is performed using the microprocessor 48, but for example, binarization processing, differentiation processing, etc. can be executed with a relatively simple hardware configuration. By partially performing processing using hardware, processing speed can be increased. Furthermore, since extremely high-speed processing is performed, it is possible to implement all processing algorithms in hardware, although this increases the scale of the device.

Further, in the embodiments according to the present invention described above, an articulated robot has been described, but it is clear that a Cartesian robot or a cylindrical robot may also be used.

Furthermore, as another modification of the present invention, depending on the work target, recognition of the target object only by the distance image detector 19,
It should be added that it is also possible to recognize the target object using only the image detector 20 on the robot arm.

〔Effect of the invention〕

As explained above, according to the present invention, since the target object is discovered and its three-dimensional position, orientation, and shape are detected by distance image processing, the color, brightness, surface roughness, and surface roughness of the target object surface are detected. Efficient recognition is always possible regardless of dirt or the like. Also, by differentiating the distance image? Since the boundary line between objects is detected, it is possible to stably separate and detect objects regardless of the pressure on the surface of the object. Furthermore, by integrating the above effects, the target object to be worked on can be identified from complexly folded parts or the background, and the third
It is possible to realize complete visual feedback by recognizing dimensional position and posture and controlling the robot. In addition, the hierarchical processing combined with the image detector on the robot arm according to the present invention makes it possible to accurately position the target object even when the absolute positioning accuracy of the robot is poor. assembly, inspection,
It becomes possible to apply it to a wide range of fields such as adjustment.

[Brief explanation of drawings]

Figures 1 and 2 are diagrams showing conventional robot vision, Figure 3 is a diagram explaining distance images, and Figures 4, 5, 6, 7, and 8 are diagrams of the present invention. Figure 9 shows the process of processing a distance image by A diagram showing the definition of the posture of an object by distance image processing, FIG. 13 is a diagram showing the overall configuration of an embodiment according to the present invention, FIG. 14 is a diagram explaining the robot control device, and FIG. 15 is a diagram showing the distance Diagrams showing one embodiment of the image detector, FIG. 16, FIG. 17, FIG. 18,
Figures 19 and 20 are diagrams showing the detection process, and Figure 2
Figure 1 is an example of another range image detector, Figure 22 is a diagram explaining the cause of shadows in range images, and Figures 23 and 24 are examples of an image detector on a robot arm (showing Jll). 25 is a diagram showing an embodiment of the image processor, FIG. 26 is a schematic configuration diagram showing the optical cutting line extraction circuit according to the present invention, and FIG. 27 explains the function of the circuit shown in FIG. 26. 28 is a block diagram specifically showing the circuit shown in FIG. 27, and FIG. 29 is a diagram specifically showing an embodiment of the robot visual device according to the present invention shown in FIG. 23. 30 is a diagram for explaining the work target in the embodiment shown in FIGS. 23 and 29, FIG. 31 is a diagram showing an example of a light cutting line, and FIG. 32 is a diagram explaining the light cutting extraction process. 33 is a diagram showing an example of a distance image, and FIG. 34 is a diagram showing the relationship between the light cutting line, the center of the imaging device, and the surface of the object.
FIG. 35 is a diagram showing jump edges and roof edges in a distance image, and FIG. 36 is a diagram showing a state in which distances for 6×3 picture elements are cut out to detect jump edges by differentiating the distance image signal. , FIG. 37 is a diagram showing another embodiment different from FIG. 29 according to the present invention, FIG. 68 is a diagram showing another embodiment according to the present invention, and FIG. 39 is a diagram showing another embodiment according to the present invention. As shown in FIG.
Fig. 41 is a perspective view showing the basic configuration of the detector shown in Fig. 41. Fig. 41 is a diagram showing equidistant lines and equispaced lines by the detector shown in Fig. 39. Fig. 42 is a diagram showing equidistant lines and equispaced lines by the detector shown in Fig. 39. FIG. 43 is a diagram showing an example of equidistant lines, FIG. 44 is a diagram showing an example of equidistant lines, and FIG. 45 is a diagram showing changes in the equidistant lines shown in FIG. 45. Figure 46 is a diagram showing changes in the monospaced lines shown in Figure 44, Figure 47 is a diagram showing a method for extracting light cutting lines, and Figures 48 and 49 are examples of objects to be detected. Figures, ste:l, Figures 51 and 52 are in Figure 4.
Figure 5 shows the process of detecting the detection target object shown in Figure 8.
Figure 3 shows the detection target object diagram (α) shown in Figure 49 and Figure 57 (
α) is a plan view showing a specific example of a slit light source used in the detector of the robot vision device according to the present invention, and FIGS. 56(b) and 59(h) are used in the detector of the robot vision device of the present invention. FIG. 3 is a front view showing a specific example of a slit light source. 9...Distance image 11...Target object 17...Robot arm 18...Robot control device 19...Distance image detector 20...Image detector 21 on robot arm...Image processor 38...Slit light 39...Slit light source 42.
... Optical cutting line extraction device 45...'rV camera Figure 1 Figure 32 Kaya 4 Zuken, 5 times ＾ Otsu m Bu 7 Zuko 8 Zuko ｑ Zuko / θmadai // Dan Kaya 72 ffi Perforation/Otsu Figure 11 Station' D Figure 78 Chi 30th area 31st closed No. 330 No. 34 factor ＼ Kaya 35 Figure 36m Kaya 37 Figure 32 sub-No. 4o U I4/Main (Rin χzq surface (b) ) "JZ" F side grass 422 rod 430 grass 440 grass 4.Sr grass 46 illustration grass 3/ II grass,!; 2 grass 63 difficult grass 540 grass, 55 illustration

Claims (1)

[Claims] 1. A detector that detects a distance image of the entire work object from a fixed point relative to the robot base, and a distance image detected by the distance image detector according to a predetermined procedure. an image processor that detects a target object and three-dimensionally recognizes its shape, position, and orientation; a four-bot control device that causes a robot arm to approach the target object from a predetermined direction according to the recognition results from the image processor;
An image detector is installed on the robot arm and detects an image of the target object from a close position, and an image processor processes the detected image according to a set procedure to recognize the shape, position, and orientation of the target object. , a robot control device is provided that corrects the four robot arms to the correct position relative to the target object according to the recognition results of the image processor,
A robot that performs tasks such as assembly, inspection, and adjustment. 2. The image processor includes a differentiating means for differentiating a distance hat image signal to form a jump edge image signal, a separating means for separating into a region closed by this jump edge, and a necessary region from this separated region. extract,
A patent claim characterized in that it is constituted by an extraction means for extracting a distance image of a main plane from this extracted area, and a detection means for detecting the position and inclination of the center of gravity of this main plane from this distance image. The robot in the first term of the range.