JP2019063954A - Robot system, calibration method and calibration program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a robot system which reduces labor and time required for calibration. SOLUTION: A robot system 1 includes an image pickup device 12 for imaging an object, a robot 10 including a laser sensor 13 for measuring a distance, an arm 15, and a control device 20, and the control device is an image pickup device. The distance from the image pickup device to the first object when the focus is on the first object 50 is based on the storage unit 22 stored as a reference distance and the first image including the second object 40. An operation control unit that arranges the arm so that the laser sensor is located in a region where the distance to the second object can be measured, and moves the arm so that the distance from the image pickup device to the second object matches the reference distance. The 210, the detection unit 213 that detects a plurality of feature points of the second object based on the distance information, the coordinates of the plurality of feature points in the coordinate system of the arm, and the second image captured by the moved imaging device. It is provided with a calibration unit 214 that associates with the coordinates of. [Selection diagram] Fig. 1

Description

  The present invention relates to a robot system, a calibration method and a calibration program.

  Conventionally, a so-called on-hand robot system has been developed in which an imaging device is attached to the tip of a robot manipulator, and the operation of the manipulator is controlled based on information obtained from the imaging device. In such a robot system, for example, when the robot performs an operation such as gripping of an object, the imaging device captures an image of a working area of the robot, detects the object from the obtained image, and detects the detected object. By calculating the position, the operation such as gripping can be controlled with high accuracy.

  In such a robot system, in order to reflect information obtained from the imaging device in the operation of the robot, pixel coordinates on the image acquired by the imaging device and space coordinates on the work space in which the robot works are previously It needs to be associated. The process of associating different coordinates in this manner is called calibration.

  As a method of calibration, for example, in Patent Document 1 below, an image obtained by imaging a marker a plurality of times with an imaging device attached to an arm while disposing an arm of a robot at a plurality of positions in a work space Discloses a method of correlating the pixel coordinates of the marker contained in and the space coordinates of the robot at the time of imaging.

JP, 2016-120567, A

  Even if calibration is completed once by the above-mentioned method, for example, if the relative positional relationship between the imaging device and the object changes by changing the type of the object, the focus of the lens included in the imaging device is targeted It becomes impossible to match the object, and it becomes impossible to image the object with high accuracy. Therefore, every time the operator changes the object, it is necessary to adjust the focus of the lens provided in the imaging device and re-calibrate the calibration, which requires labor and time for the operator.

  Then, this invention aims at providing the robot system which can reduce the effort and time which calibration requires, the calibration method, and a calibration program.

  A robot system according to an aspect of the present invention includes a robot and a control device that controls operations of the robot, and the robot is configured to capture an image of an object placed on the mounting surface and a distance to the object The control device includes a laser sensor to be measured, and an arm attached with an imaging device and a laser sensor, and the control device is a distance from the imaging device to the first object when the focus of the imaging device matches the first object. The laser sensor is located in an area where the distance to the second object can be measured based on the storage unit stored as the reference distance and the first image including the second object present in the field of view of the imaging device And an operation control unit that moves the arm in a direction intersecting the mounting surface such that the arm is disposed and the distance from the imaging device to the second object matches the reference distance, and the distance information measured by the laser sensor Calibration that associates the detection unit that detects a plurality of feature points of the second object, the coordinates of the plurality of feature points in the coordinate system of the arm, and the coordinates of the second image captured by the moved imaging device based on And a unit.

  According to this aspect, the distance from the imaging device to the first object when the focus of the imaging device matches the first object is stored as the reference distance in the storage unit. Thereby, even when the relative positional relationship between the imaging device and the object changes due to a change in the type of the object, the distance from the imaging device to the second object is determined using the laser sensor. The imaging device can be automatically moved to match the reference distance. Therefore, it is not necessary for the operator to refocus the imaging device, and the time and time required for calibration can be reduced.

  In the above aspect, the plurality of feature points may be present on the contour of the second object.

  According to this aspect, the feature point of the second object can be detected using the laser sensor. Therefore, the detection accuracy of the feature point is improved as compared with the case where the image obtained from the imaging device is subjected to image processing to detect the feature point, and as a result, the accuracy of the calibration can be improved. Moreover, compared with the case where image processing is performed, the time required for the detection of a feature point can be reduced.

  In the above aspect, the imaging device and the laser sensor may be separate.

  According to this aspect, since the distance is measured by the laser sensor, the accuracy of distance measurement is higher than, for example, a configuration in which the distance is measured by the imaging device having a distance measuring function. Therefore, the focusing of the imaging device and the detection of the feature point of the second object can be performed with high accuracy.

  According to the present invention, it is possible to provide a robot system, a calibration method and a calibration program capable of reducing the time and effort required for calibration.

It is a figure showing an example of composition of a robot system concerning one embodiment of the present invention. It is a flowchart which shows the procedure of the calibration in the robot system which concerns on one Embodiment of this invention. It is an explanatory view for explaining a procedure of calibration in a robot system concerning one embodiment of the present invention. It is an explanatory view for explaining a procedure of calibration in a robot system concerning one embodiment of the present invention. It is an explanatory view for explaining a procedure of calibration in a robot system concerning one embodiment of the present invention. It is an explanatory view for explaining a procedure of calibration in a robot system concerning one embodiment of the present invention. It is an explanatory view for explaining a procedure of calibration in a robot system concerning one embodiment of the present invention.

  Embodiments of the present invention will be described with reference to the accompanying drawings. In addition, what attached the same code | symbol in each figure has the same or same structure.

  FIG. 1 is a view showing a configuration example of a robot system according to an embodiment of the present invention. The robot system 1 shown in FIG. 1 is a system in which a robot is controlled to perform a predetermined work on a work (object) placed on a workbench. Specifically, the robot system 1 includes, for example, a robot 10 that performs various tasks, and a control device 20 that controls the operation of the robot 10. As shown in FIG. 1, a workbench 30 and a work 40 placed on a placement surface 31 on the workbench 30 are disposed in a work area where the operation of the robot 10 can extend. The work platform 30 and the work 40 may or may not be included in the robot system 1.

  The robot 10 includes, for example, a manipulator 11 constituting a main body of the robot 10, and a visual sensor 12 and a laser sensor 13 attached to the manipulator 11. The robot 10 operates in a work space represented by space coordinates (XYZ coordinates in FIG. 1).

  The manipulator 11 includes a base 14 installed on a floor or the like, and an arm 15 having a plurality of joints. A servomotor or the like is attached to the plurality of joints, and the articulated motion of the arm 15 is realized by controlling the drive of the servomotor. The movable range of the arm 15 changes according to the number of joints, but in this embodiment, it is assumed to be, for example, a six-axis articulated structure. An end effector (not shown) for performing a predetermined operation is attached to the tip 16 of the arm 15 (that is, the tip of the manipulator 11). By exchanging the end effector, the robot 10 can handle various tasks. Specific examples of the end effector include, for example, a multi-fingered hand, a gripper, a welding torch, a painting gun, a screw tightening machine and the like.

  The visual sensor 12 (imaging device) is mounted on the tip 16 of the arm 15 and performs imaging while moving in synchronization with the movement of the arm 15. The visual sensor 12 is a specific example of an imaging device, and is, for example, a camera including an imaging element and a lens. In FIG. 1, the optical axis of the lens of the vision sensor 12 is disposed along the Z axis, and the field of view of the vision sensor 12 extends in the XY plane. An image captured by the visual sensor 12 is output to the control device 20 via, for example, a cable (not shown) and processed in the control device 20.

  The robot 10 is a so-called on-hand robot whose operation is controlled based on information obtained from the visual sensor 12. When the robot system 1 is used, for example, for the gripping operation of the workpiece 40, the workpiece 40 placed on the placement surface 31 of the work bench 30 is imaged by the visual sensor 12, and the workpiece 40 is detected from the obtained image. Thus, the position and posture of the workpiece 40 on the mounting surface 31 and the shape of the workpiece 40 in a plan view of the XY plane are calculated. Therefore, by providing the visual sensor 12, the robot 10 can grip, for example, the workpiece 40 with high accuracy.

  When the information obtained from the visual sensor 12 is to be reflected in the operation of the robot 10, in the control device 20, pixel coordinates on the image acquired by the visual sensor 12 and space coordinates representing a work space in which the robot works And must be associated. The process of associating different coordinate systems in this manner is called calibration. The method of calibration in this embodiment will be described in detail later.

  The laser sensor 13 is attached to the tip 16 of the arm 15 together with the visual sensor 12 and moves in synchronization with the movement of the arm 15 and the visual sensor 12. The laser sensor 13 is a specific example of a distance sensor, and includes, for example, a trigonometric measurement type or time measurement type laser sensor, and a light reception amount discriminant type laser sensor. Specifically, the laser sensor 13 has an irradiation unit and a light receiving unit (not shown), irradiates the laser from the irradiation unit to the object, and receives the laser reflected from the object by the light reception unit. Measure the distance to the object. In the present embodiment, the laser sensor 13 is used, for example, to measure the distance to the work 40 or the mounting surface 31. The value of the distance obtained by the laser sensor 13 is output and stored in the control device 20 via, for example, a cable (not shown). The specific configuration of the laser irradiated by the laser sensor 13 is not particularly limited, and may be, for example, a point-shaped laser as shown in FIG. 1 or a line-shaped laser.

  The position where the visual sensor 12 and the laser sensor 13 are attached is not limited to the tip 16 of the arm 15. For example, the visual sensor 12 and the laser sensor 13 may be attached to other positions of the arm 15 or an end effector (not shown) attached to the tip 16 of the arm 15. Further, the visual sensor 12 and the laser sensor 13 may be incorporated in the robot body as a part of the robot 10 or may be externally attached to the robot 10. Further, in the present embodiment, a configuration in which the visual sensor 12 and the laser sensor 13 are separate bodies is shown, but instead of the configuration, a device in which an imaging function and a distance measuring function are integrated (for example, measurement) An imaging device or the like having a distance measuring function may be used.

  The control device 20 is configured by, for example, a computer, and controls the operations of the manipulator 11, the visual sensor 12, and the laser sensor 13. Specifically, the control device 20 includes, for example, a control unit 21 and a storage unit 22. The control unit 21 includes a manipulator control unit 210, a vision sensor control unit 211, a laser sensor control unit 212, a detection unit 213, and a calibration unit 214.

  The manipulator control unit 210 (motion control unit) controls the drive of the servomotor of each joint of the robot 10, and operates the manipulator 11 in the work space. When the robot 10 includes an end effector, the manipulator control unit 210 may control the operation of the end effector.

  The visual sensor control unit 211 controls the imaging of the visual sensor 12 to acquire an image. The acquired image is used to drive the manipulator 11 in the manipulator control unit 210.

  The laser sensor control unit 212 causes the laser sensor 13 to emit and receive a laser, and acquires the distance from the laser sensor 13 to an object (for example, the workpiece 40 or the mounting surface 31).

  The detection unit 213 detects a feature point (for example, an edge or the like) of the workpiece 40 based on the distance information acquired by the laser sensor control unit 212.

  The calibration unit 214 associates the coordinates of the feature point of the workpiece 40 in the space coordinate system of the robot 10 with the pixel coordinates of the image captured by the vision sensor 12.

  The storage unit 22 stores, as a reference distance, the distance from the visual sensor 12 to the marker when, for example, the focus of the visual sensor 12 matches the marker described later.

  Each function included in the control unit 21 is realized, for example, by the processor executing a predetermined program stored in the storage unit 22. In addition, the function of the control apparatus 20 is not limited to this, Arbitrary functions may be added suitably as needed. Although FIG. 1 shows a configuration in which each function included in the control unit 21 is realized by one control device 20, each function may be dispersed and realized in a plurality of devices.

  The work 40 (second object) is mounted on the mounting surface 31 of the work table 30. The shape of the work 40 is not particularly limited, but in the present embodiment, the work 40 forms a rectangular parallelepiped having a plurality of faces. The workpiece 40 has an upper surface 41 opposite to the mounting surface 31 and has an outline (edge) around the upper surface 41.

  Next, a method of performing calibration of the manipulator 11 and the visual sensor 12 in the above-described robot system 1 will be described with reference to FIGS. 2 and 3A to 3E. Here, FIG. 2 is a flowchart showing the procedure of calibration in the robot system according to one embodiment of the present invention, and FIGS. 3A to 3E are the procedure of calibration in the robot system according to one embodiment of the present invention It is explanatory drawing for demonstrating. FIG. 3E shows the visual field range screen of the visual sensor 12. Further, in the flowchart shown in FIG. 2, the relative positional relationship between the visual sensor 12 and the laser sensor 13 with respect to a predetermined position (for example, the tip 16) of the manipulator 11 is stored in the storage unit 22 of the control device 20 in advance. Start in the state.

  First, in step S <b> 10, the visual sensor control unit 211 acquires an image including the marker 50 (first object) placed on the placement surface 31 using the visual sensor 12. Based on the image, the vision sensor 12 is focused on the marker 50 (see FIG. 3A). This focusing is performed, for example, by adjusting the focus of the lens of the visual sensor 12 while the operator looks at a display (not shown) on which a captured image of the visual sensor 12 is displayed. The markers 50 include, for example, those on which a pattern, characters, etc. suitable for focusing the visual sensor 12 are printed. Focusing may be performed by moving the arm 15 in the Z-axis direction and adjusting the distance from the visual sensor 12 to the marker 50 instead of adjusting the focus. Also, the marker 50 may be printed directly on the mounting surface 31 instead of being mounted on the mounting surface 31, or may be directly mounted on the floor instead of the work table 30.

  Next, in step S20, when the focus of the visual sensor 12 is aligned with the marker 50, the laser sensor control unit 212 measures the distance to the marker 50 using the laser sensor 13 (see FIG. 3A). The positional relationship between the laser sensor 13 and the visual sensor 12 is stored in the storage unit 22. Therefore, based on the distance from the laser sensor 13 to the marker 50, the distance from the visual sensor 12 (more specifically, for example, the tip of the lens of the visual sensor 12) to the marker 50 is calculated. The calculated distance is stored in the storage unit 22 as a reference distance WD. The reference distance WD corresponds to the work distance of the vision sensor 12.

  Next, in step S30, when the work 40 is present in the visual field of the visual sensor 12, the visual sensor control unit 211 acquires an image (first image) including the work 40 using the visual sensor 12. . The manipulator control unit 210 detects the approximate position of the workpiece 40 on the mounting surface 31 based on the acquired image, and the arm 15 is XY so that the laser sensor 13 is positioned in the region on the Z axis of the workpiece 40. Move in the planar direction (see FIG. 3B). The area on the Z axis of the work 40 is an area where the distance to the work 40 can be measured by the laser sensor 13. The visual field range of the visual sensor 12 is wider than the measurement range of the laser sensor 13 (ie, one point where the laser is irradiated). Therefore, by detecting the approximate position of the workpiece 40 using the visual sensor 12, the time required to detect the workpiece 40 can be shortened as compared to the case where the position of the workpiece 40 is scanned using only the laser sensor 13. Can.

  In step S30, the focus of the visual sensor 12 does not necessarily have to match the workpiece 40, as long as the approximate position of the workpiece 40 can be determined from the acquired image. This determination may be performed by, for example, template matching or the like. The program for moving the arm 15 in accordance with the position of the work 40 in the image may be stored in advance in the storage unit 22 or the like. In addition, when the laser sensor 13 is positioned in advance in the area on the Z axis of the workpiece 40 at the start of step S30, the arm 15 may not be moved.

  Next, in step S40, the laser sensor control unit 212 measures the distance h1 to the upper surface 41 of the work 40 using the laser sensor 13. Then, the manipulator control unit 210 intersects the arm 15 with the mounting surface such that the distance from the visual sensor 12 to the workpiece 40 calculated based on the measured distance h1 matches the reference distance WD (see FIG. In 3C, it is moved in the Z-axis direction (see FIG. 3C). Specifically, if the distance h1 to the upper surface 41 of the work 40 is longer than the reference distance WD, the arm is moved in the negative Z-axis direction by the difference (= h1-WD), and the distance h1 is shorter than the reference distance WD For example, the arm is moved in the Z-axis positive direction by the difference (= WD−h1). As a result, the distance from the visual sensor 12 to the workpiece 40 coincides with the working distance of the visual sensor 12 and the upper surface 41 of the workpiece 40 can be automatically focused. Although FIG. 3C shows an example in which the upper surface 41 of the work 40 is focused, the position in the Z-axis direction at which the work 40 is focused is not necessarily limited to the upper surface 41. For example, in the case where the upper surface of the work is uneven or the upper surface is inclined, the focal point is focused on any position between the upper surface and the lower surface of the work (ie, the surface in contact with the mounting surface 31) It is also good.

  Next, in step S50, the detection unit 213 detects a plurality of feature points of the workpiece 40 based on the distance information measured by the laser sensor 13 (see FIG. 3D). Specifically, while moving the arm 15 along the XY plane, the manipulator control unit 210 and the laser sensor control unit 212 cause the laser sensor 13 to irradiate the laser toward the workpiece 40 to move the upper surface 41 of the workpiece 40. The peripheral area of the workpiece 40 including is scanned. Then, based on the distance information that the distance from the laser sensor 13 to the work 40 or the mounting surface 31 has changed rapidly (for example, changed by a predetermined threshold or more), the rapidly changing position is an edge point of the work 40 It is regarded as E. By repeating this scan in the XY plane, a plurality of different edge points present on the contour of the workpiece 40 are detected. The number and position of edge points to be detected are not particularly limited, but may be middle points (edge points E1 to E4) on four sides surrounding the upper surface 41 of the work 40, as shown in FIG. 3E, for example. The position coordinates of the arm 15 (for example, the XYZ coordinates of the tip 16 of the arm 15) when the plurality of edge points are detected are stored in the storage unit 22.

Finally, in step S60, the calibration unit 214 determines the edge points E1 to E4 in the XYZ coordinate system based on the position coordinates of the arm 15 when the plurality of edge points E1 to E4 obtained in step S50 are detected. At least XY coordinates (X ₁ , Y ₁ ), (X ₂ , Y ₂ ), (X ₃ , Y ₃ ), and (X ₄ , Y ₄ ) are calculated (see FIG. 3E). Further, based on the position coordinates of E1 to E4, the straight line L1 connecting the edge points E1 and E2 and the straight line L2 connecting the edge points E3 and E4 are calculated, and the position coordinate (X of the intersection E5 of the straight line L1 and the straight line L2 ₅ and Y ₅ ) are calculated (see FIG. 3E). Further, the visual sensor control unit 211 acquires an image (second image) including the workpiece 40 at a predetermined position using the visual sensor 12. Then, the calibration unit 214 registers position coordinates (i.e., space coordinates) of the plurality of edge points E1 to E4 and the intersection point E5 in the view range screen 60 of the vision sensor 12, and associates the space coordinates with pixel coordinates in the image. This completes the calibration.

  Although the number of feature points to be detected is not particularly limited, the accuracy of calibration is improved as the number of feature points is larger. Moreover, in the above-mentioned embodiment, although the edge of a workpiece | work is used as a feature point of a workpiece | work, a feature point is not restricted to an edge.

  By the above-described procedure, the robot system 1 has the following effects. That is, in the robot system 1, the work distance of the visual sensor 12 is stored in the storage unit 22 as the reference distance WD. Thereby, even if the relative positional relationship between the visual sensor 12 and the work 40 changes due to the change of the type of the work, the change of the height of the mounting surface, etc., the laser sensor 13 is used. The visual sensor 12 can be automatically moved so that the distance from the visual sensor 12 to the workpiece 40 matches the work distance. Therefore, the operator does not have to refocus or calibrate the visual sensor 12. Therefore, according to the present embodiment, it is possible to reduce the labor and time required for the calibration.

  Further, for example, according to the method disclosed in Patent Document 1, it is necessary to use an image processing technique such as contour detection or gray scale detection when detecting a feature point of a workpiece. However, when using a robot system in a severe environment such as a welding site, it is not appropriate to use a high-performance visual sensor, and the accuracy of the obtained image may be insufficient. In addition, even if a high-performance visual sensor is used, the brightness of the obtained image also fluctuates at the site where the external environment such as the brightness largely fluctuates due to the welding operation, for example, and application of the image processing may be difficult. In this case, the detection of the work on the image is not stably performed, and as a result, the accuracy of the calibration may be reduced. In this regard, according to the robot system 1, when detecting the feature points of the workpiece 40, the distance information obtained by the laser sensor 13 is used instead of the image processing. Thereby, even at a site where the external environment greatly changes, the feature point of the workpiece 40 can be detected with high accuracy. Therefore, the accuracy of calibration can be improved without using a high-performance visual sensor as compared to the method using image processing. In addition, the time required to detect a feature point can be reduced as compared to a method using image processing.

  In the present embodiment, the visual sensor 12 and the laser sensor 13 are separately provided, but instead of the configuration, for example, the distance may be measured by an imaging device having a distance measuring function. Good. In addition, the ranging function provided in the imaging device is generally easily affected by disturbance, and the accuracy of ranging is inferior to that of the laser sensor. In particular, when the robot system is used at a welding site or the like as described above, the accuracy of the distance measurement may be insufficient due to the fluctuation of the external environment. In this respect, in the present embodiment, since the laser sensor 13 is provided separately from the visual sensor 12, the distance can be increased with high accuracy as compared with the configuration in which the imaging device also serving as the distance measuring function as described above is used. Can be measured. Therefore, the focusing of the visual sensor 12 and the detection of the feature point of the work 40 can be performed with high accuracy.

  Furthermore, in the robot system 1, the visual sensor 12 having a detection accuracy lower than that of the laser sensor 13 but having a wider field of view and the laser sensor 13 having a smaller measurement range but higher measurement accuracy than the visual sensor 12 are used in combination. Thus, the position of the workpiece 40 can be detected at high speed as compared with the method using only the laser sensor 13, and the feature point of the workpiece 40 can be detected with high accuracy as compared to the method using only the visual sensor 12. it can. Therefore, calibration can be performed at high speed and with high accuracy.

  The embodiments described above are for the purpose of facilitating the understanding of the present invention, and are not for the purpose of limiting the present invention. The elements included in the embodiment and the arrangement, the material, the conditions, the shape, the size, and the like of the elements are not limited to those illustrated, and can be changed as appropriate. In addition, configurations shown in different embodiments can be partially substituted or combined with each other.

  For example, in the above-described embodiment, the case where the target (marker 50) used when focusing the visual sensor is different from the target (work 40) used when detecting the feature point is shown as an example. However, the object may be the same, and for example, the workpiece 40 may be used to focus the visual sensor.

  Moreover, in the above-mentioned embodiment, although demonstrated in order of step S10-S60, these steps do not necessarily need to be this order. For example, after the feature point of the work 40 is detected by the laser sensor 13 in step S50, the distance from the visual sensor 12 to the work 40 may be matched with the work distance in step S40.

DESCRIPTION OF SYMBOLS 1 ... Robot system, 10 ... Robot, 11 ... Manipulator, 12 ... Vision sensor, 13 ... Laser sensor, 14 ... Base, 15 ... Arm, 16 ... Tip, 20 ... Control device, 21 ... Control part, 22 ... Storage part, Reference numeral 210: manipulator control unit 211: visual sensor control unit 212: laser sensor control unit 213: detection unit 214: calibration unit 30: work table 31: mounting surface 40: work 41: upper surface 50 ... Marker, 60 ... Field of view screen

Claims (3)

A robot system comprising a robot and a control device for controlling the operation of the robot, the robot system comprising:
The robot is
An imaging device for imaging an object placed on the placement surface;
A laser sensor that measures the distance to an object;
An arm attached with the imaging device and the laser sensor;
Equipped with
The controller is
A storage unit in which a distance from the imaging device to the first object when the focus of the imaging device matches the first object is stored as a reference distance;
Disposing the arm such that the laser sensor is positioned in an area where the distance to the second object can be measured based on the first image including the second object present in the field of view of the imaging device; An operation control unit that moves the arm in a direction intersecting the mounting surface such that a distance from an imaging device to the second object matches the reference distance;
A detection unit that detects a plurality of feature points of the second object based on distance information measured by the laser sensor;
A calibration unit that associates coordinates of the plurality of feature points in a coordinate system of the arm with coordinates of a second image captured by the moved imaging device;
A robot system equipped with The plurality of feature points are present on the contour of the second object,
The robot system according to claim 1. The imaging device and the laser sensor are separate.
A robot system according to claim 1 or 2.

Images