CN104552292A - Control system of robot, robot, program and control method of robot - Google Patents

Abstract

Provided is a robot control system. The robot control system comprises a shot image acquiring portion used for acquiring shot images and a control portion used for controlling the robot based on the shot images. The shot image acquiring portion is used for acquiring shot images of at least assembled objects among assembling objects and the assembled objects reflecting assembly operation. The control portion is used for detecting and processing of the characteristic quantity of the assembled objects based on shot images and enables the objects to move based on the characteristic quantity of the assembled objects.

Description

Robot control system, robot, program, and robot control method

technical field

The present invention relates to a robot control system, a robot, a program, a robot control method and the like.

Background technique

In recent years, many industrial robots have been introduced in production sites in order to mechanize and automate work performed by humans. However, precise calibration is a prerequisite for robot positioning, which is an obstacle to robot introduction.

Here, as one of the means for robot positioning, there is visual servoing. Existing visual servoing is a technology that performs feedback control on a robot based on the difference between a reference image (goal image, target image) and a captured image (current image). Some kind of visual servoing is useful in that calibration precision is not required, and it is attracting attention as a technique for lowering barriers to robot introduction.

As a technique related to this visual servoing, there is a prior art described in Patent Document 1, for example.

Patent Document 1: Japanese Patent Laid-Open No. 2011-143494

When the robot performs an assembly operation of assembling an assembly object to an assembly object by visual servoing, the position and posture of the assembly object change every time the assembly operation is performed. When the position and posture of the object to be assembled change, the position and posture of the object to be assembled and the assembled object in the assembled state also change.

In this case, if visual servoing is performed using the same reference image every time, accurate assembly work cannot be realized. This is because the assembled object is moved to the position and posture of the assembled object reflected in the reference image regardless of whether the position and posture of the assembled object in the assembled state are changed.

In addition, although in theory, as long as a different reference image is used every time the position of the actual object to be assembled changes, the assembly work can be performed by visual servoing using the reference image, but in this case, it is necessary to prepare a large number of Referring to the image, it is unrealistic.

Contents of the invention

One aspect of the present invention relates to a robot control system including: a captured image acquiring unit that acquires a captured image; Among the target object and the object to be assembled, at least the captured image of the target object to be assembled, the control unit performs a feature value detection process of the target object to be assembled based on the captured image, and based on the object to be assembled The characteristic quantity moves the above-mentioned assembly object.

In one aspect of the present invention, the object to be assembled is moved based on the feature amount of the object to be assembled detected from the captured image.

As a result, even when the position and orientation of the object to be assembled changes, the assembly work can be accurately performed.

In addition, in one aspect of the present invention, the control unit may be configured such that the object to be assembled and the object to be assembled are displayed on the basis of one or a plurality of captured images showing the object to be assembled and the object to be assembled. The above-mentioned feature quantity detection process of the object, and according to the feature quantity of the above-mentioned assembly object and the feature quantity of the above-mentioned assembled object, make the relative position and posture relationship between the above-mentioned assembly object and the above-mentioned assembled object become the target relative position and posture relationship The above-mentioned assembly object is moved by the method.

As a result, assembly operations and the like can be performed based on the feature quantities of the object to be assembled and the feature quantities of the object to be assembled detected from the captured image.

In addition, in one aspect of the present invention, the control unit may be configured such that the control unit uses a feature value set as a target feature value among the feature values of the object to be assembled and a feature value among the feature values of the object to be assembled. The feature quantity set as the attention feature quantity is such that the assembly object is moved so that the relative position and posture relationship becomes the target relative position and posture relationship.

Accordingly, it is possible to move the assembly object such that the relative position and posture relationship between the set assembly portion of the assembly object and the set assembly portion of the assembly object becomes the target relative position and posture relationship.

In addition, in one aspect of the present invention, the control unit may move the object to be assembled so that the feature point of interest of the object to be assembled coincides with or approaches the target feature point of the object to be assembled. .

Thereby, the assembly part of an object to be assembled can be assembled to the part to be assembled of an object to be assembled, etc. FIG.

In addition, in one aspect of the present invention, it may be configured to include a reference image storage unit that stores a reference image displaying the assembly object taking the target position and posture, the control unit based on the The first captured image of the object to be assembled and the reference image are moved to the target position and posture of the object to be assembled, and after moving the object to be assembled, based on the second captured image showing at least the object to be assembled, In the feature amount detection process of the object to be assembled, the object to be assembled is moved based on the feature amount of the object to be assembled.

Thus, when the same assembly work is repeated, the assembly object can be moved to the vicinity of the object to be assembled using the same reference image, and then the assembly work can be performed in comparison with the detailed position and posture of the actual object to be assembled. .

In addition, in one aspect of the present invention, the control unit may be configured to perform the feature value of the first object to be assembled based on the first captured image showing the first object to be assembled in the assembly operation. In the detection process, the assembly object is moved based on the feature value of the first assembly object, and after the assembly object is moved, the above-mentioned The feature amount detection process of the second object to be assembled moves the object to be assembled and the first object to be assembled based on the feature amount of the second object to be assembled.

Thus, even if the positions of the first object to be assembled and the second object to be assembled are misaligned every time an assembly operation is performed, the objects to be assembled, the first object to be assembled, and the second object to be assembled can be assembled. Assembly work, etc.

In addition, in one aspect of the present invention, the control unit may be configured to perform the above-described operation based on one or more first captured images showing the assembly object and the first object to be assembled in the assembly operation. The feature detection process of the object to be assembled and the first object to be assembled is to make the object to be assembled and the first object to be assembled The relative position and posture relationship of the object to be assembled is such that the relative position and posture relationship of the object to be assembled becomes the first target relative position and posture relationship. The above-mentioned feature value detection process, and according to the feature value of the first object to be assembled and the feature value of the second object to be assembled, the relative relationship between the first object to be assembled and the second object to be assembled The position and posture relationship is such that the second target relative position and posture relationship is such that the assembly object and the first assembly object are moved.

Thus, it is possible to make the feature point of interest of the assembly object close to the target feature point of the first object to be assembled, and the feature point of interest of the first object to be assembled to be close to the target feature point of the second object to be assembled. , for visual servoing, etc.

In addition, in one aspect of the present invention, the control unit may be configured such that one or more of the object to be assembled, the first object to be assembled, and the second object to be assembled in the above-mentioned assembly operation are reflected based on one or more taking an image, performing the feature quantity detection process of the assembly object, the first assembly object, and the second assembly object, and based on the feature quantity of the assembly object and the feature of the first assembly object amount, so that the relative position and posture relationship between the assembly object and the first object to be assembled becomes the first target relative position and posture relationship, the assembly object is moved, and according to the characteristics of the first assembly object quantity and the feature quantity of the second object to be assembled, so that the relative position and posture relationship between the first object to be assembled and the second object to be assembled becomes the second target relative position and posture relationship. The object to be assembled moves.

Thereby, simultaneous assembly work of three workpieces etc. can be performed.

In addition, in one aspect of the present invention, the control unit may be configured to perform the feature value of the second object to be assembled based on the first captured image showing the second object to be assembled during the assembly work. The detection process is to move the first object to be assembled based on the feature value of the second object to be assembled, and to perform the first object to be assembled based on the second captured image reflecting the moved first object to be assembled. The feature amount detection process of the object to be assembled moves the object to be assembled based on the feature amount of the first object to be assembled.

This makes it possible to more easily control the robot, without moving the object to be assembled and the first object to be assembled simultaneously.

In addition, in one aspect of the present invention, the control unit may be configured to control the robot by performing visual servoing based on the captured image.

In this way, feedback control and the like can be performed on the robot according to the current work situation.

Moreover, another aspect of the present invention relates to a robot including: a photographed image acquisition unit that acquires a photographed image; Among the object to be assembled and the object to be assembled, at least the photographed image of the object to be assembled, the control unit performs feature quantity detection processing of the object to be assembled based on the photographed image, and The characteristic quantity moves the above-mentioned assembly object.

Moreover, another aspect of this invention is related with the program which makes a computer function as said each part.

In addition, another aspect of the present invention relates to a robot control method including the step of acquiring a photographed image of at least the object to be assembled among the object to be assembled and the object to be assembled on which the assembly work is reflected; , a step of performing a feature amount detection process of the object to be assembled; and a step of moving the object to be assembled based on the feature amount of the object to be assembled.

According to some aspects of the present invention, it is possible to provide a robot control system, a robot, a program, a robot control method, and the like that can accurately perform assembly work even when the position and orientation of an object to be assembled changes.

In addition, another aspect is a robot control device, which is characterized in that it includes: a first control unit that moves an end point of an arm of a robot to a target position along a path formed based on one or more set guidance positions; , generating an instruction value; an image acquisition unit that acquires an image including the endpoint when the endpoint is at the target position, that is, a target image, and an image that includes the endpoint when the endpoint is at the current position, that is, a current image; the second control a unit that generates a command value in such a manner that the endpoint moves from the current position to the target position based on the current image and the target image; and a drive control unit that uses the command value generated by the first control unit and the The command value generated by the second control unit moves the arm.

According to this aspect, the command value is generated so that the end point of the arm of the robot moves to the target position based on the path formed based on one or more set guide positions, and the end point moves from the current position to the target position based on the current image and the target image. A command value is generated by moving to the target position. Then, the arm is moved using these command values. Accordingly, while maintaining the high speed of position control, it is also possible to cope with changes in the target position.

In addition, another aspect is a robot control device characterized by comprising: a control unit that generates a trajectory of the end point of the arm of the robot so that the end point approaches a target position; A current image that is an image including the endpoint at the time of the position, and a target image that is an image that includes the endpoint when the endpoint is at the target position, the control unit based on a route formed based on one or more set guidance positions, The arm is moved with the above-mentioned current image and the above-mentioned target image. Accordingly, while maintaining the high speed of position control, it is also possible to cope with changes in the target position.

Here, the drive control unit may move the arm using a signal obtained by superimposing the command value generated by the first control unit and the command value generated by the second control unit by predetermined components. Thereby, the trajectory of the end point can be moved so that it becomes a desired trajectory. For example, the trajectory of the end points can be formed so that, although not ideal, the trajectory includes the object from the angle of view of the hand-eye camera.

Here, the drive control unit may determine the predetermined component based on a difference between the current position and the target position. Thereby, the components can be continuously changed according to the distance, so that the switching control can be smoothly performed.

Here, an input unit for inputting the aforementioned predetermined component may also be provided. Thereby, the arm can be controlled on a trajectory desired by the user.

Here, a storage unit for storing the above-mentioned predetermined components may also be provided. Thereby, it is possible to use previously initialized components.

Here, the drive control unit may be configured to drive the arm using a command value based on a trajectory generated by the first control unit when the current position satisfies a predetermined condition, and to drive the arm when the current position does not satisfy the predetermined condition. Under the condition of , the arm is driven using a command value based on the trajectory generated by the first control unit and a command value based on the trajectory generated by the second control unit. Thereby, processing can be performed at a higher speed.

Here, it is also possible to include: a force detection unit that detects the force applied to the end point; and a third control unit that moves the end point from the current position to the target position based on the value detected by the force detection unit. The way of moving is to generate the trajectory of the above-mentioned endpoint, and the above-mentioned drive control unit uses the command value based on the trajectory generated by the first control unit, the command value based on the trajectory generated by the second control unit, and the command value based on the trajectory generated by the third control unit. The arm is moved using the command value of the trajectory generated by the control unit, or the command value based on the trajectory generated by the first control unit and the command value based on the trajectory generated by the third control unit. Thereby, even when the target position moves or the target position cannot be confirmed, it is possible to maintain the high speed of the position control and safely perform the work.

In addition, another aspect is a robot system characterized by comprising: a robot having an arm; and a first control unit that directs the end point of the arm toward the target along a path formed based on one or more set guidance positions. The method of position movement generates a command value; the imaging unit performs an image including the target image when the endpoint is at the target position, and an image including the endpoint when the endpoint is at the current position at the current time. an image, that is, a current image; a second control unit that generates a command value in such a manner that the endpoint moves from the current position to the target position based on the current image and the target image; and a drive control unit that uses The command value generated by the first control unit and the command value generated by the second control unit move the arm. Accordingly, while maintaining the high speed of position control, it is also possible to cope with changes in the target position.

In addition, another aspect is a robot system characterized by comprising: a robot having an arm; a control unit that generates a trajectory of the end point so that the end point of the arm approaches a target position; and an imaging unit that captures the above-mentioned The current image, which is an image including the endpoint when the endpoint is at a current position which is the current position, and the target image, which is an image including the endpoint when the endpoint is at the target position, are photographed, and the control unit performs based on the settings. The path formed by one or more guide positions, the current image, and the target image are used to move the arm. Accordingly, while maintaining the high speed of position control, it is also possible to cope with changes in the target position.

In addition, another aspect is a robot, characterized by comprising: an arm; and a first control unit configured to move an end point of the arm to a target position along a path formed based on one or more set guide positions, An instruction value is generated; an image acquiring unit that acquires a target image that is an image including the endpoint when the endpoint is at the target position, and a current image that includes the endpoint when the endpoint is at a current position that is a position at the current time. image; a second control unit that generates a command value in such a way that the endpoint moves from the current position to the target position based on the current image and the target image, and a drive control unit that uses the command value generated by the first control unit and the command value generated by the second control unit to move the arm. Accordingly, while maintaining the high speed of position control, it is also possible to cope with changes in the target position.

In addition, another aspect is a robot characterized by comprising: an arm; a control unit that generates a trajectory of the end point so that the end point of the arm approaches a target position; and an image acquisition unit that acquires that the end point is at the current position. The current image is an image including the endpoint when the endpoint is at the target position, and the target image is an image including the endpoint when the endpoint is at the target position. The arm is moved with the current image and the target image. Accordingly, while maintaining the high speed of position control, it is also possible to cope with changes in the target position.

In addition, another aspect is a robot control method, which is characterized in that it includes the steps of: acquiring an image including the end point of the arm of the robot, that is, a target image when the end point is at the target position; and acquiring the current position where the end point is at the current moment. An image including the above-mentioned end point at that time, that is, the step of the current image; A step of generating a command value such that the end point moves from the current position to the target position using the image and the target image, and moving the arm using the command value. Accordingly, while maintaining the high speed of position control, it is also possible to cope with changes in the target position.

In addition, another aspect is a robot control method, which is characterized in that it controls an arm and acquires an image including the end point when the end point of the arm is at the current position, that is, the current image and when the end point is at the target position include the above-mentioned The robot control method of the above-mentioned arm of the robot of the image acquisition part of the image of the end point, that is, the target image, and using the command value of the position control performed based on the route formed based on one or more set guide positions, and the command value based on the above-mentioned The command value of visual servoing performed on the current image and the above-mentioned target image, thereby controlling the above-mentioned arm. Accordingly, while maintaining the high speed of position control, it is also possible to cope with changes in the target position.

In addition, another aspect is a robot control method, which is characterized in that it controls an arm and acquires an image including the end point when the end point of the arm is at the current position, that is, the current image and when the end point is at the target position include the above-mentioned The robot control method of the above-mentioned arm of the robot of the image acquisition part of the image of the end point, that is, the target image, simultaneously performs position control based on a path formed based on one or more set guide positions, and simultaneously performs position control based on the above-mentioned current image. And the visual servoing performed by the above-mentioned target image. Accordingly, while maintaining the high speed of position control, it is also possible to cope with changes in the target position.

In addition, another aspect is a robot control program, which is characterized in that it causes the computing device to execute the step of acquiring a target image that is an image including the end point of the arm of the robot when the end point is at the target position, and the end point is at the target position as the current time. the current image including the endpoint at the current location of the location; and generating a command value so that the endpoint moves to the target location according to a path formed based on one or more set guide locations, And a step of generating a command value to move the end point from the current position to the target position based on the current image and the target image, and moving the arm using the command value. Accordingly, while maintaining the high speed of position control, it is also possible to cope with changes in the target position.

In addition, another aspect relates to a robot control device including: a robot control unit that controls a robot based on image information; a change amount calculation unit that calculates an image feature amount change amount based on the image information; a change amount estimation unit , which calculates an estimated amount of change in the image feature amount, that is, an estimated image feature amount change amount, based on information for estimating the amount of change that is information on the robot or the object and is information other than the image information; and abnormality determination A section that performs abnormality determination by comparing the amount of change in the image feature amount and the amount of change in the estimated image feature amount.

In this form, abnormality determination of robot control using image information is performed based on the estimated image feature amount change amount obtained based on the image feature amount change amount and the change amount estimation information. Thereby, in the control of the robot using the image information, especially in the means using the image feature value, it is possible to appropriately perform abnormality determination and the like.

In another aspect, the information for estimating the amount of change may be joint angle information of the robot.

Thus, the joint angle information of the robot can be used as the change amount estimation information.

In another aspect, the change amount estimating unit may act on the change amount of the joint angle information by acting on a Jacobian matrix in which the change amount of the joint angle information corresponds to the change amount of the image feature value. , so as to calculate the change amount of the above-mentioned estimated image feature value.

Thus, the amount of change in the estimated image feature amount, etc. can be obtained using the amount of change in the joint angle information and the Jacobian matrix.

In addition, in another aspect, the information for estimating the amount of change may be the position and posture information of the end effector of the robot or the object.

As a result, the position and posture information of the end effector of the robot or the above-mentioned object can be used as the change amount estimation information.

In addition, in another aspect, the change amount estimating unit may be configured such that, by acting on the change amount of the position posture information, a Jacobian matrix in which the change amount of the position posture information is associated with the change amount of the image feature value, Thus, the above-mentioned estimated image feature amount change amount is calculated.

In this way, the amount of change in the estimated image feature amount and the like can be obtained using the amount of change in the position and orientation information and the Jacobian matrix.

In addition, in another manner, it may also be configured such that the image feature value f1 of the first image information is acquired at the i-th (i is a natural number) time, and the image feature value f1 of the first image information is acquired at the j-th (j is a natural number satisfying j≠i) time. In the case of the image feature amount f2 of the two image information, the change amount calculating unit calculates the difference between the image feature amount f1 and the image feature amount f2 as the image feature amount change amount, and the change amount estimating portion (k is a natural number) the information p1 for estimating the amount of change corresponding to the first image information is acquired at time, and the information p2 for estimating the amount of change corresponding to the second image information is acquired at the lth time (l is a natural number) In this case, the amount of change in the estimated image feature amount is obtained from the information p1 for estimating the amount of change and the information p2 for estimating the amount of change.

Accordingly, it is possible to obtain the corresponding image feature amount change amount, estimated image feature amount change amount, and the like in consideration of time.

In another aspect, the kth time may be an acquisition time of the first image information, and the lth time may be an acquisition time of the second image information.

This makes it possible to easily perform processing in consideration of time, taking into consideration that joint angle information can be acquired at high speed.

In another aspect, the abnormality determination unit may perform a comparison process between the difference information between the image feature amount change amount and the estimated image feature amount change amount and a threshold value, and when the difference information is greater than the threshold value When it is large, it is judged as abnormal.

Thereby, abnormality determination etc. can be performed by threshold value determination.

In addition, in another aspect, the abnormality determination unit may be configured such that the difference between the acquisition times of the two pieces of image information used in the calculation of the change amount of the image feature value in the change amount calculation unit becomes larger. If the value is larger, the above-mentioned threshold value is set to be larger.

Thereby, the threshold value etc. can be changed according to a situation.

In addition, in another aspect, when an abnormality is detected by the abnormality determination unit, the robot control unit may perform control to stop the robot.

Thereby, it is possible to realize safe robot control and the like by stopping the robot at the time of abnormality detection.

In addition, in another aspect, when an abnormality is detected by the abnormality determination unit, the robot control unit skips the calculation of the amount of change in the image feature value in the change amount calculation unit. Of the two pieces of the above-mentioned image information, the above-mentioned image information acquired at a later time in time series, that is, the abnormality determination image information is formed, and based on the above-mentioned image information acquired at a time earlier than the above-mentioned abnormality determination image information to control.

Thereby, it is possible to skip the control of the robot using the abnormality determination image information at the time of abnormality detection.

In addition, another aspect relates to a robot control device including: a robot control unit that controls a robot based on image information; and a change calculation unit that obtains position and posture information indicating an end effector of the robot or an object. The amount of change in the position and posture of the change, or the amount of change in the joint angle representing the amount of change in the joint angle information of the robot; the change amount estimation unit, which calculates the amount of change in the image feature value based on the above image information, and calculates the amount of change in the image feature value based on the above image feature value. the amount of change is to obtain an estimated amount of change in position and posture, which is an estimated amount of change in position and posture, or an estimated amount of change in joint angle, that is, an estimated amount of change in joint angle; Abnormality determination is performed by comparison processing with the above-mentioned estimated position and posture change amount, or comparison processing between the above-mentioned joint angle change amount and the above-mentioned estimated joint angle change amount.

In addition, in another form, the estimated position and posture change amount or the estimated joint angle change amount is obtained from the image feature value change amount, and the position and posture change amount is compared with the estimated position and posture change amount, or the joint angle change amount is compared with the estimated joint angle change amount. Abnormal judgment is made by comparing the amount of joint angle change. Accordingly, it is also possible to appropriately perform abnormality determination in the control of the robot using image information, particularly in means using image feature values, by comparing position and posture information or joint angle information.

In addition, in another aspect, the change amount computing unit may perform a process of acquiring a plurality of the position and posture information and obtaining a difference between the plurality of the position and posture information as the position and posture change amount, and acquire a plurality of positions and posture information. The process of obtaining the above-mentioned position and posture information and obtaining the above-mentioned joint angle change amount based on the difference of the above-mentioned multiple pieces of the above-mentioned position and posture information, acquiring the above-mentioned pieces of the above-mentioned joint angle information, and calculating the difference of the above-mentioned joint angle information as the above-mentioned joint angle change amount Any one of the processing of acquiring a plurality of pieces of joint angle information and calculating the amount of change in the position and orientation based on the difference of the pieces of joint angle information.

Thus, the amount of change in position and posture, the amount of change in joint angle, and the like can be obtained by various means.

In addition, another aspect relates to a robot including: a robot control unit that controls the robot based on image information; a change amount calculation unit that calculates an image feature amount change amount based on the above image information; a change amount estimation unit that calculating an estimated amount of change in the image feature amount, that is, an estimated image feature amount change amount, based on change amount estimation information that is information on the robot or the object and that is information other than the image information; and an abnormality determination unit, The abnormality determination is performed by comparing the change amount of the image feature amount with the above-mentioned estimated image feature amount change amount.

In another aspect, abnormality determination of robot control using image information is performed based on the image feature amount change amount and the estimated image feature amount change amount obtained based on the change amount estimation information. This makes it possible to appropriately perform abnormality determination in the control of the robot using image information, particularly in means using image feature values.

In addition, another aspect relates to a robot control method, which is a robot control method for controlling a robot based on image information, including the step of performing a change amount calculation process for obtaining a change amount of an image feature value based on the image information; The step of calculating the change amount estimation process of calculating the estimated amount of change amount of the image feature amount, that is, the estimated image feature amount change amount; and a step of performing abnormality determination by comparing the change amount of the image feature amount with the above-mentioned estimated image feature amount change amount.

In another aspect, abnormality determination of robot control using image information is performed based on the image feature amount change amount and the estimated image feature amount change amount obtained based on the change amount estimation information. This makes it possible to appropriately perform abnormality determination in the control of the robot using image information, particularly in means using image feature values.

In addition, another aspect relates to a program that causes a computer to function as: a robot control unit that controls a robot based on image information; and a change amount calculation unit that calculates an image feature amount change amount based on the image information. a change amount estimating unit that performs an estimation of the amount of change in the image feature amount, that is, an estimated image feature amount change amount, based on information for estimating the amount of change that is information on the robot or the object and is information other than the image information. an operation; and an abnormality determination unit that performs abnormality determination by comparing the amount of change in the image feature value and the amount of change in the estimated image feature value.

In another aspect, the computer executes the abnormality determination of the robot control using the image information based on the image feature amount change amount and the estimated image feature amount change amount obtained based on the change amount estimating information. This makes it possible to appropriately perform abnormality determination in the control of the robot using image information, particularly in means using image feature values.

As described above, according to several aspects, it is possible to provide a robot control device, a robot, a robot control method, and the like that appropriately detect abnormalities in control using image feature values in robot control based on image information.

In addition, another aspect relates to a robot that performs an inspection process of inspecting the inspection object using a captured image of the inspection object captured by an imaging unit, and generates an inspection area including the inspection process based on first inspection information. The second inspection information, and according to the second inspection information, perform the above inspection process.

In addition, in another form, the second inspection information including the inspection area is generated based on the first inspection information. Generally, which area of the image used for inspection (appearance inspection in a narrow sense) is used for processing depends on information such as the shape of the object to be inspected, the content of work performed on the object to be inspected, and so on. When the object or work content changes, it is necessary to reset the inspection area, which causes a heavy burden on the user. In this regard, by generating the second inspection information based on the first inspection information, it is possible to easily determine the inspection area and the like.

In addition, in another aspect, the second inspection information may include a viewpoint information group including a plurality of viewpoint information, and each viewpoint information in the viewpoint information group may include the imaging unit in the inspection process. The position of the viewpoint and the direction of the line of sight.

In this way, a viewpoint information group and the like can be generated as the second inspection information.

In another aspect, it may be configured to set a priority for moving the imaging unit to the viewpoint position and the line-of-sight direction corresponding to the viewpoint information for each viewpoint information in the viewpoint information group.

Thereby, it is possible to set a priority and the like for each piece of viewpoint information included in the viewpoint information group.

In another aspect, it may be configured to move the imaging unit to the viewpoint position and the line-of-sight direction corresponding to the viewpoint information in the viewpoint information group according to the movement order set based on the priority. .

In this way, it is possible to actually control the imaging unit to perform inspection processing and the like using a plurality of viewpoint information with priorities set therein.

In addition, in another aspect, it may be configured such that, based on the movable range information, it is determined that the imaging unit cannot move to the viewpoint position corresponding to the i-th (i is a natural number) viewpoint information among the plurality of viewpoint information. And in the case where the line of sight direction moves, the imaging unit is not moved based on the i-th viewpoint information, but is based on the next j-th of the i-th viewpoint information in the moving sequence (j is satisfied i≠j) natural number) point of view information to move the imaging unit.

Thereby, it is possible to realize the control of the imaging unit in consideration of the movable range of the robot, and the like.

In addition, in another aspect, the first inspection information may include a relative inspection processing target position with respect to the inspection target object, and based on the inspection processing target position, it may be configured to correspond to the inspection target object. The object coordinate system is used to generate the viewpoint information using the object coordinate system.

In this way, viewpoint information and the like in the object coordinate system can be generated.

In addition, in another aspect, the first inspection information may include object position and posture information indicating the position and posture of the inspection target in the global coordinate system, and may be obtained based on the object position and posture information. According to the relative relationship between the above-mentioned global coordinate system and the above-mentioned object coordinate system, the above-mentioned viewpoint information in the above-mentioned global coordinate system is obtained, and based on the movable range information in the above-mentioned global coordinate system and the above-mentioned viewpoint information in the above-mentioned global coordinate system, It is determined whether or not the imaging unit can be moved to the viewpoint position and the line-of-sight direction.

In this way, it is possible to generate viewpoint information in the global coordinate system, and to control the movement of the imaging unit based on the viewpoint information and the robot's movable range information.

In addition, in another aspect, the inspection process may be performed on a result of robot work, and the first inspection information may be information acquired during the robot work.

Thereby, the first inspection information and the like can be acquired during the robot work.

In addition, in another aspect, the first inspection information may be configured to include shape information of the inspection object, position and posture information of the inspection object, and a relative inspection processing target position of the inspection object. at least one of the information.

Thereby, at least one piece of information on the shape information, the position posture information, and the position of the inspection processing target can be acquired as the first inspection information.

In addition, in another aspect, the first inspection information may be configured to include three-dimensional model data of the inspection object.

Thereby, three-dimensional model data can be acquired as the first inspection information.

In addition, in another aspect, the inspection process may be performed on the result of the robot operation, and the three-dimensional model data may include post-operation three-dimensional model data obtained by performing the robot operation, and data before and after the robot operation. The above-mentioned three-dimensional model data of the above-mentioned inspection target object is the pre-work three-dimensional model data.

Thereby, three-dimensional model data before and after work can be acquired as the first inspection information.

In another aspect, the second inspection information may include a pass image, and the pass image is the three-dimensional model data captured by a virtual camera arranged at the viewpoint position and the line-of-sight direction corresponding to the viewpoint information. The resulting image.

Thereby, it is possible to acquire a pass image or the like as the second inspection information from the three-dimensional model data and viewpoint information.

In addition, in another aspect, the second inspection information may include a pass image and a pre-work image, and the pass image is taken by a virtual camera arranged at the viewpoint position and the line-of-sight direction corresponding to the viewpoint information. The image obtained from the three-dimensional model data after the operation, the image before operation is an image obtained by capturing the three-dimensional model data before operation by the virtual camera arranged at the viewpoint position and the line of sight direction corresponding to the viewpoint information, and The above-mentioned pre-work image is compared with the above-mentioned pass image, and the above-mentioned inspection area is obtained.

In this way, the pass image and the pre-work image can be obtained from the three-dimensional model data before and after the operation and the viewpoint information, and the inspection region and the like can be obtained by comparing them.

In another aspect, in the comparison, a difference image that is a difference between the pre-operation image and the acceptable image is obtained, and the inspection region is a part of the difference image that includes the inspection object. area.

In this way, it is possible to obtain the inspection region and the like using the difference image.

In addition, in another aspect, the second inspection information may include a pass image and a pre-work image, and the pass image is taken by a virtual camera arranged at the viewpoint position and the line-of-sight direction corresponding to the viewpoint information. The image obtained from the post-operation three-dimensional model data, the above-mentioned pre-operation image is an image obtained by capturing the pre-operation three-dimensional model data by the virtual camera arranged at the viewpoint position and the line-of-sight direction corresponding to the viewpoint information, according to the above The degree of similarity between the pre-work image and the pass image is set as a threshold value used in the inspection process based on the captured image and the pass image.

Thereby, it is possible to set the threshold value of the inspection process, etc., using the degree of similarity between the pre-work image and the acceptable image.

In another aspect, at least a first arm and a second arm may be included, and the imaging unit may be a hand-eye camera provided on at least one of the first arm and the second arm.

Accordingly, it is possible to perform inspection processing and the like using two or more arms and at least one hand-eye camera provided on the arm.

In addition, another aspect relates to a processing device that outputs information used in the inspection process for a device that performs inspection processing of the inspection target object using a captured image of the inspection target object captured by the imaging unit, and based on The first inspection information generates the second inspection information including the viewpoint information including the viewpoint position and line-of-sight direction of the imaging unit in the inspection process, and the inspection area in the inspection process, and targets the device performing the inspection process. The above-mentioned second check information is output.

In addition, in another aspect, the second inspection information including the inspection area is generated based on the first inspection information. Generally, which area of the image used for inspection (appearance inspection in a narrow sense) is used for processing depends on information such as the shape of the object to be inspected, the content of the work performed on the object to be inspected, and so on. When the inspection object or work content changes, it is necessary to reset the inspection area, which places a heavy burden on the user. In this regard, by generating the second inspection information based on the first inspection information, it is possible to easily determine an inspection area and make another device perform inspection processing or the like.

In addition, another aspect relates to an inspection method for performing an inspection process of inspecting the inspection object using a captured image of the inspection object captured by the imaging unit. The inspection method includes, based on the first inspection information, A step of generating second inspection information including viewpoint information including a viewpoint position and a line-of-sight direction of the imaging unit in the inspection process and an inspection area in the inspection process.

In addition, in another aspect, the second inspection information including the inspection area is generated based on the first inspection information. Generally, which area of the image used for inspection (appearance inspection in a narrow sense) is used for processing depends on information such as the shape of the object to be inspected, the content of the work performed on the object to be inspected, and so on. When the inspection object or work content changes, it is necessary to reset the inspection area, which places a heavy burden on the user. In this regard, by generating the second inspection information based on the first inspection information, it is possible to easily determine the inspection area and the like.

In this manner, according to several aspects, it is possible to provide a robot, a processing device, an inspection method, and the like that can easily perform an inspection while reducing the burden on the user by generating the second inspection information required for the inspection based on the first inspection information.

Description of drawings

FIG. 1 is an explanatory diagram of assembly work performed by visual servoing.

2A and 2B are explanatory diagrams of positional displacement of an object to be assembled.

FIG. 3 is an example of the system configuration of this embodiment.

4 is an explanatory diagram of an assembly operation performed by visual servoing based on feature quantities of an object to be assembled.

FIG. 5 is an example of a captured image used for visual servoing based on feature quantities of an object to be assembled.

Fig. 6 is an explanatory diagram of an assembled state.

FIG. 7 is a flowchart of visual servoing based on feature quantities of an object to be assembled.

FIG. 8 is another flow chart of visual servoing based on feature quantities of objects to be assembled.

FIG. 9 is an explanatory diagram of a process of moving an object to be assembled directly above an object to be assembled.

FIG. 10 is an explanatory diagram of assembly work performed by two types of visual servoing.

FIG. 11 is a flowchart of processing when two types of visual servoing are performed continuously.

12(A) to (D) are explanatory diagrams of a reference image and a captured image.

13A and 13B are explanatory views of the assembly work of three workpieces.

14(A) to (C) are explanatory diagrams of captured images used when performing assembly work of three workpieces.

FIG. 15 is a flowchart of processing when three workpieces are assembled.

16(A) to (C) are explanatory diagrams of captured images used when three workpieces are assembled simultaneously.

Fig. 17 is a flowchart of processing when three workpieces are assembled simultaneously.

18(A) to (C) are explanatory diagrams of captured images used when three workpieces are assembled in another order.

19A and 19B are configuration examples of the robot.

Fig. 20 is a configuration example of a robot control system that controls a robot via a network.

FIG. 21 is a diagram showing an example of the configuration of the robot system 1 according to the second embodiment.

FIG. 22 is a block diagram showing an example of the functional configuration of the robot system 1 .

FIG. 23 is a data flow diagram of the robot system 1 .

FIG. 24 is a diagram showing a hardware configuration of the control unit 20 .

FIG. 25A is a diagram illustrating a trajectory of an end point when the arm 11 is controlled by position control and visual servoing, and FIG. 25B is an example of a target image.

FIG. 26 is a diagram explaining the component α.

FIG. 27 is a flowchart showing the flow of processing of the robot system 2 according to the third embodiment of the present invention.

FIG. 28 is a diagram explaining the position of an object, the position of a switching point, and the trajectory of an end point.

FIG. 29 is a diagram showing an example of the configuration of the robot system 3 according to the fourth embodiment of the present invention.

FIG. 30 is a block diagram showing an example of the functional configuration of the robot system 3 .

FIG. 31 is a flowchart showing the processing flow of the robot system 3 .

FIG. 32 is a diagram showing an assembly operation in which the robot system 3 inserts the workpiece into the hole H. As shown in FIG.

FIG. 33 is a flowchart showing the flow of processing of the robot system 4 according to the fifth embodiment of the present invention.

FIG. 34 is a diagram showing an assembly operation in which the robot system 4 inserts the workpiece into the hole H. As shown in FIG.

FIG. 35 is a configuration example of a robot control device according to this embodiment.

FIG. 36 is a detailed configuration example of the robot control device according to this embodiment.

Fig. 37 is an example of arrangement of imaging units that acquire image information.

FIG. 38 is an example of the configuration of the robot of this embodiment.

FIG. 39 is another example of the structure of the robot of this embodiment.

Fig. 40 is a configuration example of a general visual servoing control system.

FIG. 41 is a diagram illustrating the relationship between the amount of change in image feature data, the amount of change in position and posture information, and the amount of change in joint angle information, and the Jacobian matrix.

Fig. 42 is a diagram illustrating visual servoing control.

43A and 43B are explanatory diagrams of the abnormality detection means of this embodiment.

FIG. 44 is an explanatory diagram of means for setting a threshold corresponding to a difference in image acquisition time.

FIG. 45 is a diagram showing the relationship between image acquisition time, joint angle information acquisition time, and image feature amount acquisition time.

FIG. 46 is another diagram showing the relationship between the image acquisition time, the joint angle information acquisition time, and the image feature amount acquisition time.

FIG. 47 is a diagram illustrating the relationship between the amount of change in image feature data, the amount of change in position and posture information, and the amount of change in joint angle information in conjunction with mathematical formulas.

FIG. 48 is a flowchart illustrating the processing of this embodiment.

FIG. 49 is another detailed configuration example of the robot controller of this embodiment.

FIG. 50 is an example of the configuration of the robot of this embodiment.

51A and 51B are configuration examples of the processing device of this embodiment.

FIG. 52 is an example of the configuration of the robot of this embodiment.

Fig. 53 is another configuration example of the robot of this embodiment.

Fig. 54 is a configuration example of an inspection device using second inspection information.

Fig. 55 is an example of first inspection information and second inspection information.

FIG. 56 is a flowchart illustrating the flow of offline processing.

57A and 57B are examples of shape information (three-dimensional model data).

FIG. 58 shows an example of viewpoint candidate information used for generating viewpoint information.

FIG. 59 is an example of coordinate values in the object coordinate system of viewpoint candidate information.

FIG. 60 is an example of setting an object coordinate system based on the inspection processing target position.

61A to 61G are examples of pre-work images and acceptable images corresponding to each viewpoint information.

62A to 62D are explanatory diagrams of means for setting an inspection region.

63A to 63D are explanatory diagrams of means for setting an inspection region.

64A to 64D are explanatory diagrams of means for setting an inspection region.

65A to 65D are explanatory diagrams of similarity calculation processing before and after work.

66A to 66D are explanatory diagrams of similarity calculation processing before and after work.

67A to 67E are explanatory diagrams of priorities of viewpoint information.

Fig. 68 is a flowchart illustrating the flow of online processing.

69A and 69B are comparison examples of viewpoint information in the object coordinate system and viewpoint information in the robot coordinate system.

70A and 70B are explanatory diagrams of image rotation angles.

Detailed ways

Hereinafter, this embodiment will be described. In addition, this embodiment described below does not limit the content of this invention described in a claim unreasonably. In addition, not all the configurations described in this embodiment are necessarily essential components of the present invention.

1. The means of this embodiment

first embodiment

As shown in FIG. 1 , here, a case of assembling the object to be assembled WK1 grasped by the robot's hand HD to the object to be assembled WK2 will be described. In addition, the hand HD of the robot is provided at the tip of the arm AM of the robot.

First, as a comparative example of the present embodiment, when the assembly work shown in FIG. image to control the robot. Specifically, the object to be assembled WK1 is moved to the position of the object to be assembled WK1R in which the reference image is reflected as indicated by the arrow YJ, and assembled to the object to be assembled WK2 .

Here, the reference image RIM used at this time is shown in FIG. 2A , and the position on the real space (three-dimensional space) of the object to be assembled WK2 reflected in the reference image RIM is shown in FIG. 2B . In the reference image RIM of FIG. 2A , the object to be assembled WK2 and the object to be assembled WK1R (corresponding to WK1R in FIG. 1 ) in the assembled state (or the state before assembly) are reflected. In visual servoing using the above-mentioned reference image RIM, the position and posture of the assembly object WK1 reflected in the captured image and the position and posture of the assembly object WK1R in the assembled state reflected in the reference image RIM are matched so that the assembly object WK1 WK1 moves.

However, as described above, when the assembly work is actually performed, the position and orientation of the object to be assembled WK2 may change. For example, as shown in FIG. 2B , the position of the center of gravity of the object to be assembled WK2 reflected in the reference image RIM of FIG. 2A is GC1 in real space. On the other hand, the actual object to be assembled WK2 is offset, and the position of the center of gravity of the actual object to be assembled WK2 is GC2. In this case, even if the actual object to be assembled WK1 is moved so as to match the position and posture of the object to be assembled WK1R reflected in the reference image RIM, the actual assembled object WK2 cannot be brought into an assembled state. , therefore, the assembly work cannot be performed correctly. This is because when the position and orientation of the object to be assembled WK2 changes, the position and orientation of the object to be assembled WK1 which is assembled with the object to be assembled WK2 also changes.

Therefore, the robot control system 100 and the like according to the present embodiment can accurately perform the assembly work even when the position and orientation of the object to be assembled changes.

Specifically, a configuration example of the robot control system 100 according to the present embodiment is shown in FIG. 3 . The robot control system 100 of the present embodiment includes a captured image acquisition unit 110 that acquires a captured image from the imaging unit 200 , and a control unit 120 that controls the robot 300 based on the captured image. In addition, the robot 300 has an end effector (hand) 310 and an arm 320 . In addition, the configuration of the imaging unit 200 and the robot 300 will be described in detail later.

First, the captured image acquiring unit 110 acquires a captured image of at least the target to be assembled among the target to be assembled and the target to be assembled on which the assembly work is reflected.

Then, the control unit 120 performs feature amount detection processing of the object to be assembled based on the captured image, and moves the object to be assembled based on the feature amount of the object to be assembled. Moreover, the process of outputting the control information (control signal) of the robot 300, etc. are included in the process of moving an object to be assembled. In addition, the functions of the control unit 120 can be realized by hardware such as various processors (CPU, etc.), ASIC (gate array, etc.), programs, or the like.

In this way, in the visual servoing (comparative example) using the above reference image, the object to be assembled is moved based on the feature value of the object to be assembled in the reference image. The characteristic quantity of the object to be assembled is used to move the object to be assembled. For example, as shown in FIG. 4, in the captured image captured by the camera CM, the feature value of the workpiece WK2 as the object to be assembled is detected, and based on the detected feature value of the workpiece WK2, the workpiece WK1 as the object to be assembled is Move as indicated by arrow YJ.

Here, the object to be assembled WK2 at the present time (time of imaging) is reflected in the captured image captured by the camera CM. Therefore, the object to be assembled WK1 can be moved to the position of the object to be assembled WK2 at the present time. Thereby, it is possible to prevent the object to be assembled WK1 from being moved to a position where it cannot be assembled at the present time, as in the failure example of visual servoing using the reference image (problem of the comparative example described in FIG. 1 ). Also, since a new target position for visual servoing is set based on the captured image every time an assembly operation is performed, even when the position and orientation of the object to be assembled WK2 changes, an accurate target position can be set.

As described above, even when the position and orientation of the object to be assembled changes, the assembly work can be accurately performed. Furthermore, in this embodiment, there is no need to prepare a reference image in advance, and the preparation cost for visual servoing can be reduced.

In addition, in this way, the control unit 120 controls the robot by performing visual servoing based on the captured image.

In this way, feedback control and the like can be performed on the robot according to the current work situation.

In addition, the robot control system 100 is not limited to the configuration shown in FIG. 1 , and can be implemented in various modifications such as omitting some of the above components or adding other components. In addition, as shown in FIG. 19B described later, the robot control system 100 of this embodiment is included in the robot 300 and may be integrally configured with the robot 300 . Furthermore, as shown in FIG. 20 described later, the functions of the robot control system 100 may be realized by the server 500 and the terminal device 330 included in each robot 300 .

In addition, for example, when the robot control system 100 and the imaging unit 200 are connected via at least one network including wired and wireless, the captured image acquisition unit 110 may be a communication unit (interface unit) that communicates with the imaging unit 200 . Furthermore, when the robot control system 100 includes the imaging unit 200 , the captured image acquisition unit 110 may be the imaging unit 200 itself.

Here, the captured image refers to an image captured by the imaging unit 200 . In addition, the captured image may be an image stored in an external storage unit or an image acquired via a network. The captured image is, for example, an image PIM shown in FIG. 5 to be described later.

In addition, an assembling operation refers to an operation of assembling a plurality of work objects, and specifically refers to an operation of assembling an assembly object to an assembled object. The assembly operation is, for example, an operation of placing the workpiece WK1 on (or beside) the workpiece WK2, an operation of inserting (fitting) the workpiece WK1 into (fitting) the workpiece WK2 (embedding operation, fitting operation), or placing the workpiece WK1 and the workpiece WK2 Bonding, connecting, assembling, and fusing work (bonding work, connecting work, assembling work, fusing work), etc.

In addition, the object to be assembled refers to an object to be assembled with respect to an object to be assembled in the assembly work. For example, it is workpiece WK1 in the example of FIG. 4 .

On the other hand, the object to be assembled refers to an object to which the object to be assembled is assembled in the assembly work. For example, it is workpiece WK2 in the example of FIG. 4 .

2. Details of processing

Next, the processing of this embodiment will be described in detail.

2.1. Visual servoing based on the feature quantity of the assembled object

The captured image acquisition unit 110 of the present embodiment acquires one or a plurality of captured images showing the assembly object and the assembled object. Then, the control unit 120 performs feature amount detection processing of the object to be assembled and the object to be assembled based on the acquired one or a plurality of captured images. In addition, the control unit 120 moves the object to be assembled so that the relative positional relationship between the object to be assembled and the object to be assembled becomes a target relative positional relationship based on the feature amount of the object to be assembled and the feature amount of the object to be assembled. .

Here, the target relative position and posture relationship refers to the relative position and posture relationship between the object to be assembled and the object to be assembled, which is targeted at the time of the assembly work by visual servoing. For example, in the example of FIG. 4 , the relative position and posture relationship when the workpiece WK1 is in contact with (adjacent to) the triangular hole HL of the workpiece WK2 is the target relative position and posture.

As a result, assembly operations and the like can be performed based on the feature quantities of the object to be assembled and the feature quantities of the object to be assembled detected from the captured image. In addition, one or more captured images acquired by the captured image acquisition unit 110 will be described in detail in a later stage.

In addition, in many assembly operations, a portion of the assembly object to be assembled with the assembly object (assembly portion) and a portion of the assembly object to be assembled with the assembly object (assembly portion) are often specified. For example, in the example of FIG. 4 , the assembled portion of the object to be assembled refers to the bottom surface BA of the workpiece WK1 , and the portion to be assembled of the object to be assembled refers to the triangular hole HL of the workpiece WK2 . In the example of the assembly work shown in FIG. 4 , the assembly part BA is fitted into the hole HL of the part to be assembled. For example, it is meaningless to assemble the side surface SA of the workpiece WK1 into the hole HL. Therefore, it is preferable to set the assembly part of the object to be assembled and the part to be assembled of the object to be assembled in advance.

Therefore, the control unit 120 uses the feature quantity set as the target feature quantity among the feature quantities of the object to be assembled and the feature quantity set as the attention feature quantity among the feature quantities of the object to be assembled to make the assembly object The object to be assembled is moved so that the relative positional relationship between the object and the object to be assembled becomes the target relative positional relationship.

Here, the feature amount refers to, for example, a feature point of an image, a contour line of an object to be detected (an object to be assembled, an object to be assembled, etc.) reflected in the image, and the like. Also, the feature amount detection processing refers to processing for detecting feature amounts in an image, for example, feature point detection processing, contour line detection processing, and the like.

Hereinafter, a case where a feature point is detected as a feature amount will be described. A feature point is a point that can stand out from an observation in an image. For example, in the captured image PIM11 shown in FIG. 5 , feature points P1 to P10 are detected as feature points of the workpiece WK2 to be assembled, and feature points Q1 to P10 are detected as feature points of the workpiece WK1 to be assembled. Q5. In addition, in the example of FIG. 5 , for convenience of illustration and explanation, only the feature points of P1 to P10 and Q1 to Q5 are detected, but more features than this are detected in the actual captured image. point. However, even when more feature points are detected, the content of the processing described below does not change.

In addition, in this embodiment, a corner detection method or the like is used as a feature point detection method (feature point detection processing), but other general corner detection methods (either eigenvalue, FAST feature detection) may also be used. Local feature descriptors represented by SIFT (Scale Invariant FeatureTransform: Scale Invariant Feature Transformation), SURF (Speeded Up Robust Feature: Fast Robust Feature), etc. can be used.

Furthermore, in this embodiment, based on the feature quantity set as the target feature quantity among the feature quantities of the object to be assembled, and the feature quantity set as the feature quantity of interest among the feature quantities of the object to be assembled, the Visual Servo.

Specifically, in the example of FIG. 5 , among the feature points P1 to P10 of the workpiece WK2 , a target feature point P9 and a target feature point P10 are set as target feature quantities. On the other hand, among the feature points Q1 to Q5 of the workpiece WK1 , a feature point of interest Q4 and a feature point of interest Q5 are set as feature quantities of interest.

Then, the control unit 120 moves the object to be assembled so that the feature point of interest of the object to be assembled matches or approaches the target feature point of the object to be assembled.

That is, the object to be assembled WK1 is moved as shown by the arrow YJ so that the focused feature point Q4 approaches the target feature point P9 and the focused feature point Q5 approaches the target feature point P10 .

Here, the target feature amount refers to a feature amount that is targeted when the assembly object is moved by visual servoing, among the feature amounts representing the object to be assembled. In other words, the target feature amount is the feature amount of the assembled part of the object to be assembled. In addition, the target feature point refers to a feature point set as a target feature amount when the feature point detection process is performed. As described above, in the example of FIG. 5 , the feature point P9 and the feature point P10 corresponding to the triangular hole HL of the workpiece WK2 are set as target feature points.

On the other hand, the feature value of interest refers to a point (in the example of FIG. 5 , a triangular hole of the workpiece WK2 ) facing a point on the real space corresponding to the target feature value among the feature values representing the object to be assembled or the object to be assembled. HL) is a moving feature value representing a point in real space (the bottom surface of the workpiece WK1 in the example of FIG. 5 ). In other words, the feature value of interest is the feature value of the assembly part of the object to be assembled. Note that the feature point of interest refers to a feature point set as the feature amount of interest when the feature point detection process is performed. As described above, in the example of FIG. 5 , the feature point Q4 and the feature point Q5 corresponding to the bottom surface of the workpiece WK1 are set as the feature points of interest.

In addition, the target feature quantity (target feature point) and the attention feature quantity (focus feature point) may be preset by the instructor (user), or may be feature quantities (feature points) set according to a given algorithm. . For example, the target feature point can also be set according to the deviation of the detected feature point and the relative positional relationship with the target feature point. Specifically, in the example of FIG. 5 , in the captured image PIM11 , the feature points P9 and P10 located near the centers of deviations of the feature points P1 to P10 representing the workpiece WK2 may be set as target feature points. In addition, it is also possible to pre-set points corresponding to target feature points in CAD (Computer Aided Design: Computer Aided Design) data representing the object to be assembled, and to perform matching between the CAD data and the CAD in the captured image, Accordingly, the feature points set as the target feature points are specified (detected) from among the feature points of the object to be assembled based on the result of the CAD matching. The same applies to the feature quantity of interest (feature point of interest).

In addition, in this embodiment, the control unit 120 moves the object to be assembled so that the attention feature point of the object to be assembled matches or approaches the target feature point of the object to be assembled. Objects are tangible objects, so in fact, the target feature point and the attention feature point cannot be detected at the same point. That is, moving the object to be assembled in a consistent manner ultimately means moving the point at which the feature point of interest is detected toward the point at which the target feature point is detected.

Accordingly, it is possible to move the assembly object such that the relative position and posture relationship between the set assembly portion of the assembly object and the set assembly portion of the assembly object becomes the target relative position and posture relationship.

Furthermore, it is possible to assemble the assembly portion of the object to be assembled to the portion to be assembled of the object to be assembled, and the like. For example, as shown in FIG. 6 , it is possible to fit the bottom surface BA of the assembly part of the workpiece WK1 into the hole HL of the assembly part of the workpiece WK2 .

Next, the processing flow of this embodiment will be described using the flowchart of FIG. 7 .

First, the captured image acquisition unit 110 acquires, for example, the captured image PIM11 shown in FIG. 5 ( S101 ). Both the object to be assembled WK1 and the object to be assembled WK2 are reflected in this captured image PIM11 .

Next, the control unit 120 performs feature detection processing based on the captured image PIM11 to detect the target feature FB of the object to be assembled WK2 and the feature of interest FA of the object to be assembled WK1 ( S102 , S103 ).

Then, as described above, the control unit 120 moves the assembly object WK1 based on the detected attention feature value FA and target feature value FB (S104), and checks the relative position and posture relationship between the assembly object WK1 and the assembly object WK2. It is judged whether it is the target relative position and posture relationship (S105).

Finally, when it is determined that the relative position and posture relationship between the assembly object WK1 and the assembled object WK2 becomes the target relative position and posture relationship as shown in FIG. If the relative position and posture relationship of the object WK2 has not reached the target relative position and posture relationship, it returns to step S101 and repeats the process. The above is the processing flow of this embodiment.

In addition, the captured image acquisition unit 110 may acquire a plurality of captured images. In this case, the captured image acquisition unit 110 may acquire a plurality of captured images showing both the assembly target object and the assembled object, or may acquire a captured image showing only the assembly target object and only a captured image showing only the assembled object. Captured images of objects.

Here, the latter processing flow in the case of acquiring a plurality of captured images showing the assembly object and the assembled object are shown in the flowchart of FIG. 8 .

First, the captured image acquisition unit 110 acquires the captured image PIM11 in which at least the object to be assembled WK2 is reflected ( S201 ). In addition, the object to be assembled WK1 may be reflected in this captured image PIM11. Then, the control unit 120 detects the target feature value FB of the object to be assembled WK2 from the captured image PIM11 ( S202 ).

Next, the captured image acquisition unit 110 acquires the captured image PIM12 in which at least the object to be assembled WK1 is reflected ( S203 ). In addition, as in step S201, the object to be assembled WK2 may be reflected in the captured image PIM12. Then, the control unit 120 detects the feature value FA of interest of the object to be assembled WK1 from the captured image PIM12 ( S204 ).

Then, the following is the same as the processing flow described in FIG. 7 , the control unit 120 moves the assembly object WK1 according to the detected attention feature value FA and target feature value FB (S205), and the assembly object WK1 and the assembly object It is determined whether the relative position and posture relationship of WK2 becomes the target relative position and posture relationship (S206).

Finally, when it is determined that the relative position and posture relationship between the assembly object WK1 and the assembled object WK2 becomes the target relative position and posture relationship as shown in FIG. If the relative position and posture relationship of the object WK2 has not reached the target relative position and posture relationship, it returns to step S203 and repeats the process. The above is the processing flow in the case of acquiring a plurality of captured images in which the object to be assembled and the object to be assembled are respectively reflected.

In addition, in the above example, the object to be assembled is actually assembled to the object to be assembled by visual servoing, but the present invention is not limited thereto, and the state before the object to be assembled to the object to be assembled may be formed by visual servoing. .

That is, the control unit 120 may specify an image region in a given positional relationship with the object to be assembled based on the feature amount (feature point) of the object to be assembled, and make the feature points of interest of the object to be assembled and the determined Make the assembly object move in such a way that the image areas are consistent or close to each other. In other words, the control unit 120 may specify a point in real space that is in a predetermined positional relationship with the object to be assembled based on the feature value of the object to be assembled, and move the object to be assembled to the specified point.

For example, in the captured image PIM shown in FIG. 9 , feature points P8 to P10 are detected as feature points representing the triangular hole HL which is the assembled portion of the object to be assembled WK2 . In this case, in the captured image PIM, the image regions R1 to R3 having a predetermined positional relationship with the feature points P8 to P10 are identified. Then, the assembly object WK1 is moved so that the attention feature point Q4 of the assembly object WK1 coincides with (closes to) the image region R2 , and the attention feature point Q5 of the assembly object coincides with (closes to) the image region R3 .

Thereby, for example, the state before the assembling work can be created.

In addition, it is not necessarily necessary to perform the process of detecting the feature value of the object to be assembled as shown in the above example. For example, it is also possible to detect the feature quantity of the object to be assembled, and infer the position and posture of the object to be assembled relative to the robot based on the detected feature quantity of the object to be assembled, so that the hand holding the object to be assembled Control the robot, etc. so that the part approaches the estimated position of the object to be assembled.

2.2. Assembly work by two types of visual servoing

Next, for the case where visual servoing using a reference image (first visual servoing) and visual servoing (second visual servoing) in which an assembly object is moved using a feature value of the object to be assembled are continuously performed processing will be explained.

For example, in FIG. 10 , the movement of the assembly object WK1 from the position GC1 to the position GC2 (the movement indicated by the arrow YJ1 ) is performed by the first visual servoing using the reference image, and by using the feature value of the assembly object WK2 The second visual servoing performs movement (movement shown by arrow YJ2 ) of assembly target object WK1 toward position GC3 from position GC2 . In addition, the positions GC1 to GC3 are the positions of the center of gravity of the object to be assembled WK1.

When performing such processing, the robot control system 100 of the present embodiment further includes a reference image storage unit 130 as shown in FIG. 3 . The reference image storage unit 130 stores a reference image displaying an assembly object taking a target position and posture. The reference image is, for example, an image RIM shown in FIG. 12(A) described later. In addition, the function of the reference image storage unit 130 can be realized by a memory such as RAM (Random Access Memory: Random Access Memory), HDD (Hard Disk Drive: Hard Disk Drive), or the like.

Furthermore, the control unit 120 moves the object to be assembled to the target position and posture based on at least the first captured image and the reference image showing the object to be assembled as the first visual servoing shown by the arrow YJ1 in FIG. 10 . .

And the control part 120 performs the 2nd visual servoing shown by the arrow YJ2 of FIG. 10 after the said 1st visual servoing. That is, after the control unit 120 moves the object to be assembled, based on the second captured image showing at least the object to be assembled, it performs a feature amount detection process of the object to be assembled, and based on the feature amount of the object to be assembled, the assembled object The object moves.

Here, a more specific flow of processing will be described using the flowchart of FIG. 11 and FIGS. 12(A) to 12(D).

First, as a preparation for the first visual servoing, the robot's hand HD grasps the object to be assembled WK1, moves the object to be assembled WK1 to the target position (S301), and takes an image with the imaging unit 200 (camera CM in FIG. 10 ). The assembly object WK1 in the target position and posture acquires a reference image (target image) RIM as shown in FIG. 12(A) (S302). Then, the feature value F0 of the object to be assembled WK1 is detected from the acquired reference image RIM (S303).

Here, the target position and posture refer to the position and posture of the assembly target object WK1 to be targeted in the first visual servoing. For example, in FIG. 10 , the position GC2 is the target position and posture, and the reference image RIM in FIG. 12(A) shows a state where the assembly object WK1 is located at the target position and posture GC2. The target position and posture are the positions and postures set by the instructor (user) when generating the reference image.

In addition, the reference image refers to an image in which the assembly object WK1 which is the moving object in the first visual servoing is reflected under the above-mentioned target position and posture, such as the reference image RIM in FIG. 12A . In addition, in the reference image RIM of FIG. 12(A), the object to be assembled WK2 is also reflected, but the object to be assembled WK2 is not necessarily reflected. In addition, the reference image may be an image stored in an external storage unit, an image acquired via a network, an image generated from CAD model data, or the like.

Next, first visual servoing is performed. First, the captured image acquisition unit 110 acquires the first captured image PIM101 as shown in FIG. 12(B) (S304).

Here, the first captured image in this example refers to a captured image of at least the assembly target WK1 among the assembly target WK1 and the assembly target WK2 , such as the captured image PIM101 in FIG. 12(B). .

Then, the control unit 120 detects the feature value F1 of the object to be assembled WK1 from the first captured image PIM101 (S305), and moves the object to be assembled WK1 as indicated by the arrow YJ1 in FIG. (S306).

Then, the control unit 120 judges whether the object to be assembled WK1 is in the target position and posture GC2 ( S306 ), and when it is determined that the object to be assembled WK1 is in the target position and posture GC2 , it moves to the second visual servoing. On the other hand, when it is determined that the object to be assembled WK1 is not in the target position and posture GC2 , the process returns to step S304 and the first visual servoing is repeated.

In this way, in the first visual servoing, the robot is controlled while comparing the feature quantities of the assembly target object WK1 in the reference image RIM and the first captured image PIM101 .

Next, the second visual servoing is performed. In the second visual servoing, first, the captured image acquisition unit 110 acquires the second captured image PIM21 as shown in FIG. 12(C) (S308). Here, the second captured image refers to a captured image used for the second visual servoing. In addition, in the second captured image PIM21 of this example, both the object to be assembled WK1 and the object to be assembled WK2 are reflected.

Then, the control unit 120 detects the target feature value FB of the object to be assembled WK2 from the second captured image PIM21 ( S309 ). For example, in this example, as shown in FIG. 12(C), target feature points GP1 and target feature points GP2 are detected as target feature quantities FB.

Similarly, the control unit 120 detects the feature value FA of interest of the object to be assembled WK1 from the second captured image PIM21 ( S310 ). For example, in this example, as shown in FIG. 12(C), attention feature points IP1 and attention feature points IP2 are detected as attention feature amounts FA.

Next, the control unit 120 moves the object to be assembled WK1 based on the attention feature value FA and the target feature value FB ( S312 ). That is, like the example described above with reference to FIG. 5 , the object to be assembled WK1 is moved so that the focused feature point IP1 approaches the target feature point GP1 and the focused feature point IP2 approaches the target feature point GP2 .

Then, the control unit 120 determines whether the object to be assembled WK1 and the object to be assembled WK2 are in the target relative positional relationship ( S312 ). For example, in the captured image PIME shown in FIG. 12(D), since the feature point of interest IP1 is adjacent to the target feature point GP1, and the feature point of interest IP2 is adjacent to the target feature point GP2, it is determined that the object to be assembled WK1 and the object to be assembled are WK2 is in the target relative position and posture relationship, and the process ends.

On the other hand, when it is determined that the object to be assembled WK1 and the object to be assembled WK2 are not in the target relative position and posture relationship, the process returns to step S308 and the second visual servoing is repeated.

As a result, each time the same assembly work is repeated, the same reference image is used to move the assembly object to the vicinity of the object to be assembled, and then the detailed position and posture of the actual object to be assembled can be compared. Cooperate with assembly work, etc. That is, even when the position and posture of the object to be assembled when the reference image is generated are shifted (different from) the position and posture of the object to be assembled during the actual assembly work, the second visual servoing is different from the position and posture of the object to be assembled. Since the positional displacement of the object to be assembled corresponds, in the first visual servoing, the same reference image can be used each time without using a different reference image. As a result, preparation costs for reference images and the like can be suppressed.

In addition, in the said step S310, although the feature quantity FA of interest of the object to be assembled WK1 is detected from the 2nd captured image PIM21, it does not necessarily mean that it must be detected from the 2nd captured image PIM21. For example, when the object to be assembled WK1 is not shown in the second captured image PIM21 , the feature amount of the object to be assembled WK1 may be detected from another second captured image PIM22 in which the object to be assembled is reflected.

2.3. Assembly of three workpieces

Next, as shown in FIG. 13A and FIG. 13B , the processing in the case where the assembly work of three works WK1 to WK3 is performed will be described.

In this assembly operation, as shown in FIG. 13A , the assembly object WK1 (workpiece WK1 , such as a driver) held by the first hand HD1 of the robot is assembled to the first assembled object held by the second hand HD2 of the robot. The object WK2 (work WK2, for example, a screw), and the workpiece WK2 assembled with the workpiece WK1 is assembled to the second object to be assembled WK3 (work WK3 , for example, a screw hole) on the table. Then, after the assembling work, it becomes the assembled state shown in FIG. 13B.

Specifically, when performing such processing, as shown in FIG. 14(A), the control unit 120 performs the first Feature amount detection processing of the object to be assembled WK2. The first captured image in this example refers to a captured image used when the assembly work of the assembly object WK1 and the first assembly target object WK2 is performed.

Then, the control unit 120 moves the assembly object WK1 as indicated by the arrow YJ1 in FIG. 13A based on the feature amount of the first assembly object WK2 .

Next, as shown in FIG. 14(B), after the control unit 120 moves the object to be assembled WK1, based on the second captured image PIM41 showing at least the object to be assembled WK3, the second object to be assembled WK3 is moved. feature detection processing. The second captured image in this example refers to a captured image used when performing the assembly work of the first object to be assembled WK2 and the second object to be assembled WK3 .

Then, the control unit 120 moves the assembly object WK1 and the first assembly object WK2 as indicated by the arrow YJ2 in FIG. 13A , based on the feature value of the second assembly object WK3 .

As a result, even when the positions of the first object to be assembled WK2 and the second object to be assembled WK3 are misaligned every time an assembly operation is performed, the objects to be assembled, the first object to be assembled, And the assembly operation of the second object to be assembled, etc.

Next, the flow of processing in the assembly work of the three workpieces shown in FIGS. 13A and 13B will be described in detail using the flowchart of FIG. 15 .

First, the captured image acquisition unit 110 acquires one or a plurality of first captured images showing at least the assembly object WK1 and the first assembly target object WK2 in the assembly operation. Then, the control unit 120 performs feature amount detection processing of the assembly object WK1 and the first assembly object WK2 based on the first captured image.

In this example, first, the captured image acquisition unit 110 acquires the first captured image PIM31 in which the first object to be assembled WK2 is reflected ( S401 ). Then, the control unit 120 detects the first target feature amount FB1 by performing feature amount detection processing of the first object to be assembled WK2 based on the first captured image PIM31 ( S402 ). Here, the target feature point GP1 and the target feature point GP2 as shown in FIG. 14(A) are detected as the first target feature amount FB1.

Next, the captured image acquisition unit 110 acquires the first captured image PIM32 in which the object to be assembled WK1 is reflected ( S403 ). Then, the control unit 120 detects the first attention feature amount FA by performing feature amount detection processing of the assembly target object WK1 based on the first captured image PIM32 ( S404 ). Here, the attention feature point IP1 and the attention feature point IP2 are detected as the first attention feature amount FA.

In addition, in steps S401 to S404, an example of acquiring a plurality of first photographed images (PIM31 and PIM32) respectively reflecting the assembly object WK1 and the first assembly object WK2 has been described, but as shown in FIG. 14 (A ), when the object to be assembled WK1 and the first object to be assembled WK2 are reflected in the same first captured image PIM31, the object to be assembled WK1 and the first object to be assembled can also be detected from the first captured image PIM31 The characteristic quantities of both sides of WK2.

Next, the control unit 120 uses the feature quantity of the assembly object WK1 (the first feature quantity of interest FA) and the feature quantity of the first object to be assembled WK2 (the first target feature quantity FB1 ) to make the assembly object WK1 and the first target feature quantity An object to be assembled WK1 is moved so that the relative positional relationship of the object to be assembled WK2 becomes the first target relative positional relationship (S405). Specifically, in the captured image, the object to be assembled WK1 is moved such that the focused feature point IP1 approaches the target feature point GP1 , and the focused feature point IP2 approaches the target feature point GP2 . In addition, this movement corresponds to the movement of arrow YJ1 in FIG. 13A .

Then, the control unit 120 determines whether the object to be assembled WK1 and the first object to be assembled WK2 are in the first target relative positional relationship ( S406 ). When it is determined that the object to be assembled WK1 and the first object to be assembled WK2 are not in the first target relative positional relationship, the process returns to step S403 and the process is repeated.

On the other hand, when it is determined that the assembly object WK1 and the first assembly object WK2 are in the first target relative positional relationship, the second attention feature of the first assembly object WK2 is detected from the first captured image PIM32 Quantity FB2 (S407). Specifically, as shown in FIG. 14(B) described later, the control unit 120 detects a feature point of interest IP3 and a feature point of interest IP4 as the second feature value of interest FB2 .

Next, the captured image acquisition part 110 acquires the 2nd captured image PIM41 which shows at least the 2nd object to be assembled WK3 as shown in FIG.14(B) (S408).

Then, the control unit 120 detects the second target feature amount FC by performing feature amount detection processing of the second object to be assembled WK3 based on the second captured image PIM41 ( S409 ). Specifically, as shown in FIG. 14(B), the control unit 120 detects the target feature point GP3 and the target feature point GP4 as the second target feature value FC.

Next, the control unit 120 uses the feature quantity of the first assembly object WK2 (the second attention feature quantity FB2) and the feature quantity of the second assembly object WK3 (the second target feature quantity FC) to make the first The assembly object WK1 and the first assembly object WK2 are moved so that the relative positional relationship between the assembled object WK2 and the second assembled object WK3 becomes the second target relative positional relationship ( S410 ).

Specifically, in the captured image, the object to be assembled WK1 and the first object to be assembled WK2 are arranged such that the feature point of interest IP3 is close to the target feature point GP3 and the feature point of interest IP4 is close to the target feature point GP4. move. In addition, this movement corresponds to the movement of the arrow YJ2 of FIG. 13A.

Then, the control unit 120 determines whether the first object to be assembled WK2 and the second object to be assembled WK3 are in the second target relative positional relationship ( S411 ). When it is determined that the first object to be assembled WK2 and the second object to be assembled WK3 are not in the second target relative positional relationship, the process returns to step S408 and the process is repeated.

On the other hand, as in the captured image PIME shown in FIG. 14(C), when it is determined that the first object to be assembled WK2 and the second object to be assembled WK3 are in the assembled state, that is, in the second target relative positional relationship, In case, end processing.

In this way, the feature points of interest (IP1 and IP2) of the assembly object WK1 can be brought closer to the target feature points (GP1 and GP2) of the first object to be assembled WK2, and the feature points of interest of the first object to be assembled WK2 can be brought closer to each other. Visual servoing etc. are performed so that (IP3 and IP4) approach the target feature point (GP3 and GP4) of the second object to be assembled WK3.

In addition, instead of assembling the assembly object WK1 and the first assembly object WK2 sequentially as shown in the flow chart of FIG. 15 , they may be assembled simultaneously as shown in FIGS. 16(A) to 16(C). Three artifacts.

The flow of processing at this time is shown in the flowchart of FIG. 17 . First, the captured image acquisition unit 110 acquires one or a plurality of captured images showing the assembly object WK1 , the first assembly object WK2 , and the second assembly object WK3 during the assembly operation ( S501 ). In this example, captured image PIM51 shown in FIG. 16(A) is acquired.

Next, the control unit 120 performs feature value detection processing of the assembly object WK1 , the first assembly object WK2 , and the second assembly object WK3 based on one or more captured images ( S502 to S504 ).

In this example, as shown in FIG. 16(A), target feature points GP3 and target feature points GP4 are detected as feature quantities of the second object to be assembled WK3 (S502). Then, target feature points GP1 and GP2 , and focused feature points IP3 and focused feature points IP4 are detected as feature quantities of the first assembly object WK2 ( S503 ). Then, the attention feature point IP1 and the attention feature point IP2 are detected as feature quantities of the assembly object WK1 ( S504 ). In addition, when the three workpieces are reflected in different captured images, the feature amount detection processing may be performed on the different captured images.

Next, the control unit 120 sets the relative positional relationship between the assembly object WK1 and the first assembly object WK2 as the first target based on the feature value of the assembly object WK1 and the feature value of the first assembly object WK2. The method of relative position and posture relationship makes the assembly object WK1 move, and makes the first assembly object WK2 and the second The first object to be assembled WK2 is moved so that the relative positional relationship of the object to be assembled WK3 becomes the second target relative positional relationship (S505).

That is, to bring the focused feature point IP1 close to the target feature point GP1, to make the focused feature point IP2 close to the target feature point GP2, to make the focused feature point IP3 close to the target feature point GP3, and to make the focused feature point IP4 close to the target feature point GP4 In an approaching manner, the assembly object WK1 and the first assembly object WK2 are moved simultaneously.

Then, the captured image acquisition unit 110 reacquires the captured image (S506), and the control unit 120 performs an operation of the three assembly objects WK1, the first assembled object WK2, and the second assembled object WK3 based on the newly acquired captured image. It is determined whether a workpiece is in the target relative position and posture relationship (S507).

For example, the photographed image acquired in step S506 is the photographed image PIM52 shown in Figure 16 (B), when it is determined that the three workpieces are not in the target relative position and posture relationship, return to step S503, and repeat deal with. Moreover, based on the captured image PIM52 newly acquired, the process of step S503 and following are performed.

On the other hand, when the captured image acquired in step S506 is the captured image PIME as shown in FIG. 16(C), it is determined that the three workpieces are in the target relative position and posture relationship, and the process ends.

Thereby, the assembly work of three workpiece|work etc. can be performed simultaneously. As a result, it is possible to shorten the work time and the like of the assembly work of the three workpieces.

In addition, when performing the assembly work of three workpieces, the assembly work may be performed in the reverse order of the assembly order shown in the flow chart of FIG. 15 . That is, as shown in FIGS. 18(A) to 18(C), after assembling the first object to be assembled WK2 to the second object to be assembled WK3, the object to be assembled WK1 may be assembled to the first object to be assembled. Object WK2.

In this case, as shown in FIG. 18(A), the control unit 120 performs a feature analysis of the second object to be assembled WK3 based on the first captured image PIM61 showing at least the second object to be assembled WK3 during the assembly operation. Quantity detection processing, and moves the first object to be assembled WK2 based on the feature amount of the second object to be assembled WK3. Note that, since the details of the feature amount detection processing are the same as those described with reference to FIG. 16(A), description thereof will be omitted.

Next, as shown in FIG. 18(B), the control unit 120 performs feature quantity detection processing of the first object to be assembled WK2 based on the second captured image PIM71 showing at least the first object to be assembled WK2 after the movement, Then, the object to be assembled WK1 is moved so as to form the assembled state shown in FIG. 18(C) based on the feature value of the first object to be assembled WK2.

Thereby, it is not necessary to simultaneously move the object to be assembled WK1 and the first object to be assembled WK2 , and it is possible to more easily control the robot and the like. In addition, even if the robot is not a multi-arm robot but a single-arm robot, it is possible to perform assembly work of three workpieces and the like.

In addition, the imaging unit (camera) 200 used in the above-mentioned embodiment includes, for example, an imaging element such as a CCD (charge-coupled device) and an optical system. The imaging unit 200 is located on a ceiling or a workbench, for example, at an angle such that a detection object in visual servoing (an object to be assembled, an object to be assembled, or the end effector 310 of the robot 300, etc.) enters the field of view of the imaging unit 200. configuration. Furthermore, the imaging unit 200 outputs the information of the captured image to the robot control system 100 and the like. However, in this embodiment, the information of the captured image is output to the robot control system 100 as it is, but the present invention is not limited thereto. For example, the imaging unit 200 can include a device (processor) used for image processing and the like.

3. Robot

Next, a configuration example of a robot 300 to which the robot control system 100 of the present embodiment is applied is shown in FIGS. 19A and 19B . In either case of FIGS. 19A and 19B , the robot 300 has an end effector 310 .

The end effector 310 is a member attached to an end point of the arm for grasping, lifting, hoisting, sucking a workpiece (work object), and performing processing on the workpiece. The end effector 310 may be, for example, a hand (grip), a hook, or a suction cup. Also, a plurality of end effectors may be provided for one arm. In addition, the arm is a part of the robot 300, and is a movable part including one or more joints.

For example, in the robot of FIG. 19A , the robot main body 300 (robot) and the robot control system 100 are configured independently. In this case, part or all of the functions of the robot control system 100 are realized by a PC (Personal Computer), for example.

In addition, the robot of this embodiment is not limited to the configuration shown in FIG. 19A , and may be configured integrally with the robot main body 300 and the robot control system 100 as shown in FIG. 19B . That is, the robot 300 may also include the robot control system 100 . Specifically, as shown in FIG. 19B , the robot 300 may have a robot body (with an arm and an end effector 310 ) and a base unit supporting the robot body, and the robot control system 100 may be housed in the base unit. In the robot 300 shown in FIG. 19B , wheels and the like are provided on the base unit and the entire robot is movable. In addition, FIG. 19A is an example of a single-arm type, but the robot 300 may be a multi-arm type robot such as a double-arm type as shown in FIG. 19B . In addition, the robot 300 may be moved by a human hand, or may be a robot provided with a motor for driving wheels and controlled by the robot control system 100 to move the motor. In addition, it is not limited to install the robot control system 100 in the base unit part installed under the robot 300 as shown in FIG. 19B.

In addition, as shown in FIG. 20 , the functions of the robot control system 100 may be realized by a server 500 communicatively connected to the robot 300 via a network 400 including at least one of wired and wireless.

Alternatively, in this embodiment, part of the processing of the robot control system of the present invention may be performed by the robot control system on the server 500 side. In this case, the processing is realized by distributed processing with the robot control system provided on the robot 300 side. In addition, the robot control system on the robot 300 side is realized by, for example, a terminal device 330 (control unit) provided in the robot 300 .

Furthermore, in this case, the robot control system on the server 500 side performs the processing assigned to the robot control system of the server 500 among the respective processes of the robot control system of the present invention. On the other hand, the robot control system provided in the robot 300 performs processing assigned to the robot control system of the robot 300 among the respective processes of the robot control system of the present invention. In addition, each process of the robot control system of the present invention may be allocated to the server 500 side, or may be allocated to the robot 300 side.

Thereby, for example, the server 500 having a higher processing capability than the terminal device 330 can perform processing with a large amount of processing. In addition, for example, the server 500 can collectively control the operation of each robot 300 and can easily coordinate the operation of a plurality of robots 300 .

In addition, in recent years, there is an increasing tendency to manufacture many types of parts with a small number of components. Furthermore, when changing the type of parts to be manufactured, it is necessary to change the operation performed by the robot. With the configuration shown in FIG. 20 , the server 500 can collectively change the actions and the like performed by the robots 300 without redoing the instruction work for each of the plurality of robots 300 .

Furthermore, according to the configuration shown in FIG. 20 , compared with the case where one robot control system 100 is provided for each robot 300 , it is possible to greatly reduce the trouble of updating the software of the robot control system 100 .

In addition, the robot control system and the robot of the present embodiment may implement a part or most of the above processing by a program. In this case, a processor such as a CPU executes the program, thereby realizing the robot control system and the robot of the present embodiment. Specifically, a program stored in an information storage medium is read, and a processor such as a CPU executes the read program. Here, the information storage medium (a medium that can be read by a computer) is a medium that stores programs, data, etc., and its function can be performed by an optical disk (DVD, CD, etc.), HDD (hard disk drive), or a memory (card memory, ROM, etc.) ) and so on to achieve. Furthermore, a processor such as a CPU performs various processes in the present embodiment based on a program (data) stored in an information storage medium. That is, a program for causing a computer (a device including an operation unit, a processing unit, a storage unit, and an output unit) to function as each unit of the present embodiment (a program for causing the computer to execute the processing of each unit) is stored in the information storage medium. .

The present embodiment has been described in detail above, but it is obvious to those skilled in the art that various changes can be made without substantially departing from the novel contents and effects of the present invention. Therefore, such modified examples are also included in the scope of the present invention. For example, in the specification or the drawings, a term described together with a different term having a broader or synonymous meaning at least once can be replaced by the different term at any position in the specification or the drawings. In addition, the configuration and operation of the robot control system, the robot, and the program are not limited to those described in this embodiment, and various modifications can be made.

second embodiment

FIG. 21 is a system configuration diagram showing an example of the configuration of the robot system 1 according to the embodiment of the present invention. The robot system 1 of the present embodiment mainly includes a robot 10 , a control unit 20 , a first imaging unit 30 , and a second imaging unit 40 .

The robot 10 is an arm-type robot having an arm 11 including a plurality of joints (joints) 12 and a plurality of links 13 . The robot 10 performs processing based on control signals from the control unit 20 .

An actuator (not shown) for operating them is provided on the joint 12 . The actuator includes, for example, a servo motor, an encoder, and the like. The encoder value output by the encoder is used for feedback control of the robot 10 by the control unit 20 .

A hand-eye camera 15 is provided near the front end of the arm 11 . The hand-eye camera 15 is a means for capturing an object at the tip of the arm 11 to generate image data. As the hand-eye camera 15, for example, a visible light camera, an infrared camera, or the like can be employed.

As the region of the tip portion of the arm 11 , a region that is not connected to other regions of the robot 10 (excluding the hand 14 described later) is defined as an end point of the arm 11 . In this embodiment, the position of the point E shown in FIG. 21 is located at the end point.

In addition, the structure of the robot 10 has demonstrated the main structure when describing the characteristic of this embodiment, and is not limited to the said structure. Structures possessed by general handling robots are not excluded. For example, although a six-axis arm is shown in FIG. 21 , the number of axes (the number of joints) may be further increased or decreased. The number of connecting rods can also be increased or decreased. In addition, the shape, size, arrangement, structure, etc. of various components such as arms, links, and joints may be appropriately changed.

The control unit 20 performs processing for controlling the robot 10 as a whole. The control unit 20 may be installed in a place away from the main body of the robot 10 or built in the robot 10 . When the control unit 20 is installed in a place away from the main body of the robot 10 , the control unit 20 is connected to the robot 10 by wire or wirelessly.

The first imaging unit 30 and the second imaging unit 40 are means for generating image data by imaging the vicinity of the work area of the arm 11 from different angles. The first imaging unit 30 and the second imaging unit 40 include cameras, for example, and are installed on a workbench, a ceiling, a wall, or the like. As the first imaging unit 30 and the second imaging unit 40 , a visible light camera, an infrared camera, or the like can be employed. The first imaging unit 30 and the second imaging unit 40 are connected to the control unit 20 , and input images captured by the first imaging unit 30 and the second imaging unit 40 to the control unit 20 . In addition, the first imaging unit 30 and the second imaging unit 40 may be connected to the robot 10 instead of the control unit 20 , or may be built in the robot 10 . At this time, the images captured by the first imaging unit 30 and the second imaging unit 40 are input to the control unit 20 via the robot 10 .

Next, an example of the functional configuration of the robot system 1 will be described. FIG. 22 shows a functional block diagram of the robot system 1 .

The robot 10 includes a motion control unit 101 that controls the arm 11 based on encoder values of actuators, sensor values of sensors, and the like.

The motion control unit 101 drives the actuator so as to move the arm 11 to the position output from the control unit 20 based on the information output from the control unit 20 , the encoder value of the actuator, the sensor value of the sensor, and the like. The current position of the end point can be obtained from an encoder value or the like of an actuator provided on the joint 12 or the like.

The control unit 20 mainly includes a position control unit 2000 , a visual servoing unit 210 , and a drive control unit 220 . The position control unit 2000 mainly includes a route acquisition unit 201 and a first control unit 202 . The visual servoing unit 210 mainly includes an image acquisition unit 211 , an image processing unit 212 , and a second control unit 213 .

The position control unit 2000 executes position control for moving the arm 11 along a predetermined path set in advance.

The route acquisition unit 201 acquires information on routes. The route is formed based on the guidance position, for example, is formed by connecting one or more guidance positions set in advance through guidance in a predetermined order set in advance. In addition, information related to the route, for example, information indicating coordinates and order within the route is held in the memory 22 (described later, refer to FIG. 24 and the like). The information on the route held in the memory 22 may also be input via the input device 25 or the like. In addition, the information on the route also includes the final position of the endpoint, that is, information on the target position.

The first control unit 202 sets the next guidance position, that is, the trajectory of the set end point, based on the information about the route acquired by the route acquisition unit 201 .

In addition, the first control unit 202 determines the next moving position of the arm 11 , that is, the target angle of each actuator provided on the joint 12 based on the track of the end point. In addition, the first control unit 202 generates a command value for moving the arm 11 by a target angle, and outputs it to the drive control unit 220 . In addition, since the processing performed by the first control unit 202 is general, a detailed description thereof will be omitted.

The visual servoing unit 210 measures the change in the relative position of the target object as visual information based on the images captured by the first imaging unit 30 and the second imaging unit 40 , and uses it as feedback information to track the target object. The control means, so-called visual servoing, makes the arm 11 move.

In addition, as visual servoing, methods such as a position-based method, a feature-based method, etc., which control a robot based on three-dimensional position information of an object by using two objects that produce parallax, etc., can be used. The image is calculated by a method such as a stereogram that recognizes the image as a stereoscopic image; the method of the above-mentioned feature reference is to make the difference between the image captured by the two imaging units from the orthogonal direction and the target image of each imaging unit held in advance is zero (the error matrix of the number of pixels in each image is zero) to control the robot. For example, in this embodiment, a feature-based method is employed. In addition, although the feature-based method can be performed using one imaging unit, it is preferable to use two imaging units in order to improve accuracy.

The image acquisition unit 211 acquires an image captured by the first imaging unit 30 (hereinafter referred to as a first image) and an image captured by the second imaging unit 40 (hereinafter referred to as a second image). The first image and the second image acquired by the image acquisition unit 211 are output to the image processing unit 212 .

The image processing unit 212 recognizes the tip of the endpoint from the first image and the second image based on the first image and the second image acquired from the image acquiring unit 211 , and extracts an image including the endpoint. In addition, since the image recognition processing performed by the image processing part 212 can use various general techniques, the description is abbreviate|omitted.

The second control unit 213 sets the track of the end point, that is, the movement of the end point, based on the image extracted by the image processing unit 212 (hereinafter, referred to as the current image) and the image when the end point is at the target position (hereinafter, referred to as the target image). amount and direction of movement. In addition, what is necessary is just to store the information acquired beforehand in the memory 22 etc. about a target image.

In addition, the second control unit 213 determines the target angle of each actuator provided on the joint 12 based on the movement amount and movement direction of the end point. Then, the second control unit 213 generates a command value for moving the arm 11 by a target angle, and outputs it to the drive control unit 220 . In addition, since the process performed by the 2nd control part 213 is general content, detailed description is abbreviate|omitted.

In addition, in the robot 10 having joints, once the angles of the joints are determined, the positions of the end points are uniquely determined by forward kinematics processing. That is, in an N-joint robot, since one target position can be represented by N joint angles, if a combination of the N joint angles is used as one target joint angle, the trajectories of the end points can be considered as a set of joint angles. Thus, the command values output from the first control unit 202 and the second control unit 213 may be values related to positions (target positions) or values related to joint angles (target angles).

The drive control unit 220 outputs an instruction to the motion control unit 101 based on the information acquired from the first control unit 202 and the second control unit 213 to move the position of the end point, that is, the arm 11 . The details of the processing performed by the drive control unit 220 will be described in detail later.

FIG. 23 is a data flow diagram of the robot system 1 .

A feedback loop for bringing each joint of the robot closer to a target angle through position control is transmitted to the position control unit 2000 . The preset route information includes information related to the target position. For the first control unit 202 , if the information related to the target position is acquired, the orbit and the command value (here, the target angle) are generated according to the information related to the target position and the current position acquired by the route acquisition unit 201 .

A visual feedback loop for approaching a target position using information from the first imaging unit 30 and the second imaging unit 40 is transmitted to the visual servoing unit 210 . The second control unit 213 acquires a target image as information on the target position. For the second control unit 213, since the current image and the target position on the current image are represented by the coordinate system on the image, it is transformed into the coordinate system of the robot. The second control unit 213 generates a trajectory and a command value (here, a target angle) based on the converted current current image and target image.

The drive control unit 220 outputs a command value formed from the command value output from the first control unit 202 and the command value output from the second control unit 213 to the robot 10 . Specifically, the drive control unit 220 multiplies the command value output from the first control unit 202 by a coefficient α, multiplies the command value output from the second control unit 213 by a coefficient 1−α, and sends the output to the robot 10 A value obtained by combining the above values is output. Here, α is a real number larger than 0 and smaller than 1.

In addition, the form of the command value formed from the command value output from the 1st control part 202 and the command value output from the 2nd control part 213 is not limited to this.

Here, in this embodiment, the command value is output from the first control unit 202 at constant intervals (for example, every 1 millisecond (msec)), and the command value is output from the second control unit 213 at an interval longer than that from the first control unit 202. The command value is output at intervals (for example, every 30msec). Therefore, when the drive control unit 220 does not output a command value from the second control unit 213 , it multiplies the command value output from the first control unit 202 by the coefficient α, and the last command output from the second control unit 213 The value is multiplied by a coefficient of 1−α, and a value obtained by combining the above values is output to the robot 10 . The drive control unit 220 may temporarily store the command value output from the second control unit 213 in a storage device such as the memory 22 (see FIG. 24 ), and the drive control unit 220 may read it from the storage device and use it.

The operation control unit 101 acquires a command value (target angle) from the control unit 20 . The operation control unit 101 acquires the current angle of the end point based on the encoder value of the actuator provided on the joint 12 and the like, and calculates the difference (deviation angle) between the target angle and the current angle. In addition, the operation control unit 101 calculates the moving speed of the arm 11 based on, for example, the deviation angle (the larger the deviation angle, the faster the moving speed), and moves the arm 11 at the calculated deviation angle at the calculated movement speed.

FIG. 24 is a block diagram showing an example of a schematic configuration of the control unit 20 . As shown in the figure, the control unit 20 composed of, for example, a computer, etc. is equipped with a central processing unit (Central Processing Unit:) 21 as a computing device; a RAM (Random Access Memory: random access memory) as a volatile storage device , a memory 22 composed of a ROM (Read only Memory: read-only memory) as a non-volatile storage device; an external storage device 23; a communication device 24 for communicating with external devices such as the robot 10; an input such as a mouse or a keyboard. device 25; an output device 26 such as a display; and an interface (I/F) 27 for connecting the control unit 20 to other units.

Each of the functional units described above is realized by, for example, CPU 21 reading out and executing a predetermined program stored in memory 22 in memory 22 . In addition, the predetermined program may be installed in the memory 22 in advance, for example, or may be downloaded from the network via the communication device 24 to be installed or updated.

The configuration of the above robot system 1 is not limited to the configuration described above while the main configuration was described when describing the characteristics of the present embodiment. In addition, a configuration including a general robot system is not excluded.

Next, the characteristic processing of the robot system 1 having the above configuration according to the present embodiment will be described. In this embodiment, an operation of sequentially visually inspecting the objects O1, O2, and O3 shown in FIG. 21 using the hand-eye camera 15 will be described as an example.

When a control start instruction is input through a button (not shown) or the like, the control unit 20 controls the arm 11 through position control and visual servoing. The drive control unit 220 uses the value to be output from the first control unit 202 (hereinafter referred to as position-based control) when the command value is input from the second control unit 213 (in this embodiment, once every 30 times). command value) and a value output from the second control unit 213 (hereinafter referred to as a command value based on visual servoing) with arbitrary components, and outputs an instruction to the motion control unit 101 . When the drive control unit 220 does not input a command value from the second control unit 213 (29 times out of 30 in the present embodiment), it uses the command value based on the position control output from the first control unit 202 and the last The command value output from the second control unit 213 and temporarily stored in the memory 22 or the like is output as an instruction to the operation control unit 101 .

FIG. 25A is a diagram illustrating a trajectory of an end point when the arm 11 is controlled by position control and visual servoing. The object O1 is installed at position 1 in FIG. 25A , the object O2 is installed at position 2 , and the object O3 is installed at position 3 . In FIG. 25A, the objects O1, O2, and O3 are located on the same plane (XY plane), and the hand-eye camera 15 is located at a constant position in the Z direction.

In FIG. 25A , the trajectory shown by the solid line is the trajectory of the end point when only the command value by position control is used. This trajectory becomes a trajectory that passes above positions 1, 2, and 3. When the objects O1, O2, and O3 are always installed at the same position and posture, it is possible to control the objects O1, O2, and O3 only by position control. Visual inspection.

In contrast, in FIG. 25A , consider a case where the object O2 moves from position 2 on the solid line to position 2 after the movement. Since the end point moves above the object O2 shown by the solid line, when only the command value by position control is used, the accuracy of the inspection of the object O2 may decrease or inspection may become impossible.

Since it corresponds to the movement of the position of the object, visual servoing is well applied. If visual servoing is used, even if the position of the object is shifted, the end point can be moved directly above the object. For example, if the object O2 is located at the moved position 2, in the case of giving the image shown in FIG. 25B as the target image, assuming that only the instruction value based on visual servoing is used, the endpoint is shown by the dotted line in FIG. 25A on the track.

Visual servoing is a very useful control method capable of responding to the displacement of the object, but due to the frame rate of the first imaging unit 30 or the second imaging unit 40, the image processing time of the image processing unit 212, etc., it is different from the position control method. Compared with the case, there is a problem that it takes a long time to reach the target position.

Therefore, by using command values for position control and visual servoing at the same time (simultaneously performing position control and visual servoing, that is, parallel control), the inspection accuracy can be ensured in accordance with the positional deviation of the objects O1, O2, and O3. Move at a relatively high speed.

In addition, the term "simultaneous" is not limited to exactly the same time and time. For example, the case where the command values for position control and visual servoing are used simultaneously includes the case where the command value for position control and the command value for visual servoing are output simultaneously, and the command value for position control and the command for visual servoing are output with a slight time difference. Concept of value situation. The minute time may be any length of time as long as the same processing as in the simultaneous case can be performed.

Especially in the case of visual inspection, since the viewing angle of the hand-eye camera 15 only needs to include the object O2 (the object O2 does not need to be located at the center of the viewing angle), there is no problem even if the orbit is not on the object O2.

Therefore, in the present embodiment, the drive control unit 220 synthesizes the command value by position control and the command value by visual servoing so that a trajectory including the object O2 is formed in the field of view of the hand-eye camera 15 . The trajectory at this time is indicated by a dotted line in FIG. 25A. This orbit does not pass directly above the object O2, but is a position where inspection accuracy can be ensured to the maximum.

In addition, in addition to the displacement of the installation position of the object, the expansion of each member of the arm 11 due to temperature change, etc. is also an important factor for the deviation between the position on the path and the actual position of the object. In some cases, it can also be solved by using command values of position control and visual servoing at the same time.

The position of the track indicated by the dashed-dotted line in FIG. 25A can be changed by the component α. FIG. 26 is a diagram explaining the component α.

FIG. 26A is a graph showing the relationship between the distance to the target (here, objects O1, O2, and O3) and the component α. Line A is the case of constant component α regardless of the distance to the target position. Line B is a case where the component α is gradually decreased in accordance with the distance to the target position. Lines C and D are the case where the component α is continuously reduced corresponding to the distance to the target position, line C is the case where the change in the component α is made smaller in proportion to the distance, and line D is the case where the distance is proportional to the component α Condition. Wherein, the component α is 0<α<1.

In the case of lines B, C, and D in FIG. 26A , the components are set so that as the distance to the target position gets closer, the proportion of the command value for position control decreases and the proportion of the command value for visual servoing increases. alpha. Accordingly, when the target position moves, the trajectory can be generated so that the end point is closer to the target position.

In addition, since the component α is set so that each command value of the position control and visual servoing is superimposed, the component α can be continuously changed corresponding to the distance to the target position. By continuously changing the components according to the distance, it is possible to smoothly switch the control from the control of the arm approximately based on position control to the control of the arm approximately based on visual servoing.

In addition, as shown in FIG. 26A , the component α is not limited to the case where it is defined by the distance to the target (objects O1, O2, O3 here). As shown in FIG. 26B , the component α may be specified by the distance from the start position. That is, the drive control unit 220 can determine the component α based on the difference between the current position and the target position.

In addition, the distance to the target and the distance from the start position may be obtained from the route obtained by the route obtaining unit 101, or may be obtained from the current image and the target image. For example, when calculating from a route, the calculation can be performed based on the coordinates and order of the start position, target, and object positions included in the route-related information, and the coordinates and order of the current position.

The relationship between the difference and the component between the current position and the target position as shown in FIG. 26 can be input via an input unit such as the input device 25 because the arm 11 is controlled in a trajectory desired by the user. In addition, the relationship between the difference and the component between the current position and the target position is stored in storage means such as the memory 22 in advance, and it may be used. In addition, the relationship between the difference and the component between the current position and the target position stored in the storage means may be the content input via the input unit, or may be the content initially set in advance.

According to the present embodiment, since the arm (hand-eye camera) is controlled using command values obtained by combining position control and visual servoing command values at a constant ratio, even when the position of the object is shifted, the High-speed inspection can be performed with high precision. In particular, according to the present embodiment, it is possible to make the speed equal to the position control (higher than the visual servoing), and it is possible to perform a robustness check against positional deviation compared with the position control.

In addition, in the present embodiment, the command value based on position control and the command value based on visual servoing are usually synthesized, but for example, when the positional deviation of the object O2 is greater than a predetermined threshold, only the command value based on The arm 11 is moved according to the command value of visual servoing. The second control unit 213 may determine whether or not the positional deviation of the object O2 is larger than a predetermined threshold value based on the current image.

In addition, in the present embodiment, the drive control unit 220 determines the component α based on the difference between the current position and the target position, but the method of determining the component α is not limited to this. For example, the drive control unit 220 may change the component α with time. In addition, the drive control unit 220 may change the component α with time until a certain time elapses, and then change the component α according to the difference between the current position and the target position.

third embodiment

In the first embodiment of the present invention, the arm is usually controlled using a command value obtained by combining command values of position control and visual servoing at a constant ratio, but the scope of application of the present invention is not limited thereto.

In the third embodiment of the present invention, the case of using only the command values of the position control corresponding to the position of the object and the case of using the command values obtained by combining the command values of the position control and visual servoing at a constant ratio are used. The way the situation is combined. Hereinafter, a robot system 2 according to a third embodiment of the present invention will be described. In addition, since the structure of the robot system 2 is the same as that of the robot system 1 of the second embodiment, description of the structure of the robot system 2 will be omitted, and the processing of the robot system 2 will be described. In addition, the same code|symbol is attached|subjected to the same part as 2nd Embodiment, and description is abbreviate|omitted.

FIG. 27 is a flowchart showing the flow of control processing of the arm 11 of the present invention. This process is started, for example, by inputting a control start instruction via a button (not shown) or the like. In this embodiment, visual inspection of objects O1 and O2 is performed.

When the processing is started, the position control unit 2000 performs position control (step S1000). That is, the first control unit 202 generates a command value based on the information on the route acquired by the route acquisition unit 201 , and outputs the command value to the drive control unit 220 . The drive control unit 220 outputs the command value output from the first control unit 202 to the robot 10 . In this way, the motion control unit 101 moves the arm 11 (that is, the end point) according to the command value.

Next, the first control unit 202 judges the result of moving the end point by the position control, that is, whether the end point has passed the switching point 1 (step S1002 ). The information indicating the position of the switching point 1 is included in the preset route-related information.

FIG. 28 is a diagram illustrating the positions of the objects O1 and O2, the positions of the switching points, and the trajectories of the end points. In this embodiment, the switching point 1 is set between the start point and the object O1.

When the end point has not passed through the switching point 1 (No in step S1002), the control unit 20 repeats the process of step S1000.

When the end point passes through the switching point 1 (YES in step S1002), the drive control unit 220 controls the arm 11 using position control and visual servoing (step S1004). That is, the first control unit 202 generates a command value based on the information on the route acquired by the route acquisition unit 201 , and outputs the command value to the drive control unit 220 . In addition, the second control unit 213 generates a command value based on the current image and the target image processed by the image processing unit 212 , and outputs it to the drive control unit 220 . The drive control unit 220 switches the component α stepwise over time, and uses the switched component α to synthesize the command value output from the first control unit 202 and the command value output from the second control unit 213 , and output to the robot 10. In this way, the motion control unit 101 moves the arm 11 (that is, the end point) according to the command value.

Hereinafter, the processing of step S1004 will be specifically described. Before the processing of step S1004 , that is, in the processing of step S1000 , the command value from the visual servoing unit 210 is not used. Therefore, the component α of the command value from the position control unit 2000 is 1 (the component 1−α=0 of the command value from the visual servoing unit 210 ).

When a certain time (for example, 10 msec) elapses after the start of the process in step S1004 , the drive control unit 220 switches the component α of the command value from the position control unit 2000 from 1 to 0.9. In this way, the component 1−α of the command value from the visual servoing unit 210 becomes 0.1. Then, the drive control unit 220 synthesizes the command values on the premise that the component α of the command value from the position control unit 2000 is 0.9 and the component 1−α of the command value from the visual servoing unit 210 is 0.1, And output to the robot 10.

After that, when a certain period of time has elapsed, the drive control unit 220 switches the component α of the command value from the position control unit 2000 from 0.9 to 0.8, and switches the component 1-α of the command value from the visual servoing unit 210 from 0.1. to 0.2. In this way, the component α is switched stepwise with the lapse of a certain time, and the command value output from the first control unit 202 and the command value output from the second control unit 213 are combined using the switched component.

The position control unit 2000 repeats the switching of the component α and the synthesis of the command values until the component α of the command value from the position control unit 2000 becomes 0.5 and the component 1-α of the command value from the visual servoing unit 210 becomes 0.5. . After the component α of the command value from the position control unit 2000 becomes 0.5 and the component 1−α of the command value from the visual servoing unit 210 becomes 0.5, the drive control unit 220 maintains the component α without switching the component α, Synthesis of command values is repeated.

This enables visual inspection even when the position of the object O1 changes. In addition, when the object is farther than necessary, the end point is moved only by position control, thereby enabling high-speed processing. In addition, when approaching an object, the end point is moved by position control and visual servoing, and it is also possible to cope with a change in the position of the object. In addition, by gradually switching the component α, sudden movements and vibrations of the arm 11 can be prevented.

In addition, in the process of step S1004, when the end point passes the switching point 2 (step S1006, which will be described in detail later) while switching the component α and synthesizing the command value, the component α does not change until the component α becomes 0.5. Switching of the component α and synthesis of command values are performed, and the process proceeds to step S1006.

Next, the first control unit 202 judges whether or not the end point has passed the switching point 2 as a result of moving the end point by position control and visual servoing (step S1006 ). Information indicating the position of the switching point 2 is included in the route-related information. As shown in FIG. 28 , the switching point 2 is set to the object O1.

When the end point has not passed through the switching point 2 (No in step S1006), the control unit 20 repeats the process of step S1004.

When the end point passes through the switching point 2 (Yes in step S1006), the drive control unit 220 switches the component α so that the component α increases stepwise with the passage of time, and uses the switched component α , the command value output from the first control unit 202 and the command value output from the second control unit 213 are combined and output to the robot 10 . The motion control unit 101 moves the arm 11 (that is, the endpoint) according to the command value (step S1008).

Hereinafter, the processing of step S1008 will be specifically described. Before performing the processing in step S1008, that is, in the processing in step S1006, the drive control unit 220 sets the component α of the command value from the position control unit 2000 to 0.5, and sets the component α of the command value from the visual servoing unit 210 to 1−α 0.5 to synthesize the instruction value.

When a certain time (for example, 10 msec) elapses after the start of the process in step S1008 , the drive control unit 220 switches the component α of the command value from the position control unit 2000 from 0.5 to 0.6. In this way, the component 1−α of the command value from the visual servoing unit 210 becomes 0.4. Then, the drive control unit 220 synthesizes the command values on the premise that the component α of the command value from the position control unit 2000 is 0.6 and the component 1−α of the command value from the visual servoing unit 210 is 0.4, and Output to the robot 10.

After that, when a certain time elapses, the drive control unit 220 switches the component α of the command value from the position control unit 2000 from 0.6 to 0.7, and switches the component 1-α of the command value from the visual servoing unit 210 from 0.4 to 0.4. to 0.3. In this way, the component α is switched stepwise with the lapse of a certain time, and the command value output from the first control unit 202 and the command value output from the second control unit 213 are combined using the switched component.

The drive control unit 220 repeatedly switches the component α until the component α becomes 1. When the component α becomes 1, the component 1−α of the command value from the visual servoing unit 210 becomes 0. Therefore, the drive control unit 220 outputs the command value output from the first control unit 202 to the robot 10 . In this way, the motion control unit 101 moves the arm 11 (that is, the end point) according to the command value (step S1010 ). As a result, the endpoint is moved by position control. The processing of step S1010 is the same as that of step S1000.

In this way, at the stage of passing the object O1, the end point is moved by position control, thereby enabling high-speed processing. In addition, sudden movement and vibration of the arm 11 can be prevented by gradually switching the component α.

Next, the first control unit 202 judges the result of moving the endpoint by the position control, that is, whether the endpoint has passed the switching point 3 (step S1012 ). The information indicating the position of the switching point 3 is included in the preset route-related information. As shown in FIG. 28, the switching point 3 is set between the object O1 (switching point 2) and the object O2.

When the end point has not passed through the switching point 3 (No in step S1012), the control unit 20 repeats the process of step S1010.

In the case where the end point passes through switching point 3 (Yes in step S1012), the drive control unit 220 switches the component α stepwise as time elapses, and uses the switched component α to drive the output from the first control unit 202 to The output command value is combined with the command value output from the second control unit 213 and output to the robot 10 . In this way, the motion control unit 101 moves the arm 11 (that is, the end point) according to the command value (step S1014). The processing of step S1014 is the same as that of step S1004.

Next, the first control unit 202 judges whether or not the end point has passed the switching point 4 as a result of moving the end point by position control and visual servoing (step S1016 ). Information indicating the position of the switching point 4 is included in the route-related information. As shown in FIG. 28, the switching point 4 is set to the object O2.

When the end point has not passed through the switching point 4 (No in step S1016), the control unit 20 repeats the process of step S1014.

When the end point passes the switching point 4 (YES in step S1016), the drive control unit 220 switches the component α such that the component α gradually increases with the passage of time, and uses the switched component α , the command value output from the first control unit 202 and the command value output from the second control unit 213 are combined and output to the robot 10 . The motion control unit 101 moves the arm 11 (that is, the endpoint) according to the command value (step S1018). The processing of step S1018 is the same as that of step S1008.

The drive control unit 220 repeatedly switches the component α until the component α becomes 1. When the component α becomes 1, the drive control unit 220 outputs the command value output from the first control unit 202 to the robot 10 . In this way, the motion control unit 101 moves the arm 11 (that is, the end point) according to the command value (step S1020). The processing of step S1020 is the same as that of step S1010.

Next, the first control unit 202 judges the result of moving the endpoint by the position control, that is, whether the endpoint has reached the target point (step S1022 ). The information indicating the position of the target point is included in the preset route-related information.

When the end point has not reached the target point (NO in step S1022), the control unit 20 repeats the process of step S1020.

When the end point has reached the target point (YES in step S1022), the drive control unit 220 ends the process.

According to the present embodiment, when approaching an object, the end point is moved by position control and visual servoing, so that it can cope with a change in the position of the object. In addition, when the end point (current position) is farther away from the object than necessary, and when predetermined conditions such as the fact that the end point (current position) passes the object are satisfied, the end point can be moved only by position control. Perform high-speed processing.

In addition, according to the present embodiment, when switching between control based on position control and visual servoing, and control based on position control, sudden movement and vibration of the arm can be prevented by gradually switching the component α.

In addition, in the present embodiment, when the component α is gradually switched, the component α is switched by 0.1 every time a certain time elapses, but the method of gradually switching the component α is not limited to this. For example, as shown in FIG. 26, the components may be changed corresponding to the position to the object (corresponding to the target position in FIG. 26(A)) and the position away from the object (corresponding to the start position in FIG. 26B). alpha. In addition, as shown in FIG. 26 , the component α may change continuously (see lines C, D, etc. in FIGS. 26A and 26B ).

In addition, in this embodiment, when using command values for position control and visual servoing (steps S1004, S1008, S1014, and S1018), the component α is set to 0.5, 0.6, 0.7, 0.8, and 0.9, but the component α needs only It can be any value if it is a real number greater than 0 and less than 1.

Fourth Embodiment

The second and third embodiments of the present invention perform visual inspection using a hand-eye camera, but the scope of application of the present invention is not limited thereto.

The fourth embodiment of the present invention is a form in which the present invention is applied to assembly work such as insertion into a hole of an object. Hereinafter, a fourth embodiment of the present invention will be described. In addition, the same code|symbol is attached|subjected to the same part as 2nd Embodiment and 3rd Embodiment, and description is abbreviate|omitted.

FIG. 29 is a system configuration diagram showing an example of the configuration of the robot system 3 according to the embodiment of the present invention. The robot system 3 of the present embodiment mainly includes a robot 10A, a control unit 20 , a first imaging unit 30 , and a second imaging unit 40 .

The robot 10A is an arm-type robot having an arm 11A including a plurality of joints (joints) 12 and a plurality of links 13 . At the tip of the arm 11A, a hand 14 (so-called end effector) for holding the workpiece W or an instrument is provided. The position of the end point of the arm 11A is the position of the hand 14 . In addition, the end effector is not limited to the hand 14 .

A force sensor 102 (not shown in FIG. 29 , see FIG. 30 ) is provided on the arm portion of the arm 11A. The force sensor 102 is a sensor that detects force and moment received as a reaction force to the force output by the robot 10A. As the force sensor, for example, a 6-axis force sensor capable of simultaneously detecting 6 components of a force component in a translational three-axis direction and a moment component around a three-axis rotation can be used. In addition, the physical quantities used by the force sensor are current, voltage, charge, inductance, deformation, resistance, electromagnetic guidance, magnetism, air pressure, light, etc. The force sensor 102 can detect six components by converting a desired physical quantity into an electrical signal. In addition, the force sensor 102 is not limited to six axes, and may be three axes, for example.

Next, an example of the functional configuration of the robot system 3 will be described. FIG. 30 shows a functional block diagram of the robot system 3 .

The robot 10A includes a motion control unit 101 that controls the arm 11A based on an encoder value of an actuator, a sensor value of a sensor, and the like, and a force sensor 102 .

The control unit 20A mainly includes a position control unit 2000 , a visual servoing unit 210 , an image processing unit 212 , a drive control unit 220 , and a force control unit 230 .

The force control unit 230 performs force control (force sense control) based on sensor information (force information, moment information) from the force sensor 102 .

In this embodiment, impedance control is performed as force control. Impedance control is a position and force control means to set the mechanical impedance (inertia, attenuation coefficient, rigidity) generated when force is applied to the tip of the robot (hand 14) from the outside to an appropriate value for the target operation . Specifically, it is a control of contacting an object with mass, viscous coefficient, and elastic coefficient set as targets in a model in which mass, viscous coefficient, and elastic elements are connected to the end effector portion of the robot.

The force control unit 230 determines the movement direction and movement amount of the end point through impedance control. In addition, the force control unit 230 determines the target angle of each actuator provided on the joint 12 based on the moving direction and moving amount of the end point. Also, the force control unit 230 generates a command value for moving the arm 11A by a target angle, and outputs it to the drive control unit 220 . Note that since the processing performed by the force control unit 230 is general, a detailed description thereof will be omitted.

In addition, the force control is not limited to the hybrid control, and a control method capable of finely controlling the disturbance force, such as compliance control, can be employed. In addition, in order to perform force control, it is necessary to detect the force applied to the end effector such as the hand 14, but the method of detecting the force applied to the end effector is not limited to the case of using a force sensor. For example, the external force received by the end effector can also be estimated from the torque value of each axis of the arm 11A. Therefore, in order to perform force control, it is only necessary that the arm 11A has a mechanism for directly or indirectly acquiring the force applied to the end effector.

Next, the characteristic processing of the robot system 3 configured as described above in the present embodiment will be described. FIG. 31 is a flowchart showing the flow of control processing of the arm 11A of the present invention. This process is started, for example, by inputting a control start instruction via a button (not shown) or the like. In this embodiment, as shown in FIG. 32 , an assembly operation of inserting a workpiece W into a hole H will be described as an example.

When a control start instruction is input through a button (not shown) or the like, the first control unit 202 controls the arm 11 by position control to move the end point (step S130 ). The processing of step S130 is the same as that of step S1000.

In the present embodiment, the component of the command value by position control is α, the component of the command value by visual servoing is β, and the component of the command value by force control is γ. Components α, β, and γ are set such that their total is 1. In step S130, α is 1, and β and γ are 0.

Next, the first control unit 202 judges the result of moving the end point by the position control, that is, whether the end point has passed the switching point 1 (step S132 ). The processing of step S132 is the same as that of step S1002. The information indicating the position of the switching point 1 is included in the preset route-related information.

FIG. 32 is a diagram explaining the trajectory of the end point and the position of the switching point. In the present embodiment, the switching point 1 is provided at a predetermined predetermined position in the working space.

When the end point has not passed through the switching point 1 (No in step S132), the first control unit 202 repeats the process of step S130.

When the end point passes the switching point 1 (YES in step S132), the drive control unit 220 switches the components α and β stepwise with the passage of time, and uses the switched components α and β to convert from the first The command value output from the first control unit 202 is combined with the command value output from the second control unit 213 and output to the robot 10 . In this way, the motion control unit 101 moves the arm 11 (that is, the end point) according to the command value (step S134). That is, in step S134, the end point is moved by position control and visual servoing.

Hereinafter, the processing of step S134 will be specifically described. Before performing the process of step S134, that is, in the process of step S132, the drive control unit 220 sets the component α of the command value from the position control unit 200 to 1, and the component β of the command value from the visual servoing unit 210 to 0, The command value is synthesized by setting the component γ of the command value from the force control unit 230 to 0.

After the start of the process in step S134, if a certain period of time (for example, 10 msec) has elapsed, the drive control unit 220 switches the component α of the command value from the position control unit 2000 from 1 to 0.95, and switches the command value from the visual servoing unit 210 to 0.95. The value component β is switched to 0.05. Then, the drive control unit 220 synthesizes the command values of the command value from the position control unit 2000 and the command value from the visual servoing unit 210 to 0.05, and outputs the command value to the robot 10. .

After that, when a certain period of time has elapsed, the drive control unit 220 switches the component α of the command value from the position control unit 2000 from 0.95 to 0.9, and switches the component β of the command value from the visual servoing unit 210 from 0.05 to 0.1. .

In this way, the components α and β are switched stepwise with the elapse of a certain period of time, and the command value output from the first control unit 202 and the command value output from the second control unit 213 are combined using the switched components. The drive control unit 220 repeats the switching of the above components until the component α becomes 0.05 and the component β becomes 0.95. As a result, the endpoint is moved by position control and visual servoing. In addition, in step S134, since the force control is not used, the component γ is 0 as it is.

In addition, the ratio α:β of the final components α, β is not limited to 0.05:0.95. The components α, β can take various values in which the sum of the components α, β is 1. However, in such work, since the position of the hole H is not limited to be constant, it is preferable to make the component β of the visual servoing larger than the component α of the position control.

In addition, the method of gradually switching the component α is not limited to this. For example, as shown in FIGS. 26A and 26B , the component α may be changed according to the position to the object and the position away from the object. In addition, as shown by lines C and D in FIG. 26 , the component α may change continuously.

Next, the second control unit 213 judges whether or not the end point has passed the switching point 2 as a result of moving the end point by position control and visual servoing (step S136 ).

The switching point 2 is determined by the relative position from the hole H. For example, the switching point 2 is a position away from the center of the opening of the hole H by a distance L (for example, 10 cm). The position separated by the distance L from the center of the opening of the hole H can be set in a hemispherical shape in the x, y, and z spaces. In FIG. 32 , a position separated by a distance L from the center of the opening of the hole H in the z direction is shown as an example.

The image processing unit 212 extracts an image including the tip of the workpiece W and the hole H from the current image, and outputs the image to the second control unit 213 . In addition, the image processing unit 212 calculates the relationship between the distance in the image and the distance in real space based on the camera parameters (focal length, etc.) of the first imaging unit 30 or the second imaging unit 40 , and outputs the calculation to the second control unit 213 . The second control unit 213 judges whether the end point passes the switching point 2 based on the difference between the front end position of the workpiece W and the center position of the hole H in the extracted image.

When the end point has not passed the switching point 2 (No in step S136), the first control unit 202, the second control unit 213, and the drive control unit 220 repeat the process of step S134.

When the end point passes through switching point 2 (Yes in step S136), the drive control unit 220 synthesizes the command value output from the first control unit 202 and the command value output from the force control unit 230, and outputs the command value to the robot 10. . The motion control unit 101 moves the arm 11 (that is, the endpoint) according to the command value (step S138).

Hereinafter, the processing of step S138 will be specifically described. Before performing the process of step S138, that is, in the process of step S134, the drive control unit 220 sets the component α of the command value from the position control unit 2000 to 0.05, and sets the component β of the command value from the visual servoing unit 210 to 0.95. , so as to synthesize the instruction value.

After the process of step S138 is started, the drive control unit 220 switches the component α of the command value from the position control unit 2000 from 0.05 to 0.5. In addition, the component γ of the command value from the force control unit 230 is switched from 0 to 0.5. As a result, the drive control unit 220 sets the component α of the command value from the position control unit 2000 to 0.5, the component β of the command value from the visual servoing unit 210 to 0, and the component γ of the command value from the force control unit 230 to On the premise of 0.5, their command values are synthesized and output to the robot 10 . In addition, in step S138, since visual servoing is not used, the component β is 0 as it is. In addition, the components α and γ may be switched in stages.

Next, the force control unit 230 determines whether or not the end point has reached the target point as a result of moving the end point by visual servoing and force control (step S140 ). Based on the output of the force sensor 102, it can be determined whether or not the target point has been reached.

When the end point has not reached the target point (NO in step S140 ), the position control unit 200 , the force control unit 230 , and the drive control unit 220 repeat the process of step S138 .

When the end point has reached the target point (YES in step S140), the drive control unit 220 ends the processing.

According to the present embodiment, it is possible to maintain high speed of position control and to cope with different target positions. In addition, even when visual servoing cannot be used, such as when the target position cannot be confirmed, it is possible to maintain high-speed position control and perform work safely.

In addition, in this embodiment, switching point 1 is set in advance at an arbitrary position in the working space, and switching point 2 is set at a position separated from the hole H by a predetermined distance, but the positions of switching points 1 and 2 are not limited to this. . The positions of switching points 1 and 2 may also be set using the elapsed time from a predetermined position. Specifically, for example, the position of switching point 2 can be set 30 seconds after passing through switching point 1 . Alternatively, the positions of switching points 1 and 2 may be set using the distance from a predetermined position. Specifically, for example, the position of the switching point 1 can be set at a distance X from the start point. In addition, the positions of switching points 1 and 2 may be set based on an external signal input (for example, an input signal from the input device 25).

Fifth Embodiment

In the fifth embodiment of the present invention, assembly work such as insertion of an object into a hole is performed by position control and force control, but the scope of application of the present invention is not limited thereto.

The fifth embodiment of the present invention is a form in which the present invention is applied to assembly operations such as insertion of an object into a hole through position control, visual servoing, and force control. Hereinafter, a fifth embodiment of the present invention will be described. Since the structure of the robot system 4 of the fifth embodiment is the same as that of the robot system 3 , description thereof will be omitted. In addition, in the processing performed by the robot system 4 , the same parts as those in the second embodiment, the third embodiment, and the fourth embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted.

The characteristic processing of the robot system 4 of this embodiment will be described. FIG. 33 is a flowchart showing the flow of control processing of the arm 11A of the robot system 4 . This process is started, for example, by inputting a control start instruction via a button (not shown) or the like. In this embodiment, as shown in FIG. 34 , the assembly work of inserting the workpiece W into the hole H formed in the moving table will be described as an example.

When a control start instruction is input through a button (not shown) or the like, the first control unit 202 controls the arm 11A by position control to move the end point (step S130 ).

Next, the first control unit 202 judges the result of moving the end point by the position control, that is, whether the end point has passed the switching point 1 (step S132 ).

When the end point has not passed through the switching point 1 (No in step S132), the first control unit 202 repeats the process of step S130.

When the end point passes the switching point 1 (YES in step S132), the drive control unit 220 switches the components α and β stepwise with the passage of time, and uses the switched components α and β to convert from the first The command value output from the first control unit 202 is combined with the command value output from the second control unit 213 and output to the robot 10 . In this way, the operation control unit 101 moves the arm 11A (that is, the end point) according to the command value (step S134 ).

Next, the second control unit 213 judges whether or not the end point has passed the switching point 2 as a result of moving the end point by position control and visual servoing (step S136 ).

When the end point has not passed the switching point 2 (No in step S136), the first control unit 202, the second control unit 213, and the drive control unit 220 repeat the process of step S134.

When the end point passes through switching point 2 (YES in step S136), the drive control unit 220 outputs the command value output from the first control unit 202, the command value output from the second control unit 213, and the command value output from the force control unit 230. The output command values are synthesized and output to the robot 10 . The motion control unit 101 moves the arm 11A (that is, the end point) according to the command value (step S139 ).

Hereinafter, the processing of step S139 will be specifically described. Before the processing of step S139, that is, in the processing of step S134, the drive control unit 220 sets the component α of the command value from the position control unit 200 to 0.05, and sets the component β of the command value from the visual servoing unit 210 to 0.95. , so as to synthesize the instruction value.

After the process of step S139 is started, the drive control unit 220 switches the component α of the command value from the position control unit 2000 from 0.05 to 0.34. Also, the drive control unit 220 switches the component β of the command value from the visual servoing unit 210 from 0.95 to 0.33. Then, the drive control unit 220 switches the component γ of the command value from the force control unit 230 from 0 to 0.33. As a result, the drive control unit 220 sets the component α of the command value from the position control unit 2000 to 0.34, the component β of the command value from the visual servoing unit 210 to 0.33, and the component γ of the command value from the force control unit 230 to These command values are synthesized under the premise of 0.33 and output to the robot 10 .

In addition, the ratio α:β:γ of the components α, β, γ is not limited to 0.34:0.33:0.33. Components α, β, γ can be set to various values in which the sum of components α, β, γ is 1 according to the job. Alternatively, the components α, β, and γ may be switched gradually.

Next, the force control unit 230 determines whether or not the end point has reached the target point as a result of moving the end point by visual servoing and force control (step S140 ).

When the end point has not reached the target point (No in step S140), the position control unit 2000, the visual servoing unit 210, the force control unit 230, and the drive control unit 220 repeat the process of step S139.

When the end point has reached the target point (YES in step S140), the drive control unit 220 ends the process.

According to the present embodiment, while maintaining the high speed of position control, it is possible to move the end point to a different target position. In particular, even when the target position is moving and the target position cannot be confirmed, since control is performed by position control, visual servoing, and force control, it is possible to maintain high-speed position control and perform work safely.

In addition, in the present embodiment, the arm is controlled by simultaneously performing position control, visual servoing, and force control (parallel control), but in the fifth embodiment, the arm is controlled by simultaneously performing position control and force control (parallel control). . The drive control unit 220 can select whether to simultaneously perform position control, visual servoing, Force control, or simultaneous position control and force control.

In the above embodiment, the case of using a single-arm robot was described, but the present invention can also be applied to the case of using a dual-arm robot. In the above-mentioned embodiment, the case where the endpoint is provided at the tip of the arm of the robot has been described, but the term provided on the robot is not limited to being provided on the arm. For example, the robot may be provided with a manipulator that is composed of a plurality of joints and links and moves as a whole by moving the joints, and the end of the manipulator may be used as an end point.

In addition, in the above-described embodiment, two imaging units of the first imaging unit 30 and the second imaging unit 40 are provided, but there may be one imaging unit.

As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range described in said embodiment. It is obvious to those skilled in the art that various changes or improvements can be added to the above-described embodiments. In addition, it is clear from the description of the claims that an embodiment with such a change or improvement can also be included in the technical scope of the present invention. In particular, the present invention can provide a robot system that is respectively provided with a robot, a control unit, and an imaging unit, can provide a robot that includes a control unit, etc., or can provide a robot controlled by only a control unit or a control unit and an imaging unit. device. In addition, the present invention can also provide a program for controlling a robot or the like, and a storage medium storing the program.

Sixth Embodiment

1. The means of this embodiment

Robot control using image information is widely known. For example, visual servoing control is known in which image information is continuously acquired, and a result of comparison processing between information acquired from the image information and target information is fed back. In visual servoing, the robot is controlled such that the difference between the information acquired from the latest image information and the target information becomes smaller. More specifically, control is performed in which the amount of change in the joint angle to approach the target is obtained, and the joint is driven based on the amount of change or the like.

In the means of giving the position and posture of the tip of the hand of the target robot, and controlling the robot so as to form the target position and posture, it is difficult to improve the positioning accuracy, that is, it is difficult to accurately align the tip of the hand (hand) to the target. Position posture movement. Ideally, if the model of the robot is determined, the position and posture of the tip of the hand can be uniquely obtained based on the model. The model here refers to, for example, information such as the length of a frame (link) provided between two joints, the structure of the joint (rotation direction of the joint, presence or absence of offset, etc.).

However, robots contain various errors. For example, variations in the length of the connecting rod, deflection due to gravity, and the like. Due to these error factors, when performing control to make the robot take a given posture (for example, control to determine the angle of each joint), the ideal position and posture will have different values from the actual position and posture.

In this regard, in the visual servoing control, since the image processing result of the captured image is fed back, it is the same as the case where people can fine-tune the movement direction of the arms and hands while observing the work situation with the eyes. It is also possible to recognize and correct the deviation between the current position and posture and the position and posture of the target.

In the visual servoing control, three-dimensional position and posture information such as the hand tip of the robot can be used as the above-mentioned "information acquired from the image" and "target information", and image feature values acquired from the image can also be used instead of It is converted into position and pose information. Visual servoing using position and posture information is called position-based visual servoing, and visual servoing using image feature values is called feature-based visual servoing.

In order to properly perform visual servoing, it is necessary to accurately detect position and posture information or image feature values from image information. If the accuracy of this detection process is low, the current state may be erroneously recognized. Therefore, the information fed back to the control loop does not become the information that brings the state of the robot appropriately close to the target state, and high-precision robot control cannot be realized.

It is assumed that position and orientation information and image feature quantities are obtained through some detection processing (for example, matching processing), but the accuracy of this detection processing is not necessarily sufficient. This is because in the environment where the robot actually operates, not only the object to be recognized (for example, the robot's hand) but also workpieces, jigs, or objects placed in the operating environment are captured in captured images. Since various objects fall into the background of the image, the recognition accuracy (detection accuracy) of a desired object is lowered, and the accuracy of obtained position and orientation information and image feature values is also lowered.

Patent Document 1 discloses a method for detecting an abnormality by comparing a spatial position or movement speed calculated from an image with a spatial position or movement speed calculated from an encoder in position-based visual servoing. In addition, since the position in space is information included in the position and posture information, and the movement speed is also information obtained from the amount of change in the position and posture information, the following description will describe the position or movement speed in space as the position and posture information.

It is conceivable that by using the method of Patent Document 1, when some abnormality occurs in visual servoing, such as a large error in the position and posture information obtained from the image information, the abnormality can be detected. If abnormality detection can be realized, the control of the robot can be stopped, or the detection of the position and posture information can be resumed, thereby at least suppressing the use of abnormality information as it is in the control.

However, the method of Patent Document 1 is based on the premise of position-based visual servoing. In the case of a position reference, as described above, it is only necessary to perform a comparison process between position and posture information easily obtained from information such as an encoder and position and posture information obtained from image information, so implementation is easy. On the other hand, in visual servoing based on feature quantities, image feature quantities are used for robot control. Furthermore, even if it is easy to obtain the spatial position of the hand tip of the robot from information such as an encoder, it is not possible to directly obtain the relationship with the image feature value. That is, it is difficult to apply the method of Patent Document 1 when visual servoing based on a feature quantity is assumed.

Therefore, the present applicant has proposed a means of using, in control using image feature amounts, the amount of change in image feature amounts actually acquired from image information and the estimated change in image feature amounts estimated from information acquired from the results of robot control. amount to detect anomalies. Specifically, as shown in FIG. 35 , the robot control device 1000 of the present embodiment includes a robot control unit 1110 that controls the robot 20000 based on image information; a change calculation unit 1120 that calculates the change of image feature values based on image information; The change amount estimating unit 1130 that calculates the estimated amount of image feature amount change, that is, the estimated image feature amount change amount, based on the information for estimating the amount of change that is information on the robot 20000 or the object and is information other than image information; and The abnormality determination unit 1140 performs abnormality determination by comparing the change amount of the image feature amount and the estimated image feature amount change amount.

Here, as described above, the image feature amount is an amount representing features such as the region, area, line segment length, and feature point position in the image, and the image feature amount change amount represents the number of images from multiple (in a narrow sense, two) image information. Information on changes among the acquired plurality of image feature quantities. As an image feature quantity, if it is an example of a two-dimensional position on an image using three feature points, the image feature quantity is a 6-dimensional vector, and the amount of change in the image feature quantity is the difference between two 6-dimensional vectors, that is, The difference between the elements of the vector is the 6-dimensional vector of the elements.

In addition, the change amount estimating information is information used for estimating the change amount of the image feature amount, and is information other than image information. The change amount estimating information may be, for example, information acquired (actually measured) from the results of robot control, and specifically, joint angle information of the robot 20000 may be used. The joint angle information can be obtained from encoders that measure and control the motion of motors (in a broad sense, actuators) for driving joints of the robot. Alternatively, the change amount estimating information may be the end effector 2220 of the robot 20000 or the position and posture information of the object of the work performed by the robot 20000 . The position and posture information is, for example, a six-dimensional vector including the three-dimensional position (x, y, z) of the reference point of the object and the rotation (R1, R2, R3) about each axis relative to the reference posture. Various means of obtaining the position and posture information of an object are conceivable, but for example, the following means may be used: means of distance measurement using ultrasonic waves, means of using a measuring instrument, means of placing an LED on the tip of the hand and detecting the LED to perform measurement, Means using a mechanical three-dimensional measuring device, etc.

In this way, abnormality can be detected in robot control using image feature values (visual servoing based on feature values in a narrow sense). At this time, a comparison process is performed between the image feature amount change obtained from actually acquired image information and the estimated image feature amount change obtained from change estimation information obtained from a viewpoint different from the image information.

In addition, robot control based on image information is not limited to visual servoing. For example, in visual servoing, information feedback based on image information to the control loop is continuously performed, but it is also possible to acquire image information once and calculate the movement amount for the target position and posture based on the image information, and then The vision method, which controls the position by the amount of movement, is used for robot control based on image information. In addition to visual servoing and visual methods, the means of this embodiment can also be applied as a means of detecting abnormality in the control of a robot using image information, detection of information from image information, and the like.

However, as will be described later, in the means of this embodiment, it is assumed that a Jacobian matrix is used for the calculation of the estimated change amount of the image feature value. Furthermore, the Jacobian matrix is information indicating the relationship between the amount of change in a given value and the amount of change in other values. For example, even if the first information x and the second information y have a nonlinear relationship (g in y=g(x) is a nonlinear function), it can be considered that the amount of change Δx of the first information is around a given value. There is a linear relationship with the change amount Δy of the second information (h in Δy=h(Δx) is a linear function), and the Jacobian matrix represents the linear relationship. That is, in the present embodiment, it is assumed that the image feature amount change amount is used instead of the image feature amount itself in the processing. Therefore, when the means of this embodiment is applied to control other than visual servoing, such as visual methods, it should be noted that it is not necessary to use the means of acquiring image information only once, but to obtain image feature values. It is a means to obtain image information at least twice or more in the way of variation. For example, when the means of this embodiment is applied to a visual system, it is necessary to perform acquisition of image information and calculation of a target movement amount multiple times.

Hereinafter, the outline of the visual servoing will be described after describing a system configuration example of the robot control device 1000 and the robot according to the present embodiment. On this premise, the abnormality detection means of the present embodiment will be described, and finally a modified example will also be described. In addition, in the following, visual servoing is used as an example of robot control using image information, but the following description can be extended to robot control using other image information.

2. Example of system configuration

A detailed system configuration example of the robot controller 1000 according to this embodiment is shown in FIG. 36 . However, the robot control device 1000 is not limited to the configuration shown in FIG. 36 , and various modifications such as omitting some of the above-mentioned components or adding other components are possible.

As shown in FIG. 36 , the robot control device 1000 includes a target feature input unit 111, a target trajectory generation unit 112, a joint angle control unit 113, a drive unit 114, a joint angle detection unit 115, an image information acquisition unit 116, and an image feature calculation unit. unit 117 , a change calculation unit 1120 , a change estimation unit 1130 , and an abnormality determination unit 1140 .

The target feature value input unit 111 inputs the target image feature value fg to the target trajectory generation unit 112 . The target feature value input unit 111 may be realized as an interface or the like that accepts input of the target image feature value fg by the user, for example. In the robot control, control is performed to bring the image feature value f obtained from the image information close to the input target image feature value fg (in a narrow sense, to make them match). In addition, image information (reference image, target image) corresponding to the target state may be obtained, and the feature value fg of the target image may be obtained from the image information. Alternatively, instead of holding the reference image, the input of the feature value fg of the target image may be directly accepted.

The target trajectory generation unit 112 generates a target trajectory for operating the robot 20000 based on the target image feature value fg and the image feature value f obtained from the image information. Specifically, processing is performed to obtain the amount of change Δθg of the joint angle for bringing the robot 20000 closer to the target state (state corresponding to fg). This Δθg becomes a tentative target value of the joint angle. In addition, in the target trajectory generating unit 112, the driving amount of the joint angle per unit time may be obtained from Δθg (dotted θg in FIG. 36 ).

The joint angle control unit 113 controls the joint angle based on the target value Δθg of the joint angle and the current value θ of the joint angle. For example, since Δθg is the amount of change in the joint angle, a process of finding what value the joint angle can be is performed using θ and Δθg. The drive unit 114 performs control to drive the joints of the robot 20000 following the control of the joint angle control unit 113 .

The joint angle detection unit 115 performs a process of detecting what the joint angle of the robot 20000 is. Specifically, after the joint angle is changed by drive control by the drive unit 114 , the changed joint angle value is detected, and the current joint angle value is θ, which is output to the joint angle control unit 113 . Specifically, the joint angle detection unit 115 may be realized as an interface or the like for acquiring encoder information.

The image information acquisition unit 116 acquires image information from an imaging unit or the like. The imaging unit here may be an imaging unit disposed in the environment as shown in FIG. 37 , or may be an imaging unit (for example, a hand-eye camera) provided on the arm 2210 of the robot 20000 or the like. The image feature calculation unit 117 performs calculation processing of the image feature based on the image information acquired by the image information acquisition unit 116 . In addition, various means such as edge detection processing and matching processing are known as means for calculating image feature values based on image information, and can be widely applied in this embodiment, so detailed description is omitted. The image feature value obtained by the image feature value calculation unit 117 is output to the target trajectory generation unit 112 as the latest image feature value f.

The change calculation unit 1120 holds the image feature calculated by the image feature calculation unit 117, and uses the image feature f _old obtained in the past and the image feature f (in a narrow sense, the latest image feature) as the processing target. ) to calculate the image feature quantity variation Δf.

The change amount estimating unit 1130 holds the joint angle information detected by the joint angle detecting unit 115, and uses joint angle information θ _old acquired in the past, and joint angle information θ to be processed (the latest joint angle information in a narrow sense) The difference of the joint angle information is calculated on the change amount Δθ. Then, based on Δθ, the amount of change in the estimated image feature value Δfe is obtained. In addition, in FIG. 36 , an example in which the change amount estimation information is joint angle information has been described, but as described above, the end effector 2220 of the robot 20000 or the position of the object may be used as the change amount estimation information. Posture information.

In addition, the robot control unit 1110 in FIG. 35 may also be the same as the target feature quantity input unit 111, the target trajectory generation unit 112, the joint angle control unit 113, the drive unit 114, the joint angle detection unit 115, and the image information acquisition unit 116 in FIG. , and a control unit corresponding to the image feature calculation unit 117 .

In addition, as shown in FIG. 38 , the means of this embodiment can be applied to a robot including a robot control system that controls the robot (specifically, the robot main body 3000 including the arm 2210 and the end effector 2220 ) based on image information. part 1110; a change amount calculating part 1120 for obtaining a change amount of an image feature value from image information; and an image feature value change amount based on information for estimating change amount which is information of the robot 20000 or an object and which is information other than image information. The change amount estimating unit 1130 calculates the estimated amount of the estimated image feature value, that is, the change amount of the estimated image feature value;

As shown in FIGS. 19A and 19B , the robot here may also be a robot including a control device 600 and a robot body 300 . 19A and 19B, the control device 600 includes the robot control unit 1110 of FIG. 38 and the like. In this way, it is possible to perform actions based on control based on image information, and realize a robot that automatically detects abnormalities in control.

In addition, the structural example of the robot of this embodiment is not limited to FIG. 19A, FIG. 19B. For example, as shown in FIG. 39 , the robot may include a robot main body 3000 and a base unit part 350 . The robot of this embodiment may also be a dual-arm robot as shown in FIG. 39 , which includes a first arm 2210 - 1 and a second arm 2210 - 2 in addition to parts corresponding to the head and torso. In FIG. 39 , the first arm 2210 - 1 is composed of joints 2211 , 2213 and frames 2215 , 2217 provided between the joints. The same applies to the second arm 2210 - 2 , but the present invention is not limited thereto. In addition, although the example of the dual-arm robot which has two arms is shown in FIG. 39, the robot of this embodiment may have three or more arms.

The base unit part 350 is provided at a lower portion of the robot main body 3000 and supports the robot main body 3000 . In the example of FIG. 39 , wheels and the like are provided on the base unit portion 350 so that the entire robot can move. However, the base unit portion 350 may be fixed to the ground or the like without having wheels or the like. In FIG. 39, the device corresponding to the control device 600 of FIGS. 19A and 19B is not shown, but in the robot system of FIG. The device 600 is constructed as a whole.

Alternatively, like the control device 600 , the robot control unit 1110 and the like may be realized by a substrate built into the robot (more specifically, an IC or the like provided on the substrate) without providing a specific control device.

In addition, as shown in FIG. 20 , the functions of the robot controller 1000 may be realized by a server 500 communicatively connected to the robot via a network 400 including at least one of wired and wireless.

Alternatively, in the present embodiment, the server 500 as the robot control device may perform a part of the processing of the robot control device of the present invention. In this case, the processing is realized by distributed processing with the robot controller provided on the robot side.

In addition, in this case, the server 500 serving as the robot control device performs processing assigned to the robot control system of the server 500 among the respective processes of the robot control device of the present invention. On the other hand, the robot control device provided in the robot performs processing assigned to the robot control device of the robot among the respective processes of the robot control device of the present invention.

For example, the robot controller of the present invention performs the first to Mth (M is an integer) processing, and it is considered that the first processing can be realized by sub-processing 1a and sub-processing 1b, and the second processing can be realized by sub-processing 2a and sub-processing 1b. The way to realize the process 2b is to divide each of the first to Mth processes into a plurality of sub-processes. In this case, it is considered that the server 500 as the robot control device performs sub-processing 1a, sub-processing 2a, ... sub-processing Ma, and the robot control device installed on the robot side performs sub-processing 1b, sub-processing 2b, ... sub-processing Distributed processing that deals with Mb. At this time, the robot control device of this embodiment, that is, the robot control device that executes the first to Mth processes may be a robot control device that executes sub-process 1a to sub-process Ma, or a robot that executes sub-process 1b to sub-process Mb. The control device may be a robot control device that executes all of sub-processing 1a to sub-processing Ma and sub-processing 1b to sub-processing Mb. Furthermore, the robot control device of this embodiment is a robot control device that executes at least one sub-process for each of the first to M-th processes.

Thereby, for example, the server 500 having a higher processing capability than a terminal device on the robot side (for example, the control device 600 in FIGS. 19A and 19B ) can perform processing with a high processing load. In addition, the server 500 can collectively control the operation of each robot, so that, for example, it is easy to coordinate the operation of a plurality of robots.

In addition, in recent years, there is an increasing tendency to manufacture many types of parts with a small number of parts. Furthermore, when changing the type of parts to be manufactured, it is necessary to change the operation performed by the robot. With the configuration shown in FIG. 20 , the server 500 can collectively change the actions performed by the robots without re-performing the instruction work for each of the plurality of robots. In addition, compared with the case where one robot control device 1000 is provided for each robot, it is possible to significantly reduce troubles and the like at the time of updating the software of the robot control system 1000 .

3. Visual Servo Control

Before describing the abnormality detection means of this embodiment, general visual servoing control will be described. A configuration example of a general visual servoing control system is shown in FIG. 40 . As can be seen from FIG. 40 , when compared with the robot controller 1000 of the present embodiment shown in FIG. 36 , the change calculation unit 1120 , the change estimation unit 1130 , and the abnormality determination unit 1140 are excluded.

When the dimensionality of the image feature amount used for visual servoing is set to n (n is an integer), the image feature amount f is an image feature amount vector as f=[f1, f2, . . . , fn]T And show it. Each element of f may use, for example, coordinate values of an image of a feature point (control point), or the like. In this case, the target image feature value fg input from the target feature value input unit 111 is similarly expressed as fg=[fg1, fg2, . . . , fgn]T.

In addition, the joint angle is also expressed as a joint angle vector having a dimension corresponding to the number of joints included in the robot 20000 (in a narrow sense, the arm 2210 ). For example, if the arm 2210 is a six-degree-of-freedom arm having six joints, the joint angle vector θ is represented by θ=[θ1, θ2, . . . , θ6]T.

In visual servoing, when the current image feature f is acquired, the difference between the image feature f and the target image feature fg is fed back to the motion of the robot. Specifically, the robot is operated in a direction to reduce the difference between the image feature value f and the target image feature value fg. For this reason, it is necessary to know the relationship that how the image feature value f changes by moving the joint angle θ. Generally, this relationship is nonlinear. For example, in the case of f1=g(θ1, θ2, θ3, θ4, θ5, θ6), the function g is a nonlinear function.

Therefore, in visual servoing, a method using the Jacobian matrix J is widely known. Even if the two spaces are in a nonlinear relationship, the slight changes in each space can also be expressed in a linear relationship. The Jacobian matrix J is a matrix for associating the above-mentioned minute variations with each other.

Specifically, when the position and posture X of the tip of the robot 20000 is X=[x, y, z, R1, R2, R3]T, the jacques between the amount of change in the joint angle and the amount of change in the position and posture The ratio matrix Ja is expressed by the following equation (1), and the Jacobian matrix Ji between the amount of change in the position and orientation and the amount of change in the image feature amount is expressed by the following equation (2).

Formula 1

${\mathrm{<span\; class="notranslate">\; J</span>}}_{\mathrm{<span\; class="notranslate">\; a</span>}}<span\; class="notranslate">\; =</span>\left[\begin{array}{cccccc}\frac{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; x</span>}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; x</span>}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; x</span>}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; x</span>}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; x</span>}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 5</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; x</span>}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 6</span>}}}\\ \frac{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 5</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 6</span>}}}\\ \frac{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; z</span>}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; z</span>}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; z</span>}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; z</span>}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; z</span>}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 5</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; z</span>}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 6</span>}}}\\ \frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 5</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 6</span>}}}\\ \frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 5</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 6</span>}}}\\ \frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 5</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; 6</span>}}}\end{array}\right]<span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>$

Math 2

${\mathrm{<span\; class="notranslate">\; J</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>\left[\begin{array}{cccccc}\frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; x</span>}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; z</span>}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}}\\ \frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; x</span>}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; z</span>}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}}\\ <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>\\ <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>\\ <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>\\ \frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; x</span>}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; z</span>}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}& \frac{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}}{<span\; class="notranslate">\; \&PartialD;</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}}\end{array}\right]<span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; )</span>$

Furthermore, by using Ja and Ji, the relationship of Δθ, ΔX, and Δf can be expressed as shown in the following equations (3) and (4). Ja is generally called a robot Jacobian matrix, and Ja can be analytically calculated if there is mechanism information such as the link length and the rotation axis of the robot 20000 . On the other hand, Ji can be estimated in advance from changes in image feature values when the position and posture of the tip of the robot 20000 is slightly changed, and a means for estimating Ji at any time during the movement has also been proposed.

ΔX＝JaΔθ·····(3)

Δf=JiΔX·····(4)

Furthermore, by using the above equations (3) and (4), the relationship between the amount of change Δf of the image feature value and the amount of change Δθ of the joint angle can be expressed as shown in the following equation (5).

Δf＝JvΔθ·····(5)

Here, Jv=JiJa, and represents the Jacobian matrix between the amount of change in the joint angle and the amount of change in the image feature amount. In addition, Jv is also expressed as an image Jacobian matrix. The relationship between the above formulas (3) to (5) is illustrated in FIG. 41 .

Based on the above, the target trajectory generation unit 112 may obtain the driving amount of the joint angle (change amount of the joint angle) Δθ using the difference between f and fg as Δf. In this way, the amount of change in the joint angle for bringing the image feature value f closer to fg can be obtained. Specifically, in order to obtain Δθ from Δf, it is sufficient to multiply both sides of the above equation (5) from the left by the inverse matrix Jv-1 of Jv, but further considering the control gain as λ, the following equation (6) can be used to obtain The target joint angle variation Δθg is obtained.

Δθg＝－λJv ^-1 (f－fg)·····(6)

In addition, the inverse matrix Jv ^-1 of Jv is obtained in the above formula (6), but when Jv ^-1 is not obtained, the generalized inverse matrix (pseudo inverse matrix) Jv# of Jv may be used.

By using the above formula (6), a new Δθg is obtained every time a new image is acquired. Accordingly, it is possible to perform control close to the target state (state in which the image feature value is fg) while updating the target joint angle using the acquired image. This flow is illustrated diagrammatically in FIG. 42 . When the image feature value f _m-1 is obtained from the m-1th image (m is an integer), Δθg _m-1 can be obtained by forming f=f _m-1 in the above formula (6). Then, between the m−1th image and the next image, that is, the mth image, the robot 20000 may be controlled with the calculated Δθg _m−1 as a target. Then, when the m-th image is obtained, the image feature value fg is obtained from the m-th image, and Δθg _m as a new target is calculated using the above formula (6). Between the m-th image and the m+1-th image, the calculated Δθg _m is used for control. Hereinafter, the processing may be continued until the processing is terminated (until the image feature value is sufficiently close to fg).

In addition, although the target change amount of the joint angle is obtained, it is not necessarily necessary to change the joint angle by the target amount. For example, between the m-th image and the m+1-th image, Δθg _m is controlled as the target value, but the following situation is often considered: when the actual amount of change has not yet reached Δθg _m , the next image is acquired, that is, The m+1th image, and calculate the new target value Δθg _m+1 through it.

4. Anomaly detection means

The abnormality detection means of this embodiment is demonstrated. As shown in FIG. 43A , when the joint angle of the robot 20000 is θp, the p-th image information is acquired, and the image feature value fp is calculated based on the p-th image information. Then, when the joint angle of the robot 20000 is θq at a time later than the acquisition time of the p-th image information, the q-th image information is acquired, and the image feature value fq is calculated from the q-th image information. Here, the p-th image information and the q-th image information may be adjacent image information in time series, or may be non-adjacent (after the acquisition of the p-th image information and before the acquisition of the q-th image information, other image information ) image information.

In visual servoing, the difference between fp, fq, and fg is used for calculation of Δθg as Δf as described above, but the difference fq−fp between fp and fq is nothing more than an amount of image feature change. In addition, since the joint angle θp and the joint angle θq are obtained by the joint angle detection unit 115 from an encoder or the like, they can be obtained as actually measured values, and the difference θq−θp between θp and θq is the change amount Δθ of the joint angle. That is, for two pieces of image information, in order to obtain the corresponding image feature value f and joint angle θ respectively, the change amount of the image feature value is obtained as Δf=fq-fp, and the change amount of the corresponding joint angle is obtained as Δθ=θq-θp.

Furthermore, as shown in the above formula (5), there is a relationship of Δf=JvΔθ. That is, when Δfe=JvΔθ is obtained using the measured Δθ=θq−θp and the Jacobian matrix Jv, the obtained Δfe should agree with the actually measured Δf=fq−fp in an ideal environment where no error occurs.

Thus, the change amount estimating unit 1130 acts on the change amount of the joint angle information to associate the joint angle information with the image feature amount (specifically, associate the joint angle information change amount with the image feature amount change). The matrix Jv is used to calculate the estimated image feature quantity change Δfe. As described above, in an ideal environment, the obtained estimated image feature quantity change Δfe should agree with the image feature quantity change Δf obtained by the change calculation unit 1120 as Δf=fq−fp, and vice versa. , when Δf is significantly different from Δfe, it can be determined that some abnormality has occurred.

Here, as factors causing errors in Δf and Δfe, errors in computing image feature quantities based on image information, errors in reading joint angle values by encoders, errors contained in the Jacobian matrix Jv, and the like are considered. However, when the encoder reads the value of the joint angle, the possibility of error is lower when compared to the other two. In addition, the error contained in the Jacobian matrix Jv is not a large error. On the other hand, since many objects that are not objects of recognition are captured in the image, errors occur more frequently when the image feature value is calculated based on the image information. In addition, when an abnormality occurs in the calculation of the image feature value, there is a possibility that the error becomes very large. For example, if the recognition process for recognizing a desired object from an image fails, there is a possibility that the object may be mistakenly recognized as being present at a position on the image that is greatly different from the original object position. Therefore, in the present embodiment, abnormalities in the calculation of image feature quantities are mainly detected. However, errors caused by other factors can also be detected as abnormalities.

In abnormality determination, for example, a determination process using a threshold value may be performed. Specifically, the abnormality determination unit 1140 compares the difference information between the image feature amount change amount Δf and the estimated image feature amount change amount Δfe with a threshold, and determines that it is abnormal when the difference information is larger than the threshold value. For example, if a predetermined threshold Th is set and the following formula (7) is satisfied, it may be determined that an abnormality has occurred. In this way, an abnormality can be detected by a simple calculation such as the following formula (7).

|Δf－Δfe|＞Th·····(7)

In addition, the threshold Th does not need to be a fixed value, and the value may be changed according to the situation. For example, the abnormality determination unit 1140 may be configured to set a larger threshold value as the difference between the acquisition times of the two pieces of image information used in the calculation of the change amount of the image feature in the change amount calculation unit 1120 is greater.

As shown in FIG. 41 and the like, the Jacobian matrix Jv is a matrix linking Δθ and Δf. Furthermore, as shown in FIG. 44, even when the same Jacobian matrix Jv is applied, the amount of change in Δfe' obtained by acting on Δθ' greater than the amount of change in Δθ is compared with Δfe obtained by acting on Δθ. larger. At this time, it is difficult to consider that the Jacobian matrix Jv does not generate any error at all. Therefore, as shown in FIG. 44 , Δfe, Δfe' produces a bias. Furthermore, as can be seen from the comparison of A1 and A2 in FIG. 44 , the larger the amount of change, the larger the deviation.

Assuming that no error occurs in the calculation of the image feature, the image feature change Δf obtained from the image information is equal to Δfi and Δfi'. In this case, the left side of the above formula (7) represents an error due to the Jacobian matrix, and when the amount of change is small such as Δθ, Δfe, etc., it becomes a value equivalent to A1, such as Δθ', Δfe', etc. Thus, when the amount of change is large, it becomes a value equivalent to A2. However, as described above, the Jacobian matrix Jv used for both Δfe and Δfe' is the same, and although the value on the left side of the above formula (7) becomes larger, it is determined that A2 has a higher degree of abnormality than A1 status is inappropriate. That is, it cannot be said that the above formula (7) is not satisfied (not judged as abnormal) in the situation corresponding to A1, and it is appropriate to satisfy the above formula (7) (judged as abnormal) in the situation corresponding to A2. Therefore, in the abnormality determination unit 1140 , the threshold value Th is further set to be larger as the amount of change in Δθ, Δfe, etc. is larger. In this way, since the threshold Th is larger in the situation corresponding to A2 than in the situation corresponding to A1, appropriate abnormality determination can be performed. Considering that the difference between the acquisition times of the two image information (the p-th image information and the q-th image information in FIG. 43A ) is larger, Δθ, Δfe, etc. are also larger. It is sufficient to set the threshold Th correspondingly to the difference.

In addition, various controls in the case where an abnormality is detected in the abnormality determination unit 1140 are considered. For example, when an abnormality is detected by the abnormality determination unit 1140 , the robot control unit 1110 may perform control to stop the robot 20000 . As described above, the case where an abnormality is detected is, for example, a case where a large error occurs in the calculation of the image feature value from the image information. That is, if the robot 20000 is controlled using the image feature value (fq in the example of FIG. 43A ), there is a possibility that the robot 20000 will be far away from the direction in which the image feature value is close to the target image feature value fg. Possibility of moving. In this case, there is a possibility that the arm 2210 and the like collide with other objects, and the object grasped by the hands and the like may fall due to taking an unreasonable posture. Therefore, as an example of control at the time of abnormality, it is conceivable to stop the operation itself of the robot 20000 so as not to perform such a risky operation.

In addition, if it is estimated that a large error occurs in the image feature value fq and it is not desired to perform control using fq, the robot may not be stopped immediately and fq may not be used for control. Thus, for example, when an abnormality is detected by the abnormality determination unit 1140 , the robot control unit 1110 may skip one of the two pieces of image information used in the calculation based on the change amount of the image feature in the change amount calculation unit 1120 . Control based on image information acquired at a later time in time series, that is, abnormality determination image information, is performed based on image information acquired earlier than the abnormality determination image information.

In the example shown in FIG. 43A , the abnormality determination image information is the qth image. In addition, in the example of FIG. 42 , abnormality determination is performed using two pieces of adjacent image information, and it is determined that there is no abnormality in the m-2th image information and the m-1th image information. There is no abnormality in the mth image information, and there are abnormalities in the mth image information and the m+1th image information. In this case, it can be considered that f _m-1 and f _m are not abnormal, but f _m+1 is abnormal, so Δθg _m-1 and Δθg _m can be used for control, but it is not correct to use Δθg _m+1 for control appropriate. Originally, Δθg _{m +1} is used for control between the m+1th image information and the next m+2th image information, but here, this control is not appropriate because it is not performed. In this case, between the m+1th image information and the m+2th image information, the robot 20000 may be operated using the previously obtained Δθg _m . Since Δθg _m is at least information for moving the robot 20000 in the target direction at the time of calculation of f _m , it is unlikely that a large error will occur even if it is used continuously after the calculation of f _m+1 . In this way, even when an abnormality is detected, it is possible to roughly control the operation of the robot 20000 by using the information before that, especially the information obtained at a time earlier than the abnormality detection time and in which no abnormality was detected. keep going. Afterwards, when new image information is acquired (the m+2th image information in the example of FIG. 42 ), control may be performed using new image feature values obtained from the new image information.

In the flowchart of FIG. 48 , the processing flow of this embodiment considered up to abnormality detection is shown. When this process is started, the acquisition of the image by the image information acquisition unit 116 and the calculation of the image feature by the image feature calculation unit 117 are first performed, and the variation of the image feature is changed in the change calculation unit 1120. Quantity calculation (S10001). In addition, the detection of the joint angle by the joint angle detection unit 115 is performed, and the change amount of the estimated image feature value is estimated in the change amount estimating unit 1130 ( S10002 ). Then, an abnormality determination is performed based on whether the difference between the change amount of the image feature amount and the estimated image feature amount change amount is equal to or less than a threshold value (S10003).

When the difference is equal to or less than the threshold (YES in S10003), no abnormality occurs, and control is performed using the image feature value obtained in S10001 (S10004). Then, it is determined whether or not the current image feature value is sufficiently close to the target image feature value (in a narrow sense, coincides). If yes, the target is reached normally, and the process ends. On the other hand, in the case of No in S10005, the operation itself does not generate an abnormality, but the target has not been reached, and the control returns to S10001 to continue.

In addition, when the difference between the amount of change in the image feature amount and the amount of change in the estimated image feature amount is larger than the threshold (NO in S10003 ), it is determined that an abnormality has occurred. Then, it is determined whether the occurrence of the abnormality is N consecutive times (S10006), and if it occurs continuously, it is an abnormality to such an extent that it is not preferable to continue the operation, and the operation is stopped. On the other hand, when the occurrence of the abnormality is not N consecutive times, control is performed using the past image feature value determined not to be abnormal (S10007), and the image processing at the next time is continued by returning to S10001. In the flowchart of FIG. 48 , as described above, before a certain level of abnormality occurs (here, N-1 or less consecutive abnormalities occur), the operation is not stopped immediately, but the operation is controlled in the direction of continuing the operation.

In addition, in the above description, the time difference between the acquisition time of the image information, the acquisition time of the joint angle information, and the acquisition time of the image feature amount (calculation end time) is not particularly taken into consideration. However, actually, as shown in FIG. 45 , even if image information is acquired at a given timing, the encoder reads the joint angle information at the time of acquisition of the image information and transmits the read information to the joint angle detection section 115. There will be a time lag. In addition, since the calculation of image feature values is performed after image acquisition, there is also a time lag, and the length of the time lag is also different because the calculation load of image feature values differs depending on the image information. For example, when no object other than the recognition target is photographed at all and the background is a single solid color, the calculation of the image feature value can be performed at high speed, but when various objects are photographed, the calculation of the image feature value needs to be performed. time.

That is, in FIG. 43A , the abnormality judgment using the p-th image information and the q-th image information is briefly described, but in fact, as shown in FIG. The time lag tθp of information acquisition, the time lag tfp from the acquisition of the pth image information to the end of the calculation of the image feature value, and the qth image information also need to consider tθq and tfq.

The abnormality determination process starts, for example, when the image feature value fq of the qth image information is acquired. However, it must be appropriately determined how long ago the image feature value fp to obtain the difference is obtained, or when the acquisition times of θq and θp are when.

Specifically, the image feature value f1 of the first image information is obtained at the i-th time (i is a natural number), and the image feature value f2 of the second image information is obtained at the j-th time (j is a natural number satisfying j≠i) In the case of , the change calculation unit 1120 obtains the difference between the image feature value f1 and the image feature value f2 as the image feature value change amount, and the change amount estimation unit 1130 obtains the difference between the first image and the first image at time k (k is a natural number). In the case of information p1 corresponding to the information for estimating change, and information p2 for estimating change corresponding to the second image information is obtained at time l (l is a natural number), the information p1 for estimating change and the information p2 for estimating change The information p2 is used to obtain the amount of change in the estimated image feature amount.

In the example of FIG. 45 , image feature values and joint angle information are information acquired at various times, but when the acquisition time of fq (for example, the j-th time) is used as a reference, the information corresponding to the p-th image information The image feature quantity fp is an image feature quantity acquired at a time earlier (tfq+ti−tfp), that is, it is determined that the i-th time is earlier (tfq+ti-tfp) than the j-th time. Here, ti represents a difference in image acquisition time as shown in FIG. 45 .

Similarly, it is determined that the l-th time as the acquisition time of θq is (tfq-tθq) before the j-th time, and the k-th time as the acquisition time of θp is (tfq+ti-tθp) before the j-th time moment. In the method of this embodiment, it is necessary to obtain the correspondence between Δf and Δθ. Specifically, if Δf is obtained from the p-th image information and the q-th image information, then Δθ also needs to be related to the p-th image information and the q-th image information. Information correspondence. Otherwise, the estimated image feature quantity change Δfe obtained from Δθ does not have a corresponding relationship with Δf at all, and there is no meaning for comparison processing. From this, as described above, it can be said that it is important to determine the correspondence between the time points. In addition, in FIG. 45 , since the drive itself of the joint angle is performed at a very high speed and high frequency, it is processed as a continuous process.

In addition, in the conventional robot 20000 and the robot control device 1000 , it can be considered that the difference between the acquisition time of the image information and the acquisition time of the corresponding joint angle information is sufficiently small. Therefore, it can also be considered that the kth time is the acquisition time of the first image information, and the lth time is the acquisition time of the second image information. In this case, since tθp and tθq in FIG. 45 can be set to 0, processing is easy.

In addition, as a more specific example, a means of acquiring the next image information at the time when the image feature value is calculated from the previous image information is considered. An example of this is shown in FIG. 46 . The vertical axis of FIG. 46 represents the value of the image feature value, and the "actual feature value" is a value assumed to have acquired the image feature value corresponding to the joint angle information at that point in time, and cannot be confirmed in terms of processing. From the fact that the actual feature quantity smoothly transitions, it can be considered that the drive of the joint angle is continuous.

In this case, since the image feature quantity corresponding to the image information acquired at the time of B1 is acquired at the time of B2 after t2, the actual feature quantity of B1 corresponds to the image feature quantity of B2 (if there is no error are consistent). And the next image information is acquired at the time of B2.

Likewise, with regard to the image feature amount of the image information acquired at B2, the calculation is ended at B3, and the next image information is acquired at B3. Similarly, as for the image feature value of the image information acquired at B4, the calculation is completed at B5 after t1 has elapsed, and the next image information is acquired at B5.

In the example of FIG. 46 , when the image feature value calculated at time B5 and the image feature value calculated at time B2 are used in the abnormality determination process, the acquisition times of image information are B4 and B1, respectively. As described above, when the difference between the acquisition time of the image information and the acquisition time of the corresponding joint angle information is sufficiently small, the information of the time of B4 and the information of the time of B1 may be used for the joint angle information. That is, as shown in FIG. 46 , when the difference between B2 and B5 is Ts and the time reference is B5, the feature value at a time earlier than Ts is used as the image feature value to be compared. In addition, the two pieces of joint angle information used when calculating the difference of the joint angle information may be information at a time before t1 and information at a time before Ts+t2. In addition, the difference between the acquisition times of the two pieces of image information is (Ts+t2-t1). Thus, when determining the threshold Th based on the difference in acquisition time of image information, a value of (Ts+t2−t1) may be used.

In addition, the acquisition time of various information is considered, but as described above, the point that the time can be specified so that Δf and Δθ have a correspondence relationship is the same.

5. Variations

In the above description, Δf and Δθ are obtained, and the estimated image feature quantity change Δfe is obtained from Δθ to compare Δf with Δfe. However, the means of this embodiment are not limited to this. For example, position and posture information of the hand tip of the robot 20000 or position and posture information of an object grasped by the hand tip may be acquired by some means like the above-mentioned measurement means.

In this case, since the position and posture information X is acquired as information for change amount estimation, the change amount ΔX thereof can be obtained. Furthermore, as shown in the above formula (4), by applying the Jacobian matrix Ji to ΔX, it is possible to obtain the estimated image feature amount change Δfe in the same way as in the case of Δθ. Once Δfe is obtained, subsequent processing is the same as in the above example. That is, the change amount estimating unit 1130 acts on the change amount of the position and posture information by the Jacobian matrix that associates the position and posture information with the image feature value (specifically, associates the change amount of the position and posture information with the change amount of the image feature value). , and the calculation is performed on the amount of change in the inferred image feature quantity. In FIG. 43B , the flow of this processing is shown corresponding to FIG. 43A .

Here, when the position and posture of the hand tip (hand or end effector 2220) of the robot 20000 is used as the position and posture information, Ji corresponds the amount of change in the position and posture information of the hand to the amount of change in the image feature value. Information. In addition, when using the position and posture of the object as the position and posture information, Ji is information in which the amount of change in the position and posture information of the object is associated with the amount of change in the image feature value. Alternatively, if the information about the relative position and posture of the end effector to grasp the object is known, since the position and posture information of the end effector 2220 is in one-to-one correspondence with the position and posture information of the object, it is also possible to Information from one party is transformed into information from the other. That is, it is considered that after acquiring the position and posture information of the end effector 2220, converting it into the position and posture information of the object, and then using the Jacobian matrix in which the amount of change in the position and posture information of the object corresponds to the amount of change in the image feature value Ji to obtain various embodiments such as Δfe.

In addition, the comparison processing of the abnormality determination in this embodiment is not limited to performing using the image feature amount change amount Δf and the estimated image feature amount change amount Δfe. The amount of change Δf of the image feature quantity, the amount of change ΔX of the position and posture information, and the amount of change Δθ of the joint angle information can be obtained by using the Jacobian matrix, the inverse matrix of the Jacobian matrix (in a broad sense, the generalized inverse matrix), that is, the inverse Jacobian matrix. Matrices are thus transformed into each other.

That is, as shown in FIG. 49 , the means of this embodiment can be applied to a robot control device including a robot control unit 1110 that controls the robot 20000 based on image information; The position and posture change amount of the position and posture information of the robot 20000, or the change amount calculation unit 1120 of the joint angle change amount representing the change amount of the joint angle information of the robot 20000; calculates the image feature value change amount based on the image information, and according to The estimated amount of position and posture change, that is, the estimated position and posture change amount, or the estimated amount of joint angle change, that is, the change amount estimation unit 1130 that obtains the estimated amount of change in the joint angle by obtaining the amount of change in the image feature value; The abnormality determination unit 1140 performs abnormality determination by comparing the amount of change with the estimated position and posture change amount, or the comparison process between the change amount of the joint angle and the estimated joint angle change amount.

In FIG. 49 , when comparing with FIG. 36 , the change amount calculating unit 1120 and the change amount estimating unit 1130 are replaced. That is, the change amount calculation unit 1120 obtains the change amount (here, the joint angle change amount or the position posture change amount) based on the joint angle information, and the change amount estimation unit 1130 estimates the change amount based on the difference of the image feature value (calculates the estimated joint angle change or inferred position and posture change). In addition, in FIG. 49 , the change amount calculation unit 1120 is configured to acquire joint angle information, but as described above, the change amount calculation unit 1120 may acquire position and posture information using measurement results and the like.

Specifically, when Δf and Δθ are acquired, the estimated joint angle change amount Δθe may be obtained using the following equation (8) obtained from the above equation (5), and the comparison process of Δθ and Δθe may be performed. Specifically, using a predetermined threshold Th2, it is only necessary to determine that the condition is abnormal when the following formula (9) holds true.

Δθe＝Jv ^-1 Δf·····(8)

|Δθ－Δθe|＞Th2·····(9)

Alternatively, when Δf and ΔX are obtained using the above-mentioned measuring means, the estimated position and posture change ΔXe may be obtained by using the following equation (10) obtained from the above equation (4), and ΔX and ΔX may be calculated. Comparative treatment of ΔXe. Specifically, using a predetermined threshold Th3, when the following formula (11) holds true, it may be determined as abnormal.

ΔXe=Ji ^-1 Δf...(10)

|ΔX－ΔXe|＞Th3·····(11)

In addition, the comparison is not limited to the case of using directly obtained information. For example, in the case of obtaining Δf and Δθ, the estimated position and posture change ΔXe can also be obtained by using the above formula (10) according to Δf, and the position and posture change ΔX can be obtained by using the above formula (3) according to Δθ (strictly In other words, this ΔX is not an actual measured value but an estimated value), and the judgment using the above formula (11) is performed.

Alternatively, when Δf and ΔX are obtained, the estimated joint angle change Δθe can be obtained from Δf using the above formula (8), and the following formula (12) obtained from the above formula (3) can be used to obtain The amount of change in the joint angle Δθ is obtained (strictly speaking, this Δθ is also not an actual measurement value but an estimated value), and a determination using the above formula (9) is performed.

Δθ＝Ja ^-1 ΔX·····(12)

That is, the change amount calculation unit 1120 performs a process of acquiring a plurality of position and posture information and obtaining a difference between the plurality of position and posture information as a position and posture change amount, acquiring a plurality of position and posture information, and obtaining a plurality of position and posture information from the difference of the plurality of position and posture information The process of obtaining the joint change amount, the process of obtaining a plurality of joint angle information and obtaining the difference of the plurality of joint angle information as the above-mentioned joint angle change amount, and the process of obtaining a plurality of joint angle information and calculating the difference based on the difference of the joint angle information Any of the processes for calculating the amount of change in position and orientation.

In FIG. 47 , the relationship between Δf, ΔX, and Δθ shown above is summarized by attaching the numerical formula numbers in this specification. That is, for the means of this embodiment, if any two pieces of information in Δf, ΔX, and Δθ are acquired, they are converted into any one of information in Δf, ΔX, and Δθ for comparison, so that this embodiment can be realized The means of the method, and various modifications can be made to the acquired information and the information used for comparison.

In addition, the robot control device 1000 etc. of this embodiment may implement a part or most of the processing by a program. In this case, the robot control device 1000 and the like of the present embodiment are realized by a processor such as a CPU executing the program. Specifically, a program stored in a non-transitory information storage medium is read, and a processor such as a CPU executes the read program. Here, the information storage medium (a medium that can be read by a computer) is a medium that stores programs, data, etc., and its function can be performed by an optical disk (DVD, CD, etc.), HDD (hard disk drive), or a memory (card memory, ROM, etc.) ) and so on to achieve. Furthermore, a processor such as a CPU performs various processes in the present embodiment based on a program (data) stored in an information storage medium. That is, a program for causing a computer (a device including an operation unit, a processing unit, a storage unit, and an output unit) to function as each unit of the present embodiment (a program for causing the computer to execute the processing of each unit) is stored in the information storage medium. .

In addition, although the present embodiment has been described in detail above, it is easily understood by those skilled in the art that various changes can be made without substantially departing from the novel contents and effects of the present invention. Therefore, such modified examples are also included in the scope of the present invention. For example, in the specification or the drawings, a term described together with a different term having a broader or synonymous meaning at least once can be replaced by the different term at any position in the specification or the drawings. In addition, the configuration and operation of the robot controller 1000 and the like are not limited to those described in this embodiment, and various modifications can be made.

Seventh Embodiment

1. The means of this embodiment

First, the means of this embodiment will be described. In many cases, the inspection of the object to be inspected (especially the visual inspection) is used. For appearance inspection (visual inspection), the inspection method of viewing and observing with human eyes is basic, but from the viewpoint of labor saving for inspection users and high accuracy of inspection, it is proposed to use inspection equipment A means of automating the inspection.

The inspection device here may be a dedicated device. For example, as a dedicated inspection device, as shown in FIG. 54 , a device including an imaging unit CA, a processing unit PR, and an interface unit IF may be considered. In this case, the inspection device acquires a captured image of the inspection object OB captured by the imaging unit CA, and performs inspection processing using the captured image in the processing unit PR. Considering the contents of various inspection processes here, for example, an image of the inspection object OB in a state judged to be acceptable (may be a captured image or may be created from model data) is acquired in advance as an acceptable image during the inspection, and What is necessary is to perform a comparison process between the acceptable image and the actual captured image. If the captured image is close to the qualified image, it can be determined that the inspection object OB captured by the captured image is acceptable. If the difference between the captured image and the acceptable image is large, it can be determined that the inspection object OB has some problems and is unqualified. In addition, Patent Document 1 discloses a method using a robot as an inspection device.

However, as can also be seen from the above-mentioned examples of acceptable images, in order to perform an inspection by an inspection device, it is necessary to set information for the inspection in advance. For example, information such as the direction from which the inspection object OB is viewed needs to be set in advance, although it depends on the arrangement of the inspection object OB.

Generally, how the inspection object OB is observed (in a narrow sense, what shape and size it is when it is captured on a captured image) changes depending on the relative relationship between the inspection object OB and the position and direction of observation. Hereinafter, the position at which the inspection target object OB is observed is expressed as a viewpoint position, and the viewpoint position in a narrow sense means the position where the imaging unit is arranged. In addition, the direction in which the inspection object OB is observed is expressed as a line-of-sight direction, and the line-of-sight direction means the imaging direction (direction of the optical axis) of the imaging unit in a narrow sense. If no reference is provided for the position of the viewpoint and the direction of sight, the observation method of the inspection object OB may change every inspection, so it is impossible to perform visual inspection to determine whether the inspection object OB is normal or abnormal according to the observation method.

In addition, it is not possible to determine from what viewpoint position and line-of-sight direction an image can be held as a standard acceptable image for judging that there is no abnormality with respect to the inspection object OB. That is, if the position and direction of observation during inspection are uncertain, the comparison object (inspection reference) with respect to the captured image acquired during inspection is also uncertain, and appropriate inspection cannot be performed. In addition, as long as the images of the inspection object OB judged to be acceptable are observed from all viewpoint positions and viewing directions, it is possible to avoid a situation where there is no acceptable image. However, in this case, the position of the viewpoint and the direction of the line of sight will become relatively large, and the number of acceptable images will also become relatively large, which is not realistic. From the above point, it is also necessary to hold the acceptable image in advance.

In addition, in general, acceptable images and captured images also contain unnecessary information during inspection, so when inspection processing (comparison processing) is performed using the entire image, the accuracy of the inspection may be lowered. For example, tools, jigs, and the like may be captured in the captured image in addition to the inspection target, and it is not preferable to use the above information for inspection. In addition, when a part of the inspection object is the inspection object, there is a possibility that the inspection accuracy may be lowered due to the information of the area of the inspection object that is not the inspection object. Specifically, as will be described later using FIGS. 64A to 64D , when considering the operation of assembling an object B smaller than the object A for a larger object A, the object of inspection should be the object B to be assembled. surroundings, and it is less necessary to check the entirety of object A, and the possibility of misjudgment increases because the entirety of A is the inspection target. Therefore, in consideration of improving the accuracy of the inspection process, the inspection area becomes important information also in the inspection.

However, in the past, information for inspection such as the inspection area, viewpoint position, line of sight direction, or acceptable image has been set by a user with specialized knowledge in image processing. This is because although the comparison process between the acceptable image and the captured image is performed by image processing, it is required to change the setting of information required for inspection according to the specific content of the image processing.

For example, whether the application of image processing using edges in the image, image processing using all pixel values, image processing using brightness, color difference hue, or other image processing is suitable for the comparison of acceptable images and captured images The processing (similarity determination processing in a narrow sense) may vary according to the shape, color tone, texture, etc. of the inspection object OB. Therefore, for an inspection in which the content of image processing can be changed, the user performing the inspection must appropriately set what kind of image processing to perform.

In addition, even when the content of image processing has been set, or when the content of image processing with high versatility has been set in advance, the user needs to properly grasp the content of the image processing. This is because if the specific content of the image processing changes, the viewpoint position and line-of-sight direction suitable for the inspection may also change. For example, in the case of performing comparison processing using edge information, the position and direction in which a complex part of the shape of the inspection object OB can be observed can be set as the viewpoint position and the line of sight direction, and the position and direction for observing a flat part can be Direction is not appropriate for viewpoint position. In addition, if the comparison process is performed using pixel values, it is preferable to use a position and direction where a region that can be observed brightly due to a large change in color tone or sufficiently irradiated with light from a light source is used as the viewpoint position and the line-of-sight direction. That is, in the conventional means, professional knowledge of image processing is required for the setting of information necessary for the inspection including the position of the viewpoint, the direction of the line of sight, and the acceptable image. Furthermore, if the content of the image processing is different, it is necessary to change the criteria for the comparison processing between the acceptable image and the captured image. For example, depending on the content of image processing, it is necessary to determine a criterion for determining the degree of similarity between the acceptable image and the captured image and the failure for the degree of difference. However, this criterion cannot be set without professional knowledge in image processing.

That is, even if the inspection can be automated by using a robot or the like, it is difficult to set the information required for the inspection, and it cannot be said that the automation of the inspection is not easy for a user without professional knowledge.

In addition, the robot conceived by the present applicant can perform various tasks flexibly by making it easy for the user to provide guidance when performing the robot's work, and is equipped with various sensors so that the robot itself can recognize the work environment. . Such a robot is suitable for multi-item manufacturing (in a narrow sense, multi-item low-volume manufacturing in which the average amount of one item is produced is small). However, even if it is easy to provide guidance on the timing of manufacturing, it is another question whether it is easy to inspect the manufactured product. This is because the position of the object to be inspected differs depending on the product, and as a result, the inspection area to be compared is different for each product in the captured image and the pass image. That is, in the case of assuming multi-item manufacturing, entrusting the setting of the inspection area to the user will result in a large burden on the setting process, leading to a decrease in productivity.

Therefore, the present applicant proposes a means for reducing the burden on the user in the inspection process and improving productivity during robot work by generating second inspection information used in the inspection process based on the first inspection information. Specifically, the robot 30000 of this embodiment is a robot that performs inspection processing for inspecting an inspection object using a captured image of the inspection object captured by an imaging unit (for example, the imaging unit 5000 in FIG. 52 ). The inspection information generates second inspection information including an inspection area for inspection processing, and performs inspection processing based on the second inspection information.

Here, the first inspection information is information that can be acquired by the robot 30000 before execution of the inspection process, and indicates information used to generate the second inspection information. Since the first inspection information is information obtained in advance, it can also be expressed as advance information. In this embodiment, the first inspection information may be input by the user, or may be generated by the robot 30000 . Even when the user inputs the first inspection information, the first inspection information does not require expertise in image processing at the time of input, and is information that can be easily input. Specifically, it may be information including at least one of shape information of the inspection object OB, position and posture information of the inspection object OB, and a relative inspection processing target position with respect to the inspection object OB.

As will be described later, the second inspection information can be generated by using shape information (three-dimensional model data in a narrow sense), position and posture information, and information on an inspection processing target position. Moreover, the shape information is generally acquired in advance as CAD data, etc., and when the user inputs the shape information, the existing information may be selected. For example, in a situation where data of various objects that are candidates for the inspection object OB are held, the user may select the inspection object OB from among the candidates. In addition, as for the position and posture information, if it is known at the time of inspection how the inspection object OB is arranged (for example, in what posture and at what position on the workbench), the position and posture information can also be easily set. The input of pose information does not require expertise in image processing. In addition, the inspection processing target position is information indicating a position to be inspected in the inspection object OB, for example, when inspecting a predetermined portion of the inspection object OB for damage, it is information indicating the position of the predetermined portion . In addition, the inspection target object OB, which assembles an object B with respect to an object A, is used as an object. When checking whether the assembly of the object A and the object B is carried out normally, the assembly position (contact surface, contact point, insertion position) of the object A and the object B is inspected. location, etc.) becomes the inspection processing target location. Similarly, the inspection processing target position can be easily input if the content of the inspection can be grasped, and professional knowledge of image processing is not required for input.

In addition, the means of this embodiment are not limited to automatically generating all of the second inspection information. For example, part of the second inspection information may be generated using the means of this embodiment, and the other second inspection information may be manually input by the user. In this case, the user cannot completely omit the input of the second inspection information, but at least in the point of being able to automatically generate viewpoint information that is difficult to set, the advantage of being able to perform the inspection easily by using the means of this embodiment is that Changeless.

In addition, since the inspection processing is performed on the result of the robot work by the robot 30000, the first inspection information may be information acquired during the robot work.

Here, robot work refers to work performed by a robot, and various works such as joints such as screw fastening, welding, pressure welding, and snapshots, and deformation of hands, tools, and jigs are considered. When the checking process is performed on the results of the robot work, the checking process determines whether the robot work was performed normally. In this case, in order to start the robot work, it is necessary to acquire various information related to the inspection target object OB and work details. For example, what position and posture the work object (all or part of the inspection object OB) is placed before the work, and what position and posture it changes after the work is known information. In addition, when screw fastening and welding are performed, the positions of the fastening screws and the welding positions in the work object are known. Similarly, if multiple objects are combined, the position and direction of object A combined with what object is known information. If deformation is applied to the work object, the deformed position and deformed shape of the work object will also be are all known information.

That is, when the robot operation is the object, the information corresponding to the above-mentioned shape information, position and posture information, and the inspection processing target position, and other information included in the first inspection information, on the premise that the robot operation is completed In this case, a considerable part (all of the first examination information required according to the situation) becomes known. That is, in the robot 30000 according to the present embodiment, the first inspection information may divert information held by a unit (for example, the processing unit 11120 in FIG. 50 ) that controls the robot. Also, even when the means of this embodiment is applied to a processing device 10000 different from the robot 30000 as described later using FIGS. 51A and 51B , the processing device 10000 acquires the first Just check the information. Therefore, from the user's point of view, the second inspection information can be easily generated without re-inputting the first inspection information for inspection.

Thereby, even a user without professional knowledge of image processing can easily execute the inspection (acquire at least the second inspection information), or reduce the burden of setting the second inspection information when performing the inspection. In addition, in the following description of this specification, an example in which the object of the inspection process is the result of robot work will be described. That is, the user does not need to input the first examination information, but as described above, the user may input part or all of the first examination information. Even when the user inputs the first inspection information, the user does not require specialized knowledge to input the first inspection information, and the advantage of being easy to perform the inspection remains unchanged.

In addition, in the following description, an example in which the second inspection information is generated by the robot 30000 and the inspection process is executed by the robot 30000 will be mainly described using FIGS. 52 and 53 as will be described later. However, the means of this embodiment are not limited thereto, and the following description can be extended to a means of generating second inspection information in the processing device 10000 as shown in FIG. 51A , and the robot 30000 acquires the second inspection information and executes inspection processing. . Alternatively, as shown in FIG. 51B , the second inspection information can be generated in the processing device 10000 , and the inspection process using the second inspection information is executed not in the robot but in a dedicated inspection device or the like.

Hereinafter, a system configuration example of the robot 30000 and the processing device 10000 according to this embodiment will be described, and then a specific processing flow will be described. More specifically, the flow from the acquisition of the first inspection information to the generation of the second inspection information will be described as offline processing, and the process performed by a robot using the generated second inspection information will be described as online processing (online). The flow of the actual inspection process performed will be described.

2. Example of system configuration

Next, a system configuration example of the robot 30000 and the processing device 10000 according to this embodiment will be described. As shown in FIG. 50 , the robot of this embodiment includes an information acquisition unit 11110 , a processing unit 11120 , a robot mechanism 300000 , and an imaging unit 5000 . However, the robot 30000 is not limited to the configuration shown in FIG. 50 , and various modifications can be made by omitting some of the above-mentioned constituent elements or adding other constituent elements.

The information acquisition unit 11110 acquires first inspection information before inspection processing. When the first inspection information is input by the user, the information acquisition unit 11110 performs processing for accepting the input information from the user. In addition, when the information used in the robot work is the first inspection information, the information acquisition unit 11110 reads out the control information used in the processing unit 11120 when the work is read from a storage unit not shown in FIG. 50 . processing and so on.

The processing unit 11120 performs generation processing of the second inspection information based on the first inspection information acquired by the information acquiring unit 11110 and performs inspection processing using the second inspection information. The processing in the processing unit 11120 will be described in detail later. In addition, the processing unit 11120 controls the robot 30000 in inspection processing and other than inspection processing (for example, robot work such as assembly). For example, the processing unit 11120 controls the arm 3100 included in the robot mechanism 300000, the imaging unit 5000, and the like. In addition, the imaging unit 5000 may be a hand-eye camera attached to the arm 3100 of the robot.

In addition, as shown in FIG. 51A , the means of this embodiment can be applied to a processing device for using an inspection object photographed by an imaging unit (the imaging unit 5000 is shown in FIG. 51A , but not limited thereto). An apparatus for performing the inspection processing of the inspection target object by capturing an image of the object, and a processing apparatus 10000 for outputting information for the inspection processing, which generates a view including the position of the viewpoint and the direction of the line of sight of the imaging unit to be inspected based on the first inspection information. The viewpoint information and the second inspection information including the inspection area of the inspection process are included, and the second inspection information is output to the device performing the inspection process. In this case, the acquisition of the first inspection information and the generation of the second inspection information are performed by the processing device 10000, and the processing device 10000 can be used as a processing device including an information acquisition unit 11110 and a processing unit 11120 as shown in FIG. 51A , for example. accomplish.

Here, the device that performs the inspection process may be the robot 30000 as described above. In this case, as shown in FIG. 51A , robot 30000 includes arm 3100 , imaging unit 5000 for inspection processing of an inspection target, and control unit 3500 for controlling arm 3100 and imaging unit 5000 . The control unit 3500 acquires from the processing device 10000 information including the viewpoint information indicating the viewpoint position and line-of-sight direction of the imaging unit 5000 and the inspection region as the second inspection information, and performs the operation of the imaging unit based on the second inspection information. 5000 controls movement to the viewpoint position and line of sight direction corresponding to the viewpoint information, and executes the inspection process using the acquired captured image and inspection area.

In this manner, the second inspection information can be generated in the processing device 10000, and inspection processing can be appropriately executed in other devices using the second inspection information. If the inspection processing device is the robot 30000, similarly to FIG. 50 , a robot that performs inspection processing using the second inspection information can be realized. FIG. 51A differs from FIG. 50 in that the machines executing the processing are different.

In addition, the processing device 10000 may perform not only the generation process of the second inspection information but also the control process of the robot 30000 in conjunction with it. For example, the processing unit 11120 of the processing device 10000 generates second inspection information, and generates robot control information based on the second inspection information. In this case, the control unit 3500 of the robot operates the arm 3100 and the like based on the control information generated by the processing unit 11120 of the processing device 10000 . That is, the processing device 10000 is in charge of a substantial part of the control of the robot, and the processing device 10000 in this case can also be understood as a robot control device.

In addition, the subject who executes the inspection process using the second inspection information generated by the processing device 10000 is not limited to the robot 30000 . For example, the second inspection information may be used to perform inspection processing in a dedicated device as shown in FIG. 54 , and the configuration in this case is shown in FIG. 51B , for example. FIG. 51B shows an example in which the inspection device receives input of first inspection information (for example, the interface unit IF in FIG. 54 may be used for this), and outputs the first inspection information to the processing device 10000 . In this case, the processing device 10000 generates the second inspection information using the first inspection information input from the inspection device. However, the first inspection information can be implemented in various modifications, such as the example in which the user directly inputs it to the processing device.

As shown in FIG. 52, the robot 30000 of this embodiment may be a single-arm robot having one arm. In FIG. 52 , an imaging unit 5000 (hand-eye camera) is provided as an end effector of the arm 3100 . However, it is possible to implement various modifications by providing a grip such as a hand as an end effector, installing the imaging unit 5000 at other positions of the grip or the arm 3100 , and the like. In addition, in FIG. 52 , a device such as a PC is shown as a device corresponding to the control unit 3500 of FIG. 51A , but this device may also be a device corresponding to the information acquisition unit 11110 and the processing unit 11120 of FIG. 50 . In addition, in FIG. 52 , including the interface unit 6000 , the operation unit 6100 and the display unit 6200 are shown as the interface unit 6000 . The structure of the interface unit 6000 is modified.

In addition, the structure of the robot 30000 of this embodiment is not limited to FIG. 52 . For example, as shown in FIG. 53 , the robot 30000 may also include at least a first arm 3100 and a second arm 3200 different from the first arm 3100 , and the photographing unit 5000 is provided on at least one of the first arm 3100 and the second arm 3200 One hand-eye camera. In FIG. 53, the first arm 3100 is composed of joints 3110, 3130 and frames 3150, 3170 provided between the joints, and the same is true for the second arm 3200, but the present invention is not limited thereto. In addition, in FIG. 53 , an example of a dual-arm robot having two arms is shown, but the robot of this embodiment may have three or more arms. Although it is also described that the imaging unit 5000 is installed on both the hand-eye camera (5000-1) of the first arm 3100 and the hand-eye camera (5000-2) of the second arm 3200, it may be installed on one of them. superior.

In addition, the robot 30000 of FIG. 53 includes a base unit part 4000 . The base unit part 4000 is provided at the lower part of the robot main body, and supports the robot main body. In the example of FIG. 53 , wheels and the like are provided on the base unit 4000 and the entire robot is movable. However, the base unit 4000 may be fixed to the ground or the like without having wheels or the like. In the robot of FIG. 53 , the robot mechanism 300000 and the control unit 3500 are integrally configured by accommodating the control device (the device shown as the control unit 3500 in FIG. 52 ) in the base unit unit 4000 . Alternatively, like the device corresponding to the control unit 3500 in FIG. 52 , no specific control equipment may be provided, and the above-mentioned control may be realized by incorporating a substrate of the robot (more specifically, an IC or the like provided on the substrate). Control part 3500.

In the case of using a robot having two or more arms, flexible inspection processing can be performed. For example, when a plurality of imaging units 5000 are installed, inspection processing can be performed simultaneously from a plurality of viewpoint positions and line-of-sight directions. In addition, it is also possible to inspect the inspection object OB grasped by the grasping part provided on the other arm using the hand-eye camera provided on a given arm. In this case, it is possible to change not only the viewpoint position and the line-of-sight direction of the imaging unit 5000 but also the position and posture of the inspection object OB.

In addition, as shown in FIG. 20 , the functions of parts corresponding to the processing device of this embodiment or the processing unit 11120 in the robot 30000 may also be communicated with the robot 30 through a network 20 including at least one of wired and wireless. server 700 to achieve.

Alternatively, in the present embodiment, a part of processing by the processing device or the like of the present invention may be performed on the side of the server 700 as the processing device. At this time, the processing is realized by distributed processing between the processing device provided on the robot side and the server 700 as the processing device. Specifically, the server 700 side performs the processing assigned to the server 700 among the respective processes of the processing device of the present invention. On the other hand, the processing device 10000 provided in the robot performs the processing assigned to the processing unit of the robot and the like among the respective processes of the processing device of the present invention.

For example, the processing device of the present invention performs the first to Mth (M is an integer) processing, and it is considered that the first processing can be realized by sub-processing 1a and sub-processing 1b, and the second processing can be realized by sub-processing 2a and sub-processing 2b, each of the first to Mth processes is divided into a plurality of sub-processes. In this case, it is assumed that the server 700 performs sub-processing 1a, sub-processing 2a, ... sub-processing Ma, and the processing device 100 installed on the robot side performs sub-processing 1b, sub-processing 2b, ... sub-processing Mb. deal with. At this time, the processing device of this embodiment, that is, the processing device that executes the first to the Mth processing may be a processing device that executes sub-processing 1a to sub-processing Ma, may be a processing device that executes sub-processing 1b to sub-processing Mb, or It may be a processing device that executes all of sub-processing 1a to sub-processing Ma and sub-processing 1b to sub-processing Mb. Furthermore, the processing device of this embodiment is a processing device that executes at least one sub-processing for each of the first to M-th processing.

Thereby, for example, the server 700 having a higher processing capability than the processing device 10000 on the robot side can perform processing with a high processing load. In addition, when the processing device also performs robot control, the server 700 can collectively control the operation of each robot, thereby facilitating, for example, cooperative operations of a plurality of robots.

In addition, in recent years, there is an increasing tendency to manufacture many types of parts with a small number of components. Furthermore, when changing the type of parts to be manufactured, it is necessary to change the operation performed by the robot. With the configuration shown in FIG. 20 , the server 700 can collectively change the actions performed by the robots without redoing the instruction work for each of the plurality of robots. Furthermore, compared with the case where one processing device is provided for each robot, it is possible to greatly reduce the troubles and the like at the time of updating the software of the processing device.

3. Processing flow

Next, the processing flow of this embodiment will be described. Specifically, the flow of obtaining the first inspection information and the generation of the second inspection information, and the flow of executing the inspection process based on the generated second inspection information will be described. Assuming that the inspection process is performed by a robot, the acquisition of the first inspection information and the generation of the second inspection information can be performed without the operation of the robot during the inspection process, so they are described as offline processing (offline). On the other hand, since the execution of the inspection process is accompanied by the movement of the robot, it is described as online processing.

In addition, in the following, an example in which the object of the inspection process is the result of the assembly work by the robot and the inspection process is also executed by the robot will be described, but there are points where various modifications can be performed as described above.

3.1 Offline processing

First, a specific example of the first inspection information and the second inspection information of the present embodiment is shown in FIG. 55 . The second inspection information includes viewpoint information (viewpoint position and line-of-sight direction), inspection region (ROI, confirmation region), and acceptable images. In addition, the first inspection information includes shape information (three-dimensional model data), position and posture information of an inspection object, and an inspection processing object position.

The flow of specific offline processing is shown in the flowchart of FIG. 56 . When offline processing is started, first, the information acquiring unit 11110 acquires three-dimensional model data (shape information) of an inspection object as first inspection information (S100001). In inspection (appearance inspection), how to observe the inspection object is important, and the observation method from a given viewpoint position and line of sight depends on the shape of the inspection object. In particular, since the three-dimensional model data is information on an inspection object in an ideal state with no defect or deformation, it is useful information for inspection processing of the actual object to be inspected.

In the case where the inspection processing is performed on the result of robot work by the robot, the information acquisition unit 11110 acquires the three-dimensional model data of the inspection object acquired as a result of the robot work, that is, post-work three-dimensional model data, and the robot work result. The three-dimensional model data of the object to be inspected before, that is, the three-dimensional model data before operation.

When checking the results of robot work, it is necessary to determine whether the work was performed properly. When the operation is an assembly operation of assembling the object A and the object B, it is determined whether or not to assemble the object B with respect to the object A at a predetermined position from a predetermined direction. That is, the acquisition of individual 3D model data of object A and object B is not sufficient, and the important point is to assemble the data of object B at a predetermined position from a predetermined direction for object A, that is, in the state where the work is ideally completed. 3D model data. Thus, the information acquiring unit 11110 of this embodiment acquires post-work three-dimensional model data. In addition, as in the setting of the inspection area and pass threshold value described later, there are also scenes where the difference in the observation method before and after the operation becomes the main point, so the three-dimensional model data before the operation is also acquired in advance.

Examples of pre-work three-dimensional model data and post-work three-dimensional model data are shown in FIGS. 57A and 57B. In FIGS. 57A and 57B , the operation of assembling a cubic block-shaped object B in the same posture as the object A at a position shifted along a given axis direction with respect to a cubic block-shaped object A is as follows: For example. In this case, the pre-work three-dimensional model data is the three-dimensional model data of the object A as shown in FIG. 57A because it is before the object B is assembled. In addition, as shown in FIG. 57B , post-work three-dimensional model data is data in which object A and object B are assembled under the above-mentioned conditions. In addition, in Fig. 57A and Fig. 57B, in relation to the relationship shown in a planar manner, the three-dimensional model data is data such as whether to observe from a given viewpoint position and line of sight direction, but the word "three-dimensional" also It can be seen that the shape data acquired as the first inspection information is three-dimensional data in which the observation position and direction are not limited.

In addition, in S100001, viewpoint candidate information as a candidate for viewpoint information (information including viewpoint position and line-of-sight direction) is acquired in conjunction with it. It is assumed that the viewpoint candidate information is not input by the user or generated by the processing unit 11120 , but is, for example, information preset by the manufacturer of the processing device 10000 (or robot 30000 ) before shipment.

Although the viewpoint candidate information is information that is a candidate for viewpoint information as described above, it is considered that there are very many points that may become the viewpoint candidate information (infinite in a narrow sense). For example, when viewpoint information is set in the object coordinate system (object coordinate system) based on the inspection object, all points other than the points included in the inspection object in the object coordinate system may become viewpoints. Alternate information. Of course, by using such a large number of viewpoint candidate information (the viewpoint candidate information is not limited in terms of processing), it is possible to flexibly or finely set the viewpoint information according to the situation. Therefore, if the processing load does not pose a problem when setting the viewpoint information, it is not necessary to acquire viewpoint candidate information in S100001. However, in the following description, viewpoint candidate information is set in advance so that it can be commonly used even when various objects are inspection targets, and does not increase the processing load in setting viewpoint information.

At this time, the position, orientation, and the like of the arrangement of the inspection object OB at the time of inspection are not limited to being known. Therefore, since it is unclear whether the imaging unit 5000 can be moved to a position and orientation corresponding to the viewpoint information, it is not practical to limit the viewpoint information to a very small number (for example, one or two). This is because when only a small amount of viewpoint information is generated, the inspection process cannot be executed unless the imaging unit 5000 is moved to all of the small amount of viewpoint information. In order to suppress this risk, it is necessary to generate a certain amount of viewpoint information, and as a result, the number of viewpoint candidate information also becomes a certain amount.

An example of viewpoint candidate information is shown in FIG. 58 . In FIG. 58 , 18 pieces of viewpoint candidate information are set around the origin of the object coordinate system. The specific coordinate values are shown in Figure 59. In the case of viewpoint candidate information A, the viewpoint position is on the x-axis and is located at a predetermined distance (200 in the example of FIG. 59 ) from the origin. Also, the line of sight direction corresponds to a vector represented by (ax, ay, az), and in the case of viewpoint candidate information A, it is the negative direction of the x-axis, that is, the direction of the origin. Also, even if only the gaze direction vector is determined, since the imaging unit 5000 can rotate around the gaze direction vector, the posture is not fixed to one. Therefore, here, another vector specifying the rotation angle around the line-of-sight vector is set in advance as (bx, by, bz). In addition, as shown in FIG. 58 , a total of 18 points between two points on each xyz axis and points between two xyz axes are used as viewpoint candidate information. By setting the viewpoint candidate information around the origin of the object coordinate system in this way, it is possible to set appropriate viewpoint information regardless of how the inspection object is arranged in the world coordinate system (robot coordinate system). Specifically, it is possible to suppress the possibility that the imaging unit 5000 cannot be moved to all (or most) of the viewpoint information set based on the viewpoint candidate information, or that even if it is moved, it cannot be inspected due to an obstruction or the like. Enables inspection with at least a sufficient amount of viewpoint information.

In appearance inspection, it is fine to perform inspection from only one viewpoint position and line of sight direction, but in consideration of accuracy, it is preferable to perform inspection from multiple viewpoint positions and line of sight directions. This is because there is a possibility that the region to be inspected may not be observed sufficiently (for example, with a sufficiently large size on the image) when inspecting from only one direction. Therefore, it is preferable that the second inspection information is not one piece of viewpoint information, but a viewpoint information group including a plurality of viewpoint information. This is achieved, for example, by generating viewpoint information using a plurality of candidate information (basically all candidate information) among the aforementioned viewpoint candidate information. In addition, even when the above-mentioned viewpoint candidate information is not used, it is sufficient to obtain a plurality of viewpoint information. That is, the second inspection information includes a viewpoint information group including a plurality of viewpoint information, and each viewpoint information in the viewpoint information group includes the viewpoint position and line-of-sight direction of the imaging unit 5000 in the inspection process. Specifically, the processing unit 11120 generates a viewpoint information group including a plurality of viewpoint information of the imaging unit 5000 as second inspection information based on the first inspection information.

The above-mentioned viewpoint candidate information is the position in the object coordinate system, but in the stage of setting the viewpoint candidate information, the specific shape and size of the inspection object are uncertain. Specifically, although FIG. 58 is an object coordinate system based on the inspection target object, the position and posture of the object in this object coordinate system are in an indeterminate state. Since viewpoint information needs to specify at least a relative positional relationship with the inspection object, in order to generate specific viewpoint information from viewpoint candidate information, correspondence with the inspection object is required.

Here, in consideration of the above-mentioned viewpoint candidate information, the origin of the coordinate system in which the viewpoint candidate information is set is the position that is the center of all viewpoint candidate information, and when the imaging unit 5000 is arranged with each viewpoint candidate information , located in the imaging direction (optical axis direction) of the imaging unit 5000 . That is, it can be said that the origin of the coordinate system is the best position under observation using the imaging unit 5000 . Since the position that should be most observed in the inspection process is the above-mentioned inspection processing target position (it may be an assembly position in a narrow sense, or may be an assembly position as shown in FIG. 58), the inspection processing target acquired as the first inspection information is used. Viewpoint information corresponding to the inspection target object is generated based on its position.

That is, the first inspection information includes the relative inspection processing target position of the above-mentioned inspection target object, and the robot 30000 sets an object coordinate system corresponding to the inspection target object based on the inspection processing target position, and uses the object coordinate system, Generate viewpoint information. Specifically, the information acquisition unit 11110 acquires the relative inspection processing target position of the inspection target object as the first inspection information, and the processing unit 11120 sets the object coordinate system corresponding to the inspection target object based on the inspection processing target position. , and generate viewpoint information using the object coordinate system (S100002).

For example, the shape data of the inspection object has the shape shown in FIG. 60 , and the first inspection information in which point O is the position of the inspection processing object is acquired. In this case, it is only necessary to set the point O as the origin, and set the posture of the inspection object to an object coordinate system as shown in FIG. 60 . If the position and posture of the inspection object in the object coordinate system are specified, the relative relationship between each viewpoint candidate information and the inspection object becomes clear, so each viewpoint candidate information can be used as viewpoint information.

When a viewpoint information group including a plurality of viewpoint information is generated, various kinds of second inspection information are generated. First, a pass image corresponding to each viewpoint information is generated (S100003). Specifically, the processing unit 11120 acquires an image of 3D model data captured by a virtual camera arranged at a viewpoint position and a line-of-sight direction corresponding to the viewpoint information as an acceptable image used in the inspection process.

The acceptable image needs to be an image showing an ideal state of the inspection target object. Since the three-dimensional model data (shape information) acquired as the first inspection information is ideal shape data of the object to be inspected, the image of the three-dimensional model data observed from the imaging unit arranged corresponding to the viewpoint information is regarded as a qualified image. Just use it. In addition, when using 3D model data, processing using a virtual camera (specifically, conversion processing of projecting 3D data into 2D data) is performed instead of imaging by the imaging unit 5000 . In addition, assuming inspection processing for the results of robot work, the pass image is an image showing an ideal state of the inspection target object at the end of the robot work. Furthermore, since the ideal state of the inspection object at the end of the robot operation is displayed by the above-mentioned post-work 3D model data, the image of the post-work 3D model data captured by the virtual camera may be regarded as a pass image. Since a qualified image is obtained for each viewpoint information, when 18 viewpoint information are set as described above, there are also 18 qualified images. The images on the right side of each of FIGS. 61A to 61G correspond to acceptable images when the assembly work corresponding to FIG. 57B is performed. In FIGS. 61A to 61G , images corresponding to seven viewpoints are described, but as described above, the number of images is obtained from the number of viewpoint information. In addition, in the process of S100003, the pre-work image of the pre-work 3D model data captured by the virtual camera is also obtained in advance in consideration of the processing of the inspection area and pass threshold value described later (the left sides of each of FIGS. 61A to 61G ).

Next, an inspection area, which is an area in the image of the pass image and the captured image used for the inspection process, is obtained as the second inspection information ( S100004 ). Check the area as above. In addition, since the observation method of important parts in the inspection changes according to the viewpoint information, the inspection area is set for each viewpoint information included in the viewpoint information group.

Specifically, the processing unit 11120 acquires, as a qualified image, an image of the post-work 3D model data captured by a virtual camera arranged at the viewpoint position and line-of-sight direction corresponding to the viewpoint information, and obtains an image of the post-operation 3D model data obtained by placing it on the viewpoint information as a pre-operation image. The image of the pre-work 3D model data captured by the virtual camera corresponding to the viewpoint position and line-of-sight direction, and based on the comparison process between the pre-work image and the acceptable image, the area in the image used for the inspection process, that is, the inspection area is obtained.

A specific example of the inspection region setting process is shown in FIGS. 62A to 62D . In the case where the result of the robot work of assembling the object B from the right with respect to the object A is an inspection object, an image of post-work three-dimensional model data captured by a virtual camera is acquired as an acceptable image as shown in FIG. 62B as shown in FIG. 62A The image of the pre-work three-dimensional model data captured by the virtual camera is acquired as the pre-work image. In this way, in a robot operation with state changes before and after the operation, the part of the state change is more important. If it is the example in FIG. 62B , what should be judged in the inspection is whether the object B is assembled with the object A, and whether the assembly position is correct or not. Although it is also possible to inspect parts of the object A other than the parts related to the work (for example, the joint surface during assembly), the importance is relatively low.

That is, it is possible to consider a region in an acceptable image and an image with higher importance among captured images as a region that changes before and after the job. Therefore, in the present embodiment, the processing unit 120 performs a process of obtaining a difference image, which is a difference between the pre-work image and the acceptable image, as a comparison process, and obtains an area including an inspection object in the difference image as an inspection area. . In the example of FIG. 62A and FIG. 62B , since the difference image is FIG. 62C , an inspection region including the region of object B included in FIG. 62C is set.

In this way, the region including the inspection target object in the differential image, that is, the region estimated to be of high importance for inspection can be used as the inspection region.

At this time, the inspection processing target position (mounting position in FIG. 62A and the like) is known as the first inspection information, and what position the inspection processing target position is in the image is also known. As described above, since the inspection processing target position is a position serving as a reference for inspection in the inspection processing, the inspection area may be obtained from the difference image and the inspection processing target position. For example, as shown in FIG. 62C , the processing unit 11120 obtains the maximum vertical length BlobHeight and the horizontal maximum Blobwidth of the inspection processing target position in the differential image and the remaining area of the differential image. In this way, if the region within the distances of the upper and lower BlobHeights and the left and right Blobwidths is used as the inspection region centering on the inspection processing object position, the region including the inspection object in the difference image can be obtained as the inspection region. In addition, in the present embodiment, there may be margins in the vertical and horizontal directions, and in the example of FIG. 62D , regions having margins of 30 pixels in the top, bottom, left, and right sides are used as inspection areas.

The same applies to FIGS. 63A to 63D and FIGS. 64A to 64D. Figures 63A to 63D show the operation of assembling a relatively thin object B on the back side of an object A (or the operation of inserting a rod-shaped object B into an object A with holes in the horizontal direction in the image) when viewed from the viewpoint position. example. In this case, the region of the inspection target object in the difference image is divided into a plurality of discontinuous regions, but it can be processed in the same manner as in FIGS. 62A to 62D . Here, it is assumed that the object B is thinner than the object A, and the vicinity of the upper end or the vicinity of the lower end in the image of the object A is less important in inspection. According to the means of this embodiment, as shown in FIG. 63D , the region of the object A considered to be of low importance can be removed from the inspection region.

64A to 64D are operations of assembling an object B smaller than the object A for a larger object A. FIG. This corresponds to, for example, an operation of fastening a screw as the object B to a predetermined position of the object A such as a PC or a printer. In this kind of work, the necessity of checking the entire PC and printer is low, and the position where the screws are fastened is highly important. In this regard, according to the means of this embodiment, as shown in FIG. 64D , most of the object A can be removed from the inspection area, and the periphery of the object B to be inspected can be set as the inspection area.

In addition, the above-mentioned means are highly versatile means for setting the inspection area, but the means for setting the inspection area in this embodiment is not limited thereto, and the inspection area may be set by other means. For example, in FIG. 62D , since even a narrower inspection area is sufficient, means of setting a narrower area may also be used.

Next, setting processing of a threshold (pass threshold) used in the comparison processing of the pass image and the actually shot image is performed ( S100005 ). Specifically, the processing unit 11120 acquires the above-mentioned acceptable image and the pre-operation image, and sets a threshold used for inspection processing based on the captured image and the acceptable image according to the similarity between the pre-operation image and the acceptable image.

Specific examples of threshold value setting processing are shown in FIGS. 65A to 65D . 65A is a qualified image. If the robot operation is performed ideally (in a broad sense, if the inspection object is in an ideal state), the actually captured image should also be consistent with the qualified image, and the similarity is the maximum value (here, 1000). On the other hand, if there is no element matching the acceptable image at all, the similarity is the minimum value (here, 0). Here, the threshold is a value such that if the degree of similarity between the acceptable image and the captured image is equal to or greater than the threshold, it is determined that the inspection is passed, and when the degree of similarity is smaller than the threshold, it is determined that the inspection is not passed. That is, the threshold is a given value between 0 and 1000.

Here, FIG. 65B is a pre-operation image corresponding to FIG. 65A . However, since FIG. 65B also includes components common to those in FIG. 65A , the similarity between the pre-operation image and the pass image is a value other than zero. For example, in the case of using the edge information of the image to determine the degree of similarity, the image 65C which is the edge information of FIG. 65A is used for comparison processing, but the image 65D which is the edge information of the image before operation also includes part. In the example shown in FIG. 65C and FIG. 65D , the value of the degree of similarity is less than 700. As a result, even if an inspection object in a state where no work is performed is captured as a captured image, the similarity between the captured image and the acceptable image maintains a value of about 700. Taking an image of an inspection object in a state where no work is being done at all means, for example, that the work itself cannot be carried out, or that the work has been carried out but the object on the assembly side falls and is not captured in the image, and the robot may fail to work. Sex is higher. That is, since a similarity degree of about 700 appears even when the test should be "failed", it can be said that it is inappropriate to set a value lower than this value as the threshold.

Therefore, in this embodiment, a value between the maximum value of similarity (for example, 1000) and the similarity between the pre-work image and the acceptable image (for example, 700) is set as the threshold. As an example, the average value may be used, and the threshold value may be obtained by the following equation (13).

Threshold={1000+(similarity between qualified image and pre-work image)}/2·····(13)

In addition, the setting of the threshold value can be implemented in various modifications, for example, the formula for calculating the threshold value may be changed in accordance with the value of the similarity between the pass image and the pre-work image.

For example, the following modification can be implemented: when the similarity between the qualified image and the pre-work image is 600 or less, the threshold is fixed at 800; It is fixed at 1000, and in other cases, the above formula (13) is used.

In addition, the degree of similarity between the pass image and the pre-work image changes due to changes in the way the inspection object is observed before and after the work. For example, in the case of FIG. 66A to FIG. 66D corresponding to viewpoint information different from FIG. 65(A) to FIG. The degree of similarity to the pre-work image is higher than the above example. That is, similarly to the acceptable image and the inspection region, calculation processing of the similarity and the threshold is performed for each piece of viewpoint information included in the viewpoint information group.

Finally, the processing unit 11120 sets priority information indicating the priority of moving the imaging unit 5000 to the viewpoint position and line of sight direction corresponding to the viewpoint information for each viewpoint information in the viewpoint information group ( S100006 ). As described above, the observation method of the inspection object changes in accordance with the viewpoint position and line-of-sight direction indicated by the viewpoint information. As a result, a situation may arise in which a region to be inspected in an inspection target object can be well observed from given viewpoint information, while this region cannot be observed from other viewpoint information. In addition, as described above, since the viewpoint information group contains a sufficient number of viewpoint information, it is not necessary to inspect all of them in the inspection process. If the specified viewpoint information (for example, two positions) is qualified, the final result It is also qualified, so that the viewpoint information that has not been targeted before can not be processed. From the above, in consideration of efficiency of inspection processing, it is preferable to further prioritize viewpoint information useful in inspection processing, such as a region to be inspected, which can be observed well. Therefore, in this embodiment, priority is set for each piece of viewpoint information.

Here, when the inspection processing is processing for the result of the robot work, it is useful for the inspection to clarify the difference between before and after the work. As an extreme example, as shown in FIG. 67A , consider the task of assembling a small object B on a large object A from the left side of the drawing. In this case, when using viewpoint information 1 corresponding to viewpoint position 1 and gaze direction 1 in FIG. 67A , there is no difference between the pre-work image as shown in FIG. 67B and the pass image as shown in FIG. 67C . That is, viewpoint information 1 is not useful viewpoint information when checking the work here. On the other hand, when the viewpoint information 2 corresponding to the viewpoint position 2 and the gaze direction 2 is used, the pre-work image is as shown in FIG. 67D and the acceptable image is as shown in FIG. 67E . The difference is clear. In this case, the priority of viewpoint information 2 may be set higher than the priority of viewpoint information 1 .

That is, the larger the amount of change before and after the job, the higher the priority may be set, and the case where the amount of change before and after the job is large indicates the degree of similarity between the pre-job image and the acceptable image as described with reference to FIGS. 65A to 66D Low. Therefore, in the process of S100006, the degree of similarity between the pre-operation image and the acceptable image of each of the pieces of viewpoint information is calculated (this is obtained when setting the threshold value in S100005), and the lower the degree of similarity is, the higher the priority is set. That's it. When executing the inspection process, the imaging unit 5000 is moved sequentially from viewpoint information with higher priority to perform the inspection.

3.2 Online processing

Next, the flow of the online processing that is the inspection process using the second inspection information will be described with reference to the flowchart of FIG. 68 . When the online processing is started, first, the second inspection information generated by the above-mentioned offline processing is read in (S2001).

Then, the robot 30000 moves the imaging unit 5000 to the viewpoint position and line-of-sight direction corresponding to each viewpoint information in the viewpoint information group according to the movement order set based on the priority indicated by the priority information. This can be realized, for example, by the control of the processing unit 11120 in FIG. 50 or the control unit 3500 in FIG. 51A . Specifically, one viewpoint information with the highest priority among the plurality of viewpoint information included in the viewpoint information group is selected (S2002), and the imaging unit 5000 is moved to the viewpoint position and line-of-sight direction corresponding to the viewpoint information (S2003). In this way, efficient inspection processing can be realized by controlling the imaging unit 5000 according to the above-mentioned priorities.

However, viewpoint information in offline processing is basically information defined by the object coordinate system, and does not consider the position in real space (world coordinate system, robot coordinate system). For example, as shown in FIG. 69A , in the object coordinate system, the viewpoint position and viewpoint direction are set in the direction of a given surface F1 of the inspection object. In this case, in the inspection process of the inspection object, as shown in FIG. 69B , when the inspection object is arranged on the workbench with the surface F1 facing downward, the position of the viewpoint and the direction of the line of sight are the workbench. down, the imaging unit 5000 (the hand-eye camera of the robot 30000) cannot be moved to this position.

That is, S2003 is a control of not necessarily moving the imaging unit 5000 to the position and posture corresponding to the viewpoint information, but determining whether it is possible to move (S2004), and moving if possible. Specifically, the processing unit 11120 judges that the imaging unit 5000 cannot move to the viewpoint position and line-of-sight direction corresponding to the i-th (i is a natural number) viewpoint information among the plurality of viewpoint information based on the movable range information of the robot. Next, the movement of the imaging unit 5000 corresponding to the i-th viewpoint information is skipped, and the control corresponding to the j-th viewpoint information (j is a natural number satisfying i≠j) next to the i-th viewpoint information in the movement order is performed . Specifically, when the determination in S2004 is NO, the check process after S2005 is skipped, the process returns to S2002, and then the viewpoint information is selected.

Here, assuming that the number of viewpoint information included in the viewpoint information group is N (N is a natural number, and in the example of FIG. 58 above, N=18), i is an integer satisfying 1≤i≤N, and j is an integer satisfying An integer of 1≤j≤N and j≠i. In addition, the movable range information of the robot is information indicating a movable range of a part of the robot, in particular, where the imaging unit 5000 is installed. For each joint included in the robot, the range of the selected joint angle of the joint is determined in design. Furthermore, if the value of the joint angle of each joint is determined, a predetermined position of the robot can be calculated based on forward kinematics. That is, the movable range information is information obtained from design items of the robot, and may be a set of possible values of joint angles, a position and posture of the imaging unit 5000 in a space that can be taken, or other information.

In addition, the movable range information of the robot is represented by the robot coordinate system or the world coordinate system. Therefore, in order to compare the viewpoint information with the movable range information, it is necessary to convert the viewpoint information in the object coordinate system as shown in FIG. 69A into the positional relationship in the real space as shown in FIG. viewpoint information.

Thus, the information acquiring unit 11110 acquires in advance as the first inspection information object position and posture information indicating the position and posture of the inspection target object in the global coordinate system. The relative relationship between the global coordinate system and the object coordinate system is obtained, and the viewpoint information represented by the global coordinate system is obtained, and according to the movable range information of the robot represented by the global coordinate system and the viewpoint information represented by the global coordinate system, It is determined whether or not the imaging unit 5000 can be moved to the viewpoint position and line-of-sight direction indicated by the viewpoint information.

Since this processing is coordinate transformation processing, a relative relationship between the two coordinate systems is required, and this relative relationship can be obtained from the position and orientation of the reference object coordinate system in the global coordinate system, that is, object position and orientation information.

In addition, the gaze direction indicated by the viewpoint information does not necessarily have to be the only information for determining the posture of the imaging unit 5000 . Specifically, as described above in the description of the viewpoint candidate information in FIGS. 58 and 59 , the viewpoint position is determined by (x, y, z), and the viewpoint position is determined by (ax, ay, az) and (bx, by, bz). The posture of the imaging unit 5000 is determined, but (bx, by, bz) may not be considered. In determining whether it is possible to move the imaging unit 5000 to the viewpoint position and line-of-sight direction indicated by the viewpoint information, if (x, y, z), (ax, ay, az) and (bx, by, bz) If all the conditions are used as conditions, it is difficult to realize the movement of the imaging unit 5000 satisfying the conditions. Specifically, even if (ax, ay, az) in the direction of the origin can be imaged from a position represented by (x, y, z), the vector representing the rotation angle around (ax, ay, az) at that time Also, only the specified range is taken, and (bx, by, bz) may not be satisfied. Therefore, in this embodiment, the line of sight direction does not need to include (bx, by, bz), and if (x, y, z) and (ax, ay, az) are satisfied, it is determined that the imaging unit 5000 can move to the viewpoint. information.

When the imaging unit 5000 has completed moving to the position and posture corresponding to the viewpoint information, imaging by the imaging unit 5000 of the position and posture is performed to obtain a captured image ( S2005 ). Inspection processing is performed by comparing a captured image with a pass image. In the pass image, predetermined values are used for (bx, by, bz). The possibility that the rotation angle of the viewing direction is different from the angle indicated by (bx, by, bz). For example, as in the case where the acceptable image is shown in FIG. 70A and the captured image is shown in FIG. 70B , a rotation at a given angle may occur between the two images. In this case, it is inappropriate to cut out the inspection area, and it is also inappropriate to calculate the similarity. In addition, for convenience of explanation, the background of FIG. 70B is a single solid color like the acceptable image created from the model data. However, since FIG. 70B is a captured image, other objects may be included. In addition, from the perspective of the relationship between illumination light and the like, it is conceivable that the color tone of the object to be inspected is different from that of the acceptable image.

Thus, here, calculation processing of the image rotation angle between the captured image and the acceptable image is performed (S2006). Specifically, since the above-mentioned (bx, by, bz) is used when generating the acceptable image, the rotation angle of the imaging unit (virtual camera) corresponding to the acceptable image with respect to the line-of-sight direction is known information. In addition, the position and posture of the imaging unit 5000 at the time of capturing the captured image should also be known in the robot control for moving the imaging unit 5000 to the position and posture corresponding to the viewpoint information, otherwise, the imaging unit 5000 cannot move at all. In this way, information on the rotation angle of the imaging unit 5000 with respect to the line-of-sight direction at the time of imaging can also be obtained. In the process of S2006, the rotation angle between images is obtained from the difference of the two rotation angles with respect to the line-of-sight direction. Then, at least one of the acceptable image and the captured image is rotated and deformed using the obtained image rotation angle, thereby correcting the angle difference between the acceptable image and the captured image.

Since the correspondence between the acceptable image and the angle of the captured image is obtained through the above processing, the inspection area obtained in S100004 is extracted from each image (S2007), and the confirmation process is performed using this area (S2008). In S2008, the degree of similarity between the acceptable image and the captured image is calculated, and if the similarity is greater than or equal to the threshold obtained in S100005, it is judged as passing, otherwise, it is sufficient to judge as failing.

However, instead of performing inspection processing from only one viewpoint information as described above, a plurality of viewpoint information may be used. Thereby, it is judged whether the confirmation process of a designated number of times has been performed (S2009), and when it has performed the designated number of times, the process is complete|finished. For example, if there is no problem in the confirmation process at three positions, if it is an inspection process that is passed, then in the case of three passes in S2008, it is YES in S2009, and the object to be inspected is Pass and end the inspection process. On the other hand, even if it is passed in S2008, if the judgment is the first or second time, the judgment in S2009 is negative, and the processing for the next viewpoint information is continued.

In addition, in the above description, the online processing is also performed by the information acquisition unit 11110 and the processing unit 11120 , but it is not limited thereto, and the above processing may be performed by the control unit 3500 of the robot 30000 as described above. That is, the online processing can be performed by the information acquisition unit 11110 and the processing unit 11120 of the robot 30000 in FIG. 50 , or can be performed by the robot control unit 3500 in FIG. 51A . Alternatively, it may be performed by the information acquisition unit 11110 and the processing unit 11120 of the processing device in FIG. 51A , and the processing device 10000 in this case can be considered as a robot control device as described above.

In the case of using the control unit 3500 of the robot 30000 to perform online processing, the control unit 3500 of the robot 30000 determines that it is impossible for the imaging unit 5000 to align with the i-th (i is a natural number) when the viewpoint position and line-of-sight direction corresponding to the viewpoint information move, the movement of the imaging unit 5000 corresponding to the i-th viewpoint information is skipped, and the j-th ( j is a natural number that satisfies i≠j) Relative control of viewpoint information.

In addition, the processing device 10000 etc. of this embodiment may implement a part or most of the processing by a program. In this case, a processor such as a CPU executes the program, thereby realizing the processing device 10000 and the like of the present embodiment. Specifically, a program stored in a non-transitory information storage medium is read, and a processor such as a CPU executes the read program. Here, the information storage medium (a medium that can be read by a computer) is a medium that stores programs, data, etc., and its function can be performed by an optical disk (DVD, CD, etc.), HDD (hard disk drive), or a memory (card memory, ROM, etc.) ) and so on to achieve. Furthermore, a processor such as a CPU performs various processes in the present embodiment based on a program (data) stored in an information storage medium. That is, a program for causing a computer (a device including an operation unit, a processing unit, a storage unit, and an output unit) to function as each unit of the present embodiment (a program for causing the computer to execute the processing of each unit) is stored in the information storage medium. .

Claims (12)

1. A robot control system, characterized in that, comprising: a captured image acquisition section that acquires a captured image; and a control unit that controls the robot based on the captured image, The photographed image acquisition unit acquires the photographed image of at least the assembled object among the assembled object and the assembled object on which the assembly work is reflected, The control unit performs feature amount detection processing of the object to be assembled based on the captured image, and moves the object to be assembled based on the feature amount of the object to be assembled. 2. The robot control system according to claim 1, wherein: The control unit performs the feature value detection process of the assembly object and the assembly object based on one or more captured images showing the assembly object and the assembly object, And based on the feature value of the assembly object and the feature value of the assembly object, the relative position and posture relationship between the assembly object and the assembly object becomes the target relative position and posture relationship, so that The object to be assembled moves. 3. The robot control system according to claim 2, wherein: The control unit, based on the feature value set as the target feature value among the feature values of the object to be assembled, and the feature value set as the attention feature value among the feature values of the assembly target object, The object to be assembled is moved so that the relative position and posture relationship becomes the target relative position and posture relationship. 4. The robot control system according to any one of claims 1 to 3, wherein: The control unit moves the object to be assembled so that the feature point of interest of the object to be assembled coincides with or approaches the target feature point of the object to be assembled. 5. The robot control system according to any one of claims 1 to 4, wherein: including a reference image storage unit that stores a reference image displaying the assembly object in a target position and posture, The control unit moves the object to be assembled toward the target position based on the first photographed image showing the object to be assembled and the reference image, After the object to be assembled is moved, the feature value detection process of the object to be assembled is performed based on the second captured image showing at least the object to be assembled, and the The feature quantity moves the assembly object. 6. The robot control system according to any one of claims 1 to 4, wherein: The control unit performs the feature quantity detection process of the first object to be assembled based on the first captured image showing the first object to be assembled in the assembly work, and moving the assembly object based on the feature value of the first object to be assembled, After moving the object to be assembled, performing the process of detecting the feature amount of the second object to be assembled based on a second captured image showing at least the second object to be assembled, and moving the assembly object and the first assembly object based on the feature value of the second assembly object. 7. The robot control system according to any one of claims 1 to 4, characterized in that, The control unit performs the assembly operation of the assembly object and the first assembly object based on one or a plurality of first captured images showing the assembly object and the first assembly object. The feature amount detection process of the object, and according to the feature value of the object to be assembled and the feature value of the first object to be assembled, the relative position and posture relationship between the object to be assembled and the first object to be assembled becomes the first target relative position The manner in which the pose relationship causes the assembly object to move, and performing the feature value detection process of the second object to be assembled based on the second captured image of the second object to be assembled, And based on the feature value of the first object to be assembled and the feature value of the second object to be assembled, the relative position and posture relationship between the first object to be assembled and the second object to be assembled The object to be assembled and the first object to be assembled are moved so that the relative positional relationship of the second object is achieved. 8. The robot control system according to any one of claims 1 to 4, wherein: The control unit executes the process of the assembly object, the assembly object, the assembly object, and the second assembly object based on one or more captured images showing the assembly object, the first assembly object, and the second assembly object. the feature quantity detection process of the first assembly object and the second assembly object, and according to the feature value of the object to be assembled and the feature value of the first object to be assembled, the relative position and posture relationship between the object to be assembled and the first object to be assembled becomes the first target relative position The manner in which the pose relationship causes the assembly object to move, And based on the feature value of the first object to be assembled and the feature value of the second object to be assembled, the relative position and posture relationship between the first object to be assembled and the second object to be assembled The first object to be assembled is moved so that the relative position and posture relationship of the second target is achieved. 9. The robot control system according to any one of claims 1 to 4, wherein: The control unit performs the feature quantity detection process of the second object to be assembled based on the first captured image showing the second object to be assembled in the assembly work, and moving the first object to be assembled according to the characteristic amount of the second object to be assembled, and performing the feature quantity detection process of the first assembled object according to the second captured image reflecting the moved first assembled object, And the object to be assembled is moved based on the characteristic amount of the first object to be assembled. 10. The robot control system according to any one of claims 1 to 9, wherein: The control unit controls the robot by performing visual servoing based on the captured image. 11. A robot, characterized in that it comprises: a captured image acquisition section that acquires a captured image; and a control unit that controls the robot based on the captured image, The photographed image acquisition unit acquires the photographed image of at least the assembled object among the assembled object and the assembled object on which the assembly work is reflected, The control unit performs feature amount detection processing of the object to be assembled based on the captured image, and moves the object to be assembled based on the feature amount of the object to be assembled. 12. A robot control method, characterized in that, comprising: A step of acquiring a photographed image of at least the object to be assembled among the object to be assembled and the object to be assembled on which the assembly work is reflected; performing a feature amount detection process of the object to be assembled based on the captured image; and A step of moving the object to be assembled based on the characteristic amount of the object to be assembled.