CN110919658B - Robot calibration method based on vision and multi-coordinate system closed-loop conversion - Google Patents

Abstract

The invention discloses a robot calibration method based on vision and multi-coordinate system closed-loop conversion. The present invention provides a robot calibration method based on vision and multi-coordinate system closed-loop conversion, which can solve the problem that the traditional robot calibration method needs to construct an AX=XB equation with a huge amount of calculation and reduce the calculation burden; a, b, c three The combination of two coordinate system conversion closed loops improves the accuracy of the conversion relationship between each coordinate system; the nonlinear least squares method is used to optimize the obtained robot end position error to improve the robot's positioning accuracy.

Description

A robot calibration method based on vision and multi-coordinate system closed-loop transformation

technical field

The invention relates to the technical field of robots and image recognition, in particular to a robot calibration method based on vision and multi-coordinate system closed-loop conversion.

Background technique

With the rapid development of big data, artificial intelligence, image recognition and other technologies in the information age, robot technology has also made great progress. Industrial robots have the characteristics of simple structure, high flexibility, and large working space, and are widely used in automotive, logistics, electronics, medical, aerospace and other fields. Robot positioning accuracy is divided into repetitive positioning accuracy and absolute positioning accuracy. Nowadays, the repetitive positioning accuracy of robots is high, but the absolute positioning accuracy is low. Therefore, improving the absolute positioning accuracy of the robot is also one of the important fields of robotics research.

Contents of the invention

The technical problem to be solved by the invention is: the absolute positioning accuracy of the uncalibrated robot is low.

In order to solve the above technical problems, the technical solution of the present invention is to provide a robot calibration method based on vision and multi-coordinate closed-loop conversion, which is characterized in that it includes the following steps:

Step 1: Establish the Eye-to-hand basic model with the camera fixed outside the robot body, and the chess board is fixed at the end of the robot as a target; the robot drives the chess board to move within the range where the camera can capture suitable images, and uses the camera The captured chessboard images in different poses are used for camera calibration;

Step 2: Perform hand target calibration, including the following steps:

Step 201: Fix the calibration pen as a target on the end of the robot, set 9 marking points in space based on the coordinate system of the robot end joint, and the robot drives the calibration pen to fix the point at each marking point with different postures until the completion of 9 For fixed points of marked points, the corresponding points projected from points D ₁ to D ₉ on the robot end joint coordinate system to the image coordinate system are respectively d ₁ to d ₉ , which are represented by Euclidean transformation from D ₁ to D ₉ to d ₁ to The conversion of d ₉ , thus obtaining the conversion relationship between the image coordinate system and the robot end joint coordinate system;

Step 202: Based on the conversion relationship between the image coordinate system and the robot end joint coordinate system obtained in step 201, deduce the conversion relationship T _cm between the camera coordinate system and the robot end joint coordinate system, that is, complete the hand-eye calibration;

Step 203: Based on the transformation relationship T _cm , combined with the camera parameters obtained after camera calibration in step 1, deduce the transformation relationship T _bc between the target coordinate system and the camera coordinate system;

Step 204: From T _mb =T _cm T _bc , deduce the conversion relationship T _mb between the robot end joint coordinate system and the target coordinate system, that is, complete the hand-target calibration;

Step 3: Combining with the pre-calibration data, deduce the position of the end of the robot in the robot base coordinate system;

Step 4: Compare the end position of the robot obtained above with the expected end position of the robot to obtain the error between them;

Step 5: Establish the error constraint equation according to the error of the end position of the robot obtained in step 4, and then apply the nonlinear least square method to optimize, find the local minimum value of the error function through non-stop iterative calculation, and think that the local minimum value can make The objective function obtains the optimal solution, and the parameter calibration is completed.

Preferably, said step 1 specifically includes the following steps:

Step 101: Make the Z-axis direction of the calibration plate coordinate system consistent with the Z-axis direction of the flange plate coordinate system, change the pose of the chessboard on the end of the robot within the range where the camera can capture a suitable image, so that the fixed camera Collect multiple chessboard images in different poses;

f_x、f_y表示焦距，u₀和u₀表示的是相机光轴与图像平面的交点；相机的外参数矩阵为/>

R表示从世界坐标系到相机坐标系的旋转矩阵，R＝R_xR_yR_z，R_x、R_y、R_z表示相机坐标系绕世界坐标系的x,y,z轴旋转，T表示从世界坐标系到相机坐标系的平移矩阵，T＝[t_x t_y t_z]，T_x、T_y、T_z表示相机坐标系沿着世界坐标系的x,y,z轴平移，则Step 102: Extract the corner points in the checkerboard image and use the cornerSubPix() function in OpenCV to accurately position the corner points to sub-pixel precision, and perform camera calibration; obtain the internal and external parameter matrix of the camera, wherein, the internal parameter matrix of the camera for

f _x , f _y represent the focal length, u ₀ and u ₀ represent the intersection of the camera optical axis and the image plane; the external parameter matrix of the camera is />

R represents the rotation matrix from the world coordinate system to the camera coordinate system, R=R _x R _y R _z , R _x , R _y , R _z represent the rotation of the camera coordinate system around the x, y, z axes of the world coordinate system, and T represents The translation matrix from the world coordinate system to the camera coordinate system, T=[t _x t _y t _z ], T _x , T _y , T _z represent the translation of the camera coordinate system along the x, y, z axes of the world coordinate system, then

The transformation formula from the camera coordinate system to the world coordinate system is:

In formula (1), (X _w , Y _w , Z _w ) represents a point in the world coordinate system; (X _c , Y _c , Z _c ) represents the corresponding point in the camera coordinate system;

The transformation relationship between the image coordinate system and the world coordinate system is:

The point (X, Y, Z) in the world coordinate system corresponding to the point (x, y) in the image coordinate system is calculated by formula (2).

Preferably, said step 3 includes the following steps:

Step 301: Perform pre-calibration with fixed points in space, move the calibration pen to the set 9 marked points in sequence under the coordinate system of the end joint of the robot. At this time, the robot does not need to change its posture at each marked point. The camera takes pictures of it, and records the coordinates P′ _{(X, Y, Z)} of the end of the robot at each marking point while taking pictures;

Step 302: Preprocess the calibration pen images captured by the camera at different marking points; then further process to obtain the pre-calibrated T _cm ′ _(1～9) , T _cm ′ _(1～9) represents 9 groups of pre-calibrated camera coordinates system and the robot end joint coordinate system, T _cm ′ _(1～9) combines the recorded robot end coordinates to obtain the conversion relationship T _{cw (1～9 )} , take the average value of 9 sets of data as the conversion relationship T _cw between the final camera coordinate system and the world coordinate system of the robot;

Step 303: Combining the conversion relationship T _cw obtained by pre-calibration, the conversion relationship T _bc between the target coordinate system and the camera coordinate system, and T _mb obtained by hand-target calibration, the end position of the robot can be deduced, and compared with the expected manipulator The end position is compared to get the error.

Compared with the prior art, the present invention has the following advantages:

The present invention provides a robot calibration method based on vision and multi-coordinate system closed-loop conversion, which can solve the problem that the traditional robot calibration method needs to construct an AX=XB equation with a huge amount of calculation and reduce the calculation burden; a, b, c three The combination of two coordinate system conversion closed loops improves the accuracy of the conversion relationship between each coordinate system; the nonlinear least squares method is used to optimize the obtained robot end position error to improve the robot's positioning accuracy.

Description of drawings

Fig. 1 is a structural schematic diagram 1 of a robot calibration method based on vision and multi-coordinate system closed-loop conversion;

Fig. 2 is a structural schematic diagram 2 of a robot calibration method based on vision and multi-coordinate system closed-loop conversion;

Fig. 3 is a kind of coordinate system conversion relationship diagram of a robot calibration method based on vision and multi-coordinate system closed-loop conversion;

Fig. 4 is a flowchart of a robot calibration method based on vision and multi-coordinate system closed-loop conversion.

icon:

1-robot body; 2-calibration board; 3-camera; 4-calibration pen; {W}-world coordinate system of the robot; {M}-coordinate system of robot end joints; {B}-coordinate system of calibration board or calibration pen ; {C} - camera coordinate system.

Detailed ways

Below in conjunction with specific embodiment, further illustrate the present invention. It should be understood that these examples are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the teachings of the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by the appended claims of the present application.

The invention combines a camera with a robot to invent a robot calibration method based on vision and multi-coordinate system closed-loop conversion. This method can solve the problem that the traditional robot calibration method needs to construct the AX=XB equation with a huge amount of calculation and reduce the calculation burden; the combination of the three coordinate system conversion closed loops of a, b and c improves the accuracy of the conversion relationship between the coordinate systems ; Apply the nonlinear least squares method to optimize the obtained robot end position error to improve the positioning accuracy of the robot.

A robot calibration method based on vision and multi-coordinate system closed-loop transformation provided by the present invention specifically includes the following steps:

Step 1: Establish the Eye-to-hand basic model with the camera fixed outside the robot body. The chess board is fixed at the end of the robot as a target. As shown in Figure 1, the Z axis of the calibration board coordinate system and the flange plate coordinate system The Z axis stays the same. The robot drives the chessboard to move within the range where the camera can capture suitable images, and uses the calibration board images of different poses captured by the camera to perform camera calibration.

In this step, the camera external reference and internal reference calibration specifically includes the following steps:

(1) By continuously changing the pose of the calibration board at the end position of the robot, the fixed camera captures multiple chessboard images.

(2) Extract the corner points in the checkerboard image and use the cornerSubPix() function in OpenCV to make the corner point position accurate to sub-pixel level precision, perform camera calibration, and obtain the internal and external parameter matrix of the camera.

The transformation formula from the camera coordinate system to the world coordinate system is:

为相机的外参矩阵；R表示从世界坐标系到相机坐标系的旋转矩阵，R＝R_xR_yR_z，R_x、R_y、R_z表示相机坐标系绕世界坐标系的x,y,z轴旋转；T表示从世界坐标系到相机坐标系的平移矩阵，T＝[t_x t_y t_z]，T_x、T_y、T_z表示相机坐标系在世界坐标系的x,y,z轴上的平移。In formula (1), ,, (X _w , Y _w , Z _w ) represent points in the world coordinate system; (X _c , Y _c , Z _c ) represent corresponding points in the camera coordinate system;

is the external parameter matrix of the camera; R represents the rotation matrix from the world coordinate system to the camera coordinate system, R=R _x R _y R _z , R _x , R _y , R _z represent the x, y of the camera coordinate system around the world coordinate system , z-axis rotation; T represents the translation matrix from the world coordinate system to the camera coordinate system, T=[t _x t _y t _z ], T _x , T _y , T _z represent the x, y of the camera coordinate system in the world coordinate system , the translation on the z-axis.

The transformation relationship between the image coordinate system and the world coordinate system:

为相机的内参矩阵；f_x、f_y表示焦距，一般情况下，二者相等；u₀，v₀表示的是相机光轴与图像平面的交点，通常位于图像中心处，故其值常取分辨率的一半；In formula (2), S represents the proportional coefficient;

is the internal reference matrix of the camera; f _x , f _y represent the focal length, and generally they are equal; u ₀ , v ₀ represent the intersection point of the camera optical axis and the image plane, which are usually located at the center of the image, so their values are often taken as half the resolution;

The point (X, Y, Z) in the world coordinate system corresponding to the point (x, y) in the image coordinate system can be calculated by formula (2).

Further, the transformation relationship between the world coordinate system and the pixel coordinate system can be obtained:

In formulas (4) and (5), f represents the focal length of the camera; (u, v) represents a point in the pixel coordinate system.

Step 2: Replace the calibration board with a calibration pen, which is fixed at the end of the robot as a target, as shown in Figure 2, and set 9 marker points in space based on the coordinate system of the robot’s end joints. The robot drives the calibration pen to fix the points at the marked points with different postures until the fixed points of the 9 marked points are completed, and the conversion relationship between the image coordinate system and the robot end joint coordinate system is obtained. Furthermore, the conversion relationship T _cm between the camera coordinate system and the robot end joint coordinate system can be deduced to complete the hand-eye calibration. As shown in the coordinate system conversion closed loop c in Figure 3, the conversion relationship T _bc between the target coordinate system and the camera coordinate system can be deduced by combining the camera calibration; and then the conversion relationship between the robot end joint coordinate system and the target coordinate system can be deduced T _mb is to complete the hand-target calibration.

In this step, the hand target calibration specifically includes the following steps:

Replace the calibration board with a calibration pen, use the calibration pen as the target, and make the coordinate system of the calibration pen consistent with the previous calibration board coordinate system, as shown in Figure 2. Set a cube in space on the coordinate system of the end joint of the robot, and the cube is within the appropriate shooting range of the camera, and the 8 vertices and 1 center point of the cube are used as the marking points D ₁ ~ D ₉ ; the robot drives the calibration The pen fixes the points with 4 different postures at each marked point until the fixed points of 9 marked points are completed; it can be seen that the corresponding points projected from the points D ₁ to D ₉ on the end joint coordinate system of the robot to the image coordinate system are respectively d ₁ ～d ₉ , through the Euclidean transformation, that is, the rotation vector and the translation vector:

D _i =R·d _i -t(i∈(1,9)) (6)

Equation (6) represents the transformation from D ₁ to D ₉ to d ₁ to d ₉ , and in Equation (6), t represents a translation vector. Thus, the conversion relationship between the image coordinate system and the robot end joint coordinate system can be obtained. Combining the camera parameters and formula (7) obtained from the previous camera calibration:

Obtain the conversion relationship T _cm between the camera coordinate system and the robot end joint coordinate system to complete the hand-eye calibration; as shown in the coordinate system conversion closed loop c in Figure 3, the conversion relationship T _mb between the robot end joint coordinate system and the target can be obtained = The conversion relationship T _cm between the camera coordinate system and the robot end joint coordinate system × the conversion relationship T _bc between the target coordinate system and the camera coordinate system (T _mb = T _cm T _bc ), deduce the robot end joint coordinate system and the target coordinates The conversion relationship T _mb between systems is the hand-target calibration. .

Step 3: Combining with the pre-calibration data, deduce the position of the end of the robot in the robot base coordinate system.

In this step, the pre-calibration specifically includes the following steps:

The pre-calibration is carried out by the fixed-point method of spatial position. Through offline programming, the robot is controlled to move the calibration pen to the set 9 marking points in sequence. At this time, the robot does not need to change its posture at each marking point. The camera fixed outside the robot takes pictures of them and records them at the same time. The coordinates P′ _{(X, Y, Z)} of the end of the robot at each marked point and the rotation angles Δ′ _(1~6) of the six joints.

The image of the calibration pen collected by the camera is preprocessed, and then further processed to obtain the pre-calibrated T _cm ′; according to the closed loop b of the coordinate system conversion in Figure 3, T ₆ = T _cm T _cw can be obtained, combined with the recorded end of the robot Coordinates can obtain the conversion relationship T _cw between the camera coordinate system and the robot's world coordinate system.

As shown in Figure 3, the coordinate system conversion closed loop a shows that T ₆ = T _mb T _bc T _cw , and the T _cw obtained by pre-calibration, the conversion relationship T _bc between the target coordinate system and the camera coordinate system obtained earlier, and the hand target calibration can be obtained Combining T _mb of the robot can deduce the end position of the robot P _{1 (X, Y, Z)} , and compare it with the expected end position of the robot P _{0 (X, Y, Z)} to obtain the error.

Step 4: Compare the end position of the robot obtained above with the expected end position of the robot to obtain the error between them.

Step 5: Establish the error constraint equation according to the position error of the robot end, and then apply the nonlinear least square method to optimize, find the local minimum value of the error function through non-stop iterative calculations, and think that the local minimum value can make our objective function obtain Optimal solution (minimum value), complete parameter calibration.

In this step, after the error is obtained, the nonlinear least squares method is used for optimization, which specifically includes the following steps:

The purpose of parameter optimization using the least square method is to find a set of appropriate geometric parameter values, under which the robot's end positioning error can be minimized.

The objective function that can be constructed for nonlinear least squares optimization is:

In formula (8), P _0(X,Y,Z)i represents the i-th expected robot end position; P _0(X,Y,Z)j represents the j-th expected robot end position; P _{1(X ,Y,Z)i} represents the derived i-th end position of the robot; P _1(X,Y,Z)j represents the derived j-th end position of the robot.

This is a typical problem of estimating nonlinear static model parameters with the criterion of minimizing the sum of squared errors. The minimum value of the objective function is solved by an iterative optimization algorithm. The algorithm adopts the Levenberg-Marquardt algorithm, and its calculation form is:

H _LM ＝-(J ^T J+μI) ^-1 J ^T e (9)

In formula (9), H _LM is the step size of the algorithm, J is the Jacobian matrix of the error function, e is the error function, and μ is a positive number.

Bring the robot end position coordinates P _{0 (X, Y, Z)} and P _{1 (X, Y, Z)} into the calculation objective function, then calculate the iteration step size H _LM , correct the parameter value and repeat the above steps until reaching the maximum number of iterations Or the objective function is reduced to the required standard.

By adopting the above-mentioned technical solution, vision is applied to robot calibration to obtain the conversion relationship between coordinates, and to better improve the positioning accuracy of the robot.

Claims (1)

1. A robot calibration method based on vision and multi-coordinate system closed-loop conversion, is characterized in that, comprises the following steps: Step 1: Establish the Eye-to-hand basic model with the camera fixed outside the robot body, and the chess board is fixed at the end of the robot as a target; the robot drives the chess board to move within the range where the camera can capture a suitable image, and uses the camera The captured chessboard image is used for camera calibration. The Z axis of the calibration plate coordinate system is consistent with the Z axis of the flange plate coordinate system. The robot drives the chessboard to move within the range where the camera can capture a suitable image. The image of the calibration board of the attitude is used for camera calibration, including the following steps: Step 101: Make the Z-axis direction of the calibration plate coordinate system consistent with the Z-axis direction of the flange plate coordinate system, change the pose of the chessboard on the end of the robot within the range where the camera can capture a suitable image, so that the fixed camera Collect multiple chessboard images in different poses; Step 102: Extract the corner points in the checkerboard image and use the cornerSubPix() function in OpenCV to accurately position the corner points to sub-pixel precision, and perform camera calibration; obtain the internal and external parameter matrix of the camera, wherein, the internal parameter matrix of the camera for

f _x , f _y represent the focal length, u ₀ , v ₀ represent the intersection of the camera optical axis and the image plane; the external parameter matrix of the camera is />

R represents the rotation matrix from the world coordinate system to the camera coordinate system, R=R _x R _y R _z , R _x , R _y , R _z represent the rotation of the camera coordinate system around the x, y, z axes of the world coordinate system, T Represents the translation matrix from the world coordinate system to the camera coordinate system, T=[t _x t _y t _z ], T _x , T _y , T _z represent the translation of the camera coordinate system along the x, y, z axes of the world coordinate system, but The transformation formula from the camera coordinate system to the world coordinate system is:

In formula (1), (X _w , Y _w , Z _w ) represents a point in the world coordinate system; (X _c , Y _c , Z _c ) represents the corresponding point in the camera coordinate system; The transformation relationship between the image coordinate system and the world coordinate system is:

S represents the scale factor, and the point (X, Y, Z) in the world coordinate system corresponding to the point (x, y) in the image coordinate system is calculated by formula (2); Step 2: Perform hand target calibration, including the following steps: Step 201: Fix the calibration pen as a target on the end of the robot, set 9 marking points in space based on the coordinate system of the robot end joint, and the robot drives the calibration pen to fix the point at each marking point with different postures until the completion of 9 For fixed points of marked points, the corresponding points projected from points D ₁ to D ₉ on the robot end joint coordinate system to the image coordinate system are respectively d ₁ to d ₉ , which are represented by Euclidean transformation from D ₁ to D ₉ to d ₁ to The conversion of d ₉ , thus obtaining the conversion relationship between the image coordinate system and the robot end joint coordinate system; Step 202: Based on the conversion relationship between the image coordinate system and the robot end joint coordinate system obtained in step 201, deduce the conversion relationship T _cm between the camera coordinate system and the robot end joint coordinate system, that is, complete the hand-eye calibration; Step 203: Based on the transformation relationship T _cm , combined with the camera parameters obtained after camera calibration in step 1, deduce the transformation relationship T _bc between the target coordinate system and the camera coordinate system; Step 204: From T _mb =T _cm T _bc , deduce the conversion relationship T _mb between the robot end joint coordinate system and the target coordinate system, that is, complete the hand-target calibration; In the above steps, the Euclidean transformation is the rotation vector and the translation vector: D _i = R·d _i +t(i∈(1,9)) The above formula represents the conversion from D ₁ to D ₉ to d ₁ to d ₉ , where t represents the translation vector, from which the conversion relationship between the image coordinate system and the robot end joint coordinate system can be obtained, and the camera parameters obtained by camera calibration and the formula:

Obtain the conversion relationship T _cm between the camera coordinate system and the robot end joint coordinate system to complete the hand-eye calibration, and obtain the conversion relationship T _mb between the robot end joint coordinate system and the target = conversion between the camera coordinate system and the robot end joint coordinate system The relationship T _cm × the conversion relationship T _bc between the target coordinate system and the camera coordinate system, T _mb = T _cm T _bc , deduce the conversion relationship T _mb between the robot end joint coordinate system and the target coordinate system, that is, hand target calibration; Step 3: Combining the pre-calibration data, deduce the position of the robot end in the robot base coordinate system, including the following steps: Step 301: Perform pre-calibration with fixed points in space, move the calibration pen to the set 9 marked points in sequence under the coordinate system of the end joint of the robot. At this time, the robot does not need to change its posture at each marked point. The camera takes pictures of it, and records the coordinates P′ _{(X, Y, Z)} of the end of the robot at each marking point while taking pictures; Step 302: Preprocess the calibration pen images captured by the camera at different marking points; then further process to obtain the pre-calibrated T _cm ′ _(1～9) , where T _cm ′ _(1～9) represents 9 groups of pre-calibrated cameras The conversion matrix between the coordinate system and the robot end joint coordinate system, T _cm ′ _(1～9) combines the recorded robot end coordinates to obtain the conversion relationship between the 9 sets of camera coordinate systems and the robot’s world coordinate system T _{cw (1～9 9)} Take the average value of 9 sets of data as the conversion relationship T _cw between the final camera coordinate system and the world coordinate system of the robot; Step 303: Combining the conversion relationship T _cw obtained by pre-calibration, the conversion relationship T _bc between the target coordinate system and the camera coordinate system, and T _mb obtained by hand-target calibration, the end position of the robot can be deduced, and compared with the expected manipulator The end position is compared to get the error; Step 4: Compare the end position of the robot obtained above with the expected end position of the robot to obtain the error between them; Step 5: Establish the error constraint equation according to the error of the end position of the robot obtained in step 4, and then apply the nonlinear least square method to optimize, find the local minimum value of the error function through non-stop iterative calculation, and think that the local minimum value can make The objective function obtains the optimal solution and completes parameter calibration. Among them, the objective function for constructing nonlinear least square method optimization is:

In the formula, P _0(X,Y,Z)i represents the i-th expected robot end position; P _0(X,Y,Z)j represents the j-th expected robot end position; P _{1(X,Y, Z)i} represents the derived i-th end position of the robot; P _1(X,Y,Z)j represents the derived j-th end position of the robot.

Images