CN102794763B - Calibration method of welding robot system based on line structured light vision sensor guidance - Google Patents

Abstract

The invention relates to a systematic calibration method of a welding robot guided by a line structured light vision sensor, which comprises the following steps: firstly, controlling a mechanical arm to change pose, obtaining a round target image through a camera, accomplishing the matching of the round target image and a world coordinate, and then obtaining an internal parameter matrix and an external parameter matrix RT of the camera; secondly, solving a line equation of a line laser bar by Hough transformation, and using the external parameter matrix RT obtained in the first step to obtain a plane equation of the plane of the line laser bar under a coordinate system of the camera; thirdly, calculating to obtain a transformation matrix of a tail end coordinate system of the mechanical arm and a base coordinate system of the mechanical arm by utilizing a quaternion method; and fourthly, calculating a coordinate value of a tail end point of a welding workpiece under the coordinate of the mechanical arm, and then calculating an offset value of the workpiece in the pose combined with the pose of the mechanical arm. The systematic calibration method of the welding robot guided by the line structured light vision sensor is flexible, simple and fast, and is high in precision and generality, good in stability and timeliness and small in calculation amount.

Description

Calibration method of welding robot system based on line structured light vision sensor guidance

technical field

The present invention relates to a robot vision sensor and its calibration method, in particular to a welding robot system calibration method guided by a line structured light vision sensor, in particular to a robot based on the hand-eye relationship matrix and sensor parameters of the line structured light vision sensor Quick Calibration Method.

Background technique

Welding is characterized by complex process factors, high labor intensity, long production cycle, and poor working environment. Its quality depends on the operator's skills, technology and experience, and is also related to the operator's emotional and physical conditions. Therefore, welding automation technology is very important for improving joint quality. Quality and ensuring stability are of great significance. The key issue in realizing welding automation is the automatic tracking of weld seams. The intelligent welding robot guided by laser vision combines weld image recognition with robot motion control technology, which can effectively solve the problem of automatic weld seam tracking.

Calibration (including camera parameter calibration, line laser light plane equation calibration and hand-eye transformation matrix calibration) is a very critical and important link in the visual measurement system. The accuracy of the calibration results and the stability and real-time performance of the algorithm directly affect the industrial Accuracy of measurement and tracking during production. The parameters of the line structured light vision sensor include the internal parameters of the camera (focal length, principal point and distortion coefficient, etc.) and the structural parameters of the line structured light vision sensor (that is, the light plane equation of the line laser in the camera coordinate system), through Tsai, Zhang Zhengyou , Hu Zhanyi et al.'s contribution in visual calibration, we can easily obtain the internal parameters of the camera. There are many methods for calibrating structural parameters of line structured light vision sensors, such as R. Drawing calibration method proposed by Dewar; D. Q． Huynh proposed a cross-ratio invariant calibration method; Bi Dexue's calibration of line laser light plane equations based on intersection lines has strong robustness, simple implementation methods, easy feature extraction, and meets the requirements of on-site calibration. According to the measurement model, to restore the three-dimensional coordinate information of the measured surface from the measurement data returned by the sensor must determine the hand-eye transformation matrix H from the end coordinate system of the manipulator to the camera coordinate system. In the field of machine vision this problem is called hand-eye calibration. The commonly used robot hand-eye calibration method is to use a known calibration reference object (calibration block) to control the manipulator to observe a known calibration reference object in different orientations, thereby deriving the relationship between R and t and multiple observation results, where R represents the hand-eye The rotation part of the transformation matrix H, and t represents the translation part of the hand-eye transformation matrix H.

Contents of the invention

The purpose of the present invention is to overcome the deficiencies in the prior art, and provide a welding robot system calibration method guided by a line-structured light vision sensor, which is flexible, high in precision, fast in speed, good in stability, strong in real-time, and simple in method , with a small amount of calculation and strong versatility.

According to the technical solution provided by the present invention, the calibration method of the welding robot system guided by the line structured light vision sensor, the calibration method of the welding robot system includes the following steps:

The first step is to control the robot arm to change the pose, so that the camera can shoot a circular target that is placed arbitrarily and in the same position in multiple poses. The selected pose must make all the dots on the circular target within the camera's field of view. , and ensure that the line laser light plane produced by the line laser fixed on the camera is also within the field of view on the circular target; after obtaining the circular target image through the camera, extract the circle center coordinates of the circular spot on the circular target image And identify the row and column values of the corner points to complete the matching of the circular target image and the world coordinates, and then obtain the camera's internal parameter matrix according to Zhang Zhengyou's calibration algorithm $A=\left[\begin{array}{ccc}\mathrm{\α}& \mathrm{\γ}& u_0\\ 0& \mathrm{\β}& v_0\\ 0& 0& 1\end{array}\right],$ External parameter matrix RT;

Among them, α=f/dx, β=f/dy, f is the focal length of the camera, dx, dy are the length and width _of a single CCD photosensitive element in the camera; v ₀ is the pixel coordinates of the intersection of the lens optical axis of the camera and the CCD photosensitive element;

The second step, according to the circular target image of the band laser light strip obtained in the first step, extract the line laser light strip, refine the extracted line laser light strip, and obtain the line equation of the line laser light strip by Hough transform; The external parameter matrix RT obtained in the first step obtains the plane equation of the line laser light strip plane in the camera coordinate system;

The third step is to calculate the transformation matrix between the end coordinate system of the robot arm and the base coordinate system of the robot arm by using the quaternion method according to the posture of the robot arm corresponding to each pose in the first step, that is, obtain the hand-eye transformation matrix H;

The fourth step is to place the welding workpiece on the end of the mechanical arm, control the welding workpiece to touch a point on the circular target precisely under the fixed position and posture of the end of the mechanical arm, calculate the coordinate value of the end point of the welding workpiece under the coordinates of the mechanical arm, and combine The robot arm pose calculates the offset value of the workpiece under the pose.

Described first step comprises the following steps:

1.1. Use the camera to collect the image of the circular target online in real time, and use the adaptive threshold of the Otsu method to binarize the image, so that the circular spots on the circular target image are highlighted;

1.2. Use the closed operation operator to perform closed operation on the circular target image to remove noise interference; mark all the target dots in the above-mentioned binarized circular target image, and the area of the target dot is the pixel contained in the target dot Number, count the number of pixels of the target dot in the above circular target image, the average number of pixels contained in the marked target dot is NP_average, remove the number of pixels contained in the marked target less than 0.5*NP_average and the number of pixels contained in the marked target greater than 1.5* target for NP_average;

1.3. Calculate the circumference C and area S of the target dot after marking, and use the circularity expression e=C ² /(2*π*S) to distinguish the target dot from interference;

1.4. Use the center of gravity method to obtain the center coordinates (x ₀ , y ₀ ) of each target point, ${\mathrm{the\; y}}_0=\frac{\underset{i=0}{\overset{m}{\mathrm{\Σ}}}\underset{j=0}{\overset{\mathrm{no}}{\mathrm{\Σ}}}\mathrm{Hit}\left[i\right]\left[j\right]*\mathrm{the\; y}}{\underset{i=0}{\overset{m}{\mathrm{\Σ}}}\underset{j=0}{\overset{\mathrm{no}}{\mathrm{\Σ}}}\mathrm{Hit}\left[i\right]\left[j\right]};$

Among them, m, n are the row and column values of the circle target image, Hit[i][j] indicates whether the (i, j) position point is on the target circle point, and the (i, j) position point is on the target circle point Hit [i][j]=1, otherwise Hit[i][j]=0;

1.5. There are several rows and columns of dots on the circular target. The distance between two adjacent rows of dots is 15mm, and the distance between the centers of two adjacent dots in the same row is 10mm. The first point of the row has a reference circle at 5mm in the direction of the row, and the pixel coordinates of the target circle are obtained according to step (1.4), and the center points on the circle target image are sorted by using the reference circle;

1.6. According to the sorting of the dots in the above circular target image and the given circular target information, complete the matching between the pixel coordinates of the circle center point in the circular target image and the world coordinates.

Described second step comprises the following steps:

2.1. The line laser is fixed on the camera. The camera only needs to ensure that all the target dots and light strips are within the field of view of the camera to collect pictures. Use the preset threshold to binarize the collected line laser light strip images. , perform image closing operation on the binarized image to remove edge singular points;

2.2. Carry out 8-neighborhood marking on the center of the line laser light bar area. The center point of the line laser light bar area is p1, and the 8 points in the neighborhood of the center point p1 of the line laser light bar area are respectively p2 , p3,..., p9, where the center point of point p2 is above p1, and the center point p1 of the line laser light strip area is marked with 8-neighborhood points while satisfying the following conditions:

(i), 2≤N(p1)≤6;

(ii), S(p1)=1;

(Ⅲ), p2*p4*p6=0;

(ⅳ), p4*p6*p8=0;

Among them, N(p1) is the number of non-zero neighbors of the center point p1; S(p1) is the number of times the values of these points change from 0→1 when the order is p2, p3,...,p9; when for all After checking all the boundary points of , remove all the marked points, iterate repeatedly until no point meets the marking condition, and complete the thinning of the light bar;

2.3. Use the Hough transform to extract the points obtained after refinement to obtain the line equation a _k x+b _k y+c _k =0 of the line laser light strip on the circular target plane, where a _k , b _k , and c _k represent parameters of the line equation.

The third step comprises the following steps:

3.1. The plane equation of the circular target in the world coordinate system can be expressed as In the formula: π ₁ =[0,0,1,0] ^T , using the external parameter matrix RT obtained in the first step, the plane equation of the target plane in the camera coordinate system is calculated as one of them and are the plane normal coordinate vectors of the target plane in the world coordinate system and the camera coordinate system respectively;

3.2. In the camera coordinate system, set represent the normal coordinate vector of the laser plane, the normal coordinate vector of the i-th circular target, the normal coordinate vector of the j-th circular target, the coordinate vector of the laser intersection image in the i-th circular target, and the j-th plane The coordinate vector of the laser intersection image in the target; note that the equation λ _i of the laser intersection image on the i-th plane target image and can be calculated directly:

${\stackrel{<span\; class="notranslate">\; \&Right\; Arrow;</span>}{\mathrm{<span\; class="notranslate">\; \&lambda;</span>}}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; A</span>}}^{\mathrm{<span\; class="notranslate">\; T</span>}}{\mathrm{<span\; class="notranslate">\; \&lambda;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ;</span>$

(3.3), according to the duality of the two laser intersection lines and their associated planes in the projection space, $\stackrel{\&Right\; Arrow;}{w}=\frac{1}{2}(\stackrel{\&Right\; Arrow;}{w_1}+\stackrel{\&Right\; Arrow;}{w_2});$ $\left\{\begin{array}{c}\stackrel{\&Right\; Arrow;}{w_1}=\stackrel{\&Right\; Arrow;}{w_i}+{\mathrm{\&alpha;}}_i\stackrel{\&Right\; Arrow;}{{\mathrm{\&lambda;}}_j}\\ \stackrel{\&Right\; Arrow;}{w_2}=\stackrel{\&Right\; Arrow;}{w_j}+{\mathrm{\&alpha;}}_j\stackrel{\&Right\; Arrow;}{{\mathrm{\&lambda;}}_j}\end{array}\right.,$

in $\left[\begin{array}{c}{\mathrm{\&alpha;}}_i\\ {\mathrm{\&alpha;}}_j\end{array}\right]={\mathrm{D.}}^{-1}Q,$ D is a 2×2 matrix, Q is a two-dimensional vector, defined as follows: $\mathrm{D.}=\left[\begin{array}{cc}(\stackrel{\&Right\; Arrow;}{{\mathrm{\&lambda;}}_i},\stackrel{\&Right\; Arrow;}{w_i})& -(\stackrel{\&Right\; Arrow;}{{\mathrm{\&lambda;}}_j},\stackrel{\&Right\; Arrow;}{w_i})\\ (\stackrel{\&Right\; Arrow;}{{\mathrm{\&lambda;}}_i},\stackrel{\&Right\; Arrow;}{w_j})& -(\stackrel{\&Right\; Arrow;}{{\mathrm{\&lambda;}}_j},\stackrel{\&Right\; Arrow;}{w_j})\end{array}\right],$ $Q=\left[\begin{array}{c}(\stackrel{\&Right\; Arrow;}{w_i},\stackrel{\&Right\; Arrow;}{w_j}-\stackrel{\&Right\; Arrow;}{w_i})\\ (\stackrel{\&Right\; Arrow;}{w_j},\stackrel{\&Right\; Arrow;}{w_i}-\stackrel{\&Right\; Arrow;}{w_j})\end{array}\right];$ is the desired laser plane equation.

, the fourth step includes the following steps:

4.1. Take any two poses in the first step, in Indicates the transformation matrix of the coordinate system at the end of the manipulator in two poses, so that Represents the transformation matrix of the camera coordinate system in two poses, so that Then it can be abbreviated as ΦH=HΘ;

4.2, ΦH=HΘ; expanded as

R _Φ R _H =R _H R _Θ ①

R _Φ t _H +t _Φ =R _H t _Θ +t _H ②

Wherein: R _Φ , R _Θ , R _H represents Φ, Θ, the corresponding rotation part in H, t _Φ , t _Θ , t _H represents Φ, Θ, for the translation part in H;

4.3 Order $q_{\mathrm{\&Phi;}}={[a_0,{\stackrel{\&Right\; Arrow;}{a}}^T]}^T,$ $q_h={[x_0,{\stackrel{\&Right\; Arrow;}{x}}^T]}^T,$ $q_{\mathrm{\&Theta;}}={[b_0,{\stackrel{\&Right\; Arrow;}{b}}^T]}^T$ is the quaternion corresponding to R _Φ , R _H , R _Θ , q _Φ is calculated from the homogeneous matrix Φ, and q _Θ is calculated from the homogeneous matrix Θ, so that pass Find q _H , and then convert q _H into the rotation part of the hand-eye matrix, where Ω(·) represents the antisymmetric matrix function, express The anti-symmetric matrix of ; x ₀ , x is the unknown quantity to be calculated in q _H ;

4.4. Bring the rotation matrix data into ② to find t _H , and obtain the hand-eye transformation matrix H.

Calculating the welding workpiece displacement value in the fourth step includes the following steps:

5.1. Control the welding workpiece to precisely touch the preset point on the circular target under the fixed end pose of the manipulator, read the current pose of the manipulator, and obtain the first coordinate value of the end coordinate of the manipulator;

5.2. Define the coordinates of the preset point in the world coordinate system. According to the external parameter matrix RT obtained in the first step, obtain the coordinates of the preset point in the camera coordinate system. According to the hand-eye transformation matrix H and the image of the circular target The pose calculation of the manipulator obtains the second coordinate value of the preset point in the base coordinate system of the manipulator, and compares the second coordinate value with the first coordinate value to obtain the displacement value of the welding workpiece under the current pose.

When using the reference circle to sort the center points on the circle target image, find out the two target circle points with the smallest distance between the center points of the effective circle points on the circle target image, and the circle farther away from the mean center of all the circle center points is defined as 0 point, the other is defined as 1 point, then set 0, 1 point is the matched point, find the nearest point to 1 point in the unmatched point, define 2 points, set 2 points as the matched point, find in the unmatched point The point nearest to point 2 defines 3 points, and sets 3 points as matched points...until all points are found, and the sorting of all the center points is completed.

Advantages of the present invention: By analyzing the imaging principle of the camera, the principle of structured light measurement and the working principle of the hand-eye system, the present invention designs a simple and flexible calibration method based on the three-dimensional tracking of the mechanical light-guided manipulator, which includes camera internal parameter calibration, line Laser light plane equation calibration, hand-eye transformation matrix calibration and workpiece offset calibration. This calibration method overcomes the shortcomings of the traditional laser light plane equation and hand-eye matrix calibration conditions and cumbersome calibration steps. The algorithm only needs to control the random movement of the manipulator in more than 3 poses to shoot the fixed target to complete the overall calibration task, and the structured light-guided manipulator tracking system using this method has high tracking accuracy. The algorithm makes it possible to adjust the structure of the line laser and the camera, and to calibrate the system structure parameters on site. The flexibility of the structured light guidance system is greatly increased, which is of great significance to the actual visual measurement and tracking, and has good practicability.

Description of drawings

Fig. 1 is a flow chart of the calibration of the present invention.

Fig. 2 is a small hole imaging model diagram of the camera of the present invention.

Fig. 3 is a schematic diagram of dots on the round target of the present invention.

Fig. 4 is a flow chart of the tracking of the line-structured light-guided robotic arm of the present invention.

Fig. 5 is a dual representation of two laser intersection lines in projection space according to the present invention.

Fig. 6 is a diagram of the relationship between the camera fixed at the end of the mechanical arm, that is, the hand-eye relationship of the present invention.

Detailed ways

The purpose of the present invention is to overcome the shortcomings of the traditional line laser light plane equation and hand-eye calibration conditions, such as harsh conditions and cumbersome calibration steps, and propose a flexible, high-precision, fast-speed, good stability, strong real-time performance, simple method, and small amount of calculation. , Universal calibration method.

The present invention will be further described below in conjunction with drawings and embodiments.

The present invention utilizes the circular target as shown in Figure 3 to realize rapid positioning of the center of the circle, automatically match the world coordinates and pixel coordinates of the feature points of the circle center, and is conducive to the automation of camera internal parameter calibration; analysis line structured light guides the robotic arm tracking system The mathematical model introduced the quaternion method to solve the hand-eye matrix, and simplifies the overall calibration process, only need to control the mechanical arm to shoot a fixed circular target in more than 3 arbitrary poses, to achieve camera calibration, line laser light plane equation calibration and hand-eye The calibration of the matrix provides sufficient conditions for accurate tracking; the precise extraction of the center of the line laser light strip can detect and extract image feature points in real time online, stably and accurately, ensuring the stability and reliability of the calibration and tracking system; it is completed according to the calibration Real-time and precise tracking of the three-dimensional feature points of the manipulator guided by the line structured light.

The present invention extracts the sub-pixel circle center coordinates of circular spots and automatically recognizes the row and column values of the circle spots, and automatically completes the matching technology with the world coordinates. In the edge to remove the non-target part, use the circularity equation to remove the interference point, mark the circular spot, extract the marked area, and use the center of gravity method after Gaussian filtering to extract the sub-pixel coordinates of the center of the dot. Set the center points of several rows and columns on the circular target, the row distance between the centers of two adjacent rows is 15mm, the distance between the two adjacent center points in the same row is 10mm, and there is a point at the first point of the first row at 5mm in the row direction. For the reference circle, first find the two points with the smallest distance between the centers of all valid circles. According to the arrangement of the center points on the above-mentioned circular target, the distance between the reference circle and the center of the circle below the reference circle is the closest; The circle far from the point mean center is defined as 0 point, and the other is defined as 1 point. Here, the reference circle is defined as 0, and the circle point directly below the reference circle is defined as 1; then set 0, 1 point as already Match point, find the point closest to point 1 in the unmatched point, define point 2, set point 2 as the matched point, find the point closest to point 2 in the unmatched point, define point 3, set point 3 as the matched point... Until all the points are found, the sorting of the center points is completed. According to the sorting of the circle points, and the given circle target information, the pixel coordinates of the circle center points in the circle target image are matched with the world coordinates.

The calibration method based on the dual relationship in the present invention is to calibrate the laser plane equation by using the dual relationship between two laser intersection lines and their associated planes in the projected space. After the calibration is completed, the plane equation of the laser light plane in the camera coordinate system is obtained.

The technology of using the quaternion method in the present invention to solve the hand-eye transformation matrix is to use the transformation matrix of the world coordinate system and the camera coordinate system and the pose matrix of the manipulator that have been calibrated, by constructing constraint equations, using the unit quaternion and The corresponding relationship of the rotation matrix is solved by the constraint method, and the hand-eye transformation matrix H is obtained. The feature points are quickly determined by using the images collected in real time on site, and the laser plane equation and the hand-eye transformation matrix obtained by the above method realize the accurate tracking of the three-dimensional feature points.

As shown in Figure 1, the calibration specific process of the present invention is as follows:

The calibration process consists of four parts: camera calibration, line laser light plane equation calibration, hand-eye transformation matrix calibration, and workpiece offset calibration. Band-pass filters and CCDs are used to collect laser light bar scanning weld seam images. The laser light strip is thinned by using the principle of central axis transformation, and the light strip thinning center is obtained. Control the manipulator to shoot fixed targets in more than 3 arbitrary poses, use the obtained circular target center and its matching data to calculate the internal and external parameters of the camera, and use the obtained external parameters and the line laser line equation extracted from the target image to calibrate the laser light plane equation , construct the ΦH=HΘ equation by using the external parameter change and the pose change of the manipulator in every two movements of the manipulator, and use the quaternion method to solve the hand-eye transformation matrix H.

The first step is to control the robot arm to change the pose, so that the camera can shoot a round target that is placed arbitrarily and has an unchanged position in multiple poses. The selected pose must make all the dots on the round target within the camera's field of view. , and ensure that the line laser light plane produced by the line laser fixed on the camera is also within the field of view on the circular target; after obtaining the circular target image through the camera, extract the circle center coordinates of the circular spot on the circular target image And identify the row and column values of the corner points to complete the matching of the circular target image and the world coordinates, and then obtain the camera's internal parameter matrix according to Zhang Zhengyou's calibration algorithm $A=\left[\begin{array}{ccc}\mathrm{\&alpha;}& \mathrm{\&gamma;}& u_0\\ 0& \mathrm{\&beta;}& v_0\\ 0& 0& 1\end{array}\right],$ External parameter matrix RT;

Among them, α=f/dx, β=f/dy, f is the focal length of the camera, dx, dy are the length and width _of a single CCD photosensitive element in the camera; v ₀ is the pixel coordinates of the intersection of the lens optical axis of the camera and the CCD photosensitive element;

1.1. Use the camera to collect the image of the circular target online in real time, and use the adaptive threshold of the Otsu method to binarize the image, so that the circular spots on the circular target image are highlighted;

1.2. Use the closed operation operator to perform closed operation on the circular target image to remove noise interference; mark all the target dots in the above-mentioned binarized circular target image, and the area of the target dot is the pixel contained in the target dot Number, count the number of pixels of the target dot in the above circular target image, the average number of pixels contained in the marked target dot is NP_average, remove the number of pixels contained in the marked target less than 0.5*NP_average and the number of pixels contained in the marked target greater than 1.5* target for NP_average;

1.3. Calculate the circumference C and area S of the marked target dot, and use the circularity expression e=C ² /(2*π*S) to distinguish the target dot from interference;

1.4. Use the center of gravity method to obtain the center coordinates of each target point

${\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; =</span>\frac{\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\mathrm{<span\; class="notranslate">\; Hit</span>}<span\; class="notranslate">\; [</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; [</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; x</span>}}{\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\mathrm{<span\; class="notranslate">\; Hit</span>}<span\; class="notranslate">\; [</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; [</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ]</span>}<span\; class="notranslate">\; ,</span>$ ${\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; =</span>\frac{\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\mathrm{<span\; class="notranslate">\; hit</span>}<span\; class="notranslate">\; [</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; [</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}}{\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\mathrm{<span\; class="notranslate">\; Hit</span>}<span\; class="notranslate">\; [</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; [</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ]</span>}<span\; class="notranslate">\; ;</span>$

Among them, m, n are the row and column values of the circle target image, Hit[i][j] indicates whether the (i, j) position point is on the target circle point, and the (i, j) position point is on the target circle point Hit [i][j]=1, otherwise Hit[i][j]=0;

1.5. There are several dots arranged in rows and columns on the circular target. The row distance between two adjacent rows of dots is 15mm, and the distance between the centers of two adjacent dots in the same row is 10mm. There is a reference circle at 5mm in the row direction of the first dot, and the pixel coordinates of the target dot are obtained according to step (1.4), and the two target dots with the smallest distance between the centers of the effective dots on the circle target image are found, and then follow the aforementioned sorting method Sort the origin on the circular target image;

1.6. According to the sorting of the dots in the above circular target image and the given circular target information, complete the matching between the pixel coordinates of the circle center point in the circular target image and the world coordinates.

The second step is to extract the light strips and refine the light strips according to the target picture with the line laser light strips obtained above, and obtain the light strip line equation through Hough transform. Combined with the external parameter matrix obtained in the first step, the plane equation of the line laser light plane in the camera coordinate system can be obtained;

2.1. The line laser is fixed on the camera. The camera only needs to ensure that all the target dots and light strips are within the field of view of the camera to collect pictures. Use the preset threshold to binarize the collected line laser light strip images. , perform image closing operation on the binarized image to remove edge singular points;

2.2. Carry out 8-neighborhood marking on the center of the line laser light bar area. The center point of the line laser light bar area is p1, and the 8 points in the neighborhood of the center point p1 of the line laser light bar area are respectively p2 , p3,..., p9, where the center point of point p2 is above p1, and the center point p1 of the line laser light strip area is marked with 8-neighborhood points while satisfying the following conditions:

(i), 2≤N(p1)≤6;

(ii), S(p1)=1;

(Ⅲ), p2*p4*p6=0;

(ⅳ), p4*p6*p8=0;

Among them, N(p1) is the number of non-zero neighbors of the center point p1; S(p1) is the number of times the values of these points change from 0→1 when the order is p2, p3,...,p9; when for all After checking all the boundary points of , remove all the marked points, iterate repeatedly until no point meets the marking condition, and complete the thinning of the light bar;

2.3. Use the Hough transform to extract the points obtained after refinement to obtain the line equation a _k x+b _k y+c _k =0 of the line laser light strip on the circular target plane, where a _k , b _k , and c _k represent parameters of the line equation.

In the third step, according to the posture of the robot arm corresponding to each pose in the first step, the transformation matrix between the end coordinate system of the robot arm and the base coordinate system of the robot arm is calculated by using the quaternion method, that is, the hand-eye transformation matrix H is obtained;

According to the six-axis attitude of the manipulator corresponding to each pose in the first step, the transformation matrix of the end coordinate system of the manipulator and the base coordinate system of the manipulator is obtained, and the equation ΦH=HΘ is solved by the quaternion method. Here Φ, H and Θ are 4x4 matrices, where Φ and Θ represent the transformation matrix from the first position to the second position of the end coordinate system of the manipulator and the camera coordinate system respectively, and H represents the coordinate system of the end of the manipulator to the camera coordinate system The transformation matrix of , that is, the hand-eye relationship matrix;

3.1. The plane equation of the circular target in the world coordinate system can be expressed as In the formula: π ₁ =[0,0,1,0] ^T , using the external parameter matrix RT obtained in the first step, the plane equation of the target plane in the camera coordinate system is calculated as one of them and are the plane normal coordinate vectors of the target plane in the world coordinate system and the camera coordinate system respectively;

3.2. In the camera coordinate system, set represent the normal coordinate vector of the laser plane, the normal coordinate vector of the i-th circular target, the normal coordinate vector of the j-th circular target, the coordinate vector of the laser intersection image in the i-th circular target, and the j-th plane The coordinate vector of the laser intersection image in the target; note that the equation λ _i of the laser intersection image on the i-th plane target image and can be calculated directly:

${\stackrel{<span\; class="notranslate">\; \&Right\; Arrow;</span>}{\mathrm{<span\; class="notranslate">\; \&lambda;</span>}}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; A</span>}}^{\mathrm{<span\; class="notranslate">\; T</span>}}{\mathrm{<span\; class="notranslate">\; \&lambda;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ;</span>$

3.3. According to the dual relationship between the two laser intersection lines and their associated planes in the projection space, $\stackrel{\&Right\; Arrow;}{w}=\frac{1}{2}(\stackrel{\&Right\; Arrow;}{w_1}+\stackrel{\&Right\; Arrow;}{w_2});$ $\left\{\begin{array}{c}\stackrel{\&Right\; Arrow;}{w_1}=\stackrel{\&Right\; Arrow;}{w_i}+{\mathrm{\&alpha;}}_i\stackrel{\&Right\; Arrow;}{{\mathrm{\&lambda;}}_j}\\ \stackrel{\&Right\; Arrow;}{w_2}=\stackrel{\&Right\; Arrow;}{w_j}+{\mathrm{\&alpha;}}_j\stackrel{\&Right\; Arrow;}{{\mathrm{\&lambda;}}_j}\end{array}\right.,$

in $\left[\begin{array}{c}{\mathrm{\&alpha;}}_i\\ {\mathrm{\&alpha;}}_j\end{array}\right]={\mathrm{D.}}^{-1}Q,$ D is a 2×2 matrix, Q is a two-dimensional vector, defined as follows: $\mathrm{D.}=\left[\begin{array}{cc}(\stackrel{\&Right\; Arrow;}{{\mathrm{\&lambda;}}_i},\stackrel{\&Right\; Arrow;}{w_i})& -(\stackrel{\&Right\; Arrow;}{{\mathrm{\&lambda;}}_j},\stackrel{\&Right\; Arrow;}{w_i})\\ (\stackrel{\&Right\; Arrow;}{{\mathrm{\&lambda;}}_i},\stackrel{\&Right\; Arrow;}{w_j})& -(\stackrel{\&Right\; Arrow;}{{\mathrm{\&lambda;}}_j},\stackrel{\&Right\; Arrow;}{w_j})\end{array}\right],$ $Q=\left[\begin{array}{c}(\stackrel{\&Right\; Arrow;}{w_i},\stackrel{\&Right\; Arrow;}{w_j}-\stackrel{\&Right\; Arrow;}{w_i})\\ (\stackrel{\&Right\; Arrow;}{w_j},\stackrel{\&Right\; Arrow;}{w_i}-\stackrel{\&Right\; Arrow;}{w_j})\end{array}\right];$ is the desired laser plane equation.

The fourth step is to control the welding workpiece to touch the target point under the fixed end pose, calculate the coordinate value of the end point of the workpiece under the coordinates of the manipulator, and calculate the position of the workpiece under the pose of the manipulator based on the pose of the manipulator. offset value.

4.1. Take any two poses in the first step, in Indicates the transformation matrix of the coordinate system at the end of the manipulator in two poses, so that Represents the transformation matrix of the camera coordinate system in two poses, so that Then it can be abbreviated as ΦH=HΘ;

4.2, ΦH=HΘ; expanded as

R _Φ R _H =R _H R _Θ ①

R _Φ t _H +t _Φ =R _H t _Θ +t _H ②

Wherein: R _Φ , R _Θ , R _H represents Φ, Θ, the corresponding rotation part in H, t _Φ , t _Θ , t _H represents Φ, Θ, for the translation part in H;

4.3 Order $q_{\mathrm{\&Phi;}}={[a_0,{\stackrel{\&Right\; Arrow;}{a}}^T]}^T,$ $q_h={[x_0,{\stackrel{\&Right\; Arrow;}{x}}^T]}^T,$ $q_{\mathrm{\&Theta;}}={[b_0,{\stackrel{\&Right\; Arrow;}{b}}^T]}^T$ is the quaternion corresponding to R _Φ , R _H , R _Θ , q _Φ is calculated from the homogeneous matrix Φ, and q _Θ is calculated from the homogeneous matrix Θ, so that pass Find q _H , and then convert q _H into the rotation part of the hand-eye matrix, where Ω(·) represents the antisymmetric matrix function, express The anti-symmetric matrix of ; x ₀ , x is the unknown quantity to be calculated in q _H ;

4.4. Bring the rotation matrix data into ② to find t _H and obtain the hand-eye transformation matrix H

Calculating the welding workpiece displacement value in the fourth step includes the following steps:

5.1. Control the welding workpiece to precisely touch the preset point on the circular target under the fixed end pose of the manipulator, read the current pose of the manipulator, and obtain the first coordinate value of the end coordinate of the manipulator;

5.2. Define the coordinates of the preset point in the world coordinate system. According to the external parameter matrix RT obtained in the first step, obtain the coordinates of the preset point in the camera coordinate system. According to the hand-eye transformation matrix H and the image of the circular target The pose calculation of the manipulator obtains the second coordinate value of the preset point in the base coordinate system of the manipulator, and compares the second coordinate value with the first coordinate value to obtain the displacement value of the welding workpiece under the current pose.

In the embodiment of the present invention, through the above steps, the calibration of the internal parameters of the camera, the calibration of the line laser light plane equation, the calibration of the hand-eye transformation matrix and the calibration of the workpiece offset are respectively realized, that is, the calibration process of the welding robot system is achieved.

By analyzing the imaging principle of the camera, the principle of structured light measurement and the working principle of the hand-eye system, the present invention designs a simple and flexible calibration method based on the three-dimensional tracking of the mechanical light-guided manipulator, which includes the calibration of the internal parameters of the camera and the equation of the line laser light plane Calibration, hand-eye transformation matrix calibration and workpiece offset calibration. This calibration method overcomes the shortcomings of the traditional laser light plane equation and hand-eye matrix calibration conditions and cumbersome calibration steps. The algorithm only needs to control the random movement of the manipulator in more than 3 poses to shoot the fixed target to complete the overall calibration task, and the structured light-guided manipulator tracking system using this method has high tracking accuracy. The algorithm makes it possible to adjust the structure of the line laser and the camera, and to calibrate the system structure parameters on site. The flexibility of the structured light guidance system is greatly increased, which is of great significance to the actual visual measurement and tracking, and has good practicability.

Claims (7)

1. the welding robot system scaling method based on line structured light vision sensor guiding, is characterized in that, described welding robot system scaling method comprises the steps:

The first step, control mechanical arm conversion pose, make camera at a plurality of poses, take the round target of any placement and invariant position, round dot on all round target that the pose of selecting must make, within the scope of viewing field of camera, and guarantees to be fixed on striation part that line laser optical plane that on camera, laser line generator produces produces on circle target also in field range; By camera, obtain after round target image, extract the central coordinate of circle of justifying circular spot on target image the ranks value of identifying angle point, to complete the coupling of round target image and world coordinates, then according to Zhang Zhengyou calibration algorithm, obtain the inner parameter matrix of camera $A=\left[\begin{array}{ccc}\mathrm{\&alpha;}& \mathrm{\&gamma;}& u_0\\ 0& \mathrm{\&beta;}& v_0\\ 0& 0& 1\end{array}\right]$ , outer parameter matrix RT;

Wherein, α=f/dx, β=f/dy, f is camera focus, dx, dy is single CCD photo-sensitive cell length in camera, width; γ is the physical quantity that in reflection camera, CCD photo-sensitive cell is arranged inclined degree, u ₀, v ₀for the camera lens optical axis of camera and the intersection point pixel coordinate of CCD photo-sensitive cell;

Second step, according to the round target image with line laser striation obtaining in the first step, extract line laser striation, the line laser striation that refinement is extracted, converts the line equation of obtaining line laser striation by Hough; Utilize the outer parameter matrix RT that the first step obtains to obtain the plane equation of line laser striation plane under camera coordinates system;

The 3rd step, according to the mechanical arm attitude that in the first step, each pose is corresponding, utilize Quaternion Method to calculate the transformation matrix of mechanical arm tail end coordinate system and mechanical arm basis coordinates system, obtain trick transformation matrix H;

The 4th step, welding work pieces is positioned over to mechanical arm tail end, control welding work pieces a bit accurately taps justifying on target under solid mechanical arm end pose, calculate the coordinate figure of welding work pieces distal point under mechanical arm coordinate, and calculate the deviant of workpiece under described pose in conjunction with mechanical arm pose.

2. the welding robot system scaling method based on line structured light vision sensor guiding according to claim 1, is characterized in that, the described first step comprises the steps:

(1.1), utilize the image of camera online real time collecting circle target, adopt large Tianjin method adaptive threshold to image binaryzation, the circular spot on circle target image is highlighted;

(1.2), utilize closed operation operator to carry out closed operation to circle target image, to remove noise jamming; All target round dots in round target image after above-mentioned binaryzation are carried out to mark, the area of target round dot is the number of pixels that target round dot comprises, add up the number of pixels of the target round dot in above-mentioned round target image, it is NP_average that target-marking round dot comprises number of pixels average, removes in target-marking, to comprise number of pixels and be less than and in 0.5*NP_average and target-marking, comprise the target that number of pixels is greater than 1.5*NP_average;

(1.3), calculate girth C and the area S of target round dot after mark, utilize circularity expression formula e=C ²/ (2* π * S), distinguishes target round dot and interference;

(1.4), adopt gravity model appoach to ask for the central coordinate of circle (x of each target round dot ₀, y ₀),

$x_0=\frac{\underset{i=0}{\overset{m}{\mathrm{\&Sigma;}}}\underset{j=0}{\overset{n}{\mathrm{\&Sigma;}}}\mathrm{Hit}\left[i\right]\left[j\right]*x}{\underset{i=0}{\overset{m}{\mathrm{\&Sigma;}}}\underset{j=0}{\overset{n}{\mathrm{\&Sigma;}}}\mathrm{Hit}\left[i\right]\left[j\right]},y_0=\frac{\underset{i=0}{\overset{m}{\mathrm{\&Sigma;}}}\underset{j=0}{\overset{n}{\mathrm{\&Sigma;}}}\mathrm{Hit}\left[i\right]\left[j\right]*y}{\underset{i=0}{\overset{m}{\mathrm{\&Sigma;}}}\underset{j=0}{\overset{n}{\mathrm{\&Sigma;}}}\mathrm{Hit}\left[i\right]\left[j\right]};$

Wherein, m, n is the row, column value of circle target image, Hit[i] [j] represent that (i, j) location point is whether on target round dot, (i, j) location point is Hit[i on target round dot] [j]=1, otherwise Hit[i] [j]=0;

(1.5), justify on target and there is the round dot that some row, column are arranged, line-spacing between adjacent two row round dots is 15mm, distance with 2 round dot centers of circle adjacent in a line is 10mm, on first line direction of circle target the first row, there is circle of reference at 5mm place, according to step (1.4), obtain target round dot pixel coordinate, utilize circle of reference to sort to the centre point on circle target image;

(1.6), according to above-mentioned round target image orbicular spot sequence, with given round target information, complete mating of centre point pixel coordinate and world coordinates in round target image.

3. the welding robot system scaling method based on line structured light vision sensor guiding according to claim 1, is characterized in that, described second step comprises the steps:

(2.1), laser line generator is fixed on camera, collected by camera picture is only to need to guarantee that all target round dots and striation part are within the scope of viewing field of camera, utilize predetermined threshold value to carry out binaryzation to the line laser optical strip image collecting, binary image is carried out to closing operation to remove edge singular point;

(2.2), line laser striation regional center is carried out to 8-neighborhood mark, the central point in note line laser striation region is p1,8 points of line laser striation regional center point p1 neighborhood are respectively p2 around central point clockwise, p3, p9, wherein p2 dot center's point, above p1, meets the boundary point of following condition when line laser striation regional center point p1 is carried out to 8-neighborhood mark:

（ⅰ）、2≤N（p1）≤6；

（ⅱ）、S（p1）=1；

（ⅲ）、p2*p4*p6=0；

（ⅳ）、p4*p6*p8=0；

Wherein, N(p1) be the number of the non-zero adjoint point of central point p1; S(p1) be with p2, p3 ..., when p9 is order, the value of these points is from 0 → 1 change frequency; After the boundary point to all checks, all gauge points are removed, iterate until not point meet flag condition, complete striation refinement;

(2.3), the point obtaining after refinement is utilized Hough conversion to extract to obtain at the reach the standard grade line equation a of laser striation of circle target plane _kx+b _ky+c _k=0, wherein, a _k, b _k, c _kthe parameter that represents respectively line equation.

4. the welding robot system scaling method based on line structured light vision sensor guiding according to claim 1, is characterized in that, described the 3rd step comprises the steps:

(3.1), justifying the plane equation of target under world coordinate system can be expressed as in formula: π ₁=[0,0,1,0] ^t, the outer parameter matrix RT that utilizes the first step to obtain, the plane equation that calculates target plane under camera coordinates system is wherein with be respectively the planar process of target plane under world coordinate system and camera coordinates system to coordinate vector;

(3.2), in camera coordinates system, establish represent respectively the normal direction coordinate vector of laser plane normal direction coordinate vector, an i circle target, the normal direction coordinate vector of j circle target, i are justified the coordinate vector of laser intersection image in target, the coordinate vector of the laser intersection image in a j plane target drone; Note the equation of i the laser intersection image on plane target drone image with can directly calculate:

(3.3), according to the duality relation of two laser intersections in projector space and the plane that is associated thereof, obtain,

$\stackrel{\&RightArrow;}{w}=\frac{1}{2}(\stackrel{\&RightArrow;}{w_1}+{\stackrel{\&RightArrow;}{w}}_2);\left\{\begin{array}{c}{\stackrel{\&RightArrow;}{w}}_1={\stackrel{\&RightArrow;}{w}}_i+{\mathrm{\&alpha;}}_i{\stackrel{\&RightArrow;}{\mathrm{\&lambda;}}}_i\\ {\stackrel{\&RightArrow;}{w}}_2={\stackrel{\&RightArrow;}{w}}_j+{\mathrm{\&alpha;}}_j{\stackrel{\&RightArrow;}{\mathrm{\&lambda;}}}_j\end{array}\right.,$

Wherein $\left[\begin{array}{c}{\mathrm{\&alpha;}}_i\\ {\mathrm{\&alpha;}}_j\end{array}\right]=D^{-1}Q,$ D is 2 * 2 matrixes, and Q is a bivector, is defined as follows: $\left[\begin{array}{cc}({\stackrel{\&RightArrow;}{\mathrm{\&lambda;}}}_i,{\stackrel{\&RightArrow;}{w}}_i)& -({\stackrel{\&RightArrow;}{\mathrm{\&lambda;}}}_j,{\stackrel{\&RightArrow;}{w}}_i)\\ ({\stackrel{\&RightArrow;}{\mathrm{\&lambda;}}}_i,{\stackrel{\&RightArrow;}{w}}_j)& -({\stackrel{\&RightArrow;}{\mathrm{\&lambda;}}}_j,{\stackrel{\&RightArrow;}{w}}_j)\end{array}\right],Q=\left[\begin{array}{c}{\stackrel{\&RightArrow;}{w}}_i,{\stackrel{\&RightArrow;}{w}}_j-{\stackrel{\&RightArrow;}{w}}_i\\ {\stackrel{\&RightArrow;}{w}}_j,{\stackrel{\&RightArrow;}{w}}_i-{\stackrel{\&RightArrow;}{w}}_j\end{array}\right];$ it is required laser plane equation.

5. the welding robot system scaling method based on line structured light vision sensor guiding according to claim 1, is characterized in that, described the 4th step comprises the steps:

(4.1), get any 2 poses in the first step, wherein the transition matrix that represents mechanical arm tail end coordinate system under 2 poses, order the transition matrix that represents camera coordinates system under 2 poses, order can be abbreviated as

(4.2), expand into

$R_{\mathrm{\&Phi;}}{R}_H=R_H{R}_{\mathrm{\&Theta;}}$ ①

$R_{\mathrm{\&Phi;}}{t}_H+t_{\mathrm{\&Phi;}}=R_H{t}_{\mathrm{\&Theta;}}+t_H$ ②

Wherein: R _Φ, r _Ηrepresent Φ, the rotating part of correspondence in H, t _Φ, t _Ηrepresent Φ, in H for translating sections;

(4.3), order for R _Φ, R _Η, corresponding hypercomplex number, q _Φby homogeneous matrix Φ, calculated, by homogeneous matrix Θ, calculated order by try to achieve q _h, then by q _hbe converted to the rotating part of trick matrix, Ω () the title matrix function that makes difficulties wherein, represent antisymmetric matrix; x ₀, x is _qHin unknown quantity to be calculated;

(4.4), more 2. spin matrix data are brought into and just can be obtained t _Η, obtain trick transformation matrix H.

6. the welding robot system scaling method based on line structured light vision sensor guiding according to claim 1, is characterized in that, calculates welding work pieces shift value and comprise the steps: in described the 4th step

(5.1), control welding work pieces and under solid mechanical arm end pose, preset on circle target accurately tapped, read current mechanical arm pose, obtain the first coordinate figure of mechanical arm tail end coordinate;

(5.2), the coordinate of definition preset under world coordinate system, according to the outer parameter matrix RT obtaining in the first step, obtain the coordinate of preset under camera coordinates system, mechanical arm pose according to trick transformation matrix H and while taking circle target image calculates second coordinate figure of preset under mechanical arm basis coordinates system, and described the second coordinate figure and the first coordinate figure are compared and obtained welding work pieces shift value under current pose.

7. the welding robot system scaling method based on line structured light vision sensor guiding according to claim 2, it is characterized in that: while utilizing circle of reference to sort to the centre point on circle target image, find out the distance of center circle of effective round dot on round target image from two target round dots of minimum, wherein from all centre point averages center, the circle away from is defined as 0 point, another is defined as 1 point, then arrange 0, 1 is match point, in match point, do not looking for the point nearest with 1 to define 2 points, arrange at 2 for match point, in match point, do not looking for the point nearest with 2 to define 3 points, arrange at 3 for match point ... until institute is a little all looked for complete, complete the sequence of all centre points.