CN110728715A - Camera angle self-adaptive adjusting method of intelligent inspection robot - Google Patents

Abstract

The invention discloses an adaptive adjustment method for the camera angle of an intelligent inspection robot. The steps are as follows: establishing a monocular mobile vision measurement model, acquiring internal parameter calibration data of the camera, and determining the following different target points according to the position of the initial image target point of the robot. The image target points obtained by the robot at all times are combined with the homography matrix to find the corresponding image matching points of the target points in the initial image. Using the principle of triangulation, the same target point obtained by the robot at different times is realized in the camera coordinate system. The three-dimensional pose information below, and finally the deflection angle of the camera is obtained. The invention solves the problem that the target of the intelligent inspection robot deviates from the center of the vision of the robot camera when it is working. In the presence of the robot positioning error and the rotation error of the pan-tilt head, the camera angle can be adaptively adjusted to achieve precise positioning and accuracy of the target. Identify and complete intelligent inspection, fault diagnosis, identification and early warning of robot targets.

Description

An adaptive adjustment method for the camera angle of an intelligent inspection robot

technical field

The invention belongs to the technical field of machine vision measurement, and in particular relates to a camera angle adaptive adjustment method of an intelligent inspection robot.

Background technique

Intelligent inspection robots replace manual inspections in special environments, which not only improves the efficiency of on-site inspections, reduces on-site maintenance costs, but also reduces the limitations of manual inspections and expands the application of artificial intelligence technology in special environments. During the operation of the intelligent inspection robot, the equipment information to be detected and the surrounding environment information are obtained by carrying a pan/tilt and a camera, and image processing and pattern recognition technology are used to realize the analysis and judgment of the special environmental target state. In special environments such as substations and important computer rooms, intelligent inspection robots have been put into use, and have achieved good results in intelligent detection of relevant equipment in special environments, fault judgment and early warning of abnormal surrounding environments.

During the inspection process of the intelligent inspection robot based on monocular vision, there are certain navigation and positioning errors when the robot walks and stops. The target deviates from the center of the imaging field of view of the camera. In severe cases, the target completely deviates from the imaging field of view of the camera, and the target to be detected cannot be imaged, which brings certain difficulties to the intelligent detection of the subsequent target and the early warning and judgment of equipment failure.

SUMMARY OF THE INVENTION

Purpose of the invention: Aiming at the defect that the pan/tilt carried by the intelligent inspection robot in the prior art has a certain rotation error, the present invention discloses an adaptive adjustment method of the camera angle of the intelligent inspection robot, which solves the problem of the intelligent inspection robot in the The problem that the target deviates from the center of the robot's camera field of view during work helps the robot to image the same target point at the fixed position of the image obtained by the camera, so as to achieve accurate target positioning and accurate identification.

Technical solution: The present invention discloses a camera angle adaptive adjustment method of an intelligent inspection robot, which is characterized by comprising the following steps:

Step A. According to the pinhole imaging model of the camera, the camera calibration method of the plane grid point is utilized to establish a monocular mobile measurement system model;

Step B, according to the mapping relationship between the three-dimensional space coordinate point and the plane two-dimensional coordinate point, obtain the internal parameter matrix of the camera; wherein the internal parameter matrix refers to the parameters related to the characteristics of the camera itself, including the focal length and pixel size of the camera;

Step C. Obtain two images taken by the robot at two different times at the same position through the camera of the intelligent inspection robot, and obtain the expressions of the fundamental matrix and the essential matrix; wherein the fundamental matrix represents the epipolar geometry of the two views. The intrinsic projective relationship, the essential matrix represents the fundamental matrix in the normalized image coordinates;

Step D, according to the projection equations of the two images obtained in step C, extract the position information of the feature points of the two images, including the two-dimensional coordinate values of the feature points;

Step E, combine the image feature points extracted in step 4, and use the 8-point algorithm to solve the fundamental matrix, and then combine the internal parameter matrix obtained in step B to solve the essential matrix, and decompose the essential matrix to obtain the camera extrinsic parameter matrix; The external parameter matrix realizes the method of converting the point from the world coordinate system to the camera coordinate system; the camera external parameter matrix includes the rotation matrix and the translation matrix;

Step F. According to at least 4 pairs of matching feature points between the two images, combined with the SVD algorithm, solve the homography matrix between the two images; wherein the homography matrix refers to the projection of one image plane to another image plane map;

Step G, using the image positions corresponding to the feature points of the two images obtained at the two moments before and after the robot, according to the homography matrix obtained in step F, combined with the binocular stereo vision measurement principle and the in-camera parameters obtained in step E and the external structural parameters to find the angle that the camera needs to rotate, in which the angle of the camera rotation is the rotation angle of the robot gimbal; this angle is the angle in the camera coordinate system, which can be determined according to the x and y of the camera coordinate system. The axis is decomposed, that is, the left and right and up and down rotation angles of the camera. Generally, the camera angle can be adjusted after being decomposed and transmitted to the camera's PTZ.

Preferably, the step D further comprises:

D1: Extract and match SIFT feature points in the overlapping area of the image;

D2: Use the RANSAC algorithm to eliminate the mismatched points in the image pair to achieve accurate registration of the SIFT feature points between the two images.

Preferably, the step F further comprises:

F1: The relational expression between the image coordinate system and the world coordinate system of the matching point between the images of the two moments according to the perspective projection model of the camera;

F2: According to at least 4 pairs of matching feature points, use SVD (singular value decomposition) to solve the homography matrix between two images.

Preferably, the expressions of the fundamental matrix and the essential matrix in the step C are:

E=SR

Among them, F is the fundamental matrix, E is the essential matrix, R is the rotation matrix, S is the antisymmetric matrix, and A _r and A _l are the internal parameter matrices of the camera at two different times.

Preferably, the expression of the angle of rotation of the camera in the step G is:

Among them, P is the spatial feature point corresponding to one of the matching feature points in a set of matching feature points of the two images in step D, and P' is the virtual space three-dimensional point calculated according to step G. The angle in the camera coordinate system is decomposed according to the x and y axes of the camera coordinate system, representing the left and right and up and down rotation angles of the camera respectively. .

Beneficial effects: The invention discloses a camera angle adaptive adjustment method of an intelligent inspection robot, which combines the monocular vision technology and utilizes two images taken by the robot at the same position at different times to realize the extraction and matching of the features of the two images. , using the mobile monocular vision positioning technology to complete the three-dimensional reconstruction of the target point. Through the obtained image coordinates of the two sets of corresponding points and the three-dimensional coordinates in the world coordinate system, the self-adaptive adjustment of the camera angle carried by the robot in the presence of various positioning errors is realized. The invention solves the problem that the target of the intelligent inspection robot deviates from the center position of the robot's camera field of view during operation. In the presence of robot positioning error and pan-tilt rotation error, the camera angle can be adaptively adjusted to achieve precise positioning and accuracy of the target. It has completed the intelligent inspection, fault diagnosis, identification and early warning of the robot target.

Description of drawings

Fig. 1 is the flow chart of the present invention;

Fig. 2 is the monocular mobile intelligent inspection robot structure diagram of the present invention;

Among them, 1 is a camera mounted on the intelligent inspection robot, 2 is a pan/tilt with four degrees of freedom that can move left and right and up and down, 3 is a lidar, 4 is a trench sensor, 5 is a four-wheel drive chassis, and 6 is an anti-collision switch;

Fig. 3 is the extraction and matching diagram of the feature of the present invention;

FIG. 4 is a schematic diagram of a binocular vision measurement system composed of a monocular mobile robot according to the present invention and an angle deflection of a camera.

Detailed ways

The invention discloses a camera angle adaptive adjustment method of an intelligent inspection robot, which is characterized by comprising the following steps:

Step A. According to the pinhole imaging model of the camera, the camera calibration method of plane grid points is used to establish a monocular mobile measurement system model; the mobile monocular vision measurement system is moved by a camera to virtually form multiple images machine to form a multi-eye vision measurement system. The present invention analyzes the principle of a binocular stereo vision measurement system composed of a mobile monocular vision by taking two images taken by a robot at a certain position at two different times to form a two-view vision measurement as an example.

图像平面的二维点齐次坐标为

二者之间的射影关系为Step B, obtain the internal parameter matrix of the camera according to the mapping relationship between the three-dimensional space coordinate point and the plane two-dimensional coordinate point; it is assumed that the homogeneous coordinates of the three-dimensional point of the target plane are marked as

The two-dimensional point homogeneous coordinates of the image plane are

The projective relationship between the two is

A称为摄像机的内部参数矩阵。α_x、α_y是u轴和v轴的尺度因子，(u₀，v₀)为主点坐标，r是u轴和v轴的不垂直因子。由张氏平面标定法，可以求出像机的内参数矩阵A。Among them, s is an arbitrary non-zero scale factor, [R t] is a matrix with 3 rows and 4 columns, called the camera extrinsic parameter matrix, R is called the rotation matrix, t=(t ₁ , t ₂ , t ₃ ) ^T , called the translation matrix,

A is called the internal parameter matrix of the camera. α _x , α _y are the scale factors of the u-axis and v-axis, (u ₀ , v ₀ ) are the coordinates of the principal point, and r is the non-perpendicular factor of the u-axis and v-axis. By Zhang's plane calibration method, the internal parameter matrix A of the camera can be obtained.

Step C. Through the camera of the intelligent inspection robot, two images taken by the robot at two different times at a certain position are obtained, and the expressions of the fundamental matrix and the essential matrix are obtained; it is assumed that the world at a point P in space is three-dimensionally homogeneous The coordinates are X _w , and the homogeneous coordinates of the two-dimensional images in the two images captured at two moments are p ₁ and p ₂ , then the projection equations of the cameras at the two moments can be obtained from equation (1) as follows:

Among them, s ₁ and s ₂ are the non-zero scale factors of the two cameras, A ₁ and A ₂ are the camera internal parameters of the camera at time 1 and time 2, respectively, because the camera only moves rigidly, and the internal structural parameters do not change , so A ₁ =A ₂ .

Preferably, in the step C, combined with the epipolar geometric constraint relationship, the expressions of the fundamental matrix F and the essential matrix E can be obtained from the formula (2), which are respectively as follows:

E=SR (4)

where S is an antisymmetric matrix,

It can be seen from formula (3) that the fundamental matrix F is only related to the internal parameters of the two cameras and the external structural parameters of the system, while the mounted camera only performs rigid body motion due to the rotation of the PTZ, and the internal parameters of the camera remain unchanged. Therefore, the essential matrix E of formula (4) can be obtained. It can be seen that E is only related to the external parameters of the vision system. E can be decomposed to obtain the external structural parameters R and R between the two view models of the mobile monocular vision measurement system. t.

Step D, according to the projection equation of the two images obtained in step C, extract two image feature points;

Preferably, the step D further comprises:

D1: Extract and match SIFT feature points in the overlapping area of the image;

D2: Use the RANSAC algorithm to eliminate the mismatched points in the image pair to achieve accurate registration of the SIFT feature points between the two images.

The present invention utilizes the characteristics of SIFT feature points (scale-invariant feature transformation), that is, the SIFT features remain unchanged for image rotation, scale scaling, and brightness changes, and overlap with those captured by a robot camera at a certain position at two different times. The SIFT feature extraction is performed on the image pairs in the region, and the RANSAC algorithm is used to eliminate the mismatched points in the image pairs to achieve accurate registration of the SIFT feature points between the two images. As shown in Figure 3, at a certain position, due to the navigation positioning error and the pan-tilt angle rotation error of the inspection robot, the mounted camera images the same target at different times, and two images with overlapping areas will be obtained. The SIFT feature points are extracted and matched in the overlapping area of the image. Combining formulas (3) and (4), using the 8-point algorithm analyzed above, the external structural parameters R and t between the two-view camera coordinate systems can be obtained. The external parameters play an important role in the reconstruction of three-dimensional points in space.

Step E, combine the image feature points extracted in step 4, use the 8-point algorithm to solve the fundamental matrix, and then combine with the internal parameter matrix obtained in step B to solve the essential matrix, and decompose the essential matrix to obtain the difference between the two images. The external structure parameter matrix of algorithm, the fundamental matrix F between the two view images can be obtained, and the essential matrix E can be obtained by combining with the internal parameters of the camera. By decomposing the essential matrix E, the external structural parameters R and t between the two views can be finally obtained.

Step F, according to at least 4 pairs of matching points between two images, utilize SVD (singular value decomposition method), can solve the homography matrix between two images;

Preferably, the step F further comprises:

F1: The relational expression between the image coordinate system and the world coordinate system of the matching point between the images of the two moments according to the perspective projection model of the camera;

F2: Solve the homography matrix according to the SVD decomposition method.

Combining the information of the matching feature points of the two images, using the perspective projection model of the camera, the relationship between the image coordinate system and the world coordinate system of the matching point between the images at the two moments can be obtained respectively,

It can be obtained from formulas (5) and (6),

H为3×3的矩阵，H矩阵反映了两幅图像特征点之间的映射关系，如图3所示，定义H为两幅平面之间的单应性矩阵，假设代入公式(7)可以得到make

H is a 3×3 matrix. The H matrix reflects the mapping relationship between the feature points of the two images. As shown in Figure 3, H is defined as the homography matrix between the two planes. Suppose Substitute into formula (7) to get

From formula (8), we can get

where (u _a , v _a ) and (u _b , v _b ) are matched point pairs on the two images.

It can be seen from formula (9) that two equations can be obtained for each pair of feature points. The H matrix is a singular matrix of rank 8, and at least 4 pairs of matching points can solve the homography matrix H of the two planes. A common method for solving the homography matrix H is the SVD decomposition method.

Step G, obtain the homography matrix obtained according to step F, obtain the virtual space three-dimensional point corresponding to the feature points of the two images, combine the binocular stereo vision measurement principle and the external structure parameter matrix of the system, obtain the angle of the camera rotation; This angle is the angle in the camera coordinate system, which can be decomposed according to the x and y axes of the camera coordinate system, that is, the left and right and up and down rotation angles of the camera. , the camera angle can be adjusted.

Preferably, the expression of the angle of rotation of the camera in the step G is:

Among them, P is the spatial feature point corresponding to one of the matching feature points in a set of matching feature points of the two images in step D, and P' is the virtual space three-dimensional point calculated according to step G. The angle in the camera coordinate system is decomposed according to the x and y axes of the camera coordinate system, representing the left and right and up and down rotation angles of the camera respectively. .

After solving the homography matrix H of the two images and the external structure parameters R and t of the two-view camera coordinate system, the first image is used as the reference image, and a set of matching features in the matching feature points in the two images is selected. For the point pair p and q, the corresponding spatial feature point is P, and its image coordinates in the corresponding two images are respectively p=(x _a , y _a ) and q=(x _b , y _b ), as shown in Figure 4 Show. Combined with the principle of binocular stereo vision measurement, for the accurately calibrated dual-view image machine measurement system, the image coordinates of the two points and the external structural parameters R and t are known, and the three-dimensional world coordinate system of the space point P can be accurately obtained. When the world coordinate system is established in the camera 2 coordinate system, the rotation angle of the camera is calculated. It is assumed that the calculated three-dimensional world coordinates of the space point P at this time are P=(X ₁ , Y ₁ , Z ₁ ).

Due to the navigation and positioning errors of the robot, there is a deviation in the image coordinates of the matching feature points of the two images taken at different times. side. Assuming that the robot's navigation and positioning are accurate and the above errors do not exist, and the image coordinates in image 1 are set as p=(x _a , y _a ), then the position and angle of the camera at time 1 should be the same as the position and angle of the camera at time 2, Therefore, in the second image, the image coordinates of the matching point corresponding to point p in image 1 should be q'=(x _a , _ya ). In fact, due to the existence of errors, the camera position and angle at time 2 deviate from the camera position and angle at time 1.

The present invention does not change the position of the inspection robot under the condition of the existence of navigation error, and adjusts the shooting position of the camera by rotating the angle of the pan/tilt, so that the target does not deviate from the center position of the image. The rotation angle of the gimbal is calculated by the angle error calculated by the camera.

It can be seen from the above analysis that if the target point wants to achieve the position on the image captured at time 2 and the image captured at time 1 with the same image coordinates, the camera at position 2 should have the same image coordinates as the robot does not move. The moving angle is ∠qo ₂ q′.

In fact, by obtaining the homography matrix H of the two images above, the matching point p' in image 1 corresponding to the point q'=(x _a , _ya ) in image 2 can be obtained, such as As shown in Figure 4, that is

p'=H ^-T q'=H ^-T (x _a , _ya , 1) (10)

Among them, p' and q' are virtual image matching point pairs, and the space point corresponding to them is P', which is a three-dimensional point in virtual space. In the same way, the image coordinates of the corresponding point pair p' and q', and the external structural parameters R and t of the system are known, combined with the principle of binocular stereo vision measurement, the three-dimensional world coordinates of the virtual space point P' can be obtained. Assuming that the calculated P'=(X' ₁ , Y' ₁ , Z' ₁ ). From the analysis in Figure 4, because the three-dimensional coordinates of the space point are established in the coordinate system of the camera 2, ∠qo ₂ q′=∠Po ₂ P′, the moving angle of the camera 2 should be calculated by the following formula, namely,

By decomposing (11), the angle that the camera should rotate, that is, the angle that the robot head should rotate.

The above is only the preferred embodiment of the present invention, it should be pointed out that: for those skilled in the art, without departing from the principle of the present invention, several improvements and modifications can also be made, and these improvements and modifications are also It should be regarded as the protection scope of the present invention.

Claims (7)

1. A camera angle self-adaptive adjusting method of an intelligent inspection robot is characterized by comprising the following steps:

a, according to a pinhole imaging model of a camera, establishing a monocular mobile measurement system model by utilizing a camera calibration method of a plane grid point, namely a mapping relation between a three-dimensional space coordinate point and a plane two-dimensional coordinate point;

b, solving an internal parameter matrix of the camera according to the mapping relation between the three-dimensional space coordinate point and the plane two-dimensional coordinate point; the internal parameter matrix refers to parameters related to the self characteristics of the camera and comprises the focal length and the pixel size of the camera;

step C, acquiring two images shot by the inspection robot at two different moments at the same position through a camera of the intelligent inspection robot, and solving the expression of the basic matrix and the essential matrix; wherein the basic matrix represents the intrinsic projective relation of the epipolar geometry of the two views, and the essential matrix is the basic matrix under the normalized image coordinate;

d, extracting the position information of the characteristic points of the two images according to the projection equation of the two images obtained in the step C, wherein the position information comprises two-dimensional coordinate values of the characteristic points;

step E, combining the image characteristic points extracted in the step four, solving a basic matrix F by using an 8-point algorithm, solving an essential matrix E by combining the internal parameter matrix obtained in the step B, and decomposing the essential matrix by using an SVD algorithm to obtain external parameter matrixes R and t of the camera; the external reference matrix realizes a method for converting points from a world coordinate system to a camera coordinate system; the camera external parameter matrix comprises a rotation matrix R and a translation matrix t;

step F, solving a homography matrix between the two images by utilizing at least 4 pairs of matching points and combining an SVD algorithm according to the matched feature points; wherein the homography matrix represents a projection mapping relationship from one image to another image;

and G, solving the virtual space three-dimensional points corresponding to the two image characteristic points according to the homography matrix solved in the step F, and solving the rotation angle of the camera by combining the binocular stereo vision measurement principle and the camera external structure parameter matrix solved in the step E, wherein the rotation angle of the camera is the rotation angle of the robot holder.

2. A camera angle adaptive adjustment method for an intelligent inspection robot according to claim 1, wherein: the step D also comprises the following steps:

d1: extracting and matching SIFT feature points in an overlapping area of the images;

d2: and eliminating mismatching points in the image pair by using a RANSAC algorithm, and realizing accurate registration of SIFT feature points between the two images.

3. A camera angle adaptive adjustment method for an intelligent inspection robot according to claim 1, wherein: the step F further comprises the following steps:

f1: according to the relation between the image coordinate system and the world coordinate system of the matching point between the images of the perspective projection model of the camera at two moments;

f2: and solving the homography matrix according to an SVD decomposition method.

4. A camera angle adaptive adjustment method for an intelligent inspection robot according to claim 1, wherein: the expressions of the basic matrix and the essential matrix in the step C are as follows:

E＝SR

wherein F is a base matrix, E is an essential matrix, R is a rotation matrix, S is an antisymmetric matrix, A_r and A_lThe parameter matrixes in the two different simultaneous engraving machines.

5. A camera angle adaptive adjustment method for an intelligent inspection robot according to claim 1, wherein: and G, the rotating angle of the camera, namely the rotating angle expression of the robot holder is as follows:

and D, calculating a virtual space three-dimensional point according to the step D, and resolving the angle of the camera coordinate system according to the x axis and the y axis of the camera coordinate system to respectively represent the left and right rotation angles and the up and down rotation angles of the camera, and transmitting the resolved rotation angles to a tripod head of the camera to realize the adjustment of the camera angle.

6. A camera angle adaptive adjustment method for an intelligent inspection robot according to claim 1, wherein: the camera calibration method of the plane grid points in the step A is to establish a mapping relation between object points in a scene and image points on an image, namely, predefining a camera perspective model, and solving various parameters in the camera perspective model by referring to the model and knowing the corresponding relation between world coordinates of characteristic points and image coordinates; the mapping relation in the calibration method is

wherein ,

being homogeneous coordinates of the three-dimensional points of the target plane,

is the homogeneous coordinate of a three-dimensional point of the image plane, s is any non-zero scale factor, [ R t ]]The method is characterized in that the method is an external parameter matrix of a camera and a 3-row 4-column matrix, specifically, R is a rotation matrix and describes the direction of a coordinate axis of a world coordinate system relative to the coordinate axis of the camera; t is a translation matrix describing the position of the spatial origin in the camera coordinate system.

7. A camera angle adaptive adjustment method for an intelligent inspection robot according to claim 1, wherein: and B, the mobile monocular vision measuring system in the step A is a binocular stereo vision system which is virtualized by moving a camera, and offset angles of the cameras at different positions at two different moments to the same target point are calculated by combining a homography matrix between two images.

Images