CN114862969A - A method and device for self-adaptive adjustment of the angle of an airborne pan-tilt camera of an intelligent inspection robot - Google Patents

Abstract

The invention discloses a self-adaptive adjusting method and a device for camera angle of an onboard holder of an intelligent inspection robot, wherein two target images at two different moments near a certain position are acquired through a camera on the intelligent inspection robot; according to the feature matching points extracted from the two images, the poses and angles of target points of the pan-tilt camera relative to the camera at two different moments are calculated, finally the deflection angle which needs to be corrected of the camera carried by the intelligent inspection robot is obtained, the self-adaptive adjustment of the airborne pan-tilt is realized, the robot captures the fixed positions of the targets at two different moments at a certain position in the image, and important basis is provided for the follow-up realization of intelligent analysis and intelligent detection of the image on the data by the robot.

Description

A method and device for self-adaptive adjustment of the angle of an airborne pan-tilt camera of an intelligent inspection robot

technical field

The invention relates to the technical field of machine vision measurement, and more particularly, to a method and a device for self-adaptive adjustment of the angle of an airborne pan-tilt camera of an intelligent inspection robot.

Background technique

During the inspection process of the intelligent inspection robot based on monocular vision, there are certain navigation and positioning errors when the robot walks and stops. The target deviates from the center of the imaging field of view of the camera. In severe cases, the target completely deviates from the imaging field of view of the camera, and the target to be detected cannot be imaged, which brings certain difficulties to the intelligent detection of the subsequent target and the early warning and judgment of equipment failure.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a method and device for self-adaptive adjustment of the angle of an airborne pan-tilt camera of an intelligent inspection robot, which consists of two images at two different times at a certain position captured by the intelligent inspection robot to form two views, Based on the extraction and matching of ORB-FAST feature points, the homography matrix of the two images is obtained, and the 3D reconstruction of the target point is carried out to obtain the deflection angle of the on-board pan-tilt camera of the intelligent inspection robot, which reduces the accumulation of the robot due to walking. The effect of the error on the inaccurate target positioning.

For achieving the above object, the technical scheme adopted in the present invention is as follows:

The present invention provides a method for adaptively adjusting the angle of an airborne pan-tilt camera of an intelligent inspection robot, comprising:

Acquire two images at two different times at a certain position captured by the intelligent inspection robot, and perform feature point extraction and matching in the overlapping area of the two images;

Calculate the camera extrinsic parameters of the two images and the homography matrix between the two images based on the extracted matching points;

Based on the camera extrinsic parameters of the two images and the homography matrix between the two images, taking one image as the benchmark, the rotation angle of the camera is calculated to adjust the pan/tilt.

Further, performing feature point extraction and matching in the overlapping area of the two images includes:

The ORB-FAST feature point extraction is performed on the overlapping area of two images at two different times at a certain position captured by the intelligent inspection robot, and the random sampling consensus algorithm is used to eliminate the mismatched points in the feature point pair to obtain two images. matching point.

Further, calculating the camera external parameters of the two images based on the extracted matching points includes:

Based on the principle of monocular mobile vision, the projection equations of two images at two moments at a certain position captured by the intelligent inspection robot are obtained as follows:

Among them, s ₁ and s ₂ are the non-zero scale factors of the cameras at the two moments, p ₁ and p ₂ are the homogeneous coordinates of the two-dimensional images in the two images captured at the two moments, and A ₁ and A ₂ are two The camera internal parameters at time, and A ₁ =A ₂ , [R ₁ T ₁ ] and [R ₂ T ₂ ] are the camera external parameters at two moments respectively, X _W is the world of the spatial point P captured by the camera 3D homogeneous coordinates;

The fundamental matrix F and the essential matrix E are established based on the projection equations of the two images as follows:

E=SR,

Among them, R is the unified representation of the camera's external parameters at any time;

Using the extracted matching points, the 8-point algorithm is used to solve the fundamental matrix F; and the camera internal parameters are obtained by Zhang's plane calibration method;

Based on the internal parameters of the camera and the fundamental matrix, the essential matrix E is obtained by solving;

Decompose the essential matrix E to obtain the camera extrinsic parameters of the two images.

Further, calculating the homography matrix between the two images based on the extracted matching points, including:

Based on the perspective projection model of the camera, the relationship between the image coordinate system and the world coordinate system of the matching point between the two images taken at two moments is obtained as follows:

Among them, (x _a , y _a , z _a ) and (x _b , y _b , z _b ) represent the homogeneous coordinates of the matched point pairs in the two images, respectively, and Ka and K _b are the camera internal _parameters of the two images. , obtained by calibration, [R _a T _a ] and [R _b T _b ] are the camera external parameters of the two images respectively;

After conversion we get,

并假设

make,

and assume

get,

Based on the relationship between pixel coordinates and homogeneous coordinates of matched point pairs, we get:

u _a =x _a /z _a , v _a =y _a /z _a ;

Using 4 pairs of matching points to solve the above equation, the homography matrix H between the two images is obtained.

Further, the camera extrinsic parameters based on the two images and the homography matrix between the two images are based on one image, and the rotation angle of the camera is calculated, including:

According to the image coordinates and camera external parameters of a set of matched point pairs p and q in the two images I _a and I _b , the three-dimensional world coordinate system P=(X ₁ , Y ₁ of the spatial point P captured by the computer , Z ₁ );

Determine the corresponding point q'=(x _a , y _a ) of the matching point p in the image I _b in the absence of navigation and positioning errors;

Based on the homography matrix H and the point q' between the two images, the matching point p' of the point q' in the image I _a is calculated as follows:

p'=H ^-T q'=H ^-T (x _a , y _a , 1);

According to the image coordinates of the points p' and q', and the camera external parameters, calculate the three-dimensional world coordinates of the virtual space point P' corresponding to the points p' and q'P'=(X' ₁ , Y' ₁ , Z' ₁ );

Then the rotation angle of the camera is calculated as follows:

Among them, θ is the rotation angle of the camera relative to the image I _a , and o ₂ represents the origin in the coordinate system where the image I _b is located.

The present invention also provides an airborne pan-tilt camera angle adaptive adjustment device for an intelligent inspection robot, comprising:

The preprocessing module is used to obtain two images at two different times at a certain position captured by the intelligent inspection robot, and perform feature point extraction and matching in the overlapping area of the two images;

a calculation module for calculating the camera extrinsic parameters of the two images and the homography matrix between the two images based on the extracted matching points;

The adjustment module is used for adjusting the pan/tilt by calculating the rotation angle of the camera based on the external parameters of the camera of the two images and the homography matrix between the two images, and taking one image as a reference.

Further, the preprocessing module is specifically used for,

The ORB-FAST feature point extraction is performed on the overlapping area of two images at two different times at a certain position captured by the intelligent inspection robot, and the random sampling consensus algorithm is used to eliminate the mismatched points in the feature point pair to obtain two images. matching point.

Further, the computing module is specifically used for,

Based on the principle of monocular mobile vision, the projection equations of two images at two moments at a certain position captured by the intelligent inspection robot are obtained as follows:

Among them, s ₁ and s ₂ are the non-zero scale factors of the cameras at the two moments, p ₁ and p ₂ are the homogeneous coordinates of the two-dimensional images in the two images captured at the two moments, and A ₁ and A ₂ are two The camera internal parameters at time, and A ₁ =A ₂ , [R ₁ T ₁ ] and [R ₂ T ₂ ] are the camera external parameters at two moments respectively, X _W is the world of the spatial point P captured by the camera 3D homogeneous coordinates;

The fundamental matrix F and the essential matrix E are established based on the projection equations of the two images as follows:

E=SR,

Among them, R is the unified representation of the camera's external parameters at any time;

Using the extracted matching points, the 8-point algorithm is used to solve the fundamental matrix F; and the camera internal parameters are obtained by Zhang's plane calibration method;

Based on the internal parameters of the camera and the fundamental matrix, the essential matrix E is obtained by solving;

Decompose the essential matrix E to obtain the camera extrinsic parameters of the two images.

Further, the computing module is specifically used for,

Based on the perspective projection model of the camera, the relationship between the image coordinate system and the world coordinate system of the matching point between the two images captured at two moments is obtained as follows:

Among them, (x _a , y _a , z _a ) and (x _b , y _b , z _b ) represent the homogeneous coordinates of the matched point pairs in the two images, respectively, and Ka and K _b are the camera internal _parameters of the two images. , obtained by calibration, [R _a T _a ] and [R _b T _b ] are the camera external parameters of the two images respectively;

After conversion we get,

并假设

make,

and assume

get,

Based on the relationship between the pixel coordinates and the homogeneous coordinates of the matched point pairs, we get:

u _a =x _a /z _a , v _a =y _a /z _a ;

Using 4 pairs of matching points to solve the above equation, the homography matrix H between the two images is obtained.

Further, the adjustment module is specifically used for,

According to the image coordinates and camera external parameters of a set of matched point pairs p and q in the two images I _a and I _b , the three-dimensional world coordinate system P=(X ₁ , Y ₁ of the spatial point P captured by the computer , Z ₁ );

Determine the corresponding point q'=(x _a , y _a ) of the matching point p in the image I _b in the absence of navigation and positioning errors;

Based on the homography matrix H and the point q' between the two images, the matching point p' of the point q' in the image I _a is calculated as follows:

p'=H ^-T q'=H ^-T (x _a , y _a , 1);

According to the image coordinates of the points p' and q', and the camera external parameters, calculate the three-dimensional world coordinates of the virtual space point P' corresponding to the points p' and q'P'=(X' ₁ , Y' ₁ , Z' ₁ );

Then the rotation angle of the camera is calculated as follows:

Among them, θ is the rotation angle of the camera relative to the image I _a , and o ₂ represents the origin in the coordinate system where the image I _b is located.

The beneficial effects of the present invention are:

The invention provides an adaptive adjustment method for the angle of an airborne pan-tilt camera of an intelligent inspection robot. Two images at two different moments at a certain position captured by the intelligent inspection robot form two views. Based on ORB-FAST Extraction and matching of feature points, obtain the homography matrix of the two images, and perform 3D reconstruction of the target point to obtain the deflection angle of the intelligent inspection robot's airborne pan-tilt camera, which reduces the robot's cumulative walking error. The impact of inaccurate targeting. The invention can realize the self-adaptive calibration of the camera angle of the intelligent inspection robot in the complex substation environment, and provide a strong guarantee for the intelligent inspection and safe operation and maintenance of the substation robot.

Description of drawings

Figure 1 is an example of a homography constraint formed by two images at two different times at a certain position captured by an intelligent inspection robot;

FIG. 2 is an example of three-dimensional reconstruction of a moving target of the two images shown in FIG. 1 .

Detailed ways

The present invention is further described below. The following examples are only used to illustrate the technical solutions of the present invention more clearly, and cannot be used to limit the protection scope of the present invention.

At the same position at different times, due to the positioning error of the inspection robot and the rotation error of the pan/tilt angle of the fixed camera, the imaging position of the photographed target in the image is not fixed and the target captured at different times has different degrees of deviation in the image. , which brings certain difficulties to the intelligent inspection, fault diagnosis, identification and early warning of robot targets. The present invention provides a method for self-adaptive adjustment of the angle of an airborne pan-tilt camera of an intelligent inspection robot, which utilizes the monocular mobile vision technology to obtain two different positions near a certain position through the camera on the intelligent inspection robot. The target image of the moment. According to the feature points extracted from the two images, the pose and angle of the target point relative to the camera at the positions of the gimbal camera at two different times are calculated, so as to realize the adaptive adjustment of the angle of the gimbal camera by the robot, so that the robot can adjust the angle of the gimbal camera at a certain time. Two targets captured at one location at different times are in a fixed position of the image, which provides an important basis for the subsequent realization of intelligent data analysis and intelligent image detection by the robot.

A method for adaptively adjusting the angle of an airborne pan-tilt camera of an intelligent inspection robot, comprising:

Acquire two images at two different times at a certain position captured by the intelligent inspection robot, and perform feature point extraction and matching in the overlapping area of the two images;

Calculate the camera extrinsic parameters of the two images and the homography matrix between the two images based on the extracted matching points;

Based on the camera extrinsic parameters of the two images and the homography matrix between the two images, taking one image as the benchmark, the rotation angle of the camera is calculated to adjust the pan/tilt.

As a preferred implementation, in an embodiment of the present invention, the projection equations of the cameras at two moments are obtained based on the principle of monocular moving vision, and the fundamental matrix and the essential matrix are established, and the external structural parameters between the two views are finally obtained. R and t.

The principle of monocular mobile vision is as follows:

According to the pinhole imaging model of the camera, the accurate calibration of the airborne camera can be achieved by using the camera calibration method of plane grid points. Assuming that the homogeneous coordinates of the three-dimensional points on the target plane are marked as M=[x,y,z,1] ^T , and the homogeneous coordinates of the two-dimensional points on the image plane are m=[u,v,1] ^T , between the two The projective relation is:

sm=A[R t]M (1)

A称为摄像机的内部参数矩阵。α_x、α_y是u轴和v轴的尺度因子，(u₀,v₀)为主点坐标，r是u轴和v轴的不垂直因子。由张氏平面标定法，可以求出像机的内参数矩阵A。Among them, s is an arbitrary non-zero scale factor, [R t] is a matrix with 3 rows and 4 columns, called the camera extrinsic parameter matrix, R is called the rotation matrix, t=(t ₁ , t ₂ , t ₃ ) ^T is called the translation matrix,

A is called the camera's internal parameter matrix. α _x , α _y are the scale factors of the u-axis and v-axis, (u ₀ , v ₀ ) are the coordinates of the main point, and r is the non-perpendicular factor of the u-axis and v-axis. By Zhang's plane calibration method, the internal parameter matrix A of the camera can be obtained.

The mobile monocular vision measurement system is a multi-eye vision measurement system formed by moving one camera and virtualizing it into multiple cameras. In this embodiment, the principle of a mobile monocular vision measurement system is analyzed by taking two images taken by a robot at a certain position at two different times to form a two-view vision measurement as an example.

Assuming that the world three-dimensional homogeneous coordinate of a point P in space is X _W , and the two-dimensional image homogeneous coordinates of the two images captured at two moments are p ₁ and p ₂ , then the camera at two moments can be obtained by formula (1). The projection equation of is:

Among them, s ₁ and s ₂ are the non-zero scale factors of the two cameras, [R ₁ T ₁ ] and [R ₂ T ₂ ] are the camera external parameters of the camera at time 1 and time 2, respectively, A ₁ and A ₂ is the camera internal parameters of the camera at time 1 and time 2 respectively, because the camera only moves rigidly and the internal structure parameters do not change, so A ₁ =A ₂ .

Combined with the geometric constraints of epipolar lines, the expressions of the fundamental matrix F and the essential matrix E can be obtained from equation (2), which are respectively as follows:

E=SR (4)

R是像机在任何一个时刻下的像机外参的统一表示。where S is an antisymmetric matrix,

R is a unified representation of the camera's external parameters at any time.

It can be seen from equation (3) that the fundamental matrix F is only related to the internal parameters of the two cameras and the external structural parameters of the system, while the airborne camera only performs rigid body motion due to the rotation of the PTZ, and the internal parameters of the camera remain unchanged. Therefore, the essential matrix E of formula (4) can be obtained. It can be seen that E is only related to the external parameters of the vision system, and E can be decomposed to obtain the external structural parameters R and t of the mobile monocular vision measurement system.

From the epipolar geometric constraint relationship and the definition of the essential matrix, it can be seen that the basic matrix is a rank 2 matrix with 7 degrees of freedom. Through the extraction and matching of the feature points of the two images, the 8-point algorithm can be used to find the two The fundamental matrix F between the view images, combined with the internal parameters of the camera, can obtain the essential matrix E. By decomposing the matrix E, the external structural parameters R and t between the two views can be finally obtained.

As a preferred implementation, in an embodiment of the present invention, the characteristics of ORB (Oriented Fast and Rotated BRIEF)-FAST feature points are used, that is, the ORB-FAST feature remains unchanged for image rotation, scale scaling, and brightness changes. The ORB-FAST feature point extraction is performed on the images with overlapping areas captured by the robot camera at a certain position at two different times, and the Random Sample Consensus (RANSAC) algorithm is used to eliminate the mismatching points in the image pairs. Accurate registration of Scale-invariant feature transform (SIFT) feature points between two images.

As shown in Figure 1, at a certain position, due to the navigation positioning error and the pan-tilt angle rotation error of the inspection robot, the airborne camera images the same target at different times, and two images I _a with overlapping areas are obtained. and I _b , ORB-FAST feature points are extracted and matched in the overlapping regions of the images. Combined with formulas (3) and (4), using the 8-point algorithm analyzed above, the external structural parameters R and t between the camera coordinate systems at two moments can be obtained. The external parameters are important for the reconstruction of three-dimensional points in space. significance.

As a preferred implementation, in an embodiment of the present invention, to calculate the homography matrix between two views, the specific implementation process is as follows:

Combining the information of the matching feature points of the two images, and using the perspective projection model of the camera, the relationship between the image coordinate system and the world coordinate system of the matching point between the images at the two moments can be obtained respectively,

Among them, (x _a , y _a , z _a ) and (x _b , y _b , z _{b )} represent the homogeneous coordinates of the matching point pairs in the images I _a and I _b respectively, and Ka and K _b are the matching point pairs The internal parameters can be obtained by calibration, [R _a T _a ] and [R _b T _b ] are the camera external parameters of the matching point pair, respectively.

It can be obtained from formulas (5) and (6),

H为3×3的矩阵，H矩阵反映了两幅图像特征点之间的映射关系，定义H为两幅平面之间的单应性矩阵，假设

代入公式(7)可以得到，make

H is a 3×3 matrix, and the H matrix reflects the mapping relationship between the feature points of the two images. Define H as the homography matrix between the two planes. Suppose

Substitute into formula (7) to get,

Based on u _a =x _a /z _a , v _a =y _a /z _a ,

Then it can be obtained from formula (8),

Among them, (u _a , v _a ) and (u _b , v _b ) are the pixel coordinates of the matched point pairs on the two images, u _a =x _a /z _a , v _a =y _a /z _a .

It can be seen from formula (9) that two equations can be obtained for each pair of feature points. The H matrix is a singular matrix of rank 8, and at least 4 pairs of matching points can solve the homography matrix H of the two planes.

As a preferred implementation, in an embodiment of the present invention, the offset angle of the computer is calculated, and the specific implementation process is as follows:

Taking the first image as the reference image, select a set of matching feature point pairs p and q in the matching feature points in the two images, the corresponding spatial feature point is P, and its image coordinates in the corresponding two images are p respectively =(x _a , y _a ) and q=(x _b , y _b ), as shown in FIG. 2 .

Combined with the principle of binocular stereo vision measurement, for the accurately calibrated dual-view image machine measurement system, the image coordinates of the matching point pair and the external structural parameters R and t are known, and the three-dimensional world coordinate system of the space point P can be accurately obtained. The world coordinate system is established under the camera 2 coordinate system, in order to facilitate the calculation of the rotation angle of the camera. Assume that the calculated three-dimensional world coordinates of the space point P at this time are P=(X ₁ , Y ₁ , Z ₁ ).

Due to the navigation and positioning errors of the robot, there is a deviation in the image coordinates of the matching feature points of the two images taken at different times. side. Assuming that the robot's navigation and positioning are accurate and the above errors do not exist, set the image coordinates in image 1 as p=(x _a , y _a ), then the position and angle of the camera at time 1 should be the same as the position and angle of the camera at time 2, Therefore, in the second image, the image coordinates of the matching point corresponding to point p in image 1 should be q'=(x _a , y _a ). In fact, due to the existence of errors, the camera position and angle at time 2 deviate from the camera position and angle at time 1.

In this embodiment, when the navigation error exists, the position of the inspection robot is not changed, and the shooting position of the camera is adjusted by rotating the angle of the pan/tilt, so that the target does not deviate from the center position of the image. The rotation angle of the gimbal is calculated by the angle error calculated by the camera.

It can be seen from the above analysis that if the target point wants to achieve the position on the image captured at time 2 and the image captured at time 1 with the same image coordinates, the camera at position 2 should have the same image coordinates as the robot does not move. The moving angle is ∠qo ₂ q'.

In fact, by obtaining the homography matrix H of the two images from the above analysis, the matching point p' in image 1 corresponding to the point q'=(x _a , _ya ) in image 2 can be obtained, which is,

p'=H ^-T q'=H ^-T (x _a ,y _a ,1) (10)

We call p' and q' a virtual image matching point pair, and the space point corresponding to them is P', which is called a virtual space three-dimensional point. Similarly, given the image coordinates of the corresponding point pair p' and q', and the external structural parameters R and t of the system, combined with the principle of binocular stereo vision measurement, the three-dimensional world coordinates of the virtual space point P' can be obtained. P'=(X' ₁ , Y' ₁ , Z' ₁ ).

From the analysis of Figure 2, because the three-dimensional coordinates of the space point are established in the coordinate system of the camera 2, ∠qo ₂ q'=∠Po ₂ P', then the angle of the camera 2 movement should be calculated by the following formula, namely:

By decomposing (11), the angle that the camera should rotate, that is, the angle that the robot head should rotate.

The embodiment of the present invention analyzes the geometric constraint relationship of the two-view image for the inspection and positioning error of the intelligent inspection robot in the substation, taking the image feature extraction and matching as the starting point, and proposes an image three-dimensional point extraction algorithm based on ORB-FAST features, The camera positioning error of the intelligent inspection robot is calibrated, which reduces the impact of the robot on the inaccurate target positioning caused by the accumulated walking error.

Example 1

The simulation software used in the experiment of this embodiment is Visual C++2019, the host information is the computer's main frequency 3.6GHZ, the memory is 32GB, the operating system is Windows 10, 32 bits, and the robot used in the substation is produced by Zhejiang Dali Technology Co., Ltd. The model is DL-RC63 intelligent inspection robot. In this experiment, the calibration method of Zhang Zhengyou's plane target camera is used to realize the calibration of the internal parameters of the inspection robot camera, as shown in Table 1. Then use the algorithm proposed by the present invention to analyze the images acquired by the intelligent inspection robot at the same place at two different times, take the image captured for the first time as the reference image, and solve the problem obtained by the second acquisition according to the algorithm proposed by the present invention. The angular offset value of the image relative to the base image. According to the obtained deflection angle of the camera, the angle of the robot turntable is automatically adjusted, that is, the camera mounted on the intelligent inspection robot can adjust the field of view angle of the same target at different times, reducing the comparison of the intelligent inspection robot caused by positioning errors. Great cheap. It can be seen from the experimental results that the algorithm proposed by the present invention has a good effect on the self-adaptive calibration of the camera angle of the inspection robot for the images captured at different times by using the geometric constraint relationship between the images, which verifies the present invention. The proposed algorithm has a certain robustness to the angle adaptive calibration of the inspection robot in the complex environment inside the substation.

Table 1 Calibration results of camera parameters

Example 2

In this example, the algorithm proposed by the present invention is used in the experiment. Compared with other classical feature extraction algorithms, such as the SIFT feature point matching and positioning method and the SURF feature point matching and positioning method, respectively, the electrical cabinets in the substation actually photographed by the intelligent inspection robot are analyzed. Analyze the instrument images, compare the instruments in the 50 images obtained by shooting, and use the above three algorithms to calculate the average value of the time used for target positioning. The camera angle self-adaptive calibration is suitable for the robot to transform into the intelligent inspection robot in the complex environment of the site. The algorithm proposed by the invention has high calibration efficiency and can provide important technical support for the online intelligent detection of the intelligent inspection robot of the substation.

Table 2 Comparison of target detection time by different algorithms

Algorithm used Running time (seconds) SIFT feature matching localization 57 SURF feature matching localization 26 The present invention proposes algorithm positioning 19

On the other hand, the present invention provides an airborne pan-tilt camera angle adaptive adjustment device for an intelligent inspection robot, including:

The preprocessing module is used to obtain two images at two different times at a certain position captured by the intelligent inspection robot, and perform feature point extraction and matching in the overlapping area of the two images;

a calculation module for calculating the camera extrinsic parameters of the two images and the homography matrix between the two images based on the extracted matching points;

The adjustment module is used for adjusting the pan/tilt by calculating the rotation angle of the camera based on the external parameters of the camera of the two images and the homography matrix between the two images, and taking one image as a reference.

As a preferred implementation manner, in an embodiment of the present invention, the preprocessing module is specifically used to:

The ORB-FAST feature point extraction is performed on the overlapping area of two images at two different times at a certain position captured by the intelligent inspection robot, and the random sampling consensus algorithm is used to eliminate the mismatched points in the feature point pair to obtain two images. matching point.

As a preferred implementation manner, in an embodiment of the present invention, the computing module is specifically used to:

Based on the principle of monocular mobile vision, the projection equations of two images at two moments at a certain position captured by the intelligent inspection robot are obtained as follows:

Among them, s ₁ and s ₂ are the non-zero scale factors of the cameras at the two moments, p ₁ and p ₂ are the homogeneous coordinates of the two-dimensional images in the two images captured at the two moments, and A ₁ and A ₂ are two The camera internal parameters at time, and A ₁ =A ₂ , [R ₁ T ₁ ] and [R ₂ T ₂ ] are the camera external parameters at two moments respectively, X _W is the world of the spatial point P captured by the camera 3D homogeneous coordinates;

The fundamental matrix F and the essential matrix E are established based on the projection equations of the two images as follows:

E=SR,

Among them, R is the unified representation of the camera's external parameters at any time;

Using the extracted matching points, the 8-point algorithm is used to solve the fundamental matrix F; and the camera internal parameters are obtained by Zhang's plane calibration method;

Based on the internal parameters of the camera and the fundamental matrix, the essential matrix E is obtained by solving;

Decompose the essential matrix E to obtain the camera extrinsic parameters of the two images.

As a preferred implementation manner, in an embodiment of the present invention, the computing module is specifically used to:

Based on the perspective projection model of the camera, the relationship between the image coordinate system and the world coordinate system of the matching point between the two images taken at two moments is obtained as follows:

Among them, (x _a , y _a , z _a ) and (x _b , y _b , z _b ) represent the homogeneous coordinates of the matched point pairs in the two images, respectively, and Ka and K _b are the camera internal _parameters of the two images. , obtained by calibration, [R _a T _a ] and [R _b T _b ] are the camera external parameters of the two images respectively;

After conversion we get,

并假设

make,

and assume

get,

Based on the relationship between pixel coordinates and homogeneous coordinates of matched point pairs, we get:

u _a =x _a /z _a , v _a =y _a /z _a ;

Using 4 pairs of matching points to solve the above equation, the homography matrix H between the two images is obtained.

As a preferred implementation manner, in an embodiment of the present invention, the adjustment module is specifically used to:

According to the image coordinates and camera external parameters of a set of matched point pairs p and q in the two images I _a and I _b , the three-dimensional world coordinate system P=(X ₁ , Y ₁ of the spatial point P captured by the computer , Z ₁ );

Determine the corresponding point q'=(x _a , y _a ) of the matching point p in the image I _b in the absence of navigation and positioning errors;

Based on the homography matrix H and the point q' between the two images, the matching point p' of the point q' in the image I _a is calculated as follows:

p'=H ^-T q'=H ^-T (x _a , y _a , 1);

According to the image coordinates of the points p' and q', and the camera external parameters, calculate the three-dimensional world coordinates of the virtual space point P' corresponding to the points p' and q'P'=(X' ₁ , Y' ₁ , Z' ₁ );

Then the rotation angle of the camera is calculated as follows:

Among them, θ is the rotation angle of the camera relative to the image I _a , and o ₂ represents the origin in the coordinate system where the image I _b is located.

As will be appreciated by those skilled in the art, the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: the present invention can still be Modifications or equivalent replacements are made to the specific embodiments of the present invention, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention shall be included within the protection scope of the claims of the present invention.

Claims (10)

1. an airborne pan-tilt camera angle adaptive adjustment method of an intelligent inspection robot, is characterized in that, comprises: Acquire two images at two different times at a certain position captured by the intelligent inspection robot, and perform feature point extraction and matching in the overlapping area of the two images; Calculate the camera extrinsic parameters of the two images and the homography matrix between the two images based on the extracted matching points; Based on the camera extrinsic parameters of the two images and the homography matrix between the two images, taking one image as the benchmark, the rotation angle of the camera is calculated to adjust the pan/tilt. 2. The method for self-adaptive adjustment of an airborne pan-tilt camera angle of an intelligent inspection robot according to claim 1, wherein the feature point extraction and matching are performed in the overlapping area of the two images, comprising: The ORB-FAST feature point extraction is performed on the overlapping area of two images at two different times at a certain position captured by the intelligent inspection robot, and the random sampling consensus algorithm is used to eliminate the mismatched points in the feature point pair to obtain two images. matching point. 3. The method for adaptively adjusting the angle of an airborne pan-tilt camera of an intelligent inspection robot according to claim 1, wherein the calculation of the camera external parameters of the two images based on the extracted matching points includes the following steps: : Based on the principle of monocular mobile vision, the projection equations of two images at two moments at a certain position captured by the intelligent inspection robot are obtained as follows:

Among them, s ₁ and s ₂ are the non-zero scale factors of the cameras at the two moments, p ₁ and p ₂ are the homogeneous coordinates of the two-dimensional images in the two images captured at the two moments, and A ₁ and A ₂ are two The camera internal parameters at time, and A ₁ =A ₂ , [R ₁ T ₁ ] and [R ₂ T ₂ ] are the camera external parameters at two moments respectively, X _W is the world of the spatial point P captured by the camera 3D homogeneous coordinates; The fundamental matrix F and the essential matrix E are established based on the projection equations of the two images as follows:

E=SR,

Among them, R is the unified representation of the camera's external parameters at any time; Using the extracted matching points, the 8-point algorithm is used to solve the fundamental matrix F; and the camera internal parameters are obtained by Zhang's plane calibration method; Based on the internal parameters of the camera and the fundamental matrix, the essential matrix E is obtained by solving; Decompose the essential matrix E to obtain the camera extrinsic parameters of the two images. 4. The method for self-adaptive adjustment of an airborne pan-tilt camera angle of an intelligent inspection robot according to claim 3, wherein the method for calculating a homography matrix between two images based on the extracted matching points ,include: Based on the perspective projection model of the camera, the relationship between the image coordinate system and the world coordinate system of the matching point between the two images captured at two moments is obtained as follows:

Among them, (x _a , y _a , z _a ) and (x _b , y _b , z _b ) represent the homogeneous coordinates of the matched point pairs in the two images, respectively, and Ka and K _b are the camera internal _parameters of the two images. , obtained by calibration, [R _a T _a ] and [R _b T _b ] are the camera external parameters of the two images respectively; After conversion we get,

make,

and assume

get,

Based on the relationship between pixel coordinates and homogeneous coordinates of matched point pairs, we get:

u _a =x _a /z _a , v _a =y _a /z _a ; Using 4 pairs of matching points to solve the above equation, the homography matrix H between the two images is obtained. 5. The method for self-adaptive adjustment of the camera angle of an airborne pan-tilt camera of an intelligent inspection robot according to claim 3, wherein the camera external parameter based on two images and an external parameter between the two images. The homography matrix, based on an image, calculates the rotation angle of the camera, including: According to the image coordinates and camera external parameters of a set of matched point pairs p and q in the two images I _a and I _b , the three-dimensional world coordinate system P=(X ₁ , Y ₁ of the spatial point P captured by the computer , Z ₁ ); Determine the corresponding point q'=(x _a , y _a ) of the matching point p in the image I _b in the absence of navigation and positioning errors; Based on the homography matrix H and the point q' between the two images, the matching point p' of the point q' in the image I _a is calculated as follows: p'=H ^-T q'=H ^-T (x _a , y _a , 1); According to the image coordinates of points p' and q', and the camera external parameters, calculate the three-dimensional world coordinates of the virtual space point P' corresponding to the points p' and q'P'=(X' ₁ , Y' ₁ , Z' ₁ ); Then the rotation angle of the camera is calculated as follows:

Among them, θ is the rotation angle of the camera relative to the image I _a , and o ₂ represents the origin in the coordinate system where the image I _b is located. 6. An airborne pan-tilt camera angle adaptive adjustment device for an intelligent inspection robot, characterized in that, comprising: The preprocessing module is used to obtain two images at two different times at a certain position captured by the intelligent inspection robot, and perform feature point extraction and matching in the overlapping area of the two images; a calculation module for calculating the camera extrinsic parameters of the two images and the homography matrix between the two images based on the extracted matching points; The adjustment module is used for adjusting the pan/tilt by calculating the rotation angle of the camera based on the external parameters of the camera of the two images and the homography matrix between the two images, and taking one image as a reference. 7. The airborne pan-tilt camera angle adaptive adjustment device of an intelligent inspection robot according to claim 6, wherein the preprocessing module is specifically used to: The ORB-FAST feature point extraction is performed on the overlapping area of two images at two different times at a certain position captured by the intelligent inspection robot, and the random sampling consensus algorithm is used to eliminate the mismatched points in the feature point pair to obtain two images. matching point. 8. The device for self-adaptive adjustment of an airborne pan-tilt camera angle of an intelligent inspection robot according to claim 6, wherein the computing module is specifically used to: Based on the principle of monocular mobile vision, the projection equations of two images at two moments at a certain position captured by the intelligent inspection robot are obtained as follows:

Among them, s ₁ and s ₂ are the non-zero scale factors of the cameras at the two moments, p ₁ and p ₂ are the homogeneous coordinates of the two-dimensional images in the two images captured at the two moments, and A ₁ and A ₂ are two The camera internal parameters at time, and A ₁ =A ₂ , [R ₁ T ₁ ] and [R ₂ T ₂ ] are the camera external parameters at two moments respectively, X _W is the world of the spatial point P captured by the camera 3D homogeneous coordinates; The fundamental matrix F and the essential matrix E are established based on the projection equations of the two images as follows:

E=SR,

Among them, R is the unified representation of the camera's external parameters at any time; Using the extracted matching points, the 8-point algorithm is used to solve the fundamental matrix F; and the camera internal parameters are obtained by Zhang's plane calibration method; Based on the internal parameters of the camera and the fundamental matrix, the essential matrix E is obtained by solving; Decompose the essential matrix E to obtain the camera extrinsic parameters of the two images. 9 . The airborne pan-tilt camera angle adaptive adjustment device of an intelligent inspection robot according to claim 6 , wherein the computing module is specifically used for: 10 . Based on the perspective projection model of the camera, the relationship between the image coordinate system and the world coordinate system of the matching point between the two images captured at two moments is obtained as follows:

Among them, (x _a , y _a , z _a ) and (x _b , y _b , z _b ) represent the homogeneous coordinates of the matched point pairs in the two images, respectively, and Ka and K _b are the camera internal _parameters of the two images. , obtained by calibration, [R _a T _a ] and [R _b T _b ] are the camera external parameters of the two images respectively; After conversion we get,

make,

and assume

get,

Based on the relationship between pixel coordinates and homogeneous coordinates of matched point pairs, we get:

u _a =x _a /z _a , v _a =y _a /z _a ; Using 4 pairs of matching points to solve the above equation, the homography matrix H between the two images is obtained. 10 . The airborne pan-tilt camera angle adaptive adjustment device of an intelligent inspection robot according to claim 7 , wherein the adjustment module is specifically used to: 11 . According to the image coordinates and camera external parameters of a set of matched point pairs p and q in the two images I _a and I _b , the three-dimensional world coordinate system P=(X ₁ , Y ₁ of the spatial point P captured by the computer , Z ₁ ); Determine the corresponding point q'=(x _a , y _a ) of the matching point p in the image I _b in the absence of navigation and positioning errors; Based on the homography matrix H and the point q' between the two images, the matching point p' of the point q' in the image I _a is calculated as follows: p'=H ^-T q'=H ^-T (x _a , y _a , 1); According to the image coordinates of the points p' and q', and the camera external parameters, calculate the three-dimensional world coordinates of the virtual space point P' corresponding to the points p' and q'P'=(X' ₁ , Y' ₁ , Z' ₁ ); Then the rotation angle of the camera is calculated as follows:

Among them, θ is the rotation angle of the camera relative to the image I _a , and o ₂ represents the origin in the coordinate system where the image I _b is located.

Images