CN108317953A - A kind of binocular vision target surface 3D detection methods and system based on unmanned plane - Google Patents

Abstract

The invention discloses a binocular vision target surface 3D detection system based on an unmanned aerial vehicle, comprising: an unmanned aerial vehicle, a binocular vision system, a computer system and a remote control device; The vision system traverses and collects the image data of the target surface; uses the images collected by the binocular vision system to obtain the disparity information of the left and right images through feature extraction and matching; combines the internal and external parameters of the camera to obtain the 3D coordinates of the midpoint of the image to obtain the local 3D point of the target cloud; through the feature extraction and matching of the 3D point cloud of the adjacent image of the left camera, the transformation relationship between the two 3D point cloud coordinate systems is obtained, and then the pairwise registration of the 3D point cloud is realized, and finally the 3D image of the entire target surface is obtained The point cloud data is compared with the original 3D data of the target surface to realize the detection of the target. The invention is applied to the surface detection of large objects such as large industrial equipment, and has the characteristics of simple operation and high efficiency.

Description

A 3D detection method and system for a binocular vision target surface based on an unmanned aerial vehicle

technical field

The invention relates to the technical field of visual measurement, in particular to a 3D detection method and system for a binocular vision target surface based on an unmanned aerial vehicle.

Background technique

For the 3D reconstruction of the surface of a large target, under the requirement of high precision, the fixed detector cannot perceive its entire 3D surface. Only the method of moving the detector to scan can be used to detect the three-dimensional data of the target surface block by block under the dynamic coordinate system, and then splicing to obtain the final target. Therefore, image 3D reconstruction and reconstructed 3D image mosaic technology play a very important role in this application field. At present, there are few studies on the methods of 3D visual reconstruction of large target surfaces, and even less people have been involved in the surface detection of relatively complex large equipment. Therefore, the research results in this area will greatly promote the level of industrial informatization, production efficiency and production safety.

Due to the advantages of UAVs such as flexible flight, low price, long battery life, large control range, and large payload, it provides a good solution for surface detection of large targets such as boilers, reaction towers, and distillation towers by using UAVs to mount binocular cameras. possible.

The current research on binocular vision 3D detection technology is as follows:

After retrieval, the application number is CN105716530A, and the patent name is "A Method for Measuring the Geometric Dimensions of Vehicles Based on Binocular Stereo Vision". The camera is used to collect vehicle scene images from five directions, front, rear, left, right, and rear, and calculate the same pixel points in the two scene images. Then calculate the depth information of the vehicle scene point, and then calculate the size of the vehicle. In this method, the camera is fixedly installed in five directions, and can only collect some relatively small and movable objects at fixed locations. For outdoor The measurement of a large device with a fixed location may not find a suitable installation location, which is time-consuming and labor-intensive, and the overall cost will also increase.

At present, the airborne 3D mapping method applied to large fixed targets mainly uses the mounted airborne 3D lidar to scan and collect local 3D point clouds, and then combines the UAV pose to complete the 3D mapping work. After retrieval, the publication number is CN106443705A, and the patent name is "airborne laser radar measurement system and method". This method is used for three-dimensional surveying and mapping, but 3D laser radar is expensive, heavy, and has high load requirements for drones, so it cannot be widely used. On small UAVs, the system cost is increased.

Contents of the invention

In order to solve the above problems, the present invention provides a method and system for 3D detection of the surface of a binocular vision target based on an unmanned aerial vehicle.

In order to achieve the above object, the technical scheme that the present invention takes is:

A drone-based binocular vision target surface 3D detection system, including:

The UAV is used to collect image data of the target surface sequentially from bottom to top and from left to right through the binocular vision system mounted on its bottom;

The computer system is used to process the images collected by the binocular vision system, including image feature extraction, feature matching, acquisition of depth information, calculation of three-dimensional coordinates, three-dimensional image stitching, etc.; output detection results;

Remote control equipment, through the wireless transmission module to control the flight path of the UAV and the opening and closing of the binocular vision system;

The drone includes

Power supply battery pack, used to provide power guarantee for UAV communication equipment, image acquisition equipment and power equipment;

The airborne control unit is used to receive instructions from the ground control center,

The attitude measurement system is used to measure the attitude of the UAV in real time, and can record the attitude information at each moment;

The wifi/wireless transmission module is used to realize wireless communication with the remote control device and receive control instructions sent by the remote control device;

The binocular vision system is fixed with the drone, and the camera is set facing the target surface to collect target images.

The binocular vision system includes

A pose measuring device for measuring the pose of the binocular vision system;

The left and right cameras are used to collect the image data of the target;

The memory card is used to store the image data collected by the left and right cameras respectively;

The left and right cameras of the binocular vision system are placed in parallel and have the same parameters.

The present invention also provides a 3D detection method based on a drone-based binocular vision target surface, comprising the following steps:

Step 1: Use the remote control device to control the UAV to collect images according to the predetermined trajectory. After the image collection is completed, import the images in the memory card into the computer for processing;

Step 2: Number the images collected by the airborne binocular vision system in chronological order to form a sequence pair of binocular left and right images;

Step 3: Camera calibration, the internal and external parameters of the camera can be obtained, including the camera focal length f and baseline distance T, etc.;

Step 4: Perform denoising, image pose adjustment and other processing on the collected images;

Step 5: Extract feature points from the left and right images captured at the same time, perform feature point matching, obtain disparity information, and calculate the 3D coordinates of the midpoint of the image to obtain a local 3D point cloud;

Step 6: Register the 3D point clouds of the adjacent images captured by the left camera, unify the two 3D point clouds into one coordinate system, and obtain the 3D point cloud model of the entire target surface;

Step 7: Use the obtained 3D point cloud model to compare with the surface factory data given by the large-scale equipment to detect whether there is any difference.

Wherein, in the step 4, the pose information of the UAV is referred to and the pose of the image is corrected by affine transformation.

Described step 5 specifically comprises the following steps:

S51, extract the surf feature point, take a 4*4 rectangular area block around the feature point, each sub-area counts the haar wavelet feature of the horizontal direction and the vertical direction of 25 pixels, the haar wavelet feature is after the horizontal direction value, the vertical direction After the direction value, after the absolute value of the horizontal direction, and the sum of the absolute value of the vertical direction, these four directions are used as the feature vector of each sub-block area, and the matching degree is determined by calculating the Euclidean distance between two feature points, Euclidean The shorter the distance, the better the matching degree of the two feature points, and the disparity information can be obtained from the matched feature points;

S52. According to the disparity information d= _xr - _x1 and the internal and external parameters of the camera, the three-dimensional coordinates of the point are obtained by the following formula:

The three-dimensional point cloud registration method in the step 6 can be divided into the following steps:

S1. Establish a K neighborhood for each point in the adjacent three-dimensional point cloud coordinate system;

S2. Extract feature points, establish feature descriptions for each feature point, and match adjacent 3D point cloud feature points;

S3. According to the matching point pairs, the conversion relationship of the two point cloud coordinate systems can be calculated to complete the initial registration;

S4. According to the initial position provided by the initial registration, the iterative closest point algorithm is used to iterate the adjacent point cloud coordinate system, and finally the accurate conversion relationship of the adjacent point cloud coordinate system can be obtained, so that the adjacent point cloud coordinate system is one to one in the coordinate system.

In the step S1, the KD-tree method is used to speed up the K neighborhood of the search point.

In the step S2, the normal vector of the point is estimated by the principal component analysis method, and the feature point is extracted with the normal vector variation as a standard. According to the geometric relationship between the feature point and the adjacent point, a variety of features are selected to describe the feature point. The statistical characteristics of the histogram Statistical feature points and various features of the points in the K neighborhood of the feature points, establish a histogram feature description for it, get the feature vector, use the distance of the feature vector between the two points as the matching basis, and the distance between the two points The minimum distance between feature vectors is the matching point pair.

In the step S3, the quaternion method is used to obtain the transformation matrix between adjacent point cloud coordinate systems, and the implementation steps are as follows: S31: Calculate the center of gravity of the source point set P and the target point set Q respectively:

in, are the three-dimensional coordinates of the center of gravity of the source point set P and the target point set _Q ; N _P , N _Q are the number of points in the source point set P and the target point set _Q ; The three-dimensional coordinates of the i-th point in the target point set Q.

S32: Construct a covariance matrix according to the data point sets P and Q:

S33: Construct a 4×4 symmetric matrix according to the covariance matrix:

Wherein, I ₃ is the third-order identity matrix, tr (ΣP, Q) is the trace of matrix ΣP, Q,

A＝[A ₂₃ A ₃₁ A ₁₂ ] ^T ; A _{i, j} = (∑P, Q-∑P, Q ^T ) _ij

S34: Obtain the eigenvalue and eigenvector of Q(∑P, Q), and the eigenvector of the largest eigenvalue is the required rotation vector

q _R = [q ₀ q ₁ q ₃ ] ^T ;

S35: According to the rotation vector q _R obtained by the above formula, the rotation matrix R can be obtained, so that the best translation vector can be obtained, and the formula is:

In step S4, unifying the two adjacent point cloud coordinate systems Xi _-1 and _Xi into the same coordinate system can be realized by the following formula:

X _i-1 =X _i R+T; wherein, R is a rotation matrix, and T is a translation matrix.

The present invention has the following beneficial effects:

(1) The present invention uses a binocular camera as a three-dimensional detection sensor, which is lighter in weight than the lidar detection method, and can choose a small unmanned aerial vehicle as a carrier, while reducing the cost of equipment procurement.

(2) The current camera can achieve high enough precision to fully meet the detection requirements, and the existing matching reconstruction technology has been developed quite maturely, and the technical aspects can fully meet the needs.

(3) The airborne binocular camera system is flexible and convenient to operate and relatively simple to use, and is suitable for three-dimensional surface detection under various conditions.

Description of drawings

Fig. 1 is a system structure diagram of a binocular vision target surface 3D detection system based on an unmanned aerial vehicle according to an embodiment of the present invention.

Fig. 2 is a schematic diagram of the flight route of the embodiment of the present invention.

Fig. 3 is a detection flow chart of a 3D detection system for a binocular vision target surface based on an unmanned aerial vehicle according to an embodiment of the present invention.

Fig. 4 is the schematic diagram of depth extraction of binocular camera in the embodiment of the present invention

In the figure: 1-UAV, 2-Battery pack, 3-Onboard control unit, 4-WiFi or wireless transmission module, 5-Binocular camera, 6-Fixed connection structure, 7-Attitude detection system.

Detailed ways

In order to make the objects and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the examples. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

As shown in Figure 1, the embodiment of the present invention provides a 3D detection system for binocular vision target surfaces based on drones. The 3D surface detection method is based on binocular stereo vision 3D reconstruction and estimates the Pose, through the software algorithm, the correction of the pose is combined with the 3D reconstruction to obtain the depth information of the target, and then obtain the surface detection result. The system consists of drone, battery pack, on-board control unit, WiFi or wireless telex module, binocular camera with memory card, fixed attachment mechanism to attach the camera to the bottom of the drone. When the system is working, the battery pack provides the power source for the electronic equipment of the UAV, and the staff transmits instructions to the onboard control unit through the remote control device through WiFi or wireless transmission module to control the UAV's ascent, steering, flight route, etc. And can control the start and stop of the camera. During the specific implementation, the camera is separated from the target to be measured by a suitable distance, and then the field of view of the camera is tested, and then according to the horizontal range of the camera field of view, the target to be measured is divided into several parts based on the bottom, and then one of the parts is used as The starting point controls the vertical ascent of the rotorcraft, and according to the longitudinal range of the field of view, it is determined how many distances the camera rises to shoot once. The flight trajectory of the aircraft is shown in Figure 2, from bottom to top, then translate to the next part, and then from top to bottom until covering the entire surface of the target. Stop shooting, the aircraft returns to the ground, takes out the memory card, processes the pictures, performs three-dimensional reconstruction, and obtains the surface inspection results.

As shown in Figure 3, the 3D detection method based on the UAV visual surface according to the example of the present invention includes the following steps:

Step 1. Use the remote control device to control the UAV to collect images according to the predetermined trajectory. After the image collection is completed, import the pictures taken by the left and right cameras into the computer for numbering, and then combine the collected UAV pose information with the collection at each moment to get Corresponding to the picture, corrected by the affine transformation principle, and the left and right poles are mapped to infinity to realize the line alignment of the image.

Step 2, extract surf feature points for all images;

Step 3, match the surf feature points in each left and right image pair, the specific matching method is as follows:

In the Surf algorithm, a 4*4 rectangular area block is taken around the feature point, but the direction of the obtained rectangular area is along the main direction of the feature point. Each sub-region counts the haar wavelet features of the horizontal and vertical directions of 25 pixels, where the horizontal and vertical directions are relative to the main direction. The haar wavelet feature is 4 directions after the horizontal value, after the vertical value, after the absolute value of the horizontal direction, and the sum of the absolute value of the vertical direction. These 4 values are used as the feature vector of each sub-block area, so there are 4* 4*4=64-dimensional vectors are used as descriptors of Surf features.

Surf determines the matching degree by calculating the Euclidean distance between two feature points. The shorter the Euclidean distance, the better the matching degree between the two feature points. Take a key point in the left picture, and traverse the closest distance in the right picture Two key points, among the two key points, if the second-closest distance divided by the shortest distance is less than the threshold range, it is determined as a pair of matching points.

Step 4, depth calculation, as shown in Figure 4, can get a simple triangular relationship,

Therefore, the depth can be expressed as:

Among them, x _r -x ₁ is defined as the parallax. When the position of the camera is placed in parallel, the depth Z is inversely proportional to the parallax x _r -x ₁ , and fT is a definite constant.

Then the three-dimensional coordinates of the point can be obtained as follows:

Step 5: Register the 3D point clouds of the adjacent images captured by the left camera, unify the two 3D point clouds into one coordinate system, and obtain the 3D point cloud model of the entire target surface. The specific implementation steps are as follows:

(1): Use the KD-tree method to search the K neighborhood of each point in the point cloud, and establish the local neighborhood of the point;

(2): Estimate the normal vector of the point by the principal component analysis method, extract the feature point with the normal vector variation as the standard, select a variety of features to describe the feature point according to the geometric relationship between the feature point and the adjacent point, and use the histogram Statistical properties of the graph Statistical feature points and various features of points in the K neighborhood of feature points, establish histogram feature descriptions for them, and obtain feature vectors. The distance between the feature vectors between two points is used as the matching basis, and the features between two points The minimum vector distance is the matching point pair.

(3): According to the matching point pairs, the conversion relationship of the two point cloud coordinate systems can be calculated to complete the initial registration. The calculation process is as follows:

S31: Calculate the center of gravity of the source point set P and the target point set Q respectively:

in, are the three-dimensional coordinates of the center of gravity of the source point set P and the target point set _Q ; N _P , N _Q are the number of points in the source point set P and the target point set _Q ; The three-dimensional coordinates of the i-th point in the target point set Q.

S32: Construct a covariance matrix according to the data point sets P and Q:

S33: Construct a 4×4 symmetric matrix according to the covariance matrix:

Wherein, I ₃ is the third-order identity matrix, tr (ΣP, Q) is the trace of matrix ΣP, Q,

Δ=[A ₂₃ A ₃₁ A ₁₂ ] ^T ; A _i,j =(∑P, Q-∑P, Q ^T ) _ij ;

S34: Obtain the eigenvalue and eigenvector of Q(∑P, Q), the eigenvector of the largest eigenvalue is the required rotation vector q _R =[q ₀ q ₁ q ₃ ] ^T ;

S35: According to the rotation vector q _R obtained by the above formula, the rotation matrix R can be obtained, so that the best translation vector can be obtained, and the formula is:

(4): According to the initial position provided by the initial registration, the iterative closest point algorithm is used to iterate the adjacent point cloud coordinate system, and finally the accurate conversion relationship of the adjacent point cloud coordinate system can be obtained, so that the adjacent point cloud coordinate system into a coordinate system.

The above is only a preferred embodiment of the present invention, it should be pointed out that for those of ordinary skill in the art, without departing from the principle of the present invention, some improvements and modifications can also be made, and these improvements and modifications should also be It is regarded as the protection scope of the present invention.

Claims (10)

1. The utility model provides a binocular vision target surface 3D detecting system based on unmanned aerial vehicle which characterized in that: the method comprises the following steps:

the unmanned aerial vehicle is used for sequentially acquiring image data of the target surface from bottom to top and from left to right through a binocular vision system mounted at the bottom of the unmanned aerial vehicle;

the computer system is used for processing the images acquired by the binocular vision system and outputting a detection result;

the remote control equipment controls the flight path of the unmanned aerial vehicle and the start and stop of the binocular vision system through the radio transmission module;

the unmanned aerial vehicle comprises

The power supply battery pack is used for providing power guarantee for the unmanned aerial vehicle communication equipment, the image acquisition equipment and the power equipment;

an airborne control unit for receiving the instruction of the ground control center,

the navigation attitude measurement system is used for measuring the attitude of the unmanned aerial vehicle in real time and recording the attitude information at each moment;

the wifi/radio transmission module is used for realizing wireless communication with the remote control equipment and receiving a control instruction sent by the remote control equipment;

binocular vision system together fixed with unmanned aerial vehicle, the camera sets up towards the target surface for carry out the collection of target image.

2. The binocular vision target surface 3D detection system based on unmanned aerial vehicles of claim 1, wherein: the binocular vision system includes:

the pose measuring device is used for measuring the pose of the binocular vision system;

the left camera and the right camera are used for acquiring image data of a target;

and the memory card is used for respectively storing the image data collected by the left camera and the right camera.

3. The binocular vision target surface 3D detection system based on the unmanned aerial vehicle described in claim 1, characterized in that: the left camera and the right camera of the binocular vision system are arranged in parallel and have the same parameters.

4. A binocular vision target surface 3D detection method based on an unmanned aerial vehicle is characterized in that: the method comprises the following steps:

step 1: controlling the unmanned aerial vehicle to acquire images according to a preset track through the remote control equipment, and importing the images in the storage card into the computer for processing after the images are acquired;

step 2: numbering images acquired by an airborne binocular vision system according to a time sequence to form a binocular left and right image sequence pair;

and step 3: calibrating the camera to obtain internal and external parameters of the camera, including the focal length f of the camera, the baseline distance T and the like;

and 4, step 4: denoising the acquired image, adjusting the pose of the image and the like;

and 5: extracting characteristic points of a left image and a right image shot at the same moment, matching the characteristic points to obtain parallax information, and calculating a three-dimensional coordinate of a midpoint of the images to obtain a local three-dimensional point cloud;

step 6: registering three-dimensional point clouds of adjacent images shot by a left camera, and unifying the two three-dimensional point clouds into a coordinate system to obtain a three-dimensional point cloud model of the whole target surface;

and 7: and comparing the obtained three-dimensional point cloud model with surface delivery data given by large equipment to detect whether the difference exists.

5. The binocular vision target surface 3D detection method based on the unmanned aerial vehicle is characterized in that: and in the step 4, the pose information of the reference unmanned aerial vehicle corrects the image pose by affine transformation.

6. The binocular vision target surface 3D detection method based on the unmanned aerial vehicle is characterized in that:

the step 5 specifically comprises the following steps:

s51, extracting surf feature points, taking a 4 × 4 rectangular region block around the feature points, counting haar wavelet features of 25 pixels in the horizontal direction and the vertical direction in each subregion, taking the 4 haar wavelet features in 4 directions behind a horizontal direction value, a vertical direction value, a horizontal direction absolute value and a vertical direction absolute value as feature vectors of each subblock region, determining the matching degree by calculating the Euclidean distance between the two feature points, wherein the shorter the Euclidean distance is, the better the matching degree of the two feature points is, and the parallax information can be obtained by the matched feature points;

s52, according to parallaxInformation d ═ x_r-x₁And calculating the three-dimensional coordinates of the points according to the following formula by using the internal and external parameters of the camera:

7. the binocular vision target surface 3D detection method based on the unmanned aerial vehicle is characterized in that: the three-dimensional point cloud registration method in the step 6 can be divided into the following steps:

s1, establishing a K neighborhood for each point in the adjacent three-dimensional point cloud coordinate system;

s2, extracting feature points, establishing feature description for each feature point, and matching adjacent three-dimensional point cloud feature points;

s3, obtaining a conversion relation of two point cloud coordinate systems by calculation according to the matching point pairs, and finishing initial registration;

and S4, iterating the adjacent point cloud coordinate systems by using an iterative closest point algorithm according to the initial position provided by the initial registration, and finally obtaining the accurate conversion relation of the adjacent point cloud coordinate systems so that the adjacent point cloud coordinate systems are unified into one coordinate system.

8. The binocular vision target surface 3D detection method based on the unmanned aerial vehicle is characterized in that: in the step S1, a KD-tree method is adopted to accelerate the K neighborhood of the search point.

9. The binocular vision target surface 3D detection method based on the unmanned aerial vehicle is characterized in that: in step S2, the normal vector of the point is estimated by a principal component analysis method, the feature point is extracted using the normal vector variation as a standard, the feature point is described by selecting a plurality of features according to the geometric relationship between the feature point and the neighboring point, the histogram feature description is established by using the statistical characteristics of the histogram to count the feature point and the plurality of features of the K-neighborhood inner points of the feature point, the feature vector is obtained, the distance between the feature vectors of the two points is used as the matching basis, and the minimum distance between the feature vectors of the two points is the matching point pair.

10. The binocular vision target surface 3D detection method based on the unmanned aerial vehicle is characterized in that: in the step S3, a quaternion method is used to obtain a transformation matrix between adjacent point cloud coordinate systems, and the implementation steps are as follows:

s31: calculating the gravity centers of the source point set P and the target point set Q respectively:

wherein,the gravity three-dimensional coordinates of the source point set P and the target point set Q are respectively; n is a radical of_P，N_QThe number of points in the source point set P and the target point set Q respectively; p_i，Q_iThree-dimensional coordinates of the ith point in the source point set P and the target point set Q respectively.

S32: and constructing a covariance matrix according to the data point sets P and Q:

s33: a 4 × 4 symmetric matrix is constructed from the covariance matrix:

wherein, I₃Is a third order identity matrix, tr (Σ P, Q) is the trace of the matrix Σ P, Q,

Δ＝[A₂₃A₃₁A₁₂]^T；

s34: determining the characteristic value and characteristic of Q (SIG P, Q)Vector, the feature vector of the maximum feature value is the required rotation vector q_R＝[q₀q₁q₃]^T；

S35: the rotation vector q obtained from the above formula_RThe rotation matrix R can be solved, so that the optimal translation vector can be obtained, and the formula is:

step S4 is to coordinate the two adjacent point clouds by X_i-1And X_iUnification into the same coordinate system can be achieved by the following formula: x_i-1＝X_iR+T；

Where R is a rotation matrix and T is a translation matrix.