CN108171787A - A kind of three-dimensional rebuilding method based on the detection of ORB features - Google Patents

Abstract

The invention discloses a three-dimensional reconstruction method based on ORB feature detection, which belongs to the technical field of feature detection. Real-time images and preprocessing are obtained through a binocular camera installed on the platform of an unmanned aerial vehicle, and the camera is calibrated to solve the problem of inside and outside of the camera. Parameters and distortion coefficients are used to correct pixel coordinates, ORB feature detection is used to detect image feature points, FLANN is used to match, and the spatial coordinates of feature points are obtained, and OpenGL is used to perform three-dimensional reconstruction of spatial discrete points. The invention can speed up the execution speed of the system, has rotation invariance and robustness against noise interference, and greatly improves the precision of real-time three-dimensional reconstruction.

Description

A 3D Reconstruction Method Based on ORB Feature Detection

technical field

The invention relates to the technical field of feature detection, in particular to a three-dimensional reconstruction method based on ORB feature detection.

Background technique

Image feature point extraction and detection is a key part of 3D reconstruction technology, which affects the accuracy of feature point matching and the final reconstruction result. With the popularization and development of drones, autonomous obstacle avoidance and path planning of drones, and the use of drones to collect digital image information for real-time 3D reconstruction have attracted more and more attention and attention. At present, image feature point detection and matching algorithms in 3D reconstruction technology include matching based on grayscale, matching based on features and matching based on local invariant descriptors, among which image matching based on local invariant descriptors includes SIFT feature descriptor, SURF Feature descriptors, D-nets feature descriptors, etc. When the UAV is shooting pictures, the pictures captured are easily interfered by external light and itself, resulting in image noise. This has a great influence on the result of 3D reconstruction.

Contents of the invention

Aiming at the deficiencies of the prior art, the problem solved by the present invention is the problem of low real-time three-dimensional reconstruction accuracy.

In order to solve the above-mentioned technical problems, the technical solution adopted in the present invention is a three-dimensional reconstruction method based on ORB feature detection, which obtains real-time images and preprocessing through the binocular camera installed on the UAV platform, calibrates the camera, and solves the problem of The internal and external parameters of the camera and the distortion coefficient are used to correct the pixel coordinates, the ORB feature detection is used to detect the image feature points, the FLANN is used to match, the spatial coordinates of the feature points are obtained, and the OpenGL is used to perform three-dimensional reconstruction of the spatial discrete points, including the following steps:

Step 1: Carry the binocular camera system on the UAV platform, set up the UAV ground base station system, and transmit images in real time;

Step 2: preprocessing the acquired image, including Gaussian filtering, histogram equalization;

Step 3: Calibrate the binocular camera, solve the internal and external parameters of the camera and the camera distortion coefficient;

Step 4: Correct the pixel coordinates through the camera distortion coefficient to prepare for the detection and matching of image feature points;

Step 5: Use the ORB algorithm to perform feature detection on the image, and cooperate with FLANN to perform feature point matching;

Step 6: Use epipolar geometric constraints to eliminate mismatching points, and use RANSAC algorithm to optimize matching;

Step 7: Solve the spatial coordinates of the image feature points through the spatial three-dimensional coordinate calculation formula for the matched point coordinates to obtain the point cloud model, as follows:

According to the three-dimensional space coordinate formula (x ₁ ,y ₁ ,z ₁ ),

In the formula (u ₁ , v ₁ ) and (u ₂ , v ₂ ) are the coordinates of two points on the left and right image coordinate system of the binocular camera, and b is the baseline length;

Step 8: Use OpenGL to perform 3D reconstruction of the spatially discrete point cloud of the point cloud model.

In the step 3, the following sub-steps are included:

1) Using the Zhang Zhengyou calibration method to establish the parameter matrix to be calibrated according to the corresponding relationship between the features, including the internal parameter matrix of the left and right cameras, the distortion coefficient matrix and the relative relationship matrix of the left and right cameras;

2) Solve the homography matrix:

Homography matrix:

When there are more than four planar targets, H can be solved:

λ[h ₁ h ₂ h ₃ ]=H=M ₁ [r ₁ r ₂ t];

Among them, M is the projection matrix, M ₁ is the camera internal parameter matrix, and M ₂ is the camera external parameter matrix;

According to the rotation matrix constraints, any two rotation matrices can be vertically obtained:

3) Solve the internal parameters of the camera:

Let B=b _ij ＝M ₁ ^-T M ₁ ^-1 be a symmetrical array;

Define parameter variable b=[b ₁₁ ,b ₁₂ ,b ₁₃ ,b ₂₂ ,b ₂₃ ,b ₃₃ ] ^T , then h _i ^T Bh _j =V _ij ^T b;

in

Have: When n≥3, 6 internal parameters can be obtained, and b*=arg min||Vb|| can be solved;

Further _each parameter of M can be determined;

4) According to M _1, the extrinsic parameters of the camera can be further calculated, that is, R and T of M ₂ ,

Coefficient λ=1/||M ₁ ^-1 h ₁ ||=1/||M ₁ ^-1 h ₂ ||, r ₁ = λM ₁ ^-1 h ₁ , r ₂ = λM ₁ ^-1 h ₂ , r ₃ = r ₁ ×r ₂ ,

t=λM ₁ ^-1 h ₃ ;

5) Further solve the radial distortion parameters k1, k2.

In the step 5, the following sub-steps are included:

1) The ORB algorithm uses the FAST algorithm to extract feature points, uses Harris corner detection to sort the feature points, and takes N better corner points as feature points;

ORB uses the "intensity centroid" method to determine the direction of feature points, regards the range of feature points as a patch, finds the centroid of the patch, connects the centroid of the patch with the feature points, and finds the line and the abscissa axis The included angle is the direction of the feature point;

The centroid formula of the patch is as follows:

In the formula, M _pq is the gray moment, I(x, y) is the gray value at the point (x, y) of the image, p, q are the order of the gray moment;

2) The centroid is defined as:

M ₀₁ , M ₁₀ , and M ₀₀ are three different first-order gray moments;

3) Find the direction of OC, keep the range of x and y within the patch, take the feature point as the coordinate origin, and obtain the direction angle as;

4) The ORB algorithm solves the rotation invariance and anti-noise ability, adopts the matching points x _i , y _i in the patch field, sets the n test points described by the generated feature points as (xi _, y _i ), and defines a 2* The matrix of n is as follows:

Using the rotation matrix R, obtain the matched coordinates S _θ = RS after rotation,

5) Use the ORB algorithm to detect and extract feature points, and use the improved Brief descriptor to generate feature point descriptors;

6) Use the FLANN algorithm for feature point matching.

In step 6, the following sub-steps are included:

1) After the distortion-corrected camera, the matching point is on the epipolar line on the image plane;

2) Calculate the difference between |y _l -y _r | in P _l (x _l , y _l ) and P _r (x _r , y _r ), and set the threshold. The smaller the threshold, the higher the precision requirement;

3) Eliminate matching points beyond the threshold range, and the correct matching points are generally near the epipolar line;

4) Select 4 groups of matching point pairs from the remaining matching points for further screening and optimization, and calculate the homography matrix H as a model;

5) Calculate the projection error between the data set and the model, if it is less than the threshold, add it as the interior point set I;

6) Determine whether it is the optimal solution according to the number of interior points, if so, update the interior point set I, and update the number of iterations K at the same time;

7) If the number of iterations is greater than K, then exit; otherwise K+1, repeat the above steps;

Among them, p is the confidence level, generally 0.995; w is the proportion of internal points; m is the number of samples required for the calculation model 4;

After the initial screening of epipolar constraints, the proportion w of interior points is relatively high, thereby reducing the number of iterations and reducing the operation time.

In step 8, the following sub-steps are included:

1) Utilize the OpenGL function to draw the scene in the computer to complete the drawing of the three-dimensional graphics;

2) Call the OpenGL function to draw, and finally complete the three-dimensional reconstruction.

The technical solution of the present invention uses the binocular camera mounted on the UAV platform to acquire and transmit images, and adopts ORB image feature point detection and FLANN feature point matching, which can speed up the execution speed of the system, and at the same time has rotation invariance and anti-noise interference The robustness greatly improves the accuracy of real-time 3D reconstruction.

Description of drawings

Fig. 1 is a schematic diagram of the structure of the present invention.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings, but the present invention is not limited thereto.

Figure 1 shows a 3D reconstruction method based on ORB feature detection. Real-time images are obtained and preprocessed by a binocular camera installed on the UAV platform, and the camera is calibrated to solve the camera internal and external parameters and distortion coefficients. Pixel coordinate correction, use ORB feature detection to detect image feature points, use FLANN to match, find the spatial coordinates of feature points, use OpenGL to perform three-dimensional reconstruction of spatial discrete points, including the following steps:

Step 1: Carry the binocular camera system on the UAV platform, set up the UAV ground base station system, and transmit images in real time;

Step 2: preprocessing the acquired image, including Gaussian filtering, histogram equalization;

Gaussian filters are very effective at suppressing normally distributed noise. Generally, a two-dimensional Gaussian filter function is used as a smoothing filter:

Step 3: Calibrate the binocular camera, solve the internal and external parameters of the camera and the camera distortion coefficient;

Step 4: Correct the pixel coordinates through the camera distortion coefficient to prepare for the detection and matching of image feature points;

Step 5: Use the ORB algorithm to perform feature detection on the image, and cooperate with FLANN to perform feature point matching;

Step 6: Use epipolar geometric constraints to eliminate mismatching points, and use RANSAC algorithm to optimize matching;

Step 7: Solve the spatial coordinates of the image feature points through the spatial three-dimensional coordinate calculation formula for the matched point coordinates to obtain the point cloud model, as follows:

According to the three-dimensional space coordinate formula (x ₁ ,y ₁ ,z ₁ ),

In the formula (u ₁ , v ₁ ) and (u ₂ , v ₂ ) are the coordinates of two points on the left and right image coordinate system of the binocular camera, and b is the baseline length;

Step 8: Use OpenGL to perform 3D reconstruction of the spatially discrete point cloud of the point cloud model.

In the step 3, the following sub-steps are included:

1) Using the Zhang Zhengyou calibration method to establish the parameter matrix to be calibrated according to the corresponding relationship between the features, including the internal parameter matrix of the left and right cameras, the distortion coefficient matrix and the relative relationship matrix of the left and right cameras;

2) Solve the homography matrix:

Homography matrix:

When there are more than four planar targets, H can be solved:

λ[h ₁ h ₂ h ₃ ]=H=M ₁ [r ₁ r ₂ t];

Among them, M is the projection matrix, M ₁ is the camera internal parameter matrix, and M ₂ is the camera external parameter matrix;

According to the rotation matrix constraints, any two rotation matrices can be vertically obtained:

3) Solve the internal parameters of the camera:

Let B=b _ij ＝M ₁ ^-T M ₁ ^-1 be a symmetrical array;

Define parameter variable b=[b ₁₁ ,b ₁₂ ,b ₁₃ ,b ₂₂ ,b ₂₃ ,b ₃₃ ] ^T , then h _i ^T Bh _j =V _ij ^T b;

in

Have: When n≥3, 6 internal parameters can be obtained, and b*=arg min||Vb|| can be solved;

Further _each parameter of M can be determined;

4) According to M _1, the extrinsic parameters of the camera can be further calculated, that is, R and T of M ₂ ,

Coefficient λ=1/||M ₁ ^-1 h ₁ ||=1/||M ₁ ^-1 h ₂ ||, r ₁ = λM ₁ ^-1 h ₁ , r ₂ = λM ₁ ^-1 h ₂ , r ₃ = r ₁ ×r ₂ ,

t=λM ₁ ^-1 h ₃ ;

5) Further solve the radial distortion parameters k1, k2.

In the step 5, the following sub-steps are included:

1) The ORB algorithm uses the FAST algorithm to extract feature points, uses Harris corner detection to sort the feature points, and takes N better corner points as feature points;

ORB uses the "intensity centroid" method to determine the direction of feature points, regards the range of feature points as a patch, finds the centroid of the patch, connects the centroid of the patch with the feature points, and finds the line and the abscissa axis The included angle is the direction of the feature point;

The centroid formula of the patch is as follows:

In the formula, M _pq is the gray moment, I(x, y) is the gray value at the point (x, y) of the image, p, q are the order of the gray moment;

2) The centroid is defined as:

M ₀₁ , M ₁₀ , and M ₀₀ are three different first-order gray moments;

3) Find the direction of OC, keep the range of x and y within the patch, take the feature point as the coordinate origin, and obtain the direction angle as;

4) The ORB algorithm solves the rotation invariance and anti-noise ability, adopts the matching points x _i , y _i in the patch field, sets the n test points described by the generated feature points as (xi _, y _i ), and defines a 2* The matrix of n is as follows:

Using the rotation matrix R, obtain the matched coordinates S _θ = RS after rotation,

5) Use the ORB algorithm to detect and extract feature points, and use the improved Brief descriptor to generate feature point descriptors;

6) Use the FLANN algorithm for feature point matching.

In step 6, the following sub-steps are included:

1) After the distortion-corrected camera, the matching point is on the epipolar line on the image plane;

2) Calculate the difference between |y _l -y _r | in P _l (x _l , y _l ) and P _r (x _r , y _r ), and set the threshold. The smaller the threshold, the higher the precision requirement;

3) Eliminate matching points beyond the threshold range, and the correct matching points are generally near the epipolar line;

4) Select 4 groups of matching point pairs from the remaining matching points for further screening and optimization, and calculate the homography matrix H as a model;

5) Calculate the projection error between the data set and the model, if it is less than the threshold, add it as the interior point set I;

6) Determine whether it is the optimal solution according to the number of interior points, if so, update the interior point set I, and update the number of iterations K at the same time;

7) If the number of iterations is greater than K, then exit; otherwise K+1, repeat the above steps;

Among them, p is the confidence level, generally 0.995; w is the proportion of internal points; m is the number of samples required for the calculation model 4;

After the initial screening of epipolar constraints, the proportion w of interior points is relatively high, thereby reducing the number of iterations and reducing the operation time.

In said step 8, the following sub-steps are included:

1) Utilize the OpenGL function to draw the scene in the computer to complete the drawing of the three-dimensional graphics;

2) Call the OpenGL function to draw, and finally complete the three-dimensional reconstruction.

The beneficial effect of adopting the technical solution of the present invention can meet the requirements of the three-dimensional reconstruction of the binocular vision system equipped with the drone, and ensure accuracy and stability.

The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, without departing from the principle and spirit of the present invention, various changes, modifications, replacements and modifications to these embodiments still fall within the protection scope of the present invention.

Claims (6)

1. A 3D reconstruction method based on ORB feature detection, which obtains real-time images and preprocesses them through the binocular camera installed on the UAV platform, calibrates the camera, and solves the internal and external parameters of the camera and the distortion coefficient to correct the pixel coordinates , use ORB feature detection to detect image feature points, use FLANN to match, find the spatial coordinates of feature points, utilize OpenGL to carry out three-dimensional reconstruction to spatial discrete points, it is characterized in that, comprising the following steps: Step 1: Carry the binocular camera system on the UAV platform, set up the UAV ground base station system, and transmit images in real time; Step 2: preprocessing the acquired image, including Gaussian filtering, histogram equalization; Step 3: Calibrate the binocular camera, solve the internal and external parameters of the camera and the camera distortion coefficient; Step 4: Correct the pixel coordinates through the camera distortion coefficient to prepare for the detection and matching of image feature points; Step 5: Use the ORB algorithm to perform feature detection on the image, and cooperate with FLANN to perform feature point matching; Step 6: Use epipolar geometric constraints to eliminate mismatching points, and use RANSAC algorithm to optimize matching; Step 7: Solve the spatial coordinates of the image feature points through the spatial three-dimensional coordinate calculation formula for the matched point coordinates to obtain the point cloud model; Step 8: Use OpenGL to perform 3D reconstruction of the spatially discrete point cloud of the point cloud model. 2. the three-dimensional reconstruction method based on ORB feature detection according to claim 1, is characterized in that, in described step 3, comprises following sub-step: 1) Using the Zhang Zhengyou calibration method to establish the parameter matrix to be calibrated according to the corresponding relationship between the features, including the internal parameter matrix of the left and right cameras, the distortion coefficient matrix and the relative relationship matrix of the left and right cameras; 2) Solve the homography matrix: Homography matrix: When there are more than four planar targets, H can be solved: λ[h ₁ h ₂ h ₃ ]=H=M ₁ [r ₁ r ₂ t]; Among them, M is the projection matrix, M ₁ is the camera internal parameter matrix, and M ₂ is the camera external parameter matrix; According to the rotation matrix constraints, any two rotation matrices can be vertically obtained: 3) Solve the internal parameters of the camera: Let B=b _ij ＝M ₁ ^-T M ₁ ^-1 be a symmetrical array; Define parameter variable b=[b ₁₁ ,b ₁₂ ,b ₁₃ ,b ₂₂ ,b ₂₃ ,b ₃₃ ] ^T , then h _i ^T Bh _j =V _ij ^T b; in Have: When n≥3, 6 internal parameters can be obtained, and b*=arg min||Vb|| can be solved; Further _each parameter of M can be determined; 4) According to M _1, the extrinsic parameters of the camera can be further calculated, that is, R and T of M ₂ , Coefficient λ=1/||M ₁ ^-1 h ₁ ||=1/||M ₁ ^-1 h ₂ ||, r ₁ = λM ₁ ^-1 h ₁ , r ₂ = λM ₁ ^-1 h ₂ , r ₃ = r ₁ ×r ₂ , t=λM ₁ ^-1 h ₃ ; 5) Further solve the radial distortion parameters k1, k2. 3. the three-dimensional reconstruction method based on ORB feature detection according to claim 1, is characterized in that, in described step 5, comprises following sub-step: 1) The ORB algorithm uses the FAST algorithm to extract feature points, uses Harris corner detection to sort the feature points, and takes N better corner points as feature points; ORB uses the "intensity centroid" method to determine the direction of feature points, regards the range of feature points as a patch, finds the centroid of the patch, connects the centroid of the patch with the feature points, and finds the line and the abscissa axis The included angle is the direction of the feature point; The centroid formula of the patch is as follows: In the formula, M _pq is the gray moment, I(x, y) is the gray value at the point (x, y) of the image, p, q are the order of the gray moment; 2) The centroid is defined as: M ₀₁ , M ₁₀ , and M ₀₀ are three different first-order gray moments; 3) Find the direction of OC, keep the range of x and y within the patch, take the feature point as the coordinate origin, and obtain the direction angle as; 4) The ORB algorithm solves the rotation invariance and anti-noise ability, adopts the matching points x _i , y _i in the patch field, sets the n test points described by the generated feature points as (xi _, y _i ), and defines a 2* The matrix of n is as follows: Using the rotation matrix R, obtain the matched coordinates S _θ = RS after rotation, 5) Use the ORB algorithm to detect and extract feature points, and use the improved Brief descriptor to generate feature point descriptors; 6) Use the FLANN algorithm for feature point matching. 4. the three-dimensional reconstruction method based on ORB feature detection according to claim 1, is characterized in that, in described step 6, comprises following sub-step: 1) After the distortion-corrected camera, the matching point is on the epipolar line on the image plane; 2) Calculate the difference between |y _l -y _r | in P _l (x _l , y _l ) and P _r (x _r , y _r ), and set the threshold. The smaller the threshold, the higher the precision requirement; 3) Eliminate matching points beyond the threshold range, and the correct matching points are generally near the epipolar line; 4) Select 4 groups of matching point pairs from the remaining matching points for further screening and optimization, and calculate the homography matrix H as a model; 5) Calculate the projection error between the data set and the model, if it is less than the threshold, add it as the interior point set I; 6) Determine whether it is the optimal solution according to the number of interior points, if so, update the interior point set I, and update the number of iterations K at the same time; 7) If the number of iterations is greater than K, then exit; otherwise K+1, repeat the above steps; Among them, p is the confidence level, generally 0.995; w is the proportion of internal points; m is the number of samples required for the calculation model 4; After the initial screening of epipolar constraints, the proportion w of interior points is relatively high, thereby reducing the number of iterations and reducing the operation time. 5. the three-dimensional reconstruction method based on ORB feature detection according to claim 1, is characterized in that, in described step 7, specifically as follows: According to the three-dimensional space coordinate formula (x ₁ ,y ₁ ,z ₁ ), In the formula, (u ₁ , v ₁ ) and (u ₂ , v ₂ ) are the coordinates of two points on the left and right image coordinate system of the binocular camera, and b is the length of the baseline. 6. the three-dimensional reconstruction method based on ORB feature detection according to claim 1, is characterized in that, in described step 8, comprises following sub-step: 1) Utilize the OpenGL function to draw the scene in the computer to complete the drawing of the three-dimensional graphics; 2) Call the OpenGL function to draw, and finally complete the three-dimensional reconstruction.