CN105184857B - Monocular vision based on structure light ranging rebuilds mesoscale factor determination method - Google Patents

Abstract

A method for determining the scale factor in monocular vision reconstruction based on point structured light ranging, the method includes spot centroid center positioning, spatial straight line fitting, RANSAC exclusion, calculating the three-dimensional space point coordinates of the spot, and calculating the scale factor. Aiming at the traditional three-dimensional reconstruction method of monocular vision based on image sequence, most of them can only realize three-dimensional reconstruction in projective scale or affine scale. , so that the scale of the reconstructed 3D scene using the image sequence is consistent with that of the real world scene. The technical features of the present invention are as follows: (1) monocular reconstruction introduces structured light active vision to realize Euclidean three-dimensional reconstruction, (2) center-of-mass method spot positioning, (3) laser ray equation fitting with RANSAC elimination mechanism, (4) reflection Projection-optimized point positioning of spot space, (5) Euclidean reconstruction of various methods of monocular reconstruction is not limited.

Description

Scale Factor Determination Method for Monocular Vision Reconstruction Based on Point Structured Light Ranging

technical field

The invention belongs to the field of computer vision and relates to a method for realizing Euclidean three-dimensional reconstruction based on image sequence by using point structured light.

Background technique

Vision is the most important means of human perception, about 80% of the external information is received by people through the eyes. It is precisely because of the importance of vision to human beings that with the rapid development of digital computers, it has become a very attractive research topic to make computers have vision and be able to process visual information. In this way, it leads to the emergence and development of the subject of computer vision.

A very important part of computer vision research is to analyze and process the obtained dynamic image sequence to obtain useful information. Dynamic images are aimed at moving objects or scenes, they are not only a function of spatial position, but also change with time, it provides us with richer information than a single image. In the image sequence captured by a certain scene, the grayscale and color of at least a part of the pixels between two adjacent frames of images have changed, and this image sequence is called a dynamic image sequence.

In recent years, with the rapid development of research in the field of computer vision, the problem of using two-dimensional information in image sequences to restore the three-dimensional structure of the real world is also one of the hot issues and one of the important research directions. Researchers in related fields both domestically and internationally have proposed some effective solutions to these problems. Among them, the monocular vision 3D reconstruction method based on image sequences has become the most important method to solve this problem due to its advantages of less reconstruction constraints, less predictable information required for reconstruction, and suitable for reconstruction of large-scale scenes. method. An important theoretical model in multi-view geometry is the hierarchy theory, which defines a hierarchy from the real scene to its reconstruction model. The transformations involved in the hierarchy theory mainly include projective transformation, affine transformation, similarity transformation and Euclidean transformation. . However, most of the traditional monocular vision 3D reconstruction methods based on image sequences can only achieve 3D reconstruction at the projective scale or affine scale, that is, the reconstructed result differs from the scene scale of the real world by a scale factor. application in reality. In view of this, the present invention proposes a Euclidean three-dimensional reconstruction method using point structured light as an auxiliary to realize monocular vision, so that the scale of the three-dimensional scene reconstructed by using image sequences is consistent with that of the real world scene, and at the same time improves the Robustness and accuracy of 3D reconstruction algorithms.

According to the beam pattern projected by the optical projector, the structured light mode can be divided into point structured light mode, line structured light mode, multi-line structured light mode and grid structured light mode, etc. The three-dimensional scene obtained by simply using monocular vision cannot achieve the real reconstruction effect of the Euclidean three-dimensional scene. The present invention uses point structured light and a monocular camera to realize the Euclidean three-dimensional reconstruction. The beam emitted by the laser is projected onto the surface of the object to generate a light spot, which is imaged on the image plane of the camera through the lens of the camera to form a two-dimensional image point. The line of sight of the camera and the beam line intersect at the light point in space, and the spatial position of the light point in a known world coordinate system can be uniquely determined by it, and then the spatial scale factor can be obtained to achieve the effect of European 3D reconstruction.

The present invention realizes real-scale Euclidean three-dimensional reconstruction of real scenes by designing monocular stereo vision aided by point structured light. The principle diagram is shown in Figure 1. First, fix the target (8×11 black and white checkerboard) and the laser, place the target within the field of view of the camera, take a picture (as shown in the position of target 1 in the figure), then turn on the laser to hit the laser on the target and then Take a picture (as shown in the position of target 2 in the figure), move the target and repeat the above operation several times to obtain multiple images (target 3, 4, 5, etc.), and preprocess the input image. Then, the image frame difference method is used to obtain the laser point spot, and the centroid method is used to obtain the coordinates of the center of mass of the spot in the image. According to the homography between the target and the image, the three-dimensional coordinates of all the spots in the camera coordinate system are obtained, and the ray equation l ₁ emitted by the laser is fitted according to the three-dimensional coordinates of the multiple spots in the space. Finally, the light emitted from the optical center of the camera The intersection point of the ray l ₂ passing through the center of the spot and the ray projected by the laser (due to the existence of errors, l ₁ and l ₂ will become straight lines of different planes, and the midpoint of the common vertical line of the straight lines of different planes is used in practical applications) is the real spot space 3D points, so that the size of the scale factor can be obtained.

Contents of the invention

Starting from point structured light, the present invention proposes a monocular Euclidean 3D reconstruction method assisted by point structured light, including spot centroid center positioning, spatial straight line fitting with RANSAC rejection, and calculation of spot coordinates in 3D space , Finding the scale factor.

The technical solution adopted in the present invention is a method for determining the scale factor in monocular vision reconstruction based on point structured light ranging, and the overall flow chart for calculating the scale factor is shown in Figure 2. The whole calibration is unified to the camera coordinate system. The internal parameters of the camera are calibrated using a camera calibration method based on a 2D plane target. When calibrating the parameters of the structured light system, the spatial positions of the camera and laser are kept unchanged. Then project the laser spot on the plane where the checkerboard target is located, take the image at this time, then turn off the laser and take the image at this time, repeat this step to place the target any number of times to obtain a set of data images, according to The background difference method can obtain the spot information corresponding to the target at each position, as shown in Figure 3. Using the corner point information on the target to obtain the external parameter matrix of the target plane each time, plus the coordinate information of the center of mass of the laser spot in each calibration image, the spatial coordinates of the center of mass of the laser spot are obtained each time the target is placed. In this way, the result of Levenberg-Marquardt fitting of the coordinates of the center of mass of the laser spot obtained by multiple placements can be used to obtain the spatial equation of the laser light. The spatial coordinates of the laser spot on the image plane are obtained through the calibrated sensor internal parameter information. Theoretically, the laser light and the camera optical center and the image point coordinate line of the laser spot on the image will completely coincide. However, due to calibration and calculation errors, these two straight lines will form a straight line of different planes. The midpoint of the common vertical line is the real space coordinate of the laser spot, and the real scale factor is the ratio of the real space coordinate of the spot obtained by using the point structure cursor to the space coordinate of the spot obtained by using the three-dimensional reconstruction algorithm, thereby achieving the Euclidean reconstruction effect.

(1) Centroid method spot positioning

The spot image is a relatively common image in existing image processing, and the spot centroid is one of the important features of the spot image. In many fields such as visual measurement and satellite navigation, how to realize the rapid and accurate positioning of the spot center is an important topic of research at home and abroad.

The center localization methods of point-like spots can be divided into two categories based on grayscale and edge-based. The grayscale-based method generally uses the grayscale distribution information of the target, and is suitable for spots with a small radius and uniform grayscale distribution; the edge-based method generally uses the edge shape information of the target, and is suitable for spots with a large radius. Therefore, small-sized spots are usually centered using grayscale-based methods. At present, there are three commonly used gray-based center positioning methods: the centroid method, the Hessian matrix method, and the Gaussian fitting method. The centroid method is the most commonly used subdivision positioning method. It is relatively simple to implement, fast in operation speed, and has a certain positioning accuracy. Therefore, the centroid spot positioning method is used. The specific implementation method will be described in detail below.

(2) Laser ray equation fitting with RANSAC elimination mechanism

Multiple sets of centroid spot coordinates can be obtained through the above steps, and then the ray equation of the spatial point laser can be fitted. The present invention utilizes the space straight line fitting function in OpenCV to obtain the ray equation more accurately and conveniently. However, due to errors in steps such as obtaining the coordinates of the center of mass, there may be a large error in the coordinates of the center of mass of the obtained spot, which will cause a large error between the fitted space linear equation and the real linear equation. In order to reduce the error To make the obtained spatial linear equation as accurate as possible, the present invention proposes to use the RANSAC exclusion method to eliminate spatial points with large errors. The result is shown in Figure 4, which will be described in detail in the specific implementation.

(3) Spatial point positioning of spot optimized by back projection

After obtaining the laser ray equation according to the above steps, the line of sight of the camera and the beam line intersect at the light point in space. They cannot be completely intersected at one point, so the intersection location problem is to find the coordinates of the midpoint of the common perpendicular segment of straight lines on different planes. In the present invention, the three-dimensional point in space obtained by the algorithm of the midpoint of the common vertical segment of straight lines on different planes is used as the three-dimensional coordinates of the space point corresponding to the image spot. This paper further proposes to use the back-projection iterative method to optimize the obtained three-dimensional point coordinates in space, and the specific implementation method It will be explained in detail below.

Compared with the prior art, the present invention has the following beneficial effects

(1) Monocular reconstruction introduces structured light active vision to realize Euclidean 3D reconstruction

The present invention is not traditional point structured light reconstruction, but on the basis of using monocular vision to realize three-dimensional reconstruction, the real scale factor is introduced through point structured light, so as to realize real scale Euclidean three-dimensional reconstruction. First of all, the monocular 3D reconstruction algorithm based on optical flow feedback is used to achieve fast and accurate three-dimensional modeling of the environment. However, there is a lack of a real scale factor between the 3D scene reconstructed from the image sequence and the real 3D scene. Therefore, the method to obtain the real scale factor is to use the point structure cursor calibration to obtain the real space coordinates of the spot marked as [X, Y, Z] and the space coordinates of the spot obtained by using the 3D reconstruction algorithm to be marked as The ratio of [X ₀ , Y ₀ , Z ₀ ] is used as the scale factor λ, that is, λ=[X,Y,Z]/[X ₀ ,Y ₀ ,Z ₀ ] so that the Euclidean three-dimensional reconstruction of monocular vision can be realized .

(2) Euclidean reconstruction of various methods of monocular reconstruction is not limited

Scene reconstruction based on multi-view images has always been a research hotspot in the field of computer vision. Multi-view stereo reconstruction uses a series of images of a scene taken at different locations to restore the real 3D scene. Today, the mainstream dense reconstruction methods include voxel-based methods, methods based on polygonal mesh deformation, methods based on multi-view depth maps, and methods based on patch expansion. However, the reconstruction result of the above method is different from the reconstruction result of the real scene by a scale factor, and the Euclidean reconstruction of the real scene cannot be realized. For this, the monocular vision Euclidean three-dimensional reconstruction method based on point structured light proposed by the present invention is not subject to Reconstruction methods are limited and are able to achieve Euclidean 3D reconstruction. As long as the real space point coordinates of the laser spot projected on the target are determined according to the point structured light, and the corresponding relationship between the image spot coordinates and the real space point coordinates is determined, the scale factor can be obtained, so it will not be used in the 3D reconstruction process using other methods. Restricted by the reconstruction method, Euclidean 3D reconstruction can be realized.

Description of drawings

Fig. 1 Schematic diagram of obtaining laser rays.

Figure 2 overall flow chart.

Figure 3 Calculation of spot centroid.

Figure 4 adds RANSAC straight line fitting.

Detailed ways

The present invention will be described in further detail in conjunction with the accompanying drawings.

The present invention specifically includes the following steps.

(1) Center positioning of spot light spot

The image is preprocessed through various filtering or threshold selection methods, and then the centroid of the image spot is located. After processing the entire image pixel, according to the cumulative value of the final centroid parameter group of each spot, the row and column coordinates of the centroid are calculated according to the following first-order moment centroid calculation formula:

In the above formula, I(x, y) represents the gray value of the input image pixel, x, y are the row and column coordinates corresponding to the pixel, and X _c , Y _c are the row and column coordinates of the centroid of the light spot respectively.

(2) Laser ray equation fitting with RANSAC elimination mechanism

RANSAC achieves its goal by iteratively selecting a set of random subsets in the data. The selected subset is assumed to be the interior point, and is verified by the following method, and the model description is as follows:

1) Select n points from all spot centroids I as interior points, and use n points according to the least squares method to estimate the space linear equation L: Ax+By+Cz+D=0, that is, all unknown parameters of the equation (A ,B,C,D) can be calculated from the hypothetical interior points.

2) Use the model obtained in 1) to test all other remaining spot centroids (the number of spot centroids is I-n), if a certain spot centroid point is suitable for the estimated space linear model L, that is, if a certain spot centroid coordinate (x', y', z') satisfy Ax'+By'+Cz'+D<|σ|, and consider this centroid point to be an intra-local point, where σ is the set threshold.

3) If there are enough spot centroid points classified as hypothetical interior points, that is, there are enough spot centroid points (assuming m points are satisfied) to satisfy Ax'+By'+Cz'+D<|σ| , then the estimated model L is reasonable enough.

4) Then, re-estimate the model with m inlier points satisfying the condition, because it has only been estimated by the initial hypothetical inlier points.

5) Finally, the model is evaluated by estimating the error rate of the inlier points and the model.

This process is repeated for a fixed number of t times, and the model generated each time is either discarded because there are too few intra-site points, or selected because it is better than the existing model.

(3) Calculate the coordinates of the three-dimensional space point of the spot

However, the intersection point of the ray from the optical center of the camera passing through the center of mass of the spot and the ray from the laser cannot completely intersect at one point due to errors. The midpoint of the line segment is used as the spatial coordinates of the laser bright spot, so that the three-dimensional coordinates of the spatial point corresponding to the image coordinates of the spot centroid in the image can be obtained. The specific mathematical model is as follows: the direction vector of the ray O ₁ P ₁ emitted through the optical center of the camera is V ₁ , and the equation direction vector of the light O ₂ P ₂ emitted by the laser fitted in the camera coordinate system is expressed as V ₂ , then its common perpendicular The direction vector of is V=V ₁ ×V ₂ , and the intersection points of the common vertical line and O ₁ P ₁ , O ₂ P ₂ are respectively M ₁ (x ₁ ,y ₁ ,z ₁ ) and M ₂ (x ₂ ,y ₂ , z ₂ ), then the spatial three-dimensional point coordinates of the laser spot are M(x, y, z), where x=(x ₁ +x ₂ )/2, y=(y ₁ +y ₂ )/2, z =(z ₁ +z ₂ )/2.

This paper further proposes to use the back-projection iterative method to optimize the obtained spatial three-dimensional point coordinates. The specific method is as follows: firstly, the three-dimensional spatial point corresponding to the image coordinates [u, v] of the center of mass of the spot is obtained according to the midpoint algorithm of the common vertical line of the straight line of different planes. After the coordinates [x, y, z], back-project this space point to the image, that is, find the intersection point [u', v'] of the line connecting the three-dimensional point in space and the optical center of the camera and the image coordinates, so that the center of mass of the spot can be obtained With the midpoint of the image coordinates after back projection [(u+u')/2, (v+v')/2], and use this as the spot centroid, calculate the three-dimensional space according to the midpoint algorithm of the common vertical line of different planes Point coordinates, repeat the above process continuously, calculate the distance δ between the three-dimensional coordinates M (x ^* , y ^* , z ^* ) of the space point and the straight line l projected by the laser (the equation of the straight line in space is Ax+By+Cz+D=0), and calculate The formula is If δ is less than a certain threshold, the iteration ends so far.

(4) Realize Euclidean 3D reconstruction

Dense reconstruction methods include voxel-based methods, polygon mesh deformation-based methods, multi-view depth map-based methods, and patch expansion-based methods. Among them, the first three methods are only suitable for the reconstruction of a single 3D entity, so they cannot meet the technical requirements of robot navigation and virtual reality. The patch expansion method has high precision and can reconstruct large scenes such as buildings. However, the connectivity between patches, that is, the integrity of the reconstruction is difficult to guarantee.

Aiming at the above problems, this paper adopts a 3D reconstruction method of large monocular vision scenes driven by scene flow feedback. The specific steps are as follows: a hand-held free-moving camera collects multi-view images of the target scene, and the optical flow field between adjacent frames is used to establish a multi-view reconstruction method. The pixel matching relationship, and then use the five-point algorithm to solve the Euclidean space transformation relationship between multiple views. Select the central view as the reference frame, establish the world coordinate system (O _w -X _w Y _w Z _w ), and solve the sparse space three-dimensional coordinates corresponding to the corresponding image points. On the basis of sparse reconstruction, the original mesh patch is generated and fed back to the viewing angles of each comparison frame, the feedback error is quantitatively evaluated by the optical flow field, and the deviation of each viewing image is used to drive the model deformation. Since the optical flow vector field contains the motion vector field information of spatial objects, the original polygonal mesh can be effectively corrected by means of the optical flow-scene flow analysis method. After obtaining the scene adjusted by the optical flow-scene flow, use The spatial scale factor obtained in the above steps can obtain the Euclidean reconstruction of the 3D scene.

The above description is only a preferred embodiment of the present invention, and is not used to limit the protection scope of the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention shall be included in the within the protection scope of the present invention.

Claims (1)

1. The method for determining the scale factor in monocular vision reconstruction based on point structured light distance measurement, characterized in that: in the process of calculating the scale factor; the entire calibration is unified on the camera coordinate system; the internal parameters of the camera use a camera based on a 2D plane target The calibration method is used for calibration. When calibrating the parameters of the structured light system, the spatial positions of the camera and the laser are kept unchanged; Take the image at this time, repeat this step to place the target any number of times to obtain a set of data images, and obtain the spot information corresponding to the target at each position according to the background difference method; use the corner point information on the target to obtain Putting the external parameter matrix of the target plane, plus the coordinate information of the center of mass of the laser spot in each calibration image, the spatial coordinates of the center of mass of the laser spot are obtained each time the target is placed; The spatial equation of the laser light can be obtained by performing Levenberg-Marquardt fitting on the results of the coordinates; the spatial coordinates of the laser spot on the image plane can be obtained through the calibrated sensor internal parameter information; theoretically, the laser light and the camera optical center and the laser spot are in the The coordinate lines of the image points on the image will completely coincide. However, due to calibration and calculation errors, these two lines will form straight lines with different planes. Set the midpoint of the common vertical line of the two straight lines with different planes as the real space coordinates of the laser spot. , the real scale factor is the ratio of the real space coordinates of the spot obtained by using the point structure cursor to the space coordinates of the spot obtained by using the three-dimensional reconstruction algorithm, so as to achieve the effect of Euclidean reconstruction; This method specifically includes the following steps: (1) Center positioning of spot light spot Preprocess the image through various filtering or threshold selection methods, and then locate the centroid of the image spot; after processing the entire image pixel, according to the accumulated value of the final centroid parameter group of each spot, calculate according to the following first-order moment centroid The formula calculates the row and column coordinates of its centroid: In the above formula, I(x, y) represents the gray value of the input image pixel, x, y are the row and column coordinates corresponding to the pixel, and X _c and Y _c are the row and column coordinates of the spot centroid respectively; (2) Laser ray equation fitting with RANSAC elimination mechanism RANSAC achieves the goal by repeatedly selecting a set of random subsets in the data; the selected subset is assumed to be an in-house point, and is verified by the following method. The model is described as follows: 1) Select n points from all spot centroids I as interior points, and use n points according to the least squares method to estimate the space linear equation L: Ax+By+Cz+D=0, that is, all unknown parameters of the equation (A ,B,C,D) can be calculated from the hypothetical interior points; 2) Use the model obtained in 1) to test all other remaining spot centroids, the number of spot centroids is I-n, if a certain spot centroid point is suitable for the estimated space linear model L, that is, if a certain spot centroid coordinates ( x', y', z') satisfy Ax'+By'+Cz'+D<|σ|, it is considered that this centroid point is also an internal point, where σ is the set threshold; 3) If there are enough spot centroid points classified as hypothetical interior points, that is, there are enough spot centroid points, assuming there are m points, satisfying Ax'+By'+Cz'+D<|σ|, Then the estimated model L is reasonable enough; 4) Then, use m internal points satisfying the condition to re-estimate the model, because it has only been estimated by the initial hypothetical internal point; 5) Finally, evaluate the model by estimating the error rate of the inlier points and the model; This process is repeated for a fixed number of t times, and the model generated each time is either discarded because there are too few internal points, or selected because it is better than the existing model; (3) Calculate the coordinates of the three-dimensional space point of the spot However, the intersection point of the ray from the optical center of the camera passing through the center of mass of the spot and the ray from the laser cannot completely intersect at one point due to errors. The midpoint of the line segment is used as the spatial coordinates of the laser bright spot, so that the three-dimensional coordinates of the spatial point corresponding to the image coordinates of the spot centroid in the image can be obtained; the specific mathematical model is as follows: the ray O ₁ P ₁ direction vector sent by the optical center of the camera is V ₁ , the equation direction vector of the light O ₂ P ₂ emitted by the laser fitted in the camera coordinate system is expressed as V ₂ , then the direction vector of the common perpendicular is V=V ₁ ×V ₂ , and the common perpendicular is equal to O ₁ P ₁ , O ₂ P ₂ are respectively M ₁ (x ₁ ,y ₁ ,z ₁ ) and M ₂ (x ₂ ,y ₂ ,z ₂ ), then the spatial three-dimensional point coordinates of the laser spot are M(x,y, z), where x=(x ₁ +x ₂ )/2, y=(y ₁ +y ₂ )/2, z=(z ₁ +z ₂ )/2; Use the back-projection iterative method to optimize the obtained spatial three-dimensional point coordinates. The specific method is as follows: firstly, the three-dimensional coordinates of the spatial point [x ,y,z], back-project this space point to the image, that is, find the intersection point [u',v'] of the three-dimensional point in space, the line connecting the optical center of the camera and the image coordinates, so that the centroid of the spot and the back-projection can be obtained The midpoint of the final image coordinates [(u+u')/2, (v+v')/2], and this is used as the spot centroid, and the three-dimensional point coordinates of the space are calculated according to the midpoint algorithm of the straight line and the vertical line of different planes. Repeat the above process continuously to calculate the distance δ between the three-dimensional coordinates M(x ^* , y ^* , z ^* ) of the spatial point and the straight line l projected by the laser. The spatial straight line equation is Ax+By+Cz+D=0, and the calculation formula is If δ is less than a certain threshold, the iteration ends so far; (4) Realize Euclidean 3D reconstruction Using the 3D reconstruction method of large monocular vision scenes driven by scene flow feedback, the specific steps are as follows: a hand-held free-moving camera collects multi-view images of the target scene, and the pixel matching relationship throughout the multi-view is established by the optical flow field between adjacent frames. Then use the five-point algorithm to solve the Euclidean space transformation relationship between multiple views; select the central view as the reference frame, establish the world coordinate system O _w -X _w Y _w Z _w , and solve the sparse space three-dimensional coordinates corresponding to the corresponding image points; On the basis of reconstruction, the original grid surface is generated and fed back to the viewing angles of each comparison frame, the feedback error is quantitatively evaluated by the optical flow field, and the deviation of each viewing image is used to drive the model deformation; since the optical flow vector field contains the movement of space objects Vector field information, so the original polygonal grid can be effectively corrected by means of the optical flow-scene flow analysis method. After obtaining the scene adjusted by the optical flow-scene flow, the spatial scale factor obtained by the above steps can be obtained. Euclidean reconstruction of 3D scenes.