CN110853075A - A visual tracking and localization method based on dense point cloud and synthetic view - Google Patents

Abstract

The invention provides a visual tracking and positioning method based on dense point clouds and synthetic views. The process is as follows: using three-dimensional scanning of a real scene to obtain a color key frame image and a corresponding depth image, after image restoration, the key frame image is processed. Image coding: perform image coding on the current frame image obtained by the camera in real time, and select a composite image with the closest coding distance to the current frame image as the reference frame image of the current image; obtain a stable set of matching feature points on the two images, and Perform processing to obtain the six-degree-of-freedom pose information of the current frame camera relative to the 3D scanning point cloud coordinate system; use the optical flow algorithm to determine, if the requirements cannot be met, update the current frame image to the next frame image obtained by the camera, and repeat match. The invention can solve the association problem between the three-dimensional point cloud obtained by the laser radar and the heterologous visual image, and has the effect of realizing the rapid initialization and positioning of the visual navigation.

Description

A visual tracking and localization method based on dense point cloud and synthetic view

technical field

The invention belongs to the field of information technology, and in particular relates to a visual tracking and positioning method based on dense point clouds and synthetic views.

Background technique

Visual real-time tracking and positioning of large-scale outdoor scenes has always been an important research direction in the field of computer vision. Outdoor complex and changeable environmental factors, such as illumination, viewing angle, occlusion, weak textures, and objects moving with time, have a great impact on the accuracy and robustness of visual tracking and positioning algorithms. Outdoor combat scenes such as deserts, grasslands, and mountains have a wide area and lack of rich scene texture information, which poses higher technical challenges to the visual tracking and positioning algorithm. At present, the most commonly used outdoor large scene visual tracking and positioning algorithm is the SLAM (real-time positioning and map building) algorithm. This kind of algorithm can be divided into feature-based SLAM algorithm and direct method-based SLAM algorithm. The feature-based SLAM algorithm can obtain the sparse map of the scene and the real-time motion trajectory of the camera. The SLAM algorithm based on the direct method can obtain the dense map of the scene and the real-time motion trajectory of the camera. The characteristic method and the direct method have their own advantages and disadvantages, and they are suitable for different application fields. The limitation of the current SLAM algorithm is that since the scene map and pose need to be estimated at the same time, the calculation amount is large, the camera pose estimation has error accumulation drift, and it is difficult to deal with dynamic scenes and cannot be applied to large-scale scenes.

In order to solve the problem of the large amount of calculation of the SLAM algorithm and the accumulated drift of the attitude estimation error, the offline map can be constructed and the online tracking and positioning method can be adopted. Offline maps can be acquired using vision-based SFM algorithms (structural reconstruction of motion) or 3D lidar scanning. Due to the scale uncertainty of the scene map obtained by the SFM algorithm, it is difficult to be applied in outdoor augmented reality military simulation training. The use of lidar to obtain 3D point clouds of outdoor scenes has the characteristics of high scanning accuracy and definite scale. Therefore, it has become an important research direction to use lidar to obtain a 3D map of the scene and use it for subsequent vision-based tracking and positioning. The 3D map constructed by lidar and the image obtained by the camera belong to heterogeneous data, and there are various unfavorable factors such as illumination changes, sensor differences, and different viewing angles.

SUMMARY OF THE INVENTION

The purpose of the present invention is to overcome the defects of the prior art, in order to solve the problem of the correlation between the three-dimensional scanning data of the lidar and the image data, so that it can be used for visual tracking and positioning, a visual tracking based on dense point cloud and synthetic view is proposed. positioning method.

The inventive method is achieved through the following technical solutions:

A visual tracking and positioning method based on dense point cloud and synthetic view, the specific process is as follows:

Step 1: use a 3D lidar scanner to perform 3D scanning on the real scene at a fixed position, and obtain a 3D point cloud model of the real scene and a color image corresponding to the point cloud model;

Step 2: the three-dimensional point cloud model with color information utilizes the projection transformation algorithm to synthesize the color key frame image and the corresponding depth image under a plurality of specific viewpoints;

Step 3: using the depth image to use an image repair algorithm to the synthesized color key frame image to repair the holes in the image;

Step 4: extract the feature points on the color key frame images after all the repairs, and carry out offline training, build a codebook, and carry out image coding to the key frame images;

Step 5: Perform image coding on the current frame image obtained by the camera in real time, and select a composite image with the closest coding distance to the current frame image as the reference frame image of the current image;

Step 6: Extract the feature points on the current frame and the reference frame image selected in step 5 respectively, and obtain a stable set of matching feature points by using a feature matching algorithm;

Step 7: Using the projection correspondence between the feature points on the reference frame image and the 3D scanning points, construct a 2D/3D space matching relationship between the current frame matching feature points and the 3D scanning points;

Step 8: According to the spatial matching relationship, use the six-degree-of-freedom pose information of the current frame camera relative to the three-dimensional scanning point cloud coordinate system;

Step 9: Use the optical flow algorithm to predict the position of the matching feature point on the current frame image in the next frame image;

Step 10: If the number of matching points of optical flow tracking is greater than the set threshold, then the 6-DOF pose information calculated in step 8 is combined with the visual SLAM algorithm for subsequent pose estimation. If the number of matching points of optical flow tracking is less than the set value If the threshold is set, the current frame image is updated to be the next frame image obtained by the camera, and the process returns to step 5.

beneficial effect

Compared with the prior art, the method of the present invention performs image repair, retrieval and matching by projecting the three-dimensional point cloud obtained by the laser radar to synthesize the image, so it can solve the problem of association between the three-dimensional point cloud obtained by the laser radar and the heterologous visual image , which has the effect of realizing rapid initialization and positioning of visual navigation.

Description of drawings

FIG. 1 is a schematic flowchart of an algorithm according to an embodiment of the present invention;

Figure 2 is a 3D model of a real scene scanned by Faro;

Figure 3 shows the synthesized key frame image and the corresponding depth map, the left side is the key frame, and the right side is the depth map;

Figure 4 shows the key frame image and the corresponding depth map after image restoration, the left side is the key frame, and the right side is the depth map;

FIG. 5 is a feature matching diagram between the current frame image and the reference frame image, the left side is the current frame, and the right side is the reference frame.

Detailed ways

To make the purposes, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention.

The design idea of the present invention is as follows: the three-dimensional point cloud obtained by the lidar is subjected to spatial back-projection transformation to generate a synthetic image under known viewpoints, and the real-time 6 freedom of the camera is estimated by matching the synthetic image with the real-time image obtained by the camera. Degree pose. The method can be used for tracking navigation or assisted positioning in the fields of robots, unmanned vehicles, unmanned aerial vehicles, virtual reality and augmented reality.

The present invention is based on the visual tracking and positioning method of dense point cloud and synthetic view, as shown in FIG. 1 , and the specific steps include:

Step 1. Place the Faro lidar scanner at a certain location outdoors, scan the surrounding real scene, and obtain a 3D point cloud model and RGB images corresponding to the model. Each 3D point (x _w , y _w , z _w , r, g, b) of the 3D point cloud model contains the coordinates (x _w , y _w , z _w ) ^T and RGB color in the coordinate system of the point cloud model information(r,g,b).

Step 2: Using the projection transformation algorithm to synthesize the 3D point cloud model with color information to synthesize a color key frame image and a corresponding depth image under a specific viewpoint. The specific steps include:

和平移矢量

对点云中每一个三维点(x_w,y_w,z_w,r,g,b)利用公式(1)分别计算三维点在不同虚拟相机坐标系下的三维点坐标

(1) According to the 3D point cloud model obtained in step 1, use the triangular meshing algorithm to generate a triangular patch model and import the model into the 3D model software. In the 3D model software, the rotation matrices under k different viewpoints are obtained by adjusting the pose of the virtual camera

and translation vector

For each 3D point (x _w , y _w , z _w , r, g, b) in the point cloud, use formula (1) to calculate the 3D point coordinates of the 3D point in different virtual camera coordinate systems.

变换到图像坐标系下。(2) Set the internal parameters of the virtual camera according to the internal parameter values of the real camera, including the focal length (f _x , f _y ), the principal point (u ₀ , v ₀ ), and use formula (2) to set the virtual camera coordinate system coordinate of

Transform to the image coordinate system.

或

舍弃，否则保留。(3) Crop the image coordinates (u ^k , v ^k ) obtained by formula (2) according to the resolution W×H of the real camera, if

or

Discard, otherwise keep.

的

分量，保留最小正值

所对应的三维点，消除点云前后遮挡问题。(4) When there are multiple corresponding three-dimensional points (x _w , y _w , z _w , r, g, b) projected to the same image coordinates (u ^k , v ^k ), compare the three-dimensional points (x _w , y _w ) ,z _w ,r,g,b) three-dimensional point coordinates in the virtual camera coordinate system

of

components, keep the smallest positive value

The corresponding 3D points can eliminate the occlusion problem before and after the point cloud.

的

分量，合成出特定视点下的彩色关键帧图像和对应的深度图。(5) According to the projected image coordinates (u ^k , v ^k ), the color information (r, g, b) and the corresponding color information (r, g, b) of the corresponding three-dimensional point cloud (x _w , y _w , z _w , r, g, b) are assigned respectively. 3D point coordinates in the virtual camera coordinate system of

of

component to synthesize the color keyframe image and the corresponding depth map under a specific viewpoint.

Step 3: According to the color key frame image obtained in step 2 and the corresponding depth map, use an image restoration algorithm based on depth information to perform image restoration on the color key frame image obtained in step 2 one by one, and remove the holes in the color key frame image. . The specific implementation steps include:

(1) Use the linear interpolation algorithm to complete the depth map to remove the holes in the depth image,

(2) Use the completed depth map to repair the color keyframe image:

Read each pixel point (u, v) of the color key frame image, and determine whether it is a blank point. If the point is a blank point, look for the reference point (i, j) in the eight directions around the blank point, and from the depth map Obtain the depths of the point (u, v) and the reference point (i, j). The reference point cannot be a blank point. According to formula (3), the weighted sum of the reference points is performed to fill the blank point.

Among them, point (u, v) is a blank point, point (i, j) is a reference point, R is a set of reference points, I _{(i, j)} is the color information of point (i, j), I _{(u, v)} is the color information required to fill the blank point (u, v), D _{(i, j)} is the depth weighting factor, when the depth difference between the reference point (i, j) and the point (u, v) is less than the set When the threshold is determined, D _(i,j) = 1, otherwise D _{(i, j)} = 0, W _{(i, j)} is the distance weighting factor of point (i, j), calculated by formula (4),

(3) Perform the processing of the above step (2) on each point in the key frame image until the entire image is processed, and the image restoration is completed.

Step 4: Extract the feature points on all the repaired color key frame images, perform offline training, build a codebook, and use the VLAD (Vector of Aggregate Locally Descriptor) algorithm to encode the color key frame images. The specific implementation steps include:

(1) Extract the ORB feature points on all the restored color key frame images, construct a feature description subset {x _j }, use the k-means clustering algorithm to cluster the ORB feature point set, and obtain M cluster centers The codebook of , C={c ₁ c ₂ ... c _M } is the M clustering centers of the codebook;

(2) Each feature descriptor x _j extracted from a color key frame image is compared with the M cluster centers of the codebook using the K-nearest neighbor algorithm, and is assigned to its nearest neighbor cluster in the codebook according to the nearest neighbor principle Center C _i =NN(x _j ).

(3) Calculate the residual x _{j -ci} of each feature descriptor x _j and its nearest neighbor cluster center C _i , and accumulate the residuals of each color key frame image that fall on the same cluster center to form VLAD vector

(4) The VLAD descriptor of the color key frame image is constructed by concatenating the cumulative residual vector on each cluster center.

Step 5. Use the VLAD algorithm in step 4 to perform image coding on the current frame image obtained by the camera in real time to obtain the VLAD image descriptor of the current frame, and use the KNN (k nearest neighbor) algorithm to select the one closest to the encoding distance of the current frame image. The color key frame image is used as the reference frame image of the current image.

Step 6: Extract the ORB feature points on the current frame and the reference frame image selected in step 5, respectively, and use a feature matching algorithm to obtain a stable set of matching feature points. Specific steps include:

(1) The ORB features of the current frame and the reference frame image are extracted respectively, and the initial feature matching point set is determined according to the ratio of the nearest neighbor to the next nearest neighbor of the feature descriptor.

(2) Using the 8-point algorithm to calculate the basic matrix F constructed by the matching feature point set;

(3) Construct the back-projection error of the matching point according to the calculated fundamental matrix F, and use the RANSAC algorithm to further remove outliers.

Step 7: Using the projection correspondence between the feature points on the reference frame image and the 3D scanning points, construct a 2D/3D space matching relationship between the current frame matching feature points and the 3D scanning points.

Step 8. According to the spatial matching relationship between the current frame image coordinates and the three-dimensional space point coordinates established in step seven, use the PnP (perspective-n-point) algorithm in OpenCV to solve the six-freedom of the current camera relative to the three-dimensional point cloud coordinate system. Degree pose information.

Step 9: Use the optical flow tracking algorithm in OpenCV to estimate the image coordinates of the feature points of the current frame in the next frame of image.

Step 10. According to the optical flow estimation result in step 9, count the number of feature points that can be tracked stably, and compare it with the set threshold. If the number of feature points that can be tracked is greater than the threshold, then according to step 8 The obtained six-degree-of-freedom pose information is combined with the visual SLAM algorithm for subsequent pose estimation; if the number of feature points that can be tracked is less than the threshold, update the current frame image to the next frame image obtained by the camera, then return to step 5 and re-select the reference frame image.

Since then, visual tracking and localization based on dense point clouds and synthetic views has been achieved.

As shown in Figure 2-5, Figure 2 is the 3D model image of the real scene scanned by Faro, and the key frame image and the corresponding depth image are synthesized using the real scene 3D model image, as shown in Figure 3, and the left side of Figure 3 is the key frame image , the right side is the depth image; after image repairing in step 3, the repaired image is shown in Figure 4, the left side is the repaired key frame image, and the right side is the repaired depth image; Figure 5 is the current frame image The feature matching map with the reference frame image, the left side is the current frame image, and the right side is the reference frame image.

To sum up, the above are only preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims (5)

1. A visual tracking positioning method based on dense point cloud and synthetic view is characterized by comprising the following specific processes:

the method comprises the following steps: three-dimensional scanning is carried out on a real scene at a fixed position by using a three-dimensional laser radar scanner, and a three-dimensional point cloud model of the real scene and a color image corresponding to the point cloud model are obtained;

synthesizing a three-dimensional point cloud model with color information into color key frame images and corresponding depth images under a plurality of specific viewpoints by using a projection transformation algorithm;

step three: repairing the hole in the image by using the depth image and the synthesized color key frame image by using an image repairing algorithm;

extracting characteristic points on all repaired color key frame images, performing off-line training, constructing a code book, and performing image coding on the key frame images;

step five: carrying out image coding on a current frame image acquired by a camera in real time, and selecting a synthetic image closest to the current frame image coding distance as a reference frame image of the current image;

step six: respectively extracting the feature points on the current frame and the reference frame image selected in the step five, and obtaining a stable matched feature point set by using a feature matching algorithm;

step seven: constructing a 2D/3D space matching relationship between the current frame matching characteristic point and the three-dimensional scanning point by using the projection corresponding relationship between the characteristic point on the reference frame image and the three-dimensional scanning point;

step eight: solving the six-degree-of-freedom pose information of the current frame camera relative to the three-dimensional scanning point cloud coordinate system according to the space matching relation;

step nine: predicting the position of the matched feature point on the current frame image in the next frame image by using an optical flow algorithm;

step ten: and if the number of the matching points of the optical flow tracking is larger than the set threshold, performing subsequent attitude estimation by combining a visual SLAM algorithm according to the pose information of the six degrees of freedom calculated in the step eight, and if the number of the matching points of the optical flow tracking is smaller than the set threshold, updating the current frame image to be the next frame image acquired by the camera, and returning to the step five.

2. The method for visual tracking and positioning based on dense point cloud and synthesized view as claimed in claim 1, wherein the image coding is performed by using VLAD algorithm in the fourth step and the fifth step.

3. The method as claimed in claim 1, wherein the step five is to select a synthetic image with the closest coding distance to the current frame image by using KNN algorithm.

4. The visual tracking and positioning method based on the dense point cloud and the synthetic view as claimed in claim 1, wherein in the eighth step, a PnP algorithm is used to solve pose information of six degrees of freedom.

5. The visual tracking and positioning method based on the dense point cloud and the synthetic view as claimed in claim 1, wherein the specific process of the third step is:

(1) the linear interpolation algorithm is utilized to complete the image of the depth image, remove the holes in the depth image,

(2) and repairing the color key frame image by using the completed depth map:

reading each pixel point (u, v) of the color key frame image, judging whether the pixel point is a blank point, if the pixel point is the blank point, searching a reference point (i, j) in eight directions around the blank point, acquiring the depth of the point (u, v) and the reference point (i, j) from the depth map, wherein the reference point cannot be the blank point, and weighting and summing the reference point according to a formula (3) to fill the blank point.

Wherein, the point (u, v) is a blank point, the point (I, j) is a reference point, R is a set formed by the reference points, I_(i,j)Is the color information of the point (I, j), I_(u,v)To fill in the color information required for the blank spots (u, v), D_(i,j)For the depth weighting factor, when the difference between the depth of the reference point (i, j) and the depth of the point (u, v) is less than a set threshold value, D_(i,j)1, otherwise D_(i,j)＝0，W_(i,j)Is the distance weighting factor for point (i, j), calculated by equation (4),

(3) and (3) performing the processing of the step (2) on each point in the key frame image until the whole image processing is finished, and finishing the image restoration.

Images