CN116468768B - Scene depth completion method based on conditional variation self-encoder and geometric guidance - Google Patents

Abstract

The invention discloses a scene depth completion method based on a conditional variation self-encoder and geometric guidance, which comprises the following steps: acquiring a color image, a sparse depth map and a dense depth map under an automatic driving scene; the method comprises the steps of designing a condition variation self-encoder with a priori network and a posterior network, inputting a color image and a sparse depth map into the priori network to extract features, and inputting the color image, the sparse depth map and the dense depth map into the posterior network to extract features; and converting the sparse depth map into point cloud by using camera internal parameters, namely focal length and optical center coordinates, extracting a geometric space feature from a point cloud up-sampling model, and mapping the geometric space feature back to the sparse depth map. The invention can solve the problem that the data acquired by the laser radar is too sparse, so that the low-cost laser radar with fewer wire harnesses can obtain more accurate and dense depth information, and provides a cost-effective solution for industries such as automatic driving, robot environment sensing and the like which need accurate and dense depth data.

Description

Scene depth completion method based on conditional variational autoencoders and geometric guidance

Technical field

The present invention relates to the technical field of depth map completion, specifically a scene depth completion method based on conditional variational autoencoders and geometric guidance.

Background technique

Human perception, understanding and experience of the surrounding environment rely on three-dimensional scene information acquired through vision. Computer vision imitates human behavior and uses various sensors as visual organs to obtain scene information, thereby realizing the recognition and understanding of the scene. Depth information plays a key role in fields such as robotics, autonomous driving, and augmented reality. In the field of autonomous driving, it is necessary to sense the distance between the current vehicle and other vehicles, pedestrians and obstacles during driving. Full automation Level 5 requires the ability to measure distances accurate to centimeters. Currently, LiDAR is the main active distance sensor in autonomous driving. Compared with the two-dimensional RGB image collected by a color camera, the depth map collected by lidar (the depth map and point cloud can be converted into each other through the camera's internal parameters) has a precise depth distance, so it can accurately perceive the position of the 3D target in the surrounding environment. information. However, a lidar can only emit 16, 32 or 64 limited laser beams in the vertical direction, resulting in an extremely sparse point cloud density (the pixels with effective depth values only account for about 5% of the color image), giving Downstream tasks such as 3D target detection and 3D environment perception have had a serious impact.

Contents of the invention

The purpose of the present invention is to provide a scene depth completion method based on conditional variational autoencoders and geometric guidance to solve the key problems of data sparseness and missingness caused by existing depth imaging equipment such as lidar.

To achieve the above objectives, the present invention provides the following technical solution: a scene depth completion method based on conditional variational autoencoders and geometric guidance, including the following steps:

Obtain color images, sparse depth maps and dense depth maps in autonomous driving scenarios;

Design a conditional variational autoencoder with a priori network and a posteriori network, input the color image and sparse depth map into the prior network to extract features, and then input the color image, sparse depth map and dense depth map into the posterior network extract features;

Convert the sparse depth map into a point cloud using camera intrinsic parameters or focal length, then extract the point cloud upsampling model into geometric space features, and map them back to the sparse depth map;

The dynamic graph message dissemination module is used to achieve the fusion of image features and point cloud features;

A preliminary depth completion map is generated using a U-shaped encoder and decoder based on a residual network;

The initially predicted completion depth map is input into the confidence uncertainty estimation module to achieve the final depth completion optimization.

Preferably, the acquisition of color images and sparse depth maps in autonomous driving scenarios includes:

Use color cameras and lidar to capture color images and sparse depth maps in autonomous driving scenarios;

The SparsityInvariantCNNs algorithm is used to convert the sparse depth map into a dense depth map as a real label auxiliary training.

Preferably, the conditional variational autoencoder is designed with a priori network and a posteriori network, the color image and sparse depth map are input into the prior network to extract features, and then the color image, sparse depth map and dense depth map are Input to the posterior network to extract features, including:

Based on the feature extraction module of the ResNet structure, a priori network and a posteriori network with the same structure are designed as conditional variational autoencoders;

Input the color image and sparse depth map into the prior network to extract the feature map Prior of the last layer. At the same time, input the color image, sparse depth map and real label into the posterior network to extract the feature map Posterior of the last layer, and then calculate Prior and The mean and variance of the Posterior feature map are used to obtain the probability distributions D ₁ and D ₂ of the respective features, and then the Kullback-Leibler divergence loss function is used to supervise the loss between the distributions D ₁ and D ₂ so that the prior network can learn the posterior Test the real label characteristics of the network.

Preferably, the method uses camera internal parameters or focal length to convert the sparse depth map into a point cloud, and then extracts the point cloud upsampling model into geometric space features and maps them back to the sparse depth map, including:

According to the internal parameters of the camera, the sparse depth image pixels (u _i , vi ₎ are converted from the pixel coordinate system to the camera coordinate system to obtain the point cloud coordinates (xi _, y _i , z _i ) of the three-dimensional scene, forming sparse point cloud data S;

Among them, (c _x , c _y ) are the optical center coordinates of the camera, f _x , f _y are the focal lengths of the camera in the x-axis and y-axis directions respectively, d _i is the depth value at (u _i , vi ₎ , for The real label depth map also uses the above formula to generate a dense label point cloud S ₁ ;

By randomly sampling the point cloud S multiple times, different numbers of point cloud sets are obtained. For each point cloud set, the KNN nearest neighbor node algorithm is used to aggregate the 16 closest points around each point and input into the geometry-aware neural network. Extract the local geometric features of the point;

Add the sparse point cloud features extracted from each point to the original point cloud coordinates (x _i , y _i , z _i ) to obtain the point cloud encoding feature Q, and input Q into the multi-layer perceptron network with four times upsampling The predicted dense point cloud S ₂ is obtained, and the ChamferDistance loss function is used to calculate the loss value between the real dense point cloud S ₁ and the predicted dense point cloud S ₂ to supervise the training of the network. The specific calculation formula of CD loss is as follows:

Among them, the first term represents the sum of the minimum distances from any point x in S ₁ to S ₂ , and the second term represents the sum of the minimum distances from any point y in S ₂ to S ₁ .

Preferably, the dynamic graph message dissemination module is used to realize the fusion of image features and point cloud features, including:

Design two encoding networks with the same structure. The encoder is composed of a 5-layer ResNet module. The color image and sparse depth map are input to the RGB branch encoder to extract five different scale feature maps L ₁ , L ₂ , L ₃ , L ₄ and L ₅ , input the point cloud feature Q and sparse depth map to the point cloud branch encoder to extract feature maps of five different scales P ₁ , P ₂ , P ₃ , P ₄ and P ₅ ;

For L ₁ , L ₂ , L ₃ , L ₄ and L ₅ , dilated convolution is used to obtain pixels in different receptive fields, and then deformable convolution is used to explore the coordinate offset of each pixel, so that each Each pixel dynamically aggregates surrounding feature values with strong correlation to obtain features T ₁ , T ₂ , T ₃ , T ₄ and T ₅ ;

Add T ₁ , T ₂ , T ₃ , T ₄ , and T ₅ rich in dynamic image features to the point cloud encoding feature maps P ₁ , P ₂ , P ₃ , P ₄ , and P ₅ to obtain semantic information and geometry. Information point cloud feature maps M ₁ , M ₂ , M ₃ , M ₄ , M ₅ .

Preferably, the U-shaped codec based on residual network is used to generate a preliminary depth completion map, including:

Design the corresponding multi-scale decoder structure according to the encoder structure to form an encoder-decoder network with a U-Net structure;

Input the feature maps L ₁ , L ₂ , L ₃ , L ₄ and L ₅ generated by the RGB branch into U-Net to predict the first rough depth completion map Depth ₁ and confidence map C ₁ ;

Input the feature maps M ₁ , M ₂ , M ₃ , M ₄ and M ₅ generated by the point cloud branch into U-Net to predict the second rough depth completion map Depth ₂ and confidence map C ₂ .

Preferably, the initially predicted completion depth map is input into the confidence uncertainty estimation module to achieve final depth completion optimization, including:

Add the generated confidence maps C ₁ and C ₂ to obtain the feature map C, use the Softmax function to predict the uncertainty of C, and predict the uncertainty ratios F ₁ and F ₂ of each confidence map pixel by pixel;

The final optimized depth completion map is obtained by multiplying the uncertainty maps F ₁ and F ₂ with the rough depth completion Depth ₁ and Depth ₂ respectively.

Compared with the prior art, the beneficial effects of the present invention are:

By using conditional variational autoencoders to learn the feature distribution in real dense depth maps, color images and sparse depth maps are guided to generate more valuable depth features. Secondly, point cloud features in three-dimensional space are used to capture the space under different modalities. Structural features enhance the geometric perception of the network and provide auxiliary information for predicting more accurate depth values. Moreover, the dynamic graph message propagation module cleverly integrates the features between color images and point clouds to achieve high-precision depth completion. Prediction can make up for the problem of too sparse data collected by lidar, allowing low-cost lidar with fewer wire harnesses to obtain more accurate and dense depth information, providing a cost-effective solution for industries that require accurate and dense depth data, such as autonomous driving and robot environment perception. Efficient solutions.

Description of the drawings

Figure 1 is a flow chart of a scene depth completion method based on conditional variational autoencoders and geometric guidance provided by an embodiment of the present invention;

Figure 2 is a depth completion diagram of a scene depth completion method based on conditional variational autoencoders and geometric guidance provided by an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

The execution subject of the method in this embodiment is a terminal. The terminal can be a mobile phone, a tablet computer, a PDA, a notebook or a desktop computer. Of course, it can also be other devices with similar functions, which is not limited by this embodiment. .

Please refer to Figures 1 and 2. The present invention provides a scene depth completion method based on conditional variational autoencoders and geometric guidance. The method is applied to automatic driving scene depth completion, including:

Step S1: Obtain color images, sparse depth maps and dense depth maps in autonomous driving scenarios.

Specifically, step S1 also includes the following steps:

S101, uses color cameras and lidar to capture color images and sparse depth maps in autonomous driving scenarios;

S102, use the Sparsity Invariant CNNs algorithm to convert the sparse depth map into a dense depth map as real label auxiliary training.

Among them, self-driving vehicles are mainly equipped with color cameras and lidar, which collect RGB images and depth images respectively. In this method, an additional complementary depth map needs to be generated as a training label. The specific steps are as follows:

Use a color camera and Velodyne HDL-64E lidar to capture color images and depth images in autonomous driving scenarios; use the Sparsity Invariant CNNs algorithm to convert sparse depth maps into dense depth maps as real labels.

Step S2, design a conditional variational autoencoder with a priori network and a posteriori network, input the color image and sparse depth map into the prior network to extract features, and then input the color image, sparse depth map and dense depth map into Extract features from the posterior network.

Specifically, step S2 also includes the following steps:

S201, based on the feature extraction module of ResNet structure, design a priori network and posterior network with the same structure as a conditional variational autoencoder;

S202, input the color image and sparse depth map into the prior network to extract the feature map Prior of the last layer. At the same time, input the color image, sparse depth map and real label into the posterior network to extract the feature map Posterior of the last layer, and then calculate them respectively. The mean and variance of the Prior and Posterior feature maps are used to obtain the probability distributions D ₁ and D ₂ of the respective features, and then the Kullback-Leibler divergence loss function is used to supervise the loss between the distributions D ₁ and D ₂ so that the prior network can learn to the true label features of the posterior network.

Step S3: Use camera intrinsic parameters or focal length to convert the sparse depth map into a point cloud, then extract the point cloud upsampling model into geometric space features, and map them back to the sparse depth map.

Specifically, step S3 also includes the following steps:

S301, convert the sparse depth image pixels (u _i , _vi ) from the pixel coordinate system to the camera coordinate system according to the camera internal parameters/focal length, etc. to obtain the point cloud coordinates (x _i , y _i , z _i ) of the three-dimensional scene, forming a sparse Point cloud data S;

Among them, (c _x , c _y ) are the optical center coordinates of the camera, f _x , f _y are the focal lengths of the camera in the x-axis and y-axis directions respectively, d _i is the depth value at (u _i , vi ₎ , for The real label depth map also uses the above formula to generate a dense label point cloud S ₁ ;

S302, obtain different numbers of point cloud collections by randomly sampling the point cloud S multiple times. For each point cloud collection, use the KNN nearest neighbor node algorithm to aggregate the 16 closest points around each point and input them to the geometry perception neural network. Extract the local geometric features of the point in the network;

S303, add the sparse point cloud features extracted from each point to the original point cloud coordinates (x _i , y _i , z _i ) to obtain the point cloud encoding feature Q, and input Q to the multi-layer perceptron with four times upsampling The predicted dense point cloud S ₂ is obtained from the network, and the Chamfer Distance loss function is used to calculate the loss value between the real dense point cloud S ₁ and the predicted dense point cloud S ₂ to supervise the training of the network. The specific calculation formula of CD loss as follows:

Among them, the first term represents the sum of the minimum distances from any point x in S ₁ to S ₂ , and the second term represents the sum of the minimum distances from any point y in S ₂ to S ₁ .

Step S4: Use the dynamic graph message dissemination module to achieve the fusion of image features and point cloud features.

Specifically, step S4 also includes the following steps:

S401, design two encoding networks with the same structure. The encoder is composed of a 5-layer ResNet module. The color image and sparse depth map are input to the RGB branch encoder to extract five feature maps of different scales L ₁ , L ₂ , L ₃ , L ₄ and L ₅ , input the point cloud feature Q and sparse depth map to the point cloud branch encoder to extract feature maps of five different scales P ₁ , P ₂ , P ₃ , P ₄ and P ₅ ;

S402, for L ₁ , L ₂ , L ₃ , L ₄ and L ₅ , use atrous convolution to obtain pixels in different receptive fields, and then use deformable convolution to explore the coordinate offset of each pixel. This allows each pixel to dynamically aggregate surrounding feature values with strong correlation to obtain features T ₁ , T ₂ , T ₃ , T ₄ and T ₅ ;

S403, add T ₁ , T ₂ , T ₃ , T ₄ , and T ₅ rich in dynamic image features to the point cloud encoding feature maps P ₁ , P ₂ , P ₃ , P ₄ , and P ₅ to obtain semantic information. and point cloud feature maps M ₁ , M ₂ , M ₃ , M ₄ , M ₅ of geometric information.

Step S5: Use the U-shaped codec based on the residual network to generate a preliminary depth completion map.

Specifically, step S5 also includes the following steps:

S501. Design the corresponding multi-scale decoder structure according to the encoder structure in step S4 to form an encoder-decoder network with a U-Net structure;

S502, input the feature maps L ₁ , L ₂ , L ₃ , L ₄ and L ₅ generated by the RGB branch into U-Net to predict the first rough depth completion map Depth ₁ and confidence map C ₁ ;

S503, input the feature maps M ₁ , M ₂ , M ₃ , M ₄ and M ₅ generated by the point cloud branch into U-Net to predict the second rough depth complement map Depth ₂ and confidence map C ₂ .

Step S6: Input the initially predicted completion depth map into the confidence uncertainty estimation module to achieve final depth completion optimization.

Specifically, step S6 also includes the following steps:

S601, add the confidence maps C ₁ and C ₂ generated in step S5 to obtain the feature map C, use the Softmax function to perform uncertainty prediction on C, and predict the uncertainty ratio F ₁ of each confidence map pixel by pixel. and F ₂ ;

S602: Multiply the uncertainty maps F ₁ and F ₂ with the rough depth completion Depth ₁ and Depth ₂ respectively to obtain the final optimized depth completion map.

In this embodiment, first, by using conditional variational autoencoders to learn the feature distribution in real dense depth maps, color images and sparse depth maps are guided to generate more valuable depth features, and secondly, point cloud features in three-dimensional space are used Captures the spatial structure characteristics of different modalities, enhances the geometric perception ability of the network, and provides auxiliary information for predicting more accurate depth values. Moreover, the dynamic graph message propagation module cleverly integrates the features between color images and point clouds. , achieving high-precision depth completion prediction.

In addition, it should be noted that the combination of the technical features in this case is not limited to the combinations recorded in the claims of this case or the combinations recorded in the specific embodiments. All the technical features recorded in this case can be combined in any way. Free combination or combination, unless there is a conflict between them.

It should be noted that the above examples are only specific embodiments of the present invention. Obviously, the present invention is not limited to the above embodiments, and there are many similar changes. All modifications directly derived or thought of by those skilled in the art from the disclosure of the present invention shall fall within the protection scope of the present invention.

The above are only preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims (6)

1. The scene depth completion method based on the conditional variation self-encoder and the geometric guidance is characterized by comprising the following steps of:

acquiring a color image, a sparse depth map and a dense depth map under an automatic driving scene;

the method comprises the steps of designing a condition variation self-encoder with a priori network and a posterior network, inputting a color image and a sparse depth map into the priori network to extract features, and inputting the color image, the sparse depth map and the dense depth map into the posterior network to extract features;

converting the sparse depth map into point cloud by utilizing camera internal parameters or focal length, extracting a point cloud up-sampling model to geometric space features, and mapping the geometric space features back to the sparse depth map;

the dynamic image information transmission module is adopted to realize the fusion of image characteristics and point cloud characteristics, and the specific steps of the fusion are as follows: designing two coding networks with the same structure, wherein an encoder consists of a 5-layer ResNet module, and inputting a color image and a sparse depth map into an RGB (red, green and blue) branch encoder to extract five feature maps L with different scales ₁ ，L ₂ ，L ₃ ，L ₄ And L ₅ Inputting the point cloud characteristics Q and the sparse depth map into a point cloud branch encoder to extract characteristic maps P with five different scales ₁ ，P ₂ ，P ₃ ，P ₄ And P ₅ The method comprises the steps of carrying out a first treatment on the surface of the For L ₁ ，L ₂ ，L ₃ ，L ₄ And L ₅ Obtaining pixel points of different receptive fields by adopting a cavity convolution mode, and exploring the coordinate offset of each pixel point by utilizing deformable convolution to dynamically aggregate the characteristic values with strong surrounding correlation of each pixel point to obtain a characteristic T ₁ ，T ₂ ，T ₃ ，T ₄ And T ₅ The method comprises the steps of carrying out a first treatment on the surface of the T to be rich in dynamic diagram features ₁ ，T ₂ ，T ₃ ，T ₄ ，T ₅ Adding to point cloud coding feature map P ₁ ，P ₂ ，P ₃ ，P ₄ ，P ₅ Obtaining a point cloud characteristic map M containing semantic information and geometric information ₁ ，M ₂ ，M ₃ ，M ₄ ，M ₅ ；

Generating a preliminary depth complement diagram by using a U-shaped coder decoder based on a residual error network;

and inputting the preliminary predicted complement depth map to a confidence uncertainty estimation module to realize final depth complement optimization.

2. The scene depth completion method based on conditional variance self-encoder and geometric guidance of claim 1, wherein the acquiring of color image and sparse depth map in an autopilot scene comprises:

capturing a color image and a sparse depth map in an autopilot scene using a color camera and a lidar;

the sparse depth map is changed into a dense depth map by using a Sparsity Invariant CNNs algorithm to serve as a real tag to assist training.

3. The scene depth completion method based on conditional variance self-encoder and geometric guidance according to claim 2, wherein the designing the conditional variance self-encoder with a priori network and a posterior network, inputting the color image and sparse depth map into the a priori network to extract features, and then inputting the color image, sparse depth map and dense depth map into the posterior network to extract features, comprises:

the feature extraction module based on the ResNet structure designs a priori network and a posterior network with the same structure as a condition variation self-encoder;

inputting the color image and the sparse depth map into a priori network to extract the feature map primary of the last layer, inputting the color image, the sparse depth map and the real label into a Posterior network to extract the feature map Posterior of the last layer, and then respectively calculating the mean value and the variance of the primary and Posterior feature maps to obtain the probability distribution D of the respective features ₁ And D ₂ Supervision distribution D by using Kullback-Leibler divergence loss function ₁ And D ₂ The loss between the two causes the prior network to learn the real label characteristics of the posterior network.

4. A scene depth completion method based on a conditional variation self-encoder and geometric guidance according to claim 3, wherein said converting a sparse depth map into a point cloud using camera intrinsic parameters or focal lengths, extracting a point cloud upsampling model to geometric spatial features, and mapping back onto the sparse depth map comprises:

sparse depth image pixels (u) _i ,v _i ) Converting the pixel coordinate system into the camera coordinate system to obtain the point cloud coordinate (x _i ,y _i ,z _i ) Forming sparse point cloud data S;

wherein (c) _x ,c _y ) Is the optical center coordinate of the camera, f _x ，f _y Focal lengths of the camera in x-axis and y-axis directions, d _i Is (u) _i ,v _i ) Depth values at the locations, for a real tag depth map, a dense tag point cloud S is also generated using the above formula ₁ ；

The method comprises the steps of randomly sampling point clouds for a plurality of times to obtain different numbers of point cloud sets, aggregating 16 nearest points around each point by utilizing a KNN nearest node algorithm aiming at each point cloud set, and inputting the 16 nearest points into a geometric perception neural network to extract local geometric features of the point;

the sparse point cloud features extracted from each point are combined with the original point cloud coordinates (x _i ,y _i ,z _i ) Adding to obtain a point cloud coding characteristic Q, and inputting the Q into a quadruple up-sampling multi-layer perceptron network to obtain a predicted dense point cloud S ₂ Computing a true dense point cloud S using a Chamfer Distance loss function ₁ And predicted dense point cloud S ₂ The specific calculation formula of the CD loss is as follows:

wherein the first term represents S ₁ Any point x to S ₂ Sum of minimum distances of (2), second termRepresent S ₂ Any point y to S ₁ Is the sum of the minimum distances of (a) and (b).

5. The conditional variance self-encoder and geometry guided scene depth completion method of claim 1, wherein generating a preliminary depth complement map using a residual network-based U-shaped codec comprises:

designing a corresponding multi-scale decoder structure according to the encoder structure to form a U-Net structured encoder-decoder network;

feature map L generated by branching RGB ₁ ，L ₂ ，L ₃ ，L ₄ And L ₅ Inputting U-Net to predict the Depth of the first rough Depth complement map ₁ And confidence map C ₁ ；

Feature map M generated by branching point cloud ₁ ，M ₂ ，M ₃ ，M ₄ And M ₅ Inputting U-Net to predict second coarse Depth full-complement map Depth ₂ And confidence map C ₂ 。

6. The scene depth completion method based on conditional variance self-encoder and geometric guidance of claim 1, wherein said inputting the preliminary predicted completion depth map to the confidence uncertainty estimation module, achieves final depth completion optimization, comprises:

confidence map C to be generated ₁ And C ₂ Adding to obtain a characteristic diagram C, carrying out uncertainty prediction on the characteristic diagram C by using a Softmax function, and predicting an uncertainty proportion F of each confidence diagram pixel by pixel ₁ And F ₂ ；

Map of uncertainty F ₁ And F ₂ Depth of roughness and Depth of roughness ₁ And Depth ₂ Multiplying to obtain the final optimized depth complement diagram.