CN117953029B - General depth map completion method and device based on depth information propagation - Google Patents

Abstract

The invention relates to the technical field of image enhancement, in particular to a general depth map completion method and device based on depth information transmission. The general depth map completion method based on depth information propagation comprises the following steps: acquiring data of a scene by using a depth sensor to obtain a sparse depth map; acquiring data of a scene by using a color camera to obtain an RGB image; performing depth filling on the sparse depth map by adopting a pre-filling method to obtain a dense depth map; inputting the sparse depth map, the RGB map and the dense depth map into ResUNeT networks for feature extraction to obtain an affinity map; and carrying out iterative propagation according to the dense depth map and the affinity map to obtain a complement depth map. The depth map complementation method has the advantages of high complementation accuracy and high reasoning speed, and overcomes the defect of insufficient resolution of the depth sensor.

Description

A general depth map completion method and device based on depth information propagation

Technical Field

The present invention relates to the field of image enhancement technology, and in particular to a general depth map completion method and device based on depth information propagation.

Background Art

With the rapid progress in the fields of autonomous driving, robotics, augmented reality, etc. in recent years, the importance of depth information maps has become increasingly prominent. These depth information maps play a vital auxiliary role in the above tasks, enabling these tasks to be completed with excellent results. These depth information maps are usually obtained by using commercial depth sensors, such as structured light sensors, ToF (time of flight) radars, etc. However, commercial sensors have their own limitations. For example, the HDL64E model of Velodyne, a commonly used laser radar in the field of autonomous driving, can only obtain a sparse depth information map with low resolution. The number of effective depth information pixels in the sparse depth information map is only about 5% of the number of pixels in the corresponding RGB image. Although this sparse depth information map can meet some basic simple 3D vision tasks such as obstacle avoidance and moving object detection, it is somewhat incapable of more complex tasks such as autonomous driving. The sparsity of the scanning results of ordinary commercial radars greatly limits their reliability.

In order to overcome the limitations of depth sensors, there have been many studies on using a given sparse depth map and the corresponding RGB image to obtain a dense depth map. This method is called "depth completion". Through depth completion technology, we can recover complete depth information from sparse depth measurements, thereby improving the reliability and accuracy of the depth information map. Most depth completion methods have changed from using graphic imaging operations (such as corrosion and dilation) to fill holes in sparse depth maps, to feeding sparse depth maps into convolutional neural networks and directly predicting dense depth maps.

However, the prediction results of convolutional neural networks will have blurred edges. In order to solve the above problems, the convolutional spatial propagation network (CSPN) was proposed to refine the predicted depth map to obtain sharp depth map edges and more accurate depth information. This method usually predicts the affinity coefficient between each pixel and the surrounding points, and then uses this coefficient to propagate the depth information multiple times to finally obtain a dense depth information map. For this spatial propagation-based depth completion neural network framework, the input of the network is a sparse depth map and the corresponding RGB image, and the output of the network is the initial depth map of the propagation process and the affinity coefficient map used for propagation.

However, this framework is actually a compromise to deal with the sparsity problem of depth map scanning results. The network is responsible for predicting dense depth maps and affinity coefficients at the same time, but both tasks lack direct supervision, resulting in insufficient learning of the neural network for both tasks, which in turn leads to a decrease in the generalization ability of the model, and ultimately leads to low credibility of the completed depth map. The existing spatial propagation neural network structure has a relatively complex design for the affinity generation branch, and uses a large number of propagation times during the propagation process, resulting in a large amount of computation for using such depth completion methods based on spatial propagation, which ultimately leads to a slow inference speed when used.

In the prior art, there is a lack of a depth map completion method with high completion accuracy and fast inference speed that overcomes the insufficient resolution of the depth sensor.

Summary of the invention

In order to solve the technical problems in the prior art that the depth information obtained by the depth sensor is too sparse and the measurement is inaccurate due to insufficient resolution of the depth sensor, and the existing depth completion method based on spatial propagation has low precision and large calculation amount, the embodiment of the present invention provides a general depth map completion method and device based on depth information propagation. The technical solution is as follows:

On the one hand, a general depth map completion method based on depth information propagation is provided, the method is implemented by a general depth map completion device, and the method includes:

Use a depth sensor to collect data on the scene and obtain a sparse depth map; use a color camera to collect data on the scene and obtain an RGB map;

Using a pre-filling method, depth-filling the sparse depth map to obtain a dense depth map;

Inputting the sparse depth map, the RGB map and the dense depth map into the ResUNeT network for feature extraction to obtain an affinity map;

Iterative propagation is performed according to the dense depth map and the affinity map to obtain a completed depth map.

Among them, the pre-filling method is a convolutional spatial propagation network, a non-local spatial propagation network, a dense spatial propagation network or a fully convolutional spatial propagation network.

Wherein, the affinity map includes a first affinity map, a second affinity map and a third affinity map;

The first affinity map is used for structural information completion; the size of the first affinity map is one sixteenth of the size of the dense depth map; the size of the dense depth map is , m is the length of the dense depth map, n is the width of the dense depth map;

The second affinity map is used for detail information completion; the size of the second affinity map is one quarter of the size of the dense depth map;

The third affinity map is used to complete detail information; the size of the third affinity map is the size of the dense depth map.

Wherein, the ResUNeT network includes a feature extraction branch and an affinity graph generation branch;

The feature extraction branch is an encoder-decoder structure; the encoder structure includes 5 convolutional layers; the decoder structure includes 4 deconvolutional layers;

The affinity map generation branch includes a first affinity map generation branch, a second affinity map generation branch and a third affinity map generation branch; the affinity map generation branch includes 2 deconvolution layers and 1 convolution layer.

Optionally, the step of inputting the sparse depth map, the RGB map and the dense depth map into a ResUNeT network for feature extraction to obtain an affinity map comprises:

According to the sparse depth map, the RGB map and the dense depth map, feature extraction is performed through a ResUNeT network to obtain a first image feature, a second image feature and a third image feature;

The first image feature includes a first sparse depth map feature, a first RGB map feature, and a first dense depth map feature; a size of the first image feature is one sixteenth of a size of the dense depth map;

The second image feature includes a second sparse depth map feature, a second RGB map feature, and a second dense depth map feature; the size of the second image feature is one quarter of the size of the dense depth map;

The third image feature includes a third sparse depth map feature, a third RGB map feature and a third dense depth map feature; the size of the third image feature is the size of the dense depth map;

Performing a convolution operation according to the first image feature to obtain a first affinity map;

Performing a convolution operation according to the second image feature to obtain a second affinity map;

A convolution operation is performed according to the third image feature to obtain a third affinity map.

Optionally, performing iterative propagation according to the dense depth map and the affinity map to obtain a completed depth map includes:

Downsampling the dense depth map to obtain a first dense depth map; iteratively propagating the first dense depth map based on the first affinity map to obtain a first completed depth map;

Performing upsampling processing on the first completed depth map to obtain a second dense depth map; and iteratively propagating the second dense depth map based on the second affinity map to obtain a second completed depth map;

The second completed depth map is upsampled to obtain a third dense depth map; based on the third affinity map, the third dense depth map is iteratively propagated to obtain a completed depth map.

On the other hand, a general depth map completion device based on depth information propagation is provided, and the device is applied to a general depth map completion method based on depth information propagation, and the device includes:

The data acquisition module is used to collect data of the scene using a depth sensor to obtain a sparse depth map; and to collect data of the scene using a color camera to obtain an RGB map;

A pre-filling module, configured to perform depth filling on the sparse depth map by using a pre-filling method to obtain a dense depth map;

An affinity map generation module, used for inputting the sparse depth map, the RGB map and the dense depth map into the ResUNeT network for feature extraction to obtain an affinity map;

The depth map completion module is used to perform iterative propagation according to the dense depth map and the affinity map to obtain a completed depth map.

Among them, the pre-filling method is a convolutional spatial propagation network, a non-local spatial propagation network, a dense spatial propagation network or a fully convolutional spatial propagation network.

Wherein, the affinity map includes a first affinity map, a second affinity map and a third affinity map;

The first affinity map is used for structural information completion; the size of the first affinity map is one sixteenth of the size of the dense depth map; the size of the dense depth map is , m is the length of the dense depth map, n is the width of the dense depth map;

The second affinity map is used for detail information completion; the size of the second affinity map is one quarter of the size of the dense depth map;

The third affinity map is used to complete detail information; the size of the third affinity map is the size of the dense depth map.

Wherein, the ResUNeT network includes a feature extraction branch and an affinity graph generation branch;

The feature extraction branch is an encoder-decoder structure; the encoder structure includes 5 convolutional layers; the decoder structure includes 4 deconvolutional layers;

The affinity map generation branch includes a first affinity map generation branch, a second affinity map generation branch and a third affinity map generation branch; the affinity map generation branch includes 2 deconvolution layers and 1 convolution layer.

Optionally, the affinity graph generating module is further used to:

According to the sparse depth map, the RGB map and the dense depth map, feature extraction is performed through a ResUNeT network to obtain a first image feature, a second image feature and a third image feature;

The first image feature includes a first sparse depth map feature, a first RGB map feature, and a first dense depth map feature; a size of the first image feature is one sixteenth of a size of the dense depth map;

The second image feature includes a second sparse depth map feature, a second RGB map feature, and a second dense depth map feature; the size of the second image feature is one quarter of the size of the dense depth map;

The third image feature includes a third sparse depth map feature, a third RGB map feature and a third dense depth map feature; the size of the third image feature is the size of the dense depth map;

Performing a convolution operation according to the first image feature to obtain a first affinity map;

Performing a convolution operation according to the second image feature to obtain a second affinity map;

A convolution operation is performed according to the third image feature to obtain a third affinity map.

Optionally, the depth map completion module is further used to:

Downsampling the dense depth map to obtain a first dense depth map; iteratively propagating the first dense depth map based on the first affinity map to obtain a first completed depth map;

Performing upsampling processing on the first completed depth map to obtain a second dense depth map; and iteratively propagating the second dense depth map based on the second affinity map to obtain a second completed depth map;

The second completed depth map is upsampled to obtain a third dense depth map; based on the third affinity map, the third dense depth map is iteratively propagated to obtain a completed depth map.

On the other hand, a general depth map completion device is provided, the general depth map completion device comprising: a processor; a memory, the memory storing computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the general depth map completion methods based on depth information propagation as described above is implemented.

On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned general depth map completion methods based on depth information propagation.

The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

The present invention proposes a general depth map completion method based on deep information propagation. Through a replaceable pre-filling method, the method proposed by the present invention can obtain accuracy improvement on any pre-filling method. The higher the accuracy of the pre-filling method, the higher the final precision. Therefore, even after the emergence of a new depth completion method, the method proposed by the present invention can be used as a pre-filling method and obtain a more accurate depth completion result on its basis. The present invention proposes a new affinity graph generation method. The affinity graph generated by this method solves the problem that the propagation range is large and the reasoning speed is too slow. Since the affinity of the three stages is generated using feature maps of corresponding scales, the feature maps of these three scales correspond to information from abstract structural information to specific detail information. The overall spatial propagation network framework of the present invention greatly reduces the risk of poor generalization of the network obtained by model training; due to the participation of supervisory information, the difficulty of the learning and training process is reduced. The present invention is a depth map completion method with high completion accuracy and fast reasoning speed that overcomes the insufficient resolution of the depth sensor.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

FIG1 is a flow chart of a general depth map completion method based on depth information propagation provided by an embodiment of the present invention;

FIG2 is a block diagram of a general depth map completion device based on depth information propagation provided by an embodiment of the present invention;

FIG. 3 is a schematic structural diagram of a general depth map completion device provided by an embodiment of the present invention.

DETAILED DESCRIPTION

The technical solution of the present invention is described below in conjunction with the accompanying drawings.

In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and/or" can be both, or it can be either of the two.

In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

In the embodiments of the present invention, sometimes a subscript such as _W1 may be expressed in a non-subscript form such as W1. When the difference is not emphasized, the meanings to be expressed are consistent.

In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

The embodiment of the present invention provides a general depth map completion method based on depth information propagation, which can be implemented by a general depth map completion device, which can be a terminal or a server. As shown in FIG1 , the general depth map completion method based on depth information propagation process flow of the method may include the following steps:

S1. Use a depth sensor to collect data on the scene to obtain a sparse depth map; use a color camera to collect data on the scene to obtain an RGB map.

In one feasible implementation, a sparse depth map of a scene is collected using a depth sensor and a color camera. and RGB images Where m is the length of the sparse depth map image, n is the width of the sparse depth map image, and the subscript S represents sparse.

S2. Use the pre-filling method to perform depth filling on the sparse depth map to obtain a dense depth map.

In a feasible implementation, the present invention proposes a decoupled hierarchical-convolutional spatial propagation network framework (Decoupled Hierarchical-Convolutional Spatial Propagation Network, DH-CSPN) to complete the sparse depth scanning results. This framework is very different from the traditional depth completion network framework based on spatial propagation, such as the convolutional spatial propagation network (CSPN). The depth completion method used in this framework decouples the subsequent spatial propagation process. The initial value of the propagation process (i.e., the initial depth map of depth completion) is now directly obtained using the pre-filling method. In the original CSPN framework, the initial depth map is generated by the network itself.

The initial depth map and affinity map under the CSPN framework are not supervised by supervisory information, which makes the learning difficulty of both of them greater, which is also the direct reason why the model learns poor generalization results. The framework proposed in the present invention can fix the initial dense depth map in the propagation iteration process by pre-filling method, thereby reducing the burden of the convolutional neural network, so that the convolutional neural network focuses on the prediction of affinity, and finally learns an affinity prediction method with stronger generalization ability, and obtains a more accurate depth completion result.

In this step, a dense depth map of the same size as the sparse depth map is obtained. , where the subscript D stands for dense.

Among them, the pre-filling method is a convolutional spatial propagation network, a non-local spatial propagation network, a dense spatial propagation network or a fully convolutional spatial propagation network.

In a feasible implementation, the pre-filling method proposed by the present invention is an arbitrary depth completion method. On the premise of obtaining the completion results of other depth completion methods, the method of the present invention refines the completion results by sending the depth map completed by the method into the depth completion network framework of the present invention, and finally obtains a more accurate depth completion result. This makes the present invention a universal depth completion method, which can be connected to the back end of any depth completion method to perform refined post-processing on the method.

The pre-filling method can use very simple image processing dilation operations, or even no operation, while the pre-filling method of this framework can be replaced with various newly released depth completion methods for more complexity.

S3. Input the sparse depth map, RGB map and dense depth map into the ResUNeT network for feature extraction to obtain the affinity map.

The affinity map includes a first affinity map, a second affinity map and a third affinity map;

The first affinity map is used to complete the structural information; the size of the first affinity map is one sixteenth of the size of the dense depth map; the size of the dense depth map is , m is the length of the dense depth map, n is the width of the dense depth map;

The second affinity map is used to complete the detail information; the size of the second affinity map is one quarter of the size of the dense depth map;

The third affinity map is used to complete the detail information; the size of the third affinity map is the size of the dense depth map.

In one feasible implementation, the sparse depth map , RGB image And the dense depth information map obtained after pre-filling The three images are sent to the backbone network of the Residual U-shaped Network (ResUNet) structure to extract features of the above three images and finally generate three affinity maps of different sizes.

The sizes of the affinity graphs are , , , k is the affinity propagation range, the value is set to 3, and the propagation range of the affinity graph is The affinity map is the correlation coefficient between each pixel on the depth map and its surroundings. Its significance in the actual iterative propagation process of the depth map is to control the coefficient of the depth value propagated in a certain direction during the propagation process. The affinity diagram of For each point in the depth map of size, The coefficient when propagating depth information within the range.

Among them, the ResUNeT network includes a feature extraction branch and an affinity graph generation branch;

The feature extraction branch is an encoder-decoder structure; the encoder structure includes 5 convolutional layers; the decoder structure includes 4 deconvolutional layers;

The affinity map generation branch includes a first affinity map generation branch, a second affinity map generation branch and a third affinity map generation branch; the affinity map generation branch includes 2 deconvolution layers and 1 convolution layer.

In a feasible implementation, the ResUNet structure is designed to be divided into an encoder part and a decoder part. The encoder structure is designed with 5 convolutional layers. The first 4 convolutional layers (conv2-conv5) are the same as the first layer of ResUNet34. The convolution kernel size of the last convolutional layer conv6 is , the step size is 2, the number of channels is 512, and ReLU is used as the activation function.

The decoder structure is designed with 4 deconvolution layers, and the deconvolution kernel size of the deconvolution layer dec5 is , the step size is 2, the number of channels is 256; the convolution kernel size of the deconvolution layer dec4 is , the step size is 2, the number of channels is 128, and the deconvolution layer connects the output of the deconvolution layer dec5 with the output of the convolution layer conv5; the deconvolution kernel size of the deconvolution layer dec3 is , the step size is 2, the number of channels is 64, and the deconvolution layer connects the output of the convolution layer conv4 with the output of the deconvolution layer dec4; the deconvolution kernel size of the deconvolution layer dec2 is , the step size is 2, the number of channels is 64, and the deconvolution layer connects the outputs of the convolution layer conv3 and the deconvolution layer dec3.

Optionally, feature extraction is performed through a ResUNeT network according to the sparse depth map, the RGB map, and the dense depth map to obtain an affinity map, including:

According to the sparse depth map, the RGB map and the dense depth map, feature extraction is performed through the ResUNeT network to obtain the first image feature, the second image feature and the third image feature;

The first image feature includes a first sparse depth map feature, a first RGB map feature, and a first dense depth map feature; the size of the first image feature is one sixteenth of the size of the dense depth map;

The second image feature includes a second sparse depth map feature, a second RGB map feature, and a second dense depth map feature; the size of the second image feature is one quarter of the size of the dense depth map;

The third image feature includes a third sparse depth map feature, a third RGB map feature, and a third dense depth map feature; the size of the third image feature is the size of the dense depth map;

Performing a convolution operation according to the first image feature to obtain a first affinity map;

Performing a convolution operation according to the second image feature to obtain a second affinity map;

A convolution operation is performed according to the third image feature to obtain a third affinity map.

In a feasible implementation, the affinity map is essentially a mapping that converts the initial depth map into a more refined depth map, and the expressiveness of this mapping will naturally affect the accuracy of the final refined result.

According to the concept of dynamic affinity range, the propagation range of the first affinity map and the second affinity map of the present invention is The depth information of the original image can be aggregated within the range of 16 times and 4 times, respectively, because it is performed within the range of 1/16 and 1/4.

The first affinity map, the second affinity map and the third affinity map generated in the three stages correspond to feature maps of three scales respectively, and the feature maps of these three scales correspond to the abstract structural information to the specific detail information respectively. Therefore, in the first stage with lower resolution, the first affinity map of DH-CSPN can pay more attention to the structural information, while the affinity of the third affinity map generated in the final stage can pay more attention to the detail information. The design here is based on dynamic affinity and introduces the concept of weight.

The network framework structure for generating the first affinity graph includes deconvolution layer gd2-dec1, deconvolution layer gd2-dec0 and convolution layer gd2-conf0. The deconvolution kernel size of deconvolution layer gd2-dec1 is , the step size is 2, the number of channels is 128, this layer connects the outputs of the convolution layer conv4 and the deconvolution layer dec4; the deconvolution kernel size of the deconvolution layer gd2-dec0 is , the step size is 1, and the number of channels is ; The convolution kernel size of the convolution layer gd2-conf0 is , the step size is 1, and the number of channels is , Sigmoid is used as the activation function.

The network framework structure for generating the second affinity graph includes a deconvolution layer gd1-dec1, a deconvolution layer gd1-dec0, and a convolution layer gd1-conf0. The deconvolution kernel size of the deconvolution layer gd1-dec1 is , the step size is 1, and the number of channels is , which connects the outputs of the convolution layer conv3 and the deconvolution layer dec3; the deconvolution kernel size of the deconvolution layer gd1-dec0 is , the step size is 1, and the number of channels is ; The convolution kernel size of the convolution layer gd2-conf0 is , the step size is 1, and the number of channels is , Sigmoid is used as the activation function.

The network framework structure for generating the third affinity graph includes a deconvolution layer gd0-dec1, a connection layer concat2, a deconvolution layer gd0-dec0, and a convolution layer gd0-conf0. The deconvolution kernel size of the deconvolution layer gd0-dec1 is , the step size is 2, the number of channels is 64, this layer connects the outputs of the convolution layer conv2 and the deconvolution layer dec2; the connection layer concat2 links the outputs of gd0-dec1, dec1 and concat1; the deconvolution kernel size of the deconvolution layer gd0-dec0 is , the step size is 1, and the number of channels is ; The convolution kernel size of the convolution layer gd0-conf0 is , the step size is 1, and the number of channels is , Sigmoid is used as the activation function.

Since the depth image is downsampled in the initial stage of the subsequent iterative propagation process, resulting in loss of detail information and only retaining structural information, the first affinity map of the present invention will be more focused on this aspect because the 1/16 size depth information map only contains structural information. Similarly, in the 1/1 result of the third stage, the second affinity map generated by the method will be more focused on detail information.

The three-scale structure of the hierarchical design enables the affinity generation and depth value iterative propagation processes in the first two stages to be carried out on a smaller scale, and the corresponding computational complexity is also 1/16 and 1/4 of the original. This makes the method in this paper more advantageous in terms of computational complexity than the Non-local Spatial Propagation Network (NLSPN) and the Dynamic Spatial Propagation Network (DySPN).

S4. Perform iterative propagation based on the dense depth map and affinity map to obtain the completed depth map.

Optionally, iterative propagation is performed according to the dense depth map and the affinity map to obtain a completed depth map, including:

Downsampling the dense depth map to obtain a first dense depth map; iteratively propagating the first dense depth map based on the first affinity map to obtain a first completed depth map;

Upsampling the first completed depth map to obtain a second dense depth map; iteratively propagating the second dense depth map based on the second affinity map to obtain a second completed depth map;

The second completed depth map is upsampled to obtain a third dense depth map; based on the third affinity map, the third dense depth map is iteratively propagated to obtain a completed depth map.

In a feasible implementation manner, after obtaining the first affinity map After the dense depth map is completed, the original size is The length and width of the dense depth map of is downsampled to 1/4 of the original, which can be expressed as . Based on the first affinity graph of the same size and depth map , use the first affinity map to guide the depth map to perform iterative depth propagation. The mathematical expression of the first stage iterative propagation of the depth value is as follows (1):

(1)

Where i is the row position of each pixel in the depth map, j is the column position of each pixel in the depth map, and the subscript N represents the number of iterations. Represents the bitwise multiplication of, (a, b) represents The relative position of each point within the propagation range.

From formula (1), we can see that according to the depth map before iteration Around the corresponding position Points within range, through affinity graph Perform weighted sum calculation to obtain the depth map after iterative propagation of depth values In the first stage of iterative propagation, the iterative propagation process of the depth value on the depth information map of 1/16 original image size will be continued for 6 times, and finally the depth value of each pixel point in the image is obtained. .

According to the results of the first phase of iterative propagation Perform upsampling to double the length and width of the original. .according to and the second affinity map of the corresponding size The depth value is iteratively propagated. The mathematical expression of this second stage propagation process is as follows:

(2)

After 6 iterations, we get .

According to the results of the second phase of iterative propagation Perform upsampling to double the length and width of the original. .according to and the third affinity map of corresponding size The depth value is iteratively propagated. The mathematical expression of this third stage propagation process is as follows (3):

(3)

After 6 iterations, we finally get , the final This is the dense depth map after completion by this method.

In a feasible implementation, the method of the present invention is experimentally verified on the datasets of NYU-Depth V2-New York University various indoor scene datasets and depth completion dataset (Konstruktion Innovationund Technologie in der Industrie, KITTI). The evaluation indicators used are mainly the following six, and their mathematical expressions are shown in the following formula (4):

(4)

in, Represents the predicted depth information graph result, Represents the actual depth information result of the scene. The t value of the indicator is 1, 2, and 3. Among the above indicators, Root Mean Square Error (RMSE), Index Root Mean Square Error (iRMSE), Mean Absolute Error (MAE), Index Mean Absolute Error (iMAE), and Relative Error (REL) are all error measurement indicators, and the smaller they are, the better; Represents the number of pixels within the specified error range, the larger the better.

For the existing method based on spatial propagation, it is transformed into a decoupled framework and the accuracy before and after decoupling is compared. The error comparison results on the NYUv2 dataset are shown in Table 1 (Decoupling result error comparison table).

Table 1

According to Table 1, the methods based on spatial propagation in the prior art are all improved after decoupling, which verifies the effectiveness of the present invention in improving the accuracy of the depth map based on decoupling.

The adaptability of the present invention to different pre-filling methods is tested by changing the pre-filling method. Table 2 (Comparison table of results errors of different pre-filling methods) shows the test results on the NYU dataset after using different pre-filling methods.

Table 2

According to the results in Table 2, it can be seen that the present invention has achieved improved results in all pre-filling methods adopted, which reflects the feasibility of the present invention as a universal depth completion framework.

Using the same ResUnet34 framework, we compare the accuracy and computational complexity of several existing methods based on spatial propagation. Table 3 (Comparison of accuracy and computational complexity of methods based on spatial propagation) is a comparison chart of the effects of various methods based on spatial propagation on the NYUv2 dataset.

Table 3

According to the results in Table 3, it can be seen that the method of the present invention achieves a significant improvement in accuracy with a small amount of calculation, which illustrates the superiority of the present invention in the designed multi-level affinity generated network structure.

The present invention proposes a general depth map completion method based on deep information propagation. Through a replaceable pre-filling method, the method proposed by the present invention can obtain accuracy improvement on any pre-filling method. The higher the accuracy of the pre-filling method, the higher the final precision. Therefore, even after the emergence of a new depth completion method, the method proposed by the present invention can be used as a pre-filling method and obtain a more accurate depth completion result on its basis. The present invention proposes a new affinity graph generation method. The affinity graph generated by this method solves the problem that the propagation range is large and the reasoning speed is too slow. Since the affinity of the three stages is generated using feature maps of corresponding scales, the feature maps of these three scales correspond to information from abstract structural information to specific detail information. The overall spatial propagation network framework of the present invention greatly reduces the risk of poor generalization of the network obtained by model training; due to the participation of supervisory information, the difficulty of the learning and training process is reduced. The present invention is a depth map completion method with high completion accuracy and fast reasoning speed that overcomes the insufficient resolution of the depth sensor.

FIG2 is a block diagram of a general depth map completion device based on depth information propagation according to an exemplary embodiment, and the device is used for a general depth map completion method based on depth information propagation. Referring to FIG2 , the device includes a data acquisition module 210, a pre-filling module 220, an affinity map generation module 230, and a depth map completion module 240. Among them:

The data acquisition module 210 is used to acquire data of the scene using a depth sensor to obtain a sparse depth map; and to acquire data of the scene using a color camera to obtain an RGB map;

A pre-filling module 220 is used to perform depth filling on the sparse depth map by using a pre-filling method to obtain a dense depth map;

An affinity map generation module 230 is used to input the sparse depth map, the RGB map and the dense depth map into the ResUNeT network for feature extraction to obtain an affinity map;

The depth map completion module 240 is used to perform iterative propagation according to the dense depth map and the affinity map to obtain a completed depth map.

Among them, the pre-filling method is a convolutional spatial propagation network, a non-local spatial propagation network, a dense spatial propagation network or a fully convolutional spatial propagation network.

The affinity map includes a first affinity map, a second affinity map and a third affinity map;

The first affinity map is used to complete the structural information; the size of the first affinity map is one sixteenth of the size of the dense depth map; the size of the dense depth map is , m is the length of the dense depth map, n is the width of the dense depth map;

The second affinity map is used to complete the detail information; the size of the second affinity map is one quarter of the size of the dense depth map;

The third affinity map is used to complete the detail information; the size of the third affinity map is the size of the dense depth map.

Among them, the ResUNeT network includes a feature extraction branch and an affinity graph generation branch;

The feature extraction branch is an encoder-decoder structure; the encoder structure includes 5 convolutional layers; the decoder structure includes 4 deconvolutional layers;

The affinity map generation branch includes a first affinity map generation branch, a second affinity map generation branch and a third affinity map generation branch; the affinity map generation branch includes 2 deconvolution layers and 1 convolution layer.

Optionally, the affinity graph generating module 230 is further configured to:

According to the sparse depth map, the RGB map and the dense depth map, feature extraction is performed through the ResUNeT network to obtain the first image feature, the second image feature and the third image feature;

The first image feature includes a first sparse depth map feature, a first RGB map feature, and a first dense depth map feature; the size of the first image feature is one sixteenth of the size of the dense depth map;

The second image feature includes a second sparse depth map feature, a second RGB map feature, and a second dense depth map feature; the size of the second image feature is one quarter of the size of the dense depth map;

The third image feature includes a third sparse depth map feature, a third RGB map feature, and a third dense depth map feature; the size of the third image feature is the size of the dense depth map;

Performing a convolution operation according to the first image feature to obtain a first affinity map;

Performing a convolution operation according to the second image feature to obtain a second affinity map;

A convolution operation is performed according to the third image feature to obtain a third affinity map.

Optionally, the depth map completion module 240 is further configured to:

Downsampling the dense depth map to obtain a first dense depth map; iteratively propagating the first dense depth map based on the first affinity map to obtain a first completed depth map;

Upsampling the first completed depth map to obtain a second dense depth map; iteratively propagating the second dense depth map based on the second affinity map to obtain a second completed depth map;

The second completed depth map is upsampled to obtain a third dense depth map; based on the third affinity map, the third dense depth map is iteratively propagated to obtain a completed depth map.

The present invention proposes a universal depth map completion method based on deep information propagation. Through a replaceable pre-filling method, the method proposed by the present invention can obtain accuracy improvement on any pre-filling method. The higher the accuracy of the pre-filling method, the higher the final precision. Therefore, even after the emergence of a new depth completion method, the method proposed by the present invention can be used as a pre-filling method and obtain a more accurate depth completion result on its basis. The present invention proposes a new affinity map generation method. The affinity map generated by this method solves the problem that the propagation range is large and the reasoning speed is too slow. Since the affinity of the three stages is generated using feature maps of corresponding scales, the feature maps of these three scales correspond to information from abstract structural information to specific detail information. The overall spatial propagation network framework of the present invention greatly reduces the risk of poor generalization of the network obtained by model training; due to the participation of supervisory information, the difficulty of the learning and training process is reduced. The present invention is a depth map completion method with high completion accuracy and fast reasoning speed that overcomes the insufficient resolution of the depth sensor.

FIG3 is a schematic diagram of the structure of a general depth map completion device provided by an embodiment of the present invention. As shown in FIG3 , the general depth map completion device may include the general depth map completion apparatus based on depth information propagation shown in FIG2 . Optionally, the general depth map completion device 310 may include a processor 2001 .

Optionally, the general depth map completion device 310 may further include a memory 2002 and a transceiver 2003 .

The processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

The following is a detailed introduction to the various components of the general depth map completion device 310 in conjunction with FIG3 :

The processor 2001 is the control center of the general depth map completion device 310, and may be a processor or a general term for multiple processing elements. For example, the processor 2001 is one or more central processing units (CPUs), or may be application specific integrated circuits (ASICs), or may be configured to implement one or more integrated circuits of the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

Optionally, the processor 2001 may execute various functions of the general depth map completion device 310 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

In a specific implementation, as an embodiment, the processor 2001 may include one or more CPUs, such as CPU0 and CPU1 shown in FIG. 3 .

In a specific implementation, as an embodiment, the general depth map completion device 310 may also include multiple processors, such as the processor 2001 and the processor 2004 shown in FIG3 . Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and/or processing cores for processing data (such as computer program instructions).

The memory 2002 is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the processor 2001, or may exist independently, and be coupled to the processor 2001 through an interface circuit (not shown in FIG. 3 ) of the universal depth map completion device 310, which is not specifically limited in the embodiment of the present invention.

The transceiver 2003 is used to communicate with a network device or a terminal device.

Optionally, the transceiver 2003 may include a receiver and a transmitter (not shown separately in FIG. 3 ), wherein the receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

Optionally, the transceiver 2003 may be integrated with the processor 2001 or exist independently and be coupled to the processor 2001 via an interface circuit (not shown in FIG. 3 ) of the universal depth map completion device 310 , which is not specifically limited in this embodiment of the present invention.

It should be noted that the structure of the general depth map completion device 310 shown in FIG. 3 does not constitute a limitation on the router, and the actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

In addition, the technical effects of the general depth map completion device 310 can refer to the technical effects of the general depth map completion method based on depth information propagation described in the above method embodiment, which will not be repeated here.

It should be understood that the processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

It should be understood that the term "and/or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and/or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character "/" in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and/or" relationship. Please refer to the context for specific understanding.

In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (7)

1. A general depth map completion method based on depth information propagation, characterized in that the method comprises: Use a depth sensor to collect data on the scene and obtain a sparse depth map; use a color camera to collect data on the scene and obtain an RGB map; Using a pre-filling method, depth-filling the sparse depth map to obtain a dense depth map; Inputting the sparse depth map, the RGB map and the dense depth map into the ResUNeT network for feature extraction to obtain an affinity map; Wherein, the affinity map includes a first affinity map, a second affinity map and a third affinity map; The first affinity map is used for structural information completion; the size of the first affinity map is one sixteenth of the size of the dense depth map; the size of the dense depth map is , m is the length of the dense depth map, n is the width of the dense depth map; The second affinity map is used for detail information completion; the size of the second affinity map is one quarter of the size of the dense depth map; The third affinity map is used for completing detail information; the size of the third affinity map is the size of the dense depth map; Wherein, the ResUNeT network includes a feature extraction branch and an affinity graph generation branch; The feature extraction branch is an encoder-decoder structure; the encoder structure includes 5 convolutional layers; the decoder structure includes 4 deconvolutional layers; The affinity graph generation branch includes a first affinity graph generation branch, a second affinity graph generation branch and a third affinity graph generation branch; the affinity graph generation branch includes 2 deconvolution layers and 1 convolution layer; The step of inputting the sparse depth map, the RGB map and the dense depth map into the ResUNeT network for feature extraction to obtain an affinity map includes: According to the sparse depth map, the RGB map and the dense depth map, feature extraction is performed through a ResUNeT network to obtain a first image feature, a second image feature and a third image feature; The first image feature includes a first sparse depth map feature, a first RGB map feature, and a first dense depth map feature; a size of the first image feature is one sixteenth of a size of the dense depth map; The second image feature includes a second sparse depth map feature, a second RGB map feature, and a second dense depth map feature; the size of the second image feature is one quarter of the size of the dense depth map; The third image feature includes a third sparse depth map feature, a third RGB map feature and a third dense depth map feature; the size of the third image feature is the size of the dense depth map; Performing a convolution operation according to the first image feature to obtain a first affinity map; Performing a convolution operation according to the second image feature to obtain a second affinity map; Performing a convolution operation according to the third image feature to obtain a third affinity map; Iterative propagation is performed according to the dense depth map and the affinity map to obtain a completed depth map. 2. According to claim 1, the general depth map completion method based on depth information propagation is characterized in that the pre-filling method is a convolutional spatial propagation network, a non-local spatial propagation network, a dense spatial propagation network or a fully convolutional spatial propagation network. 3. The general depth map completion method based on depth information propagation according to claim 1, characterized in that the iterative propagation according to the dense depth map and the affinity map to obtain the completed depth map comprises: Downsampling the dense depth map to obtain a first dense depth map; iteratively propagating the first dense depth map based on the first affinity map to obtain a first completed depth map; Performing upsampling processing on the first completed depth map to obtain a second dense depth map; iteratively propagating the second dense depth map based on the second affinity map to obtain a second completed depth map; The second completed depth map is upsampled to obtain a third dense depth map; based on the third affinity map, the third dense depth map is iteratively propagated to obtain a completed depth map. 4. A general depth map completion device based on depth information propagation, wherein the general depth map completion device based on depth information propagation is used to implement the general depth map completion method based on depth information propagation according to any one of claims 1 to 3, characterized in that the device comprises: The data acquisition module is used to collect data of the scene using a depth sensor to obtain a sparse depth map; and to collect data of the scene using a color camera to obtain an RGB map; A pre-filling module, configured to perform depth filling on the sparse depth map by using a pre-filling method to obtain a dense depth map; An affinity map generation module, used for inputting the sparse depth map, the RGB map and the dense depth map into the ResUNeT network for feature extraction to obtain an affinity map; Wherein, the affinity map includes a first affinity map, a second affinity map and a third affinity map; The first affinity map is used for structural information completion; the size of the first affinity map is one sixteenth of the size of the dense depth map; the size of the dense depth map is , m is the length of the dense depth map, n is the width of the dense depth map; The second affinity map is used for detail information completion; the size of the second affinity map is one quarter of the size of the dense depth map; The third affinity map is used for completing detail information; the size of the third affinity map is the size of the dense depth map; Wherein, the ResUNeT network includes a feature extraction branch and an affinity graph generation branch; The feature extraction branch is an encoder-decoder structure; the encoder structure includes 5 convolutional layers; the decoder structure includes 4 deconvolutional layers; The affinity graph generation branch includes a first affinity graph generation branch, a second affinity graph generation branch and a third affinity graph generation branch; the affinity graph generation branch includes 2 deconvolution layers and 1 convolution layer; Wherein, the affinity graph generation module is further used for: According to the sparse depth map, the RGB map and the dense depth map, feature extraction is performed through a ResUNeT network to obtain a first image feature, a second image feature and a third image feature; The first image feature includes a first sparse depth map feature, a first RGB map feature, and a first dense depth map feature; a size of the first image feature is one sixteenth of a size of the dense depth map; The second image feature includes a second sparse depth map feature, a second RGB map feature, and a second dense depth map feature; the size of the second image feature is one quarter of the size of the dense depth map; The third image feature includes a third sparse depth map feature, a third RGB map feature and a third dense depth map feature; the size of the third image feature is the size of the dense depth map; Performing a convolution operation according to the first image feature to obtain a first affinity map; Performing a convolution operation according to the second image feature to obtain a second affinity map; Performing a convolution operation according to the third image feature to obtain a third affinity map; The depth map completion module is used to perform iterative propagation according to the dense depth map and the affinity map to obtain a completed depth map. 5. The general depth map completion device based on depth information propagation according to claim 4, characterized in that the depth map completion module is further used to: Downsampling the dense depth map to obtain a first dense depth map; iteratively propagating the first dense depth map based on the first affinity map to obtain a first completed depth map; Performing upsampling processing on the first completed depth map to obtain a second dense depth map; iteratively propagating the second dense depth map based on the second affinity map to obtain a second completed depth map; The second completed depth map is upsampled to obtain a third dense depth map; based on the third affinity map, the third dense depth map is iteratively propagated to obtain a completed depth map. 6. A general depth map completion device, characterized in that the general depth map completion device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 3 is implemented. 7. A computer-readable storage medium, characterized in that program codes are stored in the computer-readable storage medium, and the program codes can be called by a processor to execute the method according to any one of claims 1 to 3.