CN115131569A - An Unguided Depth Completion Method for Custom Kernel Dilation - Google Patents

Abstract

A custom kernel-dilated unguided depth completion method for lidar depth image processing. The algorithm does not depend on any training data. The present invention adopts a greedy algorithm to complete the input sparse depth map into a dense depth map through depth inversion, pixel hole filling and kernel expansion and other operations in order from small to large. The present invention can run in real time on a 3.8GHz CPU and does not require any additional GPU hardware, which enables it to be deployed in embedded systems as a preprocessing step for more complex tasks such as SLAM or 3D object detection. Meanwhile, considering the wide application of guided depth completion, this method can be used as a preprocessing step for guided depth completion. Or directly apply it to occasions where guided depth completion is not applicable, such as dark nights and other environments with poor lighting.

Description

An Unguided Depth Completion Method for Custom Kernel Dilation

technical field

The invention belongs to the technical field of image processing, and in particular relates to a non-guided depth completion method for self-defined kernel expansion.

Background technique

Lidar is a measurement tool widely used in the field of driverless cars and robot vision technology. It can output radar point cloud images of the surrounding environment and reflect the 3-dimensional depth information of the environment. However, due to the limited number of point clouds scanned by lidar per unit period, the depth point cloud images output by lidars on the market are often not dense enough to meet people's application needs. This makes it necessary to use other methods to densify the radar point cloud image output by the lidar, that is, to achieve depth completion.

In recent years, with the development of deep learning and artificial intelligence, more and more people use deep learning as a solution for deep completion of radar data. At present, the mainstream method on the market is to use the sparse radar point cloud image and the color RGB image calibrated with it, and use the relevant deep learning model to realize the depth completion of the radar data.

This approach often requires training on a large dataset and is extremely demanding on the GPU used for testing. At the same time, due to the reliance on calibrated color RGB images, this type of depth completion often has extremely high requirements for lighting, and the use of color RGB images that do not have sufficient color discrimination in the dark will often greatly reduce the completion effect.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a depth completion method that does not depend on any training data, does not use a deep learning model, and does not require an additional GPU, and can restore a complete depth map from a depth map scanned by a lidar.

For this purpose, the present invention adopts the following technical solutions:

The present invention proposes a non-guided depth completion method for self-defined kernel expansion, which includes the following specific steps:

Step S0, obtaining a sparse point cloud image of the target scene through lidar detection and collection;

Take these sparse point cloud images as input, and use classical image processing methods for them, including depth coding inversion, 5×5 diamond kernel expansion, 5×5 full kernel closing operation, and 13×13 full kernel expansion. , pixel extension, 27×27 full kernel dilation, median blur, Gaussian blur, depth coding restoration operations.

Step S1, cover the smaller pixel value with the larger pixel value. The data depth value range used in the present invention is between 0 and 250 meters, and the position without valid depth value is filled with 0. Perform the inversion operation on the effective depth, i.e.

X _inversion = 270.0-K _input

This way, there is a 20m buffer between the effective depth and the null value.

In step S2, the expansion operation is performed with 5×5 diamond-shaped cores.

Step S3, perform a closing operation on the small holes with a 5×5 full check.

After the previous steps, there are still some small to medium sized holes in the depth map that are not front-filled. To fill these holes, step S4 first computes a mask of empty pixels, and then performs a 13×13 full-kernel dilation operation.

Consider some taller objects, such as trees, buildings, etc., all reaching the top of the LiDAR point cloud. In order to fill in the depths of these objects, step S5 extends the pixel values at the top of each column upwards.

Step S6, fill the unfilled large holes, and use a 27×27 full kernel l to perform the expansion operation to fill all the depth values that are still empty, while the effective depth values of other positions remain unchanged.

Step S7, firstly use a 3×3 full-kernel median blur to remove outliers at local edges. Then, a Gaussian blur is performed with a 3×3 full kernel to smooth the local plane and smooth the edges of sharp objects.

Step S8, first divide the original depth image into several image blocks of 8×8 size, and use the formula

x _ij =R _ij X

Obtain the mapping relationship between the image block and the original image, where x _ij is the vector representation of the image block at the original image (i, j), X represents the depth value matrix of the original image, and R _ij represents the block x _ij extracted from the image X matrix operator.

Select the columns of the standard discrete cosine transform dictionary so that the selected column is most relevant to the current redundant vector during each iteration, subtract the relevant part from the original vector and iterate repeatedly until the number of iterations reaches a certain sparsity, stop iterate.

At this time, the optimal sparse coefficient satisfies the public

In the formula, α ^k represents the optimal sparse representation coefficient of the approximate signal, and k represents the sparsity of the final iteration. b is the expanded vector representation of the depth information of the image block, D is the standard discrete cosine transform dictionary, and M is the mask matrix of the image block.

Use the formula

表示步骤S8最终获取的深度信息矩阵，

为最终更新后的字典，

为最终的稀疏表示矩阵。R_ij表示从图像X中提取出8×8数据块x_ij的矩阵算子。The final inpainted depth map can be obtained. in,

represents the depth information matrix finally obtained in step S8,

is the final updated dictionary,

is the final sparse representation matrix. R _ij represents the matrix operator that extracts the 8×8 data block x _ij from the image X.

Step S9, performing weighting processing on the depth information obtained by the two paths in the above steps:

表示两条路径加权后的深度信息，

表示主路径获取的深度信息，

表示分路径获取的深度信息。in,

represents the weighted depth information of the two paths,

Indicates the depth information obtained by the main path,

Indicates the depth information obtained by sub-path.

Step S10, restore the inverted depth value used in the previous steps to the original depth coding, that is, use the same inversion formula as step S1:

X _inversion = 270.0 - X _input .

Step S11, output the complete depth map.

The beneficial effects that the present invention has are:

(1) The present invention only uses the classical image processing method, does not use any deep learning algorithm, does not need to rely on any existing deep learning model, and does not require other data sets for training. Therefore, when contacting the present invention for the first time, only the corresponding data point cloud image for completion is required to realize depth completion. There is no need for model training and testing that deep learning solutions typically require.

(2) All the pixels completed by the present invention are all based on the addition and speculation of the original lidar point cloud, and do not need to use the calibrated color RGB image, so it can be applied to the depth change of different scenes and has higher robustness.

(3) The present invention has a simple network structure, does not introduce more test parameters, does not require any pre-training on other data sets, and does not have a complex post-processing network. Compared with other complex depth completion methods, it has Stronger real-time performance, and extremely low requirements for test hardware.

In general, the network structure of the present invention is simple, does not have too many strict requirements for input data, and can quickly output a dense depth map in a short period of time. It has stronger real-time performance and has the fastest test speed compared with other published depth completion algorithms under the same test conditions. At the same time, since the present invention does not need to use color RGB images for auxiliary completion, and because the requirements for testing hardware are not high, the present invention can be applied to more depth completion scenarios.

Description of drawings

FIG. 1 is a flowchart of an unguided depth completion method for custom kernel expansion in an embodiment of the present invention.

FIG. 2 is an example of a model in an embodiment of the present invention.

FIG. 3 is a schematic diagram of different kernels used for comparison in an embodiment of the present invention.

FIG. 4 is a visualization of qualitative results of the method on three samples in the KITTI test set in the embodiment of the present invention.

FIG. 5 is a schematic diagram of qualitative results in the embodiment of the present invention.

Detailed ways

The technical solutions of the present invention will be further described in detail below with reference to the accompanying drawings.

As shown in the flow chart of Fig. 1, the embodiment implemented according to the complete method of the present invention and its implementation process are as follows:

Taking the KITTI Depth Completion known dataset as a known dataset and complementing the sparse depth map as an example, the idea and specific implementation steps of the unguided depth completion of custom kernel expansion are described.

The sparse depth map and the ground-truth depth map of the embodiment are all from the known dataset of KITTI Depth Completion.

Step 1: Use the known training set of KITTIDepthCompletion to perform step 2 on the depth map provided by the training set.

Step 2: Perform depth completion on the sparse depth map in the training set described in Step 1. The main sparse processing mechanism used is the OpenCV morphological transform operation, which overlays smaller pixel values with larger pixel values. When considering the raw KITTI depth map data, the closer pixel values are close to 0m, while the farther pixel values are at most 250m. However, the null pixel value is also 0m, which prevents using native OpenCV operations without modification. Applying an expansion operation to the original depth map will cause larger distances to cover smaller distances, thus losing edge information for closer objects. To fix this, the valid (non-empty) pixel depth is inverted according to the X _inversion = 270.0-X _input , also creating a 20m buffer between valid and empty pixel values. This inversion algorithm preserves closer edges when applying spread operations. The 20m buffer is used to offset the effective depth to mask invalid pixels in subsequent operations.

Step 3: The empty pixels closest to the valid pixels are filled first, since these pixels are most likely to share a close depth value with the valid depth. Considering the sparsity of projected points and the structure of lidar scan lines, a custom kernel is designed for the initial expansion of each effective depth pixel. The kernel shape is designed such that pixels that are most likely to have the same value are upscaled to the same value. The four kernel shapes shown in Figure 3 were implemented and evaluated. According to the experimental results, a 5 × 5 diamond kernel is used to dilate all valid pixels.

Step 4: After the initial expansion step, there are still many holes in the depth map. Since these regions do not contain depth values, consider the structure of objects in the environment and note that nearby patches of dilated depth can connect to form the edges of objects. A 5×5 full-kernel morphological closure algorithm is used to close small holes in the depth map. This operation uses a binary kernel, preserving object edges. This step is used to concatenate nearby depth values, which can be viewed as a set of 5×5 pixel planes from farthest to nearest.

Step 5: Some small to medium sized holes in the depth map were not filled by the first two dilation operations. To fill these holes, a mask for empty pixels is first computed, followed by a 13×13 full-kernel dilation operation. This operation will only fill empty pixels, while keeping the previously computed valid pixels unchanged.

Step 6: To account for tall objects that extend above the top of the lidar point, such as trees, poles, and buildings, the top values along each column are extrapolated to the top of the image, providing a denser depth map output.

Step Seven: The final fill step will deal with larger holes in the depth map that are not fully filled. Since these regions do not contain points and do not use image data, the depth values for these pixels are inferred from nearby values. An expansion operation with a 27×27 full kernel is used to fill in any remaining empty pixels, while keeping valid pixels unchanged.

Step 8: After applying the previous steps, a dense depth map is finally obtained. In this depth map, outlier (extreme) values are by-products (unnecessary values) of the dilation operation. To remove these outliers, a 3×3 kernel median filter is used. This denoising step is very important because it removes outliers while preserving local edges. Finally, a 3×3 Gaussian filter is used to smooth local planes and refine the edges of sharp parts.

Step 9: After completing the depth inversion in Step 2, divide the inverted original depth image into several 8×8 image blocks, and use the formula

x _ij =R _ij X

Obtain the mapping relationship between the image block and the original image, where x _ij is the vector representation of the image block at the original image (i, j), X represents the depth value matrix of the original image, and R _ij represents the block x _ij extracted from the image X matrix operator.

Select the columns of the standard discrete cosine transform dictionary so that the selected column is most relevant to the current redundant vector during each iteration, subtract the relevant part from the original vector and iterate repeatedly until the number of iterations reaches a certain sparsity, stop iterate. The core algorithm steps are as follows:

支撑集

Initialization: k=0, α ⁰ =0, residual

support set

找到最小的j₀，使得∈(j₀)≤∈(j),

更新支撑集S^k＝S^k-1∪{j₀}Iterative Process: Calculating Errors

Find the smallest j ₀ such that ∈(j ₀ )≤∈(j),

Update support set ^Sk = ^Sk-1 ∪{j ₀ }

Update sparse representation coefficients: For a given support set S ^k , find the optimal sparse representation coefficients of the approximate signal

Update residuals:

Iteration: if ‖r ^k ‖ ₂ ≤∈(j), stop iterative update; otherwise repeat

The final sparse representation solution α ^k

和稀疏表示矩阵

对图块进行取平均值的操作。Through iteration, the number of iterations J and the error threshold ∈ satisfying certain conditions are obtained, and the final updated dictionary is obtained

and sparse representation matrix

Perform an average operation on the tiles.

Get the depth map with the final inpainting.

Step 10: Perform weighting processing on the depth maps obtained by the two paths, the formula is as follows:

表示两条路径加权后的深度信息，

表示主路径获取的深度信息，

表示分路径获取的深度信息。in,

represents the weighted depth information of the two paths,

Indicates the depth information obtained by the main path,

Indicates the depth information obtained by sub-path.

Finally, restore the original depth encoding from the reversed depth values used in the previous steps of the algorithm.

It can be simply calculated as:

X _inversion = 270.0 - X _input .

Step 11, output the full depth map.

A depth-completion test is performed, which consists of a large number of lidar scans projected into image coordinates to form a depth map. Using the front camera calibration matrix to project the lidar points onto the image coordinates, a sparse depth map of the same size as the RGB image is obtained. Sparsity is due to the lidar data being at a much lower resolution than the image space into which it is projected Only the bottom two-thirds of the depth map contains points (pixels) due to the angle of the lidar scan lines The sparseness of the points in the bottom region of the depth map Degrees are between 5-7%. Corresponding RGB images are also provided for each depth map, but are not used by the unguided depth completion algorithm. The provided 1000-image validation set will be used for the evaluation of all experiments, and the final results of this 1000-image test set will be submitted and evaluated by KITTI's test server. The performance of the algorithm and baselines is evaluated using the Inverse Root Mean Square Error (iRMSE), Inverse Mean Square Error (iMAE), Root Mean Square Error (RMSE), and Mean Error (MAE) metrics.

Table 1: The performance of the three most commonly used depth completion methods and this method is compared on the KITTI depth completion test set. Results are generated by KITTI's evaluation server.

It can be seen from the table that this method improves the root mean square error (RMSE) and the average error (MAE) of MAE by 131.29mm and 113.54mm, respectively, over the runner-up NN+CNN algorithm on the KITTI dataset. This corresponds to an average error difference of 11cm in the final point cloud result, which is important for accurate 3D object localization, obstacle avoidance and SLAM (even for localization and map building).

Table 2: Influence of dilation kernel shape and size on algorithm performance.

The size of the expanded nucleus Root mean square error (mm) Average error (mm) 3×3 1649.97 367.06 5×5 1545.85 349.45 7×7 1720.79 430.82 shape of the expanded nucleus Root mean square error (mm) Average error (mm) plastic 1545.85 349.45 ring 1528.45 342.49 cruciform 1521.95 333.94 diamond 1512.18 333.67

The design of the initial dilation kernel has a great impact on the performance of the algorithm. To find the optimal dilation kernel, a full kernel size varies between 3×3, 5×5 and 7×7. A 7×7 kernel was found to expand depth values beyond their actual area of influence, while a 3×3 kernel did not have enough pixels to expand, so that the boundaries were connected by subsequent hole closing operations. Table 2 shows that the 5×5 kernel provides the lowest rms error. Using the results of the kernel size experiments, the design space of the 5 × 5 binary kernel shape is explored. The full kernel was used as the baseline and compared with circular, cross and diamond kernel shapes. The shape of the dilation kernel determines the initial area of effect for each pixel. Table 2 shows that the diamond kernel provides the lowest rms error. The diamond core shape retains the general outline of the rounded edges and is large enough that the edges are joined by the next hole closing operation.

Table 3: Filtering effect.

nuclear Root mean square error (mm) Average error (mm) Running time (s) no filtering 1512.18 333.67 0.007 bilateral filtering 1511.80 334.12 0.011 Intermediate filtering 1461.54 323.34 0.009 Intermediate plus bilateral filtering 1456.69 328.02 0.014 Gaussian filter 1360.06 310.39 0.008 Intermediate Gaussian filter 1350.93 305.35 0.011

The design of the intermediate filter is to remove salt and pepper noise, making it effective in removing abnormal depth values. This operation increases the running time of 2ms, but it can reduce the value of the root mean square error and the average error. Bilateral filtering preserves local structure by only filtering nearby pixels with similar values, with minimal impact on the evaluated RMSE and mean error metrics, but adds 4ms to the runtime. Due to the Euclidean computation of the root mean square error metric, Gaussian filtering significantly reduces the root mean square error by minimizing the effect of outlier pixel depth. Gaussian filtering also ran the fastest, adding only 1 millisecond to the average running time. The final present method uses a combined median and Gaussian filtering, as this combination has been shown to provide the lowest rms error.

It can be seen that a depth completion method proposed in this embodiment takes a sparse depth map as input and outputs a dense depth map. Only traditional image processing techniques are used and no training is required, making it robust to overfitting. We show that the image processing-based algorithm provides fairly advanced results on the KITTI depth completion benchmark, outperforming several deep learning-based methods. This method can run in real-time on CPUs above 3.8GHz and does not require any additional GPU hardware, making it a strong competition for deployment on embedded systems as a preprocessing step for more complex tasks such as SLAM or 3D object detection By. In the end, this work is not intended to diminish the power of deep learning systems, but rather to shed light on current research trends in which classical approaches have not been carefully considered for comparison, but if properly designed, they can be powerful baseline.

The above descriptions are only the preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above-mentioned embodiments, but any equivalent modifications or changes made by those of ordinary skill in the art based on the contents disclosed in the present invention should be included in the within the scope of protection described in the claims.

Claims (10)

1. a non-guided depth completion method of self-defined kernel expansion, is characterized in that: comprise the following steps: Step S0, the sparse point cloud data image collected by the lidar is used as the input of the algorithm; Step S1, perform inversion processing according to the depth information contained in the real depth point cloud image; Step S2, use custom kernel expansion to fill the empty pixels closest to the effective pixels; use the custom kernel as the initial expansion of each effective depth pixel; the design of the kernel shape makes the most probable pixels with the same value enlarged to the same value ; Step S3, take the hole closing operation, connect the adjacent depth values, and fill the small holes; In step S4, the depth map obtained in step S3 is further filled with small holes, the empty pixel mask is calculated first, and then full-kernel expansion is performed to fill in the unfilled small-to-medium-sized holes in the first two steps of expansion operations; Step S5, performing an extension operation on the top pixel value of each column, so that the empty pixels above the image are completely filled; Step S6, the depth map obtained in step S5 is filled with large holes, and the remaining empty pixels are filled with large-size kernel expansion; In step S7, Gaussian blur and median blur are performed on the dense depth map obtained in step S6 to remove outliers and maintain local boundaries; Step S8, perform depth restoration based on sparse representation on the depth map obtained in step S2, and use a discrete cosine transform dictionary to obtain a dense depth map; Step S9, weighting and integrating the depth maps obtained in steps S7 and S8 to obtain the final depth map; Step S10, take depth inversion again, restore the inverted depth value used in the previous steps to the original depth coding; Step S11, output the complete depth map. 2. the non-guided depth completion method of a kind of self-defined kernel expansion according to claim 1, it is characterized in that: step S1 first carries out the operation of filling 0 on the position that does not have effective depth value on the depth image; Perform an inversion operation on all depth codes on , calculated as: X _inversion = 270.0 - X _input Among them, X _inversion refers to the depth information after data inversion, and X _input refers to the depth information input in this link, and the unit is meters. 3 . The non-guided depth completion method for self-defined kernel expansion according to claim 1 , wherein the self-defined kernel used in step S2 is a 5×5 diamond kernel. 4 . 4 . The non-guided depth completion method for self-defined kernel expansion according to claim 1 , wherein the self-defined kernel used in step S3 is a 5×5 full kernel. 5 . 5 . The non-guided depth completion method for self-defined kernel expansion according to claim 1 , wherein the self-defined kernel used in step S4 is a 13×13 full kernel. 6 . 6 . The non-guided depth completion method for self-defined kernel expansion according to claim 1 , wherein the self-defined kernel used in step S6 is a 27×27 full kernel. 7 . 7 . The non-guided depth completion method for self-defined kernel expansion according to claim 1 , wherein the self-defined kernel used in step S7 is a 3×3 full kernel. 8 . 8. The non-guided depth completion method for self-defined kernel expansion according to claim 1, characterized in that: in the depth restoration method based on sparse representation adopted in step S8, the original depth map is divided into several 8× 8 size image blocks, using the formula: x _ij =R _ij X Obtain the mapping relationship between the image block and the original image, where x _ij is the vector representation of the image block at the original image (i, j), X represents the depth value matrix of the original image, and R _ij represents the block x _ij extracted from the image X The matrix operator of ; Data repair based on discrete cosine transform dictionary, and its sparse representation coefficients are updated iteratively

Among them, α ^k represents the optimal sparse representation coefficient of the approximate signal, k represents the sparsity of the final iteration; b is the vector representation of the expanded depth information of the image block, D is the standard discrete cosine transform dictionary, and M is the mask matrix of the image block . 9. the non-guided depth completion method of a kind of self-defined kernel expansion according to claim 8, is characterized in that: in step S8, the depth information expression finally obtained is:

in,

represents the depth information matrix finally obtained in step S8,

is the final updated dictionary,

is the final sparse representation matrix. 10. The non-guided depth completion method of a self-defined kernel expansion according to claim 1, characterized in that: in the path integration adopted in step S9, the depth information obtained by the two paths is weighted:

in,

represents the weighted depth information of the two paths,

Indicates the depth information obtained by the main path,

Indicates the depth information obtained by sub-path.

Images