CN106780588A - A kind of image depth estimation method based on sparse laser observations - Google Patents

Abstract

The method discloses an image depth estimation method based on sparse laser observation, and the method proposes to use the sparse observation of single-line or multi-line laser to realize the depth-dense reconstruction based on monocular image. The deep neural network is trained by constructing the reference depth map and the residual depth map, which makes full use of the sparse partial observation depth information. Compared with the method of depth estimation using only monocular images, the method of the present invention has obvious advantages.

Description

A Method of Image Depth Estimation Based on Sparse Laser Observation

technical field

The invention relates to the field of scene depth estimation, in particular to a scene dense depth estimation method based on monocular images and sparse lasers.

Background technique

Based on rich experience and continuous learning, human beings also have the ability to estimate the distance of objects in the image from monocular images, that is, the ability to estimate depth to a certain extent. In recent years, machine learning methods have also made remarkable progress in imitating the depth estimation ability of humans, especially data-driven deep learning techniques. This technique avoids the manual feature design process, learns features based on the original monocular RGB image, and outputs a prediction of the corresponding depth of the image.

Eigen et al. first proposed deep learning-based monocular depth estimation. They constructed a two-stage depth estimation network. The first stage generates a rough estimate and the second stage performs fine-tuning. Subsequently, they extended this work to simultaneously estimate scene depth, depth normal vectors, and scene semantics, and verified that estimating depth normal vectors and semantic information simultaneously can help improve the performance of scene depth estimation. Liu et al. discussed the depth estimation combined with deep learning and conditional random fields, performed superpixel segmentation on images, and optimized conditional random fields for all superpixels. Li and Wang respectively extended it, and optimized it layer by layer from the superpixel level to the pixel level through the layered conditional random field.

Although these methods demonstrate the possibility of estimating depth from monocular images, in fact the monocular images themselves are scale-informed. Eigen et al. also mentioned that there may be a global bias in depth estimation based on monocular images.

Contents of the invention

The purpose of the present invention is to combine sparse single-line laser information to estimate the dense depth of the image, so as to reduce the global deviation of scene depth estimation and obtain scene depth estimation with higher reliability.

In order to achieve the above purpose, the present invention is based on a deep learning method, using monocular images and sparse single-line lasers as input, autonomously learning features and obtaining dense depth estimation. The specific steps of the training process are as follows:

A depth image estimation method based on sparse laser observation is characterized in that it comprises the following steps:

Step 1, in order to densify the sparse single-line laser information, the sparse laser includes single-line laser and multi-line laser, wherein the single-line laser in the sparse laser is used to construct the reference depth map and the residual depth map, and the single-line laser in the three-dimensional space Each laser point is stretched in the direction perpendicular to the ground to obtain a reference depth plane perpendicular to the ground; according to the calibration information of the monocular camera and the single-line laser, the reference depth plane obtained in the three-dimensional space is projected to the monocular camera to obtain On the image plane of the image, a reference depth map corresponding to the image is obtained, and the absolute depth map obtained by the depth sensor is compared with the reference depth map to obtain a residual depth map;

Step 2: Use the monocular image acquired by the monocular camera and the reference depth map obtained as described in step 1 as training data, and train the machine neural network to estimate the corresponding residual depth map;

Step 3: Add the residual depth map estimated by the convolution neural network to the reference depth map to obtain the estimated absolute depth map, which is called the absolute depth estimation map, and further construct the optimized convolution neural network on this basis to shrink The difference between the absolute depth estimation map and the absolute depth map obtained by the depth sensor; the optimized convolutional neural network and the convolutional neural network described in step 2 for estimating the residual depth can be superimposed together for end-to-end Optimization, that is, input a monocular image and a reference depth map, and output an optimized absolute depth estimation map.

On the basis of above-mentioned technical scheme, the present invention can also adopt following further technical scheme:

The absolute depth estimation map obtained by the end-to-end output of the deep neural network is fused with the sparse laser depth map through the conditional random field, so as to confirm that there is a single-line laser observation in the absolute depth estimation map. The depth value and the depth value of the laser observation are consistent.

In step 2, the method of training the convolutional neural network to estimate the corresponding residual depth map is as follows: the value of the residual depth of each pixel on the depth residual map to be fitted is discretized to several natural number values, and The categorical form implements depth estimation of the residual depth.

Due to the adoption of the technical solution of the present invention, the beneficial effects of the present invention are: the present invention can combine some sparse real depth observations, such as single-line laser radar, to obtain more accurate depth estimation, and the present invention can reduce the global deviation of scene depth estimation , to obtain a more reliable scene depth estimate.

Description of drawings

Figure 1a is the input monocular image;

Figure 1b is an example of a desired estimated depth image;

Figure 2a is sparse laser observation;

Figure 2b is a reference depth map;

Figure 2c Example of residual depth map;

Figure 3a is the real depth image;

Figure 3b is the depth estimation before optimization;

Figure 3c shows the optimized depth estimation.

detailed description

In order to better understand the technical solution of the present invention, further description will be made below in conjunction with the accompanying drawings. Figure 1 shows an example of depth estimation. The input is the monocular image shown in Figure 1a, and it is required to estimate the depth of the scene shown in Figure 1b.

Step 1: Construct a reference depth map and a residual depth map based on the single-line laser. Figure 2a shows the known single-line laser information in Figure 1, it can be seen that the single-line laser information is very sparse and limited. In order to densify the sparse single-line laser information, each laser point is stretched in the direction perpendicular to the ground in three-dimensional space to obtain a reference depth plane perpendicular to the ground. According to the calibration information of the monocular camera and the single-line laser, the reference depth plane obtained in three dimensions is correspondingly drawn on the image, and a dense reference depth map corresponding to the image is obtained, as shown in Figure 2b. The difference between the real depth map and the reference depth map is obtained to obtain the residual depth map, as shown in Figure 2c.

Step 2. Based on deep learning, the residual depth map is fitted with the monocular image and the reference depth map as input. Discretize the residual depth value of each pixel to several integer values to achieve depth estimation in a classification form. A deep neural network in the form of full convolution is constructed to realize the estimation of the depth value category on each pixel. In order to obtain better fitting performance and larger capacity, the 50-layer Deep Residual Network proposed by He et al. is used, and the network trained on ImageNet is used as the initial value for training.

Step 3: Add the residual depth map estimated by the network to the reference depth map to obtain the estimated real depth map, and further construct an optimization network on this basis to reduce the gap between the estimated real depth map and the actual real depth map. difference. The optimized network can be superimposed with the residual estimation network for end-to-end optimization. Figure 2 shows the comparison of depth ground truth, pre-optimization depth estimation and post-optimization depth estimation.

The present invention verifies the effectiveness of the method by performing experiments on the NYUD2 data set. NYUD2 is an indoor RGB-D dataset. This method simulates and generates single-line laser data on RGB-D data. The main advantage of the present invention lies in the generation and estimation of the reference depth map and the residual depth map. Therefore, the experiment compares under the same neural network structure, only using RGB as input to predict the depth to estimate the real depth (Scheme 1), using RGB and reference depth map as input to estimate the real depth (Scheme 2), and using RGB and reference depth map to estimate The residual depth map further obtains the result of the real depth map (Scheme 3), and compares it with the current world-leading depth estimation method based on monocular images. The specific comparison results are shown in Table 1.

In order to comprehensively evaluate the effect of depth estimation, Table 1 uses six metrics. Let the estimated depth of each pixel be The real depth value y, T is the collection of all pixels, and the six metrics are as follows:

1. Absolute relative error (rel),

2. Average Log error (log10),

3. Mean square mean error (rms),

4. Three threshold accuracy rates (δ _i ), satisfying the conditions of Occupy

The ratio of all pixels.

It can be seen from the results in Table 1 that the performance of depth estimation can be improved to a certain extent after laser densification as input directly, and the performance of depth estimation can be further improved by residual estimation and subsequent optimization. Compared with other world-leading monocular image depth estimation algorithms, this method has obvious advantages in various indicators.

Table 1 Comparison of depth estimation of NYUD2 dataset.

The above-mentioned embodiment is an illustration of the present invention, not a limitation of the present invention, and any solution after a simple transformation of the present invention belongs to the protection scope of the present invention.

Claims (3)

1. a depth image estimation method based on sparse laser observation, is characterized in that it comprises the steps: Step 1, in order to densify the sparse single-line laser information, the sparse laser includes single-line laser and multi-line laser, wherein the single-line laser in the sparse laser is used to construct the reference depth map and the residual depth map, and the single-line laser in the three-dimensional space Each laser point is stretched in the direction perpendicular to the ground to obtain a reference depth surface perpendicular to the ground; according to the calibration information of the monocular camera and the single-line laser, the reference depth surface obtained in the three-dimensional space is projected to the monocular camera to obtain On the image plane of the image, a reference depth map corresponding to the image is obtained, and the absolute depth map obtained by the depth sensor is compared with the reference depth map to obtain a residual depth map; Step 2: Use the monocular image acquired by the monocular camera and the reference depth map obtained as described in step 1 as training data, and train the machine neural network to estimate the corresponding residual depth map; Step 3, add the residual depth map estimated by the convolutional neural network to the reference depth map to obtain an estimated absolute depth map, which is called the absolute depth estimation map, and further construct an optimized convolutional neural network on this basis; The optimized convolutional neural network and the convolutional neural network used to estimate the residual depth described in step 2 can be superimposed together for end-to-end optimization, that is, input a monocular image and a reference depth map, and output an optimized absolute Depth estimation map. 2. A method for image depth estimation based on sparse laser observation as claimed in claim 1, characterized in that the absolute depth estimation map obtained by the end-to-end output of the deep neural network and the sparse laser depth map are fused through a conditional random field , so as to confirm that there is a single-line laser observation position in the absolute depth estimation map, and its depth value is consistent with the laser observation depth value. 3. A kind of image depth estimation method based on sparse laser observation as described in claim 1, it is characterized in that in step 2, the way of training the machine neural network estimation corresponding residual depth map is as follows: the depth to be fitted The value of the residual depth of each pixel on the residual map is discretized to several natural number values, and the depth estimation of the residual depth is realized in the form of classification.