CN116883477A - A monocular depth estimation method - Google Patents

Abstract

The application discloses a monocular depth estimation method, which comprises the following steps: dividing a training set and a testing set by adopting a KITTI data set; constructing a network structure based on monocular depth estimation, wherein the network structure comprises a depth network and a pose network, the depth network adopts a U-net structure and comprises an encoder and a decoder, and the pose network comprises a pose encoder and a pose decoder; reconstructing a target image by using the image reconstruction model; computing reconstructed imagesCorresponding photometric feature loss and textural feature loss; and performing monocular depth estimation training by using a multi-fusion loss function of texture feature loss, luminosity feature loss and pixel smoothness as a total loss function, and effectively evaluating a test set by using a model obtained by training. On the basis of luminosity feature loss, the depth estimation effect under the condition of no texture or weaker texture is enhanced by introducing texture feature loss, and in addition, the good effect can be realized across the data domain.

Description

A monocular depth estimation method

Technical field

The invention relates to the field of computer vision technology, specifically a monocular depth estimation method.

Background technique

Currently, monocular depth estimation is a research focus in computer vision tasks and is widely used in intelligent driving, robot movement and three-dimensional cognition. Traditional SFM and other algorithms are difficult to implement in many tasks. As deep learning is becoming the current mainstream algorithm, the effect of using monocular cameras is getting better and better, and it also has the advantages of cost saving and small size.

Learning-based methods are mainly divided into two categories: supervised learning and self-supervised learning. Supervised learning requires a large variety of datasets and paired ground-truth depth labels as input. Relatively speaking, the task of obtaining data is difficult, and the equipment such as lidar used is also relatively expensive. Self-supervised learning is relatively easy to obtain depth and pose. Usually, a monocular camera image sequence is used as input, and a corresponding network architecture is used to unify the two tasks of depth map and pose estimation into one framework, in which the supervision information is mainly From view composition.

However, in terms of the effects currently achieved, self-supervised methods are still inferior to supervised methods. The main reason is that when using photometric feature loss as the supervision signal, features cannot be effectively extracted in some places with high light intensity or weak texture.

Contents of the invention

In order to solve the above problems, the present invention provides a monocular depth estimation method that can reflect texture details and edge clues in images.

In order to achieve the above objects, the present invention is achieved through the following technical solutions:

The present invention is a monocular depth estimation method, including the following:

Use the KITTI data set to divide the training set and the test set;

Construct a network structure based on monocular depth estimation, including a deep network and a pose network. The deep network adopts a U-net structure and includes an encoder and a decoder. The pose network includes a pose encoder and a pose decoder. ;

Use the image reconstruction model to reconstruct the target image: the target image I _t (p) outputs the corresponding depth map D _t through the deep network, and the relative pose T _{t → t′} of the corresponding image generated through the pose network, combined with the internal parameter K of the camera The image reconstruction model performs image reconstruction to obtain the reconstructed image. Compute reconstructed image/> Corresponding photometric feature loss and introduction of texture feature gradient Calculate the texture feature loss function;

The multi-fusion loss function of texture feature loss, photometric feature loss, and pixel smoothness is used as the total loss function to perform monocular depth estimation training, and the trained model is used to effectively evaluate the test set data.

A further improvement of the present invention is that the output end of the encoder introduces coordinate attention and is connected to the decoder.

A further improvement of the present invention is that the encoder of the deep network adopts the ResNet-18 network structure with the fully connected layer removed, in which the deepest feature map goes through 5 downsampling stages, and the resolution of the input image is reduced to 1/32, so The described decoder consists of five 3×3 convolutional layers, each followed by a bilinear upsampling layer.

A further improvement of the present invention is that the pose encoder of the pose network adopts the ResNet-18 network structure. A further improvement of the present invention is that the image reconstruction model expression is:

In the formula, To reconstruct the image,/> is the reconstructed pixel, proj is the projection relationship between 2D and 3D, <> is the sampling operator, I _t is the target image frame, T _t→t′ is the relative pose, K is the internal parameter of the camera, and D _t is the depth. picture.

A further improvement of the present invention lies in: reconstructing the image The corresponding expression of photometric feature loss is:

In the formula, L _phRec is the photometric feature loss, l(,) is the photometric difference loss measured for each pixel, I _t (p) is the target image, and p is the pixel;

Gradient texture features Introduced into reconstructed image/> The corresponding photometric characteristic loss expression is obtained:

In the formula, is the reconstructed texture feature, and φ _t (p) is the texture feature of the target image.

A further improvement of the present invention is that the expression of the multi-fusion loss function is:

L _total ＝λL _smooth +βL _phRec +γL _fmRec (4)

in:

In the formula, L _total is the multi-fusion loss, L _fmRec is the texture feature loss, L _phRec is the photometric feature loss, L _smooth is the pixel smoothness, where α is 0.85, λ, β, and γ are the pixel smoothness and photometric feature losses respectively. , the weight of texture feature loss, where λ = 1, β and γ are 0.001, and x and y are used to represent pixel coordinates.

The beneficial effects of the present invention are: based on the loss of photometric features, the present invention enhances the effect of depth estimation in the case of no texture or weak texture by introducing texture feature loss. The method of the present invention can achieve good results across data domains. Although the present invention trains a monocular depth estimation network based on the KITTI data set, it has good performance on KITTI, cityscapes, Maker3D and other data sets. The method of the present invention introduces coordinate attention, strengthens channel and position attention, and enhances boundary features.

Description of the drawings

Figure 1 is a schematic diagram of the network structure of monocular depth estimation in an embodiment of the present invention;

Figure 2 is a flow chart of photometric feature loss in the embodiment of the present invention;

Figure 3 is a schematic diagram of coordinate attention in the embodiment of the present invention;

Figure 4 is a comparison of the Kitti data set effects between the method in the embodiment of the present invention and the existing depth estimation method;

Figure 5 is a comparison of the Cityscape data set effects between the method in the embodiment of the present invention and the existing depth estimation method;

Figure 6 is a comparison of the effects of the Make3D data set between the method in the embodiment of the present invention and the existing depth estimation method.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the embodiments described here are only used to explain the present invention and are not intended to limit the present invention.

A monocular depth estimation method of the present invention includes the following:

Use the KITTI data set and divide it into a training set and a test set;

Construct a network structure based on monocular depth estimation, including a deep network and a pose network. The deep network adopts a U-net structure and includes an encoder and a decoder. The pose network includes a pose encoder and a pose decoder. ;

Use the image reconstruction model to reconstruct the target image: the target image I _t (p) outputs the corresponding depth map D _t through the deep network, and the relative pose T _{t → t′} of the corresponding image generated through the pose network, combined with the internal parameter K of the camera The image reconstruction model performs image reconstruction to obtain the reconstructed image.

Compute reconstructed image Corresponding photometric feature loss, and introduce texture feature gradient/> Calculate the texture feature loss function;

The multi-fusion loss function of texture feature loss, photometric feature loss, and pixel smoothness is used as the total loss function to perform monocular depth estimation training, and the trained model is used to effectively evaluate the test set data.

The encoder part of the deep network (DipthNet) of the present invention adopts the Resnet18 network structure, and its output end can introduce coordinate attention and connect to the decoder. Coordinate attention can improve the expressive ability of network learning features. _{You can} ^use any _{intermediate feature tensor} …,y _c ]. The coordinate attention structure in this embodiment is shown in Figure 3. Given an input X, two spatial ranges (H, 1) or (1, W) of the pooling kernel in the and vertical coordinates are encoded for each channel, and the final output Y is the same size as C×H×W. The output formula of the c-th channel with height h can be transformed into:

Similarly, the output formula of the c-th channel at a bandwidth of w is also:

Among them, C is the number of image channels, H is the image height, W is the image width, x _c (h,i), x _c (j,w) represent the features in the x direction of the coordinate axis, Represents the characteristics of the z direction of the coordinate axis.

The above two transformations aggregate features along two spatial directions respectively, producing a pair of direction-aware feature maps. After being processed by Xavg pool and Y avg pool, they are connected and then subjected to 1×1 convolution transformation to obtain an intermediate feature map that encodes spatial information in the horizontal and vertical directions. Then, this embodiment decomposes the feature map into two independent tensors along the spatial dimension for processing respectively, and finally obtains an output of the same dimension with spatial and channel attention features.

The entire monocular depth estimation network structure of the present invention adopts a multi-scale structure design, which can extract photometric features at different scales to solve problems such as "artifacts". In order to effectively improve the compactness of the overall network structure, the pose encoder of the pose network (PoseNet) also uses resnet18, and the relative pose relationship between three consecutive frames of images is output at the pose decoder output. Specifically, the encoder of the deep network uses the ResNet-18 network structure with the fully connected layer removed. The deepest feature map goes through 5 downsampling stages and reduces the resolution of the input image to 1/32. The decoder contains five 3×3 convolutional layers, each convolutional layer is followed by a bilinear upsampling layer. The multi-scale feature maps of the decoder convolutional layer are used to generate multi-scale reconstructed images, where the feature maps of each scale are further subjected to 3×3 convolution and sigmoid function for image reconstruction.

As shown in Figure 1, in the entire monocular depth estimation network structure, the input image source I _s ∈ {I _t-1 , I _t , I _t+1 } is three adjacent image frames, where I _t is as Target image frame, I _t is input into the depth network, and the corresponding depth map D _t is output. The image source I _s is input into the pose network, and the corresponding pose data is output. Among them, I _t-1 and I _t+1 calculate the position of I _t respectively. The pose estimation is denoted as T _t→t′ .

In order to generate an effective depth map, the target image I _t (p) of the present invention outputs the corresponding depth map D _t through the depth network, the relative pose T _t→t′ of the corresponding image generated through the pose network, and the image is combined with the internal parameter K of the camera Reconstruct the model to reconstruct the image and obtain the reconstructed image The expression of the image reconstruction model is:

In the formula, To reconstruct the image,/> is the reconstructed pixel, proj is the projection relationship between 2D and 3D, <> is the sampling operator, I _t is the target image frame, T _t→t′ is the relative pose, K is the internal parameter of the camera, and D _t is the depth. picture.

Photometric feature loss is relatively common in depth estimation, but performs poorly in low-texture areas. The present invention introduces texture feature loss based on photometric feature loss to increase feature extraction in low-texture areas. The photometric feature loss is shown in Figure 2 in the network structure of monocular depth estimation. The input in the figure is the target image (i.e. I _t (p)). The disparity map (depth map D _t ) is generated through the depth network, and the joint pose network The output pose T _t→t′ is reconstructed, and the reconstructed image is generated after sampling (i.e. ). The corresponding photometric feature loss expression is:

Among them, l(,) is the photometric difference loss measured for each pixel, and p is the pixel.

In the case of normal depth estimation and pose motion, photometric feature loss can play a good role, but corresponding to low-texture or even texture-less areas, if the photometric difference is similar or equal, it cannot play a good supervision role. According to formula (5) and formula (6), further analyze the gradient of depth D(p) and self-motion M in photometric feature loss:

It can be seen from the above formula that the depth and pose gradients depend on the image gradient In the textureless area, the image gradient is zero, which results in formula (5) and formula (6) being zero. It can be seen from here that multi-view reconstruction cannot be performed well by relying solely on photometric errors. Therefore, the present invention introduces texture feature gradient/> To formula (4):

in represents the reconstructed texture feature, and φ _t (p) represents the texture feature of the target image.

The multi-fusion loss function in this embodiment is as follows:

According to Equation (7), the texture feature measurement loss can be obtained as:

The photometric error L _phRec is generated using L1 and SSIM, that is:

Take α=0.85. At the same time, the parallax smoothing loss is solved for the generated depth map:

Combine pixel smoothness, photometric feature loss, and texture feature loss into an overall loss function:

L _total ＝γL _smooth +βL _phRec +γL _fmRec (11)

In the formula, λ, β, and γ are the weights of pixel smoothness, photometric feature loss, and texture feature loss respectively.

The input image size in this example is 640*192. The KITTI data set is used for network training of monocular depth estimation. There are 39810 frames of training images and 4242 frames of test images. Monocular (M) training is mainly used for training comparison. After comparative experiments, it was found that monodepth2 uses the same parameters to train and the final Abs Rel obtained is 0.120. The results show that even under the conditions of limited computing resources, the evaluation effect of the method of the present invention is still relatively good. The training results are shown in Table 1:

Table 1: Comparison of training results on sampled KITTI data sets

Ours(Feat) refers to the monocular depth estimation network that only includes texture feature loss in the embodiment of the present invention. Ours(coor+Feat) refers to the monocular depth estimation network that introduces both texture feature loss and coordinate attention in the embodiment of the present invention. network.

Figure 4 shows the data test effect of KITTI. Participants in the comparison include the classic algorithm Monodepth2, the latest results Lite-mon and R_MSFM and other methods, and the present invention uses a method with only texture features our (Feat) and texture attention combined ours (coor +Feat) two methods. It can be seen from the experimental results that the method combining texture attention performs better in low texture details, which is specifically reflected in: (a) the outline of the trees in the column chart is relatively clear, (b) the thickness and size of the sign poles in the column and even the reflective logo They can clearly express, (c) the treatment of the corner with strong sunlight, and (d) the treatment of the shrubs and railings illuminated by the sun. Due to the inconsistency of photometric errors, the actual distance between the object and the camera in the depth map is often inconsistent, which cannot reflect the preparation distance from the object in the depth map to the camera. For details, please refer to the cyclist in column (b). Note the texture of the present invention. The method of combining forces can concretely reflect the actual distance.

Figure 5 shows the test results of the Cityscapes data set. The KITTI data are basically static scenes shot by mobile cameras and lack moving targets. However, in the actual real-life environment, there are many vehicles and pedestrians on the highway. Different from the KITTI data set, there are a large number of pedestrians and driving vehicles in the Cityscapes data set. The model trained in the KITTI data set is used to test the Cityscapes data set to verify the effectiveness of the network of the present invention in different data sets. In the input part of Figure 5, the shape details of distant street light poles, trees, and pedestrians have become the basis for distinguishing the advantages and disadvantages of different methods. In column (a), the method of the present invention can clearly display the street light poles in the distance. In column (b), the surrounding street light poles can be clearly displayed. In column (c), only the method of the present invention has the details of the tree texture. The obtained image is relatively clear, and the method of the present invention in column (d) can express both the surrounding street light poles and the distant sky. It can be seen that the effect of using the method of the present invention is obvious in Figure 5.

Figure 6 shows the test effect of the Make3D data set. The network trained in KITTI is also tested and compared on the Make3D data set. As shown in Figure 6, the method of combining texture attention in the present invention performs better in the details of the depth map, especially in places with strong lighting, which can better reflect the outlines and details of houses and trees.

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person familiar with the technical field can easily think of various changes or modifications within the technical scope disclosed in the present application. Replacements, these should all be covered by the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims (7)

1. A monocular depth estimation method, characterized by: including the following: Use the KITTI data set to divide the training set and the test set; Construct a network structure based on monocular depth estimation, including a deep network and a pose network. The deep network adopts a U-net structure and includes an encoder and a decoder. The pose network includes a pose encoder and a pose decoder. ; Use the image reconstruction model to reconstruct the target image: the target image I _t (p) outputs the corresponding depth map D _t through the deep network, and the relative pose T _{t → t′} of the corresponding image generated through the pose network, combined with the internal parameter K of the camera The image reconstruction model performs image reconstruction to obtain the reconstructed image. Compute reconstructed image Corresponding photometric feature loss, and introduce texture feature gradient/> Calculate the texture feature loss function; The multi-fusion loss function of texture feature loss, photometric feature loss, and pixel smoothness is used as the total loss function to perform monocular depth estimation training, and the trained model is used to effectively evaluate the test set. 2. A monocular depth estimation method according to claim 1, characterized in that: the output end of the encoder introduces coordinate attention and is connected to a decoder. 3. A monocular depth estimation method according to claim 1 or 2, characterized in that: the encoder of the deep network adopts the ResNet-18 network structure with the fully connected layer removed, in which the deepest feature map passes through 5 In the downsampling stage, and reducing the resolution of the input image to 1/32, the decoder contains five 3×3 convolutional layers, each convolutional layer is followed by a bilinear upsampling layer. 4. A monocular depth estimation method according to claim 3, characterized in that: the pose encoder of the pose network adopts a ResNet-18 network structure. 5. A monocular depth estimation method according to claim 3, characterized in that: the image reconstruction model expression is: In the formula, To reconstruct the image,/> is the reconstructed pixel, proj is the projection relationship between 2D and 3D, <> is the sampling operator, I _t is the target image frame, T _t→t′ is the relative pose, K is the internal parameter of the camera, and D _t is the depth. picture. 6. A monocular depth estimation method according to claim 5, characterized in that: the reconstructed image The corresponding expression of photometric feature loss is: In the formula, L _phRec is the photometric characteristic loss, To measure the photometric difference loss of each pixel, I _t (p) is the target image, and p is the pixel; Gradient texture features Introduced into reconstructed image/> The corresponding photometric characteristic loss expression is obtained: In the formula, is the reconstructed texture feature, and φ _t (p) is the texture feature of the target image. 7. A monocular depth estimation method according to claim 6, characterized in that: the expression of the multi-fusion loss function is: L _total ＝λL _smooth +βL _phRec +γL _fmRec (4) in: In the formula, L _total is the multi-fusion loss, L _fmRec is the texture feature loss, L _phRec is the photometric feature loss, L _smooth is the pixel smoothness, where α is 0.85, λ, β, and γ are the pixel smoothness and photometric feature losses respectively. , the weight of texture feature loss, where λ = 1, β and γ are 0.001, and x and y are used to represent pixel coordinates.