CN114519682B

CN114519682B - A depth map enhancement method based on autoregressive model

Info

Publication number: CN114519682B
Application number: CN202210040794.9A
Authority: CN
Inventors: 杨洋; 于红蓓; 赵岩; 曾兰玲; 王新宇
Original assignee: Jiangsu University
Current assignee: Jiangsu University
Priority date: 2022-01-14
Filing date: 2022-01-14
Publication date: 2025-03-14
Anticipated expiration: 2042-01-14
Also published as: CN114519682A

Abstract

The invention discloses a depth map enhancement method based on an autoregressive model. Given the image and parameters input by the user, the method firstly obtains the corresponding confidence coefficient according to the depth image, and then carries out correction and up-sampling operation on the depth image. For depth map correction, an unreliable edge region is given a lower confidence and a reliable flat region is given a higher confidence. For up-sampling of the depth map, firstly, the depth map with low resolution is interpolated to obtain the depth map with high resolution, the confidence degree of the unreliable interpolation depth value is given to the low confidence degree of the reliable original depth value, and finally, the image to be processed, the obtained confidence degree and the corresponding reference image are input into a model and the model is solved, so that the target of enhancing the depth map is achieved. Experimental results show that the method can overcome artifacts and ensure smooth edge while ensuring the processing speed, and a good depth map enhancement effect is achieved.

Description

Depth map enhancement method based on autoregressive model

Technical Field

The invention belongs to the technical field of computational photography, and particularly relates to a depth map enhancement method based on an autoregressive model.

Background

In the field of computer vision, depth information has played an irreplaceable role. Existing methods for acquiring depth information include laser ranging scanners, kinect cameras, TOF cameras, and the like. However, these methods have various problems, such as a laser ranging scanner having a high accuracy, a poor real-time performance, a data loss, a fast Kinect camera speed, a short ranging, and a "cavitation" of the acquired depth information, a TOF camera having a high real-time performance, a low resolution of the acquired depth image, and a large amount of random noise. In addition to acquiring depth data by using an instrument, acquiring a depth map by a deep learning manner is also a trend in the field of computer vision at present, but there are often more differences in the depth image acquired by such a manner. In order to process a depth image acquired by an imaging device and a depth image acquired by depth learning, to improve resolution and correct an erroneous depth value so that it can meet the requirements of related applications, a depth map enhancement technique is required.

Existing depth map enhancement methods can be divided into two types, traditional methods and solvers. The classical traditional methods comprise bilateral filtering, guide map filtering and weighted median filtering, and the methods are flexible, fast and efficient, but do not perform well when performing depth map correction tasks, and cannot achieve good correction effects. The quick bilateral solver is a representative method of a solver depth map enhancement method, and the method is quick and can be applied to various computer vision tasks, but the depth map obtained by the solver can generate artifact.

The invention provides a novel depth map enhancement method based on an autoregressive model, which not only can well correct a depth image and realize up-sampling of the depth map, but also can well overcome the artifact phenomenon generated by a rapid bilateral solver, and has higher calculation efficiency.

Disclosure of Invention

Aiming at the problems in the prior art, the invention provides a depth map enhancement method based on an autoregressive model, which can realize up-sampling of a depth map and correction of a depth value while ensuring efficiency and can effectively overcome the artifact phenomenon.

A depth map enhancement method based on an autoregressive model comprises the following steps:

Step 1, matrix grid parameters σ _l (brightness bandwidth), σ _u,v (color bandwidth), σ _x,y (spatial bandwidth) and smoothing parameter λ are set, and a depth image to be enhanced is selected.

Step 2, solving an objective function of the depth image, and enhancing the depth image to obtain a processed depth image;

Further, the objective function used to enhance the depth image is as follows:

Wherein O is the depth value to be enhanced, C is the credibility of the depth value, and the calculation of the credibility is divided into two cases of depth map correction and depth map up-sampling.

Further, the simplified bilateral affine matrix solution is:

according to the definition formula of the bilateral affine matrix:

Wherein the method comprises the steps of Representing the spatial position of pixel i,The values of the l, u and v channels of the pixel i are respectively represented, and the closer the distance between the pixel i and the pixel j is, the closer the color is, and the larger the value A _i,j of the corresponding bilateral affine matrix is.

However, solving the objective function of the depth image to be enhanced based on the definition of the bilateral affine matrix is not realistic, because the depth value of a depth map is often thousands of, and the time taken to solve the objective function directly by the definition of the bilateral affine matrix is not imaginable, so that the bilateral affine matrix needs to be simplified, which is also the key point of solving the objective function.

According to the former study, the bilateral affine matrix can be approximated by the following equation:

A≈S^TBS

S ^T, B, S represent Splat, blur, slice processes, respectively. Now, assuming that a pixel value X needs to be processed, a pixel point is determined by the position (X, y) and the color value (l, u, v), so that X can be regarded as

P= (o _x,o_y,o_l,p_u,p_v) point set. From the previous analysis, it is not preferable to directly solve the bilateral affine matrix of X because the time is too expensive, and the approximation of the bilateral affine matrix is completed by the following three steps:

Splat, each pixel value x _i is projected onto the vertex closest to p _i.

Blur, performing a smoothing operation on vertex values of the bilateral space.

Slice, interpolate new pixel values from the blur value of the vertex closest to p _i.

For use in objective functionsDefined by the formula:

Where D _m、D_n is a diagonal matrix.

If the depth map is corrected, the confidence level is lower when the position is an edge region, and the confidence level is higher when the position is a smooth region, specifically, the confidence level solving formula is that

Where I is the depth image that needs correction, G is the color guide image, σ _c is the smoothing parameter, which is set to 0.125 in the method of the invention.

If the depth map is up-sampled, the low-resolution depth map is first interpolated to the corresponding resolution, then the confidence level at the interpolation is naturally low, and the greater the difference from the true depth value, the lower the confidence level. Aiming at solving the confidence coefficient under the condition, the invention mainly adopts a Gaussian function envelope method. Specifically, the local confidence solving formula is:

Where F is a multiple of the upsampling and F is a matrix of F rows and F columns. Subsequently, C _f is copied m times longitudinally and n times transversely to obtain the confidence coefficient C under the corresponding resolution, A is the length of the depth map to be enhanced, b is the width of the depth map to be enhanced, specifically, the confidence level is

I is the actual depth value somewhere in the depth map to be processed, lambda is an adjustable smoothing parameter; is a bilateral affine matrix, and an approximation of the bilateral affine matrix is used in the objective function.

Further, the method for simplifying the objective function and solving the depth value of the corresponding region according to the simplified bilateral affine matrix comprises the following steps:

approximation of a bilateral affine matrix In alternative target formulasThen, the bias derivative is calculated for O _i, and is set to be 0, so that the formula of the objective function is further simplified as follows:

The writing matrix form is obtained by the following steps:

The arrangement is as follows:

then let o=s ^T Y be the same, The following steps are obtained:

and multiplying S on both sides of the above formula to obtain the following formula:

Wherein SS ^T＝D_m,Sdiag(C)S^T =diag (SC), diag (C) i=c.i, and thus the above formula is equivalent to:

And finally, only one-step slice operation is needed to be carried out on the solved Y, so that the depth value of the required solution can be obtained.

Further, the above procedure performs a one-step slice operation on the solved Y, that is, multiplies the solved Y matrix by the transpose of the S matrix, and calculates by the following formula:

O=S^TY

The beneficial effects of the invention are as follows:

the depth images directly obtained by the prior art have the problems of low resolution, error depth values and the like, so before the depth images are applied to computer vision tasks, the depth images need to be repaired by a depth image enhancement technology so as to meet the requirements of related applications. The invention can realize a better enhancement effect on the depth map, namely correct the error depth value, and better overcome the generation of artifact phenomenon in the up-sampling of the depth map, and has quicker processing efficiency.

Drawings

Fig. 1 is a flow chart of the method of the present invention.

FIG. 2 is a graph comparing different upsampling multiples of the present invention to a bilateral filter and fast bilateral solver.

Fig. 2 (a) is the result of the inventive 2-fold up-sampling, fig. 2 (b) is the result of the inventive 4-fold up-sampling, and fig. 2 (c) is the result of the inventive 8-fold up-sampling. Fig. 2 (d) is a result of double-sided filtering 2-fold up-sampling, fig. 2 (e) is a result of double-sided filtering 4-fold up-sampling, and fig. 2 (f) is a result of double-sided filtering 8-fold up-sampling. Fig. 2 (g) is a result of 2-fold up-sampling by the fast bilateral solver, fig. 2 (h) is a result of 4-fold up-sampling by the fast bilateral solver, and fig. 2 (i) is a result of 8-fold up-sampling by the fast bilateral solver. The parameters of the fast bilateral solver are λ=5, σ _l＝10,σ_u,v＝5,σ_x,y =30, the bilateral filtering parameters are r=16, the global variance is 16, the local variance is 0.8, and the parameters of the fast bilateral solver are λ=5, σ _l＝10,σ_u,v＝5,σ_x,y =30.

Fig. 3 is a comparison of the present invention with other methods for depth map correction.

Fig. 3 (a) is a depth map of residual network prediction, fig. 3 (b) is a corrected result of the present invention, the parameter is λ=15, σ _l＝5,σ_u,v＝10,σ_x,y =30, fig. 3 (c) is a corrected result of domain conversion filtering DF, σ_s=8σ_r=4.6, the parameter is r=16, the global variance is 16, the local variance is 0.8, fig. 3 (d) is a corrected result of weighted median filtering WMF, the parameter is r=10, σ=15.5, w=1, fig. 3 (e) is a corrected result of the fast bilateral solver, the parameter is λ=15, σ _l＝5,σ_u,v＝10,σ_x,y =30, and fig. 3 (f) is a corrected result of pilot map filtering, the parameter is r=10.

Detailed Description

Given an image and parameters input by a user, the method firstly obtains corresponding confidence coefficient according to the depth image, and then carries out correction and up-sampling operation on the depth image. For depth map correction, because the error of the edge part of the depth map predicted by the full convolution residual network is larger, and the prediction result of the image smoothing area is more accurate, when the method is used for correcting the depth map, a lower confidence level is given to the edge part of the unreliable image, and a higher confidence level is given to the trusted image smoothing area. For up-sampling of the depth map, firstly, the depth map with low resolution is interpolated to obtain the depth map with high resolution, the depth value obtained by interpolation is unreliable, a lower confidence value is correspondingly given, the original depth value has higher confidence and a higher confidence value is correspondingly given, then, a simplified bilateral affine matrix in the method is solved, and finally, the image to be processed, the obtained confidence and the corresponding reference image are processed by the method provided by the invention, so that the effects of correction and up-sampling are achieved. Experimental results show that the method can overcome artifacts and smooth edge protection while ensuring the processing speed, and achieves a good depth map enhancement effect.

The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.

As shown in fig. 1, the depth map enhancement method based on the autoregressive model provided by the invention comprises the following specific processes:

Step 1, matrix grid parameters, σ _l (brightness bandwidth), σ _u,v (color bandwidth), σ _x,y (spatial bandwidth) and smoothing parameters λ are set, and a depth image to be enhanced and a corresponding color reference image are selected. The objective function for depth enhancement by creating the target depth image is as follows:

wherein I _i is the actual depth value somewhere in the depth map to be processed, λ is an adjustable smoothing parameter; The method is a bilateral affine matrix, an approximate form of the bilateral affine matrix is used in an objective function, o is a depth value needing to be enhanced, i and j represent positions of pixels in an image, and C is reliability of the depth value at the position and is divided into two cases of depth map correction and depth map up-sampling.

Step 2, solving simplified bilateral affine matrix and confidence coefficient in objective function

For the objective function proposed in the above step, two main problems need to be solved first, namely solving a simplified bilateral affine matrix and solving a confidence coefficient.

The method for solving the simplified bilateral affine matrix comprises the following steps:

according to the definition formula of the bilateral affine matrix:

Wherein the method comprises the steps of Representing the spatial position of pixel i,The values of the l, u and v channels representing the pixel i respectively, sigma _x,y,σ_l,σ_uv refers to three adjustable parameters, and from the definition formula of the bilateral affine matrix, it can be seen that the closer the distance between the pixel i and the pixel j is, the closer the color is, and the larger the value A _i,j of the corresponding bilateral affine matrix is.

However, the objective function proposed by the definition solution based on the bilateral affine matrix is unrealistic, because the depth value of a depth map is often thousands of, and the time taken to solve the objective function directly by the definition of the bilateral affine matrix is not imaginable, so that the simplification process is required for the bilateral affine matrix, which is also the key point of solving the objective function.

Approximating the bilateral affine matrix by the following equation:

A≈S^TBS

S ^T, B, S represent Splat, blur, slice processes, respectively. Assuming that a pixel value X needs to be processed, a certain pixel point is determined by the position (X, y) and the color value (l, u, v), so that X can be regarded as a point set of p= (P _x,p_y,p_l,p_u,p_v). From the previous analysis, it is not preferable to directly solve the bilateral affine matrix of X because the time is too expensive, and therefore, the approximation of the bilateral affine matrix is completed by the following three processes:

Splat, each pixel value x _i is projected onto the vertex closest to p _i.

Blur, performing a smoothing operation on vertex values of the bilateral space.

For use in objective functionsDefined by the formula:

Where D _m、D_n is a diagonal matrix.

Where I is the depth image to be corrected, G is the color guide image, σ _c is the smoothing parameter, which is set to 0.125 in the present invention.

If the depth map is up-sampled, the low-resolution depth map is first interpolated to the corresponding resolution, then the confidence level at the interpolation is naturally low, and the greater the difference from the true depth value, the lower the confidence level. The invention mainly adopts a Gaussian function envelope method, and specifically, the partial confidence solving formula is as follows:

Where F is a multiple of the upsampling and F is a matrix of F rows and F columns.

Subsequently, C _f is copied n times longitudinally, n times transversely, the confidence degree C under the corresponding resolution is obtained,A is the length of the depth map to be enhanced, b is the width of the depth map to be enhanced, specifically, the confidence level is

The method for simplifying the objective function according to the simplified bilateral affine matrix comprises the following steps:

approximation of a bilateral affine matrix In alternative target formulasThen, the bias derivative is calculated for O _i, and is set to be 0, so that the formula of the objective function can be further simplified as follows:

The writing matrix form is obtained by the following steps:

The arrangement is as follows:

Then let o=s ^T Y to be, The following steps are obtained:

wherein SS ^T＝D_m,Sdiag(C)S^T = diag (SC), diag (C) I = C I,

The above formula is therefore equivalent to:

The above process performs one-step slice operation on the solved Y, that is, multiplies the solved Y matrix by the transpose of the S matrix, and calculates by the following formula:

O=S^TY

And 3, taking the depth image to be enhanced as a target image, taking a corresponding color image as a reference image, inputting the target function into a solver, solving the corresponding confidence coefficient aiming at two different conditions, and finally constructing and outputting the enhanced depth image.

In addition, the size of the grid of the bilateral space mentioned in the step 2 can be controlled, the larger the grid is, the faster the grid is, but the larger the grid is, the effect of enhancing the depth map is affected. In the embodiment, on the Intel i5-4200H CPU@2.80GHz,16G memory machine, when the depth map line of 1390 pixels×1110 pixels is enhanced, the time is about 1s, so that the requirement of the actual application on the depth map restoration calculation efficiency can be met.

As shown in FIG. 2, the method of the invention has the advantages that compared with two methods, the image details can be better reserved under the same upsampling multiple, the edges are sharper, and the better upsampling effect is achieved. As shown in fig. 3, the four methods of the co-domain conversion filtering, the weighted median filtering and the fast bilateral solver of the invention and other methods for correcting the depth map are compared, and the correction effect of the method of the invention is better than that of other methods, especially in the edge region of the depth map predicted by the residual network, and the correction effect is more obvious. The following table shows the comparison result of the invention with other methods in operation time, and the image processed by the invention and the other methods are 1390 pixels multiplied by 1110 pixels, so that the invention has higher efficiency and can meet the requirement of real-time in practical application.

Method of	Run time(s)
		The invention is that	1.07
Bilateral filtering	4.07
		Quick double-side solver	1
Domain conversion filtering	0.25
		Weighted median filtering	1.21
Guide map filtering	11.96

The above list of detailed descriptions is only specific to practical embodiments of the present invention, and they are not intended to limit the scope of the present invention, and all equivalent manners or modifications that do not depart from the technical scope of the present invention should be included in the scope of the present invention.

Claims

1. A depth map enhancement method based on an autoregressive model, comprising:

S1, setting matrix grid parameters including brightness bandwidth sigma _l, color bandwidth sigma _u,v, space bandwidth sigma _x,y and smoothing parameter lambda, selecting a depth image to be enhanced, and establishing an objective function of the depth image;

S2, solving an objective function of the depth image, and enhancing the depth image to obtain a processed depth image;

in the step S1, an objective function of the depth image to be enhanced is established as follows:

where I is the actual depth value at the preset of the depth map to be processed, lambda is an adjustable smoothing parameter, The depth value is a bilateral affine matrix, O is a depth value to be enhanced, i and j represent positions of pixels in an image, and C is confidence coefficient of the depth value at the position and is divided into two cases of depth map correction and depth map up-sampling;

if the depth map is corrected, the confidence level is lower when the position is an edge region, the confidence level is higher when the position is a smooth region, and the confidence level solving formula is that

Wherein I is a depth image to be corrected, G is a color guide image, σ _c is a confidence level smoothing parameter, which is set to 0.125;

the solution for the simplified bilateral affine matrix is as follows:

Defining a bilateral affine matrix:

Wherein the method comprises the steps of Representing the spatial position of pixel i,The sigma _x,y,σ_l,σ_u,v refers to three adjustable parameters, and the closer the distance between the pixel i and the pixel j is, the closer the color is, the larger the value A _i,j of the corresponding bilateral affine matrix is;

Approximating the bilateral affine matrix:

A≈S^TBS

S ^T, B, S represent Splat, blur, slice processes, respectively, as follows:

Splat, projecting each pixel value x _i onto the vertex closest to p _i;

Blur, performing smoothing operation on vertex values of the bilateral space;

slice, interpolating new pixel values from the blur value of the vertex closest to p _i;

thus, used in objective functions Defined by the formula:

Wherein D _m、D_n is a diagonal matrix;

according to the simplified bilateral affine matrix reduction objective function, the specific method is as follows:

approximation of a bilateral affine matrix Replacing in the objective function formulaThen, the bias derivative is calculated for O _i, and is set to 0, so that the objective function in the step S1 is further simplified to be:

Writing the above into a matrix form:

The arrangement is as follows:

then let o=s ^T Y be the same, The following steps are obtained:

wherein SS ^T＝D_m,Sdiag(C)S^T = diag (SC), diag (C) I = C I,

The above formula is therefore equivalent to:

Finally, only one-step Slice operation is carried out on the solved Y, and the depth value of the required solution can be obtained;

the Slice operation refers to multiplying the solved Y matrix by the transpose of the S matrix, and the calculation expression is as follows:

O=S^TY。

2. The method according to claim 1, wherein if the depth map is up-sampled, the depth map with low resolution is first interpolated to the corresponding resolution, then the confidence level at the interpolation is low, and the greater the difference from the true depth value, the lower the confidence level, and the local confidence level is calculated by using the gaussian envelope method:

Wherein F is a multiple of up-sampling, and F is a matrix of F rows and F columns;

Subsequently, C _f is copied m times longitudinally and n times transversely to obtain the confidence coefficient C under the corresponding resolution, A is the length of the depth map to be enhanced, b is the width of the depth map to be enhanced, and the confidence is