CN106559659B - Stereoscopic image depth map generation device and method - Google Patents

Abstract

A device and method for generating a stereoscopic image depth map, the device comprising: a depth calculation module, a first-order bilateral filtering module and a second-order bilateral filtering module. The depth calculation module receives a reference image including a plurality of reference pixels and a target image including a plurality of target pixels, and generates an initial depth map according to the pixel differences between the reference pixels and the target pixels. The first-order bilateral filtering module receives the initial depth map to perform a first-order bilateral filtering calculation on the initial depth map and the initial depth map itself to generate an average depth map. The second-order bilateral filtering module receives the average depth map and the target image to perform a second-order bilateral filtering calculation on the average depth map and the target image to generate a refined depth map. The present invention also discloses a method for generating a stereoscopic image depth map.

Description

Stereoscopic image depth map generation device and method

technical field

The invention relates to an image processing technology, in particular to a device and method for generating a stereoscopic image depth map.

Background technique

Stereo vision technology (stereo vision) is widely used in various fields such as 3D movies, etc., and a depth map (depth map) is important information for generating stereo vision. The depth map can be obtained through stereo matching, and its accuracy will affect the quality of the image. However, in the current technology, the quality of the depth map is often limited by the broken part of the parallax and the expansion of the edge, which causes the generated 3D image to shake when viewed.

Therefore, how to design a new device and method for generating a depth map of a stereoscopic image to solve the above-mentioned deficiency is an urgent problem to be solved in the industry.

Contents of the invention

The technical problem to be solved by the present invention is to provide a stereoscopic image depth map generation device and method to solve the above-mentioned defects in the prior art.

In order to achieve the above object, the present invention provides a stereoscopic image depth map generation device, including: a depth calculation module, a first-order bilateral filter module and a second-order bilateral filter module. The depth calculation module receives a reference image including a plurality of reference pixels and a target image including a plurality of target pixels, and generates an initial depth map (initial depth map) according to pixel differences between the reference pixels and the target pixels. The first-order bilateral filter (bilateral filter) module receives the initial depth map to perform first-order bilateral filter calculation on the initial depth map and the initial depth map itself to generate an average depth map. The second-order bilateral filter module receives the average depth map and the target image, and performs second-order bilateral filter calculation on the average depth map and the target image to generate a refined depth map.

In order to better achieve the above object, the present invention also provides a method for generating a stereoscopic image depth map, comprising: the depth calculation module receives a reference image containing a plurality of reference pixels and a target image containing a plurality of target pixels, and And the pixel difference of the target pixel generates an initial depth map; the first-order bilateral filtering module receives the initial depth map, and performs the first-order bilateral filtering calculation on the initial depth map and the initial depth map itself to generate an average depth map; and the second-order The bilateral filtering module receives the average depth map and the target image, and performs a second-order bilateral filter calculation on the average depth map and the target image to generate a refined depth map.

Technical effect of the present invention is:

The present invention further refines the depth map through two-order bilateral filtering to improve the quality of the depth map and easily achieve the above purpose.

The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments, but not as a limitation of the present invention.

Description of drawings

FIG. 1 is a block diagram of a device for generating a stereoscopic image depth map according to an embodiment of the present invention;

Fig. 2 is a block diagram of a depth calculation module according to an embodiment of the present invention;

Fig. 3 is a graph of the corresponding curve relationship between the parameter difference and the weight value according to an embodiment of the present invention:

FIG. 4 is a flowchart of a method for generating a depth map of a stereoscopic image according to an embodiment of the present invention.

Among them, reference signs

1 Stereoscopic image depth map generation device 10 Depth calculation module

12 First-order bilateral filter module 14 Second-order bilateral filter module

16A, 16B conversion unit 20 weight calculation unit

21 Weight value 22 Cost calculation unit

23 Cost Value 24 Cost Aggregation Unit

25 Cost aggregation value 26 Postprocessing unit

28 mean filtering unit 30, 32 curves

400 Stereoscopic image depth map generation method 401-403 steps

Detailed ways

Below in conjunction with accompanying drawing, structural principle and working principle of the present invention are described in detail

Please refer to Figure 1. FIG. 1 is a block diagram of a stereoscopic image depth map generation device 1 in an embodiment of the present invention. The stereo image depth map generation device 1 includes a depth calculation module 10 , a first-order bilateral filter module 12 and a second-order bilateral filter module 14 .

The depth calculation module 10 receives a reference image Ir and a target image It. Both the reference image Ir and the target image It are two-dimensional images. In one embodiment, the reference image Ir is a right-eye image, and the target image It is a left-eye image. In another embodiment, the reference image Ir is a left-eye image, and the target image It is a right-eye image. The reference image Ir includes a plurality of reference pixels (not shown), and the target image It includes a plurality of target pixels (not shown).

In one embodiment, the stereoscopic image depth map generation device 1 includes conversion units 16A and 16B, which respectively first convert the reference image Ir and the target image It from the RGB color space to the YcbCr color space, and then receive the converted images from the depth calculation module 10. The reference image Ir' and the target image It' are processed.

The depth calculation module 10 generates an initial depth map (initial depth map) ID according to the pixel difference between the reference pixels of the reference image Ir' and the target pixels of the target image It'.

The first-order bilateral filter module 12 receives the initial depth map ID to perform first-order bilateral filter calculation on the initial depth map ID and the initial depth map ID itself to generate an average depth map ID'.

In one embodiment, the first-order bilateral filtering calculation performed by the first-order bilateral filtering module 12 is a bilateral grid (bilateral grid) filtering calculation. The bilateral grid filtering technique is evolved from the basic bilateral filtering technique.

Among them, the bilateral filtering technology is a non-linear filtering technology, which can blur the image and retain the content information at the edge of the image. Bilateral grid filtering technology further improves computational efficiency and maps two-dimensional images to three-dimensional space. Through the control of the grid size, bilateral filtering can be performed on the pixels mapped to the three-dimensional space, and then the values are mapped back to the original image to obtain a smooth picture. In one embodiment, the first-order bilateral filter calculation performed by the first-order bilateral filtering module 12 on the initial depth map ID and the initial depth map ID itself can carry out the weight of pixel distance and the weight of pixel color difference, so that the initial depth map The disparity information contained in the ID produces a smoothing effect, so as to correct the problem of fragmentation of the disparity information.

The second-order bilateral filtering module 14 receives the average depth map ID' and the target image It to perform second-order bilateral filtering calculations on the average depth map ID' and the target image It' to generate a refined depth map IDR.

Similarly, in one embodiment, the second-order bilateral filtering calculation performed by the second-order bilateral filtering module 14 is a bilateral grid filtering calculation. The second-order bilateral filtering calculation uses the two-dimensional target image It' to correct the problem that the disparity information in the initial depth map ID will expand at the boundary.

Therefore, the stereoscopic image depth map generation device 1 of the present invention further refines the depth map through two-order bilateral filtering, so as to improve the quality of the depth map.

Please refer to Figure 2. FIG. 2 is a more detailed block diagram of the depth calculation module 10 in an embodiment of the present invention. The depth calculation module 10 includes a weight calculation unit 20 , a cost calculation unit 22 , a cost aggregation unit 24 and a post-processing unit 26 .

In an embodiment, the depth calculation module 10 may further include an average filtering unit, which is used to perform mean filtering on the reference image Ir', and then be received and processed by the weight calculation unit 20. In one embodiment, the size of the average filter can be, for example, but not limited to, a three-by-three window size, so as to filter out the noise of the reference image Ir'.

The weight calculation unit 20 determines the difference range where the parameter difference between the reference pixels is located, so as to assign a weight value 21 corresponding to the difference range to each reference pixel. Wherein, the parameter difference is a color parameter difference or a space parameter difference. For example, the color parameter difference may be the difference of the grayscale value, and the spatial parameter difference may be the difference of the pixel distance. In one embodiment, there is a binary correspondence between the above-mentioned difference range and the weight value 21 .

Please also refer to Figure 3. FIG. 3 is a graph showing the relationship between the parameter difference and the weight value 21 according to an embodiment of the present invention. Wherein, the horizontal axis represents the parameter difference, and the vertical axis represents the weight value 21 .

In some technologies, the difference range and the weight value adopt an exponential correspondence relationship as represented by the thinner curve 30 . The advantage of using exponential correspondence to generate weights is that it is more sensitive to weights, but relatively, each value in the reference window of each pixel needs to be recalculated, resulting in a huge amount of calculation.

The binary correspondence represented by the thicker curve 32 is to set a weight value for each interval of the difference range. Different from the point-to-point correspondence of the index, the binary is to specify the weight in a line-to-point manner, which can not only greatly reduce the amount of calculation, but also ignore the error caused by some noise points.

The cost calculation unit 22 calculates the cost value 23 for the reference pixels of the reference image Ir' and the target pixels of the target image It' through window matching and a preset search range.

In one embodiment, each cost value 23 calculated by the cost calculation unit 22 is a truncated absolute difference between one of the reference pixels and one of the target pixels. More specifically, the difference between pixels will take an absolute value against a preset critical value, the purpose of which is to reduce the impact of noise. In one embodiment, the number of cost values 23 is determined by the size of the search range. The larger the search range, the more times the cost is calculated.

The cost aggregation unit 24 performs cost aggregation calculation according to the weight value 21 and the cost value 23 to generate multiple sets of cost aggregation values 25 corresponding to multiple depth pixels of the initial depth map ID. In one embodiment, the cost aggregation unit 24 performs statistics on the values in the reference window to calculate the sum of pixel difference values according to the weight value 21 and the cost value 23 . In different embodiments, the cost aggregation unit 24 may multiply the corresponding weight value 21 and the cost value 23 in the reference window before summing up, or adopt a manner of separately aggregation in the horizontal and vertical directions. Wherein, each group of cost aggregation values includes a maximum cost aggregation value and a minimum cost aggregation value.

The post-processing unit 26 receives multiple sets of cost aggregation values 25 , and besides finding the minimum cost aggregation value, further calculates the ratio between the maximum cost aggregation value and the minimum cost aggregation value. When the ratio is greater than the preset threshold, the post-processing unit 26 judges that the minimum cost aggregation value is reliable. And when the ratio is smaller than the threshold value, the post-processing unit 26 judges that the minimum cost aggregation value is unreliable. Through the above method, the post-processing unit 26 can obtain the distribution of the reliability mask. In one embodiment, for unreliable pixels, the post-processing unit 26 may generate a modified cost aggregation value. Wherein, the modified cost aggregation value can be generated by averaging or other processing on other surrounding pixel parameters.

The post-processing unit 26 further outputs reliable minimum cost aggregation values corresponding to all depth pixels and modified cost aggregation values to generate an initial depth map ID. After the reliability mask is generated, the initial depth map ID can be refined to improve its correctness.

Therefore, the depth calculation module 10 of the present invention can increase the calculation speed and refine the initial depth map ID through the binary weight correspondence and the generation of the reliability mask.

Please refer to Figure 4. FIG. 4 is a flowchart of a method 400 for generating a stereoscopic image depth map in an embodiment of the present invention. The stereoscopic image depth map generation method 400 can be applied to the stereoscopic image depth map generation device 1 shown in FIG. 1 . The stereoscopic image depth map generation method 400 includes the following steps (it should be understood that the steps mentioned in this embodiment, unless the order is specifically stated, can be adjusted according to actual needs, or even simultaneously or partially executed simultaneously).

In step 401, the depth calculation module 10 receives the reference image Ir and the target image It, and generates an initial depth map ID according to the pixel differences between the reference pixels and the target pixels.

In step 402, the first-order bilateral filter module 12 receives the initial depth map ID to perform first-order bilateral filter calculation on the initial depth map ID and the initial depth map ID itself to generate an average depth map ID'.

In step 403, the second-order bilateral filter module 14 receives the average depth map ID' and the target image It to perform second-order bilateral filter calculation on the average depth map ID' and the target image It to generate a refined depth map IDR.

Certainly, the present invention also can have other multiple embodiments, without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and deformations according to the present invention, but these corresponding Changes and deformations should belong to the scope of protection of the appended claims of the present invention.

Claims (16)

1. a kind of three-dimensional image depth map generation device, which is characterized in that include

One depth calculation module, to receive include multiple reference pixels one with reference to image and comprising multiple object pixels One target image, and an initial depth figure is generated according to the pixel difference of multiple reference pixel and multiple object pixel；

One first rank bilateral filtering module, to receive the initial depth figure, with to the initial depth figure and the initial depth figure It itself carries out one first rank bilateral filtering to calculate, to generate a mean depth figure；And

One second-order bilateral filtering module, to receive the mean depth figure and the target image, with to the mean depth figure And the target image carries out a second-order bilateral filtering and calculates, to generate a refining depth map；

Wherein, which includes：

One cost polymeric unit, to be carried out according to multiple value at costs between multiple reference pixel and multiple object pixel One cost polymeric calculates, to generate multigroup cost polymeric value of multiple depth pixels of the corresponding initial depth figure, one of which Cost polymeric value includes a maximum cost polymeric value and a minimum cost polymerizing value；And

One post-processing unit, to calculate the ratio between the maximum cost polymeric value and the minimum cost polymerizing value, in The ratio judges that the minimum cost polymerizing value is reliable when being more than a threshold value, and judges to be somebody's turn to do when the ratio is less than the threshold value Minimum cost polymerizing value corrects cost polymerizing value for unreliable and generation one；

The post-processing unit further export corresponding all multiple depth pixels the reliable minimum cost polymerizing value and The amendment cost polymerizing value, to generate the initial depth figure.

2. three-dimensional image depth map generation device as described in claim 1, which is characterized in that the first rank bilateral filtering calculates And it is respectively that a bilateral mesh filtering calculates that the second-order bilateral filtering, which calculates,.

3. three-dimensional image depth map generation device as described in claim 1, which is characterized in that the depth calculation module includes：

One weight calculation unit, to judge that the multiple parameters difference between multiple reference pixel is located at a difference range, to refer to Determine the corresponding weighted value of each multiple reference pixel difference range；And

One cost calculation unit, by form matching and a default search area, to multiple reference pixel and this is more A object pixel calculates multiple value at cost；

Wherein the cost polymeric unit is somebody's turn to do according to the corresponding weighted value of multiple reference pixel and multiple value at cost Cost polymeric calculates.

4. three-dimensional image depth map generation device as claimed in claim 3, which is characterized in that each multiple value at cost is respectively One absolute difference of one of multiple reference pixel and one of multiple object pixel cuts value, and the absolute difference cuts value It is taken absolute value by the difference between one of reference pixel and one of object pixel for a preset critical value.

5. three-dimensional image depth map generation device as claimed in claim 3, which is characterized in that the difference range and the weighted value With a binary bit correspondence.

6. three-dimensional image depth map generation device as claimed in claim 3, which is characterized in that the parameter difference is joined for a color Number difference or a spatial parameter difference.

7. three-dimensional image depth map generation device as described in claim 1, which is characterized in that also comprising two converting units, divide This is not converted to after YcbCr color spaces by rgb color space by the depth calculation mould with reference to image and the target image Block processing.

8. three-dimensional image depth map generation device as described in claim 1, which is characterized in that the depth calculation module also includes One mean filter unit is handled this is referred to after image carries out a mean filter.

9. a kind of three-dimensional image depth map production method, which is characterized in that comprise the following steps：

One depth calculation module receives include multiple reference pixels one with reference to image and the mesh comprising multiple object pixels Image is marked, and an initial depth figure is generated according to the pixel difference of multiple reference pixel and multiple object pixel；

One first rank bilateral filtering module receives the initial depth figure, with to the initial depth figure and the initial depth figure itself into One first rank bilateral filtering of row calculates, to generate a mean depth figure；And

One second-order bilateral filtering module receives the mean depth figure and the target image, with to the mean depth figure and should Target image carries out a second-order bilateral filtering and calculates, to generate a refining depth map；

Wherein, it also includes：

One cost polymeric unit of the depth calculation module is according to more between multiple reference pixel and multiple object pixel A value at cost carries out a cost polymeric calculating, to generate multigroup cost polymeric of multiple depth pixels of the corresponding initial depth figure Value, one of which cost polymeric value include a maximum cost polymeric value and a minimum cost polymerizing value；

One post-processing unit of the depth calculation module, to calculate the maximum cost polymeric value and the minimum cost polymerizing value Between a ratio, using in the ratio be more than a threshold value when judge that the minimum cost polymerizing value is reliable, and is less than in the ratio Judge that the minimum cost polymerizing value corrects cost polymerizing value for unreliable and generation one during the threshold value；And

The post-processing unit further export corresponding all multiple depth pixels the reliable minimum cost polymerizing value and The amendment cost polymerizing value, to generate the initial depth figure.

10. three-dimensional image depth map production method as claimed in claim 9, which is characterized in that the first rank bilateral filtering meter It calculates and the second-order bilateral filtering calculates a respectively bilateral mesh filtering and calculates.

11. three-dimensional image depth map production method as claimed in claim 9, which is characterized in that also include

It is poor that one weight calculation unit of the depth calculation module judges that the multiple parameters difference between multiple reference pixel is located at one It is worth range, to specify the corresponding weight of the difference range to multiple reference pixel corresponding to each multiple parameter difference Value；And

One cost calculation unit of the depth calculation module is matched by a form and a default search area, to multiple ginseng It examines pixel and multiple object pixel calculates multiple value at cost；

The cost polymeric unit carries out the cost according to the corresponding weighted value of multiple reference pixel and multiple value at cost Polymerization calculates.

12. three-dimensional image depth map production method as claimed in claim 11, which is characterized in that each multiple value at cost difference Value is cut for an absolute difference of one of multiple reference pixel and one of multiple object pixel, the absolute difference is cut It is worth and is taken absolute value by the difference between one of reference pixel and one of object pixel for a preset critical value.

13. three-dimensional image depth map production method as claimed in claim 11, which is characterized in that the difference range and the weight Value has a binary bit correspondence.

14. three-dimensional image depth map production method as claimed in claim 11, which is characterized in that the parameter difference is a color Parameter difference or a spatial parameter difference.

15. three-dimensional image depth map production method as claimed in claim 11, which is characterized in that also include

This is converted to YcbCr color spaces by two converting units with reference to image and the target image by rgb color space respectively Afterwards by the depth calculation resume module.

16. three-dimensional image depth map production method as claimed in claim 9, which is characterized in that also include

This is referred to after image carries out a mean filter and handled by the mean filter unit that the depth calculation module includes.