CN105678765A - Texture-based depth boundary correction method - Google Patents

Abstract

The present invention relates to a texture-based depth boundary correction method. The method comprises the steps of A1, inputting a texture image and a corresponding depth image, wherein the images are acquired by a depth sensor (such as a Kinect); A2, extracting the boundary of the texture image and the boundary of the depth image, and acquiring a depth boundary dislocation figure with the boundary of the texture image as the reference; A3, calculating the pixel value difference between an accurate depth point (sweet spot) and an error depth point (dead point) and determining a boundary dislocation area; A4, according to the distribution characteristics of the boundary dislocation area, adaptively determining the side length of a square window for depth enhancement processing, correcting the depth of the dead point in the window, eliminating the boundary dislocation area. According to the invention, the accuracy and the time-domain stability of the boundary of the depth image, acquired by the Kinect and other low-end depth sensors, are significantly improved. The method is applied to the fields of three-dimensional reconstruction, free viewpoint video coding and the like, and can effectively improve the scene three-dimensional reconstruction quality and the coding efficiency.

Description

A kind of depth boundary modification method based on texture

Technical field

The present invention relates to computer vision and digital image processing field, particularly to a kind of depth boundary modification method based on texture.

Background technology

Depth image is commonly used for the numerous areas such as three-dimensional reconstruction, free view-point coding. Existing depth information collection is often based on some complicated and expensive sensors, such as structured light camera or laser range finder etc. Based on the depth image that these equipment gather, not only to be limited by all kinds of noise jamming, too low resolution also greatly limit the development of other research application based on by depth image. With the Kinect low side depth transducer being representative, their price is cheap and as active sensor, it is possible to the degree of depth of quick obtaining scene and texture information, thus is widely used under study for action. But owing to low side depth transducer mainly through emitting structural light and receives its reflection light acquisition scene depth image, therefore suffer from the severe jamming of environment light source and body form material, the depth image noise causing collection is relatively big, especially there is the problem that serious shake accuracy is low in object boundary area. Containing in scene the important informations such as the shaped position of object due to borderline region, and belong to visual acuity region, depth boundary error badly influences other application based on depth image, for instance scene three-dimensional reconstruction etc.

In order to improve the quality in low side depth transducer sampling depth image boundary region, propose a series of improved method currently for its depth image border image defects. Wherein representative method has two kinds: to depth image two-sided filter pretreatment, then on spatial domain, depth map is divided into non-borderline region and borderline region, depth information is strengthened by special weighting method for borderline region, the method reduces the depth image noise in object boundary area, but owing to lacking the restriction to time-domain stability, adjacent each frame boundaries depth value still suffers from larger fluctuation; The texture weighted mean utilizing multiframe corresponding region depth information in time domain improves depth image stability, depth image is made to obtain very big improvement in time domain concordance from the method carrying out associating bilateral filtering time domain and spatial domain present frame depth map, the seriality of smooth surface depth value is obviously improved, but owing to ignoring the process to huge object boundary affected by noise, the filter effect of borderline region is not ideal enough, and boundary misalignment needs to be suppressed.

Summary of the invention

It is an object of the invention to provide a kind of depth boundary modification method based on texture, borderline region is better processed.

For this, the present invention propose one, it is characterised in that comprise the steps: A1, input depth transducer gather texture image and corresponding depth image; A2, respectively extraction texture image border and depth image border, obtain depth boundary dislocation figure with Texture Boundaries for benchmark; A3, calculate the better pixel value difference with bad point, it is determined that boundary misalignment region; Wherein better referring to accurate depth point, bad point refers to error depth point; A4, according to dislocation zone distribution characteristics, adaptively determine the square window W length of side and carry out degree of depth enhancement process, revise the bad point degree of depth in window, eliminate boundary misalignment region; Wherein ensureing better in window W being dominant in step A4, namely better number Ngood crosses bad point number Nbad.

It is an advantage of the current invention that:

1) boundary misalignment region is ensuredFurther, boundary misalignment region E can be completely eliminated by algorithm_r;

2) owing to, before degree of depth enhancement process, namely revising D_badBefore, to be always ensured that in W better is dominant, and therefore improves all better depth weighted averages in window WReliability, namely ensure dislocation district comprised by window completely and in window better quantity cross bad point quantity, thus ensure that the robustness of degree of depth enhancement process.

Accompanying drawing explanation

Fig. 1 is embodiment of the present invention schematic flow sheet.

Detailed description of the invention

In order to obtain object boundary clearly smooth, shape complete, position high-quality depth image accurately, it is necessary to for the borderline region of depth image, the advantage of comprehensive various improvement depth image methods, learn from other's strong points to offset one's weaknesses. The thinking of the embodiment of the present invention is based primarily upon depth image borderline region and there are shake and two subject matters of dislocation, by stable and corresponding texture image border accurately, borderline region is carried out degree of depth enhancement process, reach to promote spatial domain accuracy and time-domain stability, suppress the purpose of the boundary error of the depth image existence of low side depth transducer collection comprehensively, conveniently further depth information is dissolved in other application.

The present embodiment is compared with corresponding depth image border by extracting each frame texture image border, obtains the misalignment of depth boundary intuitively. Along depth boundary node-by-node algorithm normal direction gradient value, obtain error degree of depth representative value and accurate depth representative value near this point, acquisition is delimitd a boundary line dislocation zone criterion, utilize this criterion to divide and point out accurate depth point (better) and error depth point (bad point) the depth boundary point direction extension ray being sent to correspondence from Texture Boundaries, bad point constitutes the depth boundary dislocation needing to eliminate, and between every pair of corresponding Texture Boundaries point and depth boundary point, the position of dislocation and size are determined. Distribution situation according to dislocation zone, adaptively determines the square process window length of side, it is ensured that dislocation district is comprised by window completely and in window, better quantity crosses bad point quantity, thus improving the reliability of the degree of depth enhancement process eliminating dislocation zone. This ensure that the accurately clear of border in depth image based on the depth boundary dislocation zone modification method of Texture Boundaries deeply while eliminating border shake, finally it is effectively improved with the depth image quality of the Kinect low side sensor acquisition being representative, is extensively benefited making based on each research field of depth information.

The proposed a kind of depth boundary dislocation zone modification method based on Texture Boundaries of the present embodiment is as follows:

A1: input texture image and corresponding depth image that depth transducer (such as Kinect) gathers;

A2: extract texture image border and depth image border respectively, obtains depth boundary dislocation figure with Texture Boundaries for benchmark;

A3: calculate the pixel value difference of accurate depth point (better) and error depth point (bad point), it is determined that boundary misalignment region;

A4: according to dislocation zone distribution characteristics, adaptively determines the square window length of side and carries out degree of depth enhancement process, revise the bad point degree of depth in window, eliminate boundary misalignment region.

In particular embodiments, can operate by following mode. It is noted that the concrete grammar (such as Sobel operator, weighted mean approach etc.) described in implementation process below is all only and enumerates explanation, the scope that the present invention contains is not limited to these cited methods.

A1: each frame texture image of input and corresponding depth image gather gained respectively through the color camera of low side depth transducer with depth camera, for Kinect, the information of Same Scene in the same time is comprised mutually by the texture image of the same frame of sensor acquisition and depth image, they have identical resolution, but owing to depth camera camera lens and color camera lens exist a determining deviation, need to through camera calibrated, two images just can exist corresponding relation, the texture being respectively described in scene same point and depth information between the pixel of same position.

A2: extract the present frame texture image border T of input in A1_eWith depth image border D_e, will extract Texture Boundaries T_eIt is mapped on depth image, due to corresponding relation between the texture image of same frame and depth image pixel after calibration, thus be arranged in the borderline pixel of texture image and can be quickly found out the corresponding pixel points of its depth image, obtain the depth boundary dislocation figure being benchmark with Texture Boundaries.

Image boundary extraction specifically can adopt the methods such as (but being not limited to) Sobel operator, Roberts operator, Prewitt operator, and the difference of said method is in that to there are differences for the extraction accuracy on dissimilar border.

A3: determine and the misalignment between corresponding depth boundary point along Texture Boundaries pointwise. Obtain with Texture Boundaries point P_TeFor end points to its nearest neighbours depth boundary point P_DeDirection extends pixel P on ray and has belonged to point set P_goodWith bad point collection P_badFoundation, need to based on the depth value D of pixel P each on ray_PDivide, determine boundary misalignment region E with this_r. For determining better depth value D_goodWith bad point depth value D_badDifference TH₂, its normal direction P need to be obtained along depth boundary pointwise_De ^⊥Concentration gradient value Grad. Relatively each point depth value D on ray_PWith end points depth value D_PTeBetween difference and TH₂Magnitude relationship divide better P_goodWith bad point P_bad, all bad points constitute dislocation district E_r. Criteria for classifying and dislocation zone are formed as follows:

TH₂=Grad_PD⊥e

$P\&Element;\left\{\begin{array}{ccc}{P}_{good},& if& \left|D_P-D_{P_{Te}}\right|\&GreaterEqual;{\mathrm{TH}}_2\\ P_{bad},& if& \left|D_P-D_{P_{Te}}\right|<{\mathrm{TH}}_2\end{array}\right.$

P_bad∈E_r

A4: determined dislocation zone E by A3_rSize and location distribution characteristics, adaptively determine window W length of side TH₁, this window W is carried out degree of depth enhancement process, revises bad point degree of depth D in window_bad, eliminate boundary misalignment region E_r. As described in A3, it is thus achieved that better depth value D_goodWith bad point depth value D_badDifference TH₂, divide with Texture Boundaries point P_Te ⁰For end points, to closest depth boundary point P_De ⁰On the ray that direction extends, pixel P has belonged to point set P_goodWith bad point collection P_bad, bad point number N on note ray_badFor N_bad ⁰, for ensureing better in window W being dominant, i.e. better number N_goodCross bad point number N more_bad, then N_goodIt is at least N_bad ⁰+ 1, therefore set square window W length of side TH₁Initial value be N_good+N_bad=2N_bad ⁰+ 1. Method described in A3, adds up adjacent Texture Boundaries point P_Te ⁰Rear TH₁-1 Texture Boundaries point P_Te ^k(k=1 ..., TH₁-1), initial TH on every corresponding ray₁Better quantity in-1 point and bad point quantity, respectively better total number N in cumulative final window W_goodNumber N total with bad point_bad.Still it is dominant if better, then by all better depth weighted averages in window W, is calculated as follows:

${\tilde{D}}_{good}=\left(\underset{i=1}{\overset{N_{good}}{\&Sigma;}}{w}_i{D}_{good}^i\right)/N_{good}$

Revise bad point depth value D_bad, eliminate the dislocation zone E in window W_r, weight w in weighted mean algorithm_i(i=1 ..., N_good) calculating can consider in target bad point that (but being not limited to) to revise and window W other better between the information such as dependency of texture and position; Otherwise, if better quantity is less than bad point quantity, then square shaped window W length of side TH₁Add 1, again better total number N in statistics new window_goodNumber N total with bad point_bad, it is dominant until better in window W. It is as follows that algorithm performs step:

S1: initialize TH₁=2N_bad ⁰+ 1;

S2: statistics T_eUpper some P_Te ⁰Rear TH₁-1 some P_Te ^k(k=1 ..., TH₁-1) the initial TH of corresponding ray₁N in-1 point_good ^kAnd N_bad ^k, better total number N in cumulative window W respectively_goodNumber N total with bad point_badAs follows:

$N_{good}=\underset{k=0}{\overset{2{\mathrm{TH}}_1-1}{\&Sigma;}}{N}_{good}^k,N_{bad}=\underset{k=0}{\overset{2{\mathrm{TH}}_1-1}{\&Sigma;}}{N}_{bad}^k$

S3: if N_good-N_bad> 0 turn of S4, otherwise turn S5;

S4:Process and stop;

S5:TH₁=TH₁+ 1, turn S2.

The length of side TH of square window W is determined according to above-mentioned algorithm₁, there is following two advantage:

1) ensureBy algorithm S4, E can be completely eliminated_R;

2) owing to, before degree of depth enhancement process, namely revising D_badBefore, to be always ensured that in W better is dominant, and therefore improves all better depth weighted averages in window WReliability, thus ensure that the robustness of degree of depth enhancement process.

The foregoing is only embodiments of the invention; not thereby the scope of the claims of the present invention is limited; every equivalent device utilizing description of the present invention and accompanying drawing content to make or equivalent method conversion; or directly or indirectly it is used in other relevant technical fields, all in like manner include in the scope of patent protection of the present invention.

Claims (9)

1. the depth boundary modification method based on texture, it is characterised in that comprise the steps:

A1, the texture image inputting depth transducer collection and corresponding depth image;

A2, respectively extraction texture image border and depth image border, obtain depth boundary dislocation figure with Texture Boundaries for benchmark;

A3, calculate better P_goodWith bad point P_badPixel value difference, it is determined that boundary misalignment region; Wherein better referring to accurate depth point, bad point refers to error depth point;

A4, according to dislocation zone distribution characteristics, adaptively determine the square window W length of side and carry out degree of depth enhancement process, revise the bad point degree of depth in window, eliminate boundary misalignment region E_r;

Wherein ensureing better in window W being dominant in step A4, namely better number Ngood crosses bad point number Nbad. .

2. the depth boundary modification method based on texture as claimed in claim 1, it is characterised in that: in step A2, adopt one of following operator to extract texture image border Te and depth image border De:Sobel operator, Roberts operator, Prewitt operator; And the Texture Boundaries Te of extraction is mapped on depth image, obtain the depth boundary dislocation figure being benchmark with Texture Boundaries.

3. the depth boundary modification method based on texture as claimed in claim 1, it is characterised in that: in step A3, the better distribution situation with bad point along Texture Boundaries point-to-point analysis ray, namely based on the depth value D of pixel P each on ray_PDivide better P_goodWith bad point P_bad, it is determined that boundary misalignment region E_r; Described ray refers to the line that Texture Boundaries point extends for end points to its nearest neighbours depth boundary point direction.

4. the depth boundary modification method based on texture as claimed in claim 1, it is characterised in that: in described step A4, determine degree of depth enhancement process square window W length of side TH according to good bad point distribution characteristics self adaptation₁, revise bad point degree of depth D in window_bad, eliminate boundary misalignment region E_r。

5. the depth boundary modification method based on texture as claimed in claim 4, it is characterised in that: in step A4, also comprise the steps: to obtain better depth value D_goodWith bad point depth value D_badDifference TH₂, divide with Texture Boundaries point P_Te ⁰For end points, to closest depth boundary point P_De ⁰On the ray that direction extends, pixel P has belonged to point set P_goodWith bad point collection P_bad, on note ray, bad point number is N_bad ⁰, set square window W length of side TH₁Initial value be 2N_bad ⁰+ 1, add up adjacent Texture Boundaries point P_Te ⁰Rear TH₁-1 Texture Boundaries point P_Te ^k(k=1 ..., TH₁-1), every initial TH of corresponding ray₁Better quantity in-1 point and bad point quantity, respectively better total number N in cumulative final window W_goodNumber N total with bad point_bad。

6. the depth boundary modification method based on texture as claimed in claim 5, is characterized in that in step A4, if also comprising the steps: better to be still dominant, then by all better depth weighted averages in window W, is calculated as follows:

${\tilde{D}}_{good}=\left(\underset{i=1}{\overset{N_{good}}{\&Sigma;}}{w}_i{D}_{good}^i\right)/N_{good}$

Revise bad point depth value D_bad, eliminate the dislocation zone E in window W_r;Otherwise, square shaped window W length of side TH₁Add 1, again better total number N in statistics new window_goodNumber N total with bad point_bad, it is dominant until better in window W.

7. the depth boundary modification method based on texture as claimed in claim 6, is characterized in that: described step A4 includes following algorithm and performs step:

S1: initialize TH₁=2N_bad ⁰+ 1;

S2: statistics T_eUpper some P_Te ⁰Rear TH₁-1 some P_Te ^k(k=1 ..., TH₁-1) the initial TH of corresponding ray₁N in-1 point_good ^kAnd N_bad ^k, better total number N in cumulative window W respectively_goodNumber N total with bad point_badAs follows:

$N_{good}=\underset{k=0}{\overset{2{\mathrm{TH}}_1-1}{\&Sigma;}}{N}_{good}^k,N_{bad}=\underset{k=0}{\overset{2{\mathrm{TH}}_1-1}{\&Sigma;}}{N}_{bad}^N$

S3: if N_good-N_bad> 0 turn of S4, otherwise turn S5;

S4: $D_{bad}={\tilde{D}}_{good},$ Process and stop;

S5:TH₁=TH₁+ 1, turn S2.

8. the depth boundary modification method based on texture as claimed in claim 1, it is characterized in that: in described step A3, the better distribution situation with bad point along Texture Boundaries point-to-point analysis ray, described ray refers to the line that Texture Boundaries point extends for end points to its nearest neighbours depth boundary point direction, namely based on the depth value D of pixel P each on ray_PBetter P is divided with the absolute value of the difference of end points depth value_goodWith bad point P_bad, it is determined that boundary misalignment region E_r。

9. the depth boundary modification method based on texture as claimed in claim 8, is characterized in that determining better depth value D as follows_goodWith bad point depth value D_badDifference TH₂: obtain its normal direction P along depth boundary pointwise_De ^⊥Concentration gradient value Grad, as determining dislocation district E_rThe foundation of distribution, namely divides with apart from this depth boundary point P_DeNearest Texture Boundaries point P_TeFor end points, to depth boundary point P_DeOn the ray that direction extends, pixel P has belonged to point set P_goodWith bad point collection P_badFoundation, criteria for classifying is as follows:

${\mathrm{TH}}_2={\mathrm{Grad}}_{P_{De}^{\&perp;}}$

$P\&Element;\left\{\begin{array}{ccc}{P}_{good},& if& |D_P-D_{P_{Te}}|\&GreaterEqual;{\mathrm{TH}}_2\\ P_{bad},& if& |D_P-D_{P_{Te}}|<{\mathrm{TH}}_2\end{array}\right.$

P_bad∈E_r。