CN101720047A - Method for acquiring range image by stereo matching of multi-aperture photographing based on color segmentation - Google Patents

Abstract

The invention discloses a method for obtaining depth images based on multi-eye camera stereo matching based on color segmentation, which includes the steps of: (1), image normalization for all input images; (2), color segmentation for reference images, extract The color consistency area in the image; (3), perform local window matching on multiple input images to obtain multiple disparity maps; (4), apply a bidirectional matching strategy to eliminate the mismatching points generated during the matching process; (5) ), synthesize multiple disparity maps into one disparity map, and fill in the disparity information of mismatching points; (6), perform post-processing optimization on the disparity map, and obtain a dense disparity map; (7), according to the difference between disparity and depth , transforming the disparity map into a depth map. This method obtains depth information from multiple viewpoint images, and utilizes the image information provided by multiple viewpoint images, which can not only solve the mismatching caused by periodically repeated texture features and occlusions in the image, but also improve the matching accuracy and obtain an accurate image. depth image.

Description

A Method of Obtaining Depth Image Based on Multi-camera Stereo Matching Based on Color Segmentation

technical field

The present invention relates to a method for acquiring a depth image, in particular to a method for acquiring a depth image through multi-eye camera stereo matching based on color segmentation.

Background technique

Depth Image Based Rendering (DIBR) technology is a key technology at the decoding end of the 3DTV system. DIBR technology uses color images and their corresponding depth images to generate new viewpoint images by introducing depth information to the visible pixels in the source image. Due to the huge amount of data that needs to be transmitted for compression and transmission of multi-channel video, the required bandwidth will increase significantly, especially if the user is supported to select the viewpoint, it is necessary to use a dense camera array to collect, which makes the amount of multi-view video data increase sharply. , Seriously hamper the application of 3DTV. Therefore, the representation method of "single-viewpoint video + depth" is used as an alternative to stereoscopic video. At the decoding end, DIBR technology is used to generate 3D scenes of one or more virtual viewpoints in real time, which can produce 3D visual effects and satisfy certain viewpoint selection interactions. and "multi-viewpoint video + depth" is currently the 3D video representation method proposed by MPEG/JVT, which can shoot 3D scenes with a relatively sparse camera array and support the interactivity of viewpoint selection. The above 3D video representation using single or multiple 2D color video plus its corresponding depth information can greatly reduce the amount of 3D video data, thereby saving transmission bandwidth. However, the acquisition of the depth map for rendering based on the depth image is a key technology and also a difficulty.

At present, there are mainly two types of methods for depth acquisition. One is to use special hardware devices to actively obtain the depth information of each point in the scene. For example, the Zcam depth camera developed by 3DV Systems is a camera with a ranging function. It uses an infrared pulse light source to send signals to the scene, and then uses an infrared sensor to detect the infrared light reflected back by objects in the scene to determine the depth of the scene. The distance of each point of the object to the camera. Because the price of such system equipment is very expensive, it is not suitable for promotion. Another type of method is based on the traditional computer stereo vision method, using two images of the same scene obtained at two different viewpoints or multiple viewpoint images for stereo matching to recover the depth information of the scene objects. This type of method generally includes two steps: (1) performing stereo matching on the image to obtain the parallax image of the corresponding point; (2) calculating the depth according to the relationship between the parallax and the depth of the corresponding point to obtain the depth image.

Stereo matching algorithms can be mainly divided into two categories: region-based stereo matching algorithms and feature-based stereo matching algorithms. The stereo matching algorithm based on the region (window) can easily restore the parallax in the high texture region, but it will cause a large number of mismatches in the low texture region, resulting in blurred boundaries, and it is also difficult to process the occluded region; The feature points extracted by the feature-based stereo matching method are not too sensitive to noise, so more accurate matching can be obtained, but because the feature points in the image are very sparse, this method can only obtain a sparse disparity map. A segmentation-based stereo matching method has recently received much attention due to its ability to obtain dense disparity maps. Such methods assume that the object structure in the scene consists of a set of non-overlapping planes, each plane corresponds to a color segmented region in the reference image, and the disparity changes in a single color region are smooth.

Contents of the invention

The object of the present invention is to overcome the deficiencies of the prior art and provide a method for obtaining depth images based on multi-eye camera stereo matching based on color segmentation. This method can not only effectively solve the mismatch problem caused by periodically repeated texture features and occlusions in the image, but also improve the matching accuracy, so as to obtain an accurate and dense depth image.

To achieve the above object, design of the present invention is:

Carry out color segmentation on the input reference image according to the color information of the image; then use the reference image to perform local window matching with other input images to obtain multiple initial disparity maps, and fuse multiple initial disparity maps to fill in the disparity information of the mismatching points; finally, the obtained After the disparity map is optimized, the disparity map is converted into a depth map according to the relationship between the disparity and the depth.

According to above-mentioned design, technical scheme of the present invention is:

The above-mentioned method for obtaining a depth image based on multi-eye camera stereo matching based on color segmentation, the steps are:

(1) Perform image normalization on all input images to eliminate color differences of corresponding points on images from different viewpoints caused by image noise, lighting conditions, occlusion and other factors;

(2), carry out color segmentation to reference image, extract the color consistent area in image;

(3), performing local window matching on multiple input images respectively to obtain multiple disparity maps;

(4), apply the two-way matching strategy to eliminate the mismatching points generated during the matching process, and improve the parallax accuracy;

(5), multiple disparity maps are fused into a disparity map according to the fusion criterion, and the disparity information of the mismatching points is filled;

(6) Perform post-processing optimization on the disparity map to obtain a dense disparity map;

(7) Convert the disparity map into a depth map according to the relationship between disparity and depth.

Compared with the prior art, the method for obtaining depth images based on multi-eye camera stereo matching based on color segmentation of the present invention has the following obvious substantive outstanding features and significant advantages: Obtain depth information, rationally use the image information provided by multiple viewpoint images, eliminate the false matching caused by periodically repeating texture features and occlusions in the image, improve the matching accuracy, and obtain the parallax of each pixel, and finally can Obtain an accurate and dense depth map, so as to meet the requirements of the depth image quality based on the image rendering technology, so that it can be applied in the depth acquisition based on the depth image rendering.

Description of drawings

Fig. 1 is the block flow diagram of the method for obtaining depth images based on multi-eye camera stereo matching of color segmentation of the present invention;

Fig. 2 is a schematic diagram of a parallel camera configuration system;

Fig. 3 is the flow chart of carrying out color normalization to image in Fig. 1;

Fig. 4 is a schematic diagram of the occlusion relationship among different viewpoint images of the target image C ₁ , the reference image C _c , and the target image C _r ;

Fig. 5 is a diagram of stereo matching results obtained according to the method of the present invention.

Detailed ways

Embodiments of the present invention will be further described in detail below in conjunction with the accompanying drawings. This embodiment is carried out on the premise of the technical solution of the present invention, and detailed implementation is given, but the protection scope of the present invention is not limited to the following embodiments.

The embodiment of the present invention takes 3 input images as an example, the 3 input images are captured by the parallel camera configuration system shown in Figure 2, the image captured by the middle viewpoint C _c is the reference image C _c , its The images captured by the left and right viewpoints C ₁ , C _r are two target images C ₁ , C _r . f represents the focal length of the three cameras, and the projection points of the space scene point P with the depth value Z on the three image planes are (x ₁ , y), (x _c , y) and (x _r , y) .

Referring to Fig. 1, the method for obtaining the depth image based on multi-eye camera stereo matching of color segmentation of the present invention, its steps are:

(1) Perform image normalization on the three input images to eliminate the color difference of the corresponding points on the images of different viewpoints caused by factors such as image noise, lighting conditions, and occlusion;

(2), carry out color segmentation to reference image, extract the color consistent area in image;

(3) Local window matching is performed on the 3 input images respectively to obtain 2 disparity maps; that is, the reference image is matched with the left target image and the reference image is matched with the right target image;

(4), using two-way matching to eliminate the mismatch points generated during the matching process;

(5), multiple disparity maps are synthesized into a disparity map according to the fusion criterion, and the disparity information of the mismatching points is filled;

(6), optimize the disparity map to obtain a dense disparity map;

(7) According to the relationship between the disparity and the depth, the depth is calculated to convert the disparity map into a depth map.

Referring to Fig. 3, the image normalization is carried out to the input image described in the above step (1), and the color difference of the corresponding point on the images of different viewpoints is eliminated due to factors such as image noise, lighting conditions, occlusion, etc., and the specific steps are:

(1-1), calculate the cumulative histogram of each input image according to the brightness value of the pixel;

(1-2) Taking the C _c viewpoint image as the reference image C _c , divide the cumulative histogram into 10 segments on average according to the number of pixels, and find out the upper and lower brightness boundary values of each segment, thereby determining the reference image C _c and the target The linear mapping relationship between corresponding segments of image C ₁ or target image C _r ;

(1-3) For each pixel in the target image, find its segment number in the cumulative histogram, and then map its brightness value to another value according to the corresponding mapping formula to normalize the pixel.

Carry out color segmentation to the reference image described in above-mentioned steps (2), extract the color consistency area in the reference image, its specific steps are as follows:

The Mean Shift algorithm is used to segment the color of the reference image C _c according to the color information of the image, and the gradient of the probability distribution is used to find the distribution peak, and each pixel in the image is classified into the corresponding density mode, so as to realize clustering, so that the obtained Pixels within each segment have the same color value.

Carry out local window matching respectively to a plurality of input images described in above-mentioned step (3), obtain a plurality of disparity maps, its specific steps are as follows:

(3-1), determining the positional relationship between the reference image C _c and the target image C ₁ and target image C _r ;

When a matching point cannot be found in one target image, its corresponding matching point can be found in another target image. As shown in Figure 4, the long black bars in the figure represent two scene objects, and the image captured by the viewpoint C _c in the middle is set as the reference image C _c , and the images captured by the two viewpoints C ₁ and C _r on the left and right are taken The images of are respectively target image C ₁ and target image C _r . The line segment area P ₁ P ₃ in the scene is occluded at the C ₁ viewpoint. When the stereo matching is used to find the parallax of the reference image C _c , the pixels in this area cannot find a matching point in the C ₁ viewpoint image, but at the C _r viewpoint Corresponding matching points can be found in the image.

(3-2), carry out local window matching to reference image C _c and left target image C ₁ and reference image C _c and right target image C _r respectively;

Taking the above reference image _Cc as the reference image, create a window with a size of 5*5 as the central pixel in the reference image, and search for a pixel neighborhood of the same size as the neighborhood of the point to be matched with a size of 5*5 in the target image, Sequentially compare with the window of the points to be matched, where the self-adapting dissimilarity measure is used as the similarity measurement function, as shown in the following formula, and the point corresponding to the maximum similarity is the best matching point.

C(x, y, d)=(1-ω)*C _SAD (x, y, d)+ω*C _GRAD (x, y, d)

${\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; SAD</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span>$

${\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; GRAD</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; N</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; \&dtri;</span>}_{\mathrm{<span\; class="notranslate">\; x</span>}}{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{<span\; class="notranslate">\; \&dtri;</span>}_{\mathrm{<span\; class="notranslate">\; x</span>}}{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span>$

$<span\; class="notranslate">\; +</span>\underset{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; N</span>}}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; \&dtri;</span>}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{<span\; class="notranslate">\; \&dtri;</span>}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span>$

表示图像梯度的水平分量，

表示图像梯度的垂直分量，ω表示权重。Among them, N(x, y) is a 5*5 window with the matching point (x, y) as the center pixel,

represents the horizontal component of the image gradient,

Represents the vertical component of the image gradient, and ω represents the weight.

The disparity map I _LI (x, y) is obtained by matching the C ₁ viewpoint image with the C _c viewpoint image (reference image). The C _r viewpoint image is matched with the C _c viewpoint image (reference image) to obtain the disparity map I _RI (x, y). The two obtained disparity maps contain many mismatching points.

The two-way matching described in the above step (4) is used to eliminate the false matching points generated in the matching process, and it is used in the two identical matching processes of the reference image matching the left target image and the reference image matching the right target image, The specific steps are as follows:

(4-1), take the left image as the reference image, and the right image as the target image, and perform local window matching from left to right to obtain the disparity map d _LR from left to right;

(4-2), take the right image as the reference image, and the left image as the target image, perform local window matching from right to left to obtain the disparity map d _RL from right to left;

(4-3). According to the following formula, find out the corresponding point with inconsistent parallax in the two disparity maps, and determine it as a mismatching point.

${\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; LR</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; RL</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; R</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}\frac{{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; LR</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; RL</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; R</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; 2</span>}}& <span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; LR</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; RL</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; R</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; \&lambda;</span>}\\ \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; else</span>}\end{array}\right.$

Where: λ is the error threshold, the pixel (x _L , y) on the left image and the pixel (x _R , y) on the right image are a pair of matching points, that is, x _R = x _L +d _LR (x _L , y ).

When the parallax error of the corresponding point in the two disparity maps satisfies |d _LR (x _L , y)-d _RL (x _R , y)|≤λ (λ is the allowed parallax error threshold), it indicates that the parallax of the corresponding point matches correct. When the disparity error does not satisfy |d _LR (x _L , y)-d _RL (x _R , y)|≤λ, it indicates that the point is a mismatch point, and the point is assigned a value of 0.

As described in the above step (5), multiple disparity maps are synthesized into a disparity map, and the disparity information of the mismatching points is filled, and the specific steps are as follows:

(5-1), calculate the proportional coefficient α according to the translation vector t in the external parameter matrix of the camera;

$\mathrm{<span\; class="notranslate">\; \&alpha;</span>}<span\; class="notranslate">\; =</span>\frac{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; C</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}<span\; class="notranslate">\; |</span>}{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; C</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; +</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; C</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; R</span>}}<span\; class="notranslate">\; |</span>}$

Among them, t _L , t _C , and t _R are translation vectors in the extrinsic parameter matrices of the left camera, the middle camera, and the right camera, respectively.

(5-2), according to the following fusion criteria, the two parallax maps I _LI (x, y) and I _RI (x, y) are synthesized into the final parallax map I (x, y), and the gaps of the mismatch points are filled Parallax information. The fusion method is expressed by the formula as follows:

Among them, I(x, y) is the final synthesized disparity map of the reference image coordinates. I _LI (x, y) and I _RI (x, y) respectively represent the disparity maps obtained by matching the reference image with its adjacent left and right images. δ represents an error threshold.

The parallax map described in the above step (6) is optimized to obtain a dense parallax map. The specific steps are as follows:

(6-1), assuming that the parallax change in each color segmented region in the reference image is smooth;

(6-2) Take the median parallax value of all pixels in each segmented area as the parallax of the entire segmented area to obtain the parallax value of each pixel, and its mathematical expression is as follows. Finally, a high-quality dense disparity map I'(x, y) is obtained.

${\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; SEG</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; SEG</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; median</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; I</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>$

Among them, I _SEG (x, y) represents a segmented region.

According to the relationship between the parallax and the depth described in the above step (7), calculate the depth and convert the parallax map into a depth map, which is specifically:

The depth value of the scene in the parallel camera configuration system has the following relationship with its disparity:

$\mathrm{<span\; class="notranslate">\; Z</span>}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; Bf</span>}}{\mathrm{<span\; class="notranslate">\; D.</span>}}$

Among them, Z represents the depth value, B represents the baseline distance (camera distance), f is the camera focal length, and D is the parallax.

According to the relationship between depth and disparity, when the disparity is known, the depth value of each pixel is calculated, so that the disparity map I'(x, y) is converted into a depth map.

Figures (a) and (b) in Figure 5 are the matching results obtained according to the method of the present invention. Among them, Figure (a) is the matching result for static scene sequences, and Figure (b) is the matching result for dynamic scene sequences. It can be seen from Figure 5 that the final matching result can maintain accurate disparity map boundaries, especially the disparity of some occlusion areas that lack matching has also been well restored, and the depth level of scene objects is very obvious. According to the method of the present invention, an ideal matching effect can be obtained, thus also verifying the effectiveness of the present invention.

Claims (8)

1. A method for obtaining depth images based on multi-eye camera stereo matching of color segmentation, characterized in that, the method carries out color segmentation to the input reference image according to the color information of the image; then utilizes the reference image to carry out local window matching with the rest of the input images Multiple initial disparity maps are obtained, and multiple initial disparity maps are fused to fill in the disparity information of mismatching points; finally, after optimizing the obtained disparity maps, the disparity maps are converted into depth maps according to the relationship between disparity and depth. Specific steps are as follows: (1) Perform image normalization on all input images to eliminate color differences of corresponding points on images from different viewpoints caused by image noise, lighting conditions, occlusion and other factors; (2), carry out color segmentation to reference image, extract the color consistent area in image; (3), performing local window matching on multiple input images respectively to obtain multiple disparity maps; (4), apply the two-way matching strategy to eliminate the mismatching points generated during the matching process, and improve the parallax accuracy; (5), multiple disparity maps are synthesized into a disparity map according to the fusion criterion, and the disparity information of the mismatching points is filled; (6) Perform post-processing optimization on the disparity map to obtain a dense disparity map; (7) Convert the disparity map into a depth map according to the relationship between disparity and depth. 2. the method for obtaining the depth image based on the multi-eye camera stereo matching of color segmentation according to claim 1, is characterized in that, all input images described in the above-mentioned steps (1) carry out image normalization, and its concrete steps are: (1-1), calculate the cumulative histogram of each input image according to the brightness value of the pixel; (1-2) Taking the C _c viewpoint image as the reference image C _c , divide the cumulative histogram into 10 segments on average according to the number of pixels, and find out the upper and lower brightness boundary values of each segment, thereby determining the reference image C _c and the target The linear mapping relationship between corresponding segments of image C ₁ and target image C _r ; (1-3). For each pixel in the target image, calculate its segment number in the cumulative histogram, and then map its brightness value to another value according to the corresponding mapping formula. 3. the method for obtaining the depth image based on the multi-eye camera stereo matching of color segmentation according to claim 2, is characterized in that, the reference image described in above-mentioned step (2) is carried out color segmentation, and the color in the extraction reference image is consistent The specific steps are as follows: Use the Mean Shift algorithm to segment the color of the reference image _Cc according to the color information of the image, use the gradient of the probability distribution to find the distribution peak, and classify each pixel in the image into the corresponding density mode, In this way, clustering is realized, so that the pixels in each obtained segmented region have the same color value. 4. the method for obtaining the depth image based on the multi-eye camera stereo matching of color segmentation according to claim 3, is characterized in that, the described in above-mentioned step (3) carries out local window matching respectively to a plurality of input images, obtains a plurality of Disparity map, the specific steps are as follows: (3-1), determining the positional relationship between the reference image C _c and the target image C ₁ and target image C _r ; When no matching point can be found in one target image, the corresponding matching point can be found in another target image, and the image captured by the middle viewpoint C _c is set as the reference image C _c , and the left and right two The images taken by two viewpoints C ₁ and C _r are the target image C ₁ and the target image C _r respectively, and the P ₁ P ₃ line segment area in the scene is blocked at the C ₁ viewpoint. When the stereo matching is used to calculate the parallax of the reference image C _c , the pixels in this area cannot find a matching point in the C ₁ viewpoint image, but can find a corresponding matching point in the C _r viewpoint image; (3-2), carry out local window matching to reference image C _c and left target image C ₁ and reference image C _c and right target image C _r respectively; Take the above reference image C _c as the reference image, create a window with a size of 5*5 as the center pixel of the reference image to be matched, and search for a pixel neighborhood with the same size as the neighborhood of the point to be matched with 5*5 in the target image, and then Compare with the window of the point to be matched, where the self-adapting dissimilarity measure is used as the similarity measurement function, as shown in the following formula, the point corresponding to the maximum similarity is the best matching point, C(x, y, d)=(1-ω)*C _SAD (x, y, d)+ω*C _GRAD (x, y, d) ${\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; SAD</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span>$ ${\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; GRAD</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; N</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; \&dtri;</span>}_{\mathrm{<span\; class="notranslate">\; x</span>}}{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{<span\; class="notranslate">\; \&dtri;</span>}_{\mathrm{<span\; class="notranslate">\; x</span>}}{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span>$ $<span\; class="notranslate">\; +</span>\underset{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; N</span>}}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; \&dtri;</span>}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{<span\; class="notranslate">\; \&dtri;</span>}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span>$ Among them, N(x, y) is a 5*5 window with the matching point (x, y) as the center pixel,

represents the horizontal component of the image gradient, Represents the vertical component of the image gradient, ω represents the weight, By matching the C ₁ viewpoint image with the C _c viewpoint image (reference image), the disparity map I _LI (x, y) is obtained, and the C _r viewpoint image is matched with the C _c viewpoint image (reference image), and the disparity map I _RI (x , y), the obtained two disparity maps contain many mismatching points. 5. the method for obtaining the depth image based on the multi-eye camera stereo matching of color segmentation according to claim 4, is characterized in that, adopting two-way matching described in above-mentioned step (4) eliminates the false matching point that produces in matching process , which is used in the same matching process of matching the reference image with the target image on the left and matching the reference image with the target image on the right. The specific steps are as follows (4-1), take the left image as the reference image, and the right image as the target image, and perform local window matching from left to right to obtain the disparity map d _LR from left to right; (4-2), take the right image as the reference image, and the left image as the target image, perform local window matching from right to left to obtain the disparity map d _RL from right to left; (4-3), according to the following formula, find out the corresponding points with inconsistent parallaxes in the two parallax maps, and determine them as mismatching points, ${\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; LR</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; RL</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; R</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}\frac{{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; LR</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; RL</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; R</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; 2</span>}}& <span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; LR</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; RL</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; R</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; \&lambda;</span>}\\ \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; else</span>}\end{array}\right.$ Where: λ is the error threshold, the pixel (x _L , y) on the left image and the pixel (x _R , y) on the right image are a pair of matching points, that is, x _R = x _L +d _LR (x _L , y ), When the parallax error of the corresponding point in the two disparity maps satisfies |d _LR (x _L , y)-d _RL (x _R , y)|≤λ (λ is the allowed parallax error threshold), it indicates that the parallax of the corresponding point matches Correct, when the parallax error does not satisfy |d _LR (x _L , y)-d _RL (x _R , y)|≤λ, it indicates that the point is a mismatch point, and the point is assigned a value of 0. 6. the method for obtaining the depth image based on the multi-eye camera stereo matching of color segmentation according to claim 5, is characterized in that, described in above-mentioned step (5), a plurality of parallax maps are synthesized into one parallax map, fill in the error To match the disparity information of the points, the specific steps are as follows: (5-1), calculate the proportional coefficient α according to the translation vector t in the external parameter matrix of the camera; $\mathrm{<span\; class="notranslate">\; \&alpha;</span>}<span\; class="notranslate">\; =</span>\frac{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; C</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}<span\; class="notranslate">\; |</span>}{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; C</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; +</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; C</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; R</span>}}<span\; class="notranslate">\; |</span>}$ Among them, t _L , t _C , and t _R are the translation vectors in the extrinsic matrix of the left camera, the middle camera, and the right camera respectively; (5-2), according to the following fusion criteria, the two parallax maps I _LI (x, y) and I _RI (x, y) are synthesized into the final parallax map I (x, y), and the gaps of the mismatch points are filled Disparity information, the fusion method is expressed by the formula as follows:

Among them, I(x, y) is the disparity map finally synthesized by the coordinates of the reference image, I _LI (x, y), I _RI (x, y) respectively represent the disparity map obtained by matching the reference image with its adjacent left and right images, δ represents an error threshold. 7. the method for obtaining the depth image based on the multi-eye camera stereo matching of color segmentation according to claim 6, is characterized in that, the parallax map described in the above-mentioned step (6) is optimized and processed to obtain a dense parallax map, which Specific steps are as follows: (6-1), assuming that the parallax change in each color segmented region in the reference image is smooth; (6-2), take the median disparity value of all pixels in each segmented area as the disparity of the entire segmented area, and obtain the disparity value of each pixel. Its mathematical expression is as follows, and finally a high-quality Dense disparity map I'(x,y), ${\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; SEG</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; SEG</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; median</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; I</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>$ Among them, I _SEG (x, y) represents a segmented region. 8. the method for obtaining the depth image based on the multi-eye camera stereo matching of color segmentation according to claim 7, is characterized in that, according to the relation between parallax and depth described in above-mentioned step (7), calculate depth and disparity map Converted to a depth map, specifically: The depth value of the scene in the parallel camera configuration system has the following relationship with its disparity: $\mathrm{<span\; class="notranslate">\; Z</span>}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; Bf</span>}}{\mathrm{<span\; class="notranslate">\; D.</span>}}$ Among them, Z represents the depth value, B represents the baseline distance (camera distance), f is the focal length of the camera, and D is the parallax. According to the relationship between depth and parallax, the depth value of each pixel is calculated when the parallax is known, so that Transform the disparity map I'(x, y) into a depth map.

Images