CN102075779A - Intermediate view synthesizing method based on block matching disparity estimation - Google Patents

Abstract

The invention discloses an intermediate view synthesizing method based on block matching disparity estimation. In order to carry out disparity estimation on two images selected from the same scene, the same time and different angles, a virtual view at any position on a straight line (baseline) between original views is obtained through image interpolation according to the acquired disparity image. The method comprises the following steps: firstly carrying out disparity estimation by taking a left view and a right view as target images respectively, and computing two disparity views from the left view to the right view as well as from the right view to the left view by adopting a disparity estimation algorithm based on the block matching; then synthesizing a preliminary intermediate view according to the disparity views; and finally carrying out hole filling on the synthetic view to obtain a final result. Aiming at the characteristic that the left view and the right view only have parallax in the horizontal direction because shooting is carried out by a camera on a horizontal line, search is only carried out in the horizontal direction, the amount of calculation is reduced, and the algorithm running time is shortened; and the experimental result indicates that a favorable synthetic result can be obtained under the condition of a farther distance of view foreground to the camera or smooth image gradation changes.

Description

An Intermediate View Synthesis Method Based on Block Matching Disparity Estimation

technical field

The invention relates to a processing method of a binocular digital image, in particular to a method for synthesizing intermediate viewpoints of a binocular digital image.

Background technique

With the continuous development of multimedia technology, image and video technology is also developing from two-dimensional to three-dimensional, and interactivity will become a main feature of future multimedia technology. One of the key technologies of interactive 3D video system is virtual view synthesis technology. Virtual viewpoint rendering synthesis is an indispensable module for all stereoscopic display system terminals, and it also plays a vital role in high-end multimedia fields such as remote video conferencing and free viewpoint stereoscopic TV. In order to enable users to roam in the scene and realize "continuous look around", the number of cameras should be as many as possible during the multi-view video collection process, but due to the impracticality of placing infinite cameras to achieve seamless switching of viewpoints, In order to display the view of any viewpoint, the virtual viewpoint must be synthesized on the client side, and the viewpoint that the user wants to observe must be synthesized by analyzing the existing viewpoint. Therefore, virtual view synthesis technology is a very important emerging technology in the field of multimedia.

Virtual viewpoint synthesis technology is divided into model-based rendering (GBR) and image-based rendering (IBR) in principle. The advantage of GBR is that the geometric model is comprehensive and can meet the needs of flexible and changeable viewing angles. The traditional algorithm has been developed relatively mature and has good interactivity. However, this method has two main disadvantages, that is, creating a high-precision 3D object and scene model is computationally intensive and almost impossible; and the calculation is time-consuming, with poor real-time performance, and the rendering time depends on the complexity of the object. Also, building a 3D model may not be necessary if the ultimate goal is only to synthesize mesoscopic vision. In view of this, the IBR method, which can directly use the real scene as the reference viewpoint image to synthesize a new view, has gradually received attention and development. The view generated by the IBR method has a strong sense of reality and is better integrated with the scene, and because it avoids the reconstruction process, it is more convenient for real-time rendering of complex scenes, and the computational complexity does not change with the scene. modeling features. The method of intermediate view synthesis based on parallax is just a branch of IBR.

Among the various IBR methods, Depth-Image Based Rendering (DIBR) has developed relatively maturely. This type of method is obtained by using the depth camera or using the geometric constraints and disparity information between the original views. Each color pixel corresponds to the depth information, and draws a virtual image of the three-dimensional scene according to the depth information. For example, Jiang Gangyi et al proposed to use the smoothness of the reference viewpoint depth map to divide the color image into blocks of different sizes, and then perform three-dimensional image transformation to project each block to the virtual viewpoint to complete the drawing of the virtual view of the viewpoint. Method (see Jiang Gangyi, Zhu Bo, Yu Mei. A virtual viewpoint image rendering method based on scalable blocks [P]. Chinese invention patent: 200910153324.8, 2009-10-14.). Another example is that Luo Kai et al. proposed a method of morphologically processing the depth image first, then using the viewpoint transformation equation to generate the target viewpoint, and finally using an image inpainting algorithm to post-process the target viewpoint containing holes (see Luo Kai, Li Dongxiao, Feng Yamei , Zhang Ming. Arbitrary viewpoint rendering based on DIBR and image inpainting [J]. Chinese Journal of Image and Graphics, 2010, 15(3): 443-449.). The DIBR method introduces the depth information of the scene into the virtual viewpoint scene, which reduces the number of reference viewpoints required for virtual viewpoint rendering, but it is very difficult to obtain the depth image. The depth camera can directly obtain the scene depth information, but it is expensive and many It is difficult to calibrate the target camera group; it is very difficult to directly extract the depth information of each pixel from the color image, because the pumping density and accuracy of the depth information greatly affect the synthesis accuracy of the virtual viewpoint, and it is difficult to extract the depth information from the original view. Information is often unreliable.

In the process of acquiring depth information using geometric methods, one step is to calculate the parallax field. Therefore, it is convenient to directly use the parallax to draw the virtual viewpoint. Among such methods, the accuracy of disparity estimation is the key to obtaining a good intermediate view, so current algorithms mainly focus on how to obtain accurate disparity fields. However, accurate parallax fields require a large amount of calculation, which is not practical in practical applications. Classical disparity estimation algorithms are always unable to strike a balance between accuracy and rapidity. These algorithms often use complex constraints and search strategies in order to achieve higher accuracy when searching for disparity, but the improvement of accuracy The effect is not obvious, but it makes the running time of the algorithm longer; the efficiency of disparity estimation can be improved by searching for the best matching point of pixels in the local area, but the cost is that the accuracy of disparity estimation is lower than that of global search.

In the process of disparity estimation, Lu Chaohui et al. proposed a method based on adaptive weights in disparity estimation, and introduced disparity smoothness constraints into the matching cost function. The disparity field obtained by this method is more accurate. , but due to complex constraint conditions, the amount of calculation is greatly increased (see Lu Zhaohui, Yuan Dun. Mesopic synthesis based on disparity estimation [J]. Optoelectronics·Laser. 2007, 18(7): 855-858). Luo Yan et al. proposed to add an image segmentation algorithm to the disparity estimation. After obtaining the disparity field, the input image is segmented based on the gray value, and the image is divided into several regions with different gray values. The points inside the region are have a grayscale similarity. After the disparity field is obtained, the disparity field is smoothed according to the assumption that pixels in the same segmented region have the same or similar disparity value (see Luo Yan, Anping, Zhang Zhaoyang. Intermediate view based on disparity field correction and region segmentation Image generation and interpolation method [J]. Journal of Communications. 2004.25(10): 127-133.). Doing so can correct a part of the wrongly matched disparity, and achieve the purpose of smoothing the disparity field to a certain extent, but it will cause another part of the original correct disparity to be given a wrong disparity value. Moreover, the increase of the image segmentation module makes the whole algorithm The amount of calculation increases.

Contents of the invention

The technical problem to be solved by the present invention is: in order to overcome the deficiencies of the prior art, the present invention provides an intermediate view synthesis method based on block matching disparity estimation, which greatly reduces the computational complexity and saves Algorithm runtime.

The technical solution adopted by the present invention to solve the technical problems includes: a method for synthesizing intermediate views based on block matching parallax estimation, characterized in that it includes the following steps:

(1) Input two left and right images taken from the same scene at the same moment, and the camera is at the same horizontal height. It is required that the two images only have differences in shooting angles;

(2) If the two input images are color images, then convert them into grayscale images respectively; if the two input images are grayscale images, then perform step (3);

(3) Determine whether the size of the two input images is the same, if not, prompt an error and jump out; if they are the same, perform step (4);

(4) is a disparity estimation method based on block matching. Specifically, the right view is used as the target image, and the left view is used as the reference image. The target view is divided into blocks of fixed size. For each block in the target image, in the reference image Search for the closest blocks one by one in , and calculate the displacement vector d ^LR between each target image block and the matching block in the reference image, which is the block disparity from the left view to the right view; then take the left view as the target image , the right view is a reference image, repeat step (4) to obtain the block parallax d ^RL from the right view to the left view;

(5) According to the block parallax d ^LR from the left view to the right view obtained in step (4), using the principle of binocular vision parallax, the middle view I based on the block parallax from the left view to the right view after preliminary viewpoint synthesis can be obtained ^M _L ; (wherein the superscript M represents "middle view"; the subscript L represents "left view to right view"); according to the block parallax d ^RL from the right view to the left view obtained in step (4), in the same way Calculate the middle view I ^M _R based on the block disparity from the right view to the left view after preliminary view synthesis (the subscript R stands for "right view to left view") Theoretically, d ^LR and d ^RL , I ^M _L and I ^M _R should be equal, but since the error of disparity block matching cannot be completely eliminated, it is necessary to further process the two intermediate views after the initial viewpoint synthesis;

(6) is a method for finding the best matching point of the view to be synthesized in the left view and the right view respectively, specifically for the intermediate points of the two preliminary synthesis of I ^M _L and I ^M _R obtained in step (5). View, for each pixel, find the best matching points I ^L (x ^L , y) and I ^R (x ^R , y) for each pixel in I ^M _L and I ^M _R in the left view and right view one by one (Wherein, I ^L and I ^R represent the pixel gray value in the left view and the right view; (x, y) represent the coordinates of the pixel; x ^L and x ^R represent the maximum value found in the left view and the right view respectively The abscissa value of the best matching point, since the binocular cameras are at the same horizontal height, the ordinate values of the best matching point found in the left view and the right view are equal, denoted by y), and based on the weighting of the best matching point Hole filling is performed on I ^ML _{and I MR (that is, a certain gray value is used to fill the blank area that is not mapped to I ML and I MR} ⁾ _{, and} ^{finally the} result of intermediate view _synthesis I ^M is obtained.

The parallax estimation method based on block matching in the above-mentioned step (4) adopts the following steps to realize:

(i) Carry out edge expansion processing on the reference image, add k pixel units on the left and right sides of the reference image respectively (k must satisfy the scene content that only exists in one view in all binocular views), and make these The grayscale value of the pixel is 0, and the supplementary edge is set to black;

(ii) the target image is divided into M * N blocks, where M is the width of each block, and N is the length;

(iii) Find the SAD of the block corresponding to the target image and the reference image, as the initial value for comparison during the search, where the size of the block is M×N, and the coordinates of the upper left corner are (m, n) of the block in the target image The absolute value error and the minimum absolute difference SAD between the reference image block whose coordinates are (p, q) in the upper left corner are:

$\mathrm{<span\; class="notranslate">\; SAD</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; q</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; q</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span>$

Among them, (m, n) is the pixel coordinates of the upper left corner of the block in the target image; ( _p , q) is the _pixel coordinates of the upper left corner of the reference image; The gray value of the coordinate point;

(iv) In the reference image, set the search starting point as the coordinates (m, n) of the upper left corner of the target image block, perform matching within the range of (m-60, m+60) in the abscissa interval, and find out each matching The SAD between the position and the target image block to be matched, so that the corresponding position of the reference image with the smallest SAD value is the best matching block, and the minimum SAD value is retained;

(v) record the best matching block position searched, and find the displacement vector d between the target block and the best matching block, where d (i, j)=(m-p, n-q), i.e. parallax;

(vi) If the block in the lower right corner of the target image is accepted for matching, excluding the supplementary edge, that is, the coordinates of the upper left corner are (X+60-M, Y+60-N), then end the matching; otherwise, find the next A target image block to be matched, return to step (iii);

(v) Extend the block-based disparity to pixel-based units, ie d(i×M+m, j×N+n)=d(i, j); where m∈[0,M-1] , n ∈ [0, N-1]. Taking the left view as the reference image and the right view as the target image, the block disparity d ^LR from the left view to the right view can be calculated; then using the right view as the reference image and the left view as the target image, the distance from the right view to the left view can be calculated Block parallax d ^RL . The present invention adopts this method to obtain d ^LR and d ^RL for use in step (5) and step (6). Only by calculating these two disparities can the intermediate view synthesis be continued.

In above-mentioned step (5), utilize binocular visual parallax principle to obtain the method for the intermediate view that preliminary viewpoint synthesis obtains as follows:

Two cameras at the same level shoot the scene at the same time. According to the principle of binocular parallax, there are:

x ^R = x ^L + d ^LR (x, y)

I ^R (x, y) = I ^L (x+d ^LR (x, y), y)

Among them, x _L is the abscissa of a point on the left view in the camera coordinate system, in pixels; X _R is the abscissa of the corresponding point in the right view; d ^LR is the target image of the left view, and the right view is The disparity value obtained from the reference image,

The composition of the virtual perspective is obtained by weighting the brightness as follows:

I ^M (x ^L , y) = (1-α) I ^L (x ^L , y) + αI ^R (x ^R , y)

Let α be the position parameter, let α=0 be the position of the left view, and α=1 be the position of the right view, then α can represent any position on the straight line (baseline) between the original views in the interval (0, 1), and to be The relationship between the synthesized intermediate view and the left and right original views is expressed as:

x ^M =x ^L +αd ^LR (x,y)=x ^R +(1-α)d ^RL (x,y)

Substituting the above relationship into the weighting formula can obtain the I ^M _L and I ^M _R calculated by d ^LR and d ^RL respectively as follows:

I ^M _L (x ^L +αd ^LR ,y)=(1-α)I ^L (x ^L ,y)+αI ^R (x ^L +d ^LR ,y)

I ^M _R (x ^R +(1-α)d ^RL ,y)=(1-α)I ^L (x ^R +d ^RL ,y)+αI ^R (x ^R ,y)

Among them, I ^M _L and I ^M _R are the intermediate views obtained by preliminary viewpoint synthesis; I ^L and I ^R are the left and right original views respectively; x ^L and x ^R can traverse the entire image.

In the above-mentioned step (6), the method of finding the best matching point of the view to be synthesized in the left view and the right view respectively is realized as:

(i) Find the maximum value d ^LR _max in the disparity map d ^LR from the left view to the right view;

(ii) Calculate the minimum value of the function g ^L (x)=|x ^I -(x+αd ^RL (x, y))| in the interval of x∈[x ^I -αd ^RL _max , x ^I +αd ^RL _max ] Value min(g ^L (x))=g ^L (x ^L ), and save the x value that makes g ^L (x) obtain the minimum value, denoted as x ^L , wherein, x ^I is the abscissa of the middle point of the middle view, take The value is x ^I ∈ [0, image width - 1];

(iii) find the maximum value d ^RL _max in the disparity map d ^RL from the right view to the left view;

(iv) Find the minimum value of the judgment function g ^R (x)=|x ^I -(x+αd ^RL (x, y))| in the interval of x∈[x ^I -αd ^RL , x ^I +αd ^RL ] min(g ^R (x)) = g ^R (x ^R ), and save x ^R ;

(v) According to the brightness weighting formula I ^M (x, y) = (1-α) I ^L (x ^L , y) + α I ^R (x ^R , y), the result I ^M of intermediate view synthesis is finally obtained.

Compared with the prior art, the present invention has the following advantages:

(1) The present invention adopts a small-step search in a larger range in the parallax estimation link, and the search accuracy is basically equal to the exhaustive global search, and the search accuracy is higher than the local search scheme of the classic parallax estimation algorithm;

(2) In the parallax estimation, according to the original view for synthesizing intermediate vision, the present invention is based on the principle of binocular parallax to shoot two images from the same horizontal height. In theory, there is no parallax in the vertical direction, so only the horizontal direction is performed. Matching search greatly reduces the amount of computation and the running time of the algorithm;

(3) experiments show that the method adopted in the present invention can synthesize an intermediate view with good visual effect, especially when the foreground is far away from the camera or the grayscale change of the scene is relatively gentle; View composition speed has been significantly improved.

Description of drawings

Fig. 1 is the flow chart of the intermediate view synthesis algorithm based on block matching parallax estimation in the present invention;

Fig. 2 is a flow chart of the parallax estimation part in the present invention;

Fig. 3 is a flow chart of the middle view synthesis part in the present invention;

Figure 4 is the left view in the original view;

Figure 5 is a right view in the original view;

Fig. 6 is the left side view in the preliminary synthesis result among the present invention;

Fig. 7 is the right side view in the preliminary synthesis result among the present invention;

Fig. 8 is the final intermediate view synthesis result in the present invention, and the position parameter is α=0.25;

Fig. 9 is the final intermediate view synthesis result in the present invention, and the position parameter is α=0.5;

Fig. 10 is the final synthesis result of the middle view in the present invention, and the position parameter is α=0.75.

Detailed ways

The intermediate view synthesis method based on block matching disparity estimation of the present invention uses the disparity estimation based on block matching to respectively obtain the disparity fields from the left view to the right view and from the right view to the left view, and perform intermediate view according to these two disparity fields respectively. View synthesis, and finally use the parallax field and the initially synthesized view to calculate the best matching point of each pixel of the view to be synthesized in the left and right views, and weight each point in the view to be synthesized according to the brightness of the best matching point at the corresponding position Assign a brightness value to complete the view synthesis at any position on the line connecting the two views.

All "views" and "images" in this embodiment refer to digital bitmaps, the abscissa is from left to right, and the ordinate is from top to bottom, all counting from 0, and the pixel point representation form is (X, Y) . Fig. 1 shows the flow chart of the intermediate view synthesis algorithm based on block matching parallax estimation in the present invention; Fig. 2 shows the process flow of the disparity estimation part in the present invention; Fig. 3 shows the flow chart of the intermediate view synthesis part, and the specific steps are as follows :

(1) Input two images taken from the same scene at the same moment, and the camera is at the same horizontal height. It is required that the two images only have differences in shooting angles. The above requirements for a set of input images meet the binocular principle, which is equivalent to observing and recording the scene with "left and right eyes". And the work to be completed by the present invention is to synthesize and output the result obtained by observing the scene at any point between the "left and right eyes";

(2) If the two input images are color images, convert them into grayscale images respectively; if the two input images are grayscale images, perform step (3). When the color image is converted into a grayscale image, weighting is performed according to the proportions of the three color components RGB being 0.11, 0.59, and 0.30 respectively, and the obtained result is the grayscale value of the corresponding pixel;

(3) Determine whether the sizes of the two input images are the same, if not, prompt an error and jump out of the program; if they are the same, execute step (4). If the left and right images used to synthesize the middle view have different sizes, the corresponding pixels of the two images cannot be interpolated to obtain the final result;

(4) Take the right view as the target image and the left view as the reference image, divide the target view into blocks of fixed size, search for the closest block in the reference image, and calculate the matching between each target image block and the reference image The displacement vector between the blocks, that is, the block parallax d ^LR from the left view to the right view; then take the left view as the target image, and the right view as the reference image, and repeat step (4) to find the block parallax from the right view to the left view d ^RL , the specific process is shown in Figure 2;

(5) According to the d ^LR obtained in step (4), obtain the intermediate view I ^M _L based on the block parallax from the left view to the right view after preliminary viewpoint synthesis according to the principle of binocular visual parallax; The obtained d ^RL is used to obtain the intermediate view I ^MR based on the block disparity _between the right view and the left view after preliminary view synthesis. Because the disparity information used in the process ^of solving I ^ML and I _MR is incomplete, only d _LR is used when calculating I ^ML , and only _{d RL is used when calculating I MR} ^{, and} the weighting formula adopts forward mapping ^, so Voids are generated in ^IML and _IMR , that is, areas without filling _, ^which require further processing;

(6) For the two initially synthesized intermediate views of I ^M _L and I ^M _R obtained, for each pixel, according to the process shown in Figure 3, search for I M L and I ^M _R in the left view and right view one by one. The best matching points I ^L (x ^L , y) and I ^R (x ^R , y) of each pixel in I ^M _R are weighted according to the gray value of the best matching point to fill holes in the initially synthesized view, Finally, the result I ^M of intermediate view synthesis is obtained.

Wherein, the parallax estimation method based on block matching in the above-mentioned step (4) is realized by the following steps:

(i) Perform edge expansion processing on the reference image, and add k pixel units on the left and right sides of the reference image respectively (k must satisfy the scene content that only exists in one view in all binocular views. After experiments, this The invention takes k=60), and let it be 0, and set the added part to black. These supplementary black areas will be used to search for blocks close to the left and right edges of the target image, so that the edges of the reference image do not exceed the search range;

(ii) Divide the target image into M×N blocks. In general, for the sake of simplicity of expression when processing images, M=N is usually used. In the example in the accompanying drawings of the specification of the present invention, a block with a size of 16×16 is used for matching;

(iii) Obtain the SAD of the block corresponding to the target image and the reference image, and use it as an initial value for comparison during searching.

SAD is defined as the sum of the absolute values of the subtraction of the corresponding elements of two arrays with the same number of elements. In the present invention, the size of the block is M×N, and the absolute value error and SAD for:

$\mathrm{<span\; class="notranslate">\; SAD</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; q</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; q</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span>$

Among them, (m, n) is the pixel coordinates of the upper left corner of the block in the target image; ( _p , q) is the _pixel coordinates of the upper left corner of the reference image; The gray value of the coordinate point;

This definition applies to all SADs hereinafter;

(iv) In the reference image, set the search starting point as the coordinates (m, n) of the upper left corner of the target image block, perform matching within the range of (m-60, m+60) in the abscissa interval, and find out each matching The SAD between the position and the target image block to be matched, so that the corresponding position of the reference image with the smallest SAD value is the best matching block, the coordinates of this best matching block are (p, q), and the minimum SAD value is reserved;

(v) Record the searched best matching block position (p, q), and find the displacement vector d between the target block and the best matching block, in pixels, where d(i, j) = (m-p, n-q), which is the disparity vector. In the present invention, since the input images taken are taken at the same horizontal height, the vertical component of the parallax vector is zero, and only the horizontal component exists;

(vi) If the accepted match is the last meaningful block in the last line of the target image, that is, the block in the lower right corner of the target image (excluding the supplementary edge), the coordinates of the upper left corner are (X+60-M, Y+ 60-N), then end the matching; otherwise, find the next target image block to be matched, and return to step (iii). Using this method to traverse the target image divided into several blocks, the disparity values of all blocks in the target image will be obtained finally;

(v) Extend the block-based disparity to pixel-based units, ie d(i×M+m, j×N+n)=d(i, j); where m∈[0,M-1] , n ∈ [0, N-1]. Taking the left view as the reference image and the right view as the target image, the block disparity d ^LR from the left view to the right view can be calculated; then using the right view as the reference image and the left view as the target image, the distance from the right view to the left view can be calculated Block parallax d ^RL .

In the above-mentioned step (5), the method for obtaining the preliminary intermediate view according to the principle of binocular visual parallax is specifically as follows:

Use two cameras at the same level to shoot the scene at the same time. According to the principle of binocular parallax, the two views obtained by shooting have the following geometric relationship:

x ^R = x ^L + d ^LR (x, y)

I ^R (x, y) = I ^L (x+d ^LR (x, y), y)

Among them, x _L is the abscissa of a point on the left view in the camera coordinate system, in pixels; x _R is the abscissa of the corresponding point in the right view; d ^LR is the target image of the left view, and the right view is The disparity value obtained from the reference image.

The composition of the virtual viewing angle at any position on the connection between the two cameras can be obtained using the following brightness weighting formula:

I ^M (x ^L , y) = (1-α) I ^L (x ^L , y) + αI ^R (x ^R , y)

Among them, I ^M is the middle view to be synthesized, α is the position parameter, let α=0 be the position of the left view, α=1 is the position of the right view, then α can represent the original view in the (0,1) interval Any position on the straight line (baseline). The relationship between the intermediate view to be synthesized and the left and right original views can be expressed as follows:

x ^M =x ^L +αd ^LR (x,y)=x ^R +(1-α)d ^RL (x,y)

Substituting the above relationship into the weighting formula can obtain the I ^M _L and I ^M _R calculated by d ^LR and d ^RL respectively as follows:

I ^M _L (x ^L +αd ^LR ,y)=(1-α)I ^L (x ^L ,y)+αI ^R (x ^L +d ^LR ,y)

I ^M _R (x ^R +(1-α)d ^RL ,y)=(1-α)I ^L (x ^R +d ^RL ,y)+αI ^R (x ^R ,y)

Among them, I ^M _L and I ^M _R are the intermediate views obtained by preliminary viewpoint synthesis; I ^L and I ^R are the left and right original views respectively; x ^L and x ^R can traverse the entire image.

Use these two formulas to synthesize the intermediate views of Figures 4 and 5, and the synthesized views are shown in Figures 6 and 7. Since each synthesized view uses only one disparity field, that is, the view synthesized with the left view as the target image only uses d ^RL , while the view synthesized with the right view as the target image only uses d ^LR , the lack of disparity information makes The result of the synthesis is not accurate enough, and there are many errors; more obviously, the synthesis formula used in this step uses the forward mapping method, that is, the coordinates of the I ^M to be assigned on the left side of the equation cannot traverse the entire image , so there are many blank unmapped areas in the synthesized view, which greatly affects viewing and cannot be regarded as the final synthesized result.

In the above-mentioned step (6), the best matching points of the views to be synthesized are respectively searched for in the left view and the right view, because the middle view obtained in the above-mentioned step (5) lacks disparity information, and the synthesis formula adopts The forward mapping method leads to errors and many blank areas in the synthesized view, which greatly affects viewing and cannot be regarded as the final synthesis result. Therefore, further processing is required to fill the unmapped empty areas. The method is shown in Figure 3. The flow chart is implemented in the following steps:

(i) traverse the disparity map d ^LR from the left view to the right view, and find the maximum value d ^LR _max in d ^LR ;

(ii) In the right image, obtain the decision function g ^L (x)=|x ^I -(x+αd ^RL (x,y))|at x∈[x ^I -αd ^RL _max , x ^I +αd ^RL _max ] the minimum value min(g ^L (x))=g ^L (x ^L ) in the interval, and save the x value that makes g ^L (x) get the minimum value, denoted as x ^L , x ^L is the minimum value of the point best match point. Among them, x ^I is the abscissa of the middle point in the middle view, and the value is x ^I ∈ [0, image width - 1]; traversing the entire reference image, you can get all the best matching points of the left view in the right view;

(iii) find the maximum value d ^RL _max in the disparity map d ^RL from the right view to the left view;

(iv) In the left image, determine the decision function g ^R (x)=|x ^I -(x+αd ^RL (x,y))| at x∈[x ^I -αd ^RL , x ^I +αd ^RL ] The minimum value min(g ^R (x)) in the interval = g ^R (x ^R ), and save the value of x that makes g ^R (x) the smallest, denoted as x ^R , and x ^R is the best match at this point point;

(v) According to the brightness weighting formula I ^M (x, y) = (1-α) I ^L (x ^L , y) + α I ^R (x ^R , y), finally get the result I ^M of the intermediate view synthesis, as shown in the figure 8. As shown in Fig. 9 and Fig. 10, the position parameters are respectively α=0.25, α=0.5, and α=0.75. This formula makes full use of the geometric information of the left view and the right view, and the two disparity fields from the left view to the right view, and from the right view to the left view, so the obtained result is relatively accurate. Prove from accompanying drawing example, the present invention has really obtained good visual effect.

The contents not described in detail in the description of the present invention belong to the prior art known to those skilled in the art.

Claims (4)

1. medial view synthetic method based on piece coupling disparity estimation is characterized in that performing step is as follows:

(1) input is taken from Same Scene, synchronization, and the position for video camera is two width of cloth images in about same level height, require this two width of cloth image only to there are differences on the shooting visual angle;

(2) if two width of cloth input pictures are coloured image, then be translated into gray level image respectively; As if two width of cloth input pictures is gray level image, then execution in step (3);

(3) judge whether two width of cloth input picture sizes are identical, if different, the prompting mistake is also jumped out; If identical, execution in step (4);

(4) with the right view be target image, left view is a reference picture, target view is divided into the piece of fixed size, and the most close piece of each piece in search and the target image respectively one by one in reference picture calculates the displacement vector d between the match block in each target image piece and the reference picture ^LR, be the piece parallax of left view in the right view; Be target image again with the left view, right view is a reference picture, and repeating step (4) is obtained the piece parallax d of right view in the left view ^RL

(5) according to the piece parallax d of the left view of obtaining in the step (4) to right view ^LR, utilize the binocular vision principle of parallax, obtain through the synthetic back of preliminary viewpoint based on the medial view I of left view to right view piece parallax ^M _L, wherein subscript M represents medial view, and subscript L represents left view to right view, according to the piece parallax d of the right view of obtaining in the step (4) to left view ^RL, in like manner obtain through the synthetic back of preliminary viewpoint based on the medial view I of right view to left view piece parallax ^M _R, wherein subscript R represents right view to left view;

(6) for the I that obtains in the step (5) ^M _LWith I ^M _RThese two preliminary synthetic medial view at each pixel wherein, are sought I one by one in left view and right view ^M _LWith I ^M _RIn the optimal match point I of each pixel ^L(x ^L, y) and I ^R(x ^R, y), I wherein ^LAnd I ^RRepresent the pixel gray value in left view and the right view, (x, the y) coordinate of represent pixel point, x ^LAnd x ^RRepresent the abscissa value of the optimal match point that in left view and right view, finds respectively, because binocular camera is in same level height, so the ordinate value of the optimal match point that finds in left view and right view equates, represent with y, and according to the weighting of optimal match point to I ^M _LWith I ^M _RCarry out the cavity and fill, promptly use gray value to fill up I ^M _LWith I ^M _RIn be not mapped to white space, finally obtain the synthetic I as a result of medial view ^M

2. a kind of medial view synthetic method based on piece coupling disparity estimation according to claim 1 is characterized in that: being implemented as follows in the described step (4):

(i) reference picture being expanded the limit handles, left side and right side at reference picture increase k pixel unit respectively, k must satisfy to comprise in all binocular views and only is present in the scene content of a width of cloth view, and to make the gray value of these pixels be 0, and the edge of augmenting is changed to black;

The piece that (ii) target image is divided into M * N, wherein M is an every width, N is a length;

(iii) obtain the SAD of target image and reference picture correspondence position piece, the initial value that is used for comparison during as search, wherein the size of piece is M * N, upper left corner coordinate is (m, n) piece in the target image and upper left corner coordinate be (p, absolute value error between reference image block q) and minimum absolute difference SAD are:

$\mathrm{SAD}(m,n,p,q)=\underset{i=1}{\overset{M}{\mathrm{\&Sigma;}}}\underset{j=1}{\overset{N}{\mathrm{\&Sigma;}}}|I_1(m+i,n+j)-I_2(p+i,q+j)|$

Wherein, (m n) is the pixel coordinate in the piece upper left corner in the target image, and (p q) is the pixel coordinate in the upper left corner in the reference picture, I ₁, I ₂Be respectively target image and reference picture gray value in a certain coordinate points; I gets the integer from 0 to M-1 respectively; J gets the integer from 0 to N-1 respectively;

(iv) in reference picture, the search starting point is set to the upper left corner coordinate (m of target image piece, n), in the abscissa interval is (m-60, m+60) mate in the scope, obtain the SAD of each matched position and target image interblock to be matched, make sad value obtain minimum reference picture correspondence position and be best matching blocks, and keep this SAD minimum value;

(v) the best matching blocks position that searches is noted, and is obtained displacement vector d between object block and the best matching blocks, wherein d (i, j)=(m-p, n-q), i.e. parallax;

If (what vi) accept coupling is the piece that is positioned at the lower right corner in the target image, does not comprise the edge of augmenting, and promptly upper left corner coordinate is that (X+60-M Y+60-N), then finishes coupling; Otherwise, find target image piece next to be matched, return step (iii);

(v) will be that the parallax of unit extends to the pixel with the piece be unit, promptly d (i * M+m, j * N+n)=d (i, j), m ∈ [0, M-1] wherein, n ∈ [0, N-1], with the left view is reference picture, and right view is a target image, calculates the piece parallax d of left view to right view ^LRBe reference picture with the right view again, left view is a target image, calculates the piece parallax d of right view to left view ^RL

3. a kind of medial view synthetic method based on piece coupling disparity estimation according to claim 1 is characterized in that: it is as follows to utilize the binocular vision principle of parallax to obtain the method for the synthetic medial view that obtains of preliminary viewpoint in the described step (5):

Be positioned at two video cameras of same level height, simultaneously scene taken,, have according to the binocular parallax principle:

x ^R＝x ^L+d ^LR(x，y)

I ^R(x，y)＝I ^L(x+d ^LR(x，y)，y)

Wherein, x _LFor some abscissa in camera coordinate system of left view, be unit with the pixel; x _RAbscissa for these corresponding points in right view; d ^LRFor being target image with the left view, right view is the parallax value that reference picture is obtained,

The following luminance weighted acquisition of the synthetic usefulness at virtual visual angle:

I ^M(x ^L，y)＝(1-α)I ^L(x ^L，y)+αI ^R(x ^R，y)

If α is a location parameter, make that α=0 is the left view position, α=1 is the right view position, then α is (0,1) can represent straight line between original view in interval, i.e. optional position on the baseline, and centre to be synthesized is looked and left and right sides original view between relation table be shown:

x ^M＝x ^L+αd ^LR(x，y)＝x ^R+(1-α)d ^RL(x，y)

To can obtain using d respectively with co-relation substitution weighting formula ^LRAnd d ^RLThe I that obtains ^M _LWith I ^M _RAs follows:

I ^M _L(x ^L+αd ^LR，y)＝(1-α)I ^L(x ^L，y)+αI ^R(x ^L+d ^LR，y)

I ^M _R(x ^R+(1-α)d ^RL，y)＝(1-α)I ^L(x ^R+d ^RL，y)+αI ^R(x ^R，y)

Wherein, I ^M _LAnd I ^M _RBe the synthetic medial view that obtains of preliminary viewpoint; I ^LAnd I ^RBe respectively left and right sides original view; x ^LAnd x ^RAll can travel through entire image.

4. a kind of medial view synthetic method according to claim 1 based on piece coupling disparity estimation, it is characterized in that: described step (6) is implemented as follows:

(i) obtain the disparity map d of left view to right view ^LRIn maximum d ^LR _Max

(ii) obtain function g ^L(x)=| x ^I-(x+ α d ^RL(x, y)) | at x ∈ [x ^I-α d ^RL _Max, x ^I+ α d ^RL _Max] interval interior minimum value min (g ^L(x))=g ^L(x ^L), and preservation makes g ^L(x) obtain minimum x value, be designated as x ^L, wherein, x ^IBe the abscissa of medial view mid point, value is x ^I∈ [0, picture traverse-1];

(iii) obtain the disparity map d of right view to left view ^RLIn maximum d ^RL _Max

(iv) obtain decision function g ^R(x)=| x ^I-(x+ α d ^RL(x, y)) | at x ∈ [x ^I-α d ^RL, x ^I+ α d ^RL] interval interior minimum value min (g ^R(x))=g ^R(x ^R), and preserve x ^R

(v) according to luminance weighted formula I ^M(x, y)=(1-α) I ^L(x ^L, y)+α I ^R(x ^R, y), finally obtain the synthetic I as a result of medial view ^M

Images