JP2013126114A - Stereo image processing method and stereo image processing apparatus - Google Patents

Abstract

The present invention provides a stereo image processing method and a stereo image processing apparatus that reduce fluctuations in disparity between successive frames while suppressing the amount of computation. A stereo image of a predetermined frame is input, and the stereo image is input. A dissimilarity calculating unit that calculates the dissimilarity of the stereo image, a weight value calculating unit that calculates a weighted value of the predetermined frame based on the disparity information of the frame immediately before the predetermined frame, and a dissimilarity between the stereo images supplied from the dissimilarity calculating unit A first filter that weights the weight value supplied from the weight value calculation unit, and an optimization unit that calculates disparity information of a predetermined frame based on the dissimilarity weighted by the weight value supplied from the first filter And a first method for storing the disparity information of the predetermined frame supplied from the optimization unit and outputting the stored disparity information to the weight value calculation unit. Stereo image processing apparatus including a re, a.
[Selection] Figure 3

Description

  The present invention relates to a stereo image processing method and a stereo image processing apparatus.

  In recent years, devices using stereo images such as 3D televisions have been actively developed. In order to display a stereo image, stereo matching is performed to extract depth (depth information) from the stereo image, and a parallax image to be displayed on the display device is synthesized using the extracted depth image and the reference image.

  In stereo matching, disparity of corresponding pixels is calculated between the left image and the right image of the stereo image. At this time, it is necessary that the corresponding points of the left image and the right image of the stereo image are parallelized so that they exist on the same scan line.

  General stereo matching consists of matching cost calculation and optimization (optimization). Matching cost calculation is to calculate a matching cost (difference) of a reference image for each pixel of a left image, for example, a reference image for each pixel of a left image, for example, a right image. is there. The optimization is to associate each pixel in the standard image and the reference image so that the 3D image becomes smooth while reducing the matching cost of the reference image with respect to all the pixels of the standard image. By performing this optimization, noise due to mismatch between the standard image and the reference image is removed, and a good 3D image can be visually recognized.

  As a method for calculating the matching cost, generally, block matching is performed, and a method using a sum of absolute differences (SAD) of luminance differences in a predetermined region corresponding to the base image and the reference image is known. ing. As an optimization (optimization) method, operations such as a dynamic programming (DP) method, a graph cut (GC) method, and a probability propagation (BP) method are generally used.

  Among general optimization methods, the dynamic programming method performs optimization independently for each scan line, so horizontal line noise may occur in the 3D image that is visible after optimization. There is a problem that there is. Further, in the dynamic programming method, since optimization is performed for each frame, there is a problem that disparity fluctuation may occur between consecutive frames. Since the graph cut method and the established propagation method are optimized in the two-dimensional direction of the vertical direction and the horizontal direction, line noise does not occur, but there is a problem that the calculation amount becomes enormous as compared with the dynamic programming method.

  As a method of improving the line artifact by suppressing the generation of line noise, there is a method of combining the dynamic programming method and the probability propagation method (Patent Document 1). In this method, first, the disparity between the standard image and the reference image is calculated by the dynamic programming method, and the result is used as the initial value of the calculation by the probability propagation method, thereby reducing the calculation amount and improving the line artifact. It is. In this method, the generation of line noise is suppressed, and the calculation amount is reduced as compared with the normal probability propagation method, so that the calculation time is shortened. However, this method requires more computation time than the dynamic programming method.

  Furthermore, as a method for reducing the amount of computation of the dynamic programming method, for regions where the difference in luminance value of corresponding pixels between consecutive frames is less than or equal to a predetermined threshold, the disparity of the previous frame is inherited as it is, and the The calculation in the initialization is omitted, and in the predetermined frame, the disparity of the immediately above scan line is inherited for the area where the difference in luminance value between the adjacent scan lines in the predetermined frame is equal to or less than the predetermined threshold. There is a method of omitting computation in optimization (Patent Document 2). In this method, the amount of computation in optimization is reduced and the computation time is shortened. However, since the change between successive frames is determined by the threshold value for the difference in luminance value, an image with no change between frames can be obtained. Even if there is a difference in luminance value of corresponding pixels between frames due to sampling error, disparity is not inherited and disparity fluctuation between frames occurs. is there. In addition, when an incorrect disparity is calculated due to a mismatch in the first frame, this incorrect disparity may be inherited in subsequent frames. Furthermore, with regard to inheritance of disparity between adjacent scan lines, for example, if an incorrect disparity is calculated due to a mismatch in the scan line located at the top of the screen, this incorrect disparity is also detected in the subsequent scan line. May be inherited.

Special table 2010-531490 gazette JP 2006-331108 A

  In addition to the optimization method described above, there is a method of D. Min et al. This method improves the disparity fluctuation between frames by calculating the disparity of the current frame using the motion vector from the reference image of the current frame to the reference image of the previous frame and the disparity of the previous frame. There is an effect. However, the method of D. Min et al. Has the problem that line noise may occur because optimization is performed independently for each scan line as in the dynamic programming method. There is a problem that the amount of calculation increases because the motion probability of the pixel with the previous frame is obtained. See D. Min, S. Yeo, A Verto, “Temporally Consistent Stereo Matching Using Coherence Function”, Proc. IEEE 3DTV Coherence, Tampere, Finland, June 2010.

  An object of the present invention is to provide a stereo image processing method and a stereo image processing apparatus that reduce fluctuations in disparity between consecutive frames while suppressing the amount of calculation.

  A stereo image processing apparatus according to an embodiment of the present invention receives a stereo image of a predetermined frame, calculates a difference between the stereo images, and performs a predetermined operation based on disparity information of a frame immediately before the predetermined frame. A weight calculation unit for calculating a weight value of the frame, a first filter for weighting the weight value supplied from the weight value calculation unit to the dissimilarity of the stereo image supplied from the difference degree calculation unit, and a first filter An optimization unit that calculates disparity information of a predetermined frame based on the dissimilarity weighted by the weight value supplied from, and stores the disparity information of the predetermined frame supplied from the optimization unit, and stores the stored disparity information And a first memory that outputs to a weight value calculation unit.

In the stereo image processing method according to an embodiment of the present invention, a stereo image of a predetermined frame is input, a difference between the stereo images is calculated, and a weight value of the predetermined frame is calculated based on the parallax information calculated immediately before. The weighting value is weighted to the dissimilarity of the stereo image, the disparity information of the predetermined frame is calculated based on the dissimilarity weighted by the weighting value, the disparity information of the predetermined frame is stored, and the stored disparity information is Used to calculate weights,
It is characterized by including.

  A stereo image display device according to an embodiment of the present invention includes a display panel, a stereo image input unit to which a video signal including a stereo image is input, and a stereo that performs height correction of the stereo image and generates a corrected stereo image. An image parallelizing unit, a stereo image processing device that performs stereo matching of the corrected stereo image and generates parallax information of the one image, and image data that creates parallax video using the one image and the parallax information A stereo image processing unit, and a timing control unit that outputs image data. The stereo image processing apparatus receives a stereo image of a predetermined frame and calculates a dissimilarity calculation unit. Supplied from a weight value calculation unit that calculates a weight value of a predetermined frame based on disparity information of a frame immediately before the predetermined frame, and a difference degree calculation unit A first filter that weights the weighting value supplied from the weight value calculation unit to the degree of difference between the stereo images, and a disparity of a predetermined frame based on the degree of difference weighted by the weight value supplied from the first filter An optimization unit that calculates information; and a first memory that stores disparity information of a predetermined frame supplied from the optimization unit and outputs the stored disparity information to a weight value calculation unit. And

  It is possible to provide a stereo image processing method and a stereo image processing apparatus that reduce fluctuations in disparity between consecutive frames while suppressing the amount of calculation.

It is a block diagram which shows the structure of the stereo image display apparatus 100 which concerns on one Embodiment of this invention. It is a figure explaining the height correction of the stereo image in a stereo image parallelization part. It is a block diagram which shows the structure of the stereo matching part which concerns on one Embodiment of this invention. It is a flowchart which shows an example of the flow of a process in the stereo matching part shown in FIG. It is a figure which shows an example of the brightness | luminance of each pixel taken out from the same scan line of a base image and a reference image when the block size of SAD is set to 1x1. (A) It is a figure explaining the method of calculating | requiring the difference degree of a reference | standard image and a reference image. (B) It is a figure which shows an example of DSI. It is a figure for demonstrating the block matching method. (A) The figure which shows an example of the matching cost DSI corresponding to the predetermined y coordinate of a reference | standard image. (B) It is a figure which shows an example of the disparity in the predetermined y coordinate of an immediately preceding frame. (C) It is a figure which shows an example of the weight value which concerns on the element of the matching cost DSI corresponding to a predetermined y coordinate. (D) It is a figure which shows the matching cost DSIF which weighted the weighting value shown to (c) to the element of the matching cost DSI shown to (a). It is the figure which represented the example of matching cost DSI corresponding to the predetermined y coordinate of a continuous t-1 frame, t frame, and t + 1 frame, a weight value, and weighted matching cost DSIF. It is a figure for demonstrating the dynamic programming method. It is a figure for demonstrating the dynamic programming method. It is a block diagram which shows the structure of the stereo matching part which concerns on another embodiment of this invention. It is a flowchart which shows an example of the flow of a process in the stereo matching part which concerns on another embodiment of this invention. It is the block diagram which showed the structure of the image data comparison part. It is a figure which shows the relationship of constant (sigma) ₀ , (sigma) ₁ . It is a graph which shows an example of a weight value.

  Hereinafter, a stereo image processing apparatus according to an embodiment of the present invention will be described with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram showing a configuration of a stereo image display device 100 according to an embodiment of the present invention. The stereo image display apparatus 100 includes a stereo image input unit 102, a stereo image parallelization unit 104, a stereo matching unit 106, a parallax image synthesis unit 108, and a timing control unit 110. Each component is implemented by a logic circuit such as an FPGA or an ASIC. Alternatively, it is mounted by a processor such as a CPU or GPU, a ROM, a RAM, and a system LSI in which these are incorporated.

  A stereo image is input to the stereo image input unit 102. The stereo image is composed of two images, and is composed of one image captured by two imaging devices, for example, a left image IL, and the other image, for example, a right image IR. One of the left image IL and the right image IR is used as a two-dimensional image, and the other image is used for extracting disparity (disparity information) by stereo matching with the one image.

  The stereo image collimating unit 104 aligns the heights of the two images supplied from the stereo image input unit 102 in the line direction. As shown in FIG. 2, the stereo image collimating unit 104 sets a representative point in the base image, for example, the left image IL, and sets a corresponding point corresponding to the representative point in the reference image, for example, the right image IR, as a block matching method or the like And based on the representative point of the standard image IL and the corresponding point of the reference image IR, the entire image of either the standard image IL or the reference image IR so that the heights of the standard image IL and the reference image IR match. Is subjected to height correction, that is, parallelization. Thereby, the height deviation between the standard image IL and the reference image IR is eliminated. The stereo image collimating unit 104 outputs the standard image IL and the reference image IR whose height has been corrected to the stereo matching unit 106.

  The stereo matching unit 106 performs stereo matching using the stereo image whose height has been corrected by the stereo image collimating unit 104 to obtain the disparity (parallax information) of the reference image IL or the reference image IR. The disparity is output to the parallax image synthesis unit 108. Each configuration of the stereo matching unit 106 will be described later.

  The parallax image combining unit 108 displays image data for displaying a multi-view parallax image based on the disparity (disparity information) supplied from the stereo matching unit 106 and the standard image IL or the reference image IR corresponding to the disparity. Is output to the timing control unit 110.

  The timing control unit 110 outputs the image data supplied from the parallax image synthesis unit 108 to the data driver 112. The display panel 114 displays an image based on the image data supplied from the data driver 112.

  FIG. 3 is a block diagram illustrating a configuration of the stereo matching unit 106. The stereo matching unit 106 includes a dissimilarity calculation unit 301, a weight value calculation unit 303, a first filter 305, an optimization unit 307, a second filter 309, and a first memory 311. Hereinafter, the dissimilarity calculation unit 301 is the matching cost calculation unit 301, the first filter 305 is the DSI filter 305, the optimization unit 307 is the optimization unit 307, the second filter 309 is the disparity filter 309, and the first memory 311. Is called a disparity memory 311.

  FIG. 4 is a flowchart showing an example of the flow of processing in the stereo matching unit 106 shown in FIG. With reference to FIGS. 3 and 4, the flow of processing in the stereo matching unit 106 will be briefly described.

  The matching cost calculation unit 301 calculates a matching cost (difference) MCt between the reference image IR and the reference image IR of the t-th frame subjected to height correction supplied from the stereo image parallelization unit 104 (S401). ) Find DSIT. A method for calculating the matching cost MCt and DSI will be described later. The matching cost calculation unit 301 outputs the obtained DSIt to the DSI filter 305. Hereinafter, DSIt is also referred to as a matching cost DSIt.

  The weight calculation unit 303 calculates the weight value Wt of the t-th frame that is the current frame using the disparity DFt-1 of the t-1 frame that is the previous frame supplied from the disparity memory (S402). Output to the DSI filter 305. A method for calculating the weight value Wt will be described later. Here, the order of calculation of the weight value in the weight value calculation unit 303 (S402) and calculation of the matching cost MCt in the matching cost calculation unit 301 (S401) may be reversed or simultaneous.

  The DSI filter 305 multiplies the matching cost DSIT supplied from the matching cost calculator 301 by the weight Wt supplied from the weight calculator 303 to calculate a weighted matching cost DSIFt (S403). The DSI filter 305 outputs the weighted matching cost DSIFt to the optimization unit 307.

  The optimization unit 307 optimizes the weighted matching cost DSIFt supplied from the DSI filter 305 using a general optimization method, and calculates the optimal disparity Dt (disparity information) of the t-th frame (S404). ). As the optimization method, a known dynamic programming method, scan line optimization method, or the like may be used. For scanline optimization methods, see D. Scharstein and R. Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. IJCV, 47 (1-3): 7-42, 2002. The optimization unit 307 outputs the calculated disparity Dt of the t-th frame to the disparity filter 309.

  The disparity filter 309 is a filter that removes noise in the horizontal direction. For example, an average value filter, an intermediate value filter, or the like may be used. The disparity filter 309 removes horizontal noise from the disparity Dt supplied from the optimization unit 307 (S405). The disparity filter 309 outputs the disparity DFt from which the horizontal noise has been removed to the parallax image synthesis unit 108 and the disparity memory 311. The disparity filter 309 may be omitted. Further, the disparity filter 309 may be arranged at the subsequent stage of the disparity memory 311.

  The disparity memory 311 temporarily stores the disparity DFt supplied from the disparity filter 309 (S406), and outputs the stored disparity DFt to the weight value calculation unit 303 (S407). The disparity DFt output from the disparity memory 311 is used by the weight value calculation unit 303 to calculate the weight value Wt + 1 related to the matching cost DSIt + 1 of the t + 1-th frame. When the disparity filter 309 is omitted, the disparity memory 311 stores the disparity Dt supplied from the optimization unit 307 and outputs the disparity Dt to the weight value calculation unit 303. When the disparity filter 309 is arranged at the subsequent stage of the disparity memory 311, the disparity Dt output from the disparity memory 311 is subjected to removal of horizontal noise by the disparity filter 309, and then a weight value calculation unit It is output to 303.

  Based on the disparity DFt supplied from the disparity filter 309, the parallax image synthesis unit 108 generates image data for displaying a multi-view parallax image of the t-th frame and outputs the image data to the timing control unit 110. When the disparity filter 309 is omitted, the parallax image synthesis unit 108 generates image data for displaying the multi-view parallax image of the t-th frame based on the disparity Dt supplied from the optimization unit 307. And output to the timing control unit 110.

  Hereinafter, processing in each configuration of the stereo matching unit 106 will be described in detail with reference to FIGS.

Matching Cost Calculation Unit The matching cost calculation unit 301 calculates a matching cost (difference) MC between the standard image IL and the reference image IR that have been subjected to height correction in the stereo image parallelization unit 104. The matching cost MC between the reference image IL and the reference image IR can be obtained using block matching using SAD, SSD, NCC, NSSD, or the like. Here, block matching by SAD is performed, and a matching cost MCt between the standard image (here, the left image) ILt and the reference image (here, the right image) IRt of the t-th frame which is the current frame is calculated. Hereinafter, a method of calculating the matching cost MCt _{SAD1 × 1} between the base image IL and the reference image IR when the SAD block size is set to 1 × 1 will be described as an example.

  The matching cost calculation unit 301 extracts the luminance of each pixel on the same scan line corresponding to each other from the standard image ILt and the reference image IRt. FIG. 5 shows an example of the luminance of each pixel extracted from the same scan line of the base image ILt and the reference image IRt when the SAD block size is set to 1 × 1.

  Next, as shown in FIG. 6A, the matching cost calculation unit 301 shifts each pixel of the reference image IRt by one pixel in the horizontal direction with respect to each pixel of the base image ILt, that is, disparity is changed. The absolute value ABS of the luminance difference between the pixels of the base image ILt and the reference image IRt is obtained while shifting one by one. In FIG. 6A, as an example, the calculation of the absolute value ABS of the luminance difference in the disparity from 0 to (+) 4 is shown. For example, the disparity is (−) 100 to (+) 100. The absolute value ABS of the luminance difference between the pixels of the base image ILt and the reference image IRt up to may be calculated. The numerical range of disparity can be changed as appropriate according to the resolution of the display panel and the input video. When the disparity is shifted one by one and the luminance difference of each pixel of the standard image IL and the reference image IR is obtained, if there is no corresponding pixel in the standard image IL and the reference image IR, the absolute difference of the luminance is calculated. Let the value ABS be ∞.

When the block size of the SAD is 1 × 1, the matching cost MCt _{SAD1 × 1} between the standard image ILt and the reference image IRt of the t-th frame is _one pixel in the horizontal direction for each pixel of the standard image IL and the reference image IR. The absolute value of the luminance difference between corresponding pixels of the base image IL and the reference image IR calculated while shifting is obtained. The matching cost MC (x, y, d) between one pixel can be obtained using the following equation (1).

Here, IL (x, y) indicates the luminance in the (x, y coordinate) of the left image that is the base image IL, and IR (x−d, y) indicates (x of the right image that is the reference image IR. -D, y coordinates). d represents a numerical value of disparity.

Here, the SAD block size is 1 × 1, but the SAD block size is not limited to 1 × 1, and for example, 3 × 3 blocks, 5 × 5 blocks, and 15 × 15 blocks can be set as appropriate. is there. The matching cost MC _{SADI × J} (x, y, d) of the corresponding pixels of the standard image IL and the reference image IR in the predetermined SAD block is determined between the corresponding pixels in the predetermined SAD block of the standard image IL and the reference image IR. The sum of absolute values of luminance differences is obtained using the following equations (2) and (3).

Here, MCSADI × J is a sum of absolute values of luminance differences between corresponding pixels of the base image IL and the reference image IR in a predetermined SAD block I × J, and Wb is a block matching window ( Block Matching Window). For example, as shown in FIG. 7, when the SAD block size is 5 × 5, (I, J) = (5, 5). When the SAD block size is 1 × 1, the matching cost MCt _{SAD1 × 1} between corresponding pixels of the standard image ILt and the reference image IRt of the t-th frame is

It is.

In FIG. 6B, the horizontal coordinate x of the reference image is taken on the horizontal axis and the disparity d is taken on the vertical axis, and the t-th disparity shown in FIG. 6A is (0) to (+) 4. This represents the matching cost MCt _{SAD1 × 1} between the standard image ILt of the frame and the reference image IRt, that is, the luminance difference between the pixels of the standard image ILt and the reference image IRt. As shown in FIG. 6B, the horizontal coordinate x of the reference image is taken on the horizontal axis and the disparity d is taken on the vertical axis, and the luminance between corresponding pixels of the reference image IL and the reference image IR in a predetermined SAD block. The sum of the absolute values of the differences is referred to as DSI (Disparity Space Image).

  As another representation of DSI, there is a method of taking the horizontal coordinate x of the reference image on the horizontal axis and the horizontal coordinate x ′ of the reference image on the vertical axis. The horizontal coordinate x ′ of the reference image is x ′ = x−d, and d is a numerical value of disparity. For details, refer to JP 2006-331108 A. Below, what is shown in FIG.6 (b) is demonstrated as DSI.

The matching cost calculation unit 301 calculates the sum of absolute values of luminance differences between corresponding pixels in a predetermined SAD block of the standard image IL and the reference image IR of the t-th frame as a matching cost MCt _{SADI of the} standard image IL and the reference image IR. _The obtained matching cost MCt _{SADI × J} of the standard image IL and the reference image IR is obtained by DSI. The DSI is calculated for each y coordinate of the reference image IL, that is, for each line of the reference image IL. Let DSI (y) be the DSI for each y coordinate of the reference image IL. Hereinafter, DSI (y) is also referred to as matching cost DSI (y) between the standard image IL and the reference image IR. The matching cost calculation unit 301 summarizes the matching costs DSI (y) in units of frames, and collectively outputs them to the DSI filter 305 as matching costs (differences) between the standard image IL and the reference image IR. Here, the matching cost DSIt of the t-th frame is a sum of the matching costs DSI (y) of the t-th frame in units of frames. However, when mounting on hardware such as an ASIC or FPGA, the matching cost DSI (y) corresponding to each y coordinate of the reference image IL may be sequentially output in order to increase the parallelism of processing. The number of DSI included in DSIt is the vertical width of the reference image IL.

The weight value calculation unit weight value calculation unit 303 is a previous frame corresponding to the matching cost DSI (y) corresponding to each y coordinate of the reference image IL of the t-th frame which is the current frame supplied from the disparity memory 311. Using the disparity DFt-1 of the (t-1) th frame, the weight value W for each element DSI (x, y, d) in the matching cost DSI (y) corresponding to each y coordinate of the reference image IL of the tth frame. (X, y, d) is calculated. The weight value W (x, y, d) is obtained from the following equation (4).

Here, α and σ are constants obtained from experimental values, and d indicates a disparity value corresponding to the element DSI (x, y, d) of the matching cost DSI (y) of the t-th frame. When the disparity filter 309 is omitted, the t−1th frame from which the horizontal noise corresponding to the matching cost DSI (y) for each y coordinate of the reference image IL of the tth frame is removed. Instead of the disparity DFt−1, the disparity Dt−1 of the (t−1) th frame from which the horizontal noise supplied from the disparity memory 311 is not removed may be substituted into (Expression 4).

  Referring to (Equation 4), the closer the disparity value d is to the corresponding disparity DFt-1 (x, y) of the immediately preceding frame, that is, the disparity DFt-1 (x, y) of the immediately preceding frame. ) And the disparity value d is smaller, the smaller the weight value W (x, y, d) is. Further, the disparity value d is a value farther from the corresponding disparity DFt-1 (x, y) of the immediately preceding frame, that is, the disparity DFt-1 (x, y) of the immediately preceding frame and the disparity value. It can be seen that the weighted value W (x, y, d) increases as the difference from d increases. The weight value calculation unit 303 calculates the weight value W (x, y, d) for each element DSI (x, y, d) in the predetermined matching cost DSI (y) of the current frame calculated using (Equation 4). The matching cost DSI (y) corresponding to all y coordinates is calculated, and is output to the DSI filter 305 as a weighted value Wt of the t-th frame collectively in units of frames.

The DSI filter DSI filter 305 is supplied from the element DSI (x, y, d) of the predetermined matching cost DSI (y) of the matching cost DSIt of the t-th frame supplied from the matching cost calculation unit 301 and the weight value calculation unit 303. The weighted matching cost DSIF (x, y) is multiplied by the weighted value W (x, y, d) corresponding to the element DSI (x, y, d) of the given matching cost DSI (y). , D).

  The DSI filter 305 is a predetermined element DSI (x, y, d) of a predetermined matching cost DSI (y) included in the t-th frame matching cost DSIT, that is, a predetermined image of the reference image ILt and the reference image IRt of the t-th frame. The sum of the absolute values of the luminance differences between corresponding pixels in the y coordinate is multiplied by the corresponding weight value W (x, y, d) to obtain the sum of the absolute values of the weighted luminance differences. Similarly to DSI, the horizontal axis x of the standard image is taken on the horizontal axis and the disparity d is taken on the vertical axis, and the sum of absolute values of weighted luminance differences between corresponding pixels of the standard image ILt and the reference image IRt is obtained. The representation is DSIF. The DSI filter 305 calculates DSIF (y) for each matching cost DSI (y) of each y coordinate of the reference image IL. The DSI filter 305 calculates a weighted matching cost DSIF (y) for all y coordinates of the reference image IL, summarizes the results in units of frames, and outputs them collectively to the optimization unit 307 as weighted matching costs for the t-th frame. To do. Here, the weighted matching cost DSIF (y) corresponding to each y coordinate of the reference image IL of the t-th frame is defined as a weighted matching cost DSIFt of the t-th frame. However, when mounting on hardware such as ASIC or FPGA, the weighted matching cost DSIF (y) corresponding to each y coordinate of the reference image IL may be sequentially output in order to increase the degree of parallelism of processing.

For example, each element DSI (x, y, d) of the matching cost DSI (y) of a predetermined y coordinate of the t th frame, that is, 1 of the predetermined y coordinate of the base image ILt and the reference image IRt of the t th frame. The weighting for the matching cost DSI (y) in the case where the matching cost MCt _{SAD1 × 1} of the luminance difference between the pixels is a numerical value indicated by the DSI in FIG. The disparity DFt-1 (x, y) of the t-1th frame corresponding to the matching cost DSI (y) of the predetermined y coordinate of the reference image IL of the tth frame is the numerical value shown in FIG. 8B. In this case, the numerical value of the disparity DFt-1 (x, y) of the t-1st frame shown in FIG. 8B is sequentially substituted into (Expression 4). Here, assuming that constant α = 0.5 and constant σ = 1, the weight values applied to each element DSI (x, y, d) of the matching cost DSI (y) of the predetermined y coordinate are shown in FIG. ) Is a numerical value indicated by each element W (x, y, d) of the weight value W shown in FIG. The numerical value of each element DSI (x, y, d) of the matching cost DSI (y) of the predetermined y coordinate shown in FIG. 8A and each element W () of the weight W shown in FIG. The numerical value of x, y, d) is substituted into (Equation 5) to obtain a weighted matching cost DSIF (y) of a predetermined y coordinate of the reference image IL of the t-th frame. The numerical value of each element DSIF (x, y, d) of the weighted matching cost DSIF (y) is the numerical value shown in DSIF shown in FIG.

  As described above, the DSI filter 305 weights each DSI (y) corresponding to each y coordinate included in the matching cost DSIT of the t-th frame using the corresponding weight value W (x, y, d), A weighted matching cost DSIF (y) is calculated. The DSI filter 305 collects DSIF (y) for each y coordinate in units of frames, and collectively outputs to the optimization unit 307 as a weighted matching cost DSIFt of the t-th frame. However, as described above, when mounted on hardware such as an ASIC or FPGA, the weighted matching cost DSIF (y) corresponding to each y coordinate of the reference image may be sequentially output.

  FIG. 9 shows the predetermined t-1 frame, t frame, and t + 1 frame when the images displayed in the t frame in the continuous t-1 frame, t frame, and t + 1 frame are switched. An example of the y-coordinate matching cost DSI (y), the corresponding weight value W (x, y, d), and the weighted matching cost DSIF (y) is shown as an image.

The optimization unit The optimization unit 307 optimizes the weighted matching cost DSIFt of the t-th frame supplied from the DSI filter 305 using a general optimization method, and calculates the disparity (disparity information) of the t-th frame. calculate. As the optimization method, a known dynamic programming method, scan line optimization method, or the like may be used. Here, optimization using a dynamic programming method will be described as an example.

The optimization of the weighted matching cost DSIF (y) shown in FIG. 8D will be described as an example. First, the optimization unit 307 includes three elements DSIF (x−1) adjacent to each element DSIF (x, y, d) on the weighted matching cost DSIF (y) illustrated in FIG. , Y, d), DSIF (x, y, d + 1), and DSIF (x-1, y, d-1) accumulated cost AC, the smallest numerical value is the accumulated cost AC of element DSIF (x, y, d). Select as (x, y, d). A predetermined element DSIF (x, y, d) is an s element, and three elements DSIF (x-1, y, d), DSIF (x, y, d + 1), DSIF (x-1, x) adjacent to the s element If y, d-1) are a element, b element, and c element, respectively, the positional relationship between the s element and the a element, b element, and c element is as shown in FIG. Here, the minimum accumulated cost AC is

, And the cumulative cost _{AC Rocc} the cumulative cost _{AC Locc,} c component of accumulated cost _{AC Mch,} b elements of a element can be determined from the following equation (6).

Here, k _occ is a constant.

The optimization unit 307 uses the expression (6) to calculate the minimum accumulated cost AC () for each element DSIF (x, y, d) of the weighted matching cost DSIF (y) corresponding to each y coordinate of the reference image IL. x, y, d) is obtained. FIG. 11 shows the accumulation of each element DSIF (x, y, d) in the weighted matching cost DSIF (y) corresponding to the predetermined y coordinate shown in FIG. 8 (d) when k _occ = 2. The cost AC (x, y, d) is shown. Here, when the maximum value of x is x = 15, the accumulated cost at x = 15 is the total cost. An element (x, y, d) having a minimum accumulated cost AC from the element (15, y, d) having the minimum total cost to the left (in the direction indicated by the broken line in FIG. 11). ). In FIG. 11, the element (x, y, d) indicated by a thick frame is the element having the minimum accumulated cost AC, and the disparity d corresponding to the element (x, y, d) indicated by the thick frame Is the optimal disparity D for the corresponding DSIF (y).

  The optimization unit 307 calculates an optimum disparity D for the weighted matching cost DSIF (y) corresponding to each y coordinate of the reference image IL of the t-th frame. The optimization unit 307 collects the optimal disparity D for the DSIF (y) corresponding to each y coordinate of the obtained reference image IL of the t-th frame in units of frames, and outputs to the disparity filter 309 in a lump. Here, the disparity Dt of the t-th frame is the sum of the optimal disparities D for the DSIF (y) corresponding to each y coordinate of the reference image IL of the t-th frame in units of frames.

Disparity Filter disparity filter 309, as described above, a filter for removing horizontal noise average filter, it may be a median filter. The disparity filter 309 removes horizontal noise from the disparity Dt supplied from the optimization unit 307. The disparity filter 309 outputs the disparity DFt from which the horizontal noise has been removed to the parallax image synthesis unit 108 and the disparity memory 311. Note that the disparity filter 309 may be arranged at the subsequent stage of the disparity memory 311. Further, the disparity filter 309 may be omitted.

  When the disparity filter 309 is arranged at the subsequent stage of the disparity memory 311 and when the disparity filter 309 is omitted, the optimization unit 307 converts the disparity Dt of the t-th frame into the disparity memory 311 and the disparity The image is output to the image composition unit 108.

  The disparity memory 311 temporarily stores the disparity DFt supplied from the disparity filter 309 or the disparity Dt supplied from the optimization unit 307, and the weighted value calculation unit 303 stores the stored disparity DFt or Dt. Output to. The disparity DFt or Dt output from the disparity memory 311 is used by the weight value calculation unit 303 to calculate the weight value Wt + 1 for the matching cost DSIt + 1 of the t + 1-th frame. When the disparity filter 309 is arranged at the subsequent stage of the disparity memory 311, the disparity Dt output from the disparity memory 311 is a weight value calculation unit after the noise in the horizontal direction is removed by the disparity filter 309. 303.

  The parallax image combining unit 108 displays image data for displaying a multi-view parallax image of the t-th frame based on the reference image IL of the t-th frame and the disparity DFt of the t-th frame supplied from the disparity filter 309. Generate and output to the timing control unit 110.

  In the stereo matching unit 106 of the stereo image display apparatus 100 according to the embodiment of the present invention, the weight value of the current frame is calculated using the above-described (Equation 4) using the disparity calculated in the frame immediately before the current frame. Then, the calculated weight value and the matching cost (difference) MC between the standard image IL and the reference image IR of the current frame, that is, the absolute value of the luminance difference between corresponding pixels of the standard image IL and the reference image IR Multiplying the sum, weighting the matching cost MC between the current frame standard image IL and the reference image IR, and calculating the disparity of the current frame based on the weighted matching cost between the current frame standard image IL and the reference image IR. calculate. If there is no significant change between the current frame image and the previous frame image, the closer the current frame disparity is to the previous frame disparity, the smaller the weight applied to the current frame. The parity is the same as or close to the disparity of the previous frame. Therefore, it is possible to reduce the disparity fluctuation between the immediately preceding frame and the current frame.

  In the stereo matching unit 106 of the stereo image display apparatus 100 according to an embodiment of the present invention, a disparity filter 309 is provided, and disparity obtained by removing horizontal noise from the disparity output from the optimization unit 307 is used. By generating image data in this way, line noise in a predetermined frame can be reduced. At the same time, by calculating the weight of the next frame using the disparity from which noise has been removed, fluctuations in disparity between the current frame and the next frame can be reduced and the horizontal noise of the current frame can be reduced. can do.

  In the above description, in the DSI filter 305, the weighting value W is weighted only once to the matching cost DSI of a predetermined frame. However, the weighting value W may be weighted to the matching cost DSI multiple times. For example, in the case of the first weighting, the first weight value Wt1 related to the matching cost DSIT of the t-th frame is a weight value based on the disparity DFt-1 of the t-1th frame supplied from the disparity memory 311. Calculated in the calculation unit 303. However, in the DSI filter 305, when the weighting value Wt is further weighted to the matching cost DSI of the t-th frame, the second weighting value Wt2 related to the matching cost DSI of the t-th frame is t from the disparity memory 311. Based on the disparity DFt1 of the th frame, the weight value calculation unit 303 calculates. The disparity DFt1 is a disparity obtained from the matching cost DSIFt of the tth frame obtained by weighting the first weighting value Wt1 to the matching cost DSIT of the tth frame. In the DSI filter 305, when the weighting value Wt is further weighted to the matching cost DSIT of the t-th frame, the third weight value Wt3 related to the matching cost DSIT of the t-th frame is obtained from the disparity memory 311 in the weight value calculation unit 303. It is calculated based on the supplied disparity DFt2 of the t-th frame. The disparity DFt2 is a disparity obtained from the matching cost DSIFt of the t-th frame obtained by weighting the matching weight DSIT of the t-th frame with the second weight value Wt2. As described above, when the weighting value is weighted a plurality of times to the matching cost DSI of the reference image IL of the predetermined frame, the weighting value used in the second and subsequent weighting is calculated based on the disparity of the predetermined frame calculated immediately before. Is done.

  The disparity memory 311 stores the disparity obtained immediately before being output from the optimization unit 307 or the disparity filter 309, and outputs the stored disparity to the weight calculation unit 303.

  In this way, the weight value calculation unit 303 calculates the weight value W based on the disparity of the same frame and repeats weighting the updated weight value W to the matching cost DSI of the same frame again. The weight value W is stabilized, and it becomes possible to calculate a more appropriate disparity. The number of times the weight value W is weighted to the DSI of the same frame can be set as appropriate.

(Second Embodiment)
Hereinafter, a stereo matching unit according to another embodiment of the present invention will be described with reference to FIGS. 12 to 16.

  FIG. 12 is a block diagram showing a configuration of a stereo matching unit 1006 according to another embodiment of the present invention. The stereo matching unit 1006 includes a dissimilarity calculation unit 1201, a weight value calculation unit 1203, a first filter 1205, an optimization unit 1207, a second filter 1209, a first memory 1211, and a second memory A memory 1213, an image data comparison unit 1215, a frame similarity calculation unit 1217, and a coefficient calculation unit 1219 are provided. Hereinafter, the dissimilarity calculation unit 1201 is the matching cost calculation unit 1201, the first filter 1205 is the DSI filter 1205, the optimization unit 1207 is the optimization unit 1207, the second filter 1209 is the disparity filter 1209, and the first memory 1211. Is called a disparity memory 1211, and the second memory 1213 is called a frame memory 1213.

  FIG. 13 is a flowchart illustrating an example of a processing flow in the stereo matching unit 1006. With reference to FIG.12 and FIG.13, the flow of the process in a stereo matching part is demonstrated easily.

  The matching cost calculation unit 1201 calculates a matching cost (difference) MCt between the reference image IRt and the reference image IRt of the t-th frame subjected to height correction supplied from the stereo image parallelization unit 104 (S1301). ) Find DSIT. The matching cost calculation unit 1201 outputs the obtained DSIt to the DSI filter 1205. Hereinafter, DSIt is also referred to as a matching cost DSIt.

  The frame memory 1213 temporarily stores the reference image ILt supplied from the stereo image parallelization unit 104 and outputs the stored reference image ILt to the image data comparison unit 1215.

  The image data comparison unit 1215 receives the t-th frame reference image ILt that is the current frame supplied from the stereo image parallelization unit 104 and the t−1-th frame reference image ILt− that is the immediately preceding frame supplied from the frame memory 1213. 1 is compared, and a luminance difference DIFFt between corresponding pixels in the reference image ILt and the reference image ILt-1 of the (t-1) th frame is calculated (S1302). The image data comparison unit 1215 uses the frame similarity calculation unit 1217 and the coefficient calculation unit to calculate the luminance difference DIFFt between corresponding pixels of the calculated reference image ILt of the t-th frame and the reference image ILt-1 of the t−1th frame. It outputs to 1219.

  The frame similarity calculation unit 1217 is based on the luminance difference DIFFt between pixels corresponding to each other in the reference image ILt of the t-th frame and the reference image ILt-1 of the t−1th frame supplied from the image data comparison unit 1215. The similarity between the (t-1) th frame and the tth frame is obtained (S1303), and the similarity SIMt is output to the coefficient calculation unit 1219.

  The coefficient calculation unit 1219 obtains the difference DIFFt in luminance between the pixels corresponding to each other in the reference image ILt of the t-th frame and the reference image ILt-1 of the t−1th frame supplied from the image data comparison unit 1215 and the frame similarity. The coefficient σt is calculated based on the similarity SIMt between the t−1th frame and the tth frame supplied from the sex calculation unit 1217 (S1304), and the calculated coefficient σt is output to the weight value calculation unit 1203.

  The weight value calculation unit 1203 uses the coefficient σt supplied from the coefficient calculation unit 1219 and the disparity DFt−1 of the t−1th frame, which is the previous frame supplied from the disparity memory 1211, to determine the current frame t. The weight value Wt of the th frame is calculated (S1305) and output to the DSI filter 1205.

  Here, the matching cost (difference) MCt between the standard image ILt and the reference image IRt is calculated in the matching cost calculation unit 1201 (S1301), and the standard image ILt and the t−1th frame in the t-th frame in the image data comparison unit 1215. Calculation of luminance difference DIFFt between mutually corresponding pixels in frame reference image ILt-1 (S1302), and calculation of similarity between t-1th frame and tth frame in frame similarity calculation unit 1217 (S1303) The order of the calculation of the coefficient σt in the coefficient calculation unit 1219 (S1304) and the calculation of the weight value Wt of the t-th frame in the weight value calculation unit 1203 (S1305) may be reversed or simultaneous.

  The DSI filter 1205 multiplies the matching cost DSIt supplied from the matching cost calculation unit 1201 by the weight value Wt supplied from the weight value calculation unit 1203 to calculate a weighted matching cost DSIFt (S1306). The DSI filter 1205 outputs the weighted matching cost DSIFt to the optimization unit 1207.

  The optimization unit 1207 optimizes the weighted matching cost DSIFt supplied from the DSI filter 1205 using a general optimization method, and calculates the optimal disparity Dt of the t-th frame (S1307). As the optimization method, a known dynamic programming method, scan line optimization method, or the like may be used. The optimization unit 1207 outputs the calculated disparity Dt of the t-th frame to the disparity filter 1209.

  The disparity filter 1209 is a filter that removes noise in the horizontal direction, and an average value filter, an intermediate value filter, or the like may be used. The disparity filter 1209 removes horizontal noise from the disparity Dt supplied from the optimization unit 1207 (S1308). The disparity filter 1209 outputs the disparity DFt from which the horizontal noise has been removed to the parallax image synthesis unit 108 and the disparity memory 1211. Note that the disparity filter 1209 may be disposed at a subsequent stage of the disparity memory 1211. Further, the disparity filter 1209 may be omitted.

  The disparity memory 1211 temporarily stores the disparity DFt supplied from the disparity filter 1209 (S1309), and outputs the stored disparity DFt to the weight value calculation unit 1203 (S1310). The disparity DFt output from the disparity memory 1211 is used by the weight value calculation unit 1203 to calculate the weight value Wt + 1 related to the matching cost DSIt + 1 of the t + 1-th frame. When the disparity filter 1209 is arranged at the subsequent stage of the disparity memory 1211 and when the disparity filter 1209 is omitted, the disparity memory 1211 stores the disparity Dt supplied from the optimization unit 1207. When the disparity filter 1209 is arranged at the subsequent stage of the disparity memory 1211, the disparity Dt output from the disparity memory 1211 is a weight value calculation unit after the noise in the horizontal direction is removed by the disparity filter 1209. 1203.

  Based on the disparity DFt supplied from the disparity filter 1209, the parallax image synthesis unit 108 generates image data for displaying a multi-view parallax image of the t-th frame and outputs the image data to the timing control unit 110. When the disparity filter 1209 is omitted, the parallax image synthesis unit 108 generates image data for displaying a multi-view parallax image of the t-th frame based on the disparity Dt supplied from the optimization unit 1207. And output to the timing control unit 110.

  Hereinafter, the processing in each configuration of the stereo matching unit 1006 will be described in detail with reference to FIGS. 14 and 16. However, in the configuration of the stereo matching unit 1006, the same or similar configuration as the stereo matching unit 106 according to the first embodiment is not described repeatedly.

  The matching cost calculation unit 1201 has the same configuration as the matching cost calculation unit 301 of the stereo matching unit 106 according to the first embodiment, and the matching cost (difference) between the reference image IL and the reference image IR in the matching cost calculation unit 1201. The calculation method of DSIt is similar to the calculation method of the matching cost (difference) DSIT between the base image IL and the reference image IR in the matching cost calculation unit 301 of the stereo matching unit 106 according to the first embodiment. Explanation here is omitted.

The frame memory frame memory 1213 temporarily stores the reference image (here, the left image) IL supplied from the stereo image parallelization unit 104, and outputs the stored reference image IL to the image data comparison unit 1215. When the stereo matching unit 1006 calculates the disparity of the t-th frame that is the current frame, the frame memory 1213 uses the t-1th reference image ILt-1 that is the previous frame stored as the image data comparison unit 1215. Output to.

Image data comparison unit The image data comparison unit 1215 receives the t-th frame reference image ILt that is the current frame supplied from the stereo image parallelization unit 104 and the t−1-th frame that is the previous frame supplied from the frame memory 1213. The reference image ILt-1 is compared, and a luminance difference DIFFt between pixels corresponding to each other in the reference image ILt and the reference image ILt-1 is calculated. The image data comparison unit 1215 uses the frame similarity calculation unit 1217 and the coefficient calculation unit to calculate the luminance difference DIFFt between corresponding pixels of the calculated reference image ILt of the t-th frame and the reference image ILt-1 of the t−1th frame. It outputs to 1219.

  FIG. 14 is a block diagram showing the configuration of the image data comparison unit 1215. The image data comparison unit 1215 includes filters 1401a and 1401b and a difference calculation unit 1403. The filters 1401a and 1401b may be average value filters or the like, and remove noise from each of the reference image ILt of the t-th frame and the reference image ILt-1 of the t−1th frame, and the tth from which the noise has been removed. The reference image ILFt of the frame and the reference image ILFt-1 of the (t−1) th frame are output to the difference calculation unit 1403.

The difference calculation unit 1403 calculates a luminance difference DIFFt between pixels corresponding to each other in the reference image ILFt and the reference image ILFt−1 using the following (Expression 7).

Here, DIFFt (x, y) is the difference between the (x, y coordinates) of the reference image ILFt-1 and the (x, y coordinates) of the reference image ILFt-1, and ILF _t-1 (x, y). y) indicates the luminance at (x, y coordinates) of the reference image ILFt-1, ILF _t (x, y) indicates the luminance at (x, y coordinates) of the reference image ILFt, and Imax indicates the reference image. The maximum value of ILF luminance is shown. The difference calculation unit 1403 outputs the luminance difference DIFFt between the corresponding pixels of the reference image ILt and the reference image ILt−1 calculated using (Expression 7) to the frame similarity calculation unit 1217 and the coefficient calculation unit 1219. .

Frame similarity calculation unit The frame similarity calculation unit 1217 corresponds to each other in the reference image ILFt of the t-th frame and the reference image ILFt-1 of the t-1 frame supplied from the difference calculation unit 1403 of the image data comparison unit 1215. The similarity SIMt between the t−1th frame and the tth frame is obtained based on the luminance difference DIFFt between the pixels to be processed. The frame similarity calculation unit 1217 obtains the similarity SIMt between the t−1th frame and the tth frame using the following (Equation 8).

Here, α is a constant, and X and Y indicate the horizontal width and vertical width of the input reference image IL, respectively. The frame similarity calculation unit 1217 outputs the calculated similarity SIMt between the t−1th frame and the tth frame to the coefficient calculation unit 1219.

Coefficient calculation unit The coefficient calculation unit 1219 is a luminance difference DIFFt between pixels corresponding to each other in the reference image ILFt of the t-th frame and the reference image ILFt-1 of the t−1th frame supplied from the image data comparison unit 1215. The coefficient σt is calculated based on the similarity SIMt between the t−1th frame and the tth frame supplied from the frame similarity calculation unit 1217 (S1304), and the obtained coefficient σt is output to the weight value calculation unit 1203. To do. The coefficient calculation unit 1219 obtains the coefficient σt using the following (Expression 9) and (Expression 10).

Here, σ ₀ and σ ₁ are constants, and σ ₀ > σ ₁ . The relationship between σ ₀ and σ ₁ is as shown in FIG. The coefficient calculation unit 1219 outputs the calculated coefficient σt to the weight value calculation unit 1203.

The weight calculation unit 1203 matches the coefficient σt supplied from the coefficient calculation unit 1219 and the matching cost DSI (y) corresponding to each y coordinate of the reference image IL of the t-th frame which is the current frame supplied from the disparity memory 1211. ), The element DSI (x, y) of the matching cost DSI (y) corresponding to each y coordinate of the reference image IL of the tth frame, using the disparity DFt-1 of the t−1th frame which is the immediately preceding frame corresponding to , D), the weight value W (x, y, d) is calculated. The weight value W (x, y, d) is obtained from the following equation (11).

Here, α is a constant obtained from experimental values, and d indicates a disparity value corresponding to the element DSI (x, y, d) of the matching cost DSI (y) of the t-th frame. When the disparity filter 1209 is omitted, instead of the disparity DFt-1 of the (t-1) th frame from which the horizontal noise is removed, corresponding to the DSI (y) of each y coordinate of the tth frame. The disparity Dt-1 of the (t-1) th frame from which the horizontal noise is not removed supplied from the disparity memory 1211 may be substituted into (Equation 11).

  Referring to (Equation 11), the closer the disparity value d is to the disparity DFt-1 (x, y) of the immediately preceding frame corresponding to each matching cost DSI of the current frame, that is, the disparity of the immediately preceding frame. It can be seen that the smaller the difference between the parity DFt-1 (x, y) and the disparity value d, the smaller the value of the weight value W (x, y, d). Further, the disparity value d is farther from the disparity DFt-1 (x, y) of the immediately preceding frame corresponding to each matching cost DSI of the current frame, that is, the disparity DFt-1 (x of the immediately preceding frame). , Y) and the value of the disparity value d increase, the weight value W (x, y, d) increases.

FIG. 16 is a graph showing the weight value W when (constant 12) shown below is set to a constant α = 0.5 and σ = 2 and σ = 10 are substituted as an example. Here, the vertical axis represents the weight value W, and the horizontal axis represents the difference Δd between the disparity DFt-1 (x, y) of the immediately preceding frame and the disparity value d. In FIG. 16, the weight value W when σ = 2 is substituted is shown by a curve 1601, and the weight value W when σ = 10 is substituted is shown by a curve 1602.

In (Expression 12), Δd represents the value of DFt−1 (x, y) −d. As indicated by a curve 1601 in FIG. 16, the smaller the value of the coefficient σt, that is, the higher the similarity between the t-th frame and the t−1-th frame, the more the weight value W (x, y, d) is the previous frame. It can be seen that the peak width centered on the difference 0 from the disparity DFt-1 (x, y) is small. The weight value calculation unit 1203 calculates the weight value W (x, y, d) for each element DSI (x, y, d) in the predetermined matching cost DSI (y) of the current frame calculated using (Equation 11). Collectively in units of frames, output to the DSI filter 1205 as the weight value Wt of the t-th frame.

The DSI filter DSI filter 1205 supplies the element DSI (x, y, d) of the predetermined matching cost DSI (y) of the t-th frame matching cost DSIt supplied from the matching cost calculation unit 1201 and the weight value calculation unit 1203. The weighted matching cost DSIF (y) is calculated by multiplying the weighted value W (x, y, d) corresponding to the element DSI (x, y, d) of the given predetermined matching cost DSI (y). To do. The DSI filter 1205 outputs the calculated weighted matching cost DSIF (y) to the optimization unit 1207. A multiplication expression of the predetermined matching cost DSI (y) in the DSI filter 1205 and the corresponding weight value W (x, y, d) is described in the description of the DSI filter 305 of the stereo matching unit 106 according to the first embodiment. (Equation 5).

  The DSI filter 1205 includes an element DSI (x, y, d) of a predetermined matching cost DSI (x, y, d) included in the matching cost DSIt of the t-th frame, that is, the reference image ILt and the reference image of the t-th frame. The sum of the absolute values of the luminance differences between the corresponding pixels at a predetermined y coordinate of IRt is multiplied by the corresponding weight value W (x, y, d), and the sum of the absolute values of the weighted luminance differences is obtained. Ask. The DSI filter 1205 takes the horizontal coordinate x of the reference image on the horizontal axis and the disparity d on the vertical axis, and calculates the sum of the absolute values of the weighted luminance differences between the corresponding pixels of the reference image IL and the reference image IR. The represented DSIF is calculated for each matching cost DSI (y) corresponding to each y coordinate of the reference image IL of the t-th frame. The DSI filter 1205 collects the weighted matching cost DSIF (y) corresponding to each y coordinate of the reference image IL of the t-th frame in units of frames, and sends it to the optimization unit 1207 as the weighted matching cost DSIFt of the t-th frame. Output all at once. Here, when mounting on hardware such as ASIC or FPGA, weighted matching costs DSIF (y) corresponding to each y coordinate of the reference image IL may be sequentially output in order to increase the degree of parallelism of processing. .

  Regarding the configuration of the optimization unit 1207, the disparity filter 1209, and the disparity memory 1211 and the processing in each configuration, the optimization unit 307, the disparity filter 309, and the disparity memory 311 in the stereo matching unit 106 according to the first embodiment. Therefore, the description is omitted here.

  In the stereo matching unit 1006 of the stereo image display apparatus 100 according to the embodiment of the present invention, the coefficient σ based on the similarity between the current frame and the previous frame is calculated using (Expression 7) to (Expression 10), The weight value of the current frame is calculated using the above-described (Equation 11) using the coefficient σ and the disparity of the frame immediately before the current frame, and the calculated weight value is matched with the reference image and the reference image of the current frame Multiplying the cost (difference) MC, that is, the sum of absolute values of the luminance differences between corresponding pixels of the base image IL and the reference image IR, and the matching cost MC between the base image IL and the reference image IR of the current frame And the disparity of the current frame is calculated based on the matching cost between the weighted standard image of the current frame and the reference image. If there is no significant change between the current frame image and the previous frame image, the closer the current frame disparity is to the previous frame disparity, the smaller the weight applied to the current frame. The parity is the same as or close to the disparity of the previous frame. Also, if the overall similarity between the current frame image and the previous frame image is high, the closer the disparity of the current frame is to the disparity of the previous frame, the smaller the weight applied to the current frame. The disparity is equal to or closer to the disparity of the previous frame. Therefore, it is possible to reduce the disparity fluctuation between the immediately preceding frame and the current frame.

  In the stereo matching unit 1006 of the stereo image display apparatus 100 according to an embodiment of the present invention, a disparity filter 1209 is provided, and disparity obtained by removing horizontal noise from the disparity output from the optimization unit 1207 is used. By generating image data in this way, line noise in a predetermined frame can be reduced. At the same time, by calculating the weight of the next frame using the disparity from which noise has been removed, fluctuations in disparity between the current frame and the next frame can be reduced and the horizontal noise of the current frame can be reduced. can do.

  In the above description, the example in which the weighting value W is weighted only once to the matching cost DSI of the predetermined frame in the DSI filter 1205 has been described, but as described in the description of the stereo matching unit 106 according to the first embodiment. The weighting value W may be weighted a plurality of times to the matching cost DSI of a predetermined frame. During the process of calculating the disparity of the same frame in the stereo matching unit 1006, the disparity memory 1211 stores the disparity obtained immediately before being output from the optimization unit 1207 or the disparity filter 1209, The stored disparity is output to the weight value calculation unit 1203.

  In this manner, the weight value calculation unit 1203 calculates the weight value W based on the disparity of the same frame and repeats the weighting of the updated weight value W to the matching cost DSI of the same frame again. The weight value W is stabilized, and it becomes possible to calculate a more appropriate disparity. The number of times the weighting value W is weighted to the matching cost DSI of the same frame can be set as appropriate.

DESCRIPTION OF SYMBOLS 100 Stereo image display apparatus 102 Stereo image input part 104 Stereo image parallelizing part 106 Stereo matching part 108 Parallax image synthetic | combination part 110 Timing control part 301 Dissimilarity calculating part (matching cost calculating part)
303 Weight calculation unit 305 First filter (DSI filter)
307 Optimization section (optimization section)
309 Second filter (disparity filter)
311 First memory (disparity memory)

Claims (18)

A stereo image of a predetermined frame is input, and a dissimilarity calculation unit that calculates the dissimilarity of the stereo image;
A weight value calculator that calculates a weight value of the predetermined frame based on disparity information of a frame immediately before the predetermined frame;
A first filter that weights the weight value supplied from the weight value calculation unit to the difference degree of the stereo image supplied from the difference value calculation unit;
An optimization unit that calculates disparity information of the predetermined frame based on a difference weighted by the weight value supplied from the first filter;
A first memory that stores the disparity information of the predetermined frame supplied from the optimization unit and outputs the stored disparity information to the weight calculation unit;
A stereo image processing apparatus comprising:   The stereo image processing apparatus according to claim 1, further comprising: a second filter that removes noise from the disparity information calculated by the optimization unit and outputs the disparity information from which the noise has been removed to the first memory. .   2. The stereo image according to claim 1, further comprising: a second filter that removes noise from the parallax information output from the first memory and outputs the parallax information from which noise has been removed to the weight value calculation unit. Processing equipment. A second memory for storing image data of either a standard image or a reference image among the stereo images immediately before the predetermined frame;
Of the stereo image of the predetermined frame, the image data of either the standard image or the reference image corresponding to the image data stored in the second memory is input, and the immediately preceding frame supplied from the second memory An image data comparison unit that compares the image data of the predetermined frame and the image data of the predetermined frame for each corresponding pixel,
4. The stereo image processing apparatus according to claim 1, wherein the weight calculation unit calculates a weight value of the predetermined frame based on a comparison result from the image data comparison unit. 5. .   The weight value calculation unit is calculated when the comparison result from the image data comparison unit indicates that there is little change in corresponding pixel data between the image data of the immediately preceding frame and the image data of the predetermined frame. The smaller the difference between the disparity information corresponding to the stereo image of the predetermined frame and the disparity information of the immediately preceding frame, the smaller the weight value is, and the disparity information corresponding to the calculated difference between the stereo images of the predetermined frame The stereo image processing apparatus according to claim 4, wherein the weighting value is set to a larger value as a difference between the disparity information of the previous frame and the previous frame is larger. A frame similarity calculation unit for determining similarity between the immediately preceding frame and the predetermined frame;
The stereo image processing apparatus according to claim 4 or 5, wherein the weight calculation unit calculates a weight value of the predetermined frame based on a comparison result from the frame similarity calculation unit.   When the result supplied from the frame similarity calculation unit indicates that the similarity between the immediately preceding frame and the predetermined frame is high, the weight value calculation unit corresponds to the calculated difference between the stereo images of the predetermined frame. The smaller the difference between the disparity information and the disparity information of the immediately preceding frame, the smaller the weight is, and the difference between the disparity information corresponding to the calculated difference between the stereo images of the predetermined frame and the disparity information of the immediately preceding frame is The stereo image processing apparatus according to claim 6, wherein the larger the value is, the larger the weight value is. When the weight is a small value, the optimization unit calculates disparity information that is the same as or close to the disparity information of the immediately preceding frame as disparity information of the predetermined frame,
The stereo image according to claim 5 or 7, wherein when the weight value is a large value, the optimization unit calculates disparity information different from the disparity information of the immediately preceding frame as disparity information of the predetermined frame. Processing equipment. A stereo image of a predetermined frame is input,
Calculating the difference between the stereo images;
A weight value of the predetermined frame is calculated based on the disparity information calculated immediately before,
Weighting the weights to the stereo image dissimilarity,
Calculating disparity information of the predetermined frame based on the dissimilarity weighted by the weight value;
Storing the disparity information of the predetermined frame, and using the stored disparity information for calculating a next weight value;
Stereo image processing method including   The stereo image processing method according to claim 9, further comprising removing noise from the parallax information before storing the parallax information of the predetermined frame.   The method further comprises removing noise from the disparity information after storing the disparity information of the predetermined frame and before using the stored disparity information for calculating a next weight value. The stereo image processing method according to claim 9. Save image data of either the standard image or the reference image among the stereo images immediately before the predetermined frame,
Comparing the image data of either the standard image or the reference image corresponding to the stored image data among the stereo image of the predetermined frame and the image data of the immediately preceding frame for each corresponding pixel, Including
The stereo image processing method according to claim 9, wherein a weight value of the predetermined frame is calculated based on a comparison result.   When the comparison result indicates that there is little change in the pixel data corresponding to the image data of the immediately preceding frame and the image data of the predetermined frame, the disparity information corresponding to the calculated difference between the stereo images of the predetermined frame And the smaller the difference between the disparity information of the immediately preceding frame and the smaller the weight value, the larger the difference between the disparity information corresponding to the calculated difference between the stereo images of the predetermined frame and the disparity information of the immediately preceding frame. The stereo image processing method according to claim 12, wherein the weight value is set to a large value. Determining similarity between the immediately preceding frame and the predetermined frame;
The stereo image processing method according to claim 12 or 13, wherein a weight value of the predetermined frame is calculated based on a result of similarity.   When the similarity result indicates that the similarity between the immediately preceding frame and the predetermined frame is high, the difference between the calculated disparity information corresponding to the difference between the stereo images of the predetermined frame and the disparity information of the immediately preceding frame The smaller the value is, the smaller the weight value is, and the larger the difference between the disparity information corresponding to the calculated disparity of the stereo image of the predetermined frame and the disparity information of the immediately preceding frame is, the larger the weight value is. The stereo image processing method according to claim 14, wherein: When the weight value is a small value, disparity information that is the same as or close to the disparity information of the frame immediately before the predetermined frame is calculated as the disparity information of the predetermined frame;
16. The stereo image processing method according to claim 13, wherein when the weight value is a large value, disparity information different from the disparity information of the immediately preceding frame is calculated as disparity information of the predetermined frame. Using the calculated disparity information, recalculate the weight value of the predetermined frame,
Weighting the recalculated weight value to the difference between the stereo images,
Calculating disparity information of the predetermined frame based on the degree of difference weighted by the weight value calculated again,
Storing the calculated disparity information of the predetermined frame, and using the stored disparity information for calculating a next weight value;
The stereo image processing method according to claim 9, comprising: A display panel;
A stereo image input unit to which a video signal including a stereo image is input;
A stereo image parallelizing unit that performs height correction of the stereo image and generates a corrected stereo image;
A stereo image processing device that performs stereo matching of the corrected stereo image and generates parallax information of the one image;
Using the one image and the parallax information, a parallax image combining unit that generates image data for creating a parallax video;
A timing control unit for outputting the image data;
The stereo image processing apparatus includes:
A stereo image of a predetermined frame is input, and a dissimilarity calculation unit that calculates the dissimilarity of the stereo image;
A weight value calculator that calculates a weight value of the predetermined frame based on disparity information of a frame immediately before the predetermined frame;
A first filter that weights the weight value supplied from the weight value calculation unit to the difference degree of the stereo image supplied from the difference value calculation unit;
An optimization unit that calculates disparity information of the predetermined frame based on a difference weighted by the weight value supplied from the first filter;
A first memory that stores the disparity information of the predetermined frame supplied from the optimization unit and outputs the stored disparity information to the weight calculation unit;
Stereo image display device comprising: