JP2013065179A - Image processing device and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device and an image processing method which enable more accurate parallax detection, that is, more accurate distance detection by correcting parallax based on a difference in spot shape within a screen.SOLUTION: An image processing device comprising input means that inputs a plurality of images captured from a plurality of viewpoints, and parallax calculation means that calculates parallax between the plurality of images comprises parallax correction means that corrects the parallax calculated by the parallax calculation means on the basis of a spot shape depending on an image position within an image.

Description

  The present invention relates to an image processing apparatus and an image processing method for processing an image captured by a stereo camera.

  With an ideal camera, if a point light source is photographed as a subject, the point light source image on the imaging surface should be a bright spot. However, with an actual camera, it is photographed due to the MTF characteristics of the imaging lens system. The point light source image has a certain spread. The distribution shape of the spread of the point light source image is called a spot shape here. An example of the spot shape is shown in FIG. On the left side of the figure, the vertical and horizontal axes represent the vertical and horizontal positions on the screen, and the luminance distribution is represented by contour lines. Moreover, the right side of this figure represents the X-axis cross section of the left figure.

  In general, the spot shape has a bell shape as shown in FIG. Factors that determine this shape include lens system design, the wavelength of incident light, and individual differences due to lens assembly tolerances, and also vary depending on the position on the screen. Generally, a symmetrical shape as shown in FIG. 10A is taken near the center of the screen, but in many cases, an asymmetrical shape as shown in FIG. 10B is taken as the distance from the optical axis increases.

  Here, the influence of such a change in the spot shape on the parallax detection of the stereo camera will be considered. For example, assuming that there is no optical distortion or camera assembly error, or a lens that has been completely corrected, and only the difference in spot shape, the stereo camera 70 and three point light source subjects A, B, and C are shown in FIG. When the subjects A and B are photographed in such a manner, it can be seen that both the subjects A and B are in front of the left camera 71, but A is far (infinity) and B is near the camera. It can also be seen that the subject C is in the same direction as B as viewed from the right camera 72.

  Therefore, as shown in FIG. 12, in the two stereo images to be shot, in the left camera image 81, both A and B appear in the center of the screen (A1, B1), and only C appears on the left side (C1). . On the other hand, in the right camera image 82, A appears in the center of the screen (A2), and B and C appear in the left side of the screen (B2, C2).

  FIG. 13 shows examples of two types of spot shapes. Here, for easy understanding, only one dimension in the horizontal direction is considered, and a triangular shape as shown in FIG. 12 is assumed. FIG. 13A shows a left-right symmetric type 91, and FIG. 13B shows an asymmetric type 92. Both the left camera 71 and the right camera 72 have a symmetric spot (a) near the center of the screen and an asymmetric spot (b) near the left end of the screen. That is, assume that A1, B1, and A2 shown in FIG. 12 have the spot shape of FIG. 13A, and C1, B2, and C2 of FIG. 12 have the spot shape of FIG.

  Next, FIG. 14 shows calculation results of self and cross-correlation of two types of spot shapes. 14 is the autocorrelation of the symmetric spot shown in FIG. 13A, bb shown in FIG. 14 is the autocorrelation of the non-target spot shown in FIG. 13B, and FIG. The ab shown shows the cross-correlation of two types of spots. From FIG. 14, it can be seen that the correlation value of aa and bb is minimum when x = 0, but the minimum value of ab is x <0. This is because the correlation 91 between the symmetric spot and the correlation 92 between the non-target spots is maximized because the peak positions of the two spots do not coincide with each other. ) It shows that the arrangement is shifted. It can be seen that the amount of deviation d is different from the position of the center of gravity of the non-target spot.

  In the stereo image of FIG. 12, since the subject A is shown in the center of the screen for both the left and right images, the spot shapes are both symmetrical as shown in FIG. 13A. A correlation coefficient in the form aa shown is obtained. Since it is assumed that the camera internal and external parameters are completely corrected, the parallax of the subject A at infinity is zero. Similarly, subject C is both asymmetric in the spot shape shown in FIG. 13B, and is obtained as parallax 0 from the movement correlation in the shape of bb shown in FIG.

  However, in the case of subject B, the spot shape in the left camera image 81 is the symmetric type shown in FIG. 13A and the right camera image 82 is the asymmetric type shown in FIG. As for the corresponding point position, a value deviated by d shown in FIG. 15 is obtained from the difference between the peak positions of the two spots.

  14 shows a correlation function, FIG. 16A shows values calculated by SAD (sum of absolute differences) often used in stereo parallax calculation, and FIG. 16B shows SSD ( The value calculated by the sum of squared differences) is shown. The minimum difference value between the symmetric spot shown in FIG. 13A and the asymmetric spot shown in FIG. 13B is shifted from 0 as in FIG. 14, and these evaluation values are used. Even in this case, it can be seen that the calculated disparity value includes the same error as in the case of using the correlation.

  Therefore, Patent Document 1 discloses a technique for correcting a parallax calculated as a difference in coordinates of corresponding point positions as described above by referring to a table. The invention according to Patent Document 1 is provided with a means for correcting the image position to an ideal coordinate value free from distortion and assembly error for the purpose of correcting the distortion characteristics of the camera and the variation of the camera mounting position. It is.

  However, even if the technique shown in Patent Document 1 is used to completely correct distortion characteristics, camera mounting position deviations, etc., to the ideal image position based on the spot peak position or spot centroid position, Due to the difference in shape, an error occurs in the result of the parallax calculation.

  Therefore, the present invention provides an image processing apparatus that enables more accurate parallax detection, that is, more accurate distance detection, by correcting parallax based on the difference in spot shape in the screen. With the goal.

  In order to solve the above problems, an image processing apparatus of the present invention is an image processing apparatus including an input unit that inputs a plurality of images captured from a plurality of viewpoints, and a parallax calculation unit that calculates a parallax between the plurality of images. The apparatus includes a parallax correcting unit that corrects the parallax calculated by the parallax calculating unit based on a spot shape corresponding to an image position in the image.

  According to the present invention, since the parallax is corrected in consideration of the difference in spot shape, a highly accurate parallax calculation can be realized.

1 is a block diagram illustrating an image processing apparatus according to a first embodiment of the present invention. It is a figure which shows the area | region division of the image for a spot shape table regarding 1st Embodiment of this invention. It is the schematic which shows the structure of the spot shape measuring apparatus regarding 1st Embodiment of this invention. It is a figure which shows the one-dimensional luminance distribution data integrated | accumulated from the laser spot picked-up image regarding 1st Embodiment of this invention. It is a figure which shows the luminance distribution data in 1 / N pixel unit regarding 1st Embodiment of this invention. It is a figure which shows the example of a correlation bias table regarding 1st Embodiment of this invention. It is a flowchart explaining an image process regarding embodiment of this invention. It is a block diagram which shows the image processing apparatus of 2nd Embodiment of this invention. It is a figure which shows the approximation by the difference of the bias value of a standard spot reference | standard regarding 2nd Embodiment of this invention. It is a figure which shows (a) symmetrical type spot shape and (b) asymmetric type spot shape as a general spot shape. It is a figure which shows the arrangement | positioning relationship between a stereo camera and a to-be-photographed object. In FIG. 10, it is a figure which shows the stereo image of the to-be-photographed object which a right-and-left camera image | photographs. It is a figure showing (a) symmetrical spot shape and (b) asymmetrical spot shape caught only in one dimension. It is a figure which shows the calculation result of the auto / cross correlation of (a) symmetrical spot shape shown in FIG. 13, (b) asymmetrical spot shape. It is a figure which shows arrangement | positioning with which the cross correlation of (a) symmetrical spot shape shown in FIG. 13 and (b) asymmetrical spot shape becomes the maximum. It is a figure which shows the value calculated by (a) SAD (difference absolute value sum) and (b) SSD (difference sum of squares) of said 2 types (symmetric / asymmetric) spot shape.

[First Embodiment]
An image processing apparatus 1 according to a first embodiment of the present invention will be described with reference to FIGS. FIG. 1 is a schematic block diagram showing an image processing apparatus 1 of the present embodiment. Two images photographed by a stereo camera composed of a set of two general parallel equivalence units are input to the image processing apparatus 1 from the left side of the figure via input means (not shown). Reference image and comparison image). Here, it is assumed that the distortion characteristics of the images taken by each camera, the displacement of the relative positions of the two cameras, and the like are corrected in each camera, and the input stereo image has a horizontal epipolar line.

  First, with reference to FIG. 1, the corresponding point search unit 11 as the parallax calculation unit, the correlation bias table 12 and the parallax correction unit 13 as the parallax correction unit will be described in the following order. The correlation bias table 12 functions as a parallax bias value determination unit further provided in the parallax correction unit.

  The corresponding point search unit 11 searches for corresponding points of the left and right camera images in the same manner as a general stereo camera. That is, the reference image is divided into regions, and (1) one region in the reference image is selected as the reference region in the raster scan order. (2) In the comparison image, the normalized correlation between the pixel values of the reference area and each comparison area is calculated while moving the comparison area pixel by pixel on the epipolar line from the same position as the reference area selected in (1). (3) The position of the comparison area where the correlation coefficient is the largest is taken as the corresponding point position.

  In the above processing by the corresponding point search unit 11, the position coordinates of the center of the reference area (reference point position), the position coordinates of the center of the corresponding comparison area (corresponding point position), and the parallax (difference between the two x-axis coordinates). Three pieces of information are output for each reference area. Here, the coordinate value of the pixel unit is obtained as the corresponding point position, but the corresponding point position can also be obtained with the accuracy of the pixel unit or less by the interpolation calculation from the correlation value in the vicinity of the maximum correlation position. Further, instead of the maximum value of the correlation coefficient, the minimum value of the sum of absolute differences (SAD) or the sum of squared differences (SSD) may be used.

  The correlation bias table 12 is a table in which the deviation amount of the correlation peak position corresponding to the spot shape at each position on the image plane, which is measured in advance, is recorded. The contents and setting method will be described later. The correlation bias table 12 inputs the reference point and the position coordinates of the corresponding point, and outputs a deviation amount (parallax bias) in the combination.

  The parallax correction unit 13 adds the parallax bias output from the correlation bias table 12 to the parallax output from the corresponding point search unit 11 to correct the parallax error due to the difference in spot shape.

  In the following, the process from spot shape measurement to bias value determination will be described for the present embodiment. The correlation bias table 12 is used to acquire a parallax correction value (bias value) corresponding to the spot shape at that position from position information in the image plane. The image plane is divided into regions as shown in FIG. 2 (here, 5 × 3 in the horizontal direction), the representative spot shape for each region is measured, and the corresponding bias value is recorded.

  The spot shape measurement will be described below with reference to FIG. FIG. 3 is a schematic diagram showing the configuration of the spot shape measuring apparatus 20 that measures the spot shape at each position on the image plane divided into regions. A spot shape measurement camera 21 is installed on a camera turntable 22 around two axes of a vertical axis and a horizontal axis, and parallel laser light is incident from the left side of the figure by a light emitting device (not shown). By installing and adjusting the angle of the turntable in this way, a laser beam spot can be formed at an arbitrary position on the image plane.

  When measuring the spot shape of the center of each region, for example, region 0 in FIG. 2, the angle of the turntable 22 is adjusted so that the laser spot appears at point A in the figure, and an image is taken. Here, since parallax in the x-axis direction is a problem, the obtained images are integrated in the y-axis direction to create luminance distribution data representing a one-dimensional spot shape. FIG. 4 shows the result of calculating the one-dimensional luminance distribution data from the laser spot photographed image.

  The upper part of FIG. 4 shows one-dimensional luminance distribution data obtained from the laser spot image 31 which is a part of the laser spot image. Further, in the middle part of FIG. 4, the result of integrating the one-dimensional luminance distribution data in the vertical direction is shown as a vertical integration result 32. Furthermore, the vertical integration result 32 is shown as a graph 33 in the lower part of FIG.

  In addition, as shown in FIG. 5, every time the image is rotated around the horizontal rotation axis using a 1 / N pixel equivalent camera, if N pieces of images are taken and rearranged, the spot shape data in units of 1 / N pixels is obtained. Can be measured. In FIG. 5, the second image 42 is obtained by calculating the luminance distribution data in units of ½ pixels obtained by shifting the ½ pixel for the second image 42 with respect to the first image 41. 43. In addition, the accuracy of the measurement data can be improved by a general data processing method such as an average of a large number of measurement values or spot shape model fitting based on lens design data.

  Next, determination of the bias value will be described below with reference to FIG. A cross-correlation function between the spot shapes measured in each region is calculated, and a correction amount is recorded for the combination of the two regions. FIG. 6 shows an example of the correlation bias table 12 in which the correction amount is recorded. Here, the vertical axis represents the region number of the reference image, the horizontal axis represents the region number of the comparison image, and the numbers in each column indicate the shift amount (pixel unit) of the cross-correlation peak position of the spot shape of the left and right images. Therefore, by adding the bias value read from the example of the correlation bias table 12 to the parallax amount obtained as a result of the normal stereo correspondence point search, an error in the parallax amount due to a change in the spot shape can be corrected.

  The flow of the image processing of this embodiment described above will be described with reference to FIG. First, a reference image and a comparison image taken with a stereo camera are input (step S1). As the reference image and the comparison image, those measured as the spot shape as described above are used. Next, the parallax between the reference image and the comparison image is calculated (step S2). That is, this parallax is output by the corresponding point search unit 11 described above. Further, a bias value, which is a shift amount of the cross-correlation peak positions of the spot shapes of the left and right images, is recorded in the correlation bias table 12 (step S3). Then, the bias value read from the correlation bias table 12 is added to the parallax amount obtained in step S2 (step S4).

  It should be noted that in order to correct for individual camera differences due to lens assembly tolerances, spot shapes and parallax bias values are measured and recorded for each individual camera. However, the spot shape distribution depends only on the lens design, and all individual cameras of the same model are considered to have the same characteristics, and the measurement process of each individual is omitted, and one correlation bias table is assigned to all of the same model. It can also be used with other cameras.

[Second Embodiment]
A second embodiment of the present invention will be described with reference to FIGS. FIG. 8 is a schematic block diagram showing the image processing apparatus 50 of the second embodiment. Note that the description of the same configuration as in the first embodiment is omitted. The difference from the first embodiment is that the corresponding point search unit 51 of the second embodiment outputs a set of reference point position / corresponding point position / parallax according to the contents of the input image 1 and the input image 2. .

  The bias table of the present embodiment records a parallax correction amount corresponding to the feature point position for each screen area in advance. Here, unlike the correlation bias table 12 of the first embodiment, two one-dimensional tables that associate one bias value with one region information are used instead of a combination of two regions (first bias table 52). , Second bias table 53). The parallax bias 1 and the parallax bias 2 associated with the region including the corresponding corresponding point coordinates are output. Further, the parallax correction unit 54 corrects the parallax bias 1 and the parallax bias 2 added to the parallax output from the corresponding point search unit 51.

  The setting of the bias table of the second embodiment will be described with reference to FIG. First, as a standard spot shape, for example, an ideal symmetrical spot (standard spot shape 61) at the center of the screen simulated from a lens design value is prepared. Then, as in the first embodiment, the spot shape in each image area of the actual camera is measured, and the measured spot shape (X type 62, Y type 63) and the standard spot shape 61 are measured. The correlation bias amount is recorded in the bias table as the bias value of the image area. However, in the second bias table 53 for the input image 2, the sign is reversed and recorded. Thereby, in the parallax correction part 54 of 2nd Embodiment, the difference of the bias amount corresponding to each spot shape is added to parallax.

In the first bias table 52 and the second bias table 53 of the second embodiment, only the bias values of the standard spot shape 61 and each spot shape (X type 62, Y type 63) are used, and asymmetric spots having different shapes are used. The correlation bias amount between the X type 62 and the Y type 63 is not directly calculated. However, as shown in FIG. 9, generally, the bias value tends to increase (V ₀ > V ₁ ) as the spot shape bias increases (the Y type is more than the X type). The amount can be approximated as a difference V _0-1 (parallax bias 66) between bias values (parallax bias 65 and parallax bias 64) with the standard spot.

  Therefore, in the second embodiment, the correction amount of the parallax between the same spot shapes is 0 without using the two-dimensional correlation bias table as in the first embodiment. An effect similar to the correction of the first embodiment can be expected. The image processing flow of the second embodiment is the same as that of the first embodiment although the bias value recording method is different as described above.

  According to the present invention, since means such as table reference and bias value addition are used, parallax correction considering the spot shape can be realized with a simple configuration.

  Each of the above-described embodiments is a preferred embodiment of the present invention, and various modifications can be made without departing from the scope of the present invention.

DESCRIPTION OF SYMBOLS 1, 50 Image processing apparatus 11, 51 Corresponding point search part 12 Correlation bias table 13, 54 Parallax correction part 20 Spot shape measurement apparatus 21 Spot shape measurement camera 22 Camera turntable 52 1st bias table 53 2nd bias table

Japanese Patent No. 3261115

Claims (6)

An image processing apparatus comprising: an input unit that inputs a plurality of images captured from a plurality of viewpoints; and a parallax calculation unit that calculates a parallax between the plurality of images.
An image processing apparatus comprising: a parallax correction unit that corrects the parallax calculated by the parallax calculation unit based on a spot shape corresponding to an image position in an image.   The parallax correction unit further includes a parallax bias value determination unit that determines a parallax bias value based on a spot shape, and the parallax bias value determined by the parallax bias value determination unit is converted into a parallax calculated by the parallax calculation unit. The image processing apparatus according to claim 1, wherein addition is performed.   The image processing apparatus according to claim 2, wherein the parallax bias value determining unit divides an image plane into regions and determines a parallax bias value corresponding to a region to which a feature point belongs. An image processing method comprising: an input step for inputting a plurality of images taken from a plurality of viewpoints; and a parallax calculation step for calculating a parallax between the plurality of images,
An image processing method, comprising: a parallax correction step of correcting the parallax calculated by the parallax calculation step based on a spot shape according to an image position in the image.   The parallax correction step further includes a parallax bias value determination step for determining a parallax bias value based on a spot shape, and the parallax bias value determined by the parallax bias value determination step is changed to the parallax calculated by the parallax calculation step. The image processing method according to claim 4, wherein addition is performed.   6. The image processing method according to claim 5, wherein the parallax bias value determining step divides the image plane into regions and determines a parallax bias value corresponding to a region to which the feature point belongs.

Images