JP6555990B2 - Distance measuring device, imaging device, and distance measuring method - Google Patents

Description

  The present invention relates to a distance measuring device.

  Conventionally, a depth from focus (DFD) method as disclosed in Patent Document 1 has been proposed as a method for acquiring the distance of a shooting scene from an image acquired by an imaging apparatus. In the DFD method, a plurality of images with different blurs are acquired by controlling shooting parameters of the imaging optical system, and the size and correlation amount of each other are calculated using the measurement target pixel and its surrounding pixels in the plurality of acquired images. . Since the size of the blur and the correlation amount change according to the distance of the subject in the image, the distance is calculated using the relationship. The distance measurement by the DFD method has an advantage that it can be incorporated into a commercially available imaging device because the distance can be calculated by one imaging system.

  Patent Document 1 proposes a distance measurement method that selects and weights a local region from a captured luminance image in order to enable distance measurement with high accuracy even when the subject distance varies spatially.

In Patent Document 2, it is possible to acquire images of different shooting parameters by one shooting by changing the optical system. In addition, a proposal has been made to measure distances using luminance images of different colors depending on the distance of the subject.
In addition, as a method for acquiring the distance, there is a phase difference detection method using an image shift amount of an image by a light beam that has passed through different pupil regions.

JP 2010-016743 A Japanese Patent No. 5159986

The distance measurement by the DFD method as described in Patent Document 1 utilizes the fact that the blur size by the imaging optical system changes according to the distance to the subject, and the distance from the blur size of the captured image. Is calculated. The disclosed DFD method uses a luminance image to measure distance. However, if the luminance image is obtained from a monochrome image sensor, the luminance value of light in which various wavelength components are mixed is obtained. It is done. When the optical system has axial chromatic aberration, the distance measurement result varies because the blurring method differs for each color (wavelength) of the subject image. A similar problem occurs when using a luminance image obtained by YUV conversion from an image sensor with a Bayer array. When a luminance image (G image) of green pixels in a Bayer array is used, correct distance measurement cannot be performed when the subject image does not include a green component. Specifically, since the luminance of the G image is lowered, there is a problem that SN is deteriorated and distance measurement cannot be performed, or distance measurement accuracy is lowered.
This problem is not limited to the case where the DFD method is used, but also occurs when the phase difference detection method is used.

  In Patent Document 2, the distance measurement range is expanded by performing distance measurement for each color using an optical system that generates axial chromatic aberration, but there remains a problem that distance measurement cannot be performed depending on the subject color. .

  In view of the above problems, an object of the present invention is to perform stable and accurate distance measurement regardless of the subject color.

  In order to solve the above problems, a distance measuring device according to the present invention is a distance measuring device that calculates distance information of a subject based on a first color image and a second color image, and the two color images Distance calculation means capable of calculating the distance information of the pixel of interest in each color plane using the color plane, and evaluation value calculation means for calculating an evaluation value representing the reliability of the distance information in the pixel of interest for each color plane. Then, the distance calculating means determines which color plane to calculate the distance information of the target pixel based on the evaluation value.

  The distance measurement method according to the present invention is a distance measurement method performed by a distance measurement device that calculates distance information of a subject based on a first color image and a second color image, and includes the distance information of a target pixel. An evaluation value calculating step for calculating an evaluation value representing reliability for each color plane; a selection step for determining which color plane is used to calculate the distance information of the target pixel based on the evaluation value; A distance calculating step of calculating distance information of the target pixel according to the determination.

  According to the distance measurement method of the present invention, stable and accurate distance measurement can be performed regardless of the subject color.

1 is a diagram illustrating a configuration of an imaging apparatus according to a first embodiment. The flowchart which shows the flow of the distance measurement process in 1st embodiment. The flowchart which shows the flow of the distance map production | generation process in 1st embodiment. The figure explaining the difference in the correlation value for every color plane. The flowchart which shows the flow of the distance integration process in 1st embodiment. The figure which shows the flow of the distance integration process in 2nd embodiment. The figure which shows the flow of the distance integration process in 3rd embodiment. The flowchart which shows the flow of the distance map production | generation process in 4th embodiment. The figure explaining the image pick-up element in 4th embodiment. The flowchart which shows the flow of the distance measurement process in 4th embodiment.

(First embodiment)
<System configuration>
FIG. 1 is a system configuration diagram of an imaging apparatus according to the first embodiment of the present invention. The imaging apparatus 1 includes an imaging optical system 10, an imaging element 11, a control unit 12, a signal processing unit 13, a distance measurement unit 14, a memory 15, an input unit 16, a display unit 17, and a storage unit 18.

  The imaging optical system 10 is an optical system that includes a plurality of lenses and forms incident light on the image plane of the imaging element 11. As the imaging optical system 10, a variable focus optical system is used, and automatic focusing is possible by the autofocus function of the control unit 12. The auto focus method may be either a passive method or an active method.

The image sensor 11 is an image sensor having a CCD, a CMOS, or the like, and acquires a color image. The image sensor 11 may be an image sensor having a color filter, or may be a three-plate image sensor having a different color. The image sensor 11 of the present embodiment acquires RGB color images of three colors, but may be an image sensor that acquires three or more color images including other than visible light.

  The control unit 12 is a functional unit that controls each unit of the imaging apparatus 1. Functions of the control unit 12 include, for example, automatic focusing by autofocus (AF), change of focus position, change of F value (aperture), image capture, control of shutter and flash (both not shown), input There are control of the unit 16, the display unit 17, and the storage unit 18.

  The signal processing unit 13 is a functional unit that performs processing on a signal output from the image sensor 11. Specific functions of the signal processing unit 13 include analog signal A / D conversion, noise removal, demosaicing, luminance signal conversion, aberration correction, white balance adjustment, color correction, and the like. The digital image data output from the signal processing unit 13 is temporarily stored in the memory 15, and then displayed on the display unit 17, recorded (stored) in the storage unit 18, and output to the distance measurement unit 14. The desired process is performed.

  The distance measuring unit 14 is a functional unit that calculates the distance in the depth direction to an object in the image. The distance measurement unit 14 includes an evaluation value calculation unit 141 and a distance calculation unit 142. The evaluation value calculation unit 141 has a function of obtaining an evaluation value representing the reliability of distance calculation for each color plane of the color image. The distance calculation unit 142 has a function of calculating a distance from each color plane of the color image, and a function of calculating a final distance based on the distance and evaluation value for each color plane. Detailed operation of the distance measuring unit 14 will be described later.

  The input unit 16 is an interface that is operated by the user to input information and change settings on the imaging apparatus 1. For example, dials, buttons, switches, touch panels, etc. can be used.

  The display unit 17 is a display unit configured by a liquid crystal display, an organic EL display, or the like. The display unit 17 is used for composition confirmation at the time of photographing, browsing of photographed / recorded images, display of various setting screens and message information, and the like.

  The storage unit 18 is a non-volatile storage medium that stores captured image data, parameter data used by the imaging apparatus 1, and the like. The storage unit 18 is preferably a high-capacity storage medium that can read and write at high speed. For example, a flash memory can be suitably used.

<Subject distance measurement method>
Next, distance measurement processing performed by the imaging apparatus 1 will be described in detail with reference to FIG. 2 which is a flowchart showing the flow of processing.

  When the user operates the input unit 16 to instruct the execution of distance measurement and starts shooting, shooting is performed after auto focus (AF) or automatic exposure control (AE) is performed, and the focus position and aperture (F Number) is determined (step S11). Thereafter, shooting is performed in step S <b> 12, and an image is captured from the image sensor 11.

  When the first shooting is completed, the control unit 12 changes the shooting parameters (step S13). The imaging parameter to be changed is at least one of an aperture (F number), a focus position, and a focal length. As the parameter value, a value stored in advance may be read and used, or a value determined based on information input by the user may be used. When the change of the shooting parameter is completed, the process proceeds to step S14, and the second shooting is performed.

  In this embodiment, the focus position is changed and a second image is taken. For example, the first image is shot so as to focus on the main subject, and the second image is shot while changing the focus position so that the main subject is blurred.

  When shooting a plurality of images, the higher the shutter speed and the shorter the shooting interval, the less the influence of camera shake and subject blur. Therefore, in order to perform distance measurement with higher accuracy, it is desirable to increase the shutter speed and shorten the shooting interval. However, if the sensitivity is increased in order to increase the shutter speed, the influence of noise increases more than the influence of camera shake in some cases, so it is necessary to set an appropriate shutter speed in consideration of the sensitivity.

  When two images are photographed, the photographed images are each processed by the signal processing unit 13 so as to become an image suitable for distance measurement, and are temporarily stored in the memory 15. Specifically, development processing is performed, but it is necessary to avoid edge enhancement processing or the like so that blur does not change due to signal processing. In the case of a color image, the image used for the subsequent processing is an RGB image generated by demosaicing or selecting only a specific color from the Bayer array. The R image, the G image, and the B image are color plane images in the present embodiment. In addition, at least one of the two captured images may be signal-processed as an ornamental image and stored in the memory 15.

  In step S15, the distance measuring unit 14 receives as input two color images for distance measurement stored in the memory 15, and calculates a distance map from these color images. A distance map is data representing the distribution of subject distances in an image. The subject distance may be expressed as an absolute distance from the imaging device to the subject, or may be expressed as a relative distance from the focus position to the subject. The subject distance may be represented by an object distance or an image distance. Furthermore, a value that can be converted into a distance, such as the size of the blur or the amount of correlation, may be used as the subject distance (distance information). The calculated subject distance distribution (distance map) is displayed through the display unit 17 and stored in the recording unit 19.

  Next, the process (hereinafter, distance map generation process) performed by the distance measurement unit 14 in step S15 will be described in more detail. FIG. 3 is a flowchart showing the flow of the distance map generation step S15 in the present embodiment.

  When the two color images are received, the distance measuring unit 14 selects a color plane image of the same color from the two color images in a color plane selection step S21. Since the processing of steps S22 to S25 is executed for all color planes, the order of selecting color planes in step S21 is not particularly limited.

  Next, in the band limiting step S22, the distance calculation unit 142 executes a filtering process so as to extract a desired spatial frequency band for the two generated color plane images, and the band limited color plane image is obtained. Generate. Since the change in blur differs depending on the spatial frequency, in the band limiting step S22, only the frequency band of interest is extracted so that stable distance measurement can be performed. The extraction of the spatial frequency band may be performed after being converted into a frequency space, or may be performed by filtering, and the method is not limited. Since the influence of noise is large in the high frequency band, it is preferable to pass the low frequency to the middle frequency.

In the region selection step S23, the distance calculation unit 142 selects the target pixel at the same coordinate position and the surrounding local region in the two band-limited color plane images. It is possible to calculate the distance image (distance map) of the entire input image by setting the pixel of interest and the local region over the entire image by shifting one pixel at a time and performing the following processing. The distance map does not necessarily have the same number of pixels as the input image, and may be calculated for every several pixels of the input image. The local area may be set for one or more areas specified in advance, or the range may be specified by the user using the input unit 16.

ここで、ＮＣＣ_ｊは注目画素ｊを中心とする局所領域における相関値を表し、Ｉ_１，ｉは一方のカラープレーン画像における画素位置ｉでの画素値を表し、Ｉ_２，ｉは他方のカラープレーン画像における画素位置ｉでの画素値を表わす。総和は注目画素ｊを中心とする局所領域を対象として行われる。また、バーのついたＩ_１およびＩ_２は、それぞれのカラープレーン画像における局所領域内での画素値の平均値である。 Next, in the correlation calculation step S24, the distance calculation unit 142 calculates the correlation between the local region of the first image selected in the region selection step S23 and the local region of the second image by the equation (1).

Here, NCC _j represents the correlation value in the local region centered on the pixel of interest j, I _{1, i} represents the pixel value at the pixel position i in one color plane image, and I _{2, i} represents the other color. The pixel value at the pixel position i in the plain image is represented. The summation is performed on a local region centered on the pixel of interest j. Further, I ₁ and I ₂ with a bar are average values of pixel values in a local region in each color plane image.

  If there is no aberration and the defocus is the same before and after the focus, the blur is equivalent between the two focus positions taken by moving the focus on the image side, and the correlation at that position is the highest value. . As the distance from the intermediate position increases, the blurring of the two images changes and the correlation decreases. That is, the correlation decreases as the blur becomes the peak at the same position and moves away from that position. Since the correlation value is a value corresponding to the blur due to defocus, if the correlation value is known, the corresponding defocus amount can be known, and the relative distance can be calculated.

  Next, in the distance conversion step S25, the distance calculation unit 142 converts the correlation value calculated in the correlation calculation step S24 into an image distance. The relationship between the correlation value and the image distance is shown in FIG. The inclination of the defocus characteristic of the correlation value and the position where the correlation value is maximized differ depending on the color plane. The tilt is due to the difference in depth depending on the wavelength, and the shift of the focus position is due to axial chromatic aberration. Therefore, the distance calculation unit 142 holds different conversion tables or conversion formulas for converting the correlation value into the image distance for each color plane, and converts the image distance to the image distance by different conversion for each color plane. The conversion table or the conversion formula is obtained in advance by measurement or by simulation using optical design information. In the case of simulation, the conversion table and conversion formula for each color plane apply the result of simulation using the wavelength of the center wavelength and the wavelength with the highest transmittance for each color filter as the representative wavelength. In obtaining the representative wavelength, the centroid wavelength may be obtained from the wavelength in the color filter and the transmittance thereof and used. Although different correlation values are calculated for each color plane even for objects at the same distance, an image distance independent of the color plane is obtained by the distance conversion step S25.

  In the present embodiment, Formula 1 has been described as an example of the calculation method, but the formula is not limited to this formula as long as the formula can determine the blur relationship between the two color planes. If the relationship between the output value according to the calculation and the focus position on the image plane is known, conversion to a relative distance is possible.

As another calculation example, the following formula 2 and the like can be given.

ここでＦはフーリエ変換を表し、ＯＴＦは光学伝達関数、Ｓは撮影シーンのフーリエ変換結果を表している。数式３では、２つの撮影条件における光学伝達関数の比が得られ、予め光学系の設計データからこの値のデフォーカスによる変化を知ることができるため相対距離への変換が可能となる。 Moreover, Formula 3 etc. are mentioned as distance calculation calculation by performing a Fourier transform and evaluating in frequency space.

Here, F represents the Fourier transform, OTF represents the optical transfer function, and S represents the Fourier transform result of the shooting scene. In Formula 3, the ratio of the optical transfer function under the two imaging conditions can be obtained, and the change due to defocusing of this value can be known in advance from the design data of the optical system, so that it can be converted into a relative distance.

  When the distance conversion step S25 ends, the process returns to the region selection step S23, and the above-described processing is repeated for the unprocessed target pixel. The area selection step S23 and the correlation calculation step S24 are repeatedly executed until the calculation for the designated area or the entire image is completed. This iterative process can generate a distance map for one color plane. When the distance conversion step S25 is completed for all local regions, the process proceeds to an end determination step S26.

  In the end determination step S26, the distance calculation unit 142 determines whether the processing has been completed for all the color planes included in the color image. If there is a color plane that has not been processed, the process returns to the color plane selection step S21 to repeat the processing up to the distance conversion step S25. When the generation of the distance map is completed for all the color planes, the process proceeds to the distance integration step S27.

  In the distance integration step S27, the distance measurement unit 14 generates a final distance map by selecting the color plane distance with high reliability for each pixel from the distance maps of the plurality of color planes calculated in the distance conversion step S25. To do. Details of the distance integration step S27 are shown in the flowchart of FIG. 5A.

  The evaluation value calculation unit 141 calculates, for each color plane, an evaluation value that represents the reliability of the distance information at the target pixel in the distance map (S31). In the present embodiment, as an evaluation value, a luminance value for each color plane before correcting the transmittance of the color filter is used. The color plane image from which the evaluation value is calculated may be a color plane image for any one of the two color images, or a color image obtained by synthesizing the two color images. May be a color plane image.

  The distance calculation unit 142 selects the distance in the color plane having the highest evaluation value as the distance in the target pixel (S32). This is because the use of a color plane having a high luminance value provides a highly accurate ranging result with good SN.

  By performing the above-described evaluation value calculation step S31 and distance selection step S32 for all the pixels of the distance map, one distance map in which the distance maps calculated for each color plane are integrated is generated.

  In the flowchart of FIG. 5A, the processing of the evaluation value calculation step S31 and the distance selection step S32 is performed for each pixel. However, after the evaluation value calculation processing is performed for all pixels, the distance selection step S32 is performed for all pixels. May be. The timing for obtaining the evaluation value may be any timing as long as it is before the distance selection step S32.

  FIG. 5B is a diagram illustrating a data flow in the distance map generation step S15 of the present embodiment. A distance map including distance information for each pixel is generated for each color plane by the distance calculation unit 142 from a color image including a plurality of color planes (S21 to S26). Further, an evaluation value representing the reliability of the distance information for each pixel is calculated for each color plane by the evaluation value calculation unit 103 from a color image composed of a plurality of color planes (S31). Then, the distance calculation unit 142 determines, for each pixel, a final distance map to be generated using distance information obtained from which color plane based on the evaluation value (S32).

  Here, another method for calculating the evaluation value will be described. As the evaluation value, an average value of luminance values (before correction of the transmittance of the color filter) in the peripheral region centered on the target pixel can be used. Here, the size of the peripheral region may be the same as or different from the local region used in the distance calculation (selected in step S23). In addition, as the evaluation value, it is also possible to use the luminance value of the target pixel and the average luminance value in the peripheral region of the target pixel after correcting the transmittance of the color filter or correcting the white balance. As yet another example, the number of pixels that satisfy the condition that the luminance value is equal to or greater than a threshold value in the peripheral region of the band-limited color plane image may be used. The threshold value here is preferably a luminance value that enables reliable distance calculation when the luminance value is more than that.

  The distance information included in the distance map output by the distance measuring unit 14 may be a relative distance from the focus position, or an object distance obtained by converting the obtained relative distance using the focal distance and the object-side focus distance. It may be.

  In this embodiment, distance information is calculated from a plurality of color planes, and distance information of the color plane that is most reliable (having the highest evaluation value) is selected. In the conventional technique that calculates the distance based on a color plane of a specific color, accurate distance measurement cannot be performed in an area where the color component is weak, but in the present embodiment, accurate distance measurement is not performed regardless of the color of the subject image. Good distance measurement is possible. In addition, with the conventional technique that calculates the distance using a luminance image generated from a plurality of color images, a luminance value in which a plurality of colors are mixed is obtained, and the correlation value is different depending on the color due to axial chromatic aberration. Distance measurement cannot be performed accurately. However, since distance measurement is performed for each color plane in the present embodiment, accurate distance measurement is possible even when the optical system has axial chromatic aberration. Therefore, according to the present embodiment, it is possible to improve robustness and distance measurement accuracy with respect to the subject color.

(Second embodiment)
Next, a second embodiment of the present invention will be described. The second embodiment is different from the first embodiment in the method (S27) for integrating the distances calculated for each color plane. Since the configuration of the imaging apparatus 1 is the same as that of the first embodiment, description will be made using the same reference numerals. Hereinafter, differences in processing from the first embodiment will be mainly described. FIG. 6 is a flowchart showing the flow of the distance integration step S27 in the second embodiment.

  The evaluation value calculation step S31 is the same as that in the first embodiment. Various evaluation values described in the first embodiment can be adopted as evaluation values. Hereinafter, a case where the luminance value for each color plane before correction of the transmittance of the color filter is used as the evaluation value will be described as an example.

ここで、Ｗ_Ｒ，Ｗ_Ｇ、Ｗ_Ｂはそれぞれ各カラープレーンに対する重みであり、Ｒ，Ｇ，Ｂは各カラープレーンにおける輝度値である。添字のｉは注目画素を表す。 In the present embodiment, the distance at the target pixel is not determined as a distance calculated from one color plane, but as a weighted average value of distances calculated from a plurality of color planes. The weight for the distance for each color plane in the weighted average is determined according to the evaluation value for each color plane. In the weighted average distance calculation step S33, the distance calculation unit 142 first calculates the weight for the weighted average using the evaluation value (in this case, the luminance value) for each color plane in the target pixel. When the color image is an RGB image, the weight W of each color plane is calculated as a luminance value ratio as shown in Equation 4.

_{Here, W R, W G, W} B is a weight for each of the color planes, respectively, R, G, B is the intensity value in each color plane. The subscript i represents the pixel of interest.

  The distance calculation unit 142 performs a weighted average on the distance map of each color plane using the calculated weight. The distance calculation unit 142 determines the weighted average value of the distances calculated in this way as final distance information in the target pixel. The distance calculation unit 142 executes the integration process using the above weighted average for all the pixels of the distance map, and outputs the integrated distance map.

  In the above example, the weighted average processing has been described as an example where the luminance value is adopted as the evaluation value. However, it is clear that the weighted average processing can be performed by the same method even when the evaluation value is other than the luminance value. Will. Specifically, R, G, and B in Formula 2 may be read as evaluation values for each color plane, and a weighted average with a weight of the evaluation value ratio as a final distance may be used.

  If the distance information calculated in the distance conversion step S25 is an image distance and is a relative distance based on the focus position, the distance information used for the weighted average may be determined in consideration of the sign of the distance value. Good. More specifically, in step S33, the distance calculation unit 142 calculates only the distance information having a large number of the same relative distance codes based on the focus position among the distance information obtained using each color plane. To calculate a weighted average distance value. For example, when the color plane is three colors of RGB, the relative distance obtained from the R and G color planes is positive, and when the relative distance obtained from the B color plane is negative, the R and G colors A weighted average distance value is calculated using only distance information obtained from the plane. The weight at this time is determined based on the evaluation value of the color plane used for the weighted average. The same applies when the number of color planes is greater than three.

  By performing the weighted average taking into account the sign of the relative distance in this way, if there is a difference in the sign due to the error included in the distance calculated in the vicinity of the focus, mixing of errors when integrating the distance with the weighted average is performed. Can be prevented. In addition, when a sign inversion occurs due to noise or the like at a distance greatly deviating from the focus, the accuracy is greatly reduced due to the inclusion of different codes during distance integration. Thus, by considering the sign when calculating the evaluation value, it is possible to prevent a decrease in accuracy during distance integration due to the weighted average.

  If the number of color planes is an even number and the number of color planes giving the distance value of either positive or negative sign is the same, the following processing can be performed. For example, it is conceivable to select either positive or negative sign as appropriate and take a weighted average using only the distance information of the selected sign. At this time, the relative distance of the code having the larger average value or maximum value of the evaluation values may be selected. In any case, it is also preferable to take measures such as adding data that the reliability of the calculated distance is low.

  According to the present embodiment, a distance step caused by an error caused by a distance conversion error for each color plane can be reduced by using a weighted average at a boundary where the color planes selected by the distance integration process are different. In addition, when an image distance sign error caused by noise or a distance conversion error occurs, it is possible to reduce distance degradation during integration by performing weighted averaging using only the distance of the same sign.

(Third embodiment)
A third embodiment of the present invention will be described. The third embodiment is different from the first embodiment and the second embodiment in that the correlation value obtained when calculating the distance is considered in the method of integrating the distance calculated for each color plane (S27). To do. Since the configuration of the imaging device 1 in the third embodiment is the same as that of the first embodiment and the second embodiment, description will be made using the same reference numerals. Hereinafter, differences in processing between the first embodiment and the second embodiment will be mainly described.

  7A and 7B are flowcharts showing the flow of the distance integration step S27 in the present embodiment. FIG. 7A shows an example of processing based on the first embodiment, and FIG. 7B shows an example of processing based on the second embodiment.

  In this embodiment, the color plane distance information having a high correlation value (see step S24 in FIG. 3 and Formula 1) when calculating the distance for each color plane is not used for the distance integration processing. The correlation value is high in the region where the blur is about the same, but the decrease in the correlation value due to noise is large, and the error in the distance obtained according to the distance conversion table or conversion equation stored in advance may be large. . Since the distance measurement error increases when the correlation value is high in this way, the distance calculation can be improved by not using the distance obtained from the high correlation value for the distance integration.

  Here, if axial chromatic aberration remains in the optical system, the subject distance that is highly correlated differs depending on the color plane. Therefore, even if one color plane has a high correlation, the other color planes do not have a high correlation. Therefore, at least one of the color planes has a reliable correlation value and can be used for distance measurement.

  In the high correlation determination step S41, the evaluation value calculation unit 141 determines whether the correlation value calculated for each color plane is larger than a predetermined threshold value. This threshold is a value estimated that the distance calculation error becomes larger due to the influence of noise if the correlation value is higher than that. The evaluation value calculation unit 141 sets the evaluation value to 0 (minimum value) for the color plane for which the correlation value is determined to be larger than the predetermined threshold value in the evaluation value calculation step S31. And the distance calculation part 142 implements distance selection step S32 and weighted average distance calculation step S33 using the evaluation value determined in this way. The processing in these steps is the same as in the first embodiment and the second embodiment. In this embodiment, since the evaluation value is determined as described above, the distance information at the target pixel can be calculated using the distance information obtained from the color plane whose correlation value is equal to or less than the threshold value.

  The evaluation value itself is calculated in the same manner as in the first embodiment and the second embodiment, and it is the same even if the distance information whose correlation value is larger than the threshold is not used in the distance selection step S32 or the weighted average distance value calculation step S33. The effect is obtained.

According to this embodiment, distance information can be calculated with high accuracy even in a highly correlated region. As described above, when the optical system has axial chromatic aberration, even if the correlation is high in one color plane, the correlation is not high in another color plane. In the present embodiment, since the final distance map is generated based on information obtained from a color plane in which the distance measurement performance is less deteriorated due to noise, a decrease in accuracy can be avoided.

(Fourth embodiment)
In the fourth embodiment of the present invention, a color plane to be used for distance measurement is determined for each pixel, and distance calculation is performed only for the color plane to be used. Compared to the first embodiment, it is different in that distance calculation is not performed for all color planes.

  The configuration of the imaging device 1 in the fourth embodiment is the same as that in the first embodiment. Hereinafter, differences in processing from the first embodiment will be described. FIG. 8 is a flowchart showing the flow of the distance map generation step S15 in the fourth embodiment.

  When a color image is input to the distance measuring unit 14, the evaluation value calculating unit 141 executes an evaluation value calculating step S51. The evaluation value calculation step S51 is a process of calculating an evaluation value for each color plane and for each pixel, as in the evaluation value calculation step S31 (FIG. 5A) of the first embodiment. As specific evaluation values, various values described in the first embodiment such as luminance values in each color plane can be adopted.

  Next, the distance calculation unit 142 performs a band limiting step S52 and a region selecting step S53 on the input color image. These processes are the same as the band limiting step S22 and the area selecting step S23 of the first embodiment.

  In the color plane selection step S54, the distance calculation unit 142 selects which color plane is used for distance measurement in the target pixel based on the evaluation value calculated in the evaluation value calculation step S31. Specifically, the distance calculation unit 142 determines to use the color plane having the highest evaluation value for the pixel of interest for distance measurement.

  The distance calculation unit 142 executes a correlation calculation step S55 and a distance conversion step S56 for the selected color plane. Since these processes are the same as the correlation calculation step S24 and the distance conversion step S25 of the first embodiment, the description thereof is omitted. In order to obtain the same effect as in the third embodiment, when the correlation value calculated in the correlation calculation step S55 is larger than a predetermined threshold value, the correlation value calculation is performed using the color plane having the next highest evaluation value. Can be redone.

  In the end determination step S57, it is determined whether or not the processing has been completed for all the pixels of the distance map. Until the processing of all the pixels is completed, the processing from the region selection step S53 to the distance conversion step S56 is repeated. When the distance calculation for all pixels is completed, the process ends and outputs a distance map.

  It should be noted that the same distance map generation can be performed even if the processing order of the area selection and the color plane selection is changed. Specifically, after selecting a color plane, distance information may be calculated using the color plane for a pixel having the highest evaluation value in the color plane. By executing this process for all color planes, the same effect as described above can be obtained.

The distance measuring unit 14 according to the present embodiment can calculate distance information from any color plane, but does not calculate distances for all the pixels for each color plane, and calculates distances only for necessary color planes. Can be reduced. Furthermore, the amount of calculation can be reduced even in the point that the integration processing of a plurality of distances is not necessary. Therefore, it is possible to generate a distance map that is robust to the color of the subject with a calculation amount equivalent to the calculation amount for calculating the distance map of one color plane.

(Fifth embodiment)
In the fifth embodiment of the present invention, a case where distance measurement is performed using a phase difference method will be described. Here, first, the imaging element 11 included in the imaging apparatus 1 according to the fifth embodiment of the present invention will be described. FIG. 9 is a diagram illustrating the image sensor. In FIG. 9A, the pixel group 201 and the pixel group 202 are each composed of four pixels of 2 rows × 2 columns. . In the pixel group 201, a green pixel 201Gv is arranged in the diagonal direction, and a red pixel 201Rv and a blue pixel 201Bv are arranged in the other diagonal direction. Similarly, in the pixel group 202, a green pixel 202Gh is arranged diagonally, and a red pixel 202Rh and a blue pixel 202Bh are arranged on the other side. These two types of pixel groups are arranged on the image sensor 11 in a checkered pattern.

  FIG. 9B is a schematic cross-sectional view taken along V-V ′ in the pixel group 201. 201ML is a micro lens, 201CF is a color filter, and 201A and 201B are photoelectric conversion units. FIG. 9C shows an H-H ′ schematic cross-sectional view of the pixel group 202. 202ML is a microlens, 202CF is a color filter, and 202A and 202B are photoelectric conversion units. 9B and 9C, reference numeral 210 denotes a light receiving surface, which is an xy surface (+ z side surface) on the light incident side of the photoelectric conversion unit. FIG. 9B shows the structure of the green pixel 201Gv and the blue pixel Bv, but the red pixel 201Rv also has the same structure. 9C shows the structure of the green pixel 202Gh and the blue pixel 202Bh, the red pixel 202Rh also has the same structure.

  In the imaging device 11 of this modification, two photoelectric conversion units are arranged in one pixel (each pixel such as a green pixel), and the microlens 201ML is arranged so that the light receiving surface 210 and the exit pupil are optically conjugate. , 202ML power is set. With such an arrangement, the photoelectric conversion units 201A and 201B (and 202A and 202B) can receive light beams that have passed through areas EPA and EPB having different exit pupils, respectively. In the pixel group 201 and the pixel group 202, the arrangement of the two photoelectric conversion units is rotated by 90 °. With such a configuration, in the pixel group 201 and the pixel group 202, a light beam that has passed through an area obtained by dividing the exit pupil in different directions is received by the photoelectric conversion unit. As a result, the subject distance can be detected for subjects whose contrast changes in different directions between the pixel group 201 and the pixel group 202.

  Image signals generated from light received by two photoelectric conversion units in each pixel group are acquired as an A image and a B image, respectively. The A image and the B image correspond to the first color image and the second color image.

  Next, a method for measuring the distance from the generated A image and B image by the phase difference method will be described. FIG. 10 is a flowchart showing a flow of distance measurement processing by the phase difference method.

ここで、Ａ（ｉ）はＡ像の像信号、Ｂ（ｉ）はＢ像の像信号、ｉは画素番号、ｋはＡ像とＢ像の相対シフト量である。ｍは、相関値Ｃ（ｋ）の演算に用いる対象画素範囲を表して
いる。 In the phase difference method, it is possible to detect the defocus amount Def of the subject image by detecting the image shift amount r between the A image and the B image. The detection of the image shift amount r is obtained by calculating the correlation between the A image and the B image (step S121). For the correlation calculation, a known method can be used. For example, the correlation value C (k) is calculated by Equation 5, and the image shift amount r can be calculated from k where C (k) = 0.

Here, A (i) is an image signal of A image, B (i) is an image signal of B image, i is a pixel number, and k is a relative shift amount between the A image and the B image. m represents the target pixel range used for the calculation of the correlation value C (k).

ここでｗは基線長（射出瞳ＥＰＡと射出瞳ＥＰＢの中心間隔）であり、Ｚは撮像素子から射出瞳までの距離である。 Next, the defocus amount Def is calculated by substituting the calculated image shift amount r into Equation 6 (step S122).

Here, w is a base line length (a center distance between the exit pupil EPA and the exit pupil EPB), and Z is a distance from the image sensor to the exit pupil.

  A distance image based on the defocus amount can be generated by performing the same process on the entire surfaces of the A image and the B image used for the distance measurement. In addition, the calculated defocus amount can be converted into a distance to the subject using a parameter at the time of shooting, and a distance image can be generated. The A image and the B image used for distance measurement may be obtained from either the pixel group 201 or the pixel group 202. Alternatively, distance measurement is performed for each of the A image and the B image obtained from the pixel group 201 and the A image and the B image obtained from the pixel group 202, and a distance with higher reliability of distance measurement is selected at each position. A distance image may be generated.

  As the reliability of distance measurement in the present embodiment, a luminance value for each color plane, the presence / absence of texture (the higher the texture, the higher the reliability), or a combination thereof can be used.

  By executing the above distance measurement method for each RGB color plane, the distance can be calculated for each color plane. The distance information of each color plane corrects the deviation of the calculated distance information due to the difference in focal position for each color plane. This correction is performed using lens design data. The selection and integration of distances can be performed in the same manner as the method shown in the first to fourth embodiments.

  In the above description, an example in which the imaging element 11 has the same number of pixel groups 201 and pixel groups 202 having different pupil division directions is shown. However, the number of pixel groups 201 and 202 may be different. Only one of the pixel groups 202 may be employed. In the above description, an example in which a microlens is used to collect light incident on a pixel on a photoelectric conversion unit has been described. However, incident light may be collected using a waveguide instead of the microlens. Or you may condense incident light using both a micro lens and a waveguide.

(Other examples)
The description of each embodiment is an exemplification for explaining the present invention, and the present invention can be implemented with appropriate modifications or combinations without departing from the spirit of the invention. For example, the present invention can be implemented as an imaging apparatus that includes at least a part of the above processing, or can be implemented as a distance measuring apparatus that does not include an imaging unit. Moreover, it can also implement as a distance measurement method, and can also be implemented as an image processing program which makes a distance measurement apparatus perform the said distance measurement method. Each elemental technology such as processing and means described in each embodiment can be freely combined and implemented as long as no technical contradiction occurs.

  In the description of the embodiment, an example in which the imaging apparatus acquires two images has been described. However, three or more images may be acquired. In this case, two images are selected from the captured images and distance measurement is performed. By acquiring three or more images, it is possible to obtain an effect that the range in which the distance can be measured is widened and an effect that the distance accuracy is improved.

  The above-described distance measurement technique of the present invention can be preferably applied to, for example, an imaging apparatus such as a digital camera or a digital camcorder, or an image processing apparatus or computer that performs image processing on image data obtained by the imaging apparatus. Further, the present invention can also be applied to various electronic devices (including mobile phones, smartphones, slate terminals, and personal computers) incorporating such an imaging device or image processing device.

  In the description of the embodiment, the configuration in which the distance measurement function is incorporated in the imaging apparatus main body is shown. However, the distance measurement may be performed by other than the imaging apparatus. For example, a distance measurement function may be incorporated into a computer having an imaging device, and the computer may acquire an image captured by the imaging device and calculate the distance. Further, a distance measurement function may be incorporated into a computer that can be accessed via a network by wire or wireless, and the computer may acquire a plurality of images via the network and perform distance measurement.

  The obtained distance information can be used, for example, for various image processing such as image segmentation, generation of stereoscopic images and depth images, and emulation of blur effects.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  For this purpose, the program is stored in the computer from, for example, various types of recording media that can serve as the storage device (ie, computer-readable recording media that holds data non-temporarily). Provided to. Therefore, the computer (including devices such as CPU and MPU), the method, the program (including program code and program product), and the computer-readable recording medium that holds the program non-temporarily are all included in the present invention. Included in the category.

14 Distance measurement unit 141 Correlation value calculation unit 142 Distance calculation unit

Claims (23)

A distance measuring device that calculates distance information of a subject based on a first color image and a second color image,
Distance calculation means capable of calculating distance information of each pixel of interest in each color plane using the two color images;
An evaluation value calculating means for calculating an evaluation value representing the reliability of the distance information in the target pixel for each color plane;
Have
The distance calculation unit determines, based on the evaluation value, which color plane is used to calculate the distance information of the target pixel.
Distance measuring device. The evaluation value calculation means calculates the luminance value at the target pixel in the color plane of the image obtained by combining any one of the two color images or an image obtained by combining the evaluation value of the color plane at the target pixel. Or, it is calculated as an average value of luminance values in the peripheral area of the target pixel.
The distance measuring device according to claim 1. The evaluation value calculation means calculates the evaluation value of the color plane in the target pixel within one of the two color images or in the peripheral region of the target pixel in the color plane of the image obtained by combining them. Is calculated as the number of pixels that satisfy the condition that the luminance value is equal to or greater than a predetermined threshold.
The distance measuring device according to claim 1. The distance calculating means calculates distance information of the pixel of interest using a color plane having the highest evaluation value;
The distance measuring device according to any one of claims 1 to 3. The distance calculating means calculates a plurality of distance information using each color plane for each of a plurality of pixels, and calculates the distance information obtained using the color plane having the highest evaluation value for each pixel. Select as information,
The distance measuring device according to any one of claims 1 to 4. The distance calculation means selects a color plane having the highest evaluation value for each pixel, and calculates distance information using the selected color plane.
The distance measuring device according to any one of claims 1 to 4. The distance calculation means calculates a weighted average of distance information obtained using each color plane according to the evaluation value as distance information of the target pixel.
The distance measuring device according to any one of claims 1 to 3. The distance calculation means performs a weighted average using only distance information having a large number of the same signs of distances based on the focus position among the distance information obtained using each color plane.
The distance measuring device according to claim 7. The distance calculation means calculates a correlation value between one color plane of two color images, and calculates distance information from the correlation value by a conversion table or conversion formula that differs for each color plane.
The distance measuring device according to any one of claims 1 to 8. The distance calculation means calculates distance information of the pixel of interest using a color plane whose correlation value is equal to or less than a threshold value.
The distance measuring device according to claim 9. The distance calculating means calculates distance information from a difference in phase difference in each color plane of two color images.
The distance measuring device according to any one of claims 1 to 8. An imaging optical system;
An image sensor for acquiring a color image composed of a plurality of color planes;
The distance measuring device according to any one of claims 1 to 11,
An imaging device. A distance measurement method performed by a distance measurement device that calculates distance information of a subject based on a first color image and a second color image,
An evaluation value calculating step for calculating an evaluation value representing the reliability of the distance information in the target pixel for each color plane;
A distance calculation step of calculating distance information of the pixel of interest using distance information obtained from a color plane determined based on the evaluation value;
Including distance measurement method. In the evaluation value calculating step, the evaluation value of the color plane in the target pixel is the luminance value in the target pixel in the color plane of the image obtained by combining any one of the two color images or the two color images. Or, it is calculated as an average value of luminance values in the peripheral area of the target pixel.
The distance measuring method according to claim 13. In the evaluation value calculating step, the evaluation value of the color plane in the target pixel is the peripheral region of the target pixel in the color plane of the image obtained by combining any one of the two color images or a combination thereof. The luminance value within is calculated as the number of pixels equal to or greater than a predetermined threshold.
The distance measuring method according to claim 13. In the distance calculation step, distance information of the target pixel is calculated using a color plane having the highest evaluation value.
The distance measuring method according to any one of claims 13 to 15. In the evaluation value calculating step, the evaluation values are calculated for a plurality of pixels,
The distance calculating step includes:
For each of a plurality of pixels, a step of calculating a plurality of distance information using each color plane, and for each pixel, distance information obtained using a color plane determined based on the evaluation value is distance information of the pixel. Step to select as,
The distance measuring method according to claim 13, comprising: In the evaluation value calculating step, the evaluation values are calculated for a plurality of pixels,
In the distance calculating step, for each pixel, the distance information of the pixel is calculated using a color plane determined based on the evaluation value.
The distance measuring method according to any one of claims 13 to 16. In the distance calculating step, a value obtained by weighted averaging distance information obtained using each color plane according to the evaluation value is calculated as distance information of the target pixel.
The distance measuring method according to any one of claims 13 to 15. In the distance calculation step, among the distance information obtained using each color plane, the weighted average is performed using only the distance information having the same number of distances with the same distance code as the reference.
The distance measuring method according to claim 19. In the distance calculating step, a correlation value between one color plane of two color images is calculated, and distance information is calculated from the correlation value by a conversion table or a conversion formula that is different for each color plane.
The distance measuring method according to any one of claims 13 to 20. In the distance calculating step, the distance information of the target pixel is calculated using a color plane whose correlation value is equal to or less than a threshold value.
The distance measurement method according to claim 21.   The program for making a computer perform each step of the method of any one of Claim 13 to 22.

Images