JP2012022716A - Apparatus, method and program for processing three-dimensional image, and three-dimensional imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To absorb dispersion in field angles between a plurality of images composing a parallax image by geometrical deformation while leaving a field angle deviation due to parallax to be essentially generated between the images.SOLUTION: Corresponding points of a plurality of groups in which features are approximately coincident between a predetermined reference image and a target image other than the reference image out of a plurality of images obtained by photographing the same object from a plurality of visual points are detected, and a geometrical deformation parameter for geometrically deforming the target image is estimated so that only y coordinate values (coordinate values of a direction perpendicular to a parallax direction) may coincide with each other out of the xy coordinate values of one corresponding point in the detected corresponding points of the plurality of groups and the xy coordinate values of the other corresponding point. By geometrically deforming the target image on the basis of the geometrical deformation parameter estimated in this way, a field angle deviation in the direction perpendicular to the parallax direction can be corrected while leaving the field angle deviation due to the parallax to be essentially generated between the reference image and the target image.

Description

  The present invention relates to a stereoscopic image processing apparatus, method and program, and a stereoscopic imaging apparatus, and in particular, geometrically corrects a variation in angle of view between a plurality of images caused by a variation in installation posture, zoom ratio, and the like of a plurality of imaging units. Related to technology.

  A so-called compound eye camera that includes a plurality of imaging units in one camera is capable of capturing the same subject from a plurality of viewpoints, and is therefore used for acquiring three-dimensional information of the subject. In the field of video viewing, there are 3D video viewing media such as prints using lenticular lens sheets using parallax images obtained from each imaging unit, and a compound eye camera as image input means for these media Is used.

  As described above, since a compound eye camera is composed of a plurality of imaging units, there are variations in characteristics of lenses, imaging elements, etc. of each imaging unit, or in installation postures for each imaging unit when mounting the imaging unit. As a result, a good parallax image cannot be obtained. In particular, the latter variation greatly affects the acquisition of accurate parallax information, so that the three-dimensional appearance and visibility of the 3D image are not impaired, and the variation in the installation posture of the imaging unit is suppressed. is important.

  When a subject (person) is imaged by two imaging units as shown in FIG. 11A, an ideal parallax image as shown in FIG. 11B (the subject is shifted in the horizontal direction due to parallax, It is difficult to obtain a parallax image having no camera posture variation), and the actually obtained parallax image has not only a parallax, but also an image shift caused by the posture of the imaging unit (FIG. 11C).

  In the process of manufacturing a compound eye camera, it is desirable to mount the imaging unit with high accuracy so that the above-mentioned variation does not occur. However, since accuracy is required, problems such as a decrease in productivity and an increase in man-hours occur.

  In order to solve such problems, conventionally, a method has been proposed in which an image captured by a compound eye camera is geometrically deformed to absorb a geometric characteristic difference between imaging units (Patent Literature). 1, 2).

  In the method described in Patent Document 1, in order to correct projective distortion caused by a shift in the positions and orientations of a plurality of cameras, images taken by each camera are projected on the same plane, and the images of each camera are projected. Corrections are made to images on the same plane.

  In addition, the method described in Patent Document 2 is intended to correct a deviation caused by a difference in posture and zoom ratio of an imaging unit among field angle deviations that occur when a subject is simultaneously captured by two imaging units. Then, corresponding points are detected between images obtained from the respective imaging units, geometric correction parameters are calculated so that the corresponding points of the images coincide with each other, and the image is deformed using the geometric correction parameters.

JP 7-294215 A JP 2006-157432 A

  However, since the method described in Patent Document 1 performs image processing for forcibly projecting each image on the same plane, there is a problem that parallax generated between the projected images becomes unintentional and inappropriate. .

  On the other hand, since the method described in Patent Document 2 deforms an image so that corresponding points between the two images coincide with each other, the angle-of-view shift caused by the parallax that is originally desired to be corrected is corrected, and the parallax information is lost. There is a problem of being.

  The present invention has been made in view of such circumstances, and when the variation in the angle of view between the plurality of images due to the variation in the installation posture and the zoom rate of the plurality of imaging units is absorbed by the geometric deformation, To provide a stereoscopic image processing device, method and program, and a stereoscopic imaging device that can be geometrically deformed so as to leave a field angle shift due to parallax that should occur between images, and thereby obtain an ideal parallax image. Objective.

  In order to achieve the above object, a stereoscopic image processing apparatus according to an aspect of the present invention includes an image acquisition unit that acquires a plurality of images obtained by capturing the same subject from a plurality of viewpoints, Corresponding point detection that selects a predetermined image as a reference image, selects an image other than the reference image as a target image, and detects a plurality of sets of corresponding points whose features substantially match between the reference image and the target image And image deformation means for geometrically deforming the target image so that only the y-coordinates of the feature points and the corresponding points included in the same set substantially coincide with each other in the direction orthogonal to the parallax directions of the plurality of viewpoints. It is characterized by having prepared.

  According to the invention of one aspect of the present invention, correspondence between a predetermined image (reference image) and images other than the reference image (target image) among a plurality of images obtained by photographing the same subject from a plurality of viewpoints. Among the displacements between the points, a constraint condition is set such that the component in the direction orthogonal to the parallax direction of the plurality of viewpoints is made substantially zero. That is, the y coordinate in the direction orthogonal to the parallax direction of the plurality of viewpoints among the xy coordinate value of one of the corresponding points of the plurality of sets of detected points and the xy coordinate value of the other corresponding point. By geometrically deforming the target image so that only the two substantially match each other, the angle of view in the direction orthogonal to the parallax direction is corrected while leaving the angle of view due to the parallax that should originally occur between the reference image and the target image. And an ideal parallax image can be obtained.

  A stereoscopic image processing method according to another aspect of the present invention includes an image acquisition step of acquiring a plurality of images obtained by photographing the same subject from a plurality of viewpoints, and using a predetermined image among the acquired plurality of images as a reference image. A corresponding point detecting step for selecting a plurality of corresponding points whose features substantially match between the reference image and the target image, and selecting the image other than the reference image as a target image; And an image deformation step of geometrically deforming the target image so that only the y-coordinates of the included feature points and corresponding points in the direction orthogonal to the parallax directions of the plurality of viewpoints substantially coincide with each other.

  A stereoscopic image processing program according to still another aspect of the present invention includes an image acquisition function for acquiring a plurality of images obtained by photographing the same subject from a plurality of viewpoints, and a predetermined image among the acquired plurality of images as a reference image. A corresponding point detection function for selecting a plurality of corresponding points whose features substantially match between the reference image and the target image, and selecting the image other than the reference image as a target image; Causing the computer to execute a stereoscopic image processing process in which the target image is geometrically deformed so that only the y-coordinates of the feature points and corresponding points in the direction orthogonal to the parallax directions of the plurality of viewpoints substantially coincide with each other. It is a feature.

  A stereoscopic image processing program according to still another aspect of the present invention acquires a plurality of images obtained by photographing the same subject from a plurality of viewpoints, selects a predetermined image from the plurality of acquired images as a reference image, An image other than the reference image is selected as a target image, a plurality of feature points are detected from the reference image, and corresponding points are detected from the target image, and the features of the corresponding points and the features of the corresponding points substantially match. A set of such feature points and corresponding points is generated, and for the feature points and corresponding points included in the same set, only the y-coordinates in the direction orthogonal to the parallax directions of the plurality of viewpoints substantially match. A computer is caused to execute a stereoscopic image processing process that estimates a geometric deformation parameter for geometrically deforming the image, and geometrically deforms the target image based on the estimated geometric deformation parameter. That.

  A stereoscopic imaging apparatus according to still another aspect of the present invention includes the stereoscopic image processing apparatus, and a plurality of imaging units that are respectively arranged at a plurality of viewpoint positions and respectively capture the same subject from the viewpoint positions, The image acquisition means acquires a plurality of images respectively captured by the plurality of imaging units.

  According to the present invention, among the displacements between corresponding points of a predetermined image (reference image) and a non-reference image (target image) among a plurality of images obtained by photographing the same subject from a plurality of viewpoints, Since the target image is geometrically deformed with a constraint condition that the component in the direction orthogonal to the parallax direction of the plurality of viewpoints is set to substantially zero, the view angle deviation due to the parallax that should originally occur between the images remains. On the other hand, it is possible to correct a field angle shift in a direction orthogonal to the parallax direction, and an ideal parallax image can be obtained.

FIG. 1 is a block diagram showing the overall configuration of a stereoscopic imaging apparatus according to the present invention. FIG. 2 is a block diagram showing a configuration of the imaging unit shown in FIG. FIG. 3 is a diagram used to explain the relationship between the arrangement of a plurality of imaging units in the apparatus main body and the subject. FIG. 4 is a diagram illustrating a correspondence relationship between ideal parallax images captured by six imaging units and actual parallax images. FIG. 5 is a flowchart showing the overall flow of stereoscopic image processing according to the present invention. FIG. 6 is a diagram illustrating a relationship between feature points and corresponding points in the reference image and the target image. FIG. 7 is a diagram used for explaining the concept of parameter estimation for geometric deformation according to the present invention. FIG. 8 is a diagram showing a part of a flowchart showing an example of the projection conversion parameter estimation procedure. FIG. 9 is a diagram showing the remaining part of the flowchart showing an example of the projection conversion parameter estimation procedure. FIG. 10 is a diagram used for explaining a method for determining the remaining parameters necessary for determining the x-coordinate value of the projection transformation destination among the projection transformation parameters. FIG. 11 is a diagram for explaining image shift caused by camera postures of a plurality of imaging units.

  Preferred embodiments of a stereoscopic image processing apparatus, method and program, and stereoscopic imaging apparatus according to the present invention will be described below with reference to the accompanying drawings.

[Overall configuration of stereoscopic imaging apparatus]
FIG. 1 is a block diagram showing the overall configuration of a stereoscopic imaging apparatus according to the present invention.

  As shown in the figure, the stereoscopic imaging apparatus 10 includes six imaging units 1 to 6, acquires six images (parallax images) obtained by photographing the same subject from six viewpoints, and has a predetermined format. This is an apparatus for recording as image data for recording.

  The central processing unit (CPU 12) is connected with the imaging units 1 to 6 and the light emitting unit 18 via the control bus 16, and also includes a main memory 20, a digital signal processing unit 22, an integrating unit 24, a compression / decompression processing unit 26, An external recording unit 28, a display unit 30, a corresponding point detection unit 32, a geometric deformation unit 34, and a geometric deformation parameter estimation unit 36 are connected, and the CPU 12 follows a predetermined control program based on an operation input from the operation unit 14. It functions as a control unit that performs overall control of the entire operation of the stereoscopic imaging apparatus 10.

  In addition, the imaging units 1 to 6, the main memory 20, the digital signal processing unit 22, the integration unit 24, the compression / decompression processing unit 26, the external recording unit 28, the display unit 30, the corresponding point detection unit 32, the geometric deformation unit 34, and the geometric The deformation parameter estimation units 36 are connected to each other via a data bus 38.

  Each of the imaging units 1 to 6 has the same configuration as shown in FIG. 2, and includes a photographic lens 40, an aperture 41, an IR cut filter 42, an optical low-pass filter 43, an imaging device (CCD) 44, and an A / D. It comprises a converter 45, a lens driving unit 46, an aperture driving unit 47, and a CCD driving unit 48.

  The taking lens 40 includes a focus lens, a zoom lens, and the like, and is driven by the lens driving unit 46 to move back and forth along the optical axis. The CPU 12 controls the driving of the lens driving unit 46 to control the position of the focus lens so as to adjust the focus so that the subject is in focus, and the position of the zoom lens according to the zoom command from the operation unit 14. Zooming is controlled.

  The diaphragm 41 is constituted by, for example, an iris diaphragm, and is driven by the diaphragm driving unit 47 to operate. The CPU 12 controls the aperture amount (aperture value) of the aperture 41 via the aperture drive unit 47 and controls the amount of light incident on the CCD 44.

  The CCD 44 is constituted by a two-dimensional color CCD solid-state imaging device. A large number of photodiodes are two-dimensionally arranged on the light receiving surface of the CCD 44, and a color filter is arranged in a predetermined arrangement on each photodiode. The optical image of the subject formed on the CCD light-receiving surface through the photographing lens 40, the diaphragm 41, the IR cut filter 42, and the optical low-pass filter 43 is converted into a signal charge corresponding to the amount of incident light by the photodiode. . The signal charge accumulated in each photodiode is sequentially read out from the CCD 44 as a voltage signal (image signal) corresponding to the signal charge based on a drive pulse given from the CCD drive unit 48 in accordance with an instruction from the CPU 12. The CCD 44 has an electronic shutter function, and the exposure time (shutter speed) is controlled by controlling the charge accumulation time in the photodiode. In the present embodiment, the CCD 44 is used as the image pickup device, but an image pickup device having another configuration such as a CMOS sensor may be used.

  The image signal read from the CCD 44 is converted into a digital signal by the A / D converter 45 and then temporarily stored in the main memory 20 via the data bus 38.

  As shown in FIG. 3, the imaging units 1 to 6 having the above-described configuration are arranged in the horizontal direction with respect to the apparatus main body of the stereoscopic imaging apparatus 10 and at a predetermined interval (predetermined baseline length), and each imaging unit. The convergence angles (angles formed by the optical axes) are adjusted so that the optical axes of the photographing lenses 1 to 6 intersect at one point.

  The CPU 12 drives the imaging units 1 to 6 in synchronization. That is, the photographing lenses 40 are always adjusted in focus so that the same subject is in focus, and are always set to the same focal length (zoom magnification). In addition, the diaphragm 41 is adjusted so that the same incident light amount (aperture value) is always obtained.

  The operation unit 14 includes known operation input means such as a shutter button, a power switch, a mode dial, a cross button, and a zoom button. The shutter button is composed of a two-stage stroke type switch composed of so-called “half press” and “full press”. When the shutter button is pressed halfway in the shooting mode, shooting preparation processing (that is, AE (Automatic Exposure), AF (Auto Focus), AWB (Automatic White Balance)) is performed. When the shutter button is fully pressed, image capturing / recording processing is performed.

  The light emitting unit 18 is constituted by, for example, a discharge tube (xenon tube), and emits light as necessary when photographing a dark subject or in backlight.

  The main memory 20 is used as a work area when the CPU 12 executes a program, and as a storage unit that temporarily stores digital image signals captured and acquired by the imaging units 1 to 6.

  The digital signal processing unit 22 interpolates a spatial shift of the color signal associated with the color filter array of the single-chip CCD by interpolating a white balance adjustment circuit, a gradation conversion processing circuit (for example, a gamma correction circuit), and a synchronization circuit (color). A processing circuit that converts signals simultaneously), a contour correction circuit, a luminance / color difference signal generation circuit, and the like, and functions as image processing means for R, G, and B image data stored in the main memory 20 Signal processing. In other words, the R, G, B image signals are converted into YUV signals composed of luminance signals (Y signals) and color difference signals (Cr, Cb signals) in the digital signal processing unit 22, and gradation conversion processing (for example, A predetermined process such as gamma correction is performed. The image signal processed by the digital signal processing unit 22 is stored in the main memory 20.

  The accumulating unit 24 calculates a focus evaluation value necessary for AF control based on an image signal captured when the shutter button is half-pressed, and calculates a subject luminance necessary for AE control. The CPU 12 searches for a position where the focus evaluation value calculated by the integrating unit 24 is maximized during AF control, moves the focus lens to that position, thereby focusing on the subject (main subject), and performs AE control. Sometimes, exposure setting is performed to obtain an appropriate exposure amount based on the subject brightness calculated by the integrating unit 24. That is, the photographing sensitivity, aperture value, shutter speed, and necessity of flash emission are set.

  The compression / decompression processing unit 26 performs compression processing on the input image data in accordance with a command from the CPU 12 to generate compressed image data in a predetermined format. For example, compression processing conforming to the JPEG standard is applied to still images, and MPEG2, MPEG4, H.264, and the like are applied to moving images. A compression process conforming to the H.264 standard is performed. In addition, the compression / decompression processing unit 26 performs decompression processing on the input compressed image data in accordance with a command from the CPU 12 to generate uncompressed image data.

  The external recording unit 28 records the image file storing the image data in the JPEG format or the like generated by the compression / decompression processing unit 26 on a removable external recording medium such as a memory card, or stores the image file from the external recording medium. Read.

  The display unit 30 is configured by a display device including a color liquid crystal panel, for example, and is used as an image display unit for displaying a captured image and is used as a GUI at various settings. The display unit 30 is used as an electronic viewfinder for confirming the angle of view in the shooting mode. A so-called lenticular lens having a semi-cylindrical lens group is arranged on the surface of the display unit 30, and an external recording unit is in a three-dimensional image (3D image) reproduction mode with a plurality of viewpoint images (parallax images). The parallax image read from 28 is displayed, and the user can stereoscopically view.

  As shown in FIG. 4 (A), the six images obtained from the imaging units 1 to 6 ideally have subject displacement due to parallax only between the images. As shown, in addition to the parallax, the subject shifts due to variations in the posture of the imaging units 1 to 6 and the zoom magnification.

  The corresponding point detection unit 32, the geometric deformation unit 34, and the geometric deformation parameter estimation unit 36 are characteristic portions of the present invention that correct the displacement of the subject by geometrically deforming the image. Refer to the flowchart shown in FIG. While explaining.

<Overall flow of stereoscopic image processing>
FIG. 5 is a flowchart showing the overall flow of stereoscopic image processing according to the present invention.

[Step S10]
When a parallax image is captured, the same subject is imaged by the six imaging units 1 to 6 and the obtained six images are recorded. At this time, each image is subjected to the various signal processing described above by the digital signal processing unit 22 and then temporarily stored in the main memory 20. Note that recording may be performed on an external recording medium by the external recording unit 28.

[Step S12]
Of the six images stored in the main memory 20, one predetermined image (in this embodiment, an image obtained from the imaging unit 3) is set as a reference image, and the set reference image is Read from the main memory 20. In this embodiment, the image obtained from the imaging unit 3 is set as the reference image. However, the present invention is not limited to this, and any of the six images may be set as the reference image. Good.

[Step S14]
One predetermined image among the five images other than the set reference image is selected as a target image, and the selected target image is read from the main memory 20.

[Step S16]
The corresponding point detection unit 32 detects a plurality of sets of corresponding points whose features substantially match between the set reference image and the selected target image.

  Various methods have been proposed for detecting corresponding points by the corresponding point detection unit 32. Although the present invention is not limited to this method, for example, conventional techniques such as block matching method, KLT method (Tomasi & Kanade, 1991, Detection and Tracking of Point Features), SIFT (Scale Invariant Feature Transform) can be used. .

  In this embodiment, for convenience, the corresponding points detected from the reference image among the corresponding points of each pair (pair) detected by the corresponding point detection unit 32 are referred to as feature points.

[Step S18]
The geometric deformation parameter estimation unit 36 estimates and calculates a projective transformation parameter based on the coordinate values of a plurality of sets of the feature points and corresponding points. Details of step S18 will be described later.

[Step S20]
It is determined whether the projective transformation parameter is estimated as normal. If normal (in the case of “yes”), the process proceeds to step S22. If not normal (in the case of “no”), step S22 is performed. And the process proceeds to step S24.

[Step S22]
The geometric deformation unit 34 performs projective transformation on the target image based on the normally estimated projection transformation parameter. The image after the projective transformation is recorded on an external recording medium by the external recording unit 28.

[Step S24]
It is determined whether or not the processing in steps S14 to S22 has been completed between all of the reference image and all five target images. If not completed, the process proceeds to step S14, where another target image is selected. If completed, the stereoscopic image processing is terminated.

<Projection transformation parameter estimation flow>
Next, a procedure for estimating projective transformation parameters based on a set of a plurality of feature points and corresponding points will be described.

  Here, the projective transformation formula is as follows.

  The projective transformation parameters indicate eight parameters a, b, s, c, d, t, p, and q in the formula [1]. Further, (x, y) and (X, Y) indicate coordinate values before and after projective transformation, respectively.

  FIG. 6A is a diagram in which feature points (●) extracted from the reference image are superimposed on the reference image, and FIG. 6B is a diagram illustrating corresponding points (◯) detected from the target image on the target image. FIG.

  In FIG. 6C, vectors connecting the corresponding feature points and the corresponding points are shown. In FIG. 6, a feature point A (x1, y1) and a corresponding point A ′ (x2, y2) indicate a certain set of feature points and corresponding points.

  Originally, the feature points and the corresponding points should be the same height (= y coordinate values are substantially the same), but subject displacement occurs due to variations in the orientation and zoom rate of the imaging units 1 to 6. Therefore, the height (y coordinate value) of the feature point A and the corresponding point A ′ does not match.

  Therefore, according to the present invention, in estimating the projective transformation parameter, a line segment connecting the feature point and the corresponding point (hereinafter referred to as “vector”) is substantially horizontal (that is, the y-coordinate value of the feature point and the corresponding point is approximately equal). Is a constraint condition. This constraint condition is based on the fact that the six imaging units 1 to 6 are installed horizontally on the apparatus main body. If a plurality of imaging units are installed vertically, it is necessary to make an assumption that the vector is vertical. In short, a constraint condition is set that the coordinate values in the direction orthogonal to the parallax directions of a plurality of viewpoints substantially coincide.

  FIG. 7 is a diagram showing the concept of parameter estimation according to the present invention.

  As shown in FIG. 7, when the factors that cause the deviation between the feature points and the corresponding points (composite vectors in FIG. 7) are considered separately, there are “parallax” and “variation in the posture of the imaging unit”. As can be seen from the figure, the y component of the composite vector is not affected by parallax, and only the component due to variations in the orientation of the imaging unit appears.

  Therefore, the present invention estimates and calculates a projective transformation parameter related to the camera posture and the like from the y component of the vector.

<Projection transformation parameter estimation processing flow>
8 and 9 are flowcharts showing an example of the projection transformation parameter estimation procedure according to the present invention.

[Step S100]
When the number of pairs of feature points extracted from the reference image and corresponding points corresponding to each feature point detected from the target image is N, the coordinates (x, y) of the N feature points ) And the coordinates (X, Y) of the corresponding point. The coordinates of feature points of the i-th (1 ≦ i ≦ N) set are (xi, yi), and the coordinates of corresponding points are (Xi, Yi).

[Step S102]
When the number of pairs of feature points and corresponding points obtained by the corresponding point detection unit 32 is small, projection conversion parameters cannot be estimated, and therefore threshold processing is performed for the number N of pairs.

  That is, a threshold value TH1 is provided for the number N of pairs of feature points and corresponding points, and the following conditional branching is performed.

  IF N <TH1 → A flag indicating that the parameter estimation has failed is set and the estimation is terminated (step S124 in FIG. 9).

  ELSE → Continue parameter estimation.

  Note that in order to estimate the projective transformation parameter, at least five sets of coordinate values are required, and thus the threshold value TH1 is set to a predetermined value of 5 or more.

[Step S104]
The number of iterations iland and the parameter reliability maximum value n_vote_max are initialized (both are set to 0).

[Step S106]
Of the N sets of feature points and corresponding points, five sets are randomly selected. Five sets of random selections can be made using random numbers.

[Step S108]
Based on the coordinate values of the five sets of feature points and corresponding points selected in step S106 (the xy coordinate values of the five feature points and the y coordinate value of the five corresponding points), Projective transformation parameters for matching y-coordinate values are calculated.

  As shown in the above [Equation 1], there are five projective transformation parameters in the y direction, c, d, t, p, and q. Therefore, the coordinate values of the five points are expressed in [Equation 1]. By solving simultaneous equations of five formulas obtained by substitution, parameters (c, d, t, p, q) can be calculated uniquely.

[Step S110]
A set of all points (N sets) is applied to the projective transformation expression into which the calculated parameters (c, d, t, p, q) are substituted, and the number n_vote of points satisfying the following conditional expression 1 is counted. .

  The intent of Conditional Expression 1 is to determine whether or not the y-coordinate values of the other point sets match based on the projective transformation parameters determined from the five point sets. That is, the value of n_vote represents how many of the N vectors are horizontal. It can be determined that the larger the n_vote is, the higher the reliability of the projective transformation parameter is.

[Step S112]
The number n_vote calculated in step S110 is compared with the reliability maximum value n_vote_max. If n_vote is larger than n_vote_max (n_vote> n_vote_max), the process proceeds to step S114, and if smaller (n_vote ≦ n_vote_max), The process proceeds to step S118.

[Step S114]
The calculated parameters (c, d, t, p, q) are temporarily stored as intermediate parameters (c_tmp, d_tmp, t_tmp, p_tmp, q_tmp), and n_vote is temporarily stored as a maximum value n_vote_max.

  When estimating the projective transformation parameters, the processing from step S106 to step S118 is repeated nrand for a predetermined number of times as described below, and the parameter when n_vote becomes the largest is considered to have the highest reliability. . Therefore, the previous reliability n_vote is held as n_vote_max, and each parameter is updated when the current reliability is greater than the previous reliability.

[Step S116]
A ratio (n_vote_max / N) of the reliability maximum value n_vote_max with respect to the number N of all pairs is obtained, and threshold processing is performed on this ratio.

  That is, the following conditional branch is performed according to a preset threshold value TH2.

  IF n_vote_max / N <TH2 → Parameter estimation is continued.

  ELSE → The parameter iterative estimation is terminated, and the process proceeds to the calculation of the final projective transformation parameter (step S126 in FIG. 9).

  The threshold value TH2 can be set as a value near 1 that is smaller than 1.

  Since the maximum reliability value n_vote_max is the number of points in which the y coordinate values are substantially the same among the N points as a result of projective transformation using the estimated parameters, n_vote_max / N is the set of all points. The ratio of the number of points corrected appropriately to the number N of Therefore, when n_vote_max / N is greater than or equal to TH2, it is assumed that the estimated parameter is sufficiently reliable, and the iterative estimation of the parameter is terminated, and the process proceeds to calculation of the final projective transformation parameter. On the other hand, when n_vote_max / N is less than TH2, iterative parameter estimation is continued.

[Step S118]
It is determined whether or not the number of iterations iland that iteratively estimates the parameter is greater than the predetermined number of times nrand. If the number of iterations irand is less than or equal to the predetermined number of times nrand (in the case of “no”), the process proceeds to step S120. If the number of iterations irand is greater than the predetermined number of times nrand (in the case of “yes”), the process proceeds to step S122. As the predetermined number of times nrand, although it depends on the number N of pairs of feature points and corresponding points in the image, about 1000 to 10,000 times can be considered.

[Step S120]
The iteration number iland is incremented by 1, and the process proceeds to step S106.

[Step S122]
When the number of iterations iland is larger than the predetermined number nrand, the ratio (n_vote_max / N) of the current reliability maximum value n_vote_max held in step S114 to the number N of all pairs is obtained, and this ratio Threshold processing for is performed.

  That is, the following conditional branching is performed according to a preset threshold value TH3.

  IF n_vote_max / N <TH3 → A flag indicating that the parameter estimation has failed is set and the estimation is terminated (step S124 in FIG. 9).

  ELSE → The parameter iterative estimation is terminated, and the process proceeds to a step (step S126 in FIG. 9) for calculating the final projective transformation parameter.

  Needless to say, the threshold value TH3 is smaller than the threshold value TH2 in step S114.

[Step S124 (FIG. 9)]
As described above, when the number N of pairs of feature points and corresponding points is less than the threshold TH1 (step S102) or when the reliability of the estimated parameter is low (n_vote_max / N <TH3), the projection transformation parameter is estimated. An estimation failure flag indicating that the process failed has been set, and the projection transformation parameter estimation is terminated.

[Step S126]
All sets of points (N sets) are applied to the projective transformation formula substituted with the intermediate parameters (c_tmp, d_tmp, t_tmp, p_tmp, q_tmp) held in step S114, and only the sets satisfying the following conditional expression 2 from N sets (A set that does not satisfy conditional expression 2 is excluded).

  Thereby, it is possible to leave only a pair of feature points and corresponding points with high reliability.

[Step S128]
The final parameter estimation is performed by the method of least squares using only a set of points satisfying the conditional expression 2. Here, assuming that the number of sets of points satisfying the conditional expression 2 is n, the parameter estimation by the least square method estimates the parameters so that the evaluation function J2 of the following expression is minimized.

  The evaluation function J2 is equivalent to the evaluation function J1.

  Specifically, as shown in the following equation, each of five equations obtained by partial differentiation of the evaluation function J2 with each parameter (c, d, t, p, q) is set to 0, and these five equations are used. If the simultaneous equations are solved, parameters (c, d, t, p, q) can be obtained.

  The final parameters (c, d, t, p, q) can be obtained by solving the above simultaneous equations.

[Step S130]
Next, among the eight projective transformation parameters (a, b, s, c, d, t, p, q), projection transformation destinations other than the estimated parameters (c, d, t, p, q) are selected. The remaining parameters (a, b, s) necessary to determine the x coordinate value are determined.

  Here, the parameter s is a parameter corresponding to the amount of parallel movement in the x direction (direction in which parallax occurs). Since the amount of parallax is unknown and s cannot be uniquely determined, s = 0 is set.

  On the other hand, the parameters a and b are calculated as follows, for example.

  Consider a coordinate system as shown in FIG. Assume that the image plane is at a distance 1 from the origin O, and that each point on the image plane is projected onto the projection plane, that is, the point P1 is projected onto P2.

  Here, the angle between the image plane and the projection plane is θ, and the projection plane is on the x ′ axis. Further, the intersection point of the projection plane and the z axis (the depth direction of the image) is defined as O ′, and the origin on the x ′ axis.

  When the x coordinate value of P1 is x and the x coordinate value of P2 when x 'axis is considered as x' axis, the following equation is established.

  If this is generalized and the angle between the projection plane and the x-axis is θx, and the angle between the y-axis (axis in the direction orthogonal to the paper surface in FIG. 10) is θy, the following equation is obtained.

  In the above [Expression 7], (x, y) is a coordinate value on the image plane, and (x ′, y ′) is a coordinate value on the projection plane. W and h represent the width and height of the image, respectively, and the width and height of the image are normalized. This operation corresponds to adjusting the direction of the optical axis of the imaging unit.

Subsequently, rotation, scale change, and parallel movement are performed on the image projected on the projection plane. This operation corresponds to adjusting the zoom rate of the imaging unit and the rotation of the optical axis. Here, when the rotation angle is θ ₀ , the scale change rate due to zoom is k, and the parallel movement amount is (s ′, t ′), the following equation is established.

  Substituting [Expression 7] into [Expression 8] gives the following expression.

  When this is compared with the projective transformation equation (Equation 1), the following relationship is derived.

In the above [Equation 10] where already c, d, t, p, since q is known, [theta] x from these equations, [theta] y, it is possible to obtain the theta _0, k, these [theta] x, [theta] y, theta ₀ , K, a and b can be calculated. However, as described above, a and b are calculated with s = s ′ = 0.

[Step S132]
When the estimation / calculation of the eight projective transformation parameters (a, b, s, c, d, t, p, q) is completed by the above procedure, the estimation success flag is set and the estimation of the projective transformation parameters is finished. .

  In this embodiment, the final parameter estimation is performed by the least square method using only a set of points satisfying the condition shown in [Equation 3]. However, the present invention is not limited to this. Intermediate parameters may be used as they are.

<Others>
According to the above estimation method, the projective transformation parameter for projective transformation of the feature point is calculated. That is, it can be said that the obtained projective transformation parameter is for deforming the reference image. Originally, the parameter to be calculated is a parameter for deforming the target image, but the reason for not doing so is as follows.

  Normally, when an input image is transformed and output, the position of each pixel position of the output image is calculated on the input image. This is a general method for avoiding the occurrence of an area without data when calculating where each pixel of the input image is located on the output image. That is, the parameters necessary for deforming the target image may actually be parameters for converting the feature points.

  In this embodiment, the projective transformation is used as a method of geometric deformation of the image. However, it is naturally possible to use geometric transformation such as Helmat transformation or affine transformation. In that case, the parameter estimation method is slightly different from that in the case of projective transformation. For example, the Helmat transform equation is expressed as the following equation.

  If the same idea as the method for estimating projective transformation parameters is used, the five parameters (c, d, t, p, q) for obtaining Y are calculated using five sets of feature points and corresponding points. What is necessary is just to change so that the parameter (a, b, t) for calculating | requiring Y of [Formula 11] may be estimated using three sets of feature points and corresponding points. Further, when final parameter estimation is performed by the method of least squares, the evaluation function J may be expressed by the following equation.

  In the Helmart transform equation, as shown in [Equation 11], the parameters for determining X include a and b. Therefore, if the parameters for determining Y are estimated, the parameters for determining X are also determined. It will inevitably be obtained. As in the case of projective transformation, the parameter s, which is a translation component for X, can be considered as s = 0. In the case of performing other geometric deformation, it is possible to estimate the parameters for the geometric deformation based on the same concept.

  In addition, the stereoscopic imaging apparatus of this embodiment is a compound eye camera having six imaging units, but the number of imaging units only needs to be two or more, and is not limited to that of this embodiment. The present invention can be applied to a camera system including a plurality of monocular cameras as well as a plurality of compound eye cameras.

  Furthermore, the stereoscopic image processing according to the present invention is not limited to being performed by a stereoscopic imaging apparatus, and may be performed by a personal computer or the like that does not have a parallax image capturing function. In this case, a plurality of images (parallax images) taken by a conventional compound eye camera or the like may be taken into a personal computer or the like. The present invention can also provide a program for causing a personal computer to implement a function of performing geometric deformation processing for obtaining an ideal parallax image.

  Moreover, it goes without saying that the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the present invention.

  DESCRIPTION OF SYMBOLS 1-6 ... Imaging unit, 10 ... Stereoscopic imaging device, 12 ... Central processing unit (CPU), 20 ... Main memory, 22 ... Digital signal processing part, 28 ... External recording part, 30 ... Display part, 32 ... Corresponding point detection 34, geometric deformation unit, 36 ... geometric deformation parameter estimation unit

Claims (5)

Image acquisition means for acquiring a plurality of images taken of the same subject from a plurality of viewpoints;
A predetermined image of the plurality of acquired images is selected as a reference image, an image other than the reference image is selected as a target image, a plurality of feature points are detected from the reference image, and corresponding points are detected from the target image. A corresponding point detecting means for generating a set of feature points and corresponding points such that the features of the corresponding points and the corresponding points included in the same set substantially match;
Image deformation means for geometrically deforming the target image so that only the y-coordinates of the feature points and corresponding points included in the same set substantially coincide with each other in a direction orthogonal to the parallax direction of the plurality of viewpoints;
A stereoscopic image processing apparatus comprising: An image acquisition step of acquiring a plurality of images obtained by photographing the same subject from a plurality of viewpoints;
A predetermined image of the plurality of acquired images is selected as a reference image, an image other than the reference image is selected as a target image, a plurality of feature points are detected from the reference image, and corresponding points are detected from the target image. A corresponding point detecting step for generating a set of feature points and corresponding points such that the features of the corresponding points and the features of the corresponding points substantially match,
An image deformation step of geometrically deforming the target image so that only y-coordinates of the feature points and corresponding points included in the same set substantially coincide with each other in a direction orthogonal to the parallax directions of the plurality of viewpoints;
A stereoscopic image processing method comprising: Acquire multiple images of the same subject from multiple viewpoints,
A predetermined image of the plurality of acquired images is selected as a reference image, an image other than the reference image is selected as a target image, a plurality of feature points are detected from the reference image, and corresponding points are detected from the target image. Then, a set of feature points and corresponding points such that feature points included in the same set and features of corresponding points substantially match,
Geometrically deforming the target image so that only the y-coordinates of the feature points and corresponding points included in the same set substantially coincide with each other in the direction orthogonal to the parallax direction of the plurality of viewpoints
A stereoscopic image processing program for causing a computer to execute a stereoscopic image processing process. Acquire multiple images of the same subject from multiple viewpoints,
A predetermined image of the plurality of acquired images is selected as a reference image, an image other than the reference image is selected as a target image, a plurality of feature points are detected from the reference image, and corresponding points are detected from the target image. Then, a set of feature points and corresponding points such that feature points included in the same set and features of corresponding points substantially match,
Estimating geometric deformation parameters for geometrically deforming the target image so that only the y-coordinates in the direction orthogonal to the parallax directions of the plurality of viewpoints substantially match with respect to the feature points and corresponding points included in the same set;
Geometrically deforming the target image based on the estimated geometric deformation parameters;
A stereoscopic image processing program for causing a computer to execute a stereoscopic image processing process. A stereoscopic image processing apparatus according to claim 1;
A plurality of imaging units that are respectively arranged at a plurality of viewpoint positions and respectively image the same subject from the viewpoint positions along the parallax direction,
The three-dimensional imaging device, wherein the image acquisition means acquires a plurality of images respectively captured by the plurality of imaging units.

Images