JP2013120435A - Image processing apparatus and image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a multi-viewpoint imaging apparatus using a rolling shutter image sensor which estimates the motion of the multi-viewpoint imaging apparatus during imaging and performs image composition or object distance estimation while correcting the influence of distortion of the rolling shutter using the estimated motion.SOLUTION: The image forming apparatus includes: multi-viewpoint image acquisition means (701) for acquiring an image group imaged from a plurality of viewpoints; viewpoint arrangement information acquisition means (703) for acquiring relative arrangement of the plurality of viewpoints; corresponding point searching means (704) for searching a set of points on an image corresponding to the same point of an object between the plurality of viewpoints; exposure time calculation means (705) for calculating time at which the acquired points on the image was exposed with light; and motion parameter calculation means (706) for estimating a parameter indicating motion of the plurality of viewpoints during imaging on the basis of a coordinate of the set of points on the image, the time at which the points was exposed with light, and the relative arrangement of the plurality of viewpoints.

Description

  The present invention relates to an image processing apparatus and an image processing method for processing images captured from a plurality of viewpoints.

  In recent years, CMOS image sensors have been widely used as image sensors for digital cameras and video cameras. The CMOS image sensor has excellent features such as low power consumption and low cost as compared with a CCD image sensor that has been conventionally used as an image sensor.

  However, since the CMOS image sensor completes the exposure by reading the pixel value, there is a time difference in the exposure timing for each pixel. A general CMOS image sensor has a mechanism for simultaneously reading out pixel values for each line, and acquires an image of the entire image by scanning the readout line over time (rolling shutter method). For this reason, if the image of the subject moves due to camera shake or the like during imaging, the acquired image becomes distorted. This phenomenon is called rolling shutter distortion.

  Such a rolling shutter type image sensor is used not only for a normal digital camera and video camera, but also for a multi-viewpoint imaging apparatus that captures a subject from a plurality of viewpoints using a plurality of imaging units. The image captured by the multi-viewpoint imaging device is used for synthesis of a virtual viewpoint image (Non-Patent Document 1), acquisition of a subject distance (Non-Patent Document 2), and the like. At this time, if a rolling shutter distortion is included in each viewpoint image, problems such as deterioration in the image quality of the synthesized image and a decrease in subject distance estimation accuracy occur. Therefore, it is necessary to correct distortion (rolling shutter distortion) caused by the rolling shutter system.

  As a technique for correcting rolling shutter distortion in a system capable of acquiring a plurality of images such as a multi-viewpoint imaging apparatus, Patent Document 1 and Non-Patent Document 3 are known. In Patent Document 1, a plurality of images are continuously captured, a motion vector of each region on the image is obtained by block matching, and each region is moved and taken into account with an exposure time corresponding to the coordinates on the image. A technique for correcting rolling shutter distortion by deformation is disclosed.

  Further, Non-Patent Document 3 discloses that in a multi-viewpoint imaging apparatus, the imaging timing of each eye is shifted to change the area captured by each eye within a certain minute time, and rolling is performed by combining the images of each eye. A technique for obtaining an image with reduced shutter distortion is disclosed.

Japanese Patent No. 4509917

A. Isaksen et al., “Dynamically Reparameterized Light Fields”, ACM SIGGRAPH, pp. 297-306 (2000) Y. Furukawa et al., “Accurate, Dense, and Robust Multi-View Stereopsis”, CVPR 2007 Wilburn et al., “High-Speed Videography Using a Dense Camera Array”, CVPR’04 Richard Szeliski, “Computer Vision: Algorithms and Applications”, Springer, New York, 2010 Bill Triggs et al., “Bundle Adjustment − A Modern Synthesis”, Vision Algorithm: Theory & Practice, Springer-Verlag LNCS 1883 (2000).

  However, the method described in Patent Document 1 assumes a situation in which the viewpoints between a plurality of images are almost the same, and thus there is a problem that accuracy is reduced when there is a difference in viewpoints between images. Further, in the method described in Non-Patent Document 3, since it is assumed that the subject is on the same plane, there is a problem that the accuracy is lowered for a subject having a depth. Therefore, these methods are not suitable for a multi-viewpoint imaging apparatus that captures a non-planar subject from a plurality of viewpoints.

  SUMMARY OF THE INVENTION An object of the present invention is to provide means for estimating the motion of a multi-viewpoint imaging device during imaging in a multi-viewpoint imaging device using a rolling shutter type image sensor. It is another object of the present invention to provide means for performing image composition or subject distance estimation while correcting the influence of rolling shutter distortion using the estimated motion.

  In order to solve the above problems, an image processing apparatus according to the present invention includes a multi-viewpoint image acquisition unit that acquires a group of images captured from a plurality of viewpoints, and a viewpoint arrangement that acquires a relative arrangement of the plurality of viewpoints. Information acquisition means; corresponding point search means for searching for a set of points on the image corresponding to the same point on the subject between the plurality of viewpoints; and time at which the points on the acquired image are exposed Estimating a parameter representing motion during imaging of the plurality of viewpoints from the exposure time calculation means, the coordinates of the set of points on the image, the exposure time, and the relative arrangement of the plurality of viewpoints Motion parameter calculating means.

  ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to estimate the motion of the multi-viewpoint imaging device during imaging in the multi-viewpoint imaging device using the rolling shutter type image sensor. Further, it is possible to perform image synthesis or subject distance estimation while correcting the influence of rolling shutter distortion using the estimated motion.

1 is a diagram showing an overview of a multi-viewpoint imaging apparatus according to the present invention. It is a block diagram which shows the structural example of the multiview imaging device which concerns on Embodiment 1 of this invention. It is a block diagram which shows the structural example of a multiview imaging part. It is a block diagram which shows the structural example of an imaging part. It is a figure which shows the example of arrangement | positioning of an imaging part. It is a figure for demonstrating the description method of arrangement | positioning of each imaging part. It is a block diagram which shows the structural example of a motion estimation part. It is a flowchart which shows the process sequence of a motion estimation process. It is a figure for demonstrating the calculation method of exposure time. It is a block diagram which shows the structural example of a corresponding point search part. It is a figure for demonstrating the search range in a corresponding point search. It is a figure which shows the flow of a corresponding point search process. It is a figure showing the structural example of a motion parameter calculation part. It is a flowchart which shows the process sequence of a motion parameter calculation process. It is a block diagram showing the example of a structure of a to-be-photographed object distance estimation part. It is a figure showing the example of the flow of a subject distance estimation process. 5 is a block diagram illustrating a configuration example of a multi-viewpoint imaging apparatus according to Embodiment 2. FIG. FIG. 6 is a diagram for explaining the principle of image composition according to the second embodiment. 5 is a block diagram illustrating a configuration example of an image composition unit according to Embodiment 2. FIG. FIG. 6 is a flowchart showing a processing procedure of image composition processing according to the second embodiment. 10 is a diagram for explaining the effect of the invention according to Embodiment 3. FIG. 6 is a diagram illustrating an arrangement example of imaging units in Embodiment 3. FIG.

(Embodiment 1)
<Overall configuration of imaging device>
FIG. 1 is a schematic diagram showing an example of a multi-viewpoint imaging apparatus according to the present invention. The multi-viewpoint imaging apparatus has a housing 101. FIG. 1A is a front view of an exemplary multi-viewpoint imaging apparatus. FIG. 1B is a rear view of the exemplary multi-viewpoint imaging apparatus.

  The imaging units 105 to 113 are arranged in a plurality in front of the multi-view imaging device, so that the multi-view imaging device can acquire images of a plurality of viewpoints. The multi-viewpoint imaging apparatus further includes a shooting button 102, a display 103, and an operation button 104.

  Setting is performed using the operation button 104 and the display 103, and when the shooting button 102 is pressed, imaging by the imaging units 105 to 113 is performed. The acquired image and the synthesized image can be displayed on the display 103.

  FIG. 2 is a block diagram illustrating an example of a configuration of a multi-viewpoint imaging apparatus having a function as an image processing unit according to the first embodiment of the present invention.

  Although the details will be described later, the multi-viewpoint imaging unit 213 includes a plurality of imaging units and acquires an image group from a plurality of viewpoints. Note that an image group acquired from a plurality of viewpoints is referred to as a multi-viewpoint image. Note that each imaging unit constituting the multi-viewpoint imaging unit 213 has a rolling shutter type image sensor, and an image formed by the optical system can be converted into image data using the image sensor.

  As will be described in detail later, the motion estimation unit 214 estimates the motion of the multi-viewpoint imaging device being imaged from the multi-viewpoint image acquired by the multi-viewpoint imaging unit 213. The motion estimation unit 214 searches for corresponding points between images of different viewpoints, and performs motion estimation based on the search results.

  The subject distance estimation unit 215 estimates the distance to a point on the subject using the information about the motion of the multi-viewpoint imaging device being captured acquired by the motion estimation unit 214 and the corresponding points between images of different viewpoints. .

  Note that the motion estimation unit 214 and the subject distance estimation unit 215 need information on the arrangement and characteristics of a plurality of viewpoints constituting the multi-viewpoint imaging unit, and information on the exposure / readout control of the image sensor constituting each imaging unit. And These pieces of information are calculated in advance and are stored in the ROM 202 or the RAM 203.

  The CPU 201 is involved in all the processes of each configuration, reads the instructions stored in the ROM 202 and the RAM 203 in order, interprets them, and executes the processes according to the results. The ROM 202 and RAM 203 provide the CPU 201 with programs, data, work areas, and the like necessary for the processing.

  The bus 210 functions as a path for exchanging data and processing instructions between the components.

  The operation unit 204 corresponds to a button, a mode dial, and the like, and receives a user instruction input via these buttons.

  Character generation 209 generates characters, graphics, and the like.

  In general, a liquid crystal display is widely used as the display unit 206, and displays captured images and characters received from the character generation 209 and the display control unit 205. Further, it may have a touch screen function. In that case, a user instruction can be handled as an input of the operation unit 204.

  The digital signal processing unit 212 performs γ adjustment of the multi-viewpoint image acquired by the multi-viewpoint imaging unit 213, interpolation of defective pixels, demosaicing, and the like. These processes are performed before the process by the motion estimation unit 214. Also, γ adjustment and white balance adjustment of the multi-viewpoint image to be output and displayed are performed.

  The encoder unit 211 performs encoding processing of the output multi-viewpoint image.

  The media interface 207 is an interface for connecting to a PC / other media (for example, hard disk, memory card, CF card, SD card, USB memory) 208. A subject distance estimation result and a multi-viewpoint image are output via the media interface 207.

  In addition, although the component of an apparatus exists other than the above, since it is not the main point of this invention, description is abbreviate | omitted.

<Configuration of multi-viewpoint imaging unit>
FIG. 3 is a diagram illustrating a configuration example of the multi-viewpoint imaging unit 213.

  The imaging units 301 to 309 each acquire an image from a single viewpoint. A multi-viewpoint image is acquired by arranging these at spatially different positions. Note that the imaging control unit 310 controls the operations of the imaging units 301 to 309 based on an instruction from the CPU 201.

  Next, a configuration example of the imaging units 301 to 309 will be described using the block diagram shown in FIG.

  The lens 401 and the lens 402 constitute an imaging optical system. The light beam emitted from the subject passes through the aperture 403, the IR cut filter 404, the low pass filter 405, and the color filter 406 by the imaging optical system, and then forms an image on the image sensor 407.

  The optical system control unit 410 controls the lens 401 and the lens 402 to change the focusing distance and the focal length in the imaging optical system. Further, the F value can be adjusted by opening and closing the diaphragm 403.

  Control values are collectively set by the imaging control unit 310 for the optical system control unit 410 included in each of the imaging units 301 to 309. The in-focus distance, focal length, and F value that are set are based on values set by the user via the operation unit 204 or values calculated by a separate automatic exposure device, autofocus device, etc. It is decided.

  Note that the configuration of the optical system shown here is simplified for the sake of explanation, and any configuration can be used as long as it has a configuration for forming an image of a subject on the image sensor. Further, the configuration for acquiring a color image has been described, but the acquired image may be black and white, multiband, or an image with different exposure for each pixel.

  The image sensor 407 and the A / D converter 408 correspond to an image sensor such as a CMOS image sensor. The imaging elements 407 are arranged in a two-dimensional lattice shape, and convert the formed image into an electrical signal. The A / D conversion unit 408 converts information of the image converted into an electric signal into a digital signal. Image information converted into a digital signal is stored in the buffer 409 as image data.

  The image sensor control unit 411 controls the start of exposure and signal readout for the image sensor 407 and the A / D converter 408. Exposure / reading is performed for each line on the grid formed by the image sensor. This line is hereinafter referred to as a simultaneous exposure line. The simultaneous exposure line is scanned as time advances, and image data of the entire image is acquired over a certain period of time. The timing for starting exposure / reading is instructed by the image sensor control unit 411. The scanning order and speed of the simultaneous exposure lines can be determined in advance by design, or may be instructed by the imaging control unit 310 to the imaging element control unit 411.

  Note that the image sensor may not be a CMOS image sensor, as long as the image can be obtained by converting the formed image into a digital signal for each line and scanning the line. Any configuration may be used.

  The scanning order of the simultaneous exposure lines is preferably continuous, but there may be several discontinuities. In addition, the simultaneous exposure line need not be exactly one line, and may be a plurality of adjacent or separated lines. The simultaneous exposure line may not be strictly a straight line as long as it can be regarded as a linear shape. In addition, exposure and readout on the simultaneous exposure line do not have to be exactly the same, and a deviation corresponding to the time difference between exposure and readout with the next line may exist between image sensors on the simultaneous exposure line. .

  As described above, the imaging units 301 to 309 have been described only with reference to FIG. 4, but the detailed design and configuration of the imaging units 301 to 309 may not be the same, and each forms an image of a subject and acquires image data. Any configuration can be used as long as it has a function.

  Next, an arrangement example of the imaging units 301 to 309 will be described with reference to FIG.

  Each of the imaging units 301 to 309 is stored in a cylindrical casing having the optical axis direction of the imaging optical system as an axis, and is arranged in a grid pattern on the casing 501 of the multi-viewpoint imaging apparatus. Further, the optical axes of the imaging optical systems of the respective imaging units are arranged so as to be in the same direction. FIG. 5 (a) is a schematic view of the multi-viewpoint imaging device viewed from the front. FIG. 5B is a schematic diagram of the multi-viewpoint imaging device viewed from the left in FIG. FIG. 5C is a schematic view of the multi-viewpoint imaging device viewed from above in FIG. Here, in order to clarify the relationship between each figure, the right-handed coordinate axes are shown in the figure.

  In the present embodiment, the number of image pickup units is nine. However, any number of image pickup units may be used as long as image pickup from a plurality of viewpoints is possible. The arrangement shown in FIG. 5 is merely an example, and any arrangement may be used as long as the arrangement of each viewpoint can be known from the design or measurement result.

Next, a description method of viewpoint arrangement information used in the following description will be described with reference to FIG. Here, an imaging unit that acquires an image of each viewpoint is considered as a simple perspective projection camera. As the coordinate system, an imaging unit coordinate system, a multi-viewpoint imaging unit coordinate system, and a world coordinate system are used, and the arrangement of viewpoints is expressed based on these relationships. The relationship between the coordinate axes is a right-handed system. 6 depicts the origin of the n-th image pickup section coordinate system O _n, origin O _M multi-view image capturing unit coordinate system, the origin of the world coordinate system as O _W.

The position of the origin of the n-th imaging unit coordinate system viewed from the multi-view imaging unit coordinate system is represented by a matrix T _Mn of 3 rows and 1 column. Further, when the coordinate axes of the multi-viewpoint imaging unit coordinate system are rotated by the 3 × 3 rotation matrix R _Mn with the origins overlapped, the coordinate axes of the nth imaging unit coordinate system overlap. At this time, the conversion from the position X _n represented by the n-th imaging unit coordinate system of a certain point to the position X _M represented by the multi-view imaging system is expressed by Equation 1. _Xn and _XM are matrices of 3 rows and 1 column, where the first row represents the x component, the row represents the y component, and the third row represents the z component.

The position of the origin of the multi-viewpoint imaging unit coordinate system from the world coordinate system is represented by a 3 × 1 matrix T _WM . Further, when the coordinate axes of the world coordinate system are rotated by the 3 × 3 rotation matrix R _WM with the origins overlapped, the coordinate axes of the multi-view imaging unit coordinate system are overlapped. In this case, the represented by multi-view imaging unit coordinate system of a point position X _M, conversion to a position X _W expressed in the world imaging system is as shown in Equation 2. X _M and X _W are 3 × 1 matrices, where the first row represents the x component, the row represents the y component, and the third row represents the z component.

Each imaging unit is fixed to the multi-viewpoint imaging apparatus, and R _Mn and T _Mn are not changed during imaging. Also, R _Mn and T _Mn are known from the design or measurement results.

On the other hand, R _WM and T _WM change during imaging due to camera shake or the like. In the present invention, changes during imaging of R _WM and T _WM are estimated by the motion estimation unit.

  The expression shown here is an example, and any expression may be used as long as information about the arrangement of each imaging unit and the multi-view imaging apparatus is given.

<Configuration of motion estimation unit>
Next, a configuration example of the motion estimation unit 214 will be described using the block diagram of FIG.

  The multi-view image acquisition unit 701 acquires the multi-view image captured by the multi-view image capturing unit 213 via the bus 210.

  The viewpoint arrangement information acquisition unit 703 acquires viewpoint arrangement information, which is preset information and indicates the arrangement and characteristics of each viewpoint constituting the multi-viewpoint imaging unit 213, via the bus 210.

  The exposure control information acquisition unit 702 is information set in advance, and acquires exposure control information such as the direction of the simultaneous exposure line, the scanning order, and the scanning speed of each imaging unit via the bus 210.

  Although the details will be described later, the exposure time calculation unit 705 uses the position on the image as an input, and calculates the time when the pixel at the position is exposed using the exposure control information.

  As will be described in detail later, the corresponding point search unit 704 obtains corresponding points between the images of a plurality of viewpoints constituting the multi-viewpoint image. The corresponding point search unit 704 uses the viewpoint arrangement information acquired by the viewpoint arrangement information acquisition unit 703 and the information about the exposure time calculated by the exposure time calculation unit 705 in order to improve the accuracy and efficiency of the corresponding point search.

  Note that the corresponding point search may be performed without using these two pieces of information. Corresponding point search section 704 connects points on the image of a plurality of viewpoints as a set of corresponding points. The number of points to be connected is two or more and can be any number as long as it is less than the number of imaging units. The corresponding point search unit 704 generates a plurality of sets of such corresponding points and outputs them to the motion parameter calculation unit 706. Corresponding point search section 704 passes the coordinates of the points constituting the corresponding points to exposure time calculation section 705, and acquires the exposure times of the points constituting the corresponding points. The exposure time information is also associated with the corresponding point information and output to the motion parameter calculation unit 706.

  Although the details will be described later, the motion parameter calculation unit 706 calculates parameters representing the motion of the multi-viewpoint imaging device from the corresponding point information generated by the corresponding point search unit 704 and the viewpoint arrangement information acquired by the viewpoint arrangement information acquisition unit 703. calculate. The calculated parameters and the corresponding point information used for the calculation are output to the bus 210.

  Next, the flow of motion estimation processing will be described with reference to FIG.

  In step S801, the multi-view image acquisition unit 701 acquires a plurality of images as multi-view images.

  Next, in step S802, the viewpoint arrangement information acquisition unit 703 acquires information about the arrangement of each viewpoint corresponding to each multi-viewpoint image.

  In step S803, the exposure control information acquisition unit 702 acquires the exposure control information at the time of each imaging that acquired the multi-viewpoint image.

  In step S804, although the details will be described later, the corresponding point search unit 704 and the exposure time calculation unit 705 cooperate to search for corresponding points between the plurality of images acquired in step S801.

  In step S805, although details will be described later, the exposure time calculation unit 705 calculates the time at which the position of the point on each image constituting the corresponding point searched in step S804 is exposed.

  In step S806, the motion parameter calculation unit 706 estimates a parameter representing motion during imaging of the multi-viewpoint imaging device that acquired the multi-viewpoint image from the corresponding point information acquired in step S804 and the exposure time acquired in step S806. .

  In step S807, the motion parameter calculated by the motion parameter calculation unit 706 is output.

<Exposure time calculation method>
A method for calculating the exposure time will be described with reference to FIG.

  FIG. 9 shows an image for which the exposure time is calculated. On the image shown in FIG. 9, there is a time calculation point p. Note that the normal vector n of the simultaneous exposure line, the scanning speed vector v of the simultaneous exposure line, and the scanning start position s are set based on the design of the image sensor and the control information of the image sensor at the time of exposure. Is used. Here, when the simultaneous exposure line and the scanning direction are orthogonal to each other, there is a relationship represented by Expression 3 between n and v.

  Further, when the time at which the time calculation point p is exposed is t, the relationship of Equation 4 is established as is apparent from FIG.

  Here, p, n, s, and v are two-dimensional vectors.

  By solving Equation 4 for t, the exposure time of the time calculation point p can be calculated.

  Note that the exposure time may be obtained using the correspondence between the pixel position and the exposure time calculated in advance without performing the calculation using the exposure control information.

<Configuration of corresponding point search unit>
A configuration example of the corresponding point search unit 704 will be described with reference to the block diagram of FIG.

  The feature point extraction unit 1001 performs feature point extraction at the first viewpoint on each viewpoint image input from the multi-viewpoint image acquisition unit 701. Harris corner detection or the like is a typical feature point extraction method, but any method may be used. For example, the feature point extraction method is described in detail in Chapter 4 of Non-Patent Document 4. The corresponding point search unit 704 connects points on the image of each viewpoint as a set of corresponding points, and the feature points extracted by the feature point extraction unit 1001 are candidates.

  The epipolar line calculation unit 1002 calculates the epipolar line on the image corresponding to the second viewpoint, using the viewpoint arrangement information input from the viewpoint arrangement information acquisition unit 703. The corresponding point search unit 704 connects points for every two viewpoints. At this time, the image of one viewpoint is set as a reference image, and the image of the other viewpoint is set as a corresponding point search target image. The epipolar line calculation unit 1002 calculates an epipolar line on the corresponding point search image of a certain feature point on the reference image. Note that the exposure time calculation unit 705 calculates the exposure time of the feature point on the reference image and the exposure time of each point on the corresponding epipolar line.

  The allowable deviation amount calculation unit 1003 may use the exposure time calculated by the exposure time calculation unit 705 to how much the point on the corresponding point search image corresponding to the feature point on the reference image deviates from the epipolar line. An allowable deviation amount serving as an index is calculated. If the multi-view imaging device is stationary during imaging, the point on the corresponding point search image corresponding to the feature point on the reference image should be on the epipolar line, but a shift occurs when the multi-view imaging device moves. Since the deviation generated between the feature point on the reference image and the point on the corresponding point search target image should be large if the exposure time difference is large, the allowable deviation amount is set as a monotonically increasing function for this exposure time difference. In addition, since the ease of displacement varies depending on the position on the image and the viewpoint, it may be calculated in consideration of the coordinates of the point on the corresponding point search image and information on the viewpoint.

  As an example, an example in which the coefficient d is used to calculate using Equation 5 is shown.

Here, x _n is the coordinate of the feature point on the reference image, and x ′ is the coordinate of a certain point on the epipolar line on the corresponding point search target image. t (x _n ) and t (x ′) are respective exposure times. d is a constant indicating how much the point on the corresponding point search target image moves per unit time due to camera shake.

  The search range calculation unit 1004 uses the allowable deviation amount calculated by the allowable deviation amount calculation unit 1003 to calculate a range in which a point corresponding to the feature point on the reference image is searched on the corresponding point search target image.

Here, the concept of setting the search range will be described with reference to FIG. FIG. 11 shows an example in which the allowable deviation amount is calculated using Expression 5, and the simultaneous exposure line and the scanning direction are orthogonal to each other. Here, when the coordinate in the scanning direction of the feature point on the reference image is x _{n and the} exposure time is t (x _n ), the coordinate in the scanning direction on the corresponding point search image having the same exposure time is x ′. _{Let n} be. When the coordinate in the scanning direction is x ′ _n , since the allowable deviation amount is 0, no deviation occurs from the epipolar line. Since the allowable deviation amount of the point on the epipolar line whose coordinate in the scanning direction is x ′ is d | t (x _n ) −t (x ′) |, the search range such as a circle shown in FIG. Will have. The search range can be determined by scanning the circle on the epipolar line and setting the region included in the circle as the search range.

  If the scanning speed is constant and the allowable deviation is proportional to the exposure time difference, the boundary of the search range is a straight line, so it is sufficient to know the exposure time for only two points on the epipolar line. In practice, a deviation from the epipolar line also occurs due to manufacturing or measurement errors, and therefore a certain margin is provided in the search range.

  Corresponding point candidate selection unit 1005 selects, as point candidates corresponding to the feature points on the reference image, those included in the search range calculated by search range calculation unit 1004 from the feature points on the corresponding point search target image. To do.

  The feature point associating unit 1006 associates the feature points on the reference image with the feature points as the corresponding point candidates on the corresponding point search target image selected by the corresponding point candidate selecting unit 1005. Here, among the feature points of the corresponding point candidates, the feature point having the highest degree of matching with the feature point on the reference image is associated. Here, the degree of coincidence refers to the distance between feature amounts calculated using pixels near the feature points. A typical example of the method for calculating the degree of coincidence is the sum of absolute differences between the image near the feature point and the image near the feature point of the corresponding point candidate. However, any method may be used. . The calculation of the feature amount and the calculation method of the distance are described in detail in Chapter 4 of Non-Patent Document 4.

  Each associated feature point is also associated with the exposure time calculated by the exposure time calculation unit 705, and these are output as corresponding point information.

  Next, the corresponding point search process flow will be described with reference to FIG.

  In step S1201, the corresponding point search unit 704 acquires a plurality of images as a multi-view image from the multi-view image acquisition unit 701.

  In step S1202, the corresponding point search unit 704 acquires arrangement information at a plurality of viewpoints used for image capturing from the viewpoint arrangement information acquisition unit 703.

  In step S1203, the feature point extraction unit 1001 performs feature point extraction on the image acquired in step S1201.

  In step S1204, the feature point extraction unit 1001 selects one reference image from the images acquired in step S1201.

  In step S1205, the feature point extraction unit 1001 selects one unprocessed image as a corresponding point search target image from images other than the reference image among the images acquired in step S1201.

  In step S1206, one unprocessed feature point is selected from the feature points on the reference image extracted in step S1203.

  In step S1207, the epipolar line calculation unit 1002 uses the viewpoint arrangement information acquired in step S1202 to generate the epipolar line on the corresponding point search target image selected in step S1205 with reference to the feature point selected in step S1206. calculate.

  In step S1208, the exposure time calculation unit calculates the exposure time of the feature point selected in step S1206 and each point on the epipolar line calculated in step S1007.

  In step S1209, the allowable deviation amount calculation unit 1003 calculates the allowable deviation amount of each point on the epipolar line calculated in step S1007 from the exposure time calculated in step S1208.

  In step S1210, the search range calculation unit 1004 calculates a search range from the allowable deviation amount calculated in step S1209 and the epipolar line calculated in step S1207.

  In step S1211, the corresponding point candidate selection unit 1005 uses the search range calculated in step S1210 to select a corresponding point candidate from the feature points of the corresponding point search target image extracted in step S1203.

  In step S1212, the feature point associating unit 1006 selects a feature point having the highest degree of coincidence with the feature point selected in step S1206 from the corresponding point candidates selected in step S1211, and associates these feature points.

  In step S1213, the exposure time calculation unit 705 calculates the exposure times of the two feature points associated in step S1212, and the feature point association unit 1006 associates them with the feature points.

  In step S1214, it is determined whether all feature points on the reference image have been processed. If there are unprocessed feature points, the process returns to step S1206.

  In step S1215, it is determined whether all images other than the reference image have been processed as corresponding point search target images. If there is an unprocessed image, the process returns to step S1205.

  In step S1216, the associated feature point associating unit 1006 outputs a set of coordinates and exposure times of all the associated points as corresponding point information.

  In this embodiment, feature points are extracted for both of the two images that connect the points. However, feature points are extracted only for one of the images, and deformation / movement parameters such as block matching are extracted. The points may be connected by an optimization method. Further, although the method for limiting the search range is effective for improving accuracy and efficiency, it is not always necessary to limit the search range. In this embodiment, one of a plurality of viewpoint images is used as a reference image, and a corresponding point in another viewpoint image is obtained for a feature point on the reference image. Correspondence may be performed. Alternatively, the reference image and the corresponding point search target image may be interchanged to connect the points, and processing for improving accuracy may be performed by checking consistency.

<Configuration of motion parameter calculation unit>
Next, the principle of motion parameter calculation will be described.

  In the present invention, by using a plurality of corresponding points between images of two different viewpoints, the motion of the multi-viewpoint imaging apparatus during imaging is estimated. Here, the corresponding points are the coordinates of points at two different viewpoints, and the information on the exposure time is associated with each other. The feature of the present invention is that the motion can be estimated even from an image that is distorted by the motion during imaging by using not only the coordinates of the point but also the exposure time. Further, in the present invention, the motion of the multi-viewpoint imaging device during imaging is expressed by a small number of parameters as a function of time, thereby enabling continuous motion estimation from a finite number of corresponding points.

Of the two viewpoints associated with the k-th corresponding point, one corresponds to the _Nk- th imaging unit and the other corresponds to the _N′k- th imaging unit. Also, of the two coordinates associated with the corresponding point, N _k-th m _k the homogeneous coordinates of points captured by the imaging _unit, N 'a _k-th homogeneous coordinates of points captured by the imaging unit m' _k . Further, the exposure time of the pixel corresponding to m _k is t _k , and the exposure time of the pixel corresponding to m ′ _k is t ′ _k . X _kw represents a point in space corresponding to m _k and m ′ _k in world coordinates, X _kNk represents in the N _k th imaging unit coordinate system, and N ′ _k th imaging unit coordinate system Assuming that the expressed value is X _{kN ′ k} , the following equations 6 and 7 are obtained using the elements shown in FIG.

Since R _WN′k , R _WN′k , T _WN′k , and T _WN′k are fixed with respect to the multi-viewpoint imaging apparatus, they are unchanged. Since R _WM and T _WM change depending on the movement of the multi-viewpoint imaging apparatus during imaging, they are functions of time.

Expression 6 and Expression 7 can be expressed as Expression 12 and Expression 13 by using R _WNk , R _WN′k , T _WNk , and T _WN′k shown in Expression 8, Expression 9, Expression 10, and Expression 11, _respectively .

式12、式13より、X_kNkからX_{kN' k}への変換を求めると、式14のようになる。

When the conversion from X _kNk to X _{kN ′ k} is _obtained from Expression 12 and Expression 13, Expression 14 is obtained.

Further, when Expression 14 is expressed using R _N′kNk and T _N′kNk shown in Expressions 15 and 16, Expression 17 is obtained.

Here, O _Nk is an optical center of the N _k-th image pickup unit, O _N'k is an optical center of the N _'k-th image pickup section, consider the point X _k in space. At N _'k-th image pickup unit coordinate system, represents the vector from O _Nk vector in the direction of X _k, the vector from O _N'k in the direction of X _k, from O _N'k in the direction of O _{Nk ΛR N′kNk} X _kNk , ξX _kN′k , and ηT _N′kNk , respectively. Here, λ, ξ, and η represent appropriate coefficients. Since the three vectors are vectors on the same plane, Expression 18 is established.

  Here, x represents an outer product of vectors, and · represents an inner product of vectors.

If the internal parameter matrix of the N _k- th imaging unit is A _Nk and the internal parameter matrix of the N ' _k- th imaging unit is A _N'k , X _kNk and X _kN'k are as shown in Equation 19 and Equation 20, respectively. Can be expressed as

  Here, λ ′ and ξ ′ represent appropriate coefficients. Expression (21) is obtained from Expression (18), Expression (19), and Expression (20).

Here, [] _× represents the calculation of the outer product as a 3 × 3 distortion symmetric matrix. Equation 21 is generally intended called epipolar constraint, it is possible to obtain a constraint on the relationship of the _k-th image pickup unit 'N in _k' N _k-th image pickup unit and the time t at time t _k by this equation.

A method for representing the movement of the imaging unit with parameters will be described. R _MNk , R _MN'k , T _MNk , T _MN'k , A _Nk , and A _N'k are fixed and unchanged during imaging, and these are calculated in advance from design values or measured values. To do.

R _WM (t) and T _WM (t) each have three degrees of freedom. If one degree of freedom is expressed by M parameters, R _WM (t, p ₁₁ , p ₁₂ , ..., p _1M , p ₂₁ , p ₂₂ , ..., p _2M , p ₃₁ , p ₃₂ , ..., p _3M ),
T _WM (t, p ₄₁ , p ₄₂ ,..., P _4M , p ₅₁ , p ₅₂ ,..., P _5M , p ₆₁ , p ₆₂ ,..., P _6M ).

Using the M parameters and time t R _WM, as an example of expressing the T _WM has the formula 22, wherein 23 represents the three rotation angle R _WM as in equation 24, with the three components of T _WM There is a method of expressing by a polynomial of t each having a coefficient of p _nm . Since R _WM and T _{WM at} time t = 0 may be set in any way, the 0th-order coefficient may be set to an appropriate constant when expressed by a polynomial expression.

  If the number of parameters is 6M, parameters can be calculated if there are 6M or more constraints as shown in Equation 21, that is, if there are 6M or more corresponding points.

An error function U _k representing a deviation from the equation 21 is defined as in the equation 25, and U _obtained by adding these to all corresponding points is set as in the equation 26. A parameter representing motion is calculated by _obtaining a parameter p _est that minimizes U as shown in Equation 27.

  In Equation 25, Equation 26, and Equation 27, all parameters are represented by a vector p. Examples of minimization methods include Newton method and Levenburg-Muarqardt method, but any method can be used.

  Since the corresponding points obtained by the corresponding point search often include erroneous correspondences, it is possible to further improve the accuracy by combining the above-described motion parameter calculation method with a technique for removing the erroneous correspondences. Examples of methods for eliminating the false correspondence include robust estimation such as RANSAC and the minimum median method, but any method may be used.

  Note that the motion expression method and the error definition shown in this embodiment are merely examples. If the motion of the multi-viewpoint imaging device is expressed by a time t and a finite number of parameters, and the parameters are calculated using the coordinates of the corresponding points and the exposure time, the arrangement and internal parameters of the viewpoint, and the relational geometric constraints between viewpoint Any form is acceptable. For example, all nine components of the rotation matrix may be handled independently, and a constraint may be added so that the rotation matrix becomes orthonormal.

  In this embodiment, the aberration of the optical system of the imaging unit is not taken into account, but this may be corrected by image processing after imaging, and the influence of the aberration may be taken into account when calculating the exposure time. Further, the influence of aberration may be taken into account when calculating the motion parameter.

  Next, a configuration example of the motion parameter calculation unit 706 will be described with reference to FIG.

Corresponding point selection section 1301 selects the corresponding points obtained by corresponding point search section 704 to be used for motion parameter calculation. For example, the output of the corresponding point searching unit 704, may include the corresponding point m _B on m _A and the viewpoint B point on the viewpoint A, the corresponding point m _C on point m _A and the viewpoint C on viewpoint A , it is possible to generate a corresponding point m _C on newly m _B and the viewpoint a point on the viewpoint B C. The corresponding point search unit 704 selects a plurality of point correspondences between two viewpoints used for estimation while determining what combination is taken.

  Here, when importance is attached to accuracy, all combinations of corresponding points are set as selection candidates. On the other hand, if importance is attached to the processing speed, the minimum number of combinations in which one point is included in at least one set of correspondences is selected. However, selection criteria may be generated based on any criteria. In this embodiment, since robust estimation is performed, a certain number is selected at random from correspondences that are candidates for selections acquired and generated.

  The evaluation formula generation unit 1302 generates an evaluation formula corresponding to Formula 26 from the corresponding points selected by the corresponding point selection unit 1301, the viewpoint arrangement obtained from the viewpoint arrangement information acquisition unit 703, and internal parameters.

  The initial parameter generation unit 1304 generates an initial value of a parameter for obtaining a motion parameter by optimization. If the three components of the rotation angle of the rotation matrix and the translation vector are described by polynomial expansion at time t as shown in Equation 22, Equation 23, and Equation 24, the initial values of all parameters are set to zero.

  The optimization calculation unit 1303 performs optimization corresponding to Equation 26 based on the evaluation formula generated by the evaluation formula generation unit 1302 and the initial parameter generated by the initial parameter generation unit 1304, and calculates a motion parameter.

  The estimated accuracy evaluation value calculation unit 1305 calculates an estimated accuracy evaluation value for the set of corresponding points selected by the corresponding point selection unit 1301. If the robust estimation to be used is RANSAC, the number of corresponding points in which the error represented by Equation 25 is equal to or less than the threshold among the selected corresponding points is calculated as an estimated accuracy evaluation value. In the case of the minimum median method, the error of the corresponding point selection candidate is calculated by Equation 20, and the median value is calculated.

  The optimal parameter selection unit 1306 holds the motion parameter calculated by the optimization calculation unit 1303 and the estimation accuracy evaluation value calculated by the estimation accuracy evaluation value calculation unit 1305. After the repeated estimation is performed, the estimation accuracy evaluation value is Outputs the motion parameter when it was the best.

  Next, the flow of motion parameter calculation processing will be described using FIG.

  In step S1401, the motion parameter calculation unit 706 acquires, from the viewpoint arrangement information acquisition unit 703, arrangement information of a plurality of viewpoints used for capturing a multi-viewpoint image.

  In step S1402, the motion parameter calculation unit 706 acquires the coordinates of the corresponding point and the exposure time of the corresponding point as corresponding point information from the corresponding point search unit 704.

  In step S1403, the corresponding point selection unit 1301 generates corresponding points on two different viewpoints based on the corresponding point information acquired in step S1402, and randomly selects N sets of them.

  In step S1404, the evaluation formula generation unit 1302 calculates an error with respect to the geometric consistency of the parameter representing the movement of the multi-viewpoint imaging unit from the viewpoint arrangement information acquired in step S1401 and the corresponding point selected in step S1403. An expression (corresponding to Expression 26) is generated.

  In step S1405, the initial parameter generation unit 1304 generates an initial value for performing optimization corresponding to Expression 27 on the parameter representing the motion of the multi-viewpoint imaging unit.

  In step S1406, the optimization calculation unit 1303 uses the initial value generated in step S1405 to minimize the evaluation expression generated in step S1404 and obtain the motion parameter at that time.

  In step S1407, the estimated accuracy evaluation value calculator 1305 calculates estimated accuracy evaluation values for the N pairs of corresponding points selected in step S1403, using the motion parameter obtained in step S1406.

  In step S1408, the optimum parameter selection unit 1306 stores the estimated accuracy evaluation value calculated in step S1407 and the motion parameter calculated in step S1406.

  In step S1409, it is determined whether the processing from step S1403 to step S1408 has been repeated M times. If not repeated, the process returns to step S1403.

  In step S1410, the optimum parameter selection unit 1306 selects a motion parameter that minimizes the estimated accuracy evaluation value stored in step S1408.

  In step S1411, the motion parameter calculated by the motion parameter calculation unit 706 is output.

  In the present embodiment, robust estimation by repetition is used. However, a configuration and processing may be used in which the motion parameter is calculated only once without performing robust estimation and is output as it is.

<Configuration of subject distance estimation unit>
Next, a configuration example of the subject distance estimation unit will be described using the block diagram of FIG.

The feature point information acquisition unit 1501 acquires the corresponding point information calculated by the corresponding point search unit 704 via the bus 210. Corresponding point information includes information on the correspondence between points on different viewpoints, the coordinates of the points, and the exposure time. The feature point information acquisition unit 1501 collects points corresponding to the same point in space into one group. For example, the corresponding point m _B on m _A and the viewpoint B point on the viewpoint A, if the point m _C on point m _A and the viewpoint C on viewpoint A corresponding is input, m _A, m _B, m Associate _C.

  The motion information acquisition unit 1502 acquires a motion parameter representing the motion of the multi-viewpoint imaging device calculated by the motion estimation unit 214 via the bus 210.

  A viewpoint arrangement information acquisition unit 1503 acquires the arrangement of each viewpoint and internal parameter information constituting a known multi-viewpoint imaging apparatus from the ROM 202 or the RAM 203 via the bus 210.

  The imaging unit arrangement correction unit 1505 is, for each point constituting the corresponding point acquired by the feature point information acquisition unit 1501, what kind of arrangement the imaging unit that captured the point was at the time when the point was exposed, Calculation is performed using the motion parameter and viewpoint arrangement information.

  The distance calculation unit 1504 converts a point set associated with the feature point information acquisition unit 1501 from the coordinates of the points constituting the set and the viewpoint arrangement calculated by the imaging unit arrangement correction unit 1505 to the associated point set. Calculate the position of the corresponding point in space. The position in space of the points imaged by two or more imaging units whose arrangement and internal parameters are known can be calculated by a method generally known in this technical field (Non-Patent Document 4 Chapter 7). The calculated position of the point is output as subject distance information.

  Next, the flow of subject distance estimation processing will be described using FIG.

  In step S1601, the subject distance estimation unit 215 acquires corresponding point information.

  In step S1602, the subject distance estimation unit 215 acquires viewpoint arrangement information.

  In step S1603, the subject distance estimation unit 215 acquires a parameter representing the movement of the multi-viewpoint imaging apparatus.

  In step S1604, the feature point information acquisition unit 1501 selects one set of unprocessed corresponding points.

  In step S1605, when the imaging unit arrangement correction unit 1505 captures each point constituting the corresponding point from the viewpoint arrangement information acquired in step S1602 and the exposure time information included in the corresponding point information acquired in step S1601, Each of the viewpoint arrangements is calculated.

  In step S1606, the distance calculation unit 1504 estimates the position of the point on the subject from the information on the coordinates on the image included in the corresponding point information acquired in step S1601 and the arrangement information at the time of exposure calculated in step S1605.

  In step S1607, it is determined whether all corresponding points have been processed. If there are any unprocessed corresponding points, the process returns to S1604.

  In step S1608, the subject distance information estimated by the subject distance estimation unit 215 is output.

  In the present embodiment, the subject distance information is calculated using the corresponding point information obtained when calculating the motion parameters of the multi-viewpoint imaging apparatus that is capturing images. However, from the calculated motion parameters and the viewpoint arrangement information, A corresponding point search may be separately performed using a fixed constraint. In addition, although the motion parameter and subject distance information are estimated separately, the three-dimensional position of the point on the subject is used as a parameter, and it is estimated simultaneously by the bundle adjustment method (Non-Patent Document 5) together with the motion parameter. May be.

  As described above, according to the present embodiment, it is possible to estimate the movement of the multi-viewpoint imaging device being imaged from the multi-viewpoint image in which rolling shutter distortion has occurred due to the movement of the multi-viewpoint imaging device being imaged. Further, by performing correction using the estimated motion, it is possible to estimate the subject distance information with high accuracy.

(Embodiment 2)
In the first embodiment, the influence of the rolling shutter distortion is corrected, and the distance of the subject is estimated with high accuracy. In the second embodiment, the effect of rolling shutter distortion is corrected and image composition is performed. Note that description of parts common to the first embodiment is simplified or omitted, and here, differences will be mainly described.

  FIG. 17 is a diagram illustrating the configuration of the image processing apparatus according to the present embodiment. The difference from the image processing apparatus according to the first embodiment is that an image composition unit 1701 is provided instead of the subject distance estimation unit. The image synthesis unit 1701 synthesizes a virtual viewpoint image using the multi-viewpoint image acquired by the multi-viewpoint imaging unit 213, the motion parameter calculated by the motion estimation unit 214, the known viewpoint arrangement information, and the exposure control information. Is what you do.

Next, the principle of virtual viewpoint image composition will be described with reference to FIG. From the motion parameters calculated by the motion estimation unit 214, the viewpoint arrangement information known in advance, and the exposure time for each pixel determined from the exposure information control information, the viewpoint arrangement T _Mn and R _{Mn at the} time when a pixel is exposed Can be requested. The image composition unit 1701 obtains the arrangement of viewpoints for each pixel, projects this onto a focal plane defined in space, and synthesizes a virtual viewpoint image by complementing this.

  Next, a configuration example of the image composition unit 1701 will be described with reference to FIG.

  A focusing distance acquisition unit 1901 acquires the position of the focusing surface input by the user.

  The multi-view image acquisition unit 1902 acquires the multi-view image acquired by the multi-view image capturing unit 213.

  The exposure control information acquisition unit 1903 acquires exposure control information that is known in advance.

  The motion information acquisition unit 1502, the viewpoint arrangement information acquisition unit 1503, and the imaging unit arrangement correction unit 1505 are the same as the subject distance estimation unit 215 of the first embodiment.

  The exposure time calculation unit 1904 calculates the exposure time for each pixel of the multi-viewpoint imaging unit 213.

  The coordinate conversion unit 1905 performs projection on the in-focus surface from the image-capturing unit arrangement for each pixel calculated by the image-capturing unit arrangement correction unit 1505 and the information on the in-focus surface acquired by the in-focus distance acquisition unit. The coordinates of the projected pixel on the focal plane and the pixel value of the pixel acquired from the multi-viewpoint image acquisition unit 1902 are passed to the pixel interpolation unit 1906 and stored. Any projection method may be used.

  The pixel interpolation unit 1906 generates an image by interpolating the projected pixel group when all the pixels have been projected. Any method of interpolation may be used. If the radius of the filter used in the interpolation is large, noise can be reduced, and if it is small, the resolution can be improved.

  Next, the flow of image composition processing will be described with reference to FIG.

  In step S2001, the multi-view image acquisition unit 1902 acquires a plurality of images as multi-view images.

  In step S2002, the viewpoint arrangement information acquisition unit 1503 acquires arrangement information of a plurality of viewpoints constituting the multi-viewpoint imaging apparatus used for capturing the multi-viewpoint image.

  In step S2003, the motion information acquisition unit 1502 acquires a parameter representing the motion of the multi-viewpoint imaging apparatus.

  In step S2004, the exposure control information acquisition unit 1903 acquires exposure control information of a plurality of imaging units constituting the multi-viewpoint imaging apparatus used for imaging.

  In step S2005, the focus distance acquisition unit 1901 acquires the focus distance representing the focus surface as an image composition parameter.

  In step S2006, the exposure time calculation unit 1904 calculates the exposure time for each pixel of the multi-viewpoint image from the exposure control information acquired in step S2004.

  In step S2007, from the motion information acquired in step S2004 and the exposure time calculated in step S2006, the imaging unit arrangement at the time when each pixel is imaged is calculated.

  In step S2008, the coordinate conversion unit 1905 performs coordinate conversion for projection on the in-focus plane from the viewpoint arrangement information for each pixel calculated in step S2007 and the in-focus distance acquired in step S2005.

  In step S2009, the pixel interpolation unit S2006 interpolates the projected pixel group to generate an image.

  In step S2010, the image composition unit 1701 outputs an image.

  In this embodiment, the image is generated by projecting onto the in-focus plane, but each pixel may be regarded as a light beam and the image may be synthesized using a virtual optical system.

  In addition, when the subject is arranged close to a plane (including a plane at infinity), noise reduction and high resolution can be performed by setting the projection surface as the surface on which the subject is arranged. Such processing may be performed by the image composition unit. In this case, in motion estimation, instead of the three-dimensional motion of the multi-viewpoint imaging apparatus, a time change of linear transformation of image coordinates such as a projective transformation matrix and an affine transformation matrix may be estimated.

  As described above, according to the present embodiment, by estimating and correcting the movement of the multi-viewpoint imaging device being imaged from the multi-viewpoint image in which rolling shutter distortion has occurred due to the movement of the multi-viewpoint imaging device being imaged, Image synthesis can be performed with high accuracy.

(Embodiment 3)
In Embodiments 1 and 2, it is assumed that the optical axes of the imaging units constituting the multi-viewpoint imaging unit are parallel. However, when imaging units having the same configuration are arranged in parallel, an image of a point on a distant subject is exposed at the same time. When an image of the same point in space is exposed at the same time, it is clear that t ′ _k = t _k in Equation 21, and from the corresponding point corresponding to the point in the space, the multi-viewpoint imaging device I can't get restraints on movement. Accordingly, there arises a problem that the estimation accuracy of the motion of the multi-viewpoint imaging apparatus is lowered or estimation is impossible.

  Therefore, in this embodiment, an arrangement and control method of viewpoints constituting the multi-viewpoint imaging unit for solving the above problem will be described. Other parts are the same as those in the first embodiment or the second embodiment, and thus description thereof is omitted.

  FIG. 21 (a) is a diagram for explaining a problem that occurs when imaging units having the same configuration are arranged in parallel. The vertical axis is the position on the far away subject, and the horizontal axis is the time. A thin broken line represents a relationship when an image of a point existing at a certain position on the subject is exposed when the multi-viewpoint imaging apparatus is stationary.

  When the multi-viewpoint imaging apparatus moves during imaging, the position of a point exposed at a certain time deviates from a thin broken line such as a thick broken line or a double line. A thick broken line and a double line represent different imaging units. Since the thick broken line and the double line always overlap, information on the movement of the multi-view imaging device cannot be obtained no matter how the corresponding points are acquired.

  FIG. 22A shows an arrangement method in which the simultaneous exposure line and its scanning direction are made different for each imaging unit. In this case, since the relationship between the exposure position and time differs depending on the imaging unit as shown in FIG. 21B, information regarding the movement of the multi-viewpoint imaging apparatus can be obtained. Note that, instead of changing the simultaneous exposure line and its scanning direction, the imaging unit may be rotated about the optical axis.

  FIG. 22B shows that the imaging direction is different for each imaging unit. In this case, since the relationship between the exposure position and the time varies depending on the imaging unit as shown in FIG. 21 (c), information regarding the movement of the multi-viewpoint imaging apparatus can be obtained. As a method of changing the imaging direction, the position of the imaging device relative to the optical axis of the optical system may be shifted for each imaging device, even if the installation direction of the imaging unit is changed.

  FIG. 22 (c) shows that the angle of view is different for each imaging unit. In this case, since the relationship between the exposure position and the time differs for each viewpoint depending on the imaging unit as shown in FIG. 21 (d), information regarding the movement of the multi-view imaging apparatus can be obtained.

  Further, the exposure start time that the imaging control unit 310 instructs the imaging element control unit 411 may be different for each imaging unit. In this case, since the relationship between the exposure position and time differs depending on the imaging unit as shown in FIG. 21 (e), information regarding the movement of the multi-viewpoint imaging apparatus can be obtained.

  The exposure order instructed by the imaging control unit 310 to the imaging element control unit 411 may be different for each imaging unit. In this case, since the relationship between the exposure position and time differs depending on the imaging unit as shown in FIG. 21 (f), information regarding the movement of the multi-viewpoint imaging apparatus can be obtained.

  As described above, according to the present embodiment, it is possible to acquire information on the movement of the multi-viewpoint imaging apparatus even from a distant subject by changing the arrangement, configuration, and control for each imaging unit.

(Other embodiments)
In each of the above embodiments, the case where a multi-viewpoint imaging unit is configured using a plurality of imaging units having a single viewpoint has been described. However, a configuration may be adopted in which the imaging unit divides the single imaging element 407 shown in FIG. 4 into a plurality of regions and provides an optical system corresponding to the divided regions. A plurality of such imaging units may be used.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (15)

Multi-viewpoint image acquisition means for acquiring an image group captured from a plurality of viewpoints;
Viewpoint arrangement information acquisition means for acquiring a relative arrangement of the plurality of viewpoints;
Corresponding point search means for searching for a set of points on the image corresponding to the same point on the subject between the plurality of viewpoints;
Exposure time calculating means for calculating a time at which a point on the acquired image was exposed;
A motion parameter calculating means for estimating a parameter representing motion during imaging of the plurality of viewpoints from the coordinates of the set of points on the image, the exposure time, and the relative arrangement of the plurality of viewpoints; An image processing apparatus. The corresponding point search means
Feature point extracting means for extracting feature points on an image corresponding to a first viewpoint among the plurality of viewpoints;
Epipolar line calculation means for calculating an epipolar line of the extracted feature point on an image corresponding to a second viewpoint of the plurality of viewpoints;
Corresponding to the second viewpoint in which the extracted feature point and the point corresponding to the same point on the subject exist from the epipolar line and the exposure time at the point on the image calculated by the exposure time calculation means The image processing apparatus according to claim 1, further comprising: a search range calculation unit that calculates a region on the image to be processed.   It further comprises subject distance estimation means for estimating the distance of the subject from the parameter representing the motion, the relative arrangement of the plurality of viewpoints, and the time when each point on the acquired image is exposed. 3. The image processing apparatus according to claim 1 or claim 2.   Image synthesis means for synthesizing the image from the acquired image, the parameter representing the motion, the relative arrangement of the plurality of viewpoints, and the time when each point on the image was exposed. 3. The image processing apparatus according to claim 1, wherein the image processing apparatus is characterized in that: Using multiple imaging optics and one or more rolling shutter imaging devices,
A multi-viewpoint imaging unit that captures images from a plurality of viewpoints;
An image processing unit that performs image processing from the captured image;
The image processing unit is the image processing apparatus according to any one of claims 1 to 4,
The multi-viewpoint image acquisition unit constituting the image processing unit acquires an image captured by the multi-viewpoint image capturing unit.   The imaging apparatus according to claim 5, wherein an angle of view of an image captured by the multi-viewpoint imaging unit is different for each viewpoint.   The imaging apparatus according to claim 5, wherein directions of optical axes of a plurality of viewpoints from which the multi-viewpoint imaging unit captures an image are different for each viewpoint.   The imaging apparatus according to claim 5, wherein the orientation in which the imaging elements constituting the multi-viewpoint imaging unit are installed differs for each imaging element.   The imaging apparatus according to claim 5, wherein an exposure start time of an imaging element constituting the multi-viewpoint imaging unit is different for each imaging element.   The imaging apparatus according to claim 5, wherein an exposure order of imaging elements constituting the multi-viewpoint imaging unit is different for each imaging element. A multi-viewpoint image acquisition step of acquiring a group of images captured from a plurality of viewpoints;
A viewpoint arrangement information acquisition step of acquiring a relative arrangement of the plurality of viewpoints;
A corresponding point search step of searching for a set of points on the image corresponding to the same point on the subject between the plurality of viewpoints;
An exposure time calculating step for calculating a time at which a point on the acquired image is exposed;
A motion parameter calculating step of estimating a parameter representing a motion during imaging of the plurality of viewpoints from the coordinates of the set of points on the image, the exposure time, and the relative arrangement of the plurality of viewpoints; ,
An image processing method. The corresponding point search step includes:
A feature point extracting step of extracting a feature point on the image corresponding to a first viewpoint among the plurality of viewpoints;
An epipolar line calculating step of calculating an epipolar line of the extracted feature point on an image corresponding to a second viewpoint of the plurality of viewpoints;
Corresponding to the second viewpoint in which the extracted feature point and the point corresponding to the same point on the subject exist from the epipolar line and the exposure time at the point on the image calculated by the exposure time calculation step The image processing method according to claim 11, further comprising: a search range calculation step of calculating a region on the image to be performed.   A subject distance estimating step of estimating a subject distance from the parameter representing the motion, the relative arrangement of the plurality of viewpoints, and the time when each point on the acquired image is exposed; The image processing method according to claim 11 or 12. An image synthesizing step of synthesizing the image from the acquired image, a parameter representing the motion, a relative arrangement of the plurality of viewpoints, and a time when each point on the image is exposed; The image processing method according to claim 11 or claim 12.   The program for functioning a computer as an image processing apparatus of any one of Claim 1 to 4.

Images