JP2004077262A - Three-dimensional imaging apparatus and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To accurately measure the distance of a three-dimensional object after relieving occlusion. <P>SOLUTION: A mobile projection system/imaging system unit 10 and a second imaging system 20 are used; a coded pattern is projected to the three-dimensional object; imaging is made by the two imaging systems; and the distance of the three-dimensional object is measured. The mobile projection system/imaging system unit 10 and the second imaging system 20 is separated from a body 30, are retained by movement retention means 14, 21 each, and can be moved freely. Additionally, when the mobile projection system/imaging system unit 10 and the second imaging system 20 travel, the amount of displacement in the three-dimensional position between them and the amount of displacement in respective postures in the mobile projection system/imaging system unit 10 and the second imaging system 20 are detected, thus correcting calibration parameters and hence a measurement distance to a target without a recalibration. <P>COPYRIGHT: (C)2004,JPO

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to an image capturing apparatus and a method for obtaining distance information along with luminance information by a stereo method using a plurality of cameras. By making the layout of each camera free, it is possible to expand a measurement area and improve usability. Device and method.
[0002]
[Prior art]
Techniques for measuring the shape of the target object are roughly classified into a passive technique (stereo image method, shape from X) and an active technique (light time-of-flight measurement method, stereo image method). The difference between the passive method and the active method is whether or not the object is irradiated with some energy. The stereo image method that exists in both methods is a method in which the same object is imaged by a plurality of cameras, corresponding points in the image are extracted, and a distance image is calculated by triangulation. FIG. 9 shows a three-dimensional image capturing apparatus using a passive stereo image method. In general, the stereo image method has an advantage that a three-dimensional image pickup device can be realized without using a special device.On the other hand, occlusion, which is an area that cannot be measured, requires camera calibration, and the process is complicated. There are disadvantages such as being always present.
[0003]
For the calibration, if the layout such as the distance between the cameras or the posture changes, it is necessary to perform the camera calibration again, and this has caused a more complicated process. Therefore, in a three-dimensional image pickup device that uses the stereo image method as a measurement method, the positional relationship of the image pickup system, image processing system, and electric system, which are the components in the device, is basically fixed, and it is made in one housing. Is embedded. As a result, the size of the device body is large and the mobility is poor, and the camera cannot be moved individually. Therefore, it is not suitable for measurement from an arbitrary viewpoint according to the shape and size of the target object, and the measurement area is limited. was there.
[0004]
Occlusion occurs when an object with irregularities is imaged by two cameras because the two cameras have different viewing directions, and there is an area that can be imaged by one camera but an area that cannot be imaged by the other camera. I do. If the individual cameras can be freely moved according to the shape of the object, the occlusion area can be considerably reduced. If the position and angle of the camera can be freely changed, a distance image of the object can be easily acquired without moving the apparatus body. However, for the above-described reasons, since each component in the three-dimensional image capturing apparatus is fixed, these requirements cannot be realized at present.
[0005]
The passive stereo image method has a problem that the measurement accuracy is extremely reduced if the target object has no feature points. In Japanese Patent Application Laid-Open No. 2000-9442, in order to solve this problem, encoding light is projected onto the object by a light projecting system, and an optical image on the object is taken from the optical axis direction of the light projecting system. A method of capturing an optical image on an object from a direction different from the optical axis direction of the system is shown (see FIG. 10). Since the encoded light pattern is projected on the object, even if the object itself has no feature points, the measurement accuracy does not decrease and the object is projected from the optical axis direction of the light projecting system. Is an excellent method that can prevent deterioration of the accuracy of searching for a corresponding point due to the reflectance of the target object by capturing the optical image of. However, even in the three-dimensional image pickup device proposed by this method, since the components in the device are fixed, the above-mentioned problem cannot be solved at present.
[0006]
[Problems to be solved by the invention]
In view of the above, the present invention provides a three-dimensional image capturing apparatus and method which have good mobility, can perform measurement from an arbitrary viewpoint, can reduce an occlusion region, have high measurement accuracy, and can eliminate the complexity of calibration. With the goal.
[0007]
[Means for Solving the Problems]
According to one aspect of the present invention, in order to achieve the above-described object, a light projecting system that projects an encoded pattern, and the encoding method is performed at substantially the same principal point as the light projecting system from substantially the same optical axis direction. A first imaging system that captures a pattern projection image, a second imaging system that captures the encoded pattern projection image from a direction different from the optical axis direction of the light projection system, and the first and second imaging systems In the three-dimensional image capturing apparatus that calculates a distance image using the calibration parameters of the camera in the above, the light projecting system and the first image capturing system are configured as one movable light projecting system / imaging system unit, A movable projection / imaging system unit and the second imaging system are freely movable with respect to each other, and a three-dimensional position displacement amount detecting means for detecting a three-dimensional position displacement amount between the imaging systems; Posture displacement detecting means for detecting the displacement of the imaging system in the posture direction. It has to so that.
[0008]
In this configuration, the movable projection system / imaging system unit and the second imaging system are movable relative to each other (since at least one moves relative to the object), so that the occlusion area is reduced. Can be.
[0009]
Only one of the movable projection / imaging system unit and the second imaging system may be moved with respect to the target, or both may be moved with respect to the target.
[0010]
Calibration parameter storage means for storing calibration parameters obtained by calibration performed in each of the movable projection system / imaging system unit and the camera in the second imaging system is provided, and distance measurement can be performed accurately. is there.
[0011]
Further, the three-dimensional position displacement amount between the imaging systems obtained by the three-dimensional position displacement amount detection means and the posture displacement amount detection means and the displacement amount in the posture direction of each imaging system are stored in the calibration parameter storage means. Correcting the calibration parameters eliminates the need for recalibration.
[0012]
Note that the present invention can be realized not only as an apparatus or a system invention, but also as a method.
[0013]
The above aspects of the present invention and other aspects of the present invention are set forth in the following claims, and are described in detail below using examples.
[0014]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, embodiments of the present invention will be specifically described with reference to the drawings.
[0015]
FIG. 1 shows a configuration diagram of an embodiment of the present invention. In FIG. 1, the three-dimensional image capturing apparatus according to the present embodiment has a basic configuration including a movable light projecting / imaging system unit 10, a second imaging system 20, a main body 30, and a personal computer 40.
[0016]
The movable projection system / imaging system unit 10 includes a first imaging system 11, a projection system 12, and a half mirror 13. The half mirror 13 causes the projection system 12 and the imaging system 11 to have a substantially principal point. The optical axes are adjusted and fixed to substantially the same optical axis, and the positional relationship between the two does not change. The encoded pattern is projected onto the object 50 from the light projecting system 12. The encoded pattern is, for example, as shown in FIG. 2, and slit patterns having different luminances are arranged so as not to have the same repetition.
[0017]
The main body 30 includes, as components, an image processing system 31, an electric system 32, and a calibration parameter storage unit 33 for storing calibration parameters obtained at the time of calibration of the camera. Have been. The personal computer 40 is a device for controlling the main body 30 and displaying the obtained three-dimensional image.
[0018]
The movable light projecting / imaging system unit 10 and the second imaging system 20 are separated from the main body 30 by a cable, held by moving holding means 14 and 21, respectively, and can move freely. The movement of the movable light projecting / imaging system unit 10 and the second imaging system 20 may be performed manually, or a predetermined driving unit may be used.
[0019]
The first imaging system 11 in the movable light projecting / imaging system unit 10 includes a three-dimensional position displacement amount detecting means (indicated by reference numeral 15 in FIG. 3) and a posture displacement amount detecting means (indicated by reference numeral 16 in FIG. 3). ) Is provided. The second imaging system 20 includes a posture displacement amount detection unit (indicated by reference numeral 22 in FIG. 3). A diagram for explaining these is shown in FIG.
[0020]
In FIG. 3, a three-dimensional position displacement detecting unit 15 is a detection device that detects a displacement of a three-dimensional position between the two imaging systems 11 and 20. In this embodiment, as a detection method, a TOF (Time of Flight) method in which a spot light scan of an LED or LD is combined with a two-dimensional array of light detectors, or a spot light scan of an LED or LD and a PSD (Position Sensing Device) are used. A combined triangulation method was used.
[0021]
In the detection device based on the TOF method, as shown in FIG. 4, the first imaging system 11 includes an LED or LD 151 as a light source and a two-dimensionally arranged photodetector 152 (PSD 153 in a triangulation method described later). . The pulsed spot light of the LED or LD 151 is scanned up, down, left, and right. The scanning of the spot light can be performed using, for example, two sets of galvanometer mirrors 154 and 155 as shown in FIG. Among the scanning light, the pulse light reflected by the reflection surface 23 on the second imaging system 20 is detected by the photodetector 152 provided in the first imaging system 11, and the pulse light is detected from the difference between the pulse irradiation time and the reflected light detection time. The displacement of the three-dimensional position between the two imaging systems was obtained from the calculated distance value and the pulse scanning angle.
[0022]
The detection device based on the triangulation method in which the LED or the PD and the PSD are combined also includes the LED or the LD 151 and the PSD 153 as the light source in the first imaging system. , 155 is scanned vertically and horizontally. Of the scanning light, the spot light reflected by the reflection surface 23 on the second imaging system 20 was detected by the PSD 153 provided in the first movable imaging system 11. The position at which the spot light was detected by the PSD 153 was accurately known, and the displacement of the three-dimensional position between the two imaging systems was determined by the principle of triangulation. Further, instead of the PSD 153, a CCD sensor or a CMOS sensor, which is an area sensor, may be used.
[0023]
The posture displacement amount detecting means 16 and 22 are detection devices capable of detecting the posture direction displacement amount of the imaging systems 11 and 20. In this embodiment, an acceleration sensor using a gyro or a geomagnetic sensor using a coil is used. The posture displacement amount detecting means 16 and 22 are provided in both the first and second imaging systems 11 and 20. In a gyro-based acceleration sensor, the amount of displacement in the posture direction was obtained by integrating the acceleration generated due to the movement. In the geomagnetic sensor, the attitude direction is obtained by detecting the difference between the direction of the magnetic field generated in the coil and the direction of the geomagnetism. The displacement amount in the posture direction was obtained from the difference between the distance values of the imaging systems 11 and 20 before and after the movement. By performing this in both the first and second imaging systems 11 and 20, the amount of posture displacement of each imaging system was obtained.
[0024]
The calibration of the cameras in the imaging systems 11 and 20 was performed using the method proposed by Tsai (RY Tsai, "A Versatile Camera Calibration Technology for High-Accuracy 3D Machinery Vision Technology- , IEEE Journal of Robotics and Automation, vol. RA-3, No. 4, August 1987). This assumes a camera model in which a pinhole camera has been corrected for lens aberration, and calibrates parameters (camera principal point position, camera principal point position, This is a method for determining the rotation angle, focal length, aberration, image center, etc.) with high accuracy using a calibration chart. FIG. 7 shows the calibration chart used. The overall size is 99.2 × 121.4 mm, and 5 × 6 square patches of 10.3 × 10.3 mm are arranged at a pitch of 22.2 mm. The calibration chart was fixed to a position corresponding to the working distance of the apparatus while being tilted by 30 ° from the vertical direction, the distance between the apparatus and the calibration chart was determined in advance, and images were taken by the two imaging systems 11 and 20, respectively. Thus, the camera principal point position, the rotation angle, and the focus, which are calibration parameters for associating the vertex coordinates (world coordinate system) of the square patches in the known calibration chart with the imaging surface coordinates (camera coordinate system) of the imaging systems 11 and 20. The distance, aberration, image center, and magnification were determined. Thus, a straight line formula connecting the coordinates on the imaging surface and the coordinates of the target object can be calculated in the respective imaging systems 11 and 20. Since the coordinates on the object are the same point (corresponding point), the distance to the object was determined by calculating the coordinates of the intersection of the two linear equations. The obtained calibration parameters affect the camera principal point position and the rotation angle, which are the calibration parameters, when the arrangements of the imaging systems 11 and 20 are different from those at the time of calibration. Therefore, it is necessary to correct the camera principal point position and the rotation angle. In the present embodiment, the camera principal point position is corrected by the three-dimensional position displacement amount obtained by the three-dimensional position displacement detecting means 15, and the camera rotation angle is corrected by the posture displacement amount obtained by the posture displacement detecting means 16 and 22. did. When each of the imaging systems 11 and 20 was different from the arrangement at the time of calibration, the distance value to the object was calculated using the corrected calibration parameters.
[0025]
The procedure up to obtaining a three-dimensional image will be described with reference to the flowcharts shown in FIGS. As a previous step, calibration of the imaging system is performed. This will be described with reference to a first flowchart (FIG. 7). The distance between the first imaging system 11 and the second imaging system 20 in the movable light projection / imaging system unit and the orientation of each imaging system 11 and 20 are determined and fixed so as not to change (S10). ). In this state, each of the imaging systems 11 and 20 images a calibration chart (S11). A calibration parameter is calculated from the captured image and stored in the calibration parameter storage unit 33 (S12).
[0026]
Next, the flow up to the calculation of the distance value of the object will be described using a second flowchart (FIG. 8). The encoded pattern is projected on the object, and the object is imaged by the first and second imaging systems (S20). Here, when the first and second imaging systems 11 and 20 remain in the arrangement at the time of calibration, the distance to the target is calculated using the calibration parameters stored in the calibration parameter storage unit 33. The value is calculated (S21, S22). If the first and second imaging systems 11 and 20 are not arranged at the time of calibration, the three-dimensional position displacement amount detection means 15 calculates the three-dimensional position displacement amount between the imaging systems 11 and 20 and calculates the posture displacement. The amount of attitude change of each of the imaging systems 11 and 20 is calculated by the amount detection means 16 and 22 (S23, S24). Next, the camera principal point position is corrected by the three-dimensional position displacement amount obtained by the three-dimensional position displacement amount detecting means 15, and the camera rotation angle is corrected by the posture displacement amount obtained by the posture displacement amount detecting means 16 and 22 (S25). ). The distance value to the object is calculated using the corrected calibration parameters (S22).
[0027]
As the luminance information, a luminance image captured by the first or second imaging system 11 or 20 without projecting the encoded pattern on the target is used as the luminance information. This makes it possible to acquire not only the distance information but also the luminance information, and it is also possible to provide data when creating a content to be displayed by pasting the luminance data on the shape data of the target object.
[0028]
【The invention's effect】
As described above, according to the present invention, even when the arrangement of each imaging system is different from the arrangement at the time of calibration, there is no need to perform all the calibration work from the beginning by the correction means, and the layout of each camera is free. By doing so, it is possible to provide a three-dimensional image which has good mobility, enables measurement from an arbitrary viewpoint, can reduce an occlusion region, has high measurement accuracy, and can improve usability.
[Brief description of the drawings]
FIG. 1 is a configuration diagram of a three-dimensional image capturing apparatus according to an embodiment of the present invention.
FIG. 2 is a diagram showing an encoding pattern in an embodiment of the present invention.
FIG. 3 is a diagram illustrating a configuration and a detection method of a three-dimensional position displacement amount detecting unit and a posture displacement amount detecting unit of the mobile imaging system according to the embodiment of the present invention.
FIG. 4 is a diagram illustrating a light source (LED or LD) and a photodetector used for a three-dimensional position displacement amount detection unit.
FIG. 5 is a diagram illustrating a mechanism for scanning light from a light source used in a three-dimensional position displacement amount detection unit.
FIG. 6 is a calibration chart in the example of the present invention.
FIG. 7 is a diagram illustrating a flow of calibration in the embodiment of the present invention.
FIG. 8 is a diagram illustrating a flow up to distance value calculation in the embodiment of the present invention.
FIG. 9 is a configuration diagram of a conventional three-dimensional image pickup device.
FIG. 10 is a configuration diagram of a conventional three-dimensional image pickup device.
[Explanation of symbols]
REFERENCE SIGNS LIST 10 movable light projecting / imaging system unit 11 first imaging system 12 light projecting system 13 half mirror 14 moving and holding means 15 three-dimensional position displacement detecting means 16 attitude displacement detecting means 20 second imaging system 21 moving and holding Means 22 Posture displacement amount detecting means 23 Reflecting surface 30 Main body 31 Image processing system 32 Electrical system 33 Calibration parameter storage means 40 Personal computer 50 Object 151 LED or LD (light source)
152 Photodetector 153 PSD
154, 155 Galvano mirror

Claims (5)

A light projection system for projecting the encoded pattern, a first imaging system for imaging the encoded pattern projected image from substantially the same optical axis direction at substantially the same principal point as the light projection system, and a light projection system A second imaging system that captures the coded pattern projected image from a direction different from the optical axis direction of the first imaging system, and calculates a distance image using calibration parameters of the cameras in the first and second imaging systems. In the three-dimensional image pickup apparatus, the light projecting system and the first image pickup system are configured as one movable light projecting system / imaging system unit, and the movable light projecting system / imaging system unit and the second And the three-dimensional position displacement amount detection means for detecting the three-dimensional position displacement amount between the imaging systems and the posture displacement for detecting the displacement amount in the posture direction of each imaging system. A three-dimensional image pickup device, comprising: an amount detection unit. 3. The three-dimensional storage device according to claim 1, further comprising a calibration parameter storage unit configured to store calibration parameters obtained by calibration performed in each of the movable projection system / imaging system unit and the camera of the second imaging system. Imaging device. The calibration stored in the calibration parameter storage unit is performed based on the three-dimensional position displacement amount between the imaging systems and the displacement amount in the posture direction of each imaging system obtained by the three-dimensional position displacement amount detection unit and the posture displacement amount detection unit. 3. The three-dimensional image capturing apparatus according to claim 2, wherein the three-dimensional image capturing device corrects the parameter. 4. The three-dimensional image pickup apparatus according to claim 1, wherein only one of the movable light projecting / imaging system unit and the second image pickup system is movable. A light projection system for projecting the encoded pattern, a first imaging system for imaging the encoded pattern projected image from substantially the same optical axis direction at substantially the same principal point as the light projection system, and a light projection system And a second imaging system that captures the encoded pattern projection image from a direction different from the optical axis direction of the optical imaging device, and calculates a distance image using the calibration parameters of the cameras in the first and second imaging systems. In the three-dimensional image capturing method, the light projecting system and the first image capturing system are configured as one movable light projecting system / imaging system unit, and the movable light projecting system / imaging system unit and the second The imaging systems are held relatively movably, the amount of displacement of the three-dimensional position between the imaging systems is detected, the amount of displacement of each imaging system in the attitude direction is detected, and the three-dimensional position displacement between the imaging systems is detected. The calibration parameter is compensated for by the amount of Three-dimensional imaging method, characterized by.

Images