JP5558951B2 - 3D imaging apparatus and electronic information device - Google Patents

Description

  The present invention relates to a three-dimensional imaging apparatus and an electronic information device, and in particular, a three-dimensional imaging that generates two images having parallax for forming a three-dimensional image from an image obtained by photographing with a single optical system. The present invention relates to a device and an electronic information device equipped with such a three-dimensional imaging device.

  9A and 9B are diagrams for explaining a conventional three-dimensional imaging camera (hereinafter referred to as a 3D camera). FIG. 9A shows an appearance thereof, and FIG. 9B shows an internal optical system thereof. ing. In FIG. 9B, the dotted line indicates the optical axis of the camera (the optical axis of the optical system in the camera).

  A 3D camera 10 shown in FIG. 9 includes a camera body 1 having a substantially rectangular parallelepiped shape, a pair of left and right imaging lenses 2 a and 2 b provided at the center of the front surface of the camera body 1, and the center of the upper end of the camera body 1. And a finder objective window 5 attached to the upper end corner of the camera body 1. A power switch 6 is provided at the center of the upper surface of the camera body 1, and a shutter button 7 is provided at the upper surface side of the camera body 1.

  Note that an LCD panel for displaying an image is provided on the back of the camera body 1, and in addition, a viewfinder eyepiece window and various operation buttons are provided. In addition, a slot in which a recording device such as a memory card for storing image data is detachably mounted is provided on the bottom surface of the camera body 1.

  In addition, the 3D camera 10 includes an imaging element 3a that captures an image of the subject Ob via a first imaging optical system (not shown) including the imaging lens 2a, and a second imaging optical system that includes the imaging lens 2b. A pair of left and right photographing lenses 2a and 2b are driven so that the optical element of the imaging element 3b that captures an image of the subject Ob and the optical axes of the first and second imaging optical systems intersect with each other on the subject (not shown). Driving means (not shown).

  In the 3D camera 10 having such a configuration, images of the subject Ob viewed from two directions with a slight parallax can be captured by the imaging devices 3a and 3b, respectively, and a stereoscopic image can be captured from these imaging devices 3a and 3b. Two images can be acquired to form.

  Such 3D cameras with compound eyes are disclosed in, for example, Patent Documents 1 to 5.

  In addition to the compound eye as described above, 3D cameras that capture stereoscopic images include 3D cameras based on the pupil division method.

  In addition, it can replace with one 3D camera which has two optical systems as shown in FIG. 9, and can acquire two parallax images for forming a stereo image similarly using two cameras. . In addition, as disclosed in Patent Document 1, a parallax image for forming such a stereoscopic image can be acquired using more than two cameras.

  FIGS. 10A and 10B are diagrams for explaining such a pupil division type 3D camera. FIG. 10A shows an appearance thereof, and FIG. 10B shows an internal optical system thereof. In FIG. 10B, the dotted line indicates the optical axis of the camera (the optical axis of the optical system in the camera).

  A 3D camera 20 shown in FIG. 10 is attached to a camera body 1 having a substantially rectangular parallelepiped shape, a single imaging lens 2 provided at the center of the front surface of the camera body 1, and the center of the upper end of the camera housing 1. And a finder objective window 5 attached to the upper end corner of the camera casing 1. Further, similarly to the 3D camera 10 shown in FIG. 9, a power switch 6 is provided at the center of the upper surface of the camera body 1, and a shutter button 7 is provided on the upper surface side of the camera body 1.

  As with the 3D camera 10, an LCD panel, a viewfinder eyepiece window, and various operation buttons are provided on the back of the camera body 1, and a slot in which a recording device is detachably mounted on the bottom of the camera body 1. Is provided.

  In the 3D camera 20, the imaging optical system including the imaging lens 2 is arranged to face the pupil behind the imaging lens 2, and is divided by the prism 12 and the prism 12 that divides the pupil left and right. It has a first mirror 11 a that reflects one light and a second mirror 11 b that reflects the other light divided by the prism 12. The 3D camera 20 also includes an imaging element 13a that captures an image of the subject Ob through the imaging lens 2, the prism 12, and the first reflecting mirror 11a, and an image of the subject Ob, the imaging lens 2, the prism 12, and the second. The image pickup device 13b picks up an image via the reflection mirror 11b, and these image pickup devices 13a and 13b form a three-dimensional image in which images taken by the respective image pickup devices have a predetermined parallax. Therefore, the position is shifted in the horizontal direction by a half of the arrangement pitch of the pixels arranged in the horizontal direction so as to obtain a parallax image.

  The 3D camera 20 having such a configuration includes first and second imaging elements 13a and 13b in which the pixels are shifted by ½ pixel left and right, and one of the prisms 12 that divides the pupil behind the imaging lens. The image is picked up by the image pickup device 13a, and the other image from the prism 12 is picked up by the image pickup device 13b in which the pixels are shifted by ½ pixel left and right. Thereby, the images picked up by the image pickup devices 13a and 13b become images having parallax, and these images enable stereoscopic viewing of the subject.

  Such a pupil division type 3D camera is disclosed in, for example, Patent Documents 6 and 7.

  In order to construct a 3D image as described above, two images having parallax are required, and so far, a two-lens two-sensor camera and a pupil-division-type 3D camera have been proposed.

Japanese Patent Laid-Open No. 06-078337 Japanese Patent Laid-Open No. 06-339052 Japanese Patent Laid-Open No. 08-102970 JP-A-11-155151 JP 2002-058046 A Japanese Patent Laid-Open No. 08-015616 JP 2010-056865 A

  However, the conventional compound eye 3D camera adjusts the optical axis of each optical system corresponding to the compound eye, and the pupil division type 3D camera adjusts the axis of another optical system through which the divided light passes. Since it is complicated and requires high accuracy, its manufacture is not easy, and since such a 3D camera requires two or more optical systems, the module size increases. There is a problem of end.

  The present invention has been made to solve the above-described problems, and has a simple structure with a small module size and a three-dimensional imaging apparatus capable of generating a parallax image for stereoscopic viewing, and An object is to obtain an electronic information device equipped with such a three-dimensional imaging device.

  A three-dimensional imaging apparatus according to the present invention is a three-dimensional imaging apparatus that includes an imaging unit that images a subject and generates a parallax image for stereoscopically viewing the subject from a captured image obtained by imaging the subject. An optical system configured to change a focal position and form an image of the subject on the light receiving surface of the imaging unit; and a distance from the focal position information indicating the focal position of the optical system to the subject. A distance information acquisition unit that acquires distance information indicating image information, and image processing that generates the parallax image by processing the captured image so that the position of the subject in the captured image changes based on the distance information of the subject The above object is achieved thereby.

  The present invention is the above three-dimensional imaging apparatus, wherein the image processing unit is configured to detect the subject on the captured image in accordance with a relative distance between a plurality of subjects included in the captured image in the optical axis direction of the optical system. It is preferable to generate the parallax image by adjusting the position in the horizontal direction.

  According to the present invention, in the three-dimensional imaging apparatus, the image processing unit is configured to detect a subject included in the captured image according to a relative distance between a plurality of subjects included in the captured image in the optical axis direction of the optical system. It is preferable to generate distortion by adding distortion to the image.

  According to the present invention, in the above three-dimensional imaging apparatus, the imaging unit is configured by one image sensor, and the optical system captures the light from the subject by a single lens mechanism having a single optical axis. It is preferable that the light is condensed on the light receiving surface of the part.

  In the three-dimensional imaging apparatus according to the present invention, the distance information acquisition unit includes a distance adjustment mechanism that adjusts a distance between the imaging lens in the single-lens lens mechanism and a light receiving surface of the image sensor. Individual imaging obtained by controlling the distance adjustment mechanism and the imaging unit so that the subject is imaged at the focal position detected for each of a plurality of blocks dividing the imaging screen, and imaging at the focal position of each block It is preferable to acquire distance information from the image to each subject.

  In the three-dimensional imaging apparatus according to the present invention, the optical system includes an optical component that refracts incident light and a refractive power adjustment mechanism that adjusts the refractive power of the optical component. It is preferable that the light from the subject is refracted so that an image of the subject is formed on the light receiving surface of the imaging unit.

  According to the present invention, in the three-dimensional imaging apparatus, the optical component is preferably a liquid lens.

  According to the present invention, in the above three-dimensional imaging apparatus, the optical component has a plurality of lens areas divided in a plane parallel to the image plane of the subject, and each lens area of the plurality of lens areas The refractive power adjustment mechanism is configured so that the refractive power can be adjusted independently, and the refractive power adjustment mechanism is focused on each area corresponding to each lens area on the imaging screen based on the control signal. It is preferable to adjust the refractive power of each lens region of the optical component.

  In the three-dimensional imaging apparatus according to the present invention, when the captured image includes two subject images, the image processing unit is based on a horizontal relative distance S0 of the two subject images in the captured image. Then, by performing processing on the captured image, the relative distances S1 and S2 in the horizontal direction of the images of the two subjects as the parallax images are determined by the following expressions (1) and (2), respectively. It is preferable to generate a pair of left and right images.

S1 = S0 + M × d2 × B / (2 × d1) (1)
S2 = S0−M × d2 × B / (2 × d1) (2)
Here, M is the magnification of the optical system, B is the base line length, d1 is the distance from the imaging position to the main subject of the two subjects, and the relative distance along the optical axis direction of the two subjects.

  According to the present invention, in the three-dimensional imaging apparatus, the image processing unit performs a process of adding distortion to the captured image, and the relative distances S1 and S2 in the horizontal direction of the images of the two subjects are obtained as the parallax images. It is preferable to generate a pair of left and right images each having a distance determined by the equations (1) and (2).

  In the three-dimensional imaging apparatus according to the present invention, it is preferable that the F value of the optical system is smaller than 3.0.

  An electronic information device according to the present invention is an electronic information device equipped with an image pickup unit, and the image pickup unit includes the above-described three-dimensional image pickup device according to the present invention, whereby the above object is achieved. The

  Next, the operation of the present invention will be described.

  In the present invention, in a three-dimensional imaging device that generates a parallax image for stereoscopic viewing of a subject from a captured image obtained by imaging the subject, the focal position can be changed, and the subject image can be changed. An optical system that forms an image on the light receiving surface of the imaging unit; a distance information acquisition unit that acquires distance information indicating a distance to the subject from focal position information that indicates a focal position of the optical system; and a distance of the subject And an image processing unit that generates the parallax image by processing the captured image so that the position of the subject in the captured image changes based on the information. The optical axis of the optical system that forms an image can be made one. Thereby, the adjustment of the optical axis of each optical system corresponding to the compound eye in a 3D camera with a compound eye and the adjustment of the axis of another optical system through which the divided light passes in a 3D camera of a pupil division type can be eliminated. As a result, it is possible to generate a parallax image for stereoscopic viewing with a simple structure having a small module size, and to reduce the size and cost of the 3D camera.

  In the present invention, the distance information acquisition unit has a lens driving mechanism that adjusts the distance between the imaging lens and the light receiving surface of the image sensor, and the focus detected for each of a plurality of blocks that divide the imaging screen by the image sensor. The distance information from the individual captured image (imaging candidate image) obtained by controlling the lens driving mechanism and the image sensor so that the subject is imaged at the position and captured at the focal position of each block to the individual subject. Therefore, in order to acquire the distance information to the subject, complicated image processing such as pattern recognition processing of the subject in the captured image is unnecessary, and the distance information to each subject can be easily obtained.

  Further, in the present invention, when calculating the lateral distance between the main subject and the other subject in the left and right parallax images from the lateral distance between the main subject and the other subject in the captured image, the baseline length is set to the human eye. Since a triangle formed by the right eye position, the left eye position, and the main subject is used as an interval, these objects can be stereoscopically viewed from the object distance, that is, information indicating the distance from the imaging position to the main subject and other subjects. In addition, a parallax image closer to the parallax image obtained by actual shooting can be generated from a shot image shot by an optical system having a single optical axis.

  In the present invention, as the mechanism for changing the focal position of the optical system of the 3D camera, a mechanism including a liquid lens and the like that can adjust the refractive power of the optical system is used, so that the refractive power of the optical system can be adjusted. The mechanism can be configured simply.

  In the present invention, as a mechanism for changing the focal position of the optical system of the 3D camera, each of a plurality of blocks that divide the plane within a plane parallel to the light receiving surface (imaging plane) of the image sensor. Since a mechanism that can adjust the refractive power independently is used, by applying a voltage to the transparent electrodes facing each other across a flat plate member that can change the refractive power by applying a voltage, each block of the flat plate member It is possible to adjust the refractive power according to the applied voltage and adjust the refractive power at high speed. For this reason, focus adjustment is performed for each subject using the voltage applied to the transparent electrode of each block, and this focus adjustment amount is output and converted into an object distance, whereby a captured image as a still image and a pair of parallax images are converted. Acquisition is possible in a time shorter than the frame rate (for example, 1/30 second) of the moving image, and as a result, a pair of continuous left and right parallax images for stereoscopically viewing the moving image can be obtained.

  In the present invention, since the F value of the optical system constituting the 3D camera is set to a value smaller than 3.0, when the focus is the same, the depth of focus becomes shallow because the F value is small. Object distance recognition accuracy is improved.

  Moreover, in the present invention, a small terminal having a compact 3D camera module can be realized. Using this, 3D imaging in various scenes becomes possible. For example, it is possible to acquire a realistic 3D image by attaching to a sports player and capturing an image.

  As described above, according to the present invention, a focal position can be changed in a three-dimensional imaging device that generates a parallax image for stereoscopic viewing of a subject from a captured image obtained by imaging the subject. An optical system that forms an image of the subject on the light receiving surface of the imaging unit, and a distance information acquisition unit that acquires distance information indicating a distance to the subject from focal position information indicating a focal position of the optical system And an image processing unit that processes the captured image so as to change the position of the subject in the captured image based on the distance information of the subject and generates the parallax image, so that the module size is small A three-dimensional imaging device that can generate a parallax image for stereoscopic viewing with a simple structure can be obtained.

1A and 1B are diagrams for explaining a three-dimensional image pickup apparatus (3D camera) according to Embodiment 1 of the present invention. FIG. 1A shows an appearance thereof, and FIG. 1B schematically shows an internal configuration thereof. Is shown. FIG. 2 is a block diagram for explaining the three-dimensional image capturing apparatus according to Embodiment 1 of the present invention, and shows a specific configuration of the signal processing unit 110 constituting the 3D camera. FIG. 3 is a diagram for explaining the basic principle of the present invention. The captured image P0 (FIG. 3A) obtained by imaging the subject, and a pair of left and right parallax images P2 and P1 for stereoscopic viewing of the subject. (FIG. 3B and FIG. 3C) are shown. FIG. 4 is a diagram for explaining the basic principle of the present invention, and shows the positional relationship between the subject Mh and another subject Sh on the optical axis of the 3D camera. FIG. 5 is a diagram for explaining the basic principle of the present invention, and shows one model for creating a pair of left and right parallax images by changing the relative distance between the main subject and another subject in the captured image P0. ing. FIG. 6 is a diagram for explaining an example of a method for obtaining subject distance information in the 3D camera according to the first embodiment of the present invention. FIG. 7 is a diagram for explaining another example of a method for obtaining subject distance information in the 3D camera according to Embodiment 1 of the present invention. FIG. 7A shows a case where a liquid lens is used, and FIG. And (c) has shown the case where optical components other than a liquid lens are used. FIG. 8 is a block diagram illustrating a schematic configuration example of an electronic information device using the 3D camera of the first embodiment as an imaging unit as the second embodiment of the present invention. 9A and 9B are diagrams for explaining a conventional 3D camera with compound eyes. FIG. 9A shows an external appearance thereof, and FIG. 9B shows an optical system therein. FIGS. 10A and 10B are diagrams for explaining a conventional pupil division type 3D camera. FIG. 10A shows an appearance thereof, and FIG. 10B shows an internal optical system thereof.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(Embodiment 1)
1A and 1B are diagrams for explaining a three-dimensional image pickup apparatus (3D camera) according to Embodiment 1 of the present invention. FIG. 1A shows an appearance thereof, and FIG. 1B schematically shows an internal configuration thereof. Is shown. In FIG. 1B, the dotted line indicates the optical axis of the camera (the optical axis of the optical system in the camera).

  A 3D camera 100 shown in FIG. 1 generates a parallax image for stereoscopically viewing a subject from a captured image obtained by capturing the subject.

  The 3D camera 100 includes a camera body 1 having a substantially rectangular parallelepiped shape, a single imaging lens 102 provided at the center of the front surface of the camera body 1, and strobe light emission attached to the center of the upper end of the camera body 1. And a viewfinder objective window 5 attached to the upper end corner of the camera body 1. Further, similarly to the 3D camera 10 shown in FIG. 9, a power switch 6 is provided at the center of the upper surface of the camera body 1, and a shutter button 7 is provided on the upper surface side of the camera body 1.

  As with the 3D camera 10, an LCD panel, a viewfinder eyepiece window, and various operation buttons are provided on the back of the camera body 1, and a slot in which a recording device is detachably mounted on the bottom of the camera body 1. Is provided.

  In addition, the 3D camera 100 drives the imaging lens 102 along the optical axis of the imaging optical system and an imaging element 112 that captures an image of the subject Ob via the imaging optical system 102a including the imaging lens 102. A lens driving unit 120 such as an actuator, and a signal processing unit 110 that processes image data obtained by the imaging element 112 are included. Further, the F value of the imaging lens 102 is set to a value smaller than 3.0.

  FIG. 2 is a block diagram showing a specific configuration of the signal processing unit 110.

  Here, the signal processing unit 110 corresponds to the lens drive control unit 111 that controls the drive of the lens drive unit 120 and the image data Cim obtained by the imaging element 112 with the drive control signal Lds from the lens drive control unit. An imaging candidate storage unit 114 to be added and stored, an image analysis unit 113a that analyzes the captured image based on the image data stored in the imaging candidate storage unit 114 and the drive control signal Lds, and a result of the analysis The distance determination unit 113b that determines the focal length for each block that divides the imaging screen, the distance storage unit 115 that stores the focal length for each block, and the focal length for each block stored in the distance storage unit 115 An image processing unit 1 that performs processing on the captured image stored in the imaging candidate storage unit 114 to generate a parallax image for stereoscopically viewing the subject. 6, and an image storage unit 117 for storing the parallax images.

  Here, the image analysis unit 113a and the distance determination unit 113b constitute a distance information acquisition unit 113 that acquires distance information indicating the distance to the subject from the focus position information indicating the focus position of the optical system.

  Next, the function and effect will be described.

  First, the basic principle of the present invention will be described with reference to FIGS.

  In the 3D camera of the present invention, as shown in FIG. 3, a pair of left and right parallax images P2 and P1 for stereoscopic viewing of the subject from the captured image P0 (FIG. 3 (a)) obtained by imaging the subject. 3 (b) and 3 (c)) are generated.

  The captured image P0 includes an image of the main subject (person) Mh and an image of another subject (dog) Sh, and the parallax images P1 and P2 also include the main subject Mh and other subjects Sh. Yes.

  However, in the parallax image P1, the horizontal interval between the image of the main subject Mh and the image of the other subject Sh is wider than that in the captured image P0, and in the parallax image P2, the image of the main subject Mh. And the distance between the images of other subjects Sh are narrower than those in the captured image P0. By viewing such parallax images P2 and P1 with the left eye and the right eye, respectively, these subjects can be stereoscopically viewed.

  Next, in the 3D camera 100 according to the first embodiment of the present invention, the distance in the horizontal direction between the image of the main subject Mh and the image of the other subject Sh in the parallax images P1 and P2 is determined as the main subject Mh in the captured image P0. A method of calculating from the horizontal distance between the image and the image of the other subject Sh will be described.

  In the first embodiment, in the parallax images P1 and P2, the horizontal distance between the image of the main subject Mh and the image of the other subject Sh is based on the relative distance along the optical axis of these subjects Mh and Sh. calculate.

  FIG. 4 shows the positional relationship between the subject Mh and another subject Sh on the optical axis of the 3D camera.

  In the positional relationship between the subject Mh and the other subject Sh shown in FIG. 4, the distance d1 from the light receiving surface (imaging plane) of the image sensor 112 such as an image sensor to the main subject Mh is the distance of the image sensor 112 such as an image sensor. It is larger than the distance d3 from the light receiving surface (imaging surface) to the other subject Sh.

  In the parallax images P1 and P2, the horizontal distances S1 and S2 between the image of the main subject Mh and the image of the other subject Sh are the same as those of the main subject Mh and the other subject Sh in the parallax images P1 and P2. The horizontal distances S1 and S2 from the image, the relative distance d2 between the main subject and another subject, the distance d1 from the imaging plane to the main subject Mh, the image of the main subject Mh in the captured image P0, and others Is obtained from the horizontal interval S0 with the image of the subject Sh and the base line length B3.

  Here, the baseline length B is, for example, the distance between the left and right eyes. By changing the length of the baseline length B, the stereoscopic effect of the stereoscopic image of the subject obtained from the parallax images P1 and P2, that is, the stereoscopic image of the subject is obtained. You can change the way you look.

  FIG. 5 shows one calculation model for creating a pair of left and right parallax images by changing the relative distance between the main subject and another subject in the captured image P0.

  In this calculation model, left and right eye positions Em2 and Em1 and an intermediate position Em0 thereof are defined on a reference plane Sb corresponding to the light receiving surface (imaging plane) of the image sensor 112, and these positions Em0, Em1, Em2 is the horizontal position of the main subject Mh image in the captured image P0, the horizontal position of the main subject Mh image in the parallax image P1, and the horizontal position of the main subject Mh image in the parallax image P2. Yes. On this reference plane Sb, the lateral position Es0 of the image of the other subject Sh in the captured image P0, the lateral position Es1 of the image of the other subject Sh in the parallax image P1, and the other subject in the parallax image P2 The position Es2 in the horizontal direction of the Sh image is set based on the position of the main subject in the image.

  Further, the position Hm of the main subject Mh is taken in the normal direction of the reference plane Sb, and the position Hs of another subject Sh is set with the position Hm of the main subject Mh as a reference.

  In the calculation model shown in FIG. 5, the distance S0 is equal to the distance between the points Hs and H0, the distance S1 is equal to the distance between the points Hs and H1, the distance S2 is equal to the distance between the points Hs and H2, Further, since the distance between the point H0 and the point H1 and the distance between the point H0 and the point H2 are the distance d2 × tan θ, the following equations (A) to (E) hold.

S1 = S0 + d2 × tan θ (A)
S2 = S0−d2 × tan θ (B)
tan θ = B / (2 × d1) (C)
S1 = S0 + d2 × B / (2 × d1) (D)
S2 = S0−d2 × B / (2 × d1) (E)
S0: Horizontal distance between main subject and other subject on image plane S1: Horizontal orientation of main subject and other subject in one image P1 of 2D 3D output images (parallax images) on image plane Distance in direction S2: Horizontal distance between main subject and other subject in the other image P2 of 3D output images (parallax images) of two images on the image plane d1: Main subject distance (concatenated from main subject) Distance to the image plane)
d2: Distance in the optical axis direction of the other subject with respect to the main subject B: Baseline length θ: A line connecting the intermediate position Em0 on the reference plane and the position Hm of the subject, and a position Em2 (Em1) of the left eye (or right eye) Angle formed by a line connecting the position Hm of the subject. In addition, the lens magnification M (M = y ′ / y) of the optical system of the 3D camera is used to make a reference plane between the actual main subject Mh and another subject Sh. The distance S0 ′ at is as shown in the following formula (F).

y ′: the size of the object in the direction normal to the optical axis y: the size of the image in the direction normal to the optical axis S0 ′ = S0 × M (F)
Further, when the main subject Mh is viewed from the right eye position Em1, a pseudo distance S1 ′ instead of the actual lateral distance between the main subject Mh and the other subject Sh is as shown in the following equation (G). is there.

S1 ′ = S1 × M (G)
Further, when the main subject Mh is viewed from the left eye position Em2, a pseudo distance S2 ′ instead of the actual lateral distance between the main subject Mh and the other subject Sh is as shown in the following expression (H). is there.

S2 ′ = S2 × M (H)
Further, in order to obtain the relative distances S1 and S2 between the main subject and other subjects in the parallax image in this way, the main subject distance (distance from the main subject to the imaging plane) d1 and the light of the other subject with respect to the main subject It is necessary to determine the axial distance d2, and the method will be described below.

  FIG. 6 is a diagram for explaining a method of obtaining subject distance information in the 3D camera according to the first embodiment. The light receiving surface Sc of the image sensor is arranged in a block Rmn (m, for example, arranged in an 8 × 8 matrix. , N is an integer of 1 to 8).

  Here, the focal length is obtained for each block of the image sensor based on the edge information of the image. For example, as shown in FIG. 6, when the captured image includes the main subject Mh and the image of another subject Sh shown in FIG. 3A, basically, in the block overlapping the image of the main subject Mh, the main subject Mh. Focus will be on position. Further, in a block that overlaps an image of another subject Sh, the focus is basically achieved at the position of the other subject Sh.

  At this time, the imaging lens 102 is moved along the optical axis direction from the position closest to the subject Ob to the position farthest from the subject by the lens driving mechanism 120 shown in FIG. Image analysis such as edge detection is performed on the captured image, and the focal length (distance to the subject in each block) in the area corresponding to each block of the captured image is the sharpest edge in each block Calculated as the lens position. As a result, focal length information is obtained for each block.

  From the focal length information thus obtained, the distance d1 to the main subject Mh and the relative distance d2 between the main subject Mh and the other subject Sh can be obtained.

  Hereinafter, for example, the main subject Mh and the other subject Sh having the positional relationship shown in FIG. 4 are imaged by the 3D camera 100, and a pair of left and right parallaxes are obtained from the captured image P0 (FIG. 3A) obtained by this imaging. An operation for obtaining the images P2 and P1 will be described.

  First, when the photographer captures the main subject Mh and another subject Sh in the finder objective window 5 of the 3D camera 100 and presses the shutter 7, in the 3D camera 100 having the autofocus function, from the lens drive control unit 111. In response to the control signal Lds, the lens moving mechanism 120 moves the photographing lens 102 in the optical axis direction within a predetermined range (for example, a range from the position closest to the subject to the position farthest), and the image sensor 112 An image is taken for each determined moving distance. Thereby, imaging candidate images corresponding to each of a plurality of determined positions from a position where the focal distance is infinite to a position where the focal distance is the closest are obtained, and these imaging candidate images are acquired as the imaging candidate storage unit 114. Stored in

  As described above, the image analysis unit 113a obtains the lens position where the image edge in the block is sharpest by image analysis such as edge detection for each block that divides the light receiving surface Sc of the image sensor. The distance determination unit 113b calculates the distance to the subject in each block from this lens position, and stores this distance information in the distance storage unit 115.

  Based on the information indicating the sharpness of the image edge obtained by the image analysis unit 113a, the image processing unit 116 determines not only the main subject Mh but also the main subject Mh from the imaging candidate images obtained at the plurality of lens positions obtained during the autofocus operation. An imaging candidate image that is in focus on another subject Sh is selected as the captured image P0 and stored in the image storage unit 117.

  It should be noted that not only the subject Mh but also other subject Sh can be obtained as a candidate image for focusing by using a confocal optical system.

  In addition, the image processing unit 116 performs image processing on the selected captured image P0 to generate parallax images P1 and P2 for stereoscopically viewing the main subject Mh and the other subject Sh, and the generated parallax image. P1 and P2 are stored in the image storage unit 117.

  Here, the method of generating the parallax images P1 and P2 from the captured image P0 is performed by changing the relative distance between the main subject and other subjects in the captured image P0 using one calculation model shown in FIG. This is a method of creating a pair of parallax images.

  Specifically, when the parallax image P1 is generated from the captured image P0, a pixel located inside the outline of the main subject Mh obtained by edge detection is compared with the other subject Sh obtained by edge detection. A pixel located inside the contour is shifted to the left by the number of pixels corresponding to d2 × tan θ / 2, and a pixel having a pixel value obtained by interpolation processing or the like is filled in a position where the pixel is lost by this shift.

  Further, when generating the parallax image P2 from the captured image P0, the inside of the contour of the other subject Sh obtained by edge detection with respect to the pixel located inside the contour of the main subject Mh obtained by edge detection. Is shifted to the right by the number of pixels corresponding to d2 × tan θ / 2, and a pixel having a pixel value obtained by interpolation processing or the like is filled in the position where the pixel is lost by this shift.

  Note that the process of creating the parallax images P1 and P2 is not limited to shifting the pixels constituting the other subject with reference to the pixels constituting the main subject, and the pixels constituting the main subject constitute the other subject. In addition, the pixels constituting the main subject and the other subject may be determined based on the reference position between the two subjects, for example, the intermediate position between the two subjects. The distance may be shifted in the opposite direction so that the distance between them increases or decreases. However, in any case, in the parallax image P1, the distance between both subjects is separated by the number of pixels corresponding to d2 × tan θ / 2, and in the parallax image P2, the distance between both subjects is d2 × tan θ / 2. It is necessary to shift the pixels so as to approach the number of pixels corresponding to.

  In recent years, 3D display can display movies and games in three dimensions. In the 3D camera as in the present embodiment, a 3D camera can be realized by an image plane position adjustment mechanism and image processing, and the camera module that constitutes the 3D camera captures a 3D image with a simple configuration. Can be used for modules.

  As described above, in the first embodiment, the focal position can be changed in the three-dimensional imaging apparatus 100 that generates the parallax images P1 and P2 for stereoscopic viewing of the subject from the captured image P0 obtained by imaging the subject. An optical system 102a configured to form an image of the subject on the light receiving surface of the imaging unit, and a distance for acquiring distance information indicating a distance to the subject from focal position information indicating a focal position of the optical system Since the information acquisition unit 113 and the image processing unit 116 that processes the captured image so that the position of the subject in the captured image changes based on the distance information of the subject and generates the parallax image are provided. The optical axis of the optical system for forming the subject image on the light receiving surface of the image sensor can be made one. Thereby, the adjustment of the optical axis of each optical system corresponding to the compound eye in a 3D camera with a compound eye and the adjustment of the axis of another optical system through which the divided light passes in a 3D camera of a pupil division type can be eliminated. As a result, it is possible to generate a parallax image for stereoscopic viewing with a simple structure having a small module size, and to reduce the size and cost of the 3D camera.

  In the 3D camera 100 according to the first embodiment, the distance information acquisition unit 113 includes a lens driving mechanism 120 that adjusts the distance between the imaging lens 102 and the light receiving surface of the image sensor 112, and an imaging screen by the image sensor 112. The lens drive mechanism 120 and the image sensor 112 are controlled so that the subject is imaged at the focal position detected for each of a plurality of blocks that divide the image, and the individual captured image obtained by imaging at the focal position of each block Since the distance information from the (imaging candidate image) to the individual subject is acquired, complicated image processing such as pattern recognition processing of the subject in the captured image is not necessary to acquire the distance information to the subject. Distance information to the subject can be obtained.

  In the first embodiment, when the lateral distances S2 and S1 between the main subject and the other subject in the left and right parallax images are calculated from the lateral distance S0 between the main subject and the other subject in the captured image, FIG. 5 is used, that is, the base length is the distance between the human eyes, and the triangle formed by the right eye position, the left eye position, and the main subject is used. From the information indicating the distance to the subject, a parallax image for stereoscopic viewing of these subjects can be obtained, and a parallax image closer to the parallax image obtained by actual shooting is converted to an optical system having a single optical axis. It can generate | occur | produce from the picked-up image image | photographed with the system | strain.

  In the first embodiment, as the mechanism for changing the focal position of the optical system of the 3D camera, the lens driving unit that drives the imaging lens in the optical axis direction by an actuator or the like is used. However, the focal position of the optical system of the 3D camera is used. The mechanism for changing the angle may include a liquid lens and the like, and may be a mechanism capable of adjusting the refractive power of the optical system.

  For example, FIG. 7A shows a liquid lens Lg arranged in front of the image sensor 112. The liquid lens Lg adjusts the amount of liquid injected into the liquid lens Lg so that the curved shape of the lens surface can be adjusted. The shape changes from a shape having a large refractive power indicated by a solid line to a shape having a small refractive power indicated by a dotted line.

  In addition to the liquid injection method described with reference to FIG. 7A, there are liquid lenses that adjust the refractive power by manipulating the curvature of the liquid interface by applying a voltage. It is common.

  In this voltage application type liquid lens, oil and an aqueous solution are sealed in a predetermined casing, and electrodes are arranged at both ends of the casing. In such a liquid lens, when a voltage is applied to the opposing electrode, one of the oil or the aqueous solution is pulled by the electrode and tries to gather at the central portion of the casing, and at the same time, the other of the oil or the aqueous solution is Is pushed out to the peripheral part to change the curvature of the interface between the two liquids, thereby adjusting the refractive power.

  Further, as shown in FIG. 7B, a mechanism for changing the focal position of the optical system of the 3D camera has a plurality of sections that divide the plane within a plane parallel to the light receiving surface (imaging plane) of the image sensor. It may be a mechanism capable of adjusting the refractive power independently for each of the blocks.

  The refractive power adjustment mechanism OPd shown in FIG. 7B includes a flat plate member whose refractive power can be changed by applying a voltage, and a plurality of transparent electrodes arranged in a matrix on both surfaces of the flat plate member. ing. The refractive power adjustment mechanism OPd is disposed between the imaging lens L and the imaging element 112.

  Specifically, the surface region (lens region) of the flat plate member is partitioned in a matrix form by a plurality of blocks Rpks (k and s are integers of 1 to 8), and each block is on the surface of the flat plate member. The transparent electrode is arranged so as to correspond to the above, and the transparent electrode is arranged on the back surface of the flat plate-like member so as to correspond to each block.

  In such a configuration, by applying a voltage to the transparent electrodes facing each other across a flat plate member whose refractive power can be changed by applying a voltage, the refractive power corresponding to the applied voltage is applied to each block of the flat plate member. Since the adjustment is performed, the refractive power can be adjusted at high speed. For this reason, focus adjustment is performed for each subject using the voltage applied to the transparent electrode of each block, and this focus adjustment amount is output and converted into an object distance, whereby a captured image as a still image and a pair of parallax images are converted. Acquisition is possible in a time shorter than the frame rate (for example, 1/30 second) of the moving image, and as a result, a pair of continuous left and right parallax images for stereoscopically viewing the moving image can be obtained.

  In the first embodiment, the F value of the optical system constituting the 3D camera is set to a value smaller than 3.0. Therefore, when the focal length is the same, the depth of focus becomes shallow because the F value is small. The object distance recognition accuracy can be improved.

  In the above-described embodiment, the image processing unit in the horizontal direction of the subject on the captured image according to the relative distance of the plurality of subjects included in the captured image in the optical axis direction of the optical system. Although the position adjustment is performed to generate the parallax image, the image processing unit has the plurality of subjects included in the captured image according to the relative distance in the optical axis direction of the optical system. The parallax image may be generated by adding distortion to a subject image included in the captured image.

  Although not particularly described in the first embodiment, the optical system, the lens driving mechanism, the image sensor, and the signal processing unit that constitute the 3D camera of the first embodiment are a 3D camera module such as a mobile phone. It can be mounted on an electronic information device (small terminal).

  By using this, 3D imaging in various scenes becomes possible. For example, it is possible to acquire a realistic 3D image by attaching to a sports player and capturing an image.

(Embodiment 2)
Hereinafter, such an electronic information device will be briefly described.

  FIG. 8 shows, as Embodiment 2 of the present invention, an outline of an electronic information device that uses an optical system, a lens driving mechanism, an image sensor, and a signal processing unit that constitute the 3D camera of Embodiment 1 as a camera module. It is a block diagram which shows the example of a structure.

  An electronic information device 90 according to the second embodiment of the present invention shown in FIG. 8 includes the camera module as an imaging unit 91 that captures an image of a subject. A memory unit 92 such as a recording medium for recording data after a predetermined signal processing for recording the image data, and a liquid crystal to be displayed on a display screen such as a liquid crystal display screen after the predetermined signal processing for display of the image data A display unit 93 such as a display device, a communication unit 94 such as a transmission / reception device that performs communication processing after the image data is subjected to predetermined signal processing for communication, and the image data is printed (printed) and output (printed out). And at least one of the image output unit 95 to be operated.

  As mentioned above, although this invention has been illustrated using preferable embodiment of this invention, this invention should not be limited and limited to this embodiment. It is understood that the scope of the present invention should be construed only by the claims. It is understood that those skilled in the art can implement an equivalent range based on the description of the present invention and the common general technical knowledge from the description of specific preferred embodiments of the present invention. Patents, patent applications, and documents cited herein should be incorporated by reference in their entirety, as if the contents themselves were specifically described herein. Understood.

  The present invention includes a three-dimensional imaging device that generates two images having parallax for forming a three-dimensional image from an image obtained by photographing with a single optical system, and such a three-dimensional imaging device. In the field of electronic information equipment, it is possible to obtain a three-dimensional imaging apparatus capable of generating a parallax image for stereoscopic viewing with a simple structure having a small module size.

DESCRIPTION OF SYMBOLS 1 Camera body 4 Strobe light emission part 5 Finder objective window 6 Power switch 7 Shutter button 90 Electronic information equipment 91 Imaging part 92 Memory part 93 Display means 94 Communication means 95 Image output means 100 3D camera 102 Imaging lens 102a Imaging optical system 110 Signal processing Unit 111 Lens drive control unit 112 Image sensor 113a Image analysis unit 113b Distance determination unit 114 Imaging candidate storage unit 115 Distance storage unit 116 Image processing unit 117 Image storage unit 120 Lens drive unit (lens drive mechanism)
Cim Image data Lds Drive control signal Mh Main subject Sh Other subjects

Claims (10)

A three-dimensional imaging apparatus that includes an imaging unit that images a subject and generates a parallax image for stereoscopic viewing of the subject from a captured image obtained by imaging the subject,
An optical system configured to change a focal position and forming an image of the subject on a light receiving surface of the imaging unit;
A distance information acquisition unit that acquires distance information indicating a distance to the subject from focal position information indicating a focal position of the optical system;
An image processing unit that processes the captured image so as to change the position of the subject in the captured image based on the distance information of the subject, and generates the parallax image ;
The image processing unit
The relative distance in the optical axis direction of the optical system between one subject and the other subjects included in the captured image, a predetermined distance corresponding to the distance between the left and right eyes, and the imaging position By adjusting the horizontal relative position of the one subject and the other subject on the captured image based on the distance to the one subject, in one of the pair of images constituting the parallax image Determining a lateral relative distance between the one subject and the other subject, and a lateral relative distance between the one subject and the other subject in the other of the pair of images constituting the parallax image ; 3D imaging device. A three-dimensional imaging apparatus that includes an imaging unit that images a subject and generates a parallax image for stereoscopic viewing of the subject from a captured image obtained by imaging the subject,
An optical system configured to change a focal position and forming an image of the subject on a light receiving surface of the imaging unit;
A distance information acquisition unit that acquires distance information indicating a distance to the subject from focal position information indicating a focal position of the optical system;
An image processing unit that processes the captured image so as to change the position of the subject in the captured image based on the distance information of the subject and generates the parallax image;
With
The imaging unit is composed of one image sensor,
The optical system condenses the light from the subject on the light receiving surface of the imaging unit by a single lens mechanism having a single optical axis,
The optical system is
An optical component that refracts incident light;
A refractive power adjustment mechanism for adjusting the refractive power of the optical component;
By adjusting the refractive power of the optical component, the light from the subject is refracted so that the image of the subject is formed on the light receiving surface of the imaging unit,
The optical component includes a plurality of lens regions divided by the image plane parallel to the plane of the subject, adjustably configure the power independently for each lens region of the plurality of lens regions And
The power adjustment mechanism based on the control signal, on the imaging screen, so that in focus for each area corresponding to the respective lens regions, and adjusts the refractive power of each lens region of the optical component 3D imaging device. A three-dimensional imaging apparatus that includes an imaging unit that images a subject and generates a parallax image for stereoscopic viewing of the subject from a captured image obtained by imaging the subject,
An optical system configured to change a focal position and forming an image of the subject on a light receiving surface of the imaging unit;
A distance information acquisition unit that acquires distance information indicating a distance to the subject from focal position information indicating a focal position of the optical system;
An image processing unit that processes the captured image so as to change the position of the subject in the captured image based on the distance information of the subject and generates the parallax image;
With
The image processing unit adjusts the horizontal position of the subject on the captured image according to the relative distance of the plurality of subjects included in the captured image in the optical axis direction of the optical system, and Configured to generate parallax images,
The image processing unit
When containing the image of the two subjects in the captured image, based on the relative horizontal distance S0 of the image of the two of the subject in the captured image, by performing processing on the captured image, as the parallax images, the A three-dimensional imaging device that generates a pair of left and right images in which the horizontal relative distances S1 and S2 of two subject images are determined by the following formulas (1) and (2), respectively.
S1 = S0 + M × d2 × B / (2 × d1) (1)
S2 = S0−M × d2 × B / (2 × d1) (2)
Here, M is the magnification of the optical system, B is the baseline length, d1 is the distance from the imaging position to the main subject of the two subjects , and d2 is the relative distance along the optical axis direction of the two subjects. It is. The three-dimensional imaging apparatus according to claim 3 ,
The image processing unit
By performing a process of adding distortion to the captured image, the relative distances S1 and S2 in the horizontal direction of the images of the two subjects as the parallax images are determined by the above formulas (1) and (2), respectively. A three-dimensional imaging device that generates a pair of left and right images. The three-dimensional imaging device according to claim 1 or 3 ,
The imaging unit is composed of one image sensor,
The three-dimensional imaging apparatus, wherein the optical system condenses light from the subject on a light receiving surface of the imaging unit by a single-lens lens mechanism having a single optical axis. The three-dimensional imaging device according to claim 5 ,
The distance information acquisition unit
A distance adjustment mechanism that adjusts the distance between the imaging lens in the single-lens lens mechanism and the light receiving surface of the image sensor;
The distance adjustment mechanism and the imaging unit are controlled so that the subject is imaged at the focal position detected for each of a plurality of blocks dividing the imaging screen by the image sensor, and the image is obtained at the focal position of each block. A three-dimensional imaging apparatus that obtains distance information from an individual captured image to an individual subject. The three-dimensional imaging device according to claim 1 or 3 ,
The optical system is
An optical component that refracts incident light;
A refractive power adjustment mechanism for adjusting the refractive power of the optical component;
A three-dimensional imaging apparatus that refracts light from the subject so that an image of the subject is formed on a light receiving surface of the imaging unit by adjusting a refractive power of the optical component. The three-dimensional imaging device according to claim 7 ,
The three-dimensional imaging device, wherein the optical component is a liquid lens. The three-dimensional imaging device according to any one of claims 1 to 8 ,
A three-dimensional imaging apparatus in which the F value of the optical system is smaller than 3.0. An electronic information device equipped with an imaging unit,
The electronic imaging device includes the three-dimensional imaging device according to any one of claims 1 to 3 .