JP7501582B2 - Imaging device - Google Patents

Description

The present invention relates to an imaging device.

The time-of-flight (TOF) method is known, which measures the distance to an object by irradiating the object with light and receiving the reflected light (see, for example, Patent Document 1). With this technology, there is a problem in that the range that can be measured becomes shorter when the distance resolution is increased.

JP 2008-89346 A

The imaging device according to claim 1 is an imaging device, the imaging element having a light source unit that emits light to an object, a first pixel including a first photoelectric conversion unit that converts light into an electric charge, a second pixel arranged alongside the first pixel in the row direction and including a second photoelectric conversion unit that converts light into an electric charge, a first transfer control circuit that controls transfer of the electric charge converted by the first photoelectric conversion unit, and a second transfer control circuit that controls transfer of the electric charge converted by the second photoelectric conversion unit , and a distance measuring unit that calculates a distance from the imaging device to the object based on a signal read out from a light source unit, the light source unit emitting a first modulated light modulated to a sine wave of a first period and a second modulated light modulated to a sine wave of a second period different from the first period, the first pixel including a first transfer unit that transfers the charge converted by the first photoelectric conversion unit to a first storage unit, a second transfer unit that transfers the charge converted by the first photoelectric conversion unit to a second storage unit, and a third storage unit that transfers the charge converted by the first photoelectric conversion unit to a third storage unit. the second pixel has a fifth transfer section which transfers the charge converted by the second photoelectric conversion section to a fifth accumulation section, a sixth transfer section which transfers the charge converted by the second photoelectric conversion section to a sixth accumulation section, a seventh transfer section which transfers the charge converted by the second photoelectric conversion section to a seventh accumulation section, and an eighth transfer section which transfers the charge converted by the second photoelectric conversion section to an eighth accumulation section, and the first transfer control circuit is the first transfer unit, the second transfer unit, the third transfer unit, and the fourth transfer unit so that the charges converted by the second photoelectric conversion unit are sequentially transferred to the first storage unit, the second storage unit, the third storage unit, and the fourth storage unit in the first period, and the second transfer control circuit controls the fifth transfer unit, the sixth transfer unit, the seventh transfer unit, and the eighth transfer unit so that the charges converted by the second photoelectric conversion unit are sequentially transferred to the fifth storage unit, the sixth storage unit, the seventh storage unit, and the eighth storage unit in the second period .

1 is a cross-sectional view illustrating a schematic configuration of an imaging device according to a first embodiment. FIG. 2 is a schematic diagram showing a configuration of an imaging element. FIG. 2 is a circuit diagram showing a configuration of an imaging element. FIG. 2 is a diagram illustrating modulated light. FIG. 4 is a diagram illustrating a relationship between modulated light and reflected light. FIG. 4 is a diagram illustrating a relationship between modulated light and reflected light. 4 is a timing chart showing a shooting operation. 4 is an explanatory diagram illustrating operations of a first detection unit, a second detection unit, a distance measurement unit, and an image creation unit. FIG. FIG. 11 is an explanatory diagram showing that a control signal can be approximated by a sine wave. FIG. 13 is an explanatory diagram of a calculated phase _θa and a true phase Θ. FIG. 11 is a cross-sectional view illustrating a schematic configuration of an imaging device according to a second embodiment. FIG. 2 is a schematic diagram showing a configuration of a pixel. FIG. 2 is a circuit diagram showing a configuration of an imaging element. 4 is a timing chart showing a shooting operation. FIG. 2 is a diagram illustrating modulated light. FIG. 11 is a schematic diagram showing a modified example of a pixel layout. FIG. 13 is a circuit diagram showing a configuration of an image sensor according to Modification 4. FIG. 13 is an explanatory diagram of a modified example 4.

(First embodiment)
1 is a cross-sectional view showing a schematic configuration of an imaging device according to a first embodiment. The imaging device 1 is a device having a function of photographing an object 2 and creating image data of the object 2 as well as depth map data of the object 2. In this embodiment, the depth map data is map data in which distance values representing the distance between the object 2 and the imaging device 1 (i.e., the depth of the object) are arranged two-dimensionally for each part of the object 2. By using the imaging device 1, not only the image data of the object 2 but also the depth map data can be obtained at the same time by one photographing. In other words, the image data and the depth map data are created from the same photoelectric conversion signal.

The imaging device 1 includes a light source unit 10, an optical system 11, an imaging element 12, a control unit 13, a ROM 14, a display unit 15, and a storage medium 16. The light source unit 10 emits modulated light 3, which will be described later, toward the subject 2. The modulated light 3 includes a first transmitted modulated light 3a and a second transmitted modulated light 3b. The modulated light 3 is reflected by the surface of the subject 2, returns to the imaging device 1 as reflected light 4, and enters the imaging surface of the imaging element 12 via the optical system 11. The reflected light 4 includes a first received modulated light 4a, which is the reflected light of the first transmitted modulated light 3a, and a second received modulated light 4b, which is the reflected light of the second transmitted modulated light 3b. The optical system 11 is an imaging optical system that is composed of, for example, multiple lenses. Note that in FIG. 1, the optical system 11 is simplified and illustrated as a single lens.

The control unit 13 is composed of a microcomputer and peripheral circuits (not shown). The control unit 13 controls each part of the imaging device 1 by reading and executing a control program prestored in a ROM 14, which is a non-volatile storage medium. The display unit 15 is, for example, a display device such as a liquid crystal monitor. The storage medium 16 is, for example, a rewritable storage medium such as a memory card.

As will be described in more detail below, a photoelectric conversion signal is output from the imaging element 12. The control unit 13 creates image data and depth map data based on this photoelectric conversion signal. The control unit 13 stores the created image data and depth map data in the storage medium 16, and displays images based on these data on the display unit 15.

The control unit 13 is in the form of software and includes a first detection unit 13a, a second detection unit 13b, a distance measurement unit 13c, an image creation unit 13d, and a light emission control unit 13e. Each of these units is realized in software form by the control unit 13 executing a predetermined control program stored in the ROM 14. The operation of each of these units will be described in detail later.

Fig. 2(a) is a plan view showing the configuration of the image sensor 12, and Fig. 2(b) is a schematic enlarged view of a partial region 20a of the image pickup surface 20. As shown in Fig. 2(b), a plurality of pixels 30 are arranged two-dimensionally on the image pickup surface 20.

There are two types of pixels 30: first pixels 30a and second pixels 30b. In this embodiment, as shown in FIG. 2(b), the first pixels 30a and the second pixels 30b are arranged in a staggered manner. In other words, the second pixels 30b are arranged above, below, left and right of the first pixels 30a, and the first pixels 30a are arranged above, below, left and right of the second pixels 30b. Therefore, a substantially equal number of first pixels 30a and second pixels 30b are arranged uniformly (evenly) on the imaging surface 20.

As shown in FIG. 2(a), the image sensor 12 includes a horizontal scanning circuit 21, a vertical scanning circuit 22, a first transfer control circuit 23a, and a second transfer control circuit 23b. The vertical scanning circuit 22 controls the pixels 30 of the image sensor 12 on a row-by-row basis. The horizontal scanning circuit 21 sequentially reads out output signals from the pixels 30 in one row. The first transfer control circuit 23a transfers photoelectric conversion signals to the floating diffusion at the same timing for the multiple first pixels 30a. The second transfer control circuit 23b transfers photoelectric conversion signals to the floating diffusion at the same timing for the multiple second pixels 30b.

As will be described in detail later, the control pulse signal output from the first transfer control circuit 23a to the first pixel 30a and the control pulse signal output from the second transfer control circuit 23b to the second pixel 30b are different pulse signals.

Figure 3 is a circuit diagram showing the configuration of the image sensor 12. Note that Figure 3 shows the circuit diagram of only one pixel 30. The first pixel 30a and the second pixel 30b both have the configuration shown in Figure 3. Below, we will explain pixel 30, but this explanation will cover both the first pixel 30a and the second pixel 30b.

The pixel 30 includes a microlens (not shown), a photoelectric conversion unit 32 covered by the microlens, a first readout unit 331, a second readout unit 332, a third readout unit 333, and a fourth readout unit 334. The microlens focuses incident light (light beam from the optical system 11) on the photoelectric conversion unit 32. The photoelectric conversion unit 32 photoelectrically converts the incident light to generate a charge according to the amount of light. That is, the photoelectric conversion unit 32 photoelectrically converts the incident light to output a photoelectric conversion signal. The first readout unit 331, the second readout unit 332, the third readout unit 333, and the fourth readout unit 334 read out the photoelectric conversion signal from the photoelectric conversion unit 32.

The first readout section 331 includes a transfer transistor TX1, a reset transistor RST1, a floating diffusion FD1, an amplification transistor AMP1, and a row selection transistor SEL1. The transfer transistor TX1 transfers the photoelectric conversion signal (charge generated by the photoelectric conversion section 32) output by the photoelectric conversion section 32 to the floating diffusion FD1 based on a signal φTX1. The reset transistor RST1 resets the floating diffusion FD1 based on a signal φRST. The amplification transistor AMP1 outputs a signal according to the amount of charge accumulated in the floating diffusion FD1. The row selection transistor SEL1 outputs the signal output from the amplification transistor AMP1 to a vertical signal line V1 based on a signal φSEL. In the following description, the signal output to the vertical signal line V1 is referred to as an output signal _N1 .

The second readout section 332 includes a transfer transistor TX2, a reset transistor RST2, a floating diffusion FD2, an amplification transistor AMP2, and a row selection transistor SEL2. The transfer transistor TX2 transfers the photoelectric conversion signal (charge generated by the photoelectric conversion section 32) output by the photoelectric conversion section 32 to the floating diffusion FD2 based on a signal φTX2. The reset transistor RST2 resets the floating diffusion FD2 based on a signal φRST. The amplification transistor AMP2 outputs a signal according to the amount of charge accumulated in the floating diffusion FD2. The row selection transistor SEL2 outputs the signal output from the amplification transistor AMP2 to a vertical signal line V2 based on a signal φSEL. In the following description, the signal output to the vertical signal line V2 is referred to as an output signal _N2 .

The third readout section 333 includes a transfer transistor TX3, a reset transistor RST3, a floating diffusion FD3, an amplification transistor AMP3, and a row selection transistor SEL3. The transfer transistor TX3 transfers the photoelectric conversion signal (charge generated by the photoelectric conversion section 32) output by the photoelectric conversion section 32 to the floating diffusion FD3 based on a signal φTX3. The reset transistor RST3 resets the floating diffusion FD3 based on a signal φRST. The amplification transistor AMP3 outputs a signal according to the amount of charge accumulated in the floating diffusion FD3. The row selection transistor SEL3 outputs the signal output from the amplification transistor AMP3 to a vertical signal line V3 based on a signal φSEL. In the following description, the signal output to the vertical signal line V3 is referred to as an output signal _N3 .

The fourth readout section 334 has a transfer transistor TX4, a reset transistor RST4, a floating diffusion FD4, an amplification transistor AMP4, and a row selection transistor SEL4. The transfer transistor TX4 transfers the photoelectric conversion signal (charge generated by the photoelectric conversion section 32) output by the photoelectric conversion section 32 to the floating diffusion FD4 based on a signal φTX4. The reset transistor RST4 resets the floating diffusion FD4 based on a signal φRST. The amplification transistor AMP4 outputs a signal according to the amount of charge accumulated in the floating diffusion FD4. The row selection transistor SEL4 outputs the signal output from the amplification transistor AMP4 to a vertical signal line V4 based on a signal φSEL. In the following description, the signal output to the vertical signal line V4 is referred to as an output signal _N4 .

In this embodiment, the signal φSEL is input to the pixels 30 independently for each row. In the following description, the signal φSEL input to the pixel 30 in the nth row is denoted as signal φSEL(n). For example, signal φSEL(1) indicates the signal input to the pixel 30 in the first row. On the other hand, the same signal is input to all pixels 30 for the signal φRST. Different signals are input to the first pixel 30a and the second pixel 30b for the signals φTX1, φTX2, φTX3, and φTX4. Each signal will be described in detail later.

Fig. 4 is a diagram showing a schematic diagram of modulated light 3 emitted from the light source unit 10. In Fig. 4, the horizontal axis represents time, and the vertical axis represents light intensity. The modulated light 3 is visible light obtained by superimposing the first transmitted modulated light 3a and the second transmitted modulated light 3b. The first transmitted modulated light 3a is light modulated as a sine wave having an amplitude _Aa and a period T _a . The frequency f _a of the first transmitted modulated light 3a is 1/T _a . In the following description, the amplitude A _a , period T _a , and frequency f _a of the first transmitted modulated light 3a are referred to as the first amplitude A _a , first period T _a , and first frequency f _a , respectively.

The second transmitted modulated light 3b is light modulated as a sine wave with an amplitude A _b and a period T _b . The frequency f b of the second transmitted modulated light 3b is 1/T _b . In the following description, the amplitude A _b , period T _b , and frequency f _b of the second transmitted modulated light 3b are referred to as the second amplitude A _b , the second period T _b , and the second frequency f _b , respectively. The first period T _a is shorter than the second period T _b (for example, T _a ≈ T _b /7). In other words, the first frequency f _a is higher than the second frequency f _b .

5 is a diagram showing a relationship between the modulated light 3 emitted from the light source unit 10 and the reflected light 4 incident on the imaging surface of the imaging element 12. As shown in FIG. 1, the reflected light 4 includes the first received modulated light 4a and the second received modulated light 4b. The first received modulated light 4a and the second received modulated light 4b are incident on the imaging element 12 after a time corresponding to the distance of the subject 2 has elapsed since the time the modulated light 3 was emitted. In the following description, this time corresponding to the distance of the subject 2 is called a delay time. Due to the delay time, a phase difference θa occurs between the first transmitted modulated light 3a and the first received modulated light 4a, and a phase difference θb occurs between the second transmitted modulated light 3b and the second received modulated light 4b. As shown in FIG. 5, when the delay time is less than the first period T _a , the delay time represented by the phase difference θa is the same as the delay time represented by the phase difference θb.

6 is a diagram showing a schematic relationship between modulated light 3 emitted from the light source unit 10 and reflected light 4 incident on the imaging surface of the imaging element 12 when the subject 2 is farther away. As shown in FIG. 6, when the delay time is equal to or longer than the first period T _a , the phase difference θ _a appears to be smaller than the true phase difference Θ. In other words, Θ=2πn×θ _a (n is an integer equal to or greater than 0). In other words, when the delay time is equal to or longer than the first period T _a , the delay time represented by the phase difference θ _a and the delay time represented by the phase difference θ _b have different values.

Figure 7 is a timing chart showing the shooting operation. During the shooting operation, the control unit 13 causes the image sensor 12 to reset the pixels 30. Based on the signal from the control unit 13, the vertical scanning circuit 22 outputs the signal φRST, which is a control pulse signal, at time t1. As described above, the signal φRST is output to all pixels 30. As a result, all pixels 30 are reset.

At time t2 when the reset is completed, the light emission control unit 13e causes the light source unit 10 to start emitting the modulated light 3. In synchronization with the emission of the modulated light 3 by the light source unit 10, at time t2, the first transfer control circuit 23a and the second transfer control circuit 23b start outputting the control pulse signals φTX1, φTX2, φTX3, and φTX4. The light emission control unit 13e controls the light source unit 10 so that the position p1 where the transmitted first modulated light 3a is at its maximum coincides with the center position p2 of the pulse of the signal φTX1 output from the first transfer control circuit 23a to the first pixel 30a. The light emission control unit 13e controls the light source unit 10 so that the position p3 where the transmitted second modulated light 3b is at its maximum coincides with the center position p4 of the pulse of the signal φTX1 output from the second transfer control circuit 23b to the second pixel 30b. That is, the light emission control unit 13e synchronizes the modulated light 3 with the signals φTX1, φTX2, φTX3, and φTX4.

The first transfer control circuit 23a outputs signals φTX1, φTX2, φTX3, and φTX4 to the first pixel 30a. The first transfer control circuit 23a divides the period from time t2 to time t3, which is the period T _a after the transmission first modulated light 3a, into four periods, and turns on the signals φTX1, φTX2, φTX3, and φTX4 in these four periods. The first transfer control circuit 23a repeatedly outputs the above-mentioned signals φTX1, φTX2, φTX3, and φTX4 during the exposure period Te.

The second transfer control circuit 23b outputs signals φTX1, φTX2, φTX3, and φTX4 to the second pixel 30b. The second transfer control circuit 23b divides the period from time t2 to time t4, which is the period _Tb after the second transmitted modulated light 3b, into four periods, and turns on the signals φTX1, φTX2, φTX3, and φTX4 in these four periods. The second transfer control circuit 23b repeatedly outputs the above-mentioned signals φTX1, φTX2, φTX3, and φTX4 during the exposure period Te.

At time t5, which is an exposure period Te after time t2, the light emission control unit 13e causes the light source unit 10 to stop emitting modulated light 3. In synchronization with the stop of emission of modulated light 3, the first transfer control circuit 23a and the second transfer control circuit 23b stop outputting the signals φTX1, φTX2, φTX3, and φTX4 at time t5. At time t5, the floating diffusion FD1, floating diffusion FD2, floating diffusion FD3, and floating diffusion FD4 of pixel 30 each store electric charges generated by the photoelectric conversion unit 32.

In other words, the charges photoelectrically converted from the reflected light 4 (including the received first modulated light 4a and the received second modulated light 4b) that is incident on the photoelectric conversion unit 32 during the period from time t2 to time t5 are transferred to the floating diffusions FD1 to FD4 individually for each small period. For example, the charges photoelectrically converted by the photoelectric conversion unit 32 during the period when the signal φTX1 is on are transferred to the floating diffusion FD1.

After time t5, the horizontal scanning circuit 21 and the vertical scanning circuit 22 start reading out the output signals N ₁ to N _4. The output signals N ₁ to N ₄ are read out row by row. First, at time t5, the vertical scanning circuit 22 outputs a signal φSEL(1) which is a control pulse signal to the pixels 30 in the first row. As a result, the output signals N ₁ to N ₄ from each of the pixels 30 in the first row are output to the vertical signal lines V1 to V4. While the signal φSEL(1) is being output, the horizontal scanning circuit 21 outputs the output signals N ₁ to N ₄ from each of the pixels 30 in the first row one by one to the control unit 13. When the output signals N ₁ to N ₄ for all the pixels 30 in the first row are output to the control unit 13, the vertical scanning circuit 22 then outputs a signal φSEL(2) to the pixels 30 in the second row. As a result, the output signals N ₁ to N ₄ from each of the pixels 30 in the second row are output to the vertical signal lines V1 to V4. While the signal φSEL(2) is being output, the horizontal scanning circuit 21 outputs the output signals N ₁ to N ₄ from each of the pixels 30 in the second row one by one to the control unit 13. The horizontal scanning circuit 21 and the vertical scanning circuit 22 repeat the above operation for all rows, thereby outputting the output signals N ₁ to N ₄ from all of the pixels 30 to the control unit 13.

Next, the operations of the first detection unit 13a, the second detection unit 13b, the distance measurement unit 13c, and the image creation unit 13d will be described. FIG. 8 is an explanatory diagram for explaining the operations of the first detection unit 13a, the second detection unit 13b, the distance measurement unit 13c, and the image creation unit 13d. The modulated light 3 emitted toward the subject 2 is reflected by the subject 2 and enters the first pixel 30a and the second pixel 30b as reflected light 4 together with background light. The first pixel 30a and the second pixel 30b output output signals N ₁ to N _4. The first detection unit 13a calculates z _a , θ _a , and A _a (to be described later) from the output signals N ₁ to N ₄ of the first pixel 30a. The second detection unit 13b calculates z _b , θ _b , and A _b (to be described later) from the output signals N ₁ to N ₄ of the second pixel 30b. Based on the calculation results by the first detection unit 13a and the second detection unit 13b, the distance measurement unit 13c creates depth map data of the subject 2. The image creation unit 13d creates image data of the subject 2 from the output signals _N1 to _N4 of the first pixel 30a and the second pixel 30b. These operations will be described in detail below.

As described above, the light source unit 10 emits modulated light 3, which is a superimposition of the first transmitted modulated light 3a and the second transmitted modulated light 3b, toward the subject 2. The modulated light 3 is reflected by the surface of the subject 2 to become reflected light 4, which is incident on the first pixel 30a and the second pixel 30b of the image sensor 12. However, since the subject 2 is also irradiated with background light from a light source such as the sun in addition to the modulated light 3, the reflected light 4 includes not only the first received modulated light 4a and the second received modulated light 4b, but also various reflected lights of background light.

The first pixel 30a performs photoelectric conversion on the reflected light 4 and outputs the above-mentioned output signals N ₁ to N _4. Similarly, the second pixel 30b also performs photoelectric conversion on the reflected light 4 and outputs the above-mentioned output signals N ₁ to N ₄ .

The first detection unit 13a calculates an output value z _a , which will be described later, based on the output signals N ₁ to N ₄ output from the first pixels 30a. The output value z _a is a complex value obtained by converting the photoelectric conversion result of the first received modulated light 4a contained in the reflected light 4. From the output value z _a , a first amplitude A _a of the first received modulated light 4a contained in the reflected light 4 and a phase difference θ _a of the first received modulated light 4a with respect to the first transmitted modulated light 3a are calculated. However, as will be described in detail later, the phase difference θ _a calculated here may differ from the true phase difference Θ of the first received modulated light 4a with respect to the first transmitted modulated light 3a, so the distance measurement unit 13c performs a separate correction.

The first detection unit 13a calculates an output value z _a for each of the first pixels 30a based on the output signals N ₁ to N ₄ output from each of the first pixels 30a. The first detection unit 13a calculates a plurality of output values z _a for each of the first pixels 30a.

The second detector 13b calculates an output value _zb (described later) based on the output signals N ₁ to N ₄ output from the second pixel 30b. The output value _zb is a complex value obtained by converting the photoelectric conversion result of the second received modulated light 4b contained in the reflected light 4. From the output value _zb , a second amplitude _Ab of the second received modulated light 4b contained in the reflected light 4 and a phase difference _θb of the second received modulated light 4b with respect to the second transmitted modulated light 3b are calculated.

The second detection unit 13b calculates an output value z _b for each of the second pixels 30b based on the output signals N ₁ to N ₄ output from each of the second pixels 30b. The second detection unit 13b calculates a plurality of output values z _b for each of the second pixels 30b.

The distance measuring unit 13c performs a calculation described later using the output value _zb calculated by the second detecting unit 13b to correct the phase difference _θa detected by the first detecting unit 13a and calculates a true phase difference Θ. Then, by performing a calculation described later using this true phase difference Θ, the distance measuring unit 13c calculates the distance L (depth L) from the imaging device 1 to the subject portion corresponding to one pixel 30. The distance measuring unit 13c calculates the depth L for all pixels 30 to create depth map data in which the depth L for each subject portion is mapped two-dimensionally.

The image creation unit 13d creates image data of the subject 2 based on the output signals N ₁ to N ₄ output from the first pixels 30a and the output signals N ₁ to N ₄ output from the second pixels 30b. Although details will be described later, the image creation unit 13d determines a pixel value corresponding to a certain first pixel 30a based on the output signals N ₁ to N ₄ output from that pixel. Also, the image creation unit 13d determines a pixel value corresponding to a certain second pixel 30b based on the output signals N ₁ to N ₄ output from that pixel 30. In this way, the image creation unit 13d determines the pixel value of each pixel 30 based on the output signals N ₁ to N ₄ output from that pixel 30.

Next, a method for calculating the output value z _a by the first detector 13a will be described. The first detector 13a detects the photoelectric conversion result of the received first modulated light 4a, that is, the photoelectric conversion result of the incident light having the first frequency f _a , from the photoelectric conversion result of the reflected light 4, based on the output signals N ₁ to N ₄ output as a result of photoelectric conversion by the first pixels 30a during the period from time t2 to t5. The output signal N _i (i=1 to 4) can be expressed by the following equation (1).

Here, η is the sensitivity of the image sensor 12, and φ _R (t) is the intensity of the reflected light 4. Furthermore, g _i (t) is the signal φTXi (φTX1 to φTX4) shown in Fig. 7. For example, g ₁ (t) is a function that is 1 when the signal φTX1 is on and is 0 otherwise. From the above equation (1), N ₁ -N ₃ is expressed by the following equation (2).

9, g ₁ (t) - g ₃ (t) is a signal that oscillates in the range from +1 to -1, with 0 as the center. This can be approximated by a sine wave (i.e., a sine wave with period T _a ) cos 2πf _a t, whose period is the period from time t2 to t3. In addition, the reflected light 4 is a sum of the received first modulated light 4a, the received second modulated light 4b, and other background light, and can therefore be expressed by the following equation (3).

The above formula (3) expresses k types of light superimposed on each other, one of which is the received first modulated light 4a, another is the received second modulated light 4b, and the remaining is background light. In the above formula (3), _fk ( _f0 , _f1 , _f2 , ...) is the frequency of each light contained in the reflected light 4, and _Ak ( _A0 , _A1 , _A2 , ...) is the amplitude of each light. Also, Δt is the difference between the time when the modulated light 3 is projected from the light source unit 10 and the time when the reflected light 4 is incident on the first pixel 30a, i.e., the delay time.

By approximating g ₁ (t)-g ₃ (t) with a sine wave and further introducing the above equation (3), N ₁ -N ₃ can be expressed by the following equation (4).

Here, as expressed in the following equation (5), the result of integrating the product of sine waves of different frequencies (a sine wave of frequency _fk and a sine wave of frequency _fp ) can be regarded as zero.

When this is applied to the above equation (4), of the many terms relating to light contained in the reflected light 4, only the term relating to the light of the first frequency f _a (i.e., the received first modulated light 4 a) remains (because the terms relating to the other lights are 0), and finally N ₁ -N ₃ is expressed by the following equation (6).

In other words, by subtracting output signal _N3 from output signal _N1 , it is possible to extract only the photoelectric conversion result of the received first modulated light 4a from the photoelectric conversion result of the reflected light 4. When the calculation of output signals _N1 and _N3 described above is similarly performed on output signals _N2 and _N4 , _N2 - _N4 is expressed by the following equation (7).

Here, the above expressions (6) and (7) are converted into complex numbers to form the output value z _a . The output value z _a is expressed by the following expression (8).

That is, the output value z _a is a value obtained by extracting only the value corresponding to the received first modulated light 4 a from the light reception outputs of the reflected light 4 of the first pixel 30 a , that is, the output signals N ₁ to N ₄ .

By applying the output value z _a detected by the first detector 13 a to the following equation (9), the distance L of the subject portion corresponding to the first pixel 30 a can be calculated. In the following equation (9), c is the speed of light. Also, argz _a is the deflection angle of the output value z _a , that is, the phase difference θ _a between the transmitted first modulated light 3 a and the received first modulated light 4 a.

However, the distance L cannot be calculated accurately only by the output value z _a detected by the first detector 13a. This is because, as described with reference to FIG. 6, when the delay time Δt is large, the phase difference θ _a (i.e., the argument argz _a of the output value z _a ) is not determined to a single value. If it is assumed that the delay time Δt is sufficiently small, it can be assumed that the phase difference θ _a is less than 2π, as shown in FIG. 5. On the other hand, when it is necessary to consider the possibility that the delay time Δt is large, that is, when it is necessary to consider the possibility that the phase difference θ _a is 2π or more, it is not possible to determine how many periods the phase difference θ _a obtained by calculation is shifted by. In other words, the true phase difference Θ between the first transmitted modulated light 3a and the first received modulated light 4a may _be the phase difference θ a obtained by calculation plus an integer multiple of 2π.

10 is an explanatory diagram of the calculated phase difference _θa and the true phase Θ, and is a graph with the horizontal axis representing the distance L and the vertical axis representing the phase. As the distance L increases from 0, that is, as the subject 2 moves away from the imaging device 1, the calculated phase difference _θa increases. When the distance L is a certain value L1, the calculated phase difference _θa is apparently reset from 2π to 0, and then increases again from 0 as the distance L increases.

Therefore, for example, when the calculated phase difference _θa is π, it can only be determined that the true phase difference Θ is any one of π, 3π, 5π, ... and the distance L is any one of L1/2, 3×L1/2, 5×L1/2, ...

Therefore, the distance measuring unit 13c of this embodiment not only detects the first received modulated light 4a from the reflected light 4 by the first detector 13a, but also detects the second received modulated light 4b by the second detector 13b, thereby calculating the distance L. Specifically, the distance measuring unit 13c corrects the phase difference _θa using the output value _zb calculated by the second detector 13b to obtain the true phase difference Θ. Then, the distance measuring unit 13c calculates the distance L by substituting the corrected true phase difference Θ instead of the phase difference _θa before correction into the above equation (9).

First, the method of calculating the output value z _b by the second detector 13b will be described. The calculation content performed by the second detector 13b is the same as the calculation method of the output value z _a by the first detector 13a described above, except for the content of g _i (t). As described above, since the second pixel 30b is driven based on the second period T _b , the output value z _b is a value obtained by extracting a value corresponding to the received second modulated light 4 b from the light receiving output of the reflected light 4 of the second pixel 30b, i.e., the output signals N ₁ to N ₄ .

As described above, the second period _Tb is longer than the first period T _a . Therefore, the argument argz _b (phase difference θ _b ) calculated by the second detection unit 13b does not change from 2π to 0 (is continuous) up to a distance L2 longer than L1, as shown in Fig. 10. In other words, if the second period _Tb is set to be sufficiently long, even if the true phase difference θ cannot be determined only from the phase difference _{θ a} detected by the first detection unit 13a, the true phase difference θ can be obtained by also using the phase difference θ _b detected by the second detection unit 13b.

10, for example, when the phase difference _θa is π, as described above, the distance L is any one of L1/2, 3×L1/2, 5×L1/2, .... In other words, there are multiple candidates for the distance L derived from the phase difference _θa . Here, when the phase difference _θb is a value around π/2, the distance L is specified to one value, 3×L1/2, from the multiple candidates (L1/2, 3×L1/2, 5×L1/2, ...).

Next, a specific calculation performed by the distance measuring unit 13c will be described. The distance measuring unit 13c first obtains an integer n that indicates how many periods the phase difference _θa is shifted from the true phase difference Θ, using the following equation (10).

In the above formula (10), round(x) is a function that returns the integer closest to x (a function that rounds off x). For example, when the distance L is less than the distance L1 shown in Fig. 10 (when the true phase difference Θ is less than 2π), the phase difference _θa is equal to the true phase difference Θ, and n is 0. When the phase difference _θa is shifted by one period from the true phase difference Θ, n is 1, and the true phase difference Θ is the phase difference _θa plus 2π.

The distance measurement unit 13c obtains the true phase difference Θ using the following formula (11). The distance measurement unit 13c then calculates the distance L to the subject by applying the true phase difference Θ to the following formula (12).

As described above, by detecting the first and second modulated lights 4a and 4b, the range in which distance can be measured can be made wider. For example, when the first frequency f _a is 300 MHz and the second frequency f _b is 3 MHz, if the distance L is obtained by detecting only the first modulated light 4a, the measurable distance L is about 1 m at maximum and the distance resolution is about 1 cm at maximum. If the distance L is obtained by detecting only the second modulated light 4b, the measurable distance L is about 100 m at maximum, but the distance resolution is about 1 m at maximum. In this embodiment, the distance measuring unit 13c detects both the first and second modulated lights 4a and 4b to obtain the distance L, thereby expanding the measurable distance L to a maximum of 100 m while maintaining the distance resolution at a maximum of 1 cm.

As shown in FIG. 2B, the first pixel 30a and the second pixel 30b are located at different positions, so that the output value z _a and the output value z _b cannot be obtained simultaneously for the same subject position. Therefore, the distance measuring unit 13c calculates the distance L corresponding to the first pixel 30a by using the output value z _a calculated based on the first pixel 30a and the average value of four output values z _b calculated based on the second pixels 30b present above, below, left and right of the first pixel 30a. The distance measuring unit 13c also calculates the distance L corresponding to the second pixel 30b by using the output value z _b calculated based on the second pixel 30b and the average value of four output values z _a calculated based on the first pixels 30a present above, below, left and right of the second pixel 30b. In this way, depth map data with a number of elements (resolution) equal to the number of pixels 30 can be created.

Next, the image creation unit 13d will be described. As shown in FIG. 7, the charges generated by the photoelectric conversion unit 32 during the exposure period Te are dispersed and accumulated in the floating diffusions FD1 to FD4. Therefore, by adding up the output signals N ₁ to N ₄ , a photoelectric conversion signal based on the exposure period Te is obtained. The image creation unit 13d adds up the output signals N ₁ to N ₄ for each pixel 30 to obtain the pixel value (brightness value) of that pixel 30. The image creation unit 13d creates image data by performing this for all pixels 30.

For convenience of explanation, in this embodiment, the pixels 30 are assumed to have no color filters. In other words, the photoelectric conversion signal output from each pixel 30 does not have color information. Therefore, the image data created by the image creation unit 13d is a monochrome image having only brightness information. However, it is also possible to provide color filters to each pixel 30 and have the image creation unit 13d generate image data of a color image. For example, pixels 30 provided with color filters that pass only red component light, pixels 30 provided with color filters that pass only blue component light, and pixels 30 provided with color filters that pass only green component light are arranged on the imaging surface 20 according to a so-called Bayer array. The image creation unit 13d generates image data of a color image by performing a well-known demosaic process based on the pixel values obtained for each pixel 30.

The above describes how to create image data and depth map data when capturing still images. However, it is also possible to capture moving images, i.e., to create image data and depth map data for each frame. When capturing moving images, it is sufficient to reset, expose, and read out the pixels 30 for each row.

According to the imaging device according to the first embodiment described above, the following advantageous effects can be obtained.
(1) The first transmitted modulated light 3a and the second transmitted modulated light 3b, which have different periods, are reflected by the subject 2 (object) and enter the first pixel 30a and the second pixel 30b as the first received modulated light 4a and the second received modulated light 4b. The first detector 13a (first phase difference measuring unit) detects (measures) a phase difference θ _a between the phase of the first transmitted modulated light 3a and the phase of the first received modulated light 4a based on the output signal of the first pixel 30a. The second detector 13b (second phase difference measuring unit) detects (measures) a phase difference θ _b between the phase of the second transmitted modulated light 3b and the phase of the second received modulated light 4b based on the output signal of the second pixel 30b. The distance measuring unit 13c calculates a distance L (distance value) related to the subject 2 based on the detected (measured) phase difference. In this way, a single modulated light emission and image capture can be used to perform high-precision distance measurement with a wide measurement range while removing the influence of background light.

(2) The imaging element 12 sequentially reads out the output signals of the first pixels 30a as output signals N ₁ to N ₄ (first to fourth signals) with a predetermined phase delay during one period of the first transmitted modulated light 3a. The first detection unit 13a measures the phase difference _{θ a} based on the difference between the output signals N ₁ and N ₃ and the difference between the output signals N ₂ and N _4. The imaging element 12 sequentially reads out the output signals of the second pixels 30b as output signals N ₁ to N ₄ (fifth to eighth signals) with a predetermined phase delay during one period of the second transmitted modulated light 3b. The second detection unit 13b measures the phase difference θ _b based on the difference between the output signals N ₁ and N ₃ and the difference between the output signals N ₂ and N _4. This makes it possible to remove the influence of background light from the photoelectric conversion signal.

(3) Each of the first pixel 30a and the second pixel 30b has a microlens and a photoelectric conversion unit 32. The imaging element 12 sequentially reads out the output signals N ₁ to N ₄ from the photoelectric conversion unit 32 of the first pixel 30a. The imaging element 12 sequentially reads out the output signals N ₁ to N ₄ from the photoelectric conversion unit 32 of the second pixel 30b. In this manner, the light incident on one pixel 30 can be effectively utilized.

(4) The image sensor 12 has a first readout unit 331, a second readout unit 332, a third readout unit 333, and a fourth readout unit 334 that respectively read out the output signals N ₁ to N ₄ from the photoelectric conversion unit 32 of the first pixel 30 a. The image sensor 12 has a first readout unit 331, a second readout unit 332, a third readout unit 333, and a fourth readout unit 334 that respectively read out the output signals N ₁ to N ₄ from the photoelectric conversion unit 32 of the second pixel 30 b. In this manner, the output signals N ₁ to N ₄ can be easily obtained.

(5) The image creation unit 13d (image data generation unit) generates image data relating to the image of the subject 2 formed by the optical system 11 (imaging optical system) based on an addition signal obtained by adding the output signals _N1 to _N4 of the first pixels 30a and an addition signal obtained by adding the output signals _N1 to _N4 of the second pixels 30b. In this manner, image data based on normal visible light can be obtained simultaneously with the depth map data.

(6) The first pixels 30a and the second pixels 30b are each substantially uniformly distributed over the entire imaging surface 20 of the image sensor 12, and the distance measuring unit 13c calculates the distance L for the first pixels 30a and the second pixels 30b located near each other based on the phase differences θ _a and θ _b measured by the first detection unit 13a and the second detection unit 13b, respectively. In this manner, it is possible to create depth map data with the same number of elements as the number of pixels.

(7) Based on the distance L for each first pixel 30a and the distance L for each second pixel 30b, the distance measuring unit 13c creates depth map data that represents a map of the distance L at the positions of the first pixel 30a and the second pixel 30b. In this way, it is possible to obtain the distance to each part of the subject 2, not just the main part of the subject 2.

(8) The light source unit 10 emits modulated light 3, which is visible light, to the subject 2. In this way, image data based on normal visible light can be obtained simultaneously with the depth map data.

Second Embodiment
11 is a cross-sectional view showing a schematic configuration of an image pickup device 100 according to the second embodiment. In the following description, differences from the image pickup device 1 according to the first embodiment will be mainly described, and descriptions of other parts will be omitted.

The imaging device 100 has a configuration in which the imaging element 12 of the imaging device 1 according to the first embodiment is replaced with an imaging element 112, and the control unit 13 is replaced with a control unit 113. The imaging element 112 has a pixel 130 described below instead of the pixel 30 described above. The control unit 113 has a first detection unit 13a, a second detection unit 13b, a distance measurement unit 13c, an image creation unit 13d, and a light emission control unit 13e, similar to the control unit 13 according to the first embodiment, and further has a focus detection unit 13f and a focus adjustment unit 13g. The focus detection unit 13f and the focus adjustment unit 13g will be described in detail later.

Figure 12 is a plan view showing the structure of pixel 130. As in the first embodiment, pixel 130 in this embodiment also has two types of pixels, first pixel 130a and second pixel 130b, and as in the first embodiment (i.e., as shown in Figure 2 (b)), first pixel 130a and second pixel 130b are arranged alternately. In other words, second pixel 130b are present above, below, left and right of first pixel 130a, and first pixel 130a are present above, below, left and right of second pixel 130b. Therefore, approximately the same number of first pixel 130a and second pixel 130b are uniformly (uniformly) arranged on imaging surface 20.

The pixel 130 has a first photoelectric conversion unit 132a and a second photoelectric conversion unit 132b instead of the photoelectric conversion unit 32. The shapes of the first photoelectric conversion unit 132a and the second photoelectric conversion unit 132b are approximately equal to a shape obtained by dividing the photoelectric conversion unit 32 in the x-y plane into two equal parts by a straight line along the y direction. Therefore, a pair of light beams that pass through a pair of regions of the exit pupil of the optical system 11 respectively enter the first photoelectric conversion unit 132a and the second photoelectric conversion unit 132b, respectively.

Figure 13 is a circuit diagram showing the configuration of the image sensor 12. Note that Figure 13 shows only the circuit diagram of one pixel 130. The pixel 130 includes a first readout section 331, a second readout section 332, a third readout section 333, and a fourth readout section 334. The first readout section 331 and the second readout section 332 read out a photoelectric conversion signal from the first photoelectric conversion section 132a. The third readout section 333 and the fourth readout section 334 read out a photoelectric conversion signal from the second photoelectric conversion section 132b. The configurations of the first readout section 331, the second readout section 332, the third readout section 333, and the fourth readout section 334 are the same as those in the first embodiment, so a description thereof will be omitted.

Figure 14 is a timing chart showing the shooting operation. During the shooting operation, the control unit 13 causes the image sensor 12 to reset the pixels 30. The vertical scanning circuit 22 outputs a signal φRST at time t11 based on a signal from the control unit 13. The signal φRST is output to the first photoelectric conversion unit 132a and the second photoelectric conversion unit 132b of all pixels 130. This resets the first photoelectric conversion unit 132a and the second photoelectric conversion unit 132b of all pixels 30.

At time t12 when the reset is completed, the light emission control unit 13e causes the light source unit 10 to start emitting the modulated light 3. In synchronization with the emission of the modulated light 3 by the light source unit 10, at time t12, the first transfer control circuit 23a and the second transfer control circuit 23b start outputting the signals φTX1, φTX2, φTX3, and φTX4. The light emission control unit 13e controls the light source unit 10 so that the position p5 where the transmitted first modulated light 3a is at its maximum coincides with the center position p6 of the pulse of the signal φTX1 output from the first transfer control circuit 23a to the first pixel 30a. The light emission control unit 13e controls the light source unit 10 so that the position p7 where the transmitted second modulated light 3b is at its maximum coincides with the center position p8 of the pulse of the signal φTX1 output from the second transfer control circuit 23b to the second pixel 30b. That is, the light emission control unit 13e synchronizes the modulated light 3 with the signals φTX1, φTX2, φTX3, and φTX4.

The first transfer control circuit 23a outputs signals φTX1, φTX2, φTX3, and φTX4 to the first pixel 130a. The first transfer control circuit 23a divides the period from time t12 to time t13, which is the period T _a after the first modulated light 3a transmitted, into two periods, the first half and the second half. The first transfer control circuit 23a turns on the signal φTX1 in one of the two periods and turns on the signal φTX2 in the other. That is, the signal φTX1 is a signal having a phase that differs from that of the signal φTX2 by 1/2 period (180 degrees). The first transfer control circuit 23a outputs a signal obtained by delaying the signal φTX1 by 1/4 period (90 degrees) as the signal φTX3. The first transfer control circuit 23a outputs a signal obtained by delaying the signal φTX2 by 1/4 period (90 degrees) as the signal φTX4. The first transfer control circuit 23a repeatedly outputs the above-mentioned signals φTX1, φTX2, φTX3, and φTX4 during the exposure period Te.

In the first embodiment, the signals φTX1 and φTX2 are signals that are on during the first and second quarter periods of one period (period T _a ) of the transmitted first modulated light 3a. In contrast, the signals φTX1 and φTX2 in this embodiment are signals that are on during the first and second half periods of one period (period T _a ) of the transmitted first modulated light 3a. That is, in this embodiment, the first readout unit 331 and the second readout unit 332 alternately read out photoelectric conversion signals in opposite phases from the first photoelectric conversion unit 132a during one period (period T _a ) of the transmitted first modulated light 3a.

In the first embodiment, the signals φTX3 and φTX4 are signals that are on in the third and fourth quarter periods of one period (period T _a ) of the transmitted first modulated light 3a. In contrast, the signals φTX3 and φTX4 in this embodiment are signals that are on in the second and third quarter periods of one period (period T _a ) of the transmitted first modulated light 3a and signals that are on in the first and fourth quarter periods. That is, in this embodiment, the third readout unit 333 and the fourth readout unit 334 alternately read out photoelectric conversion signals in opposite phases from the second photoelectric conversion unit 132b during one period (period T _a ) of the transmitted first modulated light 3a.

The second transfer control circuit 23b divides the period from time t12 to time t14, which is the period _Tb after the second transmission modulated light 3b, into two periods, the first half and the second half. The second transfer control circuit 23b turns on the signal φTX1 in one of the two periods and turns on the signal φTX2 in the other. That is, the signal φTX1 is a signal having a phase that differs from that of the signal φTX2 by 1/2 period (180 degrees). The second transfer control circuit 23b outputs a signal obtained by delaying the signal φTX1 by 1/4 period (90 degrees) as the signal φTX3. The second transfer control circuit 23b outputs a signal obtained by delaying the signal φTX2 by 1/4 period (90 degrees) as the signal φTX4. The second transfer control circuit 23b repeatedly outputs the above-mentioned signals φTX1, φTX2, φTX3, and φTX4 during the exposure period Te.

At time t15, which is an exposure period Te after time t12, the light emission control unit 13e causes the light source unit 10 to stop emitting modulated light 3. In synchronization with the stop of emission of modulated light 3, at time t15, the first transfer control circuit 23a and the second transfer control circuit 23b stop outputting the signals φTX1, φTX2, φTX3, and φTX4. At time t15, the floating diffusion FD1, floating diffusion FD2, floating diffusion FD3, and floating diffusion FD4 of the pixel 130 each store electric charges generated by the photoelectric conversion unit 32.

After time t15, the horizontal scanning circuit 21 and the vertical scanning circuit 22 start reading out the output signals N ₁ to N _4. The output signals N ₁ to N ₄ are read out row by row. First, at time t5, the vertical scanning circuit 22 outputs a signal φSEL(1) to the pixels 130 in the first row. As a result, the output signals N ₁ to N ₄ from each of the pixels 130 in the first row are output to the vertical signal lines V1 to V4. The output signals N ₁ and N ₂ are signals based on the charges photoelectrically converted by the first photoelectric conversion unit 132a. The output signals N ₃ and N ₄ are signals based on the charges photoelectrically converted by the second photoelectric conversion unit 132b. While the signal φSEL(1) is being output, the horizontal scanning circuit 21 outputs the output signals N ₁ to N ₄ from each of the pixels 130 in the first row to the control unit 13 one by one. When the output signals N ₁ to N ₄ for all the pixels 130 in the first row are output to the control unit 13, the vertical scanning circuit 22 then outputs a signal φSEL(2) to the pixels 130 in the second row. As a result, the output signals N ₁ to N ₄ from each of the pixels 30 in the second row are output to the vertical signal lines V1 to V4. While the signal φSEL(2) is being output, the horizontal scanning circuit 21 outputs the output signals N ₁ to N ₄ from each of the pixels 130 in the second row one by one to the control unit 13. The horizontal scanning circuit 21 and the vertical scanning circuit 22 repeat the above operation for all rows, thereby outputting the output signals N ₁ to N ₄ from all the pixels 130 to the control unit 13.

Next, a method for calculating the output value z _a by the first detector 13 a will be described. The first detector 13 a detects the photoelectric conversion result of the received first modulated light 4 a from the photoelectric conversion result of the reflected light ₄ , i.e., the photoelectric conversion result of the incident light having the first frequency f _a , based on the output signals N ₁ and N 2 output from the first pixel 130 a and the output signals N ₃ and N ₄ output from the first pixel 130 a.

In the first embodiment, N ₁ -N ₃ is considered in equation (2), but in this embodiment, N ₁ -N ₂ is considered instead. As shown in Fig. 15, g ₁ (t) -g ₂ (t) can be approximated by a sine wave, so the following equation (13) can be derived by the same procedure as described in the first embodiment.

Similarly, N ₃ -N ₄ is expressed by the following equation (14).

The output value z _a is expressed by the following equation (15).

Therefore, the distance measurement unit 13c can create depth map data, similar to the first embodiment.

Next, the image creation unit 13d will be described. As shown in FIG. 12, the pixel 130 according to this embodiment has a first photoelectric conversion unit 132a and a second photoelectric conversion unit 132b. The shapes of the first photoelectric conversion unit 132a and the second photoelectric conversion unit 132b are approximately equal to the shape of the photoelectric conversion unit 32 according to the first embodiment divided into two. Therefore, if the output signal of the first photoelectric conversion unit 132a and the output signal of the second photoelectric conversion unit 132b are added, a signal equivalent to the output signal of the photoelectric conversion unit 32 according to the first embodiment is obtained. The image creation unit 13d adds the output signals N ₁ to N ₄ for each pixel 130 to obtain the pixel value (luminance value) of the pixel 130. The image creation unit 13d performs this for all pixels 130 to create image data.

Next, the focus detection unit 13f will be described. The focus detection unit 13f creates a signal obtained by adding the output signal N ₁ and the output signal N ₂ and a signal obtained by adding the output signal N 3 and the output signal N ₄ for a plurality of pixels 130 (including the first pixel 130a and the second pixel 130b arranged alternately) arranged in a row in the x direction within a predetermined focus detection area provided on the imaging surface 20. The signal obtained by adding the output signal _{N 1 and} the output signal N ₂ is a signal based on the charge photoelectrically converted by the first photoelectric conversion unit 132a during the exposure period Te (FIG. 14). The signal obtained by adding the output signal N ₃ and the output signal N ₄ is a signal based on the charge photoelectrically converted by the second photoelectric conversion unit 132b during the exposure period Te. This pair of signals is a signal corresponding to a pair of light beams that have passed through a pair of regions of the exit pupil of the optical system 11. In other words, this pair of signals is a signal for performing focus detection using a so-called phase difference method. As is well known, phase difference focus detection is a focus detection method in which correlation calculation is performed while gradually changing the shift amount for a pair of signals, and the shift amount that has the largest correlation value, i.e., the phase difference, is obtained to detect the focus adjustment state of the optical system 11. The focus detection unit 13f performs well-known phase difference focus detection on the created pair of signals to detect the focus adjustment state (defocus amount) of the optical system 11.

The focus detection area can be located at any position on the imaging surface 20. Alternatively, multiple candidate areas may be provided on the imaging surface 20 in advance, and one of the candidate areas may be selected as the focus detection area. In the above explanation, an example of focus detection in the x direction was described, but focus detection can also be performed in a different direction (e.g., the y direction).

Next, the focus adjustment unit 13g will be described. The focus adjustment unit 13g adjusts the focus position of the optical system 11 based on either the depth map data created by the distance measurement unit 13c or the focus adjustment state of the optical system 11 detected by the focus detection unit 13f. The optical system 11 has a focus lens (not shown), and the focus adjustment unit 13g adjusts the focus of the optical system 11 by driving the focus lens in the optical axis direction using an actuator (not shown).

When performing focus adjustment, the focus adjustment unit 13g first checks the current aperture value (F-number) of the optical system 11. If the aperture value is equal to or less than a predetermined threshold value, that is, if the aperture diameter of the aperture (not shown) of the optical system 11 is large and the parallax between the two pupils divided by pupil division is large, the focus adjustment unit 13g causes the focus detection unit 13f to detect the focus adjustment state, and performs focus adjustment based on the detected focus adjustment state.

On the other hand, if the current aperture value of the optical system 11 is greater than a predetermined threshold value, i.e., the aperture diameter of the aperture (not shown) of the optical system 11 is small and the parallax between the two pupils divided by pupil division is small, the focus adjustment unit 13g causes the distance measurement unit 13c to create depth map data that covers at least the range of the focus detection area, and performs focus adjustment based on the created depth map data. For example, if it is determined from the depth map data that the distance (depth) of the subject corresponding to the focus detection area is 1 m, the focus adjustment unit 13g drives the focus lens (not shown) to focus at a position of 1 m.

According to the imaging device according to the second embodiment described above, the following advantageous effects can be obtained.
(1) The first transmitted modulated light 3a and the second transmitted modulated light 3b, which have different periods, are reflected by the subject 2 (object) and enter the first pixel 130a and the second pixel 130b as the first received modulated light 4a and the second received modulated light 4b. The first detector 13a (first phase difference measuring unit) detects (measures) a phase difference θ _{a between the phase of the first transmitted modulated light 3a and the phase of the first received modulated light 4a based on the output signal of the first pixel 130a. The second detector 13b (second phase difference measuring unit) detects (measures) a phase difference θ b between} the phase of the second transmitted modulated light 3b and the phase of the second received modulated light 4b based on the output signal of the second pixel 130b. The distance measuring unit 13c calculates a distance L (distance value) related to the subject 2 based on the detected (measured) phase difference. In this way, a high-precision distance measurement with a wide measurement range can be performed by emitting modulated light and taking a picture once.

(2) The image sensor 12 sequentially reads out the output signals of the first pixels 130a as output signals N ₁ to N ₄ (first to fourth signals) with a predetermined phase delay during one period of the first transmitted modulated light 3a. The first detector 13a measures the phase difference _{θ a} based on the difference between the output signals N ₁ and N ₂ and the difference between the output signals N ₃ and N _4. The image sensor 12 sequentially reads out the output signals of the second pixels 130b as output signals N ₁ to N ₄ (fifth to eighth signals) with a predetermined phase delay during one period of the second transmitted modulated light 3b. The second detector 13b measures the phase difference θ _b based on the difference between the output signals N ₁ and N ₂ and the difference between the output signals N ₃ and N _4. This makes it possible to remove the influence of background light from the photoelectric conversion signal.

(3) Each of the first pixel 130a and the second pixel 130b has a microlens 31, a first photoelectric conversion unit 132a, and a second photoelectric conversion unit 132b. The image sensor 12 reads out the output signals N ₁ and N ₂ of the output signals N ₁ to N ₄ from the first photoelectric conversion unit 132a of the first pixel 130a, and reads out the output signals N ₃ and N ₄ of the output signals N ₁ to N ₄ from the second photoelectric conversion unit 132b. The image sensor 12 reads out the output signals N ₁ and N ₂ of the output signals N ₁ to N ₄ from the first photoelectric conversion unit 132a of the second pixel 130b, and reads out the output signals N ₃ and N ₄ of the output signals N ₁ to N ₄ from the second photoelectric conversion unit 132b. In this manner, light incident on one pixel 30 can be pupil-divided and photoelectrically converted.

(4) The image sensor 12 has a first readout section 331 and a second readout section 332 that respectively read out the output signals N ₁ and N ₂ from the first photoelectric conversion section 132a of the first pixel 130a, and a third readout section 333 and a fourth readout section 334 that respectively read out the output signals N ₃ and N ₄ from the second photoelectric conversion section 132b of the first pixel 130a. The image sensor 12 has a first readout section 331 and a second readout section 332 that respectively read out the output signals N ₁ and N ₂ from the first photoelectric conversion section 132a of the second pixel 130b, and a third readout section 333 and a fourth readout section 334 that respectively read out the output signals N ₃ and N ₄ from the second photoelectric conversion section 132b of the second pixel 130b. By doing so, the output signals N ₁ to N ₄ can be easily obtained.

(5) The first readout unit 331 of the first pixel 130a reads out the output signal _N1 from the first photoelectric conversion unit 132a of the first pixel 130a during one half period of the cycle of the transmitted first modulated light 3a. The second readout unit 332 of the first pixel 130a reads out the output signal N2 from the first photoelectric conversion unit 132a of the first pixel 130a during the remaining half period of the cycle of the transmitted first modulated light 3a. The third readout unit 333 of the first pixel 130a reads out the output signal _N3 from the second photoelectric conversion unit 132b of the first pixel 130a with a predetermined time delay from the reading by the first readout unit ₃₃₁ during one half period of the cycle of the transmitted first modulated light 3a. The fourth readout unit 334 of the first pixel 130a reads out an output signal _N4 from the second photoelectric conversion unit 132b of the first pixel 130a during the remaining half of one period of the first transmitted modulated light 3a with a predetermined time delay from the readout by the second readout unit 332. The first readout unit 331 of the second pixel 130b reads out an output signal _N1 from the first photoelectric conversion unit 132a of the second pixel 130b during the remaining half of one period of the second transmitted modulated light 3b. The second readout unit 332 of the second pixel 130b reads out an output signal _N2 from the first photoelectric conversion unit 132a of the second pixel 130b during the remaining half of one period of the second transmitted modulated light 3b. The third readout unit 333 of the second pixel 130b reads out the output signal _N3 from the second photoelectric conversion unit 132b of the second pixel 130b during a half period of one period of the transmitted second modulated light 3b with a predetermined time delay from the readout by the first readout unit 331. The fourth readout unit 334 of the second pixel 130b reads out the output signal _N4 from the second photoelectric conversion unit 132b of the second pixel 130b during the remaining half period of one period of the transmitted second modulated light 3b with a predetermined time delay from the readout by the second readout unit 332. In this manner, it is possible to read out a signal that can be used for distance measurement based on the optical time-of-flight measurement method and can also be used for distance measurement by a so-called phase difference detection method.

(6) The image creation unit 13d (image data generation unit) generates image data relating to the image of the _{subject 2 formed by the optical system 11 (imaging optical system) based on an addition signal obtained by adding the output signals N 1 to} N _{4 of the first pixels 130a and an addition signal obtained by adding the output signals N 1 to N 4 of} the second pixels 130b. In this manner, it is possible to create depth map data while performing focus detection using a so-called phase difference detection method, and further create image data of the subject 2.

(7) The focus detection unit 13f (phase difference focus detection unit) calculates the defocus amount based on the pair of signals read out from the first photoelectric conversion unit 132a and the second photoelectric conversion unit 132b of the first pixel 130a and the second pixel 130b. In this way, focus detection using the so-called phase difference detection method can be performed in parallel with the creation of depth map data.

(8) The focus detection unit 13f calculates the defocus amount based on a pair of an addition signal obtained by adding together the output signals _N1 and _N2 read out from the first photoelectric conversion unit 132a of the first pixel 130a and the output signals _N3 and _N4 read out from the second photoelectric conversion unit 132b. Since this is done, focus detection using a so-called phase difference detection method can be performed in parallel with the creation of depth map data.

(9) The first pixels 130a and the second pixels 130b are each substantially uniformly distributed across the entire imaging surface 20 of the image sensor 12, and the distance measurement unit 13c calculates the distance L for the first pixels 130a and the second pixels 130b located near each other based on the phase differences θ _a and θ _b measured by the first detection unit 13a and the second detection unit 13b, respectively. In this manner, it is possible to create depth map data with the same number of elements as the number of pixels.

(10) Based on the distance L for each first pixel 130a and the distance L for each second pixel 130b, the distance measuring unit 13c creates depth map data that represents a map regarding the distance L at the positions of the first pixel 130a and the second pixel 130b. In this way, it is possible to obtain the distance to each part of the subject 2, not just the main part of the subject 2.

(11) When the aperture value of the optical system 11 is greater than a predetermined value (dark), the focus adjustment unit 13g performs focus adjustment based on the distance L calculated by the distance measurement unit 13c, and when the aperture value of the optical system 11 is equal to or less than a predetermined value (bright), the focus adjustment unit 13g performs focus adjustment based on the focus adjustment state detected by the focus detection unit 13f. As a result, focus adjustment can be performed even when the aperture value is large (dark) and focus detection using the phase difference method cannot be performed.

The following modifications are also within the scope of the present invention, and one or more of the modifications may be combined with the above-described embodiment.

(Variation 1)
The number and layout of the first pixels 30a and the second pixels 30b (the first pixels 130a and the second pixels 130b) may be different from those in the above-described embodiment. For example, in the first embodiment, as shown in FIG. 2B, the first pixels 30a and the second pixels 30b are arranged in a staggered manner in a substantially equal number. In contrast, in the arrangement shown in FIG. 16A, the entire imaging surface 20 is divided into 2×2 blocks of four pixels, and one block is composed of three first pixels 30a and one second pixel 30b. In this way, the first pixels 30a may be more numerous than the second pixels 30b.

10, the phase difference θ _b detected by the second pixel 30 b is used to determine the true phase difference Θ from the phase difference θ _a . In other words, the detection accuracy required for the phase difference θ _b is not as high as that for the phase difference θ _a . Therefore, it is sufficient that the second pixel 30 b exists in a number sufficient to determine the true phase difference Θ from the phase difference θ _a .

That is, as shown in FIG. 16B, when the imaging surface 20 is divided into a plurality of blocks 200 of M×N, if at least one second pixel 30b exists in one block 200, the depth map data can be created. In this case, the position of the second pixel 30b may be irregular as shown in FIG. 16B. In this case, for a position where the first pixel 30a exists, the distance L corresponding to the position is calculated using the output value z _a determined from the output of the first pixel 30a and the output value z _b determined from one second pixel 30b in the block 200 to which the first pixel 30a belongs. Also, for a position where the second pixel 30b exists, the distance L corresponding to the position is calculated using the output value z _b determined from the output of the second pixel 30b and the output value z _a determined from the output of the first pixel 30a existing in the vicinity of the second pixel 30b. Here, the output values z _a and z _b may be values determined from the output of a single pixel 30, or as in the first embodiment, may be values (e.g., average, median, mode, etc.) determined from a plurality of values determined from the outputs of a plurality of pixels 30. In other words, the output values z _a and z _b for calculating the distance L corresponding to a certain position may be the average value (calculated average, geometric mean, median, etc.) of the output values z _a and z _b obtained from the neighboring pixels 30.

In this way, by increasing the number of first pixels 30a and decreasing the number of second pixels 30b, the accuracy of the depth map data created by the distance measuring unit 13c is improved. Note that, by distributing the first pixels 30a and the second pixels 30b approximately uniformly over the entire imaging surface 20 of the imaging element 12 as shown in Fig. 2B and Fig. 16B, it is possible to ensure that the pairs of the first pixels 30a and the second pixels 30b regarding the phase differences _θa and _θb used by the distance measuring unit 13c to calculate the distance L are located close to each other.

As described in the first embodiment, the first pixel 30a and the second pixel 30b have the same structure. Therefore, a pixel 30 can function as the first pixel 30a at one time and as the second pixel 30b at another time. For example, when the first pixel 30a and the second pixel 30b are arranged in a layout as shown in FIG. 16(a), the first pixel 30a and the second pixel 30b can be switched depending on the characteristics of the subject.

(Variation 2)
As described with reference to Fig. 7, in the first embodiment, the first pixel 30a is configured to divide the first cycle T _a into four periods and output photoelectric conversion signals in each period as output signals N ₁ to N _4. Therefore, for example, integrating the output signals N ₁ to N ₄ will result in photoelectric conversion signals for the entire period of the first cycle T _a . However, even if photoelectric conversion signals obtained by a different control are used as the output signals N ₁ to N ₄ , depth map data can be created in the same manner as in the above-described embodiment.

7, for example, if the signal φTX1 is turned on during the last period of the period from time t2 to _¼Ta , the corresponding photoelectric conversion signal may be the output signal _N1 . The signal φTX1 does not necessarily have to be turned on during the entire period from time t2 to _¼Ta . The same applies to the signals φTX2 to φTX4.

Another example will be described with reference to Fig. 14. For example, the signal φTX1 can be a signal that is turned on in the second quarter period of one period (period T _a ) of the first transmitted modulated light 3a, the signal φTX2 can be a signal that is turned on in the fourth quarter period, the signal φTX3 can be a signal that is turned on in the third quarter period, and the signal φTX4 can be a signal that is turned on in the first quarter period. In this case, the output signals _N1 , _N2 , _N3 , and _N4 are the same signals as the output signals _N1 , _N3 , _N2 , and _N4 in the first embodiment, respectively, so that the same calculation as in the first embodiment can be performed.

(Variation 3)
As described with reference to FIG. 14, in the second embodiment, the signals φTX3 and φTX4 (control signals _g3 (t), _g4 (t)) are signals having a phase delayed by ¼ of the first period _Ta with respect to the signals φTX1 and φTX2 (control signals _g1 (t), _g2 (t)). However, the signals φTX3 and φTX4 may also be signals having a different phase. It is sufficient that the phase is at least different from that of the signals φTX1 and φTX2.

(Variation 4)
In each of the above-described embodiments, the first modulated light 3a and the second modulated light 3b are sine waves having different periods. The first modulated light 3a and the second modulated light 3b may be light modulated in a different form. A modified example using such modulated light 3 will be described below.

Figure 17 is a circuit diagram showing the configuration of the image sensor 12 according to the fourth modification. Note that Figure 17 only shows the circuit diagrams of the first pixel 32a and the second pixel 32b. The first pixel 32a includes a photoelectric conversion unit 32, a first readout unit 331, and a second readout unit 332. The second pixel 32b includes a photoelectric conversion unit 32, a third readout unit 333, and a fourth readout unit 334.

18 is a timing chart showing the imaging operation. In the fourth modification, the periods of the first transmitted modulated light 3a and the second transmitted modulated light 3b are both the second period _Tb (FIG. 4). The first transmitted modulated light 3a has an amplitude _A1 only during a period from time t21 to half the first period _Tb (FIG. 4) in the period, and the amplitude is zero during the other periods. The second transmitted modulated light 3b has an amplitude _A2 only during a period from time t21 to half the second period _Tb in the period, and the amplitude is zero during the other periods. That is, the first transmitted modulated light 3a and the second transmitted modulated light 3b are pulse modulated lights having the same period but different duty ratios.

18, the modulated light 3 obtained by superposing the first modulated light 3a and the second modulated light 3b has an amplitude _A3 (= _A1 + _A2 ) in the period from time t21 to time t22, and has an amplitude _A2 in the period from time t22 to time t24. Here, time t22 is 1/2 the first period T _a after time t21, and time t24 is 1/2 the second period T _b after time t21. It is also assumed that the amplitude _A1 is sufficiently larger than the amplitude _A2 .

In the fourth modification, the signal φTX1 is on and the signal φTX2 is off during the period from time t21 to time t22, which is ½ of the first period T _a later. Also, the signal φTX1 is off and the signal φTX2 is on during the period from time t22 to time t23, which is ½ of the first period T _a later.

On the other hand, for the second pixel 30b, at time t21, the signal φTX3 is on and the signal φTX4 is off during the period from time t21 to time t24, which is ½ of the second period _Tb later. Also, during the period from time t24 to time t25, which is ½ of the second period _Tb later, the signal φTX3 is off and the signal φTX4 is on.

The floating diffusion FD1 of the first pixel 32a accumulates electric charges obtained by photoelectrically converting the light 210 of the reflected light 4, which is shown by hatching in FIG. 18. In other words, the output signal _N1 output from the first pixel 32a is based on the amount of the light 210 shown by hatching in FIG. 18. Similarly, the output signal _N2 output from the first pixel 32a is based on the amount of the light 211 shown by hatching in FIG. 18. When the phase difference _θa between the transmitted first modulated light 3a and the received first modulated light 4a is zero, all electric charges obtained by photoelectrically converting the portion of the reflected light 4 having the amplitude _A1 are transferred to the floating diffusion FD1, and the output signal _N1 becomes much larger than the output signal _N2 . As the phase difference _θa increases, the proportion of electric charges obtained by photoelectrically converting the portion of the reflected light 4 having the amplitude _A1 that are transferred to the floating diffusion FD2 increases. That is, the phase difference _θa can be obtained by calculating the ratio between the output signal _N1 and the output signal _N2 . The first pixel 32a photoelectrically converts not only the portion of the reflected light 4 having the amplitude _A1 , but also the portion having the amplitude _A2 . However, if the amplitude _A1 is made sufficiently larger than the amplitude _A2 , the amount of the output signal _N1 and the output signal _N2 is dominated by the influence of the portion having the amplitude _A1 , and the influence of the portion having the amplitude _A2 can be ignored.

The floating diffusion FD3 of the second pixel 32b accumulates electric charges obtained by photoelectrically converting the light 212 shown by hatching in FIG. 18 out of the reflected light 4. In other words, the output signal _N3 output from the second pixel 32b is based on the amount of the light 212 shown by hatching in FIG. 18. Similarly, the output signal _N4 output from the second pixel 32b is based on the amount of the light 213 shown by hatching in FIG. 18. When the phase difference _θb between the transmitted second modulated light 3b and the received second modulated light 4b is zero, all of the electric charges obtained by photoelectrically converting the reflected light 4 are transferred to the floating diffusion FD3, and the output signal _N3 becomes much larger than the output signal _N4 . As the phase difference _θb increases, the proportion of the electric charges obtained by photoelectrically converting the reflected light 4 transferred to the floating diffusion FD4 increases. In other words, the phase difference _θb can be obtained by calculating the ratio between the output signal _N3 and the output signal _N4 . The output signal N3 reflects the result of photoelectric conversion of the portion of the reflected light 4 having the amplitude _A1 . However, if the period during which the amplitude is _A1 (the duty ratio of the transmitted first modulated light 3a) is made sufficiently small, the influence of the portion having the amplitude _A1 on the output signal _N3 becomes negligibly small.

As described above, instead of using high-frequency modulated light as in the first and second embodiments, it is also possible to use pulsed light as the modulated light.

(Variation 5)
The modulated light 3 emitted by the light source unit 10 to the subject 2 may be infrared light (for example, near-infrared light) instead of visible light. In this way, particularly when the subject 2 is a person, the modulated light 3 is invisible to the eye, the subject 2 does not feel bothered by the modulated light 3, and shooting can be performed smoothly.

(Variation 6)
In the above-described embodiment, the light source unit 10 is provided outside the housing of the imaging device 1 (for example, FIG. 1). The light source unit 10 can also be provided inside the housing of the imaging device 1. For example, a half mirror is inserted obliquely between the optical system 11 and the imaging element 12, and the light source unit 10 is provided to emit modulated light 3 in the y direction relative to the half mirror. In this way, the modulated light 3 emitted by the light source unit 10 is reflected by the half mirror and travels toward the optical system 11, i.e., toward the subject 2. The reflected light 4 passes through the optical system 11 and the half mirror and enters the imaging element 12. This increases the degree of freedom in the design of the housing of the imaging device 1.

(Variation 7)
In the above-described embodiment, the light source unit 10 emits modulated light 3 consisting of the first transmitted modulated light 3a and the second transmitted modulated light 3b to the subject 2, but the first transmitted modulated light 3a and the second transmitted modulated light 3b may be emitted separately to the subject 2. For example, the first light source unit emitting the first transmitted modulated light 3a and the second light source unit emitting the second transmitted modulated light 3b may be provided at different positions in the imaging device 1.

(Variation 8)
In the above embodiment, the imaging device is capable of generating image data together with depth map data, but the imaging device does not necessarily have to generate image data. For example, the imaging device may generate only depth map data.

(Variation 9)
In the second embodiment, instead of reading out the output signals _N1 and _N2 from the first photoelectric conversion unit 132a and reading out the output signals _N3 and _N4 from the second photoelectric conversion unit 132b, the output signals _N1 and _N3 may be read out from the first photoelectric conversion unit 132a and the output signals _N2 and _N4 may be read out from the second photoelectric conversion unit 132b. Also, the output signals _N1 and _N4 may be read out from the first photoelectric conversion unit 132a and the output signals _N2 and _N3 may be read out from the second photoelectric conversion unit 132b.

(Variation 10)
In the first embodiment, as shown in Fig. 3, four vertical signal lines V1 to V4 are provided for outputting output signals _N1 to _N4 to one column of pixels 30, but only two vertical signal lines V1 and V3 may be provided. In this case, the output signals _N1 and _N2 are output to the vertical signal line V1 in a time-division manner, and the output signals _N3 and _N4 are output to the vertical signal line V3 in a time-division manner. Similarly, only the vertical signal line V1 may be provided, and the output signals _N1 to _N4 may be output to the vertical signal line V1 in a time-division manner.

As long as the characteristics of the present invention are not impaired, the present invention is not limited to the above-described embodiment, and other forms that are conceivable within the scope of the technical concept of the present invention are also included within the scope of the present invention.

Reference Signs List 1, 100...imaging device, 10...light source unit, 11...optical system, 12, 112...imaging element, 13, 113...control unit, 13a...first detection unit, 13b...second detection unit, 13c...distance measurement unit, 13d...image creation unit, 13e...light emission control unit, 13f...focus detection unit, 13g...focus adjustment unit

Claims (20)

1. An imaging device, comprising:
A light source unit that emits light to an object;
an imaging element including: a first pixel including a first photoelectric conversion unit that converts light into an electric charge; a second pixel arranged alongside the first pixel in the row direction and including a second photoelectric conversion unit that converts light into an electric charge; a first transfer control circuit that controls transfer of the electric charge converted by the first photoelectric conversion unit; and a second transfer control circuit that controls transfer of the electric charge converted by the second photoelectric conversion unit;
a distance measuring unit that calculates a distance from the imaging device to the object based on a signal read from the first pixel and a signal read from the second pixel;
Equipped with
the light source unit emits a first modulated light modulated to a sine wave of a first period and a second modulated light modulated to a sine wave of a second period different from the first period;
The first pixel is
a first transfer unit that transfers the charges converted by the first photoelectric conversion unit to a first accumulation unit;
a second transfer unit that transfers the charges converted by the first photoelectric conversion unit to a second accumulation unit;
a third transfer unit that transfers the charges converted by the first photoelectric conversion unit to a third accumulation unit;
a fourth transfer unit that transfers the charges converted by the first photoelectric conversion unit to a fourth accumulation unit;
having
The second pixel is
a fifth transfer unit that transfers the charges converted by the second photoelectric conversion unit to a fifth accumulation unit;
a sixth transfer unit that transfers the electric charges converted by the second photoelectric conversion unit to a sixth accumulation unit;
a seventh transfer unit that transfers the charges converted by the second photoelectric conversion unit to a seventh accumulation unit;
an eighth transfer unit that transfers the electric charge converted by the second photoelectric conversion unit to an eighth accumulation unit;
having
the first transfer control circuit controls the first transfer unit, the second transfer unit, the third transfer unit, and the fourth transfer unit so that the charges converted by the first photoelectric conversion unit are sequentially transferred to the first storage unit, the second storage unit, the third storage unit, and the fourth storage unit in the first period;
the second transfer control circuit controls the fifth transfer unit, the sixth transfer unit, the seventh transfer unit, and the eighth transfer unit so that the charges converted by the second photoelectric conversion unit are sequentially transferred to the fifth accumulation unit, the sixth accumulation unit, the seventh accumulation unit, and the eighth accumulation unit in the second period.
Imaging device . 2. The imaging device according to claim 1,
the first transfer control circuit outputs a first pulse signal having a first pulse width to the first transfer unit, the second transfer unit, the third transfer unit, and the fourth transfer unit;
the second transfer control circuit outputs a second pulse signal having a second pulse width different from the first pulse width to the fifth transfer unit, the sixth transfer unit, the seventh transfer unit, and the eighth transfer unit;
Imaging device. 3. The imaging device according to claim 2,
the first transfer control circuit outputs the first pulse signal to the first transfer unit, the second transfer unit, the third transfer unit, and the fourth transfer unit so that the charges converted by the first photoelectric conversion unit are sequentially transferred to the first storage unit, the second storage unit, the third storage unit, and the fourth storage unit in the first period;
the second transfer control circuit outputs the second pulse signal to the fifth transfer unit, the sixth transfer unit, the seventh transfer unit, and the eighth transfer unit so that the charges converted by the second photoelectric conversion unit are sequentially transferred to the fifth storage unit, the sixth storage unit, the seventh storage unit, and the eighth storage unit in the second period.
Imaging device. 4. The imaging device according to claim 1,
a first detection unit that performs a calculation using a signal read from the first pixel;
a second detection unit that performs a calculation using a signal read from the second pixel;
Equipped with
The distance measuring unit calculates a distance from the imaging device to the object based on a calculation result of the first detection unit and a calculation result of the second detection unit.
Imaging device. 5. The imaging device according to claim 4,
The first detection unit calculates a phase difference between the first modulated light emitted from the light source unit and the first modulated light reflected by the object among the first modulated light emitted from the light source unit,
The second detection unit calculates a phase difference between the second modulated light emitted from the light source unit and the second modulated light reflected by the object among the second modulated light emitted from the light source unit.
Imaging device. 6. The imaging device according to claim 5,
The distance measuring unit corrects the phase difference calculated by the first detection unit based on the calculation result of the second detection unit.
Imaging device. 7. The imaging device according to claim 4,
The first detection unit is
a first signal based on the charges transferred to the first storage unit by the first transfer unit;
a second signal based on the charges transferred to the second storage unit by the second transfer unit;
a third signal based on the charges transferred to the third storage unit by the third transfer unit; and
a fourth signal based on the charges transferred to the fourth storage unit by the fourth transfer unit;
The calculation is performed using
The second detection unit is
a fifth signal based on the charges transferred to the fifth accumulation unit by the fifth transfer unit;
a sixth signal based on the charges transferred to the sixth accumulation unit by the sixth transfer unit;
a seventh signal based on the charges transferred to the seventh accumulation unit by the seventh transfer unit;
an eighth signal based on the charges transferred to the eighth storage unit by the eighth transfer unit;
The calculation is performed using
Imaging device. 8. The imaging device according to claim 7,
An imaging device comprising an image creation unit that generates image data using the first signal, the second signal, the third signal, the fourth signal, the fifth signal, the sixth signal, the seventh signal, and the eighth signal. 9. The imaging device according to claim 8,
The image creation unit includes:
a signal obtained by adding together the first signal, the second signal, the third signal, and the fourth signal;
a signal obtained by adding the fifth signal, the sixth signal, the seventh signal, and the eighth signal;
Generate image data using
Imaging device. 10. The imaging device according to claim 7,
The imaging element includes:
a first signal line through which the first signal is output;
a second signal line through which the second signal is output;
a third signal line through which the third signal is output;
a fourth signal line through which the fourth signal is output;
a fifth signal line through which the fifth signal is output;
a sixth signal line through which the sixth signal is output;
a seventh signal line through which the seventh signal is output;
an eighth signal line through which the eighth signal is output;
An imaging device having the above configuration. 11. The imaging device according to claim 1,
The first pixel is
a first reset unit that resets the potential of the first storage unit;
a second reset unit that resets the potential of the second storage unit;
a third reset unit that resets the potential of the third storage unit;
a fourth reset unit that resets the potential of the fourth storage unit;
having
The second pixel is
a fifth reset unit that resets the potential of the fifth storage unit;
a sixth reset unit that resets the potential of the sixth storage unit;
a seventh reset unit that resets the potential of the seventh storage unit;
an eighth reset unit that resets the potential of the eighth storage unit;
having
Imaging device. 12. The imaging device according to claim 11,
the first reset unit, the second reset unit, the third reset unit, the fourth reset unit, the fifth reset unit, the sixth reset unit, the seventh reset unit, and the eighth reset unit are electrically connected to a common control line through which a control signal for controlling the first reset unit, the second reset unit, the third reset unit, the fourth reset unit, the fifth reset unit, the sixth reset unit, the seventh reset unit, and the eighth reset unit is output;
Imaging device. 13. The imaging device according to claim 1,
The second pixel is disposed adjacent to the first pixel in the row direction.
Imaging device. 14. The imaging device according to claim 13,
The first pixels and the second pixels are arranged alternately in the row direction.
Imaging device. 15. The imaging device according to claim 1,
the amplitude of the second modulated light is different from the amplitude of the first modulated light;
Imaging device. 1. An imaging device, comprising:
A light source unit that emits light to an object;
an imaging element having: a first pixel including a first photoelectric conversion unit that converts light from a first microlens into an electric charge and a second photoelectric conversion unit that converts the light from the first microlens into an electric charge; a second pixel that is arranged alongside the first pixel in the row direction and includes a third photoelectric conversion unit that converts light from a second microlens into an electric charge and a fourth photoelectric conversion unit that converts light from the second microlens into an electric charge; a first transfer control circuit that controls transfer of the electric charge converted by the first photoelectric conversion unit; and a second transfer control circuit that controls transfer of the electric charge converted by the second photoelectric conversion unit;
a distance measuring unit that calculates a distance from the imaging device to the object based on a signal read from the first pixel and a signal read from the second pixel;
Equipped with
the light source unit emits a first modulated light modulated to a sine wave of a first period and a second modulated light modulated to a sine wave of a second period different from the first period;
The first pixel is
a first transfer unit that transfers the charges converted by the first photoelectric conversion unit to a first accumulation unit;
a second transfer unit that transfers the charges converted by the first photoelectric conversion unit to a second accumulation unit;
a third transfer unit that transfers the charges converted by the second photoelectric conversion unit to a third accumulation unit;
a fourth transfer unit that transfers the electric charge converted by the second photoelectric conversion unit to a fourth accumulation unit;
having
The second pixel is
a fifth transfer unit that transfers the charges converted by the third photoelectric conversion unit to a fifth accumulation unit;
a sixth transfer unit that transfers the electric charges converted by the third photoelectric conversion unit to a sixth accumulation unit;
a seventh transfer unit that transfers the electric charges converted by the fourth photoelectric conversion unit to a seventh accumulation unit;
an eighth transfer unit that transfers the electric charge converted by the fourth photoelectric conversion unit to an eighth accumulation unit;
having
the first transfer control circuit controls the first transfer unit, the second transfer unit, the third transfer unit, and the fourth transfer unit so that the charges converted in the first photoelectric conversion unit and the charges converted in the second photoelectric conversion unit are transferred to the first accumulation unit, the second accumulation unit, the third accumulation unit, and the fourth accumulation unit in the first period;
the second transfer control circuit controls the fifth transfer unit, the sixth transfer unit, the seventh transfer unit, and the eighth transfer unit so that the charges converted in the third photoelectric conversion unit and the charges converted in the fourth photoelectric conversion unit are transferred to the fifth accumulation unit, the sixth accumulation unit, the seventh accumulation unit, and the eighth accumulation unit in the second period;
Imaging device. 17. The imaging device according to claim 16,
a first detection unit that performs a calculation using a signal read from the first pixel;
a second detection unit that performs a calculation using a signal read from the second pixel;
Equipped with
The distance measuring unit calculates a distance from the imaging device to the object based on a calculation result of the first detection unit and a calculation result of the second detection unit.
Imaging device. 18. The imaging device according to claim 17,
The first detection unit is
a first signal based on the charges transferred to the first storage unit by the first transfer unit;
a second signal based on the charges transferred to the second storage unit by the second transfer unit;
a third signal based on the charges transferred to the third storage unit by the third transfer unit; and
a fourth signal based on the charges transferred to the fourth storage unit by the fourth transfer unit;
The calculation is performed using
The second detection unit is
a fifth signal based on the charges transferred to the fifth accumulation unit by the fifth transfer unit;
a sixth signal based on the charges transferred to the sixth accumulation unit by the sixth transfer unit;
a seventh signal based on the charges transferred to the seventh accumulation unit by the seventh transfer unit;
an eighth signal based on the charges transferred to the eighth storage unit by the eighth transfer unit;
The calculation is performed using
Imaging device. 20. The imaging device according to claim 18,
An imaging device comprising a focus adjustment unit that adjusts a focus position of an optical system that emits light to the imaging element using the first signal, the second signal, the third signal, the fourth signal, the fifth signal, the sixth signal, the seventh signal, and the eighth signal. 20. The imaging device according to claim 19,
a signal obtained by adding together a signal based on the charges transferred to the first storage section by the first transfer section and a signal based on the charges transferred to the second storage section by the second transfer section;
a signal obtained by adding a signal based on the charges transferred to the third storage section by the third transfer section and a signal based on the charges transferred to the fourth storage section by the fourth transfer section;
a signal obtained by adding together a signal based on the charges transferred to the fifth accumulation unit by the fifth transfer unit and a signal based on the charges transferred to the sixth accumulation unit by the sixth transfer unit;
a signal obtained by adding a signal based on the charges transferred to the seventh storage section by the seventh transfer section and a signal based on the charges transferred to the eighth storage section by the eighth transfer section;
a focus detection unit that calculates a defocus amount of the optical system using
the focus adjustment unit adjusts a focus position of the optical system based on the defocus amount calculated by the focus detection unit.
Imaging device.

Images