JP2011091584A - Imaging device, imaging method and electronic equipment - Google Patents

Abstract

An imaging apparatus, an imaging method, and an electronic apparatus suitable for determining a ratio of exposure amount theoretically and dynamically in multistage exposure are provided.
An image pickup apparatus includes an image pickup element, memories, an image composition unit, a control unit, an image analysis unit, and an operation unit. The ratio “L2 / L1” and “L3 / L2” of two adjacent exposure amounts in the three types of exposure amounts L1 to L3 (L1 <L2 <L3) are equal (L2 / L1 = L3 / L2). The ratio information “T1: T2: T3 (T2 / T1 = T3 / T2)” of the exposure times T1 to T3 (T1 <T2 <T3) is determined, and the imaging device 10 determines the exposure time determined by the control unit 15. The imaging processing is performed based on the ratio information.
[Selection] Figure 1

Description

  The present invention relates to an imaging apparatus, an imaging method, an electronic device, and the like suitable for generating image data with an expanded dynamic range by combining a plurality of types of captured image data having different exposure amounts.

  Conventionally, image data (HDR image) in which the same subject is imaged with a plurality of types of exposure amounts (exposure times), and pixel signals corresponding to the plurality of types of exposure amounts obtained by the imaging are combined to expand the dynamic range. Data). In such a technique, noise (false color, pseudo contour, etc.) generated in a boundary region between an image portion corresponding to each exposure amount and an image portion corresponding to another exposure amount in the composite image is often a problem. .

  For example, the same subject is imaged at exposure times T1, T2, and T3 (T1 <T2 <T3) corresponding to the exposure amounts L1, L2, and L3 (L1 <L2 <L3), and FIGS. It is assumed that the pixel signals S1, S2, and S3 having optical responses shown in FIG. In this case, as shown in FIGS. 8D to 8F, the pixel signals S1 and S2 are first multiplied by coefficients “T3 / T1” and “T3 / T2” with reference to S3 having the longest exposure time. And normalize. Next, the normalized “S1 × T3 / T1” and “S2 × T3 / T2” and “S3” are linearly synthesized. As a result, as shown in FIG. 8G, a captured image signal with an extended dynamic range (good S / N) is obtained.

  However, at the time of image capturing, images may be captured with different optical responses during the set exposure time due to external factors such as offset fluctuations. In such a case, the coefficient values “T3 / T1” and “T3 / T2” become inappropriate values, and the linearity of the captured image signal after synthesis may be lost. When the linearity is lost, noise that causes a false color, a pseudo contour, or the like is generated at the connecting portion of the broken line shown in FIG.

  In order to solve such a problem, Patent Document 1 seeks an accurate optical response inclination of a captured image. Further, in Patent Document 2, correction is performed so that the inclinations coincide with each other in consideration of the OB area (offset level) of the camera. In Patent Document 3, an accurate exposure amount ratio is obtained by analyzing a histogram of an image. Further, as a method for not analyzing the characteristics of an image, Patent Document 4 relaxes unnaturalness by applying a low-pass filter in the vicinity of a boundary region of a composite image.

JP-A-7-131708 JP 2001-352552 A Japanese Patent Application Laid-Open No. 8-214111 JP 2004-48445 A

By the way, even if the exact exposure amount (exposure ratio) of the image is known and the image is synthesized by multiplying an appropriate coefficient, if the ratio of T1: T2: T3 at the time of imaging is inappropriate, Noise components due to the S / N limit may occur in the combined image.
For example, as shown in FIGS. 8D to 8G, each exposure is performed in such a relationship that S2 complements the noise region (the wavy line portion in the figure) of the captured image signal S1 and S3 complements the noise region of S2. If the time is set, the above problem does not occur. However, as shown in FIGS. 12D to 12G, when the exposure time of T2 is slightly short, the noise region of S2 cannot be sufficiently supplemented by S3, and noise is generated near the boundary between S2 and S3. Resulting in.

However, in the prior art including the above-mentioned Patent Documents 1 to 4, there is no method for theoretically and dynamically determining the exposure amount ratio for the optimization of multi-stage exposure of three or more stages.
Therefore, the present invention has been made paying attention to such unsolved problems of the conventional technology. According to some aspects of the present invention, it is possible to provide an imaging apparatus, an imaging method, and an electronic apparatus that are suitable for theoretically and dynamically determining a ratio of exposure amounts in multistage exposure.

[Mode 1] In order to achieve the above object, an imaging apparatus according to mode 1
Imaging means for imaging a subject with three or more types of exposure amounts in which the ratio of each two adjacent exposure amounts in ascending or descending order is equal;
Synthesizing means for synthesizing pixel signals corresponding to the exposure amounts of the three or more types of exposure amounts obtained by imaging by the imaging means based on the ratio.
With such a configuration, the subject is imaged by the imaging unit with three or more types of exposure amounts in which the ratio of the two adjacent exposure amounts in ascending order or descending order is equal. When the subject is imaged with three or more types of exposure amounts, pixel signals corresponding to the exposure amounts of the three or more types of exposure amounts are combined based on the ratio by the combining unit.

  Specifically, for example, when a subject is imaged with three types of exposure amounts L1, L2, and L3 (L1 <L2 <L3), the ratio of each two adjacent exposure amounts in ascending order is “L2 / L1 ”and“ L3 / L2 ”. That they are equal is that “L2 / L1 = L3 / L2”. In order to satisfy this relationship, for example, iris (lens diaphragm), imaging sensitivity, exposure time, and the like are adjusted according to the function of the imaging means. For example, when adjusting the exposure time, if the relationship between the exposure amount and the exposure time is linear, the ratio between the two adjacent exposure amounts can be expressed by the ratio of the exposure time. That is, it can be expressed as “T2 / T1” and “T3 / T2”.

If the ratio of these exposure amounts is made uniform, the optical response (output pixel signal level) of the imaging means with respect to the incident light amount (logarithm) can be arranged at equal intervals. Thereby, since the exposure amount ratio between the images to be combined is always constant, in the image after combining, the change in brightness from the dark part to the bright part is uniform, and an image with less noise can be obtained. The effect is obtained.
Here, the above-mentioned equality is not limited to complete equality, but is within an equal range as long as it is within a range in which the effect of the present embodiment can be exhibited. This also applies to the following other forms.

[Mode 2] Furthermore, the imaging apparatus of mode 2 is the imaging apparatus of mode 1,
The three or more types of exposure amounts are L1, L2,..., Ln (n is a natural number of 3 or more and L1 <L2 <... <Ln),
In the image pickup means, the ratios “L2 / L1,..., Ln / L (n−1)” of two adjacent exposure amounts are equal (L2 / L1 =... = Ln / L (n−1). The subject is imaged at exposure times T1, T2,..., Tn (T1 <T2... <Tn).

With such a configuration, the ratio of exposure amounts “L2 / L1,..., Ln is controlled by controlling, for example, iris (lens diaphragm), imaging sensitivity, exposure time, and the like according to the function of the imaging unit. / L (n-1) "can be made equal (L2 / L1 = ... = Ln / L (n-1)).
As a result, the exposure amount ratio between the images to be combined can be made constant at all times, and in the combined image, the change in brightness from the dark part to the bright part becomes uniform, and an image with less noise is obtained. The effect of being able to be obtained.

[Mode 3] Furthermore, the imaging device of mode 3 is the imaging device of mode 1 or 2,
Selecting means for selecting any one of the plurality of types of exposure amount determination methods for determining the three or more types of exposure amounts;
Exposure amount determination means for determining the three or more types of exposure amounts based on the determination method selected by the selection means,
The imaging unit images the subject with three or more types of exposure amounts determined by the exposure amount determination unit.

If it is such a structure, in a selection means, any one determination method will be selected from the determination methods of multiple types of exposure amount. Then, when the determination method is selected, the exposure amount determination means uses the selected determination method to determine three or more types of exposure amounts in which the ratio of each two adjacent exposure amounts in ascending or descending order is equal. Is done. When the three or more types of exposure amounts are determined, the imaging unit images the subject with the determined exposure amounts.
Therefore, the user can select any one of a plurality of types of determination methods such as a determination method that prioritizes image quality and a determination method that prioritizes range. Then, the subject can be imaged with three or more types of exposure amounts determined by the selected determination method.

[Mode 4] Furthermore, the imaging device of mode 4 is the imaging device of mode 3,
The determination method sets a maximum exposure amount and a minimum exposure amount among the three or more types of exposure amounts, and a ratio of each of the two adjacent exposure amounts is based on the set maximum exposure amount and minimum exposure amount. Including a method of calculating an equal intermediate exposure amount and determining the three or more types of exposure amounts.
In such a configuration, for example, the intermediate exposure amount is calculated by setting the maximum value and the minimum value of the exposure amount based on the dynamic range, and the ratio of each two adjacent exposure amounts in ascending order or descending order. It is possible to determine three or more types of exposure amounts at which are equal.
That is, an arbitrary range (desired maximum value and minimum value) can be set within a dynamic range that can be realized by the imaging apparatus. Thereby, in the range determined by the set minimum and maximum values, it is possible to automatically determine at least three types of exposure amounts in which the ratio of each two adjacent exposure amounts in ascending order or descending order is equal.

[Embodiment 5] Furthermore, the imaging device of Embodiment 5 is the imaging device of Embodiment 1 or 2,
Exposure amount setting means for setting a maximum exposure amount and a minimum exposure amount among the three or more types of exposure amounts;
Based on the maximum exposure amount and the minimum exposure amount set by the exposure amount setting means, an intermediate exposure amount in which the ratio between the two adjacent exposure amounts is equal is calculated to determine the three or more types of exposure amounts. Exposure amount determining means.

In such a configuration, when the maximum and minimum exposure values are set by the exposure amount setting means, the ratio of each two adjacent exposure amounts in ascending or descending order is equal in the exposure amount determining means. Three or more types of exposure amounts are determined.
That is, an arbitrary range (desired maximum value and minimum value) can be set within a dynamic range that can be realized by the imaging apparatus. As a result, in the range determined by the set minimum and maximum values, it is possible to automatically determine three or more types of exposure amounts in which the ratio of each two adjacent exposure amounts in ascending order or descending order is equal.

[Mode 6] Furthermore, the imaging device of mode 6 is the imaging device of mode 4 or 5,
When imaging a subject with three types of exposure amounts L1, L2 and L3 (L1 <L2 <L3),
The exposure amount determining means calculates an exposure amount L2 that is the intermediate exposure amount according to the following equation (1).
L2 = L1 × (L3 / L1) ^1/2 (1)
In such a configuration, by setting a desired minimum value and maximum value, an intermediate ratio in which the ratio of each two adjacent exposure amounts in the ascending order or descending order is simply equalized by the above formula (1). The exposure amount can be calculated.

[Mode 7] Furthermore, the imaging device of mode 7 is the imaging device of any one of modes 1 to 6,
Based on the information related to the brightness of the imaging environment, the three or more types of exposure amounts are suitable for the brightness of the imaging environment while maintaining a uniform ratio of the two adjacent exposure amounts. Exposure amount adjusting means for adjusting a control amount related to the exposure amount of the imaging means.
With such a configuration, the three or more types of exposures can be performed by the exposure amount adjusting means while keeping the ratio of the two adjacent exposure amounts equal to each other according to the information related to the brightness of the imaging environment. The amount can be adjusted to an exposure amount suitable for the brightness of the imaging environment.

Thus, the exposure amount can be adjusted to an exposure amount suitable for the brightness of the imaging environment while keeping the above ratio uniform according to a bright environment, a dark environment, or the like. Therefore, in the synthesized image, the change in brightness from the dark part to the bright part is equalized, and the image can be obtained with less noise, and more suitable for the imaging environment. The effect that the exposure amount can be set is obtained.
Here, the control amount related to the exposure amount of the imaging means corresponds to, for example, control amounts such as iris (lens diaphragm), imaging sensitivity, and exposure time.

[Mode 8] On the other hand, in order to achieve the above object, the imaging method of mode 8
An imaging step of imaging the subject with three or more types of exposure amounts in which the ratio of each two adjacent exposure amounts in ascending or descending order is equal;
And a synthesis step of synthesizing pixel signals corresponding to the exposure amounts of the three or more types of exposure amounts obtained by imaging in the imaging step based on the ratio.
With such a configuration, it is possible to obtain operations and effects equivalent to those of the imaging device described in aspect 1.

[Embodiment 9] In order to achieve the above object, an electronic apparatus of embodiment 9
The imaging apparatus according to any one of forms 1 to 7 is provided.
With such a configuration, operations and effects equivalent to those of the imaging device according to any one of Embodiments 1 to 7 can be obtained.

1 is a block diagram illustrating a configuration of an imaging apparatus 1 according to the present invention. 1 is a block diagram illustrating a configuration of an image sensor 10. It is a block diagram which shows the detailed structure of a scanning line scanner. It is a figure which shows an example of the exposure for every line of each pixel in a sensor cell array, and the read-out operation of a pixel signal. It is a figure which shows an example of the reset timing of stored charge, and the read-out timing of the pixel signal of each exposure time. It is a flowchart which shows the control processing of exposure amount. 5 is a flowchart showing HDR synthesis processing in the image synthesis unit 14. (A)-(g) is a figure which shows an example of the response signals S1-S3 with respect to the incident light in the exposure time T1-T3 determined so that a ratio may become equal, a normalization signal, and the synthetic | combination signal HDR. It is a figure at the time of expressing the axis | shaft of the incident light in Fig.8 (a)-(c) with the logarithm. 5 is a flowchart showing exposure ratio information generation processing in a control unit 15; It is a figure which shows an example of the synthesized image signal when linearity collapses. (A)-(g) is a figure which shows an example of the response signals S1-S3 with respect to incident light in the exposure time T1-T3 of an inappropriate exposure ratio, the normalization signal, and the synthetic | combination signal HDR.

[First Embodiment]
Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. 1 to 9 are diagrams illustrating a first embodiment of an imaging apparatus, an imaging method, and an electronic device according to the present invention.
First, the configuration of the imaging apparatus according to the present invention will be described with reference to FIG. FIG. 1A is a block diagram showing the configuration of the imaging device 1 according to the present invention, FIG. 1B is a schematic diagram showing the configuration of the imaging device 10, and FIG. It is a block diagram which shows a structure.

As illustrated in FIG. 1A, the imaging device 1 includes an imaging device 10, memories 11 to 13, an image synthesis unit 14, a control unit 15, an image analysis unit 16, and an operation unit 17. Composed.
The imaging element 10 has a function of imaging a subject with three or more types of exposure amounts. As shown in FIG. 1B, a lens 10a, a microlens 10b, a color filter array 10c, and an HDR sensor 10d It is comprised including.

Hereinafter, for convenience of explanation, the imaging device 10 performs imaging with exposure times T1, T2, and T3 (T1 <T2 <T3) corresponding to three types of exposure amounts L1, L2, and L3 (L1 <L2 <L3). The configuration will be described as follows. Of course, the exposure amount is not limited to three, and may be four or more.
The lens 10a collects reflected light from the subject and guides it to the microlens. There are types such as a single focus lens, a zoom lens, and an auto iris lens depending on the imaging conditions.

The microlens 10b collects light transmitted through the lens on each sensor cell (pixel) of a sensor cell array included in the HDR sensor.
The color filter array 10c separates light having a wavelength corresponding to one predetermined type of color element from light transmitted through the microlens 10b, and makes the separated light incident on each corresponding pixel (hereinafter referred to as CF). Part) and at least the number of pixels.

The HDR sensor 10d controls the exposure times T1 to T3 by an electronic shutter method and outputs three types of digital image data having different exposure amounts.
The memories 11 to 13 temporarily store the image data corresponding to the three types of exposure amounts L1 to L3 obtained by imaging the subject with a plurality of different exposure times T1 to T3 in the image sensor 10, and synchronize with each other. It serves as a buffer to be output to the image composition unit 14.

As shown in FIG. 1B, the image composition unit 14 includes an HDR signal processing unit 14a and a frame memory 14b.
Although not shown, the HDR signal processing unit 14a includes a preprocessing unit, a synthesis processing unit, and a memory interface (hereinafter referred to as a memory IF).
The preprocessing unit performs fixed pattern noise removal processing, clamping processing, and the like on the pixel signals (digital pixel data S1 to S3) from the memories 11 to 13.

The composition processing unit performs HDR composition processing based on the image data from the memories 11 to 13 and the exposure ratio information from the control unit 15. Then, the image data after the HDR synthesizing process is stored as HDR image data in the frame memory via the memory IF.
The memory IF has a function of arbitrating data writing to and data reading from the frame memory 14b. Specifically, arbitration is performed so that reading and writing are normally performed in response to a data read request and a write request from the preprocessing unit and the synthesis processing unit. Then, data reading from the frame memory 14b and data writing to the frame memory 14b are executed in response to requests from the components.

The frame memory 14b is a memory that stores the HDR image data that has been combined in the HDR signal processing unit 14a.
The control unit 15 generates a function for generating exposure ratio information based on input information from the operation unit 17, a function for adjusting the current exposure ratio to an appropriate one based on statistical information acquired from the image analysis unit 16, and And a function of outputting exposure ratio information to the image composition unit 14.

Further, the control unit 15 generates an exposure amount signal based on the exposure ratio information corresponding to the adjusted exposure ratio or the newly generated exposure ratio information, and transmits the generated exposure amount signal and various control signals to the image sensor 10. It also has a function to do. Here, the various control signals include a vertical synchronization signal, a horizontal synchronization signal, a pixel clock, and the like.
Specifically, the image sensor 10 of the present embodiment images the same subject with exposure times T1 to T3 corresponding to three types of exposure amounts L1 to L3 according to the exposure amount signal output from the control unit 15.

In this embodiment, the control unit 15 equalizes the ratios “L2 / L1” and “L3 / L2” of two adjacent exposure amounts in the exposure amounts L1, L2, and L3 arranged in ascending order (L2 / L1 = The exposure times T1 to T3 corresponding to the exposure amounts L1 to L3 are set so as to be L3 / L2).
In the present embodiment, the control unit 15 adjusts the exposure amount ratio so as to be an optimum ratio according to the brightness of the imaging environment. For this purpose, the control unit 15 compares the brightness average value Lav of the HDR image data obtained from the image analysis unit 16, the threshold value Lth on the high brightness side, and the threshold value Ltl on the low brightness side.

  Based on the comparison result, the control unit 15 determines that the imaging environment is relatively bright if the average luminance value Lav is greater than Lth, and determines that the imaging environment is relatively dark if Lav is less than Ltl. Further, based on the determination result, the currently set exposure times T1 to T3 are increased / decreased while maintaining a uniform exposure amount ratio, so that the optimum exposure amount with respect to the brightness of the imaging environment is obtained. Make adjustments to achieve the ratio.

The image analysis unit 16 has a function of calculating statistical information related to the brightness of the imaging environment based on the HDR image data generated by the image synthesis unit 14 and outputting the calculated statistical information to the control unit 15. . Specifically, the image analysis unit 16 statistically analyzes the HDR image data (including at least the first frame) obtained from the image synthesis unit 14 in continuous imaging of a plurality of frames. In the present embodiment, in order to obtain information related to the brightness of the imaging environment, the luminance average value Lav of the HDR image data is calculated as statistical information. Then, the calculated Lav is output to the control unit 15.
The operation unit 17 is operated by the user at the time of imaging, when viewing a captured image, or when selecting an arbitrary mode from among a plurality of types of imaging modes, and information corresponding to the operation content of the user is sent to the control unit 15. It has a function to output.

Next, the configuration of the HDR sensor 10d of the image sensor 10 will be described with reference to FIG. Here, FIG. 2 is a block diagram showing a configuration of the HDR sensor.
As shown in FIG. 2, the HDR sensor 10 d includes a reference timing generator 50, a scanning line scanner 54, a sensor cell array 56, and a horizontal transfer unit 58.
The reference timing generator 50 generates a reference timing signal based on the vertical synchronization signal and the horizontal synchronization signal from the control unit 15 and outputs the reference timing signal to the scanning line scanner 54.

The scanning line scanner 54 generates a reset line selection signal that validates a line on which reset processing is performed based on various signals from the reference timing generator 50 and the control unit 15. Then, the generated reset line selection signal is output to the sensor cell array 56.
Further, the scanning line scanner 54 generates a readout line selection signal that enables the line on which charge accumulation for the set exposure time has been performed as a readout line for pixel signals after reset. Then, the generated read line selection signal is output to the sensor cell array 56.
The sensor cell array 56 includes a light receiving region having a configuration in which a plurality of sensor cells (pixels) including light receiving elements (photodiodes and the like) are arranged in a two-dimensional matrix, which is configured using CMOS technology. On the other hand, the address line, the reset line, and the readout line are connected in common.

Then, various drive signals (selection signals) are transmitted to the sensor cells constituting each line through the three control lines, and when the address line and the readout line become valid, the accumulated charge (pixel signal) is transmitted through the signal line. Transfer (output) to the horizontal transfer unit 58.
With such a configuration, the sensor cell array 56 validates (selects) a pixel line on which a reset operation or a read operation is performed by an address line based on a selection signal supplied from the scanning line scanner 54. When a reset operation is performed on each pixel on the line selected by the selection signal, a signal instructing the reset operation is input via the reset line, and when a pixel signal is read out, a read line is input. A signal for instructing the transfer of the accumulated charge is input via the. Further, in each pixel selected by the selection signal, the reset operation is performed when a signal instructing the reset operation is input, and the signal line is connected when the signal instructing the transfer of the accumulated charge is input. The accumulated charge is transferred to the horizontal transfer unit 58 via the via.
The horizontal transfer unit 58 performs A / D conversion on pixel signal (analog signal) data read from each pixel of the sensor cell array 56, and sequentially outputs the data to the memories 11 to 13 in units of lines.

Next, the internal configuration of the scanning line scanner 54 will be described with reference to FIG.
FIG. 3 is a block diagram showing the internal configuration of the scanning line scanner 54.
As shown in FIG. 3, the scan line scanner 54 includes a reset scan counter 54a, a reset scan address decoder 54b, a read scan counter 54c, and a read scan address decoder 54d.

The reset scanning counter 54 a repeats the count-up operation based on the vertical synchronization signal and horizontal synchronization signal from the reference timing generator 50 and the exposure amount (exposure time) information included in the exposure amount signal from the control unit 15. . Here, the count value of the reset scanning counter 54 a corresponds to the line number of each pixel of the sensor cell array 56. Further, information on the exposure times T1 to T3 included in the exposure amount signal is written in the internal register of the scanning line scanner 54.
When the count operation is executed, the reset scanning counter 54a counts up one by one from the initial value of the counter, and outputs each count value to the reset scanning address decoder 54b.

  The count value loops over the range of the minimum line number and the maximum line number (for example, the numbers of the bottom and top lines of the light receiving area of the sensor cell array). For example, the count value is incremented by 1 from the minimum line number, and when the count is incremented by 1 after reaching the maximum line number, the count value is reset to return to the minimum line number (for example, “1”). . The same applies to each counter of the readout scanning counter 54c.

The reset scanning address decoder 54 b generates a reset line selection signal for selecting and validating the line having the line number output from the reset scanning counter 54 a as the “reset line R”, and outputs this to the sensor cell array 56. . As a result, only the selected line is valid and the other lines are invalid.
The readout scanning counter 54c has three counters corresponding to the exposure times T1 to T3, and each counter has a vertical synchronization signal, a horizontal synchronization signal from the reference timing generator 50, and an exposure written in the register. Based on the information of the times T1 to T3, the count-up operation similar to that of the reset scanning counter 54a is repeated at a timing corresponding to each exposure time.

Specifically, each counter of the readout scanning counter 54c starts counting up with a count width corresponding to each exposure time with respect to the reset timing, and counts up one by one in order from the initial value of the counter. The count value is output to the read scanning address decoder 54d.
The read scan address decoder 54d generates a read line selection signal for selecting and validating the lines having the line numbers output from the respective counters of the read scan counter 54c as “read lines L1 to Ln”. Output to the cell array 56. As a result, only the selected line is valid and the other lines are invalid.

  Next, a method for controlling the exposure time of the HDR sensor of the image sensor 10 and a method for reading a pixel signal from the sensor cell array will be described with reference to FIGS. Here, FIG. 4 is a diagram illustrating an example of exposure and pixel signal read operations for each pixel line in the sensor cell array of the HDR sensor. FIG. 5 is a diagram showing an example of the reset timing of the accumulated charge and the readout timing of the pixel signal for each exposure time.

  Here, in the present embodiment, the non-destructive readout line L1 for performing non-destructive readout of the pixel signal at the exposure time T1 and the non-deletion of the pixel signal at the exposure time T2 with respect to the exposure region (scanning region) of the sensor cell array 56. A non-destructive readout line L2 for performing destructive readout is set. Further, a read & reset line L3 & R for resetting the accumulated charge of each pixel line and reading out the pixel signal at the exposure time T3 is set. The relationship between T1 and T3 is “T1 <T2 <T3” as shown in FIG. After the reset, first, the nondestructive read line L1 is set when T1 has elapsed. Next, the nondestructive read line L2 is set when T2 elapses, and the read & reset line L3 & R is set when T3 elapses.

  Specifically, the non-destructive readout lines L1 and L2 and the readout & reset line L3 & R are, as shown in FIG. 4, when the charge for the exposure time T3 is sequentially accumulated in the pixel line in the exposure region, the readout & reset line. L3 & R is set so as to sequentially read out the pixel signals of the lines of the respective pixels and to sequentially reset the accumulated charges. On the other hand, in the line of each pixel after the resetting of the exposure area, the pixel signals of the line of each pixel are sequentially read in a non-destructive manner during the exposure time T1 and the exposure time T2 during the period in which charges for the exposure time T3 are accumulated. Non-destructive read lines L1 and L2 are set respectively.

  For example, it is assumed that the pixel signal S3 of the exposure time T3 is read and reset on the first line that is the first line of the exposure region. Thereafter, every time the pixel signal S3 is read from the third line memory, scanning of the read & reset lines L3 & R is sequentially performed one line at a time in the scan direction in FIG. At this time, when the read & reset line L3 & R reaches the first line again, scanning is performed so that the exposure time T3 has just passed. In such a procedure, the pixel signal at the time of exposure and the reset of accumulated charge are sequentially performed for each pixel line for each pixel line in the exposure region of the sensor cell array at the exposure time T3.

  On the other hand, when the accumulated charge is reset, non-destructive readout of the pixel signal S1 of the pixel that has been exposed for the exposure time T1 in the non-destructive readout line L1 is performed on the pixel line after the reset, In the nondestructive readout line L2, nondestructive readout of the pixel signal S2 of the pixel that has been exposed for the exposure time T2 is performed. In such a procedure, the pixel signals S1 and S2 at the time of exposure are sequentially read for each line of each pixel of the sensor cell array at the exposure time T1 and the exposure time T2.

  In the present embodiment, as shown in FIG. 4, the pixel signal data (analog data) S1 corresponding to the exposure time T1 is read out to the first line memory and the pixel signal corresponding to the exposure time T2. The data (analog data) S2 is read to the second line memory. Further, pixel signal data (analog data) S3 corresponding to the exposure time T3 is read to the third line memory. Then, the read pixel signal data S1 to S3 (hereinafter referred to as pixel data S1 to S3) are sequentially output to the ADC in the order of S1 to S3 through the selection circuits as shown in FIG. There, it is converted into digital data. The converted pixel data S1 to S3 are output to the memory 11, S2 to the memory 12, and S3 to the memory 13, respectively, in the converted order (S1 to S3).

Next, the flow of exposure amount control processing in the control unit 15 will be described with reference to FIG. Here, FIG. 6 is a flowchart showing exposure amount control processing.
When the control process of the exposure amount is started in the control unit 15, first, the process proceeds to step S100 as shown in FIG.
In step S100, the control unit 15 sets initial values of the shortest exposure time Tmin and the longest exposure time Tmax, and the process proceeds to step S102. For the initial value setting, a value input or selected by the user via the operation unit 17 may be used, or an initial value preset as the standard exposure time may be used.

  In step S102, the control unit 15 calculates an exposure time Tmd, which is an exposure time corresponding to the intermediate exposure amount, using the initial value set in step S100. The calculated Tmd and the Tmin set as the initial value are calculated. And exposure ratio information are generated based on Tmax. Then, an exposure amount signal is generated based on the generated exposure ratio information, the generated exposure ratio information is output to the image composition unit 14, and the generated exposure amount signal and various control signals are transmitted to the image sensor 10, and step The process proceeds to S104.

In this embodiment, since there are three types of exposure amounts L1 to L3, the shortest exposure time Tmin corresponding to the exposure amount L1 is T1, and the longest exposure time Tmax corresponding to the exposure amount L3 is T3. The intermediate exposure time Tmid corresponding to the exposure amount L2 is T2. Then, the control unit 15 calculates an intermediate exposure time T2 that satisfies “T2 / T1 = T3 / T2” according to the following equation (2).

T2 = T1 × (T3 / T1) ^1/2 (2)

For example, when the shortest exposure time Tmin that can be set is 1H and the longest exposure time Tmax is 1000H, T1 = 1H and T3 = 1000H, and from the above equation (2), “T2 = 1 × (1000/1) ^1/2 ≈32 ”. Thereby, the exposure amount ratio = exposure time ratio = “1: 32: 1000” is obtained. In this case, the dynamic range that can be expanded by HDR synthesis is “60 [dB]”.

  Note that the scanning time for one line in the sensor cell array is abbreviated as HSYNC, which is the name of the horizontal synchronization signal, and is represented as 1H. For example, when the frame rate is 30 [fps] with respect to the total number of lines of the sensor cell array of 500 lines, the 1H period is the time obtained by dividing 1/30 seconds by 500 lines, that is, 1/15000 seconds, in other words. 1/15 milliseconds (about 67 microseconds).

In step S <b> 104, in the image sensor 10, a process of imaging a subject with exposure amounts L <b> 1 to L <b> 3 (exposure times T <b> 1 to T <b> 3) is executed based on the exposure amount signal and various control signals from the control unit 15. Then, the image data S1 to S3 obtained by imaging are sequentially output to the memories 11 to 13, and the process proceeds to step S106.
In step S106, the image composition unit 14 executes HDR composition processing based on the image data S1 to S3 sequentially output from the memories 11 to 13 and the exposure ratio information (T1: T2: T3) from the control unit 15. Then, the process proceeds to step S108.

In step S108, the control unit 15 determines whether or not the imaging process has been completed in the imaging device 10. If it is determined that the imaging process has ended (Yes), the series of processes is terminated. If not (No), The process proceeds to step S110.
When the process proceeds to step S110, the control unit 15 acquires the HDR image data generated by the HDR combining process from the frame memory 14b of the image combining unit 14, and calculates the average luminance value Lav from the acquired HDR image data. The process proceeds to step S112.

In step S112, the control unit 15 compares the average luminance value Lav calculated in step S110 with the threshold value Lth on the high luminance side of the preset average luminance value, and determines whether or not Lav is greater than Lth. To do. If it is determined that Lav is greater than Lth (Yes), the process proceeds to step S114. If not (No), the process proceeds to step S118.
When the process proceeds to step S114, the controller 15 decreases the shortest exposure time Tmin by a preset time, and the process proceeds to step S116. That is, since the imaging environment is relatively bright, adjustment is performed to reduce the shortest exposure time Tmin in order to obtain an image that does not blow out. It should be noted that an appropriate value may be calculated for each decrease time according to the average luminance, or a value corresponding to the magnitude of the average luminance may be prepared in advance as a table.

  In step S116, the control unit 15 recalculates the intermediate exposure time Tmid using Tmin decreased in step S114 and Tmax at the initial setting, and recalculated Tmd, the decreased Tmin, and the initial setting. The exposure ratio information is generated based on Tmax. That is, “T1: T2: T3 = Tmin (decrease): Tmid (re): Tmax (initial)” is generated as the exposure ratio information. It should be noted that (decrease) attached after each exposure time indicates that the exposure time after the reduction adjustment, (re) indicates after recalculation, and (initial) indicates each exposure time at the time of initial setting. Further, the control unit 15 generates an exposure amount signal based on the generated exposure ratio information. Then, the generated exposure ratio information is output to the image composition unit 14, and the generated exposure amount signal and various control signals are transmitted to the image sensor 10, and the process proceeds to step S104.

  On the other hand, in step S112, when Lav is equal to or less than Lth and the process proceeds to step S118, the control unit 15 calculates the average luminance value Lav calculated in step S110 and the threshold value on the low luminance side of the preset average luminance value. Compare with Ltl. Furthermore, based on the comparison result, it is determined whether or not Lav is smaller than Ltl. If it is determined that Lav is smaller than Ltl (Yes), the process proceeds to step S120, and if not (No), the process proceeds to step S124.

  When the process proceeds to step S120, the control unit 15 increases the longest exposure time Tmax by a preset time, and the process proceeds to step S122. That is, since the imaging environment is in a relatively dark state, adjustment is performed to increase the longest exposure time Tmax in order to obtain an image that is not blackened. Note that an appropriate value for the increasing time may be calculated each time according to the average luminance, or a value corresponding to the magnitude of the average luminance may be prepared in advance as a table.

  In step S122, the control unit 15 recalculates the intermediate exposure time Tmid using Tmax increased in step S120 and Tmin at the time of initial setting, and recalculates Tmid, Tmin at the time of initial setting, and after the increase. Exposure ratio information is generated based on Tmax. Here, the exposure ratio information is “T1: T2: T3 = Tmin (initial): Tmid (re): Tmax (increase)”. Note that (increased) added after the exposure time Tmax indicates the exposure time after the increase adjustment. Further, the control unit 15 generates an exposure amount signal based on the generated exposure ratio information. Then, the generated exposure ratio information is output to the image composition unit 14, the generated exposure amount signal and various control signals are transmitted to the image sensor 10, and the process proceeds to step S104.

In step S118, if Lav is equal to or greater than Ltl and the process proceeds to step S124, the control unit 15 sets the normal exposure time as the shortest exposure time Tmin and the longest exposure time Tmax, and the process proceeds to step S126. In the present embodiment, the initial exposure time is set as the normal exposure time.
In step S126, the control unit 15 recalculates the intermediate exposure time Tmid using the Tmin and Tmax set in step S124. Based on the recalculated Tmid, the initial setting Tmin, and the increased Tmax. Exposure ratio information is generated. Here, the exposure ratio information is “T1: T2: T3 = Tmin (initial): Tmid (re): Tmax (initial)”. Further, the control unit 15 generates an exposure amount signal based on the generated exposure ratio information. Then, the generated exposure ratio information is output to the image composition unit 14, and the generated exposure amount signal and various control signals are transmitted to the image sensor 10, and the process proceeds to step S104.

Next, the flow of the HDR synthesis process in step S106 will be described based on FIG. Here, FIG. 7 is a flowchart showing the HDR synthesizing process in the image synthesizing unit 14.
When the HDR synthesizing process is started in step S106, first, the process proceeds to step S200 as shown in FIG.
In step S200, the HDR signal processing unit 14a determines whether the imaging process has been started. If it is determined that it has been started (Yes), various control signals are transmitted to the image sensor 10, and the process proceeds to Step S202. If not (No), the determination process is repeated until the imaging process is started. .

When the process proceeds to step S202, the HDR signal processing unit 14a obtains one line of pixel data S1 to S3 from the memories 11 to 13, and the process proceeds to step S204.
Specifically, the HDR signal processing unit 14a acquires S1 to S3 at the timing when the pixel data S1 to S3 for one line at the same pixel position are stored in the memories 11 to 13.
In the present embodiment, the pixel data S1 to S3 are data including luminance information (luminance value) and pixel position information (row and column position information). Further, the pixel data S1 to S3 may not have position information, and the HDR signal processing unit 14a may generate pixel position information by counting the horizontal synchronization signal and the pixel clock.

Also, the HDR signal processing unit 14a performs fixed pattern noise removal processing and clamping processing on the acquired pixel data S1 to S3 in the preprocessing unit.
In step S204, the HDR signal processing unit 14a compares the luminance value of the pixel data S3 with a preset luminance value for saturation determination. From this comparison result, it is determined whether or not S3 is saturated. If it is determined that it is saturated (Yes), the process proceeds to step S206. If not (No), the process proceeds to step S212. To do.

When the process proceeds to step S206, the HDR signal processing unit 14a compares the luminance value of the pixel data S2 with the luminance value for saturation determination. From this comparison result, it is determined whether or not S2 is saturated. If it is determined that it is saturated (Yes), the process proceeds to step S208. If not (No), the process proceeds to step S210. To do.
When the process proceeds to step S208, the HDR signal processing unit 14a multiplies the pixel data S1 (luminance value) by the coefficient “T3 / T1” to calculate the pixel value HDR of the HDR pixel data, and the process proceeds to step S214. To do.

On the other hand, when the process proceeds to step S210, the HDR signal processing unit 14a multiplies the pixel data S2 (luminance value) by the coefficient “T3 / T2” to calculate the pixel value HDR of the HDR pixel data, and then performs step S214. Migrate to
In step S204, when S3 is not saturated and the process proceeds to step S212, the HDR signal processing unit 14a selects the pixel data S3 as the pixel value HDR of the HDR pixel data, and the process proceeds to step S214. To do.

In step S214, the HDR signal processing unit 14a stores the HDR pixel data in the frame memory 14b via the memory IF unit, and the process proceeds to step S216.
In step S216, in the HDR signal processing unit 14a, it is determined whether or not the HDR synthesis processing has been completed for pixel data for one line. If it is determined that the processing has ended (Yes), the process proceeds to step S218. When that is not right (No), it transfers to step S222.

When the process proceeds to step S218, the HDR signal processing unit 14a determines whether or not the HDR synthesizing process has been completed for one frame of pixel data. If it is determined that the process has been completed (Yes), the process proceeds to step S220. If not (No), the process proceeds to step S202.
When the process proceeds to step S220, the HDR signal processing unit 14a determines whether or not the imaging is completed. When it is determined that the imaging is completed (Yes), the series of processes is terminated, and when it is not (No). Shifts to step S202.
If the HDR composition processing for one line is not completed in step S216 and the process proceeds to step S222, the next pixel data S1 to S3 that have not been processed by the HDR composition processing are selected, and step S204 is performed. Migrate to

Next, the operation of the present embodiment will be described with reference to FIGS.
Here, FIGS. 8A to 8C are diagrams showing response signals S1 to S3 with respect to incident light at exposure times T1 to T3 determined so that the ratios are equal, and FIGS. () Is a diagram when the response signals of (a) to (c) are multiplied by a coefficient. FIG. 8G is a diagram in which the response signals of FIGS. 8D to 8F are synthesized. FIG. 9 is a diagram showing the incident light axis in FIGS. 8A to 8C in logarithm.

When the power is turned on, the imaging apparatus 1 first generates exposure ratio information and an exposure amount signal in the control unit 15 before the imaging process is started, and sends the generated exposure ratio information to the image synthesis unit 14. The generated exposure amount signal and various control signals are transmitted to the image sensor 10.
Specifically, first, initial values of the exposure times T1 and T3 are set by an instruction from the user via the operation unit 17 (step S100). Here, it is assumed that “T1 = 2H, T3 = 500H” is set as the initial value.

Next, the control unit 15 calculates an intermediate exposure time T2 having a relationship of “T2 / T1 = T3 / T2” according to the above equation (2) (step S102). Specifically, since “T1 = 2H, T3 = 500H” is set, the control unit 15 sets “T2 = 2 × (500/2) ^1/2 ≈2 × 15.8 = 31.6”. Calculated automatically. Here, rounding is performed and T2 is set to “32”. As a result, “T2 / T1 = 32/2 = 16” and “T3 / T2 = 500 / 32≈15.6” are obtained, and the exposure ratio information ““ T1: T2: T3 = 2: 32 ”at which the ratio becomes substantially equal. : "Can be obtained.

Further, the control unit 15 generates an exposure amount signal including information of the exposure ratio information “T1: T2: T3 = 2: 32: 500”, and outputs the generated exposure ratio information to the image composition unit 14 to generate the exposure amount information. An exposure amount signal and various control signals are transmitted to the image sensor 10.
Thereby, information of the exposure time ratio “T1: T2: T3 = 2: 32: 500” is written in the register of the HDR sensor 14d of the image sensor 10.
Subsequently, when imaging of the subject is started in the image sensor 10, the light reflected from the subject is collected by the lens 10a and is incident on the microlens 10b. Incident light from the lens 10a is collimated by the microlens 10b and is incident on each pixel of the sensor cell array via the color filter array 10c.

  On the other hand, when imaging is started (step S104), in the HDR sensor 10d, the reset line R is set for each line in order from the start line, and the accumulated charge of each pixel is reset. Subsequently, for each scanning line, after resetting each pixel, the non-destructive readout line L1 is set at the elapse timing of the exposure time T1, and the pixel signal S1 having a light response as shown in FIG. 8A is read out. Subsequently, for each scanning line, after resetting each pixel, a non-destructive readout line L2 is set at the elapse timing of the exposure time T2, and a pixel signal S2 having a light response as shown in FIG. 8B is read out. Subsequently, a read & reset line L3 & R is set in order from the start line, and a photoresponse pixel signal S3 as shown in FIG. 8C is read from each pixel of the scanned line, and thereafter, accumulation of each pixel is performed. The charge is reset.

As described above, the pixel signals S1 to S3 obtained by imaging with the exposure times T1, T2, and T3 at which the ratios are substantially equal are substantially equal in a graph representing incident light in a logarithm as shown in FIG. Line up at regular intervals.
Thereafter, while the imaging is being performed, the setting of L1 to L3 & R, the reading of the pixel signals S1 to S3, and the reset process are repeatedly performed according to the above procedure.
The pixel signals S1 to S3 read out in this way are stored in the first line memory to the third line memory for each line and output to the selection circuit in line units. From the selection circuit, analog pixel data S1 to S3 are output to the ADC in the order of S1 to S3. The ADC converts the analog pixel data S1 to S3 into digital pixel data S1 to S3. Then, the ADC sequentially outputs pixel data to the memories 11 to 13 in units of lines and in the order of the pixel data S1 to S3.

On the other hand, when the pixel data S1, S2, and S3 for one line are stored in the memories 11, 12, and 13, respectively, the HDR signal processing unit 14a starts the HDR synthesis process (step S106).
When the HDR synthesizing process is started (“Yes” branch of step S200), the HDR signal processing unit 14a acquires pixel data S1 to S3 for one line from the memories 11 to 13 (step S202). Then, in the preprocessing unit, fixed pattern noise removal processing and clamping processing are performed on the acquired pixel data S1 to S3.

Next, the HDR signal processing unit 14a selects the pixel data S1 to S3 at the same pixel position that has not been subjected to the determination process. First, the selected S3 and the luminance value Ssat for saturation determination (for example, Ssat = 250). And compare. Here, it is assumed that the luminance values of the selected pixel data S1 to S3 are “S1 = 62, S2 = 224, S3 = 255”. In this case, since “S3 (255)> Ssat (250)”, it is determined that S3 is saturated (“Yes” branch of step S204). Since S3 is saturated, the HDR signal processing unit 14a next compares S2 and Ssat. Since “S2 (224) <Ssat (250)”, it is determined that S2 is not saturated (“No” branch of step S206). Since S2 is not saturated, the HDR signal processing unit 14a next calculates a value (HDR) of HDR pixel data at the selected pixel position using S2. Specifically, as shown in FIG. 8E, a coefficient “T3 / T2” is obtained from the exposure times T3 and T2 at the time of imaging, and HDR is calculated by multiplying S2 by this coefficient (step S210). . That is, “HDR = 224 (S2) × 500H (T3) / 32H (T2) = 3500”.
The HDR signal processing unit 14a stores the calculated HDR pixel data value (HDR) in the memory area of the address corresponding to each pixel position of the frame memory 14b via the memory IF (step S214).

When the selected pixel data S1 to S3 are “S1 = 62, S2 = 255, S3 = 255”, “S3 (255)> Ssat (250)”, and S3 is saturated. ("Yes" branch in step S204). Furthermore, since “S2 (255)> Ssat (250)”, it is determined that S2 is also saturated (“Yes” branch of step S206). Since both S3 and S2 are saturated, the HDR signal processing unit 14a next calculates the value (HDR) of the HDR pixel data at the selected pixel position using S1. Specifically, as shown in FIG. 8D, a coefficient “T3 / T1” is obtained from the exposure times T3 and T1 at the time of imaging, and HDR is calculated by multiplying S1 by this coefficient (step S208). . That is, “HDR = 62 (S1) × 500H (T3) / 2H (T1) = 15500”.
The HDR signal processing unit 14a stores the calculated HDR pixel data HDR in the memory area of the address corresponding to each pixel position of the frame memory 14b via the memory IF (step S214).

  If the selected pixel data S1 to S3 is “S1 = 62, S2 = 124, S3 = 221”, “S3 (221) <Ssat (250)”, so S3 is not saturated. ("No" branch in step S204). In this case, as shown in FIG. 8 (f), the HDR signal processing unit 14a sets the value of S3 as HDR, and the HDR is set to an address corresponding to each pixel position of the frame memory 14b via the memory IF. Store in the memory area (step S214).

  By calculating the HDR as described above, it is possible to calculate the HDR pixel data of the optical response shown in FIG. In addition, the wavy line part shown to FIG.8 (d)-(g) is a noise part which becomes a cause of generation | occurrence | production of a pseudo contour or a false color. As described above, the image processing is performed with the ratio of each two adjacent exposure times being equalized, and the pixel signals S1 to S3 obtained thereby are combined by the above combining method, whereby the noise portion signal is combined. The amount used for processing can be reduced.

  Next, the HDR signal processing unit 14a selects unprocessed next pixel data S1 to S3 ("No" branch in step S216, step S222), and performs the same processing as described above. When the processing is completed for the pixel data S1 to S3 for one line ("Yes" branch in step S216), the pixel data S1 to S3 for the next line are acquired and the same processing as described above is performed. Further, when the processing for one frame is completed, it is next determined whether or not the imaging process is completed. If the imaging process is continued (“No” branch in step S220), the next frame is processed. The series of processes (steps S202 to S222) are performed on the pixel data, and when the imaging process ends (“Yes” branch of step S220), the series of processes ends.

  On the other hand, when HDR image data for one line or one frame is generated by the imaging process and the imaging process is continued ("No" branch in step S108), the control unit 15 sets the data for one line. The HDR pixel data or the HDR pixel data for one frame is acquired from the frame memory 14b. Then, based on the acquired pixel data, the average luminance Lav for one line or one frame is calculated as statistical information related to the brightness of the imaging environment (step S110). When the average luminance Lav is calculated, the control unit 15 compares Lav with a threshold Lth on the high luminance side, and determines whether or not Lav is greater than Lth. Here, the threshold Lth on the high luminance side is “192”, and the threshold Ltl on the low luminance side is “64”.

For example, when the average luminance Lav is “61”, since Lav (61) is smaller than Lth (192) (the “No” branch in Step S112), the control unit 15 next calculates Lav and Ltl. A comparison is made to determine whether Lav is smaller than Ltl. Here, since Lav (61) is smaller than Ltl (64) (branch “Yes” in step S118), the control unit 15 determines that the imaging environment is dark, and currently sets the longest exposure time “Tmax”. Adjustment to increase (T3) = 500H ”is performed (step S120). For example, it is assumed that the number increases to “T3 = 1000H”. As a result, the substantially uniform ratio is lost. Therefore, the control unit 15 substitutes “T1 = 2H (initial), T3 = 1000H (increase)” into the above equation (2) to perform intermediate exposure. Recalculate time Tmid (T2). Accordingly, “T2 = 2 × (1000/2) ^1/2 ≈45” is calculated. Then, “T1: T2: T3 = 2: 45: 1000” is generated as new exposure ratio information, and a new exposure information signal is generated based on this exposure ratio information. The generated exposure ratio information is output to the image composition unit 14, and the generated exposure information signal is transmitted to the image sensor 10 together with each control signal. Thereby, in the image sensor 10, an imaging process is performed based on new exposure ratio information suitable for the brightness of the imaging environment.

When the calculated average luminance Lav is “221”, since Lav (221) is larger than Lth (192) (“Yes” branch in step S112), the control unit 15 has a bright imaging environment. And the adjustment is performed to reduce the currently set shortest exposure time “Tmin (T1) = 2H” (step S114). For example, it is assumed that the value decreases to “T1 = 1H”. As a result, the substantially uniform ratio is lost. Therefore, the control unit 15 substitutes “T1 = 1H (decrease), T3 = 500H (initial)” into the above equation (2) to perform intermediate exposure. Recalculate time Tmid (T2). Thus, “T2 = 1 × (500/1) ^1/2 ≈22” is calculated. Then, “T1: T2: T3 = 1: 22: 500” is generated as new exposure ratio information, and a new exposure information signal is generated based on the exposure ratio information. The generated exposure ratio information is output to the image composition unit 14, and the generated exposure information signal is transmitted to the image sensor 10 together with each control signal. Thereby, in the image sensor 10, an imaging process is performed based on new exposure ratio information suitable for the brightness of the imaging environment.

As described above, in the image pickup apparatus 1 of the present embodiment, in the image pickup element 10, the ratios “L2 / L1” and “L3 / L2” of two adjacent exposure amounts are equal “L2 / L1 = L3 / L2”. The subject can be imaged with exposure times T1 to T3 corresponding to exposure amounts L1 to L3.
As a result, the ratio of the exposure amounts of the pixel data S1 to S3 used in the HDR synthesizing process is always constant, so that it is possible to obtain a synthesized image with uniform luminance and low noise from the dark part to the bright part.

Further, as the statistical information relating to the brightness of the imaging environment, the average luminance Lav is calculated from the HDR pixel data for one line or one frame, and the Lav is compared with the high luminance side threshold value Lth and the low luminance side threshold value Ltl. Thus, the brightness state of the imaging environment can be determined.
Further, when it is determined that the imaging environment is bright, adjustment is performed to decrease the shortest exposure time Tmin (T1), and when it is determined that the imaging environment is dark, adjustment is performed to increase the longest exposure time Tmax (T3). It can be carried out.

Furthermore, by using T1 and T3 after adjustment and T1 and T3 at the time of initial setting to recalculate T2 according to the above equation (2), the ratio collapsed by adjusting the exposure time can be returned uniformly.
Thus, the exposure amount ratio (exposure time ratio) can be automatically adjusted to an exposure amount ratio suitable for the brightness of the imaging environment.
In the first embodiment, the imaging device 10 and the control unit 15 correspond to the imaging unit described in the form 1 or 2, and the memories 11 to 13 and the image composition unit 14 correspond to the composition unit described in the form 1. .
In the first embodiment, steps S100 to S104 and S110 to S126 correspond to the imaging step described in Form 8, and Step S106 corresponds to the combining step described in Form 8.

[Second Embodiment]
Next, 2nd Embodiment of this invention is described based on drawing. FIG. 10 is a diagram illustrating a second embodiment of an imaging apparatus, an imaging method, and an electronic device according to the present invention.
In the present embodiment, the control unit 15 in the image sensor 10 of the first embodiment is selected from a plurality of types of exposure ratio information determination methods (modes) in response to a selection instruction from the user via the operation unit 17. The difference is that either one is selected and the exposure ratio is determined (exposure ratio information is generated) in the selected determination mode. Furthermore, the ratio of the exposure time after calculation is adjusted so that the shortest exposure time T1, the intermediate exposure time T2, and the longest exposure time T3 are all natural numbers.

Therefore, only the processing contents of the control unit 15 are partially different, and the other components are the same as those in the first embodiment.
Hereinafter, parts different from those in the first embodiment will be described in detail, and description of similar parts will be omitted as appropriate.
The control unit 15 of the present embodiment has a plurality of types of exposure ratio information determination modes. Then, it has a function of selecting any one determination mode based on the input information from the operation unit 17 and determining (generating) the exposure ratio information in the selected determination mode. In the present embodiment, either the range priority mode or the noise reduction mode can be selected as the determination mode.

  Specifically, the range priority mode is a mode for determining an exposure ratio that makes the best use of a dynamic range that can be expanded by HDR synthesis. When this mode is selected, T1 and T3 that maximize the range are automatically set. For example, when the exposure time can be set in the range of 1H to 1000H and the dynamic range is “60 dB”, T1 = 1H and T3 = 1000H are automatically set. When T1 and T3 are set, the exposure ratio “T1: T2: T3” is determined by calculating T2 using Expression (2) of the first embodiment.

  The noise reduction mode is a mode for determining the exposure ratio so that the ratio “T2 / T1 = T3 / T2” falls within the input upper limit range. That is, noise generated in the boundary region is suppressed at the expense of the width of the dynamic range. In this embodiment, when this mode is selected, the user is prompted to input the upper limit value of the ratio, the shortest exposure time T1, and the longest exposure time T3, and the upper limit values T1 and T3 input via the operation unit 17. Based on the above, T2 is calculated using the above equation (2) to determine the exposure ratio “T1: T2: T3”.

Further, the control unit 15 according to the present embodiment has a function of adjusting the exposure ratio to be a natural number based on the calculation result when the value calculated by the above equation (2) is a decimal number, for example. have.
For example, assume that the noise reduction mode is selected and ratio upper limit value = 16, T1 = 1, and T3 = 250 are input. In this case, when the above equation (2) is applied, T2 = 1 × (250/1) ^1/2 ≈15.8, and T2 becomes a decimal. In such a case, for example, the values of T1 to T3 are adjusted such that T2 / T1 = T3 / T2 = 16 (ratio upper limit value). For example, T1 = 1, T2 = 16, and T3 = 256 are adjusted. As a result, the exposure ratios are all natural numbers, and in this case, numbers that can be expressed by powers of 2.

Next, the flow of the exposure ratio information generation process in the control unit 15 will be described with reference to FIG. Here, FIG. 10 is a flowchart showing exposure ratio information generation processing in the control unit 15.
When the exposure ratio information generation process is started in the control unit 15, first, the process proceeds to step S300 as shown in FIG.
In step S <b> 300, the control unit 15 determines whether or not the range priority mode is selected in response to a user selection instruction input via the operation unit 17. If it is determined that it has been selected (Yes), the process proceeds to step S302. If not (No), the process proceeds to step S308.

When the process proceeds to step S302, the control unit 15 sets the shortest exposure time Tmin and the longest exposure time Tmax according to the maximum range, and the process proceeds to step S304.
In step S304, the control unit 15 calculates the intermediate exposure time Tmid based on Tmin and Tmax set in step S302 and the above equation (2), and the process proceeds to step S306.
In step S306, the control unit 15 determines whether or not the calculation result in step S304 has become a decimal. If it is determined that the calculation has become a decimal (Yes), the process proceeds to step S308; otherwise (No). The process proceeds to step S310.
When the process proceeds to step S308, the control unit 15 adjusts the exposure ratio so that all exposure ratios are natural numbers based on the calculation result of step S304, and the process proceeds to step S310.

  When the process proceeds to step S310, the control unit 15 determines exposure ratio information based on the Tmin and Tmax set in step S302 and the Tmid calculated in step S304, or the adjusted Tmin, Tmid and Tmax in step S308. Is generated. Further, an exposure amount signal is generated based on the generated exposure ratio information. Then, the generated exposure ratio information is output to the image composition unit 14, and the generated exposure amount signal and various control signals are transmitted to the image sensor 10, and the process proceeds to step S300.

  On the other hand, when the range priority mode is not selected in step S300 and the process proceeds to step S312, the control unit 15 determines whether or not the noise reduction mode is selected. If it is determined that it is selected (Yes), a message prompting the input of the ratio upper limit value is displayed on the liquid crystal screen (not shown), and the process proceeds to Step S314. If not (No), the process proceeds to Step S300. Transition.

When the process proceeds to step S314, the control unit 15 determines whether the ratio upper limit value is input via the operation unit 17. If it is determined that it has been input (Yes), a message prompting the input of Tmin and Tmax is displayed on a liquid crystal screen (not shown), and the process proceeds to step S316. If not (No), it is input. Repeat the determination process until
When the process proceeds to step S316, the control unit 15 sets Tmin and Tmax based on the ratio upper limit value input in step S314 and the dynamic range that can be expanded by HDR synthesis, and the process proceeds to step S318.

In step S318, the control unit 15 calculates the intermediate exposure time Tmid based on Tmin and Tmax set in step S316 and the above equation (2), and the process proceeds to step S320.
In step S320, the control unit 15 determines whether or not the calculation result of step S318 has become a decimal. If it is determined that the calculation has become a decimal (Yes), the process proceeds to step S322; otherwise (No). The process proceeds to step S324.
When the process proceeds to step S322, the control unit 15 adjusts the exposure ratio so that all exposure ratios are natural numbers based on the calculation result of step S314, and the process proceeds to step S324.

  When the process proceeds to step S324, the exposure ratio information is based on the Tmin and Tmax input in step S316 and the Tmid calculated in step S314 in the control unit 15 or the Tmin, Tmid and Tmax adjusted in step S322. Is generated. Further, an exposure amount signal is generated based on the generated exposure ratio information. Then, the generated exposure ratio information is output to the image composition unit 14, and the generated exposure amount signal and various control signals are transmitted to the image sensor 10, and the process proceeds to step S300.

Next, the operation of this embodiment will be described.
When the power is turned on and the operation mode is changed to the initial setting mode according to the operation of the operation unit 17, the imaging apparatus 1 displays a determination mode selection screen on a liquid crystal screen (not shown). The user operates the operation unit 17 according to the screen guidance and selects the determination mode.
Thereby, when the range priority mode is selected (“Yes” branch of step S300), the control unit 15 acquires information related to the exposure amount setting including the dynamic range information from the image sensor 10. . Based on the acquired information, the shortest exposure time Tmin and the longest exposure time Tmax are set (step S302). Here, it is assumed that the dynamic range that can be expanded by HDR synthesis is 60 [dB], and the exposure time can be set such that the shortest is 1H and the longest is 1000H.

Since the range priority mode is selected, the controller 15 gives priority to the width of the dynamic range and sets Tmin and Tmax that are the maximum dynamic range. Here, it is assumed that Tmin (T1) = 1H and Tmax (T3) = 1000H are set.
When T1 and T3 are set, the control unit 15 next calculates an intermediate exposure time T2 having a relationship of “T2 / T1 = T3 / T2” according to the above equation (2) (step S304). Specifically, since “T1 = 1H, T3 = 1000H” is set, the control unit 15 automatically calculates “T2 = 1 × (1000/1) ^1/2 ≈31.6”. .
Here, since “31.6” is a decimal (the branch of “Yes” in Step S306), the control unit 15 next adjusts the exposure ratio (Step S308). In the present embodiment, adjustment is performed with priority so that the exposure ratio is a power of two.

However, in this case, if the power of 2 is given priority and “T2 / T1 = T3 / T2 = 32”, T3 is 1024H, which exceeds the maximum value, so “T2 / T1 = T3 / T2 = 31”. T2 and T3 are adjusted so that As a result, “T1: T2: T3 = 1: 31: 961” is determined (generated) as the exposure ratio information.
Further, the control unit 15 generates an exposure amount signal including the exposure ratio information “T1: T2: T3 = 1: 31: 961”, outputs the generated exposure ratio information to the image composition unit 14, and generates the generated exposure amount. The signal and various control signals are transmitted to the image sensor 10 (step S310).
As a result, information of the exposure time ratio “T1: T2: T3 = 1: 31: 961” is written in the register of the HDR sensor 14d of the image sensor 10.

  On the other hand, when the noise reduction mode is selected on the selection mode selection screen (“Yes” branch of step S312), the control unit 15 displays a screen that prompts the liquid crystal screen (not shown) to input the upper limit value of the ratio. indicate. This screen may be a screen that allows the user to select a plurality of upper limit values, or may be a screen on which the user directly inputs an arbitrary value. When the ratio upper limit value is input via the operation unit 17 (“Yes” branch of step S314), the shortest exposure time Tmin and the longest exposure time Tmax that hold the input ratio upper limit value are then set. Set. Here, it is assumed that the ratio upper limit = 16 is input.

Specifically, the control unit 15 sets T1 and T3 that can hold “T3 / T1 ≦ 16”. Here, it is assumed that T1 = 2H and T3 = 500H are set.
Next, the control unit 15 calculates the intermediate exposure time T2 based on the set T1 and T3 and the above equation (2) (step S318).
Since “T1 = 2H, T3 = 500H” is set in the control unit 15, “T2 = 2 × (500/2) ^1/2 ≈31.6” is substituted into the above equation (2). calculate.

  Here, since “31.6” is a decimal (the branch of “Yes” in step S320), the control unit 15 next adjusts the exposure ratio (step S322). In the present embodiment, adjustment is performed with priority so that the exposure ratio is a power of two. Here, giving priority to the power of 2, the control unit 15 adjusts T2 = 31.6H to 32H and adjusts T3 = 500H to 512H. Thereby, “T1: T2: T3 = 2: 32: 512” is determined (generated) as the exposure ratio information. As a result, the dynamic range is 48 [dB], but when the “S / N” of the sensor cell itself is not good, the noise generated in the boundary region can be reduced compared to when the maximum range is set.

Further, the control unit 15 generates an exposure amount signal including the exposure ratio information “T1: T2: T3 = 2: 32: 512”, outputs the generated exposure ratio information to the image composition unit 14, and generates the generated exposure amount. The signal and various control signals are transmitted to the image sensor 10 (step S324).
As a result, information of the exposure time ratio “T1: T2: T3 = 2: 32: 512” is written in the register of the HDR sensor 14d of the image sensor 10.
Since the subsequent imaging processing operations are the same as those in the first embodiment, description thereof is omitted.

As described above, the imaging apparatus 1 of the present embodiment can select either the range priority mode or the noise reduction mode as the exposure ratio information determination method.
When the range priority mode is selected, the control unit 15 performs exposure corresponding to the exposure amounts L1 to L3 in which the ratios “L2 / L1” and “L3 / L2” are equal so that the dynamic range is maximized. Determine the time ratio. When the noise reduction mode is selected, the exposure time corresponding to the exposure amounts L1 to L3 in which the ratios “L2 / L1” and “L3 / L2” are equal without exceeding the input ratio upper limit value. Determine the ratio.
Thus, in the range priority mode, an image expressed with a wider gradation can be obtained, and in the noise reduction mode, an image with an image quality with reduced noise can be obtained.

In addition, when the intermediate exposure time T2 is calculated, the control unit 15 can adjust T1 to T3 so that the exposure time ratios are all natural numbers based on the calculation result when T2 becomes a decimal. . At this time, the exposure time ratio can be preferentially adjusted to be a power of 2.
Thereby, generation of noise due to a rounding error can be suppressed, and multiplication or the like can be performed by a shift operation by setting the power to 2. Therefore, the circuit configuration in each processing unit can be simplified.
In the second embodiment, the imaging device 10 and the control unit 15 correspond to the imaging unit described in the third form, and the memories 11 to 13 and the image composition unit 14 correspond to the combining unit described in the first form.

In the second embodiment, the function of selecting the exposure ratio information determination method (mode) by the operation unit 17 and the control unit 15 corresponds to the selection unit described in the third embodiment.
In the second embodiment, the function of determining the exposure ratio information based on the selected determination method (range priority mode or noise reduction mode) in the control unit 15 corresponds to the exposure amount determination unit described in the third embodiment. To do.

In the second embodiment, the function of setting the shortest exposure time and the longest exposure time after selecting the exposure ratio information determination method in the control unit 15 corresponds to the exposure amount setting means described in the fifth embodiment.
Note that the imaging device 1 in each of the above embodiments can be combined with other devices (not shown) such as a display device and a memory device to constitute an electronic device such as a digital camera or a digital video camera.

Further, in each of the above embodiments, the three or more types of exposure amounts with equal ratios are realized by controlling the exposure time, but not limited to this configuration, depending on the function of the image sensor 10, For example, it may be configured to be realized by controlling an iris (aperture), imaging sensitivity, and the like.
Further, in each of the above embodiments, the method for determining the exposure time ratio for performing imaging with three types of exposure amounts at which the ratio is uniform has been described with some examples, but the method described in the above embodiment Not limited to. For example, a method may be used in which the S / N of the actual sensor cell array 56 is examined, and based on this S / N, a setting is made so that no noise is generated in the connected portion of the image at the time of composition.

In the second embodiment, the two methods, the range priority mode and the noise reduction mode, have been described as examples of the exposure ratio information determination method. However, the determination method is not limited to these.
For example, a method in which the user sets any one of any dynamic range and any T1 to T3, and determines the remaining two exposure times using the above equation (2) based on the setting information. May be used. Specifically, when an arbitrary range is set and an arbitrary T1 is set, the remaining T2 and T3 are determined. When an arbitrary T2 is set, the remaining T1 and T3 are determined, and an arbitrary T3 is determined. Is set, the remaining T1 and T2 are determined.

In the above embodiments, the intermediate exposure time T2 is calculated using the above equation (2). However, the present invention is not limited to this configuration. For example, a configuration may be adopted in which T2 for predetermined T1 and T3 is calculated in advance using the above equation (2) to generate a data table (LUT), and T2 is determined using the generated LUT.
Further, in each of the above embodiments, the HDR sensor 10d of the image sensor 10 has the sensor cell array 56 configured using the CMOS technology. However, the configuration is not limited thereto. For example, other configurations such as a configuration having a sensor cell array composed of CCDs may be used.

Further, in each of the above embodiments, the HDR signal processing unit 14a in the image synthesis unit 14 is configured to use the synthesis method shown in the flowchart of FIG. 7, but the present invention is not limited to this configuration and may be configured to use another synthesis method. Good.
Each of the above embodiments is a preferable specific example of the present invention, and various technically preferable limitations are given. However, the scope of the present invention is particularly limited in the above description. As long as there is no description, it is not restricted to these forms. In the drawings used in the above description, for convenience of illustration, the vertical and horizontal scales of members or parts are schematic views different from actual ones.
Further, the present invention is not limited to the above-described embodiments, and modifications, improvements, and the like within the scope that can achieve the object of the present invention are included in the present invention.

DESCRIPTION OF SYMBOLS 1 ... Imaging device, 10 ... Imaging element, 10a ... Lens, 10b ... Micro lens, 10c ... Color filter array, 10d ... HDR sensor, 11-13 ... Memory, 14 ... Image composition part, 14a ... HDR signal processing part, 14b DESCRIPTION OF SYMBOLS ... Frame memory, 15 ... Control part, 16 ... Image analysis part, 17 ... Operation part, 50 ... Reference | standard timing generator, 54 ... Scanning line scanner, 54a ... Reset scanning counter, 54b ... Reset scanning address decoder, 54c ... Read scanning Counter 54d Read scan address decoder 56 Sensor cell array 58 Horizontal transfer unit

Claims (9)

Imaging means for imaging a subject with three or more types of exposure amounts in which the ratio of each two adjacent exposure amounts in ascending or descending order is equal;
An image pickup apparatus comprising: combining means for combining pixel signals corresponding to the exposure amounts of the three or more types of exposure amounts obtained by imaging with the image pickup means based on the ratio. The three or more types of exposure amounts are L1, L2,..., Ln (n is a natural number of 3 or more and L1 <L2 <... <Ln),
In the image pickup means, the ratios “L2 / L1,..., Ln / L (n−1)” of two adjacent exposure amounts are equal (L2 / L1 =... = Ln / L (n−1). 2. The imaging apparatus according to claim 1, wherein the subject is imaged at exposure times T1, T2,..., Tn (T1 <T2... <Tn). Selecting means for selecting any one of the plurality of types of exposure amount determination methods for determining the three or more types of exposure amounts;
Exposure amount determination means for determining the three or more types of exposure amounts based on the determination method selected by the selection means,
The imaging apparatus according to claim 1, wherein the imaging unit images a subject with three or more types of exposure amounts determined by the exposure amount determination unit.   The determination method sets a maximum exposure amount and a minimum exposure amount among the three or more types of exposure amounts, and a ratio of each of the two adjacent exposure amounts is based on the set maximum exposure amount and minimum exposure amount. The imaging apparatus according to claim 3, further comprising a method of calculating an equal intermediate exposure amount and determining the three or more types of exposure amounts. Exposure amount setting means for setting a maximum exposure amount and a minimum exposure amount among the three or more types of exposure amounts;
Based on the maximum exposure amount and the minimum exposure amount set by the exposure amount setting means, an intermediate exposure amount in which the ratio between the two adjacent exposure amounts is equal is calculated to determine the three or more types of exposure amounts. The imaging apparatus according to claim 1, further comprising an exposure amount determining unit. When imaging a subject with three types of exposure amounts L1, L2 and L3 (L1 <L2 <L3),
The imaging apparatus according to claim 4, wherein the exposure amount determining unit calculates an exposure amount L2 that is the intermediate exposure amount according to the following expression (1).
L2 = L1 × (L3 / L1) ^1/2 (1)   Based on the information related to the brightness of the imaging environment, the three or more types of exposure amounts are suitable for the brightness of the imaging environment while maintaining a uniform ratio of the two adjacent exposure amounts. The imaging apparatus according to claim 1, further comprising an exposure amount adjusting unit that adjusts a control amount related to an exposure amount of the imaging unit. An imaging step of imaging the subject with three or more types of exposure amounts in which the ratio of each two adjacent exposure amounts in ascending or descending order is equal;
And a synthesis step of synthesizing pixel signals corresponding to each of the three or more exposure amounts obtained by imaging in the imaging step based on the ratio.   An electronic device comprising the imaging device according to any one of claims 1 to 7.

Images