JP6333095B2 - Imaging apparatus, control method therefor, and program - Google Patents

Description

  The present invention relates to an imaging apparatus, a control method thereof, and a program, and more particularly to a technique for generating a high dynamic range image.

  A technique called high dynamic range (HDR) composition is known in which a plurality of images taken with different exposures (underexposure and overexposure) are combined to expand the dynamic range of the images. In this technique, by combining the appropriately exposed portions of each image, it is possible to suppress overexposure and underexposure and to obtain an image having good gradation from a dark portion to a bright portion.

  The HDR synthesizing technique is a technique applicable not only when shooting a still image but also when shooting a moving image. As a configuration for performing HDR composition (moving image HDR shooting) when shooting a moving image, for example, a configuration in which different exposures (under exposure and over exposure) are alternately applied to each frame of the moving image is known. When shooting is performed in this manner, for example, when a moving image is shot at a frame rate of 60 fps, the frame rate of the moving image recorded by HDR synthesis is 30 fps, which is half.

  In order to perform HDR shooting using each frame of a moving image, it is necessary to sequentially control exposure conditions applied to each frame. Japanese Patent Application Laid-Open No. H10-228688 discloses a technique for performing peak detection of the luminance of a frame every frame and controlling an exposure time for a frame to be shot by applying underexposure. Also, in Patent Document 2, first, shooting is performed with the exposure conditions and gradation conversion characteristics set as a reference, and based on the obtained signal level (that is, the photometric result of the AE sensor), each frame used at the time of actual shooting is recorded. A technique for adaptively selecting a combination of exposure and tone conversion characteristics is disclosed.

JP 2009-213032 A JP 2006-254279 A

  By the way, in moving image HDR shooting, AE (automatic exposure control) may be performed during moving image shooting. In HDR shooting of still images, exposure control by AE is performed before main shooting, whereas in moving image HDR shooting, exposure control by AE needs to be performed in parallel with the shooting of moving images. Since the appropriate exposure gradually changes during the exposure control process, the exposure difference between the underexposure and overexposure used for moving image HDR shooting becomes different from the assumption. Conventionally, image processing parameters such as digital gain and gamma can be determined in advance in order to align the exposure after development with an image captured with underexposure and overexposure. However, when an exposure difference that is different from what is assumed occurs due to exposure control by AE, the luminance gradation continuity between images shot with underexposure and overexposure is reduced during HDR composition after development. It is assumed that a tone jump in which a boundary is generated occurs.

  The present invention has been made in view of the above-described problems of the prior art, and is an imaging apparatus capable of improving the continuity of the gradation of a generated image when automatic exposure control is performed in HDR shooting using a moving image. And its control method and program.

  In order to solve this problem, for example, an imaging apparatus of the present invention has the following configuration. That is, an imaging unit, an imaging control unit that controls the imaging unit to repeatedly capture images with different exposures at a predetermined frame rate, an exposure control unit that adjusts the amount of light incident on the imaging unit, and during imaging by the imaging unit When the exposure control means adjusts the amount of light and the exposure difference of images with different exposures fluctuates from a predetermined value, the image of the frame captured with the exposure adjusted by the exposure control means is based on the changed exposure difference. The image processing apparatus includes correction means for correcting, and combining means for combining images of a plurality of frames including the image of the frame corrected by the correcting means.

  According to the present invention, it is possible to improve the continuity of the gradation of a generated image when automatic exposure control is performed in HDR shooting using a moving image.

Sectional drawing (a) of the digital camera 100 as an example of the imaging device which concerns on embodiment of this invention, and the block diagram (b) which shows the function structural example 6 is a flowchart showing a series of operations of a moving image HDR shooting process according to the first embodiment. The figure which shows the signal acquisition for photometry and area division based on Embodiment 1 Program diagram used to determine exposure conditions in moving picture HDR shooting processing according to Embodiment 1 FIG. 6A is a diagram for explaining an example of changes in the scene brightness value, aperture value, etc., and FIG. 6B is a diagram for explaining an example of exposure difference change. 1 is a block diagram illustrating an example of a functional configuration of a signal processing device according to a first embodiment. The figure which shows the synthetic | combination ratio in the HDR synthetic | combination which concerns on Embodiment 1. FIG. 6 is a sequence diagram illustrating a series of operations of the moving image HDR shooting process according to the first embodiment. FIG. 6A is a diagram for explaining an example of changes in scene brightness values, aperture values, and the like, and FIG. 6B is a diagram for explaining an example of changes in exposure difference. The figure for demonstrating the process which estimates the control value with respect to Bv value which concerns on Embodiment 2. FIG. FIG. 3 is a block diagram illustrating an example of a functional configuration of a signal processing device according to a second embodiment. FIG. 6 is a diagram for explaining gain correction by gamma conversion according to the second embodiment. FIG. 10 is a sequence diagram showing a series of operations of moving image HDR shooting processing according to the second embodiment. FIG. 6 is a diagram for explaining gain correction by gamma conversion according to a modification of the second embodiment. 8 is a flowchart showing a series of operations of a moving image HDR shooting process according to the second embodiment.

(Embodiment 1)
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. Hereinafter, as an example of the imaging apparatus, an example in which the present invention is applied to a digital camera having an imaging function capable of acquiring a moving image by changing exposure will be described. However, the imaging apparatus referred to in the present invention is not limited to a digital camera, and can be applied to any electronic device having such an imaging function. These electronic devices may include, for example, a mobile phone, a game machine, a tablet terminal, a personal computer, a clock-type or glasses-type information terminal.

(1 Digital camera configuration)
FIG. 1A is a cross-sectional view mainly illustrating an arrangement of optical members, sensors, and the like of a digital camera 100 as an example of an imaging apparatus of an embodiment. The digital camera 100 is a so-called digital single-lens reflex camera with interchangeable lenses, and has a camera body 1 and an interchangeable lens 2.

  An image sensor 10 in the camera body 1 is, for example, a CMOS image sensor or a CCD image sensor, and a plurality of pixels (storage type photoelectric conversion elements) are arranged. In addition to the pixels arranged in the light receiving section, an amplification circuit for pixel signals, a peripheral circuit for signal processing, and the like are formed. The image sensor 10 realizes an electronic shutter function that allows exposure control by the image sensor by adjusting the charge accumulation time for the pixels and the charge readout. A subject optical image formed by a photographing optical system provided in the interchangeable lens 2 is photoelectrically converted by each pixel, and an image signal in units of pixels is output.

  A mechanical shutter 11 provided near the front of the image sensor 10 adjusts the exposure timing and exposure time of the image sensor 10. The semi-transmissive main mirror 3 and the first reflecting mirror 7 disposed on the back surface of the main mirror 3 jump up to the top during photographing. The second reflection mirror 8 further reflects the light beam reflected by the first reflection mirror 7 and makes it incident on the AF sensor 9. The AF sensor 9 may be an image sensor having a smaller number of pixels than the image sensor 10, for example. The first reflection mirror 7, the second reflection mirror 8, and the AF sensor 9 are configured to perform focus detection by a phase difference detection method at an arbitrary position in the photographing screen. Note that, during live view display and moving image recording, the main mirror 3 is always in a state of jumping upward, so that exposure control and focus adjustment control are performed using image information on the imaging surface.

  The AE sensor 6 is a photometric sensor, and performs photometry by receiving the luminous flux of the subject reflected by the pentaprism 4 and the third reflecting mirror 5. The AE sensor 6 can divide the light receiving unit into a plurality of areas and output subject brightness information for each area.

  A finder optical system is configured by the pentaprism 4. The subject optical image reflected by the pentaprism 4 can be observed from an eyepiece (not shown). Of the light beam reflected by the pentaprism 4, a part of the light beam reflected by the main mirror 3 and diffused by the focus plate 12 is incident on the AE sensor 6.

  The interchangeable lens 2 is a detachable lens, and communicates with the camera body 1 as needed through a contact point of a lens mount provided in the camera body 1.

  FIG. 1B is a block diagram illustrating a functional configuration example of the camera body 1 and the interchangeable lens 2 of the digital camera 100 illustrated in FIG. One or more of the functional blocks shown in FIG. 1B may be realized by hardware such as an ASIC or a programmable logic array (PLA), or a programmable processor such as a CPU or MPU executes software. It may be realized by. Further, it may be realized by a combination of software and hardware. Therefore, in the following description, even when different functional blocks are described as the operation subject, the same hardware can be realized as the subject.

  The control unit 21 is, for example, a one-chip microcomputer incorporating an ALU (Arithmetic and Logic Unit) 33, a ROM 34, a RAM 35, an A / D converter 31, a timer 32, a serial communication port (SPI) 36, and the like. The control unit 21 controls the operations of the camera body 1 and the interchangeable lens 2 by executing a program stored in the ROM 34, for example. For example, the control unit 21 controls each unit of the camera body 1 in order to execute processing related to moving image HDR shooting described later. Output signals of the AF sensor 9 and the AE sensor 6 are input to the A / D converter 31 of the control unit 21.

  The signal processing unit 25 controls the image sensor 10 in accordance with an instruction from the control unit 21 and obtains an image signal by applying A / D conversion and signal processing to the signal output by the image sensor 10. When recording the obtained image signal, the signal processing unit 25 compresses the image signal in a predetermined format and outputs it to the storage unit 26. Further, development processing and composition processing necessary for moving picture HDR shooting described later are performed.

  The memory 28 is a storage device configured by a DRAM or the like. The memory 28 is used as a work memory when the signal processing unit 25 performs various signal processing, and is also used as a VRAM when displaying an image on the display unit 27 described later.

  The display unit 27 includes a liquid crystal display panel and displays information such as camera setting values, messages, menu screens, and captured images. The display unit 27 is controlled by an instruction from the control unit 21. The storage unit 26 is a non-volatile memory such as a flash memory, for example, and a captured image signal is input from the signal processing unit 25.

  The motor 22 performs up / down of the main mirror 3 and the first reflection mirror 7 and charging of the mechanical shutter 11 under the control of the control unit 21.

  The operation unit 23 is an input device group such as a switch used by the user to operate the camera. The operation unit 23 includes a release switch for instructing the start of shooting preparation operation and shooting start, a shooting mode selection switch for selecting a shooting mode, a direction key, a determination key, and the like.

  The contact part 29 is a contact for communicating with the interchangeable lens 2 and is connected to an input / output signal of the serial communication port of the control part 21. The shutter drive unit 24 is connected to the output terminal of the control unit 21 and drives the mechanical shutter 11.

  The interchangeable lens 2 is provided with a contact portion 50 that makes a pair with the contact portion 29. A lens control unit 51, which is a one-chip microcomputer similar to the control unit 21, is connected to the contact unit 50, and communication with the control unit 21 is possible. For example, the lens control unit 51 executes a program stored in the ROM, and controls the operation of the interchangeable lens 2 based on an instruction from the control unit 21. Further, the controller 21 is notified of information such as the state of the interchangeable lens 2.

  The focus lens drive unit 52 drives the focus lens based on an instruction from the lens control unit 51, and the zoom drive unit 53 changes the angle of view of the interchangeable lens based on an instruction from the lens control unit 51.

  The aperture driving unit 54 adjusts the aperture amount of the aperture based on an instruction from the lens control unit 51 and adjusts the amount of light incident on the image sensor 10.

  When the interchangeable lens 2 is attached to the camera body 1, the lens control unit 51 and the control unit 21 can perform data communication via the contact units 29 and 50. For example, lens-specific optical information necessary for the control unit 21 to perform focus detection and exposure calculation, information on the subject distance based on the distance encoder, and the like are output from the lens control unit 51 to the control unit 21 by data communication. Is done. When the control unit 21 transmits focus and aperture control information obtained by focus detection and exposure calculation to the lens, the lens control unit 51 controls the aperture and the like according to the received control information. The contact portion 29 supplies power to a motor or actuator (not shown) provided in the interchangeable lens 2 via the lens-side contact portion 50.

(2. A series of operations related to movie HDR shooting)
Next, a series of operations related to moving image HDR shooting will be described with reference to FIG. This process is realized by the control unit 21 expanding and executing the program stored in the ROM 34 in the work area of the RAM 35.

  In S11, when the control unit 21 becomes operable, for example, when a power switch included in the operation unit 23 is turned on, the control unit 21 communicates with the lens control unit 51 of the interchangeable lens 2 to perform various kinds of operations necessary for focus detection and photometry. Perform initialization processing such as obtaining lens information.

  In S12, when the control unit 21 detects that the moving image mode switching switch included in the operation unit 23 is turned on, the control unit 21 switches the arrangement processing of the memory 28 to the moving image mode and starts imaging.

  In S13, when the control unit 21 detects that the switch included in the operation unit 23 has been switched to the moving image HDR shooting, the control unit 21 switches the shooting mode to the moving image HDR shooting mode.

  In S14, the control unit 21 sets appropriate exposure conditions. In the moving picture HDR shooting mode, imaging control is performed in which imaging is alternately repeated with a preset exposure, for example, + 1EV overexposure and -1EV underexposure for a proper exposure. In the present embodiment, an appropriate exposure for setting each exposure is calculated. The calculation of the appropriate exposure is periodically executed during imaging in order to cope with a case where the scene changes even after calculating the appropriate exposure once and controlling to the appropriate exposure.

  Hereinafter, the process for calculating the appropriate exposure in this step will be described in more detail. In the moving image mode started in S12, the control unit 21 performs moving image shooting, which is a so-called live view image (or through image), that causes the display unit 27 to function as an electronic viewfinder. This photographing is performed by continuously performing photographing using a so-called electronic shutter. In this state, since the mirror is raised, photometry cannot be performed by the AE sensor 6.

The control unit 21 performs photometry by periodically obtaining a signal for photometry by the image sensor 10 while capturing a live view image. For example, one frame of a live view image may be acquired as an image signal for photometry. The control unit 21 reads an image signal from the image sensor 10, performs A / D conversion, and stores it in the RAM. As shown in FIG. 3, the control unit 21 divides the pixel area into, for example, 9 × 7 blocks, and calculates luminance information for each block using the pixel signals included in each block. To do. The luminance information of each block is calculated, for example, by averaging the luminance of all the pixels in the block. Note that the calculation of the luminance is performed by converting into a Bv value (subject luminance at appropriate exposure) in, for example, APEX (Additive System of Photographic Exposure) expression. Based on these luminance information, projection data of Y1 to Y7 and X1 to X9 are calculated. In general, a method of converting data of a two-dimensional array such as m rows × n columns into data of a one-dimensional array added or averaged in the row direction or the column direction is referred to as projection or projection from two dimensions to one dimension. Further, data of a one-dimensional array obtained as a result of addition in the column direction or the row direction is referred to as a projection image or projection data. In the present embodiment, the projection data Y1 to Y7 and X1 to X9 are calculated as follows.
X1 = Σ (x1) ÷ 7 where x = 1-7
X2 = Σ (x2) ÷ 7 where x = 1-7
X3 = Σ (x3) ÷ 7 where x = 1-7
X4 = Σ (x4) ÷ 7 where x = 1-7
X5 = Σ (x5) ÷ 7 where x = 1-7
X6 = Σ (x6) ÷ 7 where x = 1-7
X7 = Σ (x7) ÷ 7 where x = 1-7
X8 = Σ (x8) ÷ 7 where x = 1-7
X9 = Σ (x9) ÷ 7 where x = 1-7
Y1 = Σ (1y) ÷ 9 where y = 1-9
Y2 = Σ (2y) ÷ 9 where y = 1-9
Y3 = Σ (3y) ÷ 9 where y = 1-9
Y4 = Σ (4y) ÷ 9 where y = 1-9
Y5 = Σ (5y) ÷ 9 where y = 1-9
Y6 = Σ (6y) ÷ 9 where y = 1-9
Y7 = Σ (7y) ÷ 9 where y = 1-9
The control unit 21 obtains the maximum value _{E max} in the projection data Y1~Y7 and X1~X9 calculated, calculates a first exposure correction value γ on the basis of the maximum value _{E max} determined. Specifically, when the value of E _max exceeds 10 as the Bv value, it is calculated by the following formula.

γ = (E _max −10) × 0.25
Here, the first exposure correction value γ performs exposure correction in accordance with an empirical rule that it is often preferable that a high-luminance subject having a Bv value exceeding 10 is bright. The coefficient 0.25 is merely an example, and an optimal value may be determined depending on how bright a high-luminance subject is to be captured. Further, the average Bv (AVEBv) of the entire image signal is calculated by the following formula: AVEBv = Σ (Xx + Yy) ÷ (7 × 9)
However, x = 1-7, y = 1-9

As a result, the control value for the Bv value is
Proper exposure: AVEBv + γ
It becomes.

  The control unit 21 determines the exposure condition with reference to a preset program diagram based on the control value for the Bv value. FIG. 4 shows an example of a program diagram used in the moving image HDR shooting. A solid line indicates proper exposure, a broken line indicates overexposure, and a two-dot chain line indicates underexposure. The right vertical axis represents the aperture value Av, and the left vertical axis and the upper horizontal axis represent the Bv value. The lower horizontal axis indicates Tv (shutter speed), but in the diagram shown in FIG. 4, the ISO sensitivity is displayed for 1/30 seconds or more. This is because in moving image shooting, there is a limit of the Tv value according to the frame rate. For example, in FIG. 4, when the proper exposure is 30 / fps, the longest second setting on the over side is also 1/30 second, so that the side where the exposure increases more than this will be realized by increasing the ISO sensitivity. Yes. Based on the control value for the Bv value described above, the control unit 21 refers to a data table corresponding to the program diagram to determine an exposure condition (ISO sensitivity, aperture, second time), and performs exposure according to the condition. Take control.

  In S15, the control unit 21 drives the lens to the focus detection process and the determined focus position. The control unit 21 accumulates a focus detection signal in the image sensor 10. The reason why focus detection using the AF sensor 9 is not performed is that continuous shooting is performed with the mirror raised in order to record a moving image. Therefore, the focus detection by the contrast detection method is performed from the image signal obtained by the image sensor 10. When the accumulation is completed, A / D conversion is performed while reading the accumulated signal, and it is stored in the RAM 35 as an image signal for focus detection. Further, the focus state of each part of the shooting screen is calculated from the lens information obtained in S11 and the image signal for focus detection, and a focus detection area to be focused is determined. The position of the focus detection area may be determined by another method. For example, the focus detection area may be determined in advance as a focus detection area designated by the user through the operation unit 23, or using an image recognition technique such as face detection. A focus detection area may be set on the face of a person. The control unit 21 calculates a lens movement amount for focusing on the focus detection region according to the determined focus state and lens information in the focus detection region, and controls the focus lens driving unit 52 through the lens control unit 51. As a result, the interchangeable lens 2 is brought into focus with respect to the subject in the focus detection area. Since the information of the distance encoder changes when the focus lens is driven, the control unit 21 also updates various lens information in the interchangeable lens 2.

  In S <b> 16, the control unit 21 determines whether aperture control is necessary in a frame to be processed in order to realize overexposure. Of a set of frames used for HDR synthesis, a frame that is overexposed is referred to as an overframe, and a frame that is underexposed is referred to as an underframe. First, for example, it is assumed that the proper exposure condition set at a certain timing is Bv11 and the luminance acquired from the image signal is Bv10. On the program diagram shown in FIG. 4, the aperture value Av does not change only by changing the ISO sensitivity from 200 to 400 by 1 EV. In this case, the control unit 21 determines that aperture control is not necessary, and proceeds to S18 to perform overexposure imaging (over imaging) with the current exposure.

  On the other hand, it is assumed that the proper exposure condition set at a certain timing is Bv8, and the luminance acquired from the image signal is Bv10. On the program diagram, the aperture changes from 2.8 to 5.6, and the aperture value Av changes by 2 EV. In this case, the control unit 21 determines that aperture control is necessary, and proceeds to S17.

In S17, when the control unit 21 calculates the exposure condition for overexposure in the overframe, the control unit 21 performs exposure control using the calculation result. For the calculation of the overexposure exposure condition, the program diagram example of FIG. 4 is referred to in the same manner as the calculation of the proper exposure condition. In the figure, wavy lines indicate overexposure. Since the control value for the previously obtained Bv value is a value related to the proper exposure, the control value may be further corrected for overexposure and then the program diagram may be referred to as follows. When the control unit 21 controls the lens 2 to perform exposure control based on the calculated exposure condition, the control unit 21 performs over imaging in a reference frame, that is, a frame that is overexposed in S18.

Over: AVEBv + γ + α where α is the exposure difference with respect to proper exposure

Although the details will be described later in the processing of S21, the over-imaging exposure condition determined here is a target value, and in reality, a process of approaching the target exposure condition over several frames from a certain timing is performed. The

  In S19, the control unit 21 advances the frame to be processed by one frame, and determines whether aperture control is necessary to realize underexposure in the underframe. Whether or not aperture control for realizing underexposure is necessary is performed in the same manner as the determination of overexposure aperture control in S16. If the control unit 21 determines that the aperture control is not necessary, the control unit 21 advances the process to S23 to perform the over-development process. On the other hand, when it is determined that the aperture control is necessary, the control unit 21 advances the process to S20 to perform underexposure calculation and exposure control.

In S20, the control unit 21 calculates an exposure condition for underexposure in the same way as appropriate exposure with reference to the program diagram example shown in FIG. 4, and performs exposure control using the calculation result. In FIG. 4, the alternate long and short dash line indicates underexposure. Since the control value for the previously obtained Bv value is a value related to the proper exposure, the control value may be further corrected for the underexposure and the program diagram may be referred to as follows. When the control unit 21 completes the process, the process proceeds to S21 to acquire the actual exposure difference.

Under: AVEBv + γ-β where β is the exposure difference from the proper exposure

In S <b> 21, the control unit 21 acquires an actual exposure difference between overexposure and underexposure after exposure adjustment by exposure control. Hereinafter, details of this step will be described with reference to FIGS. 5 (a) and 5 (b). In a moving image, the smoother the change in exposure, the better the visibility, so instead of controlling the immediately following frame to be the exposure condition determined in S17, it gradually approaches the target value over a plurality of frames. . Therefore, while continuing the aperture control operation toward the target value, the actual aperture value Av of the overframe and the underframe is acquired for each frame and reflected in the gain correction.

  First, exposure conditions up to several frames ahead are determined in advance. For example, how many frames the aperture value Av is changed to the target value is set for the actually changed scene brightness value Bv, and the exposure condition of each frame is set. FIG. 5A shows an example of changing the Bv value and the associated Av value over 4 frames. A dotted line in FIG. 5A indicates an actual control value with respect to the Bv value.

  What is important here is the exposure difference between the overframe and the underframe, and FIG. 5B shows the exposure difference using a control value for the Bv value. The control value for the Bv value is not the Bv value of the photometric scene but the Bv value for determining the overframe and underframe exposure conditions. In the example of FIG. 5 (b), it is shown that the assumed exposure difference can be realized in the state before the change of the Av value due to the aperture driving starts (represented by “b” shown in FIG. 5 (b)). Exposure difference). In the above-described example of the scene, the Bv value changes from 8 (dark scene) to 10 (bright scene), and b = 2EV. However, the exposure difference between the overframe and the underframe is 2a + b during the Av value variation period starting from the first over-imaging in one set of frames used for HDR synthesis. Originally, the exposure difference between the overframe and the underframe should be a predetermined exposure difference b. However, since the proper exposure itself is fluctuated by controlling Av at the same time, error exposure 2a occurs.

  The control unit 21 acquires the overframe and underframe aperture values used for the HDR synthesis from the lens control unit 51, and acquires the actual exposure difference. When the control unit 21 acquires the actual exposure difference, the control unit 21 outputs the actual exposure difference to the signal processing unit 25 for use in gain correction processing by the signal processing unit 25.

  In S <b> 22, the signal processing unit 25 generates a development parameter for performing gain correction on the image signal based on the actual exposure difference acquired from the control unit 21. The development parameter in the present embodiment is a white balance coefficient described later.

ここでＷＢＣｏｅｆｘ（ｘ＝Ｒ、Ｇ１、Ｇ２、Ｂ）は、修正前のホワイトバランス係数であり、Ｂｖ_tは、各フレームにおけるＢｖ値、ｂは図５（ｂ）に示す目標値である露出差である。修正前のホワイトバランス係数については、「太陽光」や「くもり」など予めカメラにプリセットされている場合は各々設定されているホワイトバランス係数を用いる。オートホワイトバランスに設定されている場合は、例えば画像信号から白っぽい領域を検出してその領域の平均Ｒ、Ｇ、Ｂのバランスを整えるような公知のホワイトバランス係数を算出する方法を用いて行うことができる。 In gain correction, it is necessary to use either the overframe or the underframe as a reference, but in this embodiment, the overframe gain is corrected based on the underframe. In the example shown in FIG. 5A and FIG. 5B, the variation of the Av value is in the direction of a small diaphragm (diaphragm diameter decreases), and the control start frame of the Av value is an overframe. Underexposure is greater than expected. For this reason, the error exposure during the control of the Av value is 2a as described above. Therefore, the gain correction direction is gain up, that is, the influence of error exposure is reduced. The gain correction is realized by digital gain, and is reflected by applying the white balance coefficient generated by the white balance unit 802 shown in FIG. The white balance coefficient is generated by the white balance coefficient correction unit 809 acquiring control values for the over and under Bv values and correcting the white balance coefficient 807. An example of correcting the white balance coefficient 807 will be shown below.

Here WBCoefx (x = R, G1, G2, B) is a white balance coefficient before correction, Bv _t is exposure difference Bv value in each frame, b is the target value shown in FIG. 5 (b) It is. When the white balance coefficient before correction is preset in the camera in advance, such as “sunlight” or “cloudy”, the set white balance coefficient is used. When the auto white balance is set, for example, a known white balance coefficient that detects a whitish area from the image signal and adjusts the average R, G, B balance of the area is used. Can do.

  In S23, the signal processing unit 25 performs development processing. That is, over development including white balance is performed using the corrected white balance coefficient. Here, the details of the development processing for the image signal will be described with reference to FIG. The processing of each functional block is performed on the image signal acquired in each of the overframe and the underframe.

  The subject image is imaged on the image sensor of the image sensor 10 by an imaging optical system (lens) and subjected to photoelectric conversion. The image sensor 10 is, for example, a single-plate color image sensor including a general primary color filter. The primary color filter is composed of three types of color filters each having a transmission main wavelength band in the vicinity of 650 nm, 550 nm, and 450 nm, and each color plane corresponds to each band of R (red), G (green), and B (blue). Shoot. In a single-plate color imaging device, the color filters are spatially arranged for each pixel, and each pixel can only obtain the light intensity in a single color plane. For this reason, a color mosaic image is output from the image sensor (801 and 810). The white balance unit 802 multiplies the R, G, and B pixel values by gains such that the R, G, and B values of the region that should be white have the same signal value. The color interpolation unit 803 generates a color image in which R, G, and B color information are aligned in all pixels by interpolating the color mosaic image. A basic color image is generated from the generated color image through a matrix conversion unit 804 and a gamma conversion unit 805.

  Prior to the high dynamic range composition, it is necessary to align the brightness levels of the overframe and underframe taken with different exposures. The gain applied in the gamma conversion unit 805 needs to be set so as not to cause overexposure or blackout, and therefore is performed using a gamma curve instead of a uniform gain. Regarding the design of these gamma curves, for example, when a gain due to an exposure difference of 2 EV, that is, a gain twice as large as an under and a gain of 1/2 as an over, the characteristics are the same as those of the gamma curve. Design as follows. By doing so, the boundary can be smoothed when a plurality of images are switched in accordance with the luminance range. The color luminance adjustment unit 806 performs processing for improving the appearance of the image on the color image, and performs image correction such as saturation enhancement, hue correction, and edge enhancement, for example. When the processing by the color luminance adjustment unit 806 is completed, the development processing is completed.

  In the case of still image HDR synthesizing, it is general that spatial alignment is performed with respect to movement caused by camera shake or the like with one image as a reference before synthesizing. However, in the case of moving image HDR composition, the over and under shooting intervals are much shorter than still image shooting, and if the alignment is performed, the frame rate is reduced, the amount of processing calculation, etc. The alignment process may not be performed for a reason. When completing the over-development process, the signal processing unit 25 advances the process to S24.

  In S <b> 24, the image sensor 10 sets the underexposure in the underframe and performs imaging (under imaging). In S <b> 25, the signal processing unit 25 performs development parameter generation and development processing for the obtained image signal. However, in this embodiment, since the gain correction is applied only to the over-captured image signal, the development parameter is not corrected for the under-captured image signal.

  Furthermore, in S26, when the development processing for the overframe and underframe image signals is completed, the signal processing unit 25 performs HDR synthesis processing in the HDR synthesis unit 816 using the over-developed image and the under-developed image, respectively.

  The HDR synthesis process will be described with reference to FIG. The horizontal axis represents the reference luminance, and the vertical axis represents the composition ratio when the images are added and combined. Only an over-developed image is used for an area darker than the reference luminance threshold Y1 when combining, and only an under-developed image is used for an area brighter than the reference luminance threshold Y2. Also, in the intermediate region between the boundaries Y1 and Y2 near the reference luminance threshold, the image switching can be made smooth by gradually changing the composition ratio. In this embodiment, the underframe is used as the reference luminance. Through the above processing, the signal processing unit stores the HDR image generated by generating the HDR image in the memory 28.

  In S <b> 27, the control unit 21 determines whether there is an instruction to end imaging by pressing the REC button or the like via the operation unit 23. If it is determined that there is no instruction to end imaging, the process returns to S14 again and the processes from S14 to S26 are repeated. On the other hand, if it is determined that there is an instruction to end imaging, the process proceeds to S28 in order to compress the generated HDR video.

  In S28, the signal processing unit compresses the HDR moving image stored in the memory in a predetermined format, records it in the storage unit 26 such as a memory in S29, and ends the series of operations.

  Regarding the processing of the present embodiment described above, the timing of each processing will be described with reference to the sequence diagram shown in FIG. Note that frames t to t + 3 indicate that attention is paid to a certain section during moving image recording. For this reason, description of initialization processing performed immediately after the start of moving image recording is omitted. Further, during this period, the Av value fluctuates toward a certain target value.

  In the t−1 frame before the start of the t frame, the lens 2 controls to a predetermined Av value in accordance with an instruction from the control unit 21 (1006), and the Av value is stored as an Av value for over imaging. 26 stores (1007).

  In the t frame, the imaging device 10 starts over imaging (1008). The signal processing unit 25 reads out an image signal (over image signal) obtained by over-imaging and stores it in the storage unit 26 (1009). When the over imaging is completed, the lens 2 performs aperture control for t + 1 frame (1010), and then the storage unit 26 stores the Av value for under imaging. The storage unit 26 outputs the Av value for over imaging and the Av value for under imaging (1011), and the signal processing unit 25 corrects the white balance gain using these Av values (1012). When the correction of the white balance gain is completed, the signal processing unit 25 performs over-development and stores the developed image in the storage unit 26 (1013).

  When the start timing of the t + 1 frame is entered, the image sensor 10 starts under imaging separately (1014). The signal processing unit 25 reads an image (under image signal) obtained by under imaging and stores it in the storage unit 26 (1015). The signal processing unit 25 starts under development immediately after imaging (1016). Similarly to the t-1 frame, the lens 2 performs aperture control for over imaging in accordance with an instruction from the control unit 21 (1017), and the storage unit 26 stores the over imaging Av value (1018).

  When it is time to start the t + 2 frame, the image sensor 10 starts over-frame imaging similarly to the t frame (1019). The signal processing unit 25 simultaneously performs HDR composition using the over-developed image and the under-developed image after development (1020), and stores the generated image in the storage unit 26 (1022). Thereafter, the processing from 1060 is repeated until the end of imaging.

  In the present embodiment, an example in which two image signals of an over frame and an under frame are used has been described. However, the present embodiment can also be applied to a case where three or more frames are repeatedly acquired. For example, when the exposure of three frames is changed in stages, the exposure difference generated in the three frames is smoothed by the number of frames, and gain correction obtained from the smoothed exposure difference is used for other than the reference frame. What is necessary is just to apply with respect to an image signal. Further, since there is only an exposure difference between consecutive frames, for example, instead of an overframe and an underframe, a proper frame and an underframe, an overframe and a proper frame, and the like may be used.

  As described above, in the present embodiment, even when the proper exposure changes due to the change of the aperture value by the automatic exposure control, the actual exposure difference is obtained based on the overframe and underframe aperture values. I tried to do it. Then, HDR synthesis is performed on an image whose gain has been corrected with digital gain based on the acquired actual exposure difference. This makes it possible to improve the continuity of the gradation of the generated image when automatic exposure control is performed in moving image HDR shooting. That is, the exposure difference between the overframe and the underframe can be controlled as expected, and image quality degradation such as tone jump can be suppressed in the HDR synthesized image.

(Embodiment 2)
Next, Embodiment 2 will be described. In the first embodiment, the gain correction is performed on the image signal using the digital gain of the white balance. On the other hand, in the present embodiment, the gain correction is performed on the image signal using the gamma characteristic. In the first embodiment, the case where the first frame of the over frame and the under frame used for HDR synthesis is described as an example in which the Av value starts to fluctuate. However, in this embodiment, the Av value fluctuates by two. An example starting from the first frame will be described. The configuration of the digital camera 100 of the present embodiment may be the same as that of the first embodiment. For this reason, the same reference numerals are assigned to the same components, and redundant descriptions are omitted, and differences will be mainly described.

  First, a series of operations related to moving image HDR shooting in the present embodiment will be described with reference to FIG. Note that the processes common to those in FIG.

そして、制御部２１は実際の露出差を取得する。実施形態１ではレンズ２との通信により実際の露出を取得する例を示したが、本実施形態では実際の露出差を推定により取得する。これは、ハード上の制約などで必ずしもレンズ通信による絞り値の取得が困難な場合やレンズとの通信に遅延が生じる場合、処理の高速化を図る場合等に有効である。露出差の推定方法について図１０を参照して説明する。制御部２１は、Ｓ１４において算出した適正露出条件に加えて、何フレームかけて露出を変動させていくかの情報を用いて、所定のタイミングｉにおけるＢｖ値に対する制御値（Ｂｖ_i）を以下の式を用いて算出する。そして、制御部２１は算出したＢｖ値に対する制御値と上述したＡｖ値変動中のフレームにおける露出誤差とを用いて露出差を求める。

Ｓ１５１において信号処理部２５は、推定演算により推定した実際の露出に基づいてゲイン修正を行うための現像パラメータを生成する。ゲイン修正においてはオーバーフレームかアンダーフレームのいずれかが基準フレームとする必要があるが、本実施形態ではオーバーフレームを基準にアンダーフレームのゲインを修正することを考える。本実施形態では、Ａｖ値の変動が開放絞りの方向であり、Ａｖ値の制御開始フレームがＨＤＲ合成フレームに用いる１組のフレームの２枚目となるため、アンダー露出が想定よりも小さくなりＡｖ値の制御中の露出誤差は上述の通りａもしくは２ａ分となる。よって、ゲインを修正する方向はゲインアップとなる。本実施形態におけるゲインアップはガンマ特性を用いたガンマ変換により実現し、図１１に示すガンマ変換部８０５が修正されたガンマテーブルを用いてガンマ変換することによって反映する。ガンマテーブルによるゲイン調整を行うことで、より柔軟に階調特性を制御した現像画像を生成できる。 In S150, the control unit 21 acquires an actual exposure difference. In the present embodiment, for example, it is assumed that the set exposure condition is set to Bv10, and the acquired luminance is Bv8. In this case, the aperture value changes by 2 EV from 5.6 to 2.8 on the program diagram shown in FIG. In other words, since the scene changes from a bright scene to a dark scene, the aperture fluctuates toward the side where the aperture value becomes smaller (the side where the aperture diameter becomes larger). FIG. 9A shows the fluctuation of the control value with respect to the Av value and the Bv value in this case. After the Bv value decreases, the Av value starts to change. The frame from which the Av value starts to change is the second frame of the set of frames used for HDR synthesis, that is, the underframe. For this reason, the overframe and underframe exposure errors differ between immediately after the start of the Av value change and immediately before the end and during the change. In this case, the over frame is exposed as expected when the Av value starts to fluctuate, and the under frame is exposed as expected immediately before the end of the Av value, and the other frame in the set of frames includes the fluctuation of the Av value. The exposure difference in this case is as shown in FIG. The assumed exposure difference is b as in the first embodiment.

And the control part 21 acquires an actual exposure difference. In the first embodiment, an example in which the actual exposure is acquired by communication with the lens 2 is shown. However, in the present embodiment, the actual exposure difference is acquired by estimation. This is effective when it is difficult to obtain an aperture value through lens communication due to hardware restrictions, when there is a delay in communication with the lens, or when processing speed is to be increased. An exposure difference estimation method will be described with reference to FIG. The control unit 21 uses the information on how many frames the exposure is changed in addition to the appropriate exposure condition calculated in S14, and sets the control value (Bv _i ) for the Bv value at the predetermined timing i as follows: Calculate using the formula. And the control part 21 calculates | requires an exposure difference using the control value with respect to the computed Bv value, and the exposure error in the flame | frame in which Av value fluctuation | variation mentioned above.

In S151, the signal processing unit 25 generates a development parameter for performing gain correction based on the actual exposure estimated by the estimation calculation. In gain correction, either the overframe or the underframe needs to be used as a reference frame, but in the present embodiment, it is considered that the gain of the underframe is corrected based on the overframe. In the present embodiment, the variation of the Av value is the direction of the wide open aperture, and the control start frame of the Av value is the second frame of the set of frames used for the HDR composite frame, so the underexposure becomes smaller than expected and Av The exposure error during the value control is a or 2a as described above. Therefore, the gain correction direction is gain up. The gain increase in this embodiment is realized by gamma conversion using the gamma characteristic, and is reflected by the gamma conversion unit 805 shown in FIG. 11 performing gamma conversion using the corrected gamma table. By performing gain adjustment using a gamma table, it is possible to generate a developed image with gradation characteristics controlled more flexibly.

  The gamma table is corrected by correcting the gamma table 1307 in the gamma table correction unit 1310. The gamma table correction unit 1310 acquires from the control Bv value estimation unit 1309 control values for over and under Bv values necessary for correcting the gamma table 1307. In addition, the control Bv value estimation unit 1309 acquires the Av variation start frame, the Av variation end frame, the number of Av control frames, and the frame number to be processed as shooting information 1308 necessary for this estimation.

  A specific method for correcting the gamma table in the gamma table correcting unit 1310 will be described with reference to FIG. In the graph shown in FIG. 12, the horizontal axis represents an input signal value to gamma conversion, but is displayed in the Log scale, and one step engraved with a dotted line represents one stage. Moreover, the point shown with the continuous vertical line has shown appropriate exposure. The vertical axis represents the output signal value of gamma conversion. The solid line gamma table shows the conversion characteristics for underframes, and the dash-dot line gamma table shows the conversion characteristics for overframes. In this figure, the under frame and the over frame have an exposure difference of approximately three steps. A two-dot chain line gamma table indicates a corrected gamma table, which is obtained by shifting the conversion characteristic of the original solid line in the horizontal direction. The shift amount indicated by the arrow may be obtained as a or 2a described above. In S23, the signal processing unit 25 performs development processing using the corrected gamma table.

  A sequence diagram of the processing according to the second embodiment is shown in FIG. In this embodiment, in order to estimate the control value for the Bv value, the control value for the Bv value for the underframe is estimated at t frame and stored in the storage unit 26 (1601). Then, in order to correct the gain for the underframe, that is, the gamma table, the gain value (gamma table) is corrected before developing the underframe (1602) and the underdevelopment is performed (1016).

  In this embodiment, the gain correction is performed by correcting the gamma table. However, it is not always necessary to use the gamma table, and the same moving picture HDR shooting is possible as long as the correction table has a predetermined gradation correction characteristic. Can be realized.

  As described above, in the present embodiment, when the gain correction is performed using the exposure difference, the gamma table is corrected using the exposure difference, and the gain correction is performed using the gamma table. In this way, when automatic exposure control is performed in moving picture HDR shooting, it is possible to generate a developed image with more controlled gradation characteristics and improve the continuity of the gradation of the composite image. . As for the actual exposure difference, the actual exposure difference is estimated based on control information for changing the exposure over a plurality of frames. When a delay occurs in the communication with the lens, the processing speed can be increased.

(Modification of Embodiment 2)
Further, a modification of the second embodiment will be described. In the first embodiment and the second embodiment, the example in which the gain is increased for the correction gain has been described. In this modification, a case will be described in which the correction gain is reduced by gamma conversion. In the present modification, in the application of the correction gain, the correction gain is applied anew so that the saturation value in the high luminance region does not change.

  The situation assumed in this modification is the case where the exposure condition set at a certain timing is set to Bv8 and the luminance acquired at that time is Bv10, as in the first embodiment. In this case, on the program diagram shown in FIG. 4, the aperture changes by 2 EV from 2.8 to 5.6, and the aperture value Av changes by 2 EV. That is, the scene changes from a dark scene to a bright scene, and the aperture fluctuates from a larger value side to a smaller aperture side (a smaller aperture diameter side). The variation of the control value with respect to the Av value and the Bv value is as shown in FIG. 5A, and the exposure difference in this case is 2a + b as shown in the first embodiment.

  As described above, gain correction is performed to reduce this exposure difference. In the gain correction, either the overframe or the underframe needs to be the reference frame, but in this modification, it is considered that the gain of the underframe is corrected based on the overframe. In the present modification, the Av value varies in the direction of small aperture (diaphragm diameter decreases), and the control start frame of the Av value is overframe. Therefore, underexposure is larger than expected. For this reason, the exposure error during the control of the Av value is 2a as described in the first embodiment. The direction of correcting the gain is gain down. Gain correction for gain reduction is performed using gamma conversion, as in the second embodiment. The control value for Bv of the target frame may be acquired through lens communication as in the first embodiment, or may be estimated from shooting information as in the second embodiment.

  A specific method for correcting the gamma table will be described with reference to FIG. The two-dot chain line gamma table shown in FIG. 14 is a corrected gamma table. This gamma table is obtained by shifting the solid line gamma table in the horizontal direction by the exposure error 2a. However, if the shift is performed as it is, the high luminance portion is not saturated. Thus, basically, the original gamma table is shifted in the horizontal direction, and the table is adjusted so that the high luminance part approaches the original gamma table. For example, since the high luminance region from the luminance value of the white circle shown in FIG. 14 is a luminance region in which the under frame is used, the adjustment may be performed in the region. The brightness value of the white circle shown in FIG. 14 is a brightness value corresponding to Y2 shown in FIG.

  As described above, in this modification, the gamma table is adjusted so as to maintain the saturation when the gain correction for reducing the gain is performed. In this way, when performing gain correction for gain reduction with respect to automatic exposure adjustment, it is possible to perform gain correction while maintaining saturation of the high luminance part.

  In the above-described embodiment and modification, only the scene in which the Bv value changes only once during the automatic exposure control is illustrated, but the present invention can also be applied to a scene in which the Bv value continuously changes up and down. Even in such a scene, each change can be dealt with by a combination of gain correction by gain-up and gain-down described in this embodiment.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

DESCRIPTION OF SYMBOLS 10 ... Image sensor, 25 ... Signal processing part, 21 ... Control part, 51 ... Lens control part, 54 ... Aperture drive part, 805 ... Gamma conversion part, 809 ... White balance coefficient correction part, 816 ... HDR synthetic | combination part, 1309 ... Control Bv value estimation unit, 1310... Gamma table correction unit

Claims (10)

Imaging means;
Imaging control means for controlling the imaging means so as to repeatedly capture images with different exposures at a predetermined frame rate;
Exposure control means for adjusting the amount of light incident on the imaging means;
When the light amount is adjusted by the exposure control means during imaging by the imaging means and the exposure difference of the images with different exposures fluctuates from a predetermined value, the frame of the image captured with the exposure adjusted by the exposure control means Correction means for correcting the image based on the changed exposure difference;
An image pickup apparatus comprising: a combining unit configured to combine a plurality of frame images including a frame image corrected by the correcting unit.   The correction unit calculates a correction value that corrects the fluctuation of the exposure difference caused by the adjustment of the light amount of the exposure control unit, and uses the correction value to capture an image of a frame that is captured after the adjustment of the light amount. The imaging apparatus according to claim 1, wherein the imaging device is corrected.   The imaging apparatus according to claim 2, wherein the correction value is a value related to a gradation characteristic.   The imaging apparatus according to claim 2, wherein the correction value is a value relating to control of gamma characteristics or white balance.   5. The imaging apparatus according to claim 2, wherein the correction unit calculates the correction value from a setting value corresponding to an exposure condition after the light amount is adjusted by the exposure control unit.   6. The imaging apparatus according to claim 2, wherein the correction unit estimates and calculates a setting value corresponding to an exposure condition after the light amount is adjusted by the exposure control unit.   The said exposure control means determines whether it is necessary to adjust the said light quantity based on the difference of the said light quantity and appropriate exposure, The any one of Claim 1 thru | or 6 characterized by the above-mentioned. Imaging device. The images with different exposures are an over image captured with overexposure with respect to proper exposure and an under image captured with underexposure with respect to the appropriate exposure,
The imaging apparatus according to claim 1, wherein the image of the frame corrected by the correcting unit includes at least one of the over image and the under image. An imaging step of imaging by the imaging means;
An imaging control step in which the imaging control means controls the imaging means so as to repeatedly capture images with different exposures at a predetermined frame rate;
An exposure control step in which the exposure control means adjusts the amount of light incident on the imaging means;
When the light amount is adjusted by the exposure control unit during imaging by the imaging unit and the exposure difference of the images with different exposures fluctuates from a predetermined value, the correction unit performs imaging with the exposure adjusted by the exposure control unit. A correction step of correcting the image of the frame that has been corrected based on the changed exposure difference;
A method for controlling an imaging apparatus, comprising: a combining unit configured to combine a plurality of frame images including a frame image corrected by the correcting unit.   The program for functioning a computer as each means of the imaging device of any one of Claims 1 thru | or 8.

Images