JP6170402B2 - Imaging apparatus and control method thereof - Google Patents

Description

  The present invention relates to an imaging apparatus for recording and reproducing a moving image and a control method thereof.

  2. Description of the Related Art Recent image recording / playback apparatuses such as digital cameras employ various methods for shooting, recording, and playing back moving images with sound. As usual, there are a method in which shooting is performed until a user gives an instruction to start shooting and a shooting instruction is given, and a method of shooting and recording for a predetermined time from an instruction to start shooting and recording.

  Under these circumstances, in order to obtain an image quality in which both the background and the subject are appropriate, a moving image with an expanded dynamic range is required. In a still image, it is known to combine an image with a high brightness portion and an appropriate exposure with a low illuminance portion, and an image with an expanded dynamic range, with a properly exposed image. Similarly, processing for expanding the dynamic range is also required for moving images. In order to perform image composition with a moving image, it is necessary to obtain a composition ratio corresponding to the movement of the subject at runtime, and it is preferable that the focus of the lens with respect to the subject is adjusted and there is contrast with the background. However, since the focus adjustment is performed in parallel with the moving image shooting and recording, the TTL focus adjustment using the output of the image sensor is performed, so that all images are not in focus during the focus adjustment period.

  On the other hand, the pixel unit and the circuit unit can be arranged to overlap each other by stacking the pixel unit of the CMOS image sensor and the circuit unit responsible for driving the pixel unit. As a result, it is possible to read out images suitable for moving image recording from the pixel portion and simultaneously read out a plurality of images for AF. When the stacked CMOS image sensor as described above is used, it is possible to perform focus adjustment suitable for the movement of the subject by AF evaluation at the same time as recording a moving image, but all the AF images are not in focus. There is no change.

  Japanese Patent Application Laid-Open No. 2004-228561 discloses a technique for causing a common overlapping portion in an imaging period in all the images for synthesis for performing dynamic range expansion. The common overlapping portion includes an image in which the imaging reference is located at the center. Japanese Patent Laid-Open No. 2004-259542 also includes a phase difference pixel and a normal pixel. The focus amount of the photographing lens is calculated using the phase difference pixel, and the normal pixel and the phase difference pixel after focus adjustment control are read to generate a recording image. The technology to do is disclosed.

Japanese Patent No. 4715853 JP 2011-59337 A

  When a plurality of AF evaluation images are obtained at the same time as reading a moving image recording image from the image sensor like a stacked CMOS sensor, the accumulation time of the AF evaluation image is shorter than the moving image recording image. Due to the limitation of the circuit configuration, the AF evaluation image reads integer frames while reading one frame of the moving image recording image. Since the moving image recording image and the AF evaluation image are obtained at the same time, and the respective accumulation times are different, it can be applied to the expansion of the dynamic range.

  On the other hand, during moving image recording, an AF evaluation value is acquired while moving the focus lens to perform focus adjustment. A moving image recording image that is continuously exposed during a period in which the focus lens is moving has a relatively short influence on the in-focus state because the lens movement period with respect to the exposure time is relatively short, and is not noticeable. However, a plurality of AF evaluation images read out at high speed have a significant difference in the in-focus state for each image depending on the focus lens position during the exposure period. In other words, since the focus position is detected with a significant difference in evaluation value depending on the focus lens position, a difference in focus state appears between AF evaluation images according to the depth of field. For this reason, there is a possibility that the alignment between images and the sense of resolution of the synthesized image will be impaired.

  The present invention provides an imaging apparatus capable of accurately aligning and obtaining an image with a wide dynamic range without impairing the resolution when combining an AF evaluation image with a moving image recording image.

According to an aspect of the present invention, an imaging device that captures a subject image via a photographing lens, a focus lens that is included in the photographing lens and is movable in an optical axis direction, and a first frame from the imaging device. to obtain the first image at a rate obtaining means for obtaining a plurality of second images at the first higher than the frame rate second frame rate, the first image and the plurality of second An image in which the dynamic range is expanded by synthesizing a plurality of images obtained from the imaging element, and a focus adjustment unit that performs focus adjustment on the subject image by controlling the position of the focus lens based on a high-frequency component of the image of Combining means for generating the image, wherein the combining means selects an image with a high focusing rate from the plurality of second images, and combines the selected image with the first image. Imaging device that is provided.

  According to the present invention, when focus adjustment is performed while performing moving image shooting and recording, an image with a high focusing rate is selectively combined. As a result, it is possible to accurately perform alignment at the time of synthesis, and to obtain an image with a wide dynamic range without losing a sense of resolution.

The block diagram of the imaging device in an embodiment. 6 is a timing chart for explaining the operation of the imaging apparatus according to the embodiment. The figure which shows the data flow of the image composition in embodiment. 6 is a flowchart illustrating a focus adjustment process in the embodiment. The flowchart which shows the focus position determination process in embodiment. The figure which shows the data flow of the image composition in embodiment. 6 is a timing chart for explaining the operation of the imaging apparatus according to the embodiment.

  DESCRIPTION OF EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings. In addition, this invention is not limited to the following embodiment, It shows only the specific example advantageous for implementation of this invention. Moreover, not all combinations of features described in the following embodiments are indispensable for solving the problems of the present invention.

  FIG. 1 is a block diagram illustrating a configuration of an imaging apparatus 100 according to the present embodiment. The imaging apparatus 100 includes a photographic lens including a zoom lens 110, a shift lens 111 that corrects camera shake, and a focus lens 112 for focus adjustment. The taking lens may be detachable from the main body. The focus lens 112 is configured to be movable in the optical axis direction. The imaging apparatus 100 further includes a mechanical shutter 113 that blocks the light beam to the rear stage of the photographing lens and a diaphragm 114 that adjusts the light beam to the rear stage. The timing generator 116 generates timing pulses necessary for driving the image sensor 115.

  The image processing unit 117 includes an SSG circuit that generates a synchronization signal for imaging driving, a preprocessing circuit, an AF evaluation value calculation circuit, a luminance integration circuit, a signal processing circuit, an image synthesis circuit, a reduction circuit, a raster block conversion circuit, and a compression Includes circuitry. The SSG circuit receives an imaging drive clock from the timing generator, generates horizontal and vertical synchronization signals, and outputs them to the timing generator and the image sensor. The preprocessing circuit distributes the input image to the luminance integration circuit and the signal processing circuit in units of rows. Further, processing such as data correction between channels necessary for imaging data is performed. The AF evaluation value calculation circuit performs horizontal filtering on the luminance component of the image signal input in a plurality of set evaluation areas, selects the maximum value while extracting a predetermined frequency representing the contrast, and Perform the integral operation. In the luminance integration circuit, luminance components are mixed and generated from RGB signals, the input image is divided into a plurality of regions, and a luminance component is generated for each region. The signal processing circuit generates a luminance signal by performing color carrier removal, aperture correction, gamma correction processing, and the like on the image sensor output data. At the same time, color interpolation, matrix conversion, gamma processing, gain adjustment, and the like are performed to generate a color difference signal, and image data in YUV format is formed in the memory unit 123. In the signal processing circuit, the luminance signals (Y) of the YUV format image data to be generated are tabulated for each level, and luminance distribution data for each image data is generated. The image composition circuit inputs a plurality of YUV format image data formed in the memory unit 123 or directly generated from the image sensor output data by the signal processing circuit. Then, the input image data in the YUV format is multiplied by a coefficient set in pixel units and added to generate a composite image. Specific contents will be described later. The reduction circuit receives the output of the signal processing circuit, cuts out input pixel data, performs thinning, linear interpolation processing, and the like, and performs pixel data reduction processing in the horizontal and vertical directions. In response to this, the raster block conversion circuit converts the raster scan image data scaled by the reduction circuit into block scan image data. Such a series of image processing is realized by using the memory unit 123 as a buffer memory. The compression circuit compresses the YUV image data converted into the block scan data by the buffer memory according to the moving image compression method, and outputs a moving image bit stream.

  The exposure control unit 118 controls the mechanical shutter 113 and the diaphragm 114. The lens driving unit 119 moves the zoom lens 110 and the focus lens 112 along the optical axis to form a subject image on the image sensor 115. The lens driving unit 119 drives the shift lens according to the outputs of the angular velocity sensor and the acceleration sensor, and optically corrects camera shake of the image recording / reproducing device. The release switch 120 for instructing photographing is a double action switch, for example, and includes a first switch (half press) and a second switch (full press).

  The microphone 127 converts the input sound into an audio signal. The A / D converter 128 converts the microphone audio output into a digital audio signal. The audio processing unit 129 performs predetermined audio processing on the digital audio data and outputs an audio bitstream.

  Control of these components is performed by a system control circuit 121 including a CPU and its interface circuit, a DMAC (Direct Memory Access Controller), a bus arbiter, and the like. A program executed by the CPU is stored in the flash memory 122. Here, the program is a program for executing various flowcharts described later in the present embodiment. In the memory unit 123, constants and variables for operation of the system control circuit 121, programs read from the flash memory 122, and the like are expanded. The system control circuit 121 also performs display control by controlling the memory unit 123, the reproduction circuit 150, the monitor 151, and the like.

  The connection terminals 130 and 141 engage the imaging device 100 and a battery 142 that is a power source. The battery BOX 140 causes the imaging device 100 to hold the battery 142. The recording medium 133 records image data. The connection terminals 125 and 132 engage the imaging device 100 and the recording medium 133. The switch 134 detects a write-protect switch for the recording medium. The detection switch 126 detects attachment / detachment of the recording medium 133. The moving image recording switch 180 instructs to record moving images.

  The reproduction circuit 150 converts the image data generated by the image processing unit 117 and stored in the memory unit 123 into a display image, and transfers the image to a monitor 151 that is a display device. The reproduction circuit 150 separates the YUV format image data into a luminance component signal Y and a modulated chrominance component C, and applies LPF to the analog Y signal subjected to D / A conversion. Further, BPF is applied to the analog C signal that has been subjected to D / A conversion, and only the frequency component of the modulated color difference component is extracted. Based on the signal component thus generated and the subcarrier frequency, the signal is converted into a Y signal and an RGB signal and output to the monitor 151. Thus, EVF is realized by sequentially processing and displaying the image data from the image sensor. In the EVF state, AF (automatic focus adjustment) processing, AE (automatic exposure) processing, and AWB (auto white balance) processing are continuously performed, thereby serving as a viewfinder.

  The mode switch 190 switches the operation mode of the system control circuit 121 to any one of a still image recording mode, a moving image recording mode, a reproduction mode, and the like. Modes included in the still image recording mode include an auto shooting mode, an auto scene discrimination mode, a manual mode, various scene modes for shooting settings for each shooting scene, a program AE mode, a custom mode, and the like. The mode switch 190 can be directly switched to one of these modes included in the still image shooting mode. Alternatively, after the mode switch 190 temporarily switches to the still image shooting mode, the mode switch 190 may be used to switch to any of these modes included in the still image shooting mode using another operation member. Similarly, the moving image shooting mode may include a plurality of modes. When the first switch is turned on by half-pressing the release switch 120, a first shutter switch signal SW1 is generated. In response to the first shutter switch signal SW1, shooting preparation operations such as AF processing, AE processing, and AWB processing are started.

  When the second switch is turned on by fully pressing the release switch 120, the second shutter switch signal SW2 is generated. Based on the second shutter switch signal SW2, the system control circuit 121 performs EF (flash pre-emission) processing according to the AE processing result and the strobe setting, and performs main exposure or main exposure with strobe flash. Then, a series of shooting processing operations from reading a signal from the image sensor 115 to writing image data to the recording medium 133 is started. When the moving image recording switch 180 is pressed, AF, AE, and AWB are forcibly performed as if the release switch 120 was fully pressed at once. After that, an image suitable for encoding is generated from the image output from the image sensor 115, encoding is sequentially performed, and an operation of writing moving image data to the recording medium 133 is started.

<Embodiment 1>
FIG. 2 is a timing chart of image sensor drive timing, image sensor output timing, and image synthesis. The image sensor 115 receives a synchronization signal from the SSG in the image processing unit 117 and receives a driving signal generated by the timing generation unit 116 and outputs two types of images at two types of rates. The vertical synchronization signal (VD) from the SSG is output at a rate that is an integral multiple (for example, 8 times) of the recording frame rate, for example, and the timing generator 116 has two types of pixel reset pulses and two types of pixel readout pulses. Is output. A pixel reset pulse and a readout pulse are output with the VD8 period as one period. As a result, the first image is output at the same frame rate (first frame rate) as the moving image recording frame rate. In addition, a pixel reset pulse and a readout pulse are output with the VD1 period as one period. As a result, a plurality of second images are output at a higher rate than the first frame rate, for example, a rate (second frame rate) that is eight times the moving image recording frame rate. In the present embodiment, the output rate of the second image is eight times the output rate of the first image, but the present invention is not limited to this.

  The acquired first image and the plurality of second images are respectively input to the AF evaluation value calculation circuit of the image processing unit 117 and output an evaluation value for each image. The AF evaluation value is evaluated for each of the first image and the second image, and is used for automatic focus adjustment. Details of the automatic focus adjustment will be described later. As a result of the focus adjustment, an image exposed at the focus lens position closest to the focus position is selected from the eight second images corresponding to the output cycle of the first image based on the position of the focus lens in focus ( In FIG. 2, (F) and (N)). These selected images are used for synthesis with the first image.

  FIG. 3 is a diagram illustrating a data flow of image composition. First, a first image (D301) and a plurality of second image groups (D302) are captured from the image sensor 115. From the first image (D301), first, an intermediate YUV image A (D305) is generated through composite image generation processing in which the signal processing circuit of the image processing unit 117 develops with a setting value suitable for composition. Further, a positional deviation reference image (D306) is generated from the first image (D301) through a ratio image generation process that develops with a setting value suitable for image positional deviation detection at the time of composition. Similarly, from the second image group (D302), an intermediate YUV image B group (D303) is generated through a composite image generation process, and a misregistration detection image group (D304) is generated through a ratio image generation process.

  Based on the focus adjustment result from the intermediate YUV image B group (D303) and the misregistration detection image group (D304), an image exposed at the focus lens position that is closest to the focal position through the image selection process is displayed as an intermediate YUV image B (D307). ) And a displacement detection image (D308).

  Based on the positional deviation reference image (D306) and the positional deviation detection image (D308), the luminance synthesis ratio (D309) of each image for each area is calculated from the luminance average for each area obtained by dividing each image into a plurality of areas. In addition, a moving average filter process is performed for each of the divided areas, and the motion synthesis ratio (D310) of each image for each area is calculated from the average of the absolute differences of the luminance and color difference for each area.

  The maximum value of the luminance composition ratio and the motion composition ratio is set as a composition ratio in the composition circuit of the image processing unit 117, the intermediate YUV image A (D305) and the intermediate YUV image B (D307) are input, and composition is performed through weighted addition processing. An image is generated (D311).

  FIG. 4 is a flowchart showing the focus adjustment process. A program corresponding to this flowchart is included in, for example, the flash memory 122, loaded into the memory unit 123, and executed by the system control circuit 121. The automatic focus adjustment is a hill-climbing method that continuously searches for a lens position where the high-frequency component of the luminance signal obtained from the image sensor 115 is maximum at three focus lens positions. Specifically, the signal used for the evaluation is obtained by filtering the image data from the image sensor 115 to extract the luminance component in the high frequency band, and integrating the horizontal peak in the set region in the vertical direction in the frame. This is the focus evaluation value obtained in this way.

  First, the AF evaluation value calculation circuit is activated, and the first image obtained from the image sensor 115 is input to the AF evaluation value calculation circuit of the image processing unit 117 (S400). It is determined whether or not the focusing rate is low (S401). If the focusing rate is low, the completion of exposure of the first image at the current focus lens position is awaited (S402). When the exposure of the first image is completed, the focus lens is moved to the closest side from the initial position, and the second image is input to the AF evaluation value calculation circuit of the image processing unit 117 (S403). When the movement of the focus lens is completed (S404) and the integration of the evaluation value of the second image at the initial position is completed (S405), the lens position and the evaluation value are stored in association with each other (S406).

  Next, when the exposure of the second image while moving the focus lens to the close side is completed (S407), the exposure of the second image at the close focus lens position is awaited (S408). Thereafter, the focus lens is moved from the initial position to the infinite side (S409). When the movement of the focus lens is completed (S410) and the integration of the evaluation value at the closest position is completed (S411), the lens position and the evaluation value are stored in association with each other (S412). Next, when the exposure of the second image while moving the focus lens to the infinite side is completed (S413), the exposure of the second image at the infinite side focus lens position is waited for (S414). Waiting for completion of integration of the evaluation value at the infinite side position (S415), the lens position is associated with the evaluation value and stored (S416). The focal position is determined from the three evaluation values (S417), and the lens is moved to the focal position (S418). If the EVF is not finished (S419), the process returns to the focus determination in S401 and is repeated.

  FIG. 5 is a flowchart showing a specific example of the process of determining the focal position from the three evaluation values in S417. First, it is determined whether or not the initial position evaluation value (current position before focus detection; hereinafter, initial position Cur.) Is equal to or greater than a predetermined value X of the close position evaluation value (hereinafter, close position Mod) ( S420). That is, it is determined whether or not Cur-Mod ≧ X. When Cur-Mod ≧ X, the initial position Cur is compared with the evaluation value of the infinite position (hereinafter, infinite position Inf) (S421). If the initial position Cur is large, the initial position Cur is set as the focal position (S422). Otherwise, the infinite position Inf is set as the focal position (S423).

  If Cur-Mod ≧ X is not satisfied in S420, it is determined whether or not the closest position Mod is equal to or larger than a predetermined value X of the initial position Cur (S424). That is, it is determined whether Mod-Cur ≧ X. If Mod-Cur ≧ X, it is determined whether the initial position Cur is equal to or greater than the infinite position Inf (S425). If the initial position Cur is greater than or equal to the infinite position Inf, the closest position Mod is set as the focal position (S426). If the initial position Cur is not equal to or greater than the infinite position Inf, it is determined whether the infinite position Inf is equal to or greater than the predetermined value X of the closest position Mod (S427). If the infinite position Inf is greater than or equal to the predetermined value X of the closest position Mod, the infinite position Inf is set as the focal position (S428), otherwise the focal position is determined to be indefinite (S429).

  Further, if it is determined in S424 that Mod-Cur ≧ X is not satisfied, it is determined whether the infinite position Inf is equal to or greater than the predetermined value X of the initial position Cur (S430). That is, it is determined whether Inf-Cur ≧ X. If Inf-Cur ≧ X, it is determined whether the initial position Cur is equal to or greater than the closest position Mod (S431). When the initial position Cur is equal to or greater than the closest position Mod, the infinite position Inf is set as the focal position (S432). Otherwise, it is determined whether the closest position Mod is equal to or greater than a predetermined value X of the infinite position Inf (S433). When the close position Mod is equal to or greater than the predetermined value X of the infinite position Inf, the close position Mod is set as the focus position (S434), and otherwise, the focus position is set undefined (S435). Even when it is determined in S430 that Inf-Cur ≧ X is not satisfied, the focus position is made indefinite in S435. Note that the focal position determination method is not limited to the method of FIG.

  The process of extracting the intermediate YUV image B (D307) and the misregistration detection image (D308) from the intermediate YUV image B group (D303) and the misregistration detection image group (D304) in FIG. 3 is determined by the process of FIG. An image generated from the image exposed at the focal position is selected. Also, in FIG. 5, when it is determined that the focal position is indefinite, an evaluation obtained by inputting the first image to the AF evaluation value calculation circuit among the initial position Cur, the closest position Mod, and the infinite position Inf. The image closest to the value is selected.

  As described above, when performing the focus adjustment while performing moving image shooting and recording, it is possible to accurately perform alignment at the time of synthesis by selectively synthesizing images with a high focusing rate. At the same time, an image with a wide dynamic range can be obtained without losing a sense of resolution.

<Embodiment 2>
In the first embodiment, the image exposed at the focus lens position closest to the focus position is selected from the eight second images corresponding to the output cycle of the first image. However, the following configuration may be used. That is, a plurality of images are selected in descending order of the focusing rate from eight, and a third image is generated by synthesizing the selected images ((F) and (G) and (N) in FIG. 2). And (O)). Thereafter, the third image is combined with the first image. By synthesizing a plurality of second images, it contributes to the reduction of the noise level of the second image when synthesizing with the first image, and an image with a higher image quality and an expanded dynamic range can be obtained.

  In FIG. 4 (S418), the exposure starts after the focus lens is moved to the focal position (timing t4 and t5 in FIG. 2), and after the focus lens is moved, the exposure of the first first image is completed (FIG. 2). The second image output until t2 and t3 are selected as the images to be combined.

  FIG. 6 is a data flow for generating the intermediate YUV image B (D307) and the misregistration detection image (D308) by combining the second image with multiple image selections in FIG. The difference from FIG. 3 lies in image selection processing for selecting two images that satisfy the above conditions from the intermediate YUV image B group (D303) and the misregistration detection image group (D304). An intermediate YUV image B1 (D316) and an intermediate YUV image B2 (D317) are selected through image selection processing from the intermediate YUV image B group (D303). Further, the position shift detection image 1 (D312) and the position shift detection image 2 (D313) are selected through image selection processing from the position shift detection image group (D304).

  Based on the displacement detection image 1 (D312) and the displacement detection image 2 (D313), the luminance synthesis ratio (D314) of each image for each region is calculated from the luminance average for each region obtained by dividing each image into a plurality of regions. Further, a moving average filter process is performed for each of the divided areas, and a motion synthesis ratio (D315) of each image for each area is calculated from the average of the absolute differences of the luminance and color difference for each area.

  The maximum value of the luminance composition ratio (D314) and the motion composition ratio (D315) is set in the composition circuit of the image processing unit 117 as the composition ratio. Then, the misalignment detection image 1 (D312) and the misalignment detection image 2 (D313) are input and a misalignment detection image (D308) synthesized through a weighted addition process is generated. Similarly, the maximum value of the luminance composition ratio (D314) and the motion composition ratio (D315) is set as the composition ratio in the composition circuit of the image processing unit 117. Then, the intermediate YUV image B1 (D316) and the intermediate YUV image B2 (D317) are input and the intermediate YUV image B (D307) synthesized through the weighted addition process is generated. The subsequent data flow is the same as the data flow described in FIG.

  As described above, when performing the focus adjustment while performing moving image shooting and recording, a plurality of images with a high focusing rate are selected and combined. As a result, the alignment at the time of synthesis can be performed accurately, and at the same time, an image with a wide dynamic range with a high S / N ratio can be obtained without losing a sense of resolution.

<Embodiment 3>
FIG. 7 is a timing chart of image sensor drive timing, image sensor output timing, and image synthesis when the focus lens is not moved during the exposure period by automatic focus adjustment. In the first and second embodiments, it is assumed that the second image has a significant difference in focus due to the automatic focus adjustment. On the other hand, here, a period in which focus adjustment is not performed, that is, a time when it is not determined that the focusing rate is low in FIG. 4 (S401) is assumed. In this case, a predetermined number of second images near the center of exposure of the first image are selected, and the selected second images are combined to generate a fourth image (in FIG. 5, (A) (B) and (H) and (I)). Then, it is combined with the first image. Thereby, the deviation in real time between the images to be combined is reduced. Then, by synthesizing a plurality of second images having a short exposure time and then synthesizing with the first image, an image with a small displacement and a reduced noise level and a higher dynamic range can be obtained. .

  The data flow is the same as in FIG. 6, and the image selection processing is changed to select the second image closest to the exposure center of gravity of the first image as described above. The exposure center of gravity can be obtained for each image as follows.

  In FIG. 7, the recording frame rate is F, the first image readout time is R1, and the second image readout time is R2. Further, the first read timing of the first image is V1, the second read timing of the first image is V2, and the third read timing of the first image is V3. Further, the second exposure time of the first image is E1, the third exposure time of the first image is E2, and the exposure time of the second image is E3.

  The first image exposed during the period from V1 to V2 starts to be read from V2, and the center of exposure is obtained by the following equation (1).

Exposure gravity center = V2-E1 / 2 + R1 / 2 (1)
The exposure center of gravity of the second image exposed during the period from V1 to V2 is obtained by the following equation (2).

Exposure center of gravity 2 = V2− (F / m) × (mn) −E2 / 2 + R2 / 2 (2)
In this embodiment, m = 8, n = 8 in the case of the second image in the VD period immediately before V2, and n = 1 in the case of the second image in the VD period immediately after V1.
Values other than V1, V2, V3,... Are known, and the exposure centroid of the first image and the exposure centroid of the second image are obtained at the timing when the first image starts to be read. In parallel with the reading of the first image, the exposure centroid is obtained by the equations (1) and (2), and the second image at the exposure centroid time closest to the time of the first exposure centroid is second. The second image with the closest exposure centroid time is selected.

  In V2 of FIG. 7, t6 is the exposure center-of-gravity time of the first image, and the closest second image (H) and the second closest image (I) are selected.

  As described above, even when moving image recording and focus adjustment is not performed, a plurality of AF evaluation images close to the center of exposure of the moving image recording image are selected and combined. As a result, the alignment at the time of synthesis can be performed accurately, and at the same time, an image with a wide dynamic range with a high S / N ratio can be obtained without losing a sense of resolution.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed. In this case, the program and the storage medium storing the program constitute the present invention.

Claims (11)

An image sensor that captures a subject image via a photographic lens;
A focus lens included in the photographing lens and movable in an optical axis direction;
From the imaging device, an acquisition unit to obtain the first image at a first frame rate, to obtain a plurality of second images at the first higher than the frame rate second frame rate,
Focus adjusting means for adjusting the focus of the subject image by controlling the position of the focus lens based on the high-frequency components of the first image and the plurality of second images;
A synthesizing unit that synthesizes a plurality of images obtained from the image sensor and generates an image having an expanded dynamic range;
Have
The image-capturing apparatus, wherein the synthesizing unit selects an image with a high focusing rate from the plurality of second images, and synthesizes the selected image with the first image.   The imaging apparatus according to claim 1, wherein the second frame rate is an integral multiple of the first frame rate. The focus adjustment unit performs the focus adjustment based on an evaluation value obtained by integrating high-frequency components of the first image and the plurality of second images,
When it is determined that the plurality of second images have no significant difference in focus, the synthesizing unit selects an image closest to the evaluation value of the first image from the plurality of second images. The imaging apparatus according to claim 1, wherein the selected image is combined with the first image.   The synthesizing unit selects a predetermined number of images from the plurality of second images in descending order of the focusing rate, synthesizes the selected images to generate a third image, and the third image is The image pickup apparatus according to claim 1, wherein the image pickup apparatus is combined with the first image.   The synthesizing unit selects a predetermined number of images in the vicinity of the exposure center of gravity of the first image from the plurality of second images during a period when the focus adjustment is not performed, and synthesizes the selected images to obtain a fourth 4. The imaging apparatus according to claim 1, wherein the fourth image is generated and the fourth image is combined with the first image. 5. A method for controlling an imaging apparatus, comprising: an imaging element that captures a subject image via a photographing lens; and a focus lens that is included in the photographing lens and is movable in an optical axis direction.
Acquisition step acquiring unit, from the image pickup device, to obtain to obtain the first image at a first frame rate, a plurality of second images at the first higher than the frame rate second frame rate When,
A focus adjustment step in which focus adjustment means performs focus adjustment on the subject image by controlling a position of the focus lens based on high-frequency components of the first image and the plurality of second images;
Combining means comprises a combining step of generating an image whose dynamic range is expanded by combining a plurality of images obtained from the imaging element,
Have
In the synthesizing step, the synthesizing unit selects an image with a high focusing rate from the plurality of second images, and synthesizes the selected image with the first image. Control method. The control method according to claim 6, wherein the second frame rate is an integral multiple of the first frame rate. In the focus adjustment step, the focus adjustment means performs the focus adjustment based on an evaluation value obtained by integrating high-frequency components of the first image and the plurality of second images,
In the synthesizing step, when it is determined that the plurality of second images do not have a significant difference in focus, the synthesizing unit has the highest evaluation value of the first image among the plurality of second images. The control method according to claim 6 or 7, wherein a close image is selected, and the selected image is combined with the first image. In the synthesizing step, the synthesizing unit selects a predetermined number of images from the plurality of second images in descending order of the focusing rate, and synthesizes the selected images to generate a third image, The control method according to claim 6, wherein the third image is synthesized with the first image. In the synthesizing step, the synthesizing unit selects a predetermined number of images in the vicinity of the exposure centroid of the first image from the plurality of second images in a period in which the focus adjustment is not performed, and selects the selected images. The control method according to claim 6, wherein a fourth image is generated by synthesis, and the fourth image is synthesized with the first image. The program for making a computer perform each step of the control method of any one of Claims 6 thru | or 10.

Images