JP2015159353A - Imaging apparatus and imaging method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus and an imaging method which, even in occurrence of disturbance during imaging, prevent picture quality from being deteriorated by the disturbance and minimize exposure interruption.SOLUTION: Image data exposed in an exposure time T is synthesized repetitively (S19), and disturbance occurring during imaging is detected (S11). In synthesizing the image data, an exposure time is divided (S7) into time intervals of T/M (M is 2 or larger). In a case where disturbance is detected during imaging, the image data exposed in the exposure time T is generated by excluding image data of a time interval containing the disturbance therein out of the divided time intervals (S11: Yes).

Description

  In an imaging apparatus such as a digital still camera that generates a captured image by combining a plurality of image data, the image quality of a still image or a moving image is reduced due to an unexpected disturbance such as a camera shake or a change in subject brightness. The present invention relates to an imaging apparatus and an imaging method capable of preventing deterioration.

  In recent years, high performance and multi-functionality of digital cameras have made it possible to take pictures for a long time more easily than before, and an increasing number of photographers have enjoyed taking pictures for a long time. In long-time shooting, it is possible to shoot fantastic photos and videos such as astronomical shooting, which is a very dark subject, fireworks shooting, and sun shooting using an ND filter.

  In addition, bulb exposures with exposure times ranging from seconds to tens of minutes, interval composite shooting that combines multiple images taken continuously at regular intervals into a single photo, and frame-by-frame feeding of many consecutively shot photos Special shooting methods such as time-lapse shooting for moving images have been developed.

  In long-time shooting, it is common to shoot with a camera fixed on a tripod so that it does not shake. However, when shooting in a very dark environment, even if the photographer is careful, the tripod may be bumped and shaken, other photographers may flash during shooting, or the car may be passing by. In many cases, the light of the headlight is reflected or a disturbance unexpected by the photographer occurs.

  When such disturbances occur, still images and moving images taken over a long period of time are spoiled due to blurring or overexposure of images due to high-luminance subjects. If it is a short time shooting, it is easy to redo the shooting, but if it is a long time, it takes a long time to reshoot.

  In a digital camera that synthesizes a long-time photograph with a separately exposed image against a disturbance during such long-time shooting, a method has been proposed in which the blur is corrected before and after the occurrence of camera shake and is combined ( Patent Document 1). Also, a method has been proposed in which when the luminance of a subject changes during shooting, the image processing is performed to adjust the brightness corresponding to the change in luminance (see Patent Document 2).

JP 2013-168699 A JP 2013-172372 A

  According to the methods described in Patent Documents 1 and 2, it is possible to reduce image quality deterioration due to camera shake or luminance change during long-time shooting. However, the intention of the photographer who shoots for a long time is the desire to continuously capture the light trajectory, such as the trajectory caused by the diurnal motion of stars, the trajectory of fireworks, and the flight trajectory of fireflies. The methods described in Patent Documents 1 and 2 cannot meet these demands.

  In other words, in the method of Patent Document 1, the captured image when the camera shake occurs among the continuously captured images cannot be used for composition because the image is blurred. In the method of Patent Document 2, when the luminance change is smooth (for example, the luminance change in the sky from night to sunrise), it is possible to continuously capture the light trajectory. However, when another photographer emits a flash or when a strong light such as a car headlight is reflected, the brightness is very high and the image output is saturated, so that the brightness adjustment by image processing cannot cope. In such a case, it is necessary to exclude and synthesize an image in which disturbance has occurred among images that are divided and exposed at a predetermined timing, and the locus of light reflected in the image is interrupted.

  The present invention has been made in view of such circumstances, and provides an imaging apparatus and an imaging method that prevent image quality degradation due to disturbance even when disturbance occurs during shooting and minimize exposure interruptions. With the goal.

  In order to achieve the above object, an image pickup apparatus according to a first invention includes an image pickup device, an image data combining unit that repeatedly combines image data exposed at an exposure time T, and disturbance detection that detects a disturbance that occurs during shooting. And the image data composition unit divides the exposure time into T / M (M is 2 or more) time intervals, and when a disturbance is detected by the disturbance detection unit during photographing, The image data of the exposure time T is synthesized by excluding the image data of the time interval in which the disturbance is mixed among the divided time intervals.

  An image pickup apparatus according to a second invention is the image pickup device according to the first invention, wherein the image pickup device is a destructive readout type, and the image data synthesis unit is image data obtained by exposure with the exposure time T / M. Is added.

  According to a third invention, in the first invention, the imaging device is of a non-destructive readout type, and control for controlling reset of charge accumulation and start of charge accumulation of pixels constituting the image sensing device. When the disturbance detection unit detects a disturbance, the control unit resets the charge accumulation and starts the charge accumulation immediately after the disturbance disappears, and the image data synthesis unit The image data generated immediately before the disturbance occurs and the image data for which exposure has been started immediately after the disturbance disappears are combined.

  According to a fourth aspect of the present invention, in the first aspect, the image pickup device is a non-destructive readout type, and control for controlling reset of charge accumulation and start of charge accumulation of pixels constituting the image pickup device. A controller, and the controller performs the reset and start of charge accumulation every exposure time T.

In the imaging device according to a fifth aspect based on the first aspect, the disturbance detection unit detects camera shake during exposure.
In the imaging device according to a sixth aspect based on the first aspect, the disturbance detection unit analyzes a luminance value of the image data subjected to the divided exposure and detects a change in luminance.
In the imaging device according to a seventh aspect based on the first aspect, the image data synthesis unit performs at least one of comparatively bright synthesis, comparative dark synthesis, addition average synthesis, and time-lapse moving image generation.

  An imaging method according to an eighth aspect of the present invention includes an image data synthesis step for repeatedly synthesizing image data exposed at an exposure time T, and a disturbance detection step for detecting a disturbance that occurs during photographing. The synthesizing step divides the exposure time into T / M (M is 2 or more) time intervals, and when a disturbance is detected in the disturbance detection step during shooting, the disturbance is detected in the divided time intervals. Image data of the exposure time T is generated by excluding the mixed time interval image data.

  According to the present invention, it is possible to provide an image pickup apparatus and an image pickup method that prevent image quality deterioration due to a disturbance even when a disturbance occurs during shooting and minimize exposure interruptions.

It is a block diagram which shows mainly an electrical structure of the camera which concerns on 1st Embodiment of this invention. It is a flowchart which shows the operation | movement in the long-time imaging | photography mode of the camera which concerns on 1st Embodiment of this invention. It is a flowchart which shows the operation | movement of the disturbance detection of the camera which concerns on 1st Embodiment of this invention. It is a flowchart which shows the operation | movement of the image composition and imaging progress display of the camera which concerns on 1st Embodiment of this invention. 3 is a time chart showing a long-exposure shooting sequence in the camera according to the first embodiment of the present invention. It is a time chart which shows the imaging | photography sequence of long time exposure (addition synthetic | combination mode) in the camera which concerns on 2nd Embodiment of this invention. 6 is a time chart showing a shooting sequence (no disturbance) of long-time exposure (comparison / synthesis mode, time-lapse moving image generation mode) in the camera according to the second embodiment of the present invention. 6 is a time chart showing a shooting sequence (with disturbance) of long-time exposure (comparison / synthesis mode, time-lapse moving image generation mode) in the camera according to the second embodiment of the present invention. It is a block diagram which shows schematic structure of the nondestructive readout type image sensor used in the camera which concerns on 2nd Embodiment of this invention. It is a timing chart which shows the operation | movement of the nondestructive readout type used in the camera which concerns on 2nd Embodiment of this invention. It is a time chart which shows the imaging sequence of the conventional long exposure, (a) shows the normal imaging | photography time which does not generate | occur | produce a disturbance, (b) shows the case where a disturbance generate | occur | produces.

  Hereinafter, a preferred embodiment will be described using a camera to which the present invention is applied. The camera according to a preferred embodiment of the present invention is a digital camera, and generally, recording is started in response to an operation of a release button, and a plurality of image data read from the image sensor is combined to record corresponding to long-time shooting. Synthesizes still images or moving images for recording in external memory. When a disturbance is detected during shooting, composition is performed except for an image exposed during the period in which the disturbance occurs.

  FIG. 1 is a block diagram mainly showing an electrical configuration of the camera according to the first embodiment of the present invention. This camera includes an imaging unit 1, an image processing unit 10, a disturbance detection unit 15, a system control unit 20, an internal memory 33, an external memory 36, a display unit 37, an input IF (interface) 38, and a bus 31 for connecting them. Have. In the present embodiment, the lens 2 is configured integrally with the camera body, but may of course be an interchangeable lens.

  The imaging unit 1 includes a lens 2, a mechanical shutter 3, and an image sensor 4. The lens 2 forms an optical image of the subject on the image sensor 4. This lens 2 is provided with an aperture for determining an aperture value for adjusting the exposure amount. The mechanical shutter 3 controls the shutter speed by performing exposure and shading to the image sensor 4 by opening and closing operations.

  The image sensor 4 includes an image sensor such as a CMOS image sensor or a CCD image sensor, converts an optical image of a subject formed by the lens 2 into an electrical signal for each pixel, and converts the image data into the image processing unit 10 and the bus. To 31. In the present embodiment, the image sensor 4 is a destructive readout type. The bus 31 is a signal line for transmitting and receiving signals between the blocks.

  The image processing unit 10 performs image processing on the image data output from the image sensor 4. The image processing unit 10 includes an image composition unit 11 and a development processing unit 13, and the image composition unit 11 includes an addition composition unit 11a, a comparison composition unit 11b, an addition average composition unit 11c, and a moving image generation unit 11d.

  The image composition unit 11 functions as an image data composition unit that repeatedly composes image data exposed at the exposure time T. Further, the image composition unit 11 divides the exposure time into T / M (M is 2 or more) time intervals, and when a disturbance is detected by a disturbance detection unit described later during shooting, The image data of the exposure time T is synthesized by excluding the image data of the time interval in which the disturbance is mixed. The image composition unit 11 adds image data obtained by exposure with an exposure time T / M (see an addition composition unit 11a described later). In addition, the image composition unit 11 performs at least one of comparatively bright composition, comparative dark composition, addition average composition, and time-lapse animation generation.

  The adding and synthesizing unit 11a generates a synthesized image by adding outputs for each corresponding pixel among the plurality of image data read from the image sensor 4. The comparison / synthesis unit 11b compares outputs for each corresponding pixel among a plurality of image data read from the image sensor 4 and uses the pixel output of the larger output as the pixel data of the synthesized image, or A synthesized image is generated by comparative dark synthesis using the pixel output with the smaller output as the pixel data of the synthesized image.

  The addition average combining unit 11c generates a combined image by adding and averaging the outputs of the corresponding pixels among the plurality of image data read from the image sensor 4. The moving image generation unit 11d generates moving image data by connecting image data of a plurality of frames read continuously in time series. In addition, the moving image generating unit 11d generates moving image data based on the image data acquired at the time of time-lapse plus moving image shooting.

  The development processing unit 13 performs development processing such as demosaicing, white balance adjustment, gamma correction, and image compression on the RAW image data generated by the image composition unit 11.

  The disturbance detection unit 15 includes a camera shake detection unit 16 and an exposure abnormality detection unit 17. The disturbance detection unit 15 functions as a disturbance detection unit that detects a disturbance generated during shooting.

  The camera shake detection unit 16 may detect camera shake during exposure, and may detect whether camera shake has occurred based on the output of a gyro sensor mounted on a general digital camera. It is also possible to detect whether camera shake has occurred based on the presence or absence of blurring of the subject image by comparing continuously read images. In addition, the camera shake detection unit 16 feeds back the subject image displacement information due to camera shake to the image processing unit 10, and the image processing unit 10 corrects and synthesizes the subject image displacement due to camera shake before and after the camera shake occurs. To do.

  The exposure abnormality detection unit 17 analyzes the luminance value of the separately exposed image data and detects a change in luminance. That is, the brightness value is calculated based on the output value of each piece of image data subjected to the divided exposure, and the brightness values of the image data are compared to detect the presence or absence of a sudden change in the brightness value. For example, if there is a sudden change in the luminance value due to strobe light emission by another photographer or a headlight of a passing car, it is detected.

  In addition to the imaging unit 1, the image processing unit 10, and the disturbance detection unit 15, an internal memory 33, an external memory 36, a display unit 37, an input IF 38, and the system control unit 20 are connected to the bus 31.

  The internal memory 33 temporarily stores various setting information necessary for camera operation and image data in the middle of image processing. The internal memory 33 is configured by a rewritable nonvolatile memory such as a flash memory or SDRAM.

  The external memory 36 is a non-volatile storage medium that can be loaded into the camera body or fixed inside, and is, for example, an SD card or a CF card. The external memory records the image data developed by the development processing unit, and at the time of reproduction, the recorded image data can be read out and output to the outside of the camera.

  The display unit 37 includes a rear display unit such as a TFT (Thin Film Transistor) liquid crystal or an organic EL, and an EVF (electronic viewfinder), and displays an image developed by the development processing unit 13.

  The input IF 38 includes an operation member such as a release button, a touch panel for inputting a touch operation on the rear display unit, and the like, and instructs various shooting operations such as mode setting and release based on a user operation. Through the input IF 38, the photographer can set, for example, a long-time shooting mode such as a bulb shooting mode, an interval shooting mode, and a time-lapse moving image mode.

  The system control unit 20 includes a CPU (Central Processing Unit) and performs overall control by controlling each unit of the camera according to a program stored in the internal memory 33. Further, the system control unit 20 performs overall control of the above-described imaging unit 1, image processing unit 10, disturbance detection unit 15, and the like.

  Next, the operation of the camera according to the first embodiment of the present invention when the long-time shooting mode is set will be described with reference to the flowcharts shown in FIGS. This flowchart is executed by the system control unit 20 controlling each unit in accordance with a program stored in the internal memory 33.

  In this flowchart, it is assumed that the photographer has set the camera shooting mode to the long-time shooting mode by the input IF 38. Examples of the long-time shooting mode include a bulb shooting mode, an interval shooting mode, and a time-lapse moving image mode.

  First, the entire sequence will be described with reference to FIG. When the long shooting mode is entered, live view display is first performed (S1). Here, the image signal from the image sensor 4 is subjected to image processing for live view display in the image processing unit 13, and a live view image is displayed in the display unit 37. In the live view display, the live view image is updated every time an image signal is read from the image sensor 4 at a predetermined time interval. When the live view image is displayed on the display unit 37, the photographer sets the composition, focus position, and shooting conditions according to the subject while checking the live view image.

  Once live view display is performed, an exposure time T is set (S3). When the bulb shooting mode is set as the shooting mode, an image read out by the photographer at the desired timing is added at any time and displayed on the display unit 37 as the exposure progress so that the exposure progress can be confirmed. To do. In this case, the exposure time T is set by the input IF 37 in accordance with the photographer's preferred timing. For example, when shooting for several minutes, if it is desired to check the progress of exposure every 10 seconds, the exposure time T is set to 10 seconds.

  In addition, when the interval shooting mode is set to combine the interval shot images into one still image by comparison synthesis or addition average synthesis, or the time-lapse plus shooting mode to generate time-lapse movie (frame-by-frame movie) is set If it is, the images continuously exposed and read out with the exposure time T set in this step are synthesized. When combining still images, the exposure of the background portion of the composite image (the portion where the brightness did not change during shooting) is the same exposure as each image read at the exposure time T, and the exposure time for time-lapse movies Since the image read out at T corresponds to the exposure of one frame of the moving image, the photographer may manually set the time T when the brightness of the live view image is confirmed and the optimum exposure can be obtained. T may be automatically set to an exposure time for obtaining an appropriate exposure using an AE (Auto Exposure) function.

  Once the exposure time T is set, it is next determined whether or not it is a release (S5). If the photographer determines that there is a photo opportunity, the user operates the release button. Here, it is determined whether or not the release button has been operated. If the result of this determination is that release has not been carried out, processing returns to step S1, and operations such as live view display are executed. On the other hand, if the result of determination in step S5 is release, exposure is started (S6). Here, imaging is started by the image sensor 4, and the optical image of the subject is converted into an electrical signal for each pixel. Charge is stored in the capacitor.

  When exposure is started in step S6, it is next determined whether or not time has elapsed by T / M (S7). Since the timing operation is started by the timer simultaneously with the start of exposure, the determination is made based on the timing time of this timer. M represents the number of images used when generating a composite image (for example, 4 in the example shown in FIG. 5), and is preferably an integer of 1 or more. Furthermore, the larger M is, the shorter the T / M time is, the exposure time of frames to be excluded from synthesis, which will be described later, is shortened, and the number of exposures is shortened. For this reason, when M is large, it is preferable that exposure loss is shortened.

  However, since the imager 4 generally generates a slight read noise when reading an image, the read noise of the composite image is increased by the number of read times, and the image quality deteriorates. Therefore, it is not preferable to increase M without limit. Still, for example, if M = 4 is fixed and shooting is performed at T = 60 seconds, the method described in the above-described patent document may result in a loss of exposure of 60 seconds when a disturbance occurs. In the embodiment, the exposure can be shortened to 60 · 4 = 15 seconds, and there is a sufficient merit.

  Further, it is more preferable to set M to such an extent that exposure omission is not visually recognized in consideration of the image plane speed of the subject (moving speed of the image projected on the image sensor). For example, when shooting the locus of a star, when shooting using a 20-megapixel image sensor 4 with a wide-angle lens (for example, a focal length of about 20 mm (35 mm film equivalent)), the star or the like has an exposure time of 20 seconds. Since it moves about pixels, it is preferable to set T and M so that T / M is about 2 seconds so that the exposure of the star locus is about 1 pixel or less. On the other hand, when shooting fireworks and the like, the image speed of the trajectory is faster than that of stars, but since the shooting time is shorter from several seconds to several tens of seconds than stars, for example, T / M is set to 0.1 seconds or less. Even if it is shortened, the total number of divisions does not increase compared to shooting star trails. Some recent digital cameras have a scene recognition function. In this case, T / M may be automatically set according to the subject by recognizing a subject such as fireworks or stars.

  As a result of the determination in step S7, when T / M time has elapsed, image reading is performed (S9). Here, an image signal is read from the image sensor 4. When the image reading is completed, since the sensor is a destructive reading type sensor, the signal charge storage capacitor (FD) is reset, and exposure of the next frame is started. In the case of the second embodiment to be described later, since it is a non-destructive readout type sensor, the signal charge storage capacitor (FD, see FD 55 in FIG. 9) is not normally reset.

  In addition, a frame counter that counts the number of frames of the image read in step S9 is prepared. The number of frames is defined as N (an integer equal to or greater than 1), and the first frame of imaging start is defined as N = 1. Frames in which disturbance is detected are not counted, and the number of frames in which disturbance is not detected is counted.

  Once image reading has been performed, it is next determined whether or not a disturbance has been detected (S11). Here, it is determined whether the camera shake detection unit 16 of the disturbance detection unit 15 has detected camera shake, and whether the exposure abnormality detection unit 17 has detected an abnormality in luminance or the like. As described above, when image reading is performed, exposure of the next frame starts. Whether or not a disturbance has occurred during N frame exposure from the disturbance detection unit 15 before proceeding to processing of N frame read image data. Receive the information. Details of the disturbance detection will be described later with reference to FIG. When a disturbance is detected, the Nth frame image is not used for synthesis.

  If the result of determination in step S11 is that no disturbance has been detected, then frame counter N count up + 1 is performed (S13). Here, the frame counter is incremented by +1. Subsequently, based on the count value of the frame counter, it is next determined whether or not N = 1 frame (S15).

  As a result of the determination in step S15, if N = 1 frame, it is recorded in the internal memory 33 without being synthesized (S21).

  On the other hand, if the result of determination in step S15 is that after the second frame, camera shake correction is performed (S17). Here, the image processing unit 10 corrects the image data read in step S9 based on the camera shake amount detected by the disturbance detection unit 15. Since the amount of camera shake is detected at the time of disturbance detection in step S11, image blurring occurs in the synthesized image by electronically correcting the subject image shift due to camera shake between the accumulated synthesized image and the read image. You can avoid it. Further, exposure correction is performed on the image data read in step S9.

  When camera shake correction is performed in step S17, image composition and shooting progress display are performed (S19). Here, image synthesis is performed using the image data read out in step S9, and the synthesized image is displayed on the display unit 37. Thereby, even in the long-time shooting mode such as bulb shooting, the shooting progress is sequentially displayed until the shooting is completed. Detailed operation of the image composition / shooting progress display will be described later with reference to FIG.

  If image composition is performed in step S19 or if it is stored in the internal memory 33 in step S21, it is next determined whether or not it is a release (end of exposure) (S23). When shooting for a long time such as bulb shooting, the photographer presses the release button again to instruct the end of shooting, so determination is made based on this instruction. If the result of this determination is that the exposure has not ended, the process returns to step S7 to continue long-time shooting such as bulb shooting.

  On the other hand, if the result of determination in step S23 is that exposure is complete, shooting is terminated (S25), the final image is displayed, and stored in external memory (S27). Here, when a still image is synthesized in the addition synthesis mode or the comparison synthesis mode by the image processing unit 10, a final display of the generated synthesized image is performed on the display unit 37 for a predetermined time, and this synthesized image is stored in the external memory 36. To record. Note that the display of the composite image may be omitted. When the data is recorded in the external memory, this flow is finished.

  Next, the disturbance detection in step S11 will be described with reference to FIG. In the disturbance detection flow, first, a camera shake signal is detected (S31). Here, the camera shake detection unit 16 determines whether camera shake or the like has occurred in the camera based on a sensor output from a gyroscope or the like.

  If the result of determination in step S31 is that a camera shake signal has been detected, camera deviation amount feedback is performed (S33). Here, the camera shift amount detected by the camera shake detection unit 16 is output to the system control unit 20. The system control unit 20 corrects the shake by moving the image sensor 4 within the plane of the optical axis based on the detected deviation amount by a mechanism (not shown), for example. If camera deviation amount feedback is performed in step S33, then “disturbance detection” is set (S35).

  On the other hand, if the camera shake signal is not detected as a result of the determination in step S31, the process proceeds to an exposure detection operation in the exposure abnormality detection unit 17 in step S41 and subsequent steps. First, the output of an N frame image is calculated (S41). The calculation of the image output in this step may be the average of all areas of the image data, or the image data may be divided into a plurality of vertical and horizontal blocks, and the average output of each block may be calculated. The latter method can be detected even in the case of local exposure abnormality such as reflection of a headlight.

  Once the output of the N frame image is calculated, it is next determined whether or not N = 1 frame (S43). As a result of the determination, in the case of the first frame, it cannot be determined that the exposure is abnormal, and in the second and subsequent frames, the process proceeds to step S47 in order to determine whether there is an exposure abnormality based on the exposure of this image. Treat as not occurring.

  If the result of determination in step S43 is not the first frame of image data, that is, if there are two or more frames, | (N frame output) − (N−1 frame output) | is calculated, and a predetermined threshold level is calculated. It is determined whether it is greater than TH (S45). Here, since there are two or more frames, the absolute value of the difference between the N frame output and the N-1 frame output is calculated, and it is determined whether the value is greater than a predetermined threshold level (TH).

  As a result of the determination in step S45, if the absolute value of the output difference is larger than the threshold level, the process proceeds to step S35, and disturbance detection is fed back to the system control unit 20 (disturbance detection is set). If the average value is calculated by dividing the image data of each frame into multiple vertical and horizontal blocks, the absolute value of the difference between the output average values of the blocks at the same image position is calculated, and even with one block If it is greater than the level, it is determined that a disturbance has been detected. The threshold level (TH) detects an abnormal exposure change, and does not detect a slight exposure change (for example, a brightness change from evening to night or a sky change from night to dawn). Is preferred. It is set to a level at which it is determined that the exposure change for each frame is too large, such as half or ¼ of the saturation signal amount of the digital pixel output.

  If the result of determination in step S45 is that the absolute value of the difference between the N frame output and the N-1 frame output is less than or equal to the threshold level (TH), no disturbance is set (S47). Here, the fact that no disturbance has occurred is fed back.

  If it is determined in step S47 that no disturbance has occurred, then exposure correction value feedback is performed (S49). Here, the output difference between frames is fed back as an exposure correction value even if it is below the threshold level. As a result, when a slight change below the exposure threshold level is undesirable for the photographer, the exposure of the N frame can be corrected at the time of camera shake correction. Two types of composite images may be recorded by compositing a composite image with exposure correction and a composite image without exposure correction by parallel processing. Note that this step may be omitted because of the output difference below the threshold level.

  If disturbance detection is set in step S35, or if disturbance has not occurred in step S47, the process returns to the original flow (step S11). When returning to the original flow, the process proceeds to step S13 or S6 according to the setting in step S35 or S47.

  Next, the detailed operation of the image composition / shooting progress display in step S19 will be described with reference to FIG. In this image composition / shooting progress display flow, the image data corresponding to the exposure time T is obtained by sequentially adding and synthesizing images read by dividing the exposure time T into M, and adding and synthesizing M frames. (M composite images) are combined. When the image data corresponding to the exposure time T is synthesized, the images of the next M frames are sequentially added and synthesized in the same manner, and similarly, the synthesized image data (M synthesized images) of the next exposure time T is obtained. Generate. Different processing is performed for each shooting mode by using a plurality of M pieces of synthesized image data synthesized in this way.

  When the flow of image composition / shooting progress display is entered, first, the read image of the Nth frame is stored in the internal memory (S61). Here, the image data of the Nth frame read in step S9 is stored in the internal memory 33.

  After the image data is stored in the internal memory 33, it is next determined whether N is an integer multiple of M (S63). Here, the process is branched depending on the number of frames of the Nth frame. M is the number of divisions for generating a composite image (for example, M = 4 in the example shown in FIG. 5). If N is not an integer multiple of M as a result of this determination (No), this flow is terminated without processing, and the original flow is restored.

  On the other hand, if the result of determination in step S63 is that N is an integer multiple of M, the frame count L is incremented (+1) (S65). Here, a frame counter L (L is an integer of 1 or more) indicating how many M composite images have been combined is counted up. The frame counter L is a counter provided in the system control unit 20.

  Next, the M read images are added and synthesized to synthesize the L-th M synthesized image (S67). Here, M read images that have been exposed and read in the T / M time stored in the internal memory 33 are added and combined, and an M-th combined image of the L frame is combined.

  Once the Lth M composite images have been combined, the M read images stored in the internal memory 33 are then erased (S69). The M pieces of read image data stored in the internal memory 33 are deleted because they are not used later.

  Next, the set synthesis mode is determined (S71). Here, the processing of the M composite image is branched depending on the shooting mode set by the photographer.

  If the result of determination in step S71 is the comparison / synthesis mode, the addition / synthesis mode, or the addition / average mode, the comparison / addition / synthesis is performed using the Lth M-number synthesized image and the L-1 cumulative synthesized image. Alternatively, addition average synthesis is performed (S73). Once these syntheses are performed, the L cumulative synthesized image is subsequently stored in the internal memory 33 (S75).

  When the set synthesis mode is the comparison synthesis mode, in step S73, the M-th synthesized image of the Lth frame synthesized and the L-1 cumulative synthesized image obtained by synthesizing the L-1 frame from one frame, Are combined to generate an L cumulative composite image. In step S75, the generated L cumulative composite image is stored in the internal memory 33. At this time, if L = 1, the composition processing is not performed and the data is stored in the internal memory 33. When the L cumulative composite image is stored, the previous L-1 cumulative composite image may be deleted from the internal memory 33. However, for example, all the combined images without erasing may be left so that the photographer can select a favorite image after shooting, or the combined process may be saved as a moving image. Alternatively, the M composite images before the cumulative composition processing may be stored in the external memory 36 without being erased. After shooting, the photographer can use the plurality of M composite images to edit images such as desired composition / image processing using an editing function such as a camera or a PC (personal computer).

  When the synthesis mode is the addition synthesis mode, the comparison method is different from the comparison synthesis mode in that the synthesis method of the M composite images is addition synthesis, and when it is the addition average synthesis mode, it is addition average synthesis. However, the other processing flows are the same.

  If the L cumulative composite image is stored in the internal memory in step S75, then development processing is performed (S77). Here, the development processing unit 13 performs development processing on the L cumulative composite image stored in the internal memory 33. Once development processing has been performed, next, a shooting progress display is performed (S79). Here, using the image data of the L cumulative composite image that has been subjected to the development processing, the composite image is displayed on the display unit 37 as the shooting progress.

  On the other hand, if the result of determination in step S71 is the time-lapse moving image generation mode, the M composite images are stored in the internal memory 33 as one frame for moving images (S81).

  Next, it synthesize | combines with the moving image data produced | generated with the M sheet | seat synthetic | combination image from 1 to L-1 frames, and produces | generates the moving image to L frame (S83). If the format of the generated video does not compress the data amount due to changes between frames such as motionJPEG, the video may be generated at any time when the M composite images are saved, and shooting is completed. Then, the moving image may be generated after the recording of all the frames is completed. When a method of compressing the data amount from the data change between frames such as AVCHD is used, the moving image is generated after the shooting is completed and the recording of all the frames is completed.

  Next, development processing is performed (S85). Here, the M-frame composite image of the L frame generated in step S83 is developed. When the development process is performed, shooting progress display is performed (S87). Here, a composite image is displayed as shooting progress on the display unit 37 using image data of a moving image that has been subjected to development processing. When shooting progress display is performed in step S79 or S87, the flow returns to the original flow.

  As described above, in the flow of the present embodiment, when shooting is performed with the exposure time T, the exposure time is divided into M (S7), and the image data is read and read each time the M-divided exposure time elapses. The image data is stored in the internal memory 33 (S21, S61). If no disturbance occurs during acquisition of the M image data (No in S11), image synthesis is performed (S19, S67, S73, S81). On the other hand, if a disturbance occurs (S11 Yes), The next M-divided exposure is started without performing image composition on the image data when the disturbance occurs. For this reason, the image data when the disturbance occurs is not used for the image composition, so that the deterioration of the image quality can be prevented, and the continuity is almost lost because only the T / M time is excluded in the image composition. Absent.

  Next, the operation of the present embodiment will be described with reference to the time chart shown in FIG. 5. In order to understand the present embodiment, first, a conventional long exposure sequence will be described with reference to FIG. To do.

  11 (a) and 11 (b), the time elapses from the left side to the right side, and the processing order (in the camera) is shown from the upper side to the lower side. In FIG. The images exposed at T are sequentially read out from the image sensor 4 (images 1, 2, 3, 4,...), And the combined image is generated by the image combining unit 11 using the read images to generate a combined image. (Synthesis 1, 2, 3, 4,...) The synthesis processing here is any one of the above-described addition synthesis, comparison synthesis, addition average synthesis, and moving image generation.

  FIG. 11A shows an imaging sequence at the time of normal imaging where no disturbance occurs, and FIG. 11B shows an imaging sequence when disturbance occurs for a short time between time t2 and t3. When a disturbance occurs between times t2 and t3, the image 3 exposed during this time is not used for image composition (the image 3 is not used for the image of composition 2). Therefore, when a disturbance is detected during exposure, the image 3 is excluded and combined, and the finally combined image 3 is exposed for time T.

  FIG. 5 is a time chart showing a shooting sequence of long exposure in the present embodiment. FIG. 5 shows an example in which the division number M = 4, and the exposure time T is divided into four, and the division exposure is performed with the division exposure time T / M. The image sensor 4 exposes and sequentially reads out images for T / M time (image 1-1 to image 1-4, image 2-1 to image 2-4,...), And sequentially performs addition synthesis. Perform (“+” in the figure). M images are read out, and the combined image is added and synthesized as M synthesized images (4 synthesized images 1, 2, 3,...), And the M synthesized images are synthesized to generate a synthesized image. (Synthesis 1, 2, 3, ...).

  FIG. 5A shows a shooting sequence during normal shooting in which no disturbance occurs, and FIG. 5B shows a shooting sequence when a disturbance occurs at time t23-t24. In this case, the image 2-4 that has been exposed again at time t24-t31 and read out at time t31 is added and combined. Thereafter, when four images are added and synthesized in the same manner, a synthesis process is performed as a four-image synthesized image 2. As described above, conventionally, an image in which the time T exposure is interrupted is generated, but in the present embodiment, the exposure interruption can be shortened to the divided exposure time T / M.

  As described above, in the first embodiment of the present invention, even when a disturbance occurs during long-time shooting, it is possible to prevent image deterioration due to the disturbance and to shorten the exposure interruption as much as possible.

  Next, a second embodiment of the present invention will be described with reference to FIGS. In the first embodiment of the present invention, the image sensor 4 is a destructive readout type. However, the second embodiment is different in that it is a nondestructive readout type. The image sensor is generally a destructive readout type in which the stored electrical signal disappears when the electrical signal stored in the pixel is read out. However, various structures of nondestructive readout type image sensors that read out electrical signals while maintaining the accumulation of electrical signals have been proposed.

  The circuit configuration in this embodiment is the same as the block diagram shown in FIG. 1 according to the first embodiment. However, as described above, the structure of the image sensor 4 is different in that it is a nondestructive readout type. That is, the image sensor in this embodiment is a non-destructive readout type, and further includes a control unit that controls reset of charge accumulation and start of charge accumulation of pixels constituting the image sensor. When the disturbance detection unit 15 detects a disturbance, the control unit described above resets the charge accumulation and starts charge accumulation immediately after the disturbance disappears, and the image data synthesis unit 11 continues until just before the disturbance occurs. The image data generated in the above step is synthesized with the image data whose exposure is started immediately after the disturbance disappears. Further, the above-described control unit performs reset and start of charge accumulation every exposure time T.

  FIG. 9 shows a schematic configuration of a nondestructive readout type image sensor. This nondestructive readout type image sensor includes a photodiode 51, a charge transfer switch (SW) 53, an FD (capacitor) 55, a charge reset switch (SW) 57, an FD readout switch (SW) 59, a pixel peripheral circuit 61, and readout. A switch (SW) 63 and an analog-digital converter (ADC) 65 are provided. These switches 53, 59, and 63 are controlled by a control unit in the system control unit 20.

  As in the case of the image sensor 4 in the first embodiment, the photodiode 51 is regularly arranged for each pixel in the two-dimensional direction, and converts the light input into a charge signal. The output of the photodiode 51 is output to the FD 55 via the charge transfer SW 53. The FD 55 is a capacitor for accumulating the charge signal generated in the photodiode 51. The FD 55 is connected to a power supply Vdd via a charge reset switch (SW) 57. The output of the FD 55 is output to the pixel peripheral circuit 61 via the FD readout SW 59.

  The pixel peripheral circuit 61 includes various circuits such as an amplifier circuit for a signal read from the FD 55. The output of the pixel peripheral circuit 61 is output to the ADC 65 via the readout SW 63. The ADC 65 converts the analog signal output from the pixel peripheral circuit 61 into a digital signal. The output of the ADC 65 is output to the image processing unit 10.

  Next, the operation of the image sensor of this embodiment will be described with reference to FIG. FIG. 10A is a timing chart of the non-destructive read operation, and FIG. 10B is a timing chart of the destructive read operation for comparison. As can be seen from FIGS. 10A and 10B, the operations up to time T4 are the same for the non-destructive read and destructive read operations.

  In the case of destructive reading, the charge reset SW 57 is turned on at times T4 to T5, the power supply Vdd is applied, and the accumulation operation in the FD 55 is reset. When the accumulated charge is reset, from time T5, charge accumulation is performed in response to light input, as in times T1 to T4.

  On the other hand, in the case of non-destructive reading, the charge transfer SW 53 and the FD reading SW 59 are turned on without turning on the charge reset SW 57 at times T4 to T5. As a result, the signal charge is accumulated in the FD 55 and the accumulated charge signal is output to the pixel peripheral circuit 61. At time T5 to T6, the readout SW 63 is turned on, and the output of the pixel peripheral circuit 61 is output to the ADC 65.

  Thus, the nondestructive readout type image sensor can read out a voltage signal corresponding to the signal charge while maintaining the signal charge accumulated in the FD 55. For this reason, in the case of long-time exposure, it is possible to read out the signal accumulated by the divided exposure while being maintained by the image sensor 4.

  Next, the operation in this embodiment will be described with reference to FIGS. In the nondestructive readout image sensor, the processing is different in the addition synthesis mode, the comparison synthesis mode, the addition average synthesis mode, and the time-lapse moving image generation mode.

  First, the operation in the additive synthesis mode will be described with reference to FIG. An image is read at every time T and every T / M, and this reading is performed by nondestructive reading. The buffer image 1-1 is an image obtained by exposure at times t11 to t12, and the buffer image 1-2 is an image obtained by exposure at times t11 to t13. The buffer image 1-4 is an image obtained by exposure at times t11 to t21. In the first embodiment, the composite image of the images 1-1 to 1-4 obtained by the divided exposure is generated. However, since the non-destructive reading is performed, the composite image is generated by the image composition unit 11. Is not necessary.

  As described above, since non-destructive readout is performed, each readout image (progress display images 1, 2, 3,...) Displayed every T can be read out by a single readout. Read noise generated when reading from the sensor 4 is only required once. As a result, the number of readouts is reduced as compared with the case of the normal image sensor 4 described with reference to FIG.

  Further, the case where disturbance noise occurs will be described with reference to FIG. For example, when a disturbance is detected during the exposure of the image 2-2 (time t22 to t23), the exposure is interrupted (the charge reset SW 55 is turned on and the FD 55 is reset), and the time t23 when the disturbance has not occurred. The exposure is restarted again. Then, the subsequent progress display image is synthesized by adding and synthesizing the read image 2-1 before the occurrence of the disturbance and the image after the non-destructive reading. Progress display image 2 is an image obtained by adding and synthesizing image 2-1 and image 2-4, and the period during which exposure is interrupted is only the period of T / M of image 2-2 exposure (time t22 to t23). The number of images (number of readings) is also two.

  If no disturbance occurs after time t23, the image 2-1 and the subsequent non-destructive read image may be added, and the two read images may be added and combined. For this reason, similarly to the first embodiment, exposure loss due to disturbance can be suppressed to a time of T / M, and the amount of readout noise included in the composite image can be reduced, thereby improving the image quality.

  Next, operations in the comparison synthesis mode, the addition average synthesis mode, and the time-lapse moving image generation mode will be described with reference to FIGS. 7 and 8. In this mode, unlike the addition synthesis, read images of the exposure time T are synthesized, so destructive readout is performed every time T, and nondestructive readout is performed every T / M thereafter. When disturbance does not occur, as shown in FIG. 7, a destructive read image (four composite images 1, 2, 3, 4,...) Read at each exposure time T is combined to process a still image or a moving image. Are synthesized (synthesis 1, 2, 3,...).

  Next, when a disturbance occurs, the exposure is interrupted, and the exposure is resumed at a timing when the disturbance is no longer detected. For example, in the example shown in FIG. 8, a disturbance occurs between times t22 and t23. In this case, exposure is resumed at time t23, and a four-sheet composite image (time t21 to t25) In the progress display / combination image 2), the images exposed for the time T are combined by adding and combining the images 2-1 and 2-4. The subsequent four composite images are similarly composed of destructive readout images for each T.

  The flowchart for executing the operation in the present embodiment is substantially the same as the flowchart shown in FIGS. 2 to 4 in the first embodiment. However, in step S9 in FIG. 2, since the image sensor of the first embodiment is a destructive readout type image sensor, the reset operation is always performed. However, in the present embodiment, a non-destructive readout type image sensor is used. Therefore, the FD 55 may be reset as necessary. For example, in the case of additive synthesis mode, it may be reset at the timing of resuming exposure, and in the case of comparative synthesis mode, additive average synthesis mode, and time-lapse movie generation mode, it may be reset every time the exposure time T elapses. Good.

  As described above, in the second embodiment, even when a disturbance occurs during long-time shooting, it is possible to prevent image deterioration due to the disturbance and to minimize exposure interruption. Furthermore, by using a nondestructive imager to read out and devise a composition method, image quality deterioration due to readout noise can be further suppressed as much as possible, and the image quality of the composite image can be further improved.

  As described above, in each embodiment of the present invention, image data exposed at the exposure time T is repeatedly combined (for example, S19 in FIG. 2, S73, S81 in FIG. 4), and occurs during shooting. A disturbance is detected (for example, S11 in FIG. 2, FIG. 3). In the image data synthesis, the exposure time is divided into T / M (M is 2 or more) time intervals (for example, S7 in FIG. 2), and when a disturbance is detected during shooting, the divided time intervals are divided. The image data of the exposure time T is generated by excluding the image data of the time interval in which the disturbance is mixed (for example, S11 Yes in FIG. 2). In this way, the exposure time is divided into time intervals, and when a disturbance occurs, the image data corresponding to the time interval is excluded and the image is synthesized, so even if a disturbance occurs during shooting, the image quality deteriorates due to the disturbance. Can be prevented and exposure interruptions can be minimized.

  In each embodiment of the present invention, a digital camera has been described as an apparatus for photographing. However, the camera may be a digital single lens reflex camera or a compact digital camera, such as a video camera or a movie camera. It may be a camera for moving images, and may be a camera built in a mobile phone, a smart phone, a personal digital assistant (PDA), a personal computer (PC), a tablet computer, a game machine, or the like. In any case, the present invention can be applied to any device for photographing having a long-time photographing mode.

  Of the techniques described in this specification, the control mainly described in the flowchart is often settable by a program and may be stored in a recording medium or a recording unit. The recording method for the recording medium and the recording unit may be recorded at the time of product shipment, may be a distributed recording medium, or may be downloaded via the Internet.

  In addition, regarding the operation flow in the claims, the specification, and the drawings, even if it is described using words expressing the order such as “first”, “next”, etc. It does not mean that it is essential to implement in this order.

  The present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, you may delete some components of all the components shown by embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

DESCRIPTION OF SYMBOLS 1 ... Imaging part, 2 ... Lens (aperture), 3 ... Mechanical shutter, 4 ... Image sensor, 10 ... Image processing part, 11 ... Image composition part, 11a ... Addition Combining unit, 11b ... comparison / combining unit, 11c ... addition / average combining unit, 11d ... moving image generating unit, 13 ... development processing unit, 15 ... disturbance detection unit, 16 ... camera shake Detection unit, 17 ... Exposure abnormality detection unit, 20 ... System control unit, 31 ... Bus, 33 ... Internal memory, 36 ... External memory, 37 ... Display unit, 38 ... Input IF, 51 ... photodiode, 53 ... charge transfer SW, 55 ... FD (capacitor), 57 ... charge reset SW, 59 ... FD read SW, 61 ... pixel periphery Circuit 63 ... Reading SW 65 ... ADC (analog data Tal converter)

Claims (8)

An image sensor;
An image data synthesis unit that repeatedly synthesizes the image data exposed at the exposure time T;
A disturbance detection unit for detecting disturbances occurring during shooting,
With
The image data synthesis unit divides the exposure time into T / M (M is 2 or more) time intervals, and when a disturbance is detected by the disturbance detection unit during shooting, An image pickup apparatus characterized in that image data of an exposure time T is synthesized by excluding image data in a time section in which the disturbance is mixed.   The imaging apparatus according to claim 1, wherein the imaging element is a destructive readout type, and the image data synthesis unit adds image data obtained by exposure with the exposure time T / M. The image sensor is a non-destructive readout type, and further includes a control unit that controls reset of charge accumulation and start of charge accumulation of pixels constituting the image sensor,
When the disturbance detection unit detects a disturbance, the control unit resets the charge accumulation and starts the charge accumulation immediately after the disturbance disappears, and the image data composition unit immediately before the disturbance occurs. The image pickup apparatus according to claim 1, wherein the image data generated so far and the image data for which exposure is started immediately after the disturbance disappears are combined. The image sensor is a non-destructive readout type, and further includes a control unit that controls reset of charge accumulation and start of charge accumulation of pixels constituting the image sensor,
The imaging apparatus according to claim 3, wherein the control unit performs the resetting and the start of charge accumulation every exposure time T.   The imaging apparatus according to claim 1, wherein the disturbance detection unit detects camera shake during exposure.   The imaging apparatus according to claim 1, wherein the disturbance detection unit analyzes a luminance value of the image data subjected to the divided exposure and detects a change in luminance.   The imaging apparatus according to claim 1, wherein the image data synthesis unit performs at least one of comparative bright synthesis, comparative dark synthesis, additive average synthesis, and time-lapse moving image generation. An image data synthesis step for repeatedly synthesizing the image data exposed at the exposure time T;
A disturbance detection step for detecting disturbances occurring during shooting;
Have
The image data synthesis step divides the exposure time into T / M time intervals (M is 2 or more), and if a disturbance is detected in the disturbance detection step during shooting, An image pickup method, wherein image data of an exposure time T is generated by excluding image data of a time section in which the disturbance is mixed.

Images