JP5721552B2 - Imaging apparatus and image composition method - Google Patents

Description

  The present invention relates to an imaging apparatus and an image composition method.

  An imaging device such as a digital camera that can synthesize a plurality of images and record one synthesized image has been proposed. Among such image capturing apparatuses, there is an image capturing apparatus having a function of capturing an image with an exposure lower than an appropriate exposure at which the brightness of the subject is appropriate and integrating and combining a plurality of low-exposure images. In general, when a dark subject is photographed, the exposure time is increased in order to increase the exposure amount, so that camera shake and subject blur are likely to occur.

  When the imaging device captures a dark subject, it aligns the subject between screens for multiple low-exposure images captured with an exposure time shorter than the appropriate exposure, and achieves the same brightness as when captured with appropriate exposure. It is conceivable to integrate and synthesize so that Thereby, an integrated composite image with less camera shake and subject blur can be obtained.

  As an example of a technique for integrating and synthesizing a plurality of images, Patent Document 1 discloses an electronic camera in which the number of shots is set in advance and images obtained by shooting the set number of the same subject are superimposed.

Japanese Patent Application Laid-Open No. 11-069226

  The imaging apparatus aligns and integrates a plurality of low-exposure images with a short exposure time, whereby an integrated composite image with less camera shake and subject blur can be obtained. However, since a low-exposure image does not have a better signal-to-noise (S / N) than a proper image, there is a problem that the image quality of an image obtained by integration synthesis is degraded.

  An object of the present invention is to provide an imaging apparatus that integrates and synthesizes a plurality of low-exposure images, and that improves the image quality of an image obtained by integration and synthesis.

An imaging apparatus according to an embodiment of the present invention includes an exposure calculation unit that calculates a second exposure lower than a first exposure at which the brightness of a subject has a predetermined brightness, and the calculated second exposure. An image pickup means for picking up a subject and outputting a plurality of low-exposure images; a summing-compositioning means for summing and synthesizing the plurality of output low-exposure images to generate a composite image; and tone mapping of the composite image Tone mapping means, and motion detection means for detecting the amount of motion of the subject imaged by the imaging means . The integrating synthesis unit integrates and synthesizes the low-exposure images for the number of exposures of the synthesized image equal to or greater than the exposure amount corresponding to the first exposure, and the tone mapping unit is detected by the motion detection unit. As the amount of motion to be performed is smaller, the tone mapping characteristic of tone mapping performed on the composite image is set to a tone mapping characteristic that increases the gradation compression ratio of the dark part.

  According to the present invention, it is possible to provide an imaging apparatus that integrates and synthesizes a plurality of low-exposure images, and that reduces the amplitude of noise of an image obtained by integration and synthesis. Therefore, the image quality of the integrated composite image is improved.

It is a figure which shows the structural example of the imaging device of this embodiment. It is a figure which shows the structural example of a synthetic | combination number calculation part and an exposure calculation part. It is a figure explaining the example of an operation process of an image integration composition part. It is a figure explaining the example of an operation process of an image integration composition part. It is a figure explaining a gradation compression process. 6 is a flowchart illustrating an example of operation processing of the imaging apparatus according to the present embodiment. It is a figure explaining the example of the integrating | accumulating synthetic | combination process of several low exposure image.

  FIG. 7 is a diagram for explaining an example of an integration synthesis process of a plurality of low-exposure images. FIG. 7A shows the relationship between the appropriate exposure and low exposure subject illuminance and photocurrent (pixel level) in an imaging device having linear characteristics. A symbol L801 indicates a relationship between subject illuminance and photocurrent in the case of proper exposure. A symbol L802 indicates a relationship between subject illuminance and photocurrent in the case of low exposure. Further, in this example, the low exposure is one third of the exposure amount with respect to the proper exposure.

  When a subject with a certain illuminance is photographed with appropriate exposure and low exposure, a low-exposure image has a smaller photocurrent and a dark image is captured. Usually, the appropriate exposure is determined so that the subject does not white out. In this case, the subject of the low-exposure image is captured with a brightness that is one-third that of the proper exposure image. FIG. 7B schematically shows this relationship.

  FIG. 7B shows an example of processing for changing the exposure by changing the exposure time. The pixel level range that each image can take is indicated by hatching. The exposure time for proper exposure is 1/10 second. The exposure time for low exposure is 1/30 second. Since the exposure amount of low exposure is one third of the proper exposure, the pixel level range that the low exposure image can take is one third of that of the proper exposure image. The low-exposure image shot in this way is a darker image than the proper exposure image. However, since the exposure time is shorter than the appropriate exposure, it can be said that the image has little camera shake or subject blur. Appropriateness is achieved by aligning the subject position with a plurality of captured low-exposure images by geometric deformation, etc., and integrating and synthesizing three low-exposure images as shown in the example of the integrated composite image shown in FIG. It is possible to obtain a composite image with the same brightness as the exposure and with little camera shake and subject blur.

  However, as described above, the low-exposure image has an S / N lower than that of the appropriate image, so that there is a problem that the image quality of the image obtained by the integration synthesis is deteriorated. According to the imaging apparatus of the present embodiment described below, this problem can be solved.

  FIG. 1 is a diagram illustrating a configuration example of an imaging apparatus according to the present embodiment. The imaging apparatus illustrated in FIG. 1 includes a lens 101, an imaging unit 102, and an A (Analog) / D (Digital) conversion unit 103. In addition, the imaging apparatus includes a camera signal processing unit 104, an image registration unit 105, a composite number calculation unit 106, an exposure calculation unit 107, an image integration synthesis unit 108, a tone mapping unit 109, an image recording unit 110, and a camera control unit 111. Is provided. The image composition method of the present embodiment is an image composition method in the imaging apparatus. In addition, the image composition method and the computer program according to the present embodiment are realized by the functions of the processing units included in the imaging apparatus illustrated in FIG.

  The lens 101 guides subject light to the imaging unit 102. The lens 101 performs a zoom operation, a focus adjustment operation, and the like in response to a control signal output from the camera control unit 111. The imaging unit 102 images a subject using an imaging device in accordance with an instruction from the camera control unit 111. The imaging device is a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS), or the like. Then, the imaging unit 102 converts subject light obtained by imaging into an electrical signal and outputs the electrical signal to the A / D conversion unit 103.

  The A / D conversion unit 103 amplifies the analog signal according to the amplification factor set by the camera control unit 111 and converts it into a digital signal. The camera signal processing unit 104 performs image processing such as white balance and edge enhancement on the digital signal output from the A / D conversion unit 103 and outputs a video signal.

  The image alignment unit 105 geometrically deforms the image according to the motion vector between the screens, thereby aligning the subject position between the screens and suppressing the motion between the screens. By image alignment by the image alignment unit 105, for example, movement of the entire screen such as camera shake can be reduced. The composite number calculation unit 106 calculates the number of images (integrated composite number) to be integrated and combined by the image integration / synthesis unit 108.

  The composite number calculation unit 106 calculates, as the integrated composite number, the number of exposures of the composite image obtained by the integration synthesis by the image integration synthesis unit 108 that is equal to or greater than the exposure amount corresponding to the appropriate exposure. The appropriate exposure is the first exposure at which the brightness of the subject becomes a predetermined brightness.

  The exposure calculation unit 107 calculates the exposure used by the imaging unit 102 for imaging. Specifically, the exposure calculation unit 107 calculates a second exposure (low exposure) lower than the appropriate exposure. The camera control unit 111 controls the entire imaging apparatus. The camera control unit 111 and the imaging unit 102 function as an imaging unit that captures an image of a subject with the second exposure and outputs a plurality of low-exposure images.

  The image integration / synthesis unit 108 acquires the integration / synthesis number calculated by the synthesis number calculation unit 106 from the combination number calculation unit 106. Further, the image integration / synthesis unit 108 acquires, from the image registration unit 106, a registration image output from the image registration unit 106, that is, a plurality of low exposure images. The image integration / synthesis unit 108 functions as an integration / synthesis unit that integrates and combines the low-exposure images acquired by the image registration unit 106 by the integration / synthesis number calculated by the synthesis number calculation unit 106.

  A tone mapping unit 109 performs tone mapping on the composite image. The tone mapping unit 109 performs tone compression of the composite image by non-linear processing such as gamma processing, for example. The image recording unit 110 records the image that has been tone mapped by the tone mapping unit 109.

  FIG. 2 is a diagram illustrating a configuration example of the composite number calculation unit and the exposure calculation unit. FIG. 2A shows a configuration example of the composite number calculation unit 106. The composite number calculation unit 106 includes a motion amount calculation unit 201 and a composite number determination unit 202. The motion amount calculation unit 201 receives the registration image output from the image registration unit 105 and calculates (detects) the motion amount based on the size of the inter-frame pixel difference value of the input image. That is, the motion amount calculation unit 201 functions as a motion detection unit that detects a motion amount between a plurality of low-exposure images captured by the imaging unit 102.

  For example, when the subject is moving, the inter-frame pixel difference value is large, so that the amount of movement is large. The composite number determination unit 202 determines the number of low-exposure images (integrated composite number) to be integrated and combined by the image integration / synthesis unit 108 according to the motion amount output from the motion amount calculation unit 201.

  FIG. 2B is a diagram illustrating a configuration example of the exposure calculation unit. FIG. 2B shows a configuration example of the exposure calculation unit 107. The exposure calculation unit 107 includes a motion amount calculation unit 300, a luminance integrated value calculation unit 301, and an exposure determination unit 302.

  The luminance integrated value calculation unit 301 receives the registration image output from the image registration unit 105, converts it into luminance representing the brightness of the subject, and calculates the luminance integrated value. The luminance integrated value is an integrated value of luminance of a plurality of low-exposure images. For example, if the image is bright, the integrated luminance value is a large value.

  The motion amount calculation unit 300 has a function similar to that of the motion amount calculation unit 201 illustrated in FIG. The motion amount calculation unit 300 receives the registration image output from the image registration unit 105 and calculates a motion amount based on the size of the inter-frame pixel difference value of the image. The exposure determining unit 302 determines the exposure of the next image according to the luminance integrated value calculated by the luminance integrated value calculating unit 301 and the amount of motion calculated by the motion amount calculating unit 300 (the imaging unit 102 captures an object). Change the second exposure used).

  The exposure determining unit 302 may determine the exposure of the next image according to the luminance integrated value calculated by the luminance integrated value calculating unit 301 and the motion amount calculated by the motion amount calculating unit 201. A specific method for determining exposure will be described later.

  3 and 4 are diagrams for explaining an example of operation processing of the image integration / synthesis unit. The image integration / synthesis unit 108 integrates and synthesizes the images taken with the exposure calculated by the exposure calculation unit 107 by the number of integration / composite images calculated by the synthesis number calculation unit 106. In this embodiment, the aperture value is fixed, the exposure time is changed, and the exposure is controlled.

  FIG. 3 shows an example of the process of changing the integrated composite number. In this example, the exposure time is fixed to 1/30 seconds, and the composite number calculation unit 106 changes the integrated composite number according to the motion amount calculated by the motion amount calculation unit 201.

  The hatched portion in FIG. 3 indicates the pixel level range of the alignment image output from the image alignment unit 105. In this example, the quantization accuracy of the A / D conversion unit 103 is 12 bits. In general, when shooting is performed with proper exposure, shooting is performed with exposure that does not cause overexposure or blackout, so the pixel level range of a properly exposed image is a range from 0 to 4095. The imaging apparatus according to the present embodiment performs shooting with an exposure lower than the appropriate exposure.

  In the example shown in FIG. 3, the exposure time for proper exposure is 1/10 seconds, whereas the exposure time for low-exposure images is 1/30 seconds. Since the exposure amount of the low-exposure image is 1/3, the pixel level range is a range of 1/3 of the proper exposure image, and thus the image is dark. By integrating and synthesizing a plurality of such dark low-exposure images by the image integrating and synthesizing unit 108, a bright integrated synthesized image can be obtained. In this example, when three low-exposure images are integrated and combined, the brightness (pixel level range) is the same as that of the proper exposure image.

  A low-exposure image has a smaller exposure amount than the proper exposure, but by shortening the exposure time, it is possible to record an image in which subject blur and camera shake are suppressed compared to the proper exposure. For such a low-exposure image, the image alignment unit 105 aligns the position of the entire screen such as camera shake, and performs integration synthesis. As a result, it is possible to record an integrated composite image with less blur compared to an image photographed with appropriate exposure. However, it is very difficult to align partial movements such as subject movements, not movements of the entire screen. Therefore, for example, subject images may be doubled when partial motion areas are integrated and combined.

  The image integration / synthesis unit 108 performs integration / synthesis for the integration / synthesis number calculated by the synthesis number calculation unit 106. The composite number determination unit 202 of the composite number calculation unit 106 increases the integrated composite number to be determined as the motion amount between the low-exposure image screens calculated (detected) by the motion amount calculation unit 201 is small. Further, the composite number determination unit 202 decreases the total composite number to be determined as the motion amount between the plurality of low-exposure image screens calculated (detected) by the motion amount calculation unit 201 is large.

  In the example illustrated in FIG. 3, when the amount of motion is small (less than a predetermined amount of motion), the composite number calculation unit 106 performs the integration synthesis so that the exposure amount of the integrated composite image is equal to or greater than the exposure amount of the appropriate exposure image. Six sheets are calculated as the number of sheets. When the amount of motion is large (when the amount of motion is greater than or equal to a predetermined amount of motion), the composite number calculation unit 106 sets the total number of composite images to 3 so that the exposure amount of the integrated composite image is equal to or greater than the exposure amount of the appropriate exposure image. Is calculated. In this way, when the amount of motion is large, it is possible to suppress the subject image from being duplicated by reducing the total number of combined images.

  FIG. 4 shows another example of the process of changing the integrated composite number. In this example, the total number of combined images is fixed at 3. In addition, the composite number calculation unit 106 changes the exposure time according to the motion amount calculated by the motion amount calculation unit 201. The shaded area in FIG. 4 indicates the pixel level range of the alignment image output from the image alignment unit 105. Note that the quantization accuracy of the A / D converter 103 is 12 bits. Also in this example, the imaging device performs shooting at an exposure lower than the appropriate exposure.

  In the example shown in FIG. 4, the exposure time for proper exposure is 1/10 seconds, whereas the exposure time for low-exposure images is 1/20 seconds. That is, the exposure amount of the low-exposure image is halved. Therefore, the pixel level range is a half of the proper exposure image compared to the proper exposure, and is a dark image. The image integration / synthesis unit 108 integrates a plurality of such dark low-exposure images, thereby obtaining a bright integrated image. In this example, when two low-exposure images are integrated and combined, the brightness (pixel level range) is the same as that of the proper exposure image.

  The image integration / synthesis unit 108 performs integration / synthesis of three positional images. The exposure control unit 107 increases the exposure time when the amount of movement between the low-exposure image screens is small. The exposure determining unit 302 of the exposure calculating unit 107 shortens the exposure time when the amount of movement between the plurality of low-exposure image screens is large. That is, the exposure calculation unit 107 lowers the exposure used by the imaging unit 102 for imaging the subject as the amount of motion increases.

  In the example illustrated in FIG. 4, when the amount of motion is small, the exposure determining unit 302 sets the exposure time to 1/20 second, so that the exposure amount of the integrated composite image is equal to or greater than the exposure amount of the appropriate exposure image. To be. When the amount of motion is large, the exposure determination unit 302 sets the exposure time of the integrated composite image to the same exposure amount as that of the appropriate exposure image by setting the exposure time to 1/30 seconds. Thus, by reducing the exposure time when the amount of motion is large, it is possible to reduce subject blurring that occurs when the subject moves during the exposure period.

  FIG. 5 is a diagram for explaining gradation compression processing by the tone mapping unit. The graph shown in FIG. 5 shows the tone curve of the gradation compression process. The horizontal axis is the input pixel level, and the vertical axis is the output pixel level.

  The tone mapping unit 109 changes the tone curve characteristic (tone mapping characteristic) according to the pixel level range of the image integrated and combined by the image integration / synthesis unit 108. For example, when the image output from the image integration / synthesis unit 108 is an integration / synthesis image with a small amount of motion, the tone mapping unit 109 performs tone compression processing using a tone curve indicated by reference numeral L601 in FIG. For example, when the image output from the image integration / synthesis unit 108 is an integration / synthesis image with a large amount of motion, the tone mapping unit 109 performs tone compression processing using a tone curve indicated by reference numeral L602 in FIG.

  Comparing the tone curve characteristics L601 and L602, the tone curve characteristics L601 has a higher gradation compression rate especially in the dark part. An image obtained by tone mapping an integrated composite image with a small amount of motion has a higher gradation compression ratio than an integrated composite image with a large amount of motion. That is, the tone mapping unit 109 sets a tone mapping characteristic of tone mapping performed on the integrated composite image to a higher tone compression ratio in the dark part as the motion amount calculated by the motion amount calculation unit 201 (300) is smaller. Make mapping characteristics. This reduces the noise amplitude and improves the image quality.

  The tone mapping unit 109 may change the tone curve according to the magnitude of the analog signal amplification factor of the A / D conversion unit 103. Specifically, the tone mapping unit 109 increases the gradation compression rate of the dark part when the analog signal amplification factor of the A / D conversion unit 103 is high. Further, the tone mapping unit 109 reduces the gradation compression rate of the dark part when the analog signal amplification factor of the A / D conversion unit 103 is low. Increasing the analog signal amplification factor of the A / D conversion unit 103 increases the amplitude of noise, but the tone mapping unit 109 changes the tone curve characteristics as described above, thereby reducing the amplitude of noise generated in the dark portion. can do.

  FIG. 6 is a flowchart for explaining an operation processing example of the imaging apparatus according to the present embodiment. First, the image pickup apparatus is turned on (step S1). Next, the composite number calculation unit 106 sets the integrated composite number to an initial value (step S2). The composite number calculation unit 106 sets the total composite number to, for example, three.

  Next, the exposure calculation unit 107 sets the exposure to an initial value (step S3). The exposure calculation unit 107 sets the exposure time to 1/30 seconds, for example. Subsequently, the camera control unit 111 performs provisional shooting. During temporary shooting, the camera control unit 111 displays the captured image on a display unit (not shown) of the imaging device.

  Next, the motion amount calculation unit 201 calculates a motion amount (step S5). Then, the composite number determination unit 202 determines the integrated composite number based on the amount of motion calculated in step S5 (step S6). Subsequently, the exposure determining unit 302 determines the exposure of the next image based on the luminance integrated value calculated by the luminance integrated value calculating unit 301 and the motion amount calculated by the motion amount calculating unit 201 (300). Note that the imaging apparatus may operate so as to change both the total number of combined images and the exposure, or may operate so that one of them is fixed and the other is changed.

  Next, the camera control unit 111 determines whether the shutter has been pressed (step S8). If the camera control unit 111 determines that the shutter is not pressed, the process returns to step S4. Then, the processes in steps 4 to 7 are repeated until the shutter is pressed. As a result, the integrated composite number and the exposure are updated.

  When the camera control unit 111 determines that the shutter has been pressed, the camera control unit 111 performs the main shooting with the exposure determined in step 7 (step S9). Subsequently, the image integration / synthesis unit 108 integrates and combines the images for the integration / synthesis number determined in step 6 (step S10). Then, the tone mapping unit 109 performs tone mapping on the image integrated and synthesized in step S10 (step S11), and records it in the image recording unit 110.

  Through the above operation, the noise amplitude can be reduced by integrating and synthesizing a plurality of low-exposure images and performing tone mapping so that the exposure amount is larger than the exposure amount that is captured with appropriate exposure. it can. Therefore, the image quality of the integrated composite image is improved.

(Other examples)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed. In this case, the program and the storage medium storing the program constitute the present invention.

DESCRIPTION OF SYMBOLS 101 Lens 102 Image pick-up part 105 Image alignment part 106 Composition number calculation part 107 Exposure calculation part 108 Image integration composition part 109 Tone mapping part 111 Camera control part

Claims (8)

Exposure calculation means for calculating a second exposure lower than the first exposure at which the brightness of the subject is a predetermined brightness;
Imaging means for imaging a subject with the calculated second exposure and outputting a plurality of low-exposure images;
Integration synthesis means for integrating and synthesizing the plurality of low-exposure images to generate a composite image;
Tone mapping means for performing tone mapping of the composite image ;
Motion detection means for detecting the amount of motion of the subject imaged by the imaging means ,
The integrating and synthesizing unit integrates and combines the low-exposure images by the number of exposures of the combined image equal to or greater than the exposure amount corresponding to the first exposure ;
The tone mapping means sets the tone mapping characteristic of tone mapping performed on the composite image to a tone mapping characteristic that increases the tone compression ratio of a dark part as the amount of motion detected by the motion detection means is smaller. An imaging device that is characterized. Wherein in accordance with the motion amount of the motion detecting means detects the integrated synthesizing means imaging apparatus according to claim 1, wherein a and a synthesis number determining means for determining the number of low-exposure image integrating synthesis. The imaging apparatus according to claim 2, wherein the composite number determination unit reduces the integrated composite number of the low-exposure images to be determined as the amount of motion detected by the motion detection unit increases. The exposure calculating unit is configured to capture the subject based on an integrated luminance value of the plurality of low-exposure images captured by the imaging unit and a motion amount detected by the motion detecting unit. The imaging apparatus according to claim 1 , wherein the second exposure to be used is changed. 5. The imaging apparatus according to claim 4, wherein the exposure calculation unit lowers the second exposure used by the imaging unit for imaging the subject as the amount of motion detected by the motion detection unit increases. . The imaging apparatus according to claim 4, wherein the exposure calculation unit changes the second exposure by changing an exposure time. An image composition method in an imaging apparatus,
An exposure calculation step of calculating a second exposure lower than the first exposure at which the brightness of the subject is a predetermined brightness;
An imaging step of imaging a subject with the calculated second exposure and outputting a plurality of low-exposure images;
An integration synthesis step of integrating and synthesizing the plurality of low-exposure images to generate a composite image;
A tone mapping step for performing tone mapping of the composite image ;
A motion detection step of detecting the amount of motion of the subject imaged in the imaging step ,
The integrated case as in Naruko, number of sheets to exposure of the composite image is equal to or greater than the exposure amount corresponding to the exposure of the first, the low-exposure image integrated synthesis,
In the tone mapping step, as the amount of motion detected in the motion detection step is small, the tone mapping property of tone mapping performed on the composite image is set to a tone mapping property that increases the tone compression rate of the dark part. An image composition method characterized by the above. A computer program for causing a computer to execute the image composition method according to claim 7 .

Images