JP4619895B2 - Imaging apparatus, image processing apparatus, imaging method, and image processing method - Google Patents

Description

  The present invention relates to an imaging apparatus, an image processing apparatus, an imaging method, and an image processing method. In particular, the present invention relates to an imaging device, an image processing device, an imaging method, and an image processing method that synthesize images.

There is known an electronic camera that outputs a high-quality moving image signal by performing processing for preventing flickering according to the calculated magnitude of a motion vector of a subject (for example, see Patent Document 1). The camera described in Patent Document 1 adds a predetermined pixel and a pixel adjacent thereto when the magnitude of the motion vector exceeds a predetermined value.
Japanese Patent Application Laid-Open No. 09-200685

  Conventionally, however, there is a problem that random noise is noticeable when the captured image is dark.

  Accordingly, an object of the present invention is to provide an imaging apparatus, a moving image processing apparatus, an imaging method, and a moving image processing method that can solve the above-described problems. This object is achieved by a combination of features described in the independent claims. The dependent claims define further advantageous specific examples of the present invention.

  The imaging apparatus according to the first aspect of the present invention is an imaging apparatus that continuously captures images, and brightness determination means that determines whether or not the acquired image is darker than a reference value. When the image is determined to be darker than the reference value, the image processing apparatus includes a combining unit that combines the image captured at least one of the images immediately before or immediately after the image with the one image.

  The synthesizing unit may synthesize two images captured immediately before and after the image into one image when one image is darker than a dark reference value smaller than the reference value.

  The imaging apparatus captures a moving image by continuously capturing images, and the combining unit combines the images captured at least immediately before or immediately after the images with each of the continuously captured images. May be.

  The imaging apparatus may further include a brightness adjustment unit that brings the brightness of the image synthesized by the synthesis unit closer to the brightness of the image captured before the image so as to keep the change in brightness of the moving image. The brightness adjustment unit may further include a unit that brings an image obtained by combining two images close to the brightness of an image captured before the image so that the change in brightness of the moving image is continued.

  The imaging apparatus further includes a still image acquisition unit that acquires a still image while capturing a moving image, and the determination unit determines whether the still image acquired by the still image acquisition unit is darker than a reference value. Then, when the still image is darker than the reference value, the synthesizing unit may synthesize an image in the moving image captured immediately before or after the still image with the still image.

  The imaging apparatus further includes a motion detection unit that detects a relative movement of the subject with respect to the imaging device, and a reference value setting unit that sets a reference value smaller when the motion detected by the motion detection unit is larger. May be.

  The imaging method according to the second aspect of the present invention is an imaging method for continuously capturing images, and a brightness determination step for determining whether or not the acquired image is darker than a reference value. And a synthesis step of synthesizing an image captured at least one of the images immediately before or immediately after the image when the image is darker than the reference value.

  In the combining step, when one image is darker than a dark reference value smaller than the reference value, two images captured immediately before and after the image may be combined into one image. The imaging method may further include a brightness adjustment step for bringing the brightness of the image synthesized by the synthesis step closer to the brightness of the image captured before the image so that the change in the brightness of the image is continuous.

  An image processing apparatus according to a third aspect of the present invention is an image processing apparatus that increases the brightness of an image, and includes a brightness determination unit that determines whether one image is darker than a reference value, and one image. And, when it is determined that the image is darker than the reference value, combining means for increasing the brightness of the one image by combining the image captured at least one of immediately before or immediately after the one image into the one image; Is provided.

  The synthesizing unit may synthesize two images captured immediately before and after the image into one image when one image is darker than a dark reference value smaller than the reference value.

  The synthesizing unit may synthesize an image captured at least one immediately before or after the image with each of the continuously captured moving images.

  The image processing apparatus may further include a brightness adjustment unit that brings the brightness of the image synthesized by the synthesis unit closer to the brightness of an image captured before the image so as to keep the change in brightness of the moving image. The brightness adjustment unit may further include a unit that brings an image obtained by combining two images close to the brightness of an image captured before the image so that the change in brightness of the moving image is continued.

  The image processing apparatus further includes a still image acquisition unit that acquires a still image from a moving image, the determination unit determines whether the still image is darker than a reference value, and the combining unit determines that the still image is lower than the reference value. When the image is dark, an image that is being captured at least one of the images immediately before or immediately after the still image may be combined with the still image.

  The image processing apparatus further includes a motion detection unit that detects a motion of a subject included in the moving image, and a reference value setting unit that sets a reference value smaller when the motion detected by the motion detection unit is larger. Also good.

  A moving image processing method according to a fourth aspect of the present invention is an image processing method for increasing the brightness of an image, wherein a brightness determination step for determining whether one image is darker than a reference value, When the image is determined to be darker than the reference value, a synthesizing step for increasing the brightness of the one image by synthesizing the image picked up at least one immediately before or immediately after the one image into the one image With.

  In the combining step, when one image is darker than a dark reference value smaller than the reference value, two images captured immediately before and after the image may be combined into one image.

  The image processing method may further include a brightness adjustment step for bringing the brightness of the image synthesized by the synthesis step closer to the brightness of the image captured before the image in order to make the change in brightness of the image continuous.

  Note that the above summary of the invention does not enumerate all the necessary features of the present invention, and sub-combinations of these feature groups can also be the invention.

  According to the present invention, this random noise can be made inconspicuous by synthesizing another frame image with a dark frame image in which random noise is conspicuous.

  Hereinafter, the present invention will be described through embodiments of the invention. However, the following embodiments do not limit the claimed invention, and all combinations of features described in the embodiments are inventions. It is not always essential to the solution.

  FIG. 1 is a block diagram of an imaging device 10 and a moving image processing device 20 according to an embodiment of the present invention. The moving image imaging apparatus 10 includes an imaging unit 160 and a moving image processing apparatus 20. The moving image processing apparatus 20 includes a brightness determination unit 100, a synthesis unit 110, a brightness adjustment unit 120, a motion detection unit 130, a reference value setting unit 140, a still image acquisition unit 150, a recording unit 170, and a display unit 180.

The imaging unit 160 captures a moving image. The still image acquisition unit 150 acquires a still image while capturing a moving image. The recording unit 170 records moving images and still images. The display unit 180 displays the image recorded by the recording unit 170. The still image acquisition unit 150 may acquire only the timing when the imaging operation is performed without actually capturing an image. In this case, for example, the moving image processing device 20 acquires the time when the still image is imaged by the timer included in the moving image processing device 20. Then, the recording unit 170 records the acquired time.

The brightness determination unit 100 determines whether each frame image in the moving image captured by the imaging unit 160 or the still image acquired by the still image acquisition unit 150 is darker than a predetermined reference value. to decide. For example, the brightness determination unit 100 adds the brightness of each pixel of the acquired image and divides it by the total number of pixels to obtain an average value of brightness, and determines whether this average value is darker than the reference value. .

  When it is determined that the acquired one image is darker than the reference value, the synthesizing unit 110 refers to the imaging time information (time stamp) of each image for the image immediately before or after the acquired one image. To select. Then, the synthesizing unit 110 adds the lightness of the image immediately before or after the acquired image to the lightness of the acquired one image. Specifically, for each pixel, the synthesizing unit 110 adds the brightness value of the pixel in the image immediately before or after the image to the brightness value of the pixel in the acquired one image.

  Now, the imaged image has readout noise that occurs when electric charges are read out from the individual imaging element in the imaging means 160. This readout noise contributes to random noise generated at random positions in the image. However, the position of random noise varies from frame to frame. For this reason, when a moving image with a lot of random noise is reproduced, the random noise is displayed at various positions. Random noise also occurs when the still image acquisition means 150 acquires a still image. Here, random noise is often noticeable in dark images.

  Therefore, the synthesizing unit 110 synthesizes another frame image with a dark frame image in which random noise is conspicuous. Since the position of the random noise differs depending on the frame, when a plurality of images are combined, the positions of the random noise are dispersed and become inconspicuous. In recent years, cameras capable of capturing still images during moving image capturing have been provided. Even when a still image captured during moving image capturing is dark and random noise is conspicuous, the synthesizing unit 110 synthesizes another frame image with the still image to make the random noise inconspicuous.

  Darker images often have more random noise. Therefore, when the acquired one image is darker than the dark reference value that is smaller than the reference value, the synthesizing unit 110 acquires two images that are captured immediately before and after the acquired image. Composite to one image. By combining the other two images with a dark image in which the random noise is more conspicuous, the random noise can be made less noticeable than when the other image is combined.

  The synthesizing unit 110 may synthesize an image captured at least one immediately before or immediately after each of the images captured continuously. In this case, it is possible to make the random noise in each of the continuously captured images inconspicuous.

  When multiple images are combined (added), the brightness of the image suddenly increases. Therefore, the brightness adjustment unit 120 brings the brightness of an image obtained by synthesizing the other one or two images closer to the brightness of an image captured before that image in order to make the change in brightness of the moving image continuous. . Thereby, the discontinuity of the change in brightness can be reduced.

  When the movement of the subject is large, when the target frame image and another frame image are combined, the image of the subject in the combined frame image is blurred. Therefore, the motion detection unit 130 detects the relative motion of the subject with respect to the moving image capturing apparatus 10. For example, the motion detection unit 130 detects the motion of the subject by comparing consecutively captured images of the subject between successive frames. The reference value setting unit 140 holds a lightness reference value in advance in association with the amount of movement of the subject, and sets a smaller reference value and a dark reference value when the movement of the subject is larger. Therefore, it is possible to reduce blurring of the subject due to the synthesis. The reference value setting unit 140 may set the reference value and the dark reference value based on the movement of the main subject when there are a plurality of subjects.

  FIG. 2 shows an example of image processing performed by the lightness adjustment unit 120. The vertical axis indicates the brightness of each frame image included in one moving image, and the horizontal axis indicates each frame of the moving image. In FIG. 2, the thick solid line indicates the brightness of the acquired image, the thin solid line indicates the brightness of the image after being combined by the combining means 110, and the broken line indicates the brightness of the image after being adjusted by the lightness adjusting means 120.

  In FIG. 2, the brightness of the frame image 2 is smaller than the reference value (10) because (9) is smaller than the reference value (10). When the frame image 2 immediately before the lightness is (10) is combined with the frame image 2 having the lightness (9), the lightness of the combined frame image 2 is H (19). The brightness suddenly increases from (10) to H (19). That is, the brightness of the first frame image synthesized is abruptly higher than the brightness of the frame image immediately before the synthesis is started.

  Therefore, the brightness adjustment unit 120 brings the brightness of the synthesized image closer to the brightness of an image captured before that image in order to make the change in brightness of the moving image continuous. That is, the lightness adjusting unit 120 reduces the discontinuity of the lightness change before and after the synthesis is started.

  For the first synthesized frame image, the brightness adjustment unit 120, for example, brings the brightness H (19) of the synthesized frame image 2 closer to the brightness (10) of the frame image 1 immediately before the frame image L ( Adjust (change) to 10). Further, the lightness adjusting means 120 uses the lightness of the frame image 3 whose lightness is I (17) after being combined as the lightness of the frame image 2 whose lightness has already been adjusted immediately before the frame image L (10 ) Is adjusted (changed) to M (12).

  The change in brightness becomes discontinuous before and after the synthesis is finished after the synthesis is started. Therefore, the brightness adjustment means 120 captures the brightness of an image that has not been synthesized in order to make the change in brightness of the moving image continuous, and the brightness of the synthesized image or the already adjusted image is captured. Move closer to lightness. That is, the lightness adjusting means 120 reduces the discontinuity of the lightness change before and after the synthesis is finished.

  For example, the lightness adjusting unit 120 brings the lightness F (10) of the first frame image 8 after the composition is not performed closer to the lightness K (18) of the composited frame image 7 immediately before the frame image N. Adjust (change) to (16). Further, the lightness adjusting means 120 adjusts the lightness G (11) of the frame image 9 to O (14) close to the lightness N (16) of the frame image 8 whose lightness has already been adjusted immediately before the frame image ( change.

  In order to make the change in brightness sufficiently continuous, it is preferable that the brightness adjustment amount by the brightness adjustment unit 120 gradually decreases over a period of a predetermined number of frames or more. Therefore, the lightness adjusting unit 120 determines a decrease amount of the lightness adjustment amount performed for each frame by dividing the lightness change amount due to the start or end of the synthesis by a predetermined number of frames.

  In FIG. 2, the brightness of the frame image 5 is smaller than the dark reference value. Therefore, the synthesizing unit 110 synthesizes the immediately preceding frame image 4 and the immediately following frame image 6 with the frame image 5 and adds the brightness of each image for each pixel.

  FIG. 3 shows an example of the processing flow of the moving image processing apparatus 20 according to an embodiment of the present invention. The moving image processing apparatus 20 reads a frame image recorded in advance (S200). Specifically, the target frame image and the frame images immediately before and immediately after the frame image are read out. The moving image processing apparatus 20 selects the immediately preceding and immediately following frame images with reference to the imaging time of each frame image.

The motion detector 130 detects the motion of the subject included in the target frame image (S210). Specifically, the movement of each subject is detected by comparing continuously captured images of the subject between the frame image to be synthesized and the immediately preceding or immediately following frame image.

The reference value setting unit 140 sets the reference value smaller when the detected movement is larger (S220). The reference value setting unit 140 holds a lightness reference value in advance in association with the amount of movement of the subject, and sets a smaller reference value and a dark reference value when the movement of the subject is larger. As a result, the subject is less likely to be blurred by being synthesized. Note that the reference value setting unit 140 sets the reference value and the dark reference value based on the movement of the main subject when there are a plurality of subjects.

  The brightness determination unit 100 determines whether the target frame image is darker than the dark reference value (S230). Specifically, the brightness of each pixel of the target frame image is added and divided by the total number of pixels to obtain an average value of brightness, and it is determined whether or not this average value is darker than the reference value.

  When it is determined that the target frame image is darker than the dark reference value (S230: Yes), the synthesizing unit 110 targets two images captured immediately before and after the target frame image. The frame image is synthesized (S250). The synthesizing unit 110 selects the immediately preceding and immediately following frame images with reference to the imaging time of each frame image. Then, the brightness of the image immediately before and after it is added to the brightness of the target frame image. Specifically, for each pixel, the synthesizing unit 110 adds the brightness value of the pixel in the immediately preceding and immediately following images to the brightness value of the pixel in the target image.

  On the other hand, when it is not determined that the target frame image is darker than the dark reference value (S230: No), the brightness determination unit 100 further determines whether the target frame image is darker than the reference value. Judgment is made (S240).

  When it is determined that the target frame image is darker than the reference value (S240: Yes), the synthesizing unit 110 targets the image captured at least one immediately before or immediately after the target frame image. The frame image is synthesized (S260). Specifically, since it is substantially the same as step 250, it is omitted.

  This makes random noise less noticeable when the captured image is dark. If the captured image is darker, the random noise is less noticeable.

  On the other hand, when it is not determined that the target frame image is darker than the reference value (S240: No), the combining unit 110 does not combine the other frame image with the target frame image.

The brightness adjustment unit 120 brings the brightness of the target frame image closer to the brightness of the already captured frame image (S270). This reduces the discontinuity of the change in brightness due to the synthesis. Specifically, the lightness adjusting unit 120 determines a decrease amount of the lightness adjustment amount to be performed for each frame by dividing the lightness change amount due to the start or end of the synthesis by a predetermined number of frames. Then, the moving image processing apparatus 20 causes the recording unit 170 to record the target frame image (S280).

  The moving image processing apparatus 20 determines whether the target frame image is the last frame image included in the moving image (S290). When the moving image processing apparatus 20 determines that this is the last frame image (S290: Yes), this processing flow ends. On the other hand, when the moving image processing apparatus 20 determines that this is not the last frame image (S290: No), the process returns to step 200. This makes random noise in each of the continuously captured frame images inconspicuous.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. Various modifications or improvements can be added to the above embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

FIG. 1 is a block diagram of a moving image capturing apparatus 10 and a moving image processing apparatus 20 according to an embodiment of the present invention. FIG. 2 shows an example of image processing performed by the lightness adjustment unit 120. An example of the flow of processing of the moving image processing apparatus 20 according to an embodiment of the present invention will be shown.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Imaging apparatus 20 Moving image processing apparatus 100 Brightness determination means 110 Composition means 120 Brightness adjustment means 130 Motion detection means 140 Reference value setting means 150 Still image acquisition means 160 Imaging means 170 Recording means 180 Display means

Claims (18)

An imaging device that captures a moving image by continuously capturing a plurality of images,
Brightness determination means for determining whether one of the plurality of images is darker than a reference value;
The one image, if it is determined that the darker than the reference value, an image captured at least one in the immediately preceding or immediately after the first image, and synthesizing means for synthesizing the first image,
An imaging apparatus comprising: brightness adjusting means for bringing the brightness of the image synthesized by the synthesizing means closer to the brightness of the image taken before the image so as to make the change in brightness of the moving image continuous. . Movement detecting means for detecting relative movement of a subject with respect to the imaging device;
Reference value setting means for setting the reference value smaller when the motion detected by the motion detection means is larger;
The imaging apparatus according to claim 1, further comprising: The synthesizing unit synthesizes two images captured immediately before and after the image into the one image when the one image is darker than a dark reference value smaller than the reference value. The imaging device according to claim 1 or 2. Before SL synthesizing means, the imaging apparatus according to each of the continuously captured images, to Claim 3 for synthesizing an image captured immediately before or at least one of immediately after the image. The brightness adjustment means further includes means for bringing an image obtained by synthesizing the two images closer to the brightness of an image captured before the image in order to make the change in brightness of the moving image continuous. 5. The imaging device according to 4. A still image acquisition means for acquiring a still image while capturing the moving image ;
The determination unit determines whether the still image acquired by the still image acquisition unit is darker than a reference value;
The combining means, when the still image is darker than the reference value, wherein the image in the moving image captured at least one in the immediately preceding or immediately after the still image, from claim 1 to synthesize the still image Item 6. The imaging device according to any one of Items 5 . An imaging method for capturing a moving image by continuously capturing a plurality of images,
A brightness determination step of determining whether one of the plurality of images is darker than a reference value;
The one image, if it is determined that the darker than the reference value, the image captured immediately before or at least one of immediately after the first image, a synthesis step of combining the first image
An imaging method comprising: a brightness adjustment step of bringing the brightness of the image synthesized by the synthesis step closer to the brightness of an image taken before the image in order to make the change in brightness of the moving image continuous. . A motion detection step of detecting a relative motion of a subject included in the moving image;
A reference value setting step of setting the reference value smaller when the motion detected in the motion detection step is larger;
The imaging method according to claim 7, further comprising: The combining step combines two images captured immediately before and after the image into the one image when the one image is darker than a dark reference value smaller than the reference value. The imaging method according to claim 7 or 8 . An image processing device for increasing the brightness of an image,
Brightness determination means for determining whether one of the continuously captured moving images is darker than a reference value;
When it is determined that the one image is darker than the reference value, the one image is synthesized by combining an image captured at least one of the one image immediately before and after the one image with the one image. A synthesis means to increase the brightness of
Image processing comprising: brightness adjusting means for bringing the brightness of the image synthesized by the synthesizing means closer to the brightness of an image captured before the image so as to make the change in brightness of the moving image continuous. apparatus. Movement detection means for detecting movement of a subject included in the moving image ;
The image processing apparatus according to claim 10 , further comprising reference value setting means for setting the reference value to be smaller when the motion detected by the motion detection means is larger. The synthesizing unit synthesizes two images captured immediately before and after the image into the one image when the one image is darker than a dark reference value smaller than the reference value. Item 12. The image processing device according to item 10 or item 11 . The combining means, each of the images picked up continuously, the image processing apparatus according to claim 12 for synthesizing the image captured immediately before or at least one of immediately after the image. The brightness adjustment means further includes means for bringing an image obtained by synthesizing the two images closer to the brightness of an image captured before the image in order to make the change in brightness of the moving image continuous. The image processing apparatus according to 13 . A still image acquisition means for acquiring a still image from the moving image ;
The determining means determines whether the still image is darker than a reference value;
The combining means, when the still image is darker than the reference value, wherein the image in the moving image captured immediately before the still image or at least one of immediately claims 10 to synthesize the still image Item 15. The image processing apparatus according to any one of Items 14 . An image processing method for increasing the brightness of an image,
A brightness determination step of determining whether one of the continuously captured moving images is darker than a reference value;
When it is determined that the one image is darker than the reference value, the one image is synthesized by combining an image captured at least one of the one image immediately before and after the one image with the one image. a synthesizing step of increasing the brightness,
A brightness adjustment step of bringing the brightness of the image synthesized by the synthesis step closer to the brightness of an image captured before the image in order to continuously change the brightness of the moving image. Method. A motion detection step of detecting a relative motion of a subject included in the moving image;
A reference value setting step of setting the reference value smaller when the motion detected in the motion detection step is larger;
The image processing method according to claim 16, further comprising: In the combining step, when the one image is darker than a dark reference value smaller than the reference value, the two images captured immediately before and after the image are combined with the one image.
The image processing method according to claim 16 or claim 17.

Images