JP2018074363A - Image processing apparatus and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve such a problem of conventional method that since the number of interpolation images is determined according to the time width of unexposed section, the unexposed section becomes longer as the exposed section becomes shorter, and thereby the number of interpolation images increases thus causing increase in processing load.SOLUTION: An image processing apparatus for generating a composite image, corresponding to an image obtained by shooting of second exposure time longer than first exposure time, by synthesizing multiple images obtained by continuous shooting of first exposure time, includes synthesis means for synthesizing multiple images, interpolation means for interpolating image information, corresponding to the unexposed section between shooting, according to movement of a subject in the synthesis by the synthesis means, and control means for controlling the processing content of interpolation means by the interpolation means according to the characteristics of the subject.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a technique for artificially generating a long second exposure image by combining a plurality of short second exposure images.

  When an image that has been shot with long-second exposure (long-second exposure image) is realized by compositing multiple short-second exposure images, there is an interval (non-exposure interval) during which exposure is not performed during shooting of short-second shooting. Therefore, when the short second captured image is synthesized as it is, a non-exposure section remains. Therefore, for example, when a tail lamp of a car is photographed, a rough image in which the trajectory of the tail lamp light is partially interrupted corresponding to the non-exposure section is generated.

  Patent Document 1 determines the number of interpolated images for interpolating a non-exposure section from the time width of the non-exposure section when combining a plurality of images captured at a predetermined time interval, and interpolates the captured image and the interpolated image. The image is synthesized.

International Publication No. 2009/098749

  When an image that interpolates the non-exposure section is generated and combined, there is a problem that the number of images that interpolate the non-exposure section increases and the processing takes time as the moving amount of the moving object to be photographed increases. Since Patent Document 1 determines the number of interpolated images according to the time width of the non-exposure section, the non-exposure section becomes longer as the exposed section (exposure section) becomes shorter. The number of sheets increases and the processing load increases.

  In one embodiment of the present invention, by combining a plurality of images obtained by continuously performing photographing at the first exposure time, an image obtained by photographing at a second exposure time longer than the first exposure time is obtained. An image processing apparatus for generating a corresponding composite image, wherein the plurality of images are combined, and image information corresponding to a non-exposure section between shots in the combination by the image unit is interpolated according to the movement of the subject And an interpolating unit for controlling the processing contents of the interpolation by the interpolating unit according to the characteristics of the subject.

  According to the present invention, the interpolation process for interpolating the image information corresponding to the non-exposure section is performed according to the characteristics of the subject, so that an increase in the load of the interpolation process can be suppressed.

It is a figure which shows the concept of embodiment of this invention. It is a figure which shows the structure of 1st Embodiment. It is a flowchart which shows the process in 1st Embodiment. It is a figure explaining the process of the mobile body detection in 1st Embodiment. It is a figure explaining the interpolation process in 1st Embodiment. It is a figure explaining the process which calculates the adjustment parameter used by the interpolation process in 1st Embodiment. It is a figure explaining the production | generation process and filter process of the interpolation image in 1st Embodiment.

(First embodiment)
Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that. The description will be made using a digital camera. However, the present invention is not limited to this, and any image processing apparatus that can process image data may be used. That is, the image processing apparatus may be an information processing apparatus such as a personal computer, or an image forming apparatus such as a portable information terminal or a printer.

  In the present embodiment, alignment is performed on a plurality of images obtained by continuously capturing images, and the moving object information obtained by detecting the moving object from the image after alignment is used in the interpolation process. A parameter is calculated to generate a non-exposure section. A long-second exposure image (synthesized image) is generated by synthesizing the image information corresponding to the non-exposure section and the captured image.

  First, the non-exposure section used here will be described with reference to FIG. FIG. 1 is a diagram showing a photographing time and a photographing interval at the time of photographing, S0 is a photographing start position, S1 to S7 are photographing intervals when a plurality of continuous shots are taken in a short time, and 101 is an exposure time at the time of long-time photographing. , 102 indicates an exposure interval during short-second shooting, and 103 indicates an unexposed interval during short-second shooting.

  In the exposure section when the long exposure is performed, the shutter is kept open during the imaging time, so that shooting is performed without interruption as in 101. On the other hand, in the processing assumed in this case, two or more images are shot with an exposure time such as 102, which is shorter than the exposure time for shooting with a long exposure. For this reason, a time during which the shutter is not released occurs, and a section 103 in which exposure is not performed occurs during that period. Therefore, when a plurality of captured images are combined, an unexposed section (non-exposure section) ), And results are different from those obtained when shooting with long-second exposure. Therefore, in the present embodiment, the non-exposure section is generated by interpolation processing using the captured image and the moving object detection result, thereby generating an image close to actual long-second exposure.

  FIG. 2 is a system diagram of the image processing apparatus according to the present embodiment.

  In FIG. 2, reference numeral 201 denotes a captured image acquisition unit that acquires two or more captured images used in the main image processing.

  Reference numeral 202 denotes a moving body detection unit that detects a moving body from an image using two or more pieces of captured image information.

  Reference numeral 203 denotes an adjustment parameter calculation unit that calculates parameters used in the interpolation processing unit based on the moving body information input from the moving body detection unit 202. That is, the adjustment parameter calculation unit has a function as a control unit that controls the processing content of the interpolation processing by the interpolation processing unit.

  Reference numeral 204 denotes an interpolation processing unit that performs interpolation processing for generating a non-exposure section portion from two or more images captured by the captured image acquisition unit 201 using the parameter information input from the adjustment parameter calculation unit 204.

  A combining unit 205 combines the non-exposure section portion generated by the interpolation processing unit 204 and two or more captured images acquired from the captured image acquisition unit 201 to generate a long-second exposure image.

  FIG. 3 is a flowchart of the above operation according to the flow of processing, and details of the processing will be described with reference to FIGS.

  In step S <b> 301, two or more captured images used in the image processing apparatus are acquired by the captured image acquisition unit 201 and input to the moving body detection unit 102 for a scene that is desired to be exposed for a long second.

  In step S <b> 302, alignment is performed on two or more captured images input from the captured image acquisition unit 201 in order to absorb camera shake when handheld. First, one image serving as a reference is selected from the captured images, and a conversion coefficient is calculated in order to align the position of the reference image with other images. Specifically, a projective transformation equation is calculated using a combination of four or more feature points corresponding between images. Then, the image is deformed using the calculated projective transformation formula, and an image that has been aligned with the reference image is output to the moving object detection unit 202.

  In S303, the moving body is detected from the captured image using two or more aligned images. A moving body detection process is demonstrated using FIG. FIG. 4 shows an N-th frame captured image 401, an (N + 1) -th frame captured image 411, an N-th frame captured block 421, and a moving object detection result 431 of two or more images captured by the captured image acquisition unit 201. Yes.

First, the captured image 401 and the captured image 411 are divided into blocks of an arbitrary size (M × N) (421), and a correlation value is calculated as a difference between the regions. In order to calculate a correlation value between images, a sum of absolute difference (SAD) is calculated for an evaluation frame in an arbitrary range centered on the pixel of interest, and an absolute value difference of data values in the evaluation frame is integrated. The Specifically, when a pair of data values in the evaluation frame is expressed as f (i, j) and g (i, j) (i and j are coordinates), the data value is expressed by the following equation (1). _Is calculated as an evaluation value R _SAD .

ｆ（ｉ，ｊ）：画像４０１の評価枠内の画素値
ｇ（ｉ，ｊ）：画像４０２の評価枠内の画素値
Ｒ_ＳＡＤ：相関値

f (i, j): Pixel value in the evaluation frame of the image 401 g (i, j): Pixel value in the evaluation frame of the image 402 R _SAD : Correlation value

  In the present embodiment, an example of a method for calculating an evaluation value using SAD that is an absolute difference value of pixels has been described, but the present invention is not limited to this, and other methods may be used.

  Since the portion where the calculated evaluation value of the difference becomes smaller is a similar subject, the region is searched, and the motion vector of the moving object is calculated from the positional relationship of the block (404). In addition to the motion vector information, the moving body detection unit 202 also acquires moving body size information and contrast information.

  First, as the size information of the moving object, the number of pixels in a region having similar pixel values is calculated and set as the size of the region that is a candidate for the moving object. Here, a clustering process is used to calculate a region that is a candidate for a moving object. That is, this is a method of analyzing the distribution of luminance and color information, determining the similarity of adjacent blocks, and determining that they are the same block if they are determined to be similar.

  Using this method, the same cluster is determined as one region that is a candidate for a moving object. Note that these methods are merely examples, and the present embodiment is not limited to these methods. And the average value of the motion vector of the block contained in this mobile body area | region is linked | related as the motion vector v (431) of a mobile body.

  Next, edge strength characteristics in the subject are acquired as contrast information of the moving body. In this method, an edge signal is detected by applying a band pass filter to an image, and an integral value of an edge signal having an arbitrary block size is calculated. The calculated evaluation value is large when the integral value of the edge signal is large, that is, when there are many textures. This evaluation value is calculated using the following formula (2).

ｆｒ（ｉ，ｊ）：任意のブロックサイズ内の各座標におけるエッジ強度値
Ｆｒ：任意のブロックサイズにおけるエッジ強度の積分値

fr (i, j): edge strength value at each coordinate in an arbitrary block size Fr: integrated value of edge strength at an arbitrary block size

  In step S304, parameters used by the interpolation processing unit 204 are calculated based on the information on the moving body detected by the moving body detection unit 202 and the camera information used to capture the image.

  In the present embodiment, in order to interpolate a non-exposure section between two or more captured images, generation of an interpolation image and processing for applying a filter having directionality to the moving direction of the moving body are performed as interpolation processing.

  FIG. 5 shows an interpolation process, and an interpolated image corresponding to a non-exposure section is generated using N frames and N + 1 frames. Then, filter processing having directionality is performed on the captured image and the interpolated image. Specifically, a filter having directionality is applied to the N frame 501 and the interpolation image 502, the interpolation images 502 and 503, the interpolation images 503 and 504, and the interpolation image 504 and the N + 1 frame 505. The number of interpolation images and the number of filter taps used in this process are calculated as adjustment parameters.

  The adjustment parameter calculation process will be described with reference to FIG. FIG. 6 is graphs 601 and 602 showing the number of filter taps to be used according to the information of the moving body. The horizontal axis of the graph represents each piece of information, the vertical axis represents the number of filter taps, and the priority weight of each piece of information 603. It is represented by The priority weight used here is a value expressed in the range of 0 to 1, and a priority value is assigned to the result of the number of taps in each information.

  As the moving object information, the moving object characteristics such as the motion vector of the moving object, the size of the moving object, and contrast information are acquired from the moving object detection unit 202. First, the movement amount v from the captured image N frame to the N + 1 frame is used as a motion vector. An interpolation image for compensating for the non-exposure section is determined according to the moving amount v of the moving body.

  Next, the number of taps for filtering is calculated. The number of taps of the filter is calculated with the upper limit of the number of taps that can be processed in consideration of the memory amount of the camera. The number of interpolated images generated may be adjusted in consideration of the number of filter taps.

  Next, processing for determining the number of filter taps according to the acquired size of the moving body and contrast information will be described with reference to FIG.

  First, a case where contrast information is acquired will be described.

  Using the contrast evaluation value detected in step S303, the number of filter taps is determined using the graph 601. Here, when the subject has many textures, that is, when the contrast evaluation value is higher than the predetermined value, it is desirable to perform an interpolation process that leaves a texture on the subject. Therefore, priority is given to the process of generating an interpolated image over the filter process. Raise the degree.

  Accordingly, the number of filter taps is set to a small value so that the number of interpolation images becomes a parameter. On the other hand, if the subject has few textures, that is, if the contrast evaluation value is lower than the predetermined value, the number of interpolated images is set to increase the priority of the filtering process rather than generating an interpolated image for that subject. Is set to a value that increases the number of filter taps.

  Next, a case where the size information of the moving body is obtained will be described. Using the value of the size of the area detected in S303, the number of filter taps is determined using the graph 602. When the area of the moving object is larger than the predetermined value, the moving object becomes a conspicuous area, so that the priority of the process for generating the interpolation image that leaves the texture information of the subject is increased. Accordingly, the number of filter taps is set to a small value so that the number of interpolation images becomes a parameter.

  On the other hand, when the area of the moving object is smaller than the predetermined value, the moving object becomes an inconspicuous area, and the number of interpolated images is reduced in order to increase the priority of filter processing for the subject. It is set at a value that increases the number of taps of the filter.

  Although the calculation of the number of taps of the filter for each piece of information related to the acquired subject characteristic has been described, the number of taps of the filter to be finally used is determined in consideration of the number of taps for each piece of information and the priority for each piece of information. As a calculation method, the number of taps T corresponding to each information is calculated using the following equation (3).

Here, α _n represents the weight set for the priority for each information, and n represents the number of pieces of information acquired. As α _n, as shown in Table 603, since the presence / absence of the texture of the object, which is the contrast information of the object, is important for determining the number of taps of the filter, the weight is increased. These weights may be specified by the user.

ａ_ｎ：情報毎に決定した優先度

a _n Priority determined for each information

Next, the number of interpolated images used in the interpolation process is calculated. As a calculation method, the following equation (4) is used to determine the number M of interpolation images in consideration of the calculated number of filter taps based on the moving amount of the moving body.
M = v ÷ T (4)
v: detected motion vector

  In step S305, interpolation processing for generating a non-exposure section portion in the captured image is performed using the calculated parameters. The interpolation process will be described with reference to FIG. FIG. 7 shows the relationship between the number of interpolations used when generating an interpolated image and the motion vector 701, the motion vector after applying the gain used for the interpolated image 711, and the filter processing 721 using the motion vector. Is shown.

  In the interpolation processing, in order to interpolate a non-exposure section between two or more captured images, processing for generating an interpolation image and applying a filter having directionality is performed. First, interpolation images are generated in N frames and N + 1 frames of the captured image. In the generation of this interpolation image, information acquired from the moving object detection unit 202 and the adjustment parameter calculation unit 203 is used.

  The number M of interpolated images to be generated as adjustment parameters is acquired from the adjustment parameter calculation unit 203. Then, an interpolation image is generated so that M images as interpolation images between N frames and N + 1 frames generated as in 701 are equally spaced in time. In 701, n indicates the position of the interpolation image to be generated, and the first to Mth images are generated. First, a gain corresponding to the position of the interpolation image generated using the following equation (5) is applied to the motion vector of the mobile object acquired as the mobile object information.

Ｖｍ：生成する補間画像位置に対応する動きベクトル
ｖ：検出した動きベクトル
ｍ：補間画像を生成する位置（１≦ｍ≦Ｍ）

Vm: motion vector corresponding to the position of the interpolated image to be generated v: detected motion vector m: position where the interpolated image is generated (1 ≦ m ≦ M)

  For example, when an interpolation image is generated at a position of n = 1, a gain of 1 / (M + 1) is applied to the motion vector. Then, using the motion vector multiplied by the gain, an interpolation image is generated so that the moving body of N frames comes to the position indicated by the calculated motion vector. The interpolated image is obtained by moving the moving object 712 by the motion vector v / (M + 1). Then, the process is finally performed up to the Mth sheet, and a necessary number of interpolated images are generated.

  Next, a filter corresponding to the moving direction of the motion vector is applied to the captured image and the interpolated image as a filtering process having directionality for the captured N frame and N + 1 frame images and the generated M interpolated images. Call against. For example, when the filter processing is performed on two images of the captured image N frame indicated by 721 and the interpolation image (n = 1), the gain used when the interpolation image (n = 1) is generated is applied. Then, a filter is generated using the motion vector v / (M + 1).

  In this filtering process, a filtering process is performed between two frames based on the motion vector of the subject. Here, based on the motion vector, an interpolation image (n = 1) -th image is processed for the pixel value of the necessary number of taps T using the following equation (6) in the opposite direction to the motion vector.

Ｉｎ：ｎフレーム目の画像データ
α：重み係数

In: nth frame image data α: weighting coefficient

  By using Expression (6), filter processing with a degree corresponding to the vector is performed on the direction based on the motion vector. This filtering process is an example and is not limited to this method.

  In step S306, a long-second exposure image is generated using the captured image acquired by the captured image acquisition unit 101 and image information that interpolates the non-exposure section portion generated by the above processing. As a synthesis method, weighted synthesis of pixel values at the same pixel position is performed. In the weighted composition, the weight is changed according to the luminance value of the image. For example, a long-second exposure image is generated by performing a combining process with a weight of 1 / R for each pixel, using the number of sheets R to be finally combined for each pixel at the same position. .

  In this embodiment, the number of interpolated images is determined using the motion vector from the N frame to the N + 1 frame in S304. However, the present invention is not limited to this, and the number of interpolated images is determined based on the memory capacity of the device used. The maximum number may be determined. Specifically, since the number of images that can be stored is determined from the capacity of the interpolation image to be generated, the maximum number of images is recorded, and the interpolation that is finally generated by using the number of filter taps calculated in S304 in this embodiment. Determine the number of images.

In the present embodiment, the number of interpolated images is determined using the motion vector from the N frame to the N + 1 frame in S304. However, the present invention is not limited to this, and the number is determined based on the shooting interval and the shutter speed. You may do it. Specifically, the ratio of the shooting shutter speed (exposure section: Tv) in the shooting interval time (Tv _fps ) is calculated.

  Among them, the ratio of the non-exposure section can be calculated from 1-P. Then, the number of interpolation images to be generated is calculated using each ratio.

  Next, the number of filter taps calculated in S304 in this embodiment is calculated, and the number of interpolated images and the number of filter taps calculated in S304 are output to the interpolation processing unit 104.

  In the present embodiment, the weighted synthesis method is used in the synthesis process of S306. However, a method of generating a long second exposure image by performing comparatively bright synthesis on the target image may be performed. Specifically, the longest exposure image is generated by selecting the brightest pixel value with respect to the target pixel position from the captured image and the generated non-exposure section and storing the selected pixel value in the pixel position.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 101 Captured image acquisition part 102 Moving body detection part 103 Adjustment parameter calculation part 104 Interpolation process part 105 Compositing process part

Claims (7)

An image that generates a composite image corresponding to an image obtained by shooting at a second exposure time longer than the first exposure time by combining a plurality of images obtained by continuously performing shooting at the first exposure time. A processing device comprising:
Combining means for combining the plurality of images;
Interpolating means for interpolating image information corresponding to a non-exposure section between photographing in the composition by the image means according to the movement of the subject;
An image processing apparatus comprising: control means for controlling processing contents of interpolation by the interpolation means in accordance with characteristics of the subject.   The interpolating unit generates an interpolated image for interpolating between the plurality of images, and performs a filtering process on the plurality of images and the interpolated image in a moving direction of a subject. Image processing apparatus.   The image processing apparatus according to claim 2, wherein the control unit determines the number of generated interpolated images according to the amount of movement of the subject.   The image processing apparatus according to claim 2, wherein the control unit controls the number of generated interpolation images and the number of taps used for the filter processing.   The image processing apparatus according to claim 4, wherein the control unit performs control so that the number of generated interpolated images is prioritized when the size of the subject is larger than a predetermined value.   6. The control unit according to claim 4, wherein when the evaluation value of the contrast of the subject is greater than a predetermined value, the control unit performs control so that the number of generated interpolated images is prioritized. Image processing device. An image that generates a composite image corresponding to an image obtained by shooting at a second exposure time longer than the first exposure time by combining a plurality of images obtained by continuously performing shooting at the first exposure time. A processing method,
A combining step of combining the plurality of images;
An interpolation step of interpolating image information corresponding to a non-exposure section between photographing in the combining step according to the movement of the subject;
And a control step for controlling the processing content of the interpolation in the interpolation step according to the characteristics of the subject.

Images