JP2010193180A - Image processing unit, and image processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing unit capable of improving image quality by expanding a dynamic range even if positional misalignment occurs in a plurality of images when obtaining a composite image by capturing the plurality of images. <P>SOLUTION: A plurality of video signals, where an exposure value and a focal length to a main subject are different each, are obtained. A low-frequency component is extracted from the plurality of video signals to generate a plurality of low-frequency video signals, and the plurality of low-frequency video signals are composited to generate a composite low-frequency video signal. Also, a first video signal, which has a first exposure value and focuses on the main subject, is selected from the plurality of video signals, and a high-frequency component is extracted from the first video signal to generate a high-frequency video signal. Then, the composite low-frequency video signal is composited to the high-frequency video signal, thus generating a composite video signal. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing apparatus that expands the dynamic range of a video signal.

  In general, the dynamic range of various solid-state image sensors such as CCD image sensors and CMOS image sensors used in electronic cameras is narrower than the dynamic range of the subject. In some cases, black crushing may occur. For this reason, there has been proposed a method of generating an image with an expanded dynamic range by combining a plurality of video signals having different exposure amounts into one video signal. However, when a composite image is obtained by photographing a plurality of images, there has been a problem that artifacts such as a double image occur in the video signal at the time of synthesis due to camera shake or subject blur.

  In Japanese Patent Laid-Open No. 7-75026 (Patent Document 1), the weight of the first lightness value and the second lightness value of two images is determined based on either one of them, thereby changing the characteristics of the camera. A technique for generating an appropriate wide dynamic range image even when there is a variation in illumination intensity and a positional shift on the image sensor is disclosed. Japanese Patent Application Laid-Open No. 5-7336 (Patent Document 2) discloses a method of performing composition after performing coordinate conversion on a plurality of images in response to positional shifts at the time of imaging a plurality of images. ing.

Japanese Patent Laid-Open No. 7-75026 JP-A-5-7336

  However, in any of the methods of Patent Document 1 and Patent Document 2 described above, complete alignment is performed with respect to a complicated positional shift between a plurality of video signals, for example, a positional shift caused by a three-dimensional motion. It is difficult.

  The present invention has been made in view of the above problems, and in the case where a composite image is obtained by capturing a plurality of images, for example, when there is a positional shift due to a three-dimensional motion in the plurality of images. Even so, an object of the present invention is to provide an image processing apparatus capable of expanding the dynamic range and improving the image quality.

In order to solve the above problems, the present invention employs the following means.
The present invention is an image processing device that expands a dynamic range by combining a plurality of video signals, a video signal acquisition unit that acquires a plurality of video signals having different exposure amounts and focal lengths from a main subject, and Low frequency video signal generating means for extracting low frequency components from the plurality of video signals to generate a plurality of low frequency video signals, and synthesis for synthesizing the plurality of low frequency video signals to generate a combined low frequency video signal A low-frequency signal generation means; a selection means for selecting a first video signal having a first exposure amount and focused on the main subject from the plurality of video signals; and a high-frequency component is extracted from the first video signal And a high frequency video signal generating means for generating a high frequency video signal, and an image synthesizing means for generating a composite video signal by synthesizing the synthesized low frequency video signal and the high frequency video signal. To provide an image processing apparatus according to claim and.

  Further, the present invention is an image processing program for expanding a dynamic range by synthesizing a plurality of video signals, the step of acquiring a plurality of video signals having different exposure amounts and focal lengths from a main subject, Extracting a low frequency component from a plurality of video signals to generate a plurality of low frequency video signals; synthesizing the plurality of low frequency video signals to generate a combined low frequency video signal; and the plurality of videos Selecting a first video signal having a first exposure amount and focusing on a main subject from the signal, extracting a high frequency component from the first video signal, and generating a high frequency video signal; An image processing program for causing a computer to execute a step of generating a synthesized video signal by synthesizing a synthesized low frequency video signal and the high frequency video signal is provided. That.

  As described above, the low-frequency component in which the positional deviation is less noticeable in the video signal is the target of the image synthesis process for a plurality of images, while only one arbitrary image is taken for the high-frequency component in which the positional deviation is noticeable. By making it a subject of synthesis processing, it is possible to generate a synthesized signal having a wide dynamic range free from artifacts such as double lines even for a plurality of blurred video signals.

1 is a block diagram illustrating a schematic configuration of an image processing apparatus according to an embodiment of the present invention. It is a block diagram which shows schematic structure of the low frequency component production | generation part of the image processing apparatus which concerns on embodiment of this invention. It is explanatory drawing regarding the weighting coefficient in the image processing apparatus which concerns on embodiment of this invention. It is a block diagram which shows schematic structure of the high frequency component production | generation part of the image processing apparatus which concerns on embodiment of this invention. It is a block diagram which shows schematic structure of the image synthetic | combination part of the image processing apparatus which concerns on embodiment of this invention. It is a flowchart which shows the process of the image synthesis process in the image processing apparatus which concerns on embodiment of this invention.

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing a schematic configuration of an image processing apparatus according to an embodiment of the present invention.
As shown in FIG. 1, the image processing apparatus according to the present embodiment includes a lens system 100, an aperture 101, an AF motor 102, a color filter 103, a CCD 104, an A / D conversion unit 105, a buffer 106, an imaging control unit 107, An image processing unit 120, a compression unit 115, an output unit 116, a control unit 117, and an external I / F unit 118 are provided.

  The lens 100 system is arranged so as to form an image of a subject through a diaphragm 101 and form an image on a CCD 104 which is an image pickup device through a color filter 103. The AF motor 102 is driven and controlled by a shooting control unit 107 described later, and is driven to focus the lens system 100 on the subject based on a control signal from the shooting control unit 107. The CCD 104 generates electrical image information based on the subject image formed by the lens system 100, and outputs this electrical image information to the A / D conversion unit 105. In this embodiment, the description will be made assuming that the CCD 104 uses an RGB primary color single-plate CCD. The A / D conversion unit 105 converts the image information generated by the CCD 104 into a video signal as discrete digital data that can be subjected to predetermined processing in the image processing unit 120, and temporarily converts the converted video signal into the buffer 106. And output from the buffer 106 to the image processing unit 120. The compression unit 115 compresses the video signal that has been subjected to predetermined processing by the image processing unit 120 described later, and outputs the compressed video signal to the output unit 116.

  The shooting control unit 107 determines a focusing condition and an exposure condition at the time of shooting, and information at the time of shooting is output to the control unit 117. The contrast information in the AF area of the video signal is detected, so that the contrast is maximized or the video signal is not acquired at the time of pre-photographing, for example, the distance from the main subject is determined using an external infrared sensor (not shown). The AF motor 102 is driven and controlled according to this distance. Further, the photographing control unit 107 controls the electronic shutter speeds of the diaphragm 101 and the CCD 104 for adjusting the amount of incident light using a luminance level in the video signal or a luminance sensor (not shown). The control by the photographing control unit 107 is performed as many times as the number of times of photographing in actual photographing. That is, the above-described processing is performed a plurality of times, and video signals of a plurality of captured images are temporarily stored in the buffer 106, and the video signals are sequentially output from the buffer 106 to the image processing unit 120 one by one.

  The photographing control unit 107 also functions as a video acquisition unit. When the appropriate exposure amount is defined as B and the distance to the main subject is defined as D, the exposure signal is B and the video signal S1 is focused on the distance D. The lens system 100, the aperture 101, the AF motor 102, and the like are acquired so that at least one video signal S2 that is different from the exposure amount B and is not focused on the main subject, that is, a blurred video signal, is acquired. Control.

  The control unit 117 includes a shooting control unit 117, a signal processing unit 108 of the image processing unit 120 described later, a conversion unit 109, a low frequency component generation unit 110, a selection unit 111, a high frequency component generation unit 112, an image synthesis unit 113, a floor. The key conversion unit 114 and the compression unit 115 are bidirectionally connected to drive and control each of them. The external I / F unit 118 includes a power switch, a shutter button, and an interface for switching various modes during shooting.

  The image processing unit 120 includes a signal processing unit 108, a conversion unit 109 as a low frequency video signal generation unit, a low frequency component generation unit 110 as a combined low frequency video signal generation unit, a selection unit 111 as a selection unit, and a high frequency video signal. A high frequency component generation unit 112 as a generation unit, an image synthesis unit 113 as an image synthesis unit, and a gradation conversion unit 114 are provided.

  The signal processing unit 108 reads a single-panel video signal input from the buffer 106 based on the control of the control unit 117, and performs predetermined image processing such as interpolation processing, white balance adjustment processing, electronic zoom processing, and noise suppression processing. To generate a three-plate video signal of each pixel RGB and output it to the conversion unit 109. Further, the RGB signal may be converted into a YCbCr signal using the following equation (1).

  The conversion unit 109 acquires information on the focus positions of a plurality of video signals from the imaging control unit 117, performs a conversion process such as a smoothing filter only on the video signal focused on the main subject, and the low-frequency video signal. Is generated. On the other hand, no processing is performed for a focused video signal other than the main subject, that is, a blurred video signal. In the present embodiment, conversion processing is performed on the video signal S1 to generate a low-frequency video signal of S1, and processing is not performed on the video signal S2, and the low-frequency component generation unit 110 and high-frequency component generation are performed. Forward to the unit 112.

  The low frequency component generation unit 110 generates a synthesized low frequency video signal by synthesizing the low frequency video signal input from the conversion unit 109. For this reason, the low frequency component generation unit 110 includes a buffer 300, a weighting coefficient calculation unit 301, and an addition unit 302 as shown in FIG.

  The buffer 300 stores a plurality of video signals transferred from the conversion unit 109. In the present embodiment, two video signals stored in the buffer 300 are a low-frequency video signal S1 and a blurred video signal S2. The adder 302 synthesizes signal values of the same coordinates with a plurality of video signals transferred from the buffer 300. The combining method may be simple addition, or may be added using the weighting coefficient α transferred from the weighting coefficient calculation unit 301 as shown in the following equation (2).

Here, I represents the combined video signal, and (x, y) represents the coordinates of the video signal. Further, I ₁ is one of the video signal S1 focused on the main subject in the proper exposure, I ₂ respectively correspond to the video signal S2 and the incident light quantity and the focal length is different. The video signal to be added may be an RGB signal channel or a YCbCr signal. The weighting coefficient calculator 303 calculates the weighting coefficient α as shown in FIG. The weighting coefficient α is expressed as a function with respect to the video signal S1. The function in FIG. 3 shows the case where the video signal S2 has a smaller exposure amount than the video signal S1. That is, the weight of the video signal S2 that is captured darkly is increased for a signal value larger than the threshold Th that may cause the video signal to be saturated, and the signal value that is equal to or less than the threshold Th is captured with appropriate exposure. The weight of the video signal S1 is increased. By using such a weighting factor, it is possible to add in consideration of the saturated portion of the signal. Further, the video signals before synthesis are all blurred video signals, that is, low-frequency video signals. The added low frequency video signal is transferred to the image composition unit 113.

  The selection unit 111 selects one video signal focused on the main subject by appropriate exposure using the information of the imaging control unit from among a plurality of video signals transferred from the signal processing unit 108.

  The high frequency component generation unit 112 generates a high frequency video signal by subtracting the low frequency video signal from one video signal focused on the main subject by the appropriate exposure selected by the selection unit 111. FIG. As shown in FIG. 4, buffers 400 and 401 and a differential signal calculation unit 402 are provided. The buffer 400 stores a plurality of low frequency video signals input from the conversion unit 109. The buffer 401 stores one image signal input by the selection unit 111. The difference signal calculation unit 402 reads out the low frequency video signal from the buffer 400 based on the control by the control unit 117, and subtracts the low frequency video signal from one video signal in the buffer 401. A high-frequency video signal is generated by performing difference processing at each coordinate (x, y) as represented by:

  Here, H (x, y) is a high-frequency video signal after difference processing, S1 is a single video signal S1, S1 ′ focused on the main subject by proper exposure input by the selection unit 111, and the selection unit 111 A low-frequency video signal corresponding to the extracted video signal S1 is shown.

  The image synthesizing unit 113 adds the signal values of the same coordinates to the high frequency video signal of the high frequency component generating unit 112 and the synthesized low frequency video signal synthesized by the low frequency component generating unit 111, and the gradation converting unit 114 Forward to. Since the high-frequency video signal may contain noise, it is preferable to perform addition processing after noise reduction. Therefore, the image composition unit 113 includes buffers 500 and 501, a noise reduction unit 502, and an addition unit 503 as shown in FIG. The buffer 500 temporarily stores the high frequency video signal transferred from the high frequency component generation unit 112 and transfers the high frequency video signal to the noise reduction unit 502. The noise reduction unit 502 reduces noise using a smoothing filter, a median filter, or the like for the high-frequency video signal. The adder 503 performs addition processing at the same coordinates of the high-frequency video signal after noise reduction and the synthesized low-frequency video signal synthesized from the buffer 501, generates a synthesized video signal, and transfers it to the gradation converter 114. To do.

  The tone conversion unit 114 performs gamma conversion and tone conversion processing using a tone curve for the synthesized video signal transferred from the image synthesis unit 113, and transfers it to the compression unit 115.

  Hereinafter, the process of image composition processing in the image processing apparatus of the present embodiment configured as described above will be described with reference to the flowchart of FIG.

  Hereinafter, for convenience of explanation, it is assumed that the number of times of shooting is two, that is, the video signal is acquired as two video signals S1 and S2. Further, an appropriate exposure amount in the main imaging is defined as B, a distance to the main subject is defined as D, S1 is a video signal focused on the distance D with the exposure amount B, and S2 is an exposure amount of B. A half-way video signal focused on an infinite distance, that is, a blurred video signal in which the main subject is not focused.

  The in-focus position for acquiring the video signal S2 may be anywhere as long as it is far from the main subject, and the in-focus position may be adjusted to the shortest imaging distance of the imaging system by controlling the AF motor 102. Alternatively, the image signal S2 may be imaged by opening the aperture 101 compared to that at the time of proper exposure and by increasing the shutter speed of a mechanical shutter or electronic shutter (not shown) as compared with that at the time of proper exposure. Information at the time of each imaging is transferred to the control unit 117. In the following description, a plurality of video signals to be imaged are described as two images S1 and S2. In addition, instead of the apparatus automatically taking a plurality of images at the time of actual imaging, the photographer may control the aperture 101 and the AF motor 102 every time the actual imaging is performed.

  In the image processing apparatus, after setting shooting conditions such as ISO sensitivity and exposure via the external I / F unit 118, a pre-shooting mode is entered by pressing a shutter button (not shown) halfway. When the pre-shooting of the same number of times as the number of shootings in the main shooting is completed, the main shooting is performed by fully pressing a shutter button (not shown) via the external I / F unit 118. The actual photographing is performed based on the focusing condition and the exposure condition obtained by the photographing control unit 107, and information at the time of photographing is transferred to the control unit 117. For example, when the number of times of photographing in the actual photographing is two times, the photographing of the video signal S2 having a different incident light amount and focal length is performed with respect to the video signals S1 and S1 having the appropriate incident light amount. Is transferred to the control unit 117.

  In step S11, the video signal from the CCD 104 is read as Raw data, and ISO sensitivity information, video signal size, and the like are read as header information, and the process proceeds to the next step S2. In step S12, the S1 and S2 video signals photographed through the lens system 100, the aperture 101, the color filter 103, and the CCD 104 are converted into digital video signals by the A / D converter 105 and input to the buffer 106. Is done. In step S13, signal processing such as interpolation processing, white balance adjustment processing, Y / C signal generation processing, electronic zoom processing, and noise suppression processing is performed by the signal processing unit 108, and the process proceeds to step S14.

In step S14, the conversion unit 109 acquires information on the focus positions of a plurality of video signals from the imaging control unit 117, generates a low-frequency video signal based on the video signal S1 focused on the main subject, and is blurred. The process proceeds to step S15 without performing any processing for S2, which is a video signal. The generated low-frequency video signal S1 ′ and the unprocessed video signal S2 are both output to the low-frequency component generator 110 and the high-frequency component generator 112.
In step S15, among the video signals of S1 and S2 transferred from the signal processing unit 108 by the selection unit 111, the video signal S1 that is a video signal focused on the main subject by appropriate exposure using information of the imaging control unit. The single video signal S1 is output to the high frequency component generation unit 112.

  In step S <b> 17, the high frequency component generation unit 112 generates a high frequency video signal by taking the difference between the low frequency video signal S <b> 1 ′ input from the conversion unit 109 and the video signal S <b> 1 input from the selection unit 111. Then, the generated high-frequency video signal is output to the image composition unit 113.

  In step S16, the low frequency component generation unit 110 combines the low frequency video signal S1 ′ input from the conversion unit 109 and the video signal S2 to generate a composite low frequency video signal. The image is output to the image composition unit 113 and proceeds to the next step S18.

  In step S18, the image synthesis unit 113 performs addition processing, that is, image synthesis, on the video signal S1 that is a video signal focused on the main subject by appropriate exposure and the synthesized low-frequency video signal, and the synthesized video signal is converted into a synthesized video signal. Generate. The synthesized video signal is output to the gradation converting unit 114, and the process proceeds to step S19. In step S19, the gradation conversion unit 114 performs a predetermined gradation conversion process on the input composite video signal, and the routine ends. Note that the synthesized video signal that has been subjected to the gradation conversion process is output to the compression unit 115, the compression unit 115 performs compression processing such as JPEG compression, and the output is performed to the output unit 116.

  As described above, a low frequency component in which blurring is not noticeable in the video signal is subjected to synthesis processing for a plurality of images, while a high frequency component in which blurring is noticeable is focused on the main subject by appropriate exposure. By using only one video signal as a target of synthesis processing, it is possible to generate a synthesized signal having a wide dynamic range without artifacts such as double lines even for a plurality of blurred video signals.

In the embodiment described above, processing by hardware is assumed, but it is not necessary to be limited to such a configuration. For example, a configuration in which processing is performed separately by software is also possible. In this case, the image processing apparatus includes a main storage device such as a CPU and a RAM, and a computer-readable recording medium on which a program for realizing all or part of the above processing is recorded. Then, the CPU reads out the program recorded in the storage medium and executes information processing / calculation processing, thereby realizing processing similar to that of the above-described image processing apparatus.
Here, the computer-readable recording medium means a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, or the like. Alternatively, the computer program may be distributed to the computer via a communication line, and the computer that has received the distribution may execute the program.

109 Conversion unit 110 Low frequency component generation unit 111 Selection unit 112 High frequency component generation unit 113 Image composition unit 301 Weighting coefficient calculation unit 302 Addition unit 402 Difference signal calculation unit 502 Noise reduction unit 503 Addition unit

Claims (9)

An image processing apparatus that expands a dynamic range by combining a plurality of video signals,
Video signal acquisition means for acquiring a plurality of video signals having different exposure amounts and focal lengths from the main subject;
Low frequency video signal generating means for extracting a low frequency component from the plurality of video signals to generate a plurality of low frequency video signals;
A combined low-frequency signal generating means for combining the plurality of low-frequency video signals and generating a combined low-frequency video signal;
Selecting means for selecting a first video signal focused on a main subject from the plurality of video signals with a first exposure amount;
High-frequency video signal generating means for extracting a high-frequency component from the first video signal and generating a high-frequency video signal;
Image synthesizing means for generating a synthesized video signal by synthesizing the synthesized low frequency video signal and the high frequency video signal;
An image processing apparatus characterized by comprising: The video signal acquisition means acquires at least one first video signal and at least one second video signal having a second exposure amount and not focusing on the main subject,
The low-frequency video signal acquisition unit generates a video signal by extracting a low-frequency component from the first video signal, and sets the second video signal as a low-frequency video signal.
The image processing apparatus according to claim 1.   The image processing apparatus according to claim 2, wherein the first exposure amount is an exposure amount corresponding to so-called appropriate exposure for the subject.   4. The low frequency video signal generation unit generates the low frequency video signal by filtering the video signal with a smoothing filter. 5. Image processing device. Weighting calculating means for calculating a weighting coefficient that is a composite ratio of the plurality of low frequency video signals based on the pixel value of the low frequency video signal,
5. The image according to claim 1, wherein the synthesized low-frequency video signal generating unit performs synthesis by weighting the low-frequency video signal based on the weighting coefficient. Processing equipment.   6. The high-frequency video signal generating unit generates a high-frequency video signal by calculating a difference between the first video signal and the low-frequency video signal. The image processing apparatus according to item 1.   The image processing apparatus according to claim 1, wherein the image synthesizing unit adds the synthesized low-frequency video signal and the high-frequency video signal. The image synthesizing means includes noise reduction means for performing noise reduction processing on the high-frequency video signal;
8. The image processing apparatus according to claim 1, wherein the synthesized low-frequency video signal and the high-frequency video signal whose noise is reduced by the noise reduction unit are added. 9. An image processing program for expanding a dynamic range by combining a plurality of video signals,
Acquiring a plurality of video signals having different exposure amounts and focal lengths from the main subject; and
Extracting a low frequency component from the plurality of video signals to generate a plurality of low frequency video signals;
Combining the plurality of low frequency video signals to generate a combined low frequency video signal;
Selecting a first video signal having a first exposure amount and focused on a main subject from the plurality of video signals;
Extracting a high frequency component from the first video signal to generate a high frequency video signal;
Generating a synthesized video signal by synthesizing the synthesized low frequency video signal and the high frequency video signal;
An image processing program for causing a computer to execute.

Images