KR101695987B1 - Apparatus and method for enhancing image taken by multiple color-filter aperture camera and multiple color-filter aperture camera equipped with the same - Google Patents

Abstract

Disclosed is an apparatus and method for improving the image quality of an image captured by an image pickup apparatus having a multi-color filter stop, and an image pickup apparatus having a multi-color filter stopper. An image pickup apparatus includes an image pickup element for generating and outputting an original image from photographed light of an object transmitted through color filters having mutually different hues, and an image pickup element for matching the focal points of a plurality of color channels extracted from the original image, A second restored image is generated by filtering the first restored image, a third restored image obtained by removing the noise by mixing the first restored image and the second restored image, And an image restoring unit for generating the image.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus and method for improving image quality of an image captured by an image pickup apparatus having a multi-color filter iris, and an image pickup apparatus having a multi- multiple color-filter aperture camera equipped with the same}

The present invention relates to an image quality enhancement apparatus and method capable of correcting a focus of color channels obtained through a plurality of color filters and restoring an image by eliminating noise of a focus-corrected image, and a multi- To an image pickup apparatus.

The technique of restoring an image obtained by the auto focus technique and the auto focus technique is one of the main technologies of the image technology such as a camcorder, a digital camera, and a video surveillance system. As the resolution and precision of video devices increase, the focus of an image may be blurred due to the fine movement of the subject or the imaging device during shooting. The techniques are used to solve this problem. In particular, when a plurality of subjects are present at different distances from the image pickup apparatus, image quality deterioration such as blurring occurs because the focal point is not precisely adjusted except for a subject located on the focal length.

In order to solve the deterioration of image quality, conventionally, the focus degree of the image is estimated on the image plane, and the focus is automatically corrected by moving the lens assembly to the optical focus position based on the estimated focus degree information. However, the above-described conventional techniques solve various image quality degradation problems such as low exposure, color mixing, deviation of color convergence, and divergence of light rays There is a limit.

SUMMARY OF THE INVENTION An object of the present invention is to provide an image quality enhancement apparatus and method capable of eliminating residual noise even after the focus of an image is corrected to solve a problem of image quality deterioration that may occur in an auto focus technique, And an image pickup apparatus.

Another object of the present invention is to provide a computer-readable recording medium on which a program for causing a computer to execute an image restoration method capable of solving the problem of image deterioration that may occur in the auto focus technology have.

According to an aspect of the present invention, there is provided an image quality improving apparatus comprising: a plurality of color filters having different colors; A focus restoring unit for generating a first restored image whose focus is restored by matching the focus of the color channels; And generating a second reconstructed image by filtering the first reconstructed image to generate a third reconstructed image from which noise is removed by mixing the first reconstructed image and the second reconstructed image, .

According to another aspect of the present invention, there is provided an image pickup apparatus comprising: an image pickup element for generating an original image from photographic light of an object transmitted through color filters having different colors; A focus restoring unit for generating a first restored image whose focus is restored by matching the focus of a plurality of color channels extracted from the original image; And generating a second reconstructed image by filtering the first reconstructed image to generate a third reconstructed image from which noise is removed by mixing the first reconstructed image and the second reconstructed image, .

According to another aspect of the present invention, there is provided a method of improving image quality of an image, the method comprising: generating an original image from photographic light of a subject through color filters having different colors; Generating a first reconstructed image in which a focus is restored by matching focuses of a plurality of color channels extracted from the original image; And generating a second reconstructed image by filtering the first reconstructed image to generate a third reconstructed image from which noise is removed by mixing the first reconstructed image and the second reconstructed image; I have.

According to the present invention, it is possible to restore the focus of the images generated by the color filters having different colors, and to remove the out-of-focus blur phenomenon of the restored image by filtering. In addition, the reconstructed image from which the artifact is removed can be generated more quickly by synthesizing the out-of-focus blurred image and the restored image adaptively and in real time. Further, according to the present invention, it is possible to solve various image quality degradation problems such as low exposure, color mixing, deviation of color convergence, and divergence of light rays, As a result, it is possible to generate a reconstructed image without degrading the quality of the image. In addition, according to the present invention, since the image restoration is performed by only one original image by the auto-focus restoration and filtering based on the movement of the color channel, the degraded image can be restored more quickly, It is possible to prevent a phenomenon such as an echo.

1 is a block diagram showing the configuration of an image pickup apparatus according to the present invention;
2 is a diagram showing a schematic structure of a preferred embodiment of the image pickup apparatus according to the present invention,
3 is a view showing a convergence pattern of color channels corresponding to the respective color filters provided in the diaphragm,
4 is a diagram for explaining a process of the second CRF scheme,
5 is a diagram illustrating a process of improving an image quality of an original image by an image pickup apparatus according to an embodiment of the present invention,
6 is a flowchart for explaining a method of improving the image quality of an image pickup apparatus according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an image quality improving apparatus and method according to the present invention and an image pickup apparatus implementing the same will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numbers are used throughout the drawings to refer to the same or like elements.

Fig. 1 is a block diagram showing the configuration of an image pickup apparatus according to the present invention, and Fig. 2 is a diagram showing a rough structure of a preferred embodiment of the image pickup apparatus 100 according to the present invention. 1, an image pickup apparatus in which an image quality improving apparatus or method is implemented is illustrated. However, the focus restoring unit 120 and the image restoring unit 130 may be implemented in the form of an image quality improving apparatus independently, And may be provided in an image processing apparatus.

1 and 2, an image sensing apparatus 100 according to the present invention includes a diaphragm 110, an image sensing device 120, a focus restoring unit 130, and an image restoring unit 140.

The diaphragm 110 is provided in the lens section 150 at the front end of the imaging apparatus 100 and the imaging element 130 is installed inside the imaging apparatus main body 160. The diaphragm 110 adjusts the amount of light incident on the lens unit 150 according to the degree of opening of a plurality of openings (not shown). Color filters 111, 113, and 115 are respectively installed in the respective openings. The color filters 111, 113, and 115 are a red color filter, a green color filter, and a blue color filter. The image pickup apparatus 100 measures depth information of objects located at different distances using a diaphragm 110 having a plurality of color filters 111, 113, and 115, and performs multi-focusing.

The photographing light of the subject that has passed through the color filters 111, 113, and 115 of the diaphragm 110 is formed into an original image by being imaged on the image sensing device 120. The convergence pattern of the photographing light is red, green, Different for blue color channels. Since the positions of the color filters 111, 113, and 115 are different from each other in the diaphragm 110, an original image generated from the photographic light transmitted through each color filter 111, 113, A converging pattern is formed.

3 shows a converging pattern of color channels corresponding to each color filter provided in the diaphragm.

3, the photographing light of the subject 310 located at the In Focus distance from the lens 340 is transmitted through the color filters 111, 113, and 115 of the diaphragm 110, and corresponding red, The convergent patterns 355, 365, and 375 are formed on the color channels 350, 360, At this time, the convergence patterns 355, 365, and 375 in the respective color channels 350, 360, and 370 are shifted upward, downward, leftward, and rightward at the positions of the corresponding color filters 111, appear. Therefore, the distance to the subject can be estimated based on the arrangement of the color filters 111, 113, 115 and the convergent paddles 355, 365, 375 shown in the color channels 350, 360, 370. If the subject is located at the Near Focus distance from the lens 340 or at the Far Focus distance, the position of the converging patterns can be changed.

1, the focus restoring unit 130 is connected to the image pickup device 120 and is installed in the main body 160 of the image pickup apparatus 100, receives the original image from the image pickup device 120, Restores the focus of the deteriorated area. To this end, the focus restoring unit 130 generates a first restored image in which the focus is restored by matching and fusing a plurality of color channels extracted from the original image to match the focus of the color channels.

The focus restoring unit 130 may generate the first reconstructed image using at least one of a first color channel registration and fusion (CRF) scheme and a second CRF scheme, or other known techniques .

First, the first CRF method will be described. The focus restoring unit 130 determines a complex area among the original images input from the image sensing device 120 as an area to be focused on and moves the images corresponding to the color channels 350, And outputs an image whose focus is matched.

More specifically, the focus restoring unit 130 obtains the projection value by summing the difference in pixel intensity between the pixels constituting the original image input from the image sensing element 120 and the surrounding pixels, The focus value of each pixel is calculated as the sum of the projection values corresponding to the pixels in the window of the predetermined size. The focus restoring unit 130 determines the restored regions having a predetermined size that is lower than the focus region in the original image based on the calculated focus value and compares the focus value with the surrounding region to detect the restored regions of the color filters 111, 360, and 370 based on the motion vectors of the color channels 350, 360, and 370 calculated from the position information to generate intermediate images focused on the respective reconstruction areas. Then, the focus restoring unit 130 registers the remaining images in the reference image with the image in which the coincident focus is located at the center of the image as the reference image among the original image and the intermediate image, After the fusion region is determined in each original image and the intermediate image, the fused regions are fused to generate a first reconstructed image in which the focal point is restored in all regions.

The second CRF scheme will be described with reference to FIG. 4 is a diagram for explaining the process of the second CRF scheme. The focus restoring unit 130 may use a cluster based image segment and a Phase Correlation Matching (PCM) algorithm (hereinafter referred to as a " clustering ") to automatically match the pixels on the three color channels 350, Based algorithm "). Cluster-based algorithms are a well-known technique, which can use distance-based K-means clustering techniques and block partitioning.

Referring to FIG. 4, in the cluster-based algorithm, the k-seed can be automatically determined by Hill-Climbing processing in the 3D CIELab histogram of the original image. The focus restoring unit 130 divides the original image into blocks having the same size after labeling a salient region in the original image. The focus restoring unit 130 generates a region of interest (ROI) map that requires shift of the color channels based on the divided blocks. The focus restoring unit 130 aligns each block of the ROI map using a cluster-based algorithm, and estimates the amount of shift generated in each block. The focus restoring unit 130 adjusts and fuses the color channels 350, 360, and 370 based on the amount of the estimated shift to generate an in-focused image, that is, .

Referring back to FIG. 1, the image restoring unit 140 generates a second restored image by filtering the first restored image generated by the focus restoring unit 130, and combines the first restored image and the second restored image thereby generating a third reconstructed image from which noises have been removed. The image reconstruction unit 140 includes an image reconstruction function estimation unit 141, a filtering unit 143, and an image synthesis unit 145.

The image reconstruction function estimating unit 141 estimates an image reconstruction function corresponding to the first reconstructed image. The estimated image restoration function may be a point spread function (PFS) according to the distance between the subject and the image sensing device 120 or the distance between the subject and the lens unit 150. [ In the embodiment of the present invention, two assumptions are required to estimate the point spread function. First, the out-of-focus blur phenomenon existing in the first reconstructed image is caused by an isotropic or circularly symmetric point spread function. Next, the original image output from the image pickup device 120 And an ideal step edge located at the linear boundary of the subject. Since the point spread function can be estimated using an optimal algorithm among various known algorithms, it is not limited to a specific algorithm in the embodiment of the present invention.

The filtering unit 143 generates a filter for filtering the first reconstructed image using the reconstructed image estimated by the reconstructed image reconstruction unit 141. [ The filtering unit 143 generates a second reconstructed image by filtering the first reconstructed image using the generated filter. The second reconstructed image may be an image in which some noise such as Out-of-Focus Blur is removed from the first reconstructed image by a filter.

The filter generated by the filtering unit 143 may be a Truncated Constrained Least Squares (TCLS) restoration filter having a truncation window of (m × m) size considering the estimated image restoration function size.

Hereinafter, an embodiment in which the filtering unit 143 generates the TCLS restoration filter will be described.

The filtering unit 143 may generate a frequency response of a constrained least squares (CLS) filter using an estimated image reconstruction function, for example, a point spread function. The frequency response of the CLS filter is affected by the magnitude of the point spread function. The filtering unit 143 may apply an inverse Fourier transform to the CLS filter to obtain full-support Spatial Domain Filter coefficients, i.e. coefficients of the spatial domain. At this time, since the support area of the CLS filter becomes equal to the size of the original image, using all the coefficients of the spatial domain requires a large amount of computation. In order to solve this problem, the filtering unit 143 uses a characteristic that the impulse response characteristic of the spatial domain of the filter is concentrated in the center (Main Lobe) and its periphery, Support Spatial Domain Filter). Based on the optimized experimental values for image reconstruction performance and efficiency, the filtering unit 143 may use a raised cosine function to cut the filter. For example, the filtering unit 143 may generate a TCLS restoration filter having a (13 × 13) size cutoff window considering the size of the point spread function.

The above-described TCLS filter generates a second reconstructed image from which out-of-focus blur phenomenon is removed by removing a limited amount of noise from the first reconstructed image. However, unnecessary artifacts may exist in the second reconstructed image. The image combining unit 145 removes the artifacts from the second reconstructed image.

Meanwhile, the image combining unit 145 may calculate weights to be given to the first reconstructed image and the second reconstructed image, adaptively combine the first reconstructed image and the second reconstructed image, . For example, the image reconstructing unit 140 or the image synthesizing unit 145 may adaptively combine the high frequency regions among the flat regions and the second reconstructed image among the first reconstructed images to generate a third reconstructed image do.

The weight is calculated for each pixel, and is calculated in consideration of pixel local variation (Pixel Local Variance) calculated in a rectangular region centered on each pixel. Hereinafter, a method of generating a third reconstructed image by adaptively combining the first reconstructed image and the second reconstructed image with reference to [Equation 1] to [Equation 3] will be described.

First, in order to calculate a weight to be applied to the first reconstructed image and the second reconstructed image, the image synthesizer 145 calculates a local variance of pixel values at each pixel (x, y) using the following equation do.

Where S is a rectangular region surrounding (x, y), f (x, y) is a pixel value (e.g., a pixel value) of a pixel located at (x, y) And m _xy are the local mean of the pixel values of the pixels located in the rectangular region.

The image synthesizing unit 145 calculates a weight by the following equation (2).

Here, α (x, y) is a weight to be applied to a pixel at (x, y) position, and σ is a tuning parameter.

sigma can be selected such that alpha is distributed as uniformly as possible in the range [0, 1]. The value of sigma for the optimally equalized alpha may be, for example, approximately 1000, and this is not limiting in one example. alpha is an alpha map image consisting of weights calculated for all pixels.

The image synthesizing unit 145 obtains weights of the respective pixels according to Equation (2), and generates an alpha map image composed of weights of the pixels. The alpha map image is a black-and-white image for adjusting the degree of blending of the colors of the pixels corresponding to the first restored image and the second restored image. In the case where the color values of the blocks forming the monochrome image are classified into the levels of 0 to 255, 0 to 255 can be converted into values of 0 to 1, and the values converted into 0 to 1 can be weighted.

The image reconstruction unit 140 synthesizes the first reconstructed image and the second reconstructed image using an alpha map made up of weights and Equation (3).

In Equation 3, g is a first reconstructed image (i.e., a plurality of color channels are misaligned image), f _R is the second reconstructed image reconstructed using TCLS filter produced by the focus recovery unit 130, and , f _A is the third reconstructed image finally reconstructed by adaptive synthesis.

The third reconstructed image is generated by adaptively combining the second reconstructed image generated by the TCLS filter and the first reconstructed image generated by the CRF, and has a sharp reconstructed edge.

5 is a diagram illustrating a process of restoring the image quality of an original image by the image sensing apparatus 100 according to the embodiment of the present invention.

5, the image pickup device 120 generates an original image from photographing light of a subject that has passed through the color filters of different colors provided in the diaphragm 110. The focus restoring unit 130 generates an ROI map image from the original image and generates a first restored image by matching and fusing the color channels 350, 360, and 370 using the ROI map image. The image reconstructing unit 140 generates a second reconstructed image, such as Out-of-Focus Blur, by removing the noise by filtering the first reconstructed image using the TCLS filter. In addition, the image restoring unit 140 generates an alpha map image representing the weight of pixels from the original image. The image reconstruction unit 140 adaptively combines the first reconstructed image and the second reconstructed image using the generated alpha map image to generate a third reconstructed image.

6 is a flowchart illustrating an image restoration method of an image pickup apparatus according to an embodiment of the present invention. The image pickup apparatus that performs the image restoration method of FIG. 6 may be the image pickup apparatus 100 described with reference to FIG. 1 and may be substantially the same as or similar to the image pickup apparatus 100 of the image pickup apparatus 100 or the at least one processor (not shown) Lt; / RTI >

Referring to FIG. 6, the imaging device generates an original image from photographed light of a subject that has passed through color filters having different colors (S610). At this time, the color filters may be provided in the diaphragm. Next, the imaging device generates a first reconstructed image in which the focus is restored by matching the focuses of a plurality of color channels extracted from the original image (S620). The image pickup apparatus generates a first reconstructed image by matching and fusing color channels using the first CRF scheme or the second CRF scheme described above with reference to FIG. Next, the imaging apparatus generates a filter for filtering the generated first reconstructed image, for example, a TCLS filter (S630). Subsequently, the imaging device generates a second reconstructed image by filtering the first reconstructed image using the generated TCLS filter (S640). Finally, the imaging apparatus mixes the first reconstructed image and the second reconstructed image using Equations (1) to (3), thereby generating a third reconstructed image from which the noise is removed (S650).

The present invention can also be embodied as computer-readable codes on a computer-readable recording medium. A computer-readable recording medium includes all kinds of recording apparatuses in which data that can be read by a computer system is stored. Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, and the like. The computer-readable recording medium may also be distributed over a networked computer system so that computer readable code can be stored and executed in a distributed manner.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is clearly understood that the same is by way of illustration and example only and is not to be taken by way of limitation in the embodiment in which said invention is directed. It will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the scope of the appended claims.

Claims (22)

A plurality of color channels extracted from an original image picked up by an image pickup device for generating an original image from photographing light of an object transmitted through color filters having different colors are matched and fused, A focus restoring unit for generating a focus; And
An image reconstruction unit for generating a second reconstructed image by filtering the first reconstructed image to generate a third reconstructed image from which noise is removed by mixing the first reconstructed image and the second reconstructed image; , ≪ / RTI &
The focus restoring unit includes:
A first color channel matching and fusing method for determining a complex area of the original image as an area to be focused on and moving images corresponding to the color channels to the determined area,
The first reconstructed image is generated using a second color channel matching and fusing method of matching pixels on the color channel using a cluster-based image segment and a phase correlation matching (PCM) algorithm The image quality improvement device. The method according to claim 1,
Wherein the image restoration unit comprises:
An image reconstruction function estimating unit estimating an image reconstruction function corresponding to the first reconstructed image;
A filtering unit for generating a filter for filtering the first reconstructed image using the estimated image reconstruction function and filtering the first reconstructed image using the generated filter; And
And a video synthesis unit for calculating weights to be given to the first reconstructed image and the second reconstructed image and adaptively synthesizing the first reconstructed image and the second reconstructed image using the calculated weights And the image quality of the image is improved. 3. The method of claim 2,
Wherein the weight is calculated for each pixel and is calculated in consideration of a pixel local variation calculated in a rectangular region centered on each pixel. 3. The method according to claim 1 or 2,
Wherein the image reconstruction unit adaptively combines the high frequency region of the flat region and the second reconstructed image of the first reconstructed image to generate the third reconstructed image. . 3. The method of claim 2,
Wherein the filter generated by the filtering unit is a Truncated Constrained Least Squares (TCLS) restoration filter having a truncation window of size (mxm) considering the estimated size of the image restoration function Image quality improvement device. An imaging device for generating an original image from photographing light of a subject that has passed through color filters having different colors;
A focus restoring unit for matching and fusing a plurality of color channels extracted from the original image to generate a first restored image whose focus is restored; And
An image reconstruction unit for generating a second reconstructed image by filtering the first reconstructed image to generate a third reconstructed image from which noise is removed by mixing the first reconstructed image and the second reconstructed image; , ≪ / RTI &
The focus restoring unit includes:
A first color channel matching and fusing method for determining a complex area of the original image as an area to be focused on and moving images corresponding to the color channels to the determined area,
The first reconstructed image is generated using a second color channel matching and fusing method of matching pixels on the color channel using a cluster-based image segment and a phase correlation matching (PCM) algorithm . The method according to claim 6,
Wherein the image restoration unit comprises:
An image reconstruction function estimating unit estimating an image reconstruction function corresponding to the first reconstructed image;
A filtering unit for generating a filter for filtering the first reconstructed image using the estimated image reconstruction function and filtering the first reconstructed image using the generated filter; And
And a video synthesis unit for calculating weights to be given to the first reconstructed image and the second reconstructed image and adaptively synthesizing the first reconstructed image and the second reconstructed image using the calculated weights And the image pickup device. 8. The method of claim 7,
Wherein the weight is calculated for each pixel and is calculated in consideration of a pixel local variation calculated in a rectangular region centered on each pixel. 8. The method according to claim 6 or 7,
Wherein the image reconstructing unit adaptively combines the high frequency region of the flat region and the second reconstructed image of the first reconstructed image to generate the third reconstructed image. 8. The method of claim 7,
Wherein the filter generated by the filtering unit is a Truncated Constrained Least Squares (TCLS) restoration filter having a truncation window of size (mxm) considering the estimated size of the image restoration function . 8. The method of claim 7,
Wherein the estimated image restoration function is a point spread function according to a distance between the subject and the imaging element. The method according to claim 6,
Wherein the color filters are provided at different positions of the diaphragm. delete Generating an original image from photographic light transmitted through color filters having different colors;
Generating a first reconstructed image in which a focus is restored by matching and fusing a plurality of color channels extracted from the original image; And
Generating a second reconstructed image by filtering the generated first reconstructed image and generating a third reconstructed image from which noise is removed by mixing the first reconstructed image and the second reconstructed image However,
Wherein the generating the first reconstructed image comprises:
A first color channel matching and fusing method for determining a complex area of the original image as an area to be focused on and moving images corresponding to the color channels to the determined area,
And a first reconstructed image is generated using a second color channel matching and focussing method of matching pixels on the color channel using a cluster-based image segment and a phase correlation matching (PCM) algorithm. A method for improving image quality of an image. 15. The method of claim 14,
Wherein the generating the third reconstructed image comprises:
Estimating an image reconstruction function corresponding to the first reconstructed image;
Generating a filter for filtering the first reconstructed image using the estimated reconstructed image;
Filtering the first reconstructed image using the generated filter; And
Calculating a weight to be given to the first restored image and the second restored image, and adaptively combining the calculated restored image with the first restored image and the second restored image, / RTI > 16. The method of claim 15,
Wherein the weight is calculated for each pixel and is calculated in consideration of a pixel local variation calculated in a rectangular region centered on each pixel. 16. The method according to claim 14 or 15,
Wherein the generating the third reconstructed image comprises:
Wherein the third reconstructed image is generated by adaptively combining the high frequency region of the flat region and the second reconstructed image of the first reconstructed image. 16. The method of claim 15,
Wherein the generated filter is a Truncated Constrained Least Squares (TCLS) restoration filter having a truncation window of size (mxm) considering the size of the estimated image restoration function. How to improve. 16. The method of claim 15,
Wherein the estimated image restoration function is a point spread function according to a distance between the subject and the imaging device. 15. The method of claim 14,
Wherein the color filters are provided at different positions of the diaphragm. delete A computer-readable medium having recorded thereon a program for causing a computer to execute the image quality improvement method according to any one of claims 14 to 16 or 18 to 20.

Images