KR101025405B1 - Focus control image generation system, method and photographing apparatus employing the same - Google Patents

Abstract

A focus adjustment image generation system is disclosed. The disclosed focusing image generating system includes a calculator for calculating an initial depth and the number of layers from at least two comparison images having different focuses and a reference image having a total focus, color segmentation of the reference image, and color segmentation; A blurring unit configured to blur the kernel by applying the initial depth to each color segment, and determine a color segment constituting each layer and a corresponding distance while comparing the color similarity between each of the blurred color segments and the comparison image; And a comparison determiner, an extraction merger for extracting scene layers according to the distance, merging each color segment into the scene layer, and an image generator for refocusing the scene layer to generate a refocused image.

Description

FOCUS CONTROLING IMAGE GENERATION SYSTEM AND METHOD THEREOF AND IMAGE GENERATION APPARATUS EMPLOYING THE SAME}

The present invention relates to an image adjusting system, a method, and a photographing apparatus, and more particularly, to an image generating system, a method, and a photographing apparatus capable of generating a focused image.

Conventional photograpy obtains the resultant image by projecting a three-dimensional real space in two dimensions. However, recently, methods for generating a new image by controlling focus and the like by variously acquiring depth information together with an image of a target scene have been proposed. These methods are an active method of recovering depth information from a large number of images or actively emitting a specific pattern to a target space, and a passive device having hardware image control elements in an imaging device such as a camera. There is a way.

Among the active methods, there is a method of obtaining a depth map by projecting a dot pattern on a target scene, and then refocusing still and video images by combining the depth map and color segmentation information. However, this method is difficult to use in outdoor spaces affected by the sun, which emits strong visible light, because the projected constant pattern decreases rapidly with distance in the target space, especially the pattern in which the color of the target space is projected. In this case, many errors may occur due to the color blend.

Passive methods require a separate imaging device and are difficult to align and adjust the optical device to allow actual photographs to be formed.

The present invention has been made to improve the prior art as described above, and provides an image generating system and apparatus and method employing the same that can generate an image by adjusting the focus of the image in software.

According to one aspect of the invention, the calculation unit for calculating the initial depth and the number of layers from at least two comparison images having different focus and one reference image that is in full focus; A blurring unit configured to extract color segments by color-dividing the reference image, and apply and blur the kernel of the initial depth to each color segment; A comparison determiner configured to determine a color distance constituting each layer and a corresponding distance while comparing color similarities between the blurred color segments and the comparison image; An extract merging unit extracting a scene layer according to the distance and merging each color segment into the scene layer; And an image generator for refocusing the scene layer to generate a refocused image.

According to another aspect of the present invention, in a photographing apparatus having an image generating system, the image generating system includes: calculating an initial depth and the number of layers from at least two comparison images having different focal points and one reference image having a total focus; A calculation unit; A blurring unit configured to extract color segments by color-dividing the reference image, and apply and blur the kernel of the initial depth to each color segment; A comparison determiner configured to determine a color distance constituting each layer and a corresponding distance while comparing color similarities between the blurred color segments and the comparison image; An extract merging unit extracting a scene layer according to the distance and merging each color segment into the scene layer; And an image generator for refocusing the scene layer to generate a refocused image.

According to another aspect of the invention, the step of calculating the initial depth and the number of layers from at least two comparison images of different focus and one reference image that is in full focus; Color segmenting the reference image to extract color segments, and applying and blurring the kernel of the initial depth to each color segment; Determining a color distance constituting each layer and a corresponding distance while comparing color similarities between the blurred color segments and the comparison image; Extracting a scene layer according to the distance, and merging each color segment into the scene layer; And resetting focus to the scene layer to generate a refocused image.

According to an embodiment of the present invention, a refocus image may be generated by accurately extracting a scene layer using DFD technology and color segmentation technology.

In addition, according to an embodiment of the present invention can provide a natural refocused image.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram illustrating a structure of a focus control image generating system according to an embodiment of the present invention.

The focus control image generating system 100 according to an exemplary embodiment of the present invention includes a calculation unit 101, a blurring unit 102, a comparison determination unit 103, an extraction merger 104, and an image generator 104. It includes.

The calculator 101 calculates an initial depth and the number of layers by applying a depth from defocus (DFD) technique from at least two images having different focus (hereinafter, referred to as a comparative image) and one image having a total focus (hereinafter referred to as a reference image). . In the same camera system, the DFD forms a depth map from images captured by different optical elements such as focal length, and generates a refocused image by measuring a relative blur. In general, DFD uses images taken from special cameras consisting of a fixed lens center and an adjustable lens. However, in the case of using the DFD technology, it is necessary to normalize the size and brightness change caused by the change of the camera parameter in the process of capturing a plurality of images.

The calculation unit 101 may determine the blurring degree of the comparison image and determine the initial depth proportional to the defocusing depth by using a relationship with the blur circle modeled by a Gaussian function. In addition, a depth histogram may be extracted by applying a defocusing depth to the reference image, and the initial depth and the number of layers may be calculated.

2A is a view showing a reference image, FIG. 2B is a comparison image in which the focus is on the foreground object, and FIG. 2C is a comparison image in which the focus is on the background. 3A shows a depth map according to an initial focal length calculated by the calculation unit 101 by applying the DFD technique to the images of FIGS. 2A to 2C, and FIG. 3B shows an initial depth histogram obtained by analyzing FIG. 2A.

In the depth map according to the initial depth of FIG. 3A, it can be seen that the layer of each object is unclear. In addition, according to the depth histogram of FIG. 3B, three peaks exist from three peaks of the defocusing depth of the reference image.

In general, since the DFD technology calculates the depth for each pixel, there are many errors due to the color distribution in the image. Therefore, to compensate for this error, a method of blurring the reference image using a depth kernel set to a value near the defocusing depth according to the initial focal length is introduced.

The blurring unit 102 first color-divides the reference image to extract color segments. That is, since the initial depth information derived from the comparison image by the calculator 101 is not evenly distributed within the scene layer, it is difficult to accurately derive the layers of each scene. In order to compensate for this, the blurring unit 102 intends to use the color segment information together.

The blurring unit 102 divides each color segment of the reference image by, for example, a median shifting method among color segmentation techniques. That is, the blurring unit 102 divides the area constituting the layer of the scene image into color segment units having relatively similar color characteristics, thereby extracting meaningful layers by combining initial depth information in the future.

A hill climbing method may be used as the color division method used in the blurring unit 102. Determine the appropriate color distribution and probability density functions on the color distribution of the color space. In detail, the average position and average value having a color distribution similar to the current pixel of the image are calculated and moved to determine the maximum point position of the convergence vector. By uniformly assigning the color value of the maximum point to the color of the corresponding area, a uniformly distributed color division result can be obtained. In this case, since the boundary region of color is always a minimum point of the color distribution, the boundary can be clearly maintained according to the color distribution change of the object in the image.

The blurring unit 102 then applies and blurs the initial depth to each color segment. For example, when the initial, middle, and root diameters are derived as the initial depths, the corresponding initial depths are applied to each color segment and blurred.

4A illustrates a result of color separation and blurring based on the reference image of FIG. 2A, and FIG. 4B illustrates color segments having the same color.

The comparison determiner 103 determines a color segment constituting each layer and a corresponding distance while comparing the color similarity between each color segment and the comparison image.

The comparison determiner 103 first mixes each of the blurred color segments. In particular, the comparison determination unit 103 may mix the blurred color segments of each layer using a predetermined coefficient (alpha coefficient α) selected according to the depth in the boundary region where the depth of each layer is discontinuous. In this case, the comparison determiner 103 may start mixing with the largest color segment in the boundary region and sequentially proceed with the adjacent color segments.

Two separate color segments S ₀ and S ₁ may be mixed using an alpha coefficient α on an association map corresponding to a candidate region of a boundary between layers.

5A and 5B are diagrams showing association maps between color segments changed in a mixing process.

The comparison determiner 103 determines a color segment constituting each layer and a corresponding distance while comparing the color similarity with the comparison image after the mixing process. For example, there may be far and far blend, far and mid blend, far and far blend, medium and far blend, middle and mid blend, middle and far blend, and root and blend. The blended portion of the color segment described above and at least two images having different focal points belonging to the comparative image are compared with each other to determine exactly which layer the color segment of the boundary region belongs to, and the distance is far, middle, or near. You can decide where you belong.

The following equation illustrates the process of determining the scene layer from the blurred color segment Ii.

Here, K is a scene layer index, I _i is a blurred color segment, I _ref is a reference image, and boundary is a set of pixels in which two color segments may affect each other in an associated map.

The extraction merger 104 extracts a scene layer according to the distance derived from the comparison determiner 103 and merges each color segment into the scene layer.

6A to 6D illustrate a process of determining a layer to which a color segment belongs while comparing color similarities in the extraction merger 104.

7 is a diagram illustrating the determined final scene layer.

When the final scene layer is derived, a matting technique may be applied to the focal length information estimated at the boundary region of the object to derive a soft layer.

Finally, the image generator 105 generates a refocused image by refocusing on the scene layer.

8 is a photograph in which a refocus image is formed using the extracted scene layer.

Referring to FIG. 8, it can be seen that the focus is set to medium diameter. That is, the image generator 104 generates a refocused image by convolving the image with a kernel having a defocusing depth corresponding to each scene layer. In this case, it is preferable to correct the generated image in consideration of at least one of a point diffusion function on the degree of blurring of the image, an image enlargement according to lens movement, and a change in the aperture diameter.

The focus control image generating system according to an embodiment of the present invention is preferably mounted and used in a photographing apparatus such as a camera. When various parameters such as focal length information of the lens set by the photographing apparatus can be accurately input, the extraction of the layer is more accurate in the focus control image generating system according to the exemplary embodiment of the present invention.

9 is a flowchart illustrating a method of generating a focused image according to an embodiment of the present invention.

First, an initial depth and the number of layers are calculated from at least two comparison images having different focal points and one reference image having a total focal point (S1). Then, color segmentation is performed on the reference image to extract color segments, and the kernel of the initial depth is applied to each color segment and blurred (S2). Then, the color similarity between each of the blurred color segments and the comparison image is compared, and the corresponding color segments constituting each layer are determined (S3). Extracting the scene layer according to the distance, merging each color segment into the scene layer (S4), and refocusing the scene layer to generate a refocused image (S5).

For a detailed description of each step of the method for generating a focus control image according to an embodiment of the present invention, refer to the focus control image generation system according to the embodiment of the present invention described above.

In addition, the focus control image generating method according to an embodiment of the present invention is implemented in the form of program instructions that can be executed by various computer means may be recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on the media may be those specially designed and constructed for the purposes of the present invention, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

As described above, the present invention has been described by way of limited embodiments and drawings, but the present invention is not limited to the above-described embodiments, which can be variously modified and modified by those skilled in the art to which the present invention pertains. Modifications are possible. Accordingly, the spirit of the present invention should be understood only by the claims set forth below, and all equivalent or equivalent modifications thereof will belong to the scope of the present invention.

1 is a conceptual diagram showing a structure of a focus control image generating system according to an embodiment of the present invention;

2A is a view showing a reference image;

2b is a view showing a comparison image in which a focal point fits a foreground object;

2c is a view showing a comparison image in which the focus is in the background;

3A is a view showing a depth map according to an initial focal length calculated by applying the DFD technique to the images of FIGS. 2A to 2C;

3B is a graph showing an initial depth histogram of FIG. 2A;

4A is a view illustrating a result of color separation and blurring based on the reference image of FIG. 2A;

4b shows a color segment with the same color,

5a and 5b are views showing an association map between color segments changed in a mixing process;

6A to 6D illustrate a process of determining a layer to which a color segment belongs while comparing color similarity;

7 shows a determined final scene layer;

8 is a photo of a refocused image formed using the extracted scene layer;

9 is a flowchart illustrating a method of generating a focused image according to an embodiment of the present invention.

Claims (28)

A calculation unit configured to calculate an initial depth and the number of layers from at least two comparison images having different focal points and one reference image having a total focus; A blurring unit configured to extract color segments by color-dividing the reference image, and apply and blur the kernel of the initial depth to each color segment; A comparison determiner configured to determine a color distance constituting each layer and a corresponding distance while comparing color similarities between the blurred color segments and the comparison image; An extract merging unit extracting a scene layer according to the distance and merging each color segment into the scene layer; And An image generator for refocusing the scene layer to generate a refocused image; Focus control image generation system comprising a. The method of claim 1, And the calculator calculates the blurring degree of the comparison image and determines the initial depth proportional to the defocusing depth using a relationship with the blur circle modeled by a Gaussian function. The method of claim 1, The calculating unit extracts a depth histogram of the initial depth of the reference image and calculates the initial depth and the number of layers. The method of claim 1, And the kernel of the initial depth is set to the vicinity of the initial depth. The method of claim 1, And the comparison determiner is configured to mix the blurred color segments of each layer using coefficients determined according to the initial depth in a boundary area where the depth of each layer is discontinuous. The method of claim 5, And the comparison determiner is configured to start mixing from the largest color segment of the boundary region and sequentially perform mixing to adjacent color segments. The method of claim 1, And the image generating unit generates a refocused image by convolving the image with the kernel of the distance corresponding to each scene layer. The method of claim 7, wherein And the kernel of the distance is set to the vicinity of the distance. The method of claim 1, And the image generating unit corrects the generated image in consideration of at least one of a dot diffusion function of the degree of blurring, an image enlargement according to lens movement, and a change in aperture diameter. In the photographing apparatus having an image generating system, The image generation system is: A calculation unit configured to calculate an initial depth and the number of layers from at least two comparison images having different focal points and one reference image having a total focus; A blurring unit configured to extract color segments by color-dividing the reference image, and apply and blur the kernel of the initial depth to each color segment; A comparison determiner configured to determine a color distance constituting each layer and a corresponding distance while comparing color similarities between the blurred color segments and the comparison image; An extract merging unit extracting a scene layer according to the distance and merging each color segment into the scene layer; And An image generator for refocusing the scene layer to generate a refocused image; Shooting device comprising a. The method of claim 10, And the calculator calculates the blurring degree of the comparison image and determines the initial depth proportional to the defocusing depth using a relationship with the blur circle modeled by a Gaussian function. The method of claim 10, And the calculator extracts a depth histogram of the initial depth of the reference image and calculates the initial depth and the number of layers. The method of claim 10, And the kernel of the initial depth is set to the vicinity of the initial depth. The method of claim 10, And the comparison determiner is configured to mix the blurred color segments of each layer in the boundary region where the depths of the layers are discontinuous using a coefficient determined according to the initial depth. The method of claim 14, And the comparison determination unit starts mixing from the largest color segment of the boundary region and sequentially performs mixing to adjacent color segments. The method of claim 10, And the image generator generates a refocused image by convolving the image with a kernel of the distance corresponding to each scene layer. The method of claim 16, And the kernel of the distance is set to the vicinity of the distance. The method of claim 10, And the image generating unit corrects the generated image in consideration of at least one of a dot diffusion function of the degree of blurring, an image enlargement according to lens movement, and a change in aperture diameter. Calculating an initial depth and the number of layers from at least two comparison images having different focal points and one reference image in full focus; Color segmenting the reference image to extract color segments, and applying and blurring the kernel of the initial depth to each color segment; Determining a color distance constituting each layer and a corresponding distance while comparing color similarities between the blurred color segments and the comparison image; Extracting a scene layer according to the distance, and merging each color segment into the scene layer; And Refocusing the scene layer to generate a refocused image; Focus control image generation method comprising a. The method of claim 19, In the calculating, the focusing image generation method of determining the initial depth proportional to the defocusing depth by obtaining the blurring degree of the comparison image and using the relationship with the blur circle modeled by a Gaussian function. The method of claim 19, And extracting a depth histogram of the initial depth of the reference image and calculating the initial depth and the number of layers in the calculating step. The method of claim 19, And the kernel of the initial depth is set to a vicinity value of the initial depth. The method of claim 19, In the determining step, the blurred color segment of each layer in the boundary region where the depth of each layer is discontinuous mixing using a coefficient determined according to the initial depth. 24. The method of claim 23, And focusing on adjacent color segments sequentially, starting from the largest color segment of the boundary region. The method of claim 19, And generating a refocus image by convolving the image with a kernel of the distance corresponding to each scene layer. The method of claim 25, And the kernel of the distance is set to a value of the vicinity of the distance. The method of claim 19, In the generating of the image, a focus control image generation method of correcting the generated image in consideration of at least one of a point diffusion function for the degree of blurring, an image enlargement according to lens movement, and a change in the aperture diameter. 21. A computer readable recording medium containing instructions for performing the method for generating the focus adjusted image of any one of claims 19 to 20.

Images