KR101358430B1 - Depth Image Generation Method and System - Google Patents

Abstract

The depth image generating system according to the present invention includes a camera unit including a reference video camera and a plurality of peripheral video cameras arranged according to a predetermined rule around the reference video camera, and generates a reference image from an output image of the reference video camera. A reference image generator, a peripheral image generator for generating a peripheral image from output images of the plurality of peripheral image cameras, and performing a multi-image matching on the reference image and the peripheral image to obtain a depth image corresponding to the reference image; It corresponds to a depth image generation system including a depth image generator to generate.

Description

Depth image generation method and system {METHOD AND SYSTEM FOR GENERATING DEPTH IMAGE}

The present invention relates to a depth image generation method and system.

Recently, with the development of 3D TV and video technology, the production of the content provided there is an issue. In particular, generation of multi-view images and virtual view images has become an important issue. Accordingly, precise depth image generation is important for generating accurate multiview images and virtual viewpoint images.

Depth images required for multi-view stereoscopic image content may generally be generated through stereo matching between multiview images. However, stereo matching technique is more likely to produce incorrect matching results for areas with a homogeneous surface, and it is difficult to extract depth information due to occlusion areas (view difference between images, shadows, etc.). Recently, a method of utilizing an active sensor such as a time-of-flight (TOF) camera has been actively studied to solve this problem.

On the other hand, the hybrid sensor-based method that combines the active sensor and passive sensor (optical camera) has solved the stereo matching problem of the passive sensor-based method. However, the mixed sensor-based method also has mechanical and technical problems.

Mechanical problems include the environmental constraints of TOF cameras. In TOF cameras, the shooting distance is limited to 0.5 ~ 5m depending on the modulation frequency range. In addition, due to the reflection characteristics of the infrared region, a depth measurement of the target object may not be properly performed. Therefore, the environment that can be photographed with the TOF camera is limited to a small indoor space. In addition, there are additional problems, such as having a smaller resolution compared to the color image, noise is included in the depth image, lens distortion occurs.

In addition, as a technical problem, a depth image acquired through a TOF camera has a relatively smaller resolution than a multiview color image. That is, when the 3D warping is performed such that the depth image has the same size as the resolution of the color image, the resulting image expresses the depth information in a discontinuous sparse structure.

In order to generate an accurate stereoscopic image according to the viewing position, first of all, the contour of the subject represented in the depth image must be accurately estimated in a continuous form. Therefore, the depth image calculated with the sparse structure needs additional supplementary work to accurately express the depth information of the target scene. For this, there are a pixel value based stereo matching method and a segmentation based stereo matching method.

In the pixel matching based stereo matching method, the initial depth image may be used as an initial search position value for matching. This can greatly reduce the processing time and significantly increase the success rate of matching compared to using only images. However, this method still has a limitation in that misalignment can occur at depth discontinuities and homogeneous texture parts.

On the other hand, in the segmentation-based stereo matching method, the image is color-divided to perform the matching between the segmentation regions, and unlike the pixel value-based stereo matching result, smooth results are obtained at depth discontinuities. In addition, correct matching can be performed even for the homogeneous texture area. However, this method considers the similar color gamuts to have the same depth, thus limiting the detailed depth representation, which may cause an error in the depth discontinuity.

In this regard, Korean Patent Laid-Open Publication No. 10-2009-0068796 (name of the TOF camera) includes a TOF which is capable of varying a sensing distance by adjusting an modulation frequency of light emitted from a light source by combining an electro-optic modulator and a microcontroller. A camera is disclosed.

In addition, Korean Patent Laid-Open Publication No. 10-2010-0122988 (name of the invention: a 3D image processing apparatus and a method thereof) includes determining a region of interest within an entire sensing region of a 3D image processing apparatus, the region of interest from a reference point. Disclosed is a 3D image processing method comprising modulating a waveform of irradiated light based on a depth up to and obtaining a depth image of the ROI using the modulated irradiated light. .

The present invention is to solve the above-mentioned problems of the prior art, some embodiments of the present invention is to install a plurality of cameras in an array form so that the occlusion area does not occur, and multi-image matching from the images obtained from the plurality of cameras It is an object of the present invention to provide a depth image generating method and system for performing a depth image.

As a technical means for achieving the above technical problem, the depth image generating system according to the first aspect of the present invention includes a reference video camera and a plurality of peripheral video cameras arranged in accordance with a predetermined rule around the reference video camera. A camera unit; a reference image generation unit generating a reference image from the output image of the reference image camera; a peripheral image generation unit generating a peripheral image from the output images of the plurality of cameras; and a multiple of the reference image and the peripheral image And a depth image generator configured to generate a depth image corresponding to the reference image by performing image registration.

 In addition, the depth image generating method according to the second aspect of the present invention comprises the steps of disposing a reference video camera, a plurality of cameras based on a predetermined rule around the reference video camera, the image output by the reference video camera Generating a reference image from the image, generating a peripheral image from the images output by the plurality of cameras, and performing a multi-image matching on the reference image and the peripheral image to generate a depth image corresponding to the reference image Steps.

According to an embodiment of the aforementioned problem solving means of the present invention, a more accurate depth image can be generated when the images acquired by arranging a plurality of cameras in a two-dimensional array are matched. That is, since an image is obtained from cameras arranged in a plurality of arrangements around the reference image, no occlusion area is generated and a mismatching problem may be overcome. Therefore, it can be effectively used to generate a multiview image and a virtual view image for 3D TV.

In addition, in generating an image applicable to a 3D TV, it is important to generate a precise depth image of a reference image. According to an embodiment of the present invention, a depth image may be generated based on a reference image and a reference image. This makes it easy to produce content applicable to 3D TV.

1 is a diagram illustrating a depth image generation system according to an exemplary embodiment.
2 is a diagram illustrating an arrangement of a reference video camera and a peripheral video camera included in a depth image generating system according to an exemplary embodiment of the present invention.
3 is a diagram illustrating a multiple image registration process.
4 is a diagram illustrating a virtual space coordinate system.
5 is a diagram illustrating a relationship between similarity according to a depth value and an extracted search region.
6 illustrates a depth image generating method according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, which will be readily apparent to those skilled in the art. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly illustrate the present invention, parts not related to the description are omitted, and similar parts are denoted by like reference characters throughout the specification.

Throughout the specification, when a part is referred to as being "connected" to another part, it includes not only "directly connected" but also "electrically connected" with another part in between . Also, when an element is referred to as "comprising ", it means that it can include other elements as well, without departing from the other elements unless specifically stated otherwise.

1 is a diagram illustrating a depth image generation system according to an exemplary embodiment of the present invention, and FIG. 2 is a diagram illustrating an arrangement of a reference image camera and a peripheral image camera included in the depth image generation system.

The depth image generation system 100 according to the present invention includes a camera unit 110, a reference image generator 120, a peripheral image generator 130, and a depth image generator 140.

The camera unit 110 includes a reference video camera 210 and a plurality of peripheral video cameras 220 disposed according to a preset rule around the reference video camera 210. In this case, the peripheral image cameras 220 may be arranged in a two-dimensional array. Conventionally, the camera is arranged left and right to capture a depth image, but the depth image generating system 100 according to the present invention arranges a plurality of peripheral image cameras 220 in a two-dimensional array including a vertical arrangement as well as a left and right arrangement. can do.

For example, referring to FIG. 2, the plurality of peripheral image cameras 220 may be arranged in a circle. In this case, the reference image camera 210 may be disposed at the center of the plurality of peripheral image cameras 220 arranged in a circle.

Referring back to FIG. 1, the reference image generator 120 generates a reference image from an output image of the reference image camera 210. In this case, the reference image is an image photographed by one camera and refers to an image that can best represent a scene.

The surrounding image generating unit 130 generates the surrounding image from the outputs of the plurality of surrounding image cameras. In this case, the surrounding image refers to an image captured by a camera other than the reference image. That is, the surrounding image refers to a plurality of array images generated from the plurality of surrounding image cameras 220 arranged in an array form. The generated surrounding image is used as an auxiliary material of the reference image.

The depth image generator 140 generates a depth image corresponding to the reference image by performing multiple image matching on the reference image and the surrounding image. Multi-image matching refers to acquiring three-dimensional depth information when searching for the same point in two or more images and pointing to the same point.

In order to generate a depth image through multi-image registration, a sensor modeling process indicating a relationship between 3D depth information and 2D image information must be preceded. At this time, the sensor modeling process is performed in the installation step of the camera. When the same point is determined between a plurality of images through the multi-image matching, the image coordinates of the same point can be converted into three-dimensional coordinates and depth information through the sensor model. The accuracy of the converted depth information increases as the number of images increases.

The depth image generator 140 includes a search region extractor 141, a similarity analyzer 143, a depth value increaser 145, and a depth value setter 147.

The search region extractor 141 sets a reference point and an initial depth value in the 3D plane coordinates, and extracts a search region corresponding to the reference image or the surrounding image based on the image coordinates corresponding to the reference point and the initial depth value. The similarity analyzer 143 analyzes and records the similarity with the search region corresponding to the surrounding image based on the search region corresponding to the reference image. The depth value increasing unit 145 increases the initial depth value by a predetermined unit. The depth value setting unit 147 compares the increased depth value with a preset maximum depth value and finalizes the depth value having the highest similarity among the recorded similarities when the increased depth value is greater than or equal to the preset maximum depth value. Set to the depth value. On the other hand, the depth value setting unit 147 sets the increased depth value as the initial depth value when the increased depth value is smaller than the preset maximum depth value, extracts the search area again, analyzes and records the similarity, and the depth Increase the value by a certain unit.

Hereinafter, multiple image registration will be described with reference to FIGS. 3 and 4. 3 is a diagram illustrating a multiple image registration process, and FIG. 4 is a diagram illustrating a virtual space coordinate system.

Referring to FIG. 4, the virtual coordinate system 420 is introduced for generating image coordinates and depth images of a camera. The sensor model is established by the relationship between the row and column information of the image coordinate system 410 and the X, Y and H of the virtual coordinate system 420 in which the depth image is to be generated. Various equations, such as physical model and mathematical model, can be used as the relational expression. A representative physical model includes a collinear equation and a mathematical model includes a direct linear transform (DLT) model. On the other hand, the scope of the present invention is not limited to the above-described sensor model, various physical and mathematical models can be used.

The collinear equation, one of the physical models, can be expressed by Equation 1.

[ Equation  One]

Referring to Equation 1, (c, r) represents image coordinates, and (X, Y, Z) represents actual coordinates. Also, (X ₀ , Y ₀ , Z ₀ ) represents the position of the sensor, f represents the focal length of the camera, and R represents the rotation matrix elements. The rotation matrix R can be expressed as Euler rotation angle. The position of the sensor and the Euler rotation angle can be calculated using several reference points and least squares method. By substituting the calculated external expression element into [Equation 1], the image coordinates corresponding to the actual coordinates (X, Y, Z) can be calculated.

Meanwhile, the DLT-based model, which is one of the mathematical models, may implement a sensor model by a simple matrix, and the image used in the depth image generation method according to the embodiment of the present invention is a center projection image obtained from a camera. ]

[ Equation  2]

 Referring to Equation 2, (c, r) represents image coordinates, (X, Y, Z) represents actual coordinates, and w represents a proportionality constant. Meanwhile, the DLT model also obtains the coefficients of the matrix M using reference points selected similarly to the collinear equation-based model. However, since the DLT-based model is a linear equation model, the least square method is not used. Therefore, when the M matrix coefficient is obtained, the image coordinates (c, r) can be calculated from the actual coordinates (X, Y, Z).

Using such a sensor model equation, the coordinate transformation between the coordinates of the virtual coordinate system and the image coordinates can be made. Depth images are generated by assigning depth information H values to the respective points X and Y of the virtual coordinate system.

The multi-image matching process will be described with reference to FIG. 3.

First, the reference point (X, Y) and the initial depth value (H) is set on the plane coordinates to generate the three-dimensional model for registration of the image (S310).

Next, image coordinates corresponding to the reference point and the initial depth value are calculated, and a search region corresponding to the reference image and the surrounding image is extracted from the image coordinates (S320). Image coordinates corresponding to the reference image and the surrounding image are calculated by substituting X, Y and H values into the sensor model formula established for the extraction of the search region. The shape of the search area is mainly set to a rectangle, and the size of the search area may be set by a user-specified size. In this way, a search region corresponding to the virtual coordinates (X, Y, Z) can be set in the image.

Next, among the extracted search areas, the similarity with the search area corresponding to the surrounding image is analyzed and recorded based on the search area corresponding to the reference image (S330). In this case, the similarity may be calculated by various methods such as a correlation coefficient of the brightness value, a difference in the amount of change in the brightness value, and an energy amount.

Next, when the calculation of the similarity is completed, the initial depth value is increased by a predetermined unit ΔH (S340). As shown below, the initial depth value is increased until it is equal to or larger than the preset maximum depth value.

Next, the increased depth value is compared with a preset maximum depth value Hmax (S350). In this case, when the increased depth value is smaller than the preset maximum depth value, a new search region is extracted from the reference image and the surrounding image using the increased depth value and the reference point, and the similarity is analyzed and recorded. This process is repeatedly performed until the increased depth value is equal to or larger than the preset maximum depth value.

On the other hand, if the increased depth value is greater than or equal to the preset maximum depth value, the depth value showing the highest similarity among the similarity information repeatedly recorded until this condition is satisfied is set as the final depth value (S360). ). This completes the determination of the depth value at the reference point.

As such, when the multi-image registration process is performed at all reference points existing in the corresponding area on the virtual coordinates, an accurate depth image may be extracted.

5 is a diagram illustrating a relationship between similarity according to a depth value and an extracted search region.

5 is a result of analyzing and recording the similarity while increasing the depth value until it is equal to or larger than the preset maximum depth value. In this case, the similarity is calculated by the correlation coefficient, and the case where the highest degree of similarity is high is 1 and the case where the degree of similarity is low is 0. Referring to FIG. 5, when the depth value is 100, the similarity is 0.6938, which is the highest in comparison with other results. In this case, when comparing the reference image and the surrounding image, it can be seen that the reference image and the surrounding image are best matched compared to other results.

6 illustrates a depth image generating method according to an embodiment of the present invention.

First, the reference video camera 210 is disposed, and the plurality of peripheral video cameras 220 are disposed according to a preset rule around the reference video camera 210 (S610). At this time, the peripheral image camera 220 may be arranged in the form of a two-dimensional array.

Next, a reference image is generated from an image output by the reference image camera 210 (S620), and a peripheral image is generated from images output by the plurality of peripheral image cameras 220 (S630).

Next, multi-image matching is performed on the reference image and the surrounding image (S640) to generate a depth image corresponding to the reference image (S650). A process of generating a depth image corresponding to the reference image by performing multi-image matching is as described with reference to FIGS. 3 and 4.

The foregoing description of the present invention is intended for illustration, and it will be understood by those skilled in the art that the present invention may be easily modified in other specific forms without changing the technical spirit or essential features of the present invention. will be. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. For example, each component described as a single entity may be distributed and implemented, and components described as being distributed may also be implemented in a combined form.

The scope of the present invention is defined by the appended claims rather than the detailed description and all changes or modifications derived from the meaning and scope of the claims and their equivalents are to be construed as being included within the scope of the present invention do.

100: depth image generation system 110: camera unit
120: reference image generator 130: peripheral image generator
140: depth image generator 141: search region extractor
143: similarity analysis unit 145: depth value increasing unit
147: depth setting unit 210: reference video camera
220: peripheral video camera 410: video coordinate system
420: virtual coordinate system

Claims (10)

In the depth image generation system,
A camera unit including a reference video camera and a plurality of peripheral video cameras disposed according to a preset rule around the reference video camera;
A reference image generator for generating a reference image from the output image of the reference video camera;
A peripheral image generator configured to generate a peripheral image from output images of the plurality of peripheral image cameras;
Including a depth image generating unit for generating a depth image corresponding to the reference image by performing a multi-image matching on the reference image and the surrounding image,
The depth image generator,
A search region extraction unit configured to set a reference point and an initial depth value in three-dimensional plane coordinates, and extract a search region corresponding to the reference image or the surrounding image based on the reference point and the image coordinates corresponding to the initial depth value;
A similarity analyzer configured to analyze and record similarity with a search region corresponding to the surrounding image based on the search region corresponding to the reference image;
A depth value increasing unit for increasing the initial depth value by a predetermined unit; and
A depth for setting the depth value having the highest similarity among the recorded similarities as the final depth value when the increased depth value is greater than or equal to the preset maximum depth value by comparing the increased depth value with a preset maximum depth value. Value setting section,
The depth value setting unit sets the increased depth value as the initial depth value when the increased depth value is smaller than the preset maximum depth value, extracts the search area again, analyzes and records the similarity, and sets the depth value. To increase the depth by a predetermined unit.
The method of claim 1,
The plurality of peripheral image cameras are arranged in a two-dimensional array including a left and right and up and down direction.
The method of claim 1,
The plurality of peripheral video cameras are arranged in a circle,
The reference image camera is a depth image generation system that is disposed in the center of the plurality of peripheral image cameras.
delete delete In the depth image generation method,
Providing a camera unit including a reference video camera and a plurality of peripheral video cameras arranged according to a preset rule around the reference video camera;
Generating a reference image from the image output by the reference image camera;
Generating a peripheral image from the images output by the plurality of peripheral image cameras;
Generating a depth image corresponding to the reference image by performing multi-image matching on the reference image and the surrounding image;
Generating the depth image
Setting a reference point and an initial depth value in three-dimensional plane coordinates,
Extracting a search region corresponding to the reference image or the surrounding image based on the reference point and the image coordinates corresponding to the initial depth value;
Analyzing and recording similarity with the search region corresponding to the surrounding image based on the search region corresponding to the reference image;
Increasing the initial depth value by a predetermined unit; and
The depth value having the highest similarity among the recorded similarities is set as the final depth value when the increased depth value is greater than or equal to the preset maximum depth value by comparing the increased depth value with the preset maximum depth value. Including the steps of:
And extracting the search area again by setting the increased depth value as the initial depth value when the increased depth value is smaller than the preset maximum depth value.
The method according to claim 6,
And the plurality of peripheral image cameras are arranged in a two-dimensional array including left and right and up and down directions.
The method according to claim 6,
The plurality of peripheral video cameras are arranged in a circle,
The reference video camera is a depth image generating method that is disposed in the center of the plurality of peripheral video cameras.
delete delete

Images