KR101714224B1 - 3 dimension image reconstruction apparatus and method based on sensor fusion - Google Patents

Abstract

The present invention relates to an apparatus and method for restoring a three-dimensional image based on a sensor fusion, comprising a data acquiring unit for acquiring a depth image and two or more color images through a depth sensor and an image sensor, A depth data processing unit for improving the resolution of the depth image to convert the depth image into variance information, a disparity data processing unit for extracting a final depth image through stereo matching of the two or more color images based on the disparity information, And a reconstruction unit for reconstructing the three-dimensional image.

Description

[0001] DENSENS IMAGE RECONSTRUCTION APPARATUS AND METHOD BASED ON SENSOR FUSION [0002]

The present invention relates to a sensor fusion-based three-dimensional image restoration apparatus and method capable of improving the performance of a three-dimensional image restoration technique using an active sensor and a stereo (multi-viewpoint) camera.

Generally, 3D reconstruction (3D reconstruction) includes stereo (multi-view) camera based 3D reconstruction and active sensor based 3D reconstruction.

The stereo camera based 3D reconstruction technique has a relatively low accuracy of the depth map. In order to improve the quality, the performance is determined according to the setting of the initial data when estimating the depth information in the continuous domain. In addition, stereo camera based 3D reconstruction techniques have high resolution and frame rate.

Active sensor based 3D reconstruction techniques have accurate depth information. The algorithm for sensing depth information is very simple and has low computational complexity. In addition, active sensor based 3D reconstruction techniques have low resolution and frame rate.

Therefore, there is a need for a three-dimensional reconstruction technique capable of improving the performance by fusing two different sensors, complementing each other, rather than improving each method.

The present invention relates to a sensor fusion-based three-dimensional image restoration apparatus and method capable of improving the performance of a three-dimensional restoration technology by fusing a stereo (multi-viewpoint) camera and an active sensor, .

According to an aspect of the present invention, there is provided a sensor fusion-based three-dimensional image restoration apparatus including a data acquisition unit for acquiring a depth image and two or more color images through a depth sensor and an image sensor, A depth data processing unit for improving the resolution of the depth image to convert the depth image into variance information, a disparity data processing unit for extracting a final depth image through stereo matching of the two or more color images based on the disparity information, And a reconstruction unit for reconstructing the three-dimensional image using the reconstructed image.

The depth sensor is an active sensor.

The depth sensor is a time-of-flight (ToF) camera composed of an infrared sensor or a laser.

And the coordinate conversion unit may calibrate an integrated coordinate system of the depth sensor and the image sensor.

And the depth data processing unit may improve the resolution of the depth image using an edge-based flattening filter.

And the variation data processing unit obtains the final depth image using a variational method.

And the depth data processing unit and the variation data processing unit are integrated into one through Z-parameterization.

Meanwhile, a 3D image reconstruction method according to an embodiment of the present invention includes acquiring a depth image and two or more color images through a depth sensor and an image sensor, warping the depth image into the color image, Converting the warped depth image into a disparity image based on the color image, improving the resolution of the depth image, stereo-matching the at least two color images based on the disparity image to obtain a final depth image, and And reconstructing the 3D image using the final depth image.

According to the present invention, the performance of the three-dimensional image restoration technique can be improved by complementing the mutual disadvantages through the fusion of the stereo (multi-viewpoint) camera and the active sensor.

Also, according to the present invention, it is possible to refine surrounding environment modeling, perimeter environment recognition, and simultaneous localization and mapping (SLAM) through high-quality 3D reconstruction based on high-quality depth information.

Also, according to the present invention, it is possible to improve performance of various applications such as obstacle detection, moving object detection (MOD) based on high-quality depth information, parking space recognition, and the like.

1 is a block diagram of a three-dimensional image restoration apparatus according to an embodiment of the present invention;
2 is a graph showing the relationship between depth information and variation information related to the present invention.
Fig. 3 is a diagram showing the lowest value convergence according to an initial value in an energy optimization process; Fig.
4 is a view for explaining a process of acquiring final depth information based on depth data filtering according to an embodiment of the present invention;
5 is a flowchart illustrating a three-dimensional image restoration method according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

The present invention intends to improve the 3D reconstruction performance by integrating the depth map based 3D reconstruction technique proposed in the field of stereo vision with an active sensor.

FIG. 1 is a block diagram of a three-dimensional image reconstruction apparatus according to an embodiment of the present invention. FIG. 2 is a graph showing a relationship between depth information and variation information related to the present invention. FIG. FIG. 8 is a diagram showing the lowest value convergence according to an initial value. FIG.

1, the 3D image reconstruction apparatus 100 includes a sensor unit 110, a data acquisition unit 120, a coordinate transformation unit 130, a depth data processing unit 140, a variation data processing unit 150, And a restoration unit 160.

The sensor unit 110 includes a depth sensor 111 for measuring the depth of an object surface and an image sensor 113 for shooting two or more color images.

The depth sensor 111 is an active sensor that directly senses depth information. As the depth sensor 111, a ToF (Time of Flight) camera using an infrared sensor or a laser can be used.

The image sensor 113 photographs a stereoscopic color image (color image) using two or more color cameras. For example, the image sensor 113 simultaneously captures a left image and a right image through two or more cameras. The image sensor 113 may be implemented as a stereo camera or a multi-view camera. An image photographed by the image sensor 113 is used to estimate a depth map.

The data acquisition unit 120 acquires a depth image (depth data, depth information) through the depth sensor 111 and acquires a color image (color information) through the image sensor 113.

The coordinate transformation unit 130 calibrates an integrated coordinate system (e.g., a world coordinate system) between the depth sensor 111 and the image sensor 113 and warps a depth image to a color image. In other words, the coordinate transformation unit 130 generates a color image in which the depth information is reflected.

The depth data processing unit 140 improves the resolution of the depth image using a filtering technique. Then, the depth data processing unit 140 converts the depth image into a mutation image (mutation data, mutation information) based on the color image. In other words, the depth data processing unit 140 converts the warped depth image into the mutated image.

The variation data processing unit 150 performs stereo matching on two or more color images acquired through the image sensor 113 based on the transformed transformed images to extract the final transformed images. Then, the variation data processing unit 150 converts the final variation image into the final depth image.

The reconstruction unit 160 reconstructs the 3D image using the final depth image.

In the present invention, the depth information and the variation information have different meanings, and the depth data processing unit 140 and the variation data processing unit 150 are divided. However, the depth data processing unit 140 and the variation data processing unit 150 may be integrated into one through Z-parameterization.

As shown in FIG. 2, when the depth information and the variation information are in inverse proportion to each other and the Z-parameterization function modeling the depth information and the parameters of the warped reference color camera are considered, the depth data (Z) It is possible to implement a 3D reconstruction technique without mutual conversion. Thus, loss or error due to conversion between data formats can be reduced. Since the 3D reconstruction technique is based on the depth information, it is possible to improve 3D reconstruction performance and computation through Z-parameterization.

The variation data processing unit 150 uses a variational method which is typically used for depth / optical flow estimation or 3D reconstruction in the conventional stereo vision field. The variational method has an advantage of estimating the depth based on the floating point based on the depth information in the continuous domain to improve the stereo matching quality.

However, 3D reconstruction using stereo matching such as variational method has a fatal disadvantage that it can fall into the local minima in the energy optimization process. This can lead to a severe depth error. May be converged to a local minimum value or converged to a global minimum value according to an initial point as shown in FIG.

On the other hand, active sensors have the advantage of providing very accurate depth information with a low resolution and frame rate (cf. Depth accuracy: less than 1 cm @ 10 m). Therefore, the depth data processing unit 140 provides an initial value (initial variation data) for solving the energy equation of the variational method for existing stereo matching using the depth information acquired through the depth sensor 110, The pixel energy can be prevented from converging to the local minimum value, thereby improving the three-dimensional reconstruction performance.

In addition, the displacement data processing unit 150 is suitable for 3D reconstruction techniques requiring sub-pixel precision because the final depth result is provided as a floating point.

The variation data processing unit 150 obtains a variation d that minimizes the energy function E as in Equation (1). The energy function E consists of a data term for the stereo image (disparity), a smoothness term, and a data term and a smoothness term for the active sensor. When solving the energy equation, it is possible to solve the convergence problem at the local minimum, obtain more precise stereo matching results, and greatly improve the 3D reconstruction performance based on this.

로 나타내고, E_s는 스테레오 영상에 대한 평활도 항으로,

로 나타낸다.Here, E _d is a data term for a stereo image,

E _s is the smoothness term for the stereo image,

Respectively.

로 나타내고, E_{s_TOF}는 능동 센서에 대한 평활도 항이다.E _TOF is a data term for the active sensor,

And E s - _TOF is the smoothness term for the active sensor.

는 확산 함수(diffusivity function)이다.λ is a smoothness term weight (weighting factor), d (disparity ) is the pixel difference between the two images corresponding points, p is the position coordinate of the pixels within the image, Ω is the image plane (image plane), I _l is a left picture, I _r D _TOF is the depth information measured by the active sensor, Ψ is the potential function,

Is a diffusivity function.

The displacement data processing unit 150 may use a different method that can be merged with the depth data of the depth sensor 111 as well as a variational method using a stereo matching method.

On the other hand, in the case of using the Z-parametrization technique described above, [Equation 1] can be modified as in Equation 2 below. That is, the energy equation can be converted into the depth data (z region) instead of the variation data (d region) as shown in [Equation 2].

4 is a view for explaining a process of acquiring the final depth information based on the depth data filtering according to an embodiment of the present invention.

The warped data from the depth image to the color image is sparse warped due to the resolution difference of each camera. In order to overcome this problem and further improve the performance, the warped depth information can be filtered based on the corresponding color image.

The depth data processing unit 140 improves the resolution of the depth image using an edge-aware smoothing filter. At this time, the depth data processing unit 140 utilizes the information of the corresponding color image. The edge-based flat filter can use various filtering methods such as Weighted Mode Filtering (WMF), Joint Bilateral Upsampling (JBU) and 3D JBU. When the filtering is performed, the depth image having the same level as the resolution of the corresponding color image can be obtained.

The variation data processing unit 150 calculates a variational method (non-linear partial differential equation, non-linear PDE) together with the stereo image acquired through the image sensor 113 using the initial variation data based on the improved depth image The final depth information can be obtained, which is directly linked to the 3D reconstruction performance enhancement.

Z-parametrization can also be applied to filtering techniques. Through this, it is possible to filter based on direct depth data and apply it directly to 3D modeling.

5 is a flowchart illustrating a 3D image reconstruction method according to an embodiment of the present invention.

First, a three-dimensional image restoration apparatus 100 (hereinafter, a restoration apparatus) obtains a depth image and a stereo color image (color image) through a depth sensor 111 and an image sensor 113 (S101).

The restoration apparatus 100 calibrates an integrated coordinate system between the depth sensor 111 and the image sensor 113 (S103).

The restoration apparatus 100 warps the depth image to the color image after the integrated coordinate system is calibrated (S105). In other words, the coordinate transformation unit 130 transforms the coordinates of the color image using the depth information.

The restoration apparatus 100 improves the resolution of the warped depth image, and converts the warped depth image to the mutation image on the basis of the color image (S107). The depth data processing unit 140 improves the resolution of the depth image using an edge-based flat filter such as WMF. The depth data processing unit 140 estimates and outputs initial variation data (initial variation image) based on the depth image with improved resolution.

The restoration apparatus 100 extracts the final mutation image through the mutated image-based stereo matching (S109). The variation data processing unit 150 stereo-matches two or more color images acquired through the image sensor 113 based on the initial variation data to obtain a final variation image. At this time, the variation data processing unit 150 performs stereo matching using a variational method.

The restoration apparatus 100 converts the final mutation image into a final depth image (S111).

The restoration unit 113 restores the 3D image based on the final depth image (S113).

100: 3D image restoration device
110:
111: Depth sensor
113: Image sensor
120: Data acquisition unit
130:
140: Depth data processor
150: Variation data processor
160:

Claims (8)

A data acquisition unit for acquiring a depth image and two or more color images through a depth sensor and an image sensor,
A coordinate converter for warping the depth image into the color image,
A depth data processing unit for improving the resolution of the depth image and converting the depth image into variance information,
A disparity data processing unit for extracting a final depth image through stereo matching of the two or more color images based on the disparity information,
And a reconstruction unit for reconstructing the 3D image using the final depth image,
The variation data processing unit obtains a variation that minimizes the energy function,
Wherein the energy function comprises a data term for a stereo image, a smoothness term, and a data term and a smoothness term for the depth sensor.
The method according to claim 1,
Wherein the depth sensor is an active type sensor.
3. The method of claim 2,
Wherein the depth sensor is a time-of-flight (ToF) camera composed of an infrared sensor or a laser.
The method according to claim 1,
Wherein the coordinate conversion unit comprises:
Wherein the integrated coordinate system of the depth sensor and the image sensor is calibrated.
The method according to claim 1,
Wherein the depth data processing unit comprises:
And the resolution of the depth image is improved by using an edge-based flat filter.
The method according to claim 1,
Wherein the variation data processing unit comprises:
And the final depth image is acquired using a variational method.
The method according to claim 1,
The depth data processing unit and the variation data processing unit,
And Z-parameterization to be integrated into one.
Acquiring a depth image and two or more color images through a depth sensor and an image sensor,
Warping the depth image to the color image,
Converting the warped depth image into a disparity image based on the color image, improving the resolution of the warped depth image,
Obtaining a final depth image by stereo matching the two or more color images based on the shifted image, and
Reconstructing the 3D image using the final depth image,
A variation that minimizes the energy function in the stereo matching is found,
Wherein the energy function is composed of a data term for a stereo image, a term of smoothness, and a data term and a smoothness term for the depth sensor.

Images