CN103607584B - Real-time registration method for depth maps shot by kinect and video shot by color camera - Google Patents

Abstract

The invention proposes a stable and rapid real-time registration method for registering a depth map captured by kinect in real time and a video shot by a high-definition camera. The invention eliminates the step of estimating the internal parameters of the depth camera, reduces the parameters to be obtained, and enhances the stability of the algorithm. The present invention uses a linear optimization framework to solve the problem, and can obtain the global optimal solution in one step, and the calculation efficiency of the algorithm is greatly improved. Although the present invention removes the step of estimating the internal parameters of the depth camera in the traditional algorithm, the execution efficiency of mapping the depth information to the video is not affected, and the depth information captured by the kinect can still be mapped to the video in real time; and due to the defined mixing The parameters have good mathematical properties, and the matrix QR decomposition technology can still be used to reversely calculate the internal parameters of the depth camera for other algorithm applications.

Description

A real-time registration method of depth map captured by kinect and video captured by color camera

technical field

The invention relates to a real-time registration method of a depth map captured by a kinect and a video captured by a color camera.

Background technique

With the improvement of data storage and transmission technology and the advancement of camera lens technology, high-definition cameras and high-definition network cameras have gradually entered people's field of vision. Using these devices, people can easily obtain high-quality video material and communicate with others in high-definition video. However, in recent years, with the development of augmented reality technology and stereoscopic display technology, traditional two-dimensional high-definition video can no longer meet people's needs. High-realistic interaction with computer-synthesized virtual objects during communication, all of which rely on deep video generation technology.

The deep generation technology of video is a classic problem in the field of computer vision. In order to restore the depth information lost in the process of video shooting, countless scientists have devoted themselves to this field and proposed a series of classic algorithms. But so far, no algorithm can guarantee the correctness of complex scene depth generation while ensuring the real-time performance of video depth generation. Moreover, all current evidence shows that there is still a long time before the generation of video depth generation algorithms that satisfy both "real-time" and "correctness".

Therefore, people tend to use special equipment to capture video and depth at the same time. Although this can satisfy the "real-time" and "correctness" of depth information at the same time, such special equipment is often bulky and expensive, which is common users cannot bear.

In recent years, Microsoft has launched a lightweight depth capture device, kinect, which is small in size and cheap, and can capture scene depth in real time. This device has revived people's hope for video depth generation. But unfortunately, although the kinect device itself provides a color camera to capture video data, the resolution of this camera is very low and cannot meet people's needs for high-definition video. While using an additional high-definition camera and kinect to capture high-definition video and depth information separately, the captured video and depth data cannot be aligned spatially due to differences in the viewing angles of the two devices.

In recent years, many scholars have proposed algorithms to overcome this spatial difference. However, they all have the problems of being easily affected by the noise of depth data and low computational efficiency.

Contents of the invention

The technical content to be solved by the present invention is to propose a stable and fast real-time registration method for registering the depth map captured by the kinect in real time with the video shot by the high-definition camera.

For this reason the present invention adopts following technical scheme, it may further comprise the steps:

(1), the high-definition color camera and the kinect depth camera are fixed respectively, and the positions fixed respectively are arbitrary positions, and the real-time registration method of the present invention can handle the accuracy under the situation that there is any larger angle of view difference between the kinect and the depth camera Registration, and set synchronous shooting signal;

(2) Randomly place the checkerboard for calibration in front of the color camera and the kinect depth camera, and perform synchronous shooting to obtain the color image and depth image of the checkerboard at the same time. Repeat this operation many times until you get a set of synchronous shooting results from different angles and different distances;

(3) Manually mark one or several arbitrary-shaped areas in the area corresponding to the checkerboard grid in the depth map captured by kinect each time. The user only needs to mark one or more arbitrary-shaped areas in the flat area of the depth map. Unlike the traditional method that needs to accurately mark the four corners of the tablet, it effectively reduces the impact of depth camera noise on the algorithm, and ensures that the total number of pixels in the marked area on each depth map is not less than 10 pixels. Calculate the product of the homogeneous coordinate representation of the two-dimensional coordinates of the jth pixel point in any area manually marked in the depth map of the kinect captured for the i time and the depth of the point, denoted as . and denote the set of all coordinates as ;

(4), for the coordinate set obtained for the i-th shooting , bringing all its elements into the spatial plane equation: Perform plane fitting. and calculate the normal vector of the resulting plane . for collections all elements in ,calculate mean of ,Will all satisfied in points are filtered out, and the remaining points are used as a new set of coordinates ;

(5) Using Zhang Zhengyou's classic color camera calibration algorithm to calculate the internal parameters of the high-definition color camera and the relative position of the color camera and the checkerboard in each shot, using the rotation matrix and translation vector to represent the relative positional relationship of the i-th shooting;

(6), set the mixing parameters of 3*3 size and a 3D vector is an unknown quantity; the point coordinates obtained in (4) and (5) after the noise filtering step and the rotation parameter of the i-th shot As a known quantity, set up a plane constrained linear equation: ;

(7), according to point coordinates The corresponding depth imposes a penalty coefficient on the equation in (6) , the smaller the depth value, the larger the penalty coefficient, so that the constraint equation corresponding to the point with a smaller depth has a larger weight in the equation system;

(8), establish constraint equations , use the linear optimization framework to solve: 1) represents the translation parameter T between the color camera and kinect; 2) the mixing parameter H composed of the internal parameters of the depth camera and the relative rotation between the color camera and kinect;

(9), using the two parameters H and T obtained in (6) to map the depth signal captured by the kinect to the video captured by the high-definition camera in real time. The specific method is: use kinect to continuously shoot the depth map, and calculate the homogeneous coordinate representation of its two-dimensional coordinates for each pixel on each image and the depth of the point the product of , and then calculate the three-dimensional coordinates of the point in the camera coordinate system: , project the point onto the color image, and get its projected position on the color image and the corresponding depth , complete the real-time mapping of the depth signal captured by kinect to the video captured by the high-definition camera, where: . Since different pixels on the depth map may map to the same coordinate on the color image, for this case, the minimum depth will be saved as the depth of the point on the color image.

On the basis of adopting the above technical solutions, the present invention can also adopt the following further solutions:

In step (7), according to point coordinates The corresponding depth is used to determine the weight of the plane constrained linear equation in the entire equation system, and the penalty coefficient As follows:

(Unit: m)

Where depth refers to the depth corresponding to the pixel.

In step (8), the specific construction method of establishing the constraint equation AX=b is: the mixing parameter of size 3*3 and a 3D vector combined into a 12-dimensional vector , to establish a system of constraint equations in the sense of least squares: , using the linear optimization framework to solve: where, remember , , , then in the matrix the corresponding row in Constructed as:

Correspondingly, the column vector for:

.

In step (9), the method of mapping the obtained H and T parameters in real time from the depth signal captured by the kinect to the video captured by the high-definition camera is: use the kinect to continuously shoot the depth map, and on each image For each pixel of , calculate the homogeneous coordinate representation of its two-dimensional coordinates and the depth of the point the product of , and then calculate the three-dimensional coordinates of the point in the camera coordinate system: , project the point onto the color image, and get its projected position on the color image and the corresponding depth , complete the real-time mapping of the depth signal captured by kinect to the video captured by the high-definition camera, where: . Since different pixels on the depth map may map to the same coordinates on the color image, in this case, the minimum depth will be saved as the depth of the point on the color image

Due to the adoption of the technical solution of the present invention, the present invention has the following beneficial effects:

(1) The traditional algorithm needs to estimate the internal parameters of the depth camera, but since the depth signal captured by kinect contains a lot of noise, this step will be affected by the noise, resulting in the instability of the overall algorithm. The invention eliminates the step of estimating the internal parameters of the depth camera, reduces the parameters to be obtained, and enhances the stability of the algorithm.

(2) In order to reduce the impact of the estimation error of the internal parameters of the depth camera on the algorithm, the traditional algorithm uses a nonlinear optimization framework to iteratively correct the parameters such as the internal parameters of the depth camera. But this nonlinear optimization is slow and often falls into local optimum. The present invention uses a linear optimization framework to solve the problem, and can obtain the global optimal solution in one step, and the calculation efficiency of the algorithm is greatly improved.

(3), although the present invention removes the step of estimating the internal parameters of the depth camera in the traditional algorithm, the execution efficiency of mapping the depth information to the video is not affected, and the depth information captured by the kinect can still be mapped to the video in real time.

(4), although the present invention removes the step of estimating the internal parameters of the depth camera in the traditional algorithm, due to the good mathematical properties of the defined mixing parameters, the internal parameters of the depth camera can still be obtained by using the matrix QR decomposition technology for other algorithm applications.

Description of drawings

Fig. 1 is the overall flowchart of the method provided by the present invention.

Fig. 2 is an example diagram of the effect of the present invention. The present invention maps the synchronously captured depth map (upper left) to a color image (lower left). For convenience of observation, the present invention displays the depth map in color. The result of the mapping is shown in the figure on the right. It can be clearly seen that the color image and the depth map fit closely on the edge, and a pixel point is taken from the human ear on the depth map, and the pixel point is accurately mapped by the algorithm of the present invention to the ears of the characters in the color image.

Detailed ways

First define the abbreviations that will be used in the following instructions:

: Camera internal reference

: 3D rotation matrix of the checkerboard coordinate system relative to the camera coordinate system;

: The third column vector of the 3D rotation matrix of the checkerboard coordinate system relative to the camera coordinate system;

: The translation of the checkerboard coordinate system relative to the camera coordinate system;

: The homogeneous coordinate representation of the two-dimensional coordinates of the jth point on the checkerboard plate in the depth image captured by kinect in the ith sample and the depth of the point the product of

: Rotation by camera coordinate system and kinect coordinate system And and kinect internal reference The resulting mixing parameters, ;

: The translation between the camera coordinate system and the kinect coordinate system. .

The technical solution of the present invention will be further described below in conjunction with the accompanying drawings.

Fig. 1 is a basic flow chart of the present invention. The invention maps the depth map captured by kinect in real time to the video captured by the high-definition camera by establishing a plane-constrained linear equation set, solving the mixing parameter H and the translation parameter T. Each flow process of the present invention is described in detail below:

(1) Fix the high-definition color camera and kinect depth camera respectively, and set the synchronous shooting signal:

The present invention requires that the positional relationship between the camera and the kinect be kept fixed. At the same time, in order to map the depth map captured by kinect to the image captured by the camera at the same time, it is necessary to set a synchronous shooting signal between the two acquisition devices. Using software synchronization, the host sends a 60HZ square wave signal, and the rising edge of the clock triggers synchronous shooting.

(2), Synchronously shoot checkerboard:

Randomly place the checkerboard for calibration in front of the camera, and shoot synchronously to obtain the color image and depth image of the checkerboard at the same time. Repeat this operation many times until you get a set of simultaneous shooting results from different angles and different distances. In the experiment, a checkerboard with a size of 13*9 was used, 15 distances were selected, and checkerboards with 3 to 5 angles were photographed at each distance, and 48 samples were obtained.

(3) Manually mark the tablet:

Since plane constraints need to be used during parameter calculation, it is necessary to manually mark the area where the checkerboard plate collected by kinect in step (2) is located. It should be stated that it is not necessary to mark the entire plate area completely, but a closed curve of any shape can be used to mark the area inside the plate. The randomness of this marking effectively reduces the difficulty of human interaction. When marking, it is necessary to ensure that the total number of pixels in the marked area on each depth map is not less than 10 pixels. Calculate the product of the homogeneous coordinate representation of the two-dimensional coordinates of the jth pixel point in any area manually marked in the depth map of the kinect captured for the i time and the depth of the point, denoted as . and denote the set of all coordinates as .

(4), filter out noise points:

The depth map that Kinect shoots contains a large amount of noises, although the method of the present invention does not need to accurately mark the four corners of checkerboard as in the traditional method, but marks the area of arbitrary shape in the checkerboard area, but the noise of kinect still can affect described in the present invention The mapping correctness of the algorithm is affected. Therefore, before using the points in any region to establish constraints, it needs to be processed to filter out noise. Specifically, for the coordinate set obtained in the i-th shooting , bringing all its elements into the spatial plane equation: Perform plane fitting. and calculate the normal vector of the resulting plane . for collections all elements in ,calculate mean of ,Will all satisfied in points are filtered out, and the remaining points are used as a new set of coordinates .

(5), camera calibration:

Using Zhang Zhengyou's classic camera calibration algorithm to calculate the internal parameters of the camera ; and calculate the 3-dimensional rotation matrix of the checkerboard coordinate system relative to the camera coordinate system in each sample recorded in step (2) , and record its 3rd column vector , as the representation of the normal of the checkerboard in the i-th sample in camera coordinates. Simultaneously record the translation of the checkerboard coordinate system relative to the camera coordinate system in each sample .

(6) Establish plane constraint equation:

For each point on the checkerboard slab in the kinect depth map , the equation can be established:

in, Indicates the representation of the normal direction of the checkerboard in the i-th sample in the camera coordinates; Indicates the translation of the checkerboard coordinate system in the i-th sample relative to the camera coordinate system; Indicates the homogeneous coordinate representation of the two-dimensional coordinates of the jth point on the checkerboard plate in the depth map captured by kinect in the ith sample and the depth of the point The product of , wherein the plate area is determined in step (3); For the rotation by the camera coordinate system and the kinect coordinate system And and kinect internal reference The resulting mixing parameters, ; Indicates the translation between the camera coordinate system and the kinect coordinate system. and is the parameter to be requested, where is a 3D matrix, is a 3-dimensional vector, with a total of 12 unknowns.

For the establishment of equations, we need to pay attention to two points: First, since each point on the plane can establish an equation, the number of equations is obviously more than the number of unknowns, and even only one sample can be solved, but in order not to be in a certain location, but to satisfy the planar constraint in the entire space, still requires all samples to participate in the operation. Second, for some distant points on the plate, the depth measurement measured by kinect will be rough. When establishing equations for these points, add a penalty coefficient to punish.

In step (8), the specific construction method of establishing the constraint equation AX=b is: the mixing parameter of size 3*3 and a 3D vector combined into a 12-dimensional vector . Set up a system of constraint equations in the least squares sense: , solved using a linear optimization framework. Among them, remember , , , then in the matrix the corresponding row in Constructed as:

Correspondingly, the column vector for:

Using the least squares algorithm to solve the equation, we get and .

(7) The depth map is generated in real time:

Use kinect to continuously shoot depth maps, and calculate the homogeneous coordinate representation of its two-dimensional coordinates for each pixel on each image and the depth of the point the product of . Then you can get the three-dimensional coordinates of the point in the camera coordinate system: , project the point onto the color image, and get its projected position on the color image and the corresponding depth ,in: . Since different pixels on the depth map may be mapped to the same coordinate on the color image, in this case, the minimum depth is saved as the depth of the point on the color image.

The result of the algorithm is shown in Figure 2. The invention maps the synchronously captured depth map (upper left) to a color image (lower left). The result of the mapping is shown in the figure on the right. It can be clearly seen that the color image and the depth map fit closely on the edge, and a pixel point 1 is taken from the human ear on the depth map, and the pixel point 1 is accurately detected by the algorithm of the present invention. is mapped onto the ear 2 of the person in the color image, and through this step, the depth of the human ear in the color camera is obtained, and the reference numeral 3 represents the depth of the pixel point 1 in the depth camera, and the reference numeral 4 Indicates the depth of the pixel in the color camera.

Claims (4)

1. A real-time registration method for a depth map shot by a kinect and a video shot by a color camera is characterized by comprising the following steps:

(1) respectively fixing the high-definition color camera and the kinect depth camera, and setting synchronous shooting signals;

(2) randomly placing the checkerboards for calibration in front of the color camera and the kinect depth camera, and synchronously shooting to obtain color images and depth images of the checkerboards at the same time; repeating the operation for multiple times until a group of synchronous shooting results with different angles and different distances are obtained;

(3) manually and randomly marking one or a plurality of regions with any shapes in the region corresponding to the checkerboard in the depth map shot by kinect each time, and ensuring that the sum of the pixel number of the region marked on each depth map is not less than 10 pixels; calculating the product of homogeneous coordinate representation of two-dimensional coordinates of jth pixel point in any manually marked area and the depth of the point in the ith shot kinect depth map, and recording the product asAnd recording the set of all coordinates as；

(4) And for the coordinate set obtained by the ith shootingAll its elements are substituted into the spatial plane equation:performing plane fitting, and calculating to obtain normal vector of plane(ii) a For collectionsAll elements in (1)CalculatingMean value ofWill beAll satisfyFiltering out the remaining points as a new coordinate set；

(5) Calculating internal parameters of high-definition color camera by using Zhang Zhengyou classic color camera calibration algorithmAnd relative position relationship between the color camera and the checkerboard in each shooting, using a rotation matrixAnd translation vectorTo indicate the relative positional relationship of the i-th shot;

(6) let the mixing parameter be 3 x 3And a 3-dimensional vectorIs an unknown quantity; the point coordinates obtained in the two steps (4) and (5) and subjected to the noise filtering stepAnd rotation parameter of the ith shotAs a known quantityEstablishing a plane constraint linear equation:；

(7) according to the coordinates of the pointsCorresponding depth imposes penalty factor on equation in (6)The smaller the depth value, the larger the penalty coefficient is, so that the constraint equation corresponding to the point with the smaller depth has larger weight in the equation set;

(8) establishing a constraint equation setSolving using a linear optimization framework: 1) represents a translation parameter T between the color camera and kinect; 2) a blending parameter H consisting of depth camera internal parameters and relative rotation between the color camera and kinect;

(9) and (5) mapping the depth signal captured by kinect to the video captured by the high-definition camera in real time by using the H parameter and the T parameter obtained in the step (6).

2. The method as claimed in claim 1, wherein the real-time registration of the depth map captured by kinect with the video captured by color camera comprises: in step (7), based on the point coordinatesDetermining the weight and penalty coefficient of the plane constraint linear equation in the whole equation system according to the corresponding depthAs follows:

(unit meter)

Where depth refers to the depth to which the pixel point corresponds.

3. The method as claimed in claim 1, wherein the real-time registration of the depth map captured by kinect with the video captured by color camera comprises: in step (8), the specific construction manner for establishing constraint equation AX = b is as follows: mixing parameters of 3 x 3 sizeAnd a 3-dimensional vectorCombined into a 12-dimensional vectorEstablishing a constraint equation set in the least square sense:solving using a linear optimization framework: wherein, note，，Then in the matrixCorresponding row inThe structure is as follows:

accordingly, the column vectorComprises the following steps:

。

4. the method as claimed in claim 1, wherein the real-time registration of the depth map captured by kinect with the video captured by color camera comprises: in step (9), the method for mapping the depth signal captured by kinect to the video captured by the high-definition camera in real time by using the obtained two parameters of H and T is as follows: continuously shooting a depth map by using kinect, and calculating a homogeneous coordinate representation of two-dimensional coordinates of each pixel on each imageAnd the depth of the pointProduct of (2)Then, the three-dimensional coordinates of the point in the camera coordinate system are calculated:projecting the point on the color image to obtain the projection position on the color imageAnd corresponding depthAnd mapping the depth signal captured by kinect to the video captured by the high-definition camera in real time, wherein:since different pixels on the depth map may map to the same coordinate on the color image, for this case the minimum depth will be saved as the depth of the point on the color image.