CN106910222A - Face three-dimensional rebuilding method based on binocular stereo vision - Google Patents

Abstract

The invention provides a method for three-dimensional face reconstruction based on binocular stereo vision, comprising: constructing a binocular stereo vision system, wherein the binocular stereo vision system includes a left camera device and a right camera device; The stereo vision system collects face images, obtains the left and right images, performs stereo correction on the left and right images; detects the face areas in the left and right images; performs face keying on the face areas in the left and right images Point positioning and matching; use the matched face key points to initialize the dense disparity of the face to obtain the initialization disparity; smooth the initialization disparity through the stereo matching algorithm; and combine the smoothed initialization disparity to perform 3D reconstruction of the face. The present invention adopts an ordinary camera as an image acquisition device, which can save the cost of the device and improve the accuracy of the three-dimensional reconstruction result of the human face.

Description

3D Face Reconstruction Method Based on Binocular Stereo Vision

technical field

The invention relates to the technical field of binocular stereo vision, in particular to a three-dimensional face reconstruction method based on binocular stereo vision.

Background technique

The three-dimensional structure information of the face is widely used in face image processing, such as face recognition, face tracking, face alignment and so on. In the past few years, researchers at home and abroad have proposed many methods for 3D face reconstruction. One method is to collect 3D face structures based on hardware devices, such as using 3D laser scanners and structured light scanners. This type of method can obtain high-precision three-dimensional structure data of the face, but it needs to use expensive hardware equipment, which makes this method have many limitations such as high cost, inflexibility, and high complexity, and is not suitable for ordinary applications. Another type of method is a 3D face reconstruction method based on video or multi-angle photos. This type of method is low in cost, flexible in use, and can be applied in daily life.

The 3D face reconstruction based on binocular stereo vision belongs to the second type of method. How to use binocular stereo images to reconstruct the 3D structure of the face is still a challenging problem. This method only uses a pair of images. They come from the left and right cameras of the binocular camera to recover the 3D information of the face. There are currently many binocular matching methods, including global stereo matching algorithms and local stereo matching algorithms, such as BM algorithm, SGM algorithm, Meshstereo algorithm, etc. However, the low-texture problem in the face area is the main problem to be solved in the 3D reconstruction of the face. Therefore, a binocular stereo matching method specifically for the face structure is proposed, such as a block matching method based on face priors, a seed point growth method, etc. to restore the three-dimensional face structure. It is a camera acquisition device in the face area, which can obtain higher and more accurate results, or use a camera with a normal resolution (640 acquisition device, but the accuracy of the obtained face is relatively poor. Since the face is a curved surface structure, based on the parallax plane The stereo matching algorithm can restore the surface structure very well. Combined with the initial structure parallax of the face, the three-dimensional structure of the face is obtained through the stereo matching algorithm.

Contents of the invention

In view of the above technical problems, the present invention provides a method for three-dimensional face reconstruction based on binocular stereo vision.

According to one aspect of the present invention, a method for three-dimensional reconstruction of human face based on binocular stereo vision is provided

The face three-dimensional reconstruction method of the binocular stereo vision comprises:

Step A, building a binocular stereo vision system, wherein, the binocular stereo vision system includes a left camera and a right camera;

Step B, using the binocular stereo vision system to collect face images, obtain the left image by the left camera device, and obtain the right image by the right camera device; perform stereo correction on the left image and the right image; detect the left image and the right image face area;

Step C, for the face areas in the left image and the right image, perform positioning and matching of key points of the face;

Step D, using the matched face key points to initialize the dense disparity of the face to obtain the initialization disparity;

Step E, smoothly initializing the disparity through a stereo matching algorithm; and

Step F, combining the smoothed initialization parallax to perform 3D face reconstruction.

Preferably, in the face three-dimensional reconstruction method of binocular stereo vision of the present invention, the left camera and the right camera are cameras or cameras of the same model:

Step A includes:

Sub-step A1: Calibrate the left and right camera devices to obtain their internal parameters, distortion parameters and external parameters corresponding to three-dimensional points; based on the external parameters of the left and right camera devices corresponding to three-dimensional points, obtain the binocular stereo vision system Rotation matrix R, translation matrix T;

Sub-step A2, based on the internal parameters and distortion parameters of the left camera device, the internal parameters and distortion parameters of the right camera device, and the obtained rotation matrix R and translation matrix T of the binocular stereo vision system, the left correction matrix and the right correction matrix are obtained matrix;

Among them, the left correction matrix is used to perform stereo correction on the left image, and the right correction matrix is used to perform stereo correction on the right image. The matching points are on the same scan line;

Performing stereo correction on the left image and the right image in step B includes: performing stereo correction on the left image by using the left correction matrix, and performing stereo correction on the right image by using the right correction matrix.

Preferably, in the face three-dimensional reconstruction method of binocular stereo vision of the present invention, sub-step A1 includes:

Sub-step A1a, obtaining 10 to 20 groups of planar chessboard images with different angles and directions;

In the sub-step A1b, checkerboard monitoring is performed on the obtained planar checkerboard image, and the corner points in the checkerboard grid corresponding to the three-dimensional points are located; according to Zhang Zhengyou's calibration method and the Brown algorithm, the internal parameters of the left camera and the right camera are obtained , distortion parameters and external parameters corresponding to the corner points of the chessboard;

Wherein, the external parameters corresponding to the checkerboard corners include: the rotation matrix R _l of the left camera and the translation matrix T _l of the left camera; the rotation matrix R _r of the right camera and the translation matrix T _r of the right camera;

In the sub-step A1c, the corner point Q is input into the camera coordinate system of the left and right camera devices, corresponding to the coordinate points Q _l and Q _r of the left and right images, there is the following relationship:

Q _l ＝R _l Q+T _l

Q _r ＝R _r Q+T _r

Q _l ＝R ^T (Q _r -T)

Among them, Q is the three-dimensional coordinate of the corner point Q in the world coordinate system, the left picture is the image obtained by the left camera device, and the right picture is the image obtained by the right camera device, and the following relationship is derived:

R＝R _r (R _l )

Factory＝T _r -RT _l

According to the joint views of the corner points of a given checkerboard and the external parameter matrix corresponding to each corner point, the rotation matrix R and the translation matrix T are solved; due to image noise and rounding errors, each pair of checkerboards will make The results of R and T are slightly different. The median value of the R and T parameters is selected as the initial value of the real result, and the Levenberg-Marquardt iterative algorithm is used to find the minimum projection error of the checkerboard corner points on the views of the two camera devices, and the rotation matrix R is returned. and the result of the translation matrix T.

Preferably, in the face three-dimensional reconstruction method of binocular stereo vision of the present invention, in substep A2, utilize Bouguet algorithm, internal parameter and distortion parameter of left camera device, the internal parameter and distortion parameter of right camera device, and obtained The rotation matrix R and the translation matrix T of the binocular stereo vision system are used to obtain the left correction matrix and the right correction matrix.

Preferably, in the method for three-dimensional face reconstruction based on binocular stereo vision of the present invention, in step B, a Haar-Adaboost classifier is used to detect the face regions on the left image and the right image.

Preferably, in the face three-dimensional reconstruction method of binocular stereo vision of the present invention, step C comprises:

Sub-step C1, locating the face key points of the face area in the left image and the right image;

Sub-step C2, match the relevant face key points in the left image and the right image, and obtain the prior sparse topological information of the face-the face shape SL in the left image and the face shape SR in the right image, where the face shape SL in the left image contains the left Face key point coordinates {(lx _i , ly _i ), i∈[1, n]}, right face shape SR contains right face key point coordinates {(rx _i , ry _i ), i∈[1, n]} , n represents the total number of key points.

Preferably, in the binocular stereoscopic human face three-dimensional reconstruction method of the present invention, in the sub-step C1, the key points of the human face in the left image and the right image are located in combination with the regression tree set ERT algorithm.

Preferably, in the face three-dimensional reconstruction method of binocular stereo vision of the present invention, step D comprises:

Sub-step D1, calculating the disparity of the matched face key points in the left image and the right image;

Sub-step D2, using the disparity of the matched face key points in the left image and the right image, calculate the disparity of other points in the left image and the right image except the matched face key points, so as to realize the densification of face disparity, Get the initialized disparity.

Preferably, in the face three-dimensional reconstruction method of binocular stereo vision of the present invention, in the sub-step D1, the parallax of the matching face key points in the left image and the right image is calculated according to the following formula:

Among them, lx _i represents the column of the i-th face key point in the left image, rx _i represents the column of the face key point that matches it in the right image, and the disparity D(p _i ) represents the corresponding face key The absolute value of the column coordinate difference of the point, i=1, 2, 3, ..., n, n is the number of matching key points.

Preferably, in the face three-dimensional reconstruction method of binocular stereo vision of the present invention, sub-step D2 includes:

Sub-step D2a, using the key points located on the face to perform Delaunay triangulation on the face, and dividing the face into n triangles;

In the sub-step D2b, for each triangle, the parallax of the points inside the triangle is obtained through the parallax of the three vertices of the triangle, so as to realize the densification of the parallax of the face and obtain the initialization parallax.

Preferably, in the face three-dimensional reconstruction method of binocular stereo vision of the present invention, in the sub-step D2b, obtaining the parallax of the inner point of the triangle through the parallax of the three vertices of the triangle includes:

The three vertices of the triangle are p1, p2, p3. For point p in the triangle, there is a u and v, so that there is a relationship between point p and points p1, p2, p3 as in the formula:

p _x ＝(1-uv) p1 _x +u p2 _x +v p3 _x

p _y ＝(1-uv)·p1 _y +u·p2 _y +v·p3 _x

By substituting the coordinates of point p (p _x , p _y ), the coordinates of point p1 (p1 _x , p1 _y ), the coordinates of point p2 (p2 _x , p2 _y ), and the coordinates of point p3 (p3 _x , p3 _y ) into The formula solves the u and v parameters;

The parallax D(p) of the point p is obtained by interpolating from the formula:

D(p)=(1-u-V)·D(p1)+u·D(p2)+V·D(p3).

Preferably, in the face three-dimensional reconstruction method of binocular stereo vision of the present invention, step E comprises:

Sub-step E1, use the cost calculation to obtain the similarity of the corresponding points in the left and right images, and obtain the matching cost of the corresponding points in the left and right images;

Sub-step E2, using cost aggregation calculation, matching the cost of the corresponding points in the left and right graphs, to obtain the aggregated cost of the points around the corresponding point;

In sub-step E3, the plane with the minimum aggregate matching cost is selected as the optimal plane, and the disparity of the pixels is reversely calculated.

Preferably, in the face three-dimensional reconstruction method of binocular stereo vision of the present invention, the calculation method of step E1 matching cost is:

Point G is the center point, the brightness value I(G) of point G, and N(G) represents the brightness value I(G) of the pixels in the neighborhood N(G) in a square neighborhood with G as the center and a radius of d ') is the brightness value of the pixel in the neighborhood N(G). If the brightness value of G' of the point in the neighborhood is smaller than G, the value at the position of point G' is recorded as 1, otherwise it is 0, that is is ε(G, G′);

Connect the values of the points in the neighborhood in series to complete the transformation. The value is Rτ(G), and the formula is as follows:

After such a transformation is performed on each point, the similarity between the points in the left and right images is the Hamming distance H for calculating the transformation value of the corresponding point. The smaller the distance, the higher the similarity, and then calculate the pixel J. The matching cost between K is:

ρ(J,K)=H(R(J),R(K));

In the sub-step E2, the cost aggregation calculation formula is:

Where K is a pixel, J is the next point of W _K , W _K represents the square window of pixel K, d _K = a _f K _x + b _f K _y + c _f represents the parallax of pixel K, W(J, K ) is the weight function;

The weight function W(J, K) takes into account the edge problem in the square window, and uses the color similarity of two points to define the weight function. If the colors are similar, a high weight is given, otherwise, a low weight is given, expressed as the formula Shown:

where γ is a defined parameter and ||I _J -I _K || represents the L1 norm of the RGB color space of J and K.

In sub-step 3, the parallax calculation includes optimal plane calculation, and its calculation formula is:

Among them, F represents all parallax planes, f _J is the optimal plane of point J, and the parallax of the pixel is inversely obtained through the parameters of the optimal plane.

Preferably, in the face three-dimensional reconstruction method of binocular stereo vision of the present invention, step F comprises:

Select any two-dimensional point S, whose coordinates are (x, y), and the associated parallax d, project this point into three-dimensional, and the three-dimensional coordinates are (X/W, Y/W, Z/W), and the matrix is obtained:

Among them, O is the projection matrix,

In O, c _x represents the x coordinate of the left image of the principal point, c _y represents the y coordinate of the left image of the principal point, T _x represents the baseline length between two cameras, f represents the focal length of the camera device, and c _x ′ represents the principal point the x-coordinate of the right image;

The three-dimensional coordinates corresponding to the pixel points are obtained through the above formula, so as to obtain the three-dimensional structure of the face.

It can be seen from the above technical solutions that the method for three-dimensional face reconstruction based on binocular stereo vision in the present invention has at least one of the following beneficial effects:

(1) The present invention is aimed at the curved surface structure of human face, adopts the three-dimensional matching algorithm of parallax plane, restores smoother three-dimensional structure of human face;

(2) The present invention adopts the prior structure of the human face, and initializes the parallax plane based on this, thereby improving the accuracy of the three-dimensional reconstruction result of the human face;

(3) The present invention uses a common camera as the image acquisition device, which can save the cost of the device.

Description of drawings

FIG. 1 is a flow chart of a three-dimensional face reconstruction method based on binocular stereo vision according to an embodiment of the present invention.

FIG. 2 is a flow chart of sub-step A1 of the method for three-dimensional face reconstruction based on binocular stereo vision according to an embodiment of the present invention.

FIG. 3 is a flow chart of sub-step D2 of the method for three-dimensional face reconstruction based on binocular stereo vision according to an embodiment of the present invention.

Figure 4 is a geometric model of stereo matching vision.

detailed description

The invention provides a three-dimensional face reconstruction method based on binocular stereo vision. The 3D face reconstruction method only uses a pair of images, which come from the left camera and the right camera of the binocular camera, so as to restore the 3D information of the face.

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be described in further detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

Embodiments of the present invention will be described with reference to FIGS. 1 to 4 .

The invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

As shown in Figure 1, first, use two cameras to build a parallel binocular stereo vision system, calibrate the internal and external parameters of the two cameras, and calibrate the binocular cameras so that the matching points of the cameras are in the same row, and then shoot faces at the same time For an image, use the Viola-Jones face detection method, combined with the Ensemble of RegressionTrees (ERT) algorithm to locate the key points of the face on the left and right images, and then match the left and right key points to restore the sparse disparity estimation of the face. The linear interpolation algorithm preliminarily estimated the dense disparity of the face, combined with the Patchmatch stereo matching algorithm based on the disparity plane to smooth the obtained disparity, and reconstructed the dense disparity of the face. According to the calibrated parameters and the parallax of the corresponding points, the dense 3D point cloud information of the face is restored.

The specific implementation process is described in blocks as follows:

Step A: build a binocular stereo vision system, wherein the binocular stereo vision system includes a left camera and a right camera;

Use two cameras of the same type and place them in parallel to build a binocular stereo vision system. And the parameters of the two cameras are calibrated and corrected.

In order to reconstruct the three-dimensional structure of the face through the stereo matching algorithm, the internal parameters of the camera and the relative baseline parameters of the two cameras must first be obtained, and then the corresponding three-dimensional coordinates can be recovered from the parallax of the pixel points according to the principle of triangulation. This requires the camera to be calibrated first.

Substep A1:

Calibrate the left camera device and the right camera device to obtain the internal parameters of the two, the distortion parameters and the external parameters of the corresponding three-dimensional points; based on the external parameter matrix of the left and right camera devices corresponding to each three-dimensional point, the binocular stereo vision system is obtained The rotation matrix R, the translation matrix T, and the relative baseline parameters of the left and right cameras.

The present invention is based on a camera calibration method of a plane checkerboard marker. Two cameras acquire the left and right checkerboard images of the two inputs. According to Zhang Zhengyou's calibration method, 10 to 20 groups of planar chessboard images containing different angles and directions are obtained. Next, the camera is calibrated. First, the checkerboard detection is performed on the input object, and then the three-dimensional points are located for the corner points in the checkerboard.

According to Zhang Zhengyou's calibration algorithm and Brown's algorithm, the internal parameters, distortion parameters of the left and right cameras, and the corresponding external parameters of each corner point on each checkerboard are respectively obtained to complete the calibration and correction of the left and right cameras.

In this way, the corresponding relationship between a point in space and this point in the image is obtained. For a corner point Q on the previously recorded checkerboard, and the corresponding external parameter matrix calculated by using, including the rotation matrix R _l and translation matrix T _l of the left camera, and the rotation matrix R _r and the right camera The translation matrix T _r .

Input the Q point into the camera coordinate system of the left and right cameras, corresponding to the coordinate points Q _l and Q _r of the left and right images, there is the following relationship:

Q _l ＝R _l Q+T _l

Q _r ＝R _r Q+T _r

Q _l ＝R ^T (Q _r -T)

Among them, the relative rotation matrix R and translation matrix T of the binocular camera derive the following simple relationship:

R＝R _r (R _l )

T＝ _Tr - _RTl

According to the joint views of the corner points of a given checkerboard and the external parameter matrix corresponding to each corner point, the rotation matrix R and the translation matrix T can be solved. Due to image noise and rounding errors, each pair of checkerboards will make the R and T results slightly different. Select the median value of the R and T parameters as the initial value of the real result, use the Levenberg-Marquardt iterative algorithm to find the minimum projection error of the checkerboard corner points on the two camera views, and return the results of R and T, so as to obtain binocular stereo vision The system's rotation matrix R, translation matrix T and relative baseline parameters of the left and right cameras.

Substep A2:

Based on the rotation matrix R of the binocular stereo vision system, the translation matrix T, the internal parameters of the left and right cameras and the distortion parameter matrix, the left correction matrix and the right correction matrix are obtained.

Since the cameras cannot have exact coplanar row alignment, the image planes of the two cameras are reprojected so that they fall on exactly the same plane, and the rows of the images are perfectly aligned to the front-parallel structure superior. Such a correction can simplify the search range. After correcting the left and right images, the points in the left image and the corresponding matching points in the right image are on the same scanning line. If epipolar line correction is not performed, for a certain point on the image, it is necessary to calculate the epipolar line of this point on the corresponding image, and search for the corresponding matching point on this epipolar line. This line is neither parallel nor vertical. This increases the computational complexity and also increases the search range. Here, the Bouguet algorithm is selected for stereo correction, using the internal parameter matrix of the left and right cameras obtained above, the distortion parameter matrix, and the rotation matrix and translation matrix between the binocular cameras, the left correction matrix and the right correction matrix are solved to make the two images The number of reprojections for each image is minimized, and the observation area is minimized at the same time.

Step B:

For the face area in the left image and the right image, locate and match the key points of the face.

Use the left and right cameras in the binocular stereo vision system to collect face images, and use the face detection algorithm and key point positioning algorithm to perform face detection and key point positioning on the two images.

Sub-step B1:

Use the Haar-Adaboost classifier to detect the face area on the left and right images, and combine the regression tree ensemble ERT algorithm to locate the key points of the face in the left and right images.

Sub-step B2:

Match the relevant face key points in the left image and the right image, and obtain the sparse topological information of the face prior - the face shape SL in the left image and the face shape SR in the right image, where the face shape SL in the left image contains the key point coordinates of the left face {(lx _i , ly _i ), i∈[1,n]}, the right face shape SR contains the key point coordinates of the right face {(rx _i , ry _i ), i∈[1,n]}, n represents the key total number of points.

Step C:

Use the matched face key points to initialize the dense disparity of the face to obtain the initialization disparity.

Substep C1:

Compute the disparity of matched face keypoints in the left and right images:

Since the left and right images are stereo rectified, the face keypoints of the left and right matches are located in the same row. The calculation of the disparity D(p _i ) of the face key point p _i is shown in the formula:

Among them, lx _i represents the column of the face key point in the left image, rx _i represents the column of the face key point that matches it in the right image, and the disparity D(p _i ) represents the corresponding face key point The absolute value of the column coordinate difference.

But for the structure of the face, such parallax is too sparse to describe the structure of the face well.

Substep C2:

Using the disparity of the matched face key points in the left image and the right image, calculate the disparity of other points in the left image and the right image except the matched face key points, realize the densification of the face disparity, and obtain the initialization disparity.

Sub-step C2a:

Delaunay triangulation is performed on the face by using the key points located on the face, and the face is divided into n triangles.

Sub-step C2b:

For each triangle, the disparity of the points inside the triangle is obtained by disparity of the three vertices of the triangle

It is assumed here that the disparity of points located in the same triangle is linearly related to the disparity of the three vertices of the triangle, so that the disparity of the points within the triangle can be obtained through the disparity of the three vertices of a triangle. If the three vertices of the triangle are p1, p2, p3. For any point p in the triangle, there exists a u and v, so that there is a relationship between point p and points p1, p2, p3 as in the formula

p=(1-u-v)·p1+u·p2+v·p3

The coordinates of point p are (p _x , p _y ), the coordinates of point p1 are (p1 _x , p1 _y ), the coordinates of point p2 are (p2 _x , p2 _y ), and the coordinates of point p3 are (p3 _x , p3 _y ), they satisfy the relationship of the formula, and the parameters u and v can be solved at this time.

p _x ＝(1-uv) p1 _x +u p2 _x +v p3 _x

p _y ＝(1-uv)·p1 _y +u·p2 _y +v·p3 _y

Thus, the parallax D(P) for point p can be obtained by interpolating from the formula:

D(p)＝(1-u-v)·D(p1)+u·D(pp2)+v·D(p3)

A dense initial disparity for the entire face is obtained.

Step D:

The disparity is initialized smoothly by a stereo matching algorithm.

The traditional local stereo algorithm uses integer disparity as the support window, and assumes that the pixels in the same support window have the same disparity, but this assumption is not true for curved or inclined surfaces, which will lead to certain deviations on the front-parallel surface . The method based on the parallax plane, such as the Patchmatch method, for the pixel (x ₀ , y ₀ ) on the picture, the corresponding parallax is d, and the corresponding parallax plane f, a _f , b _f , c _f represent the plane f, the unit vector Represents the normal vector of the plane. As shown in Figure 2, the stereo matching algorithm mainly includes four steps: cost calculation, cost aggregation, disparity calculation and post-processing.

Substep D1:

Use the cost calculation to obtain the similarity of the corresponding points of the left and right images, and obtain the matching cost of the corresponding points of the left and right images.

The calculation of matching cost directly affects the accuracy and efficiency of the algorithm. After selection, this patent uses the census algorithm to calculate the similarity of corresponding points. This algorithm has gray invariance, so that the correlation between the specific size of the pixel gray value and the encoding is not strong, and only the size relationship between pixels is concerned. , which is robust to noise. The criterion is to compare the elements in the neighborhood with the central element. The specific method is as follows:

With point G as the center point, compare the brightness value I(G) of this point with the brightness value I(G") of the pixels in its neighborhood N(G). N(G) means that with G as the center, the radius In the square neighborhood of d, if the brightness value of G' of the point in the neighborhood is less than G, the value at the position of point G' is recorded as 1, otherwise it is 0, which is ε(G, G·). The values of the points in the neighborhood are concatenated to complete the transformation, and the value is R _τ (G). After such a transformation is performed on each point, the similarity between the points in the left and right pictures is the hash of calculating the transformation value of the corresponding point Bright distance sheet. The smaller the distance, the higher the similarity. The matching cost ρ(J, K)=H(R(J), R(K)) between pixels J and K.

Substep D2:

Using the cost aggregation calculation, the aggregation cost of the points around the corresponding point is obtained by matching the cost of the corresponding points in the left and right graphs.

For a certain pixel point J under the parallax plane f and the aggregated parallax cost m of the point K under the range of W _K , as shown in the formula:

Where W _K represents a square window of pixel K, and d _K =a _f K _x +b _f K _y +c _f represents the parallax of pixel K. The plane parameters a _f , b _f , c _f can be converted into plane unit normal vectors by the formula

The weight function W(J, K) considers the edge problem in the square window, and uses the color similarity of two points to define the weight function. If the colors are similar, a high weight is assigned, otherwise, a low weight is assigned, which is shown in the formula :

where γ is a defined parameter and ||I _J -I _K || represents the L1 norm of the RGB color space of J and K.

Substep D3:

The plane with the minimum aggregate matching cost is selected as the optimal plane, and the pixel point parallax is calculated inversely.

For a certain pixel point J, the plane f _J with the smallest aggregation matching cost m is selected as the optimal plane of the current point, as shown in the formula:

F represents all parallax planes, and the corresponding solutions are infinitely many. Find better plane parameters in the following way. First initialize the plane parameters and parallax parameters, use the obtained dense face parallax to initialize the parameter d, constrain the face structure in the parallax, and initialize the rest of the parameters randomly. The parameter plane is optimized by iteratively performing three steps of spatial propagation, viewpoint propagation and plane refinement.

Finally, the optimal parameter plane is found, and the parallax of the pixels is reversely calculated.

Step E:

Combining the smoothed initialization disparity for 3D face reconstruction.

Combining the obtained camera internal parameters, the relative baseline parameters of the camera, and the smoothed initial disparity corresponding to the pixels. The corrected binocular camera is based on the principle of triangulation, as shown in Figure 4, from the following formula, any two-dimensional point S, whose coordinates are (x, y), and the associated parallax d, project this point In 3D, the 3D coordinates are (X/W, Y/W, Z/W).

where O is the projection matrix,

In O, c _x represents the x coordinate of the left image of the principal point, c _y represents the y coordinate of the left image of the principal point, T _x represents the baseline length between the two cameras, L represents the focal length of the camera, and c _x ′ represents the right side of the principal point The x-coordinate of the image, the principal point where the chief ray intersects the image plane.

In this way, through the above formula, the three-dimensional coordinates corresponding to the pixel points can be obtained, so that the three-dimensional structure of the face can be obtained.

Aiming at the curved surface structure of the human face, the present invention adopts the stereo matching algorithm of the parallax plane to restore a smoother three-dimensional structure of the human face; adopts the prior structure of the human face, and initializes the parallax plane on this basis, thereby improving the accuracy of the three-dimensional reconstruction result of the human face Spend.

So far, the introduction of the embodiment of the present invention is completed.

So far, the present embodiment has been described in detail with reference to the drawings. Based on the above description, those skilled in the art should have a clear understanding of the method for 3D face reconstruction based on binocular stereo vision of the present invention.

It should be noted that, in the accompanying drawings or in the text of the specification, implementations that are not shown or described are forms known to those of ordinary skill in the art, and are not described in detail. In addition, the above definitions of each method are not limited to the various specific methods mentioned in the embodiments, and those of ordinary skill in the art can easily modify or replace them,

It should also be noted that the text may provide examples of parameters that include specific values, but these parameters need not be exactly equal to the corresponding values, but may approximate the corresponding values within acceptable error tolerances or design constraints. The directional terms mentioned in the embodiments, such as "up", "down", "front", "rear", "left", "right", etc., are only referring to the directions of the drawings, and are not intended to limit the present invention protected range. In addition, unless specifically described or steps that must occur sequentially, the order of the above steps is not limited to that listed above and may be changed or rearranged according to the desired design. Moreover, the above-mentioned embodiments can be mixed and matched with each other or with other embodiments based on design and reliability considerations, that is, technical features in different embodiments can be freely combined to form more embodiments.

The specific embodiments described above have further described the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above descriptions are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims (14)

1. a kind of face three-dimensional rebuilding method based on binocular stereo vision, it is characterised in that including：

Step A, builds Binocular Stereo Vision System, wherein, the Binocular Stereo Vision System includes that left camera head and the right side are taken the photograph As device；

Step B, facial image is gathered using the Binocular Stereo Vision System, and left image is obtained by left camera head, is taken the photograph by the right side As device obtains right image；Three-dimensional correction is carried out to left image and right image；Human face region in detection left image and right image；

Step C, for the human face region in left image and right image, carries out positioning and the matching of face key point；

Step D, carries out the dense parallax of face and initializes using the face key point of matching, obtains initializing parallax；

Step E, by Stereo Matching Algorithm smoothing initialization parallax；And

Step F, face three-dimensional reconstruction is carried out with reference to the initialization parallax after smooth.

2. face three-dimensional rebuilding method according to claim 1, it is characterised in that the left camera head and right shooting are filled It is set to the camera or camera of same model：

The step A includes：

Sub-step A1, demarcates to left camera head and right camera head, obtains both intrinsic parameters, distortion parameter and correspondence The outer parameter of three-dimensional point；Based on the outer parameter of left and right camera head corresponding three-dimensional points, the rotation of Binocular Stereo Vision System is obtained Matrix R, translation matrix T；

Sub-step A2, the intrinsic parameter and distortion parameter of intrinsic parameter and distortion parameter, right camera head based on left camera head, with And the spin matrix R and translation matrix T of the Binocular Stereo Vision System asked for, obtain left correction matrix and right correction matrix；

Wherein, left correction matrix is used to carry out left image three-dimensional correction, and right correction matrix is used to carry out three-dimensional school to right image Just, by the match point in the point in the left image after the treatment of left correction matrix with the right image after right correction matrix is processed In same scan line；

Carrying out three-dimensional correction to left image and right image in the step B includes：Left image is stood using left correction matrix Sports school just, three-dimensional correction is carried out using right correction matrix to right image.

3. face three-dimensional rebuilding method according to claim 2, it is characterised in that the sub-step A1 includes：

Sub A1a step by step, obtains 10~20 groups of plane checkerboard images comprising different angle and directions；

Sub A1b step by step, the plane checkerboard image to obtaining carries out chessboard monitoring, orients the chessboard corresponding to the three-dimensional point Angle point in lattice；According to Zhang Zhengyou scaling methods and Brown algorithm, the intrinsic parameter of the left camera head of acquisition and right camera head, Distortion parameter and the corresponding outer parameter of chessboard angle point；

Wherein, the corresponding outer parameter of the chessboard angle point includes：The spin matrix R of left camera head_lWith the translation of left camera head Matrix T_l；The spin matrix R of right camera head_rWith the translation matrix T of right camera head_r；

Sub A1c step by step, angle point Q is input to the seat of the camera head coordinate system of left and right camera head, correspondence left figure and right figure Punctuate Q_lAnd Q_r, exist such as following formula relation：

Q_l=R_lQ+T_l

Q_r=R_rQ+T_r

Q_l=R^T(Q_r-T)

Wherein, Q is three-dimensional coordinates of the angle point Q in world coordinate system, and left figure is the image that left camera head is obtained, and right figure is taken the photograph for the right side As the image that device is obtained, following relation is released：

R=R_r(R_l)

T=T_r-RT_l

According to given tessellated angle point to individual joint view, and the outer parameter matrix corresponding to each angle point, solve Spin matrix R and translation matrix T；Due to picture noise and rounding error, every a pair of chessboards can all cause that the result of R and T occurs Tiny difference, from R and T parameters intermediate value as legitimate reading initial value, with Levenberg-Marquardt iteration calculate Method searches minimum projection error of the chessboard angle point on two camera head views, returns to the knot of spin matrix R and translation matrix T Really.

4. face three-dimensional rebuilding method according to claim 2, it is characterised in that in the sub-step A2, utilizes Bouguet algorithms, the intrinsic parameter and distortion parameter of left camera head, the intrinsic parameter and distortion parameter of right camera head, Yi Jiqiu The spin matrix R and translation matrix T of the Binocular Stereo Vision System for taking, obtain left correction matrix and right correction matrix.

5. face three-dimensional rebuilding method according to claim 1, it is characterised in that Haar- is used in the step B Human face region in Adaboost detection of classifier left image and right image.

6. face three-dimensional rebuilding method according to claim 1, it is characterised in that the step C includes：

Sub-step C1, the face key point of human face region in positioning left image and right image；

Sub-step C2, the matching left image face key point related in right image, the sparse topology information of acquisition face priori- Left figure face shape SL and the right face shape SR, wherein left figure face shape SL include left face key point coordinates { (lx_i, ly_i), I ∈ [1, n] }, the right face shape SR includes right face key point coordinates { (rx_i, ry_i), i ∈ [1, n] }, n represents the total of key point Number.

7. face three-dimensional rebuilding method according to claim 6, it is characterised in that in the sub-step C1, with reference to recurrence The face key point of tree set ERT algorithms positioning left image and right image.

8. face three-dimensional rebuilding method according to claim 6, it is characterised in that the step D includes：

Sub-step D1, calculates the parallax of the face key point matched in left image and right image；

Sub-step D2, using the parallax of the face key point matched in left image and right image, in calculating left image and right image The parallax of other points in addition to the face key point of matching, realizes denseization of face parallax, obtains initializing parallax.

9. face three-dimensional rebuilding method according to claim 8, it is characterised in that in the sub-step D1, according to following formula Calculate the parallax of the face key point matched in left image and right image：

$D\left(p_i\right)=\sqrt{{({\mathrm{lx}}_i-{\mathrm{rx}}_i)}^2}=abs({\mathrm{lx}}_i-{\mathrm{rx}}_i)$

Wherein, lx_iRepresent row of i-th face key point at the place of left image, rx_iRepresent matched in right image Row where face key point, parallax D (p_i) represent corresponding face key point row coordinate difference absolute value, i=1,2, 3 ..., n, n are the number of the key point of matching.

10. face three-dimensional rebuilding method according to claim 8, it is characterised in that the sub-step D2 includes：

Sub D2a step by step, the key point gone out using face locating is carried out Delaunay Triangulation, face is divided into face N triangle；

Sub D2b step by step, for each triangle, by regarding for being put in three summit parallaxes acquisition triangles of triangle Difference, realizes denseization of face parallax, obtains initializing parallax.

11. face three-dimensional rebuilding methods according to claim 10, it is characterised in that the son in D2b, passes through step by step The parallax that three summit parallaxes of triangle obtain point in triangle includes：

Three summits of triangle are p1, p2, p3, for the point p in triangle, all in the presence of a u and v so that p points and point There is relation such as formula in p1, p2, p3：

p_x=(1-u-v) p1_x+u·p2_x+v·p3_x

p_y=(l-u-v) pl_y+u·p2_y+v·p3_x

By by the coordinate (p of p points_x, p_y), the coordinate (p1 of p1 points_x, p1_y), the coordinate (p2 of p2 points_x, p2_y), the coordinate of p3 points (p3_x, p3_y) substitute into formula solve u and v parameters；

Parallax D (p) that interpolation arithmetic obtains point p is carried out by formula：

D (p)=(1-u-V) D (p1)+uD (p2)+VD (p3).

12. face three-dimensional rebuilding methods according to claim 1, it is characterised in that the step E includes：

Sub-step E1, the similarity of two figure corresponding points of left and right is calculated using cost, obtains the matching of two figure corresponding points of left and right Cost；

Sub-step E2, is polymerized using cost and calculated, and by two figure corresponding point matching costs of left and right, obtains point around the corresponding points Polymerization cost；

Sub-step E3, from minimum polymerization Matching power flow plane as optimal planar, reverse goes out pixel parallax.

13. face three-dimensional rebuilding methods according to claim 12, it is characterised in that：

The computational methods of the sub-step E1 Matching power flows are：

$\mathrm{\&epsiv;}(G,G^{\&prime;})=\left\{\begin{array}{cc}1& ifI\left(G^{\&prime;}\right)<I\left(G\right)\\ 0& others\end{array}\right.$

Point centered on G points, the brightness value I (G) of G points, N (G) is represented centered on G, and radius is neighborhood N in the Square Neighborhood of d (G) brightness value I (G ') of the pixel in is the brightness value of the pixel in neighborhood N (G), if the brightness of the G ' of point in neighborhood Value is less than G, then the value on the position of point G ' is designated as 1, otherwise is then 0, as ε (G, G ')；

The value of the point in neighborhood is together in series, conversion is completed, it is R to be worth_τ(G), its formula is as follows：

$R_{\mathrm{\&tau;}}\left(G\right)=\&CircleTimes;\mathrm{\&epsiv;}(G,G^{\&prime;}),G^{\&prime;}\&Element;N\left(G\right)$

After conversion as being carried out to each point, the width figure point of left and right two is then the transformed value of calculating corresponding points with the similitude of point Hammerstein model H, apart from smaller, represent that similarity is higher, so as to obtain pixel J, the Matching power flow between K is：

ρ (J, K)=H (R (J), R (K))；

In the sub-step E2, the cost aggregation calculation formulae is：

$m(J,f)=\underset{K\&Element;W_J}{\&Sigma;}W(J,K)\&CenterDot;\mathrm{\&rho;}(K,K-d_K)$

Wherein K is a pixel, and J is W_KSubsequent point, W_KRepresent the square window of pixel K, d_K=a_fK_x+b_fK_y+c_fRepresent pixel K The parallax of point, W (J, K) is weighting function；

The weighting function W (J, K) considers the edge problem in square window, and power is defined using 2 points of color similarity Value function, assigns weight high if color is close, conversely, then assigning low weight, represents as shown by the equation：

$W(J,K)=e^{\frac{-\left|\right|I_J-I_K\left|\right|}{\mathrm{\&gamma;}}}$

Wherein, γ is the parameter of definition, | | I_J-I_K| | represent the L1 norms of the RGB color of J and K.

In the sub-step 3, the disparity computation is calculated including optimal planar, and its computing formula is：

$f_J=\underset{f\&Element;F}{\arg \min m(J,f)}$

Wherein, F represents all of disparity plane, f_JAs the optimal planar of J points, pixel is gone out by the inverse problem of parameter of optimal planar Parallax.

14. face three-dimensional rebuilding methods according to claim 13, it is characterised in that the step F includes：

Choose random two-dimensional point S, its coordinate is (x, y), associated parallax d, and by this spot projection to three-dimensional, three-dimensional coordinate is (X/W, Y/W, Z/W), obtains matrix：

$O\left[\begin{array}{c}x\\ y\\ d\\ 1\end{array}\right]=\left[\begin{array}{c}X\\ Y\\ Z\\ W\end{array}\right]$

Wherein, O is projection matrix,

$O=\left[\begin{array}{cccc}1& 0& 0& -c_x\\ 0& 1& 0& -c_y\\ 0& 0& 1& f\\ 0& 0& -1/Y_x& (c_x-c_x^{\&prime;})T_x\end{array}\right]$

C in O_xRepresent the x coordinate of principal point left image, c_yRepresent the y-coordinate of principal point left image, T_xBetween two camera heads of expression Baseline length, f represents the focal length of camera head, c_xThe x coordinate of ' expression principal point right image；

Three-dimensional coordinate corresponding to pixel is tried to achieve by above-mentioned formula, so as to obtain face three-dimensional structure.