CN106951840A - A Face Feature Point Detection Method - Google Patents

Abstract

The present invention discloses a kind of facial feature points detection method, using attitude detection task as constraint, method of novel three-way road GEH (the Gray Edge Hog) mode images merged using multiclass feature figure as the facial feature points detection of input.Detection in view of face 3 d pose information to face global characteristic point, is especially detected in the case where attitude deflection is larger to features of human face images, with considerable influence；The Hog characteristic informations of reflection facial image local feature presentation and shape are added simultaneously and can effectively reduce the complexity of contour feature point detection for the edge image information of the Sobel operator extractions of rim detection, the present invention is by extracting image intensity value, marginal information and Hog features generate new GEH triple channels image and are used as input, the nonproductive task estimated simultaneously using 3 d pose is used as constraint information, progress facial feature points detection.

Description

A Face Feature Point Detection Method

technical field

The invention belongs to the field of computer vision, and in particular relates to a face feature point detection method using face three-dimensional pose information as an auxiliary constraint in a new image mode, which has important applications in face recognition, face pose and expression analysis, and face synthesis .

Background technique

In recent years, with the development of deep learning, Convolutional Neural Networks (CNN) have achieved good results in facial feature point detection. CNN takes the original image of the face as input, and the features obtained by using the local receptive field strategy have better expressive ability; the weight sharing structure reduces the number of weights and thus reduces the complexity of the network model; at the same time, using the local correlation of the image Principle The downsampling of feature maps effectively reduces the amount of data processing while retaining useful structural information, so CNN is widely used in feature extraction of face images.

Yi Sun et al. proposed a three-level deep convolutional neural network cascade face feature point detection model (Deep Convolutional Network Cascade, DCNN) in 2013. The first stage of the network takes three different areas of the face image (the entire face area, the eyes and nose area, and the nose and lips area) as input, and trains three convolutional neural networks to predict the position of feature points, and then fuses The predicted values of the three networks are used to obtain more stable primary feature point detection results. The second and third stages extract features near each feature point, and train a convolutional neural network for each feature point to correct the positioning results, and realize the left eye center, right eye center, nose tip, left mouth corner, and right mouth corner. Detection of feature points. However, the result of this method is only to roughly calibrate the positions of eyes, nose, and mouth, and it cannot express the facial attributes well, and the network model is also too complicated.

In the same year, Erijin Zhou et al. proposed a four-level convolutional neural network cascade model for detecting 68 facial feature points. The model takes into account the complexity of the location of the outer contour of the face and the feature points of the inner facial features, and detects them separately. Multi-feature point detection can express the attributes of human faces such as gestures and expressions in more detail. However, the operation process of this method is relatively complicated, involving 10 different networks to be trained separately.

Zhanpeng Zhang et al. proposed a deep convolutional neural network (Tasks-Constrained Deep Convolutional Network, TCDCN) constrained by auxiliary tasks in 2015. While performing 5 facial feature point detection, the model uses 18 auxiliary tasks related to facial attributes as constraints, which enhances the network's ability to extract features and helps to improve the accuracy of feature point detection. However, this method only considers the face posture deflection in the horizontal dimension, and in most cases, the posture deflection in other dimensions also has a certain impact on the accuracy of feature point detection.

Contents of the invention

The invention provides a new three-channel GEH (Gray-Edge-Hog) mode image fused with multi-category feature maps as input, and a method for detecting human face feature points with three-dimensional posture auxiliary tasks as constraint information. The present invention considers that when the posture of the face changes, the change of the image contour structure can be clearly seen, so the three-dimensional posture information of the human face has a considerable influence on the calibration of the global feature points of the human face; The difficulty of detecting contour feature points and internal organ feature points is different. Using the edge information extracted from a face as an image pattern variable can reduce the difficulty of detecting external contour points; using the Hog feature map extracted from a face image as an image pattern Variables, while clearly reflecting the image contour structure, can more effectively highlight the regional characteristics of the various organs of the face. Therefore, a new GEH mode image is proposed that uses the image gray value, edge information and Hog feature fusion as input. The three-dimensional posture information is auxiliary constraints, and the convolutional neural network structure model jointly trained by facial feature points and facial postures can realize the precise positioning of 68 feature points of the face.

To achieve the above object, the present invention adopts the following technical solutions:

A method for face feature point detection comprises the following steps:

Step 1, the original image is subjected to face detection and positioning, clipping and three-channel multi-feature map fusion to obtain a three-channel GEH pattern picture Picture _GEH ;

Step 2, use the three-channel GEH pattern map after the fusion of three kinds of feature maps as the input of the convolutional neural network, and perform network face feature extraction, the face features include: face feature points and three-dimensional posture, the feature points Both detection and attitude detection use the least squares function corresponding to the linear regression problem to design a dual-task loss function;

Step 3, using the gradient backpropagation algorithm to carry out network training on the dual-task loss function, and finally learn the face feature point detection weight and attitude detection weight, in the test process, through the same face feature extraction network, to achieve human Face feature point detection and face 3D pose detection.

As a preference, the network feature extraction is completed alternately by 3 convolutional layers and 3 pooling layers, and 2 fully connected layers;

First, the three-channel GEH pattern Picture _GEH is used as the input of the first layer convolution operation The calculation formula of the output feature map y _j is shown in the following formula:

Among them, f represents the convolution operation, l represents the number of current network layers, i represents the number of input feature maps, j represents the number of output feature maps, w _ij is the convolution kernel parameter to be obtained, b _j is the bias parameter, w _ij and _bj are obtained by random normal initialization at the beginning of the experiment;

Then, according to the results obtained in the convolution stage, the features are sent to the function corresponding to the linear regression problem, and the designed dual-task loss function expression is shown in the following formula:

Among them, N represents the number of training images, Indicates the label value of the i-th image feature point detection task, Represents the label value corresponding to the three-dimensional pose detection of the face, x _i represents the i-th image feature extracted by the convolutional neural network, W ^f represents the weight of the feature point detection task, W ^yaw , W ^pitch , W ^roll represent the three poses of the face The weights λ _Yaw , λ _Pitch , and λ _Roll corresponding to the detection task represent the loss weight of the loss function;

Network training is carried out through the backpropagation algorithm, and the face feature point detection weight W ^f and the attitude detection weight W ^yaw , W ^pitch , W ^roll are obtained; the test process, through the same face feature extraction network, finally gets the face feature point detection The result (W ^f ) ^T _xi , and the 3D attitude detection results (W ^Yaw ) ^T _xi , (W ^Pitch ) ^T _xi , (W ^Roll ) ^T _xi .

As preferably, in step 1, carry out gray-scale processing through the face sub-image after face detection positioning and clipping, obtain the gray-scale feature map G; Then extract the Hog feature to the face sub-image, obtain the Hog feature map H; Finally extract the edge feature , to obtain the edge feature map E; using the RGB (Red‐Green‐Blue) color space as the base, the above feature map G (Gray), feature map E (Edge), feature map H (Hog) feature map variables are respectively mapped to RGB In the Cartesian coordinate system color space, a new type of GEH mode image Picture _GEH is generated, and the generation formula is as follows:

in, The grayscale value representing the grayscale feature map Gray is mapped to the R (Red) color space in RGB, Represents the Hog feature map eigenvalues mapped to the B(Blue) color space, The eigenvalues representing the Edge feature map are mapped to the G(Green) color space.

Description of drawings

Figure 1: Schematic diagram of the process of generating the three-channel feature map of the present invention;

Figure 2a: Original face image;

Figure 2b: Face image after multi-feature map fusion;

Figure 3: GEH-Dual-Task Convolutional Neural Network Architecture Model.

detailed description

The present invention provides a face feature point detection method, which takes the pose detection task as a constraint, and uses a new three-channel GEH (Gray-Edge-Hog) pattern image fused with multi-class feature maps as an input face feature point detection method . Considering that the three-dimensional pose information of the face has a considerable influence on the detection of the global feature points of the face, especially in the case of a large pose deflection, it has a considerable influence on the detection of the feature points of the face image; The Hog feature information and the edge image information extracted by the Sobel operator used for edge detection can effectively reduce the complexity of contour feature point detection. The present invention generates a new GEH three-channel image by extracting image gray values, edge information and Hog features as Input, and at the same time use the auxiliary task of 3D pose estimation as constraint information to detect facial feature points.

The basic data used in the present invention comes from the 300-W facial feature point detection competition platform, which includes four data sets of LFPW, AFW, HELEN, and IBUG. The image labels in each dataset are 68 points, including external contour points and internal facial features (eyebrows, eyes, nose, mouth). The corresponding 3D attitude labels are calculated and generated by the Interface software developed by the Human Perception Laboratory based on the feature point labels provided by the 300-W image, which respectively represent the 3D information of Yaw, Pitch and Roll.

The face feature point detection CNN structure proposed by the present invention includes two tasks, that is, a 68-point face feature point detection task and a three-dimensional pose information detection task. Utilizing the relative influence of 3D pose information on global face feature point detection, the CNN face feature point detection network is used to extract joint features that can not only represent the position of face feature points but also reflect the orientation of face poses, and finally realize the face feature points detection.

1. Image preprocessing

Appropriate image preprocessing methods can eliminate environmental influences such as weather and light in the original image, make the edge and color features of the image more prominent, and facilitate feature extraction of convolutional neural networks. The present invention first performs face detection, positioning, clipping, and normalization processing on the original image, and then performs grayscale processing on the clipped image, Hog feature extraction and visualization, and Sobel operator edge feature extraction, and the generated multi-category features Graph fusion forms a new GEH image mode, and the three-channel image in this mode is used as input.

1.1 Face detection positioning and clipping

The invention aims to remove the interference of image background, hair, clothing and other information on the feature point detection task. According to the results of face detection and positioning, the detected image's up, down, left, and right distances are enlarged by a certain proportion, and then the enlarged face image is cropped and scaled normalized.

1.2 Three-channel multi-feature map fusion

First, the face sub-image obtained in 1.1 above is processed in grayscale to obtain the gray-scale feature map G; then the Hog feature is extracted from the face sub-image, because the dimension of the feature map after extracting the Hog feature is different from the dimension of the face sub-image, Carl Vondrick was used here The Inverting Visual Features method proposed in 2012 performs feature visualization processing and finally obtains the Hog feature map H of the same size as the face sub-image; finally extracts the edge features, and uses the 5th-order Sobel operator to normalize the scale-normalized RGB face sub-map image Edge extraction is performed to generate an edge feature map E.

Since the above three types of feature maps reflect the information of different attributes of the face image and are independent of each other, the combined effect of the three independent variables can form a new image pattern. Therefore, the present invention uses the RGB (Red-Green-Blue) color space as a base, and maps the above-mentioned G (Gray), E (Edge), and H (Hog) feature map variables to the RGB Cartesian coordinate system color space respectively to generate a new The GEH mode image Picture _GEH . The generation formula is as follows:

in, The grayscale value representing the grayscale feature map Gray is mapped to the R (Red) color space in RGB, Represents the Hog feature map eigenvalues mapped to the B(Blue) color space, The eigenvalues representing the Edge feature map are mapped to the G(Green) color space.

The GEH three-channel image generation process is shown in Figure 1.

The GEH three-channel image effects are shown in Figures 2a and 2b.

2. Dual-task network model of facial feature point detection and 3D pose detection

The CNN network structure of the present invention jointly considers the task of face image feature point detection and the task of three-dimensional pose detection, wherein the task of face feature point detection is the main task, and the task of three-dimensional pose detection is an auxiliary task. The network structure is shown in Figure 3.

2.1 Face Feature Extraction

This network uses the three-channel GEH pattern Picture _GEH after the fusion of three feature maps as the input of the convolutional neural network to extract face features. Both the feature point detection and pose detection tasks use the least squares function corresponding to the linear regression problem as the loss function, and use the gradient backpropagation algorithm to train network parameters, and finally realize face feature point detection and face 3D pose detection.

In the present invention, network feature extraction is accomplished alternately by three convolutional layers, three pooling layers, and two fully connected layers. The convolutional layer extracts visual features through the convolution operation of the local receptive field. In order to more completely preserve the image features after the original image is converted into the GEH image mode, the present invention takes the size of the face submap close to the original image as the input size, and the input of the first layer of convolution operation (Picture _GEH ) is an image with a size of 224×224, which is 2.3 times the input size of DCNN and TCDCN, which ensures the integrity of the image and the validity of features; the convolution kernel sizes are 7×7, 4×4, 3 ×3. The calculation formula of the output feature map y _j is shown in formula (2):

Among them, f represents the convolution operation, l represents the number of current network layers, i represents the number of input feature maps, j represents the number of output feature maps, w _ij is the convolution kernel parameter to be obtained, b _j is the bias parameter, w _ij and _bj are obtained by random normal initialization at the beginning of the experiment.

The pooling layer operation adopts the maximum pooling method. According to the characteristics of the above-mentioned GEH mode image input size, the pooling range of the first and third layers of the present invention is set to 3×3, and the step size is 3. While ensuring the completeness of the extracted features, It reduces the feature dimension more effectively and reduces the complexity of network training; the pooling range of the second layer is 2×2, and the step size is 2.

2.2 Design of dual-task objective function

Next, according to the results obtained in the convolution stage, the features are sent to the function corresponding to the linear regression problem. In the present invention, the problem of face feature point detection and the problem of three-dimensional attitude detection of people's faces are all linear regression problems, and the least square function is used as the loss function, and the expression is as shown in formula (3):

f _loss ＝||lW ^T x _i || (3)

Among them, l represents the label of the regression problem, _xi represents the feature extracted by the convolutional neural network, and W represents the weight coefficient corresponding to the linear regression problem.

The present invention takes the face feature point detection as the main task. 3D pose detection is an auxiliary task to assist the face feature point detection task to extract features that can better reflect the 3D pose, and to more accurately locate the face image with a larger pose; the 3D pose coordinates are represented by Pitch, Yaw and Roll respectively , taking the right-handed Cartesian coordinate system in the three-dimensional rectangular coordinate system as an example, Pitch means rotation around the X axis, called pitch angle, Yaw means rotation around the Y axis, called yaw angle, Roll means rotation around the Z axis, called roll angle. In the 3D attitude detection task, according to the statistics of the experimental data, the change range of the Yaw attitude in the 3D attitude is relatively large, which is 5 to 6 times the change of the Pitch attitude and the Roll attitude. Therefore, the influence of Yaw is greater than that of Pitch and Roll. Therefore, the weight of the loss function is set to adjust the influence of the three-dimensional pose on the main task of face detection. The dual-task loss function expression designed by the present invention is shown in formula (4):

Among them, N represents the number of training images, Represents the label value of the i-th image feature point detection task (dimension is 136), Respectively represent the label value corresponding to the three-dimensional pose detection of the face (the dimension is 1), x _i represents the i-th image feature extracted by the convolutional neural network, W ^f represents the weight of the feature point detection task, W ^yaw , W ^pitch , W ^roll Respectively represent the weights corresponding to the three pose detection tasks of the face. λ _Yaw , λ _Pitch , and λ _Roll represent the loss weight of the loss function, which are 0.3, 0.1, and 0.1 respectively.

2.3 E-learning

The present invention adopts backpropagation algorithm to carry out network training. Finally, the face feature point detection weight W ^f and attitude detection weights W ^yaw , W ^pitch , W ^roll are learned. During the test process, through the same face feature extraction network, the face feature point detection result (W ^f ) ^T _xi , and the 3D pose detection result (W ^Yaw ) ^T _xi ,(W ^Pitch ) ^T _xi ,( W ^Roll ) ^T x _i .

The above method has been verified by experiment, and obvious effect has been obtained. The evaluation index uses the average estimation error index proposed by Yi Sun et al. on CVPR in 2013 for face feature point detection to measure the performance of the designed method. This index shows the accuracy and reliability of a feature point positioning algorithm.

The formula for calculating the average estimation error is as follows:

Among them, (x, y) and (x', y') represent the true coordinates and estimated coordinates of feature points respectively, and l represents the estimation error normalization factor. An estimate is considered invalid if the error in the estimate exceeds 10%.

The experiment uses the LFPW database in the 300-W challenge platform, which is a face database including multi-pose and multi-view. Because it includes pictures affected by various poses, expressions, lighting, and other factors, most face feature point detection methods are verified on this dataset. The dataset contains 811 training images and 224 testing images.

The main content of the first group of experiments: dual-task convolutional neural network (3feature-D-CNN) with multi-feature fusion image as input, dual-task convolutional neural network (D-CNN) with original image as input, and original image as input Performance comparison of traditional convolutional neural network (CNN) with image as input for facial feature point detection. The structure settings of the convolutional layer, pooling layer and fully connected layer in the above three networks are the same. The difference lies in the input image type of the network structure, the output loss function and the output dimension are different. The experimental comparison results are as follows:

Table 1: Comparison of three convolutional neural network models

The data in each column of Table 1 represents the test results of different network models on LFPW data. The lower the value, the better the effect of face landmark detection. It can be seen that the 3feature-D-CNN network proposed by the present invention reduces the average estimation error by 14.02% and 11.6% respectively compared with the original CNN network and the D-CNN network. This shows that the face feature point detection method with the pose detection task as a constraint and the new three-channel GEH (Gray-Edge-Hog) pattern image fused with multi-class feature maps as input is effective.

The main content of the second group of experiments: the comparison of the above three networks for the detection of external contour points. The experimental results are shown in Table 2:

Table 2: Comparison of the detection results of the outer contour of the face detected by the three network models

The data in each column of Table 2 represents the detection results of contour points on LFPW data by different network models. The lower the value, the better the effect of face landmark detection. It can be seen that compared with the original CNN network and the D-CNN network, the average error in external contour detection of the 3feature-D-CNN network proposed by the present invention is reduced by 21.52% and 4.95%, respectively. It is verified that the present invention improves the detection effect of the outer contour points of the human face to a certain extent.

Claims (3)

1. A face feature point detection method is characterized by comprising the following steps:

step 1, carrying out face detection positioning and cutting and three-channel multi-feature image fusion on an original face image to obtain a three-channel GEH mode image Picture_GEH；

Step 2, taking a three-channel GEH mode image obtained by fusing the three feature images as the input of a convolutional neural network, and extracting the facial features of the network, wherein the facial features comprise: the method comprises the following steps that characteristic points and three-dimensional postures of a human face are detected, and a double-task loss function is designed according to a least square function corresponding to a linear regression problem;

and 3, performing network training on the double-task loss function by adopting a gradient back propagation algorithm, finally learning the detection weight of the human face characteristic points and the detection weight of the posture, and extracting a network through the same human face characteristics in the test process so as to realize the detection of the human face characteristic points and the detection of the three-dimensional posture of the human face.

2. The method of claim 1, wherein the network feature extraction is performed by alternating 3 convolutional layers and 3 pooling layers, with 2 fully-connected layers;

firstly, a three-channel GEH mode diagram Picture_GEHAs input to a first layer convolution operationOutput feature map y_jThe calculation formula of (a) is shown as the following formula:

$x_j^l=f(b_j+{\mathrm{\Σ}}_i{w}_{ij}*x_i^{l-1})$

where f denotes convolution operation, l denotes the current number of network layers, i denotes the number of input profiles, j denotes the number of output profiles, w_ijFor the convolution kernel parameter to be solved, b_jIs a bias parameter, w_ijAnd b_jAcquiring by adopting a random normal initialization mode at the beginning of an experiment;

then, according to the result obtained in the convolution stage, the characteristics are sent to the function corresponding to the linear regression problem, and the designed dual-task loss function expression is shown as the following formula:

$\begin{array}{c}\underset{W^f,W^{Yaw},W^{Pitch},W^{Roll}}{\arg \min }\left\{\underset{i=1}{\overset{N}{\mathrm{\Σ}}}\right||l_i^f-{\left(W^f\right)}^T{x}_i|{|}^2+{\mathrm{\λ}}_{Yaw}\underset{i=1}{\overset{N}{\mathrm{\Σ}}}\left|\right|l_i^{Yaw}-{\left(W^{Yaw}\right)}^T{x}_i|{|}^2\\ +{\mathrm{\λ}}_{Pitch}\underset{i=1}{\overset{N}{\mathrm{\Σ}}}\left|\right|l_i^{Pitch}-{\left(W^{Pitch}\right)}^T{x}_i|{|}^2+{\mathrm{\λ}}_{Roll}\underset{i=1}{\overset{N}{\mathrm{\Σ}}}||l_i^{Roll}-{\left(W^{Roll}\right)}^T{x}_i|{|}^2\}\end{array}$

wherein N represents the number of training images,a tag value indicating the ith image feature point detection task,respectively representing label values, x corresponding to the detection of the three-dimensional posture of the human face_iFeatures of the ith image, W, representing convolutional neural network extraction^fRepresenting the weight of the feature point detection task, W^yaw,W^pitch,W^rollRespectively representing weight lambda corresponding to three-pose detection task of human face_Yaw,λ_Pitch,λ_RollRepresenting a loss function loss weight;

performing network training through a back propagation algorithm to obtain the face characteristic point detection weight W^fAnd attitude detection weight W^yaw,W^pitch,W^roll(ii) a In the testing process, the same face feature extraction network is used to finally obtain the face feature point detection result (W)^f)^Tx_iAnd three-dimensional attitude detection result (W)^Yaw)^Tx_i,(W^Pitch)^Tx_i,(W^Roll)^Tx_i。

3. The method for detecting human face feature points as claimed in claim 1, wherein in step 1, the human face sub-image after human face detection positioning and clipping is processed with gray scale to obtain a gray scale feature image G; extracting the Hog features from the face subgraph to obtain a Hog feature graph H; finally, extracting edge features to obtain an edge feature graph E; respectively mapping the feature map variables of the feature map G (Gray), the feature map E (edge) and the feature map H (hog) onto the color space of an RGB rectangular coordinate system by using an RGB (Red-Green-Blue) color space as a substrate to generate a novel GEH mode image Picture_GEHThe generation formula is as follows:

wherein,the Gray values representing the Gray feature map Gray are mapped to the r (red) color space in RGB,the representative Hog feature map feature values are mapped to the b (blue) color space,the feature values representing the Edge feature map are mapped to the g (green) color space.