CN106355138A - Face recognition method based on deep learning and key features extraction - Google Patents

Abstract

The invention discloses a novel method of detecting and recognizing faces in videos based on deep learning, which comprises the following steps: detecting the face images in the video and accurately extracting the facial image from the background to eliminate the interference of non-facial areas to the detection result; selecting from the face such an area less affected by visual interference conditions and extracting the local feature of a key point within such area; extracting the top facial feature from the selected area via the convolutional neural network to serve along with the local feature as the features that express the face, and reducing the dimensionality of and normalizing such features; measuring the similarities between the features of two facial images in facial pairs, and assessing one by one to select the pair of images that are most identical, thus the recognition result is obtained.The invention has good recognition effect under unconfined conditions and is suitable for application in settings with complicated interference such as postures, expressions, light and shelter.

Description

Face recognition method based on deep learning and key point feature extraction

technical field

The invention belongs to the field of computer vision research and relates to video image processing and machine learning methods, in particular to a face recognition method based on deep learning and key point feature extraction.

Background technique

With the widespread application of surveillance cameras, the market demand for face recognition systems is gradually expanding. However, in these applications, the monitored population is mostly in an unconstrained state, and the current face recognition products and face recognition systems need to have certain restrictions or requirements on the detected faces. These restrictions have become the main obstacles to the promotion and application of face recognition technology. These restrictions exist because: under uncontrollable conditions, complex interference factors will lead to a sharp drop in the accuracy of face recognition, which cannot meet the application requirements. Under uncontrollable conditions, not only may there be serious interference factors such as strong light changes, large-scale posture changes, exaggerated expression changes, intentional or unintentional occlusion, and low image resolution, but these factors may appear in random combinations in video face images. These complex interferences will lead to huge differences in the facial images of the same person. This makes it very difficult to accurately recognize faces under uncontrolled conditions. Therefore, unrestricted face recognition is still a very difficult problem; its recognition accuracy is far from meeting the needs of practical applications.

However, in unrestricted face recognition, most current work focuses on reducing the noise contained in a single face image, while ignoring the phenomenon of "visual noise condition difference". For example, some researchers use image preprocessing technology to eliminate the interference contained in the image, such as lighting normalization technology, image noise reduction technology, pose evaluation technology, etc. Other researchers reduce the impact of complex interference by extracting robust facial feature representations. When a face image contains strong disturbances, none of the current preprocessing techniques and feature extraction techniques can effectively remove or suppress them. Correspondingly, when a pair of face images have different visual interference conditions, it is difficult for current face verification techniques to identify them accurately.

For a pair of face images, when their visual interference conditions are different, selecting a local area image pair with similar interference conditions to verify the pair of faces can effectively reduce the impact of local strong interference. Moreover, many studies have shown that a satisfactory recognition rate can also be obtained by using only a single face part discrimination. Based on the above findings, if multiple face regions are predefined, for the input face image pairs to be recognized, by evaluating whether they have differences in visual interference conditions, a more reliable face region image pair can be selected. This will effectively reduce the impact of local strong interference on the face verification process.

In recent years, scholars at home and abroad have begun to apply deep learning methods to image recognition problems, and have achieved excellent results. An important characteristic of the deep learning algorithm is that the training sample set is relatively large. In the video face recognition process, tens of thousands or hundreds of thousands of sample data can be easily generated through a system mechanism. Therefore, video face recognition Combining with deep learning algorithms, it can effectively solve the problem of insufficient sample set size of deep learning algorithms. At the same time, since the deep convolutional neural network can extract features that are robust and complementary to factors such as illumination, expression, and posture, it can be used to construct the essential features of the image, which can greatly improve the accuracy of the results of the video face recognition system under unrestricted conditions. sex.

Contents of the invention

In view of the above-mentioned prior art, the purpose of the present invention is to provide a face recognition method based on deep learning and key point feature extraction, to solve the problem of insufficient sample set size of its deep learning algorithm in the prior art, and to prevent interference of the video face recognition system under unrestricted conditions Technical problems such as weak ability, low recognition accuracy and poor robustness.

In order to achieve the above object, the technical scheme adopted in the present invention is as follows:

A face recognition method based on deep learning and key point feature extraction, comprising the following steps,

Step 1, obtain the video image, and extract the Haar-like feature in the video image;

Step 2. Construct a cascaded strong classifier according to the Haar-like feature, and then use the strong classifier to detect the human eye area image in the video image;

Step 3. Symmetrically set at least 7 key points on the image of the human eye area, and then divide it into regions to obtain local image blocks;

Step 4. Obtain the preprocessed local image blocks with the same key points in the face database, match them to the local image blocks to obtain image block pairs corresponding to the key points, and then use the deep convolutional neural network to extract the feature vectors of the image block pairs ;

Step 5. Calculate the classifier decision score of the feature vector, and determine the image block pair with the highest classifier decision score, wherein the face image belonging to the preprocessed local image block in the face database is used as the output result of the recognition.

In the above method, the step 2 includes the following steps,

Step 2.1, normalize the weight, corresponding to Haar-like features, calculate the error rate of the simple classifier, select the simple classifier with the lowest error rate as the weak classifier, and update the weight;

Step 2.2. According to the self-adaptive enhancement algorithm, the weak classifier is iterated in step 2.1 to obtain a strong classifier, and all the strong classifiers are cascaded;

Step 2.3, using cascaded strong classifiers to detect the image of the human eye area in the video image.

In the above method, in step 3, 23 key points are set for the image of the human eye area.

In the above method, the step 4 includes the following steps,

Step 4.1, obtain the preprocessed local image blocks with the same key points in the face database, match them to the local image blocks to obtain image block pairs corresponding to the key points, calculate the gradient of the key points and extract the joint normalization of the image blocks to the key points local features;

Step 4.2, preset the gradient difference threshold condition, and eliminate the image block pairs where the key points of the gradient do not meet the gradient difference threshold condition;

Step 4.3, using a deep convolutional neural network to extract high-dimensional feature vectors of the remaining image block pairs;

Step 4.4. Connect the high-dimensional feature vectors to the joint normalized local features corresponding to key points to obtain feature vectors.

In the above method, the step 5 includes the following steps,

Step 5.1, calculate the cosine distance vector of the feature vector;

Step 5.2, according to the cosine distance vector, train and obtain the classifier decision function;

Step 5.3: Use the classifier decision function to evaluate the similarity of the image block pairs, and determine the image block pair with the highest classifier decision score, and the face image belonging to the preprocessed local image block in the face database is used as the output result of the recognition.

Compared with prior art, the beneficial effect of the present invention:

(1) The present invention can well realize the face image detection in the video, and can weaken the interference of the non-face image to the detection result at the same time;

(2) The present invention uses the feature extraction algorithm of deep learning, which can obtain complex interference such as posture, expression, illumination, occlusion, more distinguishing, complementary, low-dimensional human face features, which can Reduce the error rate of face recognition to a great extent;

(3) The present invention can adaptively select the decision score of the classifier according to the differences in visual conditions, and greatly reduce the error rate of face recognition under unrestricted conditions.

Description of drawings

The video face detection flowchart that Fig. 1 the present invention provides;

Fig. 2 is a schematic diagram of the division of human face regions and the positions of key points provided by the present invention;

Fig. 3 is a schematic diagram of the convolutional neural network structure of the present invention.

detailed description

All features disclosed in this specification, or steps in all methods or processes disclosed, may be combined in any manner, except for mutually exclusive features and/or steps.

The present invention will be further described below in conjunction with accompanying drawing:

Example 1

Process 1, face detection based on Adaboost adaptive enhancement algorithm

(1.1) The specific implementation of the optimized combination of weak classifiers generated by Adaboost algorithm training is as follows. The flow chart of face detection is shown in Figure 1:

Given n training samples: (x ₁ , y ₁ ), (x ₂ , y ₂ ),...(x _n , y _n ) where y _i = 0, 1, representing face area and non-face area respectively , _xi represents a 24×24 sub-window.

The initial sample weights are:

where m and l are the number of negative samples and positive samples, respectively.

Convert the Haar-Like Haar-like feature into a weak classifier. The definition formula of the weak classifier is as follows:

Among them: h _j (x) represents the value of the simple classifier; θ _j is the threshold; p _j represents the direction of the inequality sign, which can only be ±1; f _j (x) represents the image block normalized from the size to 24×24 , the extracted Haar-Like eigenvalues.

Select T optimal weak classifiers, each classifier selection method is as follows:

1. Normalized weights:

2. Train a classifier for each feature j, and calculate the error rate of the classifier:

3. Select the classifier h _t with the lowest error rate as a weak classifier

4. Update weights:

When the sample x _i is correctly classified, e _i =0, otherwise e _i =1, Repeat the above four steps until e _i =0.

Finally, the selected T weak classifiers are combined into a strong classifier:

in, Cascade trained several strong classifiers to form a cascade classifier for face detection.

(1.2) Use the trained cascade classifier to detect the faces in the video, and select several areas that may be faces. In order to improve the accuracy of human face detection, the present invention uses Haar-Like feature and Adaboost algorithm to detect human eye area for the above possible human face area. If there is an eye area inside the detected face, it means that the face area is a real face area, otherwise it means that the face area is a fake face area, and the detection result is discarded.

Process 2, face area selection method based on anomaly detection

Divide the face image into 10 regions R _i , i=1,...,10, define 23 key points on the face image, see Figure 2 for the region division and key point positions.

(2.1) Divide the standard face image I _a,j ,j=1,...,n of each person in the face database into 10 regions, which are recorded as R _ai ,i=1,...,10, where n is the database the number of people in. The corresponding 23 key points in each area are located, and the local features of these key points in each area are extracted. That is, for each face image in the database, local feature vectors H _a,i of key points in 10 regions are obtained, where i represents the number of key points in each region.

(2.2) Divide the detected video face image into 10 regions, denoted as R _bi , i=1,...,10. The corresponding 23 key points in each area are located, and the local features H _b,i of these key points in each area are extracted, where i represents the number of key points in each area.

(2.3) Video face images and face images in each database form face pairs, and n face pairs (I _a,i ,I _b ) can be obtained. For a pair (I _a,i ,I _b ) on a given face area R _i , get the local features H _a,i and H _b,i of I _a,i and I _b on P key points, respectively, i=1,...,p.

The joint normalization local feature extraction process is as follows:

Note that the local neighborhood centered on the key point P is N, which includes four blocks in the upper left, upper right, lower left, and lower right, respectively marked as B1, B2, B3, and B4, and each block contains four smaller unit blocks . Note that N _0.5 is the local neighborhood of point P after the image is doubled, which has similar blocks and unit blocks to N.

Calculate M(x,y), θ(x,y), (x,y)∈N, where M(x,y), θ(x,y) represent the gradient magnitude on the point (x,y) respectively and gradient direction, θ(x,y)∈[0,π];

Afterwards, for each unit in N, the gradient orientation histogram projection is performed with the amplitude M(x, y) as the weight, and 16 8-bit histograms H _i , i=1,...,16 are generated, and the Four units perform L2 normalization;

On B1, B2, B3, and B4 of N, respectively perform gradient direction histogram projection with the amplitude M(x, y) as the weight, and generate four 8-bit histograms, which are recorded as H _i , i=17,..., 20, they perform L2 normalization together;

Calculate M(x,y), θ(x,y), (x,y)∈N _0.5 , perform the same operation on N _0.5 , and generate 4 8-bit histograms marked as H _i ,i=21,… ,24, which together perform L2 normalization;

Perform gradient direction histogram projection on N _0.5 with the amplitude M(x, y) as the weight to generate an 8-bit histogram, perform L2 normalization, denoted as H ₂₅ ; jointly normalized H _i , i=1, …, 25, forming a 200-dimensional local feature H on point P.

(2.4) For all image pairs (I _a,i ,I _b ), perform the flip operation to get r _j and Denote their reliability estimates, where j=1,...,10.

The reliability r on each region R _i , i=1,...,10 is:

Among them, p` is the number of key points satisfying e _k <1, e _k ,k=1,...,p represents the abnormal difference detection results of (I _a,i ,I _b ) at p key points, defined As follows: e _k = e _μ e _ν

in,

k _μ and k _ν are control parameters; d _c and d _χ are the cosine distance and χ ² distance, which represent the gradient difference of the key point P.

(2.5) For all image pairs, use the following region selection method to select regions with less visual interference conditions in each pair of image pairs (gradient difference threshold condition):

1. If but

2. If r ₁ =1, output the full-face image pair (I _a,i,1 ,I _b,1 ) of R ₁ ;

3. If r ₁ <1, but there is r _j =1, then output the image pair satisfying r _j =1;

4. If all r _j < 1, then output the 5 region image pairs with the largest r _j .

Process 3, high-level face feature expression based on deep convolutional neural network

(3.1) For each pair of images (I _a,i ,I _b ), regions with less visual disturbance conditions have been selected. The selected area in each image pair is scaled to 31×39, and input to the convolutional neural network respectively, and the output layer obtains a 160-dimensional feature vector. The structure of the convolutional neural network is shown in Figure 3. According to different image pairs, different numbers of 160-dimensional vectors are selected. Denote these vectors as F _a ={F _a1 ,F _a2 ,...,F _am } and F _b ={F _b1 ,F _b2 ,...,F _bm }, m is the number of regions selected in the image pair, take this The m features are used as high-level face features describing the image pair.

(3.2) On several areas selected from the face image pair, the local features of all the key points contained in the area are expressed as and As n local feature vectors, n bits are the number of key points in the area. This operation is performed for each face image pair (I _a,i ,I _b ).

(3.3) Use PCA algorithm for F and Dimensionality reduction is performed on each feature vector in , and 90% of their information is retained. And use the min-max technique for F and Each eigenvector in is normalized to keep their eigenvalues in the [0,1] range.

Process 4. Evaluating face similarity based on distance metric learning

(4.1) For each image pair, calculate F and The cosine distances of the individual eigenvectors in :

A number of multi-distance vectors are obtained, which are denoted as Xi after _{normalization} .

(4.2) Calculate the classifier decision score _s of Xi, which is defined as follows

where φ ^b (x), φ ^c (x) and φ ^g (x) denote the decision functions of the classifier under three different visual consistency conditions, respectively. τ ₁ and τ ₂ define how to select a suitable classifier according to e, and e represents the VCM evaluation result. In the present invention, they are obtained from the training set for training φ ^b (x), φ ^c (x) and φ ^g (x).

According to the above process, using the selected face pair region R _j to calculate X _j respectively, the classifier decision score s _j =ψ _j (X _j , e _j ) of the image pair can be obtained.

(4.3) Evaluate the similarity between the faces in the video image and the faces in the face database one by one, select the pair with the highest s _j =ψ _j (X _j , e _j ), and put the corresponding i-th face in the data set The identities corresponding to the face images I _{a, i} are output as the recognition results.

The above is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto, any changes or substitutions that can be easily imagined by those skilled in the art within the technical scope disclosed in the present invention, All should be covered within the protection scope of the present invention.

Claims (5)

1. A face recognition method based on deep learning and key point feature extraction is characterized by comprising the following steps,

step 1, acquiring a video image, and extracting a haar-like feature in the video image;

step 2, constructing a cascade strong classifier according to the haar-like characteristics, and detecting a human eye region image in the video image by using the strong classifier;

step 3, symmetrically setting at least 7 key points on the eye region image, and then carrying out region division on the eye region image to obtain a local image block;

step 4, acquiring preprocessed local image blocks with the same key points in a face database, matching the preprocessed local image blocks with the preprocessed local image blocks to obtain image block pairs corresponding to the key points, and extracting feature vectors of the image block pairs by using a deep convolution neural network;

and 5, calculating a classifier decision score of the feature vector, and judging an image block pair with the highest classifier decision score, wherein a face image where the preprocessed local image block belongs to the face database is used as an identification output result.

2. The face recognition method based on deep learning and key point feature extraction as claimed in claim 1, wherein the step 2 comprises the following steps,

step 2.1, normalizing the weight, calculating the error rate of the simple classifiers corresponding to the class haar characteristics, selecting the simple classifier with the lowest error rate as a weak classifier, and updating the weight;

step 2.2, according to the self-adaptive enhancement algorithm, carrying out step 2.1 loop iteration on the weak classifiers to obtain strong classifiers, and cascading all the strong classifiers;

and 2.3, detecting the human eye region image in the video image by using the cascaded strong classifiers.

3. The method for recognizing the human face based on the deep learning and the key point feature extraction as claimed in claim 1, wherein in the step 3, 23 key points are set for the human eye region image.

4. The face recognition method based on deep learning and key point feature extraction as claimed in claim 3, wherein the step 4 comprises the following steps,

step 4.1, acquiring preprocessed local image blocks with the same key points in a face database, matching the preprocessed local image blocks with the preprocessed local image blocks to obtain image block pairs corresponding to the key points, calculating the gradient of the key points and extracting the joint normalized local features of the key points of the image block pairs;

step 4.2, presetting a gradient difference threshold condition, and eliminating image block pairs where key points of gradients which do not meet the gradient difference threshold condition belong;

4.3, extracting high-dimensional feature vectors of the remaining image block pairs by using a deep convolutional neural network;

and 4.4, connecting the high-dimensional feature vector with the joint normalized local features of the corresponding key points to obtain the feature vector.

5. The face recognition method based on deep learning and key point feature extraction as claimed in claim 1, wherein the step 5 comprises the following steps,

step 5.1, calculating cosine distance vectors of the feature vectors;

step 5.2, training and solving a decision function of the classifier according to the cosine distance vector;

and 5.3, evaluating the similarity of the image block pairs by utilizing a classifier decision function, and judging the image block pair with the highest classifier decision score, wherein the face image where the preprocessed local image block belongs to the face database is used as an identification output result.