CN106980811A - Facial expression recognition method and facial expression recognition device - Google Patents

Abstract

The invention discloses a facial expression recognition method and a facial expression recognition device, wherein the facial expression recognition method comprises the following steps: acquiring a face image sequence to be recognized, wherein the face image sequence comprises a single frame or more than two frames of face images; respectively preprocessing each frame of face image in the face image sequence; inputting each preprocessed frame of face image into a trained training model for expression recognition to obtain an expression recognition result of the face image sequence; the input end to the output end of the training model are sequentially constructed by a convolutional neural network model, a long-time memory cyclic neural network model, a first pooling layer and a logistic regression model, and the training model is obtained by training a continuous frame image set labeled with expression categories. The technical scheme provided by the invention can effectively improve the recognition performance of the facial expression.

Description

Facial expression recognition method and human facial expression recognition device

technical field

The invention relates to the technical field of image recognition, in particular to a facial expression recognition method and a human facial expression recognition device.

Background technique

Facial expression recognition technology refers to assigning an expression category to a given face image, including: anger, disgust, happiness, sadness, fear, surprise, etc. At present, facial expression recognition technology has gradually shown broad application prospects in the fields of human-computer interaction, clinical diagnosis, distance education and investigation and interrogation, and is a popular research direction of computer vision and artificial intelligence.

At present, there is a facial expression recognition method based on a deep convolutional neural network. After the face image is detected and calibrated, the calibrated human face image is input into a trained deep convolutional neural network. Expression recognition. In the above-mentioned facial expression recognition method, the deep convolutional neural network is obtained through single-frame image training, and since the facial expression is closely related to the scene context and is very related to the neutral expression of the object, the above-mentioned facial expression It is difficult for the recognition method to accurately recognize neutral expressions, and the recognition performance of facial expressions is poor.

Contents of the invention

The invention provides a human facial expression recognition method and a human facial expression recognition device, which are used to improve the recognition performance of human facial expressions.

A first aspect of the present invention provides a method for recognizing facial expressions, comprising:

Obtain a sequence of human face images to be identified, the sequence of human face images comprising a single frame or more than two frames of human face images;

Preprocessing each frame of face images in the sequence of face images respectively;

Input each frame of human face image after preprocessing into the training model that has been trained and carry out expression recognition, obtain the expression recognition result of described human face image sequence;

Wherein, the input end to the output end of the training model is successively constructed by a convolutional neural network model, a long-short-term memory recurrent neural network model, a first pooling layer and a logistic regression model, and the training model is continuously A collection of frame images is trained.

A second aspect of the present invention provides a facial expression recognition device, comprising:

An image acquisition unit, configured to acquire a sequence of human face images to be recognized, the sequence of human face images comprising a single frame or more than two frames of human face images;

An image preprocessing unit, configured to preprocess each frame of human face images in the sequence of human face images;

The recognition processing unit is used to input the preprocessed frames of human face images into the trained training model for expression recognition, and obtain the expression recognition results of the human face image sequence;

Wherein, the input end to the output end of the training model is successively constructed by a convolutional neural network model, a long-short-term memory recurrent neural network model, a first pooling layer and a logistic regression model, and the training model is continuously A collection of frame images is trained.

As can be seen from the above, in the present invention, a training model is constructed based on a Long Short Term Memory Recurrent Neural Network (LSTM-RNN, Long Short TermMemory-Recurrent Neural Networks) model, and a continuous frame image collection (such as a video) is used as the training input of the training model, The training model can make full use of the dynamic information of facial expression changes to automatically learn to identify the neutral expression of the object and the mapping relationship between different posture and expression features, thereby improving the prediction accuracy and robustness of the training model, and further improving the accuracy of facial expressions. Expression recognition performance.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those skilled in the art, other drawings can also be obtained according to these drawings without any creative effort.

Fig. 1-a is a schematic flow chart of an embodiment of a method for recognizing facial expressions provided by the present invention;

Fig. 1-b is a schematic structural diagram of a training model embodiment applied to the facial expression recognition method shown in Fig. 1-a provided by the present invention;

Figure 1-c is a schematic diagram of the timing processing flow of the training model shown in Figure 1-b in an application scenario provided by the present invention;

Figure 1-d is a schematic diagram of the timing processing flow of the training model shown in Figure 1-b in another application scenario provided by the present invention;

Fig. 1-e is a schematic structural diagram of an LSTM-RNN model applied to the training model shown in Fig. 1-b provided by the present invention;

Fig. 1-f is a schematic structural diagram of a CNN model applied to the training model shown in Fig. 1-b provided by the present invention;

FIG. 2 is a schematic structural diagram of an embodiment of a facial expression recognition device provided by the present invention.

detailed description

In order to make the purpose, features and advantages of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described The embodiments are only some of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

Embodiment one

An example of the present invention provides a method for recognizing facial expressions. As shown in Figure 1-a, the facial expression recognition method in the embodiment of the present invention includes:

Step 101, acquiring a face image sequence to be recognized;

Wherein, the above sequence of human face images includes a single frame or more than two frames of human face images. That is, the facial expression recognition method in the embodiment of the present invention can recognize continuous multi-frame facial images (such as video), and is also compatible with the recognition of single-frame facial images.

In step 101, the face image sequence to be recognized can be obtained in real time through the camera, or the face image sequence to be recognized can be obtained by receiving the face image sequence from an external device, or it can also be based on user The face image sequence to be recognized is obtained by selecting from an existing image database or video database, which is not limited here.

Step 102, performing preprocessing on each frame of face images in the above-mentioned sequence of face images respectively;

After obtaining the sequence of human face images to be recognized in step 101, preprocessing is performed on each frame of human face images in the above-mentioned sequence of human face images, so that the preprocessed human face images can be more suitable for subsequent expression recognition, specifically Therefore, in different application scenarios, corresponding processing methods can also be used for the preprocessing of face images.

For example, in one embodiment, the above-mentioned preprocessing of each frame of face images in the above-mentioned sequence of face images may specifically include the following two steps:

Step 1. Perform face detection on each frame of face images in the above frames of face images, and determine the face area. The above-mentioned face detection process can be implemented by using various face detection algorithms, such as the Adaboost face detection algorithm based on Haar-Like features. Based on the face detection algorithm, the input image (that is, each frame of the above-mentioned face image) can be scanned with a window of an appropriate size and an appropriate step size until the face area in the face image is determined (the face area is also the human face area). area of the face).

Step 2. Detect the key feature points in the above-mentioned face area, and perform alignment and calibration on the corresponding face images based on the detected key feature points. On the basis of face detection, the positions of key feature points (such as eyes, eyebrows, nose, mouth, facial outline, etc.) in the face area are further determined. According to the key feature points detected in the face area, the corresponding face images can be aligned and calibrated through rigid body transformation, so that the positions of the key feature points of the face in the image are basically the same. In the embodiment of the present invention, specifically, the landmark method may be used to perform alignment and calibration of face images. In addition, in the process of aligning and calibrating the face image, the positioning and adjustment of key feature points can also be performed according to the preset face model.

Further, in order to avoid the inconsistency of image size affecting the recognition result, the above-mentioned preprocessing of each frame of face images in the above-mentioned face image sequence may also include the following steps: aligning the calibrated face images in step 2 according to the preset The template is edited to obtain a face image of a uniform size, wherein the editing includes one or more of the following: cutting and scaling. For example, in the above editing process, based on the detected key feature points in the face area, the corresponding face image is cut out according to a uniform template, and the face image is scaled to a uniform size.

It should be noted that, if the above-mentioned sequence of human face images includes a single frame of human face images, the above-mentioned preprocessing of each frame of human face images in the above-mentioned sequence of human face images is actually performed as preprocessing the single frame of human face images. ; If the above-mentioned sequence of human face images includes more than two frames of human face images, then the above-mentioned preprocessing of each frame of human face images in the above-mentioned sequence of human face images is actually performed by performing preprocessing on each frame of human face images in the above-mentioned two or more frames of human face images. Face images are preprocessed separately.

Step 103, input the preprocessed frames of human face images into the trained training model for expression recognition, and obtain the expression recognition result of the above-mentioned human face image sequence;

In step 103, each frame of human face images preprocessed in step 102 is input into the trained training model for expression recognition, and an expression recognition result of the above human face image sequence is obtained. The facial expression recognition result may indicate the facial expression category to which the human face image sequence belongs, wherein the existing facial expression categories may include but not limited to: angry, calm, confused, disgusted, happy, sad, scared, surprised, squinting and screaming.

In the implementation of the present invention, as shown in Figure 1-b, the above-mentioned training model is sequentially composed of a convolutional neural network (CNN, Convolutional Neural Network) model, a long-short-term memory recurrent neural network model (i.e. LSTM-RNN model) from the input end to the output end of the above-mentioned training model. ), the first pooling layer and logistic regression model construction. In addition, the above training model is obtained by training a set of consecutive frame images marked with expression categories. Since the above-mentioned training model is trained through a collection of consecutive frame images marked with expression categories, on the one hand, the above-mentioned training model can automatically learn the dependence of the time scale, make full use of the dynamic information of facial expression changes, and connect the front and back of the current frame of expression Frame information makes expression recognition more robust; on the other hand, neutral expressions can be precisely defined to eliminate the influence of differences in expression tension and intensity between different objects and improve recognition accuracy; on the other hand, due to Each frame image in the continuous frame image set has a strong correlation with the labeled expression category, so expression recognition can be realized even if the input image sequence is distorted.

Optionally, the above-mentioned first pooling layer may be an average pooling layer or a maximum pooling layer or other types of pooling layers, which are not limited herein.

Optionally, if the above-mentioned sequence of human face images contains more than two frames of human face images, the above-mentioned input of each pre-processed frame of human face images into the trained training model for face recognition includes: passing through the above-mentioned first pool The transformation layer uniformly performs dimension reduction processing on the face feature vectors of the above-mentioned frames of face images input by the above-mentioned long-short-term memory recurrent neural network model, and obtains the face feature vectors after dimension reduction processing; outputs the above-mentioned dimension reduction to the above-mentioned logistic regression model The processed face feature vector. The following is a description of the sequence processing flow of the training model by taking continuous frame images (that is, the input face image sequence contains more than two frames of face images) as an example, as shown in Figure 1-c. Among them, X0, X1, ..., Xn are each frame image of a video with a length of n frames, and the face feature vectors extracted by the CNN module of each frame image are input into the LSTM module in chronological order, and will be processed by the LSTM module The obtained face feature vectors h0, h1, ..., hn output at different times are processed through the first pooling layer for dimensionality reduction to obtain the face feature vector h for expression classification, and finally the face feature vector h Input the logistic regression model to perform logistic regression processing to obtain the facial expression recognition results of the continuous frame images. When the input face image sequence is a single-frame face image (i.e. the above n=1), the sequence processing flow diagram of the training model shown in Figure 1-c can be simplified to the training model shown in Figure 1-d Schematic diagram of timing processing flow.

In the embodiment of the present invention, the structure of the LSTM-RNN model contained in the above training model can be shown in Figure 1-e, including: input gate (ie input gate), forget gate (ie forget gate), output gate (ie output gate) gate), state unit (ie cell) and LSTM-RNN model output.

For the case where the input face image sequence contains more than two frames of face images, the processing process of the above-mentioned input gate, the above-mentioned forgetting gate, the above-mentioned output gate, the above-mentioned state unit and the output result of the above-mentioned LSTM-RNN model can be realized by the following formulas respectively:

i _t = σ(W _ix x _t +W _im m _t-1 +W _ic c _t-1 + _bi );

f _t = σ(W _fx x _t +W _fm m _t-1 +W _fc c _t-1 +b _f );

c _t = f _t ⊙c _t-1 +i _t ⊙σ(W _cx x _t +W _cm m _t-1 +b _c );

o _t = σ(W _ox x _t +W _om m _t-1 +W _oc c _t-1 +b _o );

m _t =o _t ⊙h(c _t ).

Among them, in the above formula, x _t represents the face feature vector input at time t; W (ie W _ix , W _im , W _ic , W _fx , W _fm , W _fc , W _cx , W _cm , W _ox , W _om and W _oc ) are preset weight matrices, indicating that the elements of each gate are obtained from the data of the corresponding dimension, that is to say, nodes of different dimensions do not interfere with each other; b (namely b _i , b _f , b _c , b _o ) represent the preset bias vector, and it , f _t , o _t , c _t , m _t respectively represent the above-mentioned input gate, the above-mentioned forget gate, the above-mentioned output gate, and the above-mentioned state unit at time _t And the state of the output result of the above LSTM-RNN model, ⊙ is the dot product, σ() is the sigmoid function, h() is the output activation function of the above state unit, and the output activation function can specifically be a tanh function.

Optionally, for the case where the input face image sequence contains a single frame of face images, the processing process of the above-mentioned input gate, the above-mentioned forgetting gate, the above-mentioned output gate, the above-mentioned state unit and the output result of the above-mentioned LSTM-RNN model can also be simplified as The following formula is implemented:

i _t =σ(W _ix x _t +consatant ₁ );

f _t = σ(W _fx x _t + consatant ₂ );

c _t = f _t ⊙c _t-1 +i _t ⊙σ(W _cx x _t +consatant ₃ );

o _t = σ(W _ox x _t +W _om m _t-1 +consatant ₄ );

m _t =o _t ⊙h(c _t ).

Among them, in the above formula, x _t represents the face feature vector input at time t; W (ie W _ix , W _im , W _ic , W _fx , W _fm , W _fc , W _cx , W _cm , W _ox , W _om and W _oc ) are preset weight matrices, indicating that the elements of each gate are obtained from the data of the corresponding dimension, that is to say, nodes of different dimensions do not interfere with each other; consatant (ie consatant ₁ , consatant ₂ , consatant ₃ and consatant ₄ ) are preset constants, and it , f _t , o _t , c _t , _{m t represent} the above-mentioned input gate, the above-mentioned forgetting gate, the above-mentioned output gate, the above-mentioned state unit and the above-mentioned The state of the output result of the LSTM-RNN model, ⊙ is the dot product, σ() is the sigmoid function, h() is the output activation function of the above state unit, and the output activation function can specifically be a tanh function.

Optionally, as shown in FIG. 1-f, the above CNN model is sequentially constructed from the input end to the output end by the first convolutional layer, the second pooling layer, the second convolutional layer and the third pooling layer. The above-mentioned input of each pre-processed face image into the trained training model for face recognition includes: outputting the face feature vector obtained after the above-mentioned third pooling layer processing to the above-mentioned LSTM-RNN model. Wherein, the above-mentioned second pooling layer and third pooling layer may be an average pooling layer or a maximum pooling layer or other types of pooling layers, which are not limited herein. Certainly, in other embodiments, the above CNN model may also be constructed with reference to an existing CNN model, which is not limited here.

The above-mentioned process of training the above-mentioned training model is described below through the continuous frame image collection of the above-mentioned labeling expression category, specifically as follows: 1, collect one or more continuous frame image collections (the above-mentioned continuous frame image collection can include continuous frames image (such as video)) and the expression category to which each continuous frame image set belongs (the expression category to which each image in the same continuous frame image set belongs is the same), and the expression category to which each continuous frame image set belongs is marked as expected to pass the above training The expression category output by the model. In the embodiment of the present invention, a variety of expression categories (such as anger, calm, confusion, disgust, happiness, sadness, fear, surprise, squinting, screaming, etc.) can be preset, and each expression category corresponds to a mapping value. 2. Perform preprocessing on the images in the above-mentioned continuous frame image set (for the preprocessing process, refer to the description in step 102, which will not be repeated here). 3. Input the preprocessed image into the above training model, and train the training model based on the backpropagation algorithm, so that the output value of the input image processed by the above training model and the mapping value of the expression category to which the image belongs The deviation is within the preset allowable range. Of course, the training process of the training model can also be realized by referring to other existing technical solutions, which is not limited here.

It should be noted that the facial expression recognition method in the embodiment of the present invention can be executed by a facial expression recognition device, and the facial expression recognition device can be integrated in a robot, a monitoring terminal or other terminals, which is not limited here.

As can be seen from the above, the facial expression recognition method in the embodiment of the present invention builds a training model based on the LSTM-RNN model, and uses a collection of continuous frame images (such as video) as the training input of the training model, which can make the training model fully utilize the facial expressions. The dynamic information of facial expression changes automatically learns the mapping relationship between the neutral expression of the recognition object and the expression features of different poses, thereby improving the prediction accuracy and robustness of the training model, and then improving the recognition performance of facial expressions.

Embodiment two

The example of the present invention provides a kind of human facial expression recognition device, as shown in Figure 2, the human facial expression recognition device 200 in the embodiment of the present invention comprises:

An image acquisition unit 201, configured to acquire a sequence of human face images to be recognized, the sequence of human face images comprising a single frame or more than two frames of human face images;

An image preprocessing unit 202, configured to preprocess each frame of human face images in the sequence of human face images;

Recognition processing unit 203, for inputting each preprocessed human face image into the trained training model for expression recognition, and obtaining the expression recognition result of the human face image sequence;

Wherein, the input end to the output end of the training model is successively constructed by a convolutional neural network model, a long-short-term memory recurrent neural network model, a first pooling layer and a logistic regression model, and the training model is continuously A collection of frame images is trained.

Optionally, the recognition processing unit 203 is specifically configured to: when the sequence of human face images contains more than two frames of human face images, use the first pooling layer to process the input of the long short-term memory recurrent neural network model. The face feature vectors of each frame of face images are uniformly subjected to dimension reduction processing to obtain face feature vectors after dimension reduction processing; and output the face feature vectors after dimension reduction processing to the logistic regression model.

Optionally, the input terminal to the output terminal of the convolutional neural network model is sequentially constructed by the first convolutional layer, the second pooling layer, the second convolutional layer and the third pooling layer; the recognition processing unit 203 specifically uses In: outputting the face feature vector obtained after being processed by the third pooling layer to the long-short-term memory recurrent neural network model.

Optionally, the image preprocessing unit 202 is specifically configured to: perform face detection for each frame of face images in the frames of face images, and determine the face area; detect key feature points in the face area, And based on the detected key feature points, the corresponding face images are aligned and calibrated.

Optionally, the image preprocessing unit 202 is specifically further configured to: edit the aligned and calibrated face image according to a preset template to obtain a face image of a uniform size, wherein the editing process includes the following one Or two or more: clipping processing, scaling processing.

It should be noted that the facial expression recognition device in the embodiment of the present invention may be integrated into a robot, a monitoring terminal or other terminals. For the functions of each functional module of the facial expression recognition device, reference may be made to the description in the above-mentioned method embodiments, and for the specific implementation process, reference may be made to the relevant descriptions in the above-mentioned method embodiments, which will not be repeated here.

As can be seen from the above, the facial expression recognition device in the embodiment of the present invention builds a training model based on the LSTM-RNN model, and uses a collection of continuous frame images (such as video) as the training input of the training model, so that the training model can make full use of facial expressions. The dynamic information of facial expression changes automatically learns the mapping relationship between the neutral expression of the recognition object and the expression features of different poses, thereby improving the prediction accuracy and robustness of the training model, and then improving the recognition performance of facial expressions.

In the several embodiments provided in this application, it should be understood that the disclosed devices and methods may be implemented in other ways.

It should be noted that, for the sake of simplicity of description, the aforementioned method embodiments are expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence. Because of the present invention, certain steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification belong to preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

In the foregoing embodiments, the descriptions of each embodiment have their own emphases, and for parts not described in detail in a certain embodiment, reference may be made to relevant descriptions of other embodiments.

The above is a description of a facial expression recognition method and a facial expression recognition device provided by the present invention. For those of ordinary skill in the art, according to the ideas of the embodiments of the present invention, there will be specific implementation methods and application ranges. Changes, in summary, the contents of this specification should not be construed as limiting the present invention.

Claims (10)

1. a kind of facial expression recognizing method, it is characterised in that including：

Human face image sequence to be identified is obtained, the human face image sequence includes single frames or two frame above facial images；

Each frame facial image in the human face image sequence is pre-processed respectively；

The training pattern that pretreated each frame facial image input has been trained carries out Expression Recognition, obtains the face figure As the Expression Recognition result of sequence；

Wherein, the input of the training pattern circulates god by convolutional neural networks model, long short-term memory successively to output end Built through network model, the first pond layer and Logic Regression Models, and the training pattern is by marking the continuous of classification of expressing one's feelings Two field picture set training is obtained.

2. facial expression recognizing method according to claim 1, it is characterised in that if the human face image sequence includes two Frame above facial image, then, the training pattern that pretreated each frame facial image input has been trained carry out face Identification, including：

Pass through each frame facial image of first pond layer to the long short-term memory Recognition with Recurrent Neural Network mode input Face feature vector it is unified carry out dimension-reduction treatment, obtain the face feature vector after dimension-reduction treatment；

The face feature vector after the dimension-reduction treatment is exported to the Logic Regression Models.

3. facial expression recognizing method according to claim 1 or 2, it is characterised in that the convolutional neural networks model Input to output end successively by the first convolutional layer, the second pond layer, the second convolutional layer and the 3rd pond layer building；

The training pattern that pretreated each frame facial image input has been trained carries out recognition of face, including：To institute State the face feature vector obtained after long short-term memory Recognition with Recurrent Neural Network model output is handled through the 3rd pond layer.

4. facial expression recognizing method according to claim 1 or 2, it is characterised in that described respectively to the face figure As each frame facial image in sequence is pre-processed, including：

Face datection is carried out for every frame facial image in each frame facial image, human face region is determined；

The key feature points in the human face region are detected, and corresponding facial image is entered based on the key feature points detected Row alignment.

5. facial expression recognizing method according to claim 4, it is characterised in that in the detection human face region Key feature points, and alignment is carried out to corresponding facial image based on the key feature points detected, also include afterwards：

Facial image after alignment is subjected to editing and processing according to default template, to obtain the face figure of unified size Picture, wherein, the editing and processing includes following one or more kinds of：Shear treatment, scaling processing.

6. a kind of expression recognition device, it is characterised in that including：

Image acquisition unit, the human face image sequence to be identified for obtaining, the human face image sequence includes single frames or two frames Above facial image；

Image pre-processing unit, for being pre-processed respectively to each frame facial image in the human face image sequence；

Identifying processing unit, the training pattern for pretreated each frame facial image input have been trained carries out expression knowledge Not, the Expression Recognition result of the human face image sequence is obtained；

Wherein, the input of the training pattern circulates god by convolutional neural networks model, long short-term memory successively to output end Built through network model, the first pond layer and Logic Regression Models, and the training pattern is by marking the continuous of classification of expressing one's feelings Two field picture set training is obtained.

7. expression recognition device according to claim 6, it is characterised in that identifying processing unit specifically for：When When the human face image sequence includes two frame above facial images, the long short-term memory is circulated by first pond layer The face feature vector of each frame facial image of neural network model input is unified to carry out dimension-reduction treatment, obtains dimension-reduction treatment Face feature vector afterwards；

The face feature vector after the dimension-reduction treatment is exported to the Logic Regression Models.

8. the expression recognition device according to claim 6 or 7, it is characterised in that the convolutional neural networks model Input to output end successively by the first convolutional layer, the second pond layer, the second convolutional layer and the 3rd pond layer building；

The identifying processing unit specifically for：To the long short-term memory Recognition with Recurrent Neural Network model output through the 3rd pond Change the face feature vector obtained after layer processing.

9. the expression recognition device according to claim 6 or 7, it is characterised in that described image pretreatment unit has Body is used for：Face datection is carried out for every frame facial image in each frame facial image, human face region is determined；Detection is described Key feature points in human face region, and alignment is carried out to corresponding facial image based on the key feature points detected.

10. expression recognition device according to claim 9, it is characterised in that described image pretreatment unit is specific It is additionally operable to：Facial image after alignment is subjected to editing and processing according to default template, to obtain the face of unified size Image, wherein, the editing and processing includes following one or more kinds of：Shear treatment, scaling processing.