CN114529969A - Expression recognition method and system - Google Patents

Abstract

The present invention claims to protect an expression recognition method and system, belonging to the technical field of biological feature recognition. The method includes the steps of: obtaining a face expression picture sample and a corresponding category label; performing face detection and face alignment according to the face expression sample; establishing a deep neural network model, and sending the face expression picture into the depth Extract features from the neural network model to obtain expression features; select multiple triples according to the expression category labels, and each triplet includes three samples from different categories; A loss value; calculate the cross entropy loss as the second loss value according to the real category label and the obtained expression feature; send the expression feature to the classifier for classification, and output the classification result. The main purpose of the present invention is to readjust the inter-class distance between negative expressions and improve the problem of poor recognition effect of negative expressions.

Description

A method and system for facial expression recognition

technical field

The invention belongs to the technical field of biological feature recognition, and in particular relates to an expression recognition method and system.

Background technique

Facial expression recognition is to analyze the input face picture to determine which type of expression the current input face picture belongs to. Common expression recognition methods usually classify seven basic expressions, including: happy, surprised, neutral, fearful, disgusted, angry, and sad.

Most of the existing expression recognition methods use deep learning convolutional neural networks to extract facial expression features and classify them. The obvious flaw shared by current methods is that the accuracy of negative expressions is significantly lower than that of positive expressions. The so-called positive expressions are expressions with positive emotions, including happiness and surprise, and negative expressions are expressions with negative emotions, including fear, disgust, anger, and sadness.

There are two reasons for the poor effect of negative expressions:

(1) There are relatively few negative expression samples. This problem exists in almost all expression datasets. The sample size of negative expressions is smaller than that of positive expressions. Especially for large datasets, the sample sizes of positive and negative expressions are very different. This reason causes the neural network to fail to learn strong robust features on limited negative expressions, which eventually leads to the inability of the neural network to accurately judge negative expressions during the testing process.

(2) Facial features are similar between negative expressions. The change of expression is composed of movements of multiple local positions on the face, and the local movements corresponding to negative expressions overlap, such as frowning in anger and disgust. Similar facial movements are similar features, so that the neural network cannot accurately determine which type of expression the current feature belongs to.

In reality, people are more likely to act aggressively and impulsive when they have negative emotions. In practical applications, it is assumed that the driver's emotion monitoring or the monitoring of the patient's state requires others to intervene when the observed person has a negative expression, so it is particularly important to accurately identify the negative expression.

After retrieval, application publication number CN107358169A, a facial expression recognition method and facial expression recognition device, comprising: constructing and training an emotion recognition model based on convolutional neural networks; inputting a face image to be recognized into the emotion recognition model , to output the emotional category of the face image, and the emotional category includes one of positive emotions, negative emotions and neutral emotions; obtain an expression recognition model corresponding to the emotional category; input the facial image into the facial expression recognition model model to output the expression category of the face image. The invention recognizes the facial expressions in a hierarchical manner, selects different expression recognition models according to different emotions, reduces the content that each recognition model needs to memorize, and reduces the computational complexity of the entire expression recognition process , which improves the operation efficiency.

The method disclosed in the patent CN107358169A is to firstly discriminate the large categories of positive, negative and neutral emotions, and then match different recognition models according to different categories. The reason for the poor recognition effect of negative expressions is that the features between expressions are similar, and the patent does not carry out targeted operations on negative expressions, but only separates an independent recognition model, but in this independent recognition model, the negative expressions The features are still similar, and the recognition of negative expressions does not improve. However, the present invention designs a new triple loss function, and can focus on adjusting the distance between classes of negative expressions by controlling the source of categories of triples. This loss function can increase the distance between classes of negative expressions. , thereby enhancing the discrimination of negative expression features.

Application Publication No. CN111353390A, a deep learning-based micro-expression recognition method, comprising the following steps: 1: crop video data containing facial expressions, divide the video into frames, and extract; 2: perform face alignment on the extracted facial expression sequences, Face cropping, normalization and other preprocessing; 3: Perform data enhancement operations on the obtained data set; 4: Build a neural network model: 5: Divide all facial expression data into training set and test set in proportion; 6: Use the test set to test the model, and output information such as recognition accuracy, recognition time, and error. When the recognition rate meets the requirements, select the current model.

The method of the patent CN111353390A is to improve the neural network model structure, but the focus is on the overall expression recognition effect, and does not focus on the existing pain points. However, the present invention proposes a loss function specially suitable for distinguishing similar classes in view of the problem of poor negative expression effect. In addition, the present invention does not need to change the neural network model structure, can directly use the classical network structure, only needs to record the eigenvalues before the fully connected layer, and the method is simple and easy to operate.

SUMMARY OF THE INVENTION

The present invention aims to solve the above problems of the prior art. An expression recognition method and system are proposed. The technical scheme of the present invention is as follows:

An expression recognition method, comprising the following steps:

Obtaining face expression picture samples and corresponding category labels; performing face detection and face alignment according to the face expression samples;

A deep neural network model is established, and the facial expression picture is sent into the deep neural network model to extract features to obtain expression features;

Select multiple triples according to the expression category label, and each triple includes three samples from different categories;

Calculate the first loss value according to the expression feature and the triplet;

Calculate the cross-entropy loss as the second loss value according to the real class label and the obtained expression features;

The facial expression features are sent to the classifier for classification, and the classification result is output.

Further, the facial expression picture samples and the corresponding category labels specifically include:

Use a camera to record videos of facial expressions;

Observing the time when the facial expression appears in the video, extracting the frame corresponding to the time and saving it as an image, obtaining a sample of the facial expression picture, and labeling the category of the expression.

Further, establishing a deep neural network model, sending the facial expression picture into the neural network model to extract features, and obtaining expression features, specifically including:

Create an 18-layer residual network model, and initialize the parameters included in the model, the neural network learning rate, optimizer and other parameters;

Input the expression samples into the deep neural network to obtain the feature values before the last layer of the deep neural network;

Input the eigenvalues before the last layer into the last fully connected layer of the model to get the final output eigenvalues.

Further, selecting a plurality of triples according to the expression category label, and the three samples included in each triple are from different categories, specifically including:

Traverse the entire tag set, when the three traversed samples belong to different categories, store the subscripts corresponding to the current three samples into the set, and obtain a triplet set storing the subscripts;

In order to ensure that the distances between the three sample features in each triplet are not balanced, three samples are filtered from the triplet set obtained in the previous step, which all belong to the combination of positive or negative expressions. The negative expressions themselves have similar characteristics, and the method to measure the similarity of the characteristics is the distance between the features. The closer the distance is, the more similar the negative expressions will be. The meaning of ensuring that the distances between the three samples in the triplet are unbalanced is that, assuming that the currently selected triplet contains two negative expressions and one positive expression, the distance between the positive expression and the negative expression will be relatively far, while the two The distance between the negative expressions is relatively close, that is, the distance is uneven. In this case, through the subsequent loss function control, the distance between the two negative expressions can be made farther, and the value is close to the distance between the positive expression and the negative expression, so as to achieve the purpose of readjusting the distance between the negative expressions, so that the distance between the negative expressions can be adjusted. increases, and the inter-class distance of positive and negative expressions tends to be equal to the inter-class distance of negative and negative expressions. However, if the currently selected triples belong to negative expressions, there is no significant difference in the distance between samples at the beginning, so it is difficult to make the distance between classes of negative and negative expressions reach the size of the distance between classes of positive and negative expressions. The same is true for positive expressions. This is the purpose of not selecting triples that belong to both negative or positive expressions.

Further, calculating the first loss value according to the expression feature and the triplet specifically includes:

A loss value is calculated for each triplet in the triplet set, and three corresponding features are extracted from the expression features obtained before the last layer of the neural network according to the subscript stored in the triplet. Three Euclidean distances are formed, and two of them are randomly selected each time, so that the ratio of the two Euclidean distances tends to be 1. Therefore, the square value of the subtraction value 1 of the ratio of the two Euclidean distances is taken as the loss value, and finally a plurality of three distances are taken. The average of the loss values computed by the tuple is taken as the final value of the first loss.

Further, according to the subscript stored in each triple in the obtained triple set, the corresponding three 512-dimensional eigenvalues x ₁ , x ₂ , x ₃ are extracted from the features obtained before the last layer of the neural network. , three Euclidean distances can be obtained through three features, and two Euclidean distances d1 and d2 are randomly selected each time. The Euclidean distance calculation formula is as follows:

The ratio of d1 and d2 tends to 1 as the loss function, the purpose is to make the distances d1 and d2 between the two classes tend to be equal, each triplet calculates a loss value, and finally takes the average value. The loss function formula is as follows, where T is the number of triples:

The loss function can only make the two distances between the three samples tend to be equal for each triple, but two different distances are randomly selected in different triples, so on the whole, the three distances will be equal. All tend to be equal, and the result will make all inter-class distances tend to be equal.

Further, calculating the cross entropy loss as the second loss value according to the real category label and the expression feature output by the last layer of the neural network specifically includes:

Assume that the output value of the i-th expression sample through the last layer of the neural network is P _i = (p _i1 ,p _i2 ,...,p _iM ), and its real label Y _i =(y _i1 ,y _i2 ,...,y _iM ) Calculate the cross entropy loss together, where M represents the number of expression categories. In the representation of the real label, assuming that the sample belongs to the jth category, the value of y _ij in Y _i is 1, and the rest are 0. The calculation formula is as follows, where N is Number of samples:

According to the order of magnitude of the two loss values, the magnitude of the two loss values is adjusted to be consistent by the coefficient λ and added. The calculation formula is as follows:

Loss=λ*Loss1+Loss2

The predicted category is calculated according to the output P _i of the network model, and the subscript corresponding to the maximum value is the predicted category, and finally the predicted category is output. .

A facial expression recognition system using any one of the methods, comprising:

The input module is used to input the facial expression image into the expression recognition system;

The expression acquisition module is used to process multiple input facial expression images, perform face detection and face alignment on the input image samples, and obtain the facial expression image samples;

a standardization processing module, which is used to standardize the facial expression picture samples, so that the sizes of the facial expression pictures are the same, and the standardized facial expression picture samples are obtained;

The format conversion module is used to convert the obtained standardized facial expression pictures into the tensor format required by the neural network model;

Model management module for creating, managing and saving neural network models;

The feature extraction module is used to extract the features corresponding to the expression samples through the neural network model;

Combination generation module to generate expression triples that meet the requirements;

The loss calculation module is used to calculate two loss values, the first loss value is calculated according to the triplet, the second loss value is calculated according to the true label cross entropy loss value, and finally the two losses are combined, and According to the loss, the gradient backhaul is performed to update the network parameters;

The prediction category module is used to predict the classification result according to the features extracted by the feature extraction module, and output the prediction result.

Further, the combination generation module is used to select a triplet that meets the requirements according to the sample label, each sample in a batch of samples corresponds to a unique subscript, and the selection result of the triplet is three samples from different categories. subscripts, and the three samples are not both negative expressions or positive expressions at the same time, the purpose of storing the subscripts is to facilitate the subsequent calculation of the loss to extract the corresponding features according to the subscripts;

The loss calculation module is used to calculate two loss functions, and add the two values after a certain adjustment. The adjustment is mainly to make the order of magnitude of the two loss values consistent, and to ensure that the weights of the two are approximately equal, that is, to ensure that both are equal. Play a role.

The advantages and beneficial effects of the present invention are as follows:

The invention aims to improve the problem that the negative expression recognition effect of the existing facial expression recognition method is poor, and provides an expression recognition method and system, which can enhance the feature discrimination by controlling the distance of the negative expression in the feature space, and reduce the difference between the negative expression and the positive expression. Differences in expression accuracy.

In the traditional neural network-based expression recognition method, the distances between positive expressions and positive and negative expressions are relatively far, while the distance between negative expressions is relatively small, that is, there is a distance between different categories The problem of imbalance leads to a relatively low accuracy rate of negative expressions, and the present invention aims to improve this problem.

The main innovation of the paper Learning Informative and Discriminative Features for FacialExpression Recognition in the Wild is to improve the center loss function, which is designed to minimize the distance between each sample and the center of the class to which it belongs, and maximize the distance to other class centers. The loss function designed by the patent CN113887325A is the distance between each sample and the class center to which it belongs. The difference between the present invention and the above two methods is that the loss functions of the above two methods have the same restriction on all categories, and the result of this loss function adjustment is that the distance between classes is still unbalanced. The main advantage of the present invention is that the distance between three samples in a triplet is adjusted each time, and the source of the categories of the three selected samples is controlled. The feature of the present invention is that this method does not perform the same operation on all categories , but focuses on the categories with similar distances, that is, between negative and negative expressions, so it can mainly improve the problem of poor recognition of negative expressions in a targeted manner.

Description of drawings

Fig. 1 is the flow chart of facial expression recognition method that the present invention provides preferred embodiment;

Figure 2 is a structural diagram of the facial expression recognition system.

Detailed ways

The technical solutions in the embodiments of the present invention will be described clearly and in detail below with reference to the accompanying drawings in the embodiments of the present invention. The described embodiments are only some of the embodiments of the invention.

The technical scheme that the present invention solves the above-mentioned technical problems is:

Example 1:

As shown in Figure 1, the expression recognition method of the present invention comprises the following steps:

Step 1: Use the camera to collect clear facial expression videos of many people, each person contains seven basic expressions. Extract frames from the collected video, extract frames with expressions in the video, and obtain valid face expression picture samples;

Step 2: Perform face detection and face key point detection on the face expression picture sample obtained in step 1. The specific method is to use the algorithm in the Dlib library, which can preliminarily obtain the face area and face key points. The transformation matrix method is used to align the face, so that the nose of all face images is in the middle of the picture;

Step 3: Label the categories to which the facial expression picture samples obtained in step 2 belong, and use 0-6 to represent seven expressions, respectively, to obtain a complete expression data set for subsequent training of the network model;

Step 4: Create a deep neural network model and input all expression picture samples into the model. Specific steps are as follows:

(1) Standardize the facial expression picture samples obtained in step 2, and set the size of all samples to 224×224 to obtain standardized expression picture samples;

(2) Divide the standardized expression picture samples into multiple sets, each set contains different kinds of expressions, convert the samples into the tensor format required by the neural network, and obtain multiple tensor sets;

(3) Create an 18-layer residual network model, and load the network model parameters trained on a large face dataset into the current model to obtain an initialized neural network model;

(4) Input multiple tensor sets into the neural network model for feature extraction in turn to obtain the corresponding expression features. The i-th expression sample obtains the feature value x _i =(x _i1 ,x _i2 ,… , x _in ), where n=512, each sample gets a 512-bit feature;

(5) Input the expression features obtained in the previous step into the last fully connected layer of the neural network to obtain the classification information P _i = (p _i1 ,p _i2 ,...,p _iM ), where M is the number of categories, and each sample corresponds to seven values, respectively representing the possibility that the sample belongs to seven categories;

Step 5: Select triples according to the labels corresponding to the current tensor set. The specific operations are as follows:

(1) Traverse the entire label set, when the three traversed samples belong to different categories, store the subscripts corresponding to the current three samples in the set, and obtain a triplet set storing the subscripts;

(2) In order to ensure that the distances between the three sample features in each triplet are unbalanced, three samples are filtered out from the triplet set obtained in the previous step, which all belong to the combination of positive or negative expressions;

Step 6: According to the subscript stored in each triplet in the triplet set obtained in step 5, the corresponding three 512-dimensional eigenvalues are extracted from the features obtained in the third step of step 4. Three Euclidean distances, two of which are randomly selected each time, d1, d2. The Euclidean distance calculation formula is as follows:

Using the ratio of d1 and d2 tending to 1 as the loss function, each triplet calculates a loss value, and finally takes the average. The loss function formula is as follows, where T is the number of triples:

The loss function only controls two distances between the three samples for each triple, but randomly selects two different distances in different triples, so it controls the three distances as a whole. That is, the control of the distance between all categories;

Step 7: According to the final classification information P _i obtained in the fourth step of Step 4, calculate the cross entropy loss together with the real label of the expression sample Y _i =(y _i1 ,y _i2 ,...,y _iM ), in the representation of the real label , if the sample belongs to the jth class, the value of y _ij in _{Yi i} is 1, and the rest are 0. The calculation formula is as follows, where N is the number of samples:

Step 8: According to the magnitude of the two loss values, adjust the magnitude of the two loss values to be consistent with the coefficient λ and add them. The calculation formula is as follows:

Loss=λ*Loss1+Loss2

Step 9: Calculate the predicted category according to the output P _i of the network model, where the subscript corresponding to the maximum value is the predicted category, and finally the predicted category is output.

Example 2:

As shown in Figure 2, an expression recognition system includes:

The input module is used to input the facial expression image and the corresponding category label into the expression recognition system;

The expression acquisition module is used to process multiple input facial expression images, perform face detection and face alignment on the input image samples, and obtain the facial expression image samples;

a standardization processing module, which is used to standardize the facial expression picture samples, so that the sizes of the facial expression pictures are the same, and the standardized facial expression picture samples are obtained;

The format conversion module is used to convert the obtained standardized facial expression pictures into the tensor format required by the neural network model;

Model management module for creating, managing and saving neural network models;

The feature extraction module is used to extract the features corresponding to the expression samples through the neural network model;

Combination generation module to generate expression triples that meet the requirements;

The loss calculation module is used to calculate two loss values, the first loss value is calculated according to the triplet, the second loss value is calculated according to the real label, and the cross entropy loss value is calculated, and finally the two losses are combined;

The prediction category module is used to predict the classification result according to the features extracted by the feature extraction module, and output the prediction result.

The facial expression acquisition module is used to obtain the facial expression area from the input picture, and the Dlib library is used to perform face detection and face alignment on the picture sample according to the facial expression picture sample, so as to obtain a normalized facial expression sample; the normalization processing module , which is used to standardize the facial expression samples, normalize the sample data of the facial expression pictures, and set the size of all samples to 224×224;

The model management module includes a model creation unit, a model parameter loading unit and a model storage unit. The specific steps are to first create an 18-layer residual network structure in the model creation unit, and second, in the model parameter loading unit, first obtain the network parameters from the model trained on the large face dataset, and load the pre-training parameters Enter the created model, and finally the model storage unit is responsible for saving the model parameters that meet the conditions in multiple predictions and training;

The feature extraction module is used to obtain the features corresponding to the input expression samples, and input the samples into the neural network model to obtain the features corresponding to the samples. It mainly includes two parts. The first is the output before the last layer of the network, which is used to calculate The first loss value. The second is the output of the last layer of the network, which is used to calculate the cross entropy loss and calculate the classification result;

The combination generation module is used to select triples that meet the requirements. First, select all triples that can be composed of samples of different categories according to the labels of the samples, delete three combinations that belong to the same positive expression or the same negative expression, and obtain a triplet set, in which each triplet stores the selected The subscript corresponding to the sample;

The loss calculation module is used to calculate two kinds of loss functions. The first is to calculate the triplet set obtained by combining the generation module, calculate a loss for each triplet, and finally take the average of all triples. value as the first loss value. The second loss is the cross-entropy loss, which is calculated from the output of the last layer of the network and the true label. Finally, the two values are added after a certain adjustment, and the adjustment is mainly to make the order of magnitude of the two loss values consistent;

The predicting category module is used for predicting the final category, using the second part of the features obtained by the feature extraction module to predict, and outputting the final prediction result.

The systems, devices, modules or units described in the above embodiments may be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or A combination of any of these devices.

It should also be noted that the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device comprising a series of elements includes not only those elements, but also Other elements not expressly listed, or which are inherent to such a process, method, article of manufacture, or apparatus are also included. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, method, article of manufacture, or device that includes the element.

The above embodiments should be understood as only for illustrating the present invention and not for limiting the protection scope of the present invention. After reading the contents of the description of the present invention, the skilled person can make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims (9)

1. a facial expression recognition method, is characterized in that, comprises the following steps: Obtaining face expression picture samples and corresponding category labels; performing face detection and face alignment according to the face expression samples; A deep neural network model is established, and the facial expression picture is sent into the deep neural network model to extract features to obtain expression features; Select multiple triples according to the expression category label, and each triple includes three samples from different categories; Calculate the first loss value according to the expression feature and the triplet; Calculate the cross-entropy loss as the second loss value according to the real class label and the obtained expression features; The facial expression features are sent to the classifier for classification, and the classification result is output. 2. a kind of expression recognition method according to claim 1, is characterized in that, described face expression picture sample and corresponding category label, specifically comprise: Use a camera to record videos of facial expressions; Observing the time when the facial expression appears in the video, extracting the frame corresponding to the time and saving it as an image, obtaining a sample of the facial expression picture, and labeling the category of the expression. 3. a kind of facial expression recognition method according to claim 2, is characterized in that, described establishing deep neural network model, the facial expression picture is sent into the neural network model to extract feature, obtains facial expression feature, specifically comprises: Create an 18-layer residual network model, and initialize the parameters included in the model, the neural network learning rate, optimizer and other parameters; Input the expression samples into the deep neural network to obtain the feature values before the last layer of the deep neural network; Input the eigenvalues before the last layer into the last fully connected layer of the model to get the final output eigenvalues. 4. A kind of facial expression recognition method according to claim 3, is characterized in that, described according to facial expression category label to select a plurality of triples, the three samples that each triple includes are from different categories, specifically include: Traverse the entire tag set, when the three traversed samples belong to different categories, store the subscripts corresponding to the current three samples into the set, and obtain a triplet set storing the subscripts; In order to ensure that the distances between the three sample features in each triplet are unbalanced, three samples are filtered out from the triplet set obtained in the previous step, all of which belong to the combination of positive or negative expressions; the characteristics of negative expressions themselves are similar , and the method to measure the similarity of features is the distance between features. The closer the distance, the more similar, that is, the distance between negative expressions will be closer. The meaning of ensuring that the distances between the three samples in the triplet are unbalanced is that, assuming that the currently selected triplet contains two negative expressions and one positive expression, the distance between the positive expression and the negative expression will be relatively far, while the two The distance between the negative expressions is relatively close, that is, the distance is uneven. In this case, through the subsequent loss function control, the distance between the two negative expressions can be made farther, and the value is close to the distance between the positive expression and the negative expression, so as to achieve the purpose of readjusting the distance between the negative expressions, so that the distance between the negative expressions can be adjusted. increases, and the inter-class distance of positive and negative expressions tends to be equal to the inter-class distance of negative and negative expressions. However, if the currently selected triples belong to negative expressions, there is no significant difference in the distance between samples at the beginning, so it is difficult to make the distance between classes of negative and negative expressions reach the size of the distance between classes of positive and negative expressions. The same is true for positive expressions. This is the purpose of not selecting triples that belong to both negative or positive expressions. 5. a kind of facial expression recognition method according to claim 4, is characterized in that, described according to facial expression feature and triple to calculate the first loss value, specifically comprises: A loss value is calculated for each triplet in the triplet set, and three corresponding features are extracted from the expression features obtained before the last layer of the neural network according to the subscript stored in the triplet. Three Euclidean distances are formed, and two of them are randomly selected each time, so that the ratio of the two Euclidean distances tends to be 1. Therefore, the square value of the subtraction value 1 of the ratio of the two Euclidean distances is taken as the loss value, and finally a plurality of three distances are taken. The average of the loss values computed by the tuple is taken as the final value of the first loss. 6. A kind of expression recognition method according to claim 5, is characterized in that, according to the subscript stored in each triple in the obtained triple set, take out the corresponding feature from the feature obtained before the last layer of the neural network. The three 512-dimensional eigenvalues x ₁ , x ₂ , and x ₃ of , three Euclidean distances can be obtained through the three features, and two Euclidean distances d1 and d2 are randomly selected each time. The Euclidean distance calculation formula is as follows:

The ratio of d1 and d2 tends to 1 as the loss function, the purpose is to make the distances d1 and d2 between the two classes tend to be equal, each triplet calculates a loss value, and finally takes the average value. The loss function formula is as follows, where T is the number of triples:

The loss function can only make the two distances between the three samples tend to be equal for each triple, but two different distances are randomly selected in different triples, so on the whole, the three distances will be equal. All tend to be equal, and the result will make all inter-class distances tend to be equal. 7. A kind of facial expression recognition method according to claim 6, is characterized in that, described according to the facial expression characteristic of true category label and neural network last layer output gained, calculates cross entropy loss as the second loss value, specifically comprises: Assume that the output value of the i-th expression sample through the last layer of the neural network is P _i = (p _i1 ,p _i2 ,...,p _iM ), and its real label Y _i =(y _i1 ,y _i2 ,...,y _iM ) Calculate the cross entropy loss together, where M represents the number of expression categories. In the representation of the real label, assuming that the sample belongs to the jth category, the value of y _ij in Y _i is 1, and the rest are 0. The calculation formula is as follows, where N is Number of samples:

According to the order of magnitude of the two loss values, the magnitude of the two loss values is adjusted to be consistent by the coefficient λ and added. The calculation formula is as follows: Loss=λ*Loss1+Loss2 The predicted category is calculated according to the output P _i of the network model, and the subscript corresponding to the maximum value is the predicted category, and finally the predicted category is output. 8. A facial expression recognition system adopting the method of any one of claims 1-7, characterized in that, comprising: The input module is used to input the facial expression image into the expression recognition system; The expression acquisition module is used to process multiple input facial expression images, perform face detection and face alignment on the input image samples, and obtain the facial expression image samples; a standardization processing module, which is used to standardize the facial expression picture samples, so that the sizes of the facial expression pictures are the same, and the standardized facial expression picture samples are obtained; The format conversion module is used to convert the obtained standardized facial expression pictures into the tensor format required by the neural network model; Model management module for creating, managing and saving neural network models; The feature extraction module is used to extract the features corresponding to the expression samples through the neural network model; The combined generation module is used to generate expression triples that meet the requirements; The loss calculation module is used to calculate two loss values, the first loss value is calculated according to the triplet, the second loss value is calculated according to the true label cross entropy loss value, and finally the two losses are combined, and According to the loss, the gradient backhaul is performed to update the network parameters; The prediction category module is used to predict the classification result according to the features extracted by the feature extraction module, and output the prediction result. 9. The facial expression recognition system according to claim 8, wherein the combination generation module is used to select a triplet that meets the requirements according to the sample label, and each sample in a batch of samples corresponds to a unique subscript, and the three The selection result of the tuple is the subscripts of three samples from different categories, and the three samples are not both negative expressions or positive expressions at the same time. The purpose of storing the subscripts is to facilitate the subsequent calculation of the loss to extract the corresponding features according to the subscripts; The loss calculation module is used to calculate two loss functions, and add the two values after a certain adjustment. The adjustment is mainly to make the order of magnitude of the two loss values consistent, and to ensure that the weights of the two are approximately equal, that is, to ensure that both are equal. Play a role.

Images