CN109614927B - Micro-expression recognition based on the difference between before and after frames and feature dimensionality reduction - Google Patents

Abstract

The present application provides a micro-expression recognition method, which performs face recognition on each frame in a video, and extracts a face area; extracts the number of pixels, background color, and face brightness of each frame in the video; frame, calculate the face area difference, pixel number difference, background color difference, and face brightness difference between the selected frame and the previous frame; calculate the difference value of each non-first frame; And, the first frame of the video is determined as a candidate frame; in the candidate frame, the frames with continuous identification are determined as micro-expression frames; the expression features of the micro-expression frames are extracted, and the expression features are dimensionally reduced by a pre-trained dimensionality reduction model Process, identify the features after dimensionality reduction, and obtain the identification result. In this application, micro-expression frames are selected for identification according to the difference in the area of the face area, the difference in the number of pixels, the difference in the background color, and the difference in the brightness of the face, so that the frames related to the micro-expression in the face video can be accurately extracted, and the recognition efficiency of the micro-expression frame can be improved. with accuracy.

Description

Micro-expression Recognition Based on Difference Between Front and Back Frames and Feature Dimensionality Reduction

technical field

This application relates to the technical field of artificial intelligence, and in particular to a micro-expression recognition method.

Background technique

Micro-expression is a kind of non-verbal behavior that can show people's own emotions.

At present, most of the attention is on ordinary expressions. In addition to ordinary facial expressions, there are also micro-expressions produced by uncontrolled contraction of facial muscles under psychological inhibition.

Microexpressions are short in duration and have very small movements. Therefore, it is quite difficult to observe and identify correctly. The success rate of accurately capturing and recognizing micro-expressions with the naked eye is very low. After professional training, the recognition rate can only reach 47%.

Therefore, the recognition method of micro-expressions has attracted more and more researchers' attention.

Contents of the invention

In order to solve the above problems, the embodiment of the present application proposes a micro-expression recognition method.

Get face video;

Carry out face recognition to each frame in the video, and extract the face area;

Extract the number of pixels, background color, and face brightness of each frame in the video;

Select a non-first frame in turn, and calculate the face area difference, pixel number difference, background color difference, and face brightness difference between the selected frame and the previous frame;

Calculate the difference value of each non-first frame, wherein, difference value=(face area difference*face brightness difference+background color difference)^pixel number difference;

Determining a frame whose difference value is greater than a preset threshold, and the first frame of the video as a candidate frame;

In the candidate frame, the frames with continuous marks are all determined as micro-expression frames;

The expression features of micro-expression frames are extracted, and the dimensionality reduction processing is performed on the expression features through the pre-trained dimensionality reduction model, and the features after dimensionality reduction are recognized to obtain the recognition results.

Optionally, extracting the background color of each frame in the video includes:

For any frame in the video,

Determining the non-face area in any frame as the background area;

Determine the RGB color value of each pixel in the background area of any frame, and the RGB color value includes a red color value, a green color value, and a blue color value;

The RGB color mean value of the background area of any frame is calculated by the following formula, and the RGB color mean value includes a red color mean value, a green color mean value, and a blue color mean value:

为所述任一帧背景区域的红色颜色均值，

为所述任一帧背景区域的绿色颜色均值，

为所述任一帧背景区域的蓝色颜色均值，c_1j为所述任一帧背景区域第j像素点的红色颜色值，c_2j为所述任一帧背景区域第j像素点的绿色颜色值，c_3j为所述任一帧背景区域第j像素点的蓝色颜色值，n₁为所述任一帧背景区域中像素点总数量；Wherein j is the pixel point identification of the background area of any frame,

is the mean value of the red color of the background area of any frame,

is the mean value of the green color of the background area of any frame,

For the blue color mean value of the background area of any frame, _c1j is the red color value of the jth pixel in the background area of any frame, and _c2j is the green color of the jth pixel in the background area of any frame value, c _3j is the blue color value of the jth pixel in the background area of any frame, and n ₁ is the total number of pixels in the background area of any frame;

Calculate the RGB color mean square error of any frame background area, the RGB color mean square error includes red color mean square error, green color mean square error, blue color mean square error:

Among them, σ ₁₁ is the mean square deviation of the red color, σ ₂₁ is the mean square deviation of the green color, and σ ₃₁ is the mean square deviation of the blue color;

绿色颜色区间

蓝色颜色区间

Determine the RGB color interval of the background area of any frame, and the RGB color interval includes the red color interval

green color range

blue color range

In all pixels in the background area of any frame, determine that the red color value in the RGB color value is located in the red color interval in the RGB color interval, and the green color value is located in the green color interval, and the blue color value is located in the blue color interval The number of pixels in the color interval n2;

Determine the background color of any frame according to n2.

Optionally, the background color is represented by an RGB color value;

The determining the background color of any frame according to n2 includes:

Calculating the pixel number ratio n ₃ =n ₂ /n ₁ of any frame;

绿色颜色值为

蓝色颜色值为

The background color of either frame has a red color value of

The green color value is

The blue color value is

Optionally, extracting the face brightness of each frame in the video includes:

For any frame in the video,

Determine the brightness value of each pixel in the face area of any frame by the following formula:

Wherein, k is the pixel point identification of described any frame human face area, h _k is the luminance value of the kth pixel point of described any frame human face area, R _k is the RGB color value in the kth pixel point Red color value, G _k is the green color value in the RGB color value of the k pixel point, B _k is the blue color value in the RGB color value of the k pixel point;

Among the brightness values of all pixels in the human face area of any frame, determine a maximum brightness value and a minimum brightness value;

其中，n₄为所述任一帧的人脸区域中像素点总数量；Calculate the mean value of the brightness of the face area of any frame

Wherein, _n4 is the total number of pixels in the face area of any frame;

确定所述视频中任一帧的人脸亮度。According to the maximum brightness value, minimum brightness value and

Determine the brightness of a face at any frame in the video.

确定所述视频中任一帧的人脸亮度，包括：Optionally, according to the maximum brightness value, the minimum brightness value and

Determine the brightness of a face in any frame of the video, including:

Calculate the first difference d1=maximum brightness value-minimum brightness value;

Calculate the second difference

Calculate the third difference

Calculate brightness ratio d4=|d1-d2|/|d1-d3|;

Calculate the mean square error of the brightness of the face area of any frame

The face brightness of any frame in the video is

Optionally, before calculating the difference value of each non-first frame, it also includes:

According to the face area difference, pixel number difference, background color difference, and face brightness difference of each non-first frame, the non-first frame is initially screened;

The calculation of the difference value of each non-first frame includes:

Calculate the difference value of each frame after the primary screening.

Optionally, the preliminary screening of the non-first frames according to the face area difference, pixel number difference, background color difference, and face brightness difference of each non-first frame includes:

For any non-first frame,

If the face area difference of any non-first frame is not greater than the first value, and the pixel number difference is not greater than the second value, and the background color difference is not greater than the third value, and the face brightness difference is not greater than fourth value, then any non-first frame passes the preliminary screening; or,

If the face area difference of any non-first frame is not greater than the first value, but the pixel number difference, background color difference, and face brightness difference are all 0, then any non-first frame passes the preliminary screening; or ,

If the face brightness difference of any non-first frame is not greater than the fourth value, but the face area difference, pixel count difference, and background color difference are all 0, then any non-first frame passes the preliminary screening;

Wherein, the first value is (the total number of frames of the human face video of (the sum of the face area difference of all non-first frames+the face area of the first frame-avg1)/described human face video, and the second value is (all non-first frames The sum of the difference in the number of pixels+the number of pixels of the first frame-avg2)/the total number of frames of the face video, and the third value is (the sum of the background color differences of all non-first frames+the background color of the first frame-avg3) /The total number of frames of the human face video, the fourth value is (the sum of the difference in brightness of the human face of all non-first frames+the brightness of the human face of the first frame-avg4)/the total number of frames of the human face video, avg1= The sum of the face area areas of all frames/the total number of frames of the human face video, avg2=the sum of the pixels of all frames/the total number of frames of the described human face video, avg3=the sum of the background colors of all frames/ The total number of frames of the human face video, avg4=the sum of the brightness of the human face of all frames/the total number of frames of the human face video.

Optionally, before performing the dimensionality reduction processing on the expression features through the pre-trained dimensionality reduction model, it also includes:

Obtain a sample set X, where the total number of samples in X is m, each sample includes multiple expression features, and each sample belongs to a category;

Classify all samples by category;

其中，i为类标识，μ_i为第i类的均值向量，b_i为第i类的样本数量，j为样本标识，x_ij为第i类第j个样本的表情特征组成的向量；Calculate the mean vector of each class

Wherein, i is the class identification, μ _i is the mean value vector of the i-th class, b _i is the number of samples of the i-th class, j is the sample identification, and x _ij is a vector composed of the expression features of the j-th sample of the i-th class;

其中，μ₀为总均值向量，E为X中样本所属不同类别总数；According to the mean vector of each category, determine the total mean vector

Among them, μ ₀ is the total mean value vector, and E is the total number of different categories of samples in X;

Calculate the between-class variance vector and the intra-class variance vector according to the total mean vector;

The dimensionality-reduced expression features are determined according to the inter-class variance vector and the intra-class variance vector to form a dimensionality reduction model.

Optionally, the calculation of the inter-class variance vector and the intra-class variance vector according to the total mean vector includes:

Among them, S _w is the between-class variance vector, S _b is the intra-class variance vector, and Xi is the set of _i -th class samples.

Optionally, the determining the dimensionality-reduced expression feature according to the inter-class variance vector and the intra-class variance vector includes:

Calculate the weight vector W=diag(S _b /S _w ) composed of each expression feature weight, wherein, diag () is a function, and the function is used to get the elements on the matrix diagonal, / is an operator, said operator is used to _divide corresponding elements of _Sw and Sb;

According to the order of the weight of each expression feature from large to small, the expression features are sorted;

A preset number of top-ranked expression features are determined as expression features after dimensionality reduction.

Beneficial effects are as follows:

Select micro-expression frames for recognition based on face area difference, pixel number difference, background color difference, and face brightness difference, which can accurately extract frames related to micro-expressions in face videos, and improve the recognition efficiency and accuracy of micro-expression frames sex.

Description of drawings

Specific embodiments of the application will be described below with reference to the accompanying drawings, wherein:

Fig. 1 shows a schematic diagram of the principle of a dimensionality reduction model divided into two categories provided by an embodiment of the present application;

Fig. 2 shows a schematic flow chart of a micro-expression recognition method provided by an embodiment of the present application;

Fig. 3 shows a schematic diagram of calculation of an LBP descriptor provided by an embodiment of the present application;

FIG. 4 shows a schematic diagram of feature extraction provided by an embodiment of the present application.

Detailed ways

In order to make the technical solutions and advantages of the present application clearer, the exemplary embodiments of the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all implementations. Exhaustive list of examples. And in the case of no conflict, the embodiments in this description and the features in the embodiments can be combined with each other.

Due to the short duration of micro-expression, and the movement range is very small. Therefore, it is quite difficult to observe and identify correctly. Based on this, the present application provides a micro-expression recognition method, which compares the difference between each frame and the next frame, and the difference with the previous frame to obtain the difference value of the frame, and determine the micro-expression according to the difference value of each frame frame, this method can accurately extract frames related to micro-expressions in face videos, and improve the recognition efficiency and accuracy of micro-expression frames.

The facial expression recognition method provided in this application includes two major processes, the first major process is the process of training the dimensionality reduction model, and the other major process is the actual micro-expression recognition process based on the trained dimensionality reduction model.

The process of training the dimensionality reduction model is not a process that must be performed every time the expression recognition method provided by this application is executed, only when the expression recognition method provided by this application is executed for the first time, or the expression recognition scene changes, or, based on the training When the dimensionality reduction model is used for actual micro-expression recognition, the dimensionality reduction effect of expression features is not ideal, or other reasons, the process of training dimensionality reduction model will be executed to improve the dimensionality reduction effect of expression features, and then improve the accuracy of the actual micro-expression recognition results. accuracy.

This application does not limit the execution trigger conditions of the process of training the dimensionality reduction model.

The specific implementation method of training the dimensionality reduction model process is as follows:

Step 1, get sample set X.

Among them, the total number of samples in X is m, each sample includes multiple expression features, and each sample belongs to a category.

For example, if the samples in X belong to E different categories, they are the first category, the second category, ..., the i-th category, ... the E-th category. There are b ₁ samples in the first category, and the set of b ₁ samples is X ₁ , there are b ₂ samples in the second category, and the set of b ₂ samples is X ₂ ,  ….

Step 2, classify all samples by category.

Taking Liezi in step 1 as an example, in this step, all samples are classified into class E, samples belonging to class 1 are classified into class 1, samples belonging to class 2 are classified into class 1, ... .

Step 3, calculate the mean vector of each category.

Specifically, for any class (such as class i), its mean vector is calculated by the following formula:

Among them, i is the class identifier, μ _i is the mean vector of the i-th class, b _i is the number of samples of the i-th class, j is the sample ID, and x _ij is a vector composed of the expression features of the j-th sample of the i-th class.

Step 4, according to the mean vectors of each category, determine the total mean vector.

Specifically, the total mean vector is determined by the following formula:

Among them, μ ₀ is the total mean vector, and E is the total number of different categories of samples in X.

Step 5, calculate the between-class variance vector and the intra-class variance vector according to the total mean vector.

The specific calculation formula is as follows:

Among them, S _w is the between-class variance vector, S _b is the intra-class variance vector, and Xi is the set of _i -th class samples.

Step 6: Determine the dimensionality-reduced expression features according to the inter-class variance vector and the intra-class variance vector to form a dimensionality reduction model.

The specific calculation method is as follows:

1) Calculate the weight vector W=diag(S _b ·/S _w ) composed of the weights of each expression feature.

Wherein, diag() is a function, which is used to obtain elements on the diagonal of the matrix, and ·/ is an operator, which is used to divide corresponding elements of S _w and S _b .

2) According to the descending order of the weight of each expression feature, the expression features are sorted.

3) Determining a preset number of top-ranked expression features as dimensionality-reduced expression features.

The expression features after dimensionality reduction can form a feature subset F. The larger the weight, the more suitable the feature component corresponding to the weight is for micro-expression classification.

The output is a subset of features, forming a dimensionality reduction model.

Fig. 1 shows a schematic diagram of the principle of a dimensionality reduction model divided into two categories.

The implementation method of the actual micro-expression recognition process based on the trained dimensionality reduction model is shown in Figure 2:

S101. Acquire a face video.

Because the duration of the micro-expression is short and the range of motion is very small, therefore, as long as each frame of the human face video image in this step includes a human face, it is not necessary to accurately correspond to the video of the micro-expression.

S102, perform face recognition on each frame in the video, and extract a face area.

This embodiment does not limit the extraction method of the face area, and any existing extraction method may be used.

S103, extracting the number of pixels, background color, and face brightness of each frame in the video.

The pixel values of video files obtained by video acquisition devices with different configurations are different, and the pixel numbers of the previous frame and the next frame are different, which will affect micro-expression recognition. Therefore, this proposal will extract the pixels of each frame in the video number.

The number of pixels can be expressed by a single number, such as a "0.3 megapixel" digital camera, which has a rated 300,000 pixels; it can also be expressed by a pair of numbers, such as "640*480 display", which means 640 pixels horizontally and 480 pixels vertically (such as a VGA monitor). And a pair of numbers can also be converted into a number, such as the pixels of a 640*480 display are 640*480=307200 pixels.

The number of pixels in each frame in this step is the total number of pixels in the frame, which can be calculated through the resolution of the image. For example, the image resolution of one frame is 1280*960, and the number of pixels of this frame=1280*960=1228800.

This embodiment does not limit the method for extracting the number of pixels, and any existing extraction method may be used.

The implementation methods for extracting the background color of each frame in the video include but are not limited to:

For any frame in the video,

Step 1.1, determine the non-face area in any frame as the background area.

Step 1.2, determine the RGB color value of each pixel in the background area of any frame.

Wherein, the RGB color value includes a red color value, a green color value, and a blue color value.

In step 1.3, the RGB color mean value of any frame background area is calculated by the following formula.

Among them, the RGB color mean value includes the red color mean value, the green color mean value, and the blue color mean value:

为任一帧背景区域的红色颜色均值，

为任一帧背景区域的绿色颜色均值，

为任一帧背景区域的蓝色颜色均值，c_1j为任一帧背景区域第j像素点的红色颜色值，c_2j为任一帧背景区域第j像素点的绿色颜色值，c_3j为任一帧背景区域第j像素点的蓝色颜色值，n₁为任一帧背景区域中像素点总数量。j is the pixel identification of the background area of any frame,

is the mean value of the red color of the background area of any frame,

is the mean value of the green color of the background area of any frame,

is the mean value of the blue color in the background area of any frame, c _1j is the red color value of the jth pixel in the background area of any frame, c _2j is the green color value of the jth pixel in the background area of any frame, and c _3j is any The blue color value of the jth pixel in the background area of a frame, n ₁ is the total number of pixels in the background area of any frame.

Step 1.4, calculating the RGB color mean square deviation of the background area of any frame.

Among them, RGB color mean square deviation includes red color mean square deviation, green color mean square deviation, blue color mean square deviation:

σ ₁₁ is the mean square deviation of the red color, σ ₂₁ is the mean square deviation of the green color, and σ ₃₁ is the mean square deviation of the blue color.

Step 1.5, determine the RGB color range of the background area of any frame.

绿色颜色区间

蓝色颜色区间

Among them, the RGB color interval includes the red color interval

green color range

blue color range

Step 1.6, in all pixels in the background area of any frame, determine that the red color value in the RGB color value is in the red color range in the RGB color range, and the green color value is in the green color range, and the blue color value is in the blue color range The number n ₂ of pixels in the color interval.

Step 1.7, determine the background color of any frame according to n ₂ .

Wherein, the background color is represented by an RGB color value, and the RGB color value includes a red color value, a green color value and a blue color value.

绿色颜色值为

蓝色颜色值为

Specifically, calculate the pixel number ratio of any frame n ₃ =n ₂ /n ₁ ; the red color value of the background color of any frame is

The green color value is

The blue color value is

The background color extraction method provided in this embodiment does not simply use the mean value of each color channel in the RGB color values of each pixel in the background as the background color, but dynamically The average value is adjusted, and the adjusted value is used as the background color, so that the determination of the background color is more in line with the actual situation.

The implementation scheme for extracting the face brightness of each frame in the video includes but is not limited to:

For any frame in the video,

In step 2.1, the brightness value of each pixel in the face area of any frame is determined by the following formula.

Wherein, k is the pixel point identification of any frame of human face area, h _k is the brightness value of the kth pixel point of any frame of human face area, R _k is the red color value in the RGB color value of the kth pixel point, G _k is the green color value in the RGB color value of the kth pixel point, and B _k is the blue color value in the RGB color value of the kth pixel point.

Step 2.2, determine the maximum brightness value and the minimum brightness value among the brightness values of all pixels in the face area of any frame.

Step 2.3, calculate the average brightness of any frame face area

Among them, n ₄ is the total number of pixels in the face area of any frame.

确定视频中任一帧的人脸亮度。Step 2.4, according to the maximum brightness value, minimum brightness value and

Determines the brightness of a face at any frame in a video.

specific,

1) Calculate the first difference d1=maximum luminance value−minimum luminance value.

2) Calculate the second difference

3) Calculate the third difference

4) Calculate the brightness ratio d4=|d1-d2|/|d1-d3|.

5) Calculate the brightness mean square error of any frame face area

6) The face brightness of any frame in the video is

The face brightness extraction method provided in this embodiment does not simply use the average value of the brightness of each pixel in the face area as the brightness of the face, but dynamically compares the average value according to the difference between the brightness of each pixel and the maximum brightness and the minimum brightness Adjustment is performed, and the adjusted value is used as the brightness of the face, so that the determination of the brightness of the face is more in line with the actual situation.

S104. Select a non-first frame in sequence, and calculate the face area difference, pixel number difference, background color difference, and face brightness difference between the selected frame and the previous frame.

From the second frame to the end of the last frame, select a frame in turn, determine the difference between the face area area of the selected frame and the previous frame of the frame as the face area difference, and determine the difference in the number of pixels as pixels The difference between the numbers and the background color is determined as the background color difference, and the difference in the brightness of the face is determined as the brightness difference of the face.

For example, the area difference of the face area=the area of the face area of the selected frame−the area of the face area of the previous frame. Pixel number difference = pixel number of the selected frame - pixel number of the previous frame. Background color difference = background color of the selected frame - background color of the previous frame. Face brightness difference = face brightness of the selected frame - face brightness of the previous frame.

S105. Calculate the difference value of each non-first frame.

Wherein, the difference value=(area difference of the face area*the brightness difference of the face+the background color difference)^the difference in the number of pixels.

^ is the power operator.

In addition, in order to improve the execution speed of the solution provided by this embodiment, before calculating the difference value of each non-first frame, a preliminary screening can be performed on the non-first frames to propose frames that are obviously not the same person and obviously do not belong to micro-expressions.

That is, the specific execution process of S105 is: perform preliminary screening on the non-first frames according to the face area difference, pixel number difference, background color difference, and face brightness difference of each non-first frame, and calculate the difference value of each frame after the preliminary screening.

According to the face area difference, pixel number difference, background color difference, and face brightness difference of each non-first frame, the scheme for preliminary screening of non-first frames includes but is not limited to:

For any non-first frame, if the face area difference of any non-first frame is not greater than the first value, and the pixel number difference is not greater than the second value, and the background color difference is not greater than the third value, and the human If the face brightness difference is not greater than the fourth value, then any non-first frame passes the preliminary screening; or, if the face area difference of any non-first frame is not greater than the first value, but the number of pixels is poor, the background color is poor, the face If the brightness difference is 0, any non-first frame passes the preliminary screening; or, if the face brightness difference of any non-first frame is not greater than the fourth value, but the face area difference, pixel number difference, and background color difference are all If it is 0, any non-first frame passes the preliminary screening.

Wherein, the first value is (the total number of frames of the human face video of (the sum of the face area difference of all non-first frames+the face area of the first frame-avg1)/described human face video, and the second value is (all non-first frames The sum of the difference in the number of pixels+the number of pixels of the first frame-avg2)/the total number of frames of the face video, and the third value is (the sum of the background color differences of all non-first frames+the background color of the first frame-avg3) /The total number of frames of the human face video, the fourth value is (the sum of the difference in brightness of the human face of all non-first frames+the brightness of the human face of the first frame-avg4)/the total number of frames of the human face video, avg1= The sum of the face area areas of all frames/the total number of frames of the human face video, avg2=the sum of the pixels of all frames/the total number of frames of the described human face video, avg3=the sum of the background colors of all frames/ The total number of frames of the human face video, avg4=the sum of the brightness of the human face of all frames/the total number of frames of the human face video.

S106. Determine the frame whose difference value is greater than the preset threshold and the first frame of the video as candidate frames.

The preset threshold guarantees the size of the difference. Different micro-expressions have different difference ranges. Adaptive selection is carried out through the preset threshold according to the different application fields of the method, thereby ensuring the versatility of the expression recognition method provided in this application.

S107, among the candidate frames, determine frames with continuous identifiers as micro-expression frames.

For example, if the candidate frames are frame 3, frame 5, frame 6, frame 8, and frame 9, the consecutive frames (frame 5, frame 6, frame 8, and frame 9) are all determined as micro-expression frames.

At this time, it is likely that frames 5 and 6 represent a micro-expression, and frames 8 and 9 represent a micro-expression.

The above are examples only and do not represent the actual situation.

The present application does not limit "continuous", as long as it is not a single frame. For example, if there are 2 frames with continuous identifiers, both frames with consecutive identifiers are determined as micro-expression frames. For another example, if there are 3 frames with consecutive identifiers, all the 3 frames with consecutive identifiers are determined as micro-expression frames.

S108, extracting the expression features of the micro-expression frame, performing dimensionality reduction processing on the expression features through a pre-trained dimensionality reduction model, identifying the dimensionality-reduced features, and obtaining a recognition result.

In this step, the micro-expression recognition model can be pre-trained, and then the expression features of the micro-expression frame can be extracted. After the dimensionality reduction processing is performed on the expression features through the pre-trained dimensionality reduction model, the micro-expression recognition model can be used to identify the features after dimensionality reduction. Get the recognition result.

Among them, the training process of the micro-expression recognition model includes but is not limited to:

Step 3.1, obtaining a plurality of sample videos.

Sample videos can be obtained from existing microexpression datasets.

Microexpressions are tiny facial movements that people make when they try to hide their emotions. Strictly speaking, the tiny expressions that people simulate subjectively cannot be called micro-expressions, so the way micro-expressions are induced determines the reliability of the data.

In this step, multiple sample videos can be obtained from one or two of the following two existing micro-expression datasets:

The micro-expression data set SMIC, established by the University of Oulu in Finland, requires the subjects to watch videos with large emotional fluctuations and try to hide their emotions from being exposed. The recorder observes the subjects' emotions without watching the video. expression. If the recorder observed the subject's facial expression, the subject would be punished. Under this induction mechanism, 164 video sequences of 16 people were formed. There are 3 types of micro-expressions, namely positive, surprise, and negative. The number of video sequences is 70, 51, and 43, respectively. .

The micro-expression data set CASME2, established by the Institute of Psychology of the Chinese Academy of Sciences, uses a similar induction mechanism to ensure the reliability of the data, but if the subject successfully suppresses his facial expressions and is not discovered by the recorder, he will get a corresponding award. The data set is composed of 247 video sequences of 26 people, including 5 micro-expression categories, namely happiness, disgust, surprise, repression, and other. Each category corresponds to The number of video sequences are 32, 64, 25, 27, 99, respectively.

Step 3.2. For each sample video, use the local binary mode to extract the corresponding expression features.

The local binary pattern (Local Binary Pattern, LBP) descriptor is defined on the central pixel and its surrounding rectangular neighborhood, as shown in Figure 3, the gray value of the central pixel is the threshold value, and the binary quantization of the neighboring pixels around the central pixel Field pixels that are greater than or equal to the center pixel value are coded as 1, and those less than are coded as 0, and form a local binary pattern.

The binary pattern is concatenated clockwise starting from the upper left corner to obtain a series of binary numbers, and the corresponding decimal numbers can uniquely identify the central pixel. In this way, each pixel in the image can be computed using a local binary pattern.

As shown in Figure 3, the center pixel value in the table on the left is 178, and the value in the upper left corner is 65, 65<178, so the corresponding value is 0, and 188>178, so the corresponding value is 1. By analogy, the table on the right of Figure 3 is obtained, and the binary mode value is 01000100.

In addition, the LBP static texture descriptor can also be extended in the space-time domain to form a 2-dimensional local binary pattern on three orthogonal planes. As shown in Figure 4, the LBP features of the three orthogonal plane video sequences of XY, XT and YT are extracted respectively, and the feature vectors on each orthogonal plane are concatenated to form the LBP-TOP feature vector. This method not only considers the local texture information of the image, but also describes the situation of the video changing with time.

However, the vector dimension of LBP-TOP is 3×2 ^L , where L is the number of domain points. If the expression features extracted in step 3.2 are directly used for modeling, the model training speed will be slow due to the large feature dimension, and the effect will not be good. Therefore, after performing step 3.2 to extract expression features, the present application will perform step 3.3 to reduce the dimension of the actual training model to consider the expression features and improve the efficiency of model training.

Step 3.3: Carry out recognition training on each sample video to form a micro-expression recognition model.

There can be multiple training methods in this step, and the present embodiment provides the following training methods:

3.3.1. Use any clustering algorithm (such as the k-means algorithm) to cluster each sample video based on expression features to form the micro-expression class to which each sample video belongs.

3.3.2. Adjust the parameters in the clustering algorithm according to the second standard classification results of each sample video.

Since each sample video itself has a label identifying its micro-expression category, the label is obtained in this step and used as the second standard classification result of each sample video.

3.3.3. Repeat 3.3.1 and 3.3.2 to complete the training and form a micro-expression recognition model.

The micro-expression recognition model in this application is a classifier.

For example: using a Support Vector Machine (Support Vector Machine, SVM) method. The key to SVM is the kernel function, and different kernel functions will have different SVM classification effects.

For example, the following kernel functions may be used: linear kernel (Linear Kernel), chi-square kernel (Chi-square Kernel), and histogram intersection kernel (Histogram Intersection kernel).

In addition, in order to improve the classification recognition rate of the final trained classification model, cross validation (Cross Validation) can also be used to test the performance of the micro-expression recognition model. Specifically, all the sample videos are divided into two subsets, one subset is used to train the classifier called the training set, and the other subset is used to verify and analyze the validity of the classifier called the test set. Use the test set to test the trained classifier as the performance index of the classifier. Commonly used methods are simple cross-validation, K-fold cross-validation and leave-one-out cross-validation.

Take the micro-expression classification training of SVM classifiers with different kernel functions by using the leave-one-out cross-validation method as an example. Each time select all the video sequences of one subject as the test samples, and all the video sequences of the other I subjects as the training samples, repeat the experiment for I+1 times, and calculate the average classification recognition rate of I+1 times.

Based on this, the training of a micro-expression recognition model is completed.

After the micro-expression recognition model is trained, the dimensionality reduction process is performed on the expression features through the pre-trained dimensionality reduction model, and the micro-expression recognition model is used to recognize the reduced-dimensionality features, and the recognition result is obtained.

Since the expression features are dimensionally reduced before the micro-expression recognition model is used for micro-expression recognition, the recognition rate and recognition accuracy of the micro-expression recognition model can be improved.

It should be noted that "first", "second", and "third" in this embodiment and subsequent embodiments are only used to distinguish preset thresholds, classification results, standard classification results, etc. in different steps, not have any other special meaning.

Beneficial effect:

Select micro-expression frames for recognition based on face area difference, pixel number difference, background color difference, and face brightness difference, which can accurately extract frames related to micro-expressions in face videos, and improve the recognition efficiency and accuracy of micro-expression frames sex.

Claims (10)

1. a micro-expression recognition method, is characterized in that, described method comprises: Get face video; Carry out face recognition to each frame in the video, and extract the face area; Extract the number of pixels, background color, and face brightness of each frame in the video; Select a non-first frame in turn, and calculate the face area difference, pixel number difference, background color difference, and face brightness difference between the selected frame and the previous frame; Calculate the difference value of each non-first frame, wherein, difference value=(face area difference*face brightness difference+background color difference)^pixel number difference; Determining a frame whose difference value is greater than a preset threshold, and the first frame of the video as a candidate frame; In the candidate frame, the frames with continuous marks are all determined as micro-expression frames; The expression features of micro-expression frames are extracted, and the dimensionality reduction processing is performed on the expression features through the pre-trained dimensionality reduction model, and the features after dimensionality reduction are recognized to obtain the recognition results. 2. The method according to claim 1, wherein extracting the background color of each frame in the video comprises: For any frame in the video, Determining the non-face area in any frame as the background area; Determine the RGB color value of each pixel in the background area of any frame, and the RGB color value includes a red color value, a green color value, and a blue color value; The RGB color mean value of the background area of any frame is calculated by the following formula, and the RGB color mean value includes a red color mean value, a green color mean value, and a blue color mean value:

Wherein j is the pixel point identification of the background area of any frame,

is the mean value of the red color of the background area of any frame,

is the mean value of the green color of the background area of any frame,

For the blue color mean value of the background area of any frame, _c1j is the red color value of the jth pixel in the background area of any frame, and _c2j is the green color of the jth pixel in the background area of any frame value, c _3j is the blue color value of the jth pixel in the background area of any frame, and n ₁ is the total number of pixels in the background area of any frame; Calculate the RGB color mean square error of any frame background area, the RGB color mean square error includes red color mean square error, green color mean square error, blue color mean square error:

Among them, σ ₁₁ is the mean square deviation of the red color, σ ₂₁ is the mean square deviation of the green color, and σ ₃₁ is the mean square deviation of the blue color; Determine the RGB color interval of the background area of any frame, and the RGB color interval includes the red color interval

green color range

blue color range

In all pixels in the background area of any frame, determine that the red color value in the RGB color value is located in the red color interval in the RGB color interval, and the green color value is located in the green color interval, and the blue color value is located in the blue color interval The number of pixels in the color interval n2; Determine the background color of any frame according to n2. 3. The method according to claim 2, wherein the background color is represented by an RGB color value; The determining the background color of any frame according to n2 includes: Calculating the pixel number ratio n ₃ =n ₂ /n ₁ of any frame; The background color of either frame has a red color value of

The green color value is

The blue color value is

4. The method according to claim 1, wherein extracting the face brightness of each frame in the video comprises: For any frame in the video, Determine the brightness value of each pixel in the face area of any frame by the following formula:

Wherein, k is the pixel point identification of described any frame human face area, h _k is the luminance value of the kth pixel point of described any frame human face area, R _k is the RGB color value in the kth pixel point Red color value, G _k is the green color value in the RGB color value of the k pixel point, B _k is the blue color value in the RGB color value of the k pixel point; Among the brightness values of all pixels in the human face area of any frame, determine a maximum brightness value and a minimum brightness value; Calculate the mean value of the brightness of the face area of any frame

Wherein, _n4 is the total number of pixels in the face area of any frame; According to the maximum brightness value, minimum brightness value and

Determine the brightness of a face at any frame in the video. 5. The method according to claim 4, characterized in that, according to the maximum brightness value, the minimum brightness value and

Determine the brightness of a face in any frame of the video, including: Calculate the first difference d1=maximum brightness value-minimum brightness value; Calculate the second difference

Calculate the third difference

Calculate brightness ratio d4=|d1-d2|/|d1-d3|; Calculate the mean square error of the brightness of the face area of any frame

The face brightness of any frame in the video is

6. The method according to claim 1, wherein, before calculating the difference value of each non-first frame, further comprising: According to the face area difference, pixel number difference, background color difference, and face brightness difference of each non-first frame, the non-first frame is initially screened; The calculation of the difference value of each non-first frame includes: Calculate the difference value of each frame after the primary screening. 7. The method according to claim 6, characterized in that, the non-first frame is initially screened according to the area difference of the face area, the number of pixels difference, the background color difference, and the brightness difference of the face of each non-first frame, including : For any non-first frame, If the face area difference of any non-first frame is not greater than the first value, and the pixel number difference is not greater than the second value, and the background color difference is not greater than the third value, and the face brightness difference is not greater than fourth value, then any non-first frame passes the preliminary screening; or, If the face area difference of any non-first frame is not greater than the first value, but the pixel number difference, background color difference, and face brightness difference are all 0, then any non-first frame passes the preliminary screening; or , If the face brightness difference of any non-first frame is not greater than the fourth value, but the face area difference, pixel count difference, and background color difference are all 0, then any non-first frame passes the preliminary screening; Wherein, the first value is (the total number of frames of the human face video of (the sum of the face area difference of all non-first frames+the face area of the first frame-avg1)/described human face video, and the second value is (all non-first frames The sum of the difference in the number of pixels+the number of pixels of the first frame-avg2)/the total number of frames of the face video, and the third value is (the sum of the background color differences of all non-first frames+the background color of the first frame-avg3) /The total number of frames of the human face video, the fourth value is (the sum of the difference in brightness of the human face of all non-first frames+the brightness of the human face of the first frame-avg4)/the total number of frames of the human face video, avg1= The sum of the face area areas of all frames/the total number of frames of the human face video, avg2=the sum of the pixels of all frames/the total number of frames of the described human face video, avg3=the sum of the background colors of all frames/ The total number of frames of the human face video, avg4=the sum of the brightness of the human face of all frames/the total number of frames of the human face video. 8. The method according to any one of claims 1 to 7, characterized in that, before the dimension reduction processing of the expression features by the pre-trained dimension reduction model, further comprising: Obtain a sample set X, where the total number of samples in X is m, each sample includes multiple expression features, and each sample belongs to a category; Classify all samples by category; Calculate the mean vector of each class

Wherein, i is the class identification, μ _i is the mean value vector of the i-th class, b _i is the number of samples of the i-th class, j is the sample identification, and x _ij is a vector composed of the expression features of the j-th sample of the i-th class; According to the mean vector of each category, determine the total mean vector

Among them, μ ₀ is the total mean value vector, and E is the total number of different categories of samples in X; Calculate the between-class variance vector and the intra-class variance vector according to the total mean vector; The dimensionality-reduced expression features are determined according to the inter-class variance vector and the intra-class variance vector to form a dimensionality reduction model. 9. method according to claim 8, is characterized in that, described according to total mean value vector, calculates interclass variance vector and intraclass variance vector, comprising:

Among them, S _w is the between-class variance vector, S _b is the intra-class variance vector, and Xi is the set of _i -th class samples. 10. The method according to claim 9, wherein said determining the dimensionality-reduced expression feature according to said inter-class variance vector and intra-class variance vector comprises: Calculate the weight vector W=diag(S _b /S _w ) composed of each expression feature weight, wherein, diag () is a function, and the function is used to get the elements on the matrix diagonal, / is an operator, said operator is used to _divide corresponding elements of _Sw and Sb; According to the order of the weight of each expression feature from large to small, the expression features are sorted; A preset number of top-ranked expression features are determined as expression features after dimensionality reduction.

Images