KR102128158B1 - Emotion recognition apparatus and method based on spatiotemporal attention - Google Patents

Abstract

Disclosure of emotion recognition device and method. The apparatus and method for recognizing emotions of the present invention acquire 3D features from an image sequence including a plurality of frames, and at the same time extract space-time features for each frame, obtain them with space-time weights, and weight space-time weights on the 3D features. , Even if a separate region of interest is not set, accurate emotions can be determined.

Description

Apparatus and method for emotion recognition based on spatiotemporal attention {EMOTION RECOGNITION APPARATUS AND METHOD BASED ON SPATIOTEMPORAL ATTENTION}

The present invention relates to an apparatus and method for recognizing emotions, and more particularly to an apparatus and method for recognizing emotions based on space-time attention.

Emotion recognition is an important issue in interactive systems. The interactive system determines the user's requirements through interaction with the user instead of the conventional command input method. At this time, when the emotion recognition technology is applied, there is an advantage that the user's requirements can be more accurately determined.

In addition, emotion recognition can be applied to medical fields such as pain and psychological pain detection, and can be applied to various other fields.

Most of the existing studies on emotion recognition recognize categorical emotions that classify emotions into discrete categories according to the basic emotions designated by a predetermined number (for example, six) such as fear, anger, happiness, disgust, sadness, and surprise. Most of the way. However, as categorical emotion recognition is classified and recognized only as a specified emotion, there is a limitation in that not only an area of emotion that is not classified exists, but also a type of recognizable emotion is limited.

1 shows an example of an image representing a person's emotions.

1 is (a) to (d) are expression images of four types of surprises defined by Ekman, (a) is a surprising question (questioning surprise), (b) is a surprise surprise (astonished surprise), (c) ) Represents a dazed surprise, and (d) represents a full surprise.

As illustrated in FIG. 1, various surprises may exist in surprises, but all existing categorical emotion recognition is classified only as surprises, and there is a limitation that subtle emotion differences cannot be recognized. Accordingly, a method of expressing emotion in two dimensions according to two consecutive domains has been proposed.

Fig. 2 shows an example of emotion represented by a continuous two-dimensional graph.

In FIG. 2, each axis of the 2D represents arousal and valence, the arousal axis represents a level of being active or inactive, and the axis of arousal represents a positive or negative level. As shown in FIG. 2, the method of describing emotions in two-dimensional series can express more complex and subtle emotions than the conventional categorical type.

Meanwhile, recently, a neural network is applied to the emotion recognition technique to improve the accuracy of emotion recognition. However, most of the existing emotion recognition techniques have been conducted to recognize emotions from a single image, so there is a lack of research on how to accurately recognize human emotions from image sequences over time. Since the emotions of a real person are gradually and continuously changed with the passage of time, emotions can be recognized more accurately than a single image when emotions are recognized using a sequence of images.

In addition, as shown in FIG. 1, in the conventional emotion recognition, a region of interest in which emotion expression is expected to appear strongly in a human face image is previously designated, and analysis is performed on the designated region of interest. However, there is a limitation in that emotions cannot be recognized with optimal performance for various facial images as the facial expressions are estimated by activating only a region of interest and the emotions are recognized.

Korean Patent Publication No. 10-2013-0015958 (published on February 14, 2013)

An object of the present invention is to provide an apparatus and method for recognizing emotion based on temporal and spatial attention from a face image sequence.

Another object of the present invention is to provide an apparatus and method for recognizing emotions capable of recognizing emotions without specifying a region of interest in a face image sequence.

Another object of the present invention is to provide a device and a method for recognizing emotions by extracting 2D and 3D features from a face image sequence, respectively, and accurately recognizing emotions using the extracted 2D and 3D features.

In order to achieve the above object, the emotion recognition apparatus according to an embodiment of the present invention is pre-learned according to a predetermined 3D pattern recognition technique, and 3D temporally continuous T frames (where T is a natural number) of an image sequence A 3D feature extraction unit that extracts a 3D feature by recognizing a pattern as a single image of the; Spatio-temporal, which is pre-trained according to a known 2D pattern recognition technique, extracts T spatial features through pattern recognition from each of the T frames, and adds spatio-temporal features between the obtained T spatial features to obtain spatio-temporal weights Feature extraction unit; The emotion value extraction unit extracts an emotion value having a value within a predetermined range from the emotion feature by learning the emotion value by weighting the space-time weight to the 3D feature and learning in advance according to a known 3D pattern recognition technique. ; And an emotion discrimination unit that stores emotions prepared for the emotion values in advance, and determines emotions corresponding to the emotion values obtained from the emotion value extraction unit.

The 3D feature extraction unit may extract 3D features, including 3D Cvo (3D Convolutional Neural Networks) previously learned.

The space-time feature extraction unit includes a pre-trained 2D CNN (2D Convolutional Neural Networks), a spatial encoder for extracting the T spatial features for each of the T frames; A temporal decoder for extracting the spatio-temporal feature between the T spatial features, including a pre-trained convolutional long short-term memory (ConvLSTM); And a normalization unit that normalizes the spatiotemporal features extracted from the time decoder in a predetermined manner to obtain the spatiotemporal weights.

The spatial encoder includes the convolutional layer, the rectified linear unit (ReLU) layer, and the Max-Pooling layer, each of which includes a plurality of filters, and the spatial resolution of the spatial feature of the frame. It can be reduced to lower.

In the temporal decoder, the ConvLSTM includes a plurality of ConvLSTM layers to perform sequential deconvolution to recover the reduced spatial resolution of the spatial feature.

The normalizer may normalize the spatiotemporal features using a soft max function.

The emotion value extracting unit combines the 3D feature and the space-time weight by Hadamard to obtain the emotion feature; And an emotion value acquiring unit for extracting an emotion value representing emotion from the emotion feature, including a previously learned 3D CNN.

The emotion recognition method according to another example of the present invention for achieving the above object is pre-learned according to a known three-dimensional pattern recognition technique, and three-dimensionally sequential T consecutive frames of the image sequence (where T is a natural number) Extracting 3D features by recognizing a pattern as a single image; Pre-learning according to a known two-dimensional pattern recognition technique, extracting T spatial features through pattern recognition from each of the T frames, and adding space-time features between the obtained T spatial features to obtain space-time weights ; Obtaining emotional features by weighting the space-time weights with the 3D features; Learning in advance according to a predetermined 3D pattern recognition technique, and extracting an emotion value having a value within a predetermined range from the emotion feature; And determining an emotion corresponding to the emotion value.

Accordingly, the apparatus and method for recognizing emotions of the present invention can acquire 2D and 3D features from an image sequence, respectively, and accurately recognize emotions using the obtained 2D and 3D features together. In addition, not only can emotions be recognized based on temporal and spatial attention, but emotions can be accurately recognized based on successive accreditation without specifying a region for recognizing emotions.

1 shows an example of an image representing a person's emotions.
Fig. 2 shows an example of emotion represented by a continuous two-dimensional graph.
3 shows a schematic configuration of an emotion recognition device according to an embodiment of the present invention.
4 shows an example of a detailed configuration of the space-time feature extraction unit of FIG. 3.
5 is a diagram for explaining a learning method of the emotion recognition device of FIG. 3.
6 shows an emotion recognition method according to an embodiment of the present invention.
7 is a diagram visualizing the spatiotemporal weight of the present embodiment.
8 and 9 show the results of comparing emotion values and verification values obtained by applying the emotion recognition method according to the present embodiment to the two types of RECOLA data sets and AV + EC data sets, respectively.

In order to fully understand the present invention, the operational advantages of the present invention, and the objects achieved by the practice of the present invention, reference should be made to the accompanying drawings and the contents described in the accompanying drawings, which illustrate preferred embodiments of the present invention.

Hereinafter, the present invention will be described in detail by explaining preferred embodiments of the present invention with reference to the accompanying drawings. However, the present invention may be implemented in various different forms, and is not limited to the described embodiments. And, in order to clearly describe the present invention, parts irrelevant to the description are omitted, and the same reference numerals in the drawings indicate the same members.

Throughout the specification, when a part “includes” a certain component, it means that the component may further include other components, not to exclude other components, unless otherwise stated. In addition, terms such as "... unit", "... group", "module", and "block" described in the specification mean a unit that processes at least one function or operation, which is hardware or software or hardware. And software.

3 shows a schematic configuration of an emotion recognition apparatus according to an embodiment of the present invention, and FIG. 4 shows an example of a detailed configuration of the spatiotemporal feature extraction unit of FIG. 3.

Referring to FIG. 3, in the emotion recognition apparatus according to the present embodiment, an emotion value is obtained from an image acquisition unit 100 that acquires an image sequence including an object to recognize emotion, and an image sequence transmitted from the image acquisition unit 100. It includes an emotion extraction unit 200 for extracting and the emotion determination unit 300 for determining the emotion of the object included in the image according to the extracted emotion value.

First, the image acquisition unit 100 acquires an image sequence including an object to recognize emotions. In particular, in the present embodiment, the image acquisition unit 100 is not a single image, but a sequence of T (where T is a natural number) frames I _f (where f is a natural number as a frame index) image sequence (I _{1: T} = (I ₁ , I ₂ , …, I _T }). Here, each frame of the image sequence (I _{1: T} ) includes the face of the object so that emotions can be recognized.

In addition, the image acquisition unit 100 transmits the acquired image sequence to the emotion extraction unit 200. At this time, when the number of frames included in the acquired image sequence exceeds T, the image acquisition unit 100 extracts emotions by separating T frames including frames at the time of recognizing the emotion of the object in the image sequence It can be delivered to the unit 200. For example, the image acquisition unit 100 separately separates the 11th to 20th frames (I ₁₁ to I ₂₀ ) from the image sequence including 100 frames (assuming T is 10), thereby extracting the emotion 200 ).

In addition, when it is desired to recognize a continuous emotion change of an object from an image sequence, T frames may be sequentially separated from the image sequence and transmitted to the emotion extraction unit 200. As an example, the first to tenth frames I ₁ to I ₁₀ may be transmitted, and then the second to eleventh frames I ₂ to I ₁₁ may be transmitted.

The emotion extraction unit 200 extracts the emotion value from the image sequence (I _{1: T} ) transmitted from the image acquisition unit 100. In particular, in the present embodiment, the emotion extracting unit 200 uses an image sequence (I _{1: T} ) using a two-dimensional (2D) and three-dimensional (3D) pattern recognition technique previously learned about the image sequence (I _{1: T} ). Features are extracted, and emotion values are extracted by combining the extracted features.

As illustrated in FIG. 3, the emotion extraction unit 200 may include a 3D feature extraction unit 210, a spatiotemporal feature extraction unit 220, a feature combination unit 230, and an emotion value acquisition unit 240. .

The 3D feature extraction unit 210 is a frame ({I ₁ , I ₂ , …, I _T }) of a two-dimensional image sequence (I _{1: T} ) transmitted from the image acquisition unit 100 to determine the emotion of the object ) 3D features (X' _1:T ) are extracted by recognizing the entire pattern as a single 3D object. That is, an image sequence (I _1:T ) including a plurality of 2D frames accumulated over time is recognized as a 3D image, and the 3D image sequence (I _1:T ) is used for a predetermined pattern recognition technique. 3D features (X' _1:T ) are extracted by analysis accordingly. In this embodiment, the 3D feature extraction unit 210 is used for emotion recognition from an image sequence (I _{1: T} ) including _T frames ({I ₁ , I ₂ , …, I _T }) that are continuous over time. Since 3D features (X' _1:T ) are extracted, relatively accurate features can be extracted compared to a method of extracting 2D features for emotion recognition from a single image. In other words, it is possible to accurately determine the emotion of the object.

The 3D feature extraction unit 210 may be implemented as, for example, 3D Convolutional Neural Networks (hereinafter referred to as 3D CNN) previously learned.

The spatiotemporal feature extraction unit 220 extracts features based on spatiotemporal attention from each of the T two-dimensional frames ({I ₁ , I ₂ , ..., I _T }) transmitted from the image acquisition unit 100. do. In particular, space-time characteristic extracting unit 220 in the embodiment is an image sequence of, even if you do not specify a separate area of interest _{for:: (T} I _{1) T} by extracting space-time care based on features of (I _{1 T),} the image sequence It is possible to obtain a weight according to the importance of each area in the frame ({I ₁ , I ₂ , …, I _T }).

That is, the emotion recognition apparatus according to the present embodiment separately designates regions (eyes and mouths) in which emotions are strongly expressed on a person's face in each frame ({I ₁ , I ₂ , …, I _T }) as shown in FIG. 1. Even if not, the spatiotemporal feature extraction unit 220 extracts the importance of emotion expression for each region of each frame as a feature based on the spatiotemporal attention, and extracts the extracted feature as a 3D feature (X' ₁ ) as a spatiotemporal weight (A _{1: T} ). _:T ) to provide optimal emotion recognition performance.

To this end, the spatiotemporal feature extraction unit 220 may be configured as shown in FIG. 4.

Referring to FIG. 4, the spatiotemporal feature extraction unit 220 includes a spatial encoder 221 for extracting spatial attention-based features, a temporal decoder 223 for extracting spatiotemporal attention-based features, and And a normalizer 225 that normalizes the extracted features to be converted into weights within a specified range.

The spatial encoder 221 extracts and outputs the spatial feature (X _1:T ) for each of the T two-dimensional frames ({I ₁ , I ₂ , …, I _T }) of the image sequence (I _1:T ). . The spatial encoder 221 is pre-learned by a designated two-dimensional pattern recognition technique, and recognizes a spatial pattern of each of the _T frames ({I ₁ , I ₂ , …, I _T }), thereby providing two-dimensional spatial features (X _1:T ).

The spatial encoder 221 may be implemented as, for example, 2D convolutional neural networks (hereinafter referred to as 2D CNN) that have been previously learned. 2D CNN is one of artificial neural networks mainly used to extract features from 2D images.

The spatial encoder 221 sequentially receives T frames ({I ₁ , I ₂ , …, I _T }) from the image acquisition unit 100 and spatial features ({X ₁ , X ₂ , …, X _T } ) May be sequentially output, but in order to reduce time, T frames ({I ₁ , I ₂ , …, I _T }) may be simultaneously applied to extract features. When the spatial encoders 221 are configured in parallel, all spatial encoders are composed of a Siamese network in which weights and bias values are shared equally.

Also, even if T frames ({I ₁ , I ₂ , …, I _T }) are sequentially applied in the course of learning the spatial encoder 221, T frames ({I ₁ , I ₂ , …, I _T } ) Before all of the spatial features ({X ₁ , X ₂ , …, X _T }) are output, the weight and bias values of the spatial encoder 221 must not be changed, and T frames ({I ₁ , I ₂ , …, I _T }) after all of the spatial characteristics ({X ₁ , X ₂ , …, X _T }) are output, the weight and bias values of the spatial encoder 221 may be varied. This is because the emotion recognition apparatus according to the present embodiment processes the image sequence I _{1: T} including _T frames ({I ₁ , I ₂ , …, I _T }) as a unit for emotion recognition.

On the other hand, in this embodiment, the spatial encoder 221 implemented as 2D CNN is, for example, a continuous 3 X 3 convolution layer and a ReLU (Rectified Linear Unit) layer and 2 X 2 stride max pooling (Max-Pooling). ) It may be configured to include a layer. Here, the 3 X 3 convolution layer, the ReLU layer, and the max pooling layer may each include a predetermined number of filters. In one example, the 3 X 3 convolution layer may include 32 filters, the ReLU layer may include 64 filters, and the max pooling layer may be configured to include 128 filters.

The spatial encoder 221 includes a 3 X 3 convolutional layer, a ReLU layer, and a max pooling layer to determine the number of parameters when extracting spatial features (X _{1: T} ) from the image sequence (I _{1: T} ). This is to prevent overfitting problems by reducing.

The temporal decoder 223 extracts the spatio-temporal attention based feature on the spatial feature (X _1:T ) obtained from the spatial encoder 221.

When the spatial encoder 221 is implemented as a 2D CNN, spatial characteristics in each of the _T frames ({I ₁ , I ₂ , …, I _T }) of the image sequence (I _1:T ), that is, region-specific characteristics Can be extracted. However, spatial encoder 221 is the T-frame _{({I 1, I 2,} ..., I T}) with by separately extracting a feature as, T frames ({I _1, temporally consecutive I _2, ..., I _T }) There is a limit that does not reflect the temporal characteristics.

Accordingly, in this embodiment, the temporal decoder 223 extracts the spatiotemporal attention-based feature, so that the temporal feature is further added to the spatial feature (X _1:T ). The temporal decoder 223 is pre-learned by the designated pattern recognition technique, and while maintaining the spatial pattern features included in the spatial features ({X ₁ , X ₂ , ..., X _T }) as much as possible, the spatial features ({X ₁ , X ₂ , …, X _T }), temporal features between spatial features adjacent to each other are additionally extracted.

The time decoder 223 may be embodied as a Convolutional Long Short-Term Memory (ConvLSTM) that is previously learned as an example. ConvLSTM is also an artificial neural network, and it can further reflect spatial features by further improving the Long Short-Term Memory (LSTM), which has been improved so that the Recurrent Neural Network (RNN) can reflect the long term characteristics. To ensure that.

Here, the time decoder 223 uses ConvLSTM rather than LSTM, which can reflect temporal characteristics, so that the spatial characteristics (X _{1: T} ) obtained from the spatial encoder 221 can be maintained as much as possible.

Equation 1 expresses the function performed by the ConvLSTM in the time decoder 223 by the equation. In Equation 1, i _t , f _t , o _t , c _t and h _t are input gates, forget gates, output gates, and activation cells at time t, respectively. And cell output. In addition, σ(·) and tanh(·) denote a sigmoid function and a hyperbolic tangent, respectively, * denotes a convolution operator, and ⊙ denotes a Hadamard multiplication operator. And W _* is a filter matrix connecting different gates, and b _* represents a bias vector corresponding to each gate.

As shown in Equation 1, ConvLSTM has a convolutional structure for both input-to-state and state-to-state transitions, and can not only extract temporal features but also maintain spatial characteristics.

In addition, the temporal decoder 223 gradually enlarges the spatial resolution of the spatial feature (X _{1: T} ) applied through sequential deconvolution. That is, the time decoder 223 extracts features according to time correlation between frames while maintaining the spatial structure of the spatial features (X _{1: T} ).

To this end, the time decoder 223 may include a plurality of ConvLSTM layers, and each of the ConvLSTM layers may include a predetermined number of filters. In FIG. 4, for example, two ConvLSTM layers include 64 and 32 filters, respectively.

The normalizer 225 normalizes the spatiotemporal features output from the time decoder 223 using a spatial softmax function according to Equation (2).

In Equation 2, H _t-1 represents a hidden state, W _i is a weight mapped to the i-th element of the location softmax, and j represents a location.

The spatiotemporal features output from the time decoder 223 by the normalizer 225 are normalized and output as spatiotemporal weights (A _{1: T} ). As an example, the normalizer 225 may normalize such that the sum of space-time weights (A _{1: T} ) is 1.

The feature combining unit 230 combines the 3D feature (X′ _1:T ) and the spatio-temporal weight (A _1:T ) according to Equation 3 to obtain an emotional feature (X″).

In Equation 3, the 3D feature (X' _1:T ) is a feature for determining the emotion of the object, and the space-time weight (A _1:T ) normalized by the normalizer 225 is a 3D feature (X' _1:T). ) As a weight that designates importance for each corresponding area.

The emotion value acquisition unit 240 extracts the 3D feature again for the emotion feature X" to obtain the emotion value y.

The emotion value acquisition unit 240 may be implemented as a 3D CNN that has been previously learned similarly to the 3D feature extraction unit 210 as an example. Also, in this embodiment, the emotion value acquisition unit 240 may extract features so that the emotion value is obtained as a scalar value (y ∈ [-1, 1]) between -1 and 1, but is not limited thereto. Does not work.

The emotion value acquisition unit 240 may also be composed of a plurality of layers in order to acquire an efficient emotion value. For example, the emotion value acquisition unit 240 includes multiple (for example, 4) 3D CNN layers, multiple (for example, 3) 3D Max pooling layers, and multiple (for example, 2) completes It may include a fully-connected layer. In addition, a plurality of 3D CNN layers may include 32, 64, 128, and 256 filters, respectively, as an example.

Meanwhile, the fully connected layer has a single output channel, and an emotion value y may be obtained using a linear regression layer.

The emotion determination unit 300 determines the emotion of the object by substituting the emotion value y obtained in the emotion value acquisition unit 240 into the emotion criteria for each emotion value stored in advance. In the above, since it is assumed that the emotion value y is a scalar value between -1 and 1, the emotion criteria for each emotion value may be set to a continuous range value between -1 and 1 for each emotion value. Therefore, the emotion determining unit 300 can easily determine the emotion corresponding to the applied emotion value y.

In this embodiment, as an example, the emotion value y corresponding to the validity among the two axes of the arousal and the validity of the two-dimensional emotion graph shown in FIG. 2 is extracted. However, this is an example, and in some cases, the emotion extracting unit 200 may be configured to extract emotion values corresponding to arousal, or may be configured to extract emotion values corresponding to both arousal and authorization.

In this embodiment, the spatial encoder 221 and the time decoder 222 of the 3D feature extraction unit 210 and the spatio-temporal feature extraction unit 220, and the emotion value acquisition unit 240 are respectively preset according to a designated deep-learning algorithm. It is a learned artificial neural network.

Then, T frames ({I ₁ , I ₂ , …, I _T }) of the image sequence (I _{1: T} ) applied to the emotion extracting unit 200 and each feature (X _{1: T} , X′ _{1: T)} , X", A _1:T ) can be expressed as a vector matrix.

As a result, the emotion recognition apparatus according to the present embodiment is for determining emotion in three dimensions from an image sequence (I _{1: T} ) including consecutive T frames ({I ₁ , I ₂ , …, I _T }). 3D feature (X _{'1: T)} extract, and at the same time the T-frame _{({I 1, I 2,} ..., I T}) extracts features based on the respective space-time attention to the T-frame ({I _1, I ₂ , …, I _T }) Obtain weights (A _{1: T} ) for each region. Then, by weighting each region's weight (A _1:T ) on the 3D feature (X' _1:T ), the emotion feature (X") is obtained, and the emotion value (y) is extracted from the emotion feature (X") to obtain an image. It is possible to extract and discriminate the emotion of an object very accurately without setting a separate region of interest in the sequence (I _{1: T} ).

5 is a diagram for explaining a learning method of the emotion recognition device of FIG. 3.

Referring to FIGS. 3 and 4, when the learning method of FIG. 5 is described, the image acquisition unit 100 includes T frames ({I ₁ , I _{2) in} an image sequence including a plurality of frames in which emotion values have been determined in advance. , …, I _T }) are sequentially transmitted to the emotion extraction unit 200.

In order to train the emotion extracting unit 200, a plurality of image sequences or a plurality of frames are required, and thus, as an example, an image sequence includes 3500 frames.

In addition, the image acquisition unit 100 sequentially transmits T frames in an image sequence, and at this time, the image acquisition unit 100 transmits the first to tenth frames I ₁ to I _T , and thereafter, The second to eleventh frames (I ₂ to I _T+1 ) may be transmitted.

The 3D feature extraction unit 210 of the emotion extraction unit 200 is a 3D feature (X) for the entire image sequence (I _{1: T} ) including _T frames ({I ₁ , I ₂ , …, I _T }) ' _1:T ). In addition, the spatial encoder 221 of the spatiotemporal feature extraction unit 220, the temporal decoder 223, and the normalizer 225 are spatial features for each of the _T frames ({I ₁ , I ₂ , …, I _T }). (X _{1: T) obtains::,} the weight space-time _(T a ₁₎ by normalizing to again extract the space-time characteristics for the extraction, and the extracted spatial characteristics _(T X _1).

On the other hand, the feature combining unit 230 of the emotion extracting unit 200 weights the spatio-temporal weight (A _1:T ) to the 3D feature (X' _1:T ) to obtain the emotion feature (X"), and the emotion value The acquiring unit 240 extracts the three-dimensional measurement again with respect to the acquired emotional feature (X"), and acquires an emotional value y having a scalar value within a predetermined range (eg, -1 to 1 in this example). .

Here, since the 3D feature extraction unit 210, the spatial encoder 221, the time decoder 223, and the emotion value acquisition unit 240 are all untrained artificial neural networks, the extracted 3D features (X' _1:T ) and spatial Features (X _{1: T} ), spatio-temporal weights (A _{1: T} ), and emotional values (y) are all in a state with significant errors.

The obtained emotion value y is determined in advance in the corresponding frame and compared with the stored emotion value to analyze the error. The graph on the right of FIG. 5 represents the emotion value y obtained for each frame and the previously stored emotion value in an image sequence including 3500 frames for learning. Here, the case where the emotion value is a valence score is shown, the blue line represents the emotion value (y) obtained for each frame, and the red line represents the previously stored emotion value. That is, the difference between the blue line and the red line in a specific frame corresponding to the x-axis is an error.

The emotion extraction unit 200 learns by adjusting the weights and bias vectors of the 3D feature extraction unit 210, the spatial encoder 221, the time decoder 223, and the emotion value acquisition unit 240 to reduce the analyzed error. Order. At this time, the error is propagated from the emotion value acquisition unit 240 to the time decoder 223, the spatial encoder 221, and the 3D feature extraction unit 210 in the reverse order of processing the image sequence (I _{1: T} ), and gradually Learned to reduce errors.

Then, the emotion extraction unit 200 receives T frames from the image acquisition unit 100 again, acquires the emotion value y, and determines an error, thereby repeatedly learning. As a result, the emotion recognition apparatus can be trained by repeatedly acquiring the emotion value y for an image sequence including a plurality of frames and reducing the error of the obtained emotion value y.

6 shows an emotion recognition method according to an embodiment of the present invention.

Referring to FIGS. 3 and 4, the emotion recognition method of FIG. 6, first, the image acquisition unit 100 includes an image sequence (I including _T frames ({I ₁ , I ₂ , …, I _T })) _1:T ) is acquired and transmitted to the emotion extraction unit 200 (S10).

Accordingly, the 3D feature extraction unit 210 of the emotion extraction unit 200 is a 3D feature for the entire image sequence (I _{1: T} ) including _T frames ({I ₁ , I ₂ , …, I _T }) ( X'1 _:T ) is extracted (S20). The 3D feature extraction unit 210 may be implemented as a 3D CNN, for example, and extract 3D features (X' _1:T ) from _T 2D frames ({I ₁ , I ₂ , ..., I _T }). can do.

At the same time, the spatiotemporal feature extraction unit 220 of the emotion extraction unit 200 _first preferentially spatial features (X _{1: T} = {X ₁ , for each of the _T frames ({I ₁ , I ₂ , …, I _T })). X ₂ , …, X _T }) is extracted (S30 ). The spatiotemporal feature extraction unit 220 uses, for example, 2D CNN, spatial features ({X ₁ , X ₂ , …, X _T }) of each of the _T frames ({I ₁ , I ₂ , …, I _T }). ) Can be extracted. At this time, the space-time feature extraction unit 220 may reduce the spatial resolution, including a convolution layer, a ReLU layer, and a max pooling layer with 2D CNN to prevent an over-fitting problem.

Then, in order to add temporal features while maintaining the extracted spatial features (X _{1: T} ) as possible, space-time features are extracted (S40 ). The spatio-temporal feature extraction unit 220 uses ConvLSTM as an example to further extract temporal features while maintaining the spatial features (X _{1: T} ). At this time, the spatiotemporal feature extraction unit 220 may include a plurality of ConvLSTM layers, and sequentially deconvolute to enlarge the reduced spatial resolution again.

Then, the emotion extracting unit 200 normalizes the extracted space-time features in a predetermined manner to obtain space-time weights (A _{1: T} ) (S50).

Then, the feature combining unit 230 of the emotion extracting unit 200 weights the space-time weight (A _1:T ) to the 3D feature (X' _1:T ) to obtain the emotion feature (X") (S60 ). The 3D features (X' _1:T ) are extracted from the _T frames ({I ₁ , I ₂ , ..., I _T }) by weighting the space-time weights (A _1:T ) to the 3D features (X' _{1:T ).} ) May be weighted differently for each space-time area.

This means that even if a separate region of interest is not specified, the emotion extracting unit 200 can determine the importance of each region in each of the _T frames ({I ₁ , I ₂ , …, I _T }).

Then, the emotion value obtaining unit 240 of the emotion extracting unit 200 extracts the 3D feature again for the emotion feature (X") to obtain an emotion value (y) having a scalar value within a predetermined range ( S70).The emotion value acquiring unit 240 may also be embodied as a 3D CNN as an example, and the emotion value y obtained is a value representing emotion of an object included in the image sequence I _1:T . The emotion value acquisition unit 240 may include a 3D CNN layer, a 3D max pooling layer, and a complete connection layer to acquire the emotion values.

The emotion determining unit 300 determines the emotion of the object by substituting and comparing the obtained emotion value y to the emotion for each emotion value stored in advance (S80).

Hereinafter, the performance of the emotion recognition apparatus and method according to the present embodiment will be described in comparison with an existing emotion recognition method.

Here, in order to quantitatively evaluate the performance of the emotion recognition apparatus and method according to the present embodiment, the Root Mean Square Error (RMSE), the Pearson Correlation Coefficient (CC), and the Concordance Correlation Coefficient) (CCC) was used.

The double coincidence correlation coefficient (CCC) measures the correspondence between two variables according to Equation (4).

In Equation 4, ρ is the Pearson correlation coefficient, σ _x ² and σ _y ² are variances of the prediction and measurement values, and μ _x and μ _y represent the average of the prediction and measurement values.

Also, RECOLA data set adopted in 2015/2016 Audio/Visual Emotion recognition Challenges (hereinafter AV + EC) and data set of AV + EC in 2017 were used as data for performance verification.

Table 1 shows the measurement results for the case of using the 2D CNN and the case of using the 3D CNN and the emotion recognition method using the 3D CNN and the spatiotemporal attention (STA) according to the present embodiment.

As shown in the third row of Table 1, the method of recognizing emotion according to the present embodiment reduces the mean square root error (RMSE) by using 3D CNN and space-time attention (STA) together, and the Pearson correlation coefficient (CC) and It can be seen that the coincidence correlation coefficient (CCC) was increased by 0.062 and 0.053, respectively.

7 is a diagram visualizing the spatiotemporal weight of the present embodiment.

FIG. 7 is a view visualizing the spatial and temporal weights (A _{1: T} ) obtained by extracting the spatiotemporal feature by color and the color space by extracting the spatiotemporal feature for the RECOLA data set. In FIG. 7, the red area indicates a high weighted area, and the blue area indicates a low weighted area.

As described above, in the embodiment of the present invention, the spatiotemporal feature extraction unit 220 extracts spatial features for a plurality of frames ({I ₁ , I ₂ , …, I _T }) and maintains the extracted spatial features While extracting more temporal features, it extracts features based on space-time attention. That is, the space-time weight (A _{1: T} ) is obtained. For this reason, the spatiotemporal feature extraction unit 220, as shown in FIG. 7, although a separate region of interest is not designated, each of a plurality of frames ({I ₁ , I ₂ , …, I _T }) as learned. In, it is possible to differentiate the weight of each area for emotion recognition.

It can be seen from FIG. 7 that the spatio-temporal feature extraction unit 220 has determined the area around the eyes and mouth as an important area for estimating emotion.

8 and 9 show the results of comparing emotion values and verification values obtained by applying the emotion recognition method according to the present embodiment to the two types of RECOLA data sets and AV + EC data sets, respectively.

8 and 9, the red line represents the ground truth, and the blue line represents the emotion value y. 8 and 9, it can be confirmed that the emotion value y obtained by the emotion recognition method according to the embodiment of the present invention fluctuates similarly to the verification value.

In Table 2 and Table 3, the emotion recognition results according to the present embodiment for the RECOLA data set and the AV + EC data set were compared with other emotion recognition methods, respectively.

As shown in Tables 2 and 3, the emotion recognition method according to the present embodiment shows the lowest mean square root error (RMSE) compared to other conventional emotion recognition methods, while the Pearson correlation coefficient (CC) and coincidence correlation It can be seen that the coefficient (CCC) is highest. That is, it can be confirmed that the emotion recognition performance is very excellent.

The method according to the present invention may be implemented as a computer program stored in a medium for execution on a computer. The computer readable medium herein can be any available medium that can be accessed by a computer, and can also include any computer storage medium. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, and ROM (readable) Dedicated memory), RAM (random access memory), CD (compact disk)-ROM, DVD (digital video disk)-ROM, magnetic tape, floppy disk, optical data storage, and the like.

Although the present invention has been described with reference to the embodiments shown in the drawings, these are merely exemplary, and those skilled in the art will understand that various modifications and other equivalent embodiments are possible therefrom.

Therefore, the true technical protection scope of the present invention should be defined by the technical spirit of the appended claims.

100: image acquisition unit 200: emotion extraction unit
300: emotion discrimination unit 210: 3D feature extraction unit
220: space-time feature extraction unit 230: feature coupling unit
240: emotion value acquisition unit 221: spatial encoder
223: time decoder 225: normalization

Claims (14)

A 3D feature extraction unit that is pre-learned according to a known 3D pattern recognition technique and extracts 3D features by pattern recognition of T consecutive frames of an image sequence (where T is a natural number) as a single 3D image;
Spatio-temporal, which is pre-trained according to a known 2D pattern recognition technique, extracts T spatial features through pattern recognition from each of the T frames, and adds spatio-temporal features between the obtained T spatial features to obtain spatio-temporal weights Feature extraction unit;
The emotion value extraction unit extracts an emotion value having a value within a predetermined range from the emotion feature by learning the emotion value by weighting the space-time weight to the 3D feature and learning in advance according to a known 3D pattern recognition technique. ; And
An emotion discrimination unit that stores emotions prepared for emotion values in advance, and determines emotions corresponding to the emotion values obtained by the emotion value extraction unit; Emotion recognition device comprising a. The method of claim 1, wherein the 3D feature extraction unit
Emotion recognition device for extracting the 3D features, including pre-trained 3D CNN (3D Convolutional Neural Networks). The method of claim 1, wherein the space-time feature extraction unit
A spatial encoder for extracting the T spatial features for each of the T frames, including pre-trained 2D Convolutional Neural Networks (CNN);
A temporal decoder for extracting the spatio-temporal feature between the T spatial features, including a pre-trained convolutional long short-term memory (ConvLSTM); And
A normalization unit to normalize the spatiotemporal features extracted from the time decoder in a predetermined manner to obtain the spatiotemporal weights; Emotion recognition device comprising a. The method of claim 3, wherein the spatial encoder
The 2D CNN includes a convolutional layer including a plurality of filters, a ReLU (Rectified Linear Unit) layer, and a Max-Pooling layer to reduce the spatial resolution of the spatial feature to be lower than the spatial resolution of the frame. Emotion recognition device. The method of claim 4, wherein the time decoder
The ConvLSTM includes a plurality of ConvLSTM layers, and performs sequential deconvolution to restore the reduced spatial resolution of the spatial feature. The method of claim 5, wherein the normalization unit
An emotion recognition device that normalizes the spatiotemporal features using a soft max function. The method of claim 6, wherein the emotion value extraction unit
A feature combining unit for multiplying the 3D feature and the spatiotemporal weight to obtain the emotional feature; And
An emotion value acquiring unit for extracting emotion values representing emotions from the emotion features, including 3D CNNs learned in advance; Emotion recognition device comprising a. The method of claim 7, wherein the emotion value acquisition unit
An emotion recognition device that acquires the emotion value as a scalar value within a predetermined range. The method of claim 7, wherein the emotional value
An emotion recognition device that indicates the value of the validity in an emotion model that expresses emotion in two dimensions with two axes, Arousal and Valence. According to claim 1, The emotion recognition device
An image acquiring unit for separating and sequentially outputting the T frames consecutively in an image sequence including a plurality of frames; Emotion recognition device further comprising. As a method for recognizing emotions performed in a space-time attention-based emotion recognition device,
Extracting 3D features by pattern-recognizing T consecutive frames of the image sequence (where T is a natural number) as a single 3D image by learning in advance according to a predetermined 3D pattern recognition technique;
Pre-learning according to a known two-dimensional pattern recognition technique, extracting T spatial features through pattern recognition from each of the T frames, and adding space-time features between the obtained T spatial features to obtain space-time weights ;
Obtaining emotional features by weighting the space-time weights with the 3D features;
Learning in advance according to a predetermined 3D pattern recognition technique, and extracting an emotion value having a value within a predetermined range from the emotion feature; And
Determining an emotion corresponding to the emotion value; Emotion recognition method comprising a. The method of claim 11, wherein the step of extracting the 3D feature
Emotion recognition method for extracting the 3D features using 3D CNN (3D Convolutional Neural Networks) previously learned. The method of claim 11, wherein the step of obtaining the space-time weight
Extracting the T spatial features for each of the T frames using pre-trained 2D CNNs (2D Convolutional Neural Networks);
Extracting the spatiotemporal feature between the T spatial features using a previously learned Convlutional Long Short-Term Memory (ConvLSTM); And
Normalizing the extracted spatiotemporal features in a predetermined manner to obtain the spatiotemporal weights; Emotion recognition method comprising a. The method of claim 11, wherein the step of extracting the emotion value
An emotion recognition method for extracting emotion values representing emotions from the emotion features including 3D CNNs learned in advance.

Images