KR102290186B1 - Method of processing video for determining emotion of a person - Google Patents

Abstract

The multi-modal emotion recognition method using artificial intelligence of the present description is an emotion recognition method that processes an image to determine a person's emotional state, and provides an image and a voice expressing the appearance of a person, wherein the image is the first comprising an imaging unit, a second imaging unit immediately following the first imaging unit, and a third imaging unit immediately following the second imaging unit; The first image unit processes the first image unit to determine the emotional state of the person, wherein the first image unit shows the person's face and at least one hand, and the at least one hand characterized in that no part of the face overlaps; and the second image unit processes the second image unit to determine the emotional state of the person, wherein the second image unit shows the person's face and at least one hand, and the at least one hand is the person Including, wherein the processing of the first image unit includes at least one of the first image unit to determine whether the at least one hand covers the face of the person processing a frame of ; finding a first facial element of the person in the at least one frame of the first image unit; and in a state in which the first facial element is located, the at least one obtaining first facial feature data of the first image unit based on the shape of the first facial element shown in the frame of processing the audio data of the first image unit to obtain determining a state; wherein the processing of the second image unit includes processing at least one frame of the second image unit to determine whether the face of the person is covered by at least one hand. determining whether or not the at least one hand covers the face of the person in the second image unit; and finding the first facial element of the person in at least one frame of the second image unit. obtaining the first facial feature data of the first image unit based on the shape of the first face element shown in the at least one frame of the second image unit in a state where the first facial element is located step, and the audio of the second image unit to obtain the voice characteristic data based on the human voice characteristics in the second image unit processing data, the first facial feature data of the second image unit, the audio feature data of the second image unit, and additional data indicating a location where at least one hand covers a part of the person's face and determining the emotional state of the person for the second image unit based on the plurality of data included therein.

Description

Emotion recognition method that processes images to determine a person's emotional state

Embodiments of the present invention relate to an emotion recognition method of processing an image to determine a person's emotional state.

In the prior art, occlusion is recognized and treated as an error. Covering your mouth with your hand is important information and can determine the intensity of your emotional state. A simple static image may lack recognition information due to an occlusion problem.

In addition, if the subject speaks when recognizing emotions through facial expressions, false emotion recognition results are derived. Mouth shape is very important information for emotion recognition through facial expression recognition, but since the mouth shape changes frequently when speaking, mouth shapes such as surprise, anger, and laughter may appear, leading to incorrect recognition results.

As such, among the prior art, there are few alternatives to solve the case of recognizing emotions only with facial expressions. are doing In this patent, after determining whether a person is currently speaking by tracking the shape of a face or mouth, it is possible to derive an accurate emotion recognition result by minimizing the mouth shape information and increasing the weight of the voice feature information in the case of the speaking state.

Embodiments of the present invention provide a multi-modal emotion recognition apparatus, method and storage medium for performing more accurate emotion recognition using temporal information as well as hand movement and identification information, mouth shape information, voice information, and partial expression information. would like to provide

An emotion recognition method for processing an image to determine a person's emotional state according to an aspect of an embodiment of the present invention provides an image and a sound expressing the appearance of a person, wherein the image includes a first image unit and the second image comprising a second imaging part immediately following the first imaging part and a third imaging part immediately following the second imaging part; The first image unit processes the first image unit to determine the emotional state of the person, wherein the first image unit shows the person's face and at least one hand, and the at least one hand characterized in that no part of the face overlaps; and the second image unit processes the second image unit to determine the emotional state of the person, wherein the second image unit shows the person's face and at least one hand, and the at least one hand is the person Including, wherein the processing of the first image unit includes at least one of the first image unit to determine whether the at least one hand covers the face of the person processing a frame of ; finding a first facial element of the person in the at least one frame of the first image unit; and in a state in which the first facial element is located, the at least one obtaining first facial feature data of the first image unit based on the shape of the first facial element shown in the frame of processing the audio data of the first image unit to obtain determining a state; wherein the processing of the second image unit includes processing at least one frame of the second image unit to determine whether the face of the person is covered by at least one hand. determining whether or not the at least one hand covers the face of the person in the second image unit; and finding the first facial element of the person in at least one frame of the second image unit. obtaining the first facial feature data of the first image unit based on the shape of the first face element shown in the at least one frame of the second image unit in a state where the first facial element is located step, and the audio of the second image unit to obtain the voice characteristic data based on the human voice characteristics in the second image unit processing data, the first facial feature data of the second image unit, the audio feature data of the second image unit, and additional data indicating a location where at least one hand covers a part of the person's face and determining the emotional state of the person for the second image unit based on the plurality of data included therein.

In addition, the determining of the emotional state of the person for the second image unit based on the plurality of data may include: when a part of the person's face is covered by at least one hand in the second image unit, The audio feature data of the second image unit may be given more weight than the first facial feature data of the second image unit.

In addition, in the step of determining the emotional state of the person for the second image unit based on the plurality of data, any part of the person's face is not covered by at least one hand in the first image unit. , when a part of the face of the person is covered by at least one hand in the second image unit, the audio characteristic data of the second image unit may be given more weight than the audio characteristic data of the first image unit .

The processing of the first image unit may include: finding a second facial element of the person in the at least one frame of the first image unit; The method further comprises acquiring second facial feature data of the first image part based on a shape of the second facial element shown in the at least one frame of part, in particular, the first facial element of the first image part determining the emotional state of the person with respect to the first image unit based on a plurality of data including feature data, the second facial element feature data of the first image unit, and audio feature data of the first image unit, The processing of the second image unit may include determining whether the second face element is covered by at least one hand that finds the second face element of the person in the at least one frame of the second image unit. , and obtaining second facial feature data of a second image unit based on a preset weight for occlusion of the second face feature and a second facial element of the first image unit, in particular, The first facial element feature data of the second image part, the second facial element feature data of the second image part, the audio feature data of the second image part, and the at least one The emotional state of the person with respect to the second image unit may be determined based on additional data indicating the position of the hand.

In addition, further comprising the step of processing the third image unit to determine the emotional state of the person for the third image unit, in a state in which any part of the person's face is not covered by at least my hand, 3 The face of the person and the hand of the equator are shown on the image unit, and the processing of the third image unit includes the third image to determine whether the face of the person is covered by the at least one hand. processing at least one frame of the image unit; finding a first facial element of the person in at least one frame of the third image unit; obtaining first facial feature data of the third image unit based on the shape of the first facial element shown in the at least one frame of the third image unit in a state where the first facial element is located; processing the audio data of the first image unit to obtain voice characteristic data of the third image unit based on the human voice characteristic in the third image unit; and determining the emotional state of the person of the first image unit based on a plurality of data including the first facial feature data and the voice feature data of the third image unit.

When executed by a computer according to another aspect of an embodiment of the present invention, the computer readable storage medium for storing a preset instruction provides an image and a sound expressing the appearance of a person, the image comprising: a first image unit; comprising a second imaging part immediately following the first imaging part and a third imaging part immediately following the second imaging part; The first image unit processes the first image unit to determine the emotional state of the person, wherein the first image unit shows the person's face and at least one hand, and the at least one hand characterized in that no part of the face overlaps; and the second image unit processes the second image unit to determine the emotional state of the person, wherein the second image unit shows the person's face and at least one hand, and the at least one hand is the person Including, wherein the processing of the first image unit includes at least one of the first image unit to determine whether the at least one hand covers the face of the person processing a frame of ; finding a first facial element of the person in the at least one frame of the first image unit; and in a state in which the first facial element is located, the at least one obtaining first facial feature data of the first image unit based on the shape of the first facial element shown in the frame of processing the audio data of the first image unit to obtain determining a state; wherein the processing of the second image unit includes processing at least one frame of the second image unit to determine whether the face of the person is covered by at least one hand. determining whether or not the at least one hand covers the face of the person in the second image unit; and finding the first facial element of the person in at least one frame of the second image unit. obtaining the first facial feature data of the first image unit based on the shape of the first face element shown in the at least one frame of the second image unit in a state where the first facial element is located step, and the audio of the second image unit to obtain the voice characteristic data based on the human voice characteristics in the second image unit processing data, the first facial feature data of the second image unit, the audio feature data of the second image unit, and additional data indicating a location where at least one hand covers a part of the person's face Stores a command for performing an emotion recognition method of processing an image to determine a person's emotional state, which includes determining the emotional state of the person with respect to the second image unit based on a plurality of data included therein.

According to the embodiment of the present invention as described above, the emotion recognition method for processing an image to determine the emotional state of a person can accurately determine the emotional state in the case of conversation and when the expression is covered by an object such as a hand. there is.

1 is a diagram schematically illustrating a configuration of a multi-modal emotion recognition apparatus according to an embodiment of the present invention.
FIG. 2 is a diagram schematically illustrating the configuration of a data preprocessor in the multi-modal emotion recognition apparatus of FIG. 1 .
3 is a diagram schematically illustrating a configuration of a preliminary inference unit in the multi-modal emotion recognition apparatus of FIG. 1 .
FIG. 4 is a diagram schematically illustrating the configuration of a main reasoning unit in the multi-modal emotion recognition apparatus of FIG. 1 .
5 is a flowchart illustrating a multi-modal emotion recognition method by the multi-modal emotion recognition apparatus of FIG. 1 .
6 is a flowchart illustrating in detail a data pre-processing step in the multi-modal emotion recognition method of FIG. 5 .
7 is a flowchart illustrating in detail a preliminary inference step in the multi-modal emotion recognition method of FIG. 5 .
8 is a flowchart illustrating in detail a main reasoning step in the multi-modal emotion recognition method of FIG. 5 .
FIG. 9 is an exemplary view illustrating a face recognition process according to whether a situation has changed in the multi-modal emotion recognition apparatus of FIG. 1 .

Hereinafter, with reference to the accompanying drawings, embodiments of the present invention will be described in detail so that those of ordinary skill in the art to which the present invention pertains can easily implement them.

The present invention may be embodied in many different forms and is not limited to the embodiments described herein. In order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and the same reference numerals are assigned to the same or similar components throughout the specification. In addition, since the size and thickness of each component shown in the drawings are arbitrarily indicated for convenience of description, the present invention is not necessarily limited to the illustrated bar.

In the present invention, "on" means to be located above or below the target member, and does not necessarily mean to be located above the target member based on the direction of gravity. In addition, throughout the specification, when a part "includes" a certain component, this means that other components may be further included, rather than excluding other components, unless otherwise stated.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, and when described with reference to the drawings, the same or corresponding components are given the same reference numerals, and the overlapping description thereof will be omitted. .

The present invention derives a more accurate emotional recognition result by using artificial intelligence that considers facial expressions, speech states, hands, and voices based on the subject's video and voice data.

1 is a diagram schematically illustrating a configuration of a multi-modal emotion recognition apparatus according to an embodiment of the present invention.

Referring to FIG. 1 , the multi-modal emotion recognition apparatus 10 includes a data input unit 100 , a data preprocessor 200 , a preliminary reasoning unit 300 , a main reasoning unit 400 , and an output unit 500 . can do.

The data input unit 100 may receive the user's image data DV and audio data DS.

The data input unit 100 may include an image input unit 110 that receives image data DV for recognizing the user's emotions and a voice input unit 120 that receives the user's voice data DS.

In addition, the data preprocessor 200 includes _{an audio preprocessor 220 that generates voice feature data DF 2} from the voice data DS, and one or more facial feature data DF _{1 from the image data DV.} An image preprocessor 210 may be included.

In this case, the facial feature data DF ₁ may include at least one of an image, location information, size information, face ratio information, and depth information, and the voice feature data DF ₂ may include intonation and pitch. Information, utterance strength, utterance speed, etc. may include information that can represent the characteristics of the voice.

The image preprocessor 210 performs image preprocessing for extracting the user's facial feature data DF _{1 from the image data DV.}

The image preprocessing may convert image data (DV) for using a learning model, such as whole or partial face recognition, noise removal, user facial features and image extraction, into an appropriate aspect.

The voice preprocessor 220 performs voice preprocessing for extracting the user's voice feature data DF _{2 from the voice data DS.}

The voice preprocessing may transform the voice data DS into an appropriate aspect for using a learning model, such as external noise removal, noise removal, and user voice feature extraction.

The preliminary inference unit 300 may generate situation determination data P regarding whether the user's situation changes according to a temporal sequence, based on the image data DV.

At this time, the situation determination data P is a recognition target area different from the tracking target area B which is a part _{of the dialogue determination data P 1 or the entire image area of the image data DV for whether the user is in a conversation state (} It may include _{overlap determination data P 2} on whether overlap with A).

_{In detail, the preliminary inference unit 300 generates the position inference data DM 1} for inferring the position of the tracking target region B based on the image data DV, and the facial feature data DF ₁ and Based on the location inference data DM ₁ , it is possible to generate _{overlap determination data P 2} as to whether the tracking target area B and the recognition target area A overlap.

Also, the preliminary inference unit 300 may generate _{dialogue determination data P 1} for determining whether the user is in a conversation state based on the _{facial feature data DF 1 .}

The main inference unit 400 _{generates at least one sub feature map FM based on the voice feature data DF 2} or the facial feature data DF ₁ , and the sub feature map FM and the situation determination data ( P) based on the user's emotional state can be inferred.

The emotional state may include information on the user's emotional state, such as happiness, anger, fear, disgust, sadness, and surprise.

The output unit 500 may output the result of the emotional state inferred by the main reasoning unit 400 .

At this time, the output unit 500 can output in various forms using activation functions such as a sigmoid function, a step function, a softmax function, and a ReLU (Rectified Linear Unit). there is.

FIG. 2 is a diagram schematically illustrating the configuration of a data preprocessor in the multi-modal emotion recognition apparatus of FIG. 1 .

Referring to FIG. 2 , the data preprocessor 200 may include an image preprocessor 210 and an audio preprocessor 220 .

The image preprocessor 210 may include a face detector 211 , an image preprocessing module 212 , a landmark detection module 213 , a position adjustment module 214 , and a face element extraction module 215 .

The face detector 211 may detect the recognition target area A, which is an area corresponding to the user's face, from the entire area of the image data DV.

The image pre-processing module 212 may correct the recognition target area A.

In detail, the image preprocessing module 212 may perform image brightness, blur correction, and noise removal of image data DV.

The landmark detection module 213 may extract the facial element location information AL of the recognition target area A.

In detail, positional information of important face elements such as a face, eyes, mouth, nose, and forehead in the recognition target area A may be grasped to enable face recognition.

The position adjustment module 214 may adjust the position based on the facial element position information AL of the recognition target area A.

In detail, the position adjustment module 214 may align the image horizontally or vertically based on the facial element position information AL extracted from the landmark detection module 213 .

The facial element extraction module 215 is located in the recognition target area A and sets a sub recognition target area AA smaller than the recognition target area A, and facial feature data DF _{1 of the sub recognition target area AA} ) can be created.

The sub-recognition target area AA may be a plurality of areas or one area in which at least one or more facial elements such as a face, eyes, mouth, nose, and forehead are determined.

For example, when the eyes, nose, and mouth from which the facial element location information AL are extracted are extracted from the recognition target area A, the facial element extraction module 215 may be configured to perform an eye recognition area that is the sub recognition target area AA. (A ₁ ), the nose recognition area A ₂ , and the mouth recognition area A ₃ may be set, and at least one or more facial feature data DF ₁ may be generated for the set sub-recognition target area AA.

Also, when the sub-recognition target area AA is not set, the facial element extraction module 215 may generate _{facial feature data DF 1 based on the recognition target area A.}

The voice preprocessor 220 may include a voice correction module 221 and a voice feature data extraction module 222 .

The voice correction module 221 may correct the voice data DS.

In detail, the voice correction module 221 may generate corrected voice data by performing various correction methods, such as various noise and external noise removal, volume control, and frequency correction included in the voice data DS.

The voice feature data extraction module 222 may extract features of the voice data DS that have passed through the voice correction module 221 to generate the _{voice feature data DF 2 .}

In detail, the voice feature data extraction module 222 uses one or more modules of voice data, frequency and spectrum analysis modules such as Mel-frequency Cepstral Coefficients (MFCC), Geneva Minimalistic Acoustic Parameter Set (eGeMAPS), Logbank, and the like. Voice feature data (DF ₂ ) can be generated.

In this case, the voice feature data extraction module 222 may use the corrected voice data or the voice data DS.

FIG. 3 is a diagram schematically illustrating the configuration of a preliminary inference unit in the multi-modal emotion recognition apparatus of FIG. 1 .

Referring to FIG. 3 , the preliminary inference unit 300 may include a hand detection inference module 310 , a conversation state inference module 320 , and a face overlap inspection module 330 .

The dialogue state inference module 320 may use the first learning model LM ₁ and generate dialogue determination data P ₁ based on the facial feature data DF _{1 .}

In detail, the dialog state inference module 320 _{uses the whole or part of the user's facial feature data DF 1 , and uses the first learning model LM 1} that can determine whether the user is in a dialog state. , it is possible to generate dialogue determination data P _{1 that is whether dialogue is determined.}

The facial feature data DF _{1 includes mouth image data DV 2 that} is image data DV for a portion corresponding to the user's mouth in the recognition target area A, and the first learning model LM ₁ . _{By using , conversation determination data P 1} on whether the user is in a conversation state may be generated from the mouth image data DV _{2 .}

The first learning model (LM ₁ ) is an artificial intelligence that can infer temporal or spatial features such as Long Short-Term Memory (LSTM), Recurrent Neural Networks (RNNs), Deep Neural Networks (DNNs), Convolutional Neural Networks (CNNs), etc. The method may be at least one or more of an intelligent model, machine learning, and deep learning method.

The hand detection inference module 310 detects the hand image data DV ₁ for the tracking target region B from the image data DV, and uses the second learning model LM ₂ to obtain the hand image data DV ₁ ) based on the location inference data DM ₁ may be generated.

At this time, the second learning model (LM ₂ ) is used to infer temporal or spatial features such as Long Short-Term Memory (LSTM), Recurrent Neural Networks (RNNs), Deep Neural Networks (DNNs), and Convolutional Neural Networks (CNNs). the number of artificial intelligence models, machine learning, methods of at least one of the deep learning how can you generate an inferred position data (DM ₁₎ for the hand over.

In addition, the hand detection reasoning module 310 generates a location inference feature map (FM _{1 ) for the location inference data (DM 1} ), and a sub feature map (FM), situation determination data (P), and a location inference feature Based on the map FM ₁ , the emotional state of the user may be inferred.

In this case, the location inference feature map FM ₁ may include meaningful information on hand movement, such as hand feature information, that is, hand gesture information and hand location information.

The face overlap inspection module 330 determines whether or not the recognition target area A and the tracking target area B overlap based on the _{facial feature data DF 1} and the location inference data DM _{1 , and determines whether overlapping} Depending on the result, the overlap determination data P ₂ may be generated.

In detail, the overlap determination data P ₂ determines whether the recognition target area A and the tracking target area B overlap, and the corresponding facial feature data DF ₁ of the recognition target area A and the voice One or more parameters that determine the importance and use of the feature data DF _{2 may be generated.}

FIG. 4 is a diagram schematically illustrating the configuration of a main reasoning unit in the multi-modal emotion recognition apparatus of FIG. 1 .

Referring to FIG. 4 , the main inference unit 400 includes a plurality of sub feature map generators 410 ; 411 , 412 , 413 , 414 , a multi-modal feature map generator 420 , and an emotion recognition inference unit 430 . may include.

The plurality of sub-feature map generators 410; 411, 412, 413, 414 uses the third learning model LM ₃ to generate voice features based on the voice feature data DF ₂ and the facial feature data DF ₁ . A plurality of sub-feature maps FM for the data DF ₂ and the facial feature data DF _{1 may be generated.}

In detail, the third learning model (LM ₃ ) is an artificial intelligence model that can infer at least one or more spatial features, such as Deep Neural Networks (DNN), Convolutional Neural Network (CNN), etc., at least among machine learning and deep learning methods. There may be one or more methods, and a plurality of sub-feature maps FM in which the features of the speech feature data DF ₂ and the facial feature data DF _{1 are implied are generated using the third learning model LM 3 .} can

The multi-modal feature map generator 420 may generate the multi-modal feature map M from the plurality of sub feature maps FM with reference to the situation determination data P.

The situation determination data P has a predetermined situation determination value PV according to the user's situation, and the multi-modal feature map generator 420 includes at least one situation determination value among the plurality of sub-feature maps FM. (PV) can be applied to generate a multi-modal feature map (M).

In detail, the situation determination value PV may be a parameter indicating the importance of each sub-feature map FM and whether to use it.

A sub-feature map FM to which the situation determination value PV of the situation determination data P is applied is generated through the operation of the situation determination data P and the sub-feature map FM, and a plurality of sub-feature maps FM are generated. ) to create a multi-modal feature map (M).

For example, when the user's eyes are covered, the situation judgment value for the eyes is output as 0, and 0 is output through the multiplication operation of the situation judgment value for the eyes and the sub-feature map (FM) for the eyes. Thus, the main reasoning unit 400 may generate the multi-modal feature map M based on other sub-feature maps except for the sub-feature map for the eye.

In addition, the hand detection inference module 320 generates a location inference feature map (FM ₁ ), and based on the sub feature map (FM), the situation determination data (P) and the location inference feature map (FM ₁ ), the user's emotion It is possible to generate a multi-modal feature map (M) that infers a state.

The multi-modal feature map M may be generated by merging at least one of a sub feature map FM and a location inference feature map FM _{1 using Concat, Merge, and a deep network.}

The emotion recognition inference unit 430 may infer the emotional state based on the multi-modal feature map M using _{the fourth learning model LM 4 .}

At this time, the fourth learning model (LM ₄ ) may be a temporal learning model such as a recurrent neural network such as Long Short-Term Memory (LSTM), Recurrent Neural Network (RNNs), Gated Recurrent Unit (GRU), and temporal characteristics and It may be at least one method among an artificial intelligence model, machine learning, and deep learning method capable of inferring or analyzing spatial features.

5 is a flowchart illustrating a multi-modal emotion recognition method by the multi-modal emotion recognition apparatus of FIG. 1 .

Referring to FIG. 5 , a data input step S100 of receiving the user's image data DV and audio data DS is performed.

Then, a data pre-processing step comprising an audio pre-processing step of generating _{audio feature data DF 2} from the audio data DS, and an image pre-processing step of generating _{one or more facial feature data DF 1 from the image data DV.} (S200) may be performed.

In this case, the data preprocessing step S200 may generate facial feature data DF ₁ and voice feature data DF ₂ for using the learning model.

The learning model may be artificial intelligence, machine learning, and deep learning methods.

Then, based on the image data DV, a preliminary inference step (S300) of generating the situation determination data P regarding whether the user's situation changes according to the temporal order may be performed.

In this case, the temporal order may be whether the conversation is in a state of conversation, and may be data for identifying the characteristics of the movement of the body part.

In addition, the situation determination data P determines whether the image data DV overlaps and whether there is a conversation state, indicating the importance or use of _{one or more facial feature data DF 1} or voice feature data DF _{2 .} It may contain parameters.

In addition, it is possible to extract and generate feature information on a body part of the user other than the _{one or more facial feature data DF 1} generated in the data pre-processing step S200 .

Then, at least one sub feature map FM is generated based on the voice feature data DF ₂ or the facial feature data DF ₁ , and based on the sub feature map FM and the situation determination data P A main reasoning step ( S400 ) of inferring the emotional state of the user may be performed.

At this time, by calculating the sub-feature map (FM) including the feature information extracted from the user and the situation determination data (P) including the parameter for the importance or use of feature information, the importance or use of the sub-feature map (FM) The emotional state of the user may be inferred, including information on whether or not there is.

Then, a result derivation step (S500) of outputting the inference result of the emotional state in the main reasoning step (S400) is performed.

6 is a flowchart illustrating in detail a data pre-processing step in the multi-modal emotion recognition method of FIG. 5 .

Referring to FIG. 6 , the data preprocessing step S200 includes an image preprocessing step S210 and an audio preprocessing step S220 .

In the image pre-processing step S210 , a face detection step of detecting a recognition target image area in the entire area of the image data DV, and the recognition target area A is an area corresponding to the user's face, is performed.

Then, an image pre-processing step of correcting the recognition target area A is performed.

In detail, in the image pre-processing step, image brightness, blur correction, and noise removal of image data DV may be performed.

Next, a landmark detection step of extracting the facial element location information AL of the recognition target area A is performed.

In detail, face recognition may be performed by grasping location information of important face elements such as a face, eyes, nose, mouth, and forehead in the recognition target area A.

Then, a position adjustment step of adjusting the position based on the facial element position information AL of the recognition target area A may be performed.

In detail, images may be aligned horizontally or vertically based on the facial element location information AL extracted from the landmark detection module 213 .

Then, based on the facial element location information AL in the recognition target area A, a sub recognition target area AA located in the recognition target area A and smaller than the recognition target area A is set, and sub recognition is performed. A facial element extraction step of generating the facial feature data DF _{1 of the target area AA may be performed.}

In this case, the sub-recognition target area AA may be a plurality of areas or one area in which at least one or more facial elements such as the entire face, eyes, mouth, nose, and forehead are determined.

For example, when the eyes, nose, and mouth from which the facial element location information AL are extracted are extracted from the recognition target area A, the facial element extraction module 215 may be configured to perform an eye recognition area that is the sub recognition target area AA. (A ₁ ), the nose recognition area A ₂ , and the mouth recognition area A ₃ may be set, and at least one or more facial feature data DF ₁ may be generated for the set sub-recognition target area AA.

In addition, in the extracting of the face element, when the sub-recognition target area AA is not set, the facial feature data DF ₁ may be generated based on the recognition target area A.

The voice preprocessing step S220 includes a voice correction step and a voice feature data extraction step.

First, the voice correction step of correcting the voice data DS is performed.

In detail, in the voice correction step, various correction methods such as removing various noises and external noise included in the voice data DS, volume control, and frequency correction may be performed to generate corrected voice data.

The voice feature data extraction step of extracting features of the voice data DS that has undergone the voice correction step to generate the voice feature data _{DF 2 is performed.}

_{Specifically, the user's voice feature data (DF 2} ) is generated through one or more modules of voice data, frequency and spectrum analysis modules such as Mel-frequency cepstral coefficients (MFCC), Geneva Minimalistic Acoustic Parameter Set (eGeMAPS), Logbank, etc. can be

In this case, the voice feature data extraction step may generate _{the voice feature data DF 2} by using the corrected voice data or by performing the voice data DS without performing the voice correction step.

Also, this is exemplary, and at least some of the steps may be performed simultaneously with the preceding and subsequent steps or performed in a different order.

7 is a flowchart illustrating in detail a preliminary inference step in the multi-modal emotion recognition method of FIG. 5 .

A dialog state inference step S310 of using the first learning model LM ₁ and generating the dialog determination data P ₁ based on the facial feature data DF _{1 may be performed.}

In the dialog state inference step (S310), _{by using the first learning model (LM 1} ) to detect the presence or absence of a conversation in the previous situation and the features and movements of the facial elements from the facial _{feature data (DF 1 ) to detect the conversation state} can be

In detail, by using the whole or part of the user's facial feature data DF _{1 , using the first learning model LM 1 to determine whether the user is talking, the conversation determination data P 1} , which is whether the conversation is determined or not, is can be created

In this case, the facial feature data DF _{1 may include mouth image data DV 2 of} a portion corresponding to the user's mouth in the recognition target area A.

Also, by using the first learning model LM _{1 , the conversation determination data P 1} on whether the user is in a conversation state may be generated from the mouth image data DV _{2 .}

Next, the hand image data DV ₁ for the tracking target region B is detected from the image data DV, and position inference based on the hand image data DV ₁ using the second learning model LM _{2 .} The hand detection inference step S320 of generating the data DM _{1 is performed.}

In this case, using the second learning model LM ₂ , temporal inference with respect to the position of the hand with the previous situation may be possible. For example, it may be determined whether the hand temporarily overlaps the face.

In addition, the hand detection reasoning step (S320) generates a location inference feature map (FM _{1 ) for the location inference data (DM 1} ), and a sub feature map (FM), situation determination data (P), and a location inference feature Based on the map FM ₁ , the emotional state of the user may be inferred.

In detail, the location inference feature map FM ₁ may include meaningful information on the movement of the hand, such as a feature capable of recognizing a gesture for the hand and information on the location of the hand.

Then, based on the facial feature data (DF ₁ ) and the location inference data (DM ₁ ), it is determined whether the recognition target area (A) and the tracking target area (B) overlap, and the overlap determination data is based on the overlapping determination result A face overlap detection step (S330) of generating (P _{2 ) is performed.}

In detail, the overlap determination data P ₂ determines whether the recognition target area A and the tracking target area B overlap, and the corresponding facial feature data DF ₁ of the recognition target area A and the voice It may include one or more parameters that determine the importance and use of the feature data DF _{2 .}

8 is a flowchart illustrating in detail a main reasoning step in the multi-modal emotion recognition method of FIG. 5 .

Referring to FIG. 8 , the main reasoning step ( S400 ) includes a plurality of sub-feature map generation steps ( S410 ), a multi-modal feature map generation step ( S420 ), and an emotion recognition inference step ( S430 ).

First, a plurality of speech feature data (DF ₂ ) and facial feature data (DF ₁ ) based on the speech feature data (DF ₂ ) and the facial feature data (DF ₁ ) using the third learning model (LM _{3 )} A plurality of sub-feature map generation steps (S410) for generating the sub-feature map FM are performed.

Next, the third learning model (LM ₃ ) generates a multi-modal feature map (S420) for generating a multi-modal feature map (M) from a plurality of sub feature maps (FM) with reference to the situation determination data (P) is performed

At this time, the situation determination data P has a predetermined situation determination value PV according to the user's situation, and the multi-modal feature map generation step S420 is performed on at least one of the plurality of sub-feature maps FM. A multi-modal feature map M may be included by applying the situation determination value PV.

In addition, in the multi-modal feature map generation step (S420), the position inference feature map (FM ₁ ) is generated from the hand detection inference module 320, and the sub feature map (FM), the situation determination data (P) and the position inference feature A multi-modal feature map M for inferring a user's emotional state based on the map FM _{1 may be generated.}

Next, using the fourth learning model (LM ₄ ), the emotional recognition inference step ( S430 ) of inferring the emotional state based on the multi-modal feature map (M) is performed.

At this time, the fourth learning model (LM ₄ ) may be a temporal learning model such as a recurrent neural network such as Long Short-Term Memory (LSTM), Recurrent Neural Network (RNNs), Gated Recurrent Unit (GRU), and temporal characteristics and It may be at least one method among an artificial intelligence model, machine learning, and deep learning method capable of inferring or analyzing spatial features.

FIG. 9 is an exemplary view illustrating a face recognition process according to whether a situation has changed in the multi-modal emotion recognition apparatus of FIG. 1 .

Referring to FIG. 9 , (step (A)) shows a situation in which the user puts his or her hand on the face and the hand does not cover the mouth and nose.

The user's image data DV is input through the image input unit 110 , and the user's voice data DS is input through the audio input unit 120 .

Thereafter, the image preprocessor 210 generates the image preprocessed facial feature data DF ₁ , and also generates the voice preprocessed voice feature data DF ₂ through the audio preprocessor 220 , _{The image preprocessor 210 is configured to perform an eye recognition area (A 1} ), a nose recognition area (A ₂ ), and a mouth recognition area (A ₃ ) based on the recognizable face element location information (AL) of the user's eyes, nose, and mouth. The recognition target area A including

Thereafter, the preliminary inference unit 300 generates the hand image data DV _{1 for the tracking target area B 1 detected from the image data DV.}

At this time, the preliminary inference unit 300 _{generates the position inference data DM 1} for grasping the movement of the hand through the _{hand image data DV 1} , and the tracking target area B based on the position inference data DM _{1 . 1 ) and overlap determination data P 2} are generated based on the determination of whether the recognition target region A overlaps.

Here, the overlap determination data P ₂ may include parameters indicating use of the eye recognition area A ₁ , the nose recognition area A ₂ , and the mouth recognition area A _{3 .}

In addition, the dialog state inference module 310 generates _{dialog determination data P 1} by determining whether a dialog state exists through the mouth recognition region A _{3 based on the mouth image data DV 2 .}

Thereafter, the sub-feature map generation unit 410 generates a plurality of sub-feature maps FM using the third learning model LM _{3 using the facial feature data DF 1 corresponding to the eyes, nose, and mouth.} .

Thereafter, the multi-modal feature map generator 420 generates the multi-modal feature map M by integrating the plurality of sub feature maps FM and the position inference feature map FM _{1 corresponding to the hand.}

Thereafter, emotional recognition may be inferred in consideration of the previous user's behavior through the fourth learning model LM _{4 , and this may be expressed as an emotional recognition result.}

(Step (B)) Step B shows the continuous operation of step A.

For example, it can be assumed that stage B is an image continuously taken following stage A at a speed of 30 FPS.

As in step A, the user's image data DV is input through the image input unit 110 , and the user's voice data DS is input through the audio input unit 120 .

_{Thereafter, the voice pre-processing unit DF 2} is generated through the voice pre-processing unit 220 , and the image pre-processing unit 210 generates the facial feature data DF ₁ and the facial element location information AL. and set a recognition target area (A) including an _{eye recognition area (A 1} ), a nose recognition area (A ₂ ), and a mouth recognition area (A ₃ ) based on the face element location information (AL), and The area A is transmitted to the preliminary inference unit 300 .

In this case, the size of the recognition target area A may change according to a user's motion.

In step B, compared with step A, the size of the recognition target area A changes according to the operation.

Thereafter, the preliminary inference unit 300 may generate the position inference data DM _{1 based on the hand image data DV 1} , and track the movement of the hand from step A to step B.

The preliminary reasoning unit 300 generates _{overlap determination data P 2} based on determining whether the tracking target area B ₂ and the recognition target area A overlap _{based on the location inference data DM 1 .}

In addition, the preliminary reasoning unit 300 determines whether a conversation state exists and generates _{the conversation determination data P 1 .}

At this time, the preliminary reasoning unit 300 uses the first learning model (LM ₁ ) to consider whether the conversation of the user who is the subject of emotion recognition in the previous situation including the step (A) is continuing in consideration of whether the conversation state can be judged

For example, if it is inferred that the user is not in a conversational state in step A, based on the result, the user's mouth shape is temporarily changed based _{on the mouth recognition area A 3 in step B.} Although similar to , the preliminary inference unit 300 may determine that the user is not in a conversational state by using _{the first learning model LM 1 .} That is, the preliminary inference unit 300 may perform inference on whether to determine the conversation state in the next scene, step B, based on the result of determining the dialogue state in the step A.

Thereafter, the main inference unit 400 generates a plurality of sub-feature maps FM using the _{received facial feature data DF 1} and the voice feature data DF ₂ using the third learning model LM _{3 , and} , a plurality of sub-feature maps (FM) and a position inference feature map (FM ₁ ) corresponding to the hand are integrated to generate a multi-modal feature map (M).

Thereafter, the main reasoning unit 400 infers emotional recognition in consideration of the user's behavior of the previous (step (A)) through _{the fourth learning model LM 4 , and may represent this as an emotional recognition result.}

(Step (C)) After step B, the action of the user covering his mouth with his hand is shown.

The image preprocessor 210 sets the recognition target area A including the _{eye recognition area A 1} based on the recognizable user's eye facial element position information AL, and selects the recognition target area A It is transmitted to the preliminary inference unit 300 .

Thereafter, the preliminary inference unit 300 generates the hand image data DV _{1 for the tracking target area B 3 detected from the image data DV.} At this time, _{the position inference data DM 1} for recognizing the movement of the hand is generated through the _{hand image data DV 1} , and the tracking target area B ₃ and the recognition target area based on the position inference data DM _{1 ( The overlap determination data P 2} is generated based on the overlap determination of A).

Here, the overlap determination data P ₂ may include a parameter indicating whether the facial feature data _{DF 1} based on the eye recognition area A ₁ is used or a weight applied to the _{facial feature data DF 1 .}

_{In addition, the preliminary reasoning unit 300 is an area for the coin recognition area (A 2} ) or the mouth recognition area (A ₃ ), which was the recognition target area (A) in steps (A) and (B), and the user's hand position. By recognizing the overlap with the tracking target area (B ₃ ), a parameter indicating that it is excluded from the emotion recognition inference or the importance is lowered may be included _{in the overlap determination data (P 2 ).}

In addition, the preliminary inference unit 300 considers a situation in which the mouth image data DV _{2 corresponding to the mouth recognition area A 3} is not recognized and a result of determining whether the user has a previous conversation state, and the voice feature data DF ₂ ) may be included in the _{conversation determination data P 1} a value indicating whether use is determined.

Here, the determination result of whether the previous conversation state is inferred through a temporal learning model. In this case, the temporal learning model may be a temporal learning model such as a recurrent neural network such as a Long Short-Term Memory (LSTM), a Recurrent Neural Network (RNNs), or a Gated Recurrent Unit (GRU).

Thereafter, the sub-feature map generator 410 generates a plurality of sub-feature maps FM using the third learning model LM _{3 using the facial feature data DF 1 of the region corresponding to the eye.}

Thereafter, the multi-modal feature map generator 420 generates the multi-modal feature map M by integrating the plurality of sub feature maps FM and the position inference feature map FM _{1 corresponding to the hand.}

Thereafter, the emotion recognition inference unit 430 may infer emotion recognition _{in consideration of the previous user's behavior through the fourth learning model LM 4} , and may represent this as an emotion recognition result.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

The system or apparatus described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, the systems, devices and components described in the embodiments may include, for example, a processor, a controller, an Arithmetic Logic Unit (ALU), a digital signal processor, a microcomputer, a Field Programmable Array (FPA). , a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose computers or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, the processing device is sometimes described as being used, but one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other Processing Configurations are also possible, such as a Parallel Processor.

The software may include a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or provide instructions or data to the processing device. , or may be permanently or temporarily embodied in a transmitted signal wave (Signal Wave). The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiments may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magnetic media such as floppy disks. - Includes Magneto-optical Media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, Flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

10: Multi-modal emotion recognition device
100: data input unit 110: video input unit
120: voice input unit
200: data preprocessor 210: image preprocessor
211: face detection module 212: image preprocessing module
213: landmark detection module 214: position adjustment module
215: face element extraction module
220: voice preprocessor 221: voice correction module
222: speech feature data extraction module 300: preliminary inference unit
310: dialog state inference module
320: hand detection inference module 330: face overlap inspection module
400: main reasoning unit
411: first sub feature map generator 412: second sub feature map generator
413: third sub feature map generator 414: nth sub feature map generator
420: multi-modal feature map generation unit 430: emotion recognition inference unit
500: output unit
S100: data input step S200: data pre-processing step
S210: image pre-processing step S220: audio pre-processing step
S300: preliminary reasoning step S310: dialogue state reasoning step
S320: Hand detection inference step S330: Face overlap check step
S400: Main reasoning step S410: Sub feature map generation step
S420: Multi-modal feature map generation step S430: Emotion recognition inference step
S500: Result Derivation Step A ₁ : Eye Recognition Area
A ₂ : nose recognition area A ₃ : mouth recognition area
B _1, B _2, B ₃ : Tracking target area

Claims (6)

In the emotion recognition method using an emotion recognition device that processes an image to determine a person's emotional state,
The image, which provides an image and a sound expressing the appearance of a person, includes a first image unit, a second image unit immediately following the first image unit, and a third image unit immediately following the second image unit, step;
The first image unit processes the first image unit to determine the emotional state of the person, wherein the first image unit shows the person's face and at least one hand, and the at least one hand characterized in that no part of the face overlaps; and
The second image unit processes the second image unit to determine the emotional state of the person, and the second image unit shows the person's face and at least one hand, and the at least one hand Including; characterized in that it overlaps the face;
The step of processing the first image unit,
processing at least one frame of the first image unit to determine whether the at least one hand covers the face of the person;
finding a first facial element of the person in the at least one frame of the first image unit;
obtaining first facial feature data of the first image unit based on the shape of the first facial element shown in the at least one frame of the first image unit in a state where the first facial element is located;
processing the audio data of the first image unit to obtain voice characteristic data based on characteristics of the human voice in the first image unit;
determining the emotional state of the person for the first image unit based on a plurality of data including the first facial feature data and the voice feature data of the first image unit;
The step of processing the second image unit,
processing at least one frame of the second image unit to determine whether the face of the person is obscured by the at least one hand, in particular the face of the person in the second image unit The step, at which it is decided whether to cover, and
finding the first facial element of the person in at least one frame of the second image unit;
obtaining first facial feature data of the second image unit based on the shape of the first facial element shown in the at least one frame of the second image unit in a state where the first facial element is located;
processing the audio data of the second image unit to obtain voice characteristic data based on the human voice characteristic in the second image unit;
Based on a plurality of data including the first facial feature data of the second image unit, the audio feature data of the second image unit, and additional data indicating a position where at least one hand covers a part of the person's face and determining the emotional state of the person for the second image unit by processing an image to determine the emotional state of the person.
According to claim 1,
The step of determining the emotional state of the person for the second image unit based on the plurality of data,
A person who places more weight on the voice feature data of the second image unit than on the first facial feature data of the second image unit when a part of the person's face is covered by at least one hand in the second image unit Emotion recognition method that processes images to determine the emotional state of
According to claim 1,
The step of determining the emotional state of the person for the second image unit based on the plurality of data,
When at least one hand does not cover any part of the person's face in the first image unit, but a part of the person's face is covered by at least one hand in the second image unit, the first An emotion recognition method of processing an image to determine an emotional state of a person who places more weight on the audio feature data of the second video part than on the audio feature data of the video part.
According to claim 1,
The step of processing the first image unit,
finding a second facial element of the person in the at least one frame of the first image unit;
obtaining second facial feature data of the first image part based on the shape of the second facial element shown in the at least one frame of the first image part in a state where the second facial element is located; including,
In particular, based on a plurality of data including the first facial feature data of the first image part, the second facial feature data of the first image part, and the audio feature data of the first image part, the first image part is determines the emotional state of the person for
The step of processing the second image unit,
determining whether the second facial element is obscured by at least one hand, finding a second facial element of the person in the at least one frame of the second image unit;
The method further comprises: acquiring second facial feature data of a second image part based on a preset weight for occlusion of the second facial feature data and a second facial element of the first image part;
In particular, the first facial feature data of the second image part, the second facial feature data of the second image part, the audio feature data of the second image part, and the at least one Emotion recognition method for processing an image to determine the emotional state of a person, characterized in that determining the emotional state of the person for the second image unit based on additional data indicating the position of the hand.
According to claim 1,
Further comprising the step of processing the third image unit to determine the emotional state of the person with respect to the third image unit, in a state in which any part of the person's face is not covered by at least one hand, 3 The face of the person and the at least one hand are shown on the image unit,
The step of processing the third image unit,
processing at least one frame of the third image unit to determine whether the at least one hand covers the face of the person;
finding a first facial element of the person in at least one frame of the third image unit;
obtaining first facial feature data of the third image unit based on the shape of the first facial element shown in the at least one frame of the third image unit in a state where the first facial element is located;
processing the audio data of the first image unit to obtain voice characteristic data of the third image unit based on the human voice characteristic in the third image unit; and
determining the emotional state of the person including; determining the emotional state of the person in the first image unit based on a plurality of data including the first facial feature data and the voice feature data in the third image unit Emotion recognition method that processes images for
A computer readable storage medium storing instructions that, when executed by a computer, perform the method of claim 1 .

Images