JPH11232456A - Method for extracting expression from face animation - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method for extracting an expression from a face animation, by which a section where the expression is extracted is extracted. SOLUTION: A velocity vector is calculated from the inputted face animation, two-dimensional Fourier transform is executed in the respective components of the velocity vector and a feature vector string corresponding to the movement of the expression is extracted in a pre-processing part S10. An expression change detecting part S20 calculates a motion vector by integrating feature vectors by time and collates it with the previously decided respective expression patterns. An extraction recognizing part 30 previously generates HMM(hidden Marcov model) at every expression category of an object to be recognized by learning and calculates the feature vector string and a probability for generation in accordance with a collation result through the use of HMM. Then, the expression category corresponding to HMM by which the calculated probability becomes the max. one is adopted as a recognition result.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for extracting a facial expression from a face moving image, and more particularly to a method for extracting a facial expression from a continuous image in a facial moving image.

[0002]

2. Description of the Related Art Among human facial expressions, it is known that six basic facial expressions (anger, disgust, fear, sadness, happiness, and surprise) are common regardless of race or culture. Basic facial expressions may be generated independently of each other,
A plurality of facial expressions may be continuously generated.

[0003] For example, when the jet passes over the sky, an expression sequence is conceivable in which the noise is surprised first, then the disgust and anger are expressed for the continuation of the noise, and finally the noise ends and the feeling of happiness is considered. . To correctly recognize such a facial expression sequence, it is necessary to detect a change in facial expression and recognize which facial expression has changed.

[0004] The inventors of the present application have proposed a technique disclosed in Japanese Patent Application No. 9-55.
886 proposed a method for recognizing facial expressions from face images. In this method, a velocity vector of a face element is calculated from a face moving image, Fourier-transformed, a feature vector is extracted from the Fourier transform coefficient, and a plurality of output probabilities obtained by approximating a continuous output probability for each expression using a normal distribution. A hidden Markov model (HMM) is created, a probability of generating a feature vector by the HMM is calculated, and a facial expression corresponding to the calculated HMM is used as a recognition result.

[0005]

In the proposed method described above, the operation of the HMM starts from an initial state and ends at an end state, so that only one facial expression can be recognized.
In addition, since the distribution of the output feature amounts is similar in the state where the facial muscles are stationary, there is a possibility that a malfunction of the HMM may occur.

[0006] Therefore, the main object of the present invention is to provide H
An object of the present invention is to provide a facial expression extracting method for extracting a section in which a facial expression is expressed by controlling a value of an output probability in the MM according to an occurrence probability of a state to which the output probability is assigned.

[0007]

The invention according to claim 1 is
A method of extracting individual facial expressions from a face moving image, calculating a velocity vector at each position of a face element from a continuous image in the face moving image, performing a Fourier transform on each component of the calculated speed vector, The Fourier transform coefficients are extracted as a feature vector sequence, and the extracted feature vectors are time-integrated to calculate a facial expression movement vector, and the calculated facial expression movement vector is collated with each predetermined facial expression pattern. A plurality of hidden Markov models, in which continuous output probabilities for each expression are previously approximated using a normal distribution, are created, and a generation probability at which a feature vector sequence is generated is calculated based on a plurality of hidden Markov models. And the facial expression corresponding to the hidden Markov model for which the maximum generation probability was calculated among the multiple hidden Markov models was recognized as the recognition result. It is intended to determine Te.

[0008]

FIG. 1 is a flowchart for explaining a process of detecting, extracting and recognizing a facial expression change from a face moving image according to an embodiment of the present invention. In FIG. 1, one embodiment of the present invention includes a preprocessing unit S10, a facial expression change detection unit S20, and an extraction / recognition unit S30. In the preprocessing unit S10, a velocity vector is calculated from the face moving image input in S11, two-dimensional Fourier transform is performed on each component of the velocity vector calculated in S12, and in S13, a feature vector sequence according to the movement of the facial expression Is extracted.

On the other hand, in the facial expression change detecting section S20, S21
Then, the motion vector of the facial expression is calculated by time-integrating the feature vector in step (2), and collation with each predetermined facial expression pattern is performed in S22. In addition, the extraction recognition unit S30
In step S31, an HMM is created in advance for each facial expression category to be recognized by learning.
According to the result of the comparison in 22, the probability of generation of the feature vector sequence is calculated using the HMM. Then, the expression category corresponding to the HMM having the maximum calculated probability is set as the recognition result.

In the following, processing of a time-series image from no expression to each basic expression will be described by considering six types of basic expressions including anger, disgust, fear, sadness, happiness, and surprise as expression categories.

FIG. 2 is a left-to-state looped state.
FIG. 3 is a diagram illustrating a configuration of a Right-type HMM. In one embodiment of the present invention, the type of facial expression and the state of the facial muscle (contraction, relaxation, etc.) are determined by the state of the HMM prepared for each facial expression.
Are distinguished, and the estimation result (probability distribution assigned to each state) is updated based on the feature amount obtained by the image processing.

As shown in FIG. 2, the time change of each expression is L
It is modeled using an HMM of the left-to-right type. Shrinkage S ₂ of mimic muscles each facial expression from expressionless S _1, the end of the contraction S _3, returned to expressionless S ₅ through relaxation S ₄ of facial muscles, this loop is repeated. The probability α _ij (transition probability) that each expression transits to state S _j after a unit time from state S _i
If the probability that a vector O when you are in a state S _i b
_i (O) (output probability density). The transition probability and the output probability density are calculated from a time series of feature vectors obtained by performing image processing on a moving image expressing each expression.
Estimated using elch algorithm.

The image processing uses an image compressed to 1/8 in both the vertical and horizontal directions, and an optical flow is obtained from two continuous images on the time axis. In the distribution of the optical flow, a two-dimensional Fourier transform is applied to the period region of the right eye and the mouth. Is done.

[0014] In expression extraction process, using the following equation (1) to calculate the probability P _i of the state S _i at the expression E _k ^(k) (t), the state S ₃ contraction of facial muscles has been completed Probability P ₃ ^(k)
(T) is the expression E _k when exceeding the threshold value P _a with is determined to have been exposed.

[0015]

(Equation 1)

Here, N is the number of states, a _ik ^(k) , b _i ^(k)
(O) is the transition probability expression E _k, which is the output probability density. The transition probability a _ik ^(k) has a non-zero value only between states connected by arrows in FIG. However, the states S ₁ , S ₃ , and S ₅ are states in which the facial muscles are at rest, so that the output probability densities b ₁ (O),
b ₃ (O) and b ₅ (O) are similar functions, and a transition from the state S ₁ to the state S ₃ occurs due to a slight noise of a facial expression change.

Therefore, in order to eliminate such a transition,
State S_ThreeProbability P_Three ^(k)When calculating (t), P_Two
^(k)Threshold P with value of (t-1)_bIf less than
P _Two ^(k)(T-1) is set to 0. State S_FiveAlso
The same is true. Also, state S_FiveThreshold P with probability_c
When it exceeds, it is regarded as having returned to the expressionless state
Is initialized as in the following equation (2).

[0018]

(Equation 2)

FIG. 3 is a diagram showing an expression sequence, and FIG. 4 is a diagram showing changes in the probability P ₃ ^(k) of the state S ₃ in the expression E _{k with} respect to the expression sequence of FIG.
Is shown.

As a facial expression sequence, six types of basic facial expressions, anger, disgust, fear, sadness, happiness,
They are listed in order of surprise. Here, the different expressions go through a non-expression. From FIG. 3, it can be seen that the individual expressions are accurately extracted. Further, since the change probability is steep, less dependence of the extraction result for the threshold P _a.

As described above, in this embodiment, the type of facial expression and the state of the facial muscle (contraction, relaxation, etc.) are distinguished based on the state of the HMM prepared for each facial expression, and based on the feature amount obtained by image processing. By updating the estimation result (probability distribution assigned to each state), individual facial expressions can be accurately extracted from a sequence of six types of facial expressions.

In the above-described embodiment, only a sequence in which a single expression starts from an expressionless expression and returns to an expressionless expression can be extracted. It is not possible to extract a sequence that changes to another expression without passing through the expression.

Then, next, based on the HMM independently constructed for each facial expression, states corresponding to any two direct changes of the facial expressions and transitions between those states and the original HMM's expressed state A description will be given of an embodiment in which an expression can be correctly extracted even when a change to another expression occurs in the process of expressing one expression by adding.

FIG. 5 is a diagram showing a configuration of an HMM according to another embodiment of the present invention. In FIG. 5, loops HMM ₁ , HMM for extracting sequences in which the first and second categories of facial expressions are independently expressed assuming that the number of states N is 2 in order to extract facial expression changes that do not involve expressionlessness _Two
And a link to a state S ₁₂ (S ₂₁ ) that transits when directly changing from the 1 (2) th category to the 2 (1) th category and a (expression of expression) state S _{3 of} HMM ₁ and HMM ₂ It is composed of

The parameters of the HMM include a loop HM
M_iAre learned from a single facial expression sequence
Parameters are used. However, state S_ThreeFrom
State S _ijSince the link to was added, the original transition probability
a₃₄Is the transition probability a to two states₃₄And a_{3, i, j}Equally divided into two
Is done. State S_ijParameters for a single facial expression
Calculated from the parameters learned from the sequence. This
This allows the use of a sequence of facial expression changes for all combinations.
This eliminates the need to learn parameters.

The output probability distribution is obtained from the following equation (3), assuming that muscle elongation of category i and muscle contraction of category j occur simultaneously.

[0027]

(Equation 3)

Here, μ _j ⁽ⁱ⁾ and σ _j ⁽ⁱ⁾ are the categories j
Are the average and standard deviation of the output distribution density in the state _Sj . On the other hand, as shown in Expressions (4) and (5), the occurrence probability having the longer duration of the muscle elongation or muscle contraction state is selected.

[0029]

(Equation 4)

FIGS. 6 and 7 are diagrams showing an example of the result of extracting a sequence that changes from "happiness" to "surprise" in another embodiment of the present invention. FIG. 7 shows the case where the facial expression change is slow and the state transitions along the loop, while FIG. 6 shows the case where the facial expression change is fast and the muscle contraction and extension occur simultaneously. I have.

[0031]

As described above, according to the present invention, the HM
By controlling the value of the output probability in M according to the occurrence probability of the state to which the output probability is assigned, it is possible to extract the section in which the facial expression is expressed.

[Brief description of the drawings]

FIG. 1 is a flowchart illustrating a process of detecting, extracting, and recognizing a facial expression change from a face moving image according to an embodiment of the present invention.

FIG. 2 Left-to-Rig with state looped
It is a figure showing composition of ht type HMM.

FIG. 3 is a diagram showing an expression sequence.

FIG. 4 is a diagram showing a recognition result of a facial expression for a facial expression sequence and an extracted section;

FIG. 5 is a diagram showing a configuration of an HMM according to another embodiment of the present invention.

FIG. 6 is a diagram illustrating an example of an extraction result when a facial expression changes rapidly according to another embodiment of the present invention.

FIG. 7 is a diagram showing an example of an extraction result when a facial expression change is slow in another embodiment of the present invention.

[Explanation of symbols]

 S10 Preprocessing unit S20 Facial expression change detection unit S30 Extraction / recognition unit

────────────────────────────────────────────────── ───

[Procedure amendment]

[Submission date] March 11, 1999

[Procedure amendment 1]

[Document name to be amended] Statement

[Correction target item name] Claims

[Correction method] Change

[Correction contents]

[Claims]

[Procedure amendment 2]

[Document name to be amended] Statement

[Correction target item name] 0007

[Correction method] Change

[Correction contents]

[0007]

The invention according to claim 1 is
A method of extracting individual facial expressions from a face moving image, calculating a velocity vector at each position of a face element from a continuous image in the face moving image, performing a Fourier transform on each component of the calculated velocity vector, The Fourier transform coefficients are applied to the movement of the facial expression.
Extracted as Flip feature vector sequence, extracted feature vector to calculate the motion vector of the facial expression by integrating the time, performs collation between the facial expression to a predetermined pattern and movement vector of the calculated facial expression, third Steps and
It is processed in parallel and creates a plurality of hidden Markov models in which continuous output probabilities for each facial expression are approximated using a normal distribution in advance, and based on the above-described matching results, a generation probability at which a feature vector sequence is generated is calculated. The expression is calculated using a plurality of hidden Markov models, and an expression corresponding to the hidden Markov model for which the maximum generation probability is calculated among the plurality of hidden Markov models is determined as a recognition result.

Claims (1)

[Claims] 1. A method for extracting individual facial expressions from a face moving image, comprising: a first step of calculating a velocity vector of each position of a face element from a continuous image in the face moving image; A second step of performing a Fourier transform on each component of the velocity vector and extracting the Fourier transform coefficients as a feature vector sequence, and a third step of calculating a movement vector of an expression by time-integrating the extracted feature vector. A fourth step of comparing the calculated movement vector of the facial expression with each predetermined facial expression pattern; and for each facial expression, a plurality of hidden Markov models whose continuous output probabilities are approximated using a normal distribution. A fifth step of creating, based on the result of the matching in the fourth step, the generation probabilities that the feature vector sequence is generated And a seventh step of determining as a recognition result a facial expression corresponding to a hidden Markov model for which a maximum generation probability has been calculated among the plurality of hidden Markov models, as a recognition result. Expression extraction method from moving images.