JP2015219698A - Apparatus, program, and method for estimating emotion based on degree of influence of parts - Google Patents

Abstract

There is provided an emotion estimation device capable of estimating an emotion with respect to a character with high efficiency and easily grasping an estimated emotion position in all possible emotions. The emotion estimation apparatus is configured to determine, for each part, a part emotion value distribution for each part from an emotion evaluation result for each part that can be included in the expression image, and an emotion for the analysis target expression image. From the evaluation result, a means for determining the expression emotion value distribution for this expression image, and the determined expression emotion value distribution obtained by weighting and superimposing the part emotion value distribution for the parts included in this expression image Means for calculating the degree of influence as a weight for each part included in the representation object image to be analyzed, and the expression emotion value distribution for the estimation object representation image, each part included in the expression image And means for calculating based on the part emotion value distribution and the degree of influence calculated for each part. [Selection] Figure 2

Description

  The present invention relates to a technique for estimating an emotion expressed by an expression such as a character.

  Expressions such as characters displayed as images can play an extremely important role in communication with the user because they can express internal states and emotions in the same way as human actual expressions. Expressions rich in variations expressed in images are an important factor in providing high quality services even in conversation agents and animations using visual characters.

  Conventionally, various techniques have been devised for estimating human emotions. For example, Patent Document 1 discloses a technique for estimating a psychological state from facial expressions of a human face. In this technique, first, the shapes of the mouth and the left and right eyes in a human face image are quantitatively expressed using Fourier descriptors. Next, using the linear discriminant using the Fourier descriptor, the determination score for a plurality of preset basic emotions is calculated and normalized for each shape of the mouth and the left and right eyes. Finally, the certainty of emotion for each basic emotion for each shape is calculated by applying a multiple regression equation that takes into account the importance for each facial expression element, and the human psychological state is estimated from the result.

  Patent Document 2 discloses an emotion estimation technique for recognizing the facial expression of an unspecified person and estimating the emotion of the person. In this technique, a facial expression image of a specific person associated with a predetermined emotion is learned to generate a facial expression map using a neural network. Next, the generated facial expression map is divided into a plurality of regions based on a predetermined emotion, and the emotion of the unspecified person is estimated based on the facial expression image of the unspecified person and the plurality of facial expression maps having the divided regions. To do.

JP 2007-65969 A JP 2008-146318 A

  However, the conventional techniques disclosed in Patent Documents 1 and 2 are limited to emotion estimation from human facial expressions, and are not suitable for facial expressions such as characters displayed as images, for example.

  Actually, in the techniques disclosed in Patent Documents 1 and 2, it is not assumed that emotions are imparted to the model data of the visual character. For example, these techniques require a measurement process for converting facial expressions into feature quantities in a face image to be estimated for emotion. However, in the case of a visual character in which facial expression model data is set in advance, this process becomes a heavy burden. Thus, the application of the above-described prior art to the visual character becomes very inefficient.

  Furthermore, in the prior art as shown in Patent Documents 1 and 2, emotions are classified in advance into a plurality of basic emotions and divided areas, and emotions are estimated in a classification problem. Therefore, for example, when designing facial expressions and facial expression changes, the position of the emotion value for a certain facial expression and the direction of the change are recognized, and the facial expression representing the desired emotion is determined among all possible emotions. It ’s difficult. As a result, it becomes very difficult to express subtle emotions or their transitions by facial expressions or their changes.

  Therefore, the present invention can estimate emotions with high efficiency even for characters for which facial expression model data is set in advance, and easily grasps the estimated emotion position in all possible emotions. An object is to provide a possible emotion estimation apparatus, program, and method.

According to the present invention, there is an emotion estimation device for estimating an emotion expressed in an expression body image including a plurality of parts,
Part emotion distribution determining means for determining, for each of the plurality of parts, a part emotion value distribution, which is a distribution of emotion values for the part, from a plurality of emotion evaluation results for each of the plurality of parts that can be included in the expression image. When,
Expression emotion distribution determination means for determining an expression emotion value distribution, which is a distribution of emotion values for the analysis target expression image, from a plurality of emotion evaluation results for the analysis target expression image,
The determined expression body emotion value distribution is a result of weighting and superimposing the part emotion value distributions on the parts included in the analysis object expression image, and the distribution of the parts included in the analysis object expression image. For each, an influence degree calculating means for calculating an influence degree as a weight,
The expression emotion value distribution for the estimation target expression image is calculated for each part emotion value distribution for each part included in the estimation target expression image and for each part included in the estimation target expression image. An emotion estimation device having emotion value calculation means for calculating based on the degree of influence obtained is provided.

In the emotion estimation device according to the present invention, the part emotion distribution determining means determines an average value in the part emotion value distribution for the part,
The expression body emotion distribution determining means determines an average value in the expression body emotion value distribution for the expression body image to be analyzed,
The influence degree calculation means assumes that the average value in the expression body emotion value distribution is a result of weighting and combining the average values in the part emotion value distribution with respect to parts included in the expression body image to be analyzed. For each part contained in the expression image of, calculate the average value influence as a weight in the average value,
The emotion value calculation means includes an emotion value for the estimation target expression image included in the average value in the part emotion value distribution for each part included in the estimation target expression image and the estimation target expression image. It is also preferable to calculate based on the average value influence degree calculated for each part.

Furthermore, in the emotion estimation apparatus according to the present invention, the part emotion distribution determining means determines a variance covariance matrix in the part emotion value distribution for the part,
The expression emotion distribution determination means determines a variance covariance matrix in the expression emotion value distribution for the analysis target expression image,
The degree of influence calculation means that the variance covariance matrix in the expression body emotion value distribution is a result of weighted and combined with the variance covariance matrix in the part emotion value distribution for the parts included in the analysis subject expression image , For each of the parts included in the expression image to be analyzed, calculate the variance-covariance matrix influence as a weight in the variance-covariance matrix,
The emotion value calculation means determines the degree of variance of the emotion value for the estimation target expression image, the variance covariance matrix in the part emotion value distribution for each of the parts included in the estimation target expression image, and the estimation target It is also preferable to calculate based on the variance-covariance matrix influence degree calculated for each part included in the representation image.

  Further, as one embodiment of the emotion estimation apparatus of the present invention, for a plurality of expression images whose calculated emotion values are within a predetermined range in the emotion space, the expression image having a smaller calculated variance-covariance matrix is more It is also preferable to further include a representational image presenting means for presenting with high priority.

  Furthermore, as one embodiment of the influence degree calculating means, the influence degree calculating means uses a value that defines the expression body emotion value distribution for the analysis target expression image as an objective variable, and is included in the analysis target expression image. It is also preferable to perform a multiple regression analysis using a value defining the part emotion value distribution for each of the parts to be explained as an explanatory variable, and to calculate the influence as a coefficient of the explanatory variable.

As another embodiment of the influence degree calculating means, the influence degree calculating means includes a value x that defines a part emotion value distribution for each of the parts included in the expression image of the analysis target, and a representation of the analysis target. A set of the value y that defines the expression emotion value distribution for the body image is a sample (x, y), and the residual r between the sample and the regression curve f (x) = x · w + b (b is a constant term) and ε tolerance xi] (r), the weight w and a predetermined function including Σξ in the regression curve (y-f (x)) + λ | w | 2/2 on the basis of the weight w for minimizing a degree of influence It is also preferable to calculate.

  Furthermore, as one embodiment of the part emotion distribution determining unit, the part emotion distribution determining unit is configured to select a part corresponding to the emotion evaluation target part in the standard expression image as a reference for the emotion evaluation. It is also preferable to determine the part emotion value distribution for the emotion evaluation target part from a plurality of emotion evaluation results for the part evaluation image replaced with.

  Further, in the emotion estimation apparatus according to the present invention, the expression emotion value distribution of a new expression image generated using a new part in which the part emotion value distribution and the degree of influence are defined is a new emotion evaluation test. It is also preferred that it be generated automatically without the need.

According to the present invention, there is further provided a program for causing a computer mounted on an apparatus for estimating an emotion expressed in an expression image including a plurality of parts to function.
Part emotion distribution determining means for determining, for each of the plurality of parts, a part emotion value distribution, which is a distribution of emotion values for the part, from a plurality of emotion evaluation results for each of the plurality of parts that can be included in the expression image. When,
Expression emotion distribution determination means for determining an expression emotion value distribution, which is a distribution of emotion values for the analysis target expression image, from a plurality of emotion evaluation results for the analysis target expression image,
The determined expression body emotion value distribution is a result of weighting and superimposing the part emotion value distributions on the parts included in the analysis object expression image, and the distribution of the parts included in the analysis object expression image. For each, an influence degree calculating means for calculating an influence degree as a weight,
The expression emotion value distribution for the estimation target expression image is calculated for each part emotion value distribution for each part included in the estimation target expression image and for each part included in the estimation target expression image. An emotion estimation program for causing a computer to function as emotion value calculation means for calculating based on the degree of influence obtained is provided.

According to the present invention, there is further provided a method for estimating an emotion expressed in an expression image including a plurality of parts,
From a plurality of emotion evaluation results for each of a plurality of parts that can be included in the expression image, for each of the plurality of parts, a part emotion value distribution that is a distribution of emotion values for the part is determined and analyzed. A first step of determining an expression emotion value distribution, which is a distribution of emotion values for the analysis target expression image, from a plurality of emotion evaluation results for the expression image of
The determined expression body emotion value distribution is a result of weighting and superimposing the part emotion value distributions on the parts included in the analysis object expression image, and the distribution of the parts included in the analysis object expression image. For each, a second step of calculating the influence as a weight;
The expression emotion value distribution for the estimation target expression image is calculated for each part emotion value distribution for each part included in the estimation target expression image and for each part included in the estimation target expression image. And a third step of calculating based on the degree of influence.

  According to the emotion estimation apparatus, program, and method of the present invention, it is possible to estimate an emotion with high efficiency even for a character for which facial expression model data is set in advance. It is possible to easily grasp the estimated emotion position.

It is a schematic diagram which shows the production | generation of the facial expression image in one Embodiment of the emotion estimation method by this invention. It is a functional block diagram which shows the function structure in one Embodiment of the emotion estimation apparatus by this invention. It is an example of the part data preserve | saved at the part data preservation | save part. It is a schematic diagram which shows one Example of a facial expression image. It is another example of the part data preserve | saved at the part data preservation | save part. It is a schematic diagram which shows the other Example of a facial expression image. It is a schematic diagram which shows one Example of the emotion space which prescribes | regulates an emotion value. It is a schematic diagram which shows one Example of the standard facial expression image which prescribes | regulates the origin of emotion space. It is a schematic diagram which shows the Example of the image for parts evaluation used for a subjective emotion evaluation experiment. It is a graph which shows the result of the subjective emotion evaluation experiment with respect to the facial expression image which combined the face part used as the object of subjective emotion evaluation in FIG. It is a schematic diagram for demonstrating the function of a facial expression image presentation part.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a schematic diagram showing generation of a facial expression image in an embodiment of an emotion estimation method according to the present invention.

  According to FIG. 1, in the present embodiment, a facial expression image 2 as an expression image that can express an emotion is generated from a base part 21 and a plurality of facial parts 22. The base part 21 is a part that defines the outline of the face, and the facial expression image 2 is generated by superimposing the face parts 22 on the base part 21. Each of the plurality of face parts 22 is face constituent part data corresponding to a plurality of parts constituting the face (skin color, eyes, mouth, and eyebrows in FIG. 1).

The emotion estimation method of the present invention is a method for estimating an emotion expressed in an expression body image (face expression image 2) including a plurality of parts (face parts 22). Specifically, first,
(A) From a plurality of emotion evaluation results for each of a plurality of parts (face part 22) that can be included in the expression body image (face expression image 2), for each of the plurality of parts, distribution of emotion values for the part Determine a “parts emotion value distribution”,
(A) From a plurality of emotion evaluation results for the expression body image (face expression image 2) to be analyzed, an “expression body emotion value distribution” that is a distribution of emotion values for the expression body image to be analyzed is determined.

  Here, the emotion evaluation result is acquired, for example, as a result of an emotional subjective evaluation experiment. This subjective evaluation experiment is an experiment in which an evaluator records emotions felt when he / she views each of a plurality of parts (or an expression image) as emotion values. In this case, the plurality of emotion evaluation results are a set of a plurality of emotion values recorded by a plurality of evaluators who viewed one part (or expression image) that is an evaluation target. From this set of emotion values, a “part emotion value distribution (expression body emotion value distribution)” as an emotion value distribution in the emotion space is determined.

Then
(C) The determined “expression body emotion value distribution” is a result of weighting and superimposing the “part emotion value distribution” on the parts included in the analysis object image of the analysis object. For each part included in the body image, an “influence” as a weight is calculated.

  For example, it is preferable that the “expression body emotion value distribution” of the facial expression image 2 is regarded as a result of convolution of the “part emotion value distribution” of each of the facial parts 22 constituting the facial expression image 2. The “influence” of each face part 22 is calculated as a weight when convolving the “part emotion value distribution” of the face part.

Finally,
(D) “Expression emotion value distribution” for the expression object image (face expression image 2) to be estimated is assigned to each part (face part 22) included in the expression object image (face expression image 2) to be estimated. The calculation is performed based on the “part emotion value distribution” and the “degree of influence” calculated for each part (face part 22) included in the expression image to be estimated.
Here, the calculated “expression body emotion value distribution” is the emotion estimation result of the expression body image (face expression image 2) to be estimated.

  Thus, according to the present invention, when estimating the emotion of the expression body image (face expression image 2), for example, a plurality of basic emotions are set in advance, and the expression body image is classified into which category of basic emotion. The emotion is not estimated as a heel classification problem. Instead, in the present invention, first, “expression emotion value distribution” and “part emotion value distribution” indicating the distribution of emotion values in the emotion space are determined.

  Next, the “degree of influence” of each part (face part 22) connecting both distributions is calculated. Thereby, the “part emotion value distribution” and the “influence degree” of each part (face part 22) included in the expression object image (face expression image 2) to be estimated are determined, and by using these, an emotion estimation result is obtained. The “expression body emotion value distribution” can be calculated.

  In this way, the emotion estimation method captures the expression body image (face expression image 2) and each part (face part 22) in association with the distribution of emotion values in the emotion space. It is not necessary to carry out the process. Accordingly, for example, high affinity can be exhibited for application to a visual character in which facial expression model data is set in advance, and emotion can be estimated with higher efficiency.

  Furthermore, since the emotion is estimated in the form of the emotion value distribution, it is possible to easily grasp the position of the estimated emotion in all possible emotions. Therefore, it is also possible to define the emotion strength in the expression image or to define the emotion similarity by defining the emotion distance between the expression images.

  Furthermore, in this emotion estimation method, the “expression emotion value distribution” of the new expression image generated using the new parts defined with “part emotion value distribution” and “influence” is a new one. Automatically generated without the need for an emotional assessment test. As a result, by combining various parts in various ways, it is possible to easily provide a wide variety of facial expressions in which the “expression body emotion value distribution” is defined.

  The base part 21 may be included in the parts for determining the “part emotion value distribution”. However, in this embodiment, only the face part 22 is determined for the “part emotion value distribution”. In this case, the “influence degree” calculated for the face part 22 can be interpreted as a value obtained when one base part 21 is selected. Furthermore, the representation image is not limited to that based on two-dimensional (2D) computer graphics, and may be based on, for example, three-dimensional computer graphics (3DCG).

  Of course, the form of the parts is not limited to that described above. For example, as a base part, a part such as a face, upper body, or whole body of a body model such as a female model, a male model, or a mascot model may be employed. Furthermore, as a part, it is also possible to take a unit in which a plurality of body parts such as “eyes and eyebrows” are combined into one. Furthermore, in addition to the CG image, a photographic image or a video frame portion may be adopted as the part. Further, the parts are not limited to data corresponding to the body part, and may be data corresponding to clothes, personal belongings, backgrounds, and the like. That is, the part can take various forms as long as it is a part of an expression (object) capable of expressing emotion.

  FIG. 2 is a functional block diagram showing a functional configuration in an embodiment of the emotion estimation apparatus according to the present invention.

  According to FIG. 2, the information terminal 1 as an emotion estimation device includes a communication interface 101, a parts data storage unit 102, an emotion evaluation result storage unit 103, a facial expression emotion distribution storage unit 104, and a part emotion distribution storage unit. 105, a touch panel display 106, and a processor memory. Here, the processor memory realizes an emotion estimation function by executing a program for causing a computer mounted on the information terminal 1 to function.

  The processor memory includes a facial expression generation / determination unit 111, a part emotion distribution determination unit 113, a facial expression emotion distribution determination unit 114, an influence calculation unit 115, a facial expression emotion value calculation unit 116, a face It functions as a facial expression image presentation unit 117, an input / output control unit 121, and an application processing unit 122. Furthermore, it is preferable to function as the emotion evaluation test execution unit 112. Here, according to FIG. 2, the flow of processing in which the functional components are connected by arrows is understood as an embodiment of the emotion estimation method according to the present invention.

  The information terminal 1 is an information processing apparatus such as a smartphone, a tablet computer, a notebook computer or a personal computer (PC), a set top box (STB), a television (TV), and the like. The visual character displayed on the touch panel display 106 may be, for example, a conversation agent that shows a variety of expressions according to the state of dialogue. Further, the information terminal 1 may be a stand-alone device that does not include the communication interface 101. In this case, the data of the face part 22 and the result data of the subjective emotion evaluation experiment may be stored in advance in the part data storage unit 102 and the emotion evaluation result storage unit 103, respectively.

The communication interface 101 is a communication function unit of the information terminal 1 and executes data exchange with a server or the like via, for example, a base station or an access point. In this embodiment, in particular,
(A) Data of parts such as the face part 22 and the base part 21 is received from the outside in a form associated with these identifiers (IDs) and stored in the part data storage unit 102;
(B) The result of the subjective emotion evaluation experiment conducted outside is received from the outside in a form associated with the identifier (ID) of the facial expression image 2 or face part 22 to be evaluated, and the emotion evaluation result is stored. It is also preferable to store in the unit 103. The communication interface 101 may further transmit the result of the subjective emotion evaluation experiment performed on the information terminal 1, the data of the parts and facial expression images generated on the information terminal 1, etc. to the outside.

  The part data storage unit 102 stores data of parts such as the face part 22 and the base part 21.

  FIG. 3 is an example of part data stored in the part data storage unit 102.

  The part data shown in FIG. 3 is 3DCG data. In the present embodiment, in the part data storage area of the part data storage unit 102, the part data is stored separately for each type of parts such as base part data, eye part data, mouth part data, and the like. Further, for example, in the storage area of the base part data, individual base part data (base part data 1,...) Is distinguished and stored. Data 1,... Are stored separately.

  One part data has “material”, “surface”, and “reference position” as attribute items. Here, attributes defined by “material” are a texture image, a set value of a physical effect, and the like. The attributes defined by “surface” are the relative positions of all the vertices constituting the part, the corresponding positions of the vertices in the texture image, and the like.

  Furthermore, a “reference position” for adding face parts such as eye parts and eyebrow parts is set in the base part. In addition, the “reference position” corresponding to the “reference position” of the base part is also set for the face part to be added. When a face part is added to a base part, scaling and alignment are performed so that the corresponding “reference position” of the face part matches the “reference position” of the base part. Synthesized with parts.

  FIG. 4 is a schematic diagram showing an example of a facial expression image.

  In the 3DCG facial expression image 2 shown in FIG. 4, an eyebrow part 22 which is a face part is added on the base part 21. At this time, the “reference position” (vertex position) of the eyebrow part 22 is linearly converted to match the “reference position” as the eyebrow part addition position set in the base part 21.

  FIG. 5 is another example of part data stored in the part data storage unit 102. FIG. 6 is a schematic diagram showing another embodiment of the facial expression image 2.

  The part data shown in FIG. 5 is 2D data. This part data is also stored in the part data storage area of the part data storage unit 102, divided into parts such as base part data, eye part data, and mouth part data. Further, for example, in the storage area of the base part data, individual base part data (base part data 1,...) Is distinguished and stored. Data 1,... Are stored separately.

  One part data has “image data” and “reference position” as attribute items. The “image data” is data of the parts constituting a 2D facial expression image as shown in FIG. 6, for example. Also in the facial expression image 2 shown in FIG. 6, an eyebrow part 22 that is a face part is added on the base part 21. At this time, the “reference position” (vertex position) of the eyebrow part 22 is matched with the “reference position” as the eyebrow part addition position set in the base part 21.

  As described above, the facial expression image generation method described with reference to FIGS. 3 to 6 is characterized in that the “reference position” is set as the part data. That is, instead of moving the vertices of the polygon to form a facial expression as in the prior art, a facial expression image is generated by directly arranging facial parts with specified emotion values. This makes it possible to easily provide various types of facial expressions whose emotion values are defined (estimated).

  Returning to FIG. 2, the facial expression generation / determination unit 111 generates or determines the facial expression image 2 as an analysis target for calculating the “influence”. At this time, for example, the facial expression image 2 may be generated by extracting necessary parts from the base part 21 and the face part 22 stored in the part data storage unit 102. Alternatively, a desired facial expression image 2 stored in the facial expression emotion distribution storage unit 104 may be selected and determined as the facial expression image 2 to be analyzed. Further, such generation / determination of the facial expression image 2 is preferably executed according to a generation / determination instruction by the user via the touch panel / display 106 and the input / output control unit 121, for example.

The emotion evaluation test execution unit 112 performs a subjective emotion evaluation test. Here, there are two types of subjective emotion evaluation tests: an evaluation test for the facial expression image 2 and an evaluation test for the face part 22 (and the base part 21). In the evaluation test of facial expression image 2, for example,
(A) The facial expression image 2 generated or determined by the facial expression generation / determination unit 111 is displayed on the touch panel display 106 and presented to the evaluator (user).
(B) The emotion felt when the evaluator looks at the displayed facial expression image 2 is input as an emotion value via the touch panel display 106 (input screen displayed on the touch panel display 106).
(C) It is also preferable to record a plurality of emotion value data input from a plurality of evaluators (users) in the emotion evaluation result storage unit 103. Here, as a change mode, a part (or all) of emotion value data by a plurality of evaluators is received from the outside via the communication interface 101 and directly recorded in the emotion evaluation result storage unit 103 (the facial expression) It may be emotion value data (for image 2). Moreover, the number of evaluators can be set to 30 or more, for example.

  On the other hand, in the evaluation test for the face part 22, for example, the part evaluation image 4 described later with reference to FIG. 9 is used as the facial expression image presented in the step (a), and the steps (b) and ( The processing can be performed in the same manner as in c).

  FIG. 7 is a schematic diagram showing an example of an emotion space that defines an emotion value.

  The emotion space shown in FIG. 7 is a two-dimensional space based on the Russell ring model and having two axes, a pleasant-unpleasant axis and an active-inactive axis. One emotion is represented by a point on this space, that is, an emotion value taking a two-dimensional coordinate value. For example, in the emotional space of FIG. 7, an emotion value (3, -3) having a pleasant-unpleasant value of “3” and an active-inactive value of “−3” indicates a very relaxed emotional state. can do. In this way, by using this emotion space, one emotion can be expressed not as a discrete value but as a continuous value, and a change in emotion can also be expressed as a continuous curve in the emotion space. Further, for example, in the above step (b) of the subjective emotion evaluation test, the user can easily input the emotion felt by looking at the facial expression image 2 by designating one point on the emotion space. it can.

  The model expressing the emotion value is not limited to the aforementioned Russell's ring model. For example, three or more axes corresponding to attributes that can evaluate three or more aspects of emotion may be set, and the emotion value may be expressed as a point on a three-dimensional space or more. It is also possible to define emotion values by selectively enumerating emotion types and giving them strengths.

  FIG. 8 is a schematic diagram showing an example of a standard facial expression image that defines the origin of the emotion space.

とする正規分布に従うように設定される。ここで、αは任意の定数である。実際には、設定した標準顔表情画像３に対する主観感情評価実験の結果が、このような正規分布となるように、感情空間の座標値（の絶対値）を決定してもよい。 According to FIG. 8, as the standard facial expression image 3, an image that is combined (superimposed) on the base part 21 by combining standard facial parts 22 is generated. For this standard facial expression image 3, the distribution of a plurality of emotion values obtained when a plurality of evaluators performed a subjective emotion evaluation experiment on this image has an average value μ as the origin (0, 0). , The variance-covariance matrix Σ

Is set to follow a normal distribution. Here, α is an arbitrary constant. Actually, the coordinate value (absolute value) of the emotion space may be determined so that the result of the subjective emotion evaluation experiment on the set standard facial expression image 3 has such a normal distribution.

  Returning to FIG. 2, the parts emotion distribution determination unit 113 determines the plurality of faces based on the results of a plurality of subjective emotion evaluation experiments for each of the plurality of face parts 22 that can be included in the facial expression image (expression body image) 2. For each part 22, a “part emotion value distribution” which is a distribution of emotion values for the face part 22 is determined. Here, the part emotion distribution determining unit 113 defines the “part emotion value distribution” as a two-dimensional normal distribution N (μ, Σ) having an average (center) μ and a variance-covariance matrix Σ, and an average value μ It is also preferable to determine the variance covariance matrix Σ.

  FIG. 9 is a schematic diagram showing an example of a part evaluation image used in a subjective emotion evaluation experiment.

  FIGS. 9A to 9D show parts evaluation parts used in subjective emotion evaluation experiments for the “skin color” part, the “eye” part, the “mouth” part, and the “brow” part as the face part 22, respectively. Image 4 is shown. The part evaluation image 4 is an image obtained by replacing only the facial parts that are subject to subjective evaluation in the standard facial expression image 3 (for example, shown in FIG. 8) with the corresponding facial parts in the standard facial expression image 3. . For example, when the “eye” part is the subject of subjective evaluation, as shown in FIG. 9B, the “eye” part of the subject of subjective evaluation is assigned instead of the “eye” part of the standard facial expression image 3. The part evaluation image 4 is generated.

  In the subjective emotion evaluation experiment for the face part 22, the standard facial expression image 3 and the part evaluation image 4 described above are arranged side by side and presented to the evaluator. It is also preferable to record an emotion value to be felt as a change from the facial expression image 3. In this case, in other words, the emotion value for the face part 22 to be evaluated is the value of the emotion that can be felt when only the face part portion is changed to the face part 22 with respect to the standard facial expression image 3. Become.

  FIG. 9C shows a “part emotion value distribution” as a result of the subjective emotion evaluation experiment for the “mouth” part. The “part emotion value distribution” is a distribution estimated from emotion values evaluated and determined by the plurality of evaluators on the part evaluation image 4 of the “mouth” part. In this way, the parts emotion distribution determination unit 113 (FIG. 2) performs a plurality of processing on images obtained by replacing the parts corresponding to the emotion evaluation target parts in the standard facial expression image serving as the reference for emotion evaluation with the emotion evaluation target parts. "Parts emotion value distribution" is determined from the emotion evaluation results.

を用いて算出される。 Here, a method for calculating the variance / covariance matrix Σ of the “part emotion value distribution” from the emotion values acquired in the subjective emotion evaluation experiment will be described. First, it is defined that the “part emotion value distribution” corresponding to the face part data i is a normal distribution _Ni (μ _i , Σ _i ). Next, the two-dimensional emotion value evaluated by the evaluator j (j = 1, 2,..., N) is expressed as x _ij = [x ¹ _ij , x ² _ij ] and mean value μ _i = [μ ¹ _i , μ ² _i ], the variance-covariance matrix Σ _i is given by

Is calculated using

  Incidentally, the greater the value of the variance-covariance matrix Σ, the smaller the relevance indicating the peak height of the emotion value distribution, that is, the number of distributions (distribution probability). This indicates that the probability of taking the center (average) value of this emotion value distribution is small.

  Returning to FIG. 2, the facial expression emotion distribution determination unit 114 calculates the distribution of emotion values for the facial expression image 2 to be analyzed from a plurality of subjective emotion evaluation results for the facial expression image (expression body image) 2 to be analyzed. A certain “expression body emotion value distribution” is determined. Here, it is also preferable that the facial expression emotion distribution determination unit 114 determines the average value μ and / or the variance-covariance matrix Σ as the “expression body emotion value distribution”.

  FIG. 10 is a graph showing the result of a subjective emotion evaluation experiment on the facial expression image 2 in which the facial parts 22 that are the subject of the subjective emotion evaluation in FIG. 9 are combined.

  The facial expression image 2 shown in FIG. 10 is an image generated by adding all the facial parts 22 selected in FIG. Similarly, the emotion value distribution shown in FIG. 10 is obtained by obtaining a plurality of emotion values expressing emotions felt by a plurality of evaluators from the facial expression image 2 and determining “expression” determined from these emotion values. "Body emotion value distribution". Here, the variance-covariance matrix Σ of the “expression body emotion value distribution” can also be calculated using the same formulas as the formulas (2) to (4) described above.

  As described above, the facial expression emotion distribution determination unit 114 (FIG. 2) performs “expression body emotion value” for the facial expression image 2 generated using the facial part 22 for which “part emotion value distribution” is determined. Distribution "is determined. Here, from these determined “part emotion value distribution” and “expression body emotion value distribution”, the “influence degree” of each face part can be calculated as described below.

  In addition, as a change mode of the embodiment of the emotion value distribution determination described above, both the “part emotion value distribution” and the “expression body emotion value distribution” are not based on the subjective evaluation experiment, but the system It is also possible for a designer or the like to set in advance. In this case, the certainty of the estimation of the emotion for the facial expression depends on the content of the setting.

  Returning to FIG. 2, the influence calculation unit 115 determines that the determined “expression body emotion value distribution” is a result of convolution of the “part emotion value distribution” with respect to the face part 22 included in the facial expression image 2 to be analyzed. If there is, the “influence” as the weight is calculated. Here, the “influence” is calculated for each of the facial parts 22 included in the facial expression image 2 to be analyzed. It is also preferable to store the calculated “influence” in the part emotion distribution storage unit 105 in association with the corresponding part. Furthermore, as the “influence degree”, for example, two types of “average value influence degree” and “dispersion-covariance matrix influence degree” can be set.

  Specifically, the influence calculation unit 115 weights the average value μ in the “part emotion value distribution” for the face part 22 included in the facial expression image 2 to be analyzed, with the average value μ in the “expression body emotion value distribution”. It is also preferable to calculate the “average value influence degree” as a weight at the average value μ for each of the facial parts 22 included in the facial expression image 2 to be analyzed. In addition, the result of weighted combination of the variance covariance matrix Σ in the “expression body emotion value distribution” with the variance covariance matrix Σ in the “part emotion value distribution” for the face part 22 included in the facial expression image 2 to be analyzed Therefore, it is also preferable to calculate the “variance covariance matrix influence” as a weight in the variance covariance matrix Σ for each of the facial parts 22 included in the facial expression image 2 to be analyzed.

  Here, when calculating the convolution of the “part emotion value distribution”, the “part emotion value distribution” of each face part 22 is defined as a normal distribution as described above. Furthermore, from the reproducibility of the normal distribution, the convolution of the normal distribution becomes a normal distribution, and the average value μ and the variance covariance matrix Σ generated by the convolution are the average value μ and the variance covariance of the original normal distribution, respectively. Use the fact expressed as a linear combination of the matrix Σ.

を用いて算出される。ここで、μ_k（k＝1,2,・・・,m）及びΣ_k（k＝1,2,・・・,m）はそれぞれ、顔パーツデータｋに対応する「パーツ感情値分布」における平均値及び分散共分散行列である。また、β_k（k＝1,2,・・・,m）及びγ_k（k＝1,2,・・・,m）はそれぞれ、顔パーツデータｋの「平均値影響度」及び「分散共分散影響度」である。さらに、ｃ及びｄは定数である。 Specifically, when the facial expression image 2 to be analyzed is composed of m facial parts 22, the average value μ of the “expression body emotion value distribution” N (μ, Σ) corresponding to the facial expression image 2 and The variance-covariance matrix Σ is given by

Is calculated using Here, μ _k (k = 1, 2,..., M) and Σ _k (k = 1, 2,..., M) are “part emotion value distribution” corresponding to the face part data k, respectively. Mean value and variance-covariance matrix. Β _k (k = 1, 2,..., M) and γ _k (k = 1, 2,..., M) are respectively “average value influence degree” and “variance” of the face part data k. Covariance influence ”. Furthermore, c and d are constants.

In this way, the average value influence degree β _k (variance covariance influence degree γ _k ) of the face part data k is _{obtained by} changing the average value μ (variance covariance matrix Σ) of the “expression body emotion value distribution” to μ _k (Σ _k ), The weighting coefficient when expressed by the linear combination, and exactly indicates the degree of influence of the face part data k on the emotional value of the facial expression.

となり、具体的な「影響度」の算出例として、例えば、

が得られる。 More specifically, when the facial expression image 2 is composed of four face parts 22 of skin color, eyes, mouth, and eyebrows, the above equation (5) is

As a specific example of calculating the “influence”, for example,

Is obtained.

Here, the mean value influence β _k, variance covariance influence γ _k , and c and d of the face part data k are the “part emotion value distribution” for each face part data k and the facial expression image 2 to be analyzed. It can be obtained recursively from the “expression body emotion value distribution”.

Specifically, the influence degree calculation unit 115 (FIG. 2) uses a value (for example, μ or Σ) that defines “expression body emotion value distribution” for the facial expression image (expression body image) 2 to be analyzed as an objective variable, A multiple regression analysis is executed using a value (for example, μ _k or Σ _k ) defining “part emotion value distribution” for each of the facial parts 22 (face part data k) included in the facial expression image 2 to be analyzed as an explanatory variable. It is also preferable to calculate the “influence” (for example, β _k or γ _k ) as the coefficient of the explanatory variable. This multiple regression analysis can be executed using, for example, known multiple regression analysis software.

Furthermore, as another embodiment of calculating the “influence”, it is also possible to perform nonlinear regression analysis such as support vector regression instead of the linear regression analysis described above, and learn the weight to calculate the “influence”. preferable. For example, the influence calculation unit 115 (FIG. 2) first calculates a “part emotion value distribution” for each of the facial parts 22 (face part data k) included in the facial expression image (expression image) 2 to be analyzed. A set of a specified value x and a value y specifying “expression body emotion value distribution” for the facial expression image 2 to be analyzed is a sample (x, y). Here, x may be an m-dimensional vector [μ ₁ ,..., Μ _m ] or [Σ ₁ ,..., Σ _m ], and y may be an average value μ or a variance covariance matrix Σ. preferable. In this case, the relationship between x and y of the sample (x, y) is defined by the above equation (5).

のように取り決め、このε許容誤差ξ(ｒ)と、当該回帰曲線における重みｗとを含む所定の関数を最小化する最小化問題を解いて「影響度」を算出する。具体的には、次式

を満たす重みｗに基づいて、「影響度」を算出する。ここで、λは正則化パラメータである。この最適化問題を解くことで得られるｗ及びｂによって、各顔パーツｋの平均値影響度β_k又は分散共分散影響度γ_k並びにｃ及びｄ（定数項）を決定する。尚、以上に述べたサポートベクタ回帰を用いた「影響度」の算出も、例えば、公知のサポートベクタ回帰分析用ソフトウェアを用いて実行することができる。 Next, the ε allowable error ξ (r) of the residual r between the sample (x, y) and the regression curve f (x) = x · w + b (b is a constant term) is expressed by the following equation:

The “influence” is calculated by solving a minimization problem that minimizes a predetermined function including the ε allowable error ξ (r) and the weight w in the regression curve. Specifically, the following formula

The “influence” is calculated based on the weight w satisfying the above. Here, λ is a regularization parameter. Based on w and b obtained by solving this optimization problem, the average value influence β _k or variance-covariance influence γ _k and c and d (constant terms) of each face part k are determined. The calculation of the “influence” using the support vector regression described above can also be executed using, for example, known support vector regression analysis software.

  Returning to FIG. 2, the facial expression emotion value calculation unit 116 calculates the “expression body emotion value distribution” for the facial expression image (expression body image) 2 to be estimated as a facial part included in the facial expression image 2 to be estimated. It is calculated based on the “part emotion value distribution” for each of 22 and the “influence degree” calculated for each of the face parts 22 included in the facial expression image 2 to be estimated.

  Specifically, the facial expression emotion value calculation unit 116 calculates an emotion value for the facial expression image 2 to be estimated as an average value μ in the “part emotion value distribution” for each of the facial parts 22 included in the facial expression image 2. It is also preferable to calculate based on the “average value influence degree” calculated for each of the facial parts 22 included in the facial expression image 2. Further, the degree of dispersion of the emotion value with respect to the facial expression image 2 to be estimated is represented by the variance covariance matrix Σ in the “part emotion value distribution” for each of the facial parts 22 included in the facial expression image 2, and the facial expression image. It is also preferable to calculate based on the “variance / covariance matrix influence” calculated for each of the face parts 22 included in 2.

  Here, the identification of the facial expression image 2 to be estimated may be executed by receiving, for example, a designation input of the estimation target image by the user via the touch panel display 106 and the input / output control unit 121. Alternatively, the facial expression emotion value calculation unit 116 may directly use the facial expression image 2 generated or determined by the facial expression generation / determination unit 111 as an estimation target. In this case, the facial expression generation / determination unit 111 takes out the facial parts defined in the emotion value distribution stored in the part emotion distribution storage unit 105 and generates a new facial expression image 2 as an emotion value estimation target. . Further, the facial expression emotion value calculation unit 116 may select one of the facial expression images stored in the facial expression emotion distribution storage unit 104 as an estimation target.

  Furthermore, as another embodiment, when a new facial expression image is generated using a new facial part in which “part emotion value distribution” and “influence” are defined, the facial expression emotion value calculation unit 116 A “expression body emotion value distribution” of a new facial expression image can be automatically generated. That is, in this case, the facial expression emotion value calculation unit 116 can acquire all the “part emotion value distributions” and “influences” of the constituent parts, so that a new emotion evaluation test should be performed. It is possible to calculate the “expression body emotion value distribution”.

  Similarly, according to FIG. 2, the facial expression image presentation unit 117 calculates the calculated variance covariance matrix for a plurality of facial expression images (expression images) 2 in which the calculated emotion values are within a predetermined range in the emotion space. A facial expression image with a smaller Σ is presented with a higher priority.

  FIG. 11 is a schematic diagram for explaining the function of the facial expression image presenting unit 117.

  The facial expression image presenting unit 117 (FIG. 2) displays an emotion space as shown in FIG. 11 on the touch panel display 106 as an emotion value input screen. Here, when the user designates one point on the displayed emotion space (for example, by tapping or clicking), the facial expression image presenting unit 117 has a facial expression image having an emotion value corresponding to the designated point ( Expression body images) 2 are listed on the touch panel display 106 and displayed. The user can select a desired one from the displayed facial expression images 2.

  Here, at the time of this display, it is also preferable to display the facial expression image 2 having a smaller variance covariance matrix Σ as a candidate having a higher priority. In general, the smaller the value of the variance-covariance matrix Σ, the higher the peak level of the corresponding “emotion value distribution”, that is, the degree of association indicating the number of distributions (distribution probability). Therefore, by displaying as described above, it is possible to present a facial expression image having a higher probability of taking the designated emotion value as a candidate having a higher order of selection.

  Note that the facial expression image presentation unit 117 also preferably displays the facial expression images 2 having emotion values within a predetermined range centered on the emotion value designated by the user. Thereby, the facial expression image 2 expressing substantially the same emotion can be presented to the user as a selection candidate.

  Returning to FIG. 2, the application processing unit 122 displays, for example, a visual character (expression image) with a specified emotion value on the touch panel display as a conversation agent rich in facial expressions, and provides various services to the user. Start and control application software.

  As described above in detail, according to the present invention, emotions can be estimated with high efficiency even for characters for which facial expression model data is preset, and possible emotions (emotion spaces) It is possible to easily grasp the position of the estimated emotion (emotion value) in the whole. In particular, for facial expressions (expression images) that are combinations of parts of various shapes, the emotion value of this facial expression can be obtained without the need for a feature quantification process etc. The distribution can be determined (estimated) more efficiently.

  In addition, by combining the parts whose emotion value distribution and the degree of influence are defined, it is possible to provide a variety of facial expressions with a specified emotion value distribution. Further, by obtaining the variance-covariance matrix Σ, facial expression candidates having emotion values within a predetermined range can be presented with priorities. As a result, a user-friendly character having an appropriate expression suitable for the situation can be generated and displayed, and a service more suitable for the user can be constructed or implemented.

  Various changes, modifications, and omissions of the various embodiments of the present invention described above within the scope of the technical idea and the viewpoint of the present invention can be easily made by those skilled in the art. The above description is merely an example and is not intended to limit the scope of the invention. The invention is limited only as defined in the following claims and the equivalents thereto.

1 Information terminal (Emotion estimation device)
DESCRIPTION OF SYMBOLS 101 Communication interface 102 Parts data preservation | save part 103 Emotion evaluation result preservation | save part 104 Face expression emotion distribution preservation | save part 105 Part emotion distribution preservation | save part 106 Touch panel display 111 Face expression generation | occurrence | production / determination part 112 Emotion evaluation test execution part 113 Part emotion distribution deciding part 114 facial expression emotion distribution determination unit 115 influence calculation unit 116 facial expression emotion value calculation unit 117 facial expression presentation unit 121 input / output control unit 122 application processing unit 2 facial expression image (expression body image)
21 Base parts 22 Face parts 3 Standard facial expression image 4 Parts evaluation image

Claims (10)

An emotion estimation device that estimates emotion expressed in an expression image including a plurality of parts,
Part emotion distribution determining means for determining, for each of the plurality of parts, a part emotion value distribution, which is a distribution of emotion values for the part, from a plurality of emotion evaluation results for each of the plurality of parts that can be included in the expression image. When,
Expression emotion distribution determination means for determining an expression emotion value distribution, which is a distribution of emotion values for the analysis target expression image, from a plurality of emotion evaluation results for the analysis target expression image,
The determined expression body emotion value distribution is a result of weighting and superimposing the part emotion value distributions on the parts included in the analysis object expression image, and the distribution of the parts included in the analysis object expression image. For each, an influence degree calculating means for calculating an influence degree as a weight,
The expression emotion value distribution for the estimation target expression image is calculated for each part emotion value distribution for each part included in the estimation target expression image and for each part included in the estimation target expression image. An emotion estimation device comprising emotion value calculation means for calculating based on the degree of influence. The part emotion distribution determining means determines an average value in the part emotion value distribution for the part,
The expression body emotion distribution determining means determines an average value in the expression body emotion value distribution for the analysis object expression body image,
The influence calculation means determines that the average value in the expression emotion value distribution is a result of weighting and combining the average values in the part emotion value distribution for the parts included in the analysis object image. For each part contained in the target expression image, calculate the average value influence as a weight in the average value,
The emotion value calculation means includes the emotion value for the estimation target expression image in the average value in the part emotion value distribution for each part included in the estimation target expression image and the estimation target expression image. The emotion estimation device according to claim 1, wherein the emotion estimation device is calculated based on an average value degree of influence calculated for each of the parts. The part emotion distribution determining means determines a variance covariance matrix in a part emotion value distribution for the part,
The expression body emotion distribution determining means determines a variance covariance matrix in the expression body emotion value distribution for the analysis object expression body image,
The influence calculation means is a result obtained by weighting and combining the variance covariance matrix in the expression body emotion value distribution with the variance covariance matrix in the part emotion value distribution for the part included in the expression image to be analyzed. As for each of the parts included in the expression image to be analyzed, the variance / covariance matrix influence as a weight in the variance / covariance matrix is calculated,
The emotion value calculation means includes a variance covariance matrix in a part emotion value distribution for each of the parts included in the estimation target expression image, and a degree of distribution of the emotion value for the estimation target expression image, and the estimation target The emotion estimation apparatus according to claim 1, wherein the emotion estimation apparatus calculates the influence based on the variance-covariance matrix influence degree calculated for each of the parts included in the expression image.   Expression body image presentation means for presenting a plurality of expression body images whose calculated emotion values are within a predetermined range in the emotion space with higher priority as the expression body image having a smaller calculated variance-covariance matrix. The emotion estimation apparatus according to claim 3, comprising:   The influence calculation means defines a part emotion value distribution for each of the parts included in the analysis target expression image, with a value that defines the expression emotion value distribution for the analysis target expression image as an objective variable. The emotion estimation apparatus according to any one of claims 1 to 4, wherein a multiple regression analysis is performed using a value as an explanatory variable, and an influence is calculated as a coefficient of the explanatory variable. The influence calculation means includes a value x that defines a part emotion value distribution for each of the parts included in the analysis target expression image, and a value y that defines the expression emotion value distribution for the analysis target expression image. As a sample (x, y), and the ε tolerance ξ (r) of the residual r between the sample and the regression curve f (x) = x · w + b (b is a constant term), and the regression curve predetermined function Σξ (y-f (x) ) + λ containing the weights w | w | based on a weight w for minimizing a ^2/2, claim 1, characterized in that to calculate the degree of influence of 4 The emotion estimation apparatus according to any one of the above items.   The part emotion distribution determination means includes a plurality of emotion evaluation results for a part evaluation image in which a part corresponding to a part of an emotion evaluation target is replaced with a part of the emotion evaluation target in a standard expression image serving as a reference for emotion evaluation The emotion estimation device according to any one of claims 1 to 6, wherein a part emotion value distribution for the emotion evaluation target part is determined.   The expression emotion value distribution of a new expression image generated using a new part with a specified part emotion value distribution and influence level is automatically generated without requiring a new emotion evaluation test. The emotion estimation apparatus according to claim 1, wherein the emotion estimation apparatus is characterized. A program for causing a computer mounted on a device for estimating an emotion expressed in an expression image including a plurality of parts to function,
Part emotion distribution determining means for determining, for each of the plurality of parts, a part emotion value distribution, which is a distribution of emotion values for the part, from a plurality of emotion evaluation results for each of the plurality of parts that can be included in the expression image. When,
Expression emotion distribution determination means for determining an expression emotion value distribution, which is a distribution of emotion values for the analysis target expression image, from a plurality of emotion evaluation results for the analysis target expression image,
The determined expression body emotion value distribution is a result of weighting and superimposing the part emotion value distributions on the parts included in the analysis object expression image, and the distribution of the parts included in the analysis object expression image. For each, an influence degree calculating means for calculating an influence degree as a weight,
The expression emotion value distribution for the estimation target expression image is calculated for each part emotion value distribution for each part included in the estimation target expression image and for each part included in the estimation target expression image. An emotion estimation program for causing a computer to function as an emotion value calculation means for calculating based on the degree of influence. A method for estimating an emotion expressed in an expression image including a plurality of parts,
From a plurality of emotion evaluation results for each of a plurality of parts that can be included in the expression image, for each of the plurality of parts, a part emotion value distribution that is a distribution of emotion values for the part is determined and analyzed. A first step of determining an expression emotion value distribution, which is a distribution of emotion values for the analysis target expression image, from a plurality of emotion evaluation results for the expression image of
The determined expression body emotion value distribution is a result of weighting and superimposing the part emotion value distributions on the parts included in the analysis object expression image, and the distribution of the parts included in the analysis object expression image. For each, a second step of calculating the influence as a weight;
The expression emotion value distribution for the estimation target expression image is calculated for each part emotion value distribution for each part included in the estimation target expression image and for each part included in the estimation target expression image. And a third step of calculating based on the degree of influence.

Images