JP4641389B2 - Information processing method and information processing apparatus - Google Patents

Description

  The present invention relates to a technique for detecting a reaction such as a user's facial expression or action and recognizing a user's emotion from the detected reaction.

  In recent years, research on artificial agents that recognize human intentions and emotions and communicate with humans as an interface between machines and humans is progressing. In addition, various pet robots appear, and there are those that recognize the user's behavior and emotions and change the behavior of the pet robot according to the recognition result. Such a pet robot can be said to be a kind of artificial agent. In such an artificial agent, it is necessary to detect the state of a person using a camera or a microphone and recognize emotions such as emotions of the person based on the detected state.

  For example, an apparatus for detecting a facial expression from an image taken with a CCD camera or the like has been conventionally disclosed (see, for example, Patent Document 1). In this method, wavelet transform is performed on an input image, and a facial expression is detected from the difference between the average power in each frequency region and the average power when there is no expression.

  In addition, a technique for detecting facial expressions and emotions of a subject using both sound and images has been conventionally disclosed (for example, see Non-Patent Document 1).

  Furthermore, there is a technology for determining a category based on a symbol sequence after vector quantization using a feature vector corresponding to face displacement and a code book for vector quantization prepared for each category of facial expression. It has been conventionally disclosed (for example, see Patent Document 2).

In addition, a technique is also disclosed for examining smile images that are classified into pleasant smiles, unpleasant smiles, and social smiles based on changes in the position and size of the eyes, mouth, and eyebrows. (See Non-Patent Document 2, for example).
JP-A-8-249447 Japanese Patent No. 2839855 “Lawrence S. Chen, Thomas S. Huang, Tsutomu Miyasato, Ryohei Nakatsu:“ Multimodal Human Emotion / Expression Recognition ”, Proceedings of the Second International Conference on Automatic Face and Gesture Recognition pp366-371 (1998)” “Kataoka, Iwaguchi, Sagi:“ Classification of smiles by tracking eye and mouth movements ”, Information Processing Society of Japan, 2001-HI-95, pp109-116 (2001)”

  As described above, research is being conducted to detect a person's state (reaction) using a camera or a microphone, and to recognize emotions such as a person's emotions based on the detected state (reaction). However, it is not easy to accurately correspond the detected state (reaction) and emotion.

  For example, if one considers facial expressions as one of the reactions, some people have rich facial expressions and others do not. In other words, not all people with the same emotions have the same facial expression. In other words, people with the same facial expression do not always have the same emotion. For this reason, the correspondence between a person's state (reaction) such as a facial expression and emotion must be set for each individual. In the above conventional example, there is no mention regarding individual differences among individual users.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique for easily setting a correspondence between a person's state (reaction) represented by a facial expression and an emotion for each individual. To do.

  In order to achieve the object of the present invention, for example, an information processing apparatus of the present invention comprises the following arrangement.

That is, an information processing apparatus for reproducing movie data,
Memory holding means for pre-associating and holding emotion information indicating an emotion that a person who sees the scene seems to have for each scene of the movie;
Playback means for playing back the movie data;
Imaging means for capturing an image of a user when viewing the movie data;
Estimating means for estimating the user's emotion when watching the scene of the movie based on the feature amount of the user obtained from the image using a preset recognition model;
Wherein such user emotion estimated by the estimating means when viewing the scene, emotion information stored and held in the storage holding means in association with the scene is estimated as emotion indicated, the recognition Correction means for correcting the parameters of the model.

  In order to achieve the object of the present invention, for example, an information processing apparatus of the present invention comprises the following arrangement.

That is, an information processing apparatus that plays a game,
Every information indicating each progress degree of the game, a storage holding means in advance in association with memory retention emotion information indicating the emotion that seems to have the person with Oite the game該進exhibition condition,
Imaging means for capturing an image of the user when the game is being played ;
Monitoring means for monitoring the progress of the game being played by the user;
Estimating means for estimating the user's emotion in each progress of the game using a preset recognition model based on the feature amount of the user obtained from the image;
As emotion estimated by the estimating means for each progress condition of the game, emotion information held associated with the information indicating the該進exhibition degree in the storage holding means is estimated as emotion indicated, the recognition model And a correction means for correcting the parameters of.

  In order to achieve the object of the present invention, for example, an information processing method of the present invention comprises the following arrangement.

That is, an information processing method performed by an information processing apparatus that reproduces movie data,
A memory holding step in which the memory holding means of the information processing apparatus associates in advance and stores in the memory holding unit emotion information indicating the feeling that a person who sees the scene seems to have for each scene of the movie;
A reproduction step in which reproduction means included in the information processing apparatus reproduces the data of the movie;
An imaging process in which an imaging unit included in the information processing apparatus captures an image of a user when viewing the movie data;
Based on the feature amount of the user obtained from the image, the estimation means included in the information processing apparatus estimates the emotion of the user when watching the movie scene using a preset recognition model. An estimation process to
Correction means said information processing apparatus has found the emotion estimated by the estimation process, shown emotion information stored and held in the storage holding unit in association with the scene when the user is viewing the scene A correction step of correcting the parameters of the recognition model so as to be estimated as emotions.

  In order to achieve the object of the present invention, for example, an information processing method of the present invention comprises the following arrangement.

That is, an information processing method performed by an information processing apparatus that plays a game,
Storage retaining means said information processing apparatus has found the each information indicating each progress condition of the game, in advance associate emotion information indicating the emotion that seems to have the person with Oite the game該進Exhibition degree A memory holding step for storing and holding in the memory holding unit;
An imaging process in which an imaging unit included in the information processing apparatus captures an image of a user when the game is being played ,
A monitoring step in which the monitoring means of the information processing apparatus monitors the progress of the game being played by the user;
An estimation step in which the estimation unit included in the information processing apparatus estimates the emotion of the user in each progress of the game using a preset recognition model based on the feature amount of the user obtained from the image. When,
Emotion correction means said information processing apparatus has the emotion estimated by the estimating step for each progress condition of the game, indicated by the emotion information stored associated with information indicating the該進exhibition degree in the storage holder And a correction step of correcting the parameters of the recognition model.

  With the configuration of the present invention, the correspondence between the state (reaction) of a person typified by a facial expression and emotion can be easily set for each individual.

  Hereinafter, the present invention will be described in detail according to preferred embodiments with reference to the accompanying drawings.

[First Embodiment]
FIG. 1 is a diagram illustrating a functional configuration of the information processing system according to the present embodiment.

  In the figure, reference numeral 100 denotes a user. In the following, processing for setting the correspondence between the user's facial expression and emotion will be described. The system according to this embodiment includes a camera, a 101, a microphone 102, a control unit 103, a reaction measurement unit 104, an emotion recognition unit 105, a stimulus data presentation unit 106, an emotion data holding unit 107, an emotion model correction unit 108, and an emotion model 109. It is comprised by.

  Hereinafter, each part which comprises a system is demonstrated easily.

  The camera 101 captures the state of the user as a moving image. The microphone 102 collects user's voice. Image information and audio information obtained by the camera 101 and the microphone 102 are output to the reaction measurement unit 104 in the subsequent stage.

  The control unit 103 controls each unit constituting the system shown in FIG. The reaction measurement unit 104 measures the reaction of the user 100 using image information obtained from the camera 101 or voice information obtained from the microphone 102. This reaction includes, for example, the position and size of eyes and mouth on the face of the user 100, the volume of voice, and the like.

  The emotion recognition unit 105 recognizes the emotion of the user 100 using the information indicating the reaction of the user 100 measured by the reaction measurement unit 104 and the emotion model 109. This recognition method will be described later.

  The stimulus data presentation unit 106 presents (reproduces) an image and sound according to the stimulus data to the user 100. The stimulus data is associated with data (information) indicating a feeling that a general user holds when an image or sound according to the stimulus data is presented to the general user. This stimulation data will be described later in detail. The emotion data holding unit 107 holds the stimulus data and data indicating emotions associated with the stimulus data.

  When emotion data is presented to the user 100 by the stimulus data presentation unit 106, the emotion model correction unit 108 recognizes the emotion recognized by the emotion recognition unit 105 as an emotion associated with the presented stimulus data. The parameters of the emotion model 109 are corrected. The emotion model 109 is a model for receiving information from the reaction measurement unit 104 and recognizing the emotion of the user 100 based on the input.

  FIG. 7 is a diagram illustrating a hardware configuration of a system according to the present embodiment. In the present embodiment, it is assumed that the camera 101 and the microphone 102 are connected to a computer such as a PC (personal computer) or WS (workstation), and a series of the above-described recognition processing and parameter correction processing of the emotion model 109 are performed in this computer.

  Reference numeral 701 denotes a CPU that controls the entire computer using programs and data stored in the RAM 702 and ROM 703, and also receives data of each frame input from the camera 101 via the I / F 707 and input from the microphone 102. Processing to receive analog signals is also performed. In addition, a series of recognition processing, which will be described later, and parameter correction processing of the emotion model 109 are also performed. The CPU 701 functions as the control unit 103 in FIG.

  Reference numeral 702 denotes a RAM which is stored in the external storage device 706. The RAM 702 includes an area for temporarily storing programs and data loaded under the control of the CPU 701, and a work used by the CPU 701 for performing various processes. It also has an area. Further, an area for temporarily storing image information and audio information input from the camera 101 and the microphone 102 via the I / F 707 is provided.

  Reference numeral 703 denotes a ROM which stores programs and data for starting up the computer.

  An operation unit 704 includes a keyboard and a mouse, and can input various instructions to the CPU 701.

  A display unit 705 includes a CRT, a liquid crystal screen, and the like, and can display images, characters, and the like. The display unit 705 can also output sound. The display unit 705 functions as the stimulus data presentation unit 106 in FIG.

  Reference numeral 706 denotes an external storage device that functions as a large-capacity information storage device such as a hard disk device, and stores programs and data for causing the OS (operating system) and the CPU 701 to execute a series of processes described later. I can leave. In this embodiment, the reaction measuring unit 104, the emotion recognizing unit 105, the emotion model 109, and the emotion model correcting unit 108 are all configured by programs, data, and the like, stored in the external storage device 706, and stored in the RAM 702 according to the control of the CPU 701. Suppose that it is to be loaded. Also, the external storage device 706 functions as the emotion data holding unit 107 in FIG.

  Reference numeral 707 denotes an I / F that can connect the camera 101 and the microphone 102 and can receive data of each frame input from the camera 101 and an analog signal input from the microphone 102 via the I / F 707. . It is assumed that the I / F 707 includes an A / D converter for converting analog signals from the camera 101 and the microphone 102 into digital data.

  Reference numeral 708 denotes a bus connecting the above-described units.

  Hereinafter, the emotion recognition process performed by the computer and the parameter correction process of the emotion model 109 for recognizing the emotion of the user 100 will be described in detail with reference to FIG. 3 showing a flowchart of these processes.

  Here, in this embodiment, what is presented to the user 100 by the display unit 705 is a movie. That is, the stimulus data is movie data. Therefore, the stimulus data is composed of image data and audio data of each frame of the movie. In addition, the stimulus data is associated with each scene of the movie, and information indicating an emotion that appears to be (generally) held by a person watching the scene.

  There are several advantages to using movies as stimulus data. One is that many people can have the same experience, so it is easy to compare the average response with the individual's unique response. In other words, when trying to make individual settings based on experience in daily life, it is rare that the situation is exactly the same, so it is difficult to compare, but using movies makes such a problem Can be avoided.

  In general, movies have many scenes, and when you watch a movie, you can get a response to all the basic emotions. Furthermore, since there are multiple scenes where the user has the same emotion, instead of creating an emotion model only with one particular scene, the average reaction for multiple scenes where the user has the same emotion is used, Since an emotion model is created, an accurate emotion model can be created.

  FIG. 2 is a diagram illustrating a configuration example of an emotion data sheet in which data indicating emotion is associated with each scene. For example, if you watch a scene from the frame of 10 minutes 21 seconds to the frame of 12 minutes 20 seconds from the start of the movie, many people have statistics such as feeling “anger”, so the figure As shown in FIG. 5, data indicating the feeling of “anger” is associated with the scene from the frame at 10 minutes 21 seconds to the frame at 12 minutes 20 seconds. The same applies to other scenes.

  Such emotion data sheet data is created in advance, stored in the external storage device 706, and loaded into the RAM 702 as necessary.

  The CPU 701 loads movie data stored in the external storage device 706, data indicating emotions associated with each scene, and data of the emotion data sheet into the RAM 702, reproduces the movie data, and displays the display unit 705. The image information and audio information of each frame of the movie are output (reproduced) (step S301).

  The user 100 views the presented movie image and listens to the presented audio information. The state (reaction) of the user at this time is collected by the camera 101 and the microphone 102. The camera 101 captures a moving image of the user 100, and the captured image data of each frame is sequentially output to the RAM 702 via the I / F 707. Similarly, the microphone 102 collects sound emitted from the user 100, and the collected sound signal is output to the RAM 702 via the I / F 707.

  The CPU 701 measures the reaction of the user 100 watching a movie using the image information and audio information output from the camera 101 and microphone 102 to the RAM 702 (step S302). As described above, the measurement of the reaction is the position and size of the parts constituting the face such as the eyes, mouth and nose of the face of the user 100 on the image, their changes, and other positions on the hand and foot images A feature amount such as a change in the position of the user 100, a loudness of the voice uttered by the user 100, or the like is measured as a parameter measurement value used for input to the emotion model 109 described later. Note that the contents of such feature quantities are not particularly limited, and the measurement and calculation methods thereof are well known, and thus further explanation is omitted.

  Then, the CPU 701 recognizes (estimates) the emotion of the user 100 based on the obtained measurement value (step S303). That is, it recognizes the emotion of the user 100 who is listening to the scene presented in step S301. The emotion model 109 is used for this recognition. The emotion model 109 is loaded from the external storage device 706 to the RAM 702 in the form of a program or data, and the CPU 701 can use this to perform recognition processing described below.

  As the emotion model 109, for example, a known neural network can be used. FIG. 4 is a diagram illustrating a configuration example of a known hierarchical neural network. As shown in the figure, the hierarchical neural network has a three-layer structure of an input layer, an intermediate layer, and an output layer. Information input to the input layer is processed by the intermediate layer, and the processing result Is further processed by the output layer and output from each neuron constituting the output layer.

  The number of neurons constituting the input layer is prepared by the number of feature amounts obtained in step S302. Therefore, each neuron constituting the input layer includes the feature amount obtained in step S302 (the position and size of the parts constituting the face such as the eyes, mouth, and nose of the user 100, the change thereof, and the like). The position of the hand or foot, the change in the position, the loudness of the sound produced by the user 100, etc.) are input.

  On the other hand, each neuron constituting the output layer is for showing different emotions, and when each feature amount obtained in step S302 is input to each neuron constituting the input layer, the neuron constituting the output layer Any one of them will ignite. Therefore, the emotion assigned to the fired neuron is the recognition result. For example, when the number of neurons constituting the output layer is seven, any one of “happiness”, “anger”, “sadness”, “disgust”, “surprise”, “fear”, and “no emotion” is assigned to each neuron without duplication. For example, when only a neuron assigned with “surprise” fires, the recognition result is “surprise”. The type of emotion is not limited to this, and the number of neurons constituting the output layer is not limited to this.

  As is well known, a weighting factor between neurons between layers is already set, but this weighting factor is averaged when parameter measurement values of average human reaction data are input. It is initially set to output human emotions. In this initial setting, the same stimulus data is presented to a large number of people, the parameter measurement value of the response data at that time is used as an input signal to the input layer, and the average emotion at that time is learned as a teacher signal, Done.

  As a result, even when various human reaction data are input to the neural network, only the neurons that show the emotions that the approximate person will have will fire from the output layer.

  However, depending on the person, for example, even if the facial feature amount when angry is input to the neural network, the neuron assigned with the emotion of sadness may fire. This occurs when the facial expression of this person is not an average facial expression.

  Therefore, as will be described in detail below, when a neural network is used as the emotion model 109, the firing pattern of neurons in the output layer when the response data of the user 100 viewing a certain scene is input to the neural network is shown in this scene. It is necessary to modify the weighting factor between neurons to match the firing pattern of neurons in the output layer showing the associated emotion.

  On the other hand, a case where a model other than the neural network is used as the emotion model 109 will be described. As a model other than the neural network, an emotion model that defines the existence range of parameter measurement values of the response data of the user 100 measured in step S302 for each emotion is used.

FIG. 5 is a diagram showing an example of the range of each reaction parameter defined by such a model. In the emotion model sheet 501 shown in the figure, for each emotion such as “happiness” described above, parameter measurement of reaction data such as the position of the eye, mouth, etc., and the position of a smaller part constituting them, and changes thereof is performed. Describes the range of values. For example, for the feeling of “happiness”, the positional relationship between the center position of the eye and the corner of the eye is defined for the eye. If the origin is at the upper left of the image, the vertical length of the eye is Ev, the center position of the eye is Ex, Ey, the vertical position Eey of the outer corner of the eye is
Eey ≧ Ey + a × Ev and Eey <Ey + b × Ev (a = 0.1, b = 1.0)
It follows the conditional expression. In other words, this expression shows that when the emotion of “happiness” is expressed, the corners of the eyes are lowered.

  In other words, in the reaction data of the user 100, when the parameter measurement values of the center position of the eye and the position of the corner of the eye satisfy this equation, the user 100 may be in a state of “happiness”. is there. Then, using this emotion model 109, in addition to this, it is examined whether the parameter measurement values obtained for the respective components such as the eyes and mouth satisfy the expressions indicating the respective positional relationships, The emotion with the most satisfied expressions is recognized as the emotion of the user 100.

  In this way, by defining the positions and changes of the components of the face at the time of each emotion, and the existence ranges such as the movements of the hands and feet and the loudness of the voice, each emotion can be expressed on the model sheet. Note that the values of the parameters (coefficients) (a, b) in the above formula are initially set so that the average human emotion is output when the parameter measurement values of the average human reaction data are input. ing. Of course, the parameters in the other relational expressions are similarly initialized.

  As a result, even if the response data of various people are applied to the conditional expressions described in the model sheet as shown in FIG. 5, the conditional expressions corresponding to the emotions that an approximate person will have are satisfied. Become.

  However, depending on the person, for example, even if the facial feature amount when angry is substituted into each conditional expression, the conditional expression corresponding to the emotion of sadness may be satisfied. This occurs when the facial expression of this person is not an average facial expression.

  Therefore, as will be described in detail below, when a group of conditional expressions provided for each emotion is used as the emotion model 109, the reaction data of the user 100 who has seen a certain scene is substituted into the conditional expression group provided for each emotion. It is necessary to correct the parameters in each conditional expression group so that the conditional expression group provided for the emotion associated with this scene is most satisfied.

  As described above, regardless of which model is used for the emotion model 109, in this embodiment, the emotion of the user 100 is associated with this scene based on the reaction data of the user 100 who is watching a certain scene. It is necessary to correct the parameters of the emotion model 109 so that the emotion is recognized as the emotion.

  Accordingly, in step S304, the parameter of the emotion model 109 is corrected.

  First, the process in step S304 when a neural network is used as the emotion model 109 will be described.

  FIG. 8 is a diagram illustrating the process in step S304 when a neural network is used as the emotion model 109. Although the number of neurons constituting the output layer is 3 here, the number is not limited to this.

  In the figure, it is assumed that emotions “anger”, “sadness”, and “happiness” are assigned to the neurons 801, 802, and 803 constituting the output layer, respectively. Therefore, when only the neuron 801 fires (when the output pattern of “1 0 0” is output from the output layer), the neural network outputs “anger” as the recognition result. On the other hand, when only the neuron 802 fires (when the output pattern “0 1 0” is output from the output layer), the neural network outputs “sadness” as the recognition result. On the other hand, when only the neuron 803 fires (when the output pattern of “0 0 1” is output from the output layer), the neural network outputs “happiness” as the recognition result.

  Here, the response data of the user 100 watching a certain scene is input to the input layer, and an output pattern of “0.1 0.2 0.9” is output from the output layer. Assume that information indicating is associated.

  In this case, since it is generally expected that the user 100 will have a “happy” feeling by looking at this scene, the pattern “0 0 1” indicating “happiness” is recognized as an output layer as the recognition result of the user 100 emotion. Is preferably output from. However, when an output pattern of “0.1 0.2 0.9” different from this is output, this means that when the weighting coefficient is corrected and the neural network inputs the above reaction data. This means that it is necessary to output a desired output pattern (“0 0 1”).

  Therefore, in such a case, by inputting the reaction data to the input layer, a difference (error) between the output pattern output from the output layer and the desired output pattern as a teacher signal is obtained, and Based on this, a known back-propagation process is performed to correct the weighting factor between the neurons between the layers (learning process). By repeating such processing a plurality of times and correcting the weighting coefficient, when the response data of the user 100 who is watching a happy scene is input to the neural network, an output pattern of “0 0 1” is output from the output layer. In other words, it will be recognized as an emotion of “happiness”.

  The same applies to the case where the user 100 is shown other scenes (scenes with other emotions) and the weighting coefficient used for recognizing the emotions of the user 100 at that time is corrected. In the present embodiment, the processing may be shifted to the next scene after the above error reaches a predetermined value for one scene (each scene shown in FIG. 2), or once for each scene. You may make it carry out sequentially. Various algorithms for the learning process can be applied and are not particularly limited. Further, the end criterion of the learning process is not particularly limited.

  Next, the process in step S304 in the case where the emotion model 109 is an emotion model that defines the existence range of parameter measurement values for each emotion will be described.

For example, when the user 100 is viewing a scene associated with information indicating “happiness”, the condition that the presence range of the reaction data of the user 100 at this time satisfies the conditional expression corresponding to the emotion of “happiness” is satisfied. Correct the parameters included in the expression. For example, the above conditional expression for eyes to recognize as feelings of happiness
Eey ≧ Ey + a × Ev and Eey <Ey + b × Ev
In this case, the vertical length Ev and the center positions Ex and Ey of the eye included in the response data of the user 100 who is viewing the scene associated with the information indicating “happiness” are substituted into this expression, The coefficients a and b are corrected so as to satisfy the equation.

  In this embodiment, the conditional expression parameter correction process may be performed for one scene (each scene shown in FIG. 2), and then the process may be transferred to the next scene. Alternatively, each scene may be sequentially performed once. You may make it do.

  The above processing is performed until the end of the movie (step S305).

  By performing the processing described above, the parameters included in the emotion model 109 indicate what facial expression, limb movement, and voice volume when the user 100 has emotions associated with each scene. As a result, the emotion model 109 after learning can recognize the emotion of the user 100 with visual information and auditory information unique to the user 100. Become.

  Thus, by creating an emotion model for each individual user and recognizing the emotion for the user's reaction based on the individual emotion model, the user's emotion can be accurately recognized. In other words, although the individual responses (facial expressions, etc.) are generally different even if they hold the same emotions, they can learn emotions that are not affected by differences in individual responses because they learned each individual response to the same stimulus. .

  In addition, by using a movie as the stimulus data, a highly accurate emotion model can be created as described above, so that highly accurate emotion recognition is possible.

  In this embodiment, the initial values of the parameters of the emotion model 109 are initially set so that the average human emotion is output when the parameter measurement value of the average human reaction data is input. Therefore, it is possible to easily create an emotion model for each individual rather than setting all initial values to 0.

  Also, instead of recognizing emotion for each scene, only when a change occurs in the parameter measurement value of the reaction data of the user 100, such as when the facial expression of the user 100 changes or when the operation of the user 100 changes. You may be made to recognize emotions. In this way, when creating an emotion model for a person with poor response, the error when there is no response can be reduced.

  In the present embodiment, both image information and audio information of the user 100 are collected to recognize emotions. However, some users may not speak any emotions. In that case, emotion may be recognized using only image information. In that case, the microphone 102 is not necessary, and only the image information is input to the emotion model 109.

[Second Embodiment]
In the first embodiment, movie data is used as stimulus data, but in this embodiment, a video game program and data are used.

  That is, the game is played, and emotion data associated with the progress of the game and the emotion that the person has with respect to the progress is prepared. Then, the emotion model of the user is created in the same manner as in the first embodiment from the emotion data for the progress of the game and the user's reaction to the progress.

  The progress of the game here means how long it takes to clear a scene, how many enemies have been defeated within a certain period of time, Or have lost. In the present embodiment, the emotion data holding unit 107 holds emotion data in which the progress of the game is associated with the emotion at that time.

  FIG. 6 shows a configuration example of an emotion data sheet in the present embodiment.

  The emotion data sheet 601 associates the progress of the game as described above with emotions. This association is created from average emotion data in which a large number of people play a game and the emotions according to the progress of the game at that time are summarized.

  Therefore, the CPU 701 monitors the play of the user 100 for each progress shown in the figure (monitoring the clear time, how many enemies have been defeated within a certain time, etc.), and the emotion data sheet of the figure Identify the relevant progress in. Since emotions are associated with each identified progress, the largest number of emotions associated with each progress are output as recognition results. That is, what is recognized here is an emotion (estimated emotion) that “if this progress, the user 100 will have such an emotion”.

  On the other hand, as in the first embodiment, the emotion of the user 100 is recognized by the emotion model 109 based on image information and audio information from the camera 101 and the microphone 102. Then, referring to the recognized emotion and the estimated emotion (corresponding to the teacher signal in the first embodiment), the parameters of the emotion model 109 are corrected in the same manner as in the first embodiment.

  As can be seen from the above description, in the first embodiment, information indicating emotions is associated with each scene in advance and stored in the form of an emotion data sheet, so the emotions to be recognized for each scene. Is obtained by referring to this emotion data sheet, but in this embodiment, by using a game program and data as stimulus data, the configuration of the emotion data sheet and this emotion data sheet are used. The method for recognizing the emotion of the user 100 is different.

  In other words, the CPU 701 always monitors the progress of the game of the user 100 and determines which progress of the emotion data sheet corresponds to each predetermined time. Of the emotions associated with the corresponding progress, the most common emotion is taken as the recognition result. Thereby, the CPU 701 can recognize the emotion by determining that “the user 100 will have such an emotion if this progress is made”.

  Thus, by using a video game instead of a movie in order to obtain what the current user's 100 emotion is, the configuration of the emotion data sheet and the user 100 using this emotion data sheet The present embodiment is the same as the first embodiment except that the method for recognizing emotions is different.

  Thus, using a video game as stimulus data has the following advantages. That is, since many people can experience almost the same experience as the game progresses, it is easy to compare the average response with the individual specific response. In addition, since there are several states with the same emotion, an emotion model is created using an average response to multiple states with the same emotion, instead of creating an emotion model only with a specific state. Therefore, an accurate emotion model can be created.

  In addition, the video game is interactive, and the user works more proactively than the movie. Therefore, the user's reaction becomes stronger by making a voice and the emotion of emotion is easily generated. Therefore, it is easy to perform parameter measurement of user reaction data. However, when creating and correcting an emotion model, the strength of the emotion that is output with respect to the parameter measurement value of the input response data of the user needs to be at the maximum level. That is, when a value lower than the parameter measurement value level of the reaction data during the game is input, the emotion level to be output needs to be set to the normal level.

  In the present embodiment, both image information and audio information of the user 100 are collected to recognize emotions. However, some users may not speak any emotions. In that case, emotion may be recognized using only image information. In that case, the microphone 102 is not necessary, and only the image information is input to the emotion model 109.

  In addition, the form which obtains the reaction data of the user 100 can be considered other than the above embodiment. For example, in the first embodiment, image information and audio information of the user 100 watching the scene are collected as reaction data, but the pulse rate of the user watching the scene is measured, and the measured pulse rate is per unit time. The emotion may be estimated depending on how much the pulse rate is. For example, since the pulse rate per unit time increases during excitement, the pulse rate can be partial information of emotion, so this pulse rate can also be used as reaction data. Moreover, you may make it use combining suitably what was demonstrated above as reaction data.

[Other Embodiments]
An object of the present invention is to supply a recording medium (or storage medium) that records software program codes for realizing the functions of the above-described embodiments to a system or apparatus, and the computer of the system or apparatus (or CPU or MPU). Needless to say, this can also be achieved by reading and executing the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiment, and the recording medium on which the program code is recorded constitutes the present invention.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instruction of the program code. It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the actual processing and the processing is included.

  Furthermore, after the program code read from the recording medium is written into a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer, the function is based on the instruction of the program code. It goes without saying that the CPU or the like provided in the expansion card or the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

  When the present invention is applied to the recording medium, program code corresponding to the flowchart (functional configuration) described above is stored in the recording medium.

It is a figure which shows the function structure of the information processing system which concerns on the 1st Embodiment of this invention. It is a figure which shows the structural example of the emotion data sheet which linked | related the data which show an emotion for every scene. It is a flowchart of the emotion recognition process which the computer which concerns on the 1st Embodiment of this invention performs, and the correction process of the parameter of the emotion model 109 for recognizing the emotion of the user 100. FIG. It is a figure which shows the structural example of a known hierarchical type neural network. It is a figure which shows the example of the range of each reaction parameter which the emotion model which prescribes | regulates the existence range of the parameter measurement value of the reaction data of the user 100 measured by step S302 for every emotion. The structural example of the emotion data sheet in the 2nd Embodiment of this invention is shown. It is a figure which shows the hardware constitutions of the system which concerns on the 1st Embodiment of this invention. It is a figure explaining the process in step S304 at the time of using a neural network as the emotion model 109. FIG.

Claims (11)

An information processing apparatus for reproducing movie data,
Memory holding means for pre-associating and holding emotion information indicating an emotion that a person who sees the scene seems to have for each scene of the movie;
Playback means for playing back the movie data;
Imaging means for capturing an image of a user when viewing the movie data;
Estimating means for estimating the user's emotion when watching the scene of the movie based on the feature amount of the user obtained from the image using a preset recognition model;
Wherein such user emotion estimated by the estimating means when viewing the scene, emotion information stored and held in the storage holding means in association with the scene is estimated as emotion indicated, the recognition An information processing apparatus comprising: correction means for correcting a parameter of the model.   The said estimation means inputs the said feature-value into the neuron group of the input layer in a multilayer type neural network, and obtains the output from the neuron group of an output layer as the information which shows the said user's emotion. The information processing apparatus described. When the feature value obtained from the image of the user when viewing the scene is input to the neuron group of the input layer, the correction unit is stored and held in the storage holding unit in association with the scene. The information processing apparatus according to claim 2, wherein a weighting factor between neurons is corrected so that emotion information is output.   The information processing apparatus according to claim 3, wherein the correction unit corrects the weight coefficient by back propagation. The estimation means refers to an allowable range of the feature amount obtained in advance for each emotion, and outputs an emotion including the feature amount obtained from the user image as an estimation result as an estimation result. The information processing apparatus according to 1. And further comprising a collecting means for collecting the user's voice information,
The estimation means estimates the emotion of the user when viewing the scene using a preset recognition model based on the feature amount obtained from the image and the feature amount obtained from the audio information. The information processing apparatus according to claim 1, wherein the information processing apparatus is an information processing apparatus. An information processing apparatus for playing a game,
Every information indicating each progress degree of the game, a storage holding means in advance in association with memory retention emotion information indicating the emotion that seems to have the person with Oite the game該進exhibition condition,
Imaging means for capturing an image of the user when the game is being played ;
Monitoring means for monitoring the progress of the game being played by the user;
Estimating means for estimating the user's emotion in each progress of the game using a preset recognition model based on the feature amount of the user obtained from the image;
As emotion estimated by the estimating means for each progress condition of the game, emotion information held associated with the information indicating the該進exhibition degree in the storage holding means is estimated as emotion indicated, the recognition model An information processing apparatus comprising: correction means for correcting the parameters of the information processing apparatus. An information processing method performed by an information processing apparatus for reproducing movie data,
A memory holding step in which the memory holding means of the information processing apparatus associates in advance and stores in the memory holding unit emotion information indicating the feeling that a person who sees the scene seems to have for each scene of the movie;
A reproduction step in which reproduction means included in the information processing apparatus reproduces the data of the movie;
An imaging process in which an imaging unit included in the information processing apparatus captures an image of a user when viewing the movie data;
Based on the feature amount of the user obtained from the image, the estimation means included in the information processing apparatus estimates the emotion of the user when watching the movie scene using a preset recognition model. An estimation process to
Correction means said information processing apparatus has found the emotion estimated by the estimation process, shown emotion information stored and held in the storage holding unit in association with the scene when the user is viewing the scene A correction step of correcting a parameter of the recognition model so as to be estimated as emotion. An information processing method performed by an information processing apparatus that plays a game,
Storage retaining means said information processing apparatus has found the each information indicating each progress condition of the game, in advance associate emotion information indicating the emotion that seems to have the person with Oite the game該進Exhibition degree A memory holding step for storing and holding in the memory holding unit;
An imaging process in which an imaging unit included in the information processing apparatus captures an image of a user when the game is being played ,
A monitoring step in which the monitoring means of the information processing apparatus monitors the progress of the game being played by the user;
An estimation step in which the estimation unit included in the information processing apparatus estimates the emotion of the user in each progress of the game using a preset recognition model based on the feature amount of the user obtained from the image. When,
Emotion correction means said information processing apparatus has the emotion estimated by the estimating step for each progress condition of the game, indicated by the emotion information stored associated with information indicating the該進exhibition degree in the storage holder And a correction step of correcting the parameters of the recognition model so as to be estimated as follows.   The computer program for functioning a computer as each means which the information processing apparatus of any one of Claims 1 thru | or 7 has.   A computer-readable storage medium storing the computer program according to claim 10.

Images