JP2022088796A - User condition analysis method and user condition analysis system - Google Patents

Abstract

A user state analysis method and a user state analysis system capable of accurately analyzing a user's emotional state or fatigue state while making the user aware of the user's emotional state or fatigue state are provided. SOLUTION: A step of showing an image of the user's face to the user and displaying a question about the condition of the face, a step of receiving an answer to the question while the image of the face is being shown to the user, and analyzing the answer. and generating user state information indicating a user state including one or both of the user's facial expression of emotion and fatigue based on the analysis result of the answer. [Selection drawing] Fig. 1

Description

The present invention relates to methods and systems for analyzing a user's emotional or fatigue state.

Conventionally, a fatigue degree determination processing system is known (see Patent Document 1).

Japanese Unexamined Patent Publication No. 2010-201113

In the invention described in Patent Document 1, the degree of fatigue of the user is determined based on the LF / HF value and the result of the questionnaire regarding the fatigue state of the user.

It is conceivable to apply the invention described in Patent Document 1 to a method and a system for analyzing a user's emotion or fatigue state.

By the way, the analysis result of the emotion or fatigue state based on the objective data may be different from the emotion or fatigue state considered by the user himself / herself. That is, for example, when an analysis of objective information such as a skin image shows that the person feels considerable stress or is tired, but the person is not aware of it, this is the case. There can be discrepancies.

Or conversely, as a result of thinking that the person is in a stressed state or is tired even though he / she is not so stressed objectively or has accumulated fatigue, as described above. Discrepancies can occur.

However, in the invention of Patent Document 1, since the user's fatigue level is determined by using the result of the questionnaire regarding the user's own fatigue state, it is assumed that he / she is not aware of his / her own fatigue state or is tired more than necessary. It is unlikely that you will get the correct answer about your fatigue condition from your users.

Moreover, high accuracy cannot be expected for analysis results based on such uncertain information.

Therefore, an object of the present invention is to provide a user state analysis method and a user state analysis system that can accurately analyze a user's emotional or fatigue state while making the user aware of it.

The user state analysis method of the present invention
Includes a step to show the user an image of the user's face and to display a question about the condition of the face that correlates with the emotional or fatigue condition appearing on the user's face.
An input reception step that accepts an answer by the user to a question about the state of the user's face while showing the image of the face to the user.
An answer analysis step that analyzes the user's answer to the question about the user's facial condition, and
It is characterized by comprising a user state information generation step of generating user state information indicating a user state including one or both of emotions and fatigue states appearing on the user's face based on the analysis result of the answer. do.

According to the present invention, the image of the user's face is shown to the user, and the displayed question about the state of the user's face is displayed. The question is a question about the facial condition that correlates with the emotion or fatigue condition appearing on the user's face.

Then, the input of the answer by the user to the question is accepted with the image of the face displayed, and the answer is analyzed.

Further, based on the analysis result of the answer, the user state information indicating the user state which is one or both of the user's emotion and fatigue state is generated.

According to the present invention, the user answers a question about the condition of the face while looking at his / her own face, so that the answer is easy. In addition, since you will have the opportunity to calmly look back on your facial condition, you will be promoted to be aware of the facial condition that is closely related to your emotions or fatigue, and you may be able to answer your facial condition accurately. It will increase.

This makes it possible to accurately generate user state information indicating the user state, which is one or both of the user's emotions and fatigue state.

As described above, according to the present invention, the user's emotional or fatigued state can be analyzed accurately while being aware of the user himself / herself.

In the user state analysis method of the present invention
A solution determination step for determining a solution according to the user status information from a plurality of solutions that can be provided to the user in order to improve the user status of the user.
It is preferable to include a step of providing the determined solution to the user.

According to the present invention, a solution according to the user state information is determined according to the user state information from a plurality of solutions that can be provided to the user for improving the user state, and the determined solution is provided to the user. To.

As a result, it is possible to accurately generate user state information indicating one or both of the user's emotions and fatigue states, and to easily provide an appropriate solution according to the user state to improve the user state. ..

In the user state analysis method of the present invention
The step of capturing an image of the user's face and
An image analysis step that analyzes the captured image of the user's face,
Including
The user state information generation step is preferably a step of generating the user state information based on the answer and the analysis result of the face image.

According to the present invention, one or both of the user's emotional and fatigue states are used with the objective information of the analysis result of the facial image in addition to the user's subjective information of answering the question about the facial condition. User state information indicating the user state is generated.

Therefore, since the user state information is generated in consideration of the information that cannot be obtained from the answer to the question, the user state information can be generated more accurately and an appropriate solution can be provided accordingly.

In the user state analysis method of the present invention
It is preferable to include a step of displaying the generated user status information.

According to the present invention, since the generated user state information is displayed, the user can be made to recognize his / her own emotion or fatigue state, so that the user is effectively aware of his / her emotion or fatigue state. be able to.

In the user state analysis method of the present invention
A step to store the generated user status information in chronological order,
It is preferable to include a step of displaying the stored user status information in response to the user's operation.

According to the present invention, past user status information can be displayed according to the user's operation. Therefore, since the user can know at any time whether or not the emotion or fatigue state that appears on the user's face has changed and the degree of the change, it is possible to effectively make the user aware of the user's emotion or fatigue state. can.

In the user state analysis method of the present invention
The answer analysis step is a first correlation showing the correlation between the user's answer to the accepted question about the user's facial condition and the emotional or fatigue state appearing on the user's face. It is preferable to include the step of determining the analysis result of one or both of the emotions and fatigue states appearing on the user's face using the information.

According to the present invention, the analysis result of one or both of the emotional state and the fatigue state appearing on the user's face is determined by using the specific information called the first correlation information, so that the analysis result appears on the user's face. Can accurately analyze one or both of emotions and fatigue.

In the user state analysis method of the present invention
The step of capturing an image of the user's face and
An image analysis step that analyzes the captured image of the user's face,
Including
The answer analysis step includes a step of analyzing the answer to determine a fatigue sign score, which is an index of the emotion or fatigue state appearing on the user's face.
The user state information generation step is
A step of acquiring information on the user's heartbeat from the analysis result of the face image, and
A step of determining a heart rate-based fatigue score, which is an index of a heart rate-based fatigue state based on the user's heart rate information.
A step of determining an overall score, which is an index of the user's emotional or fatigue state, based on the fatigue sign score and the fatigue degree score based on the heartbeat.
Including a step of generating the user state information, including the overall score,
The solution determination step preferably includes a step of determining a solution to be provided to the user based on the overall score.

According to the present invention, the present invention is provided to the user based on a fatigue sign score that indexes the emotion or fatigue state appearing on the user's face and a total score derived from a multifaceted viewpoint of a fatigue degree score based on the heartbeat. Since the solution to be used is determined, it is possible to provide an appropriate solution according to the user's overall fatigue condition.

In the user state analysis method of the present invention
The step to determine the overall score is
A step of acquiring information on the autonomic nervous balance of the user by using the information on the heartbeat of the user and the second correlation information showing the correlation between the information on the heartbeat and the autonomic nervous balance of the user.
It is preferable to include a step of determining the total score based on the information including the information regarding the autonomic nerve balance.

According to the present invention, the solution to be provided to the user is determined based on the total score derived from a more multifaceted viewpoint, which includes information on the autonomic nervous balance. We can provide the right solution.

In the user state analysis method of the present invention
The solution determination step is
A first state in which the user should provide a passive solution that is a passive stimulus to the user's sense of touch or hearing based on the emotional or fatigue state appearing on the user's face contained in the user state information. A step to determine if the user is in a second state to provide an active solution that encourages the user to perform a given action.
It is preferable to include a step of determining a solution according to the user state information from the plurality of solutions based on the result of the determination.

A given action has the effect of improving the user's emotional or fatigued state, but if the user's emotional or fatigued state is not appropriate for the given action, a passive stimulus to the sense of touch or hearing. Providing passive solutions such as, is effective in improving user's emotions and fatigue.

According to the present invention, the user is in a first state or predetermined state to provide a passive solution that is a passive stimulus to the sense of touch or hearing based on the emotional or fatigue state appearing on the user's face. It is determined whether you are in a second state to provide an active solution that encourages action.

Then, since the solution is determined based on the result of the determination, the type of the solution can be appropriately changed and provided according to the emotion and the fatigue state of the user.

In the user state analysis method of the present invention
The active solution comprises a combination of periodic vibrations that last for a predetermined time, voice to improve the user condition, and information that prompts the user to breathe or meditate in a predetermined manner.
When the user is in the second state, the active solution comprising a combination of the vibration, the voice, and information prompting the user to breathe or meditate in a predetermined manner in the solution determination step. , It is preferable to determine as a solution to be provided to the user.

Predetermined breathing and meditation have the effect of calming emotions, relieving fatigue, or leading a person to an awake state.

According to the present invention, when the user is in a second state in which an active solution for encouraging a predetermined action should be provided, a solution including information for encouraging breathing or meditation in a predetermined manner is determined, and thus the effect. It is possible to improve the user condition.

According to the study by the present inventors, "the degree of opening of the eyes", "the thickness of the bear around the eyes", "the state of acne or pimples", "the degree of raising or lowering of the corners of the mouth" compared to the normal state. There is a significant correlation between answers to questions about a person's facial condition, including "lip moisturizing condition," and the emotional or fatigue condition that appears on the person's face. It has been found.

The emotions are, for example, four factors of happiness, burnout state, life satisfaction, negativeness or positiveness by PANAS scale, degree of resilience, subjective well-being scale, and stress tolerance.

The fatigue state is, for example, a state of insomnia, a degree of physical fatigue, a degree of mental fatigue, and a state of fatigue components in the body.

More specifically, the degree of physical fatigue and the degree of mental fatigue in this embodiment include, for example, values corresponding to the total score of the Childer Fatigue Questionnaire, and the consumer needs study "Fatigue and Fatigue" by the Ministry of Education, Culture, Sports, Science and Technology. It includes values corresponding to physical evaluation, mental evaluation, and comprehensive evaluation by the "self-diagnosis fatigue degree checklist" prepared in "Study on molecular and neural mechanisms of sensation and its defense".

The insomnia condition includes, for example, the Athens Insomnia Scale.

Examples of the fatigue component in the body include 8OHdG (8-hydroxy-2'-deoxyguanocine) which is an oxidation product, isoplastane, and dROMs (Diacron Reactive Oxygen Metalbolites) which indicate the degree of oxidative stress.

The four factors of happiness are the values of the first to fourth factors according to the questionnaire of the four factors of happiness prepared by Mr. Takashi Maeno. Specifically, the first factor is a "let's try!" Factor (self-actualization and growth factor), which corresponds to the degree of satisfaction with self-actualization or growth in the present embodiment. The second factor is a "thank you!" Factor (factor of connection and gratitude), which corresponds to a good relationship with another person or a degree or positive feeling of gratitude to another person in this embodiment. The third factor is the "somehow!" Factor (factor of premature and optimism), and is the degree of optimism or self-affirmation in the present embodiment. The fourth factor is the "like you!" Factor (independence and self-paced factor).

The burnout state includes a value corresponding to a measure of emotional exhaustion, depersonalization, and a decrease in personal achievement due to, for example, MBI (Maslach Burnout Industry).

The life satisfaction is a degree of satisfaction with life, and includes, for example, a value corresponding to a life satisfaction scale (SWLS: Strategy With Life Scale) by Ed Diener.

Negative or positive by the PANAS scale is a rating scale included in the PANAS (Positive and Negative Affective) scale by Watson, Clark, Tellegen et al., For example, a simple mood rating consisting of 8 items of positive emotions and 8 items of negative emotions. Is a measure of.

The degree of resilience is the resilience to recover from psychological damage, and includes, for example, the two-dimensional resilience factor scale proposed by Mari Hirano in "Attempt to classify qualitative factors and acquisition factors of resilience".

The subjective well-being scale is a kind of psychological health scale. For example, the subjective health scale (SUBI) created by WHO has "positive feelings for life", "feeling of accomplishment", "confidence", and "self-confidence". It includes scales for 5 areas (3 items each) of "feeling of bliss" and "feeling of disappointment with life".

The stress tolerance is stress tolerance to psychosocial stress, and includes, for example, the degree of stress tolerance according to the "stress tolerance checklist" created by Taisaku Katsura et al.

The configuration, action, and effect of the user state analysis method in the present embodiment to which the knowledge is applied are as follows.

In the user state analysis method of the present invention
The question about the condition of the user's face preferably includes a question about the degree of eye opening as compared with the normal condition of the user.

In the user state analysis method of the present invention
The question about the condition of the user's face preferably includes a question about the darkness of the bear around the eyes as compared to the normal condition of the user.

In the user state analysis method of the present invention
The question about the condition of the user's face preferably includes a question about the condition of acne or pimples as compared to the normal condition of the user.

In the user state analysis method of the present invention
The question about the condition of the user's face preferably includes a question about how the corner of the mouth is raised or lowered as compared with the normal condition of the user.

In the user state analysis method of the present invention
The question about the condition of the user's face preferably includes a question about the condition of moisturizing the lips as compared to the normal condition of the user.

According to the present invention, "open eyes" compared to the normal state of the user, which is closely related to the emotion or fatigue state of the user and can be easily judged by looking at the image of the user's face. Specific facial condition information such as "condition", "thickness of bear around eyes", "acne or pimple condition", "up or down condition of mouth corner" and / or "moisturizing condition of lips" Is acquired, so that the user's emotions or fatigue state can be analyzed accurately and easily.

In addition, the user knows which part of his / her face should be looked at in order to know his / her emotional or fatigued state. Therefore, even in daily life, these facial parts become more concerned, and the emotions appearing on one's face or the awareness of the fatigued state are more effectively promoted.

In the user state analysis method of the present invention
The step of capturing an image of the user's face is executed before or after the input reception step.
The image analysis step is preferably a step of analyzing an image of the user's face captured before or after the input reception step.

If the user captures an image of the user's face while the user is inputting an answer to a question about the condition of the face, the user is not focused on capturing the image of the face and camera shake occurs. If the image is not taken by a device having a function of correcting the image, the image pickup position or the like may shift, and as a result, it may be difficult to maintain the accuracy of image analysis.

Therefore, it is even better if the user's face image can be captured in a state where the user can concentrate on capturing the face image.

According to the present invention, the step of capturing an image of the user's face is executed before or after the input reception step. Therefore, since the face image can be acquired while the user is focused on capturing the face image, the possibility that the imaging position or the like is displaced can be reduced, and the accuracy of the image analysis can be maintained.

In the user state analysis method of the present invention
The step of capturing an image of the user's face is executed during the execution of the input reception step.
The image analysis step is preferably a step of analyzing the image of the user's face captured during the execution of the input reception step.

Alternatively, in the user state analysis method of the present invention.
The step of capturing an image of the user's face is executed before or during the input reception step.
The image analysis step is preferably executed during the execution of the input acceptance step.

It may take a certain amount of time to input the answer to the question about the facial condition and to capture the facial image required for analysis, but the long time required for such an operation is annoying to the user. Therefore, it may affect the user's emotions or fatigue. Alternatively, the operation may be stopped due to the annoyance, and as a result, the user may not be aware of the user's emotional or fatigued state.

According to the present invention, since the step of capturing an image of the user's face or the image analysis step is executed during the execution of the input reception step, the time required for the operation or the like is shortened by that amount, which is efficient. It is possible to reduce the possibility that the user will feel annoyed.

The user state analysis system of the present invention is
An image pickup unit that captures an image of the user's face,
A display unit that displays information and
An image pickup control unit that causes the image pickup unit to capture an image of the user's face,
A facial state in which an image of the user's face captured by the imaging unit is displayed on the display unit and has a correlation with an emotional or fatigue state displayed on the user's face displayed on the display unit. A display control unit that displays a question about
An image of the user's face displayed on the display unit, an input unit arranged at a position where the user can operate while looking at the question, and an input unit.
An input receiving unit that accepts an input of an answer by the user to the question via the input unit with the image of the face displayed on the display unit.
An answer analysis unit that analyzes the answers of the user to the question received by the input reception unit, and
It is characterized by including a user state information generation unit that generates user state information indicating a user state that is one or both of the emotion and fatigue states of the user based on the analysis result by the response analysis unit.

Alternatively, the user state analysis system of the present invention is
The mirror part that reflects the user's face and
A display unit that is arranged at a position where the user can be seen at the same time as the user's face reflected in the mirror unit and displays information.
A display control unit that displays a question about the facial condition on the display unit, which has a correlation with the emotion or fatigue state appearing on the user's face reflected in the mirror unit.
An input unit arranged at a position where the user can operate while looking at the user's face reflected in the mirror unit and the question displayed on the display unit.
An input reception unit that accepts input of an answer by the user to a question about the condition of the user's face through the input unit while showing an image of the user's face reflected in the mirror unit.
An answer analysis unit that analyzes the user's answer to the question about the user's facial condition received by the input reception unit, and the answer analysis unit.
It is characterized by including a user state information generation unit that generates user state information indicating a user state that is one or both of the emotion and fatigue states of the user based on the analysis result by the response analysis unit.

According to the present invention, the user answers a question about the condition of the face while looking at his / her own face, so that the answer is easy. In addition, since you will have the opportunity to calmly look back on your facial condition, you will be promoted to be aware of the facial condition that is closely related to your emotions or fatigue, and you may be able to answer your facial condition accurately. It will increase.

This makes it possible to accurately generate user state information indicating the user state, which is one or both of the user's emotions and fatigue state.

The block diagram which shows the whole picture of the user state analysis system of this invention. The flowchart which shows the processing content of the user state analysis method of this invention. The figure which shows the display content of the user state analysis method of this invention. The figure which shows the display content of the user state analysis method of this invention. The flowchart which shows the processing content of the user state analysis method of this invention. The flowchart which shows the processing content of the user state analysis method of this invention. The flowchart which shows the processing content of the user state analysis method of this invention. The flowchart which shows the processing content of the user state analysis method of this invention. The figure which shows the example of the solution provided by the user state analysis method of this invention. The figure which shows the example of the solution provided by the user state analysis method of this invention. The figure which shows the display content of the user state analysis method of this invention.

<Configuration of user status analysis system>
First, with reference to FIG. 1, the configuration of the user state analysis system used in the user state analysis method of the present embodiment will be described. The same components may be designated by the same reference numerals and description thereof may be omitted.

The user state analysis system of the present embodiment is a computer network system including, for example, a mobile terminal 10 and a server 30.

The mobile terminal 10 and the server 30 are connected to each other so as to be able to communicate with each other by a communication network 90 such as a mobile phone network. The number of mobile terminals 10 and the number of servers 30 may be plural.

The mobile terminal 10 includes, for example, an image pickup unit 110, a display unit 130, a mobile terminal control unit 150, a vibration generation unit 170, a voice reproduction unit 190, and an input unit 210. In the present embodiment, the mobile terminal 10 is a smartphone, but it may be a general-purpose terminal such as a tablet terminal, or it may be a dedicated terminal that can be carried by the user.

The image pickup unit 110 is a camera that captures an image of the user's face. In the present embodiment, since the mobile terminal 10 is a smartphone, both the in-camera and the out-camera correspond to the image pickup unit 110.

The display unit 130 is a display that displays information to the user.

The mobile terminal control unit 150 is composed of an arithmetic processing unit such as a CPU (Central Processing Unit), a memory, an I / O (Input / Output) device, and the like. The mobile terminal control unit 150 functions as, for example, an image pickup control unit 151, a display control unit 153, and an input reception unit 155 by reading and executing a predetermined program, or further, an image analysis unit 157, a solution provision unit 159, and a mode. It functions as a designated reception unit 161.

The image pickup control unit 151 causes the image pickup unit 110 to take an image of the skin of the user U's face or fingers.

The display control unit 153 displays the image of the face of the user U captured by the image pickup unit 110 on the display unit 130, and displays a question about the state of the face of the user U displayed on the display unit 130 on the display unit 130. do. The question is a question about the state of the face, which has a correlation with the emotion or fatigue state appearing on the face of the user U displayed on the display unit 130.

The input reception unit 155 receives the input of the answer by the user to the question about the face condition of the user U via the input unit 210 in a state where the face image is displayed on the display unit 130.

The image analysis unit 157 analyzes the image of the skin of the user U's face or fingers captured by the image pickup unit 110.

The solution providing unit 159 provides the user U with the solution determined by the solution determining unit 335.

The mode designation reception unit 161 receives the designation of the operation mode of the user state analysis system by the user U.

The vibration generator 170 is, for example, a vibrator included in the solution, which is a periodic vibration that continues for a predetermined time and that provides the user U with a vibration whose period changes stepwise from the start to the end of the vibration. ..

The voice reproduction unit 190 is, for example, a speaker that provides the user U with a voice that continues for a predetermined time, including a human voice, natural sounds, or music for improving the user condition, which is included in the solution.

The input unit 210 is, for example, a touch panel, a keyboard, a mouse, or other pointing device that accepts input by the user.

If the mobile terminal 10 is a smartphone or tablet, the input unit 210 is usually a touch panel, so that the user can answer the question while looking at the question about the face image and the face condition displayed on the display unit 130. An answer can be input, but even if it is composed of other devices, the display unit 130 and the input unit 210 are arranged in a positional relationship in which the answer can be input while viewing the question about the facial image and the facial condition. To.

The server 30 is, for example, a computer including a server storage unit 310 and a server control unit 330. The number of servers 30 may be plural or singular.

The server storage unit 310 includes, for example, a storage device such as a ROM (Read Only Memory), a RAM (Random Access Memory), and an HDD (Hard Disk Drive). The server storage unit 310 functions as a storage unit in the present embodiment.

The server storage unit 310 stores information on a plurality of solutions that can be provided to the user, for example, to improve the user state, which is one or both of the user's emotional and fatigue states, and also is a user state analysis system. As a result of processing, the information necessary for processing of the user state analysis system is stored.

The server control unit 330 is composed of an arithmetic processing unit such as a CPU, a memory, an I / O device, and the like. The server control unit 330 functions as, for example, a response analysis unit 331 and a user status information generation unit 333 by reading and executing a predetermined program, or further, as a solution determination unit 335, a heart rate acquisition unit 337, and a status determination unit 339. Function.

The answer analysis unit 331 analyzes the answer by the user U to the question about the facial condition of the user U received by the input reception unit 155.

The user state information generation unit 333 generates user state information indicating the user state of the user U based on the analysis results by the response analysis unit 331 and the image analysis unit 157.

The solution determination unit 335 determines a solution according to the user status information from a plurality of solutions.

The heart rate acquisition unit 337 acquires information regarding the heart rate of the user U, for example, the heart rate of the user U from the analysis result of the image of the skin of the face or the finger of the user U by the image analysis unit 157.

The state determination unit 339 determines whether the user U is in the first state to provide a passive solution which is a passive stimulus to the user U's tactile or auditory sense based on the user state information, or gives the user U a predetermined action. Determine if you are in a second state to provide an active solution that encourages you.

<Outline of processing>
Next, a series of steps by the user state analysis method of the present embodiment will be described with reference to FIGS. 2 to 11.

<Mode designation reception step>
When a series of processes are started, the mode designation reception unit 161 first executes a mode designation reception step for accepting the operation mode designation by the user U (FIG. 2 / S10).

The mode designation reception unit 161 displays, for example, a screen prompting the user U to input an operation mode on the display unit 130.

FIG. 3 is an example of a screen prompting the user U to input an operation mode. In FIG. 3, the icon M1 for designating "Midnight Blue" corresponding to the face mode, the icon M21 for designating "Red" corresponding to the first finger mode, and the "Blue" corresponding to the second finger mode. The icon M22 for designating is displayed, and the icon M23 for designating "Green" corresponding to the third finger mode is displayed.

When the user U taps any of the icons, the mode designation reception unit 161 accepts the designation of the corresponding mode and ends this step.

In the present embodiment, the first finger mode is a mode for guiding the user U to an awake state, and the second finger mode is a mode for guiding the user U to a relaxed state, and the third finger is used. The mode is a mode for guiding the user U to a calm state.

The "History" icon H1 is also displayed on the same screen. When the user U taps the icon, the display control unit 153 acquires the user status information stored in the server storage unit 310 and displays it on the display unit 130 in response to the operation.

<First imaging step>
When the face mode is designated as the operation mode (FIG. 2 / S20: face mode), the imaging control unit 151 executes the first imaging step of causing the imaging unit 110 to capture an image of the face of the user U.

At this time, the image pickup control unit 151 starts imaging the face image of the user U to be displayed on the display unit 130 in the face image / question display step, for example, via the in-camera which is the image pickup unit 110. Alternatively, at this time, the image pickup control unit 151 may capture one still image of the user U's face for display on the display unit 130, for example, in the face image / question display step.

On the other hand, when the mode corresponding to the finger mode is specified as the operation mode (FIG. 2 / S20: finger mode), the processes of FIGS. 2 / S30 to S70 are not executed, and the processes of FIGS. 2 / S90 and subsequent steps are executed. Will be done.

<Face image / question display step>
Next, the display control unit 153 shows a face image of the user U and displays a face image / question display that displays a question about the face condition that has a correlation with the emotion or fatigue state appearing on the user U's face. Execute the step (FIG. 2 / S50).

FIG. 4 is an example of a screen displayed in the face image / question display step. As shown in FIG. 4, for example, the display control unit 153 displays the captured image F of the face of the user U and the first question about the condition of the face, "the degree of opening of the eyes is compared with usual". Q1 and the answer button A1 are displayed on the display unit 130.

As shown in FIG. 4, for example, the display control unit 153 taps any one of the five grades from "open" to "closed" in the answer button A1 to answer the facial condition. Display a prompting screen.

<Input acceptance step>
For example, when the display control unit 153 displays the first question as described above, the input reception unit 155 inputs the answer by the user U to the question about the facial condition of the user U, and the user uses the image of the face. The input acceptance step of accepting while showing to U is started (FIG. 2 / S70).

When the user U taps any part of the five-level evaluation of the answer button A1, the input receiving unit 155 accepts the corresponding evaluation value as the input of the answer to the question.

The processes of FIGS. 2 / S50 and S70 are repeatedly executed until the answer to the last question is input (FIG. 2 / S80: Yes).

That is, for example, the display control unit 153 asks the second to fifth questions about the condition of the user's face, such as the question about the darkness of the bear around the eyes, the condition of acne or pimples, as compared with the normal condition of the user. The input reception unit 155 accepts the input of the answer by the user U to each of these questions by displaying in order the question regarding the degree of raising or lowering of the corner of the mouth, the question regarding the state of moisturizing the lips.

Then, for example, the input reception unit 155 transmits the information of the answer by the user U to each question received in this way to the server 30 and ends the input reception step.

<Second imaging step>
After that, the image pickup control unit 151 executes a second image pickup step of capturing an image of the skin of the user U's face or fingers for analysis in the image analysis step.

At this time, when the face mode is specified (FIG. 5 / S91: face mode), the image pickup control unit 151 captures an image of the face of the user U via the in-camera which is the image pickup unit 110. (FIG. 5 / S93) The second imaging step is completed.

On the other hand, when the finger mode is specified (FIG. 5 / S91: finger mode), the image pickup control unit 151 captures an image of the skin of the finger of the user U via the out-camera which is the image pickup unit 110 (FIG. 5 / S91: finger mode). FIG. 5 / S95) The second imaging step is completed.

At this time, for example, the display control unit 153 displays a frame for guiding the position of the face or finger of the user U to the correct position on the display unit 130, and the image pickup control unit 151 displays the face or finger of the user U at the correct position. When the image is recognized, the moving image of the skin of the user U's face or fingers is imaged for a predetermined time (for example, 30 seconds) via the image pickup unit 110.

Further, for example, if the face or finger of the user U deviates from the correct position during the imaging of the moving image, the imaging control unit 151 stops the imaging of the moving image, and the display control unit 153 positions the face or finger of the user U. A message prompting the user to return the device to the correct position is displayed on the display unit 130. Then, when the image pickup control unit 151 recognizes that the face or finger of the user U has returned to the correct position, the image pickup control unit 151 resumes the imaging of the moving image of the skin of the face or finger of the user U. Alternatively, the image captured by the image pickup control unit 151 may be a plurality of continuously captured still images.

<Image analysis step>
After that, the image analysis unit 157 executes an image analysis step of analyzing the captured image of the face or finger skin of the user U (FIG. 2 / S110).

That is, when the face mode is specified (FIG. 6 / S111: face mode), the image analysis unit 157 analyzes the image of the user U's face captured in the second imaging step (FIG. 6 / S113). When the finger mode is specified (FIG. 6 / S111: finger mode), the image of the user U's finger captured in the second imaging step is analyzed (FIG. 6 / S115).

More specifically, the image analysis unit 157 acquires information on changes in the brightness of each color component of RGB from, for example, a moving image which is an image of the skin or a continuously captured still image.

Then, for example, the image analysis unit 157 transmits the information on the change in the luminance of each color component of RGB based on the image of the face of the user U to the server 30, and ends the image analysis step.

<Answer analysis step>
When the answer analysis unit 331 receives, for example, the information of the answer by the user U to each question from the mobile terminal 10 and the information of the change in the luminance of each color component of RGB, the user U responds to the question about the state of the face of the user U. Perform an answer analysis step to analyze the answer by.

More specifically, the response analysis unit 331, for example, regarding the answer by the user U to the question about the facial condition of the user U received in the input reception step, the answer and the emotional or fatigue state appearing on the face of the user U. Using the first correlation information showing the correlation with the user U, the analysis result of one or both of the emotional state and the fatigue state appearing on the face of the user U is determined.

Various information may be used as the first correlation information indicating the correlation between the answer and the emotion or fatigue state appearing on the face of the user U. In the present embodiment, the response analysis unit 331 uses, for example, the following formula as the first correlation information, and the sign of fatigue appearing on the face of the user U is changed from "energetic" (= 10) to "tired" (= 0). ) Is indexed as a sign of fatigue score, and the score is determined as the analysis result of the answer.

F = p1 + p2 + p3 + p4 + p5
However, F is a value (a sign score of fatigue) that is an index of the emotion or fatigue state appearing on the face, and the maximum value is 10 and the minimum value is 0. That is, for example, when the total value of p1 to p5 exceeds 10, it is set to 10, and when it is less than 0, it is set to 0.

For p1, the degree of eye opening compared to the normal state of the user U was evaluated on a 5-point scale from "open" (= 5 points) to "closed" (= 1 point). The value.

In p2, the darkness of the bear around the eyes compared to the normal state of the user U is 0. From "thinning" (= 1 point) to "thickening" (= -1 point). It is a value evaluated in 5 steps in 5 point increments.

p3 shows the condition of acne or pimples compared to the normal condition of user U in 0.5 point increments from "improved" (= 1 point) to "worse" (= -1 point). It is a value evaluated in 5 stages of.

In p4, the degree of increase or decrease of the corner of the mouth compared to the normal state of the user U is 5 steps from "up" (= 5 points) to "down" (= 1 point) in 1-point increments. It is a value evaluated in.

p5 shows the moisturized state of the lips compared to the normal state of the user U in 0.5 point increments from "pulling" (= 1 point) to "bulky" (= -1 point). It is a value evaluated in 5 stages of.

<User status information generation step>
Then, based on the analysis result of the answer by the user U to the question about the face condition of the user U and the analysis result of the facial image, the user state including one or both of the emotional state and the fatigue state appearing on the face of the user U. The user state information generation step for generating the user state information indicating the above is executed (FIG. 2 / S150).

More specifically, in the user state information generation step, first, the heart rate acquisition unit 337 is transmitted from the information of the change in the brightness of each color component of RGB based on the image of the face of the user U, and the user U is used. As information about the heart rate of the user U, the heart rate of the user U is acquired (FIG. 7 / S151).

Further, the heartbeat acquisition unit 337 acquires information on the fluctuation of the heartbeat of the user U as information on the heartbeat of the user U from the information on the change in the luminance of each color component of RGB based on the image of the face of the user U (FIG. 7 / S153). ). This is because the fluctuation of the heartbeat is closely related to the fatigue state of the user U.

The information on the fluctuation of the heartbeat is, for example, CVR-R or the total power of LF (Low Frequency) which is a low frequency component of the fluctuation of the heartbeat and HF (High) which is a high frequency component obtained by frequency analysis of the fluctuation of the heartbeat. It is a total power value which is the sum of the total power of Frequency).

Then, the user state information generation unit 333 uses the second correlation information indicating the correlation between the information on the heart rate and the autonomic nervous balance of the user U for the heart rate, which is the information on the user's heart rate, to autonomously use the user U. Obtain information on neural balance (FIG. 7 / S153).

The information on the autonomic nerve balance is, for example, the ratio of LF to HF (LF / HF) obtained by frequency analysis of the fluctuation of the heartbeat.

Then, the user state information generation unit 333 determines the overall score regarding the user state of the user U based on the analysis result of the answer to the question regarding the face state of the user U, the analysis result of the face image, and the information regarding the autonomic nerve balance. (Fig. 7 / S155).

More specifically, the user state information generation unit 333 determines the autonomic nerve balance score, which is an index of information on the autonomic nerve balance with a score between 1 and 10. For example, it is determined that the more predominant the sympathetic nerve is, the closer the score is to 1, and the more predominant the parasympathetic nerve is, the closer the score is to 10.

Further, the user state information generation unit 333 determines a fatigue degree score based on the heartbeat, for example, the CVR-R (or total power value) acquired as information on the fluctuation of the heartbeat is indexed by a score between 1 and 10. For example, it is determined that the higher the degree of heart rate fluctuation, the closer the score is to 1, and the lower the degree of heart rate fluctuation, the closer the score is to 10.

Then, the user state information generation unit 333 quantifies the signs of fatigue appearing on the face of the user U determined in the response analysis step into the two values of the autonomic nerve balance score and the fatigue degree score based on the heartbeat. The average value of the three values including the symptom score of is determined as the total score, which is an index of the user's emotional or fatigue state.

Then, the user state information generation unit 333 generates the user state information (FIG. 7 / S157), and ends the user state information generation step.

More specifically, the user status information generation unit 333 has quantified the signs of fatigue appearing on the face of the user U, scoring the information on the fluctuation of the heartbeat, and scoring the information on the autonomic nerve balance. A set of values and total scores is used as user status information.

<User status information storage step>
In the user state information storage step, the user state information generation unit 333 stores the user state information generated in the user state information generation step in the server storage unit 310 in chronological order (FIG. 2 / S170).

<Solution decision step>
Then, the solution determination unit 335 determines a solution according to the user state information from a plurality of solutions that can be provided to the user U in order to improve the user state that is one or both of the emotional state and the fatigue state of the user U. Perform the solution decision step (Fig. 2 / S190).

That is, for example, when the solution determination unit 335 starts the solution determination step, it first confirms the designated operation mode (FIG. 8 / S191).

Then, when the designated operation mode is the face mode (FIG. 8 / S191: face mode), the solution determination unit 335 determines the emotional or fatigue state appearing on the face of the user U included in the user state information. Based on, User U should be in the first state to provide a passive solution that is a passive stimulus to User U's tactile or auditory sense, or User U should be provided with an active solution that encourages a given action. It is determined whether or not the two states are present (FIG. 8 / S193).

Various methods may be adopted as a method for determining whether the user is in the first state or the second state, but in the present embodiment, the solution determination unit 335 relates to the autonomic nerve balance included in the user state information. When the value obtained by scoring the information is 4 or less and the value obtained by quantifying the sign of fatigue appearing on the face of the user U is 6 or more (that is, the sympathetic nerve is dominant and the sign of fatigue appearing on the face is used. It is determined that the user U is in the second state (when it is in a healthy state), and in other cases, it is determined that the user U is in the first state.

When it is determined that the user U is in the first state (FIG. 8 / S193: first state), the solution determination unit 335 determines a solution according to the user state information from the passive solution (FIG. 8 / S195). ).

On the other hand, when it is determined that the user U is in the second state (FIG. 8 / S193: second state), the solution determination unit 335 determines a solution according to the user state information from the active solution (FIG. 8). / S197).

Further, when the designated operation mode is the finger mode (FIG. 8 / S191: finger mode), the solution determination unit 335 determines the solution according to the designated finger mode (FIG. 8 / S199). ..

Then, the solution determination unit 335 transmits the information about the solution determined in this way and the user status information generated in the user status information generation step to the mobile terminal 10 to end the solution determination step.

More specifically, for example, the operation mode and the total score and the correspondence table with the solution are stored in the server storage unit 310 in advance, and the solution determination unit 335 refers to the correspondence table and includes it in the user status information. Determine the solution that corresponds to the overall score.

In the present embodiment, the solution determined by the solution determination unit 335 is a periodic vibration that continues for a predetermined time, and the vibration whose cycle changes stepwise from the start to the end of the vibration and the user state. A combination (passive solution) of human voice, natural sounds or voices that last for a given period of time to improve, and if necessary, further breathe or meditate in a given manner to User U. Includes information that prompts an action such as, or an image that guides the user U to a predetermined state (active solution).

<Specific example of solution>
-Solution for face mode FIG. 9 shows the correspondence between the total score and the solution for face mode in this embodiment. As shown in FIG. 9, regarding the voice included in the solution for the face mode, the solution when the user U is in the first state is classified into three groups according to the level of the total score, and the total score is divided into three groups. When is 1 to 4, it is a combination of human voice and music, when it is 5 to 7, it is a combination of natural sound and music, and when it is 8 to 10, it is only music. In this way, the solution determination unit 335 determines the voice to be combined with the vibration based on the user state information.

Regarding vibration, the solution determination unit 335 determines the cycle at the start of vibration and the cycle from the start to the end based on the user state information and more specifically based on the heart rate of the user U. To determine the fluctuation pattern of.

That is, for example, when the heart rate of the user U is 80 or more, the solution determination unit 335 determines that the cycle at the start of vibration is 85 times / minute, and the cycle is continued for 10 seconds. Further, the solution determination unit 335 changes the fluctuation pattern of the cycle from the start to the end, continuing the vibration of 75 times / minute cycle for 10 seconds, continuing the vibration of 65 times / minute cycle for 20 seconds, and so on. Finally, the vibration pattern of 50 times / minute is determined, that is, the vibration is continued for 120 seconds or more until the solution is completed.

For example, when the heart rate of the user U is relatively high, and when the heart rate is 70 or more in the present embodiment, only the vibration is provided for a certain period (for example, 20 seconds) from the start of the vibration. The solution may then be decided to provide the audio in combination.

This is because when the heartbeat is high to a certain extent, it is easier to obtain the effect of improving emotions and fatigue by the voice by first calming the heartbeat and then listening to the voice.

In the present embodiment, when it is determined that the user U is in the first state, the solution determination unit 335 determines three passive solutions corresponding to the total score, and the user U is in the second state. If it is determined that, one active solution corresponding to the total score and two passive solutions are determined. Therefore, when the total score is 1 to 4, when the total score is 5 to 7, a plurality of voice solutions corresponding to each of 8 to 10 are stored in the server storage unit 310.

Then, the solution when the user U is in the second state is a combination of music, "characters supporting the action" which is information for prompting the user U to perform a predetermined action, and vibration.

"Characters that support behavior" are, for example, "Let's do a simple mindfulness meditation while feeling the vibration" and "Stretch your back and posture" that change at regular intervals, as shown in the lower part of Fig. 10. It is a combination of sentences that encourage breathing or meditation in a predetermined action or method, such as "let's correct" and "let's relax the shoulders and breathe slowly through the nose".

-Solution for finger mode FIG. 10 shows the correspondence between the heart rate of the user U and the solution for the finger mode for each finger mode in the present embodiment. As shown in FIG. 10, the solution for the finger mode is determined by the specified finger mode (Red, Blue, Green) and the heart rate of the user U, the vibration and the predetermined to the user U. It is a combination of the information for prompting the operation and the image for guiding the user U to a predetermined state.

More specifically, the solution determination unit 335 so that the vibration cycle is gradually shortened from the start to the end of the vibration when "Red" corresponding to the first finger mode is specified. That is, the fluctuation pattern of the vibration cycle to be changed so that the frequency increases is determined. This is because the pull-in effect accelerates the heartbeat of the user U and induces the user U to be in an awake state.

Further, the information that urges the user U to perform a predetermined action is, for example, the text of a comment "Let's correct the posture and take a big breath while feeling the vibration", and the image that guides the user U to the predetermined state is, for example, It is a still image that can imagine awakening.

Then, in the solution determination unit 335, when "Bule" corresponding to the second finger mode is specified, the vibration cycle is gradually lengthened from the start to the end of the vibration, that is, the frequency is increased. The fluctuation pattern of the cycle to be varied so as to decrease is determined. The speed at which the frequency decreases at this time is steeper than when "Green" is specified. This is because the pull-in effect rapidly calms the heartbeat of the user U and induces the user U to be in a calm state.

In addition, the information that urges the user U to perform a predetermined action is, for example, the text of a comment "Let's raise the shoulder once, relax and breathe slowly while feeling the vibration", and the image that guides the user U to the predetermined state. Is, for example, a still image that can be imagined as calm.

Then, in the solution determination unit 335, when "Green" corresponding to the third finger mode is specified, the vibration cycle is gradually lengthened from the start to the end of the vibration, that is, the frequency is increased. The fluctuation pattern of the cycle to be varied so as to decrease is determined. The speed at which the frequency decreases at this time is slower than when "Blue" is specified. This is because the pull-in effect gently calms the heartbeat of the user U and induces the user U to be in a relaxed state.

In addition, the information that urges the user U to perform a predetermined action is, for example, the text of a comment "Let's relax the shoulders and breathe deeply while feeling the vibration", and the image that guides the user U to the predetermined state is. For example, it is a still image that allows you to imagine relaxing.

In addition, the solution determination unit 335 may determine a fluctuation pattern such that, for example, when the finger mode of Blue or Green is specified, the vibration is terminated after the frequency is finally increased by a predetermined degree.

This is done, for example, by encouraging User U to regain normal heart rate after being calm or relaxed, for example, assuming that User U will be provided with a solution during a business lunch break. This is to guide the user U to a state where the business can be resumed smoothly after the break.

<User status information display step>
After that, for example, when the mobile terminal 10 receives the information about the solution determined from the server 30 and the user state information generated in the user state information generation step, the user state information display step for displaying the user state information is executed. (FIG. 2 / S210).

FIG. 11 is an example of the content displayed in the user status information display step when the face mode is specified. That is, as shown in FIG. 11, the display control unit 153 indicates the numerical value of the sign of fatigue appearing on the face of the user U, which is included in the received user status information, and indicates the “tiredness appearing on the face”. The display R1 of "signs", the display R2 of "energy reserve" showing the value obtained by scoring the information on the fluctuation of the heartbeat, and the display R3 of the "autonomic nerve balance" showing the value obtained by scoring the information on the autonomic nerve balance. , Display of the value of the total score T1 is displayed side by side as user status information.

Then, the display control unit 153 displays three icons (S1, S2, S3) for starting the provision of the solution determined this time side by side at the lower part of the screen. In addition, the icon includes, for example, a title indicating the content of each solution and a thumbnail image.

When the finger mode is specified, the display control unit 153 may display, for example, an "autonomic nerve balance" indicating a scored value of information related to the autonomic nerve balance, and a display indicating the heart rate of the user U. Is displayed on the display unit 130.

<Solution provision step>
When the face mode is specified, the icon (S1, S2, S3) displayed on the screen by the display control unit 153 in the user status information display step for starting the provision of the solution determined this time. When the user U taps any of them, the solution providing unit 159 provides the solution corresponding to the tapped icon to the user in response to the operation.

When the finger mode is specified, the solution providing unit 159 causes the display control unit 153 to display the display of "autonomic nerve balance" and the display showing the heart rate of the user U on the display unit 130 for a certain period of time. After that, the solution decision unit 335 automatically starts to provide the solution.

That is, for example, the solution providing unit 159 starts the vibration included in the solution via the vibration generating unit 170 at the start cycle determined by the solution determining unit 335, and fluctuates according to the fluctuation pattern determined by the solution determining unit 335. Provide continuously for a specified time.

Further, the solution providing unit 159 reproduces the voice determined by the solution determining unit 335 via the voice reproducing unit 190, and the information or the user U determined by the solution determining unit 335 to prompt the user U to perform a predetermined operation is in a predetermined state. The image to be guided to the user U is displayed via the display unit 130 and provided to the user U.

The embodiments of the present invention have been described above, but the present invention is not limited thereto. Various changes can be made without departing from the spirit of the present invention.

That is, for example, in the above description, the embodiment in which the user's face is displayed as an image on the display unit 130 has been described, but the present invention is not limited to this. That is, for example, the user state analysis method of the present invention is executed by using a user state analysis system which is a so-called smart mirror, and the user may answer a question while looking at his / her face reflected in the mirror. Further, the smart mirror may or may not be portable by the user.

More specifically, the user state analysis system of the present embodiment includes a mirror unit that reflects the face of the user U. Then, the display unit 130 is arranged at a position where the user U can be seen at the same time as the face of the user U reflected in the mirror unit. The display control unit 153 displays on the display unit 130 a question about the state of the face of the user U reflected in the mirror unit. The input unit 210 is arranged at a position where the user U can operate while looking at the question about the face of the user U reflected in the mirror unit and the state of the face of the user U displayed on the display unit 130. Then, the input reception unit 155 accepts the input of the answer by the user U to the question about the face condition of the user U via the input unit 210 in a state of showing the image of the face of the user U reflected in the mirror unit.

Alternatively, for example, in the answer analysis step, the answer analysis unit 331 inputs the answer by the user U to the question about the facial condition of the user U, and the four factors of happiness appearing on the face of the user U, the burnout state, and the life. Satisfaction, negative or positive on the PANAS scale, degree of resilience, subjective well-being scale, emotions such as stress tolerance, or insomnia, physical fatigue, mental fatigue, amount of fatigue components in the body An analysis model that outputs the fatigue state such as the above may be used as the first correlation information to determine the analysis result of one or both of the emotion and the fatigue state appearing on the face of the user U.

In this case, the response analysis unit 331 analyzes the output value of whether the emotional or fatigue state of the analysis target is higher or lower than the reference value indicating that the state is normal, using the analysis model. Determine as a result.

Alternatively, in the above embodiment, after the input acceptance step, a configuration for capturing an image of the user U's face for analysis in the image analysis step (second imaging step) and a configuration for executing the image analysis step have been described. However, it is not limited to this.

That is, for example, the step of capturing an image of the user U's face for analysis in the image analysis step (second imaging step) is executed before the input acceptance step, and the image analysis step is performed before the input acceptance step. It may be configured to be a step of analyzing the captured image of the user U's face. As described above, the step of capturing the image of the user U's face for performing the analysis in the image analysis step (second imaging step) may be performed before or after the input reception step, and the image analysis step may be performed accordingly. It is a step of analyzing the image of the user U's face captured either before or after the input reception step.

Alternatively, for example, the step of capturing an image of the user U's face is executed during the execution of the input acceptance step, and the image analysis step is a step of analyzing the image of the user U's face captured during the execution of the input acceptance step. It may be configured to be.

In this case, a step of capturing an image of the user U's face to be displayed on the display unit 130 in the face image / question display step (first imaging step) and a step of capturing the user U's face for analysis in the image analysis step. The step of imaging the image (second imaging step) is performed as a single step. That is, the image of the user's face captured in this way is used both for displaying on the display unit 130 in the face image / question display step and for performing analysis in the image analysis step.

Further, in this case, for example, the image analysis unit 157 satisfies the conditions necessary for maintaining the accuracy of image analysis, such as an image having a sufficiently low degree of camera shake from the images captured by the image pickup unit 110. It is configured to select an image and analyze the selected image.

Alternatively, for example, the step of capturing an image of the user U's face may be configured to be executed before or during the input acceptance step, and the image analysis step may be executed during the execution of the input acceptance step.

110 ... Imaging unit, 130 ... Display unit, 151 ... Imaging control unit, 153 ... Display control unit, 155 ... Input reception unit, 157 ... Image analysis unit, 159 ... Solution provision unit, 161 ... Mode designation reception unit, 170 ... Vibration Generation unit, 190 ... Voice reproduction unit, 310 ... Server storage unit, 331 ... Answer analysis unit, 333 ... User status information generation unit, 335 ... Solution determination unit, 337 ... Heart rate acquisition unit, 339 ... Status determination unit.

Claims (20)

Includes a step to show the user an image of the user's face and to display a question about the condition of the face that correlates with the emotional or fatigue condition appearing on the user's face.
An input reception step that accepts an answer by the user to a question about the state of the user's face while showing the image of the face to the user.
An answer analysis step that analyzes the user's answer to the question about the user's facial condition, and
It is characterized by comprising a user state information generation step of generating user state information indicating a user state including one or both of emotions and fatigue states appearing on the user's face based on the analysis result of the answer. How to analyze user status. In the user condition analysis method according to claim 1,
A solution determination step for determining a solution according to the user status information from a plurality of solutions that can be provided to the user in order to improve the user status of the user.
A user condition analysis method comprising further comprising providing the determined solution to the user. In the user condition analysis method according to claim 1 or 2.
The step of capturing an image of the user's face and
An image analysis step that analyzes the captured image of the user's face,
Including
The user state analysis method is characterized in that the user state information generation step is a step of generating the user state information based on the answer and the analysis result of the face image. In the user condition analysis method according to any one of claims 1 to 3,
A user state analysis method comprising a step of displaying the generated user state information. In the user condition analysis method according to any one of claims 1 to 4.
A step to store the generated user status information in chronological order,
A user state analysis method comprising a step of displaying the stored user state information in response to the user's operation. In the user condition analysis method according to any one of claims 1 to 5.
The answer analysis step is a first correlation showing the correlation between the user's answer to the accepted question about the user's facial condition and the emotional or fatigue state appearing on the user's face. A user condition analysis method comprising the step of determining the analysis result of one or both of the emotional and fatigue states appearing on the user's face using information. In the user condition analysis method according to claim 2,
The step of capturing an image of the user's face and
An image analysis step that analyzes the captured image of the user's face,
Including
The answer analysis step includes a step of analyzing the answer to determine a fatigue sign score, which is an index of the emotion or fatigue state appearing on the user's face.
The user state information generation step is
A step of acquiring information on the user's heartbeat from the analysis result of the face image, and
A step of determining a heart rate-based fatigue score, which is an index of a heart rate-based fatigue state based on the user's heart rate information.
A step of determining an overall score, which is an index of the user's emotional or fatigue state, based on the fatigue sign score and the fatigue degree score based on the heartbeat.
Including a step of generating the user state information, including the overall score,
The solution determination step is a user state analysis method including a step of determining a solution to be provided to the user based on the total score. In the user condition analysis method according to claim 7,
The step to determine the overall score is
A step of acquiring information on the autonomic nervous balance of the user by using the information on the heartbeat of the user and the second correlation information showing the correlation between the information on the heartbeat and the autonomic nervous balance of the user.
A user state analysis method comprising: a step of determining the overall score based on information including information about the autonomic nerve balance. In the user condition analysis method according to claim 2,
The solution determination step is
A first state in which the user should provide a passive solution that is a passive stimulus to the user's sense of touch or hearing based on the emotional or fatigue state appearing on the user's face contained in the user state information. A step to determine if the user is in a second state to provide an active solution that encourages the user to perform a given action.
A user state analysis method comprising: a step of determining a solution according to the user state information from the plurality of solutions based on the result of the determination. In the user condition analysis method according to claim 9,
The active solution comprises a combination of periodic vibrations that last for a predetermined time, voice to improve the user condition, and information that prompts the user to breathe or meditate in a predetermined manner.
When the user is in the second state, the active solution comprising a combination of the vibration, the voice, and information prompting the user to breathe or meditate in a predetermined manner in the solution determination step. , A user condition analysis method, characterized in that it is determined as a solution to be provided to the user. In the user condition analysis method according to any one of claims 1 to 10.
The user condition analysis method, wherein the question about the facial condition of the user includes a question about the degree of opening of the eyes as compared with the normal condition of the user. In the user condition analysis method according to any one of claims 1 to 11.
The user condition analysis method, wherein the question about the condition of the face of the user includes a question about the darkness of the bear around the eyes as compared with the normal condition of the user. In the user condition analysis method according to any one of claims 1 to 12,
The user condition analysis method, wherein the question about the condition of the user's face includes a question about the condition of acne or pimples as compared to the normal condition of the user. In the user condition analysis method according to any one of claims 1 to 13.
The user condition analysis method, wherein the question about the facial condition of the user includes a question about how the corner of the mouth rises or falls as compared with the normal condition of the user. In the user condition analysis method according to any one of claims 1 to 14,
The user condition analysis method, wherein the question about the condition of the face of the user includes a question about the condition of moisturizing the lips as compared with the normal condition of the user. In the user condition analysis method according to claim 3,
The step of capturing an image of the user's face is executed before or after the input reception step.
The user state analysis method is characterized in that the image analysis step is a step of analyzing an image of the user's face captured before or after the input reception step. In the user condition analysis method according to claim 3,
The step of capturing an image of the user's face is executed during the execution of the input reception step.
The user state analysis method is characterized in that the image analysis step is a step of analyzing an image of the user's face captured during the execution of the input reception step. In the user condition analysis method according to claim 3,
The step of capturing an image of the user's face is executed before or during the input reception step.
The image analysis step is a user state analysis method, characterized in that it is executed during the execution of the input reception step. An image pickup unit that captures an image of the user's face,
A display unit that displays information and
An image pickup control unit that causes the image pickup unit to capture an image of the user's face,
A facial state in which an image of the user's face captured by the imaging unit is displayed on the display unit and has a correlation with an emotional or fatigue state displayed on the user's face displayed on the display unit. A display control unit that displays a question about
An image of the user's face displayed on the display unit, an input unit arranged at a position where the user can operate while looking at the question, and an input unit.
An input receiving unit that accepts an input of an answer by the user to the question via the input unit with the image of the face displayed on the display unit.
An answer analysis unit that analyzes the answers of the user to the question received by the input reception unit, and
A user state including a user state information generation unit that generates user state information indicating a user state that is one or both of the emotion and fatigue states of the user based on the analysis result by the response analysis unit. Analytical system. A mirror that reflects the user's face,
A display unit that is arranged at a position where the user can be seen at the same time as the user's face reflected in the mirror unit and displays information.
A display control unit that displays a question about the facial condition on the display unit, which has a correlation with the emotion or fatigue state appearing on the user's face reflected in the mirror unit.
An input unit arranged at a position where the user can operate while looking at the user's face reflected in the mirror unit and the question displayed on the display unit.
An input reception unit that accepts input of an answer by the user to a question about the condition of the user's face through the input unit while showing an image of the user's face reflected in the mirror unit.
An answer analysis unit that analyzes the user's answer to the question about the user's facial condition received by the input reception unit, and the answer analysis unit.
A user state including a user state information generation unit that generates user state information indicating a user state that is one or both of the emotion and fatigue states of the user based on the analysis result by the response analysis unit. Analytical system.

Images