JP7507674B2 - USER STATE ANALYSIS METHOD AND SYSTEM - Google Patents

Description

The present invention relates to a method and system for analyzing a user's emotions or fatigue state.

A fatigue level determination processing system is known (see Patent Document 1).

JP 2010-201113 A

In the invention described in Patent Document 1, the user's level of fatigue is determined based on the LF/HF value and the results of a questionnaire about the user's own fatigue state.

It is conceivable that the invention described in Patent Document 1 could be applied to a method and system for analyzing a user's emotions or fatigue state.

However, the results of an analysis of emotions or fatigue state based on objective data may sometimes differ from the user's own thoughts about the emotions or fatigue state. For example, such a discrepancy may occur if an analysis of objective information such as an image of the skin indicates that the user is feeling considerable stress or fatigue, but the user is not aware of this.

On the other hand, even if objectively you are not feeling that stressed or fatigued, you may believe that you are stressed or tired, which can result in the above discrepancy.

However, in the invention of Patent Document 1, the user's level of fatigue is determined based on the results of a questionnaire about the user's own fatigue state, so there is little certainty that a correct answer regarding the user's fatigue state will be obtained from a user who is unaware of their own fatigue state or who believes that they are more tired than necessary.

Furthermore, analytical results based on such uncertain information cannot be expected to be highly accurate.

The present invention aims to provide a user condition analysis method and a user condition analysis system that can accurately analyze a user's emotions or fatigue state while making the user aware of the user's emotions or fatigue state.

The user status analysis method of the present invention includes the steps of:
presenting an image of the user's face to the user and displaying a question about a facial state that correlates with an emotional or fatigue state appearing on the user's face;
an input receiving step of receiving a response from the user to a question about the state of the user's face while showing an image of the face to the user;
a response analysis step of analyzing responses by the user to questions about the user's facial condition;
and a user condition information generation step of generating user condition information indicating the user condition including one or both of the emotions and fatigue state shown on the user's face based on the analysis results of the answers.

According to the present invention, an image of a user's face is shown to the user, and a question about the state of the displayed face of the user is displayed. The question is about a state of the face that is correlated with the emotion or fatigue state shown on the user's face.

The user's answer to the question is then accepted with the face image displayed, and the answer is analyzed.

In addition, based on the analysis results of the answers, user state information is generated that indicates the user state, which is either the user's emotions or fatigue state or both.

According to the present invention, the user answers questions about the condition of his/her face while looking at his/her own face, making it easy for him/her to answer. In addition, since the user has the opportunity to calmly reflect on the condition of his/her own face, he/she is encouraged to become aware of the facial condition that is closely related to emotions or fatigue state, and the possibility of being able to answer questions about his/her own facial condition accurately is increased.

This allows for accurate generation of user state information that indicates the user state, which is either or both of the user's emotions and fatigue state.

In this way, the present invention makes it possible to accurately analyze the user's emotions or fatigue state while making the user aware of it.

In the user condition analysis method of the present invention,
a solution determination step of determining a solution corresponding to the user state information from a plurality of solutions that can be provided to the user for improving the user state of the user;
providing said determined solution to said user.

According to the present invention, a solution corresponding to the user state information is determined from a plurality of solutions that can be provided to the user to improve the user state, and the determined solution is provided to the user.

This allows for accurate generation of user condition information indicating the user's condition, which may be one or both of the user's emotions and fatigue state, and for easily providing appropriate solutions in response to the user's condition, thereby improving the user's condition.

In the user condition analysis method of the present invention,
capturing an image of the user's face;
an image analysis step of analyzing the captured image of the user's face;
Further comprising:
It is preferable that the user state information generating step is a step of generating the user state information based on an analysis result of the answer and the face image.

According to the present invention, in addition to subjective information of the user, such as answers to questions about the condition of the face, objective information, such as the analysis results of the face image, is used to generate user condition information indicating the user's condition, which is either or both of the user's emotions and fatigue state.

As a result, user status information is generated taking into account information that cannot be obtained from answers to questions, making it possible to generate user status information more accurately and provide appropriate solutions accordingly.

In the user condition analysis method of the present invention,
It is preferable to include a step of displaying the generated user status information.

According to the present invention, the generated user condition information is displayed, so that the user can recognize his/her own emotions or fatigue state, and thus the user can be made effectively aware of his/her own emotions or fatigue state.

In the user condition analysis method of the present invention,
storing the generated user state information in chronological order;
It is preferable to further include a step of displaying the stored user status information in response to an operation by the user.

According to the present invention, past user condition information can be displayed in response to user operations. Therefore, the user himself can know at any time whether there is a change in the emotion or fatigue state shown on the user's face and the degree of change, so that the user himself can be made effectively aware of the user's emotion or fatigue state.

In the user condition analysis method of the present invention,
It is preferable that the answer analysis step includes a step of determining an analysis result of one or both of the emotions and fatigue state shown on the user's face using an answer by the user to a received question about the condition of the user's face and first correlation information indicating a correlation between the answer and the emotions or fatigue state shown on the user's face.

According to the present invention, the analysis results of one or both of the emotions and fatigue state shown on the user's face are determined using specific information, namely the first correlation information, so that one or both of the emotions and fatigue state shown on the user's face can be analyzed with high accuracy.

In the user condition analysis method of the present invention,
capturing an image of the user's face;
an image analysis step of analyzing the captured image of the user's face;
Further comprising:
The answer analyzing step includes a step of determining a fatigue symptom score by analyzing the answer and indexing an emotion or fatigue state shown on the face of the user,
The user state information generating step includes:
obtaining information about the user's heart rate from an analysis of the facial image;
determining a heart rate-based fatigue score that indexes a heart rate-based fatigue state based on information about the user's heart rate;
determining a total score that indexes the user's emotion or fatigue state based on the fatigue symptom score and the heart rate-based fatigue score;
generating the user status information including the overall score;
The solution determining step preferably includes a step of determining a solution to provide to the user based on the overall score.

According to the present invention, the solution to be provided to the user is determined based on a fatigue sign score that indexes the emotions or fatigue state shown on the user's face, and a fatigue level score based on the heart rate, which is derived from multiple perspectives, so that an appropriate solution can be provided according to the user's overall fatigue state.

In the user condition analysis method of the present invention,
The step of determining the overall score comprises:
acquiring information about the user's autonomic nerve balance using information about the user's heart rate and second correlation information indicating a correlation between the information about the heart rate and the user's autonomic nerve balance;
It is preferable that the method further includes a step of determining the overall score based on information including information regarding the autonomic nerve balance.

According to the present invention, the solution to be provided to the user is determined based on a comprehensive score derived from a more multifaceted perspective, including information on autonomic nerve balance, making it possible to provide a more appropriate solution according to the user's overall state of fatigue.

In the user condition analysis method of the present invention,
The solution determination step includes:
A step of determining whether the user is in a first state in which a passive solution, which is a passive stimulation to the user's sense of touch or hearing, should be provided, or in a second state in which an active solution, which encourages the user to perform a predetermined action, should be provided, based on the emotion or fatigue state shown on the user's face included in the user state information;
It is preferable that the method further includes a step of determining a solution corresponding to the user state information from the plurality of solutions based on a result of the determination.

The prescribed action has the effect of improving the user's emotions and fatigue state, but if the user's emotions or fatigue state are not appropriate for performing the prescribed action, providing a passive solution, such as passive stimulation of the senses of touch or hearing, is effective in improving the user's emotions and fatigue state.

According to the present invention, it is determined whether the user is in a first state in which a passive solution, which is a passive stimulation of the sense of touch or hearing, should be provided, or in a second state in which an active solution, which encourages a predetermined action, should be provided, based on the emotion or fatigue state shown on the user's face.

Then, a solution is determined based on the results of this judgment, so the type of solution provided can be appropriately changed depending on the user's emotions and fatigue state.

In the user condition analysis method of the present invention,
the active solution includes a combination of periodic vibrations lasting a predetermined time, audio to improve the user's condition, and information to encourage the user to breathe or meditate in a predetermined way;
If the user is in the second state, it is preferable that in the solution determination step, the active solution including a combination of the vibration, the sound, and information encouraging the user to breathe or meditate in a predetermined manner is determined as the solution to be provided to the user.

Specific breathing techniques and meditation can have the effect of calming emotions, relieving fatigue, and bringing a person to a state of alertness.

According to the present invention, when the user is in a second state in which an active solution encouraging a predetermined action should be provided, a solution is determined that includes information encouraging breathing or meditation in a predetermined manner, thereby effectively improving the user's state.

The inventors' research has revealed that there is a significant correlation between the answers to questions about the condition of a person's face, including "how open the eyes are," "how dark the bags around the eyes are," "the condition of acne or pimples," "how upturned or downturned the corners of the mouth are," and "how moisturized the lips are" compared to a normal condition, and the emotions or fatigue state reflected on the person's face.

Note that emotions include, for example, the four factors of happiness, burnout state, life satisfaction, negativity or positivity based on the PANAS scale, degree of resilience, subjective well-being scale, and stress tolerance.

Fatigue states include, for example, a state of insomnia, physical fatigue, mental fatigue, and the state of fatigue components in the body.

More specifically, the physical fatigue level and mental fatigue level in this embodiment include, for example, values equivalent to the total score of the Chalder Fatigue Questionnaire, as well as values equivalent to the physical evaluation, mental evaluation, and overall evaluation from the "Self-diagnosis Fatigue Checklist" created in the Ministry of Education, Culture, Sports, Science and Technology's research to meet consumer needs, "Research on the molecular and neural mechanisms of fatigue and fatigue sensation and prevention thereof."

Examples of insomnia status include the Athens Insomnia Scale.

Fatigue components in the body include, for example, the oxidation products 8OHdG (8-hydroxy-2'-deoxyguanosine) and isoprostane, and dROMs (Diacron Reactive Oxygen Metabolites), which indicate the degree of oxidative stress.

The four factors of happiness are the values of the first to fourth factors in a questionnaire on the four factors of happiness created by Maeno Takashi. Specifically, the first factor is the "Let's give it a try!" factor (a factor of self-realization and growth), which corresponds to the degree of satisfaction with self-realization or growth in this embodiment. The second factor is the "Thank you!" factor (a factor of connection and gratitude), which corresponds to good relationships with others or the degree of gratitude toward others or positivity in this embodiment. The third factor is the "It'll work out!" factor (a factor of forward-lookingness and optimism), which corresponds to the degree of optimism or self-affirmation in this embodiment. The fourth factor is the "Be yourself!" factor (a factor of independence and going at your own pace).

The state of burnout includes, for example, values equivalent to the scales of emotional exhaustion, depersonalization, and reduced sense of personal accomplishment measured by the Maslach Burnout Inventory (MBI).

Life satisfaction is the degree of satisfaction with life, and includes values equivalent to, for example, Ed Diener's Satisfaction With Life Scale (SWLS).

Negativity or positivity according to the PANAS scale is a rating scale included in the PANAS (Positive and Negative Affect Schedule) scale by Watson, Clark, Tellegen, et al., and is a simple mood rating scale consisting of, for example, eight positive affect items and eight negative affect items.

The degree of resilience is the ability to bounce back from psychological trauma, and includes, for example, the two-dimensional resilience factor scale proposed by Mari Hirano in "An attempt to classify the endowed and acquired factors of resilience."

The subjective well-being scale is a type of measure of psychological health, and includes, for example, the five domains (three items each) of "Positive attitude toward life," "Sense of accomplishment," "Self-confidence," "Sense of bliss," and "Sense of disappointment toward life" in the Subjective Well-Being Scale (SUBI) created by the WHO.

Stress tolerance refers to stress tolerance against psychosocial stress, and includes, for example, the level of stress tolerance measured using the "Stress Tolerance Checklist" created by Katsura Taisaku and others.

The configuration, action, and effects of the user status analysis method in this embodiment, which applies this knowledge, are as follows:

In the user condition analysis method of the present invention,
The questions about the user's facial condition preferably include a question about the degree to which the user's eyes are open compared to their normal state.

In the user condition analysis method of the present invention,
The questions about the user's facial condition preferably include a question about the density of dark circles around the user's eyes compared to the user's normal condition.

In the user condition analysis method of the present invention,
Preferably, the questions regarding the user's facial condition include questions regarding the user's acne or pimple condition compared to their normal condition.

In the user condition analysis method of the present invention,
The questions about the user's facial condition preferably include a question about the degree to which the corners of the mouth are raised or lowered compared to the user's normal state.

In the user condition analysis method of the present invention,
The questions regarding the user's facial condition preferably include a question regarding the moisturizing condition of the user's lips compared to their normal condition.

According to the present invention, specific facial condition information such as "eye opening degree," "dark circles around the eyes," "condition of acne or pimples," "degree of turning up or down of the corners of the mouth," and/or "moisturizing state of lips" compared to the user's normal condition is obtained, which is closely related to the user's emotions or fatigue state and can be easily judged by the user by looking at an image of his or her own face, so that the user's emotions or fatigue state can be analyzed accurately and easily.

In addition, users will know which parts of their face to look at in order to understand their own emotions or state of fatigue. This will encourage them to pay more attention to these parts of their face in their daily lives, and will more effectively encourage them to become aware of the emotions or state of fatigue that are reflected in their own face.

In the user condition analysis method of the present invention,
The step of capturing an image of the user's face is performed before or after the step of receiving the input,
The image analyzing step is preferably a step of analyzing an image of the user's face captured before or after the input receiving step.

If an image of a user's face is captured while the user is entering answers to questions about the condition of their face, the user may not be concentrating on capturing an image of their face, resulting in camera shake. For example, if the image is not captured using a device with a function to correct camera shake, the image capture position may shift, and as a result, it may be difficult to maintain the accuracy of the image analysis.

Therefore, it would be even better if an image of the user's face could be captured in a state where the user can concentrate on capturing an image of the face.

According to the present invention, the step of capturing an image of the user's face is executed before or after the input receiving step. Therefore, the facial image can be acquired while the user is concentrating on capturing an image of the face, which reduces the possibility of the capturing position being shifted and maintains the accuracy of the image analysis.

In the user condition analysis method of the present invention,
the step of capturing an image of the face of the user is executed during the execution of the input receiving step;
The image analysis step is preferably a step of analyzing an image of the face of the user captured during execution of the input reception step.

Alternatively, in the user condition analysis method of the present invention,
The step of capturing an image of the face of the user is performed before or during the step of receiving the input,
The image analysis step is preferably performed during execution of the input reception step.

It is expected that it will take a certain amount of time to input answers to questions about the condition of the face and to capture the facial image required for analysis. If such operations take a long time, the user may find them irritating, which may affect the user's emotions or state of fatigue. Or, the user may stop operating the device due to the irritation, and as a result, the user may not be able to become aware of his or her own emotions or state of fatigue.

According to the present invention, the step of capturing an image of the user's face or the image analysis step is executed during the execution of the input reception step, so the time required for operations, etc. is accordingly shortened, making it more efficient and reducing the possibility of causing annoyance to the user.

The user condition analysis system of the present invention comprises:
An imaging unit that captures an image of the user's face;
A display unit for displaying information;
An imaging control unit that causes the imaging unit to capture an image of the user's face;
a display control unit that displays on the display unit an image of the user's face captured by the imaging unit, and displays on the display unit a question about a facial condition that is correlated with an emotion or fatigue state appearing on the user's face displayed on the display unit;
an input unit arranged at a position where the user can perform an operation while viewing the image of the user's face displayed on the display unit and the question;
an input receiving unit that receives an input of an answer to the question by the user via the input unit while the image of the face is displayed on the display unit;
an answer analysis unit that analyzes the answer by the user to the question received by the input receiving unit;
The present invention is characterized by comprising a user state information generation unit that generates user state information indicating a user state, which is one or both of the user's emotions and fatigue state, based on the analysis results by the response analysis unit.

Alternatively, the user condition analysis system of the present invention comprises:
A mirror unit that reflects the face of the user;
a display unit that is disposed in a position where the user can see the face of the user reflected in the mirror unit at the same time and that displays information;
a display control unit that displays on the display unit a question about a facial condition that is correlated with an emotion or a fatigue state appearing on the face of the user reflected in the mirror unit;
an input unit arranged at a position where the user can perform an operation while looking at the face of the user reflected in the mirror unit and the question displayed on the display unit;
an input receiving unit that receives, via the input unit, an input of a response from the user to a question about the state of the user's face while showing an image of the user's face reflected in the mirror unit;
a response analysis unit that analyzes responses by the user to questions about the facial condition of the user that are received by the input receiving unit;
The present invention is characterized by comprising a user state information generation unit that generates user state information indicating a user state, which is one or both of the user's emotions and fatigue state, based on the analysis results by the response analysis unit.

According to the present invention, the user answers questions about the condition of his/her face while looking at his/her own face, making it easy for him/her to answer. In addition, since the user has the opportunity to calmly reflect on the condition of his/her own face, he/she is encouraged to become aware of the facial condition that is closely related to emotions or fatigue state, and the possibility of being able to answer questions about his/her own facial condition accurately is increased.

This allows for accurate generation of user state information that indicates the user state, which is either or both of the user's emotions and fatigue state.

1 is a block diagram showing an overall picture of a user condition analysis system according to the present invention; 4 is a flowchart showing the process of a user state analysis method according to the present invention. 11A and 11B are diagrams showing display contents of a user state analysis method according to the present invention. 11A and 11B are diagrams showing display contents of a user state analysis method according to the present invention. 4 is a flowchart showing the process of a user state analysis method according to the present invention. 4 is a flowchart showing the process of a user state analysis method according to the present invention. 4 is a flowchart showing the process of a user state analysis method according to the present invention. 4 is a flowchart showing the process of a user state analysis method according to the present invention. FIG. 2 is a diagram showing an example of a solution provided by the user status analysis method of the present invention. FIG. 2 is a diagram showing an example of a solution provided by the user status analysis method of the present invention. 11A and 11B are diagrams showing display contents of a user state analysis method according to the present invention.

<Configuration of user status analysis system>
First, the configuration of a user state analysis system used in the user state analysis method of the present embodiment will be described with reference to Fig. 1. Note that the same components will be given the same reference numerals and the description thereof will be omitted.

The user status analysis system of this embodiment is a computer network system including, for example, a mobile terminal 10 and a server 30.

The mobile terminal 10 and the server 30 are connected to each other so that they can communicate with each other via a communication network 90 such as a mobile phone network. Note that there may be multiple mobile terminals 10 and multiple servers 30.

The mobile terminal 10 includes, for example, an imaging unit 110, a display unit 130, a mobile terminal control unit 150, a vibration generating unit 170, an audio playback unit 190, and an input unit 210. In this embodiment, the mobile terminal 10 is a smartphone, but it may be a general-purpose terminal such as a tablet terminal, or it may be a dedicated terminal that can be carried by the user.

The imaging unit 110 is a camera that captures an image of the user's face. In this embodiment, since the mobile terminal 10 is a smartphone, both the in-camera and the out-camera correspond to the imaging unit 110.

The display unit 130 is a display that displays information to the user.

The mobile device control unit 150 is composed of an arithmetic processing unit such as a CPU (Central Processing Unit), a memory, and an I/O (Input/Output) device. By reading and executing a specific program, the mobile device control unit 150 functions, for example, as an imaging control unit 151, a display control unit 153, and an input reception unit 155, or further functions as an image analysis unit 157, a solution provision unit 159, and a mode designation reception unit 161.

The imaging control unit 151 causes the imaging unit 110 to capture an image of the face or skin of the fingers of the user U.

The display control unit 153 displays on the display unit 130 an image of the face of the user U captured by the imaging unit 110, and also displays on the display unit 130 a question about the state of the face of the user U displayed on the display unit 130. Note that the question is a question about the state of the face that has a correlation with the emotion or state of fatigue appearing on the face of the user U displayed on the display unit 130.

The input receiving unit 155 receives input of a user's response to a question about the condition of the user U's face via the input unit 210 while displaying an image of the face on the display unit 130.

The image analysis unit 157 analyzes the image of the face or skin of the fingers of the user U captured by the imaging unit 110.

The solution providing unit 159 provides the solution determined by the solution determining unit 335 to the user U.

The mode designation receiving unit 161 accepts the designation of the operating mode of the user state analysis system by the user U.

The vibration generating unit 170 is, for example, a vibrator that provides the user U with periodic vibrations that continue for a predetermined period of time and that are included in the solution, and whose period varies in stages from the start to the end of the vibration.

The audio playback unit 190 is, for example, a speaker that provides the user U with audio, including human voices, natural sounds, or music, that continues for a predetermined period of time and is included in the solution to improve the user's condition.

The input unit 210 is, for example, a touch panel, a keyboard, a mouse, or other pointing device that accepts input from a user.

If the mobile terminal 10 is a smartphone or tablet, the input unit 210 is usually a touch panel, so the user can input answers to questions while looking at the image of the face and questions about the face condition displayed on the display unit 130. However, even if the mobile terminal 10 is configured with other devices, the display unit 130 and the input unit 210 are positioned so that the user can input answers while looking at the image of the face and questions about the face condition.

The server 30 is, for example, a computer that includes a server storage unit 310 and a server control unit 330. There may be multiple servers 30 or there may be a single server 30.

The server storage unit 310 is composed of a storage device such as a ROM (Read Only Memory), a RAM (Random Access Memory), or a HDD (Hard Disk Drive). The server storage unit 310 functions as the storage unit in this embodiment.

The server storage unit 310 stores information on multiple solutions that can be provided to the user to improve the user's condition, for example, one or both of the user's emotions and fatigue state, and also stores the processing results of the user condition analysis system and information necessary for the processing of the user condition analysis system.

The server control unit 330 is composed of a processor such as a CPU, a memory, an I/O device, and the like. By loading and executing a specific program, the server control unit 330 functions, for example, as a response analysis unit 331 and a user state information generation unit 333, and further functions as a solution determination unit 335, a heart rate acquisition unit 337, and a state determination unit 339.

The answer analysis unit 331 analyzes answers given by user U to questions about the condition of user U's face that are received by the input receiving unit 155.

The user status information generation unit 333 generates user status information indicating the user status of the user U based on the analysis results by the response analysis unit 331 and the image analysis unit 157.

The solution determination unit 335 determines a solution from multiple solutions according to the user state information.

The heart rate acquisition unit 337 acquires information about the heart rate of the user U, for example, the heart rate of the user U, from the analysis results of the image of the face or skin of the fingers of the user U by the image analysis unit 157.

The state determination unit 339 determines, based on the user state information, whether the user U is in a first state in which a passive solution, which is a passive stimulation to the user U's sense of touch or hearing, should be provided, or in a second state in which an active solution that prompts the user U to perform a specified action should be provided.

<Processing Overview>
Next, a series of steps according to the user state analysis method of this embodiment will be described with reference to FIGS.

<Mode designation acceptance step>
When the series of processes is started, first, the mode designation receiving unit 161 executes a mode designation receiving step of receiving an operation mode designated by the user U (FIG. 2/S10).

The mode designation receiving unit 161, for example, displays a screen on the display unit 130 that prompts the user U to input an operation mode.

Figure 3 is an example of a screen that prompts the user U to input an operation mode. In Figure 3, an icon M1 for specifying "Midnight Blue" corresponding to the face mode, an icon M21 for specifying "Red" corresponding to the first finger mode, an icon M22 for specifying "Blue" corresponding to the second finger mode, and an icon M23 for specifying "Green" corresponding to the third finger mode are displayed.

When user U taps any of the icons, the mode designation receiving unit 161 accepts the designation of the corresponding mode and ends this step.

In this embodiment, the first finger mode is a mode for guiding the user U to an alert state, the second finger mode is a mode for guiding the user U to a relaxed state, and the third finger mode is a mode for guiding the user U to a calm state.

The screen also displays a "History" icon H1. When the user U taps on the icon, the display control unit 153 retrieves the user status information stored in the server storage unit 310 in response to the tap and displays it on the display unit 130.

<First imaging step>
When the face mode is designated as the operation mode (FIG. 2/S20: face mode), the imaging control unit 151 executes a first imaging step in which the imaging unit 110 captures an image of the user U's face.

At this time, the imaging control unit 151 starts capturing an image of the face of the user U via the in-camera, which is the imaging unit 110, for example, to be displayed on the display unit 130 in the face image/question display step. Alternatively, the imaging control unit 151 may capture a single still image of the face of the user U to be displayed on the display unit 130 in the face image/question display step.

On the other hand, if a mode corresponding to finger mode is specified as the operation mode (FIG. 2/S20: finger mode), the processes in FIG. 2/S30 to S70 are not executed, and the processes in FIG. 2/S90 and after are executed.

<Face image and question display step>
Next, the display control unit 153 executes a face image/question display step in which an image of the user U's face is shown and a question about the facial condition that is correlated with the emotion or fatigue state appearing on the user U's face is displayed (Figure 2/S50).

Figure 4 is an example of a screen displayed in the face image/question display step. For example, as shown in Figure 4, the display control unit 153 displays an image F of the captured face of the user U, a display Q1 of the first question about the face condition, "Are your eyes open compared to usual?", and an answer button A1 on the display unit 130.

For example, as shown in FIG. 4, the display control unit 153 displays a screen that prompts the user to answer the facial condition by tapping one of the five-level evaluations from "open" to "closed" in the answer button A1.

<Input reception step>
For example, when the display control unit 153 displays the first question as described above, the input receiving unit 155 initiates an input receiving step in which the input of a response from the user U to a question about the condition of the user U's face is received while an image of the face is shown to the user U (Figure 2/S70).

When user U taps any part of the answer button A1 that indicates the five-point scale, the input receiving unit 155 receives the corresponding evaluation value as an input of an answer to the question.

The processes in FIG. 2/S50 and S70 are repeated until an answer to the last question is entered (FIG. 2/S80: Yes).

That is, for example, the display control unit 153 displays, as questions 2 to 5 about the user's facial condition, a question about the darkness of dark circles around the eyes, a question about the condition of acne or pimples, a question about the degree to which the corners of the mouth are raised or lowered, and a question about the moisturization of the lips, in that order, in comparison with the user's usual condition, and the input receiving unit 155 receives input of answers by the user U to each of these questions.

Then, the input reception unit 155, for example, transmits the information of the answers given by the user U to each question received in this manner to the server 30, and ends the input reception step.

<Second imaging step>
Thereafter, the imaging control unit 151 executes a second imaging step of capturing an image of the face or skin of the fingers of the user U for analysis in the image analysis step.

At this time, if face mode is specified (FIG. 5/S91: face mode), the imaging control unit 151 captures an image of the face of the user U via the in-camera, which is the imaging unit 110 (FIG. 5/S93), and ends the second imaging step.

On the other hand, if finger mode is specified (Fig. 5/S91: finger mode), the imaging control unit 151 captures an image of the skin of the user U's fingers via the outer camera, which is the imaging unit 110 (Fig. 5/S95), and ends the second imaging step.

At this time, the display control unit 153, for example, displays a frame on the display unit 130 to guide the position of the user U's face or fingers to the correct position, and when the imaging control unit 151 recognizes that the user U's face or fingers are in the correct position, it captures a video of the user U's face or the skin of the fingers via the imaging unit 110 for a predetermined period of time (e.g., 30 seconds).

For example, if the face or fingers of user U deviate from the correct position during capture of the video, the imaging control unit 151 stops capturing the video, and the display control unit 153 displays a message on the display unit 130 urging the user U to return the position of the face or fingers to the correct position. Then, when the imaging control unit 151 recognizes that the face or fingers of user U have returned to the correct position, it resumes capturing the video of the skin of the face or fingers of user U. Alternatively, the images captured by the imaging control unit 151 may be multiple still images captured continuously.

<Image analysis step>
Thereafter, the image analysis unit 157 executes an image analysis step of analyzing the captured image of the face or skin of the fingers of the user U (FIG. 2/S110).

That is, if face mode is specified (Fig. 6/S111: face mode), the image analysis unit 157 analyzes the image of user U's face captured in the second imaging step (Fig. 6/S113), and if finger mode is specified (Fig. 6/S111: finger mode), the image analysis unit 157 analyzes the image of user U's fingers captured in the second imaging step (Fig. 6/S115).

More specifically, the image analysis unit 157 acquires information on the change in luminance of each of the RGB color components from, for example, a moving image or continuously captured still images that are images of skin.

Then, the image analysis unit 157 transmits, for example, information on the changes in luminance of each RGB color component based on the image of the user U's face obtained in this manner to the server 30, and ends the image analysis step.

<Response analysis step>
For example, when the answer analysis unit 331 receives information on user U's answers to each question and information on changes in brightness of each RGB color component from the mobile terminal 10, it executes an answer analysis step of analyzing user U's answers to questions about the condition of user U's face.

More specifically, for example, the answer analysis unit 331 determines an analysis result of one or both of the emotion and fatigue state shown on the face of the user U by using a first correlation information indicating a correlation between the answer by the user U to a question about the facial condition of the user U received in the input receiving step and the emotion or fatigue state shown on the face of the user U.

Various information may be used as the first correlation information indicating the correlation between the answer and the emotion or fatigue state shown on the face of the user U. In this embodiment, the answer analysis unit 331 uses, for example, the following formula as the first correlation information to index the signs of fatigue shown on the face of the user U as a fatigue sign score in the range from "energetic" (=10) to "tired" (=0), and determines the score as the analysis result of the answer.

F = p1 + p2 + p3 + p4 + p5
Here, F is a value (fatigue sign score) that indexes the emotion or fatigue state shown on the face, and its maximum value is 10 and its minimum value is 0. That is, for example, if the sum of p1 to p5 exceeds 10, it is set to 10, and if it is below 0, it is set to 0.

p1 is a value that evaluates the degree to which user U's eyes are open compared to their normal state, on a 5-point scale ranging from "open" (= 5 points) to "closed" (= 1 point) in 1-point increments.

p2 is a value that evaluates the darkness of the dark circles around the user U's eyes compared to their normal state on a five-point scale in 0.5 point increments ranging from "lighter" (= 1 point) to "darker" (= -1 point).

p3 is a value that evaluates the condition of user U's acne or pimples compared to their normal condition on a 5-point scale ranging from "better" (= 1 point) to "worse" (= -1 point) in 0.5 point increments.

p4 is a value that evaluates the degree to which the corners of the user U's mouth are raised or lowered compared to their normal state, on a 5-point scale ranging from "raised" (= 5 points) to "lowered" (= 1 point) in 1-point increments.

p5 is a value that evaluates the moisturization of user U's lips compared to their normal state on a 5-point scale ranging from "soft" (= 1 point) to "dry" (= -1 point) in 0.5 point increments.

<User status information generation step>
Then, a user status information generation step is executed to generate user status information indicating the user status including one or both of the emotions and fatigue state appearing on the user U's face based on the analysis results of the answers by the user U to the questions about the condition of the user U's face and the analysis results of the facial image (Figure 2/S150).

More specifically, in the user condition information generation step, the heart rate acquisition unit 337 first acquires the heart rate of the user U as information related to the user U's heart rate from the information on the changes in luminance of each RGB color component based on the image of the user U's face transmitted in the image analysis step (Fig. 7/S151).

Furthermore, the heart rate acquisition unit 337 acquires information about the fluctuations in the heart rate of the user U as information about the heart rate of the user U from information about changes in luminance of each RGB color component based on the image of the user U's face (FIG. 7/S153). This is because the fluctuations in the heart rate are closely related to the fatigue state of the user U.

The information relating to heart rate fluctuations is, for example, the total power value obtained by CVR-R or frequency analysis of heart rate fluctuations, which is the sum of the total power of LF (Low Frequency) which is the low frequency component of heart rate fluctuations, and the total power of HF (High Frequency) which is the high frequency component.

Then, the user condition information generating unit 333 obtains information regarding the autonomic nerve balance of the user U using the second correlation information indicating the correlation between the information regarding the heart rate, which is information regarding the user's heart rate, and the autonomic nerve balance of the user U (Figure 7/S153).

The information regarding autonomic nerve balance is, for example, the ratio of LF to HF (LF/HF) obtained by frequency analysis of heart rate fluctuations.

The user condition information generation unit 333 then determines an overall score for user U's user condition based on the analysis results of the answers to the questions about user U's facial condition, the analysis results of the facial image, and information about autonomic nerve balance (Figure 7/S155).

More specifically, the user state information generating unit 333 determines an autonomic nerve balance score by converting information related to autonomic nerve balance into an index using a score between 1 and 10. For example, the more the sympathetic nerve is dominant, the closer the score is to 1, and the more the parasympathetic nerve is dominant, the closer the score is to 10.

The user state information generating unit 333 also determines a fatigue score based on the heart rate, for example by indexing the CVR-R (or total power value) acquired as information related to heart rate fluctuation with a score between 1 and 10. For example, the score is determined so that the higher the degree of heart rate fluctuation, the closer the score is to 1, and the lower the degree of heart rate fluctuation, the closer the score is to 10.

The user state information generating unit 333 then determines the average of these three values, the autonomic nerve balance score, the fatigue score based on heart rate, and the fatigue sign score that quantifies the signs of fatigue displayed on the face of user U determined in the response analysis step, as a total score that indexes the user's emotions or fatigue state.

Then, the user status information generation unit 333 generates user status information (Figure 7/S157) and ends the user status information generation step.

More specifically, the user condition information generating unit 333 generates the user condition information as a set of a numerical value representing the signs of fatigue shown on the user U's face, a score representing the information relating to heart rate fluctuations, a score representing the information relating to autonomic nerve balance, and a total score.

<User state information storage step>
In the user status information storage step, the user status information generation unit 333 stores the user status information generated in the user status information generation step in the server storage unit 310 in chronological order (FIG. 2/S170).

<Solution decision step>
Then, the solution determination unit 335 executes a solution determination step to determine a solution according to the user condition information from a plurality of solutions that can be provided to the user U to improve the user condition, which is one or both of the user U's emotions and fatigue state (Figure 2/S190).

For example, when the solution determination unit 335 starts the solution determination step, it first checks the specified operation mode (Figure 8/S191).

If the specified operation mode is face mode (FIG. 8/S191: face mode), the solution determination unit 335 determines, based on the emotion or fatigue state shown on the face of the user U included in the user state information, whether the user U is in a first state in which a passive solution that is a passive stimulation to the user U's sense of touch or hearing should be provided, or in a second state in which an active solution that encourages the user U to perform a specified operation should be provided (FIG. 8/S193).

Various methods may be employed to determine whether the user is in the first state or the second state, but in this embodiment, the solution determination unit 335 determines that user U is in the second state if the score value of the information related to autonomic nerve balance included in the user state information is 4 or less and the numerical value of the signs of fatigue on user U's face is 6 or more (i.e., the sympathetic nerves are dominant and the user is in good health based on the signs of fatigue on his or her face), and otherwise determines that user U is in the first state.

If it is determined that user U is in the first state (FIG. 8/S193: first state), the solution determination unit 335 determines a solution according to the user state information from the passive solutions (FIG. 8/S195).

On the other hand, if it is determined that user U is in the second state (FIG. 8/S193: second state), the solution determination unit 335 determines a solution from the active solutions that corresponds to the user state information (FIG. 8/S197).

Also, if the specified operation mode is finger mode (FIG. 8/S191: finger mode), the solution determination unit 335 determines a solution according to the specified finger mode (FIG. 8/S199).

Then, the solution determination unit 335 transmits information about the solution determined in this manner and the user state information generated in the user state information generation step to the mobile terminal 10, and ends the solution determination step.

More specifically, for example, a correspondence table between the operation mode and total score and the solution is stored in advance in the server storage unit 310, and the solution determination unit 335 refers to the correspondence table to determine a solution that corresponds to the total score included in the user state information.

In this embodiment, the solution determined by the solution determination unit 335 is a combination of a periodic vibration that continues for a predetermined time, the period of which varies stepwise from the start to the end of the vibration, and a sound that continues for a predetermined time, including a human voice, a natural sound, or music, for improving the user's condition (passive solution), and if necessary, further includes information that prompts the user U to perform an action such as breathing or meditation in a predetermined manner, or an image that guides the user U to a predetermined state (active solution).

<Specific examples of solutions>
- Face mode solutions Fig. 9 shows the correspondence between the overall score and face mode solutions in this embodiment. As shown in Fig. 9, for sounds included in face mode solutions, solutions when the user U is in the first state are classified into three groups according to the high and low overall scores, with a combination of human voice and music for an overall score of 1 to 4, a combination of natural sounds and music for an overall score of 5 to 7, and music only for an overall score of 8 to 10. In this way, the solution determination unit 335 determines the sound to be combined with vibration based on the user state information.

In addition, for vibration, the solution determination unit 335 determines the period at the start of vibration and the pattern of change in that period from the start to the end based on the user state information, more specifically, based on the heart rate of the user U.

That is, for example, if the heart rate of user U is 80 or higher, the solution determination unit 335 determines that the vibration start period is 85 beats/minute and that this period will continue for 10 seconds. The solution determination unit 335 also determines the variation pattern of the period from the start to the end as follows: vibration at a period of 75 beats/minute continues for 10 seconds, vibration at a period of 65 beats/minute continues for 20 seconds, ..., ultimately resulting in vibration at a period of 50 beats/minute continuing for 120 seconds or more until the solution ends.

For example, if the user U's heart rate is relatively high, in this embodiment, if the heart rate is 70 or higher, a solution may be determined in which only vibration is provided for a certain period of time (e.g., 20 seconds) from the start of vibration, and then sound is provided in combination.

This is because, if the heart rate is relatively high, it is easier to improve emotions and fatigue by first allowing the heart rate to calm down before listening to the audio.

In this embodiment, if the user U is determined to be in the first state, the solution determination unit 335 determines three passive solutions corresponding to the overall score, and if the user U is determined to be in the second state, it determines one active solution and two passive solutions corresponding to the overall score. Therefore, multiple audio solutions corresponding to overall scores of 1 to 4, 5 to 7, and 8 to 10 are stored in the server storage unit 310.

The solution when user U is in the second state is a combination of music, "text that supports actions" that is information encouraging user U to take a specific action, and vibration.

"Text supporting actions" is a combination of sentences that encourage a specific action or a specific way of breathing or meditation, such as "Try simple mindfulness meditation while feeling the vibrations," "Straighten your back and correct your posture," and "Relax your shoulders and breathe slowly through your nose," which change at regular intervals, as shown in the lower part of Figure 10.

10 shows the correspondence between the heart rate of the user U and the finger mode solution for each finger mode in this embodiment. As shown in Fig. 10, the finger mode solution is a combination of vibration, information prompting the user U to perform a predetermined action, and an image guiding the user U to a predetermined state, which are determined according to a specified finger mode (either Red, Blue, or Green) and the heart rate of the user U.

More specifically, when "Red" corresponding to the first finger mode is specified, the solution determination unit 335 determines a variation pattern of the vibration period in such a way that the vibration period becomes gradually shorter from the start to the end of the vibration, i.e., the vibration frequency increases. This is to increase the heart rate of the user U by the attraction effect, and induce the user U into an awakened state.

In addition, the information encouraging user U to perform a specified action may be, for example, a comment such as "straighten your posture and take a deep breath while feeling the vibration," and the image guiding user U to a specified state may be, for example, a still image that evokes the image of awakening.

When "Blue" corresponding to the second finger mode is specified, the solution determination unit 335 determines a variation pattern of the period so that the vibration period becomes gradually longer from the start to the end of the vibration, i.e., the vibration frequency decreases. Note that the rate at which the vibration frequency decreases is faster than when "Green" is specified. This is to rapidly calm the heart rate of the user U by using the attraction effect, and induce the user U to a calm state.

In addition, the information encouraging user U to take a specific action may be, for example, a comment such as "Raise your shoulders once, relax, and breathe slowly while feeling the vibration," and the image guiding user U to the specific state may be, for example, a still image that evokes the image of calm.

When "Green", which corresponds to the third finger mode, is specified, the solution determination unit 335 determines a variation pattern of the period so that the vibration period becomes gradually longer from the start to the end of the vibration, i.e., the vibration frequency decreases. Note that the rate at which the vibration frequency decreases is slower than when "Blue" is specified. This is to gradually calm the heart rate of the user U by using the attraction effect, and induce the user U into a relaxed state.

In addition, the information encouraging user U to perform a specific action may be, for example, a comment such as "Relax your shoulders and take a deep breath while feeling the vibration," and the image guiding user U to a specific state may be, for example, a still image that evokes the image of relaxation.

In addition, when, for example, a Blue or Green finger mode is specified, the solution determination unit 335 may determine a variation pattern in which the vibration frequency is finally increased to a predetermined degree before ending the vibration.

For example, if user U is provided with the solution during a lunch break at work, the solution will encourage user U to calm down or relax and then bring their heart rate closer to normal again, guiding them to a state where they can smoothly resume work after the break.

<User status information display step>
Thereafter, for example, when the mobile terminal 10 receives information regarding the determined solution from the server 30 and the user status information generated in the user status information generation step, a user status information display step is executed to display the user status information (FIG. 2/S210).

Figure 11 is an example of the content displayed in the user condition information display step when the face mode is specified. That is, as shown in Figure 11, the display control unit 153 displays, as user condition information, a display R1 of "Signs of fatigue on the face" showing a numerical value of the signs of fatigue on the face of the user U included in the received user condition information, a display R2 of "Energy reserve" showing a scored value of information regarding heart rate fluctuations, a display R3 of "Autonomic nerve balance" showing a scored value of information regarding autonomic nerve balance, and a display T1 of the total score value.

Then, the display control unit 153 displays three icons (S1, S2, S3) for starting provision of the currently determined solution side by side at the bottom of the screen. Each icon also includes, for example, a title indicating the content of each solution and a thumbnail image.

When the finger mode is specified, the display control unit 153 causes the display unit 130 to display, for example, "autonomic balance," which shows a scored value of information related to autonomic balance, and a display showing the heart rate of the user U.

<Solution provision steps>
When face mode is specified, in the user status information display step, when the user U taps on any of the icons (S1, S2, S3) displayed on the screen by the display control unit 153 for starting the provision of the currently determined solution, the solution providing unit 159 provides the user with the solution corresponding to the tapped icon in response to the operation.

In addition, when the finger mode is specified, the solution providing unit 159 automatically starts providing the solution determined by the solution determining unit 335 after the display control unit 153 causes the display unit 130 to display "autonomic nerve balance" and a display showing the user U's heart rate for a certain period of time.

That is, for example, the solution providing unit 159 starts the vibration included in the solution via the vibration generating unit 170 at the start period determined by the solution determining unit 335, and provides the vibration continuously for a predetermined time while varying it according to the variation pattern determined by the solution determining unit 335.

The solution providing unit 159 also plays the audio determined by the solution determining unit 335 via the audio playing unit 190, and displays, via the display unit 130, information that prompts the user U to perform a specific action or an image that guides the user U to a specific state, determined by the solution determining unit 335, and provides it to the user U.

The above describes an embodiment of the present invention, but the present invention is not limited to this. Various modifications are possible without departing from the spirit of the present invention.

That is, for example, the above describes an embodiment in which the user's face is displayed as an image on the display unit 130, but the present invention is not limited to this. That is, for example, the user status analysis method of the present invention may be executed using a user status analysis system that is a so-called smart mirror, and the user may answer questions while looking at their own face reflected in the mirror. In addition, the smart mirror may or may not be portable by the user.

More specifically, the user condition analysis system of this embodiment includes a mirror unit that reflects the face of user U. The display unit 130 is positioned so that user U can simultaneously see the face of user U reflected in the mirror unit. The display control unit 153 displays questions about the condition of user U's face reflected in the mirror unit on the display unit 130. The input unit 210 is positioned so that user U can perform operations while viewing the face of user U reflected in the mirror unit and the questions about the condition of user U's face displayed on the display unit 130. The input receiving unit 155 receives input of answers from user U to questions about the condition of user U's face via the input unit 210 while showing an image of user U's face reflected in the mirror unit.

Alternatively, for example, in the answer analysis step, the answer analysis unit 331 may use an analysis model that takes as input the answer by user U to a question about the condition of the user U's face, and outputs emotions such as the four factors of happiness, burnout state, life satisfaction, negativity or positivity according to the PANAS scale, degree of resilience, subjective well-being scale, and stress tolerance shown on the user U's face, or fatigue state such as insomnia state, physical fatigue level, mental fatigue level, and amount of fatigue components in the body, as the first correlation information, to determine the analysis results of one or both of the emotions and fatigue state shown on the user U's face.

In this case, the response analysis unit 331 uses the analysis model to determine as the analysis result an output value indicating whether the emotion or fatigue state of the subject of analysis is higher or lower than a reference value indicating a normal state.

In the above embodiment, a configuration was described in which, after the input receiving step, an image of the face of the user U is captured (second imaging step) for analysis in the image analysis step, and the image analysis step is executed, but this is not limited to the above.

That is, for example, the step of capturing an image of user U's face for analysis in the image analysis step (second imaging step) may be performed before the input receiving step, and the image analysis step may be configured to analyze the image of user U's face captured before the input receiving step. In this way, the step of capturing an image of user U's face for analysis in the image analysis step (second imaging step) may be performed either before or after the input receiving step, and accordingly the image analysis step is a step of analyzing the image of user U's face captured either before or after the input receiving step.

Alternatively, for example, the step of capturing an image of the face of user U may be performed during the execution of the input reception step, and the image analysis step may be configured to analyze the image of the face of user U captured during the execution of the input reception step.

In this case, the step of capturing an image of the user U's face to be displayed on the display unit 130 in the face image/question display step (first imaging step) and the step of capturing an image of the user U's face to be analyzed in the image analysis step (second imaging step) are executed as a single step. That is, the image of the user's face captured in this manner is used both to be displayed on the display unit 130 in the face image/question display step and to be analyzed in the image analysis step.

In this case, for example, the image analysis unit 157 is configured to select an image that satisfies the conditions necessary to maintain the accuracy of image analysis, such as an image with a sufficiently low degree of camera shake, from among the images captured by the imaging unit 110, and analyze the selected image.

Alternatively, for example, the step of capturing an image of the face of user U may be performed before or during the input receiving step, and the image analysis step may be configured to be performed during the input receiving step.

110...imaging unit, 130...display unit, 151...imaging control unit, 153...display control unit, 155...input reception unit, 157...image analysis unit, 159...solution provision unit, 161...mode designation reception unit, 170...vibration generation unit, 190...audio playback unit, 310...server storage unit, 331...answer analysis unit, 333...user status information generation unit, 335...solution determination unit, 337...heart rate acquisition unit, 339...status determination unit.

Claims (20)

presenting an image of the user's face to the user and displaying a question about a facial state that correlates with an emotional or fatigue state appearing on the user's face;
an input receiving step of receiving a response from the user to a question about the state of the user's face while showing an image of the face to the user;
a response analysis step of analyzing responses by the user to questions about the user's facial condition;
and a user status information generation step of generating user status information indicating the user status including one or both of the emotions and fatigue state appearing on the user's face based on the analysis results of the answers. The user status analysis method according to claim 1,
a solution determination step of determining a solution corresponding to the user state information from a plurality of solutions that can be provided to the user for improving the user state of the user;
providing the determined solution to the user. The user status analysis method according to claim 1 or 2,
capturing an image of the user's face;
an image analysis step of analyzing the captured image of the user's face;
Further comprising:
A user status analysis method, characterized in that the user status information generating step is a step of generating the user status information based on an analysis result of the answer and the face image. The user status analysis method according to any one of claims 1 to 3,
A user status analysis method comprising the step of displaying the generated user status information. The user status analysis method according to any one of claims 1 to 4,
storing the generated user state information in chronological order;
A user status analysis method comprising the step of displaying the stored user status information in response to an operation by the user. The user status analysis method according to any one of claims 1 to 5,
The answer analysis step includes a step of determining an analysis result of one or both of the emotions and fatigue state shown on the user's face using an answer by the user to a received question about the user's facial condition and first correlation information indicating a correlation between the answer and the emotions or fatigue state shown on the user's face. The user status analysis method according to claim 2,
capturing an image of the user's face;
an image analysis step of analyzing the captured image of the user's face;
Further comprising:
The answer analyzing step includes a step of determining a fatigue symptom score by analyzing the answer and indexing an emotion or fatigue state shown on the face of the user,
The user state information generating step includes:
obtaining information about the user's heart rate from an analysis of the facial image;
determining a heart rate-based fatigue score that indexes a heart rate-based fatigue state based on information about the user's heart rate;
determining a total score that indexes the user's emotion or fatigue state based on the fatigue symptom score and the heart rate-based fatigue score;
generating the user status information including the overall score;
The method for analyzing a user status, wherein the solution determination step includes a step of determining a solution to be provided to the user based on the overall score. The user status analysis method according to claim 7,
The step of determining the overall score comprises:
acquiring information about the user's autonomic nerve balance using information about the user's heart rate and second correlation information indicating a correlation between the information about the heart rate and the user's autonomic nerve balance;
A user condition analysis method comprising: a step of determining the overall score based on information including information regarding the autonomic nerve balance. The user status analysis method according to claim 2,
The solution determination step includes:
A step of determining whether the user is in a first state in which a passive solution, which is a passive stimulation to the user's sense of touch or hearing, should be provided, or in a second state in which an active solution, which encourages the user to perform a predetermined action, should be provided, based on the emotion or fatigue state shown on the user's face included in the user state information;
determining a solution corresponding to the user state information from the plurality of solutions based on a result of the determination. The user status analysis method according to claim 9,
the active solution includes a combination of periodic vibrations lasting a predetermined time, audio to improve the user's condition, and information to encourage the user to breathe or meditate in a predetermined way;
A user state analysis method characterized in that, when the user is in the second state, in the solution determination step, the active solution including a combination of the vibration, the sound, and information encouraging the user to breathe or meditate in a predetermined manner is determined as the solution to be provided to the user. The user status analysis method according to any one of claims 1 to 10,
A user condition analysis method, characterized in that the questions about the user's facial condition include questions about the degree to which the user's eyes are open compared to the user's normal condition. The user status analysis method according to any one of claims 1 to 11,
A method for analyzing a user's condition, wherein the questions about the user's facial condition include a question about the density of dark circles around the eyes compared to the user's normal condition. The user status analysis method according to any one of claims 1 to 12,
A user condition analysis method, characterized in that the questions about the user's facial condition include questions about the condition of acne or pimples compared to the user's normal condition. The user status analysis method according to any one of claims 1 to 13,
A user condition analysis method, characterized in that the questions about the user's facial condition include questions about the degree to which the corners of the mouth are raised or lowered compared to the user's normal condition. The user condition analysis method according to any one of claims 1 to 14,
A user condition analysis method, wherein the questions about the user's facial condition include a question about the moisturizing state of the user's lips compared to the user's usual condition. The user status analysis method according to claim 3,
The step of capturing an image of the user's face is performed before or after the step of receiving the input,
A user state analysis method, characterized in that the image analysis step is a step of analyzing an image of the user's face captured before or after the input reception step. The user status analysis method according to claim 3,
the step of capturing an image of the face of the user is executed during the execution of the input receiving step;
A user state analysis method, characterized in that the image analysis step is a step of analyzing an image of the user's face captured during execution of the input reception step. The user status analysis method according to claim 3,
The step of capturing an image of the face of the user is performed before or during the input receiving step;
The user state analysis method, wherein the image analysis step is executed during execution of the input reception step. An imaging unit that captures an image of a user's face;
A display unit for displaying information;
An imaging control unit that causes the imaging unit to capture an image of the user's face;
a display control unit that displays on the display unit an image of the user's face captured by the imaging unit, and displays on the display unit a question about a facial condition that is correlated with an emotion or fatigue state appearing on the user's face displayed on the display unit;
an input unit arranged at a position where the user can perform an operation while viewing the image of the user's face displayed on the display unit and the question;
an input receiving unit that receives an input of an answer to the question by the user via the input unit while the image of the face is displayed on the display unit;
an answer analysis unit that analyzes the answer by the user to the question received by the input receiving unit;
A user status analysis system comprising: a user status information generation unit that generates user status information indicating a user status, which is one or both of the user's emotions and fatigue state, based on the analysis results by the response analysis unit. A mirror that reflects the user's face,
a display unit that is disposed in a position where the user can see the face of the user reflected in the mirror unit at the same time and that displays information;
a display control unit that displays on the display unit a question about a facial condition that is correlated with an emotion or a fatigue state appearing on the face of the user reflected in the mirror unit;
an input unit arranged at a position where the user can perform an operation while looking at the face of the user reflected in the mirror unit and the question displayed on the display unit;
an input receiving unit that receives, via the input unit, an input of a response from the user to a question about the state of the user's face while showing an image of the user's face reflected in the mirror unit;
a response analysis unit that analyzes responses by the user to questions about the facial condition of the user that are received by the input receiving unit;
A user status analysis system comprising: a user status information generation unit that generates user status information indicating a user status, which is one or both of the user's emotions and fatigue state, based on the analysis results by the response analysis unit.

Images