WO2022118398A1

WO2022118398A1 - Mobile information terminal and facial expression monitoring method

Info

Publication number: WO2022118398A1
Application number: PCT/JP2020/044867
Authority: WO
Inventors: 眞弓中出; 義憲岡田; 仁秋山
Original assignee: マクセル株式会社
Priority date: 2020-12-02
Filing date: 2020-12-02
Publication date: 2022-06-09

Abstract

This mobile information terminal stores a target image in which the facial expression of a target user is captured, and analyzes a target feature amount indicating the characteristics of the target facial expression of the user captured in the target image. When information is displayed on the screen of a display of a mobile information terminal, an object image obtained by imaging the face of the user who visually recognizes the screen is read, an object feature amount indicating the characteristics of the user's facial expression captured in the object image is analyzed, and a deviation amount between the object feature amount and the target feature amount is calculated, and when it is determined that the deviation amount is equal to or higher than a predetermined facial expression determination threshold value for determining that the user's facial expression has deteriorated, notification information based on the result of determination is output.

Description

Mobile information terminal and facial expression monitoring method

The present invention relates to a mobile information terminal and a facial expression monitoring method, and more particularly to a mobile information terminal and a facial expression monitoring method having a function of photographing a user or the like and a function of notifying the user.

Patent Document 1 describes "a setting means for setting for each person whether or not to record the image data when the person in the image data has a predetermined expression, an imaging means for obtaining the image data by imaging, and an image. A specific means for identifying a person in data, a facial expression detecting means for detecting a person's facial expression in image data, and a specific means for detecting a person in image data, and when the predetermined facial expression is detected, the specific means is specified. A digital camera provided with a control means for determining whether or not to record image data based on the setting contents of the setting means for a person is described. That is, the facial expression is analyzed at the time of shooting with the camera, and a smile or one's favorite face is recorded.

Further, in Patent Document 2, "a display step for displaying a plurality of facial expressions with different facial expressions, a step for designating a score for the facial expression of the facial image displayed in the display step, and a score specified above are used. A method of registering a facial expression database including a step of associating a relationship with a numerical value indicating the facial expression of the facial image and storing it in the facial expression database is described. That is, it is shown that a face image having a good facial expression can be easily selected and extracted from a plurality of images.

Japanese Unexamined Patent Publication No. 2010-178259 Japanese Unexamined Patent Publication No. 2013-169003

When a user is looking at a mobile information terminal such as a smartphone, he or she often tends to look down or wrinkles between the eyebrows to see characters that are difficult to see, and tends to have a scary face. There is a problem that many users use a mobile information terminal for a long time and keep an unfavorable facial expression for a long time, so that a bad facial expression is fixed.

Although Patent Documents 1 and 2 describe that a good face and a good facial expression can be easily selected and extracted and recorded, they are unconsciously used when a mobile information terminal such as a smartphone is used for a long time. It is not suggested without any consideration for having a bad expression.

The present invention has been made in view of the above problems, and an object of the present invention is to provide a mobile information terminal and a facial expression monitoring method that contribute to improving the facial expression of a user who is using the mobile information terminal.

In order to achieve the above object, the present invention includes the configuration described in the claims. To give an example, the present invention is a portable information terminal, which is connected to a camera, a display, a memory, a notification member for outputting notification information, and the camera, the display, and the memory, respectively. The memory includes a target image in which the target user's facial expression is captured, and the processor shows the characteristics of the target user's facial expression captured in the target image. The target feature amount is analyzed, and when the information is displayed on the screen of the display, the camera reads the target image captured by the user's face, and the feature of the facial expression of the user captured by the target image is captured. It is said that the indicated target feature amount is analyzed, the target feature amount and the deviation amount of the target feature amount are calculated, and the deviation amount is equal to or higher than a predetermined facial expression determination threshold for determining that the user's facial expression has deteriorated. When the determination is made, the notification member is made to output the notification information based on the result of the determination.

According to the present invention, it is possible to provide a mobile information terminal and a facial expression monitoring method that contribute to improving the facial expression of a user who is using the mobile information terminal. The purposes, configurations, and effects other than those described above will be clarified by the following description of the embodiments.

It is a figure which shows typically the processing outline of the smartphone which concerns on 1st Embodiment. It is a figure explaining the operation of the smartphone which concerns on 1st Embodiment. It is a functional block diagram of the configuration example of the smartphone which concerns on 1st Embodiment. It is a flowchart explaining the basic operation of the smartphone which concerns on 1st Embodiment. It is a figure which shows typically the process outline of 2nd Embodiment. It is a figure which shows typically the process outline of 2nd Embodiment. It is a flowchart explaining the operation of the smartphone which concerns on 3rd Embodiment. It is a flowchart explaining the operation of the smartphone which concerns on 4th Embodiment. It is a screen display example in 4th Embodiment. It is a figure which supplementarily explains the icon example shown in FIG. It is a figure which shows the type of an icon. It is a figure which shows typically the usage environment of the smartphone which concerns on 5th Embodiment. It is a figure which shows the appearance of the HMD which concerns on 6th Embodiment. It is a figure which shows the target image which concerns on 6th Embodiment. It is a figure which shows the target image which concerns on 6th Embodiment.

Hereinafter, an example of the embodiment of the present invention will be described with reference to the drawings. A smartphone will be described as a specific example of a mobile information terminal. FIG. 1 is a diagram schematically showing a processing outline of the smartphone 100 according to the present embodiment.

<First Embodiment>
In FIG. 1, the smartphone 100 includes an in-camera 101, a display 102, a speaker 103, a microphone 104, a vibrator 105, and a touch panel 106. The smartphone 100 captures the face of the user 10 looking at the screen of the display 102 by the in-camera 101 and captures it as a face image in the smartphone 100.

As shown in the “registration process” of FIG. 1, the user selects a target image 111 consisting of a “good facial expression” face image from the user's own face image displayed on the display 102 and incorporates it into the smartphone 100. sign up.

When the user 10 visually recognizes the screen of the smartphone 100, as shown in the “expression monitoring process” of FIG. 1, the smartphone 100 photographs the face of the user 10 who is gazing at the display 102 with the in-camera 101. The target image 112, which is a face image while the display 102 is being visually recognized and is the target of the facial expression monitoring process, is captured in the smartphone 100.

Then, from the captured target image 111 and the target image 112, the facial expression feature amount (hereinafter referred to as "feature amount") that quantitatively represents each feature forming the facial expression is analyzed and calculated. Further, for example, a feature is obtained in which the evaluation value of the facial expression can be judged to be a better facial expression than the registered image. If it is judged that there is no wrinkle if it is in a wrinkled state or a wrinkled state of the face, as a better facial expression, the amount of deviation between the characteristic amount of "good facial expression" and the characteristic amount of "facial expression at the time of shooting" is Even if it is equal to or higher than the facial expression determination threshold, a warning may not be issued or a display indicating that the direction is shifted may be displayed.

For example, in the case of eye condition, the smartphone 100 quantitatively analyzes and calculates from "good facial expressions" of wide open eyes to "bad facial expressions" such as bind-off eyes and slanted eyes as feature quantities.

In the case of the state of the mouth, it is quantitatively analyzed and calculated as a feature amount from the "good facial expression" that is obediently closed to the "bad facial expression" such as the character mouth and the wide open mouth.

In the case of wrinkles between the eyebrows and wrinkles on the face, quantitatively analyze and calculate from "good facial expressions" without wrinkles to "bad facial expressions" with many wrinkles.

The smartphone 100 has a feature amount of the target image 111 (a feature amount of a "good expression", hereinafter referred to as a "target feature amount") and a feature amount of the target image 112 (a feature amount of a "expression at the time of shooting", which is the following. When the target feature amount deviates from the target feature amount by a predetermined facial expression determination threshold value or more by comparing (referred to as "target feature amount"), the display 102 shows "the facial expression deteriorates" as shown in the "notification process" of FIG. The text information 113 of the warning such as "is" and the icon 114 indicating the face with deteriorated facial expression are displayed. In addition, the smartphone 100 emits a warning voice 115 such as "the facial expression is getting worse" from the speaker 103. Further, the smartphone 100 generates vibration by the vibrator 105 to convey a warning to the user 10. Therefore, the display 102, the speaker 103, and the vibrator 105 correspond to the notification member. The facial expression determination threshold value is set to the amount of deviation of the feature amount that should be determined to have changed from "good facial expression" to "bad facial expression".

The smartphone 100 displays the character information 116 such as "laughing" instructing the display 102 to improve the facial expression and the icon 117 indicating the face for improving the facial expression together with or instead of the warning. In addition, the smartphone 100 is a voice instructing facial expression improvement such as "laughing", "big eyes", "raising the corner of the mouth", "relaxing the power between the eyebrows", and "tightening the face" from the speaker 103. It emits 118, 119, 120, 121, 122.

As a result, when the target feature amount deviates from the target feature amount by the facial expression judgment threshold or more, the user can be notified of the warning information of the facial expression deterioration and the instruction information instructing the improvement of the facial expression, and the user who is not aware of the deterioration of the facial expression. It is possible to inform the person of the deterioration of facial expression and instruct the improvement of facial expression.

Further, the smartphone 100 may notify the user of warning information of facial expression deterioration or instruction information for instructing improvement of facial expression when the state in which the target feature amount deviates from the target feature amount by the facial expression determination threshold or more exceeds the notification determination time. .. That is, the notification information is output only when both the facial expression determination threshold value and the notification determination time are satisfied. The notification determination time may be set to a time during which it is determined that the facial expression deterioration state has continued to some extent without being notified only when the temporary or momentary deviation between the two feature quantities exceeds the facial expression determination threshold value.

As a result, when the target feature amount does not deviate from the target feature amount by the facial expression judgment threshold or more, in other words, when the user is visually recognizing the screen in a state where the user has not reached a “good facial expression” or a “bad facial expression” yet. Even if there is a temporary or momentary shift in the amount of features that exceeds the facial expression judgment threshold due to the occurrence of a bind-off eye, a slanted eye, a wide open mouth, a wrinkle between the eyebrows or the face, etc. It is possible to prevent the facial expression deterioration warning and the facial expression improvement instruction from being improperly mistaken. That is, it is possible to accurately generate a facial expression deterioration warning and a facial expression improvement instruction after the state of "bad facial expression" exceeding the threshold value continues to some extent and the facial expression deterioration warning and the facial expression improvement instruction are originally required.

Next, the operation of the smartphone 100 according to the present embodiment will be described in more detail with reference to FIG. FIG. 2 is a diagram illustrating the operation of the smartphone 100 according to the present embodiment.

Graphs 1 to 4 in FIG. 2 show a deviation amount Δfa (that is, a quantitatively large “good facial expression” target for the target feature amount of the “facial expression at the time of shooting” with respect to the target feature amount of the “good facial expression” in each feature. The vertical axis is the difference between the feature amount and the target feature amount of the "facial expression at the time of shooting"), and the horizontal axis is from "good facial expression" such as face 201 to "bad facial expression" such as

faces

202 and 203. It shows the "expression at the time of shooting" in.

Graph 1 shows the characteristic line 211 when the characteristic example forming the facial expression is the state of the eyes. In this case, the feature amount is the size of the eye or the "good facial expression" of the wide open eye, for example, when the eyes are turned down, the line of sight is directed downward and the pupil is covered with the half eyelid, resulting in a "bad facial expression". In addition, when the outer corners of the eyes are raised, the impression of a scary face becomes stronger and the face becomes "bad". Along the characteristic line 211, the "facial expression at the time of shooting" approaches the "bad facial expression" from the "good facial expression" due to the bind-off eyes and slanted eyes, and the target feature amount deviates from the target feature amount by the facial expression determination threshold of 212 or more. When the state 213 is reached, the measurement of the elapsed time is started. When the elapsed time exceeds the notification determination time, the user is notified of the warning information of the deterioration of the facial expression and the instruction information instructing the improvement of the facial expression such as "enlarge the eyes".

Graph 2 shows the characteristic line 221 when the characteristic example forming the facial expression is the state of the mouth. In contrast to a "good facial expression" that is obediently closed, for example, when the mouth is tied in a syllabary, the corners of the mouth are lowered, giving the impression of being nervous, dissatisfied, and unhappy, resulting in a "bad facial expression." Also, if the mouth is wide open, it will look sloppy and give a "bad expression". Along the characteristic line 221 the "facial expression at the time of shooting" approaches the "bad facial expression" due to the character mouth from "good facial expression" or the wide open mouth, and the target feature amount is the facial expression judgment threshold 222 or more than the target feature amount. When the facial expression state 223 of the displaced mouth is reached, the measurement of the elapsed time is started. When the elapsed time exceeds the notification judgment time, the user is notified of warning information of facial expression deterioration and instruction information instructing the improvement of facial expression such as "raise the corner of the mouth".

Graph 3 shows the characteristic line 231 when the characteristic example forming the facial expression is the state of wrinkles between the eyebrows. A "good facial expression" with no wrinkles between the eyebrows, but a lot of wrinkles between the eyebrows gives the impression of an angry, scary person or a troubled person, resulting in a "bad facial expression". .. Along the characteristic line 231 the "facial expression at the time of shooting" approaches "bad facial expression" depending on the state of wrinkles between the eyebrows from "good facial expression", and the target feature amount deviates from the target feature amount by the facial expression judgment threshold of 232 or more. When the state 233 is reached, the measurement of the elapsed time is started. When the elapsed time exceeds the notification judgment time, the user is notified of the warning information of the deterioration of the facial expression and the instruction information instructing the improvement of the facial expression such as "relax the force between the eyebrows".

Graph 4 shows a characteristic line 241 when the characteristic example forming the facial expression is a wrinkled state of the face. A "good expression" with no wrinkles on the face, but a lot of wrinkles on the face makes it look old and gives a moody image or a dark impression, resulting in a "bad expression". Along the characteristic line 241 the "facial expression at the time of shooting" approaches "bad facial expression" depending on the state of wrinkles on the face from "good facial expression", and the target feature amount deviates from the target feature amount by the facial expression judgment threshold 242 or more. When the state 243 is reached, the measurement of the elapsed time is started. When the elapsed time exceeds the notification judgment time, the user is notified of the warning information of the deterioration of the facial expression and the instruction information instructing the improvement of the facial expression such as "tighten the face". The same applies to other characteristic examples that form facial expressions, such as the posture of the nose and face.

For example, when the face is turned downward, the double chin is regarded as one of the facial expressions, and the change in the chin line representing the double chin is captured from the photographed face, and the double chin is not formed. When the deviation of the feature amount from the "good facial expression" exceeds the facial expression judgment threshold, a warning or instruction is given to pay attention to the posture. As a result, it is possible to prevent the face from turning downward and becoming a stoop and giving an unpleasant facial expression, and by not keeping the face facing downward, an effect of eliminating stiff shoulders can be obtained as an incidental effect.

Here, the facial expression determination threshold value may be initially set in advance, or may be changed at any time to the optimum setting according to the shooting situation or state. Further, the facial expression determination threshold value for each feature may be set to a different threshold value as shown in FIG. 2, or may be set to the same threshold level for a plurality of features. Further, when the facial expression determination threshold value is exceeded for each feature, the facial expression deterioration warning information or the facial expression improvement instruction information may be notified, or when the facial expression deterioration warning information or the facial expression deterioration instruction information is exceeded for a plurality of features. You may notify the instruction information of facial expression improvement.

For example, after the deviation of the feature amount for at least two or more features such as wrinkles between the eyes, mouth, and eyebrows exceeds the table judgment threshold, "laugh" with the warning information of facial expression deterioration. You may notify the instruction information of the facial expression improvement common to a plurality of features such as. That is, when the target feature amount deviates from the target feature amount by the facial expression determination threshold value or more in at least one feature of each feature of the "facial expression at the time of shooting", the user is instructed to give warning information indicating deterioration of the facial expression and to improve the facial expression. Notify information, or both warning and instructional information. As a result, it is possible to issue a warning of facial expression deterioration and an instruction to improve facial expressions in a form suitable for any feature of "facial expression at the time of shooting".

Next, a configuration example of the smartphone 100 according to the present embodiment will be described with reference to FIG. FIG. 3 is a functional block diagram of a configuration example of the smartphone 100 according to the present embodiment.

The smartphone 100 includes an in-camera 101, a display 102, a speaker 103, a microphone 104, a vibrator 105, a touch panel 106, an out-camera 107, a timer 108, a processor 310, a memory 320, and a communication device 340, which are connected to each other via a bus 350. Connected to and configured.

Each of the in-camera 101 and the out-camera 107 captures the peripheral visual field, and the light incident from the lens is converted into an electric signal by an image sensor to acquire an image. The in-camera 101 is provided on the same surface as the screen of the display 102 in the smartphone 100. The in-camera 101 photographs the face of the user 10 who is looking at the screen, and captures the face image in the smartphone 100. The out-camera 107 is provided on the surface (rear surface) of the smartphone 100 on the side opposite to the screen of the display 102.

The display 102 is composed of a liquid crystal panel or the like, and displays images and videos, as well as notification information to the user 10, such as warning information on deterioration of facial expressions and instruction information for improving facial expressions, and icons for starting applications and various status displays on the screen. Display on.

The speaker 103 emits various output information (including notification information and instruction information) in the smartphone 100 to the outside by voice.

The microphone 104 collects voices from the outside and voices of the user 10 itself, converts them into voice signals, and captures them in the smartphone 100. The operation information by the voiced sound from the user 10 can be taken into the smartphone 100, and the operation for the operation information can be executed conveniently.

The vibrator 105 generates vibration under the control of the processor 310, and converts the notification information to the user 10 transmitted by the smartphone 100 into vibration. When the facial expression deteriorates, vibration can be generated to notify the user 10.

The touch panel 106 is laminated on the screen of the display 102. The touch panel 106 is composed of, for example, a capacitive touch panel, and can input information that the user 10 wants to input by approaching or touching (hereinafter, referred to as touch) with a finger, a stylus, or the like. The touch panel 106 is an example of an operation input member. The operation input member is, for example, a keyboard or key button communicated and connected via a communication device 340, a voice input device in which the user 10 utters a voice indicating an input operation, collects the sound with the microphone 104, and converts it into operation information, or a user. It may be a gesture input device that captures and analyzes the gesture of.

The processor 310 is composed of a CPU, an MPU, etc., and constitutes a controller of the smartphone 100. The processor 310 reads and executes the operating system (Operating System: OS) 322 stored and stored in the memory 320 as the control program 321 and the operation control application program 323 in the work area of the memory 320, respectively. It controls the components and realizes the functions of the OS 322 and the application program 323.

When the processor 310 executes the application program 323, it constitutes each functional block of the feature amount analysis unit 311, the target image registration unit 312, the feature amount comparison unit 313, the notification unit 314, and the face image correction unit 315.

The feature amount analysis unit 311 analyzes the feature amount that quantitatively represents each feature of the facial expression from the face image, and the feature amount (target feature amount) of the target image 111 and the feature amount (target feature amount) of the target image 112. Is calculated.

The target image registration unit 312 registers the selected “good facial expression” face image as the target image 111. The user 10 may operate the touch panel 106 to select and register the target image 111, or may automatically select and register the target image 111 from the captured or stored moving images. In the case of user selection, it is possible to select a favorite "good facial expression" by the user's subjectivity, and in the case of automatic selection, it is possible to objectively select a "good facial expression" without the user being aware of it.

The feature amount comparison unit 313 compares the target feature amount and the target feature amount, and calculates the deviation amount Δfa of these feature amounts.

The notification unit 314 provides warning information for warning when the facial expression deteriorates, instruction information for promoting improvement toward the "good facial expression", and facial expression information indicating the perspective state of the "good facial expression" such as the speaker 103. It is output from the notification member of.

The face image correction unit 315 corrects the target image 112 so that it can be easily compared with the target image 111. Specifically, in the face image correction unit 315, the face captured in the target image 112 is oriented diagonally or tilted, which is different from the orientation of the face having a “good facial expression” captured in the target image 111. In this case, the target image 112 is corrected so that the orientation of the photographed face is the same as the orientation of the face of the target image 111. The facial expression of the face image corrected by the face image correction unit 315 is regarded as the "facial expression at the time of shooting", and the feature amount analysis unit 311 analyzes the feature amount. Even if there is, it becomes possible to make a more accurate comparison between the target feature amount and the target feature amount.

The memory 320 is composed of a non-volatile storage device or the like, and stores various programs 321 and information data 324 handled by the processor 310 or the like.

The information data 324 includes face image information 325 including the target image 111 and the target image 112, feature amount information 326 showing the result of the feature amount analysis unit 311 analyzing each of the target image 111 and the target image 112, and facial expression determination threshold information 327. , Notification judgment time information 328, warning instruction information 329 indicating the contents of facial expression deterioration warning and facial expression improvement instruction, facial expression indicating how close the "expression at the time of shooting" is to the registered "good facial expression". Information 330 and the like are stored.

The communication device 340 is a communication interface that performs wireless communication with other mobile information terminals or the like by short-range wireless communication, wireless LAN, or base station communication, and provides a communication processing circuit, an antenna, and the like corresponding to various predetermined communication interfaces. include. The communication device 340 receives, for example, a face image or video of the user 10's "good facial expression" on another personal digital assistant or an external information server, or receives a face image of the user 10's "facial expression at the time of shooting". Send it to other personal digital assistants.

As an example of short-range wireless communication, for example, communication using an electronic tag may be used. Also, if the smartphone 100 is near other mobile information terminals and at least wireless communication is possible, Bluetooth (registered trademark), IrDA (Infrared Data Association, registered trademark), Zigbee (registered trademark), HomeRF. (Home Radio Frequency, registered trademark) or Wi-Fi (registered trademark) standard compliant communication.

Further, as base station communication, long-distance wireless communication such as W-CDMA (Wideband Code Division Multiple Access, registered trademark) or GSM (Global System for Mobile Communications) may be used. Although not shown, the communication device 340 may use other methods such as optical communication and sound wave communication as a means of wireless communication.

Also, when dealing with high-definition video, the amount of data will increase dramatically, so if you use a high-speed, large-capacity communication network such as 5G (5th Generation: 5th generation mobile communication system) or local 5G for wireless communication, , The usability can be dramatically improved.

Next, the basic operation of the smartphone 100 according to the present embodiment will be described with reference to FIG. FIG. 4 is a flowchart illustrating the basic operation of the smartphone 100 according to the present embodiment.

First, the target image registration unit 312 accepts the selection of the user 10, or the target image registration unit 312 automatically selects and registers the target image 111 (S401).

After that, when the user 10 starts to visually recognize the screen of the smartphone 100, the in-camera 101 photographs the face of the user 10 and generates the target image 112 (S402: Yes). When the smartphone 100 starts displaying the display data on the screen of the display 102, the smartphone 100 considers that the user 10 has started to visually recognize the screen, and also starts taking an image of the in-camera 101. The smartphone 100 stands by when the display data is not displayed on the screen (S402: No).

The feature quantity analysis unit 311 executes face recognition processing on the image captured by the in-camera 101, and if the face is captured, the feature quantity analysis and calculation is started with the image as the target image 112. When the in-camera 101 starts imaging and this step is executed for the first time, the feature amount analysis unit 311 analyzes the target image 111 and calculates the target feature amount. On the other hand, when the feature amount analysis unit 311 returns from step S410 described later and executes this step again, the feature amount analysis of the target image 111 is omitted. Further, the feature amount analysis unit 311 calculates the target feature amount by performing the feature amount analysis every time the target image 112 is updated or every predetermined frame (S403).

The feature amount comparison unit 313 compares the target feature amount and the target feature amount, and calculates the deviation amount Δfa (S403).

The feature amount comparison unit 313 determines whether the deviation amount Δfa is equal to or greater than the facial expression determination threshold value (S404). If affirmative (S404: Yes), the measurement of the elapsed time T is started (S405).

The feature amount comparison unit 313 determines whether the elapsed time T is equal to or longer than the notification determination time (S405). If affirmative (S405: Yes), the notification unit 314 notifies the user 10 of the warning information of facial expression deterioration and the instruction information of facial expression improvement (S406).

If the processor 310 continues the facial expression monitoring process (S402 to S405) and the notification process (S406) following the denial in step S404, the denial in step S405, or S406 (S407: Yes), the processor returns to step S403.

When the facial expression monitoring process (S402 to S405) and the notification process (S406) are terminated (S407: No), the processor 310 terminates a series of processes. The "end" case is, for example, a case where the display 102 is finished or a case where the main power of the smartphone 100 is turned off.

According to the present embodiment, the target feature amount of the "facial expression during shooting" captured by the target image 112 deviates from the target feature amount of the "good facial expression" captured by the target image 111 by the facial expression determination threshold or more. , The user can be notified of the facial expression deterioration warning and the facial expression improvement instruction only when the shifted state elapses for more than the notification determination time. Therefore, when the user is visually recognizing the smartphone 100 for a long time and continues to have an unfavorable facial expression for a long time, it is possible to notify the state and always induce a good facial expression, and a bad facial expression is fixed. You can solve this problem and achieve a good facial expression without being aware of it.

<Second Embodiment>
The second embodiment is an embodiment in which the feature amount is analyzed after correcting the target image 112. The second embodiment will be described with reference to FIGS. 5A and 5B. 5A and 5B are diagrams schematically showing a processing outline of the second embodiment.

FIG. 5A shows a case where the user 10 is visually recognizing the smartphone 100 from an angle.

In the facial expression monitoring process, the smartphone 100 photographs the face of the user 10 who is gazing at the display 102 with the in-camera 101, and captures the facial image 501 of the "facial expression at the time of photographing". The face image correction unit 315 determines the orientation of the face of the face image 501, and corrects the image so as to be the same as the orientation of the face of the target image 111 facing the front.

The feature amount analysis unit 311 analyzes the target feature amount by using the corrected image of the face image 501 as the target image 502. Therefore, even if the face of the face image 501 obtained by capturing the "facial expression at the time of shooting" is oriented diagonally, the "facial expression at the time of shooting" and the "good facial expression" are obtained by correcting the direction of the face and keeping the same face orientation. It is possible to analyze each feature amount of "" and to compare and discriminate facial expression feature amounts more accurately.

FIG. 5B shows a case where the user 10 tilts his head and visually recognizes the smartphone 100.

In the facial expression monitoring process, the smartphone 100 photographs the face of the user 10 who is gazing at the display 102 with the in-camera 101, and captures the facial image 503 of the "facial expression at the time of photographing". The face image correction unit 315 determines the inclination of the face of the face image 503, and corrects the image so as to match the inclination of the face of the target image 111 facing the front.

After that, as in the case of FIG. 5A, the feature amount analysis unit 311 uses the image corrected by the face image 503 as the target image 504 and analyzes the target feature amount. As a result, the feature amount analysis unit 311 can more accurately compare and discriminate the feature amount of the facial expression. As an example, the case where the photographed face has an oblique direction or an inclination has been described, but even in other different and similar states, the face image correction unit 315 has the facial expression of "expression at the time of photography" and "the facial expression". It is clear that a similar effect can be obtained by correcting the facial image taken to match the facial condition of "good facial expression".

<Third Embodiment>
The third embodiment is an embodiment in which the smartphone 100 automatically updates the target image 111 in which a face with a “good facial expression” is captured. FIG. 6 is a flowchart illustrating the operation of the smartphone 100 according to the third embodiment.

In FIG. 6, following step S403, it is determined whether or not the photographed facial expression is a "good facial expression" rather than the registered "good facial expression" (S601). When the value of the target feature amount is higher than the value of the target feature amount, the feature amount comparison unit 313 determines that the registered "good expression" is a "good expression".

When the facial expression of the photographed face is a "good facial expression" (highly evaluated) rather than the registered "good facial expression" (S601: Yes), the target image registration unit 312 uses the target image 112 to display the target image. 111 is updated and registered (S602).

After the update registration of the face image of "good facial expression" (S602), the process proceeds to step S404 as in the case where the target image 112 is not "good facial expression" even more than the target image 111 (S601: No).

According to the present embodiment, when it is determined that the image has a better "good facial expression" than the registered "good facial expression" at the time of user shooting, the image is updated and registered to have a better "good facial expression". , Can lead to even better "good facial expressions".

<Fourth Embodiment>
The fourth embodiment is an embodiment that notifies how far away the "good facial expression" in which the "facial expression at the time of shooting" is registered is. FIG. 7 is a flowchart illustrating the operation of the smartphone 100 according to the fourth embodiment.

In FIG. 7, when the feature amount comparison unit 313 determines in step S404 that the deviation amount Δfa is less than the facial expression determination threshold value (S404: No), the notification unit 314 calculates (1-Δfa)% and (1-Δfa)%. Display information including Δfa)% is output (S701). After that, the process proceeds to step S407.

FIG. 8 is a diagram showing a screen display example of step S701. As facial expression information indicating how far the "facial expression at the time of shooting" is in the registered "good facial expression", the display 102 shows "the level of" good facial expression "is now 80%". Character information 801 is displayed. Further, the speaker 103 may emit a voice 802 such as "The level of good facial expression is now 80%".

As a result, the user 10 who is visually recognizing the screen of the smartphone 100 can always numerically know what kind of facial expression his / her facial expression is, and can be aware that it does not lead to a “bad facial expression”. ..

FIG. 9A is a diagram for supplementarily explaining the icon example shown in FIG. 8, and is a diagram showing another example of the screen display in step S701. In FIG. 9A, the notification unit 314 uses an icon 901 representing the state of the facial expression as facial expression information, and displays this on the display 102.

FIG. 9B is a diagram showing the types of icons. There are multiple types of icons, such as icon 901 representing "good facial expression", icon 902 representing normal facial expression, and icon 903 representing "bad facial expression", depending on the degree of matching of facial expression features (1-Δfa). Prepare the icon of. The notification unit 314 selects an icon corresponding to (1-Δfa)% and displays it on the screen (S701).

The "bad facial expression" icon 903 is displayed on the display 102 as needed by the notification unit 314 using it as notification information in step S406.

By using the icon, the user can easily recognize the facial expression state of his / her face in a relatively small display area that does not interfere with the original information display.

<Fifth Embodiment>
The fifth embodiment is an embodiment in which the deterioration of the facial expression of the user is monitored from a remote outside. FIG. 10 is a diagram schematically showing a usage environment of the smartphone 100 according to the fifth embodiment.

In FIG. 10, the smartphone 100 seen by the user 10 is wirelessly connected to another mobile information terminal 20, another mobile information terminal 30, and an external information server 40 seen by the user 11 via a communication device 340. Information is transmitted and received by communication, wireless LAN, or base station communication.

Due to such a usage environment, the smartphone 100 receives a facial image or a moving image of the user 10's "good facial expression" stored in each of the other mobile information terminal 20, the other mobile information terminal 30, and the information server 40. However, it is possible to register more accurate "good facial expressions" from a wider range and accurately notify the facial expression deterioration warning and facial expression improvement instruction with ease.

Further, the user 11 who transmits the face image 112a of the "facial expression at the time of shooting" of the user 10 to the other mobile information terminal 20 and operates the other mobile information terminal 20 can know the facial expression state of the user 10. It is possible to speak out and give guidance as needed. As a result, it is possible to build a system that can monitor the deterioration of the user's facial expression from a remote outside.

<Sixth Embodiment>
The sixth embodiment is an embodiment that captures the facial expression features of a part of the face and evaluates whether the deviation amount of the feature amount is equal to or larger than the facial expression determination threshold value. For example, in a state where only the upper part of the face can be constantly monitored depending on how the smartphone 100 is held, the condition of the eyes is surely photographed as the part of the upper part of the face that is particularly worrisome, and the facial expression of the eyes is captured. Calculate and evaluate the amount deviation.

In that case, use the image of the eyes as the target image of the "good facial expression" to be compared. As a result, only the eye condition needs to be registered, which has the effect of reducing the burden of the registration process.

This embodiment is suitable when a head-mounted display (hereinafter referred to as "HMD") is used as a mobile information terminal.

FIG. 11A is a diagram showing the appearance of the HMD1100.

The HMD1100 is attached to the head of the user 10 and displays a virtual space or a real space on the display 1102. The HMD 1100 includes a camera 1101 provided inside, a left headphone 1103 and a right headphone 1104 instead of a speaker, and a left vibrator 1105 and a right vibrator 1106.

The camera 1101 can capture the peripheral portion of the eyes of the user 10.

FIG. 11B is a target image 1111 composed of an image of the user's eyes captured by the camera 1101. Further, FIG. 11C is a target image 1112 showing a “good facial expression” of the eyes registered in advance.

The HMD1100 captures the facial expression of the eye from the target image 1111 and calculates the amount of deviation of the feature amount from the registered target image 1112 with the "good facial expression" of the eye. Then, the HMD1100 displays the warning information of the deterioration of the facial expression and the instruction information of the improvement of the facial expression on the display 1102. Further, the HMD 1100 may emit warning information and facial expression improvement instruction information by voice from the left headphone 1103 and the right headphone 1104. Further, the left vibrator 1105 and the right vibrator 1106, which are in direct contact with the user 10, may transmit the vibration indicating the warning information to the user.

According to the first to sixth embodiments, when the user visually recognizes the mobile information terminal for a long time and keeps an unfavorable facial expression for a long time, it is possible to solve the problem that a bad facial expression is fixed, and the facial expression is always good. You can guide and realize a good facial expression without being aware of it. In addition to the state of wrinkles in the eyes, mouth, eyebrows, and face, if it is a feature that forms a facial expression, such as the nose and face orientation, the same action, action, and effect can be obtained by capturing the feature. Needless to say, it will be done.

In the above explanation, a smartphone or a head-mounted display is taken as an example as a mobile information terminal, but the description is not limited to this, and at least it can be applied to a mobile information terminal having a notification function that can take a picture of a user.

The present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above-described embodiment has been described in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to the one including all the described configurations. Further, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Further, it is possible to add / delete / replace a part of the configuration of each embodiment with another configuration.

Further, each of the above configurations, functions, processing units, processing means, etc. may be realized by hardware by designing a part or all of them by, for example, an integrated circuit. Further, each of the above configurations, functions, and the like may be realized by software by the processor interpreting and executing a program that realizes each function. Information such as programs, tables, and files that realize each function may be stored in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD. However, it may be stored in a device on the communication network.

In addition, the control lines and information lines indicate those that are considered necessary for explanation, and do not necessarily indicate all control lines and information lines in the product. In practice, it can be considered that almost all configurations are interconnected.

Further, in the above embodiment, the warning information for the deterioration of the facial expression is notified when the two conditions of the facial expression judgment threshold value or more and the notification judgment time or more are satisfied, but the warning information is notified when the facial expression judgment threshold value or more is satisfied. You may.

10: User 11: User 20: Mobile information terminal 30: Mobile information terminal 40: Information server 100: Smartphone 101: In-camera 102: Display 103: Speaker 104: Microphone 105: Vibrator 106: Touch panel 107: Out-camera 108: Timer 111 : Target image 112: Target image 112a: Face image 113: Character information 114: Icon 115: Voice 116: Character information 117: Icon 118: Voice 119: Voice 120: Voice 121: Voice 122: Voice 201: Face 202: Face 203 : Face 211: Characteristic line 213: Facial expression state 221: Characteristic line 223: Facial expression state 231: Characteristic line 233: State 241: Characteristic line 243: State 310: Processor 311: Feature amount analysis unit 312: Target image registration unit 313: Feature Amount comparison unit 314: Notification unit 315: Face image correction unit 320: Memory 321: Program 322: Operating system 323: Application program 324: Information data 325: Face image information 326: Feature amount information 327: Facial expression determination threshold information 328: Notification Judgment time information 329: Warning instruction information 330: Expression information 340: Communication device 350: Bus 411: Feature quantity analysis calculation processing unit 501: Face image 502: Target image 503: Face image 504: Target image 701: Character information 702: Voice 901: Icon 902: Icon 903: Icon 1100: HMD
1101: Camera 1102: Display 1103: Left headphone 1104: Right headphone 1105: Left vibrator 1106: Right vibrator 1111: Target image 1112: Target image

Claims

It ’s a mobile information terminal,
With the camera
With the display
With memory
A notification member that outputs notification information and
A processor connected to each of the camera, the display, and the memory.
The memory stores a target image in which the facial expression of the target user is captured.
The processor
The target feature amount indicating the feature of the target facial expression of the user captured in the target image is analyzed.
When the information is displayed on the screen of the display, the camera reads the target image obtained by capturing the face of the user, and analyzes the target feature amount showing the characteristics of the facial expression of the user captured on the target image. ,
The amount of deviation between the target feature amount and the target feature amount is calculated.
When it is determined that the deviation amount is equal to or greater than a predetermined facial expression determination threshold value for determining that the user's facial expression has deteriorated, the notification member is made to output notification information based on the result of the determination.
A mobile information terminal characterized by that.
The mobile information terminal according to claim 1.
The notification information output by the notification member includes warning information indicating that the user's facial expression has deteriorated, instruction information for improving the user's facial expression, and the user's facial expression as the target facial expression of the user. At least one or more facial expression information indicating how far away the person is.
A mobile information terminal characterized by that.
The mobile information terminal according to claim 1.
Further equipped with a timer connected to the processor
The processor
When it is determined that the deviation amount is equal to or greater than a predetermined facial expression determination threshold value for determining that the user's facial expression has deteriorated, the timer is instructed to measure the elapsed time.
When it is determined that the elapsed time is equal to or longer than the notification determination time predetermined for determining that the user's facial expression deterioration state has continued, the notification member is made to output notification information based on the result of the determination. ,
A mobile information terminal characterized by that.
The mobile information terminal according to claim 1.
Further provided with an operation input member that accepts the user's operation,
The processor
When the target image selection and registration operation is received from the operation input member, it is stored in the memory.
A mobile information terminal characterized by that.
The mobile information terminal according to claim 1.
The processor selects a face image of the target facial expression of the user from the captured moving images and stores the target image in the memory as the target image, or updates the target image already stored in the memory. And remember
A mobile information terminal characterized by that.
The mobile information terminal according to claim 1.
The feature amount has a characteristic that the target feature amount has a relatively large value and the facial expression worsened than the target facial expression of the user has a relatively small value.
When the value of the target feature amount is larger than the value of the target feature amount, the processor updates and stores the target image stored in the memory to the target image.
A mobile information terminal characterized by that.
The mobile information terminal according to claim 1.
The processor corrects at least one or both of the directions and angles of the face image captured by the camera, and analyzes the target feature amount using the corrected face image as the target image.
A mobile information terminal characterized by that.
It is a facial expression monitoring method when visually recognizing a mobile information terminal.
A step in which the mobile information terminal stores a target image in which the facial expression of the target user is captured, and
A step of analyzing a target feature amount indicating the feature of the target facial expression of the user captured in the target image, and a step of analyzing the target feature amount.
A step of reading a target image of the face of the user who visually recognizes the screen when the information is displayed on the screen of the display of the mobile information terminal.
A step of analyzing a target feature amount indicating the facial expression feature of the user captured in the target image, and a step of analyzing the target feature amount.
The step of calculating the deviation amount of the target feature amount and the target feature amount, and
When it is determined that the deviation amount is equal to or greater than a predetermined facial expression determination threshold value for determining that the user's facial expression has deteriorated, a step of outputting notification information based on the result of the determination, and a step of outputting notification information.
A facial expression monitoring method characterized by including.