JP6202882B2 - Facial expression detection system and facial expression detection method - Google Patents

Description

  The present invention relates to a technique for a facial expression detection system and a facial expression detection method for detecting information related to a facial expression of a subject from the face image of the subject.

  2. Description of the Related Art Conventionally, techniques of a facial expression detection system and a facial expression detection method for detecting information related to a facial expression of a subject from the subject's facial image are known. For example, as described in Patent Document 1.

  In the technique described in Patent Document 1, a facial expression transition map (database) in which the degree of human facial expression is classified into a plurality of stages is created in advance, and the database and the actual facial expression of the person (subject) photographed by the camera are used. By comparing, the degree of facial expression of the person can be determined.

  However, when creating a database used in such a system, it has been difficult to acquire natural facial expression data (face organ position information, etc.), particularly during normal times. This is because, when shooting for the purpose of collecting human facial expression data, the person consciously or unconsciously creates a facial expression due to tension, etc., and cannot always be said to be a natural facial expression. Because there is.

JP 2001-43345 A

  The present invention has been made in view of the above situation, and the problem to be solved is facial expression detection capable of easily and accurately acquiring the position information of the organ of the face of the subject at normal times. A system and a facial expression detection method are provided.

  The problem to be solved by the present invention is as described above. Next, means for solving the problem will be described.

That is, in claim 1, an imaging unit capable of imaging the subject's face, a display unit disposed at a position visible from the subject, and the subject imaged by the imaging unit. The personal information of the subject person is estimated from the face image, information corresponding to the subject person is displayed on the display means based on the personal information, and the information displayed on the display means is visually recognized. An image of the target person's face is acquired by the imaging means, and normal position information that is position information of the organ of the target person's face is determined based on the image as a standard face database Control means for storing, and when the control means obtains an image of the face of the subject who is viewing the information, the control means detects position information of the organ of the subject's face. , The detected location information To determine how much the subject's face is inclined with respect to the imaging means, and obtain only an image in which the subject's face is not inclined more than a predetermined value with respect to the imaging means. The normal position information is determined on the basis of the detected position information for the.

  According to a second aspect of the present invention, the control means uses the standard face database to detect that the subject currently imaged by the imaging means has expressed a specific facial expression.

  According to a third aspect of the present invention, the control means stores a specific facial expression database relating to displacement of positional information of the facial organs when a specific facial expression is obtained with respect to positional information of the facial organs during normal times for a plurality of prestored persons. Further, it is used to detect that the subject currently imaged by the imaging means has expressed a specific facial expression.

  According to a fourth aspect of the present invention, the specific facial expression is a facial expression that indicates the subject's discomfort.

According to a fifth aspect of the present invention, the control means averages the position information after normalizing the position information of the organ of the subject's face in the plurality of images acquired by the imaging means .

In the present invention, when the subject is the same person as the subject who has previously stored the standard face database , the control means stores a new standard face database of the subject. It is something that does not do .

In claim 7, personal information of the target person is estimated from the face image of the target person, information corresponding to the target person is visually recognized based on the personal information, and the information is visually recognized. An image of the face of the subject in the middle is acquired by the imaging means, and normal position information, which is position information of the organ of the face of the subject in the normal state, is determined based on the image, and a standard face database And using the standard face database, the facial expression detection method for detecting that the subject has expressed a specific facial expression, wherein the face of the subject who is viewing the information is detected. When acquiring an image, it detects position information of the organ of the subject's face, determines how much the subject's face is inclined with respect to the imaging means based on the detected position information, The subject's face is the imaging means In which retrieves only image not tilted more than a predetermined value, determines the normal time position information based on the detected position information of the image against.

  As effects of the present invention, the following effects can be obtained.

According to the first aspect, the position information (standard face database) of the organ of the subject's face in normal times can be obtained (created) easily and accurately.
Further, by eliminating images with a large inclination of the subject's face relative to the imaging means, it is possible to create a standard face database with higher accuracy.

  According to claim 2, since it is not necessary to prepare in advance a database unique to the subject to be detected symmetrically (such as a database related to positional information of facial organs when a specific facial expression is made), labor for creating the database Can be reduced.

  According to the third aspect of the present invention, it is possible to easily detect an uncomfortable facial expression without requiring a specific facial expression database unique to the subject to be detected symmetrically.

  In the fourth aspect, the detection result of the specific facial expression can be used for control (improvement) of the environment around the subject.

According to the fifth aspect of the present invention, it is possible to create a standard face database with higher accuracy by normalizing the position information of the organ of the face of the subject in each image .

According to the sixth aspect of the present invention, for the subject who has previously stored the standard face database, the process of storing the standard face database again can be omitted, and the processing efficiency of the entire facial expression detection system can be improved. .

According to the seventh aspect, the position information (standard face database) of the organ of the subject's face in the normal time can be obtained (created) easily and accurately. Further, it is possible to detect a specific facial expression without preparing a database unique to the subject who is symmetrical to the detection in advance, and it is possible to reduce labor for creating the database .

Schematic which showed the structure of the facial expression detection system which concerns on one Embodiment of this invention. The flowchart which showed the facial expression detection method which concerns on one Embodiment of this invention. Similarly, a flow diagram. The flowchart which showed the process of step S105 in FIG. The schematic diagram which showed a mode that the organ of the face was converted into a coordinate. (A) The schematic diagram which showed the state in which the face is facing the front of the camera. (B) The schematic diagram which showed the state inclined largely to the left similarly. (C) The schematic diagram which showed the state inclined largely downward similarly. The schematic diagram which showed a mode that a coordinate was converted into distance information. Schematic which showed the structure of the discomfort degree estimation system which is an application example. The figure which showed the structure of the Bayesian network. The figure which showed the judgment standard of the grimace by visual observation.

  In the following description, the vertical direction and the horizontal direction are defined according to the arrows shown in the drawings as necessary.

  Below, the structure of the facial expression detection system 10 which concerns on this embodiment is demonstrated using FIG.

  The facial expression detection system 10 detects information relating to the facial expression of the subject person (in the present embodiment, the frequency with which the subject person 20 expresses a specific facial expression) from the face image of the subject person 20. The facial expression detection system 10 mainly includes a camera 11, a monitor 12, and a control device 13.

  The camera 11 is an embodiment of the imaging means according to the present invention. The camera 11 images the face of the subject 20. The camera 11 is arranged in front of the seated subject 20 and can always capture the face of the subject 20.

  The monitor 12 is an embodiment of the display means according to the present invention. The monitor 12 is capable of displaying various information such as images and sentences. The monitor 12 is disposed in front of the seated subject 20 (below the camera 11), and the subject 20 can visually recognize the information displayed on the monitor 12.

The control device 13 performs predetermined arithmetic processing, storage, and the like based on various information. The control device 13 includes a storage unit such as a RAM and a ROM, an arithmetic processing unit such as a CPU, and the like.
The control device 13 is connected to the camera 11 and can acquire image data captured by the camera 11.
The control device 13 is connected to the monitor 12 and can display arbitrary information on the monitor 12.

  The facial expression detection system 10 configured as described above can detect information related to the facial expression of the subject 20. More specifically, the facial expression detection system 10 can detect the frequency with which the subject 20 expresses a specific facial expression (in the present embodiment, “blink” and “face frown”). Hereinafter, the detection state (expression detection method) by the expression detection system 10 will be described with reference to FIGS. 2 to 7.

In step S <b> 101, the control device 13 recognizes the target person 20 as a detection target sitting in front of the camera 11.
Specifically, the control device 13 recognizes the target person 20 from the face image of the target person 20 captured by the camera 11. This process can be performed using a commercially available program or the like.
After performing the processing of step S101, the control device 13 proceeds to step S102.

In step S102, the control device 13 determines whether or not the subject 20 recognized in step S101 is the subject recognized for the first time.
When the control device 13 determines that the target person 20 recognized in step S101 is the first target person recognized (a target person who has not been recognized so far), the control device 13 proceeds to step S103.
When the control device 13 determines that the target person 20 recognized in step S101 is not the first target person recognized (the target person has been recognized so far), the control device 13 proceeds to step S107.

In step S <b> 103, the control device 13 estimates the personal information of the target person 20. This process can be performed using a commercially available program or the like.
Here, the “personal information” includes all information related to individuals that can be estimated from the face of the subject 20. In the present embodiment, the gender and age of the subject 20 are used as “personal information”. That is, the control device 13 estimates the gender and age of the subject 20.
After performing the process of step S103, the control device 13 proceeds to step S104.

In step S104, the control device 13 causes the monitor 12 to display information that is likely to be of interest to the person of the gender and age estimated in step S103 (target person 20).
Specifically, the control device 13 stores in advance information that is likely to be interesting for each sex and age (age) as a database (hereinafter referred to as “information database”).
Here, the “information” includes all kinds of information such as images (still images such as advertisement images and moving images) and sentences that can be displayed on the monitor 12.
Based on the gender and age estimated in step S103, the control device 13 extracts information that the subject 20 is likely to be interested from the information database, and displays the information on the monitor 12.
After performing the process of step S104, the control device 13 proceeds to step S105.

  In step S <b> 105, the control device 13 takes a plurality of images of the face of the subject 20 with the camera 11, and acquires position information of the organ of the face of the subject 20 in the plurality of images. Hereinafter, the process of step S105 will be described in more detail.

In step S201 shown in FIG. 4, the control device 13 captures and acquires an image of the face of the subject 20 who is viewing (gazing) the monitor 12 on which the information of interest is displayed in step S104. . Thus, it is considered that the facial expression of the subject 20 gazing at the information of interest is a normal natural expression in which emotions are not conspicuously expressed.
After performing the processing of step S201, the control device 13 proceeds to step S202.

In step S202, the control device 13 converts the facial organs (eyes E, eyebrows B, nose N) of the subject 20 shown in the image acquired in step S201 into coordinates and stores them.
Specifically, as shown in FIG. 5, among the faces of the subject 20 shown in the acquired image data, the end point located on the innermost side of the right eye E is the point P1 and the upper point of the right eye E. A point located on the eyelid (more specifically, a point at the end of the upper eyelid that is located at the top) is a point P2 and a point located on the lower eyelid of the right eye E (more specifically, the lower eyelid Of the right eyebrow B, the innermost end point of the right eyebrow B, the innermost end point of the right eyebrow B, and the left eye E of the innermost end point of the left eye E as the point P5. , A point located on the upper eyelid of the left eye E (more specifically, an end point located at the top of the upper eyelid) and a point located on the lower eyelid of the left eye E (more specifically) Is the point at the end of the lower eyelid that is located at the lowermost end), the point P8 is the point at the innermost end of the left eyebrow B, and the outermost point of the nose N. The point at the end which is positioned below the point P9, respectively determined. Then, the control device 13 temporarily stores the determined coordinates.
The coordinate conversion (determination from the point P1 to the point P9) is executed by a program stored in advance in the control device 13.
After performing the process of step S202, the control device 13 proceeds to step S203.

In step S203, the control device 13 determines whether or not the face of the subject 20 shown in the image acquired in step S201 is tilted to such an extent that the face cannot be accepted as data (a certain accuracy may not be secured as data). Determine.
Specifically, based on the coordinates acquired in step S202, the control device 13 moves left and right (FIG. 6B) from the position where the face of the subject 20 is directly in front of the camera 11 (see FIG. 6A). (See FIG. 6) and the vertical angle (see FIG. 6C) is determined.
The control device 13 determines that the face of the subject 20 is unacceptable as data when at least one of the angle tilted left and right and the angle tilted up and down is equal to or greater than a predetermined threshold value. It is judged that it is greatly inclined to.

If the control device 13 determines in step S203 that the face of the subject 20 is tilted so much that it cannot be accepted as data, the control device 13 proceeds to step S204.
When the control device 13 determines that the face of the subject 20 is not tilted so much as unacceptable as data, the control device 13 proceeds to step S205.

In step S204, the control device 13 deletes the coordinate data (step S202) converted from the image determined that the face of the subject 20 is tilted so much that it is not acceptable as data (see step S203).
After performing the process of step S204, the control device 13 proceeds to step S201 again.

In step S205 transferred from step S203, the control device 13 normalizes the coordinates of the facial organ of the subject 20 shown in the acquired image.
Specifically, even if it is data that is determined in step S203 that the face of the subject 20 is not greatly inclined, the face may be inclined somewhat. Therefore, the control device 13 normalizes the coordinates so as to cancel the influence of the inclination of the face of the subject 20 on the coordinates, that is, the measurement error, using homography transformation, affine transformation, or the like.
After performing the process of step S205, the control device 13 proceeds to step S206.

In step S206, the control device 13 converts the image acquired in step S201 into five-dimensional position information (distance information) and stores it.
Specifically, as shown in FIG. 7, the control device 13 determines the coordinates of the normalized image from five pieces of distance information, that is, (1) the distance from the point P1 to the point P4, and (2) from the point P5. The distance is converted into a distance from the point P8, (3) a distance from the point P4 to the point P8, (4) a distance from the point P2 to the point P3, and (5) a distance from the point P6 to the point P7. Then, the control device 13 stores the converted five-dimensional position information.
After performing the process of step S206, the control device 13 proceeds to step S207.

In step S207, the control device 13 determines whether or not the position information stored in step S206 has reached a predetermined number (for example, 50).
Here, the “predetermined number” is a value that can be arbitrarily set in advance, and is desirably set to a value that can accurately calculate the five-dimensional position information averaged in step S106 described later.
When it is determined that the stored position information has reached a predetermined number, the control device 13 proceeds to step S208.
When it is determined that the stored position information has not reached the predetermined number, the control device 13 proceeds to step S201. That is, the processes from step S201 to step S206 are repeated until a predetermined number of pieces of position information are stored.

In step S208, the control device 13 ends the display on the monitor 12 of information that the subject 20 is likely to be interested in.
After performing the process of step S208, the control device 13 ends the process of step S105 and proceeds to step S106 shown in FIG.

In step S106 shown in FIG. 2, the control device 13 averages each of the plurality of 5-dimensional position information acquired in step S105, and stores the averaged 5-dimensional position information as a standard face database.
Since this standard face database is created based on facial expressions while the subject 20 is viewing information of interest (the monitor 12), the normal face database of the subject 20 is displayed. The position information of the organ is shown.
After performing the process of step S106, the control device 13 proceeds to step S108 and step S109 shown in FIG.

On the other hand, in step S107 transferred from step S102, the control device 13 reads the standard face database of the subject 20 that was created when the subject 20 was previously recognized.
That is, when it is determined that the target person 20 is a person who has been recognized before (step S102), the control device 13 creates a standard face database by reading out the standard face database of the target person 20 previously created. The process (from step S103 to step S106) can be omitted.
After performing the processing of step S107, the control device 13 proceeds to step S108 and step S109 shown in FIG.

In step S <b> 108 shown in FIG. 3, the control device 13 detects that the subject 20 whose face is being imaged by the camera 11 blinks.
Specifically, the control device 13 determines (4) the distance from the point P2 to the point P3 and (5) the point in the five-dimensional position information from the face image of the subject 20 currently captured. Position information regarding the distance from P6 to point P7, that is, the degree of opening (opening / closing degree) of the left and right eyes E is always detected. Hereinafter, the two pieces of position information are simply referred to as “actual eye position information”.
Then, when the actual eye position information is constantly compared with the position information of the standard face database and the actual eye position information is smaller than the position information of the standard face database, the subject 20 Judging that you blinked.
After performing the process of step S108 (specifically, after continuously performing the process of step S108 (detection of blinking) for a predetermined time), the control device 13 proceeds to step S110.

In step S109 illustrated in FIG. 3, the control device 13 detects that the subject 20 whose face has been imaged by the camera 11 has frowned (face frowning).
Specifically, the control device 13 detects the face information of a plurality of people in the normal position of the facial organ (converted to five dimensions in the same manner as described above (see FIG. 7)), The five-dimensional position information of the facial organ at the time of the frowning and the difference (displacement) between the two pieces of position information are acquired (stored) in advance as a specific facial expression database.
Here, the subject person 20 may not be included in the “plurality of people”.
Then, the control device 13 creates an identification function for identifying whether or not a person has frowned using the specific facial expression database as training data. As the identification function, a support vector machine polynomial kernel method or the like can be used.
Then, the control device 13 always detects five-dimensional position information from the face image of the subject 20 currently being imaged, applies the position information and the standard face database to the identification function, and the subject 20 frowns. It is determined whether or not.
After performing the process of step S109 (specifically, after continuing the process of step S109 (detection of frowning) for a predetermined time), the control device 13 proceeds to step S110.

In step S <b> 110, the control device 13 detects the frequency of blinking and frowning expression by the subject 20.
Specifically, the control device 13 calculates and stores the number of times (frequency) per unit time of blinking and frowning by the subject 20 detected in step S108 and step S109, that is, the frequency.

  Here, the expression of blinking or frowning is an example of a specific expression according to the present invention, and is an expression indicating discomfort of the subject 20. For example, when the subject 20 frequently blinks and frowns, it is considered that the subject 20 exhibits discomfort such as strong illumination and dazzling.

  Data regarding the frequency of expression of specific facial expressions (blinks and frowning) by the subject 20 detected by the facial expression detection system 10 as described above can be used for various purposes. For example, the glare or darkness that the subject 20 feels is estimated according to the frequency of the expression, and the lighting, blinds, curtain opening / closing, etc. are controlled accordingly, and the frequency of the expression at the restaurant, etc. Accordingly, it is possible to estimate customer satisfaction and utilize data related to the satisfaction for marketing and customer service, to estimate a person in need, and to smoothly listen (voice).

  Since the facial expression detection system 10 does not need to create a database of 20 target persons in advance and can be applied to an unspecified number of persons (target persons), it can be used for a wide range of applications (for example, an unspecified number of persons). It can be used for a store to visit). Further, even if it is known to apply to a specific target person 20 in advance, it is not necessary to create a database of the target person 20 in advance, so that the facial expression detection system 10 can be easily used. can do.

As described above, the facial expression detection system 10 according to the present embodiment is
A camera 11 (imaging means) capable of imaging the face of the subject 20,
A monitor 12 (display means) disposed at a position visible from the subject 20;
The personal information (gender and age) of the subject 20 is estimated from the face image of the subject 20 captured by the camera 11 (step S103), and information corresponding to the subject 20 is monitored 12 based on the personal information. (Step S104), the camera 11 acquires an image of the face of the target person 20 who is viewing the information displayed on the monitor 12 (step S201), and based on the image, Control device 13 (control means) for determining position information of the organ of the face of the subject 20 (step S106) and storing it as a standard face database;
It comprises.
With this configuration, position information (standard face database) of the organ of the face of the subject 20 at normal times can be acquired (created) easily and accurately.
That is, by displaying information corresponding to the subject 20 on the monitor 12, an image of the face that the subject 20 is naturally viewing the information (the monitor 12) can be acquired. The position information of the facial organ of the subject 20 in the facial expression can be acquired easily and accurately.

In addition, the control device 13
Using the standard face database, it is detected that the subject 20 currently captured by the camera 11 has expressed a specific facial expression (step S108 and step S109).
By configuring in this way, it is not necessary to prepare in advance a database unique to the subject 20 that is to be detected symmetrically (such as a database related to positional information of facial organs when a specific facial expression is made), so that the database is created. Can be reduced.
In addition, since the standard face database necessary for detecting that the subject 20 currently captured by the camera 11 has expressed a specific facial expression can be easily created, the database creation of the entire facial expression detection system 10 is created. Can reduce the labor for.

In addition, the control device 13
A specific facial expression database relating to the displacement of facial organ position information when a specific facial expression of facial organ position information of a plurality of people is stored in a normal state is further captured by the current camera 11. It is detected that the subject 20 presents a specific facial expression (step S109).
By comprising in this way, it is not necessary to prepare the database specific to the subject 20 to be detected symmetrically, and a specific facial expression can be easily detected.

The specific facial expression is
It is a facial expression which shows the subject 20's discomfort.
With this configuration, the detection result of a specific facial expression can be used for control (improvement) of the environment around the subject 20 and the like.

The control device 13
When acquiring an image of the face of the target person 20 who is viewing the information, only an image in which the face of the target person 20 is not inclined more than a predetermined value with respect to the camera 11 is acquired (step S204, etc.). Is.
By configuring in this way, it is possible to create a standard face database with higher accuracy by eliminating images with a large inclination of the face of the subject 20 with respect to the camera 11.

The control device 13
After normalizing the position information of the organ of the face of the subject 20 in the plurality of images acquired by the camera 11 (step S205), the position information is averaged.
With this configuration, it is possible to create a standard face database with higher accuracy by normalizing the position information of the facial organs of the subject 20 in each image.

The control device 13
When the target person 20 is the same person as the target person 20 who has previously stored the standard face database, the new standard face database of the target person 20 is not stored (step S107, etc.) It is.
With this configuration, the process of storing the standard face database again can be omitted for the subject 20 who has previously stored the standard face database, and the processing of the entire facial expression detection system 10 can be made more efficient. Can be planned.

In addition, the facial expression detection method according to this embodiment includes:
Estimating the personal information of the subject 20 from the face image of the subject 20,
Based on the personal information, let the subject 20 visually recognize information corresponding to the subject 20,
Obtaining an image of the face of the subject 20 who is viewing the information;
Determine the position information of the organ of the face of the subject 20 in normal time based on the image, and obtain as a standard face database,
The standard face database is used to detect that the subject 20 has expressed a specific facial expression.
With this configuration, position information (standard face database) of the organ of the face of the subject 20 at normal times can be acquired (created) easily and accurately. Further, it is possible to detect a specific facial expression without preparing a database specific to the subject 20 to be detected symmetrically in advance, and it is possible to reduce labor for creating the database.

  In the present embodiment, the camera 11 is used as the imaging unit according to the present invention. However, as long as the information necessary for the control according to the present invention can be acquired, various imaging units (for example, display unit (monitor 12) are used. And the like (integrated (built-in)) can be used.

  In this embodiment, the monitor 12 is used as the display means according to the present invention. However, various display means (for example, a personal computer monitor or a display means and an input) can be used as long as the information according to the present invention can be displayed. It is possible to use a touch panel in which means are integrated.

  In the present embodiment, the control device 13 is used as the control means according to the present invention. However, various control means (for example, a commercially available personal computer, a plurality of computers, etc.) can be used as long as the control according to the present invention can be performed. It is possible to use a combination of control devices of a table.

  Further, in the present embodiment, the facial expression showing the uncomfortable feeling of the subject 20 is exemplified as the specific facial expression, but the present invention is not limited to the facial expression showing the unpleasant feeling but other facial expressions (facial expressions showing emotions). Can also be applied.

  In the present embodiment, the control device 13 recognizes the subject 20 using a commercially available program or the like in step S101 in FIG. 2, but for example, the coordinates of the facial organs described above (step S205 in FIG. 4). ) Or five-dimensional position information (step S206), the target person 20 can be recognized.

  In the present embodiment, the control device 13 determines the inclination of the face of the subject 20 based on the coordinates acquired in step S202 in step S203 of FIG. It is also possible to adopt a configuration in which the inclination is determined from the facial features.

  In the present embodiment, the control device 13 once acquires an image in step S201 in FIG. 4 and then deletes an image whose face is greatly inclined from the image (step S203 and step S204). However, it is also possible to calculate the inclination of the face at the time of acquiring the image in step S201, and not to acquire an image in which the face is greatly inclined from the beginning.

In the following, the facial expression detection system 10 described above is used in FIG. 8 to FIG. 10 in the surrounding environment (in this embodiment, particularly the lighting environment) An example of application as a discomfort level estimation system 100 that estimates how much discomfort (discomfort level) will be described.
The discomfort level estimation system 100 is one application example of the facial expression detection system 10, and the facial expression detection system 10 can be applied to other various systems.

The discomfort level estimation system 100 shown in FIG. 8 mainly includes a camera 11, a monitor 12, a ceiling light 14, a window light 15, a discomfort level measurement dial 16, and a control device 13.
Note that, in the discomfort level estimation system 100, the configuration of the camera 11 and the monitor 12 is substantially the same as that of the facial expression detection system 10 described above, and thus the description thereof is omitted.

  The ceiling lighting 14 is a lighting device for the indoor space of the room 30. The ceiling lighting 14 is a ceiling of the room 30 and is disposed above the subject 20. The illumination by the ceiling lighting 14 is configured to be changeable.

  The window lighting 15 is a lighting fixture for reproducing a window provided in the room 30 in a pseudo manner. The window illumination 15 is disposed on the side wall surface of the room 30 and in front of the subject 20. The brightness of the window illumination 15 is configured to be changeable. By causing the window illumination 15 to emit light, a state in which sunlight is inserted from a window provided in the room 30 can be simulated.

  The discomfort level measurement dial 16 is for measuring how much the subject 20 is uncomfortable (discomfort level). The discomfort measurement dial 16 is constituted by a dial-type switch, and is disposed in a range that can be reached by the subject 20. The discomfort level measurement dial 16 is provided with a scale indicating the discomfort level within a predetermined numerical range. The subject 20 operates the discomfort measurement dial 16 so that the numerical value (scale) indicated by the discomfort measurement dial 16 becomes a value corresponding to the degree of discomfort felt by the subject 20.

The control device 13 is connected to the ceiling lighting 14 and can control the illuminance by the ceiling lighting 14 to be an arbitrary illuminance. Since the control device 13 controls the illuminance of the ceiling illumination 14 itself, it is possible to detect the illuminance by the ceiling illumination 14 without using a separate sensor or the like.
Further, the control device 13 is connected to the window illumination 15 and can control the luminance of the window illumination 15 to be an arbitrary luminance. Since the control device 13 controls the illuminance of the window illumination 15 by itself, it is possible to detect the brightness of the window illumination 15 without using a separate sensor or the like.
The control device 13 is connected to the discomfort level measurement dial 16 and can detect a numerical value (scale) indicated by the discomfort level measurement dial 16.

  Below, the method of estimating the discomfort degree of the subject 20 by the discomfort degree estimation system 100 is demonstrated using FIG.9 and FIG.10.

  The controller 13 included in the discomfort level estimation system 100 stores a Bayesian network constructed as shown in FIG. 9 in advance, and the discomfort level of the target person 20 is estimated using the Bayesian network. The A Bayesian network is a model of probabilistic reasoning in which a causal relationship between a plurality of variables is represented by an acyclic directed graph and a relationship between individual variables is represented by a conditional probability. In the figure, the node on the start point side of the arrow indicates the cause, and the node on the end point side indicates the result. Also, the degree of influence of the cause on the result is quantified with a conditional probability.

  As shown in FIG. 9, the Bayesian network according to the present embodiment includes a frown recognition frequency node N101, a frown expression frequency node N102, a blink recognition frequency node N201, a blink expression frequency node N202, and illuminance ( It is constructed by a node N301 before changing the ceiling), a node N401 before changing luminance (window), a node N501 after changing illuminance (ceiling), a node N601 after changing luminance (window), and a discomfort node N701.

  In the Bayesian network according to the present embodiment, the control device 13 controls the ceiling lighting 14 and the window lighting 15 to change the illumination change in a plurality of patterns for the subject 20 (the illuminance by the ceiling lighting 14 and the luminance of the window lighting 15). It is constructed based on the result of a learning experiment that gives a large change) and is stored in the control device 13. Specifically, the probability (prior probability or conditional probability) of each node is obtained based on the learning experiment and stored in the control device 13.

  Below, each node of the Bayesian network which concerns on this embodiment is demonstrated.

  The frown recognition frequency node N101 is a node that represents the frown frequency of the subject 20. The face fraud of the subject 20 (whether or not the face is frowned) is detected by the control device 13 as described above (see step S109 in FIG. 2). In the learning experiment, it is detected by the control device 13 whether or not the subject 20 is frowning (step S109 in FIG. 2), and the frequency of frowning of the subject 20 (frequing frequency) is calculated. (Step S110 in FIG. 2). The fraud frequency is classified into a plurality of numerical ranges, and the probability (prior probability) that the fraud frequency is included in each numerical range is stored in the control device 13.

The frowning expression frequency node N102 is a node that represents the fringing frequency of the actual subject 20. In the learning experiment, whether or not the subject 20 actually frowns is confirmed by visual observation or the like, and the frequency of frowning (frequing frequency) of the actual subject 20 is calculated.
Here, in the learning experiment, for example, as shown in FIG. 10, the eye E is narrowed (FIG. 10 (b)) with respect to a facial expression (FIG. 10 (a)) in which the face is not frustrated, When the eye 20 is narrowed by bringing B close (FIG. 10C), the eye E is narrowed by pulling the eyebrow B, and the mouth M is pulled sideways (FIG. 10D), the subject 20 Judged to be frowned.
The calculated frequency of frowning is classified into a plurality of numerical ranges, and the probability (conditional probability) that the frequency of the actual frowning caused by the face fraud recognition frequency node N101 is included in each numerical range is as follows: It is stored in the control device 13.

  The blink recognition frequency node N201 is a node that represents the blink frequency of the subject 20. The blink of the subject 20 is detected by the control device 13 as described above (see step S108 in FIG. 2). In the learning experiment, whether or not the subject 20 blinks is detected by the control device 13 (step S108 in FIG. 2), and the blink frequency (blink frequency) of the subject 20 is calculated ( Step S110 in FIG. The blinking frequency is classified into a plurality of numerical ranges, and the probability that the blinking frequency is included in each numerical range (prior probability) is stored in the control device 13.

  The blinking expression frequency node N202 is a node that represents the blinking frequency of the actual subject 20. In the learning experiment, whether or not the subject 20 actually blinks is confirmed by visual observation or the like, and the blink frequency (blink frequency) of the actual subject 20 is calculated. The actual blink frequency is classified into a plurality of numerical ranges, and the probability (conditional probability) that the actual blink frequency caused by the blink recognition frequency node N201 is included in each numerical range is stored in the control device 13. Is done.

  The illuminance (ceiling) pre-change node N301 is a node that represents the illuminance before the illuminance by the ceiling illumination 14 changes significantly. The illuminance is detected by the control device 13. In the learning experiment, the illuminance before the illuminance due to the ceiling lighting 14 changes greatly is classified into a plurality of numerical ranges, and the probability (prior probability) that the illuminance before the large change is included in each numerical range is given to the control device 13. Remembered.

  The node N401 before change in luminance (window) is a node representing the luminance before the luminance of the window illumination 15 changes greatly. The brightness is detected by the control device 13. In the learning experiment, the luminance before the luminance of the window illumination 15 changes greatly is classified into a plurality of numerical ranges, and the probability (prior probability) that the luminance before the luminance changes is included in each numerical range is given to the control device 13. Remembered.

  The illuminance (ceiling) post-change node N501 is a node representing the illuminance after the illuminance by the ceiling illumination 14 has changed significantly. The illuminance is detected by the control device 13. In the learning experiment, the illuminance after the illuminance due to the ceiling illumination 14 is greatly changed is classified into a plurality of numerical ranges, and the probability (prior probability) that the illuminance after the significant change is included in each numerical range is given to the control device 13. Remembered.

  The post-change luminance (window) node N601 is a node representing the luminance after the luminance of the window illumination 15 has changed significantly. The brightness is detected by the control device 13. In the learning experiment, the luminance after the luminance of the window illumination 15 is greatly changed is classified into a plurality of numerical ranges, and the probability (prior probability) that the luminance after the large change is included in each numerical range is given to the control device 13. Remembered.

  The discomfort level node N701 is a node representing how much the subject 20 is uncomfortable (discomfort level). In the learning experiment, how much the subject 20 is actually uncomfortable (discomfort level) is measured. Specifically, in the learning experiment, the control device 13 detects the numerical value (scale) of the discomfort degree measurement dial 16 operated by the subject 20 when the illuminance by the ceiling illumination 14 and the brightness of the window illumination 15 change greatly. By doing so, it is possible to measure how unpleasant the subject 20 is (discomfort level). The measured discomfort level is classified into a plurality of numerical ranges, such as a frowning expression frequency node N102, a blinking expression frequency node N202, an illuminance (ceiling) pre-change node N301, a luminance (window) pre-change node N401, and an illuminance ( A probability (conditional probability) that the measured discomfort level caused by the node N501 after change and the node N601 after change in luminance (window) is included in each numerical range is stored in the control device 13.

  As described above, the environment of the indoor space of the room 30 has changed using the Bayesian network constructed in advance by the learning experiment (specifically, the illuminance by the ceiling lighting 14 and the brightness of the window lighting 15 have changed greatly). ) Can be estimated as probability theory.

  Specifically, first, when the illuminance by the ceiling illumination 14 in the room 30 and the brightness of the window illumination 15 change, the control device 13 detects information regarding the change. That is, the control device 13 determines the illuminance (illuminance (ceiling) pre-change node N301) before the illuminance by the ceiling illumination 14 changes significantly, and the luminance (luminance (window) pre-change node before the luminance of the window illumination 15 changes significantly. N401), the illuminance after the illuminance by the ceiling illumination 14 changes significantly (node N501 after the illuminance (ceiling) change) and the luminance after the luminance of the window illumination 15 changes greatly (node N601 after the change in luminance (window)). To do.

  In addition, the control device 13 detects an unpleasant facial expression that the subject 20 emits when the environment of the indoor space of the room 30 (illuminance by the ceiling lighting 14 and brightness of the window lighting 15) changes. In other words, the control device 13 detects the frowning and blinking of the subject 20 when the environment of the indoor space of the room 30 changes (see step S108 and step S109 in FIG. 2).

  Next, the control device 13 calculates the frequency (frequency) of an unpleasant facial expression that the detected subject 20 expresses. That is, the control device 13 detects the frequency of frowning of the subject 20 (face fraud recognition frequency node N101) and the frequency of blinking of the subject 20 detected by the blink detection means (blink recognition frequency node N201). Is calculated (see step S110 in FIG. 2).

  Then, the control device 13 performs the above-described nodes (face freckled recognition frequency node N101, blink recognition frequency node N201, node before illumination (ceiling) change N301, node before luminance (window) change N401, and node after illumination (ceiling) change N501. And the discomfort degree of the subject 20 can be taken from the conditional probabilities of the frowning expression frequency node N102, the blinking expression frequency node N202, and the discomfort degree node N701 using the condition of the brightness (window) changed node N601). Value (numerical range) and respective probabilities are calculated. The control device 13 multiplies each of a plurality of values that can be taken by the subject 20 as the degree of discomfort, and estimates a numerical value obtained by adding them as the degree of discomfort of the subject 20.

  As described above, the discomfort degree estimation system 100 according to the present embodiment uses the frequency of the unpleasant facial expression generated by the subject 20 in addition to the information related to the environmental change, and therefore accurately estimates the discomfort degree of the subject 20. Can do. The degree of discomfort of the subject 20 estimated in this way can be used for various purposes.

  In the present embodiment (application example), the room 30 installed as experimental equipment has been described as an example. However, the present invention is not limited to this, and the above-described room is used in daily life. It is also possible to estimate the degree of discomfort.

  Further, although the window illumination 15 is used in the present embodiment, the present invention is not limited to this, and a normal window (a window through which light outside the room 30 enters) and a sensor for detecting the luminance of the window It is also possible to estimate the degree of discomfort described above using.

  In the present embodiment, the degree of discomfort with respect to changes in the lighting environment (illuminance by the ceiling lighting 14 and the brightness of the window lighting 15) is estimated, but other environmental changes (for example, temperature changes, humidity It is also possible to estimate the degree of discomfort with respect to changes.

10 facial expression detection system 11 camera (imaging means)
12 Monitor (display means)
13 Control device (control means)
20 Target people

Claims (7)

Imaging means capable of imaging the face of the subject;
Display means arranged at a position visible from the subject;
The personal information of the target person is estimated from the face image of the target person imaged by the imaging means, information corresponding to the target person is displayed on the display means based on the personal information, and the display means An image of the face of the subject person who is viewing the displayed information is acquired by the imaging means, and the normal position is the position information of the organ of the face of the subject person in the normal state based on the image. Control means for determining position information and storing it as a standard face database;
Equipped with,
The control means includes
When acquiring an image of the face of the target person who is visually recognizing the information, the position information of the organ of the target person's face is detected, and the face of the target person is detected based on the detected position information. Is determined with respect to the imaging unit, and only the image in which the target person's face is not inclined more than a predetermined value with respect to the imaging unit is acquired, and the detected position information about the image is acquired. The normal position information is determined based on
Facial expression detection system. The control means includes
Using the standard face database to detect that the subject currently imaged by the imaging means expresses a specific facial expression,
The facial expression detection system according to claim 1. The control means includes
A specific facial expression database relating to the displacement of facial organ position information when a specific facial expression with respect to facial organ positional information of a plurality of people in normal times is stored is further captured by the imaging means. Detecting that the subject is exposed to a specific facial expression,
The facial expression detection system according to claim 2. The specific facial expression is
It is a facial expression showing discomfort of the subject,
The facial expression detection system according to claim 2 or claim 3. The control means includes
After normalizing the position information of the organ of the subject's face in the plurality of images acquired by the imaging means, the position information is averaged ,
The facial expression detection system according to any one of claims 1 to 4. The control means includes
When the subject is the same person as the subject who has previously stored the standard face database, the new standard face database of the subject is not stored .
The facial expression detection system according to any one of claims 1 to 5. Estimate the subject ’s personal information from the subject ’s face image,
Based on the personal information, let the subject visually recognize information corresponding to the subject,
An image of the face of the subject who is viewing the information is acquired by an imaging unit,
Determine the normal position information that is the position information of the organ of the subject's face in normal time based on the image, and obtain as a standard face database,
A facial expression detection method for detecting that the subject has expressed a specific facial expression using the standard face database,
When acquiring an image of the face of the target person who is visually recognizing the information, the position information of the organ of the target person's face is detected, and the face of the target person is detected based on the detected position information. Is determined with respect to the imaging unit, and only the image in which the target person's face is not inclined more than a predetermined value with respect to the imaging unit is acquired, and the detected position information about the image is acquired. The normal position information is determined based on
Expression detection method.

Images