JP7639322B2 - Human state estimation method and estimation system - Google Patents

Description

The technology disclosed herein belongs to the technical field of human state estimation methods and estimation systems.

Conventionally, it has been considered to create a saliency map from an image showing the subject's surroundings, which represents the attractiveness of objects in the subject's field of vision, and to use the saliency map to estimate various states of the subject.

For example, in Patent Document 1, a saliency map is used to estimate the object at which the gaze of the person being evaluated is directed. Specifically, Patent Document 1 discloses that, based on the gaze direction detection result, a gaze area at which the driver (person being evaluated) is directed in a captured image showing the periphery of the vehicle is extracted, objects included in the captured image are recognized, a top-down saliency map of the captured image is generated based on the captured image and vehicle information, objects having saliency in the top-down saliency map among the recognized objects are detected as candidates for the visual object at which the driver's gaze is directed, and the visual object at which the driver's gaze is directed is determined based on the gaze area extraction result and the detection result of the candidate visual object.

Patent document 1 also defines a top-down saliency map as a saliency map that absorbs factors such as the driver's viewing order, individual differences such as physique and habits, ambiguity of the scene from which the captured image was obtained, and fluctuations in the gaze direction detected by the gaze detection unit.

JP 2020-71528 A

However, after extensive research, the inventors of the present application found that the state of the person being evaluated can be estimated by using a saliency map. In other words, when the person being evaluated is in a normal state and when he or she is absent-minded, there is a difference in the relationship between the saliency in the person's field of vision and the movement of the person's gaze, and by evaluating this difference, the state of the person being evaluated can be estimated.

In evaluations using this saliency map, it is expected that the estimation accuracy will improve if a uniquely determined saliency map, such as the top-down saliency map in Patent Document 1, is used. However, it is very difficult to define how to absorb individual differences such as driver habits and image ambiguity, and in the worst case scenario, there is a risk that the information necessary for estimating the state of the person being evaluated will be absorbed, which could actually lower the estimation accuracy. In addition, the amount of calculation required to generate a saliency map can be enormous, which could lengthen the estimation time.

The technology disclosed here has been developed in consideration of these points, and its purpose is to estimate the human state of the person being evaluated with as high accuracy as possible using calculations that are as simple as possible when estimating the human state using a saliency map.

In order to solve the above problem, the technology disclosed herein is directed to a human state estimation method, and includes a peripheral information acquisition step of acquiring peripheral information of the person being evaluated, a visual field image acquisition step of acquiring an image included in the forward visual field of the person being evaluated, a map generation step of generating a saliency map for the image acquired in the visual field image acquisition step, a first gaze information acquisition step of acquiring the gaze of the person being evaluated with respect to the forward visual field, a second gaze information acquisition step of acquiring the gaze of a comparison person who is another person having attributes similar to the person being evaluated, a first probability calculation step of calculating the probability that the saliency at a position on the saliency map indicated by the gaze of the person being evaluated acquired in the first gaze information acquisition step will be equal to or greater than a predetermined threshold, a second probability calculation step of calculating the probability that the saliency at a position on the saliency map indicated by the gaze of the comparison person acquired in the second gaze information acquisition step will be equal to or greater than the predetermined threshold, and an abnormality estimation step of estimating whether the person being evaluated is in an abnormal state based on the first probability calculated in the first probability calculation step and the second probability calculated in the second probability calculation step.

With this configuration, it is not necessary to absorb individual differences such as the habits of the person being evaluated when generating a saliency map, as in the past, and this makes it easier to generate a saliency map. Furthermore, by comparing the relationship of the person being evaluated's gaze to the saliency map with the relationship of the gaze of another person (comparator) who has attributes similar to that of the person being evaluated to the saliency map, it is possible to improve the accuracy of estimating the human state of the person being evaluated without using a top-down saliency map. Therefore, the human state of the person being evaluated can be accurately estimated with as simple a calculation as possible.

Note that "attributes" here refer to the space in which the comparator is located, the comparator's age, the comparator's level of expertise in the target job (driving a vehicle or sorting products in a factory), etc.

In the human state estimation method, the abnormality estimation process may be configured to estimate that the subject is in an abnormal state when the rate at which the first probability and the second probability deviate by a predetermined value or more is equal to or greater than a preset specific value.

With this configuration, if the person being evaluated is distracted for just a moment, a deviation occurs between the first probability and the second probability, which can prevent the person being estimated to be in an abnormal state. This can improve the accuracy of estimating the human state of the person being evaluated.

In the human state estimation method, the second gaze information acquisition step may be a step of acquiring the gazes of two or more of the comparators, and the second probability calculation step may be a step of determining the saliency at the position on the saliency map indicated by each gaze of each of the comparators, calculating the probability that each saliency is equal to or greater than the predetermined threshold, and setting the average of the probabilities or one probability selected from the probabilities as the second probability.

This configuration makes it possible to absorb slight differences between the person being evaluated and each comparison subject to a certain degree. This makes it possible to further improve the accuracy of estimating the human state of the person being evaluated.

In another aspect of the technology disclosed herein, the human state estimation method includes a peripheral information acquisition step of acquiring peripheral information including the space in which the person being evaluated is located, a visual field image acquisition step of acquiring an image included in the forward visual field of the person being evaluated, a map generation step of generating a saliency map for the image acquired in the visual field image acquisition step, a first gaze information acquisition step of acquiring the gaze of the person being evaluated with respect to the forward visual field, a first probability calculation step of calculating the probability that the saliency at a position on the saliency map indicated by the gaze of the person being evaluated acquired in the first gaze information acquisition step will be equal to or greater than a predetermined threshold, a second probability calculation step of calculating the probability that the saliency at a position on the saliency map indicated by the gaze of the person being evaluated who was in the same space in the past will be equal to or greater than the predetermined threshold, and an abnormality estimation step of estimating whether the person being evaluated is in an abnormal state based on the first probability calculated in the first probability calculation step and the second probability calculated in the second probability calculation step.

With this configuration, since the evaluation is compared with past evaluation subjects, it is possible to absorb as many individual differences as possible, such as the subject's habits and physique, without using a top-down saliency map. This makes it possible to accurately estimate the subject's human condition with as simple a calculation as possible.

The technology disclosed here also applies to human evaluation systems. Specifically, the human state estimation system is configured to include a peripheral information acquisition unit that acquires peripheral information of the person being evaluated, a visual field image acquisition unit that acquires an image included in the forward visual field of the person being evaluated, a map generation unit that generates a saliency map for the image acquired by the visual field image acquisition unit, a gaze information acquisition unit that acquires gaze information of the person being evaluated with respect to the forward visual field, a comparator gaze information acquisition unit that acquires gaze information of a comparator who is another person having attributes similar to the person being evaluated, a first probability calculation unit that calculates the probability that the saliency at a position on the saliency map indicated by the gaze information acquisition unit of the person being evaluated acquired by the gaze information acquisition unit is equal to or greater than a predetermined threshold, a second probability calculation unit that calculates the probability that the saliency at a position on the saliency map indicated by the gaze of the comparator acquired by the comparator gaze information acquisition unit is equal to or greater than the predetermined threshold, and an abnormality estimation unit that estimates whether the person being evaluated is in an abnormal state based on the first probability calculated by the first probability calculation unit and the second probability calculated by the second probability calculation unit.

Even with this configuration, it is not necessary to absorb individual differences such as the habits of the person being evaluated when generating a saliency map, as in the past, so it is easy to generate a saliency map. Furthermore, by comparing the relationship of the person being evaluated's gaze to the saliency map with the relationship of the comparison person's gaze to the saliency map, it is possible to improve the accuracy of estimating the human state of the person being evaluated without using a top-down saliency map. Therefore, the human state of the person being evaluated can be accurately estimated with as simple a calculation as possible.

In the human state estimation system, the abnormality estimation unit may be configured to estimate that the person being evaluated is in an abnormal state when the rate at which the first probability and the second probability deviate by a predetermined value or more is equal to or greater than a preset specific value.

This configuration can improve the accuracy of estimating the human state of the person being evaluated.

In the human state estimation system, the comparator gaze information acquisition unit may acquire the gazes of two or more of the comparators, and the second probability calculation unit may determine the saliency at the position on the saliency map indicated by each gaze of each of the comparators, calculate the probability that each saliency is equal to or greater than the predetermined threshold, and set the second probability to the average of the probabilities or one probability selected from the probabilities.

This configuration can further improve the accuracy of estimating the human state of the person being evaluated.

The human state estimation system may further include a memory unit that stores information regarding the gaze of the person being evaluated, and the second probability calculation unit, when the comparison person is not present around the person being evaluated, reads out from the memory unit information regarding the gaze of the person being evaluated who was in the same space in the past, and calculates the probability that the saliency at a position on the saliency map indicated by the gaze of the person being evaluated in the past will be equal to or greater than the predetermined threshold as the second probability, when the comparison person is not present around the person being evaluated.

With this configuration, the human state of the person being evaluated can be estimated with high accuracy even in a situation where a comparison person is unlikely to be nearby, such as when driving a vehicle late at night. In addition, since information about the evaluator's past line of sight is stored, the second probability can be easily calculated. Therefore, the human state of the person being evaluated can be estimated with high accuracy using calculations that are as simple as possible.

As described above, the technology disclosed herein makes it possible to accurately estimate the human state of the person being evaluated using calculations that are as simple as possible, without the need to use a top-down saliency map, because it compares the person being evaluated with a comparator who has similar attributes to the person being evaluated, or with the person being evaluated when they were in the same space in the past.

FIG. 1 is a schematic diagram of the inside of a vehicle when the human state of a driver of the vehicle is estimated by a human state estimating system according to the first embodiment. FIG. 2 is a front view of the vehicle as seen from the front side. FIG. 3 is a block diagram showing the configuration of the human state estimation system. FIG. 4 is a schematic diagram showing the relationship between a vehicle and a cloud server in the human state estimation system. FIG. 5 is a diagram showing the external environment in front of the vehicle captured by a camera. FIG. 6 is a diagram showing the distribution of saliency calculated based on images captured by a camera. FIG. 7 is a graph showing the saliency of the driver's gaze point. FIG. 8 is a graph showing the saliency of the gaze points of the comparison subjects. FIG. 9 is a graph showing an ROC curve comparing the rate at which the comparison subject's gaze point exceeds the threshold with the rate at which the driver's gaze point exceeds the threshold. FIG. 10 is a diagram showing the concept of calculating the AUC value from the ROC curve shown in FIG. FIG. 11 is a flowchart of human state estimation. FIG. 12 is a schematic diagram showing a state in a factory when the human state of workers in the factory is estimated by the human state estimation system according to the second embodiment. FIG. 13 is a plan view illustrating an example of the arrangement of cameras and the like. FIG. 14 is a block diagram showing the configuration of a human state estimating system according to the second embodiment. FIG. 15 is a flowchart of human state estimation by the human state estimation system according to the second embodiment. FIG. 16 is a block diagram showing a configuration in which the human state estimating system according to the third embodiment is applied to a vehicle. FIG. 17 is a flowchart of human state estimation by the human state estimation system according to the third embodiment. FIG. 18 is a block diagram showing a configuration in which the human state estimating system according to the fourth embodiment is applied to a vehicle. FIG. 19 is a part of a flowchart of human state estimation by the human state estimation system according to the fourth embodiment. FIG. 20 shows the remaining part of the flowchart of human state estimation by the human state estimation system according to the fourth embodiment.

First Embodiment
An exemplary embodiment 1 will be described in detail below with reference to the drawings. In the embodiment 1, a human state estimation system 50 (see FIG. 3) is applied to a driver of a vehicle. In the following description of the embodiment 1, the forward traveling side of the vehicle is simply referred to as the front side, and the backward traveling side is simply referred to as the rear side. In addition, the left side when looking at the front side from the rear side is referred to as the left side, and the opposite is referred to as the right side.

Figure 1 shows a schematic diagram of the interior of a vehicle in which a human state estimation system 50 is used. This vehicle is a right-hand drive vehicle, with the steering wheel 8 located on the right side.

In the vehicle interior, a windshield glass 1 is disposed at the front of the vehicle as viewed from the driver's seat. When viewed from inside the vehicle interior, the windshield glass 1 is defined by a number of vehicle components. Specifically, the windshield glass 1 is defined by left and right front pillar trims 2, a roof trim 3, and an instrument panel 4.

The left and right front pillar trims 2 each form the outer boundary of the front windshield glass 1 in the vehicle width direction. Each front pillar trim 2 is disposed along each front pillar 12. As shown in FIG. 1, each front pillar trim 2 is disposed at an angle so that they are spaced apart from each other toward the upper side.

The roof trim 3 forms the upper boundary of the front windshield glass 1. The roof trim 3 covers the inside of the vehicle cabin of the roof panel 13 of the vehicle. A rearview mirror 5 is attached to the center of the front windshield glass 1 in the vehicle width direction and slightly below the roof trim 3. An in-vehicle camera 101 (see Figure 3) is provided in the roof trim 3 near the rearview mirror 5 to capture images of the interior of the vehicle cabin, and in particular the face of the driver (hereinafter referred to as the person being evaluated). The in-vehicle camera 101 is a high-performance camera that is capable of detecting changes in the pupils of the person being evaluated.

The instrument panel 4 forms the lower boundary of the front windshield glass 1. The instrument panel 4 is provided with a meter box and a display 7.

The vehicle also has side mirrors 6 located on the outside of the left and right front pillars 12 in the vehicle width direction. Each side mirror 6 is positioned so that the subject sitting in the driver's seat can see through the side door window.

As shown in FIG. 2, the vehicle is provided with an exterior camera 100 for capturing images of the external environment in front of the vehicle. The exterior camera 100 is located at the front end of the vehicle, slightly below the bonnet 9 of the vehicle.

The human state estimation system 50 according to this embodiment estimates the state of the person being evaluated by comparing the gaze movements of the person being evaluated with the gaze movements of other drivers (hereinafter referred to as comparators) who are present around the vehicle and have attributes similar to those of the person being evaluated. The "attributes" referred to here refer to the space in which the comparator is located, the age of the comparator, the proficiency of the comparator in the target job (driving a vehicle in this case), and the like. In other words, a comparator refers to, for example, a driver who is located in the same space as the person being evaluated, is at the same level as the person being evaluated, and has a driving level similar to that of the person being evaluated. The attributes of the comparator do not need to be strictly the same as those of the person being evaluated, as long as they can be considered to be at the same level as those of the person being evaluated.

Figure 3 shows a schematic configuration of the human state estimation system 50. The human state estimation system 50 uses a cloud server 70 for some of its calculations. The cloud server 70 acquires information about the person being evaluated from the vehicle VA driven by the person being evaluated, and acquires information about the comparator from another vehicle VB driven by the comparator.

The vehicle VA has a controller 60 for calculating the gaze information of the person being evaluated. This controller 60 is a controller based on a well-known microcomputer, and includes a central processing unit (CPU) for executing programs, a memory configured, for example, from a random access memory (RAM) or a read only memory (ROM) for storing programs and data, and an input/output bus for inputting and outputting electrical signals.

The controller 60 is connected to various sensors as shown in FIG.
An exterior camera 100 for capturing an image included in the forward field of view of the subject;
An in-vehicle camera 101 that captures the face of the person being evaluated;
A vehicle speed sensor 102 for detecting the speed of the host vehicle VA;
An acceleration sensor 103 for detecting a longitudinal acceleration input to the host vehicle VA;
A yaw rate sensor 104 for detecting a lateral acceleration input to the host vehicle VA;
a steering angle sensor 105 for detecting a steering angle of the host vehicle VA from a rotation angle of a steering shaft which rotates in conjunction with the steering wheel 8;
a steering torque sensor 106 for detecting a steering torque input to the steering wheel 8 via the steering shaft;
an accelerator opening sensor 107 that detects an accelerator opening from an amount of depression of an accelerator pedal of the host vehicle VA;
A brake pressure sensor 108 that detects a brake pressure based on a depression amount of a brake pedal of the host vehicle VA;
A GPS sensor 109 that detects the position of the host vehicle VA using a Global Positioning System (GPS);
Includes.

The controller 60 is also connected to a cloud server receiver 71 that acquires information from the cloud server 70. The cloud server receiver 71 and the cloud server 70 can communicate with each other via wireless communication.

The controller 60 has a gaze calculation unit 61 that calculates the gaze direction of the person being evaluated from the eyeball of the person being evaluated photographed by the in-vehicle camera 101. The gaze calculation unit 61 calculates the gaze direction of the person being evaluated, for example, by detecting changes in the pupil of the person being evaluated based on the state in which the person is looking into the lens of the in-vehicle camera 101. The gaze direction may be calculated from either the left or right eye of the person being evaluated, or may be calculated as the average of the gaze directions (gaze vectors) obtained from both eyes of the person being evaluated. In addition, when it is difficult to calculate the gaze direction of the person being evaluated from changes in the pupil of the person being evaluated, the gaze direction may be calculated by further taking into account the direction of the person's face. The in-vehicle camera 101 and the controller 60 of the vehicle VA constitute a gaze information acquisition unit.

The controller 60 has a map generator 62 that calculates saliency, which indicates how easily an object in front of the vehicle attracts the subject's gaze, based on image data acquired by the exterior camera 100, and generates a saliency map. Saliency is a visual feature consisting of salience that changes from moment to moment due to color, brightness, movement, etc. In other words, high saliency indicates high salience, and an area with high saliency is an area that stands out compared to its surroundings. More specifically, an area that has a large color difference or brightness difference compared to the surrounding areas, or that moves significantly compared to its surroundings, has high saliency.

The vehicle VA has a cloud server transmitter 72 for transmitting information from the various sensors 100 to 109 to the cloud server 70. The cloud server transmitter 72 and the cloud server 70 can communicate with each other via wireless communication.

The vehicle VA has a driving assistance device 110 that assists in steering, accelerating, decelerating, and other driving of the vehicle VA according to the estimated state of the person being evaluated.

The configuration of the other vehicle VB is basically the same as that of the host vehicle VA, so a detailed description will be omitted. The gaze calculation unit 61 of the other vehicle VB constitutes a comparison person gaze information acquisition unit that acquires the gaze information of the comparison person.

The cloud server 70 detects other vehicles VB located around the host vehicle VA based on information from the GPS sensors 109 of the host vehicle VA and the other vehicles VB. The GPS sensor 109 and the cloud server 70 constitute a surrounding information acquisition unit.

Based on information from various sensors 100-109 acquired from the vehicle VA and the other vehicle VB, the cloud server 70 determines whether the attributes of the driver of the other vehicle VB are equivalent to the attributes of the person being evaluated. As shown in FIG. 4, the cloud server 70 acquires information from each of the multiple other vehicles VB located around the vehicle VA, and compares the attributes of the driver of each other vehicle VB with the attributes of the person being evaluated. The cloud server 70 then selects a driver of each other vehicle VB who has attributes equivalent to the person being evaluated as a comparator. When there are two or more comparators, the cloud server 70 selects two or more comparators. The cloud server 70 determines whether the attributes of the driver of each other vehicle VB are the same as the attributes of the person being evaluated based on the driving distance, driving frequency, and timing of acceleration and deceleration of each other vehicle VB.

The cloud server 70 calculates the probability (hereinafter referred to as the first probability) that the saliency at a position on the saliency map indicated by the gaze information of the person being evaluated will be equal to or greater than a predetermined threshold based on the saliency map from the subject's vehicle VA and the gaze direction of the person being evaluated. The cloud server 70 also calculates the probability (hereinafter referred to as the second probability) that the saliency at a position on the saliency map indicated by the gaze information of the person being evaluated will be equal to or greater than a predetermined threshold based on the saliency map from the other vehicle VB and the gaze direction of the person being evaluated. The calculation methods of the first probability and the second probability will be described later. The cloud server 70 constitutes a first probability calculation unit and a second probability calculation unit.

The cloud server 70 estimates whether the person being evaluated is in an abnormal state based on the calculated first probability and second probability. The cloud server 70 transmits the estimation result to the vehicle VA. The cloud server 70 constitutes an abnormality estimation unit.

(Generating saliency maps)
Next, generation of the saliency map by the map generating unit 62 will be described.

Figure 5 is an image captured by the exterior camera 100 and included in the forward field of view of the subject. The forward field of view shown in this image includes the roadway 150 and the white line 151 on the roadway 150. The external environment shown in image D1 also includes another vehicle 161, a wall 162 formed on the right side of the roadway 150, trees 163 planted in the area to the right of the wall 162, a hill 164 extending into the area to the right of the wall 162, and a forest 165 formed on the left side of the roadway 150. The external environment shown in image D1 also includes a building 166 and the sky 167 extending above the building 166 and the forest 165.

The map generation unit 62 calculates saliency for the captured image and generates a saliency map. As described above, saliency changes depending on the color, brightness, movement, etc. of the target. The map generation unit 62 calculates saliency for each feature, such as saliency based on color, saliency based on brightness, and saliency based on movement, and generates a saliency map for each feature, and then adds them together to generate the final saliency map.

For example, for saliency based on color, the map generator 62 increases the saliency of a nearby area in an area with a large color difference from the surroundings in the image, compared to an area with a small color difference. Note that the color difference is calculated by the following formula when the RGB of a pixel is (R1, G1, B1) and the RGB of another pixel is (R2, G2, B2).
(Color difference) = {(R2-R1) ² + (G2-G1) ² + (B2-B1) ² } ^1/2
The map generating unit 62 may calculate the saliency so that it continuously increases as the color difference increases, or may set a number of thresholds and calculate the saliency so that it increases by a certain value each time the threshold is exceeded.

For example, with regard to saliency based on luminance, the map generating unit 62 increases the saliency of nearby areas in areas of the image where the luminance difference from the surroundings is large, compared to areas where the luminance difference is small. The map generating unit 62 may calculate the saliency so that it is continuously higher as the luminance difference increases, or may set multiple thresholds and calculate the saliency so that it increases by a certain value each time the threshold is exceeded.

Furthermore, for saliency based on movement, the map generation unit 62 increases the saliency for objects with large relative movement with respect to the host vehicle VA. For example, when a bird crosses the sky 167, the saliency of the area surrounding the bird is increased.

The calculation of saliency itself can be performed using known computer programs available on the Internet, etc. Also, known computer programs can be used to calculate the saliency map for each feature (here, the saliency map) and to integrate each saliency map.

Figure 6 is a saliency map when saliency is calculated for an image captured by the exterior camera 100. As shown in Figure 6, for example, the boundary between trees 163 and forest 165 and sky 167, and the white line 151 on the left side of the roadway 150 are areas of high saliency (high saliency areas) because of high color difference and brightness difference. Conversely, the roadway 150 and sky 167 are largely similar and have low color difference, resulting in many areas of low saliency.

The positions of the vehicle VA and the other vehicle VB are different to be exact, so the saliency maps generated by each map generation unit 62 do not strictly match. However, as long as the majority of the targets are the same, the discrepancy in the saliency maps is not significant enough to cause problems when estimating the human state, as described below.

(Estimation of the human condition)
Next, estimation of a human state will be described with reference to FIGS.

Figure 7 plots the saliency of the gaze point of the person being evaluated, i.e., the point in the direction of the person being evaluated's line of sight. This graph is generated by determining the gaze direction of the person being evaluated using the gaze calculation unit 61, applying the gaze point in the direction of the gaze to a saliency map, and determining the saliency of the gaze point from the saliency map. Each of these points is determined at a predetermined sampling period.

On the other hand, Figure 8 plots the saliency of the comparator's gaze point, i.e., the point in the direction of the comparator's line of sight. This graph is generated by determining the gaze direction of the comparator using the gaze calculation unit 61 of the other vehicle VB, applying the gaze point in the direction of that gaze to the saliency map generated by the map generation unit 62 of the other vehicle VB, and determining the saliency of that gaze point from the saliency map. Each of these points is determined at a predetermined sampling period.

After creating the graphs of FIG. 7 and FIG. 8, the cloud server 70 obtains a receiver operating characteristic (ROC) curve of the percentage of points at which the threshold is exceeded at the gaze point of the comparator and the percentage of points at which the threshold is exceeded at the gaze point of the assessee. Specifically, in the first step, the threshold is set to a value greater than the maximum value in the graphs of FIG. 7 and FIG. 8. Next, in the second step, the percentage of points at which the threshold is exceeded is obtained for each threshold while lowering the threshold. This second step is then repeated until the threshold becomes equal to or less than the minimum value in the graphs of FIG. 7 and FIG. 8. The percentage of points at which the threshold is exceeded at the gaze point of the assessee corresponds to a first probability, and the percentage of points at which the threshold is exceeded at the gaze point of the comparator corresponds to a second probability.

The graphs shown in Figures 7 and 8 are generated in intervals of a predetermined time. The predetermined time is set to a time that is sufficient to create an ROC curve and that allows for an estimation of the human state as early as possible. The predetermined time is, for example, 30 seconds.

After the first and second steps, the cloud server 70 creates an ROC curve as shown in FIG. 9 as a third step by plotting the second probability on the horizontal axis and the first probability on the vertical axis. In the graph of FIG. 9, since both the horizontal and vertical axes are probabilities, the minimum value of the curve is 0 and the maximum value is 1.

In FIG. 9, the dashed line indicates that there is almost no difference between the gaze movement of the subject and the gaze movement of the comparator. In contrast, when the curve spreads upward in a convex shape, as in the curve C1 in FIG. 9, it indicates that the gaze of the subject exceeds the threshold more frequently than the gaze of the comparator. In other words, when a curve with a shape like the curve C1 in FIG. 9 is calculated, it means that the gaze of the subject tends to look at the high saliency area compared to the gaze of the comparator. On the other hand, when the curve spreads downward in a convex shape, as in the curve C2 in FIG. 9, it indicates that the gaze of the subject exceeds the threshold less frequently than the gaze of the comparator. In other words, when a curve with a shape like the curve C2 in FIG. 9 is calculated, it means that the gaze of the subject tends to look at the low saliency area compared to the gaze of the comparator.

In this embodiment, the cloud server 70 calculates the AUC (Area Under Curve) value, which is the area under the ROC curve, as a fourth step. For example, when the ROC curve is curve C1, the AUC value corresponds to the area of the hatched region in FIG. 10. By calculating the AUC value, it is possible to evaluate the degree of deviation between the first probability and the second probability, that is, the degree of deviation between the gaze of the person being evaluated and the gaze of the comparator. Specifically, when the gaze of the person being evaluated tends to be directed toward the high saliency region more than the comparator, the AUC value becomes larger, whereas when the gaze of the person being evaluated tends to be directed toward the low saliency region more than the comparator, the AUC value becomes smaller.

Next, as a fifth step, an estimated index value is calculated, which is the ratio of AUC values that deviate from the reference value by a predetermined value or more among each AUC value for a preset measurement period. The measurement period is longer than the predetermined time, and includes a plurality of predetermined times. In this embodiment, the reference value is set to 0.5. That is, since the comparator has the same attributes as the person being evaluated, if the person being evaluated is in a normal state, there is not much difference between the first probability and the second probability. Therefore, if the person being evaluated is in a normal state, the AUC value will be 0.5. In this embodiment, the negative side of the predetermined value is set to 0.2, and the positive side is set to 0.1. The reason why the positive side is a stricter standard than the negative side is that when the person being evaluated is in an abnormal state such as a absent-minded state, the person being evaluated tends to direct his or her gaze toward a high saliency area. The estimated index value represents the deviation between the saliency of the person being evaluated's gaze point and the saliency of the person being compared. Specifically, the larger the estimated index value, the greater the discrepancy between the saliency of the subject's gaze point and the saliency of the comparison subject's gaze point.

Then, in the sixth step, the cloud server 70 determines whether the estimated index value is equal to or greater than a specific value set in advance. Specifically, if the estimated index value is equal to or greater than a specific value, it is estimated that the person being evaluated is in a different state from the comparator, i.e., in an abnormal state. In this embodiment, the specific value is set to 60%.

In this way, by comparing the saliency of the areas gazed upon by the person being evaluated with the saliency of the areas gazed upon by the comparison person, it is possible to reduce dependency on the driving scene, such as the number and extent of high saliency areas.

In particular, the first probability based on the gaze of the assessee and the second probability based on the gaze of the comparator are compared to calculate the AUC value, and the human state of the assessee can be accurately estimated. Specifically, for example, if the rate at which the assessee looks at the high saliency area is calculated, even if the assessee looks at the entire field of view without bias, the assessee will be determined to look at the high saliency area frequently when the high saliency area is spread over the entire saliency map. In addition, when the assessee is a driver with little driving experience, the field of view tends to be narrow, and the gaze may be biased to a specific location. As in this embodiment, by comparing with the saliency of the location gazed by the comparator, the dependency of the driving scene, such as the number and spread of high saliency areas, and the dependency of individual differences between assessees can be reduced. This makes it possible to accurately estimate the human state of the assessee based on the gaze information of the assessee.

When there is no comparison person around the person being evaluated, the cloud server 70 randomly specifies multiple coordinates on the saliency map generated by the controller 60 of the subject vehicle VA, and calculates the saliency of each coordinate. The second probability is then calculated from the saliency of the random points. If the person being evaluated is in a normal state, the person's gaze is basically directed evenly across the entire field of vision without being pulled toward high saliency areas. For this reason, even when the second probability calculated from the saliency of the random points is compared with the first probability, if the person being evaluated is in a normal state, the ACU will be close to 0.5. Therefore, even if the second probability is calculated from the saliency of the random points, there is no significant decrease in accuracy.

(flowchart)
Next, the processing operation of estimating the human state of the person to be evaluated by the human state estimating system 50 will be described with reference to the flowchart of FIG.

First, in step S101, the human state estimation system 50 acquires various data including information about the surroundings of the vehicle VA and an image of the subject's forward field of vision.

Next, in step S102, the human state estimation system 50 acquires gaze information of the person being evaluated from the image captured by the in-vehicle camera 101 of the subject's vehicle VA. Although not shown in the flowchart, at the time of step S102, the controller 60 of the subject's vehicle VA has generated a saliency map based on the image data acquired in step S101. The gaze information of the person being evaluated acquired in step S102 includes the saliency in the direction of the subject's gaze.

Next, in step S103, the human state estimation system 50 determines whether or not a driver of another vehicle VB who could be a comparator is present around the host vehicle VA. The human state estimation system 50 determines the presence or absence of a comparator based on the position information of the other vehicle VB and the detection results of the various sensors 102-108 related to driving. If the result is YES, meaning that a comparator is present, the human state estimation system 50 proceeds to step S104. On the other hand, if the result is NO, meaning that a comparator is not present, the human state estimation system 50 proceeds to step S108.

In step S104, the human state estimation system 50 acquires gaze information of the comparator from the image captured by the in-vehicle camera 101 of the other vehicle VB. Note that this step S104 is not performed after the judgment of step S103, but is actually performed in the other vehicle VB simultaneously with step S102. Also, although it is omitted in the flowchart, at the time of this step S104, the controller 60 of the other vehicle VB also generates a saliency map based on the image data acquired by the other vehicle VB. The gaze information of the comparator acquired in this step S104 is information that includes the saliency in the direction of the comparator's gaze.

Next, in step S105, the human state estimation system 50 determines whether a predetermined time has elapsed, and in particular whether gaze information of the person being evaluated and the comparison person has been acquired for a predetermined time. This predetermined time is approximately 30 seconds. If the predetermined time has elapsed (YES), the human state estimation system 50 proceeds to step S106. On the other hand, if the predetermined time has not elapsed (NO), the human state estimation system 50 returns to step S101. The reason for returning to step S101 is that the situation around the person being evaluated changes before the predetermined time has elapsed, and there is also a change in the presence or absence of a comparison person, etc.

In step S106, the human state estimation system 50 calculates the first probability from the gaze information of the person being evaluated.

Next, in step S107, the human state estimation system 50 calculates a second probability from the gaze information of the comparison person.

On the other hand, in step S108, which is reached when the determination in step S103 is NO, the human state estimation system 50 determines whether a predetermined time has elapsed, and in particular whether gaze information of the assessee and comparison person has been acquired for a predetermined time. This predetermined time is approximately 30 seconds. If the predetermined time has elapsed (YES), the human state estimation system 50 proceeds to step S109. On the other hand, if the predetermined time has not elapsed (NO), the human state estimation system 50 returns to step S101.

In step S109, the human state estimation system 50 calculates the first probability from the gaze information of the person being evaluated.

Next, in step S110, the human state estimation system 50 calculates a second probability from a random point selected from the saliency map generated by the controller 60 of the host vehicle VA.

Next, in step S111, the human state estimation system 50 calculates an ROC curve based on the first probability calculated in step S106 or step S109 and the second probability calculated in step S107 or step S110.

Next, in step S112, the human state estimation system 50 calculates an AUC value from the ROC curve calculated in step S111.

Next, in step S113, the human state estimation system 50 determines whether data for the measurement period, i.e., AUC values for the measurement period, have been obtained. If AUC values for the measurement period have been obtained (YES), the human state estimation system 50 proceeds to step S114, whereas if AUC values for the measurement period have not been obtained (NO), the human state estimation system 50 returns to step S101 and repeats steps S101 to S112 until AUC values for the measurement period are obtained.

Then, in step S114, the human state estimation system 50 determines whether the estimated index value calculated from the AUC value for the measurement period is equal to or greater than a predetermined specific value. If the estimated index value is equal to or greater than the specific value (YES), the human state estimation system 50 proceeds to step S115 and determines that an abnormality has occurred in the person being evaluated. On the other hand, if the estimated index value is less than the specific value (NO), the human state estimation system 50 proceeds to step S115 and determines that no abnormality has occurred in the person being evaluated.

After steps S115 and S116, return.

The cloud server 70 transmits the estimation result to the controller 60 of the vehicle VA. When the controller 60 of the vehicle VA receives a result indicating that an abnormality has occurred in the person being evaluated, the controller 60 of the vehicle VA assists the person being evaluated in driving the vehicle VA using the driving assistance device 110. When the controller 60 of the vehicle VA estimates that the condition of the person being evaluated is such that the person being evaluated is unable to continue driving the vehicle VA, the controller 60 of the vehicle VA moves the vehicle VA to the shoulder of the road using the driving assistance device 110, and then stops the vehicle VA. The controller 60 of the vehicle VA may transmit a control signal to an alarm device such as a speaker to ask the person being evaluated, for example, "Are you okay?" or "Would you like to take a break?".

In step S103, when there are two or more comparators, the cloud server 70 selects two or more comparators. When the cloud server 70 selects two or more comparators, after the second probability is calculated for each comparator, the cloud server 70 calculates the average value of each second probability and calculates the ROC curve using the average value as the final second probability.

Therefore, the human state estimation system 50 according to the first embodiment acquires peripheral information of the person being evaluated (peripheral information acquisition process), acquires an image included in the forward field of view of the person being evaluated (field of view image acquisition process), generates a saliency map for the acquired image (map generation process), acquires gaze information of the person being evaluated (first gaze information acquisition process), acquires gaze information of a comparator who is another person having attributes similar to the person being evaluated (second gaze information acquisition process), calculates a first probability that the saliency at a position on the saliency map indicated by the gaze of the person being evaluated is equal to or greater than a predetermined threshold (first probability calculation process), calculates a second probability that the saliency at a position on the saliency map indicated by the gaze of the comparator is equal to or greater than a predetermined threshold (second probability calculation process), and estimates whether the person being evaluated is in an abnormal state based on the first probability and the second probability. This makes it easier to generate the saliency map because it is not necessary to absorb individual differences such as the habits of the person being evaluated when generating the saliency map. Furthermore, by comparing the relationship of the gaze of the person being evaluated to the saliency map with the relationship of the gaze of the comparison person to the saliency map, it is possible to improve the accuracy of estimating the human state of the person being evaluated. Therefore, the human state of the person being evaluated can be estimated with high accuracy using calculations that are as simple as possible.

In addition, in this embodiment 1, the human state estimation system 50 estimates that the person being evaluated is in an abnormal state when the rate at which the first probability and the second probability diverge by a predetermined value or more is equal to or greater than a specific value set in advance. This makes it possible to prevent the person being evaluated from being estimated to be in an abnormal state due to a momentary distraction that causes a divergence between the first probability and the second probability. As a result, the accuracy of estimating the human state of the person being evaluated can be improved.

In addition, in this embodiment 1, when there are two or more comparators, the human state estimation system 50 acquires the gaze information of each of the comparators, determines the saliency at the position on the saliency map indicated by each gaze of each comparator, calculates the probability that each saliency will be equal to or greater than the predetermined threshold, and sets the average of these probabilities as the final second probability. This makes it possible to absorb slight differences between the person being evaluated and each of the comparators to a certain extent. As a result, the estimation accuracy of the human state of the person being evaluated can be further improved.

Second Embodiment
Hereinafter, the second embodiment will be described in detail with reference to the drawings. In the following description, the same reference numerals will be used to designate the same parts as those in the first embodiment, and detailed description thereof will be omitted.

In the second embodiment, the human state estimation system 250 (see FIG. 14) is applied to factory workers. In the illustrated factory, as shown in FIG. 12, the worker who is the assessee HA is a worker who inspects the product P flowing on the conveyor device 220. On the line where the assessee HA works, multiple workers including the assessee HA (two workers in FIGS. 12 and 13) are working. In the following explanation, the side of the conveyor device 220 from the worker's perspective is referred to as the front side, and the side opposite the conveyor device 220 from the worker's perspective is referred to as the rear side. In addition, the workers other than the assessee HA shown in FIGS. 12 and 13 are described as comparators HB, who are other people with attributes equivalent to the assessee HA. The attributes referred to here are, for example, age, height, and work proficiency (years of work experience).

13 and 14 show a schematic configuration of the human state estimation system 250 according to the second embodiment. The human state estimation system 250 includes a rear camera 200 that captures images of the forward field of view of the assessee HA and the comparator HB. The rear camera 200 is a fixed camera and is located behind the assessee HA and the comparator HB. A plurality of rear cameras 200 (two in this example) are provided so that an image of the forward field of view of the assessee HA and an image of the forward field of view of the comparator HB can be captured. The human state estimation system 250 includes a front camera 201 for capturing the line of sight of the assessee HA and the comparator HB. The front camera 201 is a fixed camera and is located in front of the assessee HA and the comparator HB. The front camera 201 is configured as a high-performance camera capable of detecting changes in the pupils of the assessee HA and the comparator HB.

The human state estimation system 250 has a controller 260 for calculating the gaze information of the assessee HA and the comparator HB. The controller 260 is a controller based on a well-known microcomputer, and includes a central processing unit (CPU) for executing a program, a memory configured, for example, of a RAM (Random Access Memory) or a ROM (Read Only Memory) for storing programs and data, and an input/output bus for inputting and outputting electrical signals. Image data from the rear camera 200 and the front camera 201 are input to the controller 260. The controller 260 is also connected to a cloud server receiver 271 for acquiring information from a cloud server 270, which will be described later. The cloud server receiver 271 and the cloud server 270 can communicate with each other via wireless communication.

The controller 260 has a gaze calculation unit 261 that calculates the gaze direction of the subject HA and the comparator HB from the subject's eyeball photographed by the front camera 201. The gaze calculation method used by the gaze calculation unit 261 can be the same as that used by the gaze calculation unit 61 in the first embodiment described above, and therefore a detailed description thereof will be omitted.

The controller 60 has a map generation unit 262 that calculates saliency, which indicates the likelihood that an object present in the forward field of view of the subject HA and the comparator HB will attract the gaze of the subject HA and the comparator HB, based on image data acquired by the rear camera 200, and generates a saliency map. The method of generating the saliency map by the map generation unit 262 can be the same as that of the map generation unit 262 in the first embodiment described above, and therefore a detailed description thereof will be omitted.

The human state estimation system 250 of this embodiment 2 also uses a cloud server 270 for part of the calculation. The controller 260 has a cloud server transmitter 272 for transmitting information from the rear camera 200 and the front camera 201, the gaze information of the assessee HA and the comparator HB, and the saliency map to the cloud server 270. The cloud server transmitter 272 and the cloud server 270 can communicate with each other via wireless communication.

The cloud server 270 has a function of selecting a comparator HB from multiple workers. The cloud server 270 identifies the person being evaluated HA and other workers from images captured by the rear camera 200 and the front camera 201. The cloud server 270 selects a comparator HB having attributes equivalent to those of the person being evaluated HA from the other workers identified. In Figures 12 and 13, there is one comparator HB, but when there are two or more comparators HB, the cloud server 70 selects two or more comparators HB.

Based on the information acquired from the controller 260, the cloud server 270 calculates the probability that the saliency at the position on the saliency map indicated by the gaze information of the assessee HA will be equal to or greater than a predetermined threshold (i.e., the first probability). Also, based on the information acquired from the controller 260, the cloud server 270 calculates the probability that the saliency at the position on the saliency map indicated by the gaze information of the comparator HB will be equal to or greater than a predetermined threshold (i.e., the second probability). The method of calculating the first and second probabilities by the cloud server 270 can be the same as the calculation method in the first embodiment described above, and therefore a detailed description will be omitted. In this second embodiment, the cloud server 270 constitutes the first probability calculation unit and the second probability calculation unit.

The cloud server 270 estimates whether the assessee HA is in an abnormal state based on the calculated first probability and second probability. The method of estimating the abnormal state of the assessee HA is the same as in the above-mentioned embodiment 1, and the reference value and predetermined value that serve as the calculation criteria for the estimated index value, and the specific value that serves as the criterion for determining the abnormal state can be the same values as in the above-mentioned embodiment 1.

The cloud server 270 transmits the estimation result to the controller 260 via the cloud server receiver 271. When the controller 260 receives an estimation result from the cloud server 270 indicating that the assessee HA is abnormal, the controller 260 has an alarm device 210 for notifying the worker or the supervisor monitoring the worker of this fact. The alarm device 210 is, for example, a display device such as a speaker or a display that can be confirmed by the supervisor. The alarm device 210 asks the worker questions such as "Are you OK?" and displays a message on the display device that there is an abnormal person.

The controller 260 is also configured to control the conveyor device 220. The controller 260 temporarily stops the conveyor device 220 when it obtains an inferred result that the subject HA is abnormal, and resumes operation of the conveyor device 220 when treatment of the subject HA (such as removing him/her from the line) is completed.

(flowchart)
Next, the processing operation of estimating the human state of the person to be evaluated by the human state estimating system 250 will be described with reference to the flowchart of FIG.

First, in step S201, the human state estimation system 250 acquires various data including surrounding information of the person being evaluated HA and an image of the person being evaluated HA's forward field of vision.

Next, in step S202, the human state estimation system 250 acquires gaze information of the person being evaluated HA from the image captured by the forward camera 201. Although not shown in the flowchart, at the time of this step S202, the controller 260 has generated a saliency map based on the image data acquired in step S101. The gaze information of the person being evaluated HA acquired in this step S202 includes information on the saliency ahead of the person being evaluated HA's gaze.

Next, in step S203, the human state estimation system 250 acquires gaze information of the comparator HB from the image captured by the front camera 201. The gaze information of the comparator HB acquired in step S203 includes saliency in the direction of the gaze of the comparator HB.

Next, in step S204, the human state estimation system 250 determines whether a predetermined time has elapsed, in particular whether the gaze information of the assessee HA and the comparator HB has been acquired for a predetermined time. This predetermined time is approximately 30 seconds. If the predetermined time has elapsed (YES), the human state estimation system 250 proceeds to step S205. On the other hand, if the predetermined time has not elapsed (NO), the human state estimation system 250 returns to step S201.

In step S205, the human state estimation system 250 calculates the first probability from the gaze information of the person being evaluated HA.

Next, in step S206, the human state estimation system 250 calculates the second probability from the gaze information of the comparison person HB.

Next, in step S207, the human state estimation system 250 calculates an ROC curve based on the first probability calculated in step S205 and the second probability calculated in step S206.

Next, in step S208, the human state estimation system 250 calculates the AUC value from the ROC curve calculated in step S207.

Next, in step S208, the human state estimation system 250 determines whether data for the measurement period, i.e., AUC values for the measurement period, have been obtained. If AUC values for the measurement period have been obtained (YES), the human state estimation system 250 proceeds to step S210, whereas if AUC values for the measurement period have not been obtained (NO), the human state estimation system 250 returns to step S201 and repeats steps S201 to S208 until AUC values for the measurement period are obtained.

Then, in step S210, the human state estimation system 250 determines whether the estimated index value calculated from the AUC value for the measurement period is equal to or greater than a predetermined specific value. If the estimated index value is equal to or greater than the specific value (YES), the human state estimation system 250 proceeds to step S211 and determines that an abnormality has occurred in the person being evaluated. On the other hand, if the estimated index value is less than the specific value (NO), the human state estimation system 250 proceeds to step S212 and determines that no abnormality has occurred in the person being evaluated.

After steps S211 and S212, return.

When there are two or more comparators HB, the cloud server 270 selects two or more comparators. When the cloud server 270 selects two or more comparators HB, similar to the above-mentioned embodiment 1, the cloud server 70 calculates the second probability for each comparator HB, then calculates the average value of each second probability, and calculates the ROC curve using the average value as the final second probability.

Therefore, like the human state estimation system 250 of this embodiment 2, the technology disclosed here can also be used as a worker monitoring system in a factory.

Third Embodiment
Hereinafter, the third embodiment will be described in detail with reference to the drawings. In the following description, the same reference numerals will be used to designate the same parts as those in the first and second embodiments, and detailed description thereof will be omitted.

The third embodiment is similar to the first embodiment in that the human state estimation system 350 is applied to the driver of a vehicle, but the estimation method is different from that of the first embodiment.

In this embodiment 3, as shown in FIG. 16, the human state estimation system 350 includes a storage device 380. The storage device 380 stores past gaze information of the driver as the person being evaluated. This gaze information includes saliency in the direction of the person being evaluated. The storage device 380 is configured to be able to communicate with the controller 360 of the vehicle.

In addition to the gaze calculation unit 361 and the map generation unit 362, the controller 360 has a probability calculation unit 363 that calculates the probability (first probability) that the saliency at a position on the saliency map indicated by the gaze information of the person being evaluated will be equal to or greater than a predetermined threshold, based on the gaze information of the person being evaluated. The method of calculating the first probability by the probability calculation unit 363 can be the same as the calculation method described in the first and second embodiments, and therefore a detailed description thereof will be omitted.

The probability calculation unit 363 reads out the gaze information of the person being evaluated who was in the same space in the past from the storage device 380, and calculates the probability that the saliency at the position on the saliency map indicated by the gaze of the person being evaluated in the past will be equal to or greater than a predetermined threshold. In this embodiment 3, the probability based on the gaze information of the person being evaluated in the past corresponds to the second probability. The method of calculating the second probability is the same as the method of calculating the first probability, and therefore a detailed explanation will be omitted. When the gaze information of the person being evaluated who was in the same space in the past does not exist in the storage device 380, the controller 360 randomly specifies multiple coordinates on the saliency map generated by the map generation unit 362, and determines the saliency of each coordinate. The controller 360 then calculates the second probability from the saliency of the random points.

The controller 360 also has an abnormality determination unit 364 that estimates the human state of the person being evaluated based on the first probability and the second probability calculated by the probability calculation unit 363 and determines whether or not an abnormality has occurred in the person being evaluated. The abnormality determination unit 364, like the cloud servers 70 and 270 in the above-mentioned embodiments 1 and 2, obtains an ROC curve from the first probability and the second probability and calculates an AUC value for the ROC curve. The abnormality determination unit 364 then calculates an estimated index value based on each AUC value for the measurement period and determines whether or not an abnormality has occurred in the person being evaluated based on the estimated index value. The reference value and the predetermined value that are the calculation criteria for the estimated index value, and the specific value that is the judgment criterion for the abnormal state can be the same values as those in the above-mentioned embodiment 1.

(flowchart)
Next, the processing operation of estimating the human state of the person to be evaluated by the human state estimating system 350 will be described with reference to the flowchart of FIG.

First, in step S301, the human state estimation system 350 acquires various data including information about the surroundings of the vehicle and an image of the person being evaluated's forward field of vision.

Next, in step S302, the human state estimation system 350 acquires the current gaze information of the person being evaluated from the image captured by the in-vehicle camera 101 of the subject's vehicle. Although not shown in the flowchart, at the time of this step S302, the controller 360 has generated a saliency map based on the image data acquired in step S301. The gaze information of the person being evaluated acquired in this step S302 includes the saliency ahead of the person being evaluated.

Next, in step S303, the human state estimation system 350 determines whether past gaze information of the person being evaluated in the same space is stored in the storage device 380. If past gaze information is stored in the storage device 380 (YES), the process proceeds to step S304, whereas if past gaze information is not stored in the storage device 380 (NO), the process proceeds to step S308. Note that past gaze information includes saliency in the direction of the person being evaluated in the past.

In step S304, the human state estimation system 350 acquires past gaze information from the storage device 380.

Next, in step S305, the human state estimation system 350 determines whether a predetermined time has elapsed, and in particular whether the current gaze information of the person being evaluated has been acquired for a predetermined time. This predetermined time is approximately 30 seconds. If the predetermined time has elapsed (YES), the human state estimation system 350 proceeds to step S306. On the other hand, if the predetermined time has not elapsed (NO), the human state estimation system 350 returns to step S301.

In step S306, the human state estimation system 350 calculates the first probability from the current gaze information of the person being evaluated.

Next, in step S307, the human state estimation system 350 calculates the second probability from the subject's past gaze information.

On the other hand, in step S308, which is reached when the determination in step S303 is NO, the human state estimation system 350 determines whether a predetermined time has elapsed, and in particular whether the current gaze information of the person being evaluated has been acquired for a predetermined time. This predetermined time is approximately 30 seconds. If the result is YES, that is, the predetermined time has elapsed, the human state estimation system 350 proceeds to step S309. On the other hand, if the result is NO, that is, the predetermined time has not elapsed, the human state estimation system 350 returns to step S301.

In step S309, the human state estimation system 350 calculates the first probability from the current gaze information of the person being evaluated.

Next, in step S310, the human state estimation system 350 calculates a second probability from the random point selected from the saliency map.

Next, in step S311, the human state estimation system 350 calculates an ROC curve based on the first probability calculated in step S306 or step S309 and the second probability calculated in step S307 or step S310.

Next, in step S312, the controller 360 calculates the AUC value from the ROC curve calculated in step S311.

Next, in step S313, the human state estimation system 350 determines whether data for the measurement period, i.e., AUC values for the measurement period, have been obtained. If AUC values for the measurement period have been obtained (YES), the human state estimation system 350 proceeds to step S314, whereas if AUC values for the measurement period have not been obtained (NO), the human state estimation system 350 returns to step S301 and repeats steps S301 to S312 until AUC values for the measurement period are obtained.

Then, in step S314, the human state estimation system 350 determines whether the estimated index value calculated from the AUC value for the measurement period is equal to or greater than a predetermined specific value. If the estimated index value is equal to or greater than the specific value (YES), the human state estimation system 350 proceeds to step S315 and determines that an abnormality has occurred in the person being evaluated. On the other hand, if the estimated index value is less than the specific value (NO), the human state estimation system 350 proceeds to step S316 and determines that no abnormality has occurred in the person being evaluated.

After steps S315 and S316, return.

In this way, in the third embodiment, peripheral information including the space in which the person being evaluated is located is acquired (peripheral information acquisition process), an image included in the forward field of view of the person being evaluated is acquired (field of view image acquisition process), a saliency map is generated for the image of the forward field of view (map generation process), gaze information of the person being evaluated for the forward field of view is acquired (gaze information acquisition process), the probability that the saliency at the position on the saliency map indicated by the gaze of the person being evaluated is equal to or greater than a predetermined threshold is calculated (first probability calculation process), the probability that the saliency at the position on the saliency map indicated by the gaze of the person being evaluated who was in the same space in the past is equal to or greater than a predetermined threshold is calculated (second probability calculation process), and based on the first probability and the second probability, it is estimated whether the person being evaluated is in an abnormal state (abnormality estimation process). As a result, it is possible to absorb as much individual difference as possible, such as the habits and physique of the person being evaluated, without creating a top-down saliency map as in the past in order to compare with the person being evaluated. As a result, it is possible to accurately estimate the human state of the person being evaluated with as simple calculation as possible.

Fourth embodiment
Hereinafter, the fourth embodiment will be described in detail with reference to the drawings. In the following description, the same reference numerals will be used to designate the same parts as those in the first to third embodiments, and detailed description thereof will be omitted.

The fourth embodiment is similar to the first and third embodiments in that the human state estimation system 450 is applied to the driver of a vehicle, but the estimation method is different from the first and third embodiments.

In the human state estimation system 450 according to the fourth embodiment, the vehicle VA is equipped with a storage device 480. The storage device 480 stores the past gaze information of the driver as the person being evaluated. This gaze information includes the saliency of the person being evaluated. The storage device 480 is configured to be able to communicate with the controller 460 of the vehicle VA. In FIG. 18, the other vehicle VB does not have a storage device, but the other vehicle VB may also have a storage device that stores the past gaze information of the comparison person.

In the human state estimation system 450, basically, as in the first embodiment described above, the cloud server 70 selects another vehicle VB driven by a driver who is a comparator present in the vicinity of the vehicle VA based on the surrounding information of the vehicle VA. Then, the cloud server 70 acquires the gaze information of the comparator from the controller 460 of the other vehicle VB. However, in the human state estimation system 450, when the cloud server 70 determines that the other vehicle VB driven by the driver who is a comparator does not exist in the vicinity of the vehicle VA, the controller 460 of the vehicle VA acquires the past gaze information of the subject in the same space from the storage device 480 and transmits it to the cloud server 70. In addition, in the human state estimation system 450, when the other vehicle VB driven by the driver who is a comparator does not exist in the vicinity of the vehicle VA and the past gaze information of the subject in the same space does not exist in the storage device 480, the cloud server 70 randomly specifies multiple coordinates on the saliency map generated by the map generation unit 462, and obtains the saliency of each coordinate.

In the human state estimation system 450, the controllers 460 of the subject vehicle VA and the other vehicle VB generate the saliency map and acquire the gaze information of the subject and the comparison subject, as in the first embodiment described above. On the other hand, the cloud server 70 is used to calculate the first probability and the second probability, calculate the ROC curve and AUC value, and estimate the human state of the subject. The same values as in the first embodiment described above can be used for the reference value and the predetermined value that are the calculation criteria for the estimated index value, and the specific value that is the criterion for determining an abnormal state.

(flowchart)
Next, the processing operation of estimating the human state of the person being evaluated by the human state estimation system 450 will be described with reference to the flowcharts of FIGS.

First, in step S401, the human state estimation system 450 acquires various data including information about the surroundings of the vehicle VA and an image of the subject's forward field of vision.

Next, in step S402, the human state estimation system 450 acquires the current gaze information of the person being evaluated from the image captured by the in-vehicle camera 101 of the subject's vehicle VA. Although not shown in the flowchart, at the time of this step S402, the controller 460 of the subject's vehicle VA has generated a saliency map based on the image data acquired in step S401. The gaze information of the person being evaluated acquired in this step S402 includes the saliency ahead of the subject's gaze.

Next, the first probability and the second probability are calculated by the subroutine of step S500.

As shown in FIG. 20, in the subroutine of step S500, first, in step S501, the human state estimation system 450 determines whether or not there is a driver of another vehicle VB who can be a comparator around the host vehicle VA. The human state estimation system 450 determines the presence or absence of a comparator based on the position information of the other vehicle VB and the detection results of the various sensors 102-108 related to driving. If the result is YES, meaning that a comparator is present, the human state estimation system 450 proceeds to step S502. On the other hand, if the result is NO, meaning that a comparator is not present, the human state estimation system 450 proceeds to step S506.

In step S502, the human state estimation system 450 acquires gaze information of the comparator from the image captured by the in-vehicle camera 101 of the other vehicle VB. Note that this step S502 is not performed after the judgment of step S501, but is actually performed in the other vehicle VB simultaneously with step S402. Also, although it is omitted in the flowchart, at the time of step S502, the controller 460 of the other vehicle VB also generates a saliency map based on the image data acquired by the other vehicle VB. The gaze information of the comparator acquired in step S502 includes the saliency ahead of the comparator's gaze.

Next, in step S503, the human state estimation system 450 determines whether a predetermined time has elapsed, and in particular whether gaze information of the person being evaluated and the comparison person has been acquired for a predetermined time. This predetermined time is approximately 30 seconds. If the predetermined time has elapsed (YES), the human state estimation system 450 proceeds to step S504. On the other hand, if the predetermined time has not elapsed (NO), the human state estimation system 450 returns to step S401.

In step S504, the human state estimation system 450 calculates the first probability from the gaze information of the person being evaluated.

Next, in step S505, the human state estimation system 450 calculates a second probability from the gaze information of the comparison person.

On the other hand, in step S506, which is reached when the determination in step S501 is NO, the human state estimation system 450 determines whether or not past gaze information of the person being evaluated in the same space is stored in the storage device 480. If the determination is YES, that is, past gaze information is stored in the storage device 480, the human state estimation system 450 proceeds to step S507. On the other hand, if the determination is NO, that is, that past gaze information is not stored in the storage device 480, the human state estimation system 450 proceeds to step S511. Note that past gaze information is information that also includes saliency in the direction of the person being evaluated in the past.

In step S507, the human state estimation system 450 acquires past gaze information from the storage device 480.

Next, in step S508, the human state estimation system 450 determines whether a predetermined time has elapsed, and in particular whether the current gaze information of the person being evaluated has been acquired for a predetermined time. This predetermined time is approximately 30 seconds. If the predetermined time has elapsed (YES), the human state estimation system 450 proceeds to step S508. On the other hand, if the predetermined time has not elapsed (NO), the human state estimation system 450 returns to step S401.

In step S509, the human state estimation system 450 calculates the first probability from the current gaze information of the person being evaluated.

Next, in step S510, the human state estimation system 450 calculates the second probability from the subject's past gaze information.

On the other hand, in step S511, which is reached when the determination in step S506 is NO, the human state estimation system 450 determines whether or not a predetermined time has elapsed, and in particular whether or not the current gaze information of the person being evaluated has been acquired for a predetermined time. This predetermined time is approximately 30 seconds. If the result is YES, that is, the predetermined time has elapsed, the human state estimation system 450 proceeds to step S512. On the other hand, if the result is NO, that is, the predetermined time has not elapsed, the human state estimation system 450 returns to step S401.

In step S512, the human state estimation system 450 calculates the first probability from the current gaze information of the person being evaluated.

Next, in step S513, the human state estimation system 450 calculates a second probability from the random point selected from the saliency map.

After steps S505, S510, and S513, the process proceeds to step S403 shown in FIG. 19. In step S403, the human state estimation system 450 calculates an ROC curve based on the first probability calculated in step S504, S509, or S512, and the second probability calculated in step S505, S510, or S513.

Next, in step S404, the human state estimation system 450 calculates the AUC value from the ROC curve calculated in step S403.

Next, in step S405, the human state estimation system 450 determines whether data for the measurement period, i.e., AUC values for the measurement period, have been obtained. If AUC values for the measurement period have been obtained (YES), the human state estimation system 450 proceeds to step S406, whereas if AUC values for the measurement period have not been obtained (NO), the human state estimation system 450 returns to step S401 and repeats steps S401 to S404 and S500 until AUC values for the measurement period are obtained.

Then, in step S406, the human state estimation system 450 determines whether the estimated index value calculated from the AUC value for the measurement period is equal to or greater than a predetermined specific value. If the estimated index value is equal to or greater than the specific value (YES), the human state estimation system 450 proceeds to step S407 and determines that an abnormality has occurred in the person being evaluated. On the other hand, if the estimated index value is less than the specific value (NO), the human state estimation system 450 proceeds to step S408 and determines that no abnormality has occurred in the person being evaluated.

After steps S407 and S408, return.

In step S501, when there are two or more comparators, the cloud server 70 selects two or more comparators, as in the first embodiment described above. When the cloud server 70 selects two or more comparators, after the second probability is calculated for each comparator, the cloud server 70 calculates the average value of each second probability and calculates the ROC curve using the average value as the final second probability.

Therefore, in this embodiment 4, the human state of the person being evaluated can be estimated with high accuracy even in situations where a comparison person is unlikely to be nearby, such as when driving a car late at night or driving in the suburbs. In addition, since the second probability is calculated based on information about the person being evaluated's past line of sight, the second probability can be calculated easily and with high accuracy. Therefore, the human state of the person being evaluated can be estimated with high accuracy using calculations that are as simple as possible.

Other Embodiments
The technology disclosed herein is not limited to the above-described embodiment, and may be substituted without departing from the spirit and scope of the claims.

In the above-described first, second, and fourth embodiments, the first probability and the second probability are calculated by the cloud server 70, 270. However, in the configurations of the first, second, and fourth embodiments, the calculation of the first probability and the second probability may be performed by the controller 60, 260, 460. In the configurations of the first, second, and fourth embodiments, the estimation of the human state of the person being evaluated may be performed by the controller 60, 260, 460. In this case, in the first and fourth embodiments, the controller 60, 460 of the host vehicle VA needs to obtain information regarding the second probability from the controller 60, 460 of the other vehicle VB.

In addition, in the above-mentioned embodiment 2, the second probability is calculated from the gaze information of the comparator HB. However, even when the human state estimation system is applied to a factory as in embodiment 2, the second probability may be calculated based on the gaze information of the assessee HA in the past as in embodiments 3 and 4.

In addition, in the above-mentioned first to fourth embodiments, after calculating the ROC curve, the AUC value is calculated, and the human condition of the person being evaluated is estimated based on the AUC value. This is not limiting, and for example, the human condition of the person being evaluated may be estimated by calculating the rate at which the ROC curve deviates from a reference value. The reference value in this case may be, for example, a value on a straight line formed when the first probability and the second probability are the same.

The above-described embodiments are merely examples and should not be interpreted as limiting the scope of the present disclosure. The scope of the present disclosure is defined by the claims, and all modifications and variations that fall within the scope of the claims are within the scope of the present disclosure.

The technology disclosed here is useful in estimating the human state of a person being evaluated using a saliency map, with accurate estimation using calculations as simple as possible.

50, 250, 350, 450 Human state estimation system 60, 260, 360, 460 Controller 61, 261, 361, 461 Gaze calculation unit (first gaze information acquisition unit, second gaze information acquisition unit)
62, 262, 362, 462 Map generation unit 363 Probability calculation unit (first probability calculation unit, second probability calculation unit)
364 Abnormality determination unit 70, 270 Cloud server (peripheral information acquisition unit, first probability calculation unit, second probability calculation unit, abnormality determination unit)

Claims (5)

A human state estimation method, comprising:
A peripheral information acquisition step of acquiring peripheral information of the person being evaluated;
A visual field image acquisition process for acquiring an image included in the subject's forward visual field;
a map generation step of generating a saliency map for the image acquired in the field of view image acquisition step;
A first gaze information acquisition step of acquiring gaze information of the subject with respect to the forward field of view;
A second gaze information acquisition step of acquiring gaze information of a comparator who is another person having attributes equivalent to the assessee;
A first probability calculation step of calculating a probability that the saliency at a position on the saliency map indicated by the gaze of the subject acquired in the first gaze information acquisition step is equal to or greater than a predetermined threshold;
a second probability calculation step of calculating a probability that the saliency at a position on the saliency map indicated by the gaze of the comparison person acquired in the second gaze information acquisition step is equal to or greater than the predetermined threshold;
and an abnormality estimation step of estimating whether the subject is in an abnormal state based on the first probability calculated in the first probability calculation step and the second probability calculated in the second probability calculation step ,
The human state estimation method, characterized in that the abnormality inference process is a process of inferring that the person being evaluated is in an abnormal state when the rate at which the first probability and the second probability deviate by a predetermined value or more is equal to or greater than a predetermined specific value . The human state estimation method according to claim 1 ,
The second gaze information acquisition step is a step of acquiring gaze information of two or more of the comparators,
The human state estimation method, characterized in that the second probability calculation process is a process of determining saliency at positions on the saliency map indicated by each gaze of each of the comparators, calculating the probability that each saliency will be greater than or equal to the specified threshold, and setting the average value of each probabilities or one probability selected from the respective probabilities as the second probability. A human state estimation system, comprising:
A peripheral information acquisition unit that acquires peripheral information of the person being evaluated;
A visual field image acquisition unit that acquires an image included in the forward visual field of the subject;
A map generation unit that generates a saliency map for the image acquired by the field of view image acquisition unit;
A gaze information acquisition unit that acquires gaze information of the subject with respect to the forward field of view;
A comparison person's gaze information acquisition unit that acquires gaze information of a comparison person who is another person having attributes equivalent to the evaluation person;
A first probability calculation unit that calculates a probability that the saliency at a position on the saliency map indicated by the gaze information of the subject acquired by the gaze information acquisition unit is equal to or greater than a predetermined threshold;
a second probability calculation unit that calculates a probability that the saliency at a position on the saliency map indicated by the gaze of the comparator acquired by the comparator gaze information acquisition unit is equal to or greater than the predetermined threshold;
an abnormality estimation unit that estimates whether the subject is in an abnormal state based on the first probability calculated by the first probability calculation unit and the second probability calculated by the second probability calculation unit;
The human state estimation system is characterized in that the abnormality estimation unit estimates that the person being evaluated is in an abnormal state when the rate at which the first probability and the second probability deviate by a predetermined value or more is equal to or greater than a predetermined specific value . The human state estimation system according to claim 3 ,
The comparator gaze information acquisition unit acquires gazes of two or more comparators,
The second probability calculation unit obtains saliency at positions on the saliency map indicated by each gaze of each of the comparators, calculates the probability that each saliency will be greater than or equal to the specified threshold, and sets the average value of each probabilities or one probability selected from the respective probabilities as the second probability. The human state estimation system according to claim 3 or 4 ,
Further comprising a storage unit for storing information regarding the gaze of the person to be evaluated;
The human state estimation system is characterized in that, when the comparison person is not present around the person being evaluated, the second probability calculation unit reads information regarding the gaze of the person being evaluated who was in the same space in the past from the memory unit, and calculates as the second probability the probability that the saliency at a position on the saliency map indicated by the gaze of the person being evaluated in the past will be greater than or equal to the specified threshold.

Images