JP7298508B2 - state estimator - Google Patents

Description

The technology disclosed herein relates to a state estimation device that estimates the state of a subject.

Japanese Unexamined Patent Application Publication No. 2002-200000 discloses a vehicle control device that detects an abnormality of a vehicle driver. This vehicle control device includes a body temperature measurement unit that measures the body temperature of the driver of the vehicle, a running state determination unit that determines whether the running state of the vehicle is abnormal, a measurement result of the body temperature measurement unit and the running state determination. and a vehicle control unit that controls the vehicle based on the determination result of the unit.

JP 2019-73105 A

However, the device of Patent Document 1 cannot estimate an abnormal state that does not involve heat generation. Therefore, it is not possible to estimate a sudden abnormal state such as epilepsy, heart attack, stroke, etc., in which physical function suddenly declines.

The technology disclosed herein has been made in view of this point, and its purpose is to estimate a sudden abnormal state in which a body function suddenly declines.

The technology disclosed herein relates to a state estimation device that estimates the state of a subject. This state estimation device includes a saccade detection unit that detects a saccade of the subject, a first condition that the amplitude of the saccade of the subject within a first period is equal to or less than a first amplitude threshold, and within the first period After at least one of a second condition that the frequency of saccades of the subject is equal to or less than the first frequency threshold is satisfied, a second condition in which the amplitude of the saccades of the subject within a second period is smaller than the first amplitude threshold When both the third condition that the frequency of the saccades of the subject within the second period is less than the amplitude threshold and the fourth condition that the frequency is less than the second frequency threshold smaller than the first frequency threshold are satisfied and an estimating unit for estimating that the subject's condition is a sudden abnormal condition in which physical function suddenly declines.

As a result of intensive research, the inventors of the present application found that when a subject's condition becomes a sudden abnormal state, at least one of the subject's saccade amplitude and frequency decreases compared to normal, and then the subject's saccade amplitude and frequency was found to decrease further. Therefore, it is possible to estimate a sudden abnormal state in which physical function suddenly declines based on whether or not both the third and fourth conditions are met after at least one of the first and second conditions is met. can.

In the state estimating device, the estimating unit determines in advance the duration of a state in which both the third condition and the fourth condition are satisfied after at least one of the first condition and the second condition is satisfied. The condition of the subject may be estimated to be the sudden abnormal condition when the time threshold is exceeded.

In the above configuration, by estimating a sudden abnormal state based on the duration of a state in which both the third condition and the fourth condition are satisfied, it is possible to reduce estimation errors due to sudden noise or the like. As a result, it is possible to improve the accuracy of estimating a sudden abnormal state.

In the state estimating device, the estimating unit, after at least one of the first condition and the second condition is satisfied, when at least one of the third condition and the fourth condition is not satisfied, the subject's It may be configured to presume that the condition is a reduced attention condition.

As a result of intensive research, the inventors of the present application have found that when the subject's state is in a state of reduced attention function, at least one of the amplitude and frequency of the subject's saccades is lower than normal, but unlike the sudden abnormal state, We found that there was no further reduction in the amplitude and frequency of the subject's saccades. Therefore, after at least one of the first condition and the second condition is satisfied, based on whether or not the third condition and the fourth condition are satisfied, the sudden abnormal state and the reduced attention function state can be distinguished and estimated.

The state estimating device performs a first action according to the sudden abnormal state when the state of the subject is estimated to be the sudden abnormal state, and the state of the subject is the deterioration of attention function. A control unit may be provided that performs a second action in accordance with the state of impaired attention function when the state is estimated to be the state.

With the above configuration, it is possible to appropriately perform an action according to the state of the subject estimated by the estimation unit.

In the state estimation device, the subject may be a driver of a vehicle. The first action may include an action of controlling travel of the vehicle so that the vehicle retreats to a safe area. The second action may include an action of outputting information prompting the driver to take a rest.

With the above configuration, the vehicle can be stopped in the safe area when the driver's condition is estimated to be in a sudden abnormal condition. As a result, the safety of the own vehicle (the vehicle driven by the driver) and other vehicles around the own vehicle can be ensured. Further, when the driver's state is estimated to be in a state of reduced attention function, it is possible to prompt the driver to take a rest. As a result, the condition of the driver can be recovered from the attention function deterioration condition to the normal condition, and the running safety of the vehicle can be ensured.

In the state estimation device, the subject may be a driver of a vehicle. The first amplitude threshold and the first frequency threshold may be set based on the amplitude and frequency of the saccade when the driving scene of the vehicle is a predetermined driving scene.

With the above configuration, the first amplitude threshold and the first frequency threshold can be appropriately set in consideration of the characteristics of the driver. As a result, it is possible to improve the accuracy of estimating the state of the driver.

According to the technology disclosed herein, it is possible to estimate a sudden abnormal state in which a physical function suddenly declines.

1 is a block diagram illustrating the configuration of a vehicle control system according to Embodiment 1; FIG. It is a graph for explaining a saccade. 4 is a block diagram illustrating the configuration of a state estimation unit; FIG. 7 is a graph for explaining saccade detection processing; 9 is a flowchart for explaining saccade detection processing; 7 is a graph for explaining extraction of saccades; 1 is a perspective view illustrating a driving simulator; FIG. 4 is a graph showing saccade amplitudes of epileptic patients according to driving scenes. 4 is a graph showing the frequency of saccades of epileptic patients according to driving scenes. It is a graph which shows an epileptic patient's gaze movement at the time of an attack. 5 is a flowchart for explaining state estimation processing; FIG. 11 is a flowchart for explaining Modification 1 of state estimation processing; FIG. FIG. 11 is a flowchart for explaining Modification 2 of the state estimation process; FIG. 4 is a flowchart for explaining an example of a first operation; 9 is a flowchart for explaining an example of a second operation; 4 is a flowchart for explaining threshold setting processing; FIG. 10 is a front view illustrating the appearance of the display device of Embodiment 2; FIG. 8 is a block diagram illustrating the configuration of a display device according to Embodiment 2;

Hereinafter, embodiments will be described in detail with reference to the drawings. The same reference numerals are given to the same or corresponding parts in the drawings, and the description thereof will not be repeated.

(Embodiment 1)
FIG. 1 illustrates the configuration of a vehicle control system 10 according to the first embodiment. A vehicle control system 10 is provided in a vehicle (specifically, a four-wheeled motor vehicle). The vehicle can be switched between manual driving, assisted driving, and automatic driving. Manual driving is driving in which the vehicle travels according to the driver's operation (for example, accelerator operation). Assisted driving is driving in which the vehicle travels while assisting the driver's operation. Autonomous driving is driving that runs without driver's operation. The vehicle control system 10 controls the vehicle during assist driving and automatic driving. Specifically, the vehicle control system 10 controls the operation of the vehicle (especially running) by controlling an actuator 11 provided in the vehicle. The vehicle control system 10 is an example of a mobile body control system provided in a mobile body.

In this example, the vehicle control system 10 includes an information acquisition section 20 , a vehicle control device 30 and a notification section 40 . In the following description, the vehicle provided with the vehicle control system 10 is referred to as "own vehicle", and other vehicles existing around the own vehicle are referred to as "other vehicles".

[Actuator]
The actuator 11 includes a drive system actuator, a steering system actuator, a braking system actuator, and the like. Examples of drive system actuators include engines, transmissions, and motors. An example of a braking system actuator is a brake. Steering is an example of a steering system actuator.

[Information Acquisition Unit]
The information acquisition unit 20 acquires various types of information used for vehicle control. In this example, the information acquisition unit 20 includes a plurality of cameras 21, a plurality of radars 22, a position sensor 23, a communication unit 24, a vehicle state sensor 25, a driving operation sensor 26, and a driver state sensor 27. including.

<camera>
A plurality of cameras 21 have the same configuration as each other. The plurality of cameras 21 are provided on the vehicle so that the imaging areas of the plurality of cameras 21 surround the vehicle. The plurality of cameras 21 acquires image data representing the external environment by capturing images of the environment (external environment) surrounding the vehicle. Image data obtained by each of the cameras 21 is transmitted to the vehicle control device 30 .

In this example, camera 21 is a monocular camera with a wide-angle lens. For example, the camera 21 is configured using a solid-state imaging device such as a CCD (Charge Coupled Device) or a CMOS (Complementary metal-oxide-semiconductor). Note that the camera 21 may be a monocular camera having a narrow-angle lens, or a stereo camera having a wide-angle lens or a narrow-angle lens.

<Radar>
A plurality of radars 22 have the same configuration as each other. A plurality of radars 22 are provided in the vehicle so that search areas of the plurality of radars 22 surround the vehicle. A plurality of radars 22 detect the external environment. Specifically, the radar 22 detects the external environment by transmitting a search wave toward the external environment of the vehicle and receiving a reflected wave from the external environment. Detection results of the multiple radars 22 are transmitted to the vehicle control device 30 .

For example, the radar 22 may be a millimeter wave radar that transmits millimeter waves, a lidar (Light Detection and Ranging) that transmits laser light, or an infrared radar that transmits infrared rays. Alternatively, it may be an ultrasonic sensor that transmits ultrasonic waves.

<Position sensor>
The position sensor 23 detects the position (eg latitude and longitude) of the vehicle. For example, the position sensor 23 receives GPS information from the global positioning system and detects the position of the vehicle based on the GPS information. Information (vehicle position) obtained by the position sensor 23 is transmitted to the vehicle control device 30 .

<Communication part>
The communication unit 24 receives information through an external network (such as the Internet) provided outside the vehicle. For example, the communication unit 24 receives communication information from other vehicles (not shown) located around the vehicle, car navigation data from a navigation system (not shown), traffic information, and high-precision map information such as a dynamic map. do. Information obtained by the communication unit 24 is transmitted to the vehicle control device 30 .

<Vehicle status sensor>
A vehicle state sensor 25 detects the state of the vehicle (for example, speed, acceleration, yaw rate, etc.). For example, the vehicle state sensor 25 includes a vehicle speed sensor that detects the speed of the vehicle, an acceleration sensor that detects the acceleration of the vehicle, a yaw rate sensor that detects the yaw rate of the vehicle, and the like. Information (state of the vehicle) obtained by the vehicle state sensor 25 is transmitted to the vehicle control device 30 .

<Driving operation sensor>
The driving operation sensor 26 detects driving operations applied to the vehicle. For example, the driving operation sensor 26 includes an accelerator opening sensor, a steering angle sensor, a brake oil pressure sensor, and the like. The accelerator opening sensor detects the operation amount of the accelerator of the vehicle. The steering angle sensor detects the steering angle of the steering wheel of the vehicle. The brake oil pressure sensor detects the operation amount of the brake of the vehicle. Information (driving operation of the vehicle) obtained by the driving operation sensor 26 is transmitted to the vehicle control device 30 .

<Driver status sensor>
The driver state sensor 27 detects the state of the driver riding in the vehicle (for example, the driver's physical behavior, biological information, etc.). Information (state of the driver) obtained by the driver state sensor 27 is transmitted to the vehicle control device 30 . In this example, the driver state sensor 27 includes an in-vehicle camera 28 and a biological information sensor 29 .

《In-vehicle camera》
The in-vehicle camera 28 is provided inside the vehicle. The in-vehicle camera 28 acquires image data including the driver by capturing an area including the driver. Image data obtained by the in-vehicle camera 28 is transmitted to the vehicle control device 30 . In this example, the in-vehicle camera 28 is arranged in front of the driver, and the imaging range is set so that the driver's face (especially eyeballs) is positioned within the imaging range. Note that the in-vehicle camera 28 may be provided in goggles (not shown) worn by the driver.

《Biological information sensor》
The biological information sensor 29 is provided inside the vehicle. The biological information sensor 29 detects biological information of the driver. Examples of the driver's biological information include perspiration, heartbeat, blood flow, skin temperature, and the like. Information (driver's biological information) obtained by the biological information sensor 29 is transmitted to the vehicle control device 30 . For example, the biometric information sensor 29 may be arranged at a location that comes into contact with the driver's hand, or may be provided on a member (not shown) attached to the driver's body.

[Vehicle control device]
The vehicle control device 30 is connected to the actuator 11 and each part of the vehicle control system 10 (in this example, the information acquisition part 20, the notification part 40, etc.) so that signal transmission is possible. The vehicle control device 30 controls the actuator 11 and each part of the vehicle control system 10 based on the information obtained by each part of the vehicle control system 10 . Specifically, in assisted driving or automatic driving, the vehicle control device 30 determines a target route on which the vehicle should travel based on various information acquired by the information acquisition unit 20, and travels along the target route. Determine the target motion, which is the motion of the vehicle required to Then, the vehicle control device 30 controls the operation of the actuator 11 so that the motion of the vehicle becomes the target motion. Note that the vehicle control device 30 is an example of a state estimation device.

For example, the vehicle control device 30 is composed of one or more electronic control units (ECUs). The electronic control unit may be composed of a single IC (Integrated Circuit), or may be composed of a plurality of ICs. Also, within an IC, a single core or die may be provided, or multiple cores or dies may be provided in conjunction. The core or die may be composed of, for example, a CPU (processor) and a memory that stores information such as programs for operating the CPU and results of processing by the CPU.

In this example, the vehicle control device 30 has a vehicle behavior recognition section 31 , a driving operation recognition section 32 , an external environment recognition section 33 , a driver state recognition section 34 and a vehicle control section 35 . The vehicle control unit 35 is an example of a control unit that operates according to the state of the driver (subject).

<Vehicle behavior recognition unit>
The vehicle behavior recognition unit 31 estimates vehicle behavior (for example, speed, acceleration, yaw rate, etc.) based on the output of the vehicle state sensor 25 . For example, the vehicle behavior recognition unit 31 uses a learning model generated by deep learning to generate data indicating vehicle behavior from the output of the vehicle state sensor 25 .

<Driving operation recognition part>
The driving operation recognition unit 32 recognizes the driving operation applied to the vehicle based on the output of the driving operation sensor 26 . For example, the driving operation recognition unit 32 uses a learning model generated by deep learning to generate data representing the driving operation applied to the vehicle from the output of the driving operation sensor 26 .

<External environment recognition unit>
The external environment recognition unit 33 recognizes the vehicle behavior based on the outputs of the plurality of cameras 21, the outputs of the plurality of radars 22, the output of the position sensor 23, the output of the communication unit 24, and the output of the vehicle behavior recognition unit 31. Recognize the external environment.

For example, the external environment recognition unit 33 uses a learning model generated by deep learning to generate data representing the external environment of the vehicle from the above outputs. In deep learning, a multilayer neural network (Deep Neural Network) is used. Examples of multi-layer neural networks include CNNs (Convolutional Neural Networks).

Specifically, the external environment recognition unit 33 performs image processing on the image data obtained by the plurality of cameras 21 to generate road map data (for example, three-dimensional map data) representing roads on which the vehicle can travel. to generate Also, the external environment recognition unit 33 acquires object information, which is information about objects existing around the vehicle, based on the detection results of the plurality of radars 22 . The object information includes the position coordinates of the object, the speed of the object, and the like. Note that the external environment recognition unit 33 may acquire object information based on image data obtained by a plurality of cameras 21 . Then, the external environment recognition unit 33 integrates the road map data and the object information to generate integrated map data (three-dimensional map data) representing the external environment.

Road map data includes information on road shape, road structure, road gradient, lane markings, road markings, and the like. The object information includes static object information and dynamic object information. Static object information is information about a stationary object that does not change over time. The static object information includes information about the shape of the stationary body, the positional coordinates of the stationary body, and the like. Examples of stationary objects include road signs and structures. Examples of structures include traffic lights, medians, center poles, buildings, signboards, railroad crossings, tunnels, railroad tracks, bus stops, and the like. Dynamic object information is information about a moving object that may change over time. The dynamic object information includes information about the shape of the moving object, the position coordinates of the moving object, the speed of the moving object, and the like. Examples of moving objects include other vehicles and pedestrians.

The high-precision map information received by the communication unit 24 may include road map data and object information. In this case, the external environment recognition unit 33 generates integrated map data based on the road map data and object information included in the high-precision map information, and outputs the information acquisition unit 20 such as the plurality of cameras 21 and the plurality of radars 22. may be configured to appropriately correct the integrated map data based on. For example, when an object recognized based on the outputs of the plurality of cameras 21 and the outputs of the plurality of radars 22 is not included in the integrated map data, the external environment recognition unit 33 stores the object information about the object in the integrated map data. may be added to Further, when an object included in the integrated map data is not recognized based on the outputs of the plurality of cameras 21 and the outputs of the plurality of radars 22, the external environment recognition unit 33 deletes the object information related to that object from the integrated map data. may

<Driver state recognition unit>
The driver state recognition unit 34 recognizes the state of the driver (for example, the driver's state of health, emotions, posture, etc.) based on the output of the driver state sensor 27 . For example, the driver state recognition unit 34 uses a learning model generated by deep learning to generate data indicating the state of the driver from the output of the driver state sensor 27 . In this example, the driver state recognizing section 34 has a state estimating section 300 . State estimator 300 will be described later in detail.

<Vehicle control unit>
The vehicle control unit 35 controls the actuator 11 based on the output of the vehicle behavior recognition unit 31, the output of the driving operation recognition unit 32, the output of the external environment recognition unit 33, and the output of the driver state recognition unit 34. . In this example, the vehicle control unit 35 performs travel control and notification control.

《Driving control》
Travel control is performed in assist driving and automatic driving. In travel control, the vehicle control unit 35 controls travel of the vehicle. In this example, the vehicle control unit 35 performs candidate route generation processing, target route determination processing, and motion control processing in the travel control.

In the candidate route generation process, the vehicle control unit 35 generates one or more candidate routes based on the output of the external environment recognition unit 33 . A candidate route is a route on which the vehicle can travel, and is a target route candidate. In this example, the candidate routes generated by the candidate route generation process include safe routes. A safe route is a driving route that leads to a safe area (eg, a road shoulder).

For example, in the candidate route generation process, the vehicle control unit 35 generates travel map data including roads in front of the vehicle in the traveling direction and objects existing on the roads, based on the output (integrated map data) of the external environment recognition unit 33. (two-dimensional map data). Then, the vehicle control unit 35 generates candidate routes using the state lattice method. Specifically, the vehicle control unit 35 sets a grid area made up of a large number of grid points on the road of the travel map data, and sequentially connects the plurality of grid points in the traveling direction of the vehicle, thereby forming a plurality of travels. Set route. Also, the vehicle control unit 35 gives a route cost to each of the plurality of travel routes. For example, as the safety of a vehicle on a certain travel route increases, the route cost assigned to that travel route decreases. Then, the vehicle control unit 35 selects one or a plurality of travel routes as candidate routes from among the plurality of travel routes based on the route costs assigned to each of the plurality of travel routes.

In the target route determination process, the vehicle control unit 35 selects one or more candidate routes generated in the candidate route generation process based on the output of the driving operation recognition unit 32 and the output of the driver state recognition unit 34. Select a candidate route to be the target route from . For example, the vehicle control unit 35 selects a candidate route that the driver feels most comfortable from among the plurality of candidate routes.

In the motion control process, the vehicle control unit 35 determines the target motion based on the candidate route selected as the target route in the target route determination process, and controls the actuator 11 based on the determined target motion. For example, the vehicle control unit 35 derives a target driving force, a target braking force, and a target steering amount, which are the driving force, the braking force, and the steering amount for achieving the target motion. Then, the vehicle control unit 35 sets the drive command value indicating the target drive force, the braking command value indicating the target braking force, and the steering command value indicating the target steering amount to the actuator of the drive system, the actuator of the braking system, and the steering system. and to the actuators, respectively.

《Notification control》
In notification control, the vehicle control unit 35 outputs various information for notifying the driver. In this example, the vehicle control unit 35 outputs various information to the notification unit 40 to notify the driver.

[Notification part]
The notification unit 40 is provided inside the vehicle. Then, the notification unit 40 notifies the driver of the vehicle of various information. In this example, the notification section 40 includes a display section 41 and a speaker 42 . The display unit 41 outputs various information as an image. The speaker 42 outputs various information by voice.

[Explanation of terms]
Next, the terms used below will be explained. In the following description, the terms saccades, episodic abnormal states, impaired attentional states, and normal states are used.

<Saccade>
A saccade is a jumping eye movement in which a person (eg, a driver) intentionally shifts the line of sight. Specifically, a saccade is an eyeball movement that moves the line of sight from a point of gaze at which the line of sight is stagnant for a predetermined time to the next point of gaze. FIG. 2 shows temporal changes in the position (angle) of a person's line of sight. As shown in FIG. 2, a period sandwiched between two adjacent gaze periods is a saccade period. Note that the gaze period is a period during which the line of sight is considered to be stagnant. The saccade amplitude ds is the moving distance of the line of sight during the saccade period.

For example, the movement speed of the line of sight is calculated based on the change in the movement distance of the line of sight, and the state in which the movement speed of the line of sight is less than a predetermined speed threshold (for example, 40 deg / s) is a predetermined stagnation time (for example, 0 0.1 second) may be extracted as a "gazing period". Then, in the line-of-sight movement in the period sandwiched between two adjacent gaze periods, the movement speed is equal to or greater than a speed threshold (eg, 40 deg/s) and the movement distance is a predetermined distance threshold (eg, 3 deg). The line-of-sight movement described above may be extracted as a “saccade”.

<Sudden abnormal state>
A sudden abnormal state is an abnormal state in which a body function suddenly declines. In a sudden abnormal state, it becomes difficult for the driver to continue driving the vehicle. In addition, it is difficult to recover from a sudden abnormal state to a normal state by taking a break or the like. Examples of idiopathic abnormal conditions include epilepsy, heart attack, stroke, and the like.

<Caution function deterioration state>
The reduced attentional function state is a state in which a person's attentional function is reduced due to a factor different from the sudden abnormal state described above. In the attention function degraded state, the driver's operation of the vehicle is hindered. In addition, it is possible to restore the state of impaired attention function to a normal state by taking a break or the like. Examples of states of reduced attention function include decreased alertness, fatigue, and listlessness.

<Normal state>
The normal state is neither the above-described sudden abnormal state nor reduced attention function state, and is a state in which a person can normally perform an operation that should be performed. In the normal state, the driver operates the vehicle normally.

[Configuration of State Estimation Unit]
FIG. 3 illustrates the configuration of the state estimator 300. As shown in FIG. State estimation section 300 has saccade detection section 301 , estimation section 302 , attention level detection section 303 , and setting section 304 .

[Saccade detector]
The saccade detection unit 301 detects saccades of the driver (subject) of the vehicle. In this example, the saccade detection unit 301 detects the line of sight of the driver by performing line-of-sight detection processing on image data obtained by the in-vehicle camera 28 . Note that this line-of-sight detection processing may be processing performed using a learning model (learning model for detecting line-of-sight) generated by deep learning, or processing performed using a known line-of-sight detection technique. may be Further, the line of sight of the driver may be the line of sight of the driver's right eye, the line of sight of the driver's left eye, or the line of sight of the driver's right eye and the line of sight of the left eye. It may be a line of sight derived based on. Then, the saccade detection unit 301 detects saccades of the driver based on the movement of the driver's line of sight.

The saccade detector 301 also calculates the amplitude ds and frequency fs of saccades within a predetermined first period P1 and the amplitude ds and amplitude fs of saccades within a predetermined second period P2.

[Saccade detection processing]
Next, the saccade processing performed in the saccade detector 301 will be described with reference to FIGS. 4 and 5. FIG. First, as shown in FIG. 4, the saccade detector 301 sets a first period P1 and a second period P2. Specifically, the saccade detector 301 sets the first period P1 and the second period P2 such that the first period P1 and the second period P2 are shifted by a predetermined time. Note that the second period P2 is shorter than the first period P1. In the example of FIG. 4, the length of the first period P1 is "30 seconds", the length of the second period P2 is "10 seconds", and the predetermined time is "5 seconds". The end of the second period P2 coincides with the end of the first period P1.

Then, the saccade detection section 301 performs the processing (steps ST11 to ST14) shown in FIG. 5 each time the first period P1 and the second period P2 are set.

<Step ST11>
First, the saccade detector 301 detects the line of sight of the driver (subject). In this example, the saccade detection unit 301 detects the line of sight of the driver by performing line-of-sight detection processing on image data obtained by the in-vehicle camera 28 . For example, the saccade detection unit 301 detects the pupils of the driver in the image (image data) obtained by the in-vehicle camera 28, and detects the line of sight of the driver based on the detected pupils. Next, the saccade detector 301 calculates the moving distance of the line of sight of the driver. Then, the saccade detection unit 301 calculates the speed of the driver's line of sight based on the temporal change in the moving distance of the driver's line of sight. For example, the saccade detection unit 301 calculates the speed of the line of sight of the driver by differentiating the moving distance of the line of sight, which changes over time.

<Step ST12>
Next, the saccade detection unit 301 extracts saccade candidates, which are saccade candidates, based on the movement speed of the line of sight. For example, the saccade detection unit 301 defines a period in which the line-of-sight movement speed is less than a predetermined speed threshold value (for example, 40 deg/s) for a predetermined stagnant time period (for example, 0.1 seconds) as a “gazing period. ” (see FIG. 2). Then, the saccade detection unit 301 detects that the movement speed of the line-of-sight movement in the period sandwiched between the two adjacent gaze periods is equal to or greater than the speed threshold value (for example, 40 deg/s) and that the movement distance is predetermined. A line-of-sight movement that is greater than or equal to a distance threshold (for example, 3 degrees) is extracted as a "saccade candidate."

<Step ST13>
Next, the saccade detector 301 reads a saccade range R10 (see FIG. 6) based on the regression curve L10.

For example, the saccade range R10 is derived as follows. First, the saccade detector 301 derives a regression curve L10 based on a plurality of saccade candidates. Specifically, the saccade detection unit 301 derives a regression curve L10 from a plurality of saccade candidates by the method of least squares. Next, the saccade detection unit 301 derives the first reference curve L11 by shifting the regression curve L10 by a predetermined amount in the direction in which the moving speed increases (increase direction on the vertical axis in FIG. 6), and shifts the regression curve L10 to A second reference curve L12 is derived by shifting a predetermined amount in the direction in which the moving speed decreases (decreasing direction on the vertical axis in FIG. 6). Then, the saccade detection unit 301 defines a saccade range R10 between the first reference curve L11 and the second reference curve L12. The derivation of the saccade range R10 may be performed periodically.

Then, saccade detection section 301 compares each of the plurality of saccade candidates with saccade range R10 to extract a saccade. Specifically, the saccade detection unit 301 extracts, as a saccade, a saccade candidate included in the saccade range R10 among the plurality of saccade candidates. Note that the saccade detection unit 301 does not extract, as a saccade, a saccade candidate that is not included in the saccade range R10 among the plurality of saccade candidates.

<Step S14>
Next, the saccade detector 301 calculates the amplitude ds and the frequency fs of saccades within the first period P1. Specifically, the saccade detection unit 301 calculates the average value of the amplitudes ds of the saccades included in the first period P1 as "amplitude ds of the saccades included in the first period P1". A value obtained by dividing the number of saccades in the first period P1 by the time of the first period P1 is calculated as "frequency fs of saccades in the first period P1".

The saccade detection unit 301 also calculates the amplitude ds and the frequency fs of saccades within the second period P2. Specifically, the saccade detection unit 301 calculates the average value of the amplitudes ds of the saccades included in the second period P2 as the "amplitude ds of the saccades included in the second period P2". A value obtained by dividing the number of saccades in the second period P1 by the time of the second period P1 is calculated as "frequency fs of saccades in the second period P2".

[Estimation part]
The estimation unit 302 estimates the state of the vehicle driver (subject). In this example, the estimation unit 302 estimates the state of the driver based on the driver's saccades detected by the saccade detection unit 301 . The operation of estimation section 302 will be described later in detail.

[Attention detection unit]
The caution level detection unit 303 detects the level of caution in the external environment of the vehicle. The degree of caution is an index that indicates the number of caution points (attention points that the driver of the vehicle should check while driving) in the external environment of the vehicle. As the number of places to be careful in the vehicle's external environment increases, the degree of caution in the vehicle's external environment increases. The detection result from the attention detection unit 303 is supplied to the setting unit 304 . Examples of caution points include a point where a moving object is expected to jump out and a point where there is an object that can be an obstacle to the vehicle.

For example, the caution level detection unit 303 may detect the level of caution in the external environment of the vehicle as follows. First, the attention detection unit 303 receives information (high-precision map information) obtained by the communication unit 24 and detects a high-precision area from the high-precision map information. A high caution area is an area where there are relatively many caution points that the driver of the vehicle should check while driving (for example, an area where the number of caution points is equal to or greater than a predetermined threshold value). Examples of high caution areas include road environments with poor visibility such as T-junctions, and road environments such as intersections where obstacles (for example, other vehicles) may block the vehicle's travel. area, etc. Also, the alertness detection unit 303 inputs information (vehicle position) obtained by the position sensor 23 . Then, when the position of the vehicle is included in the high caution level region, the caution level detection unit 303 detects that the level of caution in the external environment of the vehicle is a high caution level. When not included in the area, it is detected that the degree of caution in the external environment of the vehicle is low.

Alternatively, the caution level detection unit 303 may detect the level of caution in the external environment of the vehicle as follows. First, the caution level detection unit 303 detects caution points from within the external environment of the vehicle recognized by the external environment recognition unit 33 . Then, when there are relatively many caution points included in the vehicle's external environment (for example, when the number of caution points is equal to or greater than a predetermined threshold value), the caution level detection unit 303 detects the degree of caution in the vehicle's external environment. is a high degree of caution, and when the number of caution points included in the vehicle's external environment is relatively small (for example, when the number of caution points is less than a predetermined threshold value), in the vehicle's external environment Detect that attention level is low attention level.

A driving scene in which the degree of caution in the external environment of the vehicle is high is a driving scene in which there are a relatively large number of caution points that have been revealed, and it is required to direct the line of sight to these caution points in a wide range. It can be said. In addition, in a driving scene in which the degree of caution in the external environment of the vehicle is low, there are relatively few caution points that have become obvious, and it is required to frequently move the line of sight to search for potential caution points. It can be said that it is a driving scene that Below, a driving scene with a high level of caution in the vehicle's external environment is referred to as a "danger confirmation scene", and a driving scene with a low level of caution in the vehicle's external environment is referred to as a "danger search scene". Describe.

[Setting part]
The setting unit 304 sets an abnormal condition used for estimation of the state of the driver (subject) in the estimation unit 302 . In this example, the setting unit 304 sets the threshold included in the abnormal condition based on the driver's saccades detected by the saccade detection unit 301 . Note that the operation of the setting unit 304 will be described later in detail.

[Experiments conducted by the inventor of the present application]
The inventors of the present application conducted the following experiments in order to investigate the relationship between the driver's state and the driver's behavior (particularly, movement of the line of sight).

First, patients with symptoms of epilepsy (hereinafter referred to as "epileptic patients") were selected as subjects in order to collect data on an idiopathic abnormal condition. Then, as shown in FIG. 7, a driving simulator 60 was used to allow the subjects to simulate driving operations of a vehicle. Specifically, the driving simulator 60 allows the test subject to view a moving image (a moving image showing the external environment of the vehicle that can be seen from inside the vehicle) while the vehicle is running, and observes the subject's behavior during the experiment. Behavior during vehicle driving was simulated. In this experiment, a camera was installed in front of a subject viewing moving images while the vehicle was running, and the camera was set so that the subject's eyeballs were included in the imaging range.

Then, by performing line-of-sight detection processing on the image data obtained by the camera, the subject's line of sight was detected. Further, by performing saccade detection processing on the subject's line of sight obtained by the line-of-sight detection processing, the saccade of the subject was detected, and the amplitude and frequency of the saccade within a predetermined period were calculated. These processes are the same as the processes performed in the saccade detector 301 (see FIG. 5).

In the experiment described above, the test subject's condition was classified into a normal condition and a reduced attention function condition, and the vehicle driving scenes shown in the moving images reproduced by the driving simulator 60 were classified into a danger seeking scene and a danger confirming scene. carried out. Specifically, by having the subject perform a task (mental arithmetic) that is different from the driving operation during the experiment using the driving simulator 60, the subject's attention function deterioration state is simulated, and driving during the experiment using the driving simulator 60 is performed. By preventing the subject from performing a task (mental arithmetic) that is different from the operation, the subject's normal state was simulated. In addition, by causing the driving simulator 60 to reproduce moving images of driving scenes in which monotonous driving operations are possible, danger search scenes can be simulated, and moving images of driving scenes requiring complex driving operations can be reproduced by the driving simulator. 60, the danger confirmation scene was simulated.

The above experiments were performed on multiple subjects. Among these multiple subjects was one who experienced epileptic seizures during the study. Therefore, it was possible to obtain data on the subject's epileptic seizures.

FIG. 8 shows the amplitude of the saccades of the epileptic patient and the amplitude of the saccades of the epileptic patient before and just before the epileptic seizure for each driving scene. FIG. 9 shows the frequency of saccades of epileptic patients and the amplitude of saccades of epileptic patients before and just before an epileptic seizure by driving scene. FIG. 10 shows eye movement during an epileptic seizure of a subject.

In FIGS. 8 and 9, the data (amplitude or frequency) of "highway (monotonic)" and "highway (complex)" are data obtained when the subject's condition is normal, and "caution The data of "decreased" are the data acquired when the subject's state is a state of reduced attention function. "Pre-seizure" data is data acquired before the subject's epileptic seizure appears, and is data acquired during the period from about 3875 seconds to about 3890 seconds in FIG. "Just before seizure" data is data acquired immediately before the subject's epileptic seizure appears, and is data acquired during the period from about 3890 seconds to about 3915 seconds in FIG.

As shown in FIGS. 8 to 10, the period before the epileptic seizure appears (from about 3875 seconds to about 3890 seconds in FIG. 10) to the period immediately before the epileptic seizure appears (from about 3890 seconds to about 3915 seconds in FIG. 10). ), the amplitude and frequency of the subject's saccades decrease. Then, as shown in FIG. 10, the amplitude and frequency of the subject's saccades during the period immediately before the epileptic seizure appeared until the epileptic seizure appeared (the period from about 3915 seconds to about 3930 seconds in FIG. 10) has further decreased to zero. The state in which the amplitude and frequency of the saccades of this subject was zero continued for 10 seconds or more.

As shown in FIGS. 8 and 9, the amplitude and frequency of saccades during attention deficit and immediately before an epileptic seizure are lower than the amplitude and frequency of saccades during normal times. Specifically, the amplitude of the saccades at the time of attentional deterioration and immediately before the epileptic seizure was the amplitude of the saccades (approximately 12 deg ) is below 50%. The frequency of saccades during attention deficit and immediately before epileptic seizures is less than 75% of the normal saccade frequency (about 20/s) in the danger-seeking scene.

[Knowledge obtained by the inventor of the present application]
The inventors of the present application obtained the following findings from the experiments described above.

(1) At least one of the amplitude and frequency of saccades in a subject is reduced when the subject has an epileptic seizure and when the subject is in a state of attentional depression. Specifically, the amplitude and frequency of saccades are lower than normal.

(2) if the subject has an epileptic seizure, after at least one of the amplitude and frequency of the subject's saccades is reduced, both the amplitude and frequency of the subject's saccades are further reduced; Specifically, the state in which the amplitude and frequency of the saccades of the subject is zero continues for 10 seconds or longer.

(3) when the subject is in a state of attention deficit, at least one of the amplitude and frequency of the subject's saccades is reduced, but unlike when the subject has an epileptic seizure, the amplitude and frequency of the subject's saccades are further reduced; does not occur.

In the above, the cases where epileptic seizures appear are described, but other sudden abnormal conditions (specifically, heart attack, stroke, etc.) other than epilepsy are treated in the same way as epileptic seizures. It is assumed that the result

Based on the above findings, the inventors of the present application have found the following.

(1) When the driver's state becomes a sudden abnormal state or a reduced attention function state, at least one of the amplitude and frequency of the driver's saccades becomes lower than normal. Specifically, the amplitude of the driver's saccades in the episodic or impaired attentional state is less than 50% of the amplitude of the driver's saccades in the normal state. The frequency of driver saccades in an episodic or impaired attentional state is less than 75% of the amplitude of driver saccades in normal conditions.

(2) When the driver's condition becomes a sudden abnormal condition, both the amplitude and frequency of the driver's saccades further decrease after at least one of the amplitude and frequency of the driver's saccades decreases compared to normal. Specifically, the state in which the amplitude and frequency of the driver's saccades is zero continues for 10 seconds or more.

(3) When the driver's condition is reduced attention function, at least one of the amplitude and frequency of the driver's saccades is lower than normal. No further reduction in frequency occurs.

[State estimation process]
Next, with reference to FIG. 11, the operation (state estimation processing) of estimation section 302 will be described. The estimating section 302 performs the following steps ST21 to ST24 for each predetermined cycle. For example, each time the saccade detection unit 301 calculates the amplitude ds and frequency fs of saccades in the first period P1 and the amplitude ds and frequency fs of saccades in the second period P2, the estimation unit 302 performs the following processing. conduct.

<Step ST21>
The estimation unit 302 determines whether or not the amplitude ds of the saccade within the first period P1 is equal to or less than a predetermined first amplitude threshold dth1. The estimation unit 302 also determines whether or not the frequency fs of saccades within the first period P1 is equal to or less than a predetermined first frequency threshold fth1.

For example, the first amplitude threshold dth1 is set to the amplitude ds of the saccade within the first period P1 when the driver's condition can be considered to be a sudden abnormal condition or a reduced attention function condition. Specifically, the first amplitude threshold dth1 may be set to 50% or less of the amplitude ds of the saccade within the first period P1 when the driver's condition is normal. The first frequency threshold fth1 is set to the frequency fs of saccades within the first period P1 when the driver's condition can be considered to be a sudden abnormal condition or a reduced attention function condition. Specifically, the first amplitude threshold fth1 may be set to 75% or less of the frequency fs of saccades during the first period P1 when the driver's condition is normal. Note that the length of the first period P1 is the time (30 seconds in this example) during which it is possible to observe the decrease in the amplitude and frequency of saccades in the sudden abnormal state or the state of reduced attention function. The length of the second period P2 is the time (in this example, 10 seconds) during which further reduction (disappearance) of the amplitude and frequency of saccades in the sudden abnormal state can be observed.

The first condition that "the amplitude ds of saccades within the first period P1 is equal to or less than the first amplitude threshold dth1" and the second condition that "the frequency fs of saccades within the first period P1 is equal to or less than the first frequency threshold fth1" If at least one of the conditions is satisfied, the process of step ST22 is performed, and if both the first condition and the second condition are not satisfied, the process of step ST24 is performed.

<Step ST22>
The estimation unit 302 determines whether or not the amplitude ds of the saccade within the second period P2 is equal to or less than the second amplitude threshold dth2. The estimation unit 302 also determines whether or not the frequency fs of saccades within the second period P2 is equal to or less than the second frequency threshold fth2.

For example, the second amplitude threshold dth2 is set to the amplitude ds of the saccade within the second period P2 when the driver's condition can be regarded as a sudden abnormal condition. Specifically, the second amplitude threshold dth2 may be set to zero. The second frequency threshold fth2 is set to the frequency fs of saccades within the second period P2 when the driver's condition can be regarded as a sudden abnormal condition. Specifically, the second frequency threshold fth2 may be set to zero.

A third condition that "the amplitude ds of saccades within the second period P2 is less than or equal to the second amplitude threshold dth2" and a fourth condition that "the frequency fs of saccades within the second period P2 is less than or equal to the second frequency threshold fth1" If both conditions are satisfied, the process of step ST22 is performed, and if at least one of the third condition and the fourth condition is not satisfied, the process of step ST24 is performed.

<Step ST23>
After at least one of the first condition and the second condition is satisfied in step ST21, if both the third condition and the fourth condition are satisfied in step ST22, estimating section 302 determines that the driver's condition is a sudden abnormal state. Assume there is. In this example, the estimation unit 302 raises a flag indicating a sudden abnormal condition.

<Step ST23>
If both the first condition and the second condition are not satisfied in step ST21, or if at least one of the third condition and the fourth condition is not satisfied in step ST22, estimating section 302 determines that the driver's condition is a sudden abnormality. Do not presume to be a state. In this example, the estimation unit 302 does not raise a flag indicating a sudden abnormal state.

[Effect of Embodiment 1]
As described above, based on whether or not both the third condition and the fourth condition are satisfied after at least one of the first condition and the second condition is satisfied, a sudden abnormal state in which the physical function suddenly declines is estimated. can do.

(Modified example 1 of state estimation processing)
Note that the estimation unit 302 may be configured to perform the state estimation process shown in FIG. 12 . In the state estimation process shown in FIG. 12, the processes of steps ST31 and ST32 are performed instead of the process of step S24 shown in FIG.

<Step ST31>
If at least one of the third condition and the fourth condition is not satisfied in step ST22 after at least one of the first condition and the second condition is satisfied in step ST21, estimating section 302 determines that the state of the driver is a reduced attention function state. We assume that In this example, the estimating unit 302 raises a flag indicating a state of attentional deterioration.

<Step ST32>
When both the first condition and the second condition are not satisfied in step ST21, estimation section 302 estimates that the driver's condition is normal. For example, the estimation unit 302 raises a flag indicating the normal state.

〔effect〕
As described above, after at least one of the first condition and the second condition is satisfied, based on whether or not the third condition and the fourth condition are satisfied, the sudden abnormal state and the reduced attention function state are distinguished and estimated. can be done.

(Modified example 2 of state estimation processing)
Also, the estimation unit 302 may be configured to perform the state estimation process shown in FIG. 13 . In the state estimation process shown in FIG. 13, in addition to the process shown in FIG. 12, the processes of steps ST41 and ST42 are performed.

<Step ST41>
When at least one of the first condition and the second condition is satisfied in step ST21, the estimating section 302 measures the first duration T1, which is the duration of the state in which at least one of the first condition and the second condition is satisfied. Then, the estimation unit 302 determines whether or not the first duration T1 exceeds a predetermined first time threshold Tth1. If the first duration T1 exceeds the first time threshold Tth1, the process of step ST22 is performed, otherwise the process of step ST21 is performed.

<Step ST42>
When both the third condition and the fourth condition are satisfied in step ST22, the estimating section 302 measures the second duration T2, which is the duration of the state in which both the third condition and the fourth condition are satisfied. Then, if the second duration T2 exceeds a predetermined second time threshold Tth2, the estimating section 302 performs the process of step ST23, and otherwise performs the process of step ST21.

〔effect〕
As described above, after at least one of the first condition and the second condition is satisfied, based on whether or not the third condition and the fourth condition are satisfied, the sudden abnormal state and the reduced attention function state are distinguished and estimated. can be done.

In addition, by estimating a sudden abnormal state and a state of reduced attention function based on the duration of a state in which at least one of the first condition and the second condition is satisfied (first duration T1), sudden noise, etc. It is possible to reduce the estimation error due to As a result, it is possible to improve the accuracy of estimating a sudden abnormal state and a state of reduced attention function.

In addition, by estimating a sudden abnormal state based on the duration of the state where both the third condition and the fourth condition are satisfied (second duration T2), estimation errors due to sudden noise or the like are reduced. be able to. As a result, it is possible to improve the accuracy of estimating a sudden abnormal state.

(Operation of vehicle control section)
When the estimating unit 302 estimates that the driver is in a sudden abnormal state, the vehicle control unit 35 performs a first action corresponding to the sudden abnormal state. In this example, the vehicle control unit 35 performs the first action when the flag indicating the sudden abnormal state is set. Examples of the first operation include an operation of controlling the running of the vehicle so that the vehicle stops in a safe area, and an operation of outputting first notification information for notifying that the driver is in a sudden abnormal state. , an operation for notifying the surroundings of the vehicle that the driver's condition is a sudden abnormal condition, and the like. Examples of the operation of outputting the first notification information include an operation of outputting the first notification information to the notification unit 40, and an operation of outputting the first notification information to an information terminal (not shown) outside the vehicle via the communication unit 24. etc. An example of an operation for notifying the surroundings of the vehicle that the driver's condition is a sudden abnormal condition is an operation for blinking a hazard lamp (not shown) of the vehicle.

When the estimation unit 302 estimates that the state of the driver is in the state of reduced attention function, the vehicle control unit 35 performs a second operation according to the state of reduced attention function. In this example, the vehicle control unit 35 performs the second operation when the flag indicating the attention function deterioration state is set. Examples of the second operation include an operation for resolving the driver's reduced attention function state, an operation for outputting second notification information for notifying that the driver is in a reduced attention function state, and the like. be done. Examples of actions for resolving the driver's attention function deterioration state include the action of outputting warning information to encourage the driver to take a break, and the action to encourage the driver to concentrate on driving the vehicle. For example, an operation of outputting alert information may be mentioned. The notification unit 40 receives warning information and/or attention information output from the vehicle control unit 35 . As a result, the notification unit 40 outputs the warning information and/or the alert information in the form of an image and/or sound inside the vehicle. Examples of the operation of outputting the second notification information include an operation of outputting the second notification information to the notification unit 40, and an operation of outputting the second notification information to an information terminal (not shown) outside the vehicle via the communication unit 24. etc.

[First action]
Next, an example of the first operation will be described with reference to FIG. For example, the vehicle control unit 35 performs the processing of steps ST101 to ST103 below when a flag indicating a sudden abnormal state is set.

<Step ST101>
The vehicle control unit 35 performs evacuation travel control. In evacuation control, the vehicle control unit 35 sets a safe area in the traveling map data including the road in front of the vehicle in the direction of travel and objects existing on the road, and determines a traveling route (evacuation route) whose target position is the safe area. route). Then, the vehicle control unit 35 controls the traveling of the vehicle so that the vehicle travels along the evacuation route and stops in the safe area.

<Step ST102>
The vehicle control unit 35 determines whether or not the vehicle has stopped in the safe area. If the vehicle is stopped in the safe area, the process of step ST103 is performed, and if not, the process of step ST101 is continued.

<Step ST103>
When the vehicle stops in the safe area, the vehicle control unit 35 outputs to the communication unit 24 first notification information indicating that the driver is in a sudden abnormal state. The communication unit 24 transmits the first notification information to an information terminal (not shown) of the medical institution. This first notification information may include position information indicating the position of the vehicle stopped in the safe area.

[Second action]
Next, an example of the second operation will be described with reference to FIG. 15 . For example, the vehicle control unit 35 performs the processing of steps ST201 to ST203 below when the flag indicating the attention function deterioration state is set.

<Step ST201>
The vehicle control unit 35 outputs alarm information to the notification unit 40 to prompt the driver to stop the vehicle and take a rest. The notification unit 40 outputs warning information in the form of images and/or sounds. Specifically, the display unit 41 displays warning information. The speaker 42 reproduces the warning information by voice. Thereby, the driver performs a driving operation for stopping the vehicle.

<Step ST202>
The vehicle control unit 35 determines whether or not the vehicle has stopped. If the vehicle is stopped, the process of step ST203 is performed, and if not, the process of step ST201 is continued.

<Step ST203>
When the vehicle stops, the vehicle control unit 35 controls the notification unit 40 so that the output of the warning information by the notification unit 40 is stopped.

〔effect〕
As described above, the operation corresponding to the state of the driver estimated by the estimation unit 302 can be performed appropriately. Specifically, when the driver's condition is estimated to be in a sudden abnormal condition, the vehicle can be stopped in the safe area. As a result, the safety of the own vehicle (the vehicle driven by the driver) and other vehicles around the own vehicle can be ensured. Further, when the driver's state is estimated to be in a state of reduced attention function, it is possible to prompt the driver to take a rest. As a result, the condition of the driver can be recovered from the attention function deterioration condition to the normal condition, and the running safety of the vehicle can be ensured.

(Operation of setting section)
The setting unit 304 performs threshold setting processing. In the threshold setting process, an abnormal condition used for estimating the state of the driver (subject) in the estimating unit 302 is set. In this example, a first amplitude threshold dth1 and a first frequency threshold fth1 are set. Specifically, the first amplitude threshold dth1 and the first frequency threshold fth1 are set based on the amplitude ds and the frequency fs of saccades when the driving scene of the vehicle is a predetermined driving scene.

[Threshold setting process]
Next, the threshold setting process will be described with reference to FIG. Setting section 304 performs the processing of steps ST301 to ST303 below at predetermined intervals. Note that the setting unit 304 may be configured to perform the following processing in response to an instruction from the driver.

<Step ST301>
The setting unit 304 determines whether or not the driving scene of the vehicle corresponds to a predetermined driving scene. In this example, the predetermined driving scene is a danger exploration scene. The setting unit 304 determines whether or not the driving scene of the vehicle corresponds to the danger search scene based on the degree of caution in the external environment of the vehicle detected by the caution level detection unit 303 . When the driving scene of the vehicle corresponds to a predetermined driving scene, the process of step ST302 is performed.

<Step ST302>
The setting unit 304 collects the amplitude ds and frequency fs of saccades detected by the saccade detection unit 301 . The amplitude ds and frequency fs of saccades collected in step ST302 are preferably the amplitude ds and frequency fs of saccades when the driver can be considered to be in a normal state.

<Step ST303>
Next, the setting unit 304 sets the first amplitude threshold dth1 based on the saccade amplitude ds. The setting unit 304 also sets the first frequency threshold fth1 based on the saccade frequency fs. For example, setting section 304 sets the first amplitude threshold dth1 to “50% of the average value of the amplitudes ds of the saccades collected in step ST302”, and sets the first frequency threshold fth1 to “the average value of the saccade amplitudes ds collected in step ST302. 75% of the average value of the frequency fs”.

〔effect〕
As described above, the first amplitude threshold dth1 and the first frequency threshold fth1 can be appropriately set in consideration of the characteristics of the driver. As a result, it is possible to improve the accuracy of estimating the state of the driver.

(Embodiment 2)
17 and 18 illustrate the appearance and configuration of the display device 100 of the second embodiment. In this example, the display device 100 constitutes a smart phone. Specifically, the display device 100 includes a housing 100 a , a display section 110 , a speaker 111 , a storage section 112 , an information acquisition section 120 and a control device 130 .

[Chassis]
The housing 100a is formed in the shape of a flat rectangular parallelepiped box. Components of the display device 100 are housed in the housing 100a.

[Display, speaker, and memory]
Display unit 110 displays an image. As shown in FIG. 17, the display unit 110 is formed in a rectangular shape and provided on the front surface of the housing 100a. The speaker 111 reproduces sound. The storage unit 112 stores various information and data.

[Information Acquisition Unit]
The information acquisition unit 120 acquires various information used in the display device 100 . In this example, information acquisition unit 120 includes front camera 121, rear camera 122, operation unit 123, communication unit 124, microphone 125, position sensor 126, state sensor 127, and environment sensor 128. .

<Front camera and rear camera>
The front camera 121 is provided on the front surface of the housing 100a, and the rear camera 122 is provided on the rear surface of the housing 100a. The front camera 121 acquires image data indicating the front area of the display device 100 by capturing an image of the area (front area) extending in front of the display device 100 . The rear camera 122 acquires image data indicating the rear area of the display device 100 by capturing an image of the area (rear area) extending behind the display device 100 . Image data obtained by each of front camera 121 and rear camera 122 is transmitted to control device 130 . For example, the front camera 121 and the rear camera 122 are configured using a solid-state imaging device such as a CCD (Charge Coupled Device) or CMOS (Complementary metal-oxide-semiconductor).

<Operation unit>
The operation unit 123 is operated by the user of the display device. The operation unit 123 outputs a signal according to the operation given by the user. With such a configuration, the user can operate the operation unit 123 to input information. The output of the operation unit 123 is transmitted to the control device 130 . In this example, operation unit 123 includes a touch panel operation unit that forms a touch panel together with display unit 110 . Note that the operation unit 123 may include an operation dial, an operation button, etc. in addition to the touch panel operation unit.

<Communication part>
The communication unit 124 receives information and data through a communication network provided outside the display device 100 (for example, the Internet, a mobile phone line, etc.). For example, the communication unit 124 receives audio data, image data, map information, etc. from another information terminal (not shown). Information and data obtained by communication unit 124 are transmitted to control device 130 .

<microphone>
Microphone 125 converts sound into electrical signals. An electrical signal (audio data) obtained by the microphone 125 is transmitted to the control device 130 .

<Position sensor>
Position sensor 126 detects the position (eg, latitude and longitude) of display device 100 . For example, the position sensor 126 receives GPS information from the global positioning system and detects the position of the display device 100 based on the GPS information. Information (the position of the display device 100 ) obtained by the position sensor 126 is transmitted to the control device 130 .

<Status sensor>
The state sensor 127 detects the state of the display device 100 (eg, acceleration, angular velocity, attitude, etc.). For example, state sensor 127 includes an acceleration sensor, a gyro sensor, and the like. Information (the state of the display device 100 ) obtained by the state sensor 127 is transmitted to the control device 130 .

<Environment sensor>
The environment sensor 128 detects information about the environment surrounding the display device 100 (for example, light, magnetism, etc.). For example, environmental sensors 128 include optical sensors, magnetic sensors, proximity sensors, and the like. Information obtained by the environment sensor 128 (information about the surrounding environment of the display device 100 ) is transmitted to the control device 130 .

〔Control device〕
The control device 130 is connected to each part of the display device 100 (in this example, the display part 110, the speaker 111, the storage part 112, and the information acquisition part 120) so that signal transmission is possible. Then, the control device 130 controls each part of the display device 100 based on the information obtained by each part of the display device 100 . Note that the control device 130 is an example of a state estimation device.

For example, the control device 130 is composed of one or more ICs (Integrated Circuits). A single core or die may be provided within an IC, or multiple cores or dies may be provided in conjunction. The core or die may be composed of, for example, a CPU (processor) and a memory that stores information such as programs for operating the CPU and results of processing by the CPU.

In this example, controller 130 includes controller 131 and state estimator 300 . The control unit 131 is an example of a control unit that operates according to the state of the user (subject).

<Control unit>
The control unit 131 controls each unit of the display device 100 based on information obtained by each unit of the display device 100 . For example, the control unit 131 stores information and data obtained by the information acquisition unit 120 in the storage unit 112 .

In this example, the control unit 131 performs display control, call control, and the like. In display control, the control unit 131 outputs image data to the display unit 110 . Display unit 110 reproduces the image indicated by the image data. In call control, the control unit 131 outputs audio data obtained by the communication unit 124 to the speaker 111 . Speaker 111 reproduces the sound indicated by the sound data. Also, the control unit 131 transmits voice data obtained by the microphone 125 to the communication network via the communication unit 124 .

<State estimation unit>
The configuration of the state estimator 300 in the second embodiment is the same as the configuration of the state estimator 300 in the first embodiment. The state estimation unit 300 estimates the state of the user (subject).

In the second embodiment, the front camera 121 can acquire image data including the face (especially eyeballs) of the user of the display device 100 . The saccade detection unit 301 detects the user's line of sight by performing line-of-sight detection processing on the image data obtained by the front camera 121 . The estimation unit 302 estimates the state of the user based on the user's saccades detected by the saccade detection unit 301 . The operation of the estimating unit 302 of the second embodiment is the same as the operation of the estimating unit 302 of the first embodiment (see FIGS. 11 to 13).

Further, in the second embodiment, when the estimating unit 302 estimates that the driver's condition is a sudden abnormal state, the control unit 131 performs the first action according to the sudden abnormal state. Examples of the first operation include an operation of outputting first notification information for notifying that the user's condition is a sudden abnormal condition; and an operation for notifying the surroundings of the device. Examples of the operation of outputting the first notification information include an operation of outputting the first notification information to the display unit 110 and/or the speaker 111, and an operation of outputting the first notification information to another information terminal (not shown) via the communication unit 124. ), and the like. An example of the operation for notifying the surroundings of the display device 100 that the user is in a sudden abnormal state is the operation of blinking a light (not shown) of the display device 100 .

Further, in the second embodiment, when the estimation unit 302 estimates that the state of the driver is in the state of reduced attention function, the control unit 131 performs the second action according to the state of reduced attention function. Examples of the second operation include an operation for resolving the user's attention function deterioration state, an operation for outputting second notification information for notifying that the user is in the attention function deterioration state, and the like. be done. Examples of the operation for resolving the user's attention function deterioration state include an operation of outputting warning information to prompt the user to take a break, and prompting the user to concentrate on operating the display device 100. For example, an operation of outputting alert information for Display unit 110 and/or speaker 111 receives alarm information and alert information output from control unit 131 . As a result, warning information and alert information are output to the user in the form of images and/or sounds. Examples of the operation of outputting the second notification information include an operation of outputting the second notification information to the display unit 110 and/or the speaker 111, and an operation of outputting the second notification information to another information terminal (not shown) via the communication unit 124. ), and the like.

[Effect of Embodiment 2]
The display device 100 of the second embodiment can obtain the same effects as those of the first embodiment. For example, it is possible to estimate a sudden abnormal state in which a body function suddenly declines.

(Other embodiments)
In addition, in the above description, a vehicle is used as an example of a moving object, but the present invention is not limited to this. For example, other examples of moving bodies include ships and airplanes.

Also, in the above description, the case where the display device 100 is a smart phone was taken as an example, but the present invention is not limited to this. For example, the display device 100 may be a television receiver, a game machine, or other device.

Also, the above embodiments may be combined as appropriate. The above embodiments are essentially preferable illustrations, and are not intended to limit the scope of the invention, its applications, or its uses.

As described above, the technology disclosed herein is useful as a state estimation device.

10 vehicle control system 11 actuator 20 information acquisition unit 27 driver state sensor 28 in-vehicle camera (imaging unit)
30 vehicle control device (state estimation device)
35 vehicle control unit (control unit)
300 state estimation unit 301 saccade detection unit 302 estimation unit 303 caution level detection unit 304 setting unit 40 notification unit 41 display unit 42 speaker 100 display device 110 display unit 120 information acquisition unit 121 front camera (imaging unit)
130 control device (state estimation device)
131 control unit

Claims (6)

A state estimation device for estimating the state of a subject who is a driver of a vehicle ,
a saccade detection unit that detects a saccade that is a jumping eye movement that intentionally moves the line of sight of the subject;
while the subject is driving the vehicle, a first condition that the amplitude of the subject's saccades within a first time period is equal to or less than a first amplitude threshold and the frequency of the subject's saccades within the first time period is After at least one of a second condition of being equal to or less than a first frequency threshold is established, the amplitude of the saccade of the subject within a second period shorter than the first period is a second amplitude smaller than the first amplitude threshold When both the third condition that the frequency of saccades of the subject within the second period is equal to or less than the threshold is equal to or less than the second frequency threshold that is smaller than the first frequency threshold, A state estimation device, comprising: an estimation unit for estimating that the subject's state is a sudden abnormal state in which physical function suddenly declines. In claim 1,
If the duration of a state in which both the third condition and the fourth condition are satisfied exceeds a predetermined time threshold after at least one of the first condition and the second condition is satisfied, the estimation unit (2) a state estimating device that estimates that the subject's state is the sudden abnormal state; In claim 1 or 2,
When at least one of the third condition and the fourth condition is not satisfied after at least one of the first condition and the second condition is satisfied, the estimating unit determines that the state of the subject is a reduced attention function state. A state estimation device characterized by estimating that In claim 3,
When the state of the subject is estimated to be the sudden abnormal state, a first action corresponding to the sudden abnormal state is performed, and the state of the subject is estimated to be the reduced attention function state. A state estimating device, comprising: a control unit that performs a second action in accordance with the state of impaired attention function when the state of the attentional function is impaired. In claim 4 ,
The first operation includes an operation of controlling travel of the vehicle so that the vehicle evacuates to a safe area;
The state estimation device, wherein the second action includes an action of outputting information for prompting the driver to take a rest. In any one of claims 1 to 5 ,
The state estimation device, wherein the first amplitude threshold and the first frequency threshold are set based on the amplitude and frequency of the saccade when the vehicle travels in a predetermined travel scene.

Images