WO2020188629A1

WO2020188629A1 - Alertness state determination device, awakening system, and alertness state determination method

Info

Publication number: WO2020188629A1
Application number: PCT/JP2019/010829
Authority: WO
Inventors: あゆみ竹本
Original assignee: オムロン株式会社
Priority date: 2019-03-15
Filing date: 2019-03-15
Publication date: 2020-09-24
Also published as: JPWO2020188629A1

Abstract

An alertness state determination device (100) is provided with: an imaging unit (3) for generating image data by capturing images including an eye of a person; and a control unit (10) for sequentially acquiring the image data from the imaging unit and detecting movement of the eye of the person on the basis of the acquired image data. The control unit (10) measures the frequency of saccades of the eye on the basis of the movement of the eye detected in a determination period, measures a stop period in which the movement of the eye is stopped on the basis of the detected movement of the eye, and determines whether or not the alertness state of the person is a non-alert state in the determination period on the basis of the measured frequency of saccades and stop period.

Description

Awakening state determination device, awakening system, and awakening state determination method

The present disclosure relates to an awakening state determination device, an awakening system, and an awakening state determination method.

The alertness state regarding a person's alertness changes over time. The arousal state includes a normal state, a drowsiness state in which the arousal level is lower than the normal state, and a drowsiness state in which the arousal level is further lower than the drowsiness state. If you fall into a drowsy or drowsy state with reduced alertness, you may reduce the work efficiency of a person or cause a serious accident. Therefore, it is required to determine the awake state of a person, for example, whether or not the awake state of a person is in the awake state.

According to Patent Document 1, when a human being is in a rambling state, the frequency of occurrence of small-amplitude saccades is higher and the frequency of occurrence of saccades in a range including the vertical direction is lower than in a normal state. Findings are disclosed. Based on this finding, Patent Document 1 discloses a rambling state determination device for the purpose of determining the rambling state of a determination target person with stable accuracy by a simple process. This vague state determination device images the driver's face as the determination target person, calculates the eyeball rotation angle from the obtained image, and determines whether or not saccade is generated from the magnitude of the vector of the eyeball rotation speed. .. Further, the rambling state determination device calculates the amplitude and direction of the detected saccade, and the ratio of the saccade whose calculated amplitude is equal to or less than a predetermined value and the ratio of the saccade within the range in which the calculated direction is classified as the vertical direction. The saccade state is determined by the threshold determination such as.

Japanese Unexamined Patent Publication No. 2011-115450

In Patent Document 1, various arithmetic processes are performed in order to detect minute changes such as a small amplitude or a frequency of occurrence of saccades in the vertical direction from the captured image of the driver and perform threshold determination. However, when a driver drives a vehicle or the like, there are various factors that reduce the accuracy of imaging, and it is easy to implement an imaging means or a calculation means capable of detecting minute changes as described above. Difficult to do.

An object of the present invention is to provide an awakening state determination device capable of easily determining a vague state in a person's awakening state.

One aspect of the present disclosure provides an awakening state determining device for determining a dumb state in a person's awakening state. The awakening state determination device sequentially acquires image data from an imaging unit that captures an image including the human eye and generates image data, and sequentially acquires image data from the imaging unit, and moves the human eye based on the acquired image data. It is provided with a control unit for detecting. The control unit measures the frequency of saccades in which the eyes generate saccades based on the eye movements detected during the period to be determined, and measures the saccade period in which the eye movements stay based on the detected eye movements. Then, based on the measured saccade frequency and the period of stay, it is determined whether or not the awakening state of the person is in a vague state during the period to be determined.

Another aspect of the present disclosure provides an awakening system including the awakening state determination device of the above aspect and an output unit that executes a predetermined operation based on the determination result of the awakening state determination device.

Yet another aspect of the present disclosure provides a method for determining a wakefulness state for determining a rambling state in a person's wakefulness state. The saccade state determination method includes a step in which the control unit sequentially acquires image data generated by capturing an image including the human eye, and a control unit determines the movement of the human eye based on the acquired image data. Based on the step to detect, the step to measure the frequency of saccades in which the eyes generate saccades, based on the eye movements detected by the control unit during the period to be determined, and the steps to measure the frequency of saccades detected by the control unit. Based on the step of measuring the dwell period in which the eye movement is stagnant and the measured frequency of occurrence of saccade and the dwell period, whether or not the awakening state of the person is in a vague state during the period to be determined. Includes a step to determine.

According to the awakening state determination device, the awakening system, and the awakening state determination method according to the present disclosure, it is possible to determine the awakening state of a person regarding the arousal level by a simple process.

The figure for demonstrating the application example of the awakening state determination apparatus which concerns on this disclosure. The figure which illustrates the structure of the awakening system which concerns on embodiment of this disclosure. Flow chart exemplifying the operation flow of the awakening system A flowchart illustrating the detailed flow of the initial processing step of FIG. Diagram for explaining an example of a saccade detection method The figure for demonstrating an example of the method of measuring the eyeball retention period. Diagram for explaining an example of a method for measuring the degree of eye opening A flowchart illustrating the detailed flow of the state determination step of FIG. The figure for demonstrating an example of the saccade detection method in the modification 1 of this disclosure.

Hereinafter, embodiments of the awakening state determination device, the awakening system, and the awakening state determination method according to the present disclosure will be described with reference to the attached drawings. In each of the following embodiments, the same reference numerals are given to the same components.

[1. Application example]
An example to which the awakening state determination device, the awakening system, and the awakening state determination method according to the present disclosure can be applied will be described with reference to FIG. FIG. 1 is a diagram for explaining an application example of the awakening state determination device 100 according to the present disclosure.

The awakening state determination device 100 according to the present disclosure can be applied to, for example, in-vehicle use. FIG. 1 illustrates an awakening system 1 equipped with an awakening state determination device 100. The awakening system 1 is embodied as, for example, a vehicle control system that controls the vehicle 2.

The awakening state determination device 100 includes a camera 3 attached to the vehicle 2. The camera 3 is an example of the “imaging unit” of the present disclosure. The camera 3 captures the driver's face, especially the eyes. For example, the camera 3 is mounted in front of the driver, such as near the steering column cover, dashboard, and room mirror. The position of the camera 3 is not limited to this, and may be any position as long as it can capture the driver's face. For example, the camera 3 may be a spectacle-type camera or a head-mounted camera mounted on the driver's head.

The camera 3 has a frame rate of, for example, about 20 to 60 fps, for example, 30 fps. In other words, the camera 3 has a time resolution of 20-60 Hz. The camera 3 may be a high-speed camera having a frame rate of 100 fps or more, but the awakening state determination device, the awakening system, and the awakening state determination method according to the present disclosure are realized by using a normal, that is, a non-high-speed camera. it can.

The awakening state determination device 100 measures the driver's eye movement or eye opening degree based on the image captured by the camera 3. At the time of measurement, data indicating the positions of the driver's pupil center, upper eyelid, and lower eyelid obtained by analyzing the image captured by the camera 3 is used. Specifically, the arousal state determination device 100 measures the driver's saccade frequency, eyeball retention period, and eye opening degree.

In the image analysis, for example, a process of extracting the positions of the driver's pupil center, upper eyelid, and lower eyelid from the image of the driver's face captured by the camera 3 is performed. Even when a normal, that is, non-high-speed camera is used, if the driver's eyes are within the imaging range of the camera 3, the camera 3 may vibrate or the driver's face may shake due to the shaking of the vehicle 2. Even so, it is possible to extract the positions of the center of the pupil, the upper eyelid, and the lower eyelid.

The awakening state determination device 100 determines whether or not the driver is in a drowsy state or a drowsy state based on the measurement result. The rambling state is a state in which the driver's arousal level is lower than that at the time of awakening. For example, in a state of rambling, the driver does not concentrate on driving and becomes vague. The drowsiness state is a state in which the arousal level is further lowered from the drowsiness state. For example, a drowsiness state is a state in which the driver is drowsy.

As a result of diligent research, the inventor found that when the driver falls into a state of rambling, the number of saccades becomes about 50% of the time of awakening, and the eyeball retention period also becomes about 70% of the time of awakening.

In addition, as a result of diligent research, the inventor has found that when the driver falls into a drowsy state, the time rate at which the degree of eye opening, which indicates the degree of eye opening, decreases to about 50% of the time of awakening increases.

The awakening state determination device 100 can determine whether the driver is in an awake state, a drowsiness state, or a drowsiness state by measuring the saccade frequency, the eyeball retention period, and the degree of eye opening.

When the awakening state determination device 100 determines that the driver is in a drowsy state or a drowsy state, the awakening state determination device 100 drives the output unit 20 to improve the awakening degree of the driver. The output unit 20 is, for example, a speaker, and when the driver is in a drowsy state or a drowsy state, the output unit 20 outputs a voice to awaken the driver.

As described above, the awakening system 1 is applied to, for example, in-vehicle applications. When the camera 3 is attached to the vehicle 2, there are various factors that reduce the accuracy of imaging, such as the shaking of the vehicle 2 and the movement of the driver. In the awakening system 1, even if there are these personnel, the awakening state of the driver can be determined based on the image captured by the normal, that is, the non-high speed camera 3, and the awakening degree of the driver is determined according to the determination result. Can be improved.

[2. Configuration example]
A configuration example and an operation example of the awakening system 1 according to the embodiment of the present disclosure will be described below.

FIG. 2 is a diagram illustrating the configuration of the awakening system 1. As illustrated in FIG. 2, the awakening system 1 includes an awakening state determination device 100 for determining the awakening state of the subject regarding the arousal level, and an output unit 20. As illustrated in FIG. 2, the awakening state determination device 100 includes a camera 3, a storage unit 5, and a control unit 10.

The camera 3 is an imaging device that images the face of the subject, especially the eyes, to form an captured image. The camera 3 is, for example, a camera that forms an image captured by a solid-state image sensor such as CMOS (Complementary MOS) or CCD (Charge Coupled Device). The camera 3 has a frame rate of, for example, about 20-60 fps, for example 30 fps. The camera 3 may be a high-speed camera having a frame rate of 100 fps or more.

The awakening state determination device 100 may include an infrared irradiator 4. The infrared irradiator 4 is a light source device that irradiates an infrared ray toward the subject's face, particularly the eyes. Infrared rays reflected by the subject's face, cornea, etc. are incident on the camera 3. In this case, the camera 3 is configured to have sensitivity in the infrared region.

The configuration of the awakening state determination device 100 is not limited to the above, and a camera 3 having sensitivity in the same wavelength range as the wavelength of the light reflected by the subject's face may be provided. For example, the awakening state determination device 100 may be configured to include a visible light irradiator and a visible light camera, or an ultraviolet irradiator and an ultraviolet camera. Alternatively, the awakening state determination device 100 does not include a light source device such as an infrared irradiator 4, a visible light irradiator, and an ultraviolet irradiator, and the camera 3 captures sunlight or fluorescent light reflected by the subject's face. It may be an image.

The storage unit 5 stores the information of the program or the like by electrical, magnetic, optical, mechanical or chemical action so that the computer or other device, the machine or the like can read the information of the recorded program or the like. It is a medium to do. The storage unit 5 includes, for example, an auxiliary storage device such as a hard disk drive or a solid state drive.

The control unit 10 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like, and controls each component according to information processing. The control unit 10 is composed of, for example, an ECU (electronic control unit). The control unit 10 interprets and executes, for example, a program expanded in the RAM by the CPU. As the software module realized in this way, the control unit 10 includes, for example, an image processing unit 11, a measurement unit 12, a state determination unit 13, and a drive control unit 14.

The output unit 20 executes a predetermined operation based on the determination result of the awakening state determination device 100.

The output unit 20 includes, for example, a display unit 21. The display unit 21 is a display device such as a liquid crystal display, an organic EL display, and a projector. The display unit 21 may be, for example, a head-up display that projects an image onto the windshield of the vehicle 2 or a dedicated glass panel. For example, the control unit 10 of the awakening state determination device 100 that operates as the drive control unit 14 transmits a control signal indicating the determination result of the state of the target person to the display unit 21, and the display unit that receives the control signal. 21 displays the determination result.

The output unit 20 includes, for example, a speaker 22. The speaker 22 is a voice output device that notifies the target person of various information by voice. For example, the control unit 10 of the awakening state determination device 100 that operates as the drive control unit 14 causes the speaker 22 to output a voice corresponding to the determination result of the state of the target person. For example, the warning sound output from the speaker 22 improves the arousal level of the subject, and eliminates the drowsiness state and the drowsiness state. The output unit 20 may include a conversation system 23 that has a conversation with the target person, and the conversation system 23 may output a voice according to a determination result about the state of the target person. For example, when the control unit 10 determines that the subject is in a drowsy state or a drowsy state, the control unit 10 causes the conversation system 23 to output a voice to recommend a break.

The output unit 20 includes, for example, an air conditioning system 24. The air conditioning system 24 is a system that adjusts the temperature, humidity, air flow, and the like of the air around the subject. For example, the control unit 10 of the awakening state determination device 100 that operates as the drive control unit 14 controls the air conditioning system 24 according to the determination result of the state of the target person to lower the temperature around the target person or the target person. Or blow the wind on. As a result, the arousal level of the subject is improved, and the drowsiness state and the drowsiness state are eliminated.

The output unit 20 includes, for example, a lighting system 25. The lighting system 25 is, for example, a light source device that adjusts the illuminance and the brightness of the space around the subject. For example, the control unit 10 of the awakening state determination device 100 that operates as the drive control unit 14 controls the lighting system 25 according to the determination result of the state of the target person to increase the illuminance or blink the light source. As a result, the arousal level of the subject is improved, and the drowsiness state and the drowsiness state are eliminated.

The output unit 20 includes, for example, a vibration unit 26. The vibrating unit 26 is attached to, for example, a seat on which the subject sits, and can transmit vibration to the subject. When the awakening system 1 is applied to an in-vehicle application, the vibrating unit 26 may be attached to, for example, a steering handle and transmit the vibration to the target person's hand. For example, the control unit 10 of the awakening state determination device 100 that operates as the drive control unit 14 vibrates the vibrating unit 26 according to the determination result of the state of the target person. As a result, the arousal level of the subject is improved, and the drowsiness state and the drowsiness state are eliminated.

When the awakening system 1 is applied to an in-vehicle application, the output unit 20 may be, for example, a brake 27. For example, the control unit 10 of the awakening state determination device 100 that operates as the drive control unit 14 controls the brake 27 according to the determination result of the state of the target person, and stops the vehicle 2 driven by the target person. As a result, it is possible to prevent a collision accident or the like caused by a decrease in arousal level such as a drowsy state or a drowsy state of the subject.

Further, when the awakening system 1 is applied to an in-vehicle application, the output unit 20 may be, for example, a steering wheel 28. For example, the control unit 10 of the awakening state determination device 100 that operates as the drive control unit 14 controls the steering 28 according to the determination result of the state of the target person, and controls the traveling direction of the vehicle 2 driven by the target person. .. As a result, it is possible to prevent a collision accident or the like caused by a decrease in arousal level such as a drowsy state or a drowsy state of the subject.

[3. Operation example]
[3-1. Overview]
FIG. 3 is a flowchart illustrating the flow of operation of the awakening system 1. Each process of the flowchart of FIG. 3 is executed by the control unit 10 of the awakening state determination device 100 of the awakening system 1.

The control unit 10 executes the initial processing step S1 and the state determination step S2, turns on the drowsiness state flag when the target person is in a drowsiness state, and turns on the drowsiness state flag when the target person is in a drowsiness state. To. Details of steps S1 and S2 will be described later.

Next, the control unit 10 that operates as the drive control unit 14 determines whether or not the drowsiness state flag is ON (S3). When it is determined that the drowsiness state flag is ON (Yes in S3), the control unit 10 operating as the drive control unit 14 performs drive control for the drowsiness state with respect to the output unit 20 (S4). The drive control for the drowsiness state is a drive control for increasing the arousal level of the subject and eliminating the drowsiness state.

For example, the control unit 10 controls the voice output of the conversation system 23, the vibration application by the vibration unit 26, the vehicle stop by the brake 27, the adjustment of the vehicle traveling direction by the steering 28, and the like as the drive control for the drowsiness state.

On the other hand, when it is determined in step S3 that the drowsiness state flag is OFF (No in S3), the control unit 10 operating as the drive control unit 14 determines whether or not the drowsiness state flag is ON (S5). .. When it is determined that the rambling state flag is ON (Yes in S5), the control unit 10 operating as the drive control unit 14 performs drive control for the rambling state to the output unit 20 (S6). The drive control for the rambling state is a drive control for increasing the arousal level of the subject and eliminating the rambling state.

For example, the control unit 10 controls the output of a warning sound from the speaker 22, the adjustment of the temperature and the air flow by the air conditioning system 24, the increase in the illuminance by the lighting system 25, and the like as the drive control for the rambling state.

When step S4 is completed, when step S6 is completed, and when it is determined in step S5 that the vague state flag is OFF (No in S5), the process returns to step S2.

The repetition of steps S2 to S6 may be performed immediately when each step is completed, or may be performed at predetermined time intervals.

In the flowchart illustrated in FIG. 3, the initial processing step S1 is executed only once. However, the operation of the awakening system 1 is not limited to this. For example, every time a predetermined period elapses, the process may return to step S1 and the initial process may be performed again.

[3-2. Initial processing]
FIG. 4 is a flowchart illustrating the detailed flow of the initial processing step S1 of FIG. First, the control unit 10 acquires an image of the target person's eyes captured by the camera 3 (S101). Step S101 may include irradiating the subject's eyes with infrared rays by the infrared irradiator 4 before acquiring the image.

Next, the control unit 10 that operates as the image processing unit 11 performs image processing on the acquired image of the eye, and detects the position of the center of the pupil and the positions of the upper eyelid and the lower eyelid (S102). The detected data indicating the positions of the pupil center, the upper eyelid, and the lower eyelid are stored in the storage unit 5.

The position of the center of the pupil is used to detect saccades and saccades observed as eye movements. Here, the stagnation is sometimes referred to as fixation.

Even if the entire face of the subject moves significantly, if the relative position of the eyeball with respect to the face does not change, no eye movement such as saccade is performed. Therefore, the control unit 10 does not detect eye movements such as saccades when the relative position of the eyeball with respect to the face does not change even when the entire face of the subject moves significantly. The position relative to the reference point on the subject's face or the like is detected as the position of the center of the pupil. In this sense, it can be said that the detection of the position of the center of the pupil performed by the control unit 10 operating as the image processing unit 11 is the detection of the line of sight of the subject.

Specifically, for example, the control unit 10 detects the position of the center of the pupil by using the corneal reflex method. When the corneal reflex method is used, infrared rays emitted from the infrared irradiator 4 which is a point light source are reflected by the cornea of the subject and incident on the camera 3. The infrared rays reflected from the cornea are called Pulkinier images. The Pulkinier image serves as a reference point for detecting the position of the center of the pupil. That is, when eye movement occurs, the position of the center of the pupil relative to the Purkinje image changes. Utilizing this, the control unit 10 can determine the position of the center of the pupil of the subject based on the position of the Pulkinier image.

Alternatively, the control unit 10 may detect the relative position of the center of the pupil of the subject with at least one of the inner and outer corners of the eye (outer canthus) of the subject as a reference point. In this case, it is not necessary to use the infrared irradiator 4.

Steps S101 and S102 are repeated from the start of the initial processing step S1 until a predetermined period T0 elapses (S103). As a result, data indicating the positions of the subject's pupil center, upper eyelid, and lower eyelid are accumulated in the storage unit 5.

After a predetermined period T0 has elapsed from the start of the initial processing step S1 (Yes in S103), the control unit 10 operating as the measurement unit 12 has an initial value n0 (S104) of the saccade frequency n of the subject and an eyeball retention period t. The initial value ta0 (S105) of the average value ta and the initial value La0 (S106) of the average value La of the eye opening degree L are calculated. At the time of calculation, data indicating the positions of the center of the pupil, the upper eyelid, and the lower eyelid stored in the storage unit 5 is used. Steps S104, S105, and S106 are in no particular order. The initial values n0, ta0, and La0 calculated in steps S104, S105, and S106 are stored in the storage unit 5.

As described above, in step S104, the initial value n0 of the saccade frequency n is calculated. Here, the saccade frequency n is the number of saccades per unit time. Assuming that the number of saccades generated from the start of the initial processing step S1 to the elapse of the predetermined period T0 is N0, the initial value n0 of the saccade frequency is represented by, for example, the following equation (1).
n0 = N0 / T0 ... (1)

FIG. 5 is a diagram for explaining an example of a saccade detection method. FIG. 5 shows points _pi to _{pi + 2} representing the position of the center of the pupil detected in step S102. Figure 5 is moved, the position of the pupil center, from the measured _{p i} at time _{t i,} the _{p i + 1} at time _{t i + 1} is later Δt seconds at time _{t i + 2} is after addition thereof Δt seconds _{p i + 2} It shows that it was done. Here, Δt is represented by, for example, Δt = 1 / F using the frame rate F of the camera 3.

For example, in the control unit 10, the distance between the position of the pupil center at a certain time (current frame) and the position of the pupil center in the frame immediately before that (that is, the frame Δt seconds before the current frame) is a predetermined threshold value d1. If it exceeds _th , it is determined that a saccade has occurred between the previous frame and the current frame. In the example shown in FIG. 5, since the distance between the position _{p i + 1} of the pupil center at time _{t i + 1,} and the position _{p i + 2} of the pupil center at time _{t i + 2} is greater than the threshold _{d1 th,} time _{t i + 1} and the time _{t i + 2} It is judged that a saccade has occurred between.

In the next step S105, the initial value ta0 of the average value of the eyeball retention period is calculated.

A method for measuring the period of eye retention will be described. For example, in the control unit 10, the distance between the position of the pupil center at a certain time (current frame) and the position of the pupil center in the frame immediately before that (that is, the frame Δt seconds before the current frame) is a predetermined threshold value d2. _When the period t equal to or less than _th exceeds a predetermined threshold value t _th , it is determined that "staying" has occurred. When it is determined that "staying" has occurred, the control unit 10 counts this period t as the eyeball staying period.

In other words, in a case where the position of the pupil center in the current frame, the position of the pupil center in the immediately preceding frame, the state distance is below a predetermined threshold value d2 _th, it was detected over N successive frames, when N / F exceeds a predetermined threshold value _{t th,} the control unit 10, record a N / F as eye fixation duration.

The initial value ta0 of the average value of the eyeball retention period is a value obtained by dividing the total of the eyeball retention periods generated from the start of the initial processing step S1 to the elapse of the predetermined period T0 by the number of retentions.

FIG. 6 is a diagram for explaining an example of a method for measuring the eyeball retention period. FIG. 6 shows the positions p _j to p _{j + 11 of the} center of the pupil detected at times t _j to t _{j + 11} in step S102, respectively.

In the example shown in FIG. 6, it is assumed that the frame rate F of the camera 3 is 30 fps and the predetermined time t _th is set to 250 msec. Further, in the example shown in FIG. 6, the distance between p _j and p _{j + 1} and the distance between p _{j + 10} and p _{j + 11} exceed a predetermined threshold value d2 _th , but between other frames. It is assumed that the distances between the positions of the center of the pupil are all equal to or less than the predetermined threshold value d2 _th . That _is, between the _{p j + 2} and _{p j + _3,} between the p _{j + 3} and _{p j + 4} _between the p _{j + 4} and _{p j + _5,} between the p _{j + 5} and _{p j + _6,} between the p _{j + 6} and _{p j + _7,} and p _{j + 7} the distance between the between the p _{j + _8,} between the p _{j + 8} and _{p j + _9,} and a _{p j + 9} and _{p j + 10} are both less than a predetermined threshold value _{d2 th.}

In the example shown in FIG. 6, the period t in which the distance between the positions of the pupil centers between the frames is equal to or less than the predetermined threshold value d2 _th is 300 msec from the time t _{j +} ₁ to the time t _{j + 10} , and the predetermined threshold value t _th = It exceeds 250 msec. Therefore, the control unit 10 determines that "staying" has occurred, and calculates the eyeball staying period as 300 msec.

The predetermined threshold value d1 _th , which is a criterion for determining the presence or absence of _saccade, and the predetermined threshold value d2 _th, which is a criterion for determining the presence or absence of _saccade , may be the same value. In this case, in FIG. 6, it is determined that the saccade has occurred between the time t _j and the time t _{j + 1,} and between the time t _{j + 10} and t _{j + 11} . The d1 _th and the d2 _th may have different values.

As described above, in step S106, the initial value La0 of the average value of the eye opening degree L is calculated. The average value of the eye opening degree L is a time average value of the eye opening degree L. FIG. 7 is a diagram for explaining an example of a method for measuring the degree of eye opening L. The degree of eye opening L is an index of the degree of opening of the eyes. The degree of eye opening L is represented by y / x. Here, x is the lateral width of the eye, for example, the distance from the inner canthus 91 to the outer canthus 92. y is the eye opening height, and is, for example, the maximum value of the distance from the lower end 93 of the upper eyelid to the upper end 94 of the lower eyelid in the direction perpendicular to the line segment connecting the inner and outer eye angles 92. The eye opening height y is reduced by the action of closing the eyes or the blinking.

[3-3. Status judgment]
FIG. 8 is a flowchart illustrating the detailed flow of the state determination step S2 of FIG. First, the control unit 10 acquires an image of the target person's eyes captured by the camera 3 (S201). Next, the control unit 10 that operates as the image processing unit 11 performs image processing on the acquired image of the eye, and detects the position of the center of the pupil and the positions of the upper eyelid and the lower eyelid (S202). The detected data indicating the positions of the pupil center, the upper eyelid, and the lower eyelid are stored in the storage unit 5.

Steps S201 and S202 are repeated from the start of the state determination step S2 until a predetermined period T1 elapses (S203). As a result, data indicating the positions of the subject's pupil center, upper eyelid, and lower eyelid are accumulated in the storage unit 5.

After a predetermined period T1 has elapsed from the start of the state determination step S2 (Yes in S203), the control unit 10 operating as the measurement unit 12 calculates the current value La1 of the average value La of the eye opening degree L (S204). At the time of calculation, the data indicating the positions of the upper eyelid and the lower eyelid stored in the storage unit 5 from the start of the state determination step S2 to the elapse of the predetermined period T1 are used.

Next, in the control unit 10 that operates as the state determination unit 13, the current value La1 of the average value of the eye opening degree obtained in step S204 is a predetermined value Lax and the initial value La0 obtained in step S106 of FIG. It is determined whether or not the product of Lax × La0 is smaller than (S205). Lax × La0 is an example of the “third threshold” of the present disclosure. Lax is, for example, 0.2 to 0.8, for example 0.5 or 0.7.

When it is determined that La1 is smaller than Lax × La0 (Yes in S205), the control unit 10 operating as the state determination unit 13 turns on the drowsiness state flag (S206). In step S206, it is determined that the subject is in a drowsy and drowsy state. After finishing step S206, the control unit 10 finishes the state determination step S2 and proceeds to step S3 of FIG.

On the other hand, when it is determined that La1 is Lax × La0 or more (No in S205), the control unit 10 operating as the measurement unit 12 calculates the current value n1 of the saccade frequency n (S207). At the time of calculation, the data stored in the storage unit 5 indicating the position of the center of the pupil from the start of the state determination step S2 to the elapse of the predetermined period T1 is used.

Next, in the control unit 10 that operates as the state determination unit 13, the current value n1 of the saccade frequency obtained in step S207 is the product of a predetermined value nx and the initial value n0 obtained in step S104 of FIG. It is determined whether or not it is smaller than nx × n0 (S208). nx × n0 is an example of the “first threshold” of the present disclosure. nx is, for example, 0.2 to 0.8, for example 0.5.

When it is determined that n1 is nx × n0 or more (No in S208), the control unit 10 finishes the state determination step S2 and proceeds to step S3 in FIG. In this case, it is determined that the subject is not in a drowsy state or a drowsy state.

When it is determined that n1 is smaller than nx × n0 (Yes in S208), the control unit 10 operating as the measurement unit 12 calculates the current value ta1 of the average value ta of the eyeball retention period t (S209). At the time of calculation, the data stored in the storage unit 5 indicating the position of the center of the pupil from the start of the state determination step S2 to the elapse of the predetermined period T1 is used.

Next, in the control unit 10 that operates as the state determination unit 13, the current value ta1 of the average value of the eyeball retention period obtained in step S209 is a predetermined value tax and the initial value ta0 obtained in step S105 of FIG. It is determined whether or not the product is smaller than the product tax × ta0 (S210). tax × ta0 is an example of the “second threshold” of the present disclosure. The tax is, for example, 0.2 to 0.8, for example 0.7.

When it is determined that ta1 is tax × ta0 or more (No in S210), the control unit 10 finishes the state determination step S2 and proceeds to step S3 in FIG. In this case, it is determined that the subject is not in a drowsy state or a drowsy state.

When it is determined that ta1 is smaller than tax × ta0 (Yes in S210), the control unit 10 operating as the state determination unit 13 turns on the rambling state flag (S211). In step S211 it is determined that the subject is in a vague state. When the step S211 is completed, the control unit 10 finishes the state determination step S2 and proceeds to the step S3 of FIG.

[4. Summary]
As described above, the awakening state determination device 100 according to the present embodiment determines the awake state of the subject in the awake state. The awakening state determination device 100 includes a camera 3 that captures an image including the eyes of the subject and generates image data, and a control unit 10. The control unit 10 sequentially acquires image data from the camera 3 and detects the movement of the eyes of the target person based on the acquired image data. The control unit 10 measures the frequency of saccades in which the eyes cause saccades based on the movements of the eyes detected during the period T1 to be determined. The control unit 10 measures the eyeball retention period during which the eye movement is stopped, based on the detected eye movement. Based on the measured saccade frequency and eyeball retention period, the control unit 10 determines whether or not the awakening state of the subject is in a vague state during the determination target period T1.

According to this awakening state determination device 100, it is possible to determine whether or not the subject is in a vague state based on the saccade frequency and the eyeball retention period. The saccade frequency and the eyeball retention period can be measured mainly by a simple configuration including the camera 3 and the control unit 10 and a simple process.

For example, when the magnitude of the eye movement in the predetermined time width Δt exceeds the predetermined value d1 _th , the control unit 10 detects the eye movement as a saccade. For example, the control unit 10 measures the saccade frequency based on the saccade detected during the determination target period T1. In this way, the saccade can be measured by a simple configuration and a simple process. The camera 3 does not have to be a high-speed camera having a frame rate of 100 fps or more, for example.

For example, the control unit 10 measures a period in which the magnitude of eye movement in the time width Δt is a predetermined value d2 _th or less and continues beyond a predetermined period t _th longer than the time width Δt as an eyeball retention period. To do. It may be d2 _th = d1 _th . In this way, the eyeball retention period can be measured by a simple configuration and a simple process.

When the saccade frequency is smaller than the first threshold value nx × n0 and the average value of the eyeball retention period in the determination target period T1 is smaller than the second threshold value tax × ta0, the control unit 10 changes the awakening state of the subject. It may be determined that the state is in a saccade state.

The first threshold value nx × n0 may be defined by the frequency of saccades of the subject in the past fixed period, and the second threshold tax × ta0 may be defined by the eyeball retention period of the subject in the fixed period. This makes it possible to determine a personal threshold value suitable for the subject. Therefore, the awakening state can be appropriately determined for each subject.

[5. Modification example]
Although the embodiments of the present disclosure have been described in detail above, the above description is merely an example of the present disclosure in all respects. Various improvements and modifications can be made without departing from the scope of the present disclosure. For example, the following changes can be made. In the following, the same reference numerals will be used for the same components as those in the above embodiment, and the same points as in the above embodiment will be omitted as appropriate. The following modifications can be combined as appropriate.

[5-1. Modification 1]
In the above embodiment, an example in which the control unit 10 detects the movement of the position of the center of the pupil as a distance has been described. However, the present disclosure is not limited to this, as long as the control unit 10 can detect the movement of the position of the center of the pupil. For example, the control unit 10 may detect the movement of the position of the center of the pupil as the movement of the viewpoint (gaze point) corresponding to the angle conversion of the line of sight.

FIG. 9 is a diagram for explaining an example of the saccade detection method in the modified example 1. In FIG. 9, for convenience of explanation, a θx axis and a θy axis are provided. θx represents the angle of the line of sight in the horizontal direction, and θy orthogonal to θx represents the angle of the line of sight in the vertical direction. It is assumed that the viewpoint of the subject can move from the coordinates (0,0) at the lower left of the paper to the coordinates (1,1) at the upper right. Here, (0,0) means that the angle of the line of sight is about 34.5 degrees to the left and downward when the coordinates (0.5,0.5) are the front (0 degrees, 0 degrees). Corresponds to the state where the temperature is about 34.5 degrees. (1,1) corresponds to a state in which the line-of-sight angle is about 34.5 degrees to the right and about 34.5 degrees to the top.

In FIG. 9, points q _i to q _{i + 2} representing the positions of the viewpoints detected in step S102 of FIG. 4 are shown. Figure 9 is moved, the viewpoint is, from the viewpoint _{q i} measured at time _{t i,} the viewpoint _{q i + 1} at time _{t i + 1} is later Δt seconds, further to the viewpoint _{q i + 2} at time _{t i + 2} is a later Δt seconds It shows that it was done. Here, Δt is represented by, for example, Δt = 1 / F using the frame rate F of the camera 3.

For example, in the control unit 10, the angle difference between the viewpoint at a certain time (current frame) and the viewpoint at the frame immediately before that (that is, the frame Δt seconds before the current frame) exceeds a predetermined threshold value e1 _th . In this case, it is determined that a saccade has occurred between the previous frame and the current frame. The threshold value e1 _th is, for example, 0.25, and corresponds to the case where the angle change of the line of sight is about 15 degrees to about 19 degrees, for example 17.25 degrees. In the example shown for example in FIG. 9, a viewpoint _{q i + 1} at time _{t i + 1,} since the angle difference between the view point _{q i + 2} at time _{t i + 2} is greater than the threshold value _{e1 th,} time _{t i + 1} and the time _{t i + 2} and the It is judged that a saccade occurred in the meantime.

[5-2. Modification 2]
Step S205 shown in FIG. 8 is for determining whether or not the subject is in a drowsy state, and is a step for the control unit 10 to determine whether or not La1 is smaller than Lax × La0. However, the method for determining whether or not the subject is drowsy is not limited to the above.

For example, in the control unit 10 that operates as the state determination unit 13, the eye opening degree L1 from the start of the state determination step S2 to the elapse of the predetermined period T1 is the predetermined value Lx and the eye opening degree obtained in step S106. The period U, which is smaller than the product Lx × La0 of the initial value La0 of the average value, is measured. The threshold Lx is, for example, 0.2 to 0.8, for example 0.5 or 0.7.

Then, when the U / T1 is equal to or higher than the threshold value _Uth , the control unit 10 operating as the state determination unit 13 may determine that the subject is in a drowsy state and proceed to step S206. The threshold value _Uth is, for example, 0.2 to 0.8, for example, 0.6.

That is, the control unit 10 measures the eye opening degree L1 and when the period U in which the eye opening degree L1 is smaller than a predetermined threshold value is equal to or greater than the product of the measurement period T1 and the predetermined value _Uth , the subject It may be determined that the awake state of is drowsy. For example, in the control unit 10, when the eye opening degree L1 of the subject is 60% or more of 70% or less of the initial value La0 of the average value of the eye opening degree, the subject is in a drowsy state. Judge that there is.

[5-3. Modification 3]
In the above embodiment, it has been explained that the control unit 10 that operates as the drive control unit 14 controls the output unit 20 based on the determination result of the state of the target person. However, the present disclosure is not limited to this. For example, when the awakening system 1 is applied to an in-vehicle application, when it is determined that the subject is in a drowsy state or a drowsy state, the control unit 10 operating as the drive control unit 14 starts the automatic operation of the vehicle 2. You may. As a result, it is possible to prevent a collision accident or the like caused by a decrease in arousal level such as a drowsy state or a drowsy state of the subject.

[5-4. Modification 4]
The awakening system 1 according to the present disclosure may be used not only for in-vehicle use illustrated above, but also for awakening factory workers, for example, workers engaged in simple work. For example, the camera 3 captures the eyes of an inspector who inspects the appearance of a product flowing through a factory line. Based on the imaged data, the control unit 10 determines whether or not the inspector is in a state of drowsiness or drowsiness, and if so, outputs voice from the speaker 22 or the conversation system 23 to the inspector. Improves alertness. Alternatively, if the inspector is in a state of drowsiness or drowsiness, the control unit 10 may stop the operation of the factory line.

(Additional note)
Hereinafter, various aspects relating to the present disclosure will be added.

The first aspect according to the present disclosure is an awakening state determination device (100) for determining a rambling state in a person's awakening state. The awakening state determination device is based on an imaging unit (3) that captures an image including the human eye and generates image data, and the image data that is sequentially acquired from the imaging unit and acquired. A control unit (10) for detecting the movement of the human eye is provided. The control unit (10) measures the frequency of saccades in which the eyes cause saccades based on the eye movements detected during the determination target period (T1), and based on the detected eye movements, the control unit (10) The period of stay in which the movement of the eyes is stopped is measured, and based on the measured saccade frequency and the period of stay, it is determined whether or not the awakening state of the person is in a vague state during the period to be determined.

In the second aspect, in the awakening state determination device of the first aspect, the control unit (10) moves the eyes when the magnitude of the movements of the eyes in a predetermined time width exceeds a predetermined value. Is detected as a saccade, and the frequency of the saccade is measured based on the saccade detected during the period to be determined.

In the third aspect, in the awakening state determination device of the second aspect, the control unit (10) has a state in which the magnitude of the movement of the eyes in the time width is equal to or less than the predetermined value from the time width. The period that continues beyond a long predetermined period is measured as the retention period.

In the fourth aspect, in the awakening state determination device of any one of the first to third aspects, the control unit (10) has the saccade frequency smaller than the first threshold value and during the period of the determination target. When the average value of the dwelling period is smaller than the second threshold value, it is determined that the awake state of the person is the vague state.

In the fifth aspect, in the arousal state determination device of the fourth aspect, the first threshold value is defined by the frequency of the saccade of the person in the past fixed period, and the second threshold value is the said in the fixed period. It is defined by the period of stay of the person.

In the sixth aspect, in the awakening state determination device of any one of the first to fifth aspects, the control unit (10) measures the degree of eye opening by the human eye based on the acquired image data. When the time during which the degree of eye opening is smaller than the third threshold value is equal to or longer than a predetermined ratio in the period to be determined, it is determined that the person's awake state is drowsy.

In the seventh aspect, in the arousal state determination device of the sixth aspect, the third threshold value is defined by the degree of eye opening of the person in the past fixed period.

The eighth aspect is the awakening including the awakening state determination device of any one of the first to seventh aspects and an output unit (20) that executes a predetermined operation based on the determination result of the awakening state determination device. System (1).

The ninth aspect is an awakening state determination method for determining a rambling state in a person's awakening state. The awakening state determination method includes a step (S201) in which the control unit (10) sequentially acquires image data generated by capturing an image including the human eye, and the control unit (10) acquires the image data. Based on the step (S202) of detecting the movement of the eyes of the person based on the image data, and the movement of the eyes detected by the control unit (10) during the period (T1) of the determination target, the eyes play soccer. A step (S207) of measuring the frequency of saccades that cause saccades, and a step (S209) in which the control unit (10) measures the retention period during which the eye movements are stopped based on the detected eye movements. , The control unit (10) determines whether or not the awakening state of the person is in a vague state during the period to be determined (S208, S210) based on the measured frequency of occurrence of saccade and the period of stay. ) And.

The tenth aspect is a program for causing the control unit to execute the awakening state determination method of the ninth aspect.

1 Awakening system 2 Vehicle 3 Camera (imaging unit)
4 Infrared irradiator 5 Storage unit 10 Control unit 11 Image processing unit 12 Measurement unit 13 Status judgment unit 14 Drive control unit 20 Output unit 21 Display unit 22 Speaker 23 Conversation system 24 Air conditioning system 25 Lighting system 26 Vibration unit 27 Brake 28 Steering 100 Awakening state judgment device

Claims

It is an awakening state determination device that determines the awakening state of a person.
An image pickup unit that captures an image including the human eye and generates image data,
A control unit that sequentially acquires the image data from the imaging unit and detects the movement of the human eye based on the acquired image data is provided.
The control unit
Based on the eye movements detected during the period to be determined, the frequency of saccades in which the eyes generate saccades is measured.
Based on the detected eye movement, the retention period during which the eye movement stays is measured.
An awakening state determining device that determines whether or not the awakening state of the person is a vague state during the period to be determined based on the measured saccade frequency and the residence period.
The control unit
When the magnitude of the eye movement in a predetermined time width exceeds a predetermined value, the eye movement is detected as a saccade, and the eye movement is detected as a saccade.
The awakening state determination device according to claim 1, wherein the saccade frequency is measured based on the saccade detected during the period to be determined.
The control unit measures the period in which the magnitude of the movement of the eyes in the time width is equal to or less than the predetermined value for a period longer than the predetermined period longer than the time width, as the dwelling period. 2. The arousal state determination device according to 2.
When the saccade frequency is lower than the first threshold value and the average value of the dwelling period in the determination target period is smaller than the second threshold value, the control unit is in the awake state of the person. , The awakening state determination device according to any one of claims 1 to 3.
The first threshold is defined by the frequency of the person's saccade over a period of time in the past.
The awakening state determination device according to claim 4, wherein the second threshold value is defined by the dwelling period of the person in the fixed period.
The control unit
Based on the acquired image data, the degree of eye opening by the human eye is measured.
Any of claims 1 to 5, wherein it is determined that the person's arousal state is drowsy when the time during which the eye opening degree is smaller than the third threshold value is equal to or longer than a predetermined ratio in the determination target period. Awakening state determination device described in Crab.
The arousal state determination device according to claim 6, wherein the third threshold value is defined by the degree of eye opening of the person in a certain period in the past.
The awakening state determination device according to any one of claims 1 to 7.
An awakening system including an output unit that executes a predetermined operation based on a determination result of the awakening state determination device.
It is an awakening state determination method for determining a rambling state in a person's awakening state.
A step in which the control unit sequentially acquires image data generated by capturing an image including the human eye.
A step in which the control unit detects the movement of the human eye based on the acquired image data,
A step of measuring the frequency of saccades caused by the eyes based on the movement of the eyes detected by the control unit during the period to be determined.
A step in which the control unit measures the dwelling period during which the eye movement is stopped based on the detected eye movement.
The control unit includes a step of determining whether or not the awakening state of the person is a vague state during the period of the determination target, based on the measured frequency of occurrence of saccade and the period of stay.
Awakening state determination method.
A program for causing the control unit to execute the awakening state determination method according to claim 9.