JP7550879B2 - Sleep detection device and sleep detection system - Google Patents

Description

The present disclosure relates to a sleep detection device and a sleep detection system.

In order to prevent occupants of vehicles such as automobiles from falling asleep while driving, a technology has been proposed that detects the state of the occupant's face and posture and estimates the drowsiness felt by the occupant. Conventionally, because the state of drowsiness changes in stages before reaching strong drowsiness, information related to signs of drowsiness, such as the degree of eye opening and the distance between the eyes and eyebrows, was used to estimate which stage of drowsiness the occupant is in among the stages divided into stages (see, for example, Patent Document 1).

JP 2008-220424 A

Occupants may momentarily fall asleep and collapse, resulting in what is known as microsleep, and in order to prevent drowsy driving, it is necessary to detect the aforementioned momentary sleep. However, simply estimating the drowsiness stage in which the occupant is in is not enough to detect momentary sleep, resulting in poor accuracy in detecting occupant sleep.

The present disclosure has been made to solve the above-mentioned problems, and aims to provide a sleep detection device that detects momentary sleep of an occupant and improves the accuracy of detecting occupant sleep.

The sleep detection device according to the present disclosure includes an image acquisition unit that acquires an image of an occupant from an imaging device mounted on a vehicle and imaging an occupant inside the vehicle, a slowing determination unit that determines whether the occupant extracted from the image acquired by the image acquisition unit has experienced a dulling of perception, a posture determination unit that determines whether the posture of the occupant extracted from the image acquired by the image acquisition unit is abnormal, and a sleep detection unit that detects sleep of the occupant using the determination results of the slowing determination unit and the posture determination unit, wherein the slowing determination unit determines that the occupant has experienced a dulling of perception when an index value indicating a change in the occupant's facial direction is determined to be less than a slowing determination threshold and the state in which the index value is less than the slowing determination threshold continues for more than a slowing determination time , and the sleep detection unit determines that the occupant has fallen into microsleep when the slowing determination unit determines that the occupant has experienced a dulling of perception and the posture determination unit determines that the occupant's posture is abnormal.

In addition, the sleep detection system according to the present disclosure includes an imaging device mounted on a vehicle for imaging an occupant inside the vehicle, an image acquisition unit for acquiring an image of the occupant from the imaging device, a slowing determination unit for determining whether or not the occupant extracted from the image acquired by the image acquisition unit has experienced a dulling of perception, a posture determination unit for determining whether or not the posture of the occupant extracted from the image acquired by the image acquisition unit is abnormal, and a sleep detection unit for detecting sleep of the occupant using the determination results of the slowing determination unit and the posture determination unit, wherein the slowing determination unit determines that the occupant has experienced a dulling of perception when an index value indicating a change in the occupant's facial direction is determined to be less than a slowing determination threshold and the state in which the index value is less than the slowing determination threshold continues for more than a slowing determination time , and the sleep detection unit determines that the occupant has fallen into microsleep when the slowing determination unit determines that the occupant has experienced a dulling of perception and the posture determination unit determines that the occupant's posture is abnormal.

According to the present disclosure, it is possible to detect momentary sleep of an occupant and improve the accuracy of detecting occupant sleep.

1 is a block diagram showing an example of the configuration of a sleep detection system according to a first embodiment. 2 is an explanatory diagram showing an imaging range of the imaging device according to the first embodiment; FIG. 4 is an explanatory diagram showing a detection process of a feature point detection unit of the sleep detection device according to the first embodiment. FIG. 4 is an explanatory diagram showing a face direction detection process of a face direction detection unit of the sleep detection device according to the first embodiment; FIG. 4 is an explanatory diagram showing a sleep detection result of the sleep detection device according to the first embodiment. FIG. 4 is a flowchart showing an example of the operation of the sleep detection device according to the first embodiment. 4 is a flowchart showing an example of the operation of the sleep detection device according to the first embodiment. 1 is a diagram illustrating an example of a hardware configuration of a sleep detection device according to a first embodiment. FIG. 11 is a block diagram showing a configuration example of a sleep detection system according to a second embodiment. 13 is a flowchart showing an example of the operation of the sleep detection device according to the second embodiment. FIG. 11 is a block diagram showing a configuration example of a sleep detection system according to a third embodiment. 13 is a flowchart showing an example of the operation of the sleep detection device according to the third embodiment.

Below, the embodiment is explained based on the drawings.

Embodiment 1.
1 is a block diagram showing an example of a configuration of a sleep detection system 100 according to embodiment 1. The sleep detection system 100 includes a sleep detection device 10 and an imaging device 20, and the sleep detection device 10 and the imaging device 20 are each mounted on a vehicle.

Figure 2 is an explanatory diagram showing the imaging range of the imaging device 20 according to the first embodiment. In Figure 2, the inside of the vehicle equipped with the sleep detection device 10 is viewed from above. The imaging device 20 is composed of, for example, a wide-angle camera, an infrared camera, etc., and captures the inside of the vehicle. The imaging device 20 may also be a distance image sensor such as a TOF (Time-of-Flight) camera that can capture an image that reflects the distance between the imaging device 20 and a subject. The imaging device 20 captures images of the inside of the vehicle at intervals of, for example, 30 to 60 fps (frames per second), and outputs the captured images to the image acquisition unit 11 of the sleep detection device 10. Hereinafter, the images captured by the imaging device 20 are referred to as captured images.

In the example of Figure 2, the imaging area of the imaging device 20 is indicated by area A. One or more imaging devices 20 are arranged on the instrument panel, steering column, rearview mirror, etc. so as to simultaneously capture images of at least occupant 211 and occupant 212 seated in the driver's seat 201 and passenger seat 202, respectively. Hereinafter, the imaging device 20 may include a rear seat (not shown) in its imaging range. Hereinafter, occupants 211 and 212 who are the subject of imaging by the imaging device 20 are collectively referred to as "occupants". In other words, occupants include the driver.

Returning to Fig. 1, the components of the sleep detection device 10 will be described. The sleep detection device 10 includes an image acquisition unit 11 that acquires an image of an occupant from an imaging device 20, a slowing of perception determination unit 16 that determines whether or not the occupant extracted from the captured image has experienced slowing of perception, a posture determination unit 17 that determines whether or not the posture of the occupant extracted from the captured image is abnormal, and a sleep detection unit 18 that detects the sleep of the occupant using the respective determination results of the slowing of perception determination unit 16 and the posture determination unit 17.

The image acquisition unit 11 acquires an image of an occupant inside the vehicle from the imaging device 20. It is preferable that the sleep detection device 10 performs a sleep detection process, which will be described later, each time it acquires an image from the imaging device 20. The image acquired by the image acquisition unit 11 is then output to the feature point detection unit 12, which will be described next.

The feature point detection unit 12 extracts feature points related to the body organs of the occupant from the captured image acquired by the image acquisition unit 11. The feature point extraction process by the feature point detection unit 12 can use various known algorithms, and detailed descriptions of these algorithms will be omitted. For example, when extracting feature points related to the occupant's face, the feature point detection unit 12 executes a process of detecting multiple feature points corresponding to each of multiple facial parts (e.g., right eye, left eye, right eyebrow, left eyebrow, nose, and mouth).

The detection process of the feature point detection unit 12 will be described. FIG. 3 is an explanatory diagram showing the detection process of the feature point detection unit 12 of the sleep detection device 10 according to embodiment 1. FIG. 3A is an explanatory diagram of the captured image 300, FIG. 3B is an explanatory diagram of the face area 301, and FIG. 3C is an explanatory diagram of the feature points. The feature point detection unit 12 acquires the captured image 300 shown in FIG. 3A from the image acquisition unit 11. Then, the feature point detection unit 12 detects the face area 301, which is an area including facial parts, from the captured image 300 acquired from the image acquisition unit 11. For example, the feature point detection unit 12 detects the face area 301 shown in FIG. 3B from the captured image 300 of FIG. 3A.

The feature point detection unit 12 acquires position information of components of face parts such as the outer corners 311, inner corners 312, upper eyelids 313, and lower eyelids 314 of the eyes in the detected face region 301, as shown in FIG. 3C. The feature point detection unit 12 also acquires position information of components of face parts such as the root 315, tip 316, back of the nose, and wings of the nose in the face region 301, as shown in FIG. 3C. The feature point detection unit 12 also acquires position information of components of face parts such as the upper lip 317, lower lip 318, and corners of the mouth 319 in the face region 301, as shown in FIG. 3C. The position information of the components of each face part acquired by the feature point detection unit 12 is information indicating coordinates starting from a specific position O in the captured image 300 shown in FIG. 3A, or coordinates starting from the center of the captured image 300 shown in FIG. 3A. The position information acquired by the feature point detection unit 12 is detected as a feature point. The feature points detected by the feature point detection unit 12 may be recorded in a storage unit (not shown) of the sleep detection device 10.

The feature amount calculation unit 13 then calculates the feature amount of the occupant from the feature points extracted by the feature point detection unit 12. Here, the feature amount of the occupant is specifically information indicating the positional relationship between the feature points of the occupant, such as the distance between the right eye and the left eye, the position of the nose on the occupant's face, etc. In addition, the feature amount calculation unit 13 may acquire position information of the face area 301, such as the size and coordinates of the face area 301 in the captured image 300. Various known algorithms can also be used to calculate the feature amount. The feature amount calculated by the feature amount calculation unit 13 may also be recorded in the memory unit of the sleep detection device 10.

The face direction detection unit 14 detects the face direction of the occupant in the captured image acquired by the image acquisition unit 11 using the features calculated by the feature calculation unit 13. Here, for example, the face direction of the occupant may be detected by detecting at least one of the roll angle, pitch angle, and yaw angle of the occupant's face relative to a reference position. The face direction detected by the face direction detection unit 14 may also be recorded in the memory unit of the sleep detection device 10.

The face direction detection process of the face direction detection unit 14 will now be described. Figure 4 is an explanatory diagram showing the face direction detection process of the face direction detection unit 14 of the sleep detection device 10 relating to embodiment 1. For example, as shown in Figures 4A to 4C, the face direction detection unit 14 acquires the positional relationship between the tip of the nose 316 and both eyes. In addition, the face direction detection unit 14 detects the face direction of the occupant from the positional relationship between the tip of the nose 316 and both eyes.

In the case of Fig. 4A, the face direction detection unit 14 detects that the occupant's face is facing to the right because the tip of the nose 316 is located near a straight line Qa passing through the outer corner 311 of the left eye. In the case of Fig. 4B, the face direction detection unit 14 detects that the occupant's face is facing forward because the tip of the nose 316 is located between the canthi 312. In the case of Fig. 4C, the face direction detection unit 14 detects that the occupant's face is facing to the left because the tip of the nose 316 is located near a straight line Qb passing through the outer corner 311 of the right eye. In this way, the face direction detection unit 14 detects the left-right direction of the occupant's face, i.e., the yaw angle of the occupant's face.

As shown in FIG. 4B, the face direction detection unit 14 detects the up-down direction of the occupant's face, i.e., the pitch angle of the occupant's face, by using a straight line Qc passing through the point where the nose tip 316 is located. The straight line Qc may be set, for example, from an image in which the occupant faces forward in both the up-down and left-right directions. Various known algorithms may be used for the face direction detection process of the occupant by the face direction detection unit 14, and various known algorithms may be used for the roll angle detection process of the occupant's face. In addition, it is preferable to use at least one of the roll angle and pitch angle of the occupant's face for the determination process of the occupant's posture by the posture determination unit 17 described later. When an occupant falls into instantaneous sleep, the occupant nods, collapses, or leans back. As a result, when an occupant falls momentarily asleep, at least one of the roll angle and pitch angle of the face changes, and therefore, by using at least one of the roll angle and pitch angle in the posture determination process described below, the reliability of the posture determination result can be improved.

The facial orientation change amount calculation unit 15 calculates the amount of change in the facial orientation of the occupant using the facial orientation of the occupant detected by the facial orientation detection unit 14. The amount of change in the facial orientation of the occupant is, for example, the difference between the pitch angle of the occupant's face relative to a reference position detected from the captured image in frame A and the pitch angle of the occupant's face relative to a reference position detected from the captured image in frame B. Here, the amount of change in the facial orientation of the occupant includes not only the amount of change in the angle of the occupant's face, but also the amount of change in the position or size and coordinates of the occupant's face area in the captured image. Note that the amount of change may be not only the difference between each parameter, but also the difference from the average value calculated from each parameter detected in multiple captured images, or may be the median and standard deviation calculated from each parameter detected in multiple captured images.

Next, the slowing down determination unit 16 will be described. The slowing down determination unit 16 determines whether or not a slowing down of perception has occurred in the occupant extracted from the captured image. Here, the occurrence of a slowing down of perception in the occupant means that the occupant is in a state where he or she does not show a strong reaction to external stimuli. When an occupant falls into momentary sleep, it is known that the occupant experiences a slowing of perception before this momentary sleep. Therefore, in order to accurately detect the momentary sleep of an occupant, it is necessary to determine whether or not a slowing down of perception has occurred in the occupant.

When an occupant experiences sensory dulling, they will exhibit characteristic behavioral changes, such as less facial movement compared to when they are awake. This behavioral change means that the occupant no longer shows a strong reaction to external stimuli, and occupants with sensory dulling will perform fewer voluntary movements compared to when they are awake.

An example will be described in which the slowing determination unit 16 uses the facial direction of the occupant to determine whether or not the occupant's perception has been slowed. For the sake of explanation, the determination of whether or not the occupant's perception has been slowed is also referred to as a slowing determination. When the amount of change in the facial direction is less than a set threshold (hereinafter referred to as the slowing determination threshold) continues for a set time (hereinafter referred to as the slowing determination time), the slowing determination unit 16 determines that the occupant's perception has been slowed, assuming that the occupant's voluntary actions have decreased. On the other hand, when the amount of change in the facial direction is equal to or greater than the slowing determination threshold, or when the amount of change in the facial direction is less than the slowing determination threshold, the unit 16 determines that the occupant's perception has not been slowed, assuming that the occupant is voluntary.

When a dulled state occurs in an occupant, the state in which the occupant is no longer able to make voluntary movements (hereinafter referred to as the dulled state) continues for a predetermined period of time. On the other hand, even if the occupant is awake, there may be cases in which the occupant changes the direction of their face less while driving. Therefore, if the dulled state continues for a set dulling judgment time or longer, the result of the dulling judgment can be reflected in the sleep detection process described below, thereby improving the accuracy of detecting whether the occupant is sleeping.

The slowing judgment unit 16 acquires, for example, the standard deviation of the pitch angle of the occupant's face during a set period from the face direction change amount calculation unit 15 as the amount of change in the face direction. Then, if the acquired standard deviation is less than the slowing judgment threshold and the state in which the standard deviation is less than the slowing judgment threshold continues for more than the slowing judgment time, that is, if the period during which the standard deviation is less than the slowing judgment threshold is more than the slowing judgment time, the slowing judgment unit 16 judges that the occupant's perception has been slowed. On the other hand, if the acquired standard deviation is more than the slowing judgment threshold, the slowing judgment unit 16 judges that the occupant's perception has not been slowed. In addition, even if the period during which the acquired standard deviation is less than the slowing judgment threshold is less than the slowing judgment time, the slowing judgment unit 16 judges that the occupant's perception has not been slowed. Furthermore, the slowing judgment unit 16 outputs the judgment result to the sleep detection unit 18 described later.

Next, the posture determination unit 17 will be described. The posture determination unit 17 determines whether or not the posture of the occupant extracted from the captured image is abnormal. When an occupant falls asleep momentarily, the occupant assumes an abnormal posture such as nodding, collapsing, or leaning back. Therefore, the posture determination unit 17 determines whether or not the occupant is in an abnormal posture, and the determination result of the posture determination unit 17 is used in the sleep detection process described below.

The posture determination unit 17 determines whether or not the posture of the occupant extracted from the captured image is abnormal, for example, by using the amount of change in face direction calculated by the face direction change amount calculation unit 15. Then, the posture determination unit 17 outputs the determination result to the sleep detection unit 18. For the sake of explanation, the determination by the posture determination unit 17 of whether or not the posture of the occupant is abnormal is also referred to as posture determination.

An example of the posture determination by the posture determination unit 17 will be described. For example, the posture determination unit 17 acquires the maximum and minimum values of the pitch angle of the occupant's face as the amount of change in the face direction from the facial direction change amount calculation unit 15 within a set time (hereinafter referred to as the posture determination time). Then, the posture determination unit 17 determines whether the posture of the occupant is abnormal or not based on whether the amount of change in the face direction is equal to or greater than a set threshold (hereinafter referred to as the posture determination threshold) within the posture determination time. That is, the posture determination unit 17 acquires from the facial direction change amount calculation unit 15 the maximum value of the pitch angle relative to the reference position of the occupant's face direction within the posture determination time, and the minimum value of the pitch angle relative to the reference position of the occupant's face direction. Then, if the difference between the minimum value and the maximum value is equal to or greater than the posture determination threshold, the posture determination unit 17 determines that the posture of the occupant is abnormal. On the other hand, if the difference between the minimum value and the maximum value is less than the posture determination threshold, the posture determination unit 17 determines that the posture of the occupant is not abnormal.

Here, even if the occupant does not fall asleep instantly, for example, there are cases where the occupant slowly changes the direction of the face to ensure a good forward field of vision. On the other hand, if the posture determination unit 17 performs posture determination using the amount of change in the direction of the face within the posture determination time as described above, it is possible to prevent erroneous detection that the occupant has fallen asleep when the occupant does not fall asleep instantly and instead slowly changes the direction of the face.

The posture determination unit 17 may also perform posture determination using the size of the face area. When an occupant falls asleep momentarily, the occupant may lean back in a direction away from the image capture device 20, or nod or collapse in a direction toward the image capture device 20. At this time, a change occurs in the size of the detected face area of the occupant. For this reason, the posture determination unit 17 may, for example, acquire the size of the face area as a feature from the feature calculation unit 13, and determine that the posture of the occupant is abnormal if the size of the face area is outside a set range, and determine that the posture of the occupant is not abnormal if the size of the face area is within the set range. In this case, the posture determination unit 17 may be connected to the feature calculation unit 13 without being connected to the face direction change amount calculation unit 15.

Furthermore, the posture determination unit 17 may perform posture determination using the distance between the imaging device 20 and the occupant. As described above, when an occupant falls asleep momentarily, the occupant may lean back in a direction away from the imaging device 20, or nod or collapse in a direction toward the imaging device 20. For this reason, the imaging device 20 may be configured as an imaging device 20 capable of capturing an image that reflects the distance between the imaging device 20 and a subject, such as a TOF camera, and distance data indicating the distance between the occupant and the imaging device 20, which is included in the captured image, may be used to determine whether the posture is abnormal.

For example, the posture determination unit 17 determines that the posture of the occupant is abnormal if the distance between the face of the occupant included in the captured image acquired by the image acquisition unit 11 and the imaging device 20 is outside a set range, and determines that the posture of the occupant is not abnormal if the distance between the face of the occupant and the imaging device 20 is within a set range. In this case, the feature calculation unit 13 is configured to be able to calculate the distance between the face of the occupant included in the captured image and the imaging device 20, and the posture determination unit 17 and the feature calculation unit 13 are connected.

The attitude determination unit 17 can also determine whether the attitude is abnormal by using distance data indicating the distance between the occupant and a sensor, which is acquired from a sensor capable of acquiring the distance from the occupant, such as a depth sensor provided in the vehicle. That is, in the above example, the TOF camera is also included in the depth sensor.

The sleep detection unit 18 detects whether the occupant is asleep using the judgment results of the slowing down judgment unit 16 and the judgment results of the posture judgment unit 17. When the slowing down judgment unit 16 judges that the occupant has a slowing of perception and the posture judgment unit 17 judges that the posture of the occupant is abnormal, the sleep detection unit 18 detects that the occupant has fallen asleep.

Then, the sleep detection unit 18 outputs the detection result to a vehicle control device (not shown) that controls the in-vehicle equipment mounted on the vehicle, and activates the notification unit 50 mounted on the vehicle to make a notification. The notification unit 50 may also be provided in the sleep detection device 10. The sleep detection device 10 and the vehicle control device are connected. Here, the notification unit 50 may be, for example, at least one of a speaker or a display mounted on the vehicle, or a mobile terminal carried by the occupant. When the sleep detection unit 18 detects that the occupant has fallen asleep, the notification unit 50 issues a notification by emitting a warning sound to prevent the occupant from falling asleep while driving. When the notification is made to a mobile terminal carried by the occupant, the sleep detection device 10 may be connected to a communication unit (not shown) and communication may be performed between the communication unit and the mobile terminal. When the sleep detection unit 18 detects that the occupant has fallen asleep, the vehicle control device may control in-vehicle equipment such as air conditioning to eliminate drowsiness.

The results of occupant sleep detection by the sleep detection unit 18 will now be described. Figure 5 is an explanatory diagram showing the sleep detection results of the sleep detection device 10 of embodiment 1. Figure 5A is a diagram showing the sleep detection results in a comparative example, and Figure 5B is a diagram showing the sleep detection results of the sleep detection device 10 of this embodiment. The vertical axis of Figures 5A and 5B shows the detection results, and the horizontal axis shows time. In Figures 5A and 5B, the points indicate that the occupant was detected as having fallen asleep at the time they are plotted.

In the comparative example of Figure 5A, the sleep detection process detects the sleep of an occupant based only on the judgment result of the posture judgment unit 17. When an occupant is in a vehicle, the direction of their face changes due to various factors. At time t1 shown in Figure 5, the occupant actually falls asleep for a moment. However, as shown in Figure 5A, in the comparative example, the occupant is detected as having fallen asleep at times other than t1, and the accuracy of detecting the sleep of the occupant is poor.

On the other hand, in the sleep detection device 10 according to the present embodiment, if the posture determination unit 17 determines that the posture of the occupant is abnormal after the slowing determination unit 16 determines that the occupant has experienced a slowing of perception, the occupant is determined to have fallen asleep. Therefore, as shown in FIG. 5B, if the occupant does not fall into instantaneous sleep but simply changes the direction of his/her face, the occupant is not erroneously detected as having fallen asleep, and it is possible to detect that the occupant has fallen asleep only for the time t1 when the occupant actually fell into instantaneous sleep. In this way, by using the determination results of the posture determination unit 17 and the determination results of the slowing determination unit 16 in the sleep detection process, the accuracy of detecting the sleep of the occupant can be improved. Note that the same results can be obtained even if the posture determination unit 17 uses the size of the face area or the distance between the occupant and the sensor instead of the direction of the face, and the occupant's sleep can be detected with high accuracy, as shown in FIG. 5B.

Next, an example of the operation of the sleep detection device 10 will be described. Figures 6 and 7 are flowcharts each showing an example of the operation of the sleep detection device 10 according to embodiment 1. Below, an example will be described in which the sleep detection device 10 performs a slowing down judgment and a posture judgment using the facial direction of the occupant. In addition, although the flowcharts in Figures 6 and 7 do not show the process of terminating the operation of the sleep detection device 10, the sleep detection device 10 terminates its operation when it receives a command to terminate its operation from the vehicle control device. Note that the process shown in Figures 6 and 7 is repeated, for example, at a predetermined interval, while the power supply of the sleep detection device 10 is turned on.

First, the image acquisition unit 11 of the sleep detection device 10 acquires an image from the imaging device 20 (ST1). Then, the image acquisition unit 11 outputs the acquired image to the feature point detection unit 12. Next, the feature point detection unit 12 uses the captured image to detect facial features of the occupant in the captured image and extracts feature points of the occupant's face (ST2). Then, the feature amount calculation unit 13 uses the extracted feature points to calculate the feature amount of the occupant extracted from the captured image (ST3). For the sake of explanation, the processes ST1 to ST3 shown in Figure 6 are hereinafter collectively referred to as feature amount calculation process ST101.

An example of the operation of the sleep detection device 10 will be described with reference to FIG. 7. After calculating the features of the occupant in the feature calculation process ST101, the face direction detection unit 14 acquires the features and detects the face direction of the occupant extracted from the captured image (ST102). The face direction detection unit 14 then outputs the detected face direction of the occupant to the face direction change amount calculation unit 15. Next, the face direction change amount calculation unit 15 calculates the amount of change in the face direction of the occupant using the face direction of the occupant acquired from the face direction detection unit 14 (ST103). Here, the amount of change in the face direction may be recorded in the memory unit.

Next, the slowing determination unit 16 determines whether or not the occupant's perception is being slowed (ST104). The slowing determination unit 16 acquires, as the amount of change, for example, the standard deviation of the pitch angle indicating the occupant's facial direction over a set period from the facial direction change amount calculation unit 15. If the acquired amount of change is equal to or greater than the slowing determination threshold, the slowing determination unit 16 determines that the occupant's perception is not being slowed (ST104; NO) and outputs the determination result to the sleep detection unit 18. Next, the processing of the sleep detection device 10 proceeds to ST101. On the other hand, if the acquired amount of change is less than the slowing determination threshold, the slowing determination unit 16 determines that the occupant is in a state in which perception is being slowed (ST104; YES) and proceeds to the processing of the next ST105.

Next, the slowing judgment unit 16 judges whether the state in which the occupant's perception is being slowed has continued for more than the slowing judgment time (ST105). Here, the slowing judgment unit 16 judges that the state in which the occupant's perception is being slowed has continued for more than the slowing judgment time, for example, if the state in which the acquired standard deviation is less than the slowing judgment threshold has continued for more than the slowing judgment time.

If the slowing judgment unit 16 determines that the state in which the occupant's perception is being slowed has not continued for more than the slowing judgment time (ST105; NO), it judges that the occupant's perception is not being slowed, and the processing of the sleep detection device 10 proceeds to ST101. On the other hand, if the slowing judgment unit 16 determines that the state in which the occupant's perception is being slowed has continued for more than the slowing judgment time (ST105; YES), it judges that the occupant's perception is being slowed, and outputs the judgment result to the sleep detection unit 18. Then, the processing of the sleep detection device 10 proceeds to ST106, which will be described next.

Next, the posture determination unit 17 of the sleep detection device 10 acquires the amount of change in the facial direction of the occupant from the facial direction change amount calculation unit 15 (ST106). For example, the posture determination unit 17 acquires the amount of change in the facial direction from the facial direction change amount calculation unit 15, and determines whether the posture of the occupant is abnormal based on whether the amount of change in the facial direction is equal to or greater than the posture determination threshold (ST107). For example, the posture determination unit 17 acquires, as the amount of change, the maximum value of the pitch angle relative to the reference position of the facial direction of the occupant during the set period, and the minimum value of the pitch angle relative to the reference position of the facial direction of the occupant. Then, if the difference between the minimum value and the maximum value is less than the posture determination threshold, the posture determination unit 17 determines that the posture of the occupant is not abnormal (ST107; NO), and the processing of the sleep detection device 10 then proceeds to ST101.

On the other hand, if the difference between the minimum and maximum pitch angle values obtained as the amount of change is greater than or equal to the posture determination threshold, the posture determination unit 17 determines that the occupant's posture is abnormal (ST107; YES), and the processing of the sleep detection device 10 proceeds to ST108, which will be described next.

The posture determination unit 17 determines whether the posture of the occupant is determined to be abnormal within the posture determination time (ST108). For example, the posture determination unit 17 refers to frame A, which is an image in which the pitch angle of the occupant's face is at its maximum, and frame B, which is an image in which the pitch angle of the occupant's face is at its minimum, and calculates the time that has elapsed between frame A and frame B. If the calculated elapsed time is equal to or less than the posture determination time, it is determined that the amount of change in the occupant's facial direction during the posture determination time has exceeded the posture determination threshold. Note that the above-mentioned process can be omitted if the facial direction change amount calculation unit 15 calculates the minimum and maximum pitch angles during the posture determination time.

If the posture determination unit 17 determines that the occupant's posture is abnormal not within the posture determination time (ST108; NO), the processing of the sleep detection device 10 proceeds to ST101. That is, if the processing of ST108 determines that the occupant's posture is abnormal not within the posture determination time, the determination result that the occupant's posture is abnormal in the processing of ST107 is rejected. On the other hand, if the posture determination unit 17 determines that the occupant's posture is abnormal within the set posture determination time (ST108; YES), it determines that the occupant's posture is abnormal and outputs the determination result to the sleep detection unit 18. Then, the sleep detection unit 18 detects that the occupant has fallen asleep, and the alarm unit 50 notifies the occupant by emitting an alarm sound or the like (ST109).

Next, a hardware configuration for realizing the functions of the sleep detection device 10 will be described. FIG. 8 is a diagram showing an example of a hardware configuration of the sleep detection device 10 according to the first embodiment. The functions of the image acquisition unit 11, the feature point detection unit 12, the feature amount calculation unit 13, the face direction detection unit 14, the face direction change amount calculation unit 15, the slowing down determination unit 16, the posture determination unit 17, and the sleep detection unit 18 in the sleep detection device 10 are realized by a processing circuit. That is, the image acquisition unit 11, the feature point detection unit 12, the feature amount calculation unit 13, the face direction detection unit 14, the face direction change amount calculation unit 15, the slowing down determination unit 16, the posture determination unit 17, and the sleep detection unit 18 of the sleep detection device 10 may be a processing circuit 10a, which is dedicated hardware, as shown in FIG. 8A, or a processor 10b that executes a program stored in the memory 10c as shown in FIG. 8B.

8A, when the image acquisition unit 11, the feature point detection unit 12, the feature amount calculation unit 13, the face direction detection unit 14, the face direction change amount calculation unit 15, the slowing down determination unit 16, the posture determination unit 17, and the sleep detection unit 18 are dedicated hardware, the processing circuit 10a corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-programmable Gate Array), or a combination thereof. The functions of each unit of the image acquisition unit 11, the feature point detection unit 12, the feature amount calculation unit 13, the face direction detection unit 14, the face direction change amount calculation unit 15, the slowing down determination unit 16, the posture determination unit 17, and the sleep detection unit 18 may be realized by a processing circuit, or the functions of each unit may be realized by a single processing circuit.

8B, when the image acquisition unit 11, the feature point detection unit 12, the feature amount calculation unit 13, the face direction detection unit 14, the face direction change amount calculation unit 15, the slowing down determination unit 16, the posture determination unit 17, and the sleep detection unit 18 are processor 10b, the functions of each unit are realized by software, firmware, or a combination of software and firmware. The software or firmware is described as a program and stored in memory 10c. Processor 10b realizes the functions of the image acquisition unit 11, the feature point detection unit 12, the feature amount calculation unit 13, the face direction detection unit 14, the face direction change amount calculation unit 15, the slowing down determination unit 16, the posture determination unit 17, and the sleep detection unit 18 by reading and executing the program stored in memory 10c. That is, the image acquisition unit 11, the feature point detection unit 12, the feature amount calculation unit 13, the face direction detection unit 14, the face direction change amount calculation unit 15, the slowing down determination unit 16, the posture determination unit 17, and the sleep detection unit 18 include a memory 10c for storing a program which, when executed by the processor 10b, results in the execution of each step shown in Fig. 4. It can also be said that these programs cause a computer to execute the procedures or methods of the image acquisition unit 11, the feature point detection unit 12, the feature amount calculation unit 13, the face direction detection unit 14, the face direction change amount calculation unit 15, the slowing down determination unit 16, the posture determination unit 17, and the sleep detection unit 18.

Here, the processor 10b is, for example, a CPU (Central Processing Unit), a processing device, an arithmetic device, a processor, a microprocessor, a microcomputer, or a DSP (Digital Signal Processor). The memory 10c may be, for example, a non-volatile or volatile semiconductor memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable ROM), or an EEPROM (Electrically EPROM), or may be a magnetic disk such as a hard disk or a flexible disk, or may be an optical disk such as a mini disk, a CD (Compact Disc), or a DVD (Digital Versatile Disc).

In addition, the functions of the image acquisition unit 11, feature point detection unit 12, feature amount calculation unit 13, face direction detection unit 14, face direction change amount calculation unit 15, slowing down determination unit 16, posture determination unit 17, and sleep detection unit 18 may be partially realized by dedicated hardware and partially realized by software or firmware. In this way, the processing circuit 10a in the sleep detection device 10 can realize each of the above-mentioned functions by hardware, software, firmware, or a combination of these. In addition, at least some of the functions of the image acquisition unit 11, feature point detection unit 12, feature amount calculation unit 13, face direction detection unit 14, face direction change amount calculation unit 15, slowing down determination unit 16, posture determination unit 17, and sleep detection unit 18 may be executed by an external server.

In this way, the sleep detection device 10 includes an image acquisition unit 11 that acquires an image of the occupant from an imaging device 20 that is mounted on the vehicle and captures an image of the occupant inside the vehicle, a slowing of perception determination unit 16 that determines whether the occupant extracted from the image acquired by the image acquisition unit 11 has experienced a dulling of perception, a posture determination unit 17 that determines whether the posture of the occupant extracted from the image acquired by the image acquisition unit 11 is abnormal, and a sleep detection unit 18 that detects the occupant's sleep using the determination results of the slowing of perception determination unit 16 and the posture determination unit 17. When the posture determination unit 17 determines that the posture of the occupant is abnormal and the slowing of perception determination unit 16 determines that the occupant has experienced a dulling of perception, the sleep detection unit 18 detects momentary sleep of the occupant and can improve the accuracy of detection of occupant sleep.

In this embodiment, an example has been described in which the face direction detection unit 14 detects the face direction using the position of the nose on the face of the occupant, but the facial parts that the face direction detection unit 14 uses to detect the face direction are not limited. For example, the face direction detection unit 14 may obtain the position of the right eye, the position of the left eye, and the distance between the right eye and the left eye as features from the feature calculation unit 13 and use them to detect the face direction. In addition, the face direction detection unit 14 may detect feature points related to organs of the body other than the face, such as the position of the occupant's shoulders.

Embodiment 2.
FIG. 9 is a block diagram showing a configuration example of a sleep detection system 101 according to the second embodiment. As in the first embodiment, the sleep detection device 30 according to the second embodiment includes an image acquisition unit 11 for acquiring an image, a desensitization determination unit 32 for determining whether or not the occupant extracted from the image has a desensitization, a posture determination unit 17 for determining whether or not the posture of the occupant extracted from the image is abnormal, and a sleep detection unit 18 for detecting the sleep of the occupant using the determination results of the desensitization determination unit 32 and the determination results of the posture determination unit 17. This embodiment is different from the first embodiment in that the desensitization determination unit 32 determines whether or not the occupant has a desensitization using face information of the occupant extracted from the image. The same components as those in the first embodiment are denoted by the same reference numerals, and the description thereof will be omitted.

Before an occupant falls into momentary sleep, for example, compared to when the level of alertness is high, there are changes in behavior due to a dulling of perception, such as less blinking, a slower blinking rate, less eye opening, and less change in facial expression. Therefore, the determination of the dulling of perception can be performed not only using the amount of change in face direction described in the first embodiment, but also using face information, which is information about the state of the face of the occupant.

First, the facial information used in the slowing down judgment will be described. Facial information is information related to the facial state of the occupant, such as the degree of eye opening, blinking frequency, changes in facial expression, etc. As shown in FIG. 9, the sleep detection device 30 according to this embodiment includes a facial information detection unit 31. The facial information detection unit 31 detects the facial information of the occupant using the features of the occupant acquired from the feature calculation unit 13.

An example of the detection process of face information by the face information detection unit 31 will be described. When acquiring the degree of eye opening as face information, the face information detection unit 31 acquires, for example, the distance between the upper eyelid and the lower eyelid as a feature from the feature calculation unit 13, and detects this distance as face information. When acquiring the speed of blinking as face information, the face information detection unit 31 acquires, for example, the distance between the upper eyelid and the lower eyelid as the feature calculation unit 13. Then, when this distance is less than a set threshold, it is determined that a blink has occurred, and when this distance is equal to or greater than the set threshold, it is determined that the eyes are open, and acquires, as face information, the time that has elapsed from the state in which the eyes are open until the blink occurs. Furthermore, when acquiring the frequency of blinking as face information, the face information detection unit 31 acquires, for example, the distance between the upper eyelid and the lower eyelid as the feature calculation unit 13. Then, when this distance is less than a set threshold, it is determined that a blink has occurred, and acquires, as face information, the time that has elapsed between the previous blink and the next blink. The face information detection process by the face information detection unit 31 is not limited to the above example, and various known algorithms can be used.

Next, an example of a slowing judgment made by the slowing judgment unit 32 will be described. The slowing judgment unit 32 acquires face information from the face information detection unit 31, and uses the face information to judge whether or not the occupant has experienced a slowing of perception. For example, when the slowing judgment unit 32 acquires the degree of eye opening as face information from the face information detection unit 31, if the state in which the degree of eye opening is less than the slowing judgment threshold continues for more than the slowing judgment time, the slowing judgment unit 32 judges that the occupant has experienced a slowing of perception. On the other hand, if the degree of eye opening is greater than or equal to the slowing judgment threshold, or if the state in which the degree of eye opening is less than the slowing judgment threshold does not continue for more than the slowing judgment time, the slowing judgment unit 32 judges that the occupant has not experienced a slowing of perception.

In addition, for example, when the slowing determination unit 32 acquires the blinking rate as face information from the face information detection unit 31, if the blinking rate continues to be less than the slowing determination threshold for a slowing determination time or more, it determines that the occupant has experienced a slowing of perception. On the other hand, if the blinking rate is greater than or equal to the slowing determination threshold, or if the blinking rate does not continue to be less than the slowing determination threshold for a slowing determination time or more, the slowing determination unit 32 determines that the occupant has not experienced a slowing of perception.

Furthermore, for example, when the slowing determination unit 32 acquires the blinking frequency as face information from the face information detection unit 31, if the blinking frequency is below the slowing determination threshold and continues for a slowing determination time or longer, the slowing determination unit 32 determines that the occupant has experienced a slowing of perception. On the other hand, if the blinking frequency is above the slowing determination threshold or if the blinking frequency is below the slowing determination threshold and does not continue for a slowing determination time or longer, the slowing determination unit 32 determines that the occupant has not experienced a slowing of perception.

The slowing judgment threshold is different from the slowing judgment threshold in the first embodiment. That is, the slowing judgment threshold in the first embodiment is a threshold for the amount of change in the face direction, whereas the slowing judgment threshold in the second embodiment is a threshold for the face information. The slowing judgment threshold to be compared with the degree of eye opening is a threshold for the degree of eye opening, the slowing judgment threshold to be compared with the speed of blinking is a threshold for the speed of blinking, and the slowing judgment threshold to be compared with the frequency of blinking is a threshold for the frequency of blinking. Similarly, the slowing judgment time may be different for each judgment. Furthermore, the slowing judgment threshold may be a predetermined value, or may be a value set based on the face information acquired from the time of getting on until a predetermined time has elapsed. This is because the passenger's level of wakefulness is considered to be high when getting on, and the accuracy of the slowing judgment can be improved by using the face information acquired when the level of wakefulness is high as the standard.

Next, an example of the operation of the sleep detection device 30 will be described. Figure 10 is a flowchart showing an example of the operation of the sleep detection device 30 according to embodiment 2. Hereinafter, steps that are the same as those in the processing of the sleep detection device 30 according to embodiment 1 are given the same reference numerals as those shown in Figures 6 and 7, and descriptions thereof will be omitted or simplified.

First, after the features are calculated in the feature calculation process ST101, the face information detection unit 31 detects face information using the features acquired from the feature calculation unit 13 (ST201). Below, an example will be described in which the face information detection unit 31 detects the degree of eye opening as face information, and the slowing judgment unit 32 performs the slowing judgment using the degree of eye opening and a slowing judgment threshold value related to the degree of eye opening.

Next, the slowing judgment unit 32 judges whether or not the occupant's perception is being slowed (ST202). The slowing judgment unit 32 acquires, for example, from the face information detection unit 31, the degree of eye opening of the occupant during the period set as face information. If the acquired eye opening degree is equal to or greater than the slowing judgment threshold, the slowing judgment unit 32 judges that the occupant is not experiencing a slowing of perception (ST202; NO) and outputs the judgment result to the sleep detection unit 18. Next, the processing of the sleep detection device 30 proceeds to ST101. On the other hand, if the acquired eye opening degree of the occupant is less than the slowing judgment threshold, the slowing judgment unit 32 judges that the occupant is in a state in which a slowing of perception is occurring (ST202; YES) and proceeds to the processing of the next ST105.

Next, the dulling determination unit 32 determines whether the state in which the occupant's perception is dulled has continued for more than the dulling determination time (ST105). If the dulling determination unit 32 determines that the state in which the occupant's perception is dulled has not continued for more than the dulling determination time (ST105; NO), it determines that the occupant has not experienced dulling of perception, and the processing of the sleep detection device 30 proceeds to ST101. On the other hand, if the dulling determination unit 32 determines that the state in which the occupant's perception is dulled has continued for more than the dulling determination time (ST105; YES), it determines that the occupant has experienced dulling of perception, and outputs the determination result to the sleep detection unit 18. Then, the processing of the sleep detection device 30 proceeds to the next ST102.

The face direction detection unit 14 acquires the feature amount and detects the face direction of the occupant extracted from the captured image (ST102). Then, the posture determination unit 17 of the sleep detection device 30 acquires the amount of change in the face direction of the occupant from the face direction change amount calculation unit 15 (ST106). The posture determination unit 17 acquires the amount of change in the face direction from the face direction change amount calculation unit 15, and determines whether the posture of the occupant is abnormal or not depending on whether the acquired amount of change is equal to or greater than the posture determination threshold (ST107). If the acquired amount of change is less than the posture determination threshold, the posture determination unit 17 determines that the posture of the occupant is not abnormal (ST107; NO) and outputs the determination result to the sleep detection unit 18. Next, the processing of the sleep detection device 30 proceeds to ST101. Note that, as in the first embodiment, the posture determination by the posture determination unit 17 does not use the face direction, and it is also possible to use the size of the face area and the distance between the occupant and the sensor.

On the other hand, if, for example, the difference between the minimum and maximum pitch angle values is greater than or equal to a set threshold value, the posture determination unit 17 determines that the occupant's posture is abnormal (ST107; YES), and the processing of the sleep detection device 30 proceeds to ST108, which will be described next.

The posture determination unit 17 determines whether the posture of the occupant is determined to be abnormal within the posture determination time (ST108). If the posture determination unit 17 determines that the abnormality of the occupant's posture is not a determination made within the posture determination time (ST108; NO), it determines that the occupant's posture is not abnormal, and the processing of the sleep detection device 30 proceeds to ST101. In other words, if it is determined in the processing of ST108 that the abnormality of the occupant's posture is not a determination made within the posture determination time, the determination result that the occupant's posture is abnormal in the processing of ST107 is rejected.

On the other hand, if the posture determination unit 17 determines that the occupant's posture is abnormal within the posture determination time (ST108; YES), it outputs the determination result to the sleep detection unit 18. Then, the sleep detection unit 18 detects that the occupant has fallen asleep, and the notification unit 50 notifies the occupant (ST109).

In this way, if the sleep detection device 30 further includes a face information detection unit 31 that detects facial information related to the facial state of the occupant using the features calculated by the feature calculation unit 13, and the slowing down determination unit 32 uses the facial information detected by the face information detection unit 31 to determine whether or not the occupant is experiencing a slowing of perception, the slowing of perception that occurs before the occupant falls asleep can be determined from the state of the occupant's face, and the occupant's momentary sleep can be detected, thereby improving the accuracy of detecting the occupant's sleep.

In this embodiment, an example has been described in which the slowing determination unit 32 acquires the degree of eye opening as face information from the face information detection unit 31 and uses the degree of eye opening for the slowing determination, but the face information used by the slowing determination unit 32 for the slowing determination is not limited to the degree of eye opening. For example, the slowing determination unit 32 may acquire the degree of eye opening, blink speed, and blink frequency as face information from the face information detection unit 31, and determine that the occupant has a slowed perception if at least any of the acquired face information is below a set threshold.

Embodiment 3.
FIG. 11 is a block diagram showing a configuration example of a sleep detection system 102 according to a third embodiment. As in the first embodiment, the sleep detection device 40 according to the third embodiment includes an image acquisition unit 11 for acquiring an image, a slowing determination unit 16 for determining whether or not the occupant extracted from the image has a dulled sense, a posture determination unit 17 for determining whether or not the posture of the occupant extracted from the image is abnormal, and a sleep detection unit 42 for detecting sleep of the occupant using the determination results of the slowing determination unit 16 and the posture determination unit 17. In this embodiment, the sleep detection unit 42 is different from the first embodiment in that it determines whether or not the occupant has a dulled sense by using the calculation result of the sleepiness level of the sleepiness level calculation unit 41 in addition to the determination results of the slowing determination unit 16 and the posture determination unit 17. The same components as those in the first embodiment are denoted by the same reference numerals, and their description will be omitted.

The sleep detection device 40 of this embodiment includes a face information detection unit 31 and a drowsiness level calculation unit 41 that calculates the drowsiness level of the occupant. Note that the configuration of the face information detection unit 31 is the same as that described in embodiment 2, and therefore description thereof is omitted. Here, the scale of drowsiness divided into stages from a high alertness state to a low alertness state is called drowsiness level. In this disclosure, the highest alertness state is drowsiness level 1, and the lowest alertness state is drowsiness level 9, and as the alertness level decreases, the drowsiness level transitions from drowsiness level 1 to drowsiness level 9.

When a passenger's perception is dulled, the passenger may be in a high drowsiness state. Therefore, the sleep detection unit 42 uses the drowsiness level calculation result of the drowsiness level calculation unit 41 in addition to the judgment results of the dulling judgment unit 16 and the posture judgment unit 17, thereby improving the reliability of the detection result of the passenger's sleep.

The calculation of the drowsiness level by the drowsiness level calculation unit 41 will be described. The drowsiness level calculation unit 41 acquires face information from the face information detection unit 31 and calculates the drowsiness level of the occupant. For example, when acquiring the degree of eye opening as face information, the drowsiness level calculation unit 41 acquires the degree of eye opening from the face information detection unit 31, and calculates the drowsiness level of the occupant to be higher as the degree of eye opening decreases. In this case, multiple thresholds are set for the degree of eye opening, and the drowsiness level is calculated based on whether or not it is below each threshold.

In addition, for example, when the blink rate is acquired as face information, the drowsiness level calculation unit 41 acquires the blink rate from the face information detection unit 31, and calculates the drowsiness level of the occupant to be higher as the blink rate decreases. In this case, multiple thresholds are set for the blink rate, and the drowsiness level is calculated based on whether the blink rate is less than each threshold.

Furthermore, for example, when blink frequency is acquired as face information, the drowsiness level calculation unit 41 acquires the blink frequency from the face information detection unit 31, and calculates the drowsiness level of the occupant to be higher as the blink frequency decreases. In this case, multiple thresholds may be set for the blink frequency, and the drowsiness level may be calculated based on whether the blink frequency is below each threshold. Note that the drowsiness level calculation process by the drowsiness level calculation unit 41 is not limited to the above example, and various known algorithms may be used. The calculation result by the drowsiness level calculation unit 41 is then output to the sleep detection unit 42.

Next, the sleep detection unit 42 according to this embodiment will be described. The sleep detection unit 42 acquires the calculation result by the sleepiness level calculation unit 41, and if the drowsiness level of the occupant calculated by the sleepiness level calculation unit 41 is equal to or higher than the set drowsiness level, i.e., the set threshold value, detects the sleep of the occupant using the judgment result of the slowing judgment unit 16 and the judgment result of the posture judgment unit 17. Here, if the drowsiness level is equal to or higher than the drowsiness level at which the occupant is considered to be feeling drowsy, it is preferable that the sleep detection unit 42 detects the sleep of the occupant using the judgment result of the slowing judgment unit 16 and the judgment result of the posture judgment unit 17. As described above, when the occupant has a slowed sense of perception, the drowsiness level is high, that is, the occupant may feel drowsy. Then, if the drowsiness level is equal to or higher than the set threshold value, the slowing judgment unit 16 judges that the occupant has a slowed sense of perception, and the posture judgment unit 17 judges that the posture of the occupant is abnormal, the sleep detection unit 42 detects that the occupant has fallen asleep.

Next, an example of the operation of the sleep detection device 40 will be described. FIG. 12 is a flowchart showing an example of the operation of the sleep detection device 40 according to embodiment 3. Hereinafter, steps that are the same as those in the processing of the sleep detection device 40 according to embodiment 1 are given the same reference numerals as those shown in FIG. 6 and FIG. 7, and descriptions thereof will be omitted or simplified.

First, after calculating the features in the feature calculation process ST101, the face information detection unit 31 detects face information using the features acquired from the feature calculation unit 13 (ST301), and outputs the detected face information to the drowsiness level calculation unit 41. Below, an example will be described in which the face information detection unit 31 detects the degree of eye opening of the occupant as face information, and the drowsiness level calculation unit 41 calculates the drowsiness level of the occupant using the degree of eye opening.

Next, the drowsiness level calculation unit 41 calculates the drowsiness level of the occupant using the degree of eye opening and closing of the occupant acquired as face information (ST302), and outputs the calculated drowsiness level of the occupant to the sleep detection unit 42. Here, the drowsiness level calculation unit 41 acquires the degree of eye opening as face information from the face information detection unit 31, and calculates the drowsiness level of the occupant to be higher as the degree of eye opening becomes smaller.

Then, the sleep detection unit 42 determines whether the acquired drowsiness level is equal to or greater than the set threshold (ST303). If the sleep detection unit 42 determines that the drowsiness level is less than the set threshold (ST303; NO), the processing of the sleep detection device 40 proceeds to ST101. On the other hand, if the sleep detection unit 42 determines that the drowsiness level is equal to or greater than the set threshold (ST303; YES), the processing of the sleep detection device 40 proceeds to ST102.

Next, the face direction detection unit 14 acquires the features of the occupant and detects the face direction of the occupant in the captured image (ST102). The face direction detection unit 14 then outputs the detected face direction of the occupant to the face direction change amount calculation unit 15. Next, the face direction change amount calculation unit 15 calculates the amount of change in the face direction of the occupant using the face direction of the occupant acquired from the face direction detection unit 14 (ST103).

Next, the slowing determination unit 16 determines whether or not the occupant's perception is slowed (ST104). The slowing determination unit 16 acquires, for example, the amount of change in the occupant's facial direction from the facial direction change amount calculation unit 15. If the acquired amount of change in the facial direction is equal to or greater than the slowing determination threshold, the slowing determination unit 16 determines that the occupant's perception is not slowed (ST104; NO) and outputs the determination result to the sleep detection unit 42. Next, the processing of the sleep detection device 40 proceeds to ST101. On the other hand, if the acquired amount of change in the facial direction is less than the slowing determination threshold, the slowing determination unit 16 determines that the occupant is in a state in which perception is slowed (ST104; YES) and proceeds to the next processing of ST105. Note that, as in the second embodiment, the slowing determination by the slowing determination unit 16 can be made using face information instead of facial direction.

Next, the slowing judgment unit 16 judges whether the state in which the occupant's perception is slowed has continued for more than the slowing judgment time (ST105). If the slowing judgment unit 16 judges that the state in which the occupant's perception is slowed has not continued for more than the slowing judgment time (ST105; NO), the processing of the sleep detection device 40 proceeds to ST101. On the other hand, if the slowing judgment unit 16 judges that the state in which the occupant's perception is slowed has continued for more than the slowing judgment time (ST105; YES), it judges that the occupant's perception is slowed, outputs the judgment result to the sleep detection unit 18, and the processing of the sleep detection device 40 proceeds to ST106, which will be described next.

Next, the posture determination unit 17 of the sleep detection device 40 acquires the amount of change in the facial direction of the occupant from the facial direction change amount calculation unit 15 (ST106). The posture determination unit 17 acquires the amount of change in the facial direction from the facial direction change amount calculation unit 15, and determines whether the posture of the occupant is abnormal or not depending on whether the amount of change in the facial direction is equal to or greater than the posture determination threshold (ST107). If the amount of change in the facial direction is less than the posture determination threshold, the posture determination unit 17 determines that the posture of the occupant is not abnormal (ST107; NO) and outputs the determination result to the sleep detection unit 42. Next, the processing of the sleep detection device 40 proceeds to processing of ST101.

On the other hand, if the change in face direction is equal to or greater than the set threshold, the posture determination unit 17 determines that the occupant's posture is abnormal (ST107; YES) and proceeds to the processing of ST108, which will be described next. Note that, as in embodiment 1, the posture determination by the posture determination unit 17 does not have to use the face direction, but can also use the size of the face area and the distance between the occupant and the sensor.

The posture determination unit 17 determines whether the posture of the occupant is determined to be abnormal within the posture determination time (ST108). If the posture determination unit 17 determines that the determination that the posture of the occupant is abnormal is not made within the posture determination time (ST108; NO), it determines that the posture of the occupant is not abnormal, and the processing of the sleep detection device 40 proceeds to ST101. In other words, if it is determined in the processing of ST108 that the determination that the posture of the occupant is abnormal is not made within the posture determination time, the determination result that the posture of the occupant is abnormal in the processing of ST107 is rejected.

On the other hand, if the posture determination unit 17 determines that the occupant's posture is abnormal within the set posture determination time (ST108; YES), it determines that the occupant's posture is abnormal and outputs the determination result to the sleep detection unit 42. Then, the sleep detection unit 42 determines that the occupant has fallen asleep, and the alarm unit 50 issues an alarm to the occupant (ST109).

In this way, the sleep detection device 40 in this embodiment further includes a drowsiness level calculation unit 41 that calculates the drowsiness level of the occupant using face information related to the state of the occupant's face, and the sleep detection unit 42 detects that the occupant has fallen asleep if the drowsiness level of the occupant is determined to be equal to or higher than a set threshold, the slowing determination unit 16 determines that the occupant has a slowing of perception, and the posture determination unit 17 determines that the posture of the occupant is abnormal. In this way, momentary sleep of the occupant can be detected, and the accuracy of detecting sleep of the occupant can be improved, and if the calculation result of the drowsiness level is used, the reliability of the detection result of sleep of the occupant can be improved because the slowing of perception may occur in a state of low alertness.

In the first to third embodiments, the slowing judgment by the slowing judgment unit, the posture judgment by the posture judgment unit, and the calculation of the drowsiness level by the drowsiness level calculation unit have been described as being performed using the characteristic amounts of the occupant. However, the slowing judgment, posture judgment, and calculation of the drowsiness level are not limited to being performed using the characteristic amounts of the occupant. The slowing judgment, posture judgment, and calculation of the drowsiness level may be performed using, for example, machine learning. When the slowing judgment is performed using machine learning, the slowing judgment unit is provided with a learning device, and, for example, an image captured of the occupant when perception slowing occurs and an image captured of the occupant when perception slowing does not occur are input to the learning device for learning. Then, the learning device of the slowing judgment unit may be caused to acquire the captured image from the image acquisition unit, and it may be determined whether or not the occupant's perception in the captured image has been slowed.

In addition, when posture determination is performed using machine learning, a learning device is provided in the posture determination unit, and images of the occupant when their posture is abnormal, for example, when they are leaning back, nodding, or collapsing, and images of the occupant when their posture is normal are input to the learning device for learning. The learning device of the posture determination unit then acquires the captured images from the image acquisition unit, and determines whether the posture of the occupant in the captured images is abnormal or not.

Furthermore, when the drowsiness level is calculated using machine learning, the drowsiness level calculation unit is provided with a learning device, and images of occupants corresponding to the graded drowsiness levels are input to the learning device to learn the images. The learning device of the drowsiness level calculation unit then acquires the captured images from the image acquisition unit, and calculates the drowsiness level of the occupant in the captured images.

Furthermore, the embodiments disclosed in this specification may be freely combined within their scope, and each embodiment may be modified or omitted as appropriate.

10, 30, 40 Sleep detection device, 10a Processing circuit, 10b Processor, 10c Memory, 11 Image acquisition unit, 12 Feature point detection unit, 13 Feature amount calculation unit, 14 Face direction detection unit, 15 Face direction change amount calculation unit, 16, 32 Desensitization determination unit, 17 Posture determination unit, 18, 42 Sleep detection unit, 20 Imaging device, 31 Face information detection unit, 41 Drowsiness level calculation unit, 50 Notification unit, 100, 101, 102 Sleep detection system, 201 Driver's seat, 202 Passenger seat, 211, 212 Passenger, 300 Captured image, 301 Face area, 311 Eye corner, 312 Eye corner, 313 Upper eyelid, 314 Lower eyelid, 315 Nose root, 316 Nose tip, 317 Upper lip, 318 Lower lip, 319 corner of mouth.

Claims (12)

an image acquisition unit that acquires an image of an occupant from an imaging device that is mounted on a vehicle and captures an image of the occupant inside the vehicle;
a sensory desensitization determination unit that determines whether or not the occupant has sensory desensitization extracted from the captured image acquired by the image acquisition unit;
a posture determination unit that determines whether or not the posture of the occupant extracted from the captured image acquired by the image acquisition unit is abnormal;
a sleep detection unit that detects the sleep of the occupant by using a determination result of the deceleration determination unit and a determination result of the posture determination unit,
the slowing determination unit determines that the occupant has a slowed sense of perception when a state in which the index value indicating a change in the facial direction of the occupant is less than a slowing determination threshold continues for a slowing determination time or more after it has been determined that the index value is less than the slowing determination threshold,
The sleep detection device is characterized in that the sleep detection unit determines that the occupant has fallen into microsleep when the slowing determination unit determines that the occupant has experienced slowing of perception and the posture determination unit determines that the occupant's posture is abnormal. The sleep detection device according to claim 1, characterized in that the sleep detection unit determines that the occupant has fallen into microsleep when the posture determination unit determines that the posture of the occupant is abnormal after the slowing of perception determination unit determines that the occupant has experienced slowing of perception. The sleep detection device according to claim 1 or 2, characterized in that the posture determination unit determines that the posture of the occupant is abnormal if the distance between the occupant and the sensor acquired from the sensor that acquires the distance to the occupant is outside a set range. The sleep detection device according to claim 1 or claim 2, characterized in that the posture determination unit determines that the posture of the occupant is abnormal if the size of a face area, which is an area including facial parts of the occupant extracted from the captured image, is outside a set range. a face direction detection unit that detects a face direction of the occupant extracted from the captured image acquired by the image acquisition unit;
a face direction change amount calculation unit that calculates a change amount of the face direction of the occupant detected by the face direction detection unit,
The sleep detection device according to claim 1 or claim 2, characterized in that the posture determination unit determines that the posture of the occupant is abnormal if the amount of change in the facial direction of the occupant calculated by the facial direction change amount calculation unit within a set posture determination time becomes equal to or greater than a set posture determination threshold. a face direction detection unit that detects a face direction of the occupant extracted from the captured image acquired by the image acquisition unit;
a face direction change amount calculation unit that calculates a change amount of the face direction of the occupant detected by the face direction detection unit,
The sleep detection device of any one of claims 1 to 4, characterized in that the slowing determination unit determines that the occupant has experienced a slowing of perception when, after it is determined that the amount of change in the facial direction of the occupant calculated as the index value by the facial direction change amount calculation unit is less than the slowing determination threshold, the state in which the amount of change in the facial direction of the occupant is less than the slowing determination threshold continues for a set slowing determination time or more. a face direction detection unit that detects a face direction of the occupant extracted from the captured image acquired by the image acquisition unit;
a face direction change amount calculation unit that calculates a change amount of the face direction of the occupant detected by the face direction detection unit,
the slowing determination unit determines that the occupant has a slowed perception when a state in which the amount of change in the facial direction of the occupant calculated as the index value by the facial direction change amount calculation unit is less than the slowing determination threshold continues for more than the slowing determination time,
The sleep detection device according to claim 1 or claim 2, characterized in that the posture determination unit determines that the posture of the occupant is abnormal if the amount of change in the facial direction of the occupant calculated by the facial direction change amount calculation unit within a set posture determination time becomes equal to or greater than a set posture determination threshold. an image acquisition unit that acquires an image of an occupant from an imaging device that is mounted on a vehicle and captures an image of the occupant inside the vehicle;
a sensory desensitization determination unit that determines whether or not the occupant has sensory desensitization extracted from the captured image acquired by the image acquisition unit;
a posture determination unit that determines whether or not the posture of the occupant extracted from the captured image acquired by the image acquisition unit is abnormal;
a sleep detection unit that detects whether the occupant is asleep by using a determination result of the deceleration determination unit and a determination result of the posture determination unit;
a face information detection unit that detects face information related to a face state of the occupant extracted from the captured image acquired by the image acquisition unit,
the face information detection unit detects at least one of an eye opening degree, a blinking speed, and a blinking frequency of the occupant as the face information;
the slowing determination unit determines that the occupant has a slowed sense of perception when at least one of the degree of eye opening, the blinking rate, and the blinking frequency of the occupant is determined to be less than a slowing determination threshold and the state in which the degree of eye opening, the blinking rate, and the blinking frequency of the occupant is less than the slowing determination threshold continues for a slowing determination time or more;
The sleep detection device is characterized in that the sleep detection unit determines that the occupant has fallen into microsleep when the slowing determination unit determines that the occupant has experienced slowing of perception and the posture determination unit determines that the occupant's posture is abnormal. a drowsiness level calculation unit that calculates a drowsiness level of the occupant by using the face information detected by the face information detection unit,
The sleep detection device according to claim 8, characterized in that the sleep detection unit determines that the occupant has fallen into microsleep when the drowsiness level calculation unit determines that the occupant's drowsiness level is equal to or higher than a set threshold, the dulling determination unit determines that the occupant is experiencing a dulling of perception, and the posture determination unit determines that the occupant's posture is abnormal. a face information detection unit that detects face information related to a state of the occupant extracted from the captured image acquired by the image acquisition unit;
a drowsiness level calculation unit that calculates a drowsiness level of the occupant by using the face information detected by the face information detection unit,
The sleep detection device according to any one of claims 1 to 7, characterized in that when the drowsiness level calculation unit determines that the occupant's drowsiness level is equal to or higher than a set threshold, the sleep detection unit determines that the occupant is experiencing a dulling of perception, and when the posture determination unit determines that the occupant's posture is abnormal, the sleep detection unit determines that the occupant has fallen into microsleep. The sleep detection device according to any one of claims 1 to 10, further comprising an alarm unit that is mounted on the vehicle and that notifies the occupant when the sleep detection unit detects that the occupant has fallen into microsleep. An imaging device mounted in a vehicle for imaging an occupant inside the vehicle;
an image acquisition unit that acquires an image of the occupant from the imaging device;
a sensory desensitization determination unit that determines whether or not the occupant has sensory desensitization extracted from the captured image acquired by the image acquisition unit;
a posture determination unit that determines whether or not the posture of the occupant extracted from the captured image acquired by the image acquisition unit is abnormal;
a sleep detection unit that detects the sleep of the occupant by using a determination result of the deceleration determination unit and a determination result of the posture determination unit,
the slowing determination unit determines that the occupant has a slowed sense of perception when a state in which the index value indicating a change in the facial direction of the occupant is less than a slowing determination threshold continues for a slowing determination time or more after it has been determined that the index value is less than the slowing determination threshold,
The sleep detection system is characterized in that the sleep detection unit determines that the occupant has fallen into microsleep when the slowing determination unit determines that the occupant has experienced slowing of perception and the posture determination unit determines that the occupant's posture is abnormal.

Images