CN104809445A - Fatigue driving detection method based on eye and mouth states - Google Patents

Abstract

The fatigue driving detection method based on the state of the eyes and the mouth belongs to the technical field of image processing and pattern recognition. The present invention includes the following steps: driver video image acquisition, illumination compensation preprocessing, face area detection, comprehensive fatigue judgment, and fatigue alarm ; Wherein the face area detection includes eye detection and mouth detection; eye detection includes using the projection method to obtain the eye area, for eye feature analysis, compared with standard features, k value calculation and eye fatigue judgment; The detection of the mouth includes using the mouth-map method to obtain the area of the mouth, analyzing the characteristics of the mouth, comparing with the standard features, calculating the p value and judging yawning; the present invention combines the two characteristic parameters of the eyes and the mouth to judge, and it is compared with the standard features. Compared with single parameter, the accuracy and reliability of fatigue judgment are higher. The implementation of the present invention can greatly reduce traffic accidents caused by driver fatigue driving, and provide a new preventive method to ensure the safety of driver's life and property. measure.

Description

Fatigue driving detection method based on eye and mouth state

technical field

The invention belongs to the technical field of image processing and pattern recognition, and in particular relates to a fatigue detection method based on the state of the driver's eyes and mouth.

Background technique

With the rapid development of the economy, the number of cars is increasing. While automobiles bring fast and convenient transportation to human beings, they also bury hidden dangers for traffic safety. Driver's fatigue driving is an important factor that causes traffic accidents. According to statistics, 20% of traffic accidents are caused by fatigue driving, so it is particularly important to provide real-time and accurate fatigue warnings to drivers.

After decades of research by experts and scholars, there are currently two types of detection methods for driver fatigue: contact and non-contact:

1. Contact type, based on the detection of the driver's physiological characteristics, this method needs to add some measuring equipment to the driver's body to detect the driver's physiological parameters, such as electrocardiogram, electroencephalogram, pulse, etc. When the driver is tired, these physiological signals will change, and the change of the measured value of the device is used to judge whether he is tired or not.

2. Non-contact, which is divided into two methods: detection based on vehicle behavior characteristics and detection based on driver behavior characteristics. Among them, ① detection based on vehicle behavior characteristics: when the driver is tired, the driving control ability of the vehicle will be reduced. For example, when it detects that the steering wheel does not move for a long time or changes frequently, abnormalities such as vehicle speed and turning angle are detected, the driver is likely to be in a state of fatigue. Although this method does not interfere with driving, it is difficult to guarantee the accuracy of the detection results due to differences in road conditions and drivers' driving habits. ② Detection based on the driver's behavior characteristics: By detecting the driver's eye closure, blink frequency, head position, etc., it can be judged whether the driver is fatigued. When the driver is in a state of fatigue, the most common physiological behavioral responses are closed eyes for a long time, reduced blinking frequency, longer blinking cycle, yawning, and abnormal head position, etc., using machine vision to detect the above physiological responses, and identify them after processing It can be judged whether the driver is tired or not.

Among the above two types of methods, the contact detection method based on the driver's physiological characteristics requires high precision and high cost of the detection equipment, and directly contacts with the driver, which will cause interference to driving. Although non-contact detection methods based on vehicle behavior characteristics will not interfere with driving, it is difficult to guarantee the accuracy of detection results due to differences in road conditions and drivers' driving habits. The detection method based on the driver's behavior characteristics has the advantages of no interference to the driver, high accuracy and low cost, and is the most widely used. This method usually detects the face with the help of image processing technology, then extracts the eyes, calculates the proportion of eye closure time per unit time based on the PERCLOS principle, and judges whether the driver is tired by comparing with the threshold. This method has high requirements for eye detection. Since the proportion of eyes on a human face is relatively small, the size of human eyes is also different, and the judgment element is single, which will cause unsatisfactory fatigue judgment results.

Contents of the invention

The purpose of the present invention is to provide a fatigue driving detection method based on the state of the eyes and mouth, which is improved and innovated on the basis of the previous fatigue detection method, so that the fatigue detection result is more ideal.

The fatigue driving detection method based on eye and mouth state of the present invention comprises the following steps:

1. Collect the driver's video stream and convert the video stream into a frame image;

2. Carry out the light compensation preprocessing of the image: use the "reference white" algorithm to first detect the brightness of the pixels in the image, get the pixels whose brightness values are in the top 5%, and set the gray value of the pixels whose brightness values are in the top 5%. is 255, and then linearly adjust the RGB three components of the image in proportion to obtain the image after illumination compensation;

3. Detecting the face area: For the light-compensated image obtained in step 2, distinguish skin-colored points and non-skinned-colored points based on the skin-color feature, obtain the binary image of the skin-colored area, and perform the mathematical morphology of the connectivity analysis on the binary image Processing; using the projection method to extract the face area;

4. Set the initial standard values of eye and mouth features: Assume that the driver is awake when entering the cab, process the image obtained at this moment, and use the obtained initial values of the eye state and mouth state as the standard value and save;

5. Extraction and feature analysis of the eye area: Horizontally and vertically project the binary image of the face area obtained in step 3, segment the eye area including the eyebrows, and then use the eye area features for state analysis , including the following steps:

5.1 Perform grayscale processing on the eye region including the eyebrows obtained in step 5 to obtain a grayscale image of the eye region, and calculate the mean value of the x coordinates of the grayscale image pixels to obtain the horizontal average intensity of the image pixels, There will be two obvious troughs on the average image. According to the difference in the distance d between the two troughs, it is judged whether the eyes are open or closed. The distance difference d ₀ between the two troughs obtained by the initial image processing is taken as Reference standard, if dd ₀ is greater than the set threshold, the eyes are judged as closed, otherwise they are normal;

5.2 Judgment of eye fatigue: Use k to record the number of image frames with eyes closed continuously, and add 1 to k every time the eyes are closed. If k is less than the threshold, if the eyes are detected to be open, then initialize k to 0; If it is greater than the threshold, it means that it is not blinking, but eye fatigue, where: k is an integer variable, counted by k, and the initial value of k is 0;

6. Extraction and feature analysis of the mouth area: take the lower half of the face area obtained in step 3, use the following mathematical expression to extract the mouth area, and then use the mouth area features to perform state analysis, specifically including the following steps :

$\mathrm{<span\; class="notranslate">\; mouth</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; map</span>}<span\; class="notranslate">\; =</span>{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; \&times;</span>{<span\; class="notranslate">\; (</span>{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; -</span>\frac{\mathrm{<span\; class="notranslate">\; \&eta;</span>}<span\; class="notranslate">\; \&times;</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}}{{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; b</span>}}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ;</span>$

in: C _r is the red chrominance component, C _b is the blue chrominance component, n is the number of pixels in the face area image, and η is the average value of C _r (x, y) ² The estimated ratio of the mean of ;

6.1 Use the binary image of the mouth area to calculate the area s of the mouth area; take the area of the mouth area s ₀ obtained from the initial image as a reference standard value, calculate the area s of the mouth area and the area of the mouth area s obtained from the initial image Ratio of ₀ If the ratio is greater than the set threshold, it is judged that the mouth is open, otherwise it is judged as normal; where: the ratio of the area of the mouth area s to the area of the mouth area s ₀ obtained from the initial image The ratio of the number of pixels can be used Instead, n ₀ is the number of pixels in the initial mouth area, and n is the number of pixels in the mouth area of the current frame image;

6.2 Judgment of yawning: Use the p value to record the number of image frames with the mouth open continuously, and add 1 to p every time the mouth is opened; if p is less than the threshold, if the mouth is detected to be normal, initialize p to 0; If it is greater than the threshold, it means that the driver is yawning and the driver is in a state of fatigue, where: p is an integer variable, counted by p, and the initial value of p is 0;

7. Comprehensive fatigue judgment: According to step 5.2 and step 6.2, when eye fatigue, yawning, or both are detected, a fatigue alarm is given to make the driver stop for a rest or replace the driver.

The above step 5 and step 6 are performed synchronously.

The present invention detects the fatigue state of the driver in combination with the two parameters of eye state and mouth state. Among them, when detecting the state of the eyes, the feature change relationship between the eyebrows and the eyes is used, and the eyes do not need to be detected accurately, which reduces the search range and is a new method for judging the state of the eyes. In addition, even if the driver wears sunglasses or glasses, combined with the detection of mouth features, the fatigue state of the driver will not be missed, making the fatigue detection effect more ideal.

The present invention combines the two characteristic parameters of eyes and mouth to judge, and compared with a single parameter, the accuracy and reliability of fatigue judgment are higher. The implementation of the present invention can greatly reduce traffic accidents caused by driver fatigue driving , in order to ensure the safety of the driver's life and property, a new preventive measure is provided.

Description of drawings

Fig. 1 is a flow chart of the fatigue driving detection method based on the state of eyes and mouth

Figure 2 is the grayscale image of the eyes in the open state

Figure 3 is the detection effect diagram when the eyes are normally open

Figure 4 is the eye grayscale image in the closed eye state

Figure 5 is the detection effect diagram when the eyes are closed

Detailed ways

The purpose, specific technical methods and effects of the present invention will be described below in conjunction with the accompanying drawings, so that those skilled in the art can better understand the present invention. The present invention has adopted eye and mouth features to detect whether the driver is tired, as shown in Figure 1, the method comprises the following steps:

1. Collect the driver's video stream and convert the video stream into a frame image.

2. Pre-processing the image with light compensation: The light will change with the driving environment and time, which has a great impact on the extraction of skin color features. Therefore, light compensation is performed first to better extract the face area. What is used is a "reference white" algorithm. First, the brightness of the pixels in the image is detected to obtain the pixels whose brightness values are in the top 5%, and the average gray value of the pixels whose brightness values are in the top 5% is set to 255, that is, These pixels are used as "reference white", and then linearly adjust the RGB three components of the image in proportion to obtain the image after illumination compensation.

3. Detecting face area: using skin color features to detect faces is a relatively simple and effective method. Convert the image from RGB to HSV and YCbCr color space for processing. In YCbCr color space, the brightness component Y and chrominance information CbCr are independent, and the skin color area can be extracted well by using the clustering characteristics of skin color. In the HSV color space, Hue has significantly different values in the skin-colored area and the non-skinned area. Use the following formula (1) to extract the skin color area:

C _r ≥140 and C _r ≤165 and C _b ≥140and C _b ≤195and

Hue≥0.01 and Hue≤0.1 (1)

For the light-compensated image obtained in step 2, use the above formula (1) to distinguish skin-colored points and non-skinned-colored points, set the skin-colored point to 1, and set the non-skinned point to 0 to obtain the binary image of the skin-colored area. The image is processed by mathematical morphology such as connectivity analysis; the projection method is used to find the boundary of the face, and then the face area is accurately extracted.

4. Set the initial standard value of eye and mouth features: take the feature of the driver’s waking state when he first enters the cab as a reference standard, save the feature value at this time, and compare it with the feature value detected during driving , and make a judgment on fatigue. Assuming that the driver is awake when he enters the cab, the image obtained at this moment is processed, and the obtained initial values of the eye state and mouth state are taken as standard values and saved.

5. Extraction and feature analysis of the eye area: judge the opening and closing of the eyes according to the height h from the eyelids to the eyebrows. When the eyes are closed, the eyelids move to the lowest part of the eyes, and the height h is the largest at this time; when the eyes are opened normally When the eyelid moves to the upper part of the eye, the height h is the smallest. Horizontally and vertically project the binary image of the face area obtained in step 3, segment the eye area including the eyebrows, and then use the above principle to analyze the state of the eye area features, specifically including the following steps:

5.1 Perform grayscale processing on the eye region including the eyebrows obtained in step 5 to obtain the grayscale image of the eye region, as shown in Figure 2 and Figure 4, calculate the mean value of the x coordinates of this grayscale image, in the mean value There will be two obvious troughs on the image, and according to the difference in distance d between the two troughs, it can be judged whether the eyes are open or closed. Combined with the analysis of Figure 3 and Figure 5, the distance d ₀ between the two troughs obtained by the initial image processing is used as a reference standard. If dd ₀ is greater than the set threshold, the eyes are judged as closed, otherwise they are normal .

5.2 Judgment of eye fatigue: The general duration of blinking is about 0.3 seconds. If the time exceeds this time, it means that the driver may be in a sleep state with eyes closed. Use the k value to record the number of image frames in which the eyes are closed continuously, and judge whether to blink or not by comparing the k value with the set threshold k ₀ . Set the initial value of k to 0 and increment k by 1 whenever eye closure is detected. When k is less than the threshold k ₀ , if it is detected that the eyes are open, then initialize k to 0; if k ₀ is greater than the threshold k ₀ , it means that it is not blinking but eye fatigue; where: k is an integer Type variable, counted by k, k ₀ is the number of image frames corresponding to the longest blink duration.

6. Extraction and feature analysis of the mouth area: the mouth is in the lower half of the face, and only the lower half of the face is used for detection to improve detection efficiency and accuracy. In the mouth area, red is the strongest, blue is the weakest, and there is a certain difference between lip color and skin color. For the face area obtained in step 3, take the lower half, and use the following mathematical expression (2) to extract the mouth Ministry area. When a person is in a state of fatigue, in addition to closing the eyes for a long time, it is also accompanied by the phenomenon of yawning. When yawning, the mouth opens greatly, and the area of the mouth area will be larger than normal, and the number of corresponding pixels will also be more than normal. Using the above principles to analyze the state of the characteristics of the mouth region, it specifically includes the following steps:

$\mathrm{<span\; class="notranslate">\; mouth</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; map</span>}<span\; class="notranslate">\; =</span>{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; \&times;</span>{<span\; class="notranslate">\; (</span>{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; -</span>\frac{\mathrm{<span\; class="notranslate">\; \&eta;</span>}<span\; class="notranslate">\; \&times;</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}}{{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; b</span>}}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; )</span>$

in: C _r is the red chrominance component, C _b is the blue chrominance component, n is the number of pixels in the face area image, and η is the average value of C _r (x, y) ² The estimated ratio of the mean of ;

6.1 Convert the extracted lip color area into a binary image, find out the maximum connected domain after erosion and expansion, and fill the mouth area with holes, and then calculate the area s of the mouth area; The area of the mouth area s ₀ is used as a reference standard value, and the ratio of the area s of the mouth area to the area of the mouth area s ₀ obtained from the initial image is calculated If the ratio is greater than the set threshold, it is judged that the mouth is open, otherwise it is judged as normal; where: the ratio of the area s of the mouth area to the area of the mouth area s ₀ obtained from the initial image The ratio of the number of pixels can be used Instead, n ₀ is the number of pixels in the initial mouth area, and n is the number of pixels in the mouth area of the current frame image.

6.2 Judgment of yawning: It is rare for a driver to open his mouth wide when speaking, and even if there is a need to open his mouth wide, it will not last for a long time, and it will be far shorter than the time of yawning. Under normal circumstances, the time for a person to yawn is about 5 seconds or more. Use the p value to record the number of image frames with the mouth open continuously. The initial value of p is 0. Every time p is detected to open the mouth, add 1; In the case of ₀ , if the mouth is detected to be normal, p is initialized to 0; in the case of p greater than the threshold p ₀ , it means that the driver is yawning and the driver is in a state of fatigue. Among them: p is an integer variable, counted by p, and p ₀ is the corresponding number of image frames within 5 seconds.

7. Comprehensive fatigue judgment: According to step 5.2 and step 6.2, when eye fatigue, yawning, or both are detected, a fatigue alarm is given to make the driver stop for a rest or replace the driver.

The above step 5.2 and step 6.2 are performed synchronously.

Claims (2)

1. A fatigue driving detection method based on eye and mouth state, is characterized in that comprising the following steps: 1.1 Collect the driver's video stream and convert the video stream into a frame image; 1.2 Carry out the preprocessing of image light compensation: use the "reference white" algorithm to first detect the brightness of the pixels in the image, obtain the pixels whose brightness values are in the top 5%, and set the gray value of the pixels whose brightness values are in the top 5% to be 255, and then linearly adjust the RGB three components of the image in proportion to obtain the image after illumination compensation; 1.3 Detecting the face area: For the light-compensated image obtained in step 1.2, distinguish between skin-colored points and non-skinned-colored points based on the skin-color feature, obtain the binary image of the skin-colored area, and perform mathematical morphology processing on the binary image for connectivity analysis ;Use the projection method to extract the face area; 1.4 Set the initial standard values of eye and mouth features: Assume that the driver is awake when entering the cab, process the image obtained at this moment, and use the obtained initial values of the eye state and mouth state as standard values and save; 1.5 Extraction and feature analysis of the eye area: perform horizontal and vertical projection on the binary image of the face area obtained in step 1.3, segment the eye area including the eyebrows, and then use the eye area features for state analysis, Specifically include the following steps: 1.5.1 Perform grayscale processing on the eye region including the eyebrows obtained in step 1.5 to obtain a grayscale image of the eye region, and calculate the average value of the x coordinates of the grayscale image pixels to obtain the horizontal average of the image pixels Intensity, there will be two obvious troughs on the mean image, according to the difference in the distance d between the two troughs, to judge whether the eyes are open or closed, the distance difference between the two troughs obtained by the initial image processing d ₀ as a reference standard, if dd ₀ is greater than the set threshold, the eyes are judged as closed, otherwise they are normal; 1.5.2 Judgment of eye fatigue: use k to record the number of image frames in which the eyes are continuously closed, and add 1 to k every time the eyes are closed. If k is less than the threshold, if the eyes are detected to be open, then initialize k to 0; In the case where k is greater than the threshold, it means that it is not blinking at this time, but eye fatigue, where: k is an integer variable, counted by k, and the initial value of k is 0; 1.6 Extraction and feature analysis of the mouth area: take the lower half of the face area obtained in step 1.3, extract the mouth area with the following mathematical expression, and then use the mouth area feature to perform state analysis, specifically including the following steps: $\mathrm{<span\; class="notranslate">\; mouth</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; map</span>}<span\; class="notranslate">\; =</span>{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; \&times;</span>{<span\; class="notranslate">\; (</span>{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; -</span>\frac{\mathrm{<span\; class="notranslate">\; \&eta;</span>}<span\; class="notranslate">\; \&times;</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}}{{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; b</span>}}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ;</span>$ in: C _r is the red chrominance component, C _b is the blue chrominance component, n is the number of pixels in the face area image, and η is the average value of C _r (x, y) ² The estimated ratio of the mean of ; 1.6.1 Use the binary image of the mouth area to calculate the area s of the mouth area; take the area of the mouth area s ₀ obtained from the initial image as a reference standard value, calculate the area s of the mouth area and the area of the mouth area obtained from the initial image ratio of area s ₀ If the ratio is greater than the set threshold, it is judged that the mouth is open, otherwise it is judged as normal; where: the ratio of the area of the mouth area s to the area of the mouth area s ₀ obtained from the initial image The ratio of the number of pixels can be used Instead, n ₀ is the number of pixels in the initial mouth area, and n is the number of pixels in the mouth area of the current frame image; 1.6.2 Judgment of yawning: use the p value to record the number of image frames with the mouth open continuously, and add 1 to p every time the mouth is opened; if p is less than the threshold, if the mouth is detected to be normal, then initialize p to 0; When p is greater than the threshold, it means that the driver is yawning and the driver is in a state of fatigue, where: p is an integer variable, counted by p, and the initial value of p is 0; 1.7 Comprehensive fatigue judgment: according to steps 1.5.2 and 1.6.2, when eye fatigue, yawning, or both are detected, a fatigue alarm is given to make the driver stop for a rest or replace the driver. 2. The fatigue driving detection method based on eye and mouth states according to claim 1, characterized in that said step 1.5 and step 1.6 are carried out simultaneously.