CN109934199A - A method and system for driver fatigue detection based on computer vision - Google Patents

Abstract

The invention discloses a method and system for detecting driver fatigue based on computer vision. First, the camera is called to collect the driver's face video; then the face is located through the AdaBoost classifier algorithm of Harr-like features; then the landmark is used to detect blinking and yawning to indicate fatigue; finally, the human eye is located through image preprocessing and according to the PERCLOS algorithm criteria Judging fatigue for early warning. The invention combines the PERCLOS human eye detection algorithm and the face key point extraction algorithm to perform fatigue warning for the driver, has non-contact and real-time performance, does not interfere with the normal driving of the driver, and makes the fatigue detection more accurate, efficient and fast.

Description

A method and system for driver fatigue detection based on computer vision

technical field

The invention relates to the technical field of image detection, in particular to a method and system for detecting driver fatigue based on computer vision.

Background technique

According to the statistics of the traffic department, the number of deaths due to road traffic accidents has continued to increase in recent years. Fatigue driving of motor vehicle drivers is an important cause of traffic accidents. Many experts and scholars are actively studying the methods of detecting fatigue driving, and how to effectively prevent fatigue driving has become a domestic issue. hotspots of foreign research.

Traditional driver behavior detection algorithms based on physiological signals and behavioral characteristics have their advantages and disadvantages. Among them, the detection algorithm based on physiological signal features has high accuracy, but it affects driving due to the contact between the sensor and the driver; the detection algorithm based on behavioral features does not require the driver to directly contact the detection device, and based on the existing device of the car, The equipment requirements are low and the practicability is strong, but the detection accuracy is not high.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a method and system for detecting driver fatigue based on computer vision, so as to solve the problem that the existing method for detecting driver fatigue cannot take into account both non-contact and detection accuracy.

For achieving the above object, the present invention provides the following scheme:

A method for detecting driver fatigue based on computer vision, the method comprising:

Get the driver's face video within a preset time period;

Use a face detection classifier to train the face video to generate multiple frames of face pictures;

Perform blinking and yawning detection according to the face picture, and generate fatigue danger prompt information;

According to the face picture and the fatigue danger prompt information, the PERCLOS algorithm is used to detect the fatigue state of the driver, and the fatigue state judgment result is generated;

If the fatigue state judgment result is that the driver is in a fatigue state, generating fatigue driving alarm information to give an alarm;

If the fatigue state determination result is that the driver is not in the fatigue state, returning to the step of acquiring the driver's face video within the preset time period.

Optionally, the detection of blinking and yawning is performed according to the face picture, and the fatigue danger prompt information is generated, which specifically includes:

The face recognition predictor is used to extract 68 face feature points on the face picture; the 68 face feature points include eye feature points corresponding to eye contours and mouth feature points corresponding to mouth contours;

Blink detection is performed according to the eye feature points to generate a blink detection result;

Perform yawn detection according to the mouth feature points to generate a yawn detection result;

Fatigue danger prompt information is generated according to the blink detection result and the yawn detection result.

Optionally, performing blink detection according to the eye feature points to generate a blink detection result specifically includes:

Calculate the eye opening degree of the human eye in each frame of the human face picture according to the eye feature points;

If the eye opening degrees in the face pictures in two consecutive frames are both less than the opening degree threshold, add 1 to the number of blinks;

If the number of blinks in one minute exceeds 20 times, it is determined that the blink detection result is that the number of blinks exceeds the blink warning value.

Optionally, the yawning detection is performed according to the mouth feature points to generate a yawning detection result, which specifically includes:

Calculate the mouth opening degree of the mouth in each frame of the face picture according to the mouth feature points;

Judging whether the number of consecutive frames in which the mouth opening degree is greater than the threshold of the mouth opening degree in the multiple frames of the face pictures exceeds the threshold of the number of frames, and obtains a yawn detection result;

If the number of consecutive frames in which the mouth opening degree is greater than the mouth opening degree threshold in the multiple frames of the face pictures exceeds the frame number threshold, it is determined that the yawning detection result is that the driver has a yawning action;

If the number of consecutive frames in which the mouth opening degree is greater than the mouth opening degree threshold in the multiple frames of the face picture does not exceed the frame number threshold, it is determined that the yawning detection result is that the driver does not yawn.

Optionally, generating fatigue danger prompt information according to the blink detection result and the yawn detection result specifically includes:

If the blink detection result is that the number of blinks exceeds the blink warning value, or the yawn detection result is that the driver has a yawning action, generating fatigue danger prompt information;

According to the fatigue risk prompt information, a fatigue risk prompt is performed to the driver.

Optionally, the PERCLOS algorithm is used to detect the fatigue state of the driver according to the face picture and the fatigue danger prompt information, and generate a fatigue state judgment result, which specifically includes:

Use a pattern matching algorithm to roughly locate the human eyes in the face picture, and obtain the eye pictures of the left eye and the right eye;

Preprocessing the eye image to generate a preprocessed eye image;

The Hough transform circle detection algorithm is used to accurately locate the human eye in the preprocessed eye image to generate an image of the precise position of the human eye;

Calculate the percentage of eye closure PERCLOS per unit time according to the image of the precise position of the human eye;

If the PERCLOS exceeds 80%, determine that the fatigue state judgment result is that the driver is in a fatigue state;

If the PERCLOS does not exceed 80%, it is determined that the fatigue state judgment result is that the driver is not in a fatigue state.

A computer vision-based driver fatigue detection system, the system includes:

A face video acquisition module, used to acquire the driver's face video within a preset time period;

a face picture generation module, used for training the face video by using a face detection classifier to generate multiple frames of face pictures;

A blinking and yawning detection module is used to detect blinking and yawning according to the face picture, and generate fatigue danger prompt information;

The fatigue state detection module is used for detecting the fatigue state of the driver by using the PERCLOS algorithm according to the face picture and the fatigue danger prompt information, and generating a fatigue state judgment result;

A fatigue alarm module, configured to generate fatigue driving alarm information for alarming if the fatigue state judgment result is that the driver is in a fatigue state;

The loop detection module is configured to return to the step of obtaining the driver's face video within a preset time period if the fatigue state judgment result is that the driver is not in the fatigue state.

Optionally, the blinking and yawning detection module specifically includes:

A face feature point extraction unit, used for extracting 68 face feature points on the face picture by using a face recognition predictor; the 68 face feature points include eye feature points corresponding to eye contours and corresponding mouth contours mouth feature points;

a blink detection unit, configured to perform blink detection according to the eye feature points, and generate a blink detection result;

a yawn detection unit, configured to perform yawn detection according to the mouth feature points, and generate a yawn detection result;

A fatigue risk prompt information generating unit, configured to generate fatigue risk prompt information according to the blink detection result and the yawn detection result.

Optionally, the blink detection unit specifically includes:

an eye opening degree calculation subunit, used for calculating the eye opening degree of the human eye in each frame of the face picture according to the eye feature points;

A subunit for calculating the number of blinks, for adding 1 to the number of blinks if the degree of eye opening in the face pictures in two consecutive frames is less than the threshold of the opening degree;

The blink detection result determination subunit is configured to determine that the blink detection result is that the blink count exceeds the blink warning value if the blink count exceeds 20 times in one minute.

Optionally, the yawning detection unit specifically includes:

The mouth opening degree calculation subunit is used to calculate the mouth opening degree of the mouth in each frame of the face picture according to the mouth feature points;

A yawning judging subunit, configured to judge whether the number of consecutive frames in which the mouth opening degree is greater than the mouth opening degree threshold in the multiple frames of the face pictures exceeds the frame number threshold, and obtain a yawn detection result;

The yawning action determination subunit is used to determine that the yawning detection result is that the driver has a yawning action if the continuous frame number of which the mouth opening degree is greater than the mouth opening degree threshold in the multiple frames of the face pictures exceeds the frame number threshold;

No yawning action determination subunit, for if the number of consecutive frames in which the mouth opening degree is greater than the mouth opening degree threshold in the multiple frames of the face picture does not exceed the frame number threshold, it is determined that the yawning detection result is that the driver does not have a yawn. Yawning action.

According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

The invention provides a computer vision-based driver fatigue detection method and system. The method generates multiple frames of face pictures according to the driver's face video collected by the camera, detects fatigue based on computer vision, and has non-contact and real-time features. It will not interfere with the normal driving of the driver, making the fatigue detection more convenient and quick; in the fatigue detection process, before accurately determining the driver's fatigue, the blinking and yawning detection is performed according to the face picture. The detection results can prompt the driver of the danger of fatigue, which can give an early warning before the driver is fatigued, effectively prevent the occurrence of dangerous accidents, and reduce the accident rate. After the fatigue danger is prompted according to the face picture, the present invention further adopts the PERCLOS algorithm to detect the fatigue state of the driver, which can accurately determine whether the driver has fatigue driving behavior, and improves the detection accuracy of fatigue driving.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings required in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some of the present invention. In the embodiments, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative labor.

Fig. 1 is the method flow chart of the driver fatigue detection method based on computer vision provided by the present invention;

Fig. 2 is the basic principle diagram of the driver fatigue detection method based on computer vision provided by the present invention;

3 is a schematic diagram of the principle of detecting blinking and yawning by landmark provided by the present invention;

4 is a schematic diagram of the positions of 68 facial feature points provided by the present invention;

5 is a schematic diagram of the principle of further processing an eye picture provided by the present invention;

Fig. 6 is the principle schematic diagram that the human eye is accurately searched by the transform circle algorithm provided by the present invention;

FIG. 7 is a system structure diagram of a computer vision-based driver fatigue detection system provided by the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

The purpose of the present invention is to provide a computer vision-based driver fatigue detection method and system, which determines the fatigue state according to the mechanism of human fatigue expression, and adopts the landmark-based detection algorithm for the driver's blinking and yawning to solve the problem of the contact between the sensor and the driver. It has real-time and high accuracy, and has high practicability.

In order to make the above objects, features and advantages of the present invention more clearly understood, the present invention will be described in further detail below with reference to the accompanying drawings and specific embodiments.

FIG. 1 is a method flowchart of a computer vision-based driver fatigue detection method provided by the present invention. FIG. 2 is a basic schematic diagram of a computer vision-based driver fatigue detection method provided by the present invention. 1 and 2, the computer vision-based driver fatigue detection method provided by the present invention specifically includes:

Step 101: Acquire a face video of a driver within a preset time period.

A camera is called to collect a video of the driver's face for a period of time; the camera is installed on the windshield directly in front of the driver.

The computer obtains the driver's face video collected by the camera within the preset time period, and based on the face video, the fatigue detection method based on computer vision is used to detect the driver's fatigue, which has the characteristics of non-contact and real-time, and will not It interferes with the normal driving of the driver and makes the fatigue detection more convenient and fast.

Step 102: Using a face detection classifier to train the face video to generate multiple frames of face pictures.

The AdaBoost face detection classifier algorithm with Harr-like features is used, and the face images are obtained by performing multiple face feature training, face region recognition and face localization through the face detection classifier.

The driver's face image collected by the camera is input into the trained Adaboost face detection classifier based on Haar feature, and then multiple frames of face pictures can be output.

Step 103: Perform blinking and yawning detection according to the face picture, and generate fatigue danger prompt information.

The present invention uses landmarks to detect blinking and yawning for the obtained multi-frame face pictures. FIG. 3 is a schematic diagram of the principle of detecting blinking and yawning through landmarks provided by the present invention. Referring to FIG. 3, the step 103 specifically includes:

Step 3-1, using a face recognition predictor to extract 68 face feature points on the face picture; the 68 face feature points include the eye feature points corresponding to the eye contour and the mouth feature corresponding to the mouth contour point.

By calling the dlib official face recognition predictor shape_predictor_68_face_landmarks.dat to mark the 68 feature points of the face. Through the face recognition predictor, 68 face feature points on the face picture can be extracted, and landmark is the face feature point. The positions of the 68 facial feature points are shown in Figure 4. The 68 facial feature points are marked in order to obtain 68 facial feature points labeled 0-68. The 68 face feature points include eye feature points corresponding to the contour of the eye and mouth feature points corresponding to the contour of the mouth. As shown in FIG. 4 , the six feature points labeled 36-41 are the eyes corresponding to the contour of the right eye. The six feature points marked 42-47 are the eye feature points corresponding to the left eye contour. The feature points numbered 48-57 are the feature points of the mouth corresponding to the outline of the mouth, and the 8 feature points numbered 60-67 are the feature points of the inner contour of the mouth.

Step 3-2, performing blink detection according to the eye feature points to generate a blink detection result.

The present invention utilizes 68 landmarks to extract the outline of the eye region to perform blink detection and calculation of blink times. The step 3-2 specifically includes:

Step 3-2-1, set the human eye calibration point, the left eye starts to the left eye, the right eye starts to the right eye, and the six feature points d1, d2, d3, d4 corresponding to the eyes in the face feature points are obtained. ,d5,d6. For example, the six feature points labeled 36-41 in FIG. 3 or the six feature points labeled 42-47 are used as the six feature points d1, d2, d3, d4, d5, and d6 used for the detection calculation.

Step 3-2-2: Calculate the eye opening degree of the human eyes in each frame of the face picture according to the eye feature points.

Calculate the eye aspect ratio as the eye opening, and the calculation formula is:

In formula (1), f is the eye opening degree, d1, d2, d3, d4, d5, d6 are the six feature points corresponding to the eyes in the face feature points (corresponding to the feature points labeled 36-41 in Figure 4). ), where |d2-d6|, |d3-d5| are the distances of the eye feature points in the vertical direction; |d1-d4| are the distances of the eye feature points in the horizontal direction, multiplied by 2 to ensure two sets of feature points weights are the same.

When the eye aspect ratio f is smaller than the opening degree threshold Radio=0.2, the blinking action must occur in two consecutive frames. This embodiment of the present invention takes two kinds of eye state data as examples to describe the process of judging whether to open or close the eyes, as shown in Table 1 below:

Table 1 Eye aspect ratio data when eyes are opened and closed

state |d2-d6| |d3-d5| |d1-d4| f eyes closed 8.06 9.0 48.04 0.178 open eyes 16.12 15.03 47.17 0.330

Step 3-2-3, if the eye opening degrees in the face pictures in two consecutive frames are both less than the opening degree threshold, add 1 to the number of blinks;

When the number of picture frames less than the threshold of opening degree exceeds two frames, the blinking action occurs; the count value of each blink is incremented by 1, so as to calculate the number of blinks, which is used as the indicator of fatigue danger.

Step 3-2-4. Count the number of blinks in one minute. If the number of blinks in one minute exceeds 20, it is determined that the blink detection result is that the number of blinks exceeds the blink warning value, and the driver is alerted to the danger of fatigue. .

Step 3-3, performing yawning detection according to the mouth feature points, and generating a yawning detection result; specifically including:

Step 3-3-1: Calculate the mouth opening degree of the mouth in each frame of the face picture according to the mouth feature points.

The outline of the mouth area is extracted by landmark for yawn detection, and the mouth opening is calculated to determine whether to yawn. The calculation formula of the mouth opening is:

In formula (2), α is the mouth opening degree, H is the maximum height of the inner contour of the mouth, L is the width of the inner contour of the mouth, and the calculation formula is:

H=max(|p2-p8|,|p3-p7|,|p4-p6|) (3)

L=|p1-p5| (4)

Among them, p1, p2, p3, p4, p5, p6, p7, p8 are the facial feature points corresponding to the inner contour feature points of the mouth (corresponding to the feature points labeled 60-67 in Figure 4), |p2-p8|, |p3-p7|, |p4-p6| are the distances in the vertical direction of the inner contour of the mouth, and |p1-p5| are the distances in the horizontal direction of the inner contour of the mouth.

Step 3-3-2: Determine whether the number of consecutive frames in which the mouth opening degree is greater than the threshold of the mouth opening degree in the multiple frames of the face pictures exceeds the threshold of the number of frames, and obtain a yawn detection result.

The invention adopts the double threshold method to determine the yawning behavior. When the mouth opening degree α is greater than the mouth opening degree threshold Z=0.65, and the mouth opening time exceeds the continuous frame number threshold of 125 frames, it is detected as yawning, which is used as a dangerous fatigue prompt indicator.

In the embodiment of the present invention, the data of two kinds of mouth states are used as examples to describe the judgment process of detecting yawning, as shown in Table 2 below:

Table 2 Mouth opening data for normal and yawning

Step 3-3-3, if the number of consecutive frames in which the mouth opening degree is greater than the mouth opening degree threshold in the multiple frames of the face pictures exceeds the frame number threshold, determine that the yawn detection result is that the driver has yawning action;

Step 3-3-4. If the number of consecutive frames in which the mouth opening degree is greater than the mouth opening degree threshold in the multiple frames of the face picture does not exceed the frame number threshold, it is determined that the yawning detection result is that the driver does not have a yawning action.

Step 3-4, generating fatigue danger prompt information according to the blink detection result and the yawn detection result, which specifically includes:

If the blink detection result is that the number of blinks exceeds the blink warning value, or the yawn detection result is that the driver has a yawning action, generating fatigue danger prompt information;

According to the fatigue risk prompt information, the driver is prompted for the fatigue risk, and the prompt method can be a voice prompt or a warning image jumping out and a sound prompt at the same time.

Step 104: According to the face picture and the fatigue danger prompt information, use the PERCLOS algorithm to detect the driver's fatigue state, and generate a fatigue state judgment result.

For the face picture obtained in the step 2, carry out picture preprocessing to locate the human eye, and use the PERCLOS algorithm to determine the fatigue state, which specifically includes:

Step 4-1, using a pattern matching algorithm to roughly locate the human eyes in the face picture, and obtain the eye pictures of the left eye and the right eye.

A simple contour area of the left and right eyes can be obtained through the coarse positioning of the human eye. The outer contours of the left and right eyes are set as two rectangular areas, L and R, respectively, and the images of the two rectangular areas are extracted as the left eye. and an eye picture of the right eye.

Step 4-2: Preprocess the eye image to generate a preprocessed eye image.

FIG. 5 is a schematic diagram of the principle of further processing an eye image provided by the present invention. Referring to FIG. 5 , further preprocessing is performed on the eye image, including grayscale processing, smoothing the image using Gauss filter, and performing edge detection on the pupil using the Canny operator to generate a preprocessed eye image.

Step 4-3, using the Hough transform circle detection algorithm to precisely locate the human eye in the preprocessed eye image to generate an image of the precise position of the human eye.

FIG. 6 is a schematic diagram of the principle of accurately searching the human eye through the transform circle algorithm provided by the present invention. Referring to FIG. 6 , the present invention performs precise positioning of the human eye and identification of the opening and closing states through the transform circle algorithm on the basis of Hough transform. The steps 4-3 specifically include:

Step 4-3-1. After obtaining the rectangular ranges L and R of the left and right eyes, use the Canny operator to perform edge detection in these two ranges (ie, the eye images of the left and right eyes). , the rough outline of the human eye is detected.

Step 4-3-2: Binarize the edge-detected image obtained in step 4-3-1 to obtain the radius area where the pupil is located.

Step 4-3-3, initialize the Hough matrix, initialize the accumulator array M, and the array M represents the coordinates of each pixel in the parameter space of the Cartesian coordinate system.

Step 4-3-4, draw circles in the two areas of the left eye L and the right eye R in the eye pictures of the left eye and the right eye; for drawing a circle, a special circle drawing algorithm can be used directly to extract the precise outline of the eye .

Step 4-3-5, in the accumulator M, calculate the maximum value of the accumulator M, the maximum value is the center coordinate of the position of the circle of the pupil to be located, and the center coordinate of the pupil obtained by positioning is M(0, R ), thus realizing the precise positioning of the human eye. R is the radius of the circle, the distance from the pupil's coordinates to the edge.

Step 4-4: Calculate the percentage of eye closure PERCLOS per unit time according to the image of the precise position of the human eye.

The present invention relies on PERCLOS P80 criterion to carry out fatigue judgment, and the calculation formula of PERCLOS (percentage of eyelidclosure over the pupil over time, the percentage of eye closure per unit time) is as follows:

Among them, P80 means that the eyelid covers more than 80% of the pupil as the eye is closed, and the proportion of the time when the eye is closed in a certain period of time is counted as the PERCLOS value.

The method for determining fatigue based on the PERCLOS P80 criteria is as follows:

Step 4-4-1. According to all the human eye images, record four time intervals: the time interval t1 from fully opening the human eye to closing p ₁ %, and the time interval from fully opening the human eye to closing p ₂ % t2, the interval time t3 from the time when the human eye is fully opened to the next opening p ₁ %, and the time interval t4 from the fully opening of the human eye to the next opening p ₂ %; wherein, p ₁ % and p ₂ % are both The ratio of the eyeball to the entire eye. where: p ₁ =20, p ₂ =80.

Step 4-4-2, according to t1, t2, t3, t4 to obtain the human eye PERCLOS index value PERCLOS, the formula used is:

Fatigue was judged when the PERCLOS value exceeded 80%.

Step 4-5, if the PERCLOS exceeds 80%, determine that the fatigue state judgment result is that the driver is in a fatigue state;

Step 4-6, if the PERCLOS does not exceed 80%, determine that the fatigue state judgment result is that the driver is not in a fatigue state.

Step 105: If the fatigue state determination result is that the driver is in a fatigue state, generate fatigue driving alarm information to give an alarm. The alarm method is to give an alarm prompt by sound.

If the fatigue state judgment result is that the driver is not in a fatigue state, return to step 101 of obtaining the driver's face video within a preset time period, and continue to perform real-time fatigue driving detection according to the face video.

The method of the invention judges blinking and yawning actions by extracting the key points of the human face, and prompts the driver when he is about to enter the fatigue danger period, which can effectively prevent fatigue driving accidents. At the same time, the opening degree of the human eye is calculated through the PERCLOS algorithm criterion, so as to provide early warning for the driver's fatigue detection, which is real-time and non-contact. In addition, combining the two for detection has high detection accuracy, reliability and practicability.

Based on the driver fatigue detection method provided by the present invention, the present invention also provides a computer vision-based driver fatigue detection system. FIG. 7 is a system structure diagram of the computer vision-based driver fatigue detection system provided by the present invention. Referring to Figure 7, the system includes:

A face video acquisition module 701, configured to acquire a driver's face video within a preset time period;

A face picture generation module 702, configured to train the face video by using a face detection classifier to generate multiple frames of face pictures;

The blinking and yawning detection module 703 is used to detect blinking and yawning according to the face picture, and generate fatigue danger prompt information;

The fatigue state detection module 704 is configured to detect the fatigue state of the driver by using the PERCLOS algorithm according to the face picture and the fatigue danger prompt information, and generate a fatigue state judgment result;

A fatigue alarm module 705, configured to generate fatigue driving alarm information for alarming if the fatigue state judgment result is that the driver is in a fatigue state;

The loop detection module 706 is configured to return to the step of obtaining the driver's face video within a preset time period if the fatigue state determination result is that the driver is not in the fatigue state.

Wherein, the blinking and yawning detection module 703 specifically includes:

A face feature point extraction unit, used for extracting 68 face feature points on the face picture by using a face recognition predictor; the 68 face feature points include eye feature points corresponding to eye contours and corresponding mouth contours mouth feature points;

a blink detection unit, configured to perform blink detection according to the eye feature points, and generate a blink detection result;

a yawn detection unit, configured to perform yawn detection according to the mouth feature points, and generate a yawn detection result;

A fatigue risk prompt information generating unit, configured to generate fatigue risk prompt information according to the blink detection result and the yawn detection result.

The blink detection unit specifically includes:

an eye opening degree calculation subunit, used for calculating the eye opening degree of the human eye in each frame of the face picture according to the eye feature points;

A subunit for calculating the number of blinks, for adding 1 to the number of blinks if the degree of eye opening in the face pictures in two consecutive frames is less than the threshold of the opening degree;

The blink detection result determination subunit is configured to determine that the blink detection result is that the blink count exceeds the blink warning value if the blink count exceeds 20 times in one minute.

The yawn detection unit specifically includes:

The mouth opening degree calculation subunit is used to calculate the mouth opening degree of the mouth in each frame of the face picture according to the mouth feature points;

A yawning judging subunit, configured to judge whether the number of consecutive frames in which the mouth opening degree is greater than the mouth opening degree threshold in the multiple frames of the face pictures exceeds the frame number threshold, and obtain a yawn detection result;

The yawning action determination subunit is used to determine that the yawning detection result is that the driver has a yawning action if the continuous frame number of which the mouth opening degree is greater than the mouth opening degree threshold in the multiple frames of the face pictures exceeds the frame number threshold;

No yawning action determination subunit, for if the number of consecutive frames in which the mouth opening degree is greater than the mouth opening degree threshold in the multiple frames of the face picture does not exceed the frame number threshold, it is determined that the yawning detection result is that the driver does not have a yawn. Yawning action.

The fatigue danger prompt information generating unit specifically includes:

a fatigue danger prompt information generating subunit, configured to generate fatigue danger prompt information if the blink detection result is that the number of blinks exceeds the blink warning value, or the yawning detection result is that the driver has a yawning action;

The fatigue risk prompting subunit is used for prompting the driver of the fatigue risk according to the fatigue risk prompt information.

The fatigue state detection module 704 specifically includes:

an eye image generation unit, configured to use a pattern matching algorithm to roughly locate the human eyes in the face image to obtain eye images of the left eye and the right eye;

a preprocessing unit, configured to preprocess the eye image to generate a preprocessed eye image;

a precise positioning unit, used for using the Hough transform circle detection algorithm to accurately locate the human eye in the preprocessed eye image, and generate an image of the precise position of the human eye;

a PERCLOS calculation unit, configured to calculate the percentage PERCLOS of eye closure per unit time according to the image of the precise position of the human eye;

A fatigue state determination unit, configured to determine that the fatigue state judgment result is that the driver is in a fatigue state if the PERCLOS exceeds 80%;

A not in fatigue state determination unit, configured to determine that the fatigue state judgment result is that the driver is not in a fatigue state if the PERCLOS does not exceed 80%.

Compared with the prior art, the present invention has the following significant advantages:

1) The present invention is based on the fatigue driving technology based on eye recognition, combined with the facial feature point extraction algorithm, before judging that the driver is fatigued, the fatigue danger prompt is firstly performed, so that there is an opportunity to prevent the occurrence of dangerous accidents before the driver is fatigued and reduce fatigue. the probability of a driving accident;

2) The present invention detects fatigue based on computer vision, has non-contact and real-time properties, does not interfere with the normal driving of the driver, and makes the fatigue detection more convenient and quick;

3) The present invention can be used for the detection of the fatigue state of various drivers and the early warning of the fatigue risk, and the scope of application is wider;

4) The present invention can directly call the face recognition library, detect facial expressions through the key points of the face, omits the tedious processing of images by traditional methods, has high reliability, and is flexible in application;

5) Precise positioning of the human eye through a variety of algorithms. In order to prevent the inaccuracy of the PERCLOS algorithm, it also provides early warning of fatigue through blinking and yawning detection, thereby further improving the accuracy of fatigue alarms.

The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the various embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant part can be referred to the description of the method.

In this paper, specific examples are used to illustrate the principles and implementations of the present invention. The descriptions of the above embodiments are only used to help understand the methods and core ideas of the present invention; meanwhile, for those skilled in the art, according to the present invention There will be changes in the specific implementation and application scope. In conclusion, the contents of this specification should not be construed as limiting the present invention.

Claims (10)

1. a kind of Driver Fatigue Detection based on computer vision, which is characterized in that the described method includes:

Obtain face's video of driver in preset time period；

Using face's video described in Face datection classifier training, multiframe face picture is generated；

The detection blinked and yawned according to the face picture generates tired dangerous tip information；

According to the face picture and the tired dangerous tip information, using the tired shape of PERCLOS algorithm detection driver State generates fatigue state judging result；

If the fatigue state judging result is that driver is in a state of fatigue, generates fatigue driving warning message and alarm；

If the fatigue state judging result is that driver is not in fatigue state, returns in the acquisition preset time period and drive The step of face's video of member.

2. Driver Fatigue Detection based on computer vision according to claim 1, which is characterized in that described The detection blinked and yawned according to the face picture generates tired dangerous tip information, specifically includes:

68 human face characteristic points in the face picture are extracted using recognition of face fallout predictor；68 human face characteristic point packets Include the eye feature point of corresponding eye profile and the mouth feature point of corresponding mouth profile；

Blink detection is carried out according to the eye feature point, generates blink detection result；

Detection of yawning is carried out according to the mouth feature point, generates testing result of yawning；

Tired dangerous tip information is generated according to the blink detection result and the testing result of yawning.

3. Driver Fatigue Detection based on computer vision according to claim 2, which is characterized in that described Blink detection is carried out according to the eye feature point, generates blink detection as a result, specifically including:

The eyes degree of opening of human eye in every frame face picture is calculated according to the eye feature point；

If eyes degree of opening described in face picture described in two continuous frames is respectively less than degree of opening threshold value, by number of winks plus 1；

If the number of winks is more than 20 times in one minute, determine that the blink detection result is number of winks more than blink early warning Value.

4. Driver Fatigue Detection based on computer vision according to claim 3, which is characterized in that described Detection of yawning is carried out according to the mouth feature point, testing result of yawning is generated, specifically includes:

The mouth opening of mouth in every frame face picture is calculated according to the mouth feature point；

Whether the continuous frame number for judging that mouth opening described in face picture described in multiframe is greater than mouth opening threshold value is more than frame number threshold value, Acquisition is yawned testing result；

If the continuous frame number that mouth opening described in face picture described in multiframe is greater than mouth opening threshold value is more than frame number threshold value, institute is determined Stating testing result of yawning is that driver has movement of yawning；

If the continuous frame number that mouth opening described in face picture described in multiframe is greater than mouth opening threshold value is less than frame number threshold value, determine The testing result of yawning is that there is no movements of yawning by driver.

5. Driver Fatigue Detection based on computer vision according to claim 4, which is characterized in that described Tired dangerous tip information is generated according to the blink detection result and the testing result of yawning, is specifically included:

If the blink detection result is that number of winks is more than blink early warning value or the testing result of yawning is driver In the presence of movement of yawning, then tired dangerous tip information is generated；

Tired dangerous tip is carried out to driver according to the tired dangerous tip information.

6. Driver Fatigue Detection based on computer vision according to claim 5, which is characterized in that described It is generated according to the face picture and the tired dangerous tip information using the fatigue state of PERCLOS algorithm detection driver Fatigue state judging result, specifically includes:

Coarse positioning is carried out to the human eye in the face picture using pattern matching algorithm, obtains the eye figure of left eye and right eye Piece；

The eye picture is pre-processed, pretreated eyes image is generated；

The human eye in the pretreated eyes image is accurately positioned using Hough transformation loop truss algorithm, generates people Eye exact position image；

According to the percentage PERCLOS of eyes closed in the human eye exact position image unit of account time；

If the PERCLOS is more than 80%, determine that the fatigue state judging result is that driver is in a state of fatigue；

If the PERCLOS is less than 80%, determine that the fatigue state judging result is that driver is not in fatigue state.

7. a kind of Study in Driver Fatigue State Surveillance System based on computer vision, which is characterized in that the system comprises:

Face's video acquiring module, for obtaining face's video of driver in preset time period；

Face picture generation module, for generating multiframe face picture using face's video described in Face datection classifier training；

It blinks and detection module of yawning, the detection for being blinked and being yawned according to the face picture generates fatigue Dangerous tip information；

Fatigue state detection module, for being calculated using PERCLOS according to the face picture and the tired dangerous tip information Method detects the fatigue state of driver, generates fatigue state judging result；

Fatigue warning module generates fatigue driving if being that driver is in a state of fatigue for the fatigue state judging result Warning message is alarmed；

Loop detection module obtains described in return if being that driver is not in fatigue state for the fatigue state judging result The step of taking face's video of driver in preset time period.

8. Study in Driver Fatigue State Surveillance System based on computer vision according to claim 7, which is characterized in that described to blink Eye and detection module of yawning specifically include:

Human face characteristic point extraction unit, for extracting 68 face characteristics in the face picture using recognition of face fallout predictor Point；68 human face characteristic points include the eye feature point of corresponding eye profile and the mouth feature point of corresponding mouth profile；

Blink detection unit generates blink detection result for carrying out blink detection according to the eye feature point；

Detection unit of yawning generates testing result of yawning for carrying out detection of yawning according to the mouth feature point；

Tired dangerous tip information generating unit, for being generated according to the blink detection result and the testing result of yawning Tired dangerous tip information.

9. Study in Driver Fatigue State Surveillance System based on computer vision according to claim 8, which is characterized in that described to blink Eye detection unit specifically includes:

Eyes degree of opening computation subunit, the eyes for calculating human eye in every frame face picture according to the eye feature point are opened Aperture；

Number of winks computation subunit, if being respectively less than degree of opening for eyes degree of opening described in face picture described in two continuous frames Threshold value, by number of winks plus 1；

Blink detection result determines subelement, if being more than 20 times for the number of winks in one minute, determines the blink inspection Surveying result is number of winks more than blink early warning value.

10. Study in Driver Fatigue State Surveillance System based on computer vision according to claim 9, which is characterized in that described Detection unit of yawning specifically includes:

Mouth opening computation subunit, for calculating the mouth opening of mouth in every frame face picture according to the mouth feature point；

It yawns judgment sub-unit, for judging that mouth opening described in face picture described in multiframe is greater than the continuous of mouth opening threshold value Whether frame number is more than frame number threshold value, obtains testing result of yawning；

It yawns and acts determining subelement, if being greater than the company of mouth opening threshold value for mouth opening described in face picture described in multiframe Continuous frame number is more than frame number threshold value, determine described in testing result of yawning be that driver has movement of yawning；

Determining subelement is acted without yawning, if being greater than mouth opening threshold value for mouth opening described in face picture described in multiframe Continuous frame number is less than frame number threshold value, and testing result of yawning described in determination is that there is no movements of yawning by driver.