CN107292251B - Driver fatigue detection method and system based on human eye state - Google Patents
Driver fatigue detection method and system based on human eye state Download PDFInfo
- Publication number
- CN107292251B CN107292251B CN201710430629.3A CN201710430629A CN107292251B CN 107292251 B CN107292251 B CN 107292251B CN 201710430629 A CN201710430629 A CN 201710430629A CN 107292251 B CN107292251 B CN 107292251B
- Authority
- CN
- China
- Prior art keywords
- driver
- eye
- image
- face
- rectangular
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
- G06V20/597—Recognising the driver's state or behaviour, e.g. attention or drowsiness
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
- G06V40/165—Detection; Localisation; Normalisation using facial parts and geometric relationships
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/18—Eye characteristics, e.g. of the iris
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- General Health & Medical Sciences (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Ophthalmology & Optometry (AREA)
- Geometry (AREA)
- Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
- Traffic Control Systems (AREA)
- Emergency Alarm Devices (AREA)
Abstract
The invention discloses a method and a system for detecting fatigue of a driver based on human eye state, which combine human face detection and human face tracking to determine an eye preliminary rectangular range of a driver image, preprocess the eye preliminary rectangular range, position an eye accurate rectangular range image by using contour search and rectangular fitting, judge whether the driver opens eyes or closes eyes in one frame of the driver image according to vertical projection, judge whether the driver blinks or continuously closes the eyes according to continuous frames of the continuous driver image in which the driver is in the eye closing state, and calculate the blinking frequency, thereby judging whether the driver is in the fatigue state. Has the advantages that: the method carries out different pre-processing on the preliminary rectangular range image of the eyes by selecting whether the driver wears the glasses or not and selecting different illumination intensities, so that the fatigue detection method is more accurate and has higher robustness; the method is suitable for different illumination conditions, and has higher real-time performance and higher detection speed.
Description
Technical Field
The invention relates to the technical field of computer vision processing, in particular to a driver fatigue detection method and system based on human eye states.
Background
Fatigue detection (Fatigue detection) is a research subject which is complex and has both theoretical and practical values, and is used for timely finding out Fatigue states and giving out early warning signals by monitoring various Fatigue characteristics of a human body, and relates to multiple fields of physiology, psychology, image processing, motion tracking, mode recognition and the like. During a long period of driving, the fatigue level of the driver gradually accumulates from shallow to deep. If the real-time detection of the state of the driver's essence can be realized by using technical means, and an early warning is given out immediately once fatigue signs appear, the safe driving coefficient can be effectively improved.
There are many methods for detecting the fatigue state of a driver, and the methods can be roughly classified into methods based on the physiological signal of the driver, methods based on the operation behavior of the driver, methods based on the vehicle state information, and methods based on the physiological reaction characteristic of the driver, according to the type of detection. The detection method based on the physiological response characteristics of the driver is non-contact detection, the fatigue is judged by using machine vision, the normal driving behavior of the driver cannot be interfered in the measurement process, and the method has great development potential. However, the existing method for detecting and analyzing the state of human eyes to detect the fatigue state of the driver is not accurate enough and has poor real-time performance.
Disclosure of Invention
The invention aims to overcome the technical defects, provides a method and a system for detecting the fatigue of a driver based on the human eye state, and solves the technical problems in the prior art.
In order to achieve the technical purpose, the invention provides a driver fatigue detection method based on human eye state, which comprises the following steps:
s1, acquiring a driver image in real time, performing face detection on the driver image, acquiring a face rectangular frame containing a face, and acquiring an eye preliminary rectangular range image in the face rectangular frame and coordinates of the eye preliminary rectangular range image in the face rectangular frame;
s2, after a face rectangular frame of one frame of driver image is obtained, face tracking is carried out on the subsequent driver image, and the face rectangular frame of the subsequent driver image is obtained;
s3, for the face rectangular frame of the subsequent driver image, acquiring an eye preliminary rectangular range image of the subsequent driver image according to the coordinates of the eye preliminary rectangular range image in the face rectangular frame;
s4, carrying out clipping and binarization preprocessing on the preliminary eye rectangular range image acquired in the S3;
s5, carrying out contour search on the eye preliminary rectangular range image subjected to binarization in S4, accurately positioning the eye accurate rectangular range image by utilizing rectangular fitting, and obtaining the coordinates of the eye accurate rectangular range image in a face rectangular frame;
s6, extracting a non-binarized accurate eye rectangular range image from the rectangular face frame according to the coordinates of the accurate eye rectangular range image in the rectangular face frame, binarizing the extracted accurate eye rectangular range image to form an open-closed eye judgment image, vertically projecting the open-closed eye judgment image to an X axis, and judging whether the driver opens the eyes or closes the eyes in one frame of driver image according to the proportion of black pixels and white pixels in the vertical projection;
s7, counting the continuous frame numbers of the continuous images of the driver with the closed eyes, judging whether the driver blinks or the eyes are continuously closed according to the range of the continuous frame numbers with the closed eyes, and calculating the blinking frequency;
and S8, if the blinking frequency is out of the set normal range, judging that the driver is in a fatigue state, and if the eyes of the driver are continuously closed, judging that the driver is in the fatigue state.
The invention also provides a driver fatigue detection system based on the human eye state, which comprises:
the face detection module: acquiring a driver image in real time, performing face detection on the driver image, acquiring a face rectangular frame containing a face, and acquiring an eye preliminary rectangular range image in the face rectangular frame and coordinates of the eye preliminary rectangular range image in the face rectangular frame;
a face tracking module: after a face rectangular frame of one frame of driver image is obtained, face tracking is carried out on a subsequent driver image, and the face rectangular frame of the subsequent driver image is obtained;
the eye preliminary range image acquisition module: for the face rectangular frame of the subsequent driver image, acquiring an eye preliminary rectangular range image of the subsequent driver image according to the coordinates of the eye preliminary rectangular range image in the face rectangular frame;
a preprocessing module: the preliminary eye rectangular range image acquired by the preliminary eye range image acquisition module is subjected to cutting and binarization preprocessing in sequence;
eye precision range image acquisition module: carrying out contour search on the eye preliminary rectangular range image subjected to binarization in the preprocessing module, accurately positioning the eye accurate rectangular range image by utilizing rectangle fitting, and acquiring coordinates of the eye accurate rectangular range image in a face rectangular frame;
an eye opening and closing judgment module: extracting a non-binarized accurate eye rectangular range image from the rectangular face frame according to the coordinates of the accurate eye rectangular range image in the rectangular face frame, binarizing the extracted accurate eye rectangular range image to form an open-closed eye judgment image, vertically projecting the open-closed eye judgment image to an X axis, and judging whether a driver opens or closes the eyes in one frame of driver image according to the proportion of black pixels and white pixels in the vertical projection;
a blink continuous eye closing judgment module: counting the continuous frame numbers of the continuous images of the driver with the closed eyes, judging whether the driver blinks or the eyes are continuously closed according to the range of the continuous frame numbers in the closed eyes, and calculating the blink frequency;
a fatigue judgment module: and if the blinking frequency is out of the set normal range, judging that the driver is in a fatigue state, and if the eyes of the driver are continuously closed, judging that the driver is in the fatigue state.
Compared with the prior art, the invention has the beneficial effects that: the method detects whether a driver wears glasses or not, performs different cutting on the preliminary rectangular range image of the eyes according to whether the driver wears the glasses or not, and selects different average gray threshold values according to different illumination intensities to perform binarization on the preliminary rectangular range image of the eyes, so that the fatigue detection method is more accurate and has higher robustness; the method is suitable for different illumination conditions, and has higher real-time performance and higher detection speed.
Drawings
FIG. 1 is a flowchart of a method for detecting fatigue of a driver based on a human eye state according to the present invention;
FIG. 2 is a block diagram of a driver fatigue detection system based on human eye status according to the present invention;
FIG. 3 is a schematic diagram of a face rectangle frame and a preliminary eye rectangle range image of the present invention;
FIG. 4 is a diagram illustrating the effect of edge detection on the glasses detection area;
fig. 5 is a schematic diagram of a vertical projection process of the open eye and the closed eye when the infrared light supplement is not started.
In the drawings: 1. the system comprises a driver fatigue detection system based on human eye states, 11, a human face detection module, 12, a human face tracking module, 13, an eye preliminary range image acquisition module, 14, a preprocessing module, 15, an eye accurate range image acquisition module, 16, an eye opening and closing judgment module, 17, a blink continuous eye closing judgment module, 18 and a fatigue judgment module.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The invention provides a driver fatigue detection method based on human eye state, which comprises the following steps:
s1, acquiring a driver image in real time, performing face detection on the driver image, acquiring a face rectangular frame containing a face, and acquiring an eye preliminary rectangular range image in the face rectangular frame and coordinates of the eye preliminary rectangular range image in the face rectangular frame;
s2, after a face rectangular frame of one frame of driver image is obtained, face tracking is carried out on the subsequent driver image, and the face rectangular frame of the subsequent driver image is obtained;
s3, for the face rectangular frame of the subsequent driver image, acquiring an eye preliminary rectangular range image of the subsequent driver image according to the coordinates of the eye preliminary rectangular range image in the face rectangular frame;
s4, carrying out clipping and binarization preprocessing on the preliminary eye rectangular range image acquired in the S3;
s5, carrying out contour search on the eye preliminary rectangular range image subjected to binarization in S4, accurately positioning the eye accurate rectangular range image by utilizing rectangular fitting, and obtaining the coordinates of the eye accurate rectangular range image in a face rectangular frame;
s6, extracting a non-binarized accurate eye rectangular range image from the rectangular face frame according to the coordinates of the accurate eye rectangular range image in the rectangular face frame, binarizing the extracted accurate eye rectangular range image to form an open-closed eye judgment image, vertically projecting the open-closed eye judgment image to an X axis, and judging whether the driver opens the eyes or closes the eyes in one frame of driver image according to the proportion of black pixels and white pixels in the vertical projection;
s7, counting the continuous frame numbers of the continuous images of the driver with the closed eyes, judging whether the driver blinks or the eyes are continuously closed according to the range of the continuous frame numbers with the closed eyes, and calculating the blinking frequency;
and S8, if the blinking frequency is out of the set normal range, judging that the driver is in a fatigue state, and if the eyes of the driver are continuously closed, judging that the driver is in the fatigue state.
The invention discloses a driver fatigue detection method based on human eye state, comprising the following steps of S1:
utilize the infrared camera that has the infrared light filling function to acquire driver's image, infrared camera is equipped with the illumination intensity inductor, the illumination intensity inductor sensing when illumination intensity is higher than the settlement light intensity threshold value in the driver's cabin, do not open the infrared light filling function, acquire driver's color image, the illumination intensity inductor sensing is when illumination intensity is less than the settlement light intensity threshold value in the driver's cabin, automatically, open infrared light filling, and acquire driver's infrared black and white image, make under different illumination intensity, all can acquire clear driver's image and do analysis processes, therefore, the high adaptability is achieved, and wide applicability is achieved.
The invention discloses a driver fatigue detection method based on human eye state, comprising the following steps of S1:
training a classifier by using an Adaboost algorithm to perform face detection on a driver image, acquiring a face rectangular frame containing a face, performing stasm feature point positioning on an image in the acquired face rectangular frame, so as to identify eye feature points, and acquiring an image in a preset size rectangular range larger than the range of the eye feature points as an eye preliminary rectangular range image;
as shown in fig. 3, the preliminary eye rectangular range images are images including eye feature points and eyebrow feature points, and are an approximate region image of the eyes, that is, a preliminary positioning of the eyes, two preliminary eye rectangular range images are provided in the rectangular frame of the face, which are the preliminary eye rectangular range image of the left eye and the preliminary eye rectangular range image of the right eye, and the preliminary eye rectangular range images of the left eye and the right eye are identical in size and located at the same horizontal position.
The invention discloses a driver fatigue detection method based on human eye state, comprising the following steps of S2:
carrying out face tracking on subsequent driver images by using a KCF (High-speed tracking with kernel correlation filters) algorithm, generating evaluation index data for evaluating a tracking effect after face tracking of each frame of driver image, and adjusting a tracking strategy when the evaluation index data is lower than a set threshold value;
performing KCF face tracking on the driver image of each frame to generate evaluation index data peak _ value, wherein the peak _ value is a decimal from 0 to 1, and the larger the value is, the higher the confidence coefficient of the tracking result is, the better the tracking effect is; the peak _ value is larger in the case of a positive face, and smaller in the case of a non-positive face (turning, nodding, etc.); a fixed threshold is preset for the evaluation index data peak _ value, when the size of the evaluation index data is higher than the preset threshold, a frame of driver image corresponding to the evaluation index data is judged to be a front face, and when the size of the evaluation index data is lower than or equal to the preset threshold, a frame of driver image corresponding to the evaluation index data is judged to be a non-front face; when one frame of the tracked driver image is a non-frontal face, face tracking is not carried out continuously, if tracking is carried out continuously, the face tracking of the driver images of all subsequent frames is wrong due to the fact that drift under the condition of the non-frontal face is the most serious, face detection is carried out on the driver images of the subsequent frames at intervals of 5 frames at the moment, whether a face rectangular frame containing the face can be obtained or not is judged, if the face rectangular frame containing the face is obtained, the frame of the driver image is the frontal face (the driver image of the non-frontal face cannot detect the face rectangular frame), face tracking is started, and otherwise, face detection is carried out on the driver images of the subsequent frames at intervals of 5 frames.
The invention discloses a driver fatigue detection method based on human eye state, comprising the following steps of S4:
judging whether a driver wears glasses in the driver image, selecting the size of a cutting area according to a judgment result to cut the preliminary eye rectangular range image, and selecting different average gray threshold values according to different illumination intensities when the driver image is obtained to binarize the cut preliminary eye rectangular range image;
the method for judging whether the driver wears glasses in the driver image comprises the following steps: determining a glasses detection area, wherein the glasses detection area takes the height of the preliminary eye rectangular range image as the height, takes the width of the face rectangular frame as the width, and covers the preliminary eye rectangular range images of the left eye and the right eye; determining a frame detection area, wherein the frame detection area is positioned in a glasses detection area, specifically between two eyes and above a nose bridge, and the position between the two eyes and above the nose bridge can be determined according to the positioning of the stasm characteristic points so as to determine the frame detection area;
as shown in fig. 4, the image in the glasses detection area is subjected to edge detection, the frame detection area has almost no edge information when glasses are not worn, and the frame detection area has rich edge information (without considering a transparent frame) when glasses are worn, because the frame detection area between two eyes and above the nose bridge contains a glasses frame when the glasses are worn, the number of each column of white pixels in the frame detection area after edge detection is counted, if the number of n continuous columns of white pixels is 0, it can be considered that a driver does not wear glasses in one frame of driver image, otherwise, it is considered that the driver wears glasses, and n is an appropriate value according to actual conditions.
The invention discloses a driver fatigue detection method based on human eye state, comprising the following steps of S4:
because the left side of left eye and the right side of right eye bring the influence of illumination easily, the preliminary rectangle range image of the eyes of left and right eyes will be treated differently when tailorring, cuts out the partial region on left side of left eye and right eye right side respectively to when the driver wears glasses, enlarge the scope of tailorring, the interference of minimize picture frame.
The invention discloses a driver fatigue detection method based on human eye state, comprising the following steps of S4:
the average gray threshold value is the average gray of the clipped primary eye rectangular range image multiplied by a coefficient m, whether infrared light supplement is started or not is determined according to different illumination intensities, so that a color image or an infrared black-and-white image of a driver is obtained, when the driver image is a color image or an infrared black-and-white image, different coefficients m are adopted, and accordingly binarization of the clipped primary eye rectangular range image is performed by adopting different average gray threshold values according to different illumination intensities when the driver image is obtained; and if the gray value of a certain pixel point of the eye preliminary rectangular range image is smaller than the average gray threshold value, setting the gray value of the pixel point to be 255, otherwise, setting the gray value of the pixel point to be 0.
The invention discloses a driver fatigue detection method based on human eye state, comprising the following steps of S5:
because the preliminary eye rectangular range image is an image comprising eye characteristic points and eyebrow characteristic points, after the preliminary eye rectangular range image is binarized in the S4, because the colors of the eyes and the eyebrows are deeper than other parts, the eyes and the eyebrows are white after binarization, and other parts are black, contour searching is carried out on the preliminary eye rectangular range image binarized in the S4, all white areas are found, then the upper contour is identified as the eyebrows according to the geometric relation between the eyes and the eyebrows, the lower contour is the eyes, after the eye contour is determined, the accurate position image of the eyes is positioned to be the accurate eye rectangular range image by utilizing rectangular fitting, and the coordinates of the accurate eye rectangular range image in the rectangular frame of the face are obtained.
The invention discloses a driver fatigue detection method based on human eye state, comprising the following steps of S6:
carrying out binarization on the extracted eye accurate rectangular range image to form an open-closed eye judgment image, wherein the binarization method adopts an average gray threshold value for binarization, the average gray threshold value is the average gray of the extracted eye accurate rectangular range image multiplied by a coefficient m, when the driver image is a color or infrared black-and-white image, different coefficients m are adopted, if the gray value of a certain pixel point of the eye accurate rectangular range image is smaller than the average gray threshold value, the gray value of the pixel point is set to be 255, otherwise, the gray value of the pixel point is set to be 0;
as shown in fig. 5, after the binarization is completed, the open-closed eye determination image is vertically projected to the X axis, specifically: if the original driver image of the open-close eye judgment image is a color image, vertically projecting the open-close eye judgment image to an X axis, counting the number of black pixels of each column of the open-close eye judgment image, analyzing whether mutation occurs among the number of the black pixels of each column, considering that the open-close eye judgment image is open, and otherwise, considering that the driver closes the eye; the method for determining whether mutation occurs is as follows: dividing the vertical projection into 5 regions by using 4 straight lines vertical to the X axis, calculating the average value of the number of black pixels in each row in each region, calculating the difference value between every two of the 5 average values, if a difference value is larger than a difference threshold value, determining that mutation occurs between the number of black pixels in each row, and otherwise, determining that mutation does not occur;
if the original driver image of the eye opening and closing judgment image is an infrared image, vertically projecting the eye opening and closing judgment image to an X axis, counting the number of white pixels of each column of the eye opening and closing judgment image, if the number of the white pixels of each column is 0, determining that the eyes of the driver are closed in the eye opening and closing judgment image, and if not, determining that the eyes of the driver are opened;
if the number of the white pixels in each column is 0, the reason that the eyes of the driver are closed in the open-close eye judgment image is considered to be that, after a plurality of times of experimental verification, the open-close eye judgment image after eye precision rectangular range image binarization is completely black when the eyes are closed in an infrared environment, the open-close eye judgment image is vertically projected to an X axis, and the number of the white pixels in each column of the open-close eye judgment image is 0.
The invention discloses a driver fatigue detection method based on human eye state, comprising the following steps of S7:
counting the continuous frames of the continuous images of the driver in the eye-closing state, calculating the continuous time corresponding to the continuous frames according to the relationship between the frames and the time, for example, 60 frames/second when the images of the driver are collected by a camera, considering that the driver blinks when the continuous time is 0.2-0.4 second, calculating the blink frequency according to the blink frequency of the driver in a period of time, and considering that the eyes of the driver are continuously closed when the continuous time exceeds 2 seconds.
The invention discloses a driver fatigue detection method based on human eye state, comprising the following steps of S8:
the fatigue degree is deepened when the blinking frequency is too low or too high, the duration of the eye opening state is longer when the blinking frequency is too low, the vision of a driver is dull and the driver is in a vague state, and the fatigue state is indicated to be displayed; the rapid blinking frequency indicates that the eyes of the driver are dry or the driver tries to keep clear, which indicates that the fatigue state occurs; and if the blinking frequency is out of the set normal range, judging that the driver is in a fatigue state, and if the eyes of the driver are continuously closed, judging that the driver is in the fatigue state.
The present invention also provides a driver fatigue detection system 1 based on a human eye state, including:
the face detection module 11: acquiring a driver image in real time, performing face detection on the driver image, acquiring a face rectangular frame containing a face, and acquiring an eye preliminary rectangular range image in the face rectangular frame and coordinates of the eye preliminary rectangular range image in the face rectangular frame;
the face tracking module 12: after a face rectangular frame of one frame of driver image is obtained, face tracking is carried out on a subsequent driver image, and the face rectangular frame of the subsequent driver image is obtained;
eye preliminary range image acquisition module 13: for the face rectangular frame of the subsequent driver image, acquiring an eye preliminary rectangular range image of the subsequent driver image according to the coordinates of the eye preliminary rectangular range image in the face rectangular frame;
the preprocessing module 14: the preliminary eye rectangular range image acquired by the preliminary eye range image acquisition module is subjected to cutting and binarization preprocessing in sequence;
eye precision range image acquisition module 15: carrying out contour search on the eye preliminary rectangular range image subjected to binarization in the preprocessing module, accurately positioning the eye accurate rectangular range image by utilizing rectangle fitting, and acquiring coordinates of the eye accurate rectangular range image in a face rectangular frame;
open eye and closed eye determination module 16: extracting a non-binarized accurate eye rectangular range image from the rectangular face frame according to the coordinates of the accurate eye rectangular range image in the rectangular face frame, binarizing the extracted accurate eye rectangular range image to form an open-closed eye judgment image, vertically projecting the open-closed eye judgment image to an X axis, and judging whether a driver opens or closes the eyes in one frame of driver image according to the proportion of black pixels and white pixels in the vertical projection;
the blink continuous eye closing judgment module 17: counting the continuous frame numbers of the continuous images of the driver with the closed eyes, judging whether the driver blinks or the eyes are continuously closed according to the range of the continuous frame numbers in the closed eyes, and calculating the blink frequency;
the fatigue judgment module 18: and if the blinking frequency is out of the set normal range, judging that the driver is in a fatigue state, and if the eyes of the driver are continuously closed, judging that the driver is in the fatigue state.
The invention relates to a driver fatigue detection system 1 based on human eye state, a human face detection module 11 comprises:
the method comprises the steps that an infrared camera with an infrared light supplementing function is used for obtaining images of a driver, when the illumination intensity in a cab is higher than a set light intensity threshold value, a color image of the driver is obtained, when the illumination intensity in the cab is lower than the set light intensity threshold value, the infrared light supplementing is started, and infrared black and white images of the driver are obtained.
The invention relates to a driver fatigue detection system 1 based on human eye state, a human face detection module 11 comprises:
the method comprises the steps of training a classifier by using an Adaboost algorithm to carry out face detection on a driver image, obtaining a face rectangular frame containing a face, carrying out stasm feature point positioning on an image in the obtained face rectangular frame, identifying eye feature points, and obtaining an image in a preset size rectangular range larger than the range of the eye feature points as an eye preliminary rectangular range image.
The invention relates to a driver fatigue detection system 1 based on human eye state, a human face tracking module 12 is provided with:
and carrying out face tracking on subsequent driver images by utilizing a KCF algorithm, generating evaluation index data after face tracking of each frame of driver image so as to evaluate the tracking effect, and adjusting a tracking strategy when the evaluation index data is lower than a set threshold value.
The invention relates to a driver fatigue detection system 1 based on human eye state, comprising a preprocessing module 14:
judging whether a driver wears glasses in the driver image, selecting the size of a cutting area according to a judgment result to cut the preliminary eye rectangular range image, and selecting different average gray threshold values according to different illumination intensities when the driver image is obtained to binarize the cut preliminary eye rectangular range image.
Compared with the prior art, the invention has the beneficial effects that: the method detects whether a driver wears glasses or not, performs different cutting on the preliminary rectangular range image of the eyes according to whether the driver wears the glasses or not, and selects different average gray threshold values according to different illumination intensities to perform binarization on the preliminary rectangular range image of the eyes, so that the fatigue detection method is more accurate and has higher robustness; the method is suitable for different illumination conditions, the real-time performance is higher, and the detection speed is higher; compared with other methods for judging the blink by using the number of black pixels in the eye area or by using the aspect ratio of the eye fitting ellipse, the method has higher accuracy and robustness.
The above-described embodiments of the present invention should not be construed as limiting the scope of the present invention. Any other corresponding changes and modifications made according to the technical idea of the present invention should be included in the protection scope of the claims of the present invention.
Claims (8)
1. A driver fatigue detection method based on a human eye state is characterized by comprising the following steps:
s1, acquiring a driver image in real time, performing face detection on the driver image, acquiring a face rectangular frame containing a face, and acquiring an eye preliminary rectangular range image in the face rectangular frame and coordinates of the eye preliminary rectangular range image in the face rectangular frame;
s2, after a face rectangular frame of one frame of driver image is obtained, face tracking is carried out on the subsequent driver image, and the face rectangular frame of the subsequent driver image is obtained;
s3, for the face rectangular frame of the subsequent driver image, acquiring the eye preliminary rectangular range image of the subsequent driver image according to the coordinates of the eye preliminary rectangular range image in the face rectangular frame;
s4, carrying out clipping and binarization preprocessing on the preliminary eye rectangular range image acquired in the S3;
s5, carrying out contour search on the eye preliminary rectangular range image subjected to binarization in S4, accurately positioning the eye accurate rectangular range image by utilizing rectangular fitting, and acquiring coordinates of the eye accurate rectangular range image in a face rectangular frame;
s6, extracting a non-binarized accurate eye rectangular range image from the rectangular face frame according to the coordinates of the accurate eye rectangular range image in the rectangular face frame, binarizing the extracted accurate eye rectangular range image to form an open-closed eye judgment image, vertically projecting the open-closed eye judgment image to an X axis, and judging whether the driver opens the eyes or closes the eyes in one frame of driver image according to the proportion of black pixels and white pixels in the vertical projection;
s7, counting the continuous frame numbers of the continuous images of the driver with the closed eyes, judging whether the driver blinks or the eyes are continuously closed according to the range of the continuous frame numbers with the closed eyes, and calculating the blinking frequency;
s8, if the blinking frequency is out of the set normal range, judging that the driver is in a fatigue state, and if the eyes of the driver are continuously closed, judging that the driver is in the fatigue state; wherein,
in step S4: judging whether a driver wears glasses in the driver image, selecting the size of a cutting area according to a judgment result to cut the preliminary eye rectangular range image, and selecting different average gray threshold values according to different illumination intensities when the driver image is obtained to binarize the cut preliminary eye rectangular range image.
2. The method for detecting fatigue of a driver based on the state of the human eye according to claim 1, wherein in step S1:
the method comprises the steps that an infrared camera with an infrared light supplementing function is used for obtaining images of a driver, when the illumination intensity in a cab is higher than a set light intensity threshold value, a color image of the driver is obtained, when the illumination intensity in the cab is lower than the set light intensity threshold value, the infrared light supplementing is started, and infrared black and white images of the driver are obtained.
3. The method for detecting fatigue of a driver based on the state of the human eye according to claim 1, wherein in step S1:
the method comprises the steps of training a classifier by using an Adaboost algorithm to carry out face detection on a driver image, obtaining a face rectangular frame containing a face, carrying out stasm feature point positioning on an image in the obtained face rectangular frame, identifying eye feature points, and obtaining an image in a preset size rectangular range larger than the range of the eye feature points as an eye preliminary rectangular range image.
4. The method for detecting fatigue of a driver based on the state of the human eye according to claim 1, wherein in step S2:
and carrying out face tracking on subsequent driver images by utilizing a KCF algorithm, generating evaluation index data after face tracking of each frame of driver image so as to evaluate the tracking effect, and adjusting a tracking strategy when the evaluation index data is lower than a set threshold value.
5. A driver fatigue detection system based on a human eye state, characterized by comprising:
the face detection module: acquiring a driver image in real time, performing face detection on the driver image, acquiring a face rectangular frame containing a face, and acquiring an eye preliminary rectangular range image in the face rectangular frame and coordinates of the eye preliminary rectangular range image in the face rectangular frame;
a face tracking module: after a face rectangular frame of one frame of driver image is obtained, face tracking is carried out on a subsequent driver image, and the face rectangular frame of the subsequent driver image is obtained;
the eye preliminary range image acquisition module: for the face rectangular frame of the subsequent driver image, acquiring an eye preliminary rectangular range image of the subsequent driver image according to the coordinates of the eye preliminary rectangular range image in the face rectangular frame;
a preprocessing module: the preliminary eye rectangular range image acquired by the preliminary eye range image acquisition module is subjected to cutting and binarization preprocessing in sequence;
eye precision range image acquisition module: carrying out contour search on the eye preliminary rectangular range image subjected to binarization in the preprocessing module, accurately positioning the eye accurate rectangular range image by utilizing rectangle fitting, and acquiring the coordinates of the eye accurate rectangular range image in a face rectangular frame;
an eye opening and closing judgment module: extracting a non-binarized accurate eye rectangular range image from the rectangular face frame according to the coordinates of the accurate eye rectangular range image in the rectangular face frame, binarizing the extracted accurate eye rectangular range image to form an open-closed eye judgment image, vertically projecting the open-closed eye judgment image to an X axis, and judging whether a driver opens or closes the eyes in one frame of driver image according to the proportion of black pixels and white pixels in the vertical projection;
a blink continuous eye closing judgment module: counting the continuous frame numbers of the continuous images of the driver with the closed eyes, judging whether the driver blinks or the eyes are continuously closed according to the range of the continuous frame numbers in the closed eyes, and calculating the blink frequency;
a fatigue judgment module: if the blinking frequency is out of the set normal range, judging that the driver is in a fatigue state, and if the eyes of the driver are continuously closed, judging that the driver is in the fatigue state; wherein, among the preprocessing module: judging whether a driver wears glasses in the driver image, selecting the size of a cutting area according to a judgment result to cut the preliminary eye rectangular range image, and selecting different average gray threshold values according to different illumination intensities when the driver image is obtained to binarize the cut preliminary eye rectangular range image.
6. The system of claim 5, wherein the face detection module is further configured to:
the method comprises the steps that an infrared camera with an infrared light supplementing function is used for obtaining images of a driver, when the illumination intensity in a cab is higher than a set light intensity threshold value, a color image of the driver is obtained, when the illumination intensity in the cab is lower than the set light intensity threshold value, the infrared light supplementing is started, and infrared black and white images of the driver are obtained.
7. The system of claim 5, wherein the face detection module is further configured to:
the method comprises the steps of training a classifier by using an Adaboost algorithm to carry out face detection on a driver image, obtaining a face rectangular frame containing a face, carrying out stasm feature point positioning on an image in the obtained face rectangular frame, identifying eye feature points, and obtaining an image in a preset size rectangular range larger than the range of the eye feature points as an eye preliminary rectangular range image.
8. The system of claim 5, wherein the face tracking module is further configured to:
and carrying out face tracking on subsequent driver images by utilizing a KCF algorithm, generating evaluation index data after face tracking of each frame of driver image so as to evaluate the tracking effect, and adjusting a tracking strategy when the evaluation index data is lower than a set threshold value.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710430629.3A CN107292251B (en) | 2017-06-09 | 2017-06-09 | Driver fatigue detection method and system based on human eye state |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710430629.3A CN107292251B (en) | 2017-06-09 | 2017-06-09 | Driver fatigue detection method and system based on human eye state |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107292251A CN107292251A (en) | 2017-10-24 |
| CN107292251B true CN107292251B (en) | 2020-08-28 |
Family
ID=60096130
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710430629.3A Active CN107292251B (en) | 2017-06-09 | 2017-06-09 | Driver fatigue detection method and system based on human eye state |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107292251B (en) |
Families Citing this family (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108382396B (en) * | 2018-02-02 | 2020-08-07 | 上海灏领科技有限公司 | Driver driving state recognition system and application method thereof |
| CN108419012A (en) * | 2018-03-18 | 2018-08-17 | 广东欧珀移动通信有限公司 | Photographing method, device, storage medium and electronic equipment |
| CN108614999B (en) * | 2018-04-16 | 2022-09-16 | 贵州大学 | Eye opening and closing state detection method based on deep learning |
| CN109271875B (en) * | 2018-08-24 | 2019-06-14 | 中国人民解放军火箭军工程大学 | A fatigue detection method based on key point information of eyebrows and eyes |
| CN109145852B (en) * | 2018-08-31 | 2022-06-17 | 辽宁工业大学 | Driver fatigue state identification method based on eye opening and closing state |
| CN108919517A (en) * | 2018-09-25 | 2018-11-30 | 吉林大学 | A kind of car steering anti-fatigue eyeglasses and fatigue state recognition method |
| CN109646024A (en) * | 2019-01-09 | 2019-04-19 | 浙江强脑科技有限公司 | Method for detecting fatigue driving, device and computer readable storage medium |
| CN112989874B (en) * | 2019-12-13 | 2024-09-10 | 深圳开阳电子股份有限公司 | Eye state identification method and device based on eye center point |
| CN111784973B (en) * | 2020-07-30 | 2021-12-14 | 广州敏视数码科技有限公司 | MDVR equipment integration fatigue detection method of fleet management platform |
| CN113076801A (en) * | 2021-03-04 | 2021-07-06 | 广州铁路职业技术学院（广州铁路机械学校） | Train on-road state intelligent linkage detection system and method |
| CN115243000B (en) * | 2021-08-30 | 2024-02-02 | 广州汽车集团股份有限公司 | A car cockpit monitoring system and car |
| CN113780150A (en) * | 2021-09-07 | 2021-12-10 | 广西电网有限责任公司贺州供电局 | A fatigue detection method and system based on multi-dimensional body state perception |
| CN116469085B (en) * | 2023-03-30 | 2024-04-02 | 万联易达物流科技有限公司 | Monitoring method and system for risk driving behavior |
| CN116168374B (en) * | 2023-04-21 | 2023-12-12 | 南京淼瀛科技有限公司 | Active safety auxiliary driving method and system |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101059836A (en) * | 2007-06-01 | 2007-10-24 | 华南理工大学 | Human eye positioning and human eye state recognition method |
| CN102054163A (en) * | 2009-10-27 | 2011-05-11 | 南京理工大学 | Method for testing driver fatigue based on monocular vision |
| WO2012000408A1 (en) * | 2010-06-29 | 2012-01-05 | Byd Company Limited | Eye detecting method and eye detecting device |
| CN104809445A (en) * | 2015-05-07 | 2015-07-29 | 吉林大学 | Fatigue driving detection method based on eye and mouth states |
| CN104881955A (en) * | 2015-06-16 | 2015-09-02 | 华中科技大学 | Method and system for detecting fatigue driving of driver |
| CN105224285A (en) * | 2014-05-27 | 2016-01-06 | 北京三星通信技术研究有限公司 | Eyes open and-shut mode pick-up unit and method |
| CN105335695A (en) * | 2015-08-19 | 2016-02-17 | 华南理工大学 | Glasses detection based eye positioning method |
-
2017
- 2017-06-09 CN CN201710430629.3A patent/CN107292251B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101059836A (en) * | 2007-06-01 | 2007-10-24 | 华南理工大学 | Human eye positioning and human eye state recognition method |
| CN102054163A (en) * | 2009-10-27 | 2011-05-11 | 南京理工大学 | Method for testing driver fatigue based on monocular vision |
| WO2012000408A1 (en) * | 2010-06-29 | 2012-01-05 | Byd Company Limited | Eye detecting method and eye detecting device |
| CN105224285A (en) * | 2014-05-27 | 2016-01-06 | 北京三星通信技术研究有限公司 | Eyes open and-shut mode pick-up unit and method |
| CN104809445A (en) * | 2015-05-07 | 2015-07-29 | 吉林大学 | Fatigue driving detection method based on eye and mouth states |
| CN104881955A (en) * | 2015-06-16 | 2015-09-02 | 华中科技大学 | Method and system for detecting fatigue driving of driver |
| CN105335695A (en) * | 2015-08-19 | 2016-02-17 | 华南理工大学 | Glasses detection based eye positioning method |
Non-Patent Citations (4)
| Title |
|---|
| "一种用于驾驶员疲劳检测的人眼状态判别方法";王威等;《计算机光盘软件与应用》;20121231(第13期);第134页第三节 * |
| "军用车辆驾驶中的眼睛状态检测法研究";田亮等;《火力与指挥控制》;20140531;第39卷(第5期);第176-179页 * |
| "基于OpenCV的视频图像人眼跟踪方法的研究";梁涛等;《科学技术与工程》;20130228;第13卷(第5期);第1356页第2节 * |
| "基于人脸图片边缘信息的眼镜检测";陈岳林等;《软件导刊》;20140731;第13卷(第7期);第141-143页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107292251A (en) | 2017-10-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107292251B (en) | Driver fatigue detection method and system based on human eye state | |
| CN102289660B (en) | Method for detecting illegal driving behavior based on hand gesture tracking | |
| CN103714660B (en) | System for achieving fatigue driving judgment on basis of image processing and fusion between heart rate characteristic and expression characteristic | |
| CN101593425B (en) | Machine vision based fatigue driving monitoring method and system | |
| CN104809445B (en) | method for detecting fatigue driving based on eye and mouth state | |
| CN103400110B (en) | Abnormal face detecting method before ATM cash dispenser | |
| CN112241658A (en) | Fatigue driving early warning system and method based on depth camera | |
| US7798643B2 (en) | System for analyzing eye responses to automatically track size, location, and movement of the pupil | |
| US7362885B2 (en) | Object tracking and eye state identification method | |
| CN106250801A (en) | Based on Face datection and the fatigue detection method of human eye state identification | |
| US7620216B2 (en) | Method of tracking a human eye in a video image | |
| US9646215B2 (en) | Eye part detection apparatus | |
| CN103034852B (en) | The detection method of particular color pedestrian under Still Camera scene | |
| CN102054163A (en) | Method for testing driver fatigue based on monocular vision | |
| JP2008146172A (en) | Eye detection device, eye detection method, and program | |
| Darshana et al. | Efficient PERCLOS and gaze measurement methodologies to estimate driver attention in real time | |
| CN106503644B (en) | Glasses attribute detection method based on edge projection and color characteristic | |
| CN105286802A (en) | Driver fatigue detection method based on video information | |
| CN112183502B (en) | Method for determining driving state of driver, computer storage medium, and electronic device | |
| CN106214166A (en) | One is worn glasses Driver Fatigue Detection | |
| CN112241647B (en) | Dangerous driving behavior early warning device and method based on depth camera | |
| CN106203338B (en) | Human eye state method for quickly identifying based on net region segmentation and threshold adaptive | |
| Cheng et al. | A fatigue detection system with eyeglasses removal | |
| CN103729646A (en) | Eye image validity detection method | |
| CN108256378A (en) | Driver Fatigue Detection based on eyeball action recognition |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20210611 Address after: 435100 Jinhu Ma Hai, Daye City, Huangshi City, Hubei Province Patentee after: Daye Xinye Special Steel Co.,Ltd. Address before: No.139 kuangye Avenue, Jinhu Street office, Daye City, Huangshi City, Hubei Province Patentee before: HUBEI TIANYE YUNSHANG NETWORK TECHNOLOGY Co.,Ltd. |