CN111243236A - Fatigue driving early warning method and system based on deep learning - Google Patents

Abstract

The invention discloses a deep learning-based fatigue driving early warning method and system, comprising: a parameter analysis module, a video acquisition module, an image processing module, an identification module, a picture frame module, and an alarm module; the parameter analysis module is used when a program Various parameter settings of the program are read during startup, the video acquisition module collects video, and transmits the valid video to the image processing module, the image processing module reads and records the length and width of the image, and generates a A zero matrix of the same size transmits the image to the recognition module. The recognition module generates the frame coordinates of the detection result and the confidence of the detection result. The frame module identifies the frame coordinates and the confidence of the detection result. Perform traversal, frame and label all categories with confidence greater than the threshold according to the coordinates of the frame, and draw frames with different colors to distinguish them according to different categories, and the alarm module sends an alarm signal.

Description

A method and system for fatigue driving early warning based on deep learning

technical field

The invention relates to the technical field of fatigue driving early warning, and more particularly to a fatigue driving early warning system based on deep learning.

Background technique

At present, with the continuous improvement of people's living standards and the continuous development of the automobile industry and social economy, driving has become an important part of people's daily life. However, frequent traffic accidents threaten people's travel safety, causing economic losses in lightest cases, and endangering people's lives in heavy cases.

In recent years, driver fatigue driving has become one of the main causes of motor vehicle accidents. Fatigue driving, which often occurs when the driver continues to drive the vehicle, or cannot guarantee a awake state. At this time, the driver's physiological and psychological functions are out of balance, which will objectively lead to a decline in driving skills. The driver's thinking, attention, perception, responsiveness, judgment, willpower and motor nerves will be affected by the causes of fatigued driving. After long-term fatigue driving, the driver will appear drowsy and tired, it is difficult to concentrate, the vision gradually narrows, and the thinking ability declines. There may even be trance or short-term memory loss, slow response, operational errors or untimely corrections and other unsafe factors, which can easily lead to the probability of road traffic accidents. When the driver is fatigued, the judgment ability will be reduced and the response will be slow, and the operation error rate will increase at this time. When the driver faces slight fatigue, it will lead to untimely and inaccurate operation of the gearbox; when the driver faces moderate fatigue, the operation action is incoherent, and the phenomenon of forgetting to operate also occurs from time to time; when the driver faces severe fatigue, It is often mechanized or dozed off in the middle, and the vehicle will lose control in severe cases.

At present, many technologies have been used to detect the fatigue state of drivers, and these technologies can be roughly divided into three categories: the driving pattern of the vehicle, the physiological state of the driver, and computer vision technology. However, the main disadvantage of vehicle driving pattern-based detection methods is that their accuracy depends on the personal characteristics of the vehicle and its driver; detection based on physiological characteristics is currently the most accurate of all fatigue detection methods, however, because of the physical Many sensors are attached to interfere with the invasiveness of the driver, and the acquisition of signals may cause discomfort to the driver; non-invasive methods of biosignaling exist, but the accuracy is not high; methods based on visual characteristics are used such as yawning. , non-invasive visual information such as facial expressions, head movements and eye states to detect drowsiness. Due to the non-contact nature, visual feature-based methods have become a promising research area for drowsiness detection. Methods based on yawning and head movement cannot reliably detect episodes of drowsiness because these methods do not directly represent drowsiness.

Therefore, it is an urgent problem for those skilled in the art to provide a method for judging that the driver is in a fatigue state such as drowsiness through eye closure and blinking frequency.

SUMMARY OF THE INVENTION

In view of this, the present invention provides a deep learning-based fatigue driving warning method and system.

In order to achieve the above object, the present invention adopts the following technical solutions:

A fatigue driving warning method based on deep learning includes the following steps:

S1. Collect the driver's eye image;

S2. Establish a Faster R-CNN human eye opening and closing detection model;

S3. Use Faster R-CNN to detect the driver's eye image, and obtain the frame coordinates and the confidence of the detection result;

S4, traverse all the categories, draw frames and mark the categories according to the frame coordinates of all categories whose confidence is greater than the threshold, and draw frames with different colors to distinguish them according to the categories;

S5. Output the detection category.

Preferably, the specific process of the step S2 is:

S21. Select RPN as the network structure, and conduct simulation training through the CEW data set;

S22. Label the anchor points. An anchor point with the highest overlap ratio with the box of the real label or an anchor point with an intersection ratio of 0.7 is a positive label, and an anchor point with an intersection ratio lower than 0.3 is a negative label;

S22, train the shared convolution layer, and fine-tune the parameters of the unique layer;

Preferably, the category is eyes open or eyes closed.

Preferably, the specific process of detection in the step S3 is: using the im_detect function to detect the driver's eye image.

A system using a deep learning-based fatigue driving warning method, comprising: a parameter analysis module, a video acquisition module, an image processing module, an identification module, a picture frame module, and an alarm module;

The parameter parsing module is used to read in various parameter settings of the program when the program is started, the video acquisition module is used to collect video, and transmit valid video to the image processing module, and the image processing module is used to Read in and record the length and width of the image, and generate a zero matrix of the same size, and then transmit the image to the recognition module, which uses the trained Fast R-CNN detection network to generate the block diagram coordinates of the detection result and the confidence of the detection result, the frame module is used to traverse all the categories by identifying the coordinates of the frame and the confidence of the detection result, and frame all the categories whose confidence is greater than the threshold according to the coordinates of the frame and mark the category, according to the category Different color boxes are drawn to distinguish them, and the alarm module is used to send an alarm signal.

Preferably, the categories are eyes open and eyes closed.

Preferably, the alarm module is connected with a buzzer, when the alarm module outputs a high level, the buzzer alarms, and when the alarm module outputs a low level, the buzzer stops the alarm.

It can be seen from the above technical solutions that, compared with the prior art, the present invention provides a method of collecting driver's eye image information through a video acquisition module, recording the length and width of the image through an image processing module, and generating an image of the same size. Zero matrix, through the training of the Faster R-CNN network, the frame coordinates and confidence of the detection results generated by the recognition module are more accurate, and the categories are divided by the frame module. When the number of closed eyes reaches the threshold, it will alarm based on deep learning fatigue Driving Warning System.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only It is an embodiment of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to the provided drawings without creative work.

The accompanying drawing of FIG. 1 is a schematic flow chart provided by the present invention.

Figure 2 is a schematic diagram of the structure provided by the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

A fatigue driving warning method based on deep learning includes the following steps:

S1. Collect the driver's eye image;

S2. Establish a Faster R-CNN human eye opening and closing detection model;

S3. Use Faster R-CNN to detect the driver's eye image, and obtain the frame coordinates and the confidence of the detection result;

S4, traverse all the categories, draw frames and mark the categories according to the frame coordinates of all categories whose confidence is greater than the threshold, and draw frames with different colors to distinguish them according to the categories;

S5. Output the detection category.

In order to further optimize the above-mentioned technical scheme, the specific process of step S2 is:

S21. Select RPN as the network structure, and conduct simulation training through the CEW data set;

S22. Label the anchor points. An anchor point with the highest overlap ratio with the box of the real label or an anchor point with an intersection ratio of 0.7 is a positive label, and an anchor point with an intersection ratio lower than 0.3 is a negative label;

S22, train the shared convolution layer, and fine-tune the parameters of the unique layer;

In order to further optimize the above technical solution, the categories are open eyes or closed eyes.

In order to further optimize the above technical solution, the specific process of detection in step S3 is: using the im_detect function to detect the driver's eye image.

A fatigue driving warning system based on a deep learning network model training method, comprising: a parameter analysis module 1, a video acquisition module 2, an image processing module 3, an identification module 4, a picture frame module 5, and an alarm module 6;

The parameter parsing module 1 is responsible for reading various parameter settings of the program when the program starts, such as whether to use the RPN network, the name of the network and model used by the code, whether to use the CPU, the number of the GPU device, etc.; the video capture module 2 uses OpenCV The VideoCapture function captures the video. When the camera is turned on, it will automatically read a frame of image. If the read-in image is valid, the read-in image will be transmitted to the image processing module 3; After this frame of image is received, the length and width of the image are read and recorded through the numpy package, and a zero matrix of the same size is generated to prepare for the output of future images, and then the image is transmitted to the recognition module 4; recognition module 4 The im_detect function that comes with fast_rcnn is called for detection. The im_detect function receives two parameters, namely the net object and the image to be recognized, and returns two parameters: box and scores. Box is the frame coordinates used to store the detection results, and scores is the confidence of the detection result; the frame-frame module 5 first traverses all the categories through the frame coordinates obtained by the recognition module 4 and the confidence of the detection result, and draws a frame and labels the categories according to the frame coordinates of all categories whose confidence is greater than the threshold. According to the different categories—open eyes and closed eyes, draw boxes with different colors to distinguish them; the alarm module 6 is used to send an alarm signal.

In order to further optimize the above technical solution, the alarm module 6 is connected with a buzzer 7. When the alarm module 6 outputs a high level, the buzzer 7 alarms, and when the alarm module 6 outputs a low level, the buzzer 7 stops alarming.

Example

Figure 1 shows an implementation process of the present invention. After the system is turned on, the parameter analysis module 1 will read the used network model and related configuration files, complete the preloading of the model, the video acquisition module 2 starts monitoring, and the image processing module 3. Record the specification information (length, width) of the image, and then transmit it to the recognition module 4. The recognition module 4 will return the confidence level array and box frame coordinate array of the recognition category. After a series of processing: obtain the confidence score and box coordinates (x _min , y _min , x _max , y _max ) from the confidence score array and box frame coordinate array, stack them in columns, store them in the dets array together, and compare the dets The array is subjected to non-maximum suppression, that is, after removing all the smaller areas, the recognition result is obtained, and the frame module 5 is called to frame. Every time an eye closure is identified, the number of closed eyes is increased by 1, and every time an open eye is identified, the number of closed eyes is decreased by 1. When the statistics of closed eyes is 0, the operation of decrementing by 1 is not performed; when When the count of closed eyes is greater than the threshold, the alarm module 6 sends a high level to the buzzer 7, and the buzzer 7 starts to alarm, and the alarm module 6 will not send a low level to the buzzer until the count of closed eyes is 0. Buzzer 7, buzzer 7 stops the alarm.

The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the various embodiments can be referred to each other. As for the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant part can be referred to the description of the method.

The above description of the disclosed embodiments enables any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (7)

1. a fatigue driving warning method based on deep learning, is characterized in that, comprises the steps as follows: S1. Collect the driver's eye image; S2. Establish a Faster R-CNN human eye opening and closing detection model; S3. Use Faster R-CNN to detect the driver's eye image, and obtain the frame coordinates and the confidence of the detection result; S4, traverse all the categories, draw frames and mark the categories according to the frame coordinates of all categories whose confidence is greater than the threshold, and draw frames with different colors to distinguish them according to the categories; S5. Output the detection category. 2. a kind of fatigue driving warning method based on deep learning, is characterized in that, the concrete process of described step S2 is: S21. Select RPN as the network structure, and conduct simulation training through the CEW data set; S22. Label the anchor points. An anchor point with the highest overlap ratio with the box of the real label or an anchor point with an intersection ratio of 0.7 is a positive label, and an anchor point with an intersection ratio lower than 0.3 is a negative label; S22, train the shared convolutional layer, and fine-tune the parameters of the unique layer. 3. A fatigue driving warning method based on deep learning, wherein the category is eyes open or eyes closed. 4. A fatigue driving warning method based on deep learning, characterized in that the specific process of detection in the step S3 is: using the im_detect function to detect the driver's eye image. 5. A system using the deep learning-based fatigue driving warning method according to any one of claims 1-4, characterized in that, comprising: a parameter analysis module (1), a video acquisition module (2), an image processing module module (3), identification module (4), picture frame module (5), alarm module (6); The parameter analysis module (1) is used for reading in various parameter settings of the program when the program is started, and the video capture module (2) is used for capturing video and transmitting valid video to the image processing module (3). ), the image processing module (3) is used to read in and record the length and width of the image, and generate a zero matrix of the same size, then, the image is transmitted to the recognition module (4), the recognition module ( 4) Using the trained Fast R-CNN detection network to generate the frame coordinates of the detection results and the confidence of the detection results, the frame module (5) is used to traverse all the categories by identifying the frame coordinates and the confidence of the detection results , all the categories with confidence greater than the threshold are framed according to the coordinates of the frame and marked with the categories, according to the different categories, the frames with different colors are drawn to distinguish them, and the alarm module (6) is used for sending an alarm signal. 6 . The fatigue driving warning system based on deep learning according to claim 5 , wherein the categories are eyes open and eyes closed. 7 . 7. The deep learning-based fatigue driving warning system according to claim 5, wherein the alarm module (6) is connected with a buzzer (7), and when the alarm module (6) outputs a high The buzzer (7) alarms when the level is at a low level, and the buzzer (7) stops the alarm when the alarm module (6) outputs a low level.

Images