CN116168508B - A driving fatigue detection and early warning control method and device for human-machine co-driving - Google Patents

Abstract

The application discloses a method and a device for detecting and controlling driving fatigue and early warning of man-machine co-driving. Facial features and heart rate values are obtained according to image recognition in a cab, wherein the facial features comprise eye states, mouth states and head pitching angles, and the image in the cab and the extracted facial features and heart rate values are input into a preset time sequence neural network for processing at the same time to obtain first fatigue information; inputting an outdoor road surface image of a driver cab into a YOLOP model, acquiring lane line information of the road surface, extracting line pressing running time, and acquiring second fatigue information based on the line pressing running time; calculating steering wheel shake features according to the steering wheel angle information, and obtaining third fatigue information based on the steering wheel shake features; and calculating the fatigue degree based on the first, second and third fatigue information, and when the fatigue degree exceeds a preset early warning value, sending out an audible/visual alarm and starting an expert database system. The application eliminates the defects caused by a single detection method and improves the reliability of the early warning system.

Description

A driving fatigue detection and early warning control method and device for human-machine co-driving

Technical field

The present invention relates to the technical field of safe driving, and in particular to a driving fatigue detection and early warning control method and device for human-machine co-driving.

Background technique

With the rapid development of the economy and the continuous improvement of the quality of people's lives, cars have gradually become a means of transportation for ordinary people. While bringing convenience to people's lives and promoting social progress, they have also increased the occurrence of various traffic accidents. This has The hidden dangers of traffic accidents cannot be ignored. According to incomplete statistics, 20% of road traffic accidents are caused by driver fatigue. In order to ensure the driver's travel safety and property safety, driver fatigue detection and early warning are usually carried out. Currently, driver fatigue detection is mainly implemented through the following three methods: methods based on physiological characteristics, methods based on driver facial information characteristics, and methods based on vehicle behavioral characteristics. Among them, the detection method based on physiological characteristics generally requires the driver to wear relevant instruments and equipment to obtain the driver's brain wave signal (EEG), heart rate, etc. as reference indicators. It has the advantages of high accuracy and continuous availability, but the cost is high; Based on the driver's facial information characteristics, the driver's fatigue status is determined by real-time detection of key positions of the eyes, mouth and head of the human face. The biggest advantage of this method is that it is non-invasive and highly accurate, but it is easily affected by occlusion and light. etc.; the vehicle behavior-based detection method continuously monitors vehicle behavior parameters such as lane departure detection, steering wheel angle and other dynamic information to determine the driver's drowsiness, but it relies too much on the driver's personal driving habits.

Therefore, the traditional fatigue driving warning method has the problem of single information source and low fatigue driving warning accuracy.

Contents of the invention

In order to solve the above technical problems, the present invention proposes a driving fatigue detection and early warning control method and device for human-machine co-driving. In the method and device, the disadvantages caused by a single detection method are eliminated, the reliability of the early warning device is improved, and the safety of the driver is ensured.

In order to achieve the above objects, the technical solutions of the present invention are as follows:

A driving fatigue detection and early warning control method for human-machine co-driving, including the following steps:

Collect images inside the cab, road images outside the cab, and steering wheel angle information in real time;

Facial features and heart rate values are obtained based on in-cab image recognition. The facial features include eye state, mouth state and head pitch angle, and the in-cab image and the extracted facial features and heart rate values are simultaneously input into the preset time-series neural network. Process to obtain the first fatigue information;

Input the road surface image outside the cab into the trained YOLOP model, obtain the lane line information of the road surface, extract the lane line driving time in the lane line information, and obtain the second fatigue information based on the lane line driving time;

Calculate the steering wheel jitter characteristics based on the steering wheel angle information, and obtain the third fatigue information based on the steering wheel jitter characteristics;

Construct a deep learning model and use the adaptive weight algorithm for training, obtain the weight values α, β, and γ corresponding to the first fatigue information, the second fatigue information, and the third fatigue information respectively, and combine the first fatigue information, the second fatigue information, and the The third fatigue information and the corresponding weight values α, β, and γ calculate the fatigue degree. When the fatigue degree exceeds the preset warning value, an audible/visual alarm is issued and the expert database system is started.

Preferably, obtaining facial features includes the following steps:

Preprocessing the in-cab image;

Perform face recognition on the pre-processed cab image to obtain a face image;

The coordinates of the characteristic key points in the face image are extracted and input into the preset multi-task classification neural network to identify the eye state, mouth state and head pitch angle.

Preferably, the loss function of the multi-task classification neural network is:

F _{loss_face_total} =F _{loss_leye} +F _{loss_mouse} +F _{loss_angle}

in the formula represent the true values of the pitch angles of the eyes, mouth and head respectively, represent the network prediction values of the eyes, mouth and head pitch angles respectively, and c ₁ =c ₂ =c ₃ represents L2 regularization.

Preferably, obtaining the heart rate value includes the following steps:

Extract the forehead area based on the coordinates of the feature key points;

Calculate the mean value of the G channel in the forehead area and perform blind source separation on the mean signal of the G channel to obtain the separated G channel source signal;

Fast Fourier transform is performed on the separated G channel source signal to obtain the heart rate value.

Preferably, the loss function of the sequential neural network is:

In the formula, L _{face_+} represents the loss of positive samples; L _{face_-} represents the loss of negative samples, p represents the probability value of the time series network output, ∈+ represents the positive sample weight parameter, ∈- represents the negative sample weight parameter; γ ₁ , γ ₂ , γ ₃ represents the loss weight of eyes, mouth, and pitch angle respectively.

Preferably, the road surface image outside the cab is input into the trained YOLOP model, the lane line information of the road is obtained, the lane line driving time is extracted from the lane line information, and the second fatigue information is obtained based on the lane line driving time, including the following step:

Preprocess the road surface image outside the cab;

Input the preprocessed road surface image outside the cab into the trained YOLOP model to detect the lane line information of the road;

During the detection process, the time when the vehicle turns on the turn signal and drives on the line is recorded, and based on the relationship between the time of driving on the line and the preset driving threshold, the second fatigue information is determined.

Preferably, the preprocessing of the road surface image outside the cab includes one or more combinations of image denoising, image dehazing, image rain removal, image deblurring, image color enhancement, image brightness enhancement or image detail enhancement.

Preferably, calculating the steering wheel shake characteristics based on the steering wheel angle information, and obtaining the third fatigue information based on the steering wheel shake characteristics includes the following steps:

Perform filtering and denoising on the steering wheel angle information to obtain denoised steering wheel angle information;

Based on the denoised steering wheel angle information, calculate the approximate entropy of the steering wheel angle within a time window, and its approximate entropy is used as the steering wheel jitter characteristic value;

Based on the relationship between the approximate entropy and the preset jitter threshold, third fatigue information is determined.

Preferably, starting the expert database system includes the following steps:

Real-time tracking and prediction of the driver's vehicle's offset from the lane centerline and the distance to the vehicle in front, and obtaining prediction results;

When the prediction result exceeds the preset offset threshold or distance threshold, human-machine takeover is performed.

Based on the above content, the present invention also discloses a driving fatigue detection and early warning control device for human-machine co-driving, including: an image acquisition device, an information acquisition module, a control module and an early warning device, wherein,

The image acquisition device is used to collect images inside the cab and road images outside the cab in real time and transmit them to the control module;

The information collection module is used to collect steering wheel angle information during driving and transmit it to the control module;

The control module is used to obtain facial features and heart rate values based on in-cab image recognition. The facial features include eye state, mouth state and head pitch angle, and input the in-cab image and the extracted facial features and heart rate values simultaneously. The preset time-series neural network is processed to obtain the first fatigue information; it is used to input the road surface image outside the cab into the trained YOLOP model, obtain the lane line information of the road surface, and extract the lane line driving time in the lane line information, based on The second fatigue information is obtained from the driving time on the line; it is used to calculate the steering wheel jitter characteristics based on the steering wheel angle information, and the third fatigue information is obtained based on the steering wheel jitter characteristics; it is used to build a deep learning model and use the adaptive weight algorithm for training, and obtain the first The weight values α, β, and γ corresponding to the fatigue information, the second fatigue information, and the third fatigue information are combined with the first fatigue information, the second fatigue information, and the third fatigue information and the corresponding weight values α, β, and γ to calculate the fatigue degree , and send early warning instructions to the early warning device according to the degree of fatigue;

The early warning device is used to receive the early warning instruction and perform an early warning according to the early warning instruction; and is used to start the expert database system.

Based on the above technical solutions, the beneficial effects of the present invention are:

1) The present invention uses remote photoplethysmography (rPPG) based on video remote heart rate measurement technology to solve the problem that the contact method requires a wearable device to detect heart rate, and then uses a deep learning fusion algorithm to combine the driver's facial features ( Eyes, mouth, head pitch angle), heart rate, road lane line characteristics and steering wheel angle information are combined according to a certain weight to monitor the driver's fatigue status in real time. If the driver is detected to be driving fatigued, an alarm and warning light will be broadcast through voice Drivers are reminded to take breaks. Eliminate the disadvantages caused by a single detection method and improve the reliability of the early warning system;

2) When monitoring driver fatigue, the present invention not only performs sound/light alarms to remind the driver, but also tracks and predicts the offset of the driving vehicle from the lane centerline and the distance to the vehicle in front in real time, based on the prediction results and The relationship between the preset offset threshold or distance threshold is used for human-machine takeover, and the speed and steering wheel angle during driving are automatically adjusted to ensure the safety of the driver in a fatigued state.

Description of the drawings

Figure 1 is a flow chart 1 of a driving fatigue detection and early warning control method for human-machine co-driving in one embodiment;

Figure 2 is a flow chart 2 of a driving fatigue detection and early warning control method for human-machine co-driving in one embodiment;

Figure 3 is a flow chart of a detection method based on facial features and heart rate in a driving fatigue detection and early warning control method for human-machine co-driving in one embodiment;

Figure 4 is a flow chart of a detection method based on the vehicle driving process in a driving fatigue detection and early warning control method for human-machine co-driving in one embodiment;

Figure 5 is a schematic structural diagram of a driving fatigue detection and early warning control device for human-machine co-driving in one embodiment.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

As shown in Figures 1 and 2, this embodiment provides a driving fatigue detection and early warning control method for human-machine co-driving, including the following steps:

Step 1: Collect images inside the cab, road images outside the cab, and steering wheel angle information in real time.

In this embodiment, the infrared camera is used to capture the cab in real time, which ensures that clearer face images are collected at night. The captured video is collected at a frame rate of 30fps to obtain the cab images. A high-definition camera is used to capture the outside of the cab in real time, and the video is collected at a frame rate of 30fps to obtain the road surface image outside the cab.

The steering wheel angle data information is collected through the steering wheel sensor, and the sampling time point is consistent with the time point of the collection frequency of images inside the cab and road images outside the cab.

Step 2: Obtain facial features and heart rate values based on in-cab image recognition. The facial features include eye state, mouth state and head pitch angle, and input the in-cab image and the extracted facial features and heart rate values to the preset at the same time. Sequential neural network processing is used to obtain the first fatigue information.

Referring to Figure 3, first input the image in the cab into the pre-trained light enhancement neural network model for pre-processing, and output the light-enhanced image. The formula is as follows:

X _enhance =ZeroDCE(X _raw )

Among them, X _raw represents the original image collected by the infrared camera, and the optimized image X _enhance is obtained after passing through the pre-trained ZeroDCE light enhancement network.

Then, the face detection algorithm is used to identify the coordinate position of the driver's face in the current image, and then it is intercepted from the light-enhanced image as a region of interest (ROI), and then the region of interest ROI is input into the pre-trained In the multi-task classification neural network (Q2L-SPP-MobileViT), the multi-task classification neural network outputs the position of the driver's eyes, mouth, and head pitch angle. For the extracted forehead ROI area, obtain the mean value of the G channel in the forehead area, then use an improved wavelet transform filtering algorithm, and then perform fast Fourier transform to obtain the heart rate value, and finally calculate the driver's blink frequency, yawn frequency, Whether the head pitch angle transformation value and heart rate exceed the set threshold are used to determine whether the driver is in a fatigue state. The specific instructions are as follows:

FaceDetect(X _enhance ):

detector=dlib.get_frontal_face_detector()

predictor=dlib.shape_predictor("shape_predictor_68_face_landmarks.dat")

rect=detector(X _enhance , 1)#Detected face frame

landmark ₆₈ =predictor(gray,rect)#Get 68 key point coordinates based on the face frame

return rect，landmark ₆₈

rect, landmark ₆₈ = FaceDetect(X _enhance )

Use the dlib _algorithm to detect faces _{. The input} is the light- _enhanced image ); landmark ₆₈ is the coordinates of the 68 characteristic key points of the human face;

Based _on the _coordinates of 68 _feature key points, the person's left _eye yawning or closing the mouth);

Get the mean Xforehead _{G_mean} of the G channel of the cropped forehead X _forehead area

Xforehead _{G_mean} =nunpy.mean(X _forehead [:,:,1])

The numpy.mean() function represents the average of the parameters in brackets; the parameter X _forehead [:,:,1] represents the total value of the G channel traversing the forehead X _forehead area.

Perform blind source separation on the original G channel mean signal to obtain filtered clean mean data.

Perform fast Fourier transform to get the final heart rate value:

Finally, the obtained 4-dimensional vector of the left eye state, mouth state, heart rate value and driver pitch angle is used as part of the input of the fatigue recognition network, and the output is Y ₁ {fatigue, normal} in the fatigue state or normal state.

V＝{E _leye , E _mouse , E _hr , E _angle }

The input of the multi-task classification neural network is I ₁ :

I ₁ ={V,X _enhance }

Y ₁ =Net (I ₁ )

Among them, Net() represents a multi-task classification neural network.

The loss function of multi-task classification neural network is:

F _{loss_face_total} =F _{loss_leye} +F _{loss_mouse} +F _{loss_angle}

in represent the true values of the pitch angles of the eyes, mouth and head respectively, Represents the network prediction values of the eyes, mouth and head pitch angles respectively, c ₁ =c ₂ =c ₃ indicates that the L2 regularization weight is 0.01.

Temporal neural network loss function:

Among them, L _{face_+} represents the loss of positive samples (eyes open, yawning state), L _{face_-} (eyes closed, mouth closed state) represents the loss of negative samples, p represents the probability value of the time series network output, ∈+ represents the positive sample The weight parameter value is 1, ∈- indicates that the negative sample weight parameter value is 2; γ ₁ , γ ₂ , and γ ₃ respectively indicate that the loss weights of the eyes, mouth, and pitch angle are 0.3, 0.3, and 0.1 respectively.

In this embodiment, the input of the time series neural network is not only the driver's facial image information captured by the camera sensor, but also the output results of the previous multi-task classification neural network, that is, the eye opening and closing status, the mouth opening and closing status, and the pitch angle. It is fed into the temporal neural network as input at the same time as the image in the cab, which is equivalent to adding regularization to the temporal neural network to prevent network learning bias. It avoids the general end-to-end network that only inputs the entire image captured by the camera into the network, ignoring that during the normal driving process of the car, the background is complex and changeable, which affects the accuracy of the network.

Step 3: Input the road surface image outside the cab into the trained YOLOP model, obtain the lane line information of the road surface, extract the crossing driving time in the lane line information, and obtain the second fatigue information based on the crossing driving time.

Referring to Figure 4, first input the road surface image outside the cab into the trained denoising, dehazing, and rain removing multi-task model iPT (image processing transformer) to obtain a clear road image, and then input the clear road image into the trained The lane line information of the road surface is detected in the YOLOP model. If it is detected that the vehicle turns on the turn signal but is still driving on the line, the time of driving on the line is recorded. If it exceeds the threshold, it is judged to be in a fatigue state.

The principle of collecting road images with high-definition cameras is similar to that of using infrared cameras to collect driver facial features, and finally the output Y ₂ {fatigue, normal} is obtained.

Step 4: Calculate the steering wheel vibration characteristics based on the steering wheel angle information, and obtain the third fatigue information based on the steering wheel vibration characteristics.

In this embodiment, a filtering algorithm is used to perform filtering and denoising, and the denoised steering wheel angle information is obtained. Based on the calculation of the denoised steering wheel angle information, the approximate entropy of the steering wheel angle within a time window is calculated. Here, approximate entropy is used to quantify a period of time. The random sequence irregularity of the inner steering wheel angle is used, and the calculated approximate entropy is used as the steering wheel jitter characteristic value; it is judged whether the approximate entropy continues to exceed the preset jitter threshold, if so, it is judged to be in a fatigue state, and then the result is input into Y ₃ {fatigue ,normal}.

The calculation formula of approximate entropy of steering wheel angle is as follows:

In the formula, m is the number of embedding dimensions, r is the similarity tolerance threshold, N is the total number of data points in the observation period or the length of the input time series, and B _i represents the two X(i) and X(j) at i The number of vectors whose distance is smaller than the similarity tolerance threshold r, where X(·) represents the m-dimensional vector reconstructed from the input time series.

X(i)＝[X _SWA (i), X _SWA (i+1),..., X _SWA (i+m-1)], X _SWA (i)∈R ^m

X(j)＝[X _SWA (j), X _SWA (j+1),..., X _SWA (j+m-1)], X _SWA (j)∈R ^m

Step 5: Construct a deep learning model and use the adaptive weight algorithm for training, obtain the weight values α, β, and γ corresponding to the first fatigue information, the second fatigue information, and the third fatigue information respectively, and combine the first fatigue information, the second fatigue information, and the third fatigue information. The fatigue information, the third fatigue information and the corresponding weight values α, β, and γ are used to calculate the fatigue degree. When the fatigue degree exceeds the preset warning value, an audible/visual alarm is issued and the expert database system is started.

In this embodiment, the calculation formula of fatigue degree is as follows:

Y{fatigue,normal}＝αY ₁ {fatigue,normal}+βY ₂ {fatigue,normal}+γY ₃ {fatigue,normal}

Specifically, the preset initialization weight of the face image detection result is α=0.7, the weight of the detection result based on road lane lines is β=0.1, and the weight of the detection result based on the steering wheel angle is γ=0.2. After running the system, the system will train the deep learning model based on the accumulated historical data, use the Y value output as the input of the adaptive network, and output the values of weights α, β, and γ to achieve the effect of adaptive weights and consider the driver personal habits to improve the robustness and detection accuracy of the model.

See Figure 2, Q2L-SPP-MobileViT is the specific model of multi-task neural network. The general end-to-end model extracts the features of the driver's face into an image sequence and directly performs inference through the LSTM network. This invention combines the images in the cab and the output eye state, mouth state, pitch angle and heart rate value of the multi-task neural network. As it is fed into the improved LSTM network, the Self_attention mechanism is added to allow the model to pay as much attention as possible to some features that can reflect the fatigue state. Finally, a final output is output based on the first fatigue information, the second fatigue information and the third fatigue information and the weight values α, β and γ corresponding to the first fatigue information, the second fatigue information and the third fatigue information obtained using the adaptive weight algorithm. The degree of fatigue can be adaptively changed according to the driver's style, making the model more flexible and greatly reducing the false detection rate.

Among them, considering that when driver fatigue is detected, only a sound/light alarm is used to remind the driver that he is currently in a fatigue state, which can only protect the driver's safety to a very small extent. Assume that the driver is driving on the highway. When the system detects that he is driving fatigued, sound/light alarm alone cannot allow the driver to recover from the fatigue state. However, he cannot stop and rest at will on the highway, assuming that he is in a fatigue state. Drivers need to drive to the nearest service area to take a rest. During this process, the driver is still in a state of fatigue and his reaction time slows down, so the possibility of a traffic accident is still high. Based on the sound and light alarm, this system proposes an expert database system based on driver fatigue status to take over driving vehicles in dangerous and emergency situations. The specific implementation plan is as follows:

When the driver is in a fatigue state, the expert library system is started to track and predict the speed of the driving vehicle, the offset of the driver's vehicle from the lane center line, and the distance to the vehicle in front in real time.

Assume that T[x_1,x_2,…,x_t] is the offset sequence of the driving vehicle from the lane centerline within a period of time W. Using the modified steering mechanism model in the expert library, the model training data is the vehicle's offset from the lane centerline and the steering wheel angle when the turn signal is not turned on, and the model uses the non-increasing offset sequence T in the W time period as Bonus function (the greater the deviation of the driver's vehicle from the lane centerline within a period of time W, the greater the possibility of a traffic accident.) During the actual operation of the system, experts predict the driving trajectory within a period of time W based on the current offset sequence, send the offset sequence within the W time period into the model, and the output is the decision result, that is, whether to correct the driving trajectory. Make corrections. When the vehicle does not turn on the turn signal and the offset continues to be greater than the threshold A, the human-machine takes over and controls the steering wheel angle to keep it within the safe range of the lane line offset.

In view of the dangerous collision situation with the vehicle in front, the vehicle distance maintenance mechanism model in the expert library is called. The model training data is the speed of the driver's vehicle, the accelerator and brake pedal signals, and the distance between the main vehicle and the vehicle in front. The bonus function of the model is the distance between vehicles. Among them, RS represents the distance between the main vehicle and the vehicle in front, V _m represents the speed of the driving main vehicle, and T _s represents the minimum safety time. The speed of the main vehicle and the distance between the main vehicle and the preceding vehicle in the W time period are fed into the distance maintenance mechanism model, and the decision result is output, that is, whether to control the accelerator and brake pedal signals. If the output result is yes, a corrected value needs to be output. The value of the accelerator pedal. Finally, the operation results are fed back to the control module, and the control module adjusts the accelerator and brake pedals to maintain a safe distance between the main vehicle and the vehicle in front.

It should be understood that although each step in the above flowchart is shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this article, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in the above flow chart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but may be executed at different times. The order of execution is not necessarily sequential, but may be performed in turn or alternately with other steps or sub-steps of other steps or at least part of the stages.

In one embodiment, as shown in Figure 5, a driving fatigue detection and early warning control device for human-machine co-driving is provided, including: an image acquisition device 110, an information acquisition module 120, a control module 130 and an early warning device 140, wherein,

The image acquisition device 110 is used to collect images inside the cab and road images outside the cab in real time and transmit them to the control module;

The information collection module 120 is used to collect steering wheel angle information during driving and transmit it to the control module;

The control module 130 is used to obtain facial features and heart rate values based on in-cab image recognition. The facial features include eye state, mouth state and head pitch angle, and combine the in-cab image with the extracted facial features and heart rate values. Input to the preset time-series neural network processing to obtain the first fatigue information; used to input the road surface image outside the cab into the trained YOLOP model, obtain the lane line information of the road surface, and extract the lane line driving time in the lane line information, The second fatigue information is obtained based on the driving time on the line; it is used to calculate the steering wheel jitter characteristics based on the steering wheel angle information, and the third fatigue information is obtained based on the steering wheel jitter characteristics; it is used to build a deep learning model and use the adaptive weight algorithm for training, and obtain the third fatigue information respectively. The weight values α, β, and γ corresponding to the first fatigue information, the second fatigue information, and the third fatigue information are combined with the first fatigue information, the second fatigue information, and the third fatigue information and the corresponding weight values α, β, and γ to calculate the fatigue degree, and sends an early warning instruction to the early warning device 140 according to the degree of fatigue;

The early warning device 140 is used to receive the early warning instruction, perform an early warning according to the early warning instruction, and start the expert database system.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage. In the media, when executed, the computer program may include the processes of the above method embodiments. Any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and/or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous chain Synchlink DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

The technical features of the above embodiments can be combined in any way. To simplify the description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, all possible combinations should be used. It is considered to be within the scope of this manual.

The above-described embodiments only express several implementation modes of the present application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that, for those of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all fall within the protection scope of the present application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims (9)

1. A driving fatigue detection and early warning control method for man-machine co-driving is characterized by comprising the following steps:

acquiring an image in a cab, an image of a road surface outside the cab and steering wheel corner information in real time;

facial features and heart rate values are obtained according to the image recognition in the cab, wherein the facial features comprise eye states, mouth states and head pitching angles, and the image in the cab and the extracted facial features and heart rate values are input into a preset time sequence neural network for processing at the same time to obtain first fatigue information;

inputting an image of the road surface outside the driver cab into a trained YOLOP model, acquiring lane line information of the road surface, extracting line pressing running time in the lane line information, and acquiring second fatigue information based on the line pressing running time;

calculating steering wheel shake features according to the steering wheel angle information, and obtaining third fatigue information based on the steering wheel shake features;

constructing a deep learning model, training by adopting a self-adaptive weight algorithm, respectively acquiring weight values alpha, beta and gamma corresponding to the first fatigue information, the second fatigue information and the third fatigue information, calculating the fatigue degree by combining the first fatigue information, the second fatigue information and the third fatigue information and the corresponding weight values alpha, beta and gamma, and when the fatigue degree exceeds a preset early warning value, sending out an audible/visual alarm, tracking and predicting the offset of the driving host vehicle from the lane center line and the distance from the front vehicle in real time, and acquiring a prediction result; and when the predicted result exceeds a preset offset threshold or distance threshold, taking over the man-machine connection.

2. The method for detecting and controlling the fatigue driving and the early warning of the man-machine co-driving according to claim 1, wherein the step of obtaining the facial features comprises the following steps:

preprocessing the image in the cab;

carrying out face recognition on the preprocessed image in the cab to obtain a face image;

and extracting feature key point coordinates in the face image, inputting the feature key point coordinates into a preset multitask classification neural network, and identifying the eye state, the mouth state and the head pitching angle.

3. The method for detecting and controlling fatigue and early warning of driving of a man-machine co-driving according to claim 2, wherein the loss function of the multi-task classification neural network is as follows:

F _{loss_face_total} ＝F _{loss_leye} +F _{loss_mouse} +F _{loss_angle}

in the middle ofTrue values of eye, mouth and head pitch angles are respectively represented,network predictions, c, representing eye, mouth and head pitch angles, respectively ₁ ＝c ₂ ＝c ₃ Representing L2 regularization.

4. The method for detecting and controlling fatigue and early warning of driving of a man-machine co-driving according to claim 2, wherein the obtaining of the heart rate value comprises the following steps:

extracting a forehead region based on the feature key point coordinates;

calculating the average value of the G channel in the forehead area, and performing blind source separation on the average value signal of the G channel to obtain a separated G channel source signal;

and performing fast Fourier transform on the separated G channel source signals to obtain heart rate values.

5. The method for detecting and controlling driving fatigue and early warning of man-machine co-driving according to claim 1, wherein the loss function of the time sequence neural network is as follows:

in which L _{face_+} Indicating a loss of positive samples; l (L) _{face_-} Representing the loss of the negative sample, p representing the probability value output by the time sequence network, epsilon+ representing the positive sample weight parameter, epsilon-representing the negative sample weight parameter; gamma ray ₁ ,γ ₂ ,γ ₃ Loss weights of eyes, mouth, pitch angle are respectively represented.

6. The method for detecting and controlling fatigue and early warning of driving of man-machine co-driving according to claim 1, wherein the steps of inputting the road surface image outside the driving cab into a trained YOLOP model, obtaining lane line information of the road surface, extracting the line pressing driving time in the lane line information, and obtaining the second fatigue information based on the line pressing driving time include the following steps:

preprocessing the road surface image outside the cab;

inputting the preprocessed road surface image outside the cab into a trained YOLOP model to detect lane line information of the road surface;

and recording the time of turning on the steering lamp and driving the wire by the vehicle in the detection process, and determining second fatigue information based on the relation between the time of driving the wire and a preset driving threshold value.

7. The method for detecting and controlling fatigue and early warning of driving of a man-machine co-driving according to claim 6, wherein the preprocessing of the road surface image outside the driving cab comprises one or more of image denoising, image defogging, image rain removing, image deblurring, image color enhancement, image brightness enhancement or image detail enhancement.

8. The method for detecting and controlling fatigue and early warning of co-driving of man-machine according to claim 1, wherein the calculating steering wheel shake features according to steering wheel angle information and obtaining third fatigue information based on the steering wheel shake features comprises the following steps:

filtering and denoising the steering wheel angle information to obtain denoised steering wheel angle information;

calculating the approximate entropy of the steering wheel angle in a time window based on the denoised steering wheel angle information, wherein the approximate entropy is used as a steering wheel shake characteristic value;

and determining third fatigue information based on the relation between the approximate entropy and a preset jitter threshold.

9. A man-machine co-driving fatigue detection and early warning control device is characterized by comprising: the system comprises an image acquisition device, an information acquisition module, a control module and an early warning device, wherein,

the image acquisition device is used for acquiring images in a cab and road images outside the cab in real time and transmitting the images to the control module;

the information acquisition module is used for acquiring steering wheel angle information in the driving process and transmitting the steering wheel angle information to the control module;

the control module is used for obtaining facial features and heart rate values according to the image recognition in the cab, wherein the facial features comprise eye states, mouth states and head pitching angles, and the image in the cab and the extracted facial features and heart rate values are simultaneously input into a preset time sequence neural network for processing to obtain first fatigue information; the method comprises the steps of inputting an outdoor pavement image of a driver's cabin into a trained YOLOP model, obtaining lane line information of a pavement, extracting line pressing running time in the lane line information, and obtaining second fatigue information based on the line pressing running time; the method comprises the steps of calculating steering wheel shake features according to steering wheel angle information, and obtaining third fatigue information based on the steering wheel shake features; the method comprises the steps of constructing a deep learning model, training by adopting a self-adaptive weight algorithm, respectively obtaining weight values alpha, beta and gamma corresponding to first fatigue information, second fatigue information and third fatigue information, calculating the fatigue degree by combining the first fatigue information, the second fatigue information and the third fatigue information and the corresponding weight values alpha, beta and gamma, and sending an early warning instruction to the early warning device according to the fatigue degree;

the early warning device is used for receiving the early warning instruction, sending out sound/light warning according to the early warning instruction, tracking and predicting the offset of the driving main vehicle from the center line of the lane and the distance from the front vehicle in real time, and obtaining a prediction result; and when the predicted result exceeds a preset offset threshold or distance threshold, taking over the man-machine connection.