CN117842085A - Driving state detection and early warning method, driving state detection and early warning system, electronic equipment and storage medium - Google Patents

Abstract

The application provides a driving state detection and early warning method, a driving state detection and early warning system, electronic equipment and a storage medium, wherein the driving state detection and early warning method comprises the following steps: acquiring driver state detection data, vehicle state detection data and road environment state detection data uploaded by a monitoring terminal; analyzing the driver state detection data, the vehicle state detection data and the road environment state detection data to obtain a driving state evaluation result; and carrying out early warning based on the driving state evaluation result. The method provided by the application is beneficial to improving the accuracy of driving state evaluation, carrying out risk early warning according to the driving state and improving the driving safety.

Description

Driving state detection and warning method, system, electronic device and storage medium

Technical Field

The present application relates to the field of driving safety technology, and in particular to a driving status detection and early warning method, system, electronic device and storage medium.

Background technique

With the development of the automobile industry, cars have become a convenient means of travel for people, but the incidence of traffic accidents is also increasing, and people are paying more and more attention to driving safety. In order to ensure driving safety, relevant technologies detect the driver's fatigue state through the camera in the car, so as to judge the driving safety of the vehicle, and issue a warning in time when the driver is driving fatigued. However, there are many reasons for traffic accidents, not only fatigue driving, but also vehicle failure. Therefore, it is not comprehensive to only detect the driver's fatigue state, and it is impossible to accurately determine the cause of the accident, and it is impossible to accurately judge the driving state, and it is impossible to issue a risk warning based on the driving state.

Summary of the invention

The present application provides a driving status detection and warning method, system, electronic device and storage medium, which are helpful to improve the accuracy of driving status assessment and improve driving safety.

In a first aspect, the present application provides a driving state detection and warning method, comprising:

Acquire driver status detection data, vehicle status detection data, and road environment status detection data uploaded by a monitoring terminal, wherein the monitoring terminal includes a plurality of detection devices, and the detection devices include at least a physiological signal detection device, a video signal detection device, and a vehicle signal detection device;

Analyzing the driver state detection data, the vehicle state detection data and the road environment state detection data to obtain a driving state evaluation result;

A warning is issued based on the driving status evaluation result.

In one possible implementation, the driver state includes a fatigue state, a distraction state, and an emotional state;

The vehicle state includes a normal driving state and an abnormal driving state;

The road environment status at least includes no vehicle or pedestrian ahead, vehicle collision ahead, lane deviation, vehicles too close, frequent lane changes, and other vehicles cutting in.

In one possible implementation, the method further includes:

When the vehicle is traveling along a set route and in a set state, collecting first state detection data of the driver;

Performing standardization processing according to the collected first state detection data of the driver;

The driver state detection data acquired in real time is associated with the first state detection data of the driver after standardization.

In one possible implementation, after associating the driver state detection data acquired in real time with the first state detection data of the driver after standardization, the method further includes:

The driver state detection data is subjected to noise processing based on the first state detection data of the driver to obtain the second state detection data of the driver, wherein the first state detection data of the driver at least includes the physiological signal, the first eye movement signal, the electroencephalogram and brain imaging signal, and the behavior detection data of the driver;

The second state detection data of the driver, the vehicle state detection data and the road environment state detection data are analyzed to obtain the driving state evaluation result.

In one possible implementation, the driver state detection data is subjected to noise processing based on the first state detection data of the driver, including:

The driver state detection data includes a second eye movement signal, and the second eye movement signal is obtained by parsing the video content acquired in real time by the video signal detection device;

In combination with the first eye movement signal, artifact noise of the second eye movement signal is determined, and the artifact noise is removed.

In one possible implementation, the fatigue state includes mild fatigue, moderate fatigue and severe fatigue, and the fatigue state is obtained by analyzing data detected by the physiological signal detection device or the video signal detection device; and

The distraction state includes behavioral distraction and cognitive distraction, wherein the behavioral distraction is obtained by analyzing the data detected by the video signal detection device, and the cognitive distraction is obtained by analyzing the data detected by the physiological signal detection device; and

The emotional state includes emotional intensity and emotional valence. The emotional intensity is obtained by analyzing the data detected by the physiological signal detection device, and the emotional valence is obtained by analyzing the data detected by the video signal detection device.

In one possible implementation, the emotion intensity includes abnormal emotion and normal emotion, the emotion valence includes positive emotion and negative emotion, the positive emotion is used to represent positive emotion, and the negative emotion is used to represent negative emotion.

In one possible implementation, the analyzing the driver state detection data, the vehicle state detection data, and the road environment state detection data includes: inputting the driver state detection data, the vehicle state detection data, and the road environment state detection data into corresponding state recognition models, respectively, to identify the driver state, the vehicle state, and the road environment state;

The driving state evaluation result is obtained by combining the driver state, the vehicle state and the road environment state.

In one possible implementation, the combining the driver state, the vehicle state, and the road environment state to obtain the driving state evaluation result includes:

If the driver state is mild fatigue, the vehicle state is normal driving state, and the road environment state is no vehicle or pedestrian ahead, the driving state assessment result is a first driving risk state;

If the driver state is moderate fatigue, the vehicle state is an abnormal driving state, and the road environment state is no vehicle or pedestrian ahead, the driving state assessment result is a second driving risk state;

If the driver state is severe fatigue, the vehicle state is an abnormal driving state, and the road environment state is another vehicle cutting in, then the driving state assessment result is a third driving risk state.

In one possible implementation, the state recognition model includes a driver state recognition model, the driver state recognition model includes a cognitive distraction state recognition model, and the training of the cognitive distraction state recognition model includes:

Collect eye movement and EEG data of one or more drivers when performing dual tasks and single tasks multiple times within a preset time interval or at different times;

Comparing the eye movement and EEG data collected multiple times to remove invalid data;

averaging the eye movement and EEG data collected multiple times after removing invalid data;

The cognitive distraction state recognition model is trained by a machine learning algorithm based on the average value.

In one possible implementation, the method further includes:

Marking bad driving habits based on the driver status detection data, wherein the bad driving habits at least include smoking, making phone calls and cutting in;

Counting the number of occurrences of the bad driving habits;

When the number of occurrences of the bad driving habit behavior is greater than a preset number, the driver is reminded or warned.

In one possible implementation, the method further includes:

Determine whether the signal detected by the detection device is a normal signal by using a neural network algorithm,

If the signal is an abnormal signal, the driver is prompted to debug the equipment through voice or image.

In one possible implementation, the method further includes:

The monitoring terminal starts the plurality of detection devices synchronously or individually, and the driver status detection data, the vehicle status detection data and the road environment status detection data are synchronously uploaded to the monitoring terminal by the detection devices.

In a second aspect, the present application provides a driving state detection and warning device, comprising:

An acquisition module, used to acquire driver status detection data, vehicle status detection data and road environment status detection data uploaded by a monitoring terminal, wherein the monitoring terminal includes a plurality of detection devices, and the detection devices include at least a physiological signal detection device, a video signal detection device and a vehicle signal detection device;

An analysis module, used for analyzing the driver state detection data, the vehicle state detection data and the road environment state detection data to obtain a driving state evaluation result;

The warning module is used to issue a warning based on the driving status evaluation result.

In a third aspect, the present application provides an electronic device comprising: a processor and a memory, wherein the memory is used to store a computer program; the processor is used to run the computer program to implement a driving status detection and warning method as described in the first aspect.

In a fourth aspect, the present application provides a driving status detection and warning system, including: the electronic device as shown in the third aspect.

In a fifth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer-readable storage medium is run on a computer, the computer implements a driving status detection and warning method as described in the first aspect.

In this application, driver status detection data, vehicle status detection data and road environment status detection data are detected simultaneously by multiple different detection devices, and the driver status detection data, vehicle status detection data and road environment status detection data are comprehensively analyzed to obtain a driving status evaluation result, and then an early warning is issued based on the driving status evaluation result. This application evaluates the driving status from three dimensions: driver status, vehicle status and road environment status, which helps to improve the accuracy of driving status evaluation, issue risk early warnings based on the driving status, and improve driving safety.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is an architecture diagram of a driving status detection and warning system provided in an embodiment of the present application;

FIG2 is a flow chart of a driving state detection and warning method provided in an embodiment of the present application;

FIG3 is a flow chart of a method for training a cognitive distraction state recognition model according to an embodiment of the present application;

FIG4 is a flow chart of another driving state detection and warning method provided in an embodiment of the present application;

FIG5 is a schematic diagram of the structure of a driving state detection and warning device provided in an embodiment of the present application;

FIG6 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

Detailed ways

In the embodiments of the present application, unless otherwise specified, the character "/" indicates that the objects before and after the association are in an or relationship. For example, A/B can represent A or B. "And/or" describes the association relationship of the associated objects, indicating that three relationships can exist. For example, A and/or B can represent: A exists alone, A and B exist at the same time, and B exists alone.

It should be pointed out that the words "first", "second", etc. involved in the embodiments of the present application are only used to distinguish the description purpose, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated, nor can they be understood as indicating or implying order.

In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. In addition, "at least one of the following" or similar expressions refers to any combination of these items, which may include any combination of single items or plural items. For example, at least one of A, B, or C may represent: A, B, C, A and B, A and C, B and C, or A, B and C. Among them, each of A, B, and C may be an element itself, or a set containing one or more elements.

In the embodiments of the present application, "exemplary", "in some embodiments", "in another embodiment", etc. are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present concepts in a concrete way.

In the embodiments of the present application, "of", "corresponding", and "corresponding" can sometimes be used interchangeably. It should be noted that when the distinction between them is not emphasized, the meanings to be expressed are consistent. In the embodiments of the present application, communication and transmission can sometimes be used interchangeably. It should be noted that when the distinction between them is not emphasized, the meanings to be expressed are consistent. For example, transmission can include sending and/or receiving, which can be a noun or a verb.

The equal to involved in the embodiments of the present application can be used in conjunction with greater than, and is applicable to the technical solution adopted when greater than, and can also be used in conjunction with less than, and is applicable to the technical solution adopted when less than. It should be noted that when equal to is used in conjunction with greater than, it cannot be used in conjunction with less than; when equal to is used in conjunction with less than, it cannot be used in conjunction with greater than.

In the related art, the driving state is judged by detecting the driver's fatigue state, but the driving state is evaluated only by the single dimension of fatigue state, and the accuracy of the evaluation is not high. The safety of vehicle driving is not only related to the driver's state, but also to the vehicle state and the road environment state. And the driver's state does not only include fatigue state, but also distraction state, emotional state, etc. The detection of these states will also affect the accuracy of driving state evaluation and the accuracy of driving risk evaluation. For example, when the driver is in a state of emotional tension, the vehicle is driving normally at this time, and there are no pedestrians and vehicles in front. In this case, the driving risk level is low, and the corresponding warning behavior is also mild; similarly, when the driver is in a state of emotional tension, the vehicle is driving abnormally at this time, such as the vehicle deviates from the road, and the vehicle condition ahead is complicated, then the driving risk level in this case will increase, and higher requirements for warnings are needed.

Based on the above problems, an embodiment of the present application proposes a driving status detection and warning method, which evaluates the driving status from three dimensions: driver status, vehicle status and road environment status, which helps to improve the accuracy of driving status evaluation, issue risk warnings based on the driving status, and improve driving safety.

The driving status detection and warning method provided in the embodiment of the present application is now described in conjunction with Figures 2 to 4.

FIG1 is an architecture diagram of a driving state detection and warning system provided in an embodiment of the present application. As shown in FIG1 , the driving state detection and warning system includes a human factors intelligent monitoring platform and a real-time monitoring terminal. The human factors intelligent monitoring platform can monitor a real-time monitoring terminal, and can also monitor multiple real-time monitoring terminals simultaneously. When the real-time monitoring terminal is started, the human factors intelligent monitoring platform and the real-time monitoring terminal communicate with each other through a network protocol, such as HTTP (Hypertext Transfer Protocol) or TCP (Transmission Control Protocol). The real-time monitoring terminal uploads the terminal information to the human factors intelligent monitoring platform, and the human factors intelligent monitoring platform matches the uploaded terminal information with the terminal information pre-stored in the platform, thereby determining the monitoring terminal that has been successfully matched. The human factors intelligent monitoring platform can perform real-time visual presentation and processing of the warning status of the successfully matched real-time monitoring terminal, and can also perform visual presentation, data statistics and analysis of the historical information of the warning status of the real-time monitoring terminal.

Each real-time monitoring terminal includes multiple detection devices, including physiological signal detection devices, video signal detection devices, vehicle signal detection devices, motion capture detection devices, etc. The physiological signal detection device is used to detect multimodal physiological data. Multimodal physiological data refers to the driver's status data collected by multiple sensors. The sensor can be a physiological sensor that obtains the driver's physiological data when driving. For example, eye tracking devices, skin electricity, electrocardiogram, breathing and brain waves. The data collected by the eye tracking device include but are not limited to the pupil diameter, blinking frequency, gaze time, and gaze number of the driver's eye movement. The detection device is used to detect the driver's status, vehicle status and road environment status. The real-time monitoring terminal can turn on each detection device synchronously with one button, or turn on each detection device separately. The multimodal physiological data acquisition sensor can be integrated in one terminal device, or a system can be formed by multiple acquisition sensors. The terminal device integrating multiple acquisition sensors can be configured in the local driving vehicle. The acquisition sensor can also be set on the local driving vehicle, and the terminal device for analysis and processing can be set on the server side.

The real-time monitoring terminal detects the signal and sends it to the cloud server. The cloud server can use the neural network algorithm to determine whether the detected signal is a normal signal, that is, whether it is a high-quality and effective signal. If the signal is not a normal signal, the driver is prompted to debug the equipment through voice or image to obtain a normal detection signal. After the real-time monitoring terminal is connected to the detection equipment, in order to ensure the quality of data presentation, the original data is stored in the local client, and after the monitoring is completed, the data is uploaded to the cloud server to form a historical database.

FIG2 is a flow chart of a driving state detection and warning method provided in an embodiment of the present application, which specifically includes the following steps:

Step S21, obtaining driver status detection data, vehicle status detection data and road environment status detection data uploaded by the monitoring terminal.

In this embodiment, multiple detection devices detect and obtain the driver's state detection data, vehicle state detection data and road environment state detection data during the driving process. After the monitoring terminal is connected to the detection device, the monitoring terminal uploads these data to the human factors intelligent monitoring platform. Among them, the driver's state detection data not only includes multimodal physiological data, but also includes other behavioral data of the driver during driving, data used to express the driver's emotions or emotions, and data used to judge the driver's perception ability. The data for judging the perception ability can be obtained by detecting the feedback of the driver during driving through different operations. The driver's state during driving is judged by the above detected data.

In some embodiments, the driver state includes a fatigue state, a distraction state, and an emotional state.

Optionally, the fatigue state includes mild fatigue, moderate fatigue and severe fatigue. The fatigue state is obtained by analyzing the data detected by the physiological signal detection device or the video signal detection device. For example, EEG (Electroencephalogram) detection is used to obtain EEG data, PSD (power spectral density) data, etc., or the driver's pupil state changes or blinking state changes are obtained through in-car camera detection, and these data or state changes are analyzed to obtain the driver's fatigue level during driving.

In some embodiments, the distraction state includes behavioral distraction and cognitive distraction. Behavioral distraction includes making phone calls, smoking, chatting, not looking forward for a long time, etc.; cognitive distraction includes low distraction, moderate distraction, and high distraction.

Optionally, behavioral distraction is obtained by analyzing data detected by a video signal detection device, such as detecting dangerous driving behaviors of the driver due to distraction through a behavioral distraction state detection device (in-vehicle camera, video monitor, etc.); cognitive distraction is obtained by analyzing data detected by a physiological signal detection device, such as detecting HRV (Heart Rate Variability), HR (Heart Rate) heart rate, IBI (Inter-Beat Interval, i.e. R-R interval), SCR (Skin conductance response) data, etc. through a PPG (photoplethysmography) wireless pulse sensor or an EDA (electrodermal activity) wireless skin activity sensor, and analyzing these data to obtain the degree of cognitive distraction.

In some embodiments, the emotional state includes emotional intensity and emotional valence. Emotional intensity includes abnormal emotions and normal emotions; emotional valence includes positive emotions and negative emotions. Positive emotions represent positive emotions such as happiness and excitement, and negative emotions represent negative emotions such as anger, irritation, and sadness.

Optionally, the intensity of emotion is obtained by analyzing data detected by a physiological signal detection device, for example, by using a PPG (photoplethysmography) wireless pulse sensor or an EDA (electrodermal activity) wireless skin activity sensor to detect HRV (Heart Rate Variability), HR (Heart Rate) heart rate, IBI (Inter-Beat Interval, i.e. R-R interval), SCR (Skin conductance response) data, etc., and analyzing these data. For example, whether the emotion is abnormal is judged based on the change of the SCR (Skin conductance response) value per minute; the valence of emotion is obtained by analyzing data detected by a video signal detection device (such as a facial expression detection device). For example, the valence of the driver's emotion is judged as positive or negative by detecting changes in the driver's facial expressions.

In some embodiments, the vehicle state includes a normal driving state and an abnormal driving state. The vehicle state is obtained by analyzing the data detected by the vehicle signal detection device. Specifically, Vehub detects the vehicle's running speed, steering wheel data, pedal data, GPS (Global Positioning System) data, etc., and analyzes the collected data, including lateral acceleration analysis, center of mass sideslip angle constraint analysis, friction circle constraint analysis, tire sideslip angle analysis, and built-in advanced driving normal driving analysis, so as to determine whether the vehicle state is normal.

Furthermore, real-time coding and post-coding can be performed for driving behaviors, and driving behaviors can be analyzed and counted. Driving behaviors include driving pedal behaviors (brake, accelerator), steering behaviors (left turn, right turn), lane change behaviors (left lane change, right lane change, reversing, U-turn, right lane driving, left lane driving), driving speed (high speed, medium speed, low speed, parking or idling), longitudinal acceleration (strong speed, general acceleration, weak acceleration) and steering wheel behaviors (clockwise, counterclockwise), so as to judge whether the vehicle is driving normally. For example, by analyzing the driving pedal behavior, it can be judged whether the driver frequently brakes suddenly. If the driver frequently brakes suddenly, it indicates that the current vehicle is not driving normally.

In some embodiments, the road environment status includes no vehicle or pedestrian ahead, vehicle collision ahead, lane departure, vehicle too close, frequent lane changes, other vehicle cutting in, vehicle ahead speeding, pedestrian collision, etc. Among them, vehicle collision ahead, pedestrian collision, vehicle too close, and frequent lane changes include yes and no; lane departure includes left deviation and right deviation; speeding level includes low, medium, and high; other vehicle cutting in includes left and right cutting in.

Optionally, the road environment state is detected by an ADAS (Advanced Driving Assistance System) system. Optionally, the road environment state may also include environmental data detected by a third party, such as weather, temperature, humidity, etc.

Step S22, analyzing the driver state detection data, the vehicle state detection data and the road environment state detection data to obtain a driving state evaluation result.

Specifically, the driver state detection data, vehicle state detection data and road environment state detection data are respectively input into the corresponding state recognition model to identify the driver state, vehicle state and road environment state; and the driving state evaluation result is obtained by combining the driver state, vehicle state and road environment state.

Optionally, if the driver state is mild fatigue, the vehicle state is normal driving state, and the road environment state is no vehicle or pedestrian ahead, the driving state assessment result is the first driving risk state;

If the driver is in moderate fatigue, the vehicle is in an abnormal driving state, and the road environment is in a state where there are no vehicles or pedestrians ahead, the driving state assessment result is a second driving risk state;

If the driver is in severe fatigue, the vehicle is in an abnormal driving state, and the road environment is in which another vehicle cuts in, the driving state assessment result is the third driving risk state.

It should be noted that the present application may also include other indicators for evaluating driving status, and may also evaluate driving status based on driver status detection data and vehicle status detection data, or based on vehicle status detection data and road environment status detection data, or based on driver status detection data and road environment status detection data. For example, if the driver status is mild fatigue and the vehicle status is an abnormal driving state, the driving evaluation result is the fourth driving risk state, or, if the vehicle status is an abnormal driving state and the road environment status is lane deviation, the driving evaluation result is the fifth driving risk state. The above is only an exemplary description, and the present application does not limit the division of risk levels and evaluation indicators of driving status.

In some embodiments, the acquired multimodal data (including driver state detection data, vehicle state detection data, and road environment state detection data) is input into a convolutional neural network model to map a multimodal matrix, and the convolutional neural network model is a convolutional neural network model with a 3*3 structure; the multimodal matrix is imported into a Transformer model to calculate the correlation weights between different data elements in the multimodal discrete data, that is, to obtain the correlation weights between the multimodal data, and the driving state evaluation result is obtained by combining the correlation weights.

Specifically, by using this 3x3 convolutional neural network, the number of discrete data can be mapped into an m-dimensional vector. This vector can be regarded as a feature representation of discrete data, where each dimension corresponds to a feature of discrete data. Assuming there are n discrete data, after being processed by the 3x3 convolutional neural network, a 3n×3m-dimensional multimodal original matrix can be obtained. In this matrix, each row represents a feature representation of discrete data, and each column represents a different feature. The multimodal original matrix retains the feature information of discrete data, and through feature extraction and mapping of the convolutional neural network, the discrete data is converted into a vectorized form. The multimodal original matrix is then imported into the Transformer model, and correlation calculation is performed in the Transformer model based on the Self-Attention mechanism. The Self-Attention mechanism allows the model to analyze each element in the input sequence, and then adjust the weight according to the relationship between them, and combine the correlation weight to obtain the driving state evaluation result.

In some embodiments, the state recognition model includes a driver state recognition model, and the driver state recognition model includes a cognitive distraction state recognition model. As shown in FIG3 , FIG3 is a flow chart of a cognitive distraction state recognition model training method provided in an embodiment of the present application, and the cognitive distraction state recognition model training method includes:

Step S31 , collecting eye movement and EEG data of one or more drivers when performing dual tasks and single tasks multiple times within a preset time interval or at different times.

In the early stage of model building, a large amount of data from different scenarios needs to be accumulated to train the model. In this application, the eye movement and EEG data information of a driver performing dual tasks and single tasks are collected, as well as the eye movement and EEG data information of multiple different drivers in dual tasks and single tasks, and multiple collections and records are performed to ensure the completeness of the data.

Furthermore, the collected data information corresponding to one or more different drivers can be collected multiple times in different time periods or at different times. For example, data information is collected within a preset time interval or at different times. For example, information is collected at 8 to 11 o'clock, 14 to 17 o'clock, or at 6 o'clock, 10 o'clock, 13 o'clock, 15 o'clock, 19 o'clock, 12 o'clock at night, 2 o'clock in the morning, etc., and data information in different states is recorded according to different time periods or times. Because each driver has different cognitive distraction states in different time periods or times, when training the model, collecting data from different time periods or times for training can increase the completeness and accuracy of the data, ensure the reliability of the model, and improve the accuracy of driving state assessment, thereby better improving driving safety.

Step S32, comparing the eye movement and EEG data collected multiple times to remove invalid data.

Specifically, by comparing data collected multiple times and deleting invalid data, such as abnormal data or data with large deviations, the accuracy of the data can be improved and the amount of data processing can be reduced.

Step S33, averaging the eye movement and EEG data collected multiple times after removing invalid data.

Specifically, after removing invalid data, the remaining data are averaged and saved.

Step S34: training the cognitive distraction state recognition model through a machine learning algorithm based on the average value.

However, in the process of identifying the driver's distracted state, it is easy to misjudge the distracted state. For example, when the driver is driving normally, for the sake of driving safety, sometimes he needs to turn his head or eyes to look at the left and right rearview mirrors, or because of the bumps of the vehicle, the driver's heart rate or EEG data will change, which will cause the system to misjudge that the driver is distracted during driving.

Further, as shown in FIG. 4 , FIG. 4 is a flow chart of another driving state detection and warning method provided in an embodiment of the present application, which specifically includes the following steps:

Step S41, when the vehicle travels on a set route in a set state, collecting first state detection data of the driver.

When the vehicle is driving in a set state on a set route, the set state refers to the normal driving state of the driver, for example, the driver is not distracted by making phone calls, smoking, chatting, not looking forward for a long time, and is not tired or other compliant driving states. At this time, the first state detection data of the driver is collected, that is, the first state detection data of the driver refers to the detection data of the normal driving state. The first state detection data of the driver includes at least the physiological signal, the first eye movement signal, the EEG and brain imaging signal, and the behavior detection data of the driver.

Step S42: performing standardization processing based on the collected first state detection data of the driver.

Specifically, the collected first state detection data of the driver is converted into a specific unified specification so that the data is kept within a certain interval. For example, the heart rate is converted into a data interval format, and the heart rate values within the interval belong to the normal driving state of the driver. The first state detection data of the driver is standardized to eliminate the differences in the properties, dimensions, magnitudes and other attribute characteristics between different detection data, thereby converting them into dimensionless standardized values.

Step S43, associating the driver status detection data acquired in real time with the first status detection data of the driver after standardization.

In step S21, multiple detection devices collect in real time the driver status detection data of other drivers to be detected during the driving process, including multimodal physiological data (including EEG data, eye movement data, heart rate, etc.), behavioral data, etc. The driver status detection data collected in step S21 is compared and associated with the first status detection data of the driver collected in step S41 to determine which of the driver status detection data belong to the detection data interval of the normal driving state.

Step S44: the driver state detection data is subjected to noise processing based on the driver's first state detection data to obtain the driver's second state detection data.

In some embodiments, the driver state detection data includes a second eye movement signal, which is obtained by parsing the video content acquired in real time by the video signal detection device; the first eye movement signal in the driver's first state detection data is combined to determine the artifact noise of the second eye movement signal, that is, to determine the eye movement signal in the second eye movement signal that belongs to the normal driving state but is misjudged as the distracted state. The artifact noise is removed to obtain the second state detection data of the driver, that is, the driver state detection data after the artifact noise is removed is obtained.

Optionally, noise processing may also be performed on other data in the driver status detection data, such as heart rate, EEG signals, behavioral data, etc.

Step S45, analyzing the second state detection data of the driver, the vehicle state detection data and the road environment state detection data to obtain the driving state evaluation result.

The embodiment of the present application obtains the driver's normal driving state data in advance and performs data standardization, compares and correlates the collected driving state detection data with the standardized normal driving state data, and then performs noise processing to find and remove artifact noise, thereby removing the impact of misjudgment of distraction state and improving the accuracy of driving state assessment.

Based on the driving state evaluation result obtained in step S22 or step S45, step S23 is continued to be executed, that is, an early warning is issued based on the driving state evaluation result.

Specifically, the human-factor intelligent monitoring platform uses voice or visual images to remind or warn the driver based on the driving status assessment results. For example, when the driver is tired due to driving for too long, the human-factor intelligent monitoring platform can use voice or image to remind the driver: "You have been driving for a long time, please stop and rest"; or when the driver is distracted by making a phone call, and the current speed is fast and the surrounding traffic conditions are complicated, the human-factor intelligent monitoring platform can send an alarm to the vehicle to remind the driver to pay attention to the vehicle ahead.

Furthermore, based on the driver status detection data, bad driving habits are marked, including smoking, talking on the phone, cutting in, not looking ahead for a long time, frequent emergency braking, frequent speeding, etc. In addition, the number of occurrences of bad driving habits is counted; when the number of occurrences of bad driving habits is greater than a preset number (for example, 3 times), the driver is reminded or warned. Optionally, for vehicles with automatic driving systems, when it is determined that the operation of surrounding vehicles is safe, the vehicle can pull over by controlling the steering wheel, brakes and turn signals.

At the same time, by collecting driver status detection data, vehicle status detection data, road environment status detection data, human-machine interaction data (such as HMI (Human Machine Interface) human-machine interaction data, HUD (head-up display) human-machine interaction data), EEG and eye movement data, it is possible to evaluate the driver's basic abilities, such as alertness, attention span, memory, etc.; or evaluate the driver's special abilities, such as spatial orientation ability, motor coordination ability, peripheral vision level; or evaluate the driver's personality characteristics; or evaluate driving suitability, etc. The system can also further construct a data portrait of the driver's cognitive ability to provide data support for the evaluation, training and selection of driver abilities.

In this application, driver status detection data, vehicle status detection data and road environment status detection data are detected simultaneously by multiple different detection devices, and the driver status detection data, vehicle status detection data and road environment status detection data are comprehensively analyzed to obtain driving status evaluation results, and then an early warning is issued based on the driving status evaluation results. This application evaluates the driving status from three dimensions: driver status, vehicle status and road environment status, which helps to improve the accuracy of driving status evaluation and improve driving safety. At the same time, the driver status, vehicle status and road environment status are graded, such as dividing the driving status into fatigue status, distraction status and emotional status, and further subdividing the fatigue status, distraction status and emotional status, so as to further improve the accuracy of driving status evaluation, issue risk warnings based on the driving status, and improve driving safety.

FIG5 is a schematic diagram of the structure of a driving state detection and warning device provided in an embodiment of the present application. As shown in FIG5 , the driving state detection and warning device 50 may include:

An acquisition module 51 is used to acquire driver status detection data, vehicle status detection data and road environment status detection data uploaded by a monitoring terminal, wherein the monitoring terminal includes a plurality of detection devices, and the detection devices include at least a physiological signal detection device, a video signal detection device and a vehicle signal detection device;

An analysis module 52 is used to analyze the driver state detection data, the vehicle state detection data and the road environment state detection data to obtain a driving state evaluation result;

The warning module 53 is used to issue a warning based on the driving status evaluation result.

In one possible implementation, the driver state includes a fatigue state, a distracted state, and an emotional state;

The vehicle status includes normal driving status and abnormal driving status;

The road environment status at least includes no vehicles or pedestrians ahead, collision with the vehicle ahead, lane deviation, vehicles too close, frequent lane changes, and other vehicles cutting in.

In one possible implementation, the driving state detection and warning device 50 may further include:

A collection module, used for collecting first state detection data of the driver when the vehicle is traveling in a set state on a set route;

A standardization processing module, used for performing standardization processing according to the collected first state detection data of the driver;

The association module is used to associate the driver status detection data acquired in real time with the driver's first status detection data after standardization.

In one possible implementation, the driving state detection and warning device 50 may further include:

The noise processing module is used to perform noise processing on the driver state detection data based on the driver's first state detection data to obtain the driver's second state detection data. The driver's first state detection data at least includes the driver's physiological signal, first eye movement signal, EEG and brain imaging signal, and behavior detection data.

In one possible implementation, the analysis module 52 may also be used to:

The driver's second state detection data, the vehicle state detection data and the road environment state detection data are analyzed to obtain a driving state evaluation result.

In one possible implementation, the driver status detection data includes a second eye movement signal, and the second eye movement signal is obtained by analyzing video content acquired in real time by a video signal detection device.

In one possible implementation manner, the noise processing module may also be used to: determine artifact noise of the second eye movement signal in combination with the first eye movement signal, and remove the artifact noise.

In one possible implementation, the fatigue state includes mild fatigue, moderate fatigue and severe fatigue, and the fatigue state is obtained by analyzing data detected by a physiological signal detection device or a video signal detection device;

The distraction state includes behavioral distraction and cognitive distraction. The behavioral distraction is obtained by analyzing the data detected by the video signal detection device, and the cognitive distraction is obtained by analyzing the data detected by the physiological signal detection device.

The emotional state includes emotional intensity and emotional valence. The emotional intensity is obtained by analyzing the data detected by the physiological signal detection device, and the emotional valence is obtained by analyzing the data detected by the video signal detection device.

In one possible implementation, emotion intensity includes abnormal emotion and normal emotion, emotion valence includes positive emotion and negative emotion, positive emotion is used to represent positive emotion, and negative emotion is used to represent negative emotion.

In one possible implementation, the analysis module 52 may also be used to:

The driver state detection data, the vehicle state detection data and the road environment state detection data are respectively input into the corresponding state recognition model to identify the driver state, the vehicle state and the road environment state;

The driving state evaluation result is obtained by combining the driver state, vehicle state and road environment state.

In one possible implementation, the driving state evaluation result obtained by combining the driver state, the vehicle state and the road environment state includes:

If the driver is in a state of mild fatigue, the vehicle is in a normal driving state, and the road environment is in a state of no vehicles or pedestrians ahead, the driving state assessment result is a first driving risk state;

If the driver is in moderate fatigue, the vehicle is in an abnormal driving state, and the road environment is in a state where there are no vehicles or pedestrians ahead, the driving state assessment result is a second driving risk state;

If the driver is in severe fatigue, the vehicle is in an abnormal driving state, and the road environment is in which another vehicle cuts in, the driving state assessment result is the third driving risk state.

In one possible implementation, the state recognition model includes a driver state recognition model, the driver state recognition model includes a cognitive distraction state recognition model, and the training of the cognitive distraction state recognition model includes:

Collect eye movement and EEG data of one or more drivers when performing dual tasks and single tasks multiple times within a preset time interval or at different times;

Compare the eye movement and EEG data collected multiple times and remove invalid data;

The eye movement and EEG data collected multiple times were averaged after invalid data were removed;

The cognitive distraction state recognition model is trained through a machine learning algorithm based on the average value.

In one possible implementation, the driving state detection and warning device 50 may further include:

A marking module, used for marking bad driving habits based on the driver status detection data, where the bad driving habits at least include smoking, making phone calls and cutting in;

Statistics module, used to count the number of bad driving habits;

The reminder module reminds or warns the driver when the number of bad driving habits exceeds the preset number.

In one possible implementation, the driving state detection and warning device 50 may further include:

The judgment module uses a neural network algorithm to determine whether the signal detected by the detection equipment is a normal signal. If the signal is an abnormal signal, it prompts the driver to debug the equipment through voice or image.

In one possible implementation, the monitoring terminal simultaneously turns on or individually turns on multiple detection devices, and the driver status detection data, vehicle status detection data, and road environment status detection data are synchronously uploaded to the monitoring terminal by the detection devices.

The driving status detection and warning device 50 provided in the embodiment shown in FIG5 can be used to execute the technical solution of the method embodiment shown in the present application. Its implementation principle and technical effects can be further referred to the relevant description in the method embodiment.

It should be understood that the division of the various modules of the driving state detection and warning device 50 shown in Figure 5 above is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware. For example, the detection module can be a separately established processing element, or it can be integrated in a chip of an electronic device. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or they can be implemented independently. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more microprocessors (DSP), or one or more field programmable gate arrays (FPGA). For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

In the above embodiments, the processor involved may include, for example, a CPU, a DSP, a microcontroller or a digital signal processor, and may also include a GPU, an embedded neural network processor (Neural-network Process Units; hereinafter referred to as: NPU) and an image signal processor (Image Signal Processing; hereinafter referred to as: ISP). The processor may also include necessary hardware accelerators or logic processing hardware circuits, such as ASIC, or one or more integrated circuits for controlling the execution of the program of the technical solution of the present application. In addition, the processor may have the function of operating one or more software programs, and the software programs may be stored in a storage medium.

The embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the method provided by the embodiment shown in the present application.

An embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is run on a computer, it enables the computer to execute the method provided by the embodiment shown in the present application.

The embodiment of the present application also provides an electronic device for implementing the above-mentioned driving state detection and warning method. The embodiment of the present application does not limit the type of electronic device, and the electronic device can be a desktop computer, a tablet computer, a laptop computer, a PDA, an ultra-mobile personal computer (UMPC) or a netbook, etc.

The exemplary electronic device provided in the embodiments of the present application is further described below in conjunction with Fig. 6. Fig. 6 shows a schematic diagram of the structure of an electronic device 6000.

The above-mentioned electronic device 6000 may include: at least one processor; and at least one memory communicatively connected to the above-mentioned processor, wherein: the above-mentioned memory stores program instructions that can be executed by the above-mentioned processor, and the processor calls the above-mentioned program instructions to execute the driving status detection and warning method provided in the embodiment shown in this application.

Fig. 6 shows a block diagram of an exemplary electronic device 6000 suitable for implementing the embodiments of the present application. The electronic device 6000 shown in Fig. 6 is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

As shown in Fig. 6, electronic device 6000 is in the form of a general computing device. Components of electronic device 6000 may include but are not limited to: one or more processors 6010, memory 6020, a communication bus 6040 connecting different system components (including memory 6020 and processor 6010), and a communication interface 6030.

The communication bus 6040 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnection (PCI) bus.

The electronic device 6000 typically includes a variety of computer system readable media, which can be any available media that can be accessed by the electronic device, including volatile and nonvolatile media, removable and non-removable media.

The memory 6020 may include a computer system readable medium in the form of a volatile memory, such as a random access memory (Random Access Memory; hereinafter referred to as: RAM) and/or a cache memory. The electronic device may further include other removable/non-removable, volatile/non-volatile computer system storage media. Although not shown in FIG. 6, a disk drive for reading and writing a removable non-volatile disk (such as a "floppy disk"), and an optical disk drive for reading and writing a removable non-volatile optical disk (such as: a compact disc read only memory (Compact Disc Read Only Memory; hereinafter referred to as: CD-ROM), a digital versatile read-only optical disk (Digital Video Disc Read Only Memory; hereinafter referred to as: DVD-ROM) or other optical media) may be provided. In these cases, each drive can be connected to the communication bus 6040 via one or more data medium interfaces. The memory 6020 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present application.

A program/utility having a set (at least one) of program modules may be stored in the memory 6020, such program modules including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules generally perform the functions and/or methods of the embodiments described herein.

The electronic device 6000 may also communicate with one or more external devices (e.g., keyboards, pointing devices, displays, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device, and/or communicate with any device that enables the electronic device to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed through the communication interface 6030. In addition, the electronic device 6000 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and/or public networks, such as the Internet) through a network adapter (not shown in FIG. 6 ), and the network adapter may communicate with other modules of the electronic device through the communication bus 6040. It should be understood that, although not shown in FIG. 6 , other hardware and/or software modules may be used in conjunction with the electronic device 6000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, disk arrays (RAID) systems, tape drives, and data backup storage systems.

The processor 6010 executes various functional applications and data processing by running the programs stored in the memory 6020, such as implementing the method provided in the embodiment of the present application.

It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 6000. In other embodiments of the present application, the electronic device 6000 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

In the above embodiments, the processor involved may include, for example, a CPU, a DSP, a microcontroller or a digital signal processor, and may also include a GPU, an embedded neural network processor (Neural-network Process Units; hereinafter referred to as: NPU) and an image signal processor (Image Signal Processing; hereinafter referred to as: ISP). The processor may also include necessary hardware accelerators or logic processing hardware circuits, such as ASIC, or one or more integrated circuits for controlling the execution of the program of the technical solution of the present application. In addition, the processor may have the function of operating one or more software programs, and the software programs may be stored in a storage medium.

Those of ordinary skill in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented in a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

In several embodiments provided in the present application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory; hereinafter referred to as: ROM), random access memory (Random Access Memory; hereinafter referred to as: RAM), disk or optical disk, and other media that can store program codes.

The above are only specific implementations of the present application. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application should be based on the protection scope of the claims.

Claims (17)

1. A driving status detection and early warning method, characterized in that the method comprises: Acquire driver status detection data, vehicle status detection data, and road environment status detection data uploaded by a monitoring terminal, wherein the monitoring terminal includes a plurality of detection devices, and the detection devices include at least a physiological signal detection device, a video signal detection device, and a vehicle signal detection device; Analyzing the driver state detection data, the vehicle state detection data and the road environment state detection data to obtain a driving state evaluation result; A warning is issued based on the driving status evaluation result. 2. The method according to claim 1, characterized in that The driver state includes fatigue state, distraction state and emotional state; The vehicle state includes a normal driving state and an abnormal driving state; The road environment status at least includes no vehicle or pedestrian ahead, vehicle collision ahead, lane deviation, vehicles too close, frequent lane changes, and other vehicles cutting in. 3. The method according to claim 2, characterized in that the method further comprises: When the vehicle is traveling along a set route and in a set state, collecting first state detection data of the driver; Performing standardization processing according to the collected first state detection data of the driver; The driver state detection data acquired in real time is associated with the first state detection data of the driver after standardization. 4. The method according to claim 3, characterized in that after associating the driver state detection data acquired in real time with the first state detection data of the driver after standardization, the method further comprises: The driver state detection data is subjected to noise processing based on the first state detection data of the driver to obtain the second state detection data of the driver, wherein the first state detection data of the driver at least includes the physiological signal, the first eye movement signal, the electroencephalogram and brain imaging signal, and the behavior detection data of the driver; The second state detection data of the driver, the vehicle state detection data and the road environment state detection data are analyzed to obtain the driving state evaluation result. 5. The method according to claim 4, characterized in that the driver state detection data is subjected to noise processing based on the driver's first state detection data, comprising: The driver state detection data includes a second eye movement signal, and the second eye movement signal is obtained by parsing the video content acquired in real time by the video signal detection device; In combination with the first eye movement signal, artifact noise of the second eye movement signal is determined, and the artifact noise is removed. 6. The method according to claim 2, characterized in that The fatigue state includes mild fatigue, moderate fatigue and severe fatigue, and the fatigue state is obtained by analyzing the data detected by the physiological signal detection device or the video signal detection device; and The distraction state includes behavioral distraction and cognitive distraction, wherein the behavioral distraction is obtained by analyzing the data detected by the video signal detection device, and the cognitive distraction is obtained by analyzing the data detected by the physiological signal detection device; and The emotional state includes emotional intensity and emotional valence. The emotional intensity is obtained by analyzing the data detected by the physiological signal detection device, and the emotional valence is obtained by analyzing the data detected by the video signal detection device. 7. The method according to claim 6, characterized in that The emotion intensity includes abnormal emotion and normal emotion, and the emotion valence includes positive emotion and negative emotion. The positive emotion is used to represent positive emotion, and the negative emotion is used to represent negative emotion. 8. The method according to claim 6, characterized in that the analyzing the driver state detection data, the vehicle state detection data and the road environment state detection data comprises: Inputting the driver state detection data, the vehicle state detection data and the road environment state detection data into corresponding state recognition models respectively, and identifying the driver state, the vehicle state and the road environment state; The driving state evaluation result is obtained by combining the driver state, the vehicle state and the road environment state. 9. The method according to claim 8, characterized in that the step of combining the driver state, the vehicle state and the road environment state to obtain the driving state evaluation result comprises: If the driver state is mild fatigue, the vehicle state is normal driving state, and the road environment state is no vehicle or pedestrian ahead, the driving state assessment result is a first driving risk state; If the driver state is moderate fatigue, the vehicle state is an abnormal driving state, and the road environment state is no vehicle or pedestrian ahead, the driving state assessment result is a second driving risk state; If the driver state is severe fatigue, the vehicle state is an abnormal driving state, and the road environment state is another vehicle cutting in, then the driving state assessment result is a third driving risk state. 10. The method according to claim 8, characterized in that The state recognition model includes a driver state recognition model, the driver state recognition model includes a cognitive distraction state recognition model, and the training of the cognitive distraction state recognition model includes: Collect eye movement and EEG data of one or more drivers when performing dual tasks and single tasks multiple times within a preset time interval or at different times; Comparing the eye movement and EEG data collected multiple times to remove invalid data; averaging the eye movement and EEG data collected multiple times after removing invalid data; The cognitive distraction state recognition model is trained by a machine learning algorithm based on the average value. 11. The method according to claim 1, characterized in that the method further comprises: Marking bad driving habits based on the driver status detection data, wherein the bad driving habits at least include smoking, making phone calls and cutting in; Counting the number of occurrences of the bad driving habits; When the number of occurrences of the bad driving habit behavior is greater than a preset number, the driver is reminded or warned. 12. The method according to claim 1, further comprising: Determine whether the signal detected by the detection device is a normal signal by using a neural network algorithm, If the signal is an abnormal signal, the driver is prompted to debug the equipment through voice or image. 13. The method according to claim 1, characterized in that the method further comprises: The monitoring terminal starts the plurality of detection devices synchronously or individually, and the driver status detection data, the vehicle status detection data and the road environment status detection data are synchronously uploaded to the monitoring terminal by the detection devices. 14. A driving status detection and warning device, characterized by comprising: An acquisition module, the acquisition module is used to acquire driver status detection data, vehicle status detection data and road environment status detection data uploaded by a monitoring terminal, the monitoring terminal includes a plurality of detection devices, the detection devices at least include a physiological signal detection device, a video signal detection device and a vehicle signal detection device; An analysis module, the analysis module is used to analyze the driver state detection data, the vehicle state detection data and the road environment state detection data to obtain a driving state evaluation result; An early warning module is used to issue an early warning based on the driving status evaluation result. 15. An electronic device, characterized in that it comprises: a processor and a memory, wherein the memory is used to store a computer program; the processor is used to run the computer program to implement the driving state detection and warning method as described in any one of claims 1 to 13. 16. A driving status detection and warning system, characterized in that: Comprising the electronic device as claimed in claim 15. 17. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, and when the computer program is run on a computer, the driving state detection and warning method as described in any one of claims 1 to 13 is implemented.