CN110334592A - A driver abnormal behavior monitoring and safety assurance system and method thereof - Google Patents

Abstract

The present invention relates to a driver's abnormal behavior monitoring and safety guarantee system and method thereof. The system includes a driver's driving image acquisition device, a driver's biological signal acquisition device, an intelligent Internet of Things terminal, a cloud server and an application mobile terminal; the driver's driving image acquisition device is installed Right in front of the driver's cab; the driver's biological signal collection device includes a biological signal sensor, which includes multiple heart rate sensors and blood oxygen concentration sensors and is installed in the steering wheel cover; the smart Internet of Things terminal is set in the cab and performs data fusion. The result of data fusion is sent to the cloud server; the application mobile terminal interacts with the cloud server to obtain driver data. The present invention can accurately and real-time monitor the driver's facial expression information and physiological information, can accurately and effectively discover the driver's abnormal behavior, facilitates the driver himself, relatives, friends, and management personnel to check the driver's driving data in real time, and issues a warning when necessary, providing a powerful Assure.

Description

A driver abnormal behavior monitoring and safety assurance system and method thereof

technical field

The invention belongs to the technical field of artificial intelligence and the Internet of Things, in particular to a driver abnormal behavior monitoring and safety guarantee system and method thereof.

Background technique

Traffic safety has long been one of the most concerned issues for people. Although artificial intelligence and electronic information industry are developing rapidly at present, detection methods for abnormal driver behavior and recognition algorithms based on computer vision emerge in an endless stream, but in the field of driving safety early warning, there are still various difficult problems in the products on the market.

At present, domestic and foreign research on detecting abnormal driving behaviors of drivers mainly includes three types: (1) Intelligent monitoring and safety assurance based on video images: it has low identification accuracy, poor robustness to interference, few practical applications and usually does not have Emergency alarm and tracking capabilities. (2) Based on vehicle driving data: there are problems of poor real-time performance and high deployment costs, so it is rarely accepted by the market. (3) Based on wearable devices, there are problems such as difficult practical application, coarse data granularity, no tracking function, and poor experience of warning users.

To sum up, there is a lack of a safe driving guarantee technology at home and abroad that can effectively monitor, effectively warn before accidents, and manage abnormal behaviors of drivers. As a result, the death rate of domestic traffic accidents remains high and affects road traffic safety.

Contents of the invention

The purpose of the present invention is to overcome the deficiencies in the prior art, and propose a driver abnormal behavior monitoring and safety guarantee system and method thereof, which integrates biological signals, artificial intelligence, and the driver abnormal behavior monitoring and safety guarantee system and method of the Internet of Things, It can monitor and manage abnormal driving behavior in real time.

The present invention solves its technical problem and realizes by taking the following technical solutions:

A driver's abnormal behavior monitoring and safety guarantee system, including a driver's driving image collection device, a driver's biological signal collection device, an intelligent Internet of Things terminal, a cloud server, and an application mobile terminal; the driver's driving image collection device is installed directly in front of the cab and Connect with the intelligent Internet of Things terminal by wire; the driver's biological signal acquisition device includes a biological signal sensor, which includes a plurality of heart rate sensors and blood oxygen concentration sensors and is installed in the steering wheel cover; the intelligent Internet of Things terminal Set in the cab, the smart IoT terminal receives data from the driver’s driving image collection device and the driver’s biological signal collection device and performs data fusion, and the result of data fusion is sent to the cloud server; the mobile terminal of the application interacts with the cloud server to acquire Driving driver data.

The driver's biosignal acquisition device also includes an electroencephalogram detection module worn on the driver's head. The electroencephalogram detection module includes a head electrode, an ear clip electrode, a Bluetooth communication module and a battery compartment, and the detection results of the head motor and the ear clip electrode Send it to the smart IoT terminal through the Bluetooth communication module.

The plurality of heart rate sensors and blood oxygen concentration sensors are installed on both sides of the steering wheel cover, and a narrow transparent belt is arranged on the steering wheel cover so that the light emitted by the sensors can pass through the transparent belt.

The intelligent Internet of Things terminal includes a microprocessor and a 4G communication module connected thereto, an acceleration sensor, an alcohol sensor and a voice module.

A method for realizing a driver abnormal behavior monitoring and safety assurance system, comprising the following steps:

Step 1, the smart Internet of Things terminal receives the real-time driver's driving image transmitted by the driver's driving image acquisition device;

Step 2, the smart Internet of Things terminal receives the driver's biological signal collected by the driver's biological signal collection device;

Step 3. The Internet of Things terminal performs decision-level data fusion on the driver's driving image and the driver's biological signal;

Step 4. The Internet of Things terminal fuses and uploads the decision-level data to the cloud server;

Step 5. The application mobile terminal communicates with the cloud server to obtain driving data.

After the step 5, it also includes the step of using the mobile terminal to interact with the driver's voice under abnormal circumstances.

The concrete realization method of described step 1 comprises the following steps:

(1) Face image acquisition and recognition steps; the driver's driving image acquisition device automatically searches for and detects the user's face image;

⑵Facial image denoising: corrosion of facial images, Gaussian denoising, brightness and contrast adjustment;

(3) Human eye recognition: call the calibration model of 68 feature points of the face, mark the points around the eyes, calculate the value of EAR in each frame, and use it to judge the blinking frequency and eye closing time, and identify driver fatigue and inattention ;

⑷ Driving Behavior Recognition: Call the pre-trained model to identify four behaviors of the driver: using a mobile phone, not looking straight ahead, not wearing a seat belt, and smoking.

The concrete realization method of described step 2 comprises the following steps:

⑴When the driver's hand touches the steering wheel, the light beams of the heart rate sensor and the blood oxygen concentration sensor hit the hand, and the heart rate and blood oxygen concentration are measured using the light volume method. After filtering, feature-level fusion is used, and deep learning-based The heart rate and blood oxygen concentration measurement method, by building a multi-hidden layer model and a large amount of training data;

(2) The brain wave detection module collects brain wave signals at the temple and earlobe;

(3) The data processing module in the brain wave detection module acquires the potential signal collected in (2), conducts frequency domain analysis on the EEG signal, performs filtering processing, and extracts α brain waves (8-13Hz) and β brain waves (13-20Hz) ) and θ wave (4～7Hz);

(4) The processing unit in the brainwave detection module inputs the extracted characteristic parameters into the fatigue judgment model for fatigue judgment.

The specific implementation method of step 4 is as follows: set n different types of sensors to identify m attributes of the same target,

First, classify the data observed by n sensors for m hypotheses A _j (j=1, 2..., m), and obtain a set of target descriptions B _i (i=1, 2..., n);

Secondly, calculate the conditional probabilities of each description when each hypothesis A _j (j=1, 2..., m) is established, and calculate the joint likelihood function of n target descriptions under m conditional assumptions:

Finally, the Bayesian formula is used to calculate the posterior density of various hypotheses A _j in the case of n target descriptions: The maximum a posteriori density max(P(A _j |B _i ) is obtained and used as a judgment basis to decide whether to accept or reject the A _j hypothesis, realize discarding redundant information, and complete decision-level data fusion.

The specific implementation method of the step 5 is: the user uses the application mobile terminal to obtain driver data through interaction with the server, and the user includes enterprise users and individual users; the individual user binds the terminal and the fleet to view the driving data report; the enterprise user creates Fleet, invite to join the fleet, and view the driving data of all members of the fleet.

Advantage and positive effect of the present invention are:

1. The present invention can accurately and real-time monitor the driver's facial expression information and physiological information through the driving image acquisition device and the driver's biological signal acquisition device, and can accurately and effectively find the driver's abnormal behavior, which is convenient for the driver himself, relatives, friends, and management personnel to check the driver's driving status in real time. Data, and when necessary, a warning is issued to provide strong protection for drivers.

2. The present invention adopts multi-dimensional sensing technology and integrates vision, voice modules, biosensors, alcohol, acceleration sensors, etc., and can effectively identify the status of the auto equipment itself and the driver's drunk driving.

3. The terminal and acquisition device adopted in the present invention are highly integrated, which is convenient for installation and use.

4. The present invention uses the analysis method of facial micro-expressions and head features, which improves the accuracy and speed of the reaction results, and monitors the driving state more accurately and effectively.

5. The present invention uses decision-level fusion technology to integrate biological signals such as heart rate, blood oxygen concentration, and brain wave with intelligent image processing to increase monitoring reliability and accuracy.

6. The application mobile terminal of the present invention can efficiently manage the driving state of the driver through data visualization in the Android application.

7. The present invention can provide drivers with early warning of dangerous behaviors through AI voice and driver intelligent interaction.

Description of drawings

Fig. 1 is the system connection schematic diagram of the present invention;

Fig. 2 is a schematic structural view of the intelligent steering wheel cover of the present invention;

Fig. 3 is a processing flow diagram of the present invention;

Fig. 4 is the block diagram of the process of identification of driver's abnormal behavior in the present invention;

Fig. 5 is mobile terminal APP communication logic and flowchart in the present invention;

Fig. 6 is a flowchart of voice interaction in the present invention.

Detailed ways

Embodiments of the present invention will be described in further detail below in conjunction with the accompanying drawings.

A driver's abnormal behavior monitoring and safety guarantee system, as shown in Figure 1, includes a driver's driving image acquisition device, an intelligent Internet of Things terminal, a driver's biological signal acquisition device, a cloud server, and an application mobile terminal.

The driver's driving image collection device is installed directly in front of the driver's cab and is connected to the smart Internet of Things terminal by wire to collect real-time images of the driver and transmit them to the smart Internet of Things terminal. In this embodiment, the driver's driving image acquisition device uses a camera.

The intelligent Internet of Things terminal is arranged in the cab, and the intelligent Internet of Things terminal includes a microprocessor and a 4G communication module connected thereto, an acceleration sensor, an alcohol sensor and a voice module. The 4G communication module communicates with the cloud server interactively, and performs voice interaction with the driver through the voice module. The smart IoT terminal receives the data from the driver's driving image collection device and the driver's biosignal collection device, analyzes the driver's image and the driver's biosignal and performs data fusion, and the result of the data fusion is sent to the cloud server.

The driver's biological signal collection device includes a biological signal sensor and a brain wave detection module.

As shown in Figure 2, the biological signal sensor includes a plurality of heart rate sensors and blood oxygen concentration sensors and is installed in the steering wheel cover. On the other hand, the appearance of the steering wheel is basically the same as that of an ordinary steering wheel. There is a narrow transparent band 3 in the steering wheel cover. The light emitted by the sensor can pass through the transparent band. In the hand, heart rate, blood oxygen concentration measurements are performed using photoplethysmography.

The brain wave detection module is worn on the driver's head. The brain wave detection module includes head electrodes, ear clip electrodes, Bluetooth communication modules and battery compartments. terminal for interactive communication.

The application mobile terminal obtains the driver data through interaction with the server.

Based on the above-mentioned driver abnormal behavior monitoring and safety assurance system, the driver abnormal behavior monitoring and safety assurance method of the present invention, as shown in Figures 3 to 6, includes the following steps:

Step 1. The smart Internet of Things terminal receives the real-time image of the driver transmitted by the driver's driving image acquisition device and analyzes the image data of the driver. Specific methods include:

(1) Face image acquisition and recognition: When the user is within the shooting range of the driver's driving image acquisition device, the driver's driving image acquisition device will automatically search for and detect the user's face image. Face detection can be used as preprocessing for face recognition, that is, to accurately find the position and size of the face in the image. The features contained in the face image of the video stream are very diverse, such as histogram features, color features, structural features, LBP features, and Haar features. After selecting the useful data through face detection, these features can be used to realize face detection.

(2) Image denoising: In order to improve the accuracy of recognition, the image must be denoised before recognition, including image erosion, Gaussian denoising (filtering), brightness and contrast adjustment.

(3) Human eye recognition: When the driver enters the fatigue state, the eye image has obvious characteristic changes. The eye aspect ratio EAR (eye aspect ratio) is roughly constant when the eyes are opened, but it will quickly change when the eyes blink. decline. Call the calibration model of 68 feature points of the face, mark the points around the eyes, and calculate the value of EAR for each frame, which is used to judge the blinking frequency and eye closing time, and identify the driver's fatigue and inattention.

(4) Driving behavior recognition: Invoke the pre-trained model to recognize the driver's four behaviors of using a mobile phone, not looking straight ahead, not wearing a seat belt, and smoking.

Step 2. The smart IoT terminal receives the driver's biological signal collected by the driver's biological signal collection device.

The driver's biological signal collected by the driver's biological signal collection device includes the driver's biological signal collected by the biological signal sensor and the brain wave detection module. The specific method is:

(1) Acquisition and processing of biological signal sensors

When the driver's hand touches the steering wheel, the beam of the sensor hits the hand, and the light volume method is used to measure the heart rate and blood oxygen concentration. After filtering, use feature-level fusion, use deep learning-based heart rate and blood oxygen concentration measurement methods, and learn more useful features by building a multi-hidden layer model and a large amount of training data, thereby improving the accuracy of judgment.

(2) Acquisition and processing of brain wave detection module:

①The brainwave detection module collects physiological signals including the driver's temporal and earlobe;

②The data processing module learns the potential signal collected in step ①, and then analyzes the EEG signal in the frequency domain, performs filtering processing, and extracts α brain waves (8-13Hz), β brain waves (13-20Hz) and θ waves (4～7Hz);

③The processing unit inputs the extracted characteristic parameters into the fatigue judgment model, and then outputs the obtained results as signals and uploads them to the smart IoT terminal;

Wherein, the fatigue judgment model index described in step ③ is as follows: when the driver is in a fatigued state, the activity of the human brain decreases, resulting in a decrease in β waves, but an increase in α waves. When the fatigue state changes to a sleep state, the EEG The main frequency will drop to the θ wave, and the EEG power ratio change rate L: For the set time window t ₀ , calculate the EEG power ratio:

K ₀ is the EEG power ratio at the beginning of the time window, for the EEG change rate

Preset threshold L ₀ . At the same time, since fatigue is a continuous process, this method adds a fatigue detection record stack, which is used to record the time and degree of fatigue detected each time. Whenever the fatigue state of the driver is detected, this module pushes the fatigue level and current time into the stack as a record, and the fatigue detection stack will be realized through a linked list. If the time interval from the last fatigue is short, the system will determine that it is in the same state and no longer upload the fatigue signal.

Step 3. The Internet of Things terminal performs decision-level data fusion on the received image information and biological information. The specific method is:

Use a data fusion algorithm based on Bayesian inference: there is a driver with abnormal behavior B, and n conditional assumptions A ₁ , A ₂ ,..., A _n , P(A _j ) represent the prior probability of different hypotheses, P(B |A _i ) is the probability of B occurring when the hypothesis Ai is established, that is, the posterior probability of event B relative to the hypothesis Ai, then:

The main idea of designing a multi-sensor data fusion method based on Bayesian reasoning is as follows: suppose _n different types of sensors can identify m attributes of the same target. , 2..., m) to classify the observed data to obtain a set of target descriptions B _i (i=1, 2..., n) about different attributes; secondly, calculate each hypothesis A _j (j=1, The conditional probability (likelihood function) of each description when 2..., m) is established, because each sensor is observed independently, so each target description B _i (i=1, 2..., n) is also independent of each other , then it is necessary to calculate the joint likelihood function of n target descriptions under m conditional assumptions:

Finally, the Bayesian formula is used to calculate the posterior density of various hypotheses A _j in the case of n target descriptions:

Furthermore, the maximum a posteriori density max(P(A _j |B _i ) can be obtained and used as a judgment basis to decide whether to accept or reject the A _j hypothesis, realize discarding redundant information, and complete data fusion.

Step 4. The IoT network terminal fuses and uploads the decision-level data to the cloud server.

Step 5. The application mobile terminal communicates with the cloud server to obtain driving data.

As shown in Figure 5, mobile application users are divided into enterprise users and individual users. Individual users can bind terminals and fleets to view driving data reports. Enterprise users can create a fleet, invite to join the fleet, and view the driving data of all members of the fleet.

Step 6. Voice interaction.

As shown in Figure 6, the voice interaction process includes: initialization, loading audio, playing audio, and recognizing the driver's answer. If an abnormality is detected, the microphone will be turned on, and the driver and surrounding passengers will be reminded by voice. At the same time, the driver can use the voice control terminal to switch to play music, switch to rest mode, stop detection, and send text messages to inform others of their location and driving data.

What is not mentioned in the present invention is applicable to the prior art.

It should be emphasized that the embodiments described in the present invention are illustrative rather than restrictive, so the present invention includes and is not limited to the embodiments described in the specific implementation, and those skilled in the art according to the technology of the present invention Other implementations derived from the scheme also belong to the protection scope of the present invention.

Claims (10)

1. a kind of monitoring of driver's abnormal behaviour and safe-guard system, it is characterised in that: including driver driving image collecting device, Driver's biological signal collecting device, intelligent things network termination, cloud server and apply mobile terminal；The driver driving image is adopted Acquisition means are mounted on immediately ahead of driver's cabin and are connected by wired mode with intelligent things network termination；Driver's bio signal Acquisition device includes bio-signal sensor, which includes multiple heart rate sensors and blood oxygen concentration sensor And it is mounted in direction indicators cover；The intelligent things network termination is set in driver's cabin, which receives driver Data that driving image acquisition device, driver's biological signal collecting device transmit simultaneously carry out data fusion, the result of data fusion Send cloud server to；The application mobile terminal is interacted with cloud server obtains driver's data.

2. a kind of driver's abnormal behaviour monitoring according to claim 1 and safe-guard system, it is characterised in that: the department Machine biological signal collecting device further includes the E.E.G detection module for being worn on driver head, which includes head electricity Pole, ear-clip electrodes, bluetooth communication and battery compartment, head motor, ear-clip electrodes testing result by bluetooth communication to The transmission of intelligent things network termination.

3. a kind of driver's abnormal behaviour monitoring according to claim 1 or 2 and safe-guard system, it is characterised in that: institute It states multiple heart rate sensors and blood oxygen concentration sensor is mounted on two sides in direction indicators cover, it is narrow that a circle is equipped on direction indicators cover The light that narrow oolemma issues sensor is through the circle oolemma.

4. a kind of driver's abnormal behaviour monitoring according to claim 1 or 2 and safe-guard system, it is characterised in that: institute Stating intelligent things network termination includes microprocessor and coupled 4G communication module, acceleration transducer, alcohol sensor And voice module.

5. a kind of implementation method of driver's abnormal behaviour as described in any one of Claims 1-4 monitoring and safe-guard system, Be characterized in that the following steps are included:

Step 1, intelligent things network termination receive the real-time driver driving image of driver driving image collecting device transmission；

Step 2, intelligent things network termination receive driver's bio signal of driver's biological signal collecting device acquisition；

Step 3, internet of things terminal carry out decision making level data fusion to driver driving image and driver's bio signal；

Decision making level data fusion is uploaded to cloud server by step 4, internet of things terminal；

Step 5 is communicated using mobile terminal with cloud server, and driving data is obtained.

6. the implementation method of a kind of driver's abnormal behaviour monitoring and safe-guard system according to claim 5, feature It is: further includes the steps that under abnormal conditions after the step 5 using mobile terminal and driver's interactive voice.

7. the implementation method of a kind of driver's abnormal behaviour monitoring and safe-guard system according to claim 5 or 6, special Sign is: the concrete methods of realizing of the step 1 the following steps are included:

(1) man face image acquiring and identification step；Driver driving image collecting device Automatic-searching, the facial image for detecting user；

(2) facial image denoises: to facial image burn into Gauss denoising, brightness and setting contrast；

(3) eye recognition: calling the calibration model of 68 characteristic point of face, mark point around eyes, calculate the value of each frame EAR, For judging frequency of wink and closed-eye time, tired driver, absent minded state are identified；

Driving behavior identify: call in advance training complete model, identification driver using mobile phone, without visually front, do not have It fastens the safety belt, 4 kinds of behaviors of smoking.

8. the implementation method of a kind of driver's abnormal behaviour monitoring and safe-guard system according to claim 5 or 6, special Sign is: the concrete methods of realizing of the step 2 the following steps are included:

(1) when driver's hand touches steering wheel, the light beam of heart rate sensor and blood oxygen concentration sensor is beaten on hand, uses light Volumetric method carries out heart rate, blood oxygen concentration measurement, after filtering processing, using feature-based fusion, using based on deep learning Heart rate and blood oxygen concentration measurement method, pass through the model and a large amount of training datas of the more hidden layers of building；

E.E.G detection module acquisition temporo at, the brain wave signal at ear-lobe；

(3) the data processing module in E.E.G detection module obtains collected electric potential signal in (2), carries out frequency domain to EEG signals Analysis, is filtered, extracts α E.E.G (8~13Hz) and β E.E.G (13~20Hz) and θ wave (4~7Hz)；

(4) the characteristic parameter of extraction is input to progress fatigue in tired judgment models and sentenced by the processing unit in E.E.G detection module It is disconnected.

9. the implementation method of a kind of driver's abnormal behaviour monitoring and safe-guard system according to claim 5 or 6, special Sign is: the concrete methods of realizing of the step 4 are as follows: set n different types of sensors to m attribute of the same target into Row identification,

Firstly, being directed to m hypothesis A to n sensor_jThe data that (j=1,2 ..., m) is observed are classified, and one group of pass is obtained In the object declaration B of different attribute_i(i=1,2 ..., n)；

Secondly, calculating each hypothesis A_jThe conditional probability of each explanation, calculates n in the case that (j=1,2 ..., m) is set up Joint likelihood function of a object declaration under m condition hypothesis:

Finally various hypothesis A in the case of n object declaration are calculated using Bayesian formula_jPosterior density:Obtain maximum a posteriori density max (P (A_j|B_i) as judgment basis, it determines to receive still Refuse A_jRedundancy is abandoned it is assumed that realizing, to complete decision making level data fusion.

10. the implementation method of a kind of driver's abnormal behaviour monitoring and safe-guard system according to claim 5 or 6, It is characterized in that: the concrete methods of realizing of the step 5 are as follows: user, which uses, to be driven using mobile terminal by interacting to obtain with server Driver's data, the user include enterprise customer and personal user；Personal user binds terminal and fleet, checks driving data report It accuses；Enterprise customer creates fleet, invites addition fleet, checks fleet's whole member's driving data.