CN117292346A - Vehicle running risk early warning method for driver and vehicle state integrated sensing - Google Patents

Abstract

The invention discloses a vehicle driving risk early warning method for integrated perception of driver and vehicle status, which includes: using a driving status data set to train a driving behavior recognition model to obtain a trained driving behavior recognition model; using lane offset data The vehicle trajectory recognition model is trained collectively to obtain the trained vehicle trajectory recognition model; the collected driver images are input into the trained driving behavior recognition model, and the driver's driving behavior recognition results are output; the collected vehicle driving status images are Input to the trained vehicle trajectory recognition model and output the lane deviation recognition results; determine whether the driving behavior recognition results and/or lane deviation recognition results meet the warning requirements. If so, remind the driver to correct the driving behavior and stay in the correct lane. driving; if not, no processing will be performed. The invention can perform real-time synchronous perception of the driver's driving status and the vehicle's driving trajectory, thereby enhancing the reliability of the driving warning.

Description

Vehicle driving risk warning method for integrated perception of driver and vehicle status

Technical field

The invention relates to the field of vehicle driving warning, and specifically relates to a vehicle driving risk warning method for integrated perception of driver and vehicle status.

Background technique

In recent years, collision accidents are still the main form of road transportation safety accidents, with the number of accidents and deaths accounting for 70.5% and 68.4% of the total accidents respectively, exposing the deficiencies in the prevention and control of road transportation vehicle collision accidents. Before a vehicle collision accident occurs, drivers are fatigued and distracted, and vehicle lane departures and vehicle distances that are too close account for 40-50% of accidents. Vehicle driving risk warning is of great significance in improving road safety and traffic management.

Intelligent assistance is more and more widely used in the field of vehicle driving warning, but currently, vehicle driving risk warning technology has false positives or omissions of potential dangers. In addition, existing vehicle driving risk warning technology also cannot effectively identify Complex traffic conditions and risk factors have greatly reduced the reliability of risk warnings. Therefore, a vehicle driving risk warning method that is oriented to the integrated perception of driver and vehicle status is needed to solve the above problems.

Contents of the invention

In view of this, the purpose of the present invention is to overcome the shortcomings in the existing technology and provide a vehicle driving risk warning method for integrated perception of driver and vehicle status, which can perform real-time synchronous perception of the driver's driving status and vehicle driving trajectory, and enhance the The reliability of driving warning improves the safe driving level of vehicles.

The vehicle driving risk warning method for integrated perception of driver and vehicle status of the present invention includes the following steps:

Create a driver's driving status data set; use the driving status data set to train the driving behavior recognition model to obtain a trained driving behavior recognition model;

Create a lane offset data set; use the lane offset data set to train the vehicle trajectory recognition model to obtain a trained vehicle trajectory recognition model;

Input the collected driver images into the trained driving behavior recognition model and output the driver's driving behavior recognition results;

Input the collected vehicle driving status images into the trained vehicle trajectory recognition model and output the lane offset recognition results;

Determine whether the driving behavior recognition results and/or lane deviation recognition results meet the warning requirements. If so, the driver is reminded to correct the driving behavior and drive in the correct lane; if not, no processing is performed.

Further, the driving behavior recognition model includes a policy network and a two-dimensional convolutional neural network; the policy network includes a feature extractor and a long and short-term memory module; the backbone network of the two-dimensional convolutional neural network is embedded with hybrid attention. Mechanism module; the hybrid attention mechanism module includes a spatiotemporal excitation sub-module, a channel excitation sub-module and a motion excitation sub-module;

The spatiotemporal excitation submodule uses single-channel three-dimensional convolution to represent spatiotemporal features;

The channel excitation submodule adaptively calibrates the characteristic response of the channel based on the interdependence between the channels;

The motion excitation sub-module calculates time differences at the feature level to stimulate motion sensitive channels.

Furthermore, the policy network adaptively selects different frame scales to achieve driving behavior recognition efficiency, including:

At the time step t<T ₀ , adjust the frame _It to the lowest resolution and send it to the feature extractor; where T ₀ is the set time period; I _t is the driver status image frame at time t ;

The long short-term memory module uses the extracted features and previous states to update the hidden state and output it;

Given the hidden state, estimate the policy distribution, sample the action a _t at time t, and perform the Gumbel Softmax operation;

When the action a _t <L, adjust the frame size to the spatial resolution 3×H _at ×W _at and forward it to the corresponding backbone network to obtain frame-level prediction; where L is the number of state image resolution types; H _at is the height of the image at time t of action a _t ; W _at is the width of the image at time t of action a _t ;

When the action a _t ≥ L, the backbone network will skip the current frame for prediction, and the policy network will skip the subsequent F _at-L-1 frame; where F _at-L-1 is the video when a _t ≥ L frame.

Furthermore, the spatiotemporal excitation submodule uses single-channel three-dimensional convolution to represent spatiotemporal features, specifically including:

^{For a} given input image ^×W ; then, reshape F into F ^* ∈R ^{N×T×1×H×W} and feed it to a three-dimensional convolutional layer K with a core size of 3×3×3 to get Finally, add/> Reshape it as F _o ∈R ^{N×T×1×H×W} and feed it to Sigmoid activation to obtain the space-time mask M∈R ^{N×T×1×H×W} , and finally output Y: Y=X+ X⊙M;

Among them, ⊙ represents the element-by-element multiplication of the spatio-temporal mask M and all channel inputs Height; W represents the width of the image.

Further, the channel excitation submodule adaptively calibrates the characteristic response of the channel based on the interdependence between channels, specifically including:

For a given input image X∈R ^{N×T×C×H×W} , first obtain the global spatial information of the input elements F∈R ^{N×T×C×1×1} by averaging the input; The number is compressed according to proportion r, and F _r =K ₁ *F is obtained; where K ₁ is a 1×1 two-dimensional convolution layer,

Then, reshape F _r to Until temporal inference can be enabled, 1D convolutional layer K ₂ kernel size 3 is used for processing /> Get/> Among them,/>

Will Reshape to/> It is then decompressed by using a 1×1 two-dimensional convolution layer K ₃ to obtain F _o =K ₃ *F _temp , which is fed to Sigmoid activation to obtain the channel mask M; where, F _o ∈R ^{N×T× C×1×1} and M∈R ^{N×T×C×1×1} ;

The final output is Y: Y=X+X⊙M.

Further, the motion excitation sub-module calculates the time difference at the feature level to stimulate the motion-sensitive channel, specifically including:

For a given input image X∈R ^{N×T×C×H×W} , using a 1×1 two-dimensional convolution layer to compress the number of channels in proportion to r, we get Use a 1×1 two-dimensional convolution layer to decompress F _r ;

Model the motion characteristics and obtain F _m =K*F _r [:,t+1,:,:,:]-F _r [:,t,:,:,:];

Among them, K is a 3×3 two-dimensional convolution layer, Among them, F _r [:,t+1,:,:,:] represents the compressed feature map at time t+1, and F _r [:,t,:,:,:] represents the compressed feature map at time t;

Connect the motion features to each other according to the time dimension, and fill the last element with 0, then:

F _m -[F _m (1),...,F _m (t-1),0]; where F _m (t-1) is the t-1th motion representation;

F _m is then averaged to obtain the global spatial information of the input elements.

Further, the vehicle trajectory recognition model includes an improved Deeplabv3+ network;

The improved Deeplabv3+ network uses Deeplabv3+ as the basic framework, replaces the backbone network Xception of Deeplabv3+ with the lightweight network MobileNetv2, adds a channel attention mechanism module, and uses Dense-ASPP to replace the ASPP structure in the Deeplabv3+ network;

The channel attention mechanism module is used to place attention between channels of the feature map.

Furthermore, it also includes:

Use the preceding vehicle distance data set to train the vehicle distance recognition model to obtain a trained vehicle distance recognition model; input the collected preceding vehicle distance image into the trained vehicle distance recognition model and output the preceding vehicle distance recognition result; determine the preceding vehicle Whether the distance recognition result is less than the distance threshold, if so, the driver is reminded to correct the driving behavior; if not, no processing is performed.

Further, the vehicle distance recognition model includes an improved YOLOv5 network;

The improved YOLOv5 network uses YOLOv5 as the basic framework, uses the Ghost Module in GhostNet to replace the convolution operation of YOLOv5, introduces the attention mechanism Coordinate Attention, and embeds position information into the channel.

Furthermore, if the vehicle in front is directly in front of the vehicle, the distance d between the vehicle in front and the vehicle in front is determined according to the following formula:

Among them, h is the distance in the vertical direction between the camera installed on the own vehicle and the vehicle in front; θ is the camera pitch angle; the intersection of the camera's lens optical axis and the image plane is O(x,y), the focal length is f, and the front The imaging point of the center point of the car bottom on the image plane is D(u,v), and the angle between the straight line from the center point of the front car bottom to the camera and the optical axis of the lens is α;

If the vehicle in front is to the side of the vehicle in front of the vehicle in front, the distance D between the vehicle in front and the vehicle in front is determined according to the following formula:

Among them, γ is the yaw angle of the preceding vehicle.

The beneficial effects of the present invention are: the present invention discloses a vehicle driving risk warning method for integrated perception of driver and vehicle status, driver distraction and fatigue driving detection based on in-vehicle video, and vehicle lane deviation and front vehicle distance detection. Detection, the two form an integrated detection, which can realize real-time synchronous perception of the driver's driving status and vehicle's driving trajectory, conduct real-time risk assessment of the vehicle's driving status, and provide early warning of risks. The driver will be reminded by voice to correct his driving behavior so that he can Return to a safe driving state as soon as possible, thus improving the safe driving level of the vehicle.

Description of drawings

The present invention will be further described below in conjunction with the accompanying drawings and examples:

Figure 1 is a schematic diagram of the vehicle driving risk early warning process of the present invention;

Figure 2 is a schematic diagram of video key frame extraction according to the present invention;

Figure 3 is a flow chart of driver distraction and fatigue driving behavior identification according to the present invention;

Figure 4 is a schematic diagram of the SCM module architecture of ResNet-50 of the present invention;

Figure 5 is a schematic diagram of the working principle of the spatiotemporal excitation submodule of the present invention;

Figure 6 is a schematic diagram of the working principle of the channel excitation sub-module of the present invention;

Figure 7 is a schematic diagram of the working principle of the motion excitation sub-module of the present invention;

Figure 8 is a schematic diagram of the improved Deeplabv3+ network structure of the present invention;

Figure 9 is a schematic diagram showing the lane offset frame image of the present invention;

Figure 10 is a schematic diagram of the vehicle distance measurement process of the present invention;

Figure 11 is a schematic diagram of the improved YOLOv5 network structure of the present invention;

Figure 12 is a schematic diagram of the vehicle distance measurement principle based on pitch angle according to the present invention;

Figure 13 is a schematic diagram of the vehicle distance measurement principle based on pitch angle and yaw angle according to the present invention.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings, as shown in the figure:

The vehicle driving risk warning method for integrated perception of driver and vehicle status of the present invention includes the following steps:

Create a driver's driving status data set; use the driving status data set to train the driving behavior recognition model to obtain a trained driving behavior recognition model;

Create a lane offset data set; use the lane offset data set to train the vehicle trajectory recognition model to obtain a trained vehicle trajectory recognition model;

Input the collected driver images into the trained driving behavior recognition model and output the driver's driving behavior recognition results;

Input the collected vehicle driving status images into the trained vehicle trajectory recognition model and output the lane offset recognition results;

Determine whether the driving behavior recognition results and/or lane deviation recognition results meet the warning requirements. If so, the driver is reminded to correct the driving behavior and drive in the correct lane; if not, no processing is performed.

As shown in Figure 1, the present invention obtains the driver's real-time driving behavior sequence and the vehicle's real-time driving status sequence through the vehicle-mounted two-way camera. Then, in order to obtain the video frame size that meets the model input requirements, it performs data preprocessing operations such as cropping and zooming. Then input it into the trained driving behavior recognition model for recognition. When the driving behavior recognition model recognizes the driver's fatigue or distracted driving behavior, the vehicle trajectory recognition model recognizes the vehicle lane deviation, or the vehicle distance recognition model recognizes the driver's behavior, If the vehicle in front is too close, driver behavior and vehicle trajectory need to be comprehensively considered to assess driving safety risks.

Furthermore, the identified driving behavior and vehicle status can be continuously quantified to achieve a more accurate and timely warning effect. For example, if distracted driving behavior is identified in driver images for 2 consecutive seconds, or if distracted driving behavior is identified in driver images for 1 second consecutively and lane deviation is detected in vehicle driving status images at the same time, it is deemed that If there is a driving risk in the current vehicle, an early warning will be issued at this time, and a voice will be started to remind the driver to correct the driving behavior and return to a safe driving state as soon as possible, thereby achieving the purpose of improving driving safety.

In this embodiment, the driving behavior is a continuous action. Compared with the method of identifying only through a single image, the video frame sequence is used as input to identify the driver's driving status from the time, space and motion dimensional characteristics of the input data, achieving better recognition accuracy. Therefore, the present invention performs driver fatigue and distracted driving behavior detection based on adaptive frame resolution.

Processing every frame in a video is often unnecessary and inefficient due to the large amount of redundancy that comes from static scenes or very low frame quality (blur, low-light conditions, etc.). Therefore, while using the policy network to adaptively select the frame resolution in a unified framework, a frame skipping mechanism is also designed to skip frames (i.e., set the resolution to zero) when needed to further improve action recognition. s efficiency. At the same time, since the two-dimensional convolutional neural network (CNN) cannot obtain long-term time relationships, the use of three-dimensional CNN processing will face the problem of a relatively large amount of calculation. Therefore, after the input video is processed by the policy network, a hybrid attention mechanism module embedded in the two-dimensional CNN backbone network is used.

The driving behavior recognition model includes a policy network and a two-dimensional convolutional neural network; the policy network includes a feature extractor and a long and short-term memory module; a hybrid attention mechanism module is embedded in the backbone network of the two-dimensional convolutional neural network ; The hybrid attention mechanism module includes a spatiotemporal excitation submodule (STE), a channel excitation submodule (CE) and a motion excitation submodule (ME); the spatiotemporal excitation submodule uses single-channel three-dimensional convolution to represent spatiotemporal features; The channel excitation sub-module adaptively calibrates the characteristic response of the channel based on the interdependence between channels; the motion excitation sub-module calculates the time difference at the feature level to stimulate the motion-sensitive channel.

This invention uses public data sets such as YAWDD to construct driving state data sets of normal driving, distracted driving, and fatigue driving, and divides them into training sets, verification sets, and test sets in a ratio of 6:2:2, thereby identifying the driving behavior model. Conduct training and verification, and package and integrate it into the system.

In this embodiment, the policy network adaptively selects different frame scales to achieve driving behavior recognition efficiency. Express a range of resolutions in descending order as: Where S ₀ =(H ₀ ,W ₀ ) represents the original (and highest) frame resolution, and S _L-1 =(HL _-1 ,W _L-1) ) is the lowest resolution. Denote the frame at time t in the ^lth scale as/> Frame skipping is a special case of "selecting resolution S ^∞ ". Define the jump sequence (ascending order) as/> The i ^th operation represents skipping the current frame and subsequent ( _Fi -1) frames in prediction. The choice of resolution and jumps forms the action space Ω.

The policy network includes a lightweight feature extractor φ (·; θ _φ ) and a long short-term memory (LSTM) module.

At time step t<T ₀ , adjust frame I _t to the lowest resolution and send it to the feature extractor:

Among them, T ₀ is the set time period; I _t is the driver status image frame at time t; f _t is the feature vector, and θ _φ represents the learnable parameter.

LSTM updates the hidden state h _t using the extracted features and previous states and outputs ot:

[h _t ,o _t ]=LSTM (f _t ,h _t-1 ,o _t-1 ,θ _LSTM ) (2)

Given the hidden states, the policy network estimates the policy distribution and evaluates the actions

a _t ∈Ω＝{0,1,...,L+M-1} is sampled and operated by Gumbel Softmax:

a _t ~GUMBEL(h _t ,θ _G ) (3)

If a _t <L, adjust the frame size to the spatial resolution 3×H _at ×W _at and forward it to the corresponding backbone network To get frame-level prediction:

in, is the resize frame,/> is the predicted value. L is the number of state image resolution types; H _at is the height of the image at time t of action a _t ; W _at is the width of the image at time t of action a _t .

When action a _t ≥ L, the backbone network will skip the current frame for prediction, and the policy network will skip subsequent frame;/> is the video frame when a _t ≥ L.

In addition, in order to save computational effort, a shared policy network can be used to generate the lowest resolution policy and prediction, i.e. (φ′ is an eigenvector).

In this embodiment, in order to obtain more accurate prediction results, an SCM module is added to the backbone network. The SCM module consists of three sub-modules, namely the temporal and spatial excitation sub-module (STE), the channel excitation sub-module (CE) and the motion excitation sub-module (ME).

Among them, STE excites spatiotemporal information by using 3D convolution. Unlike traditional three-dimensional convolution, this module averages all channels to obtain global spatiotemporal features, which can significantly reduce the calculation of three-dimensional convolution. The output of STE contains global spatiotemporal information. . CE is used to activate channel correlation with time information, and the output contains channel correlation based on time perspective. ME has shown the effectiveness of reasoning about action motion in videos. ME models the differences between adjacent frames at the feature level and is then combined with the modules introduced above for reasoning about the rich information held in videos.

Among them, all tensors outside the SCM action module we use are 4D, that is (N (batch size) × T (number of segments), C (channel), H (height), W (width)). We reshape the input 4D tensor into a 5D tensor (N, T, C, H, W) before feeding it into the SCM module to be able to operate on specific dimensions inside the SCM module. The 5D output tensor is then reshaped into 4D before being fed to the next 2D convolution block. By doing so, the output of the SCM module can sense information from a spatiotemporal perspective, channel correlation, and motion.

Figure 4 shows the ResNet-50-SCM module architecture, where the SCM module is inserted at the beginning of each residual block. ResNet-50 gives the size of the output feature map of each layer (CLS represents the number of classes, T represents the number of segments). First, the input video is divided into T segments evenly, and then one frame is randomly sampled from the video processed by the policy network.

Among them, STE uses three-dimensional convolution to effectively simulate spatiotemporal information. In this stage, STE generates a spatiotemporal mask M∈R ^{N×T×1×H×W} , which is used to multiply the input X∈R ^{N×T×C×H×W} of all channels element by element.

As ^shown in ^Figure 5, given an image input ^×1×H×W . Then, F is reshaped into F ^* ∈R ^{N×T×1×H×W} and fed to a three-dimensional convolutional layer K with core size of 3×3×3. The formula is:

Finally, add Reshape it into F _o ^{∈ R N} It can be expressed as:

M＝δ(F _o ) (6)

The final output is:

Y＝X+X⊙M (7)

Among them, ⊙ represents the element-by-element multiplication of the space-time mask M with all channel inputs X.

T represents the number of segments into which the video corresponding to the image is divided; N represents the batch number of segments T; C represents the number of image channels; H represents the height of the image; W represents the width of the image.

The design of the CE is similar to the STE block, as shown in Figure 6.

Given an input X∈R ^{N×T×C×H×W} , the global spatial information of the input elements is first obtained by averaging the input, which can be expressed as:

where F∈R ^{N×T×C×1×1} . Compress the number of channels of F in proportion to r (r-channel compression rate), which can be interpreted as:

F _r ＝K ₁ *F (9)

Where K ₁ is a 1×1 two-dimensional convolution layer,

Then reshape F _r to Until temporal inference can be enabled. One-dimensional convolutional layer K ₂ kernel size 3 is used for processing/> as:

in Then add/> Reshape to/> It is then decompressed by using a 1×1 2D convolutional layer _K3 and fed to the Sigmoid activation. These are the last two steps to obtain the channel mask M and can be formulated separately:

F _o ＝K ₃ *F _temp (11)

M＝δ(F _o ) (12)

where F _o ∈R ^{N×T×C×1×1} and M∈R ^{N×T×C×1×1} . Finally, the output of CE is formulated to be the same as Equation (7) using the newly generated mask.

ME is used in parallel with the two modules STE and CE mentioned above. As shown in Figure 7, the motion information is modeled by adjacent frames.

Use the same compression and decompression strategy as the CE sub-module and use two 1×1 two-dimensional convolution layers. You can refer to Equation (9) and Equation (11) respectively. Features processed by a given compression operation Modeling the motion characteristics according to the proposed similar operations can be expressed as:

F _m ＝K*F _r [:,t+1,:,:,:]-F _r [:,t,:,:,:] (13)

where K is a 3×3 two-dimensional convolution layer, F _r [:,t+1,:,:,:] represents the compressed feature map at time t+1, F _r [:,t,:,:,:] represents the feature map at time t;

Connect the motion features to each other according to the time dimension and fill the last element with 0, then it is F _m -[F _m (1),...,F _m (t-1),0], Among them, F _m (t-1) is the t-1th motion representation.

It is then processed through the same spatial average merging as in formula (8), that is, averaging F _m to obtain the global spatial information of the input elements.

In this embodiment, the vehicle trajectory recognition model includes an improved Deeplabv3+ network;

The improved Deeplabv3+ network uses Deeplabv3+ as the basic framework, replaces the backbone network Xception of Deeplabv3+ with the lightweight network MobileNetv2, adds a channel attention mechanism module, and uses Dense-ASPP to replace the ASPP structure in the Deeplabv3+ network;

The channel attention mechanism module is used to place attention between channels of the feature map.

First, the video collected by the driving recorder is divided into frames and screenshots are processed, and the captured pictures are combined with the public data sets Tusimple and CUlane to form a lane offset data set used for lane line detection. Secondly, these data sets are preprocessed to enhance the illumination of darker images, etc., and then manually annotated to inform the model of the deep features of the lane lines. Finally, the labeled data set is input into the vehicle trajectory recognition model to start training. The training degree of the model is observed through the loss function. If the training effect is not good, the obtained parameters are back-propagated to the network to continue training until the training is completed. An ideal model for lane line detection.

This invention uses Deeplabv3+ as the basic framework to establish a lightweight network model that is more effective in extracting features. Aiming at the problem of incomplete feature extraction in the original network, an intensive structure is added to improve the situation. The present invention also adds an attention mechanism to increase attention to important features, which is beneficial to model training and real-time detection accuracy.

As shown in Figure 8, the backbone network Xception of Deeplabv3+ is replaced by the lightweight network MobileNetv2, which has small parameters and fast speed, and is very suitable for real-time detection scenarios. In order to improve the performance of the improved model, the present invention adds a SE (Squeeze-and-Excitation) module (channel attention mechanism module). The SE module mainly focuses on the channels of the feature map and automatically learns the importance of different channels.

In order to solve the problem that the acquisition of local features becomes more difficult as the expansion rate increases, Dense Atrous Spatial Pyramid Pooling (Dense-ASPP) is introduced, and Dense-ASPP is used to replace the ASPP structure of the original network.

Based on the above lane line detection, as shown in Figure 9, assuming that the pixel coordinates of the car's predecessor in the image are (180,0), D _L and _DR represent the distance between the vehicle center and the left and right lane lines, (L _x ,L _y ), (R _x ,R _y ) represent the pixel coordinates of the left and right lane lines in the image. Then the distance from the vehicle center to the left lane line D _L =180-L _x , and similarly D _R =R _x -180.

Normally it will take 0.5 seconds to collect 20 frames of images. At this time, you can judge whether the vehicle has drifted through the continuous changes of _DL and _DR within 20 frames. At this time, there is an array L that can store 20 changing values of D _L , and an array _R that can store 20 changing values of DR.

Furthermore, the left and right lateral velocities V _L and _VR at this time are shown in equations (14) and (15):

In this embodiment, it also includes:

Use the preceding vehicle distance data set to train the vehicle distance recognition model to obtain a trained vehicle distance recognition model; input the collected preceding vehicle distance image into the trained vehicle distance recognition model and output the preceding vehicle distance recognition result; determine the preceding vehicle Whether the distance recognition result is less than the distance threshold, if so, the driver is reminded to correct the driving behavior; if not, no processing is performed.

The present invention selects monocular vision for vehicle distance recognition analysis, and mainly completes ranging based on the monocular vision ranging model combined with the detected target frame. Monocular visual ranging only requires one camera and some coordinate system conversion work to complete the corresponding ranging. It has the advantages of small calculation amount and low consumption.

The present invention uses the driving recorder on the driving vehicle as the main device for collecting data, and the driving vehicle collects the real-time distance data set of the vehicle ahead in various scenes such as busy urban sections, high-speed sections, and highways.

As shown in Figure 11, the vehicle distance recognition model uses YOLOv5 as the basic framework, makes corresponding modifications to the backbone network, and integrates the attention mechanism to meet the requirements of lightweight, high-precision, multi-scale vehicle detection, complex Vehicle detection under environmental conditions and real-time requirements.

In the backbone network of YOLOv5, its CSP mechanism uses more convolution (Conv) operations, and the calculation amount of convolution will increase as the number of layers increases. In order to solve this problem, the present invention uses Ghost in GhostNet Module replaces the convolution operation of YOLOv5, improving the speed of the model and reducing the amount of related parameters without affecting accuracy. The attention mechanism Coordinate Attention (CA) is introduced, embedding position information into the channel, and solving the shortcomings of some mechanisms focusing on channel attention that ignore position information.

After detecting the vehicle in front of the vehicle, the pixel value of the corresponding target is extracted, and the ranging modeling is performed based on the camera pitch angle and the yaw angle of the preceding vehicle, as shown in Figure 12.

The dotted straight line below the upper left corner in the figure is the optical axis of the lens. Its intersection with the image plane is O(x,y) (image plane coordinate system). The focal length is f. The dotted straight line above the upper left corner is the detected center of the vehicle bottom. The straight-line distance from the point to the camera, its imaging point on the image plane is D(u,v) (pixel coordinate system), and the angle with the optical axis is α. Among them, if the camera is installed on the vehicle, the horizontal distance from the camera to the vehicle, that is, the distance between the vehicle and the vehicle in front is:

in, h is the vertical distance between the camera and the vehicle in front, and θ is the camera pitch angle (the angle between the optical axis of the camera lens and the horizontal plane).

Since in the actual scene, the vehicle in front is not driving directly in front of the vehicle, but may also be on the left and right sides, then the vehicle in front is in front of and to the side of the vehicle. The specific implementation is shown in Figure 13.

β is the horizontal angle between the external camera and the optical axis, and γ is the yaw angle of the preceding vehicle. B′(B _x ,B _y ) is the position of the landing point B of the yaw trajectory of the bottom center of the front vehicle in the pixel coordinate system, and the pixel center point is O′(u,v). The ranging based on the camera pitch angle and the yaw angle of the front and side vehicles is as follows:

In equation (17), D is the distance between the vehicle and the vehicle in front, θ and γ represent the camera pitch angle and the yaw angle of the vehicle in front, respectively.

Among them, since the pitch angle and yaw angle are difficult to obtain, a real-time acquisition method of changing angles can be designed: the lane lines are basically parallel. If the lane lines are photographed, the two lane lines will eventually intersect at one point. It's called the vanishing point. Use vanishing points to calculate the yaw and pitch angles of the camera.

First, the Gabor filter is used to calculate the texture direction of each pixel in the captured image, and then the confidence level of these pixels is used to determine whether voting is required. Finally, a fast local voting method is used to determine the location of the vanishing point.

Among them, the functional expression of Gabor filter is as follows:

In the formula, ω, Indicates scale and direction,/>

x, y represent the pixel coordinate position.

Since Gabor filtering obtains 36 directional textures for each pixel, it cannot guarantee that the texture in each direction is required. At this time, confidence needs to be introduced. When this parameter is greater than the set threshold, it can be considered that the texture in that direction is Texture is the voting point. The confidence of the pixel value d(x,y) at a certain point in the image can be defined by the following formula:

Among them, r _i (d) represents the response value in the i-th direction at the pixel point, r ₅ (d) to r ₁₅ (d) are the local maximum response values appearing between them, and the remaining parameters rarely appear. Set the threshold to:

t＝0.4(maxC(d)-minC(d)) (20)

When the confidence is greater than t, the pixel is the voting point. Select the final road vanishing point based on voting points and confidence.

The principle of the voting algorithm is to select the candidate vanishing point H, with H as the center of the circle, the radius as one-third of the image size, and half of the circle as the candidate area. The specific voting formula is as follows:

In the formula, α is the angle between the texture direction of a certain voting point P and the line segment PH, d(P,H) represents the distance from the point P to the center of the circle. This formula takes each pixel in the image as a candidate point. Finally, The point with the highest vote score is the vanishing point.

After obtaining the vanishing point, the pitch angle and yaw angle are:

Among them, R is the camera image rotation matrix; R _xz represents the matrix of rotation around the x-axis and z-axis; R _yz represents the matrix of rotation around the y-axis and z-axis.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified. Modifications or equivalent substitutions without departing from the spirit and scope of the technical solution of the present invention shall be included in the scope of the claims of the present invention.

Claims (10)

1. A vehicle running risk early warning method facing to the integrated perception of a driver and a vehicle state is characterized in that: comprising the following steps:

creating a driving state data set of a driver; training the driving behavior recognition model by using the driving state data set to obtain a trained driving behavior recognition model;

creating a lane departure data set; training the vehicle track recognition model by using the lane offset data set to obtain a trained vehicle track recognition model;

inputting the acquired driver images into a trained driving behavior recognition model, and outputting a driving behavior recognition result of the driver;

inputting the acquired vehicle running state image into a trained vehicle track recognition model, and outputting a lane deviation recognition result;

judging whether the driving behavior recognition result and/or the lane deviation recognition result meet the early warning requirement, if so, reminding a driver to correct the driving behavior and drive on a correct lane; if not, the processing is not performed.

2. The vehicle running risk early warning method for driver and vehicle state integrated perception according to claim 1, characterized in that: the driving behavior recognition model comprises a strategy network and a two-dimensional convolutional neural network; the strategy network comprises a feature extractor and a long-term and short-term memory module; a mixed attention mechanism module is embedded in a backbone network of the two-dimensional convolutional neural network; the mixed attention mechanism module comprises a space-time excitation sub-module, a channel excitation sub-module and a motion excitation sub-module;

the space-time excitation submodule uses single-channel three-dimensional convolution to represent space-time characteristics;

the channel excitation submodule adaptively calibrates characteristic responses of the channels based on interdependencies between the channels;

the motion-excitation sub-module calculates a time difference at a feature level to stimulate a motion-sensitive channel.

3. The vehicle running risk early warning method for driver and vehicle state integrated perception according to claim 2, characterized in that: the strategy network adaptively selects different frame scales to realize driving behavior recognition efficiency, and comprises the following steps:

at time step T < T ₀ Frame I _t Adjust to the lowest resolution and send it to the feature extractor; wherein T is ₀ For a set period of time; i _t A driver status image frame at the time t;

the long-period and short-period memory module updates and outputs the hidden state by using the extracted characteristics and the previous state;

given a hidden state, the policy distribution is estimated for action a at time t _t Sampling and performing Gumbel Softmax operation;

action a _t < L, frame size is adjusted to spatial resolution 3 XH _at ×W _at And forwards it to the corresponding backbone network to obtain frame-level predictions; wherein L is the resolution category number of the state image; h _at For action a _t The high of the image at time t; w (W) _at For action a _t The width of the image at time t;

action a _t If L is not less, the backbone network will skip the current frame for prediction and the policy network will skip the followingA frame; wherein (1)>Is a as _t And (5) video frames when the video frame is more than or equal to L.

4. The vehicle running risk early warning method for driver and vehicle state integrated perception according to claim 2, characterized in that: the space-time excitation submodule uses single-channel three-dimensional convolution to represent space-time characteristics and specifically comprises the following steps:

for a given input image X ε R ^{N×T×C×H×W} The input tensors for each channel are averaged to obtain a global spatio-temporal tensor F.epsilon.R relative to the channel axis ^{N×T×1×H×W} The method comprises the steps of carrying out a first treatment on the surface of the F is then remolded to F ^* ∈R ^{N×T×1×H×W} And fed to a three-dimensional convolutional layer K of core size 3 x 3, obtainingFinally, will->Remodelling to F _o ∈R ^{N×T×1×H×W} And feeds it to Sigmoid activation to get a spatiotemporal mask M ε R ^{N×T×1×H×W} And finally outputting Y: y=x+x+m;

wherein, as follows, the space-time mask M is multiplied by all channel inputs X element by element, and T is the number of divided segments of the video corresponding to the image; n represents the batch number of the fragment number T; c represents the number of image channels; h represents the high of the image; w represents the width of the image.

5. The vehicle running risk early warning method for driver and vehicle state integrated perception according to claim 2, characterized in that: the channel excitation submodule adaptively calibrates characteristic response of the channels based on interdependence among the channels, and specifically comprises the following steps:

for a given input image X ε R ^{N×T×C×H×W} First global spatial information F epsilon R of an input element is obtained by averaging the inputs ^{N×T×C×1×1} The method comprises the steps of carrying out a first treatment on the surface of the Compressing the channel number of F in proportion r to obtain F _r ＝K ₁ * F, performing the process; wherein K is ₁ Is a 1 x 1 two-dimensional convolution layer,

then remodel F _r To the point ofUntil time reasoning can be enabled, one-dimensional convolution layer K ₂ Kernel size 3 for handling +.>Obtain->Wherein (1)>

Will beReshape to +.>Then by using a 1 x 1 two-dimensional convolution layer K ₃ Decompressing it to obtain F _o ＝K ₃ *F _temp Feeding to Sigmoid activation to obtain a channel mask M; wherein F is _o ∈R ^{N×T×C×1×1} And M.epsilon.R ^{N ×T×C×1×1} ；

Final output Y: y=x+x+m.

6. The vehicle running risk early warning method for driver and vehicle state integrated perception according to claim 2, characterized in that: the motion-excitation sub-module calculates a time difference at a feature level to stimulate a motion-sensitive channel, comprising:

for a given input image X ε R ^{N×T×C×H×W} The channel number is compressed in proportion r by using a 1X 1 two-dimensional convolution layer to obtainUsing a 1 x 1 two-dimensional convolution layer for F _r Decompressing;

modeling the motion characteristics to obtain F _m ＝K*F _r [:,t+1,:,:,:]-F _r [:,t,:,:,:]；

Wherein K is a 3 x 3 two-dimensional convolution layer,wherein F is _r [:,t+1,:,:,:]Representing the compressed characteristic diagram at time t+1, F _r [:,t,:,:,:]The characteristic diagram after compression at the time t is shown;

connecting motion characteristics with each other according to a time dimension, and filling 0 into a last element, wherein the steps are as follows:

F _m -[F _m (1),...,F _m (t-1),0]the method comprises the steps of carrying out a first treatment on the surface of the Wherein F is _m (t-1) is the t-1 th motion representation;

then to F _m Averaging is performed to obtain global spatial information of the input elements.

7. The vehicle running risk early warning method for driver and vehicle state integrated perception according to claim 1, characterized in that: the vehicle track recognition model comprises a modified deep labv3+ network;

the improved deep bv3+ network takes deep bv3+ as a basic framework, replaces a backbone network Xaccept of the deep bv3+ with a lightweight network MobileNet v2, increases a channel attention mechanism module, and replaces an ASPP structure in the deep bv3+ network by using a Dense-ASPP;

the channel attention mechanism module is used for focusing attention among channels of the feature map.

8. The vehicle running risk early warning method for driver and vehicle state integrated perception according to claim 1, characterized in that: further comprises:

training the vehicle distance recognition model by using the front vehicle distance data set to obtain a trained vehicle distance recognition model; inputting the acquired front vehicle distance image into a trained vehicle distance recognition model, and outputting a front vehicle distance recognition result; judging whether the front vehicle distance recognition result is smaller than a distance threshold value, if so, reminding a driver to correct driving behaviors; if not, the processing is not performed.

9. The vehicle running risk early warning method for driver and vehicle state integrated perception of claim 8, wherein: the vehicle distance identification model comprises a modified YOLOv5 network;

the improved YOLOv5 network uses YOLOv5 as a basic framework, uses a Ghost Module in GhostNet to replace the convolution operation of YOLOv5, introduces an attention mechanism Coordinate Attention, and embeds position information into the channel.

10. The vehicle running risk early warning method for driver and vehicle state integrated perception of claim 8, wherein: if the front vehicle is right in front of the vehicle, determining a distance d between the vehicle and the front vehicle according to the following formula:

h is the distance between the camera arranged on the vehicle and the front vehicle in the vertical direction; θ is the camera pitch angle; the intersection point of the lens optical axis of the camera and the image plane is O (x, y), the focal length is f, the imaging point of the center point of the bottom of the front vehicle at the image plane is D (u, v), and the included angle between the straight line from the center point of the bottom of the front vehicle to the camera and the lens optical axis is alpha;

if the front vehicle is in front of the side of the vehicle, determining a distance D between the vehicle and the front vehicle according to the following formula:

wherein, gamma is the yaw angle of the front vehicle.