CN115257819A - Decision-making method for safe driving of large-scale commercial vehicle in urban low-speed environment - Google Patents

Abstract

The invention discloses a decision-making method for safe driving of a large-scale commercial vehicle in an urban low-speed environment. And secondly, constructing a multi-head attention-based safe driving decision model of the commercial vehicle. The model contains two sub-networks, deep double-Q network and generative confrontation mock-learning. The deep double-Q network learns the safe driving strategies under the edge scenes such as dangerous scenes and conflict scenes in an unsupervised learning mode; an antagonistic emulation learning sub-network is generated that emulates safe driving behavior of a human driver under different driving conditions and driving regimes. And finally, training a safe driving decision model to obtain driving strategies under different driving conditions and driving conditions. The method provided by the invention can simulate the safe driving behavior of a human driver, and considers the influence of factors such as a visual blind area, a sudden obstacle and the like on the driving safety.

Description

Decision-making method for safe driving of large commercial vehicles in urban low-speed environment

technical field

The invention relates to a decision-making method for driving a commercial vehicle, in particular to a decision-making method for safe driving of a large commercial vehicle in an urban low-speed environment, and belongs to the technical field of automobile safety.

Background technique

In the urban traffic environment, road traffic accidents caused by drivers' blind spots account for the highest proportion, and the main causes of these accidents are mostly large operating vehicles such as heavy goods vehicles, large passenger cars, and automobile trains. Different from passenger vehicles, large commercial vehicles have the characteristics of large size, long body, large wheelbase, and high driving position. There are many static and dynamic visual blind spots around the body, such as the front of the front, near the right front wheel and right rear view. Wait under the mirror. When commercial vehicles turn, especially turn right, they are very likely to collide or even crush pedestrians and non-motor vehicles in the blind area of vision, which is the main area where vicious safety accidents occur. In addition, compared with the relatively closed highway scene, in the urban traffic environment with mixed traffic, the types and numbers of traffic participants are relatively large, and business vehicles encounter obstacles frequently, which has a higher dangerous. Therefore, how to improve the driving safety of operating vehicles in an open and interfering urban traffic environment is a key problem that needs to be solved urgently, and it is also the focus of ensuring urban road traffic safety.

At present, the active development of autonomous driving technology has become an important means to ensure the safety of vehicle operation widely recognized at home and abroad. As a key part of realizing high-quality autonomous driving, driving decision-making determines the rationality and safety of autonomous driving of commercial vehicles. If the driver can be warned of danger 1.5 seconds before a traffic accident, and a reliable and effective safe driving strategy can be provided, the frequency of traffic accidents caused by factors such as visual blind spots and sudden obstacles can be greatly reduced. Therefore, studying the safe driving decision-making method of large commercial vehicles plays an important role in ensuring the driving safety of commercial vehicles.

There have been many patents and literatures on collision avoidance driving decision-making, but mainly for passenger vehicles. Compared with passenger vehicles, commercial vehicles have larger visual blind spots, and have longer braking distance and braking time. The collision avoidance decision-making method for passenger vehicles cannot be directly applied to commercial vehicles. On the other hand, some patents have studied the safe driving decision-making of commercial vehicles, such as a highly human-like autonomous driving commercial vehicle safe driving decision-making method (application number: 202210158758.2), a deep learning-based large commercial vehicle lane Transformation decision-making method (public number: CN113954837A), etc., but these decision-making methods are all oriented to the highway scene.

Different from the expressway scene with fewer types of traffic participants, the urban traffic environment has the characteristics of openness, multi-traffic target interference, and mixed traffic. In particular, the existence of factors such as vehicle visual blind spots and unexpected obstacles pose higher challenges to the safe driving of commercial vehicles in urban traffic environments. Therefore, the decision-making method for safe driving of commercial vehicles for highway scenarios cannot be directly applied to urban traffic environments with open interference.

In general, for the open urban traffic environment with multiple traffic targets, the existing methods are difficult to meet the requirements of operating vehicles for safe driving decision-making, and there is still a lack of safe driving decision-making methods that can provide specific driving suggestions such as driving actions and driving paths. In particular, there is a lack of research on safe driving decision-making of large commercial vehicles considering the impact of visual blind spots and sudden obstacles.

Contents of the invention

Purpose of the invention: In order to realize the safe driving decision-making of large commercial vehicles in urban low-speed environment and ensure the driving safety of vehicles, the present invention proposes a safe driving of large commercial vehicles in urban low-speed environment for heavy goods vehicles, heavy trucks and other self-driving commercial vehicles. decision making method. This method comprehensively considers the impact of factors such as visual blind spots, unexpected obstacles, and different driving conditions on driving safety, and can simulate the safe driving behavior of human drivers, providing more reasonable and safe driving strategies for autonomous driving vehicles. It can effectively guarantee the driving safety of self-driving commercial vehicles. At the same time, this method does not need to consider complex vehicle dynamic equations and body parameters. The calculation method is simple and clear, and it can output the safe driving strategy of autonomous driving commercial vehicles in real time. The cost of the sensors used is low, which is convenient for large-scale promotion.

Technical solution: In order to realize the object of the present invention, the technical solution adopted in the present invention is: a decision-making method for safe driving of large commercial vehicles in an urban low-speed environment. First, collect the safe driving behavior of human drivers in the urban traffic environment, and construct a safe driving behavior data set. Secondly, construct a decision-making model for safe driving of commercial vehicles based on multi-head attention. The model consists of two sub-networks, a deep double-Q network and a generative adversarial imitation learning. Among them, the deep double-Q network learns safe driving strategies in edge scenarios such as dangerous scenes and conflict scenes through unsupervised learning; the generated confrontational imitation learning sub-network imitates safe driving behaviors under different driving conditions and driving conditions. Finally, the safe driving decision-making model is trained to obtain driving strategies under different driving conditions and driving conditions, and realize advanced decision-making output for safe driving behavior of commercial vehicles. Specifically include the following steps:

Step 1: Collect the safe driving behavior of human drivers in urban traffic environment

In order to achieve driving decisions comparable to human drivers, the present invention collects safe driving behaviors under different driving conditions and driving conditions through actual road tests and driving simulations, and then constructs data representing safe driving behaviors of human drivers set. Specifically, it includes the following 4 sub-steps:

Sub-step 1: Use millimeter-wave radar, 128-line laser radar, vision sensor, Beidou sensor and inertial sensor to build a multi-dimensional target information synchronous acquisition system.

Sub-step 2: In a real urban environment, multiple drivers sequentially drive a commercial vehicle equipped with a multi-dimensional target information synchronous acquisition system, and the correlation of various driving behaviors such as lane change, lane keeping, vehicle following, acceleration and deceleration, etc. Data is collected and processed to obtain multi-source heterogeneous description data of various driving behaviors, such as the distance of obstacles in different directions measured by radar or visual sensors, the position, speed, acceleration and yaw measured by Beidou sensors and inertial sensors Angular velocity, etc., and steering wheel angle measured by on-board sensors, etc.

Sub-step 3: In order to imitate the safe driving behavior in borderline scenarios such as dangerous scenarios and conflict scenarios, build a virtual city scenario based on hardware-in-the-loop simulation. The constructed urban traffic scenarios include the following three categories:

(1) During the driving process of the vehicle, there will be laterally approaching traffic participants in front of the vehicle (that is, suddenly encountering obstacles);

(2) During the turning process of the vehicle, there are stationary traffic participants in the blind spot of the vehicle;

(3) During the turning process of the vehicle, there are moving traffic participants in the blind spot of the vehicle.

In the above traffic scenarios, there are various road network structures (straight roads, curves, and intersections) and various types of traffic participants (commercial vehicles, passenger vehicles, non-motorized vehicles, and pedestrians).

Multiple drivers drive commercial vehicles in the virtual scene through real controllers (steering wheel, accelerator and brake pedal), and collect the vehicle's horizontal and vertical positions, horizontal and vertical speeds, horizontal and vertical accelerations, and relative distances and distances from surrounding traffic participants. Relative speed and other information.

Sub-step 4: Based on the data collected in the real urban environment and driving simulation environment, construct a driving behavior data set for safe driving decision-making learning, which can be specifically expressed as:

In the formula, X represents the 2-tuple covering the state and action, that is, the constructed data set representing the safe driving behavior of human drivers, (s _j , a _j ) represents the "state-action" pair at time j, where s _j Indicates the state at time j, a _j represents the action at time j, that is, the action made by a human driver based on state s _j , and n represents the number of "state-action" pairs in the database.

Step 2: Construct a decision-making model for safe driving of commercial vehicles based on multi-head attention

In order to realize the safe driving decision-making of large-scale commercial vehicles in low-speed urban environments, the present invention comprehensively considers the influence of factors such as visual blind spots, unexpected obstacles, and driving conditions on driving safety, and establishes a safe driving decision-making model for commercial vehicles. Considering that deep reinforcement learning combines the perception ability of deep learning with the decision-making ability of reinforcement learning, and explores the traffic environment through unsupervised learning, the present invention uses deep reinforcement learning to improve safety in edge scenarios such as dangerous scenes and conflict scenes. Learn driving strategies. In addition, considering that imitation learning has the ability to imitate models, the present invention uses imitation learning to simulate the safe driving behavior of human drivers under different driving conditions and driving conditions. Therefore, the constructed safe driving decision-making model consists of two parts, which are described in detail as follows:

Sub-step 1: Define the basic parameters of the safe driving decision model

First, the safe driving decision-making problem in urban low-speed environment is transformed into a finite Markov decision process. Second, the basic parameters of the safe driving decision-making model are defined.

(1) Define the state space

In order to describe the motion state of the self-vehicle and nearby traffic participants, the present invention utilizes time series data and occupancy grid graphs to construct a state space. The specific description is as follows:

S _t = [S ₁ (t), S ₂ (t), S ₃ (t)] (2)

In the formula, S _t represents the state space at time t, S ₁ (t) and S ₂ (t) represent the state space related to the time series data at time t, and S ₃ (t) represents the state space related to the occupancy grid map at time t state space.

First, the ego vehicle's motion state is described using continuous position, velocity, acceleration, and heading angle information:

S ₁ (t)＝[p _x ,p _y ,v _x ,v _y ,a _x ,a _y ,θ _s ] (3)

In the formula, p _x , p _y represent the lateral position and longitudinal position of the self-vehicle respectively in meters, v _x , v _y represent the lateral speed and longitudinal speed of the self-vehicle respectively in meters per second, a _x , a _y Respectively represent the lateral acceleration and longitudinal acceleration of the own vehicle, the unit is meter per square second, θ _s represents the heading angle of the own vehicle, the unit is degree.

Secondly, use the relative motion state information of the self-vehicle and the surrounding traffic participants to describe the motion state of the surrounding traffic participants:

分别表示自车与第i个交通参与者的相对距离、相对速度和加速度，单位分别为米、米每秒和米每二次方秒。In the formula,

respectively represent the relative distance, relative speed and acceleration between the self-vehicle and the i-th traffic participant, and the units are meters, meters per second and meters per square second.

In the existing state space definition methods, a fixed coding method is often used, that is, the number of surrounding traffic participants considered is fixed. However, in the actual urban traffic scene, the number and position of traffic participants around the operating vehicle are constantly changing, and special consideration should be given to lateral collisions caused by sudden obstacles and visual blind spots. Although the method of fixed coding can achieve effective state representation, the number of traffic participants considered is limited (the minimum amount of information required to represent the scene is used), and it cannot accurately and comprehensively describe the impact of all surrounding traffic participants on the driving safety of operating vehicles. influences.

Finally, in order to more vividly describe the relative positional relationship between the self-vehicle and the surrounding traffic participants, and improve the reliability and effectiveness of decision-making, the present invention grids the road area and divides it into several a×b grid areas. The road area and vehicle targets are abstracted into a grid map, that is, the "existence" grid map S ₃ (t) used to describe the relative positional relationship. Wherein, a represents the length of the grid area, and b represents the width of the grid area.

The "existence" grid map contains four attributes, including grid coordinates, whether there is a vehicle, the category of the corresponding vehicle, and the distance from the left and right lanes. Among them, the grid with no traffic participants is set to "0", and the grid with traffic participants is set to "1". The position distribution between the grid and the grid where the self-vehicle is located is used to describe the relative distance between the two vehicles .

(2) Define the action space

Define the action space with lateral and longitudinal driving actions:

A _t ＝[a _left ,a _straight ,a _right ,a _accel ,a _cons ,a _decel ] (5)

In the formula, A _t represents the action space at time t, a _left , a _straight , and a _right represent left turn, straight line and right turn respectively, and a _accel , a _cons , a _decel represent acceleration, constant speed and deceleration respectively.

(3) Define the reward function

R _t =r ₁ +r ₂ +r ₃ (6)

In the formula, R _t represents the reward function at time t, r ₁ , r ₂ , and r ₃ represent the forward collision avoidance reward function, the backward collision avoidance reward function and the side collision avoidance reward function respectively, which can be obtained through formula (7), Formula (8) and formula (9) are obtained.

In the formula, TTC represents the collision time between the self-vehicle and the front obstacle, which can be obtained by dividing the distance between the self-vehicle and the front obstacle by the relative speed, TTC _thr represents the distance collision time threshold, RTTC represents the backward collision time, RTTC _thr represents the backward collision time threshold, the unit is second, x _lat represents the distance between the vehicle and the traffic participants on both sides, x _min represents the minimum lateral safety distance, the unit is meter, β ₁ , β ₂ , β ₃ respectively Indicates the weight coefficients of forward collision avoidance reward function, backward collision avoidance reward function and side collision avoidance reward function.

Sub-step 2: Construct a decision-making sub-network based on a deep double-Q network

Considering that the Double Deep Q Network (DDQN) improves the efficiency of data utilization by using the experience multiplexing pool, and can avoid parameter oscillation or divergence, it can reduce the negative effects of overestimation in the Q learning network. learning result. Therefore, the present invention utilizes a deep double-Q network to learn safe driving strategies in edge scenarios.

Different from dealing with a fixed-dimensional state space, dealing with feature information covering all surrounding traffic participants requires stronger feature extraction capabilities. Considering that the attention mechanism can capture more abundant feature information (dependence between the self-vehicle and the surrounding traffic participants), the present invention designs a policy network based on the multi-head attention mechanism. In addition, considering that the driving decision is only related to the movement state of the vehicle and the surrounding traffic participants, and should not be affected by the order of the traffic participants in the state space, the present invention uses the position encoding method (document: Vaswani, Ashish, et al. "Attention is all you need." Advances in Neural Information Processing Systems.2017.), Building permutation invariance into decision subnetworks.

The attention layer can be expressed as:

MultiHead(Q,K,V)＝Concat(head ₁ ,...,head _h )W ^O (10)

In the formula, MultiHead(Q,K,V) represents the multi-head attention value, Q represents the query vector, K represents the key vector, the dimension is d _k , V represents the value vector, the dimension is d _v , W ^O represents the parameters to be learned Matrix, head _h represents the hth head in the multi-head attention, in the present invention, h=2, can be calculated by following formula:

表示需要学习的参数矩阵。In the formula, Attention(Q,K,V) represents the output attention matrix,

Represents the parameter matrix that needs to be learned.

A decision-making sub-network based on a deep double-Q network is constructed, which is described in detail as follows.

First, the state space S _t is connected to Encoder 1, Encoder 2, and Encoder 3, respectively. Encoder 1 consists of two fully connected layers and outputs the ego-vehicle motion state encoding. Encoder 2 has the same structure as Encoder 1, and outputs relative motion state codes. Encoder 3 consists of two convolutional layers, and the output occupies a raster map encoding.

Among them, the number of neurons in the fully connected layer is 64, and the activation functions are all Tanh functions. The convolution kernels of the convolutional layer are all 3×3, and the stride is 2.

Secondly, the multi-head attention mechanism is used to analyze the dependence relationship between the self-vehicle and the surrounding traffic participants, so that the decision-making sub-network can notice the traffic participants who suddenly approach the self-vehicle or conflict with the driving path of the self-vehicle, and use different input sizes and arrangements Invariance is built into the decision subnetwork. The outputs of Encoder 1, Encoder 2, and Encoder 3 are all connected to a multi-head attention module to output an attention matrix. Again, connect the output attention matrix to decoder 1. Encoder 1 consists of a fully connected layer.

Among them, the number of neurons in the fully connected layer is 64, and the activation function is the Sigmoid function.

Sub-step 3: Build a decision-making sub-network based on generative confrontational imitation learning

In the complex urban traffic environment with open and multi-traffic target interference, it is difficult to construct an accurate and comprehensive reward function, especially it is difficult to quantitatively describe the uncertainty (such as sudden obstacles, traffic participants in the visual blind zone, etc.) impact on driving safety. In order to reduce the influence of the uncertainty of traffic environment and driving conditions on safe driving decision-making, and improve the effectiveness and reliability of driving decision-making, the present invention utilizes the generation confrontational imitation learning sub-network to learn the driving behavior data set and its generalized samples The driving strategy in the data, and then imitate the safe driving behavior under different driving conditions and driving conditions. The generative confrontational imitation learning sub-network consists of two parts, the generator and the discriminator, and the deep neural network is used to construct the generator network and the discriminator network respectively. The specific description is as follows:

(1) Build generator

Build a generator network. The input of the generator is the state space, and the output is the probability value f=π(·|s;θ) of each action in the action space, where θ represents the parameters of the generator network. First, the state space is sequentially connected with fully connected layers FC ₁ and FC ₂ to obtain feature F ₁ . The state space is sequentially combined with the fully connected layers FC ₃ and FC ₄ to obtain the feature F ₂ . At the same time, the state space is sequentially connected with the convolutional layer C ₁ and the convolutional layer C ₂ to obtain the feature F ₃ . Then, the features F ₁ , F ₂ and F ₃ are sequentially connected to the merge layer, the fully connected layer FC ₅ , and the Softmax activation function to obtain the output f=π(·|s; θ).

Among them, the number of neurons in the fully connected layers FC ₁ , FC ₂ , FC ₃ , FC ₄ and FC ₅ is 64, the convolution kernel of the convolution layer C ₁ is 3×3, and the step size is 2, and the convolution layer C The convolution kernel of ₂ is 3×3, and the step size is 1.

(2) Build a discriminator

维度为6，其中，φ表示判别器网络的参数。首先，状态空间依次与全连接层FC₆和FC₇相连，得到特征F₄。状态空间依次与全连接层FC₈和FC₉，得到特征F₅。同时，状态空间依次与卷积层C₃和卷积层C₄相连，得到特征F₆。然后，将特征F₄、F₅和F₆依次与合并层、全连接层FC₁₀、Sigmoid激活函数相连，得到输出

Build the discriminator network. The input of the discriminator is the state space, and the output is a vector

The dimension is 6, where φ represents the parameters of the discriminator network. First, the state space is sequentially connected with the fully connected layers FC ₆ and FC ₇ to obtain the feature F ₄ . The state space is sequentially combined with the fully connected layers FC ₈ and FC ₉ to obtain the feature F ₅ . At the same time, the state space is sequentially connected with the convolutional layer C ₃ and the convolutional layer C ₄ to obtain the feature F ₆ . Then, the features F ₄ , F ₅ and F ₆ are sequentially connected to the merge layer, the fully connected layer FC ₁₀ , and the Sigmoid activation function to obtain the output

Among them, the number of neurons in the fully connected layers FC ₆ , FC ₇ , FC ₈ , FC ₉ and FC ₁₀ is 64, the convolution kernel of the convolution layer C ₃ is 3×3, and the step size is 2, and the convolution layer C The convolution kernel of ₄ is 3×3, and the step size is 1.

Step 3: Train the decision-making model for safe driving of commercial vehicles

First, a decision sub-network based on generative adversarial imitation learning is trained. The goal of the Generative Adversarial Imitation Learning sub-network is to learn a generator network such that the discriminator cannot distinguish between the driving actions generated by the generator and those in the driving behavior dataset. Specifically, the following sub-steps are included:

中，初始化生成器网络参数θ₀和判别器网络参数ω₀；Sub-step 1: On the driving behavior dataset

, initialize the generator network parameter θ ₀ and the discriminator network parameter ω ₀ ;

Sub-step 2: Carry out L iterations to solve, each iteration includes sub-steps 2.1 to 2.2, specifically:

Sub-step 2.1: Update the discriminator parameters ω _i →ω _i+1 using the gradient formula described by Equation (13):

表示参数为ω的神经网络损失函数的梯度函数；In the formula,

Represents the gradient function of the neural network loss function with parameter ω;

利用信赖域策略优化算法更新生成器参数θ_i→θ_i+1。Sub-step 2.2: Set the reward function

Utilize the trust region policy optimization algorithm to update the generator parameters θ _i →θ _i+1 .

First, on the basis of the above network training results, continue to train and build a decision-making sub-network based on DDQN, which specifically includes the following sub-steps:

Sub-step 3: Initialize the capacity of experience reuse pool D as N;

Sub-step 4: The Q value corresponding to the initialization action is a random value;

Sub-step 5: Carry out M iterations to solve, each iteration includes sub-steps 5.1 to 5.2, specifically:

Sub-step 5.1: Initialize state s ₀ , initialize strategy parameter φ ₀ ;

Sub-step 5.2: Perform T iterations to solve, each iteration includes sub-steps 5.21 to 5.27, specifically:

Sub-step 5.21: Randomly select a driving action;

Sub-step 5.22: Otherwise select a _t = max _a Q ^* (φ(s _t ),a; θ);

In the formula, Q ^* ( ) represents the optimal action-value function, and a _t represents the action at time t;

Sub-step 5.23: Execute action a _t to obtain reward value r _{t at time t} and state s _{t+1 at time t+1} ;

Sub-step 5.24: Store samples (φ _t , a _t , r _t ,φ _t+1 ) in the experience multiplexing pool D;

Sub-step 5.25: Randomly select a small batch of samples (φ _j ,a _j ,r _j ,φ _j+1 ) from the experience multiplexing pool D;

Sub-step 5.26: Calculate the iteration target using the following formula:

表示t时刻目标网络的权重；γ表示折扣因子；argmax(·)表示使目标函数具有最大值的变量，y_i表示i时刻的迭代目标，p(s,a)表示动作分布；In the formula,

Indicates the weight of the target network at time t; γ indicates the discount factor; argmax( ) indicates the variable that makes the objective function have the maximum value, y _i indicates the iterative target at time i, and p(s,a) indicates the action distribution;

Sub-step 5.27: Use the following formula to perform gradient descent on (y _i -Q(φ _j ,a _j ; θ)) ² :

表示参数为θ_i的神经网络损失函数的梯度函数，ε表示在ε-greedy探索策略下，随机选择一个行为的概率；θ_i表示i时刻迭代的参数，L_i(θ_i)表示i时刻的损失函数，Q(s,a；θ_i)表示目标网络的动作价值函数，a′表示状态s′所有可能存在的动作。In the formula,

Indicates the gradient function of the neural network loss function whose parameter is θ _i , ε indicates the probability of randomly selecting a behavior under the ε-greedy exploration strategy; θ _i indicates the parameter of iteration at time i, and L _i (θ _i ) indicates the The loss function, Q(s,a; θ _i ) represents the action value function of the target network, and a' represents all possible actions in the state s'.

After the training of the safe driving decision model of commercial vehicles is completed, the state space information collected by the sensor is input into the safe driving decision model, which can output high-level driving decisions such as steering, straight ahead, acceleration and deceleration in real time, which can effectively guarantee the operation in the urban low-speed environment The vehicle is safe to operate.

Beneficial effects: Compared with general driving decision-making methods, the method proposed by the present invention has more effective and reliable characteristics, which are specifically reflected in:

(1) The method proposed by the present invention can simulate the safe driving behavior of human drivers, provide more reasonable and safe driving strategies for commercial vehicles in urban low-speed environments, and realize safe driving decisions for large commercial vehicles with a high human-like level, It can effectively guarantee the driving safety of the vehicle.

(2) The method proposed by the present invention comprehensively considers the influence of factors such as visual blind spots, unexpected obstacles, and different driving conditions on driving safety, and conducts strategy learning and training in normal driving scenes and edge scenes, further improving driving decision-making. effectiveness and reliability.

(3) The method proposed in the present invention introduces a multi-head attention mechanism, considers the dynamic interaction between the vehicle and the surrounding traffic participants, and can handle safe driving decisions with variable input (the number of surrounding traffic participants changes dynamically) .

(4) The method proposed by the present invention does not need to consider complex vehicle dynamic equations and vehicle body parameters. The calculation method is simple and clear, and can output the safe driving decision-making strategy of large commercial vehicles in real time, and the cost of the sensors used is low, which is convenient for large-scale promotion.

Description of drawings

Fig. 1 is a technical roadmap of the present invention;

Fig. 2 is a schematic diagram of the strategy network structure based on the multi-head attention mechanism designed by the present invention;

Fig. 3 is a schematic diagram of the generator network structure designed by the present invention;

Fig. 4 is a schematic diagram of the structure of the discriminator network designed by the present invention.

Detailed ways

The technical solutions of the present invention will be further described below in conjunction with the accompanying drawings and embodiments.

Aiming at the urban traffic environment which is open and interfered by multiple traffic targets, the invention proposes a safe driving decision-making method for large commercial vehicles with a highly human-like level. First, collect the safe driving behavior of human drivers in the urban traffic environment, and construct a safe driving behavior data set. Secondly, construct a decision-making model for safe driving of commercial vehicles based on multi-head attention. The model consists of two sub-networks, a deep double-Q network and a generative adversarial imitation learning. Among them, the deep double-Q network learns safe driving strategies in edge scenarios such as dangerous scenes and conflict scenes through unsupervised learning; the generated confrontational imitation learning sub-network imitates safe driving behaviors under different driving conditions and driving conditions. Finally, the safe driving decision-making model is trained to obtain driving strategies under different driving conditions and driving conditions, and realize advanced decision-making output for safe driving behavior of commercial vehicles. The method proposed by the present invention can simulate the safe driving behavior of human drivers, and considers the influence of factors such as visual blind spots and sudden obstacles on driving safety, and provides more reasonable and safe driving strategies for large-scale commercial vehicles, realizing urban Safe driving decisions for commercial vehicles in traffic environments. Technical route of the present invention is as shown in Figure 1, and concrete steps are as follows:

Step 1: Collect the safe driving behavior of human drivers in urban traffic environment

In order to achieve driving decisions comparable to human drivers, the present invention collects safe driving behaviors under different driving conditions and driving conditions through actual road tests and driving simulations, and then constructs data representing safe driving behaviors of human drivers set. Specifically, it includes the following 5 sub-steps:

Sub-step 1: Use millimeter-wave radar, 128-line laser radar, vision sensor, Beidou sensor and inertial sensor to build a multi-dimensional target information synchronous acquisition system.

Sub-step 2: In a real urban environment, multiple drivers sequentially drive a commercial vehicle equipped with a multi-dimensional target information synchronous acquisition system.

Sub-step 3: Collect and process data related to various driving behaviors such as driver's lane change, lane keeping, vehicle following, acceleration and deceleration, etc., and obtain multi-source heterogeneous description data of each driving behavior, such as radar or visual sensors Obstacle distances in different orientations are measured, the position, velocity, acceleration, and yaw rate measured by Beidou sensors and inertial sensors, and the steering wheel angle measured by on-board sensors.

Sub-step 4: In order to imitate the safe driving behavior in borderline scenarios such as dangerous scenarios and conflict scenarios, build a virtual city scenario based on hardware-in-the-loop simulation. The constructed urban traffic scenarios include the following three categories:

(1) During the driving process of the vehicle, there will be laterally approaching traffic participants in front of the vehicle (that is, suddenly encountering obstacles);

(2) During the turning process of the vehicle, there are stationary traffic participants in the blind spot of the vehicle;

(3) During the turning process of the vehicle, there are moving traffic participants in the blind spot of the vehicle.

In the above traffic scenarios, there are various road network structures (straight roads, curves, and intersections) and various types of traffic participants (commercial vehicles, passenger vehicles, non-motorized vehicles, and pedestrians).

Multiple drivers drive commercial vehicles in the virtual scene through real controllers (steering wheel, accelerator and brake pedal), and collect the vehicle's horizontal and vertical positions, horizontal and vertical speeds, horizontal and vertical accelerations, and relative distances and distances from surrounding traffic participants. Relative speed and other information.

Sub-step 5: Based on the data collected in the real urban environment and driving simulation environment, construct a driving behavior data set for safe driving decision-making learning, which can be specifically expressed as:

In the formula, X represents the 2-tuple covering the state and action, that is, the constructed data set representing the safe driving behavior of human drivers, (s _j , a _j ) represents the "state-action" pair at time j, where s _j Indicates the state at time j, a _j represents the action at time j, that is, the action made by a human driver based on state s _j , and n represents the number of "state-action" pairs in the database.

Step 2: Construct a decision-making model for safe driving of commercial vehicles based on multi-head attention

In order to realize the safe driving decision-making of large-scale commercial vehicles in low-speed urban environments, the present invention comprehensively considers the influence of factors such as visual blind spots, unexpected obstacles, and driving conditions on driving safety, and establishes a safe driving decision-making model for commercial vehicles. Considering that deep reinforcement learning combines the perception ability of deep learning with the decision-making ability of reinforcement learning, and explores the traffic environment through unsupervised learning, the present invention uses deep reinforcement learning to improve safety in edge scenarios such as dangerous scenes and conflict scenes. Learn driving strategies. In addition, considering that imitation learning has the ability to imitate models, the present invention uses imitation learning to simulate the safe driving behavior of human drivers under different driving conditions and driving conditions. Therefore, the constructed safe driving decision-making model consists of two parts, which are described in detail as follows:

Sub-step 1: Define the basic parameters of the safe driving decision model

First, the safe driving decision-making problem in urban low-speed environment is transformed into a finite Markov decision process. Second, the basic parameters of the safe driving decision-making model are defined.

(1) Define the state space

In order to describe the motion state of the self-vehicle and nearby traffic participants, the present invention utilizes time series data and occupancy grid graphs to construct a state space. The specific description is as follows:

S _t = [S ₁ (t), S ₂ (t), S ₃ (t)] (2)

In the formula, S _t represents the state space at time t, S ₁ (t) and S ₂ (t) represent the state space related to the time series data at time t, and S ₃ (t) represents the state space related to the occupancy grid map at time t state space.

First, the ego vehicle's motion state is described using continuous position, velocity, acceleration, and heading angle information:

S ₁ (t)＝[p _x ,p _y ,v _x ,v _y ,a _x ,a _y ,θ _s ] (3)

In the formula, p _x , p _y represent the lateral position and longitudinal position of the self-vehicle respectively in meters, v _x , v _y represent the lateral speed and longitudinal speed of the self-vehicle respectively in meters per second, a _x , a _y Respectively represent the lateral acceleration and longitudinal acceleration of the own vehicle, the unit is meter per square second, θ _s represents the heading angle of the own vehicle, the unit is degree.

Secondly, use the relative motion state information of the self-vehicle and the surrounding traffic participants to describe the motion state of the surrounding traffic participants:

分别表示自车与第i个交通参与者的相对距离、相对速度和加速度，单位分别为米、米每秒和米每二次方秒。In the formula,

respectively represent the relative distance, relative speed and acceleration between the self-vehicle and the i-th traffic participant, and the units are meters, meters per second and meters per square second.

In the existing state space definition methods, a fixed coding method is often used, that is, the number of surrounding traffic participants considered is fixed. However, in the actual urban traffic scene, the number and position of traffic participants around the operating vehicle are constantly changing, and special consideration should be given to lateral collisions caused by sudden obstacles and visual blind spots. Although the method of fixed coding can achieve effective state representation, the number of traffic participants considered is limited (the minimum amount of information required to represent the scene is used), and it cannot accurately and comprehensively describe the impact of all surrounding traffic participants on the driving safety of operating vehicles. influences.

Finally, in order to more vividly describe the relative positional relationship between the self-vehicle and the surrounding traffic participants, and improve the reliability and effectiveness of decision-making, the present invention grids the road area and divides it into several a×b grid areas. The road area and vehicle targets are abstracted into a grid map, that is, the "existence" grid map S ₃ (t) used to describe the relative positional relationship. Wherein, a represents the length of the grid area, and b represents the width of the grid area.

The "existence" grid map contains four attributes, including grid coordinates, whether there is a vehicle, the category of the corresponding vehicle, and the distance from the left and right lanes. Among them, the grid with no traffic participants is set to "0", and the grid with traffic participants is set to "1". The position distribution between the grid and the grid where the self-vehicle is located is used to describe the relative distance between the two vehicles .

(2) Define the action space

Define the action space with lateral and longitudinal driving actions:

A _t ＝[a _left ,a _straight ,a _right ,a _accel ,a _cons ,a _decel ] (5)

In the formula, A _t represents the action space at time t, a _left , a _straight , and a _right represent left turn, straight line and right turn respectively, and a _accel , a _cons , a _decel represent acceleration, constant speed and deceleration respectively.

(3) Define the reward function

R _t =r ₁ +r ₂ +r ₃ (6)

In the formula, R _t represents the reward function at time t, r ₁ , r ₂ , and r ₃ represent the forward collision avoidance reward function, the backward collision avoidance reward function and the side collision avoidance reward function respectively, which can be obtained through formula (7), Formula (8) and formula (9) are obtained.

In the formula, TTC represents the collision time between the self-vehicle and the front obstacle, which can be obtained by dividing the distance between the self-vehicle and the front obstacle by the relative speed, TTC _thr represents the distance collision time threshold, RTTC represents the backward collision time, RTTC _thr represents the backward collision time threshold, the unit is second, x _lat represents the distance between the vehicle and the traffic participants on both sides, x _min represents the minimum lateral safety distance, the unit is meter, β ₁ , β ₂ , β ₃ respectively Indicates the weight coefficients of forward collision avoidance reward function, backward collision avoidance reward function and side collision avoidance reward function.

Sub-step 2: Build a decision-making subnetwork based on DDQN

Considering that the Double Deep Q Network (DDQN) improves the efficiency of data utilization by using the experience multiplexing pool, and can avoid parameter oscillation or divergence, it can reduce the negative effects of overestimation in the Q learning network. learning result. Therefore, the present invention utilizes a deep double-Q network to learn safe driving strategies in edge scenarios.

Different from dealing with a fixed-dimensional state space, dealing with feature information covering all surrounding traffic participants requires stronger feature extraction capabilities. Considering that the attention mechanism can capture more abundant feature information (dependence between the self-vehicle and the surrounding traffic participants), the present invention designs a policy network based on the multi-head attention mechanism. In addition, considering that the driving decision is only related to the movement state of the vehicle and the surrounding traffic participants, and should not be affected by the order of the traffic participants in the state space, the present invention uses the position encoding method (document: Vaswani, Ashish, et al. "Attention is all you need." Advances in Neural Information Processing Systems.2017.), Building permutation invariance into decision subnetworks.

The attention layer can be expressed as:

MultiHead(Q,K,V)＝Concat(head ₁ ,...,head _h )W ^O (10)

In the formula, MultiHead(Q,K,V) represents the multi-head attention value, Q represents the query vector, K represents the key vector, the dimension is d _k , V represents the value vector, the dimension is d _v , W ^O represents the parameters to be learned Matrix, head _h represents the hth head in the multi-head attention, in the present invention, h=2, can be calculated by following formula:

表示需要学习的参数矩阵。In the formula, Attention(Q,K,V) represents the output attention matrix,

Represents the parameter matrix that needs to be learned.

Construct a decision-making sub-network based on a deep double-Q network, as shown in Figure 2, and the specific description is as follows.

First, the state space S _t is connected to Encoder 1, Encoder 2, and Encoder 3, respectively. Encoder 1 consists of two fully connected layers and outputs the ego-vehicle motion state encoding. Encoder 2 has the same structure as Encoder 1, and outputs relative motion state codes. Encoder 3 consists of two convolutional layers, and the output occupies a raster map encoding.

Among them, the number of neurons in the fully connected layer is 64, and the activation functions are all Tanh functions. The convolution kernels of the convolutional layer are all 3×3, and the stride is 2.

Secondly, the multi-head attention mechanism is used to analyze the dependence relationship between the self-vehicle and the surrounding traffic participants, so that the decision-making sub-network can notice the traffic participants who suddenly approach the self-vehicle or conflict with the driving path of the self-vehicle, and use different input sizes and arrangements Invariance is built into the decision subnetwork. The outputs of Encoder 1, Encoder 2, and Encoder 3 are all connected to a multi-head attention module to output an attention matrix. Again, connect the output attention matrix to decoder 1. Encoder 1 consists of a fully connected layer.

Among them, the number of neurons in the fully connected layer is 64, and the activation function is the Sigmoid function.

Sub-step 3: Build a decision-making sub-network based on generative confrontational imitation learning

In the complex urban traffic environment with open and multi-traffic target interference, it is difficult to construct an accurate and comprehensive reward function, especially it is difficult to quantitatively describe various uncertainties (such as sudden obstacles, traffic participants in the visual blind zone, etc.) ) on driving safety. In order to reduce the influence of the uncertainty of traffic environment and driving conditions on safe driving decision-making, and improve the effectiveness and reliability of driving decision-making, the present invention utilizes the generation confrontational imitation learning sub-network to learn the driving behavior data set and its generalized samples The driving strategy in the data, and then imitate the safe driving behavior under different driving conditions and driving conditions. The generative confrontational imitation learning sub-network consists of two parts, the generator and the discriminator, and the deep neural network is used to construct the generator network and the discriminator network respectively. The specific description is as follows:

(1) Build generator

Build a generator network as shown in Figure 3. The input of the generator is the state space, and the output is the probability value f=π(·|s;θ) of each action in the action space, where θ represents the parameters of the generator network. First, the state space is sequentially connected with fully connected layers FC ₁ and FC ₂ to obtain feature F ₁ . The state space is sequentially combined with the fully connected layers FC ₃ and FC ₄ to obtain the feature F ₂ . At the same time, the state space is sequentially connected with the convolutional layer C ₁ and the convolutional layer C ₂ to obtain the feature F ₃ . Then, the features F ₁ , F ₂ and F ₃ are sequentially connected to the merge layer, the fully connected layer FC ₅ , and the Softmax activation function to obtain the output f=π(·|s; θ).

Among them, the number of neurons in the fully connected layers FC ₁ , FC ₂ , FC ₃ , FC ₄ and FC ₅ is 64, the convolution kernel of the convolution layer C ₁ is 3×3, and the step size is 2, and the convolution layer C The convolution kernel of ₂ is 3×3, and the step size is 1.

(2) Build a discriminator

维度为6，其中，φ表示判别器网络的参数。首先，状态空间依次与全连接层FC₆和FC₇相连，得到特征F₄。状态空间依次与全连接层FC₈和FC₉，得到特征F₅。同时，状态空间依次与卷积层C₃和卷积层C₄相连，得到特征F₆。然后，将特征F₄、F₅和F₆依次与合并层、全连接层FC₁₀、Sigmoid激活函数相连，得到输出

Construct the discriminator network as shown in Figure 4. The input of the discriminator is the state space, and the output is a vector

The dimension is 6, where φ represents the parameters of the discriminator network. First, the state space is sequentially connected with the fully connected layers FC ₆ and FC ₇ to obtain the feature F ₄ . The state space is sequentially combined with the fully connected layers FC ₈ and FC ₉ to obtain the feature F ₅ . At the same time, the state space is sequentially connected with the convolutional layer C ₃ and the convolutional layer C ₄ to obtain the feature F ₆ . Then, the features F ₄ , F ₅ and F ₆ are sequentially connected to the merge layer, the fully connected layer FC ₁₀ , and the Sigmoid activation function to obtain the output

Among them, the number of neurons in the fully connected layers FC ₆ , FC ₇ , FC ₈ , FC ₉ and FC ₁₀ is 64, the convolution kernel of the convolution layer C ₃ is 3×3, and the step size is 2, and the convolution layer C The convolution kernel of ₄ is 3×3, and the step size is 1.

Step 3: Train the decision-making model for safe driving of commercial vehicles

First, a decision sub-network based on generative adversarial imitation learning is trained. The goal of the Generative Adversarial Imitation Learning sub-network is to learn a generator network such that the discriminator cannot distinguish between the driving actions generated by the generator and those in the driving behavior dataset. Specifically, the following sub-steps are included:

中，初始化生成器网络参数θ₀和判别器网络参数ω₀；Sub-step 1: On the driving behavior dataset

, initialize the generator network parameter θ ₀ and the discriminator network parameter ω ₀ ;

Sub-step 2: Carry out L iterations to solve, each iteration includes sub-steps 2.1 to 2.2, specifically:

Sub-step 2.1: Update the discriminator parameters ω _i →ω _i+1 using the gradient formula described by Equation (13):

表示参数为ω的神经网络损失函数的梯度函数；In the formula,

Represents the gradient function of the neural network loss function with parameter ω;

利用信赖域策略优化算法更新生成器参数θ_i→θ_i+1。Sub-step 2.2: Set the reward function

Utilize the trust region policy optimization algorithm to update the generator parameters θ _i →θ _i+1 .

First, on the basis of the above network training results, continue to train and build a decision-making sub-network based on DDQN, which specifically includes the following sub-steps:

Sub-step 3: Initialize the capacity of experience reuse pool D as N;

Sub-step 4: The Q value corresponding to the initialization action is a random value;

Sub-step 5: Carry out M iterations to solve, each iteration includes sub-steps 5.1 to 5.2, specifically:

Sub-step 5.1: Initialize state s ₀ , initialize strategy parameter φ ₀ ;

Sub-step 5.2: Perform T iterations to solve, each iteration includes sub-steps 5.21 to 5.27, specifically:

Sub-step 5.21: Randomly select a driving action;

Sub-step 5.22: Otherwise select a _t = max _a Q ^* (φ(s _t ),a; θ);

In the formula, Q ^* ( ) represents the optimal action-value function, and a _t represents the action at time t;

Sub-step 5.23: Execute action a _t to obtain reward value r _{t at time t} and state s _{t+1 at time t+1} ;

Sub-step 5.24: Store samples (φ _t , a _t , r _t ,φ _t+1 ) in the experience multiplexing pool D;

Sub-step 5.25: Randomly select a small batch of samples (φ _j ,a _j ,r _j ,φ _j+1 ) from the experience multiplexing pool D;

Sub-step 5.26: Calculate the iteration target using the following formula:

表示t时刻目标网络的权重；γ表示折扣因子；argmax(·)表示使目标函数具有最大值的变量，y_i表示i时刻的迭代目标，p(s,a)表示动作分布；In the formula,

Indicates the weight of the target network at time t; γ indicates the discount factor; argmax( ) indicates the variable that makes the objective function have the maximum value, y _i indicates the iterative target at time i, and p(s,a) indicates the action distribution;

Sub-step 5.27: Use the following formula to perform gradient descent on (y _i -Q(φ _j ,a _j ; θ)) ² :

表示参数为θ_i的神经网络损失函数的梯度函数，ε表示在ε-greedy探索策略下，随机选择一个行为的概率；θ_i表示i时刻迭代的参数，L_i(θ_i)表示i时刻的损失函数，Q(s,a；θ_i)表示目标网络的动作价值函数，a′表示状态s′所有可能存在的动作。In the formula,

Indicates the gradient function of the neural network loss function whose parameter is θ _i , ε indicates the probability of randomly selecting a behavior under the ε-greedy exploration strategy; θ _i indicates the parameter of iteration at time i, and L _i (θ _i ) indicates the The loss function, Q(s,a; θ _i ) represents the action value function of the target network, and a' represents all possible actions in the state s'.

After the training of the safe driving decision model of commercial vehicles is completed, the state space information collected by the sensor is input into the safe driving decision model, which can output high-level driving decisions such as steering, straight ahead, acceleration and deceleration in real time, which can effectively guarantee the operation in the urban low-speed environment The vehicle is safe to operate.

Claims (1)

1. The decision-making method for safe driving of large commercial vehicles in urban low-speed environments. First, collect the safe driving behaviors of human drivers in urban traffic environments, and construct a safe driving behavior data set; secondly, construct a safe driving of commercial vehicles based on multi-head attention Decision-making model; this model includes two sub-networks of deep double-Q network and generative confrontational imitation learning; among them, the deep double-Q network learns safe driving strategies in edge scenarios such as dangerous scenes and conflict scenes through unsupervised learning; generative confrontational imitation learning The learning sub-network imitates the safe driving behavior of human drivers under different driving conditions and driving conditions; finally, trains the safe driving decision-making model to obtain driving strategies under different driving conditions and driving conditions, and realizes the safe driving behavior of commercial vehicles. High-level decision output; characterized by: Step 1: Collect the safe driving behavior of human drivers in urban traffic environment Through the actual road test and driving simulation, the safe driving behavior under different driving conditions and driving conditions is collected, and then the data set representing the safe driving behavior of human drivers is constructed; specifically, it includes the following four sub-steps: Sub-step 1: Use millimeter-wave radar, 128-line laser radar, vision sensor, Beidou sensor and inertial sensor to build a multi-dimensional target information synchronous acquisition system; Sub-step 2: In a real urban environment, multiple drivers sequentially drive a commercial vehicle equipped with a multi-dimensional target information synchronous acquisition system, and the correlation of various driving behaviors of the driver's lane change, lane keeping, vehicle following, acceleration and deceleration The data is collected and processed to obtain multi-source heterogeneous description data of various driving behaviors, including obstacle distances in different directions measured by radar or visual sensors, position, speed, acceleration and yaw measured by Beidou sensors and inertial sensors Angular velocity, and steering wheel angle measured by on-board sensors; Sub-step 3: In order to imitate safe driving behavior in borderline scenarios such as dangerous scenarios and conflict scenarios, build a virtual city scenario based on hardware-in-the-loop simulation; the constructed urban traffic scenarios include the following three categories: (1) During the driving process of the vehicle, there will be traffic participants approaching laterally in front of the vehicle, that is, suddenly encountering obstacles; (2) During the turning process of the vehicle, there are stationary traffic participants in the blind spot of the vehicle; (3) During the turning process of the vehicle, there are moving traffic participants in the blind spot of the vehicle; In the above traffic scenarios, there are various road network structures, including straight roads, curves, and intersections, and various types of traffic participants, including commercial vehicles, passenger vehicles, non-motorized vehicles, and pedestrians; Multiple drivers drive commercial vehicles in the virtual scene through real controllers with steering wheels, accelerators and brake pedals, and collect the vehicle's horizontal and vertical positions, horizontal and vertical speeds, horizontal and vertical accelerations, and relative distances from surrounding traffic participants and relative velocity information; Sub-step 4: Based on the data collected in the real urban environment and driving simulation environment, construct a driving behavior data set for safe driving decision-making learning, specifically expressed as:

In the formula, X represents the 2-tuple covering the state and action, that is, the constructed data set representing the safe driving behavior of human drivers, (s _j , a _j ) represents the "state-action" pair at time j, where s _j Indicates the state at time j, a _j represents the action at time j, that is, the action made by the human driver based on the state s _j , n represents the number of "state-action" pairs in the database; Step 2: Construct a decision-making model for safe driving of commercial vehicles based on multi-head attention Use deep reinforcement learning to learn safe driving strategies in dangerous scenes and edge scenes of conflict scenes; in addition, considering that imitation learning has the ability to imitate models, use imitation learning to simulate the behavior of human drivers under different driving conditions and driving conditions. Safe driving behavior; therefore, the constructed safe driving decision-making model consists of two parts, which are specifically described as follows: Sub-step 1: Define the basic parameters of the safe driving decision model Firstly, transform the safe driving decision-making problem in urban low-speed environment into a finite Markov decision process; secondly, define the basic parameters of the safe driving decision-making model; (1) Define the state space In order to describe the movement state of the self-vehicle and nearby traffic participants, the state space is constructed by using time series data and occupancy grid graph; the specific description is as follows: S _t = [S ₁ (t), S ₂ (t), S ₃ (t)] (2) In the formula, S _t represents the state space at time t, S ₁ (t) represents the state space of the ego vehicle related to time series data at time t, and S ₂ (t) represents the surrounding traffic participants related to time series data at time t The state space of , S ₃ (t) represents the state space related to the occupancy grid graph at time t; First, the ego vehicle's motion state is described using continuous position, velocity, acceleration, and heading angle information: S ₁ (t)＝[p _x ,p _y ,v _x ,v _y ,a _x ,a _y ,θ _s ] (3) In the formula, p _x , p _y represent the lateral position and longitudinal position of the self-vehicle respectively in meters, v _x , v _y represent the lateral speed and longitudinal speed of the self-vehicle respectively in meters per second, a _x , a _y Respectively represent the lateral acceleration and longitudinal acceleration of the own vehicle, the unit is meter per square second, θ _s represents the heading angle of the own vehicle, the unit is degree; Secondly, use the relative motion state information of the self-vehicle and the surrounding traffic participants to describe the motion state of the surrounding traffic participants:

In the formula,

a _i represent the relative distance, relative speed and acceleration between the ego vehicle and the i-th traffic participant, and the units are meters, meters per second and meters per square second; Finally, the road area is rasterized and divided into several p×q grid areas, and the road area and vehicle targets are abstracted into a grid map, which is the "existence" grid map S ₃ ( t); wherein, p represents the length of the grid area, and q represents the width of the grid area; The "existence" grid map contains four attributes, including grid coordinates, whether there is a vehicle, the category of the corresponding vehicle, and the distance from the left and right lane lines; among them, the grid without traffic participants is set to "0", and the grid with traffic The grid of the participant is set to "1", and the position distribution between the grid and the grid of the self-vehicle is used to describe the relative distance between the two vehicles; (2) Define the action space Define the action space with lateral and longitudinal driving actions: A _t ＝[a _left ,a _straight ,a _right ,a _accel ,a _cons ,a _decel ] (5) In the formula, A _t represents the action space at time t, a _left , a _straight , and a _right represent left turn, straight line and right turn respectively, and a _accel , a _cons , a _decel represent acceleration, constant speed and deceleration respectively; (3) Define the reward function R _t =r ₁ +r ₂ +r ₃ (6) In the formula, R _t represents the reward function at time t, r ₁ , r ₂ , and r ₃ represent the forward collision avoidance reward function, the backward collision avoidance reward function and the side collision avoidance reward function respectively, which can be obtained through formula (7), Formula (8) and formula (9) are obtained;

In the formula, TTC represents the collision time between the self-vehicle and the front obstacle, which can be obtained by dividing the distance between the self-vehicle and the front obstacle by the relative speed, TTC _thr represents the distance collision time threshold, RTTC represents the backward collision time, RTTC _thr represents the backward collision time threshold, the unit is second, x _lat represents the distance between the vehicle and the traffic participants on both sides, x _min represents the minimum lateral safety distance, the unit is meter, β ₁ , β ₂ , β ₃ respectively Indicates the weight coefficients of forward collision avoidance reward function, backward collision avoidance reward function and side collision avoidance reward function; Sub-step 2: Construct a decision-making sub-network based on a deep double-Q network Use deep double Q network to learn safe driving strategy in edge scenarios; Design a policy network based on a multi-head attention mechanism; in addition, use a positional encoding method to build permutation invariance into the decision sub-network; The attention layer can be expressed as: MultiHead(Q,K,V)＝Concat(head ₁ ,...,head _h )W ^O (10) In the formula, MultiHead(Q,K,V) represents the multi-head attention value, Q represents the query vector, K represents the key vector, the dimension is d _k , V represents the value vector, the dimension is d _v , W ^O represents the parameters to be learned Matrix, head _h represents the hth head in the multi-head attention, in the present invention, h=2, calculated by the following formula:

In the formula, Attention(Q,K,V) represents the output attention matrix,

Represents the parameter matrix that needs to be learned; Construct a decision-making sub-network, which is described in detail as follows; First, the state space S _t is connected to encoder 1, encoder 2, and encoder 3 respectively; encoder 1 is composed of two fully connected layers, and outputs the self-vehicle motion state code; encoder 2 has the same structure as encoder 1, Output relative motion state encoding; Encoder 3 consists of two convolutional layers, and the output occupies a raster image encoding; Among them, the number of neurons in the fully connected layer is 64, and the activation function is Tanh function; the convolution kernel of the convolution layer is 3×3, and the step size is 2; Secondly, the multi-head attention mechanism is used to analyze the dependence relationship between the self-vehicle and the surrounding traffic participants, so that the decision-making sub-network can notice the traffic participants who suddenly approach the self-vehicle or conflict with the driving path of the self-vehicle, and use different input sizes and arrangements The invariance is built into the decision-making sub-network; the outputs of Encoder 1, Encoder 2, and Encoder 3 are all connected to the multi-head attention module, and the attention matrix is output; again, the output attention matrix is connected to Decoder 1; the encoding Device 1 consists of a fully connected layer; Among them, the number of neurons in the fully connected layer is 64, and the activation function is the Sigmoid function; Sub-step 3: Build a decision-making sub-network based on generative confrontational imitation learning Use the generative adversarial imitation learning subnetwork to learn the driving strategy in the driving behavior data set and its generalized sample data, and then imitate the safe driving behavior of human drivers under different driving conditions and driving conditions; generate the adversarial imitation learning subnetwork It consists of two parts, the generator and the discriminator, and uses the deep neural network to build the generator network and the discriminator network respectively; the specific description is as follows: (1) Build generator Build a generator network as shown in Figure 3; the input of the generator is the state space, and the output is the probability value f=π(|s; θ) of each action in the action space, wherein, θ represents the parameter of the generator network; First, the state space is sequentially connected with the fully connected layers FC ₁ and FC ₂ to obtain the feature F ₁ ; the state space is sequentially connected with the fully connected layers FC ₃ and FC ₄ to obtain the feature F ₂ ; at the same time, the state space is sequentially connected with the convolutional layer C ₁ Connect with the convolutional layer C ₂ to obtain the feature F ₃ ; then, connect the features F ₁ , F ₂ and F ₃ to the merge layer, the fully connected layer FC ₅ , and the Softmax activation function in turn to obtain the output f=π(·|s ;θ); Among them, the number of neurons in the fully connected layers FC ₁ , FC ₂ , FC ₃ , FC ₄ and FC ₅ is 64, the convolution kernel of the convolution layer C ₁ is 3×3, and the step size is 2, and the convolution layer C The convolution kernel of ₂ is 3×3, and the step size is 1; (2) Build a discriminator Build a discriminator network; the input of the discriminator is a state space, and the output is a vector

The dimension is 6, where φ represents the parameters of the discriminator network; first, the state space is sequentially connected with the fully connected layers FC ₆ and FC ₇ to obtain the feature F ₄ ; the state space is sequentially connected with the fully connected layers FC ₈ and FC ₉ to obtain the feature F ₅ ; at the same time, the state space is sequentially connected with the convolutional layer C ₃ and the convolutional layer C ₄ to obtain the feature F ₆ ; then, the features F ₄ , F ₅ and F ₆ are sequentially connected with the merge layer, the fully connected layer FC ₁₀ , The Sigmoid activation function is connected to get the output

Among them, the number of neurons in the fully connected layers FC ₆ , FC ₇ , FC ₈ , FC ₉ and FC ₁₀ is 64, the convolution kernel of the convolution layer C ₃ is 3×3, and the step size is 2, and the convolution layer C The convolution kernel of ₄ is 3×3, and the step size is 1; Step 3: Train the decision-making model for safe driving of commercial vehicles First, train the decision-making subnetwork based on generative confrontational imitation learning; the goal of the generative confrontational imitation learning subnetwork is to learn a generator network, so that the discriminator cannot distinguish between the driving actions generated by the generator and the actions in the driving behavior data set; the specifics include the following Several sub-steps: Sub-step 1: On the driving behavior dataset

, initialize the generator network parameter θ ₀ and the discriminator network parameter ω ₀ ; Sub-step 2: Carry out L iterations to solve, each iteration includes sub-steps 2.1 to 2.2, specifically: Sub-step 2.1: Update the discriminator parameters ω _i →ω _i+1 using the gradient formula described by Equation (13):

where ▽ _ω represents the gradient function of the neural network loss function whose parameter is ω; Sub-step 2.2: Set the reward function

Use the trust region strategy optimization algorithm to update the generator parameters θ _i → θ _i+1 ; First, on the basis of the above network training results, continue to train and build a decision-making sub-network based on DDQN, which specifically includes the following sub-steps: Sub-step 3: Initialize the capacity of experience reuse pool D as N; Sub-step 4: The Q value corresponding to the initialization action is a random value; Sub-step 5: Carry out M iterations to solve, each iteration includes sub-steps 5.1 to 5.2, specifically: Sub-step 5.1: Initialize state s ₀ , initialize strategy parameter φ ₀ ; Sub-step 5.2: Perform T iterations to solve, each iteration includes sub-steps 5.21 to 5.27, specifically: Sub-step 5.21: Randomly select a driving action; Sub-step 5.22: Otherwise select a _t = max _a Q ^* (φ(s _t ),a; θ); In the formula, Q ^* ( ) represents the optimal action-value function, and a _t represents the action at time t; Sub-step 5.23: Execute action a _t to obtain reward value r _{t at time t} and state s _{t+1 at time t+1} ; Sub-step 5.24: Store samples (φ _t , a _t , r _t ,φ _t+1 ) in the experience multiplexing pool D; Sub-step 5.25: Randomly select a small batch of samples (φ _j ,a _j ,r _j ,φ _j+1 ) from the experience multiplexing pool D; Sub-step 5.26: Calculate the iteration target using the following formula:

In the formula,

Indicates the weight of the target network at time t; γ indicates the discount factor; arg max( ) indicates the variable that makes the objective function have the maximum value, y _i indicates the iterative target at time i, and p(s,a) indicates the action distribution; Sub-step 5.27: Use the following formula to perform gradient descent on (y _i -Q(φ _j ,a _j ; θ)) ² :

In the formula,

Represents the gradient function of the neural network loss function with parameter θ _i , ε represents the probability of randomly selecting a behavior under the ε-greedy exploration strategy; θ _i represents the parameter of iteration at time i, L _i (θ _i ) represents the Loss function, Q(s,a; θ _i ) represents the action value function of the target network, a' represents all possible actions in the state s'; After the training of the safe driving decision model of commercial vehicles is completed, the state space information collected by the sensor is input into the safe driving decision model, which can output high-level driving decisions such as steering, straight ahead, acceleration and deceleration in real time, which can effectively guarantee the operation in the urban low-speed environment The vehicle is safe to operate.

Images