CN114707359A - A Decision Planning Method for Autonomous Vehicles Based on Value Distribution Reinforcement Learning - Google Patents

Abstract

The invention relates to an automatic driving automobile decision planning method based on value distribution reinforcement learning, and belongs to the field of automatic driving automobiles. The method comprises the following steps: s1: constructing a non-signal lamp crossroad scene considering uncertainty; s2: constructing a fully parameterized quantile function model as an automatic driving automobile control model; s3: and introducing condition risk value based on the learned state-action return distribution information in the fully parameterized quantile function model, and generating the driving behavior with risk awareness. The method improves the safety and stability of the decision planning strategy of the automatic driving automobile in the uncertain environment by using value distribution reinforcement learning.

Description

A Decision Planning Method for Autonomous Vehicles Based on Value Distribution Reinforcement Learning

technical field

The invention belongs to the field of self-driving cars, and relates to a decision-making planning method for self-driving cars based on value distribution reinforcement learning.

Background technique

Autonomous driving technology has developed rapidly in recent years, but safety has become a key issue faced by autonomous driving technology. Safety is an important factor hindering the commercialization of autonomous vehicles, and it has also been a research hotspot in recent years. The autonomous driving decision planning module, as the "brain" of autonomous vehicles, has a very important impact on the safety of autonomous vehicles, especially in complex urban scenarios such as intersections, how to make autonomous and safe decision-making has been widely used in recent years. Research.

The decision-making and planning module for autonomous vehicles mainly generates optimal driving behaviors based on the current environmental state, so as to safely complete driving tasks. The existing decision-making and planning methods are mainly divided into three categories: rule-based, optimization-based and learning-based. . Among them, the rule-based method is only suitable for specific scenarios; the optimization-based method has poor performance in real-time. Therefore, learning-based methods have been widely studied in academia and industry in recent years, and reinforcement learning has been widely used in decision-making and planning problems of autonomous vehicles. Thanks to the real-time nature and scene adaptability of reinforcement learning, reinforcement learning-based The decision planning method can accomplish the driving task well. However, due to the increasingly complex driving environment faced by autonomous vehicles, incomplete perception caused by inclement weather, building occlusion, etc., as well as the behavioral uncertainty of surrounding traffic participants, it has brought great challenges to the safety of autonomous vehicles. , traditional reinforcement learning algorithms have been unable to meet the safety requirements of autonomous vehicles.

Since traditional reinforcement learning selects the optimal action to maximize the expected value of the reward, the distributional information of the reward is largely lost, so the influence on the decision policy due to the inherent uncertainty in the environment cannot be considered. Therefore, it is urgent to propose a new reinforcement learning algorithm to deal with the uncertainties in the environment to improve the safety of autonomous vehicle decision planning.

SUMMARY OF THE INVENTION

In view of this, the purpose of the present invention is to provide a decision-making planning method for autonomous driving vehicles based on value distribution reinforcement learning, which can improve the safety and stability of decision-making planning strategies of autonomous driving vehicles in uncertain environments.

To achieve the above object, the present invention provides the following technical solutions:

A decision planning method for autonomous vehicles based on value distribution reinforcement learning, which specifically includes the following steps:

S1: Construct a no-signal intersection scene considering uncertainty;

S2: Build a fully parameterized quantile function (FQF) network model as an autonomous vehicle control model;

S3: Based on the state-action reward distribution information learned in a fully parameterized quantile function (FQF) model, a conditional value at risk (CVaR) is introduced to generate risk-aware driving behavior.

Further, in step S1, constructing a scene of an intersection without a signal light that considers uncertainty, which specifically includes: establishing an occlusion model, determining a surrounding vehicle model, and establishing a distribution of surrounding vehicle types.

Further, in step S1, an occlusion model is established, which specifically includes: considering the occlusion on both sides of the intersection, by analyzing the relative positional relationship between the surrounding vehicles and the vehicle and the center of the intersection, according to the geometric relationship, calculate that the surrounding vehicles can be observed by the vehicle. The critical distance d to be used as a critical condition for judging whether the surrounding vehicles are blocked:

为道路边界到遮挡物的距离，d为周围车辆车头至十字路口中心点的距离。Among them, l is the width of each lane, d' is the distance from the front of the vehicle to the center of the intersection,

is the distance from the road boundary to the occluder, and d is the distance from the head of the surrounding vehicles to the center of the intersection.

Further, in step S1, the surrounding vehicle model is determined, which specifically includes: in order to enable the surrounding vehicle to respond to the active change of the environment, it is specified that in the simulation environment, the behavior of the surrounding vehicle is controlled by the intelligent driver model (IntelligentDriver Model):

Among them, a is the acceleration, a _max is the maximum acceleration, v is the longitudinal speed of the vehicle, v _target is the desired longitudinal speed of the vehicle, m is the acceleration parameter, d _target is the desired longitudinal distance of the vehicle, d ₀ is the minimum longitudinal distance of the vehicle, and T ₀ is The minimum collision time of the vehicle, Δv is the relative speed with the preceding vehicle.

Further, in step S1, the distribution of the types of surrounding vehicles is established, which specifically includes: specifying that in the simulation environment, the surrounding vehicles include three types: aggressive, conservative, and normal, and each type of vehicle is used at each time Steps are added to the environment with a certain probability, and the surrounding vehicle type space is:

Further, in step S2, a fully parameterized quantile function model is constructed, which specifically includes the following steps:

S21: Build a Fraction proposal network: take the state information as the network input, and output the optimal quantile τ corresponding to each state-action;

S22: Build a quantile value network: take the optimal quantile point obtained by the quantile proposal network as the input of the quantile value network, and map the quantile function corresponding to each quantile point corresponding to the current state value;

S23: Construct the state space S: The position, speed, heading angle of the surrounding vehicles and the position, speed and heading angle of the vehicle are used as the observable state information of the vehicle, and the value distribution reinforcement learning is based on the observation information of the vehicle to make the next decision. planning;

S24: Build an action space A: The action space is defined as the set of actions that the vehicle can perform, and is the output value of the value distribution reinforcement learning network. Here, the action space of the vehicle includes three discrete action values: acceleration, cruise and deceleration; among which acceleration The specific acceleration of the two actions of deceleration and deceleration is calculated by the Intelligent Driver Model;

S25: Design a reward function, the total reward is equal to the sum of the three parts of the collision reward R _collision , the task completion reward R _success and the timeout reward R _timeout ;

S26: According to the current state S _t , perform the action A _t , and add the training data (S _t , A _t , R _t , S _t+1 ) obtained after the self-vehicle performs the action to the experience pool;

S27: Fit the return distribution;

S28: Update the quantile proposal network: by minimizing the 1-Wasserstein distance, update the quantile proposal network to determine the optimal quantile τ, so that the fitted distribution is closer to the true distribution;

S29: Update the quantile numerical network: The update goal of the quantile numerical network is to minimize the quantile regression Huber-loss, make the output of the quantile numerical network as close to the target value as possible, and update the quantile numerical network by gradient descent.

Further, step S27 specifically includes: fitting the distribution of rewards through the weighted values of N mixed Dirac functions:

Among them, N is the number of quantiles, τ _i is the quantiles generated by the quantile proposal network, satisfying τ _i-1 <τ _i , and τ ₀ =0, τ _N =1, δ _θi(s,a) is the Dirac function of the parameter θ _i in the current state (s, a).

Further, step S28 specifically includes the following steps:

S281: The 1-Wasserstein distance formula is:

为分位点

对应的分位数函数值，

Among them, N is the number of quantiles, ω is the neural network parameter,

quantile

the corresponding quantile function value,

实际上是无法得到的，因此利用带有分位数网络参数ω₂的分位数值函数

作为当前状态下真实的分位数值函数；S282: Due to the true quantile function

is not practically available, so use the quantile numerical function with the quantile network parameter ω ₂

As the real quantile value function in the current state;

S283: To avoid calculating the 1-Wasserstein distance directly, use gradient descent to minimize the 1-Wasserstein distance by applying gradient descent to the parameter ω ₁ of the quantile proposal network:

S284: The return expectation of the fully parameterized quantile function is:

Further, step S29 specifically includes the following steps:

S291: Solve the time difference equation:

Among them, δ _ij is TD-error, r _t is the return at the current moment, γ is the decay factor, Z is the return distribution at the current moment, and Z′ is the return distribution at the next moment;

S292: Calculate quantile regression Huber-loss:

为分位数回归Huber-loss，

为Huber-loss函数，κ为阈值；in,

Regression Huber-loss for quantile,

is the Huber-loss function, and κ is the threshold;

S293: Use stochastic gradient descent to update the quantile value network:

为t时刻的TD-error。in,

is the TD-error at time t.

Further, step S3 specifically includes the following steps:

S31: Based on the return distribution information obtained in the fully parameterized quantile function (FQF) model in step S2, calculate the conditional value at risk (CVaR) corresponding to each distribution as:

Z为回报的分布，α为累积概率，R为回报，是一个随机变量；Among them, the value at risk (VaR):

Z is the distribution of returns, α is the cumulative probability, and R is the return, which is a random variable;

S32: Choose the optimal action, with the goal of maximizing the CVaR value, and choose the optimal risk-sensitive behavior:

为当前状态s_t下所选择的最优动作，Z为回报的分布，α为累积概率。in,

is the optimal action selected under the current state s _t , Z is the distribution of rewards, and α is the cumulative probability.

The beneficial effects of the present invention are:

1) The present invention designs a simulation training environment at an intersection without a signal light, taking into account the incomplete perception caused by the occlusion in the environment and the behavioral uncertainty of the surrounding traffic participants, so that the scene is more in line with the real driving scene.

2) The present invention designs a decision planning method based on value distribution reinforcement learning, which adopts a more accurate fitting value distribution with a fully parameterized quantile function (FQF), and provides a more accurate distribution for subsequent decision-making behaviors with risk awareness. distribution information.

3) The present invention designs a behavior generation method based on Conditional Value at Risk (CVaR), and generates a risk-conscious driving behavior based on the obtained return distribution information and considering the uncertainty existing in the environment.

Other advantages, objects, and features of the present invention will be set forth in the description that follows, and will be apparent to those skilled in the art based on a study of the following, to the extent that is taught in the practice of the present invention. The objectives and other advantages of the present invention may be realized and attained by the following description.

Description of drawings

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be preferably described in detail below with reference to the accompanying drawings, wherein:

Fig. 1 is the overall logical frame diagram of the decision-making planning method of autonomous driving vehicle based on value distribution reinforcement learning of the present invention;

Fig. 2 is the logical frame diagram of constructing simulation training environment;

Figure 3 is a fully parameterized quantile function (FQF) network structure diagram.

Detailed ways

The embodiments of the present invention are described below through specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the drawings provided in the following embodiments are only used to illustrate the basic idea of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

Referring to FIGS. 1 to 3 , the present invention provides a decision-making planning method for autonomous driving vehicles based on value distribution reinforcement learning. Considering the uncertainty in the real driving environment, a simulation training environment at intersections without signal lights is established that considers both occlusion and different types of drivers. At the same time, considering the safety requirements of autonomous vehicle decision-making planning, a method based on value distribution reinforcement learning is proposed. The full parameterized quantile function (FQF) is used to fit the true distribution of returns, and then the conditional value at risk is calculated. (CVaR) introduces the obtained distribution information to generate risk-aware driving behaviors and improve the autonomous vehicle's ability to handle uncertainties in the environment. The method specifically includes the following steps:

Step S1: Construct a simulation training scene of an intersection without signal lights, as shown in Figure 2, which specifically includes the following steps:

S11: Establish an occlusion model: considering the occlusion on both sides of the intersection, by analyzing the relative positional relationship between the surrounding vehicles and the vehicle and the center of the intersection, according to the geometric relationship, calculate the critical distance d that the surrounding vehicles can be observed by the vehicle, as This is a critical condition for judging whether the surrounding vehicles are blocked:

为道路边界到遮挡物的距离，d为周围车辆车头至十字路口中心点的距离。Among them, l is the width of each lane, d' is the distance from the front of the vehicle to the center of the intersection,

is the distance from the road boundary to the occluder, and d is the distance from the head of the surrounding vehicles to the center of the intersection.

S12: Determine the surrounding vehicle model: In order to enable the surrounding vehicles to respond to changes in the environment, it is specified that in the simulation environment, the behavior of surrounding vehicles is controlled by the Intelligent Driver Model:

Among them, a is the acceleration, a _max is the maximum acceleration, v is the longitudinal speed of the vehicle, v _target is the desired longitudinal speed of the vehicle, m is the acceleration parameter, d _target is the desired longitudinal distance of the vehicle, d ₀ is the minimum longitudinal distance of the vehicle, and T ₀ is The minimum collision time of the vehicle, Δv is the relative speed with the preceding vehicle.

S13: Establish the distribution of surrounding vehicle types: In order to enable the vehicle to make different decisions according to different driver types, it is specified that in the simulation environment, surrounding vehicles include three types: Aggressive, Conservative, and Normal. Each type of vehicle is added to the environment at each time step with the probability: P _aggressive = 0.2, P _conservative = 0.3, P _normal = 0.5, and the surrounding vehicle type space is:

S14: Initialize the environment: randomly initialize the initial speed, position and target speed of the surrounding vehicles.

S2: Build a fully parameterized quantile function (FQF) model as an autonomous vehicle control model, as shown in Figure 3, which includes the following steps:

S21: Build a Fraction proposal network: take the state information as the network input, and output the optimal fractional point τ corresponding to each state-action.

S22: Build a quantile value network: take the optimal quantile point obtained by the quantile proposal network as the input of the quantile value network, and map the quantile function corresponding to each quantile point corresponding to the current state value.

S23: Construct a state space S: Take the position, speed, heading angle of the surrounding vehicles and the position, speed and heading angle of the vehicle as the state information that can be observed by the vehicle, and the value distribution reinforcement learning makes the next decision based on the observation information of the vehicle planning.

代表车辆的航向角。Among them, i=0 represents the self-vehicle, i∈[1,N] represents the surrounding vehicles, x _i , y _i represent the lateral and longitudinal positions of the vehicle, v _xi , v _yi represent the lateral and longitudinal speeds of the vehicle,

Represents the heading angle of the vehicle.

S24: Constructing action space A: Action space is defined as the set of actions that the vehicle can perform, and is the output value of the value distribution reinforcement learning network. Here, the action space of the vehicle includes acceleration, cruise, and deceleration, of which acceleration and deceleration are two actions. The specific acceleration of is calculated by the Intelligent Driver Model:

Among them, a is the acceleration, a _max is the maximum acceleration, v is the longitudinal speed of the vehicle, v _target is the desired longitudinal speed of the vehicle, m is the acceleration parameter, d _target is the desired longitudinal distance of the vehicle, d0 is the minimum longitudinal distance of the vehicle, and T ₀ is the vehicle longitudinal distance. The minimum collision time, Δv is the relative speed with the preceding vehicle, and the acceleration range is: a∈[-3,1]m ² / _s .

S25: Design reward function: The reward function mainly considers the sum of three parts: safety R _collision , success rate R _success and efficiency R _timeout , namely:

R=R _collision +R _success +R _timeout

The first item, R _collision , is a collision reward, which requires that the ego vehicle cannot collide with vehicles in the surrounding environment;

R _collision = -10

The second R _success is the reward for completing the task, requiring the vehicle to reach the target location without collision;

R _success = 10

The third item, R _timeout , is a timeout reward, which requires that the ego vehicle cannot exceed the specified maximum number of steps in the round.

R _timeout = -10

S26: According to the current state S _t , perform the action A _t , and add the training data (S _t , A _t , R _t , S _t+1 ) obtained after the self-vehicle performs the action to the experience pool.

S27: Fit the distribution of returns: By weighting N mixed Dirac functions, fit the distribution of returns:

为当前状态(s，a)下参数θ_i的Dirac函数。where N is the number of quantiles, τ _i is the quantile generated by the quantile proposal network, satisfying τ _i-1 <τ _i , and τ ₀ =0, τ _N =1, and

is the Dirac function of the parameter θ _i in the current state (s, a).

S28: Update the quantile proposal network: By minimizing the 1-Wasserstein distance, update the quantile proposal network to determine the optimal quantile τ, so that the fitted distribution is closer to the true distribution. The specific operations are as follows:

S281: The 1-Wasserstein distance formula is:

为分位点

对应的分位函数值，

Among them, N is the number of quantiles, ω is the neural network parameter,

quantile

The corresponding quantile function value,

实际上是无法得到的，因此利用带有分位数网络参数ω₂的分位数值函数

作为当前状态下真实的分位数值函数。S282: Due to the true quantile function

is not practically available, so use the quantile numerical function with the quantile network parameter ω ₂

as the true quantile value function in the current state.

S283: To avoid calculating the 1-Wasserstein distance directly, use gradient descent to minimize the 1-Wasserstein distance by applying gradient descent to the parameter ω ₁ of the quantile proposal network:

为分位点τ_i对应的分位函数值，

ω₂为分位数值网络参数。in,

is the quantile function value corresponding to the quantile τ _i ,

ω ₂ is the quantile value network parameter.

S284: The return expectation of the fully parameterized quantile function is:

为分位点τ_i对应的分位函数值，

ω₂为分位数值网络参数。where N is the number of quantiles,

is the quantile function value corresponding to the quantile τ _i ,

ω ₂ is the quantile value network parameter.

S29: Update the quantile numerical network: The update goal of the quantile numerical network is to minimize the quantile regression Huber-loss, so that the output of the quantile numerical network is as close to the target value as possible. After the loss function is obtained, the gradient descent method is used. To update the quantile value network, the specific operations are as follows:

S291: Solve the time difference equation:

为分位点τ_i对应的分位函数值，

Z为当前时刻的回报分布，Z′为下一时刻的回报分布。Among them, r _t is the return at the current moment, γ is the decay factor, ω ₁ is the neural network network parameter,

is the quantile function value corresponding to the quantile τ _i ,

Z is the reward distribution at the current moment, and Z′ is the reward distribution at the next moment.

S292: Calculate quantile regression Huber-loss:

δ_ij为TD-error，κ为阈值。Among them, Huber-loss:

_δij is the TD-error, and κ is the threshold.

S293: Use stochastic gradient descent to update the quantile value network:

为分位数回归Huber-loss，

为t时刻的TD-error，κ为阈值，τ_i为分位点，

where N is the number of quantiles,

Regression Huber-loss for quantile,

is the TD-error at time t, κ is the threshold, τ _i is the quantile,

S3: Based on the reward distribution obtained in step S2, a conditional value at risk (CVaR) is introduced to generate a driving behavior with risk awareness, which specifically includes the following steps:

S31: Based on the return distribution information obtained in step S2, calculate the conditional value at risk (CVaR) corresponding to each distribution:

Z为回报的分布，α为累积概率，R为回报，是一个随机变量。Among them, the value at risk (VaR):

Z is the distribution of returns, α is the cumulative probability, and R is the return, which is a random variable.

S32: Choose the optimal action: with the goal of maximizing the CVaR value, choose the optimal risk-sensitive action:

为当前状态s_t下所选择的最优动作，Z为回报的分布，α为累积概率。in,

is the optimal action selected under the current state s _t , Z is the distribution of rewards, and α is the cumulative probability.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be Modifications or equivalent replacements, without departing from the spirit and scope of the technical solution, should all be included in the scope of the claims of the present invention.

Claims (9)

1. An automatic driving automobile decision planning method based on value distribution reinforcement learning is characterized by comprising the following steps:

s1: constructing a signal lamp-free crossroad scene considering uncertainty;

s2: constructing a fully parameterized quantile function model as an automatic driving automobile control model;

s3: and introducing condition risk value based on the state-action return distribution information learned in the fully parameterized quantile function model to generate driving behaviors with risk awareness.

2. The decision planning method for the automatic driving vehicle according to claim 1, wherein in step S1, constructing a signal-free intersection scene considering uncertainty specifically comprises: and establishing an occlusion model, determining a surrounding vehicle model, and establishing the type distribution of surrounding vehicles.

3. The decision planning method for an autonomous vehicle according to claim 2, wherein in step S1, the establishing of the occlusion model specifically includes: considering the occlusion on two sides of the intersection, calculating the critical distance d that the surrounding vehicles can be observed by the vehicle according to the geometric relationship by analyzing the relative position relationship between the surrounding vehicles and the vehicle and the center of the intersection, and taking the critical distance d as a critical condition for judging whether the surrounding vehicles are occluded:

wherein l is the width of each lane, d' is the distance from the head of the vehicle to the central point of the crossroad,

the distance from the road boundary to the shelter and the distance from the head of the surrounding vehicle to the central point of the crossroad are respectively d.

4. The automated driving vehicle decision planning method according to claim 2, wherein the determining the surrounding vehicle model in step S1 specifically comprises: the behavior of the surrounding vehicles is controlled by an intelligent driver model:

wherein a is acceleration, a_maxMaximum acceleration, v vehicle longitudinal speed, v_targetFor the desired longitudinal speed of the vehicle, m is an acceleration parameter, d_targetDesired distance in the longitudinal direction of the vehicle, d₀For minimum longitudinal distance, T, of the vehicle₀Δ v is the relative speed with the preceding vehicle for the vehicle minimum time to collision.

5. The decision planning method for the automatic driving vehicle of claim 1, wherein in step S2, a fully parameterized quantile function model is constructed, which specifically includes the following steps:

s21: constructing a quantile proposing network: taking the state information as network input, and outputting the optimal quantile point tau corresponding to each state-action;

s22: constructing a quantile numerical network: using the optimal quantile point obtained by the quantile proposing network as the input of the quantile numerical value network, and mapping to obtain the corresponding quantile function value of each quantile point under the corresponding current state;

s23: constructing a state space S: taking the position, the speed and the course angle of the surrounding vehicles and the position, the speed and the course angle of the vehicle as observable state information of the vehicle, and carrying out next decision planning on the basis of observed information of the vehicle by value distribution reinforcement learning;

s24: constructing an action space A: the action space is defined as a set of executable actions of the self vehicle and is an output value of the value distribution reinforcement learning network, wherein the action space of the self vehicle comprises three discrete action values of acceleration, cruising and deceleration; the specific acceleration of the acceleration action and the deceleration action is calculated by an intelligent driver model;

s25: designing a reward function with a total reward equal to the collision reward R_collisionReward R for completing a task_successAnd an overtime reward R_timeoutThe sum of the three parts;

s26: according to the current state S_tPerforming action A_tTraining data (S) obtained after the vehicle performs the action_t,A_t,R_t,S_t+1) Adding to an experience pool;

s27: fitting a return distribution;

s28: updating the quantile proposal network: updating the quantile proposal network by minimizing the 1-Wasserstein distance to determine the optimal quantile point tau, so that the fitted distribution is closer to the real distribution;

s29: updating the quantile numerical network: the updating target of the quantile numerical network is to minimize the quantile regression Huber-loss, make the output of the quantile numerical network approach the target value as much as possible, and update the quantile numerical network by a gradient descent method.

6. The decision-making planning method for an autonomous vehicle according to claim 5, characterized in that step S27 specifically comprises: fit the distribution of the returns by the weighted values of the N hybrid Dirac functions:

wherein N is the number of sub-sites, τ_iProposing network-generated quantiles for quantiles satisfying tau_i-1<τ_iAnd τ is₀＝0，τ_N＝1，

Is a parameter theta under the current state (s, a)_iThe Dirac function of (1).

7. The automated driving vehicle decision planning method according to claim 6, wherein step S28 specifically comprises the steps of:

s281: the 1-Wasserstein distance formula is:

wherein N is the number of the sub-sites, omega is the neural network parameter,

is a quantile

The value of the corresponding quantile function,

s282: using network parameters omega with quantiles₂Fractional numerical function of

As a function of the true fractional value in the current state;

s283: proposing a parameter omega of a network by dividing digits₁Gradient descent was used to minimize the 1-Wasserstein distance:

s284: the return expectation for a fully parameterized quantile function is:

8. the autopilot decision-making method of claim 7 wherein step S29 specifically includes the steps of:

s291: solving a time difference equation:

wherein, delta_ijIs TD-error, r_tThe current time is the return, gamma is an attenuation factor, Z is the return distribution of the current time, and Z' is the return distribution of the next time;

s292: calculating quantile regression Huber-loss:

wherein,

for quantile regression Huber-loss,

is a Huber-loss function, k is a threshold value;

s293: updating the quantile numerical network by using random gradient descent:

wherein,

and is TD-error at time t.

9. The decision-making planning method for an autonomous vehicle according to claim 1, characterized in that step S3 specifically comprises the following steps:

s31: based on the return distribution information obtained by the fully parameterized quantile function model in step S2, calculating conditional risk values (CVaR) corresponding to the distributions as:

among them, the value of risk

Z is the distribution of the return, alpha is the cumulative probability, and R is the return;

s32: selecting an optimal action, targeting the maximum CVaR value, selecting an optimal risk-sensitive behavior:

wherein,

is the current state s_tAnd (4) selecting the optimal action, wherein Z is the distribution of the return, and alpha is the cumulative probability.

Images