CN114701517A - Multi-target complex traffic scene automatic driving solution based on reinforcement learning - Google Patents

Abstract

The invention discloses an automatic driving solution under a multi-target complex traffic scene based on reinforcement learning, which can process all traffic scenes by using a set of reinforcement learning automatic driving modeling method and has better universality. The reinforcement learning comprehensive modeling is based on a traditional reinforcement learning framework, and environment perception information and characteristic quantity extracted by combining human knowledge are used as observation space. Model training is based on a time-varying training strategy, and training speed and generalization of strategy application are improved. In order to further guarantee the form safety, a dangerous action recognizer based on a long-time memory (LSTM) network and a rule restraint device based on a human knowledge body are also provided, the dangerous action recognizer is sampled and trained from the environment, so that the vehicle has the capacity of recognizing dangerous actions and dangerous scenes, the rule restraint is designed according to specific situations to limit output actions, the safety can be greatly improved, the collision frequency is reduced, and the driving safety of the vehicle is guaranteed.

Description

A Reinforcement Learning-Based Solution for Autonomous Driving in Multi-objective Complex Traffic Scenarios

technical field

The invention relates to an automatic driving solution method based on reinforcement learning in multi-objective complex traffic scenarios, belonging to the technical field of automatic driving, and in particular to a general automatic driving algorithm modeling and training scheme based on deep reinforcement learning in multi-objective complex traffic scenarios.

Background technique

With the rapid development of the smart car industry and the continuous maturity of autonomous driving technology, driverless technology has become the trend of future vehicle development. The current automatic driving system on the ground can reach the L3 level, that is, the vehicle can drive automatically under the condition of driver monitoring, and the driver needs to take over the vehicle in case of emergency. However, one of the important reasons why autonomous driving has not yet achieved complete unmanned driving is that the current rule-based decision-making method cannot handle enough traffic scenarios, and there are great hidden dangers in safety.

The mainstream autonomous driving technology can be divided into three modules: perception, decision-making and control, of which the decision-making module is the core part of the intelligent system. At present, autonomous driving decision-making technology can be mainly divided into two categories: rule-based and learning-based. Rule-based decision-making techniques include general decision-making models, finite state machine models, decision tree models, and knowledge-based reasoning models. Learning-based decision-making techniques are mainly based on deep learning and reinforcement learning.

Today, rules-based decision-making techniques are still in practice, but such techniques expose more and more problems. It is difficult for a rule-based system to enumerate all possible scenarios, and some scenarios may easily lead to traffic accidents. Secondly, the design of the rule system is high in labor cost and system complexity, and system maintenance and upgrades are particularly cumbersome. Therefore, people are eager to develop and improve other technical methods, and data-driven deep reinforcement learning is one direction. However, the application of deep reinforcement learning in autonomous driving today is mainly in a specific scenario, such as overtaking, lane changing, lane keeping, etc., and its versatility is not strong. In addition, deep neural networks currently do not have complete interpretability, there is a problem of catastrophic forgetting, and it is easy to generate some unknown unsafe actions. On the other hand, reinforcement learning itself also has generalization and stability problems. Therefore, in order to make deep reinforcement learning have practical application in automatic driving decision control, it is possible to improve and solve the problems of poor generality, weak generalization, and unimproved safety to a certain extent.

SUMMARY OF THE INVENTION

The content of this application is intended to introduce concepts in a simplified form that are described in detail in the detailed description that follows. The content section of this application is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to be used to limit the scope of the claimed technical solution.

Aiming at the problems and deficiencies in the prior art, the purpose of the present invention is to provide a reinforcement learning-based solution for automatic driving in multi-objective complex traffic scenarios. Through anthropomorphic observation design and reward function design based on reward reshaping, the Only one model can show better comprehensive performance in multiple types of traffic scenarios with multiple policy environments. In addition, in order to improve the training speed and generalization, the present invention proposes a reward time-varying training method. In order to enhance security, the present invention proposes methods for ensuring security, such as LSTM-based dangerous action recognizer, knowledge-based security filtering, etc., to solve the problems raised in the above background art.

To achieve the above object, the present invention provides the following technical solutions:

The invention discloses an automatic driving solution method based on reinforcement learning in multi-objective complex traffic scenarios, comprising the following steps:

Step 1, prepare a simulator environment and complex driving scenarios for automatic driving simulation;

Step 2: Add the environmental feature information required for training the reinforcement learning model in the observation space as the environmental observation information, including the own vehicle information, other vehicle information and road information, and calculate the key feature information according to the environmental feature information, including each lane. The collision time to the preceding vehicle, the collision time difference between the vehicle and the collision point, and the variance of the heading angle between the vehicle and the road point ahead;

Step 3, setting the reward framework required for the reinforcement learning model training;

Step 4: Train the reinforcement learning model based on the time-varying training method, and then use the meta-model to continue training in different traffic scenarios, and modify the training time according to the driving performance of the intelligent body and the type of collision every certain number of iterations. Reward the weights, and modify the weights repeatedly to end the training;

Step 5: After outputting the training-completed reinforcement learning model, construct a dangerous action recognizer and a rule constraint, and judge the degree of danger of the scene according to the environmental observation information, so as to limit or adjust the planning amount output by the reinforcement learning model, and Constantly manually adding and refining rules by observing the effects in the simulation environment.

Further, the environmental feature information required for training the reinforcement learning model is added to the observation space in step 2, and the self-vehicle information includes the self-vehicle speed, the steering wheel angle of the self-vehicle, the lane number of the self-vehicle, the center point and the location of the self-vehicle. The distance from the center line of the lane, the deviation of the 15 waypoints ahead from the position of the own vehicle and the heading angle of the own vehicle, and the relative position of the preview point at a distance of 4/8 meters; the other vehicle information includes the distance of the adjacent vehicles in each lane , the collision time of each lane from the preceding vehicle, the relative speed of the nearest neighbor car and the own vehicle in the own vehicle lane; the road information includes the lateral distance from the own vehicle to the center of the lane, and the waypoint orientation error.

Further, the reward framework described in step 3 includes environmental reward, speed reward, collision penalty and lane center deviation penalty; wherein, the environmental reward is the survival time of the self-vehicle, which represents the length of time that the self-vehicle travels from the starting point to the collision. ; The speed reward is the driving speed of the self-vehicle, and the unit is the distance traveled per second; the collision penalty is that when the self-vehicle leaves the route, the self-vehicle runs out of the road boundary, or the self-vehicle collides with the environmental vehicle, corresponding The collision penalty of ; the lane center deviation penalty is the absolute value of the distance between the vehicle center and the center line.

Further, in step 4, the reinforcement learning model is trained based on the time-varying training method combined with the meta-model, and the specific steps are:

Step 4.1, initialize the reinforcement learning model, and use each scene to train a certain number of rounds in turn to obtain a meta-model;

Step 4.2, use the time-varying training method to train the meta-model obtained in step 4.1 in the selected scene, and adjust the reward weight according to the defects in the behavior of the agent;

Step 4.3, set the scene to all simple scenes without intersections, and repeat the training process in step 4.2 to improve the performance in simple scenes without intersections;

Step 4.4, set the scene to all scenes with intersections, and repeat the training process in step 4.2 to improve the performance in the intersection scene;

Step 4.5, set the scene to include a roundabout and multi-directional vehicles, and repeat the training in step 4.2 to improve the performance in the roundabout and multi-directional vehicle scenarios;

Step 4.6, continue training in the remaining scenarios until the end of the process.

Further, the reinforcement learning model is trained based on the time-varying training method described in step 4, and the specific steps are:

Step 4.2.1, setting the hyperparameters of the reinforcement learning model;

Step 4.2.2, set the reward function as the basic reward, so that the Agent can learn lane keeping and start iterative training;

Step 4.2.3, increase the weight of lane center deviation penalty and collision penalty, and continue iterative training;

Step 4.2.4, continue to increase the collision penalty, and continue the iterative training;

Step 4.2.5, add a new scene based on the original scene data set, add a speed reward, and increase the weight of the lane center deviation penalty and collision penalty until the iteration ends.

Further, in step 5, the dangerous action recognizer predicts the degree of danger according to the actions outputted by the reinforcement learning model and the environmental observation information, and takes actions such as emergency avoidance and emergency adjustment according to the degree of danger. The sample stage and the training stage, the specific steps of the sample collection stage are:

Step 5.1.1, prepare multiple types of scenarios, select a scenario to start training;

Step 5.1.2, initialize the PPO strategy model, and start training the strategy model in the selected scenario;

Step 5.1.3, record the trajectory of the current round of operation during the operation;

Step 5.1.4, when a collision occurs, collect the first 10 steps of the collision as negative samples, and randomly collect any 10 consecutive steps in the current trajectory as positive samples;

Step 5.1.5, after training until the number of running steps reaches the set total number of steps, select the next scene to repeat the training;

Step 5.1.6, until all the scenes are collected.

Further, the dangerous action recognizer described in step 5 is a dangerous action recognizer model constructed based on a long-short memory network, and the specific steps in the training phase are:

Step 5.2.1, according to each group of sample data collected, use the sliding window to generate the data group as the model input, and use the label of the last step of each group as the target label of the model;

Step 5.2.2, use the Adam optimizer and adjust the optimizer learning rate based on the cosine simulated annealing method;

Step 5.2.3, using the mean square error loss as the training loss function, calculate the mean square error loss between the model output and the target label;

Step 5.2.4, set the relevant model parameters and the number of training model rounds to complete the training.

Further, in step 5, the rule constraint device is to write rules based on human knowledge and empirical statistics of simulation experiments, and is used to limit the behavior of the self-vehicle in some specific situations, and the rule constraint device mainly includes the knowledge rules of shortest distance protection, The knowledge rules under the intersection, the knowledge rules before sharp bends, and the correction rules for long-term residence when the vehicle has no neighbors, the environmental observation information respectively judges different scenarios and determines the speed limit rules for the vehicle.

Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention provides an automatic driving solution method based on reinforcement learning in multi-objective complex traffic scenarios, which can use a set of reinforcement learning automatic driving modeling methods to process all Traffic scene, has good generality, and can obtain good multi-objective performance and generalization performance. The comprehensive modeling of reinforcement learning is based on the traditional reinforcement learning framework, using the environmental perception information and the feature quantities extracted by combining human knowledge as the observation space, and setting the lane line keeping, driving distance and collision avoidance according to the evaluation indicators as the intelligent vehicle in the reinforcement learning algorithm rewards and punishments. When the model is trained, the meta-learning idea is combined with the time-varying training strategy. Different reward weights and different training sets are set in each stage to strengthen some of the behavioral defects formed by the agent in the previous training stage, and improve the agent's performance in some parts. The performance in weak scenarios can improve the training speed and generalization of policy application. In addition, in order to further ensure its formal security, a dangerous action recognizer based on long short-term memory (LSTM) network and a rule constraint based on human knowledge are also proposed. Vehicles have the ability to identify dangerous actions and dangerous scenes, and design rules and constraints for specific situations to limit output actions, which can greatly improve safety, reduce the number of collisions, and handle special emergencies to ensure vehicle driving safety.

Description of drawings

The accompanying drawings, which constitute a part of this application, are used to provide a further understanding of the application and make other features, objects and advantages of the application more apparent. The accompanying drawings and descriptions of the exemplary embodiments of the present application are used to explain the present application, and do not constitute an improper limitation of the present application.

In the attached image:

FIG. 1 is a schematic flowchart of an overall automatic driving solution method for reinforcement learning in multi-objective complex traffic scenarios according to the present invention;

2 is a schematic diagram of the steps of the reinforcement learning multi-objective and complex traffic scene automatic driving solution method according to the present invention;

3 is a schematic flowchart of the time-varying training method of reinforcement learning and the meta-model training reinforcement learning model of the present invention;

4 is a schematic diagram of the framework of the present invention for controlling automatic driving by combining end-to-end and rule constraints;

5 is a schematic diagram of a reinforcement learning strategy network based on near-end strategy optimization according to the present invention;

6 is a schematic structural diagram of a dangerous action recognizer model based on a long short-term memory (LSTM) neural network of the present invention;

FIG. 7 is a schematic flowchart of the sampling stage of the dangerous action identifier according to the present invention;

Figure 8 is a computer language block diagram of the dangerous action recognizer sampling algorithm of the present invention.

Detailed ways

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided for a thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes, and are not intended to limit the protection scope of the present disclosure.

In addition, it should be noted that, for the convenience of description, only the parts related to the related invention are shown in the drawings. The embodiments of this disclosure and features of the embodiments may be combined with each other without conflict.

The present invention discloses an automatic driving solution method based on reinforcement learning in multi-objective complex traffic scenarios. The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments. 1 to 2, it mainly includes the following steps:

Step 1, prepare a simulator environment and complex driving scenarios for automatic driving simulation;

Step 2: Add the environmental feature information required for training the reinforcement learning model in the observation space as the environmental observation information, including the own vehicle information, other vehicle information and road information, and calculate the key feature information according to the environmental feature information, including each lane. The collision time to the preceding vehicle, the collision time difference between the vehicle and the collision point, and the variance of the heading angle between the vehicle and the road point ahead;

Step 3, setting the reward framework required for the reinforcement learning model training;

Step 4: Train the reinforcement learning model based on the time-varying training method, and then use the meta-model to continue training in different traffic scenarios, and modify the training time according to the driving performance of the intelligent body and the type of collision every certain number of iterations. Reward the weights, and modify the weights repeatedly to end the training;

Step 5: After outputting the training-completed reinforcement learning model, construct a dangerous action recognizer and a rule constraint, and judge the degree of danger of the scene according to the environmental observation information, so as to limit or adjust the planning amount output by the reinforcement learning model, and Constantly manually adding and refining rules by observing the effects in the simulation environment.

The multi-objectives of the present invention refer to the goals of accurate vehicle driving, fast driving speed, driving safety, high algorithm robustness and high generalization. Specifically, in a single traffic scenario, when the same number of simulations are required under the condition that the vehicle travels the maximum distance, the driving speed is reflected in the average speed of the self-vehicle as large as possible. Driving safety is reflected in the collision between the self-vehicle and the environmental vehicle, the number of deviations of the self-vehicle from the route as few as possible, and the driving distance of the self-vehicle as far as possible. Driving accuracy is reflected in the average distance between the center point of the vehicle and the center line of the road as close as possible. The high robustness and generalization of the algorithm is reflected in the fact that an algorithm can achieve good results in multiple complex traffic scenes and untrained map scenes.

The complex traffic scene refers to the map scene used for training in the simulator with various road types and traffic conditions. The map types used include simple, sharp bends, intersections, roundabouts, convergence, diversion, and mixed road scenes. There are a total of nine map types, and different scenes also contain environment vehicles with different degrees of density. In the rules set by the simulated environment, the driving trajectory and driving strategy of the environmental vehicle have certain randomness, and its strategy types can be divided into three types: conservative, moderate and aggressive. The operating strategy parameters still have a certain randomness.

In step 2, the environmental feature information required for training the reinforcement learning model is added to the observation space as the environmental observation information, and the key feature quantity is calculated according to the environmental feature information. In the reinforcement learning model here, the input observation space includes environmental perception information such as self-vehicle information, other vehicle information, and road information, as well as observation features extracted from environmental information; its output actions include accelerator opening, brake control, steering wheel Corner control.

Specifically, the environment observation information includes environment perception information such as own vehicle information, other vehicle information, and road information. Among them, the information of the own vehicle includes the speed of the own vehicle, the steering wheel angle of the own vehicle, the serial number of the lane where the vehicle is located, the distance between the center point of the vehicle and the center line of the lane where the vehicle is located, and the deviation of the 15 waypoints ahead from the location of the vehicle and the heading angle of the vehicle. , The relative position of the preview point at a distance of 4/8 meters, a total of 2+1+1+15+4=19 dimensions. The other car information includes the distance of the neighboring car in each lane, the collision time (TTC) between each lane and the preceding car, and the relative speed of the nearest neighbor car and the own car in the own lane, a total of 6*3=18 dimensions; When the car drives to the intersection, query the direction of the intersection within 50 meters, that is, the relative position, relative direction, relative speed, and collision time difference of the 5 vehicles closest to the vehicle that may collide, with a total of 5*5=25 dimensions. The road information includes: the lateral distance from the vehicle to the center of the lane, and the waypoint orientation error. The waypoint orientation error represents the distance between the line where the vehicle is heading and the waypoint ahead, and 15 waypoints ahead are selected from the waypoint at the location of the vehicle. Calculate the heading error separately, and take the principle that the waypoints close to the vehicle are dense, and the waypoints farther away are sparse. 21, 25, 30, 35, 42, 50], which represents the i-th waypoint ahead, and also represents the distance between the waypoint and the agent vehicle.

The extracted key feature information is mainly the collision avoidance vehicle information. The three other vehicles closest to the position of the own vehicle are selected as the collision avoidance vehicle, and the other vehicle information is calculated in the own vehicle coordinate system. Therefore, the key feature information includes the relative position of the collision avoidance vehicle, The absolute speed of the collision avoidance vehicle, the relative orientation angle of the ego vehicle and the collision avoidance vehicle, and the time difference to collision (TDTC) from the collision point. The calculation formula of the time difference to collision (TDTC) from the collision point is:

In step 3, the reward framework required for training the reinforcement learning model is set. In the reward framework of reinforcement learning model training, environmental reward, speed reward, collision penalty, lane center deviation penalty are included. Among them, the environmental reward is the survival time of the self-vehicle, which means the time it takes for the self-vehicle to travel from the starting point to the collision. Survival is rewarded. The speed reward is the driving speed of the vehicle, and the unit is the distance traveled per second. The lane center deviation penalty is the absolute value of the distance between the vehicle center and the centerline. There are three types of collision penalties: when the ego car leaves the route, the ego car runs out of the road boundary, or the ego car collides with the environmental car, the corresponding collision penalty will be given, and its value is a constant 5, and its weight increases with the number of iterations. Increase, that is, a penalty of -5 is obtained for each situation, and the coefficients of each penalty are 1, 0.5, 0.1, 1.2. These coefficients will be adjusted during the training process.

Referring to Figure 3, training the reinforcement learning model based on the time-varying training method combined with the meta-model can improve the performance of the vehicle in some special scenes, such as intersections and roundabouts. The training steps are given below:

Step 4.1, initialize the reinforcement learning model, and use each scene to train a certain number of rounds in turn to obtain a meta-model;

Step 4.2, use the time-varying training method to train the meta-model obtained in step 4.1 in the selected scene, which is to adjust the reward weight according to the defects in the behavior of the agent;

Step 4.3, set the scene to all simple scenes without intersections, and repeat the training process in step 4.2 to improve the performance in simple scenes without intersections;

Step 4.4, set the scene to all scenes with intersections, and repeat the training process in step 4.2 to improve the performance in the intersection scene;

Step 4.5, set the scene to include a roundabout and multi-directional vehicles, and repeat the training in step 4.2 to improve the performance in the roundabout and multi-directional vehicle scenarios;

Step 4.6, continue training in the remaining scenarios until the end of the process.

Specifically, in step 4.2, the specific process of using the time-varying training method to train the reinforcement learning model is as follows:

Step 4.2.1, set the reinforcement learning model hyperparameters;

Step 4.2.2, set the reward function as the basic reward, so that the Agent can learn lane keeping and start iterative training;

Step 4.2.3, increase the weight of lane center deviation penalty and collision penalty, and continue iterative training;

Step 4.2.4, continue to increase the collision penalty and continue the iterative training;

Step 4.2.5, add a new scene based on the original scene data set, add a speed reward, and increase the weight of the lane center deviation penalty and collision penalty until the end of the iteration.

The set reinforcement learning model hyperparameters are shown in the following table:

parameter name parameter value learning rate 1e-4 training batch size 10240*3 λ 0.95 Clip parameter 0.2 Number of SGD iterations 10 SGD mini-batch size 1024 Step range 1000

First, set the reward function as a basic function to make the agent learn lane keeping, and express the reward function as: 1.0×environment reward+0.1×lane center deviation penalty+1.2×collision penalty, and iteratively trained for 460 rounds from round 0. Then, the weights of lane center deviation penalty and collision penalty are increased. The reward function at this time is expressed as: 1.0×environmental reward+0.4×lane center deviation penalty+1.6×collision penalty, and the training continues from round 461 to round 768. Continue to increase the collision penalty weight. At this time, the reward function is expressed as: 1.0×environmental reward+0.4×lane center deviation penalty+1.8×collision penalty, and continue training from round 769 to round 1152. Finally, based on the original scene data set, 3 new scenes are added, including 2 all_loop scenes and 1 mix_loop scene, and speed rewards are added (the faster the vehicle travels, the greater the reward value), and the lane center deviation is increased. The weights of penalty and collision penalty are expressed as: 1.0×environmental reward+0.4×speed reward+0.56×lane center deviation penalty+2.9×collision penalty, continue training from round 1153 to round 1400, and the training process ends.

On the basis of traditional reinforcement learning, the reinforcement learning model is trained with time-varying strategies, resulting in an agent model with higher robustness and generalization. The time-varying strategy of reinforcement learning model is a staged learning method, which divides the Agent task objectives into different subtasks according to their importance and learns them in different stages. In each stage, the learning task is trained hierarchically by adjusting the reward weight. , firstly, the agent will be trained to have lane keeping ability, secondly, its collision avoidance ability will be trained, and finally its high-speed driving ability will be trained. The training process of the smart car model consists of multiple iterations. Each iteration adjusts the reward weight of this iteration according to the previous simulation results. Specifically, each iteration records the collisions and collisions that occurred in the previous simulation. Type, which adjusts the proportion of rewards based on the proportion of collisions of a certain type among all collisions.

In step 5, the dangerous action recognizer includes two stages of sample collection and training. Referring to Figures 7 and 8, the specific sample collection process is as follows:

Step 5.1.1, prepare multiple types of scenarios, select a scenario to start training;

Step 5.1.2, initialize the PPO strategy model, and start training the strategy model in the selected scenario;

Step 5.1.3, record the trajectory of the current round of operation during the operation;

Step 5.1.4, when a collision occurs, collect the first 10 steps of the collision as negative samples, and randomly collect any 10 consecutive steps in the current trajectory as positive samples;

Step 5.1.5, after training until the number of running steps reaches the set total number of steps, select the next scene to repeat the training;

Step 5.1.6, until all the scenes are collected.

Specifically, here we use 7 categories of scenes, and select one of the scenes to start training. Initialize the PPO policy model, train the policy from scratch in the selected scenario, and set the total number of steps to run to 400,000 steps. During the running process, the trajectory during the running process of the current round is recorded, including the observation-action pair (s1, a1, s2, a2, ...) and the runtime reward value (r1,, r2, ...) of each step. If there is a collision, record the observations s-action a-reward r 10 steps before the collision as a negative sample to collect, and record the trajectory length of the round as m, and randomly select a random sample from the interval [1, m-19] Count k, and collect the observation s-action a-reward r pair of 10 steps from the kth step as a positive sample. Use the reward value collected in each group of samples to calculate 10 rolling averages as the training labels of the group of samples. The calculation formula is as follows:

where i represents the ith scene and j represents the jth step in the 10-step sample. The observation s-action a-label label pair is stored as a training data set, and the training is performed until the number of running steps reaches the set total number of steps of 400,000. Select the next scene and repeat step 5.1.2 to step 5.1.5 training until all scenes are collected, and the total number of samples collected is about 100,000.

Referring to Figures 5 and 6, a dangerous action recognizer model is constructed based on a long short-term memory (LSTM) network. The model includes one LSTM layer and two fully connected layers. The output of the long short-term memory (LSTM) network passes through the first full connection layer. After the connection layer, the ReLU activation function is used to access the second fully connected layer. Then the dangerous action recognizer training process is as follows:

Step 5.2.1, according to each group of sample data collected, use the sliding window to generate the data group as the model input, and use the label of the last step of each group as the target label of the model;

Step 5.2.2, use the Adam optimizer and adjust the optimizer learning rate based on the cosine simulated annealing method;

Step 5.2.3, using the mean square error loss as the training loss function, calculate the mean square error loss between the model output and the target label;

Step 5.2.4, set the relevant model parameters and the number of training model rounds to complete the training.

Specifically, each set of samples collected contains 10-step observation s-action a-label pairs. The observation is 44-dimensional, and the action is 3-dimensional. After splicing the observation and action into a 47-dimensional vector, the 10-step data is used to generate 6 data groups of length 5 using a sliding window of size 5, and each group of 5-step data is used as Model input, with the label of the last step of each group as the target label of the model. The Adam optimizer is used, and the learning rate of the optimizer is adjusted based on the cosine simulated annealing method, and the mean square error loss is used as the training loss function to calculate the mean square error loss between the model output and the target label. Set the model parameters as shown in the table below, and train the model after 100 rounds.

parameter name parameter value LSTM hidden layer size 128 Fully connected layer 1 128x64 Fully connected layer 2 64x1 initial learning rate 0.01 Number of Cosine Annealing Rounds 100 final learning rate 0.0001 Train-validation set ratio 9:1 Batch size 32

The dangerous action recognizer is used to improve the safety of the vehicle. It predicts the degree of danger according to the action and environmental observation information output by the reinforcement learning model, and takes actions such as emergency avoidance and emergency adjustment according to the degree of danger. The dangerous action recognizer consists of two stages: sample collection stage and training stage. In the sample collection stage, a large number of safe samples and dangerous samples are collected in the process of gradually training the strategy in the simulator environment, and the sample labels are calculated to prepare for the training of the dangerous action recognizer. The samples need to be collected in multiple scenarios and strategies with different effects, and the balance of the number of positive and negative samples should be satisfied. In the training phase, the collected sample data is used for training until the end.

Referring to FIG. 4 , the rule constraint in step 5 is written based on human knowledge and empirical statistics of simulation experiments, and is used to restrict the behavior of the self-vehicle in some specific situations. The rule constraint mainly includes the knowledge rules for the closest distance protection, the knowledge rules under the intersection, the knowledge rules before sharp bends, and the correction rules for long-term residence when the vehicle has no neighbors, and judges different scenarios according to the environmental observation data. , to determine the speed limit rules for your own vehicle. The following is an example of a rule constraint.

In the rule constraint device, the first part is mainly the constraint of knowledge rules based on the protection of the closest distance: first, the three environmental vehicles closest to the self-vehicle are screened out. When the environmental vehicle is less than 20 meters away from the self-vehicle, it enters the TDTC condition judgment. Different TDTC and environmental vehicle speed judge the degree of danger, and then limit the maximum speed of the own vehicle. The specific rules are as follows:

1.-2 < TDTC < 1, d/v _cv < 1, v _cv ≥ 30: The maximum speed of the own vehicle is limited to 5, where d is the distance between the own vehicle and the conflicting vehicle, and v _cv is the speed of the conflicting vehicle;

2.0＜TDTC＜1.2: Limit the maximum speed of the vehicle to 2;

3.-1.2＜TDTC＜0: The maximum speed of the vehicle is limited to 0.1×the planned speed;

4.1.2≤TDTC＜3: The maximum speed of the vehicle is limited to 15+20(TDTC-1.2)/1.8;

5.-1.8＜TDTC≤-1.2, v _cv ＞20: limit the maximum speed of the vehicle to 0.1+1.4(TDTC-1.2)/0.6;

6.-3＜TDTC≤-1.8, v _cv ＞20: The maximum speed of the vehicle is limited to 1.4+6(TDTC-1.8)/1.2;

7.-3＜TDTC≤-1.2, v _cv ≤20: The maximum speed of the vehicle is limited to 5+15(TDTC-1.2)/1.8;

8.3≤TDTC＜7: The maximum speed of the vehicle is limited to 30+20(TDTC-3)/4;

9.-7＜TDTC≤3: The maximum speed of the vehicle is limited to 30+20(TDTC-3)/4;

10. d<4, v _cv > 5: limit the maximum speed of the vehicle to 0;

11. d<5, v _cv > 13: limit the maximum speed of the vehicle to 2;

12. d<7, v _cv > 20: Limit the maximum speed of the ego vehicle to 5.

The second part of the rule constraint is the knowledge rule under the intersection: this rule is used to avoid collisions for vehicles traveling in the side direction when the vehicle passes through the intersection. Whether there is an environmental car, the speed of the self-vehicle is limited in different sizes according to the detection of the environmental car in different size areas. In addition, the size of the rectangular area should also be corrected according to the turning of the own vehicle, and the turning state of the own vehicle can be obtained according to the deviation of the heading angle between the own vehicle and the road point ahead. The specific rules are as follows:

1. There is a non-stationary environmental vehicle within the range of 5.5 in the longitudinal distance in front of the self-vehicle and 10 in the lateral distance on both sides, and the environmental vehicle satisfies one of TDTC < 8 and v _cv < 4: the maximum speed of the self-vehicle is limited to 0;

2. There is a non-stationary environmental vehicle within the range of longitudinal distance 7 in front of the vehicle and 10 lateral distances on both sides, and the environmental vehicle satisfies one of TDTC < 8 and v _cv < 10: the maximum speed of the vehicle is limited to 5;

3. There is a non-stationary environmental vehicle within the range of 9 longitudinal distances in front of the self-vehicle and 10 lateral distances on both sides, and the environmental vehicle satisfies one of TDTC < 8 and v _cv < 12: the maximum speed of the self-vehicle is limited to 7;

4. The variance of the orientation angle between the vehicle and the road point ahead is greater than 0.16, that is, when the vehicle is turning, there is a non-stationary environmental vehicle within the range of the longitudinal distance 9 in front of the vehicle and the lateral distance on both sides of the vehicle, and the environmental vehicle satisfies TDTC<8, v One of _cv < 4: limit the maximum speed of the self-vehicle to 0;

5. The deviation of the orientation angle between the vehicle and the road point ahead is greater than 0.16, that is, when the vehicle is turning, there is a non-stationary environmental vehicle within the range of 10.5 in the longitudinal distance in front of the vehicle and 9 in the lateral distance on both sides, and the environmental vehicle satisfies TDTC<8, v One of _cv < 4: Limit the maximum speed of the ego vehicle to 7.

The third part of the rule constraint is the correction rule for long-term residence in the case of no neighboring car. The rules are:

1. There is no environmental car within 30 of the self-vehicle range, and all conflicting vehicles satisfy v _cv <5, |TDTC|>9: manually accelerate the self-vehicle, and set the minimum value of the throttle control amount to 0.3.

The fourth part of the rule constraint is the knowledge rule before the sharp bend. The speed is limited according to the variance V _h of the orientation angle between the vehicle and the road point ahead. The rules are as follows:

1. V _h > 0.33: limit the maximum speed of the vehicle to 8;

2.0.25＜V _h ≤0.33: The maximum speed of the vehicle is limited to 8+10(0.3-V _h )/0.08;

3.0.2＜V _h ≤0.25: The maximum speed of the vehicle is limited to 14+10(0.25-V _h )/0.05;

4.0.15＜V _h ≤0.2: The maximum speed of the vehicle is limited to 22+20(0.2-V _h )/0.05;

5.0.1＜V _h ≤0.15: The maximum speed of the vehicle is limited to 30+20(0.15-V _h )/0.05;

6.0.03＜V _h ≤0.1: The maximum speed of the vehicle is limited to 40+20(0.1-V _h )/0.07;

7.0.01＜V _h ≤0.03: The maximum speed of the vehicle is limited to 60+20(0.03-V _h )/0.02;

The main function of the above rule constraint is to judge whether the environment is a dangerous situation, and to limit the size of the actions output by reinforcement learning. When the front is safe and the output speed of the reinforcement learning model is too low, auxiliary acceleration is performed to ensure the driving safety and efficiency of the vehicle.

In summary, the present invention provides a solution for safety reinforcement learning in multi-objective scenarios. This technology can be applied to the fields of intelligent vehicle assisted driving, unmanned driving, etc. In comparison, it provides a new idea of hybrid solution, combining the advantages of both to achieve the purpose of high safety, high intelligence and high efficiency of vehicles in complex and various scenarios. Therefore, this technology has a high promotion. value.

The above descriptions are merely some preferred embodiments of the present disclosure and illustrations of the applied technical principles. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solution formed by the specific combination of the above-mentioned technical features, and should also cover, without departing from the above-mentioned inventive concept, the above-mentioned Other technical solutions formed by any combination of technical features or their equivalent features. For example, a technical solution is formed by replacing the above features with the technical features disclosed in the embodiments of the present disclosure (but not limited to) with similar functions.

Claims (8)

1. An automatic driving solution method based on reinforcement learning under a multi-target complex traffic scene is characterized by comprising the following steps:

step 1, preparing a simulator environment and a complex driving scene for automatic driving simulation;

step 2, adding environmental characteristic information required by training a reinforcement learning model into an observation space as environmental observation information, wherein the environmental characteristic information comprises own vehicle information, other vehicle information and road information, and calculating key characteristic information according to the environmental characteristic information, wherein the key characteristic information comprises collision time of each lane from a front vehicle, collision time difference of the own vehicle from a collision point, and orientation angle variance of the own vehicle and the front road point;

step 3, setting a reward frame required by the reinforcement learning model training;

step 4, training the reinforcement learning model based on a time-varying training method, continuing training in different traffic scenes by combining with the use of a meta-model, modifying the rewarding weight during training according to the driving performance condition of the intelligent agent and the type of collision at regular intervals of iteration rounds, and repeatedly modifying the weight for multiple times to finish the training;

and 5, after outputting the trained reinforcement learning model, constructing a dangerous action recognizer and a rule restraint device, judging scene danger degree according to the environment observation information to limit or adjust the planning quantity output by the reinforcement learning model, and continuously and manually adding and optimizing rules by observing the effect in the simulation environment.

2. The solution of automatic driving under the complex traffic scene of multiple targets based on reinforcement learning of claim 1, characterized in that environmental characteristic information required for training a reinforcement learning model is added in the observation space in step 2, and the vehicle information includes the speed of the vehicle, the steering wheel angle of the vehicle, the serial number of the lane where the vehicle is located, the distance between the center point of the vehicle and the center line of the lane where the vehicle is located, the deviation between fifteen road points ahead of the vehicle position and the heading angle of the vehicle, and the relative position of the pre-aiming point at a distance of 4/8 m; the other-vehicle information comprises the distance between the adjacent vehicle in each lane, the collision time between each lane and the previous vehicle, and the relative speed between the nearest other vehicle in the vehicle lane and the vehicle; the road information comprises the transverse distance from the vehicle to the center of the lane and the orientation error of the road point.

3. The solution for automatic driving under the multi-target complex traffic scene based on reinforcement learning as claimed in claim 2, characterized in that: the reward frame in the step 3 comprises an environment reward, a speed reward, a collision penalty and a lane center deviation penalty; wherein the environment reward is the survival time of the vehicle, and represents the time length from the starting point to the collision of the vehicle; the speed reward is the driving speed of the self vehicle, and the driving distance per second is taken as a unit; the collision punishment is that when the self-vehicle drives away from the air route, runs out of the road boundary or collides with the environmental vehicle, the self-vehicle gives corresponding collision punishment; and the lane center deviation penalty is an absolute value of the distance between the vehicle center and the central line.

4. The solution for automatic driving under the complex traffic scenario of multiple targets based on reinforcement learning as claimed in claim 3, wherein the step 4 of training the reinforcement learning model based on the time-varying training method in combination with the meta model comprises the following specific steps:

step 4.1, initializing the reinforcement learning model, and training a certain number of turns by using each scene in sequence to obtain a meta-model;

step 4.2, training the meta-model obtained in the step 4.1 under the selected scene by using a time-varying training method, and adjusting the rewarding weight according to the defects of the behavior of the agent;

4.3, setting the scenes as all simple scenes without intersections, repeating the process training of the step 4.2, and improving the performance under the simple scenes without intersections;

4.4, setting the scenes as all the scenes containing the intersection, repeating the training process of the step 4.2, and improving the performance under the scene of the intersection;

step 4.5, setting a scene containing the rotary island and the multi-direction vehicles, repeating the training process of the step 4.2, and improving the performance under the rotary island and multi-direction vehicle scene;

and 4.6, continuing training in the rest scenes until the process is finished.

5. The solution for automatic driving under the complex traffic scene of multiple targets based on reinforcement learning according to claim 4, wherein the reinforcement learning model trained based on the time-varying training method in step 4 comprises the following specific steps:

step 4.2.1, setting hyper-parameters of the reinforcement learning model;

step 4.2.2, setting a reward function as a basic reward, enabling the Agent to learn lane keeping, and starting iterative training;

4.2.3, heightening the central deviation punishment and collision punishment weight of the lane, and continuing iterative training;

step 4.2.4, continuously increasing the collision penalty, and continuously carrying out iterative training;

and 4.2.5, adding scenes on the basis of the original scene data set, adding a speed reward and increasing the lane center deviation punishment and collision punishment weight until the iteration is finished.

6. The solution of automatic driving under the complex traffic scene of multiple targets based on reinforcement learning of claim 5, wherein in step 5 the dangerous action recognizer predicts the dangerous degree according to the action and environment observation information output by the reinforcement learning model, and takes actions such as emergency avoidance and emergency adjustment according to the dangerous degree, the dangerous action recognizer includes a sample collection stage and a training stage, and the sample collection stage includes the following specific steps:

step 5.1.1, preparing various scenes, and selecting one scene to start training;

step 5.1.2, initializing a PPO strategy model, and starting training the strategy model under the selected scene;

step 5.1.3, recording the track of the current round in the running process;

step 5.1.4, when collision occurs, collecting 10 steps before the collision as negative samples, and randomly collecting any continuous 10 steps in the track of the current round as positive samples;

step 5.1.5, after training until the number of operation steps reaches the set total number of steps, selecting the next scene for repeated training;

and 5.1.6, ending until all scenes are acquired.

7. The solution for automatic driving under the complex traffic scenario of multiple targets based on reinforcement learning as claimed in claim 6, wherein the dangerous motion recognizer in step 5 is a dangerous motion recognizer model constructed based on a long and short memory network, and the specific steps in the training phase are as follows:

step 5.2.1, according to each group of collected sample data, generating a data group of the sample data by using a sliding window as a model input, and using a label of the last step of each group as a target label of the model;

step 5.2.2, using an Adam optimizer and adjusting the learning rate of the optimizer based on a cosine simulated annealing method;

step 5.2.3, calculating the mean square error loss of the model output and the target label by using the mean square error loss as a training loss function;

and 5.2.4, setting relevant model parameters and the number of training model rounds to finish training.

8. The solution of automatic driving under the complex traffic scenario of multiple objectives based on reinforcement learning of claim 7, characterized in that the rule constraint device in step 5 is an empirical statistical compiling rule based on human knowledge and simulation experiments for limiting the behavior of the vehicle under certain specific situations, the rule constraint device mainly includes a knowledge rule of nearest distance protection, a knowledge rule under the intersection, a knowledge rule before sharp bend and a correction rule of long-time residence under the condition that the vehicle has no neighboring vehicle, the environmental observation information judges different scenarios respectively and decides the speed limit rule of the vehicle.

Images