CN107544516A - Automated driving system and method based on relative entropy depth against intensified learning - Google Patents

Abstract

The invention relates to an automatic driving system based on relative entropy deep inverse reinforcement learning, including: (1) client: display driving strategy; (2) driving basic data collection subsystem: collect road information; (3) storage module: with The client terminal and the driving basic data acquisition subsystem are connected to and store the road information collected by the driving basic data acquisition subsystem; wherein, the driving basic data acquisition subsystem collects road information and transmits the road information to the client terminal and the storage module, The storage module receives the road information, and stores a continuous section of road information as a historical track, analyzes and calculates the driving strategy based on the historical track, and the storage module transmits the driving strategy to the client for user selection, and the client receives the road information And implement automatic driving according to the user's choice. The system of the present invention adopts a deep inverse reinforcement learning algorithm of relative entropy to realize automatic driving without models.

Description

Automatic driving system and method based on relative entropy deep inverse reinforcement learning

technical field

The invention relates to an automatic driving system and method based on relative entropy deep inverse reinforcement learning, belonging to the technical field of automatic driving.

Background technique

With the increase of the number of cars in our country, road traffic congestion is becoming more and more serious, and the annual traffic accidents are also rising. In order to better solve this problem, it is necessary to research and develop auto-driving systems. And with the improvement of people's pursuit of quality of life, people hope to be liberated from tiring driving activities, and automatic driving technology has emerged as the times require.

An existing auto-driving system for a vehicle uses a camera installed in the driver's cab and an image recognition system to identify the driving environment, and then the vehicle's main control computer, GPS positioning system, and path planning software provide information on the vehicle based on pre-stored road maps and other information. Carry out navigation, plan a reasonable driving path between the current position of the vehicle and the destination, and guide the vehicle to the destination.

In the above autopilot system, since the road map is pre-stored in the vehicle, the update of its data depends on the manual operation of the driver, and the update frequency cannot be guaranteed. There is no latest information about roads in the resources, so the final data cannot reflect the current road conditions, which eventually leads to unreasonable driving routes and low navigation accuracy, which brings inconvenience to driving. Moreover, most of the current autopilot systems in the field of autopilot technology still require manual intervention, and cannot achieve complete autopilot.

Contents of the invention

The purpose of the present invention is to provide an automatic driving system and method based on relative entropy deep inverse reinforcement learning, which uses a deep neural network structure and inputs the historical driving track information of the user driver to obtain a variety of driving strategies representing individual driving habits. These driving strategies carry out personalized and intelligent automatic driving.

In order to achieve the above object, the present invention provides the following technical solution: an automatic driving system based on relative entropy deep inverse reinforcement learning, the system includes:

Client: display driving strategy;

Driving basic data acquisition subsystem: collect road information;

Storage module: connected with the client and the driving basic data collection subsystem and storing the road information collected by the driving basic data collection subsystem;

Wherein, the driving basic data collection subsystem collects road information and transmits the road information to the client and the storage module, and the storage module receives the road information and stores a continuous section of road information as a historical track , analyze and calculate the driving strategy according to the historical trajectory, the storage module transmits the driving strategy to the client for selection by the user, and the client accepts and selects the driving strategy based on the road information and user personality Driving strategy implements autonomous driving.

Further, the storage module includes a driving trajectory library for storing historical driving trajectories, a trajectory information processing subsystem for calculating and simulating a driving strategy according to driving trajectories and driving habits, and a driving strategy library for storing driving strategies; the driving trajectory The library transmits the driving trajectory data to the trajectory information processing subsystem, and the trajectory information processing subsystem analyzes and calculates and simulates a driving strategy based on the driving trajectory data and transmits it to the driving strategy library, and the driving strategy library receives And store the driving strategy.

Further, the trajectory information processing subsystem adopts a multi-objective relative entropy deep inverse reinforcement learning algorithm to calculate and simulate the driving strategy.

Further, the multi-objective inverse reinforcement learning algorithm uses the EM algorithm framework to nest the relative entropy depth inverse reinforcement learning to calculate the parameters of the multi-reward function.

Further, the driving basic data collection subsystem includes sensors for collecting road information.

The present invention also provides a method for automatic driving based on relative entropy deep inverse reinforcement learning, said method comprising the following steps:

Including the following steps:

S1: collect road information and transmit the road information to the client and the storage module;

S2: The storage module receives the road information and stores a continuous piece of road information as a historical trajectory, analyzes and simulates various driving strategies according to the historical trajectory, and transmits the driving strategy to the client;

S3: The client receives the road information and driving strategy, and implements automatic driving according to the personalized driving strategy and road information selected by the user.

Further, the storage module includes a driving trajectory library for storing historical driving trajectories, a trajectory information processing subsystem for calculating and simulating a driving strategy according to driving planning and driving habits, and a driving strategy library for storing driving strategies; the driving trajectory The library transmits the driving trajectory data to the trajectory information processing subsystem, and the trajectory information processing subsystem analyzes and calculates and simulates a driving strategy based on the driving trajectory data and transmits it to the driving strategy library, and the driving strategy library receives And store the driving strategy.

Further, the trajectory information processing subsystem adopts a multi-objective relative entropy deep inverse reinforcement learning algorithm to calculate and simulate the driving strategy.

Further, the multi-objective inverse reinforcement learning algorithm uses the EM algorithm framework to nest the relative entropy depth inverse reinforcement learning to calculate the parameters of the multi-reward function.

The beneficial effects of the present invention are: by setting the driving basic data collection subsystem in the system, the road information is collected in real time, and the road information is transmitted to the storage module, and the storage module receives the road information and stores a continuous section of road information as a historical track , to simulate driving strategies based on historical driving trajectories to realize personalized and intelligent automatic driving.

The above description is only an overview of the technical solutions of the present invention. In order to understand the technical means of the present invention more clearly and implement them according to the contents of the description, the preferred embodiments of the present invention and accompanying drawings are described in detail below.

Description of drawings

FIG. 1 is a flow chart of the automatic driving system and method based on relative entropy deep inverse reinforcement learning of the present invention.

Figure 2 is a schematic diagram of a Markov decision process MDP.

detailed description

The specific implementation manners of the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

Please refer to Fig. 1, the automatic driving system based on relative entropy deep inverse reinforcement learning of a preferred embodiment of the present invention includes:

Client 1: display driving strategy;

Driving basic data acquisition subsystem 2: collect road information;

Storage module 3: connected with the client 1 and the driving basic data collection subsystem 2 and storing the road information collected by the driving basic data collection subsystem 2;

Wherein, the driving basic data collection subsystem 2 collects road information and transmits the road information to the client 1 and the storage module 3, and the storage module 3 receives the road information and stores a continuous section of road information It is stored as a historical trajectory, and the driving strategy is simulated according to the analysis and calculation of the historical trajectory. The storage module 3 transmits the driving strategy to the client 1 for the user to choose, and the client 1 receives the road information and according to the The personalized driving strategy selected by the user implements automatic driving. In this embodiment, the storage module 3 is a cloud.

The main function of the client 1 is to complete the human-computer interaction process with the user, and provide personalized and intelligent multiple driving strategy selections and services. The client 1 downloads the corresponding driving strategy from the cloud 3 driving strategy library 33 according to the user's personalized driving strategy selection, and then makes real-time driving decisions according to the driving strategy and basic data to realize real-time unmanned driving control.

The driving basic data collection subsystem 2 collects road information through sensors (not shown). The collected information has two purposes: transfer the information to the client 1 to provide basic data for the current driving decision; transfer the information to the driving track database 31 in the cloud 3 and store it as the historical driving track data of the user driver.

The cloud 3 includes a driving track library 31 for historical driving tracks, a track information processing subsystem 32 that calculates and simulates a driving strategy according to driving planning and driving habits, and a driving strategy library 33 that stores driving strategies; the driving track library 31 transmits the driving trajectory data to the trajectory information processing subsystem 32, and the trajectory information processing subsystem 32 analyzes and calculates and simulates a driving strategy according to the driving trajectory data and transmits it to the driving strategy library 33. The strategy library 33 receives and stores the driving strategies. The trajectory information processing subsystem 32 uses a multi-objective relative entropy deep inverse reinforcement learning algorithm to calculate and simulate driving strategies. In this embodiment, the multi-objective inverse reinforcement learning algorithm uses the EM algorithm framework to nest relative entropy depth inverse reinforcement learning to calculate the parameters of multiple reward functions. The historical driving trajectories include expert historical driving trajectories and user historical trajectories.

The inverse reinforcement learning IRL refers to the problem that the reward function R is unknown in the environment-known Markov decision process MDP. In the general reinforcement learning problem RL, the known environment, the given reward function R and the Markov property are often used to estimate the value Q(s, a) of a state-action pair (also called the action cumulative reward value ), and then use the converged value Q(s, a) of each state-action pair to obtain the strategy π, and the agent (Agent) can use the strategy π to make decisions. In reality, the reward function R is often extremely difficult to obtain, but some excellent trajectories T ^N are relatively easy to obtain. In the Markov decision process MDP/R where the reward function is unknown, the problem of using an excellent trajectory T ^N to restore the reward function R is called the inverse reinforcement learning problem IRL.

In this embodiment, the user’s historical driving trajectory data known in the driving trajectory database 31 is used to perform relative entropy depth inverse reinforcement learning, restore the reward function R of various user personalities, and then simulate the corresponding driving strategy π . The relative entropy deep inverse reinforcement learning algorithm is a model-free algorithm. It does not need to know the state transition function T(s, a, s′) in the environment model. The relative entropy inverse reinforcement learning algorithm can use the method of importance sampling to calculate Avoid the state transition function T(s, a, s′) in it.

In this embodiment, the automatic driving decision process of the car is a Markov decision process MDP/R without a reward function, which can be expressed as a set {state space S, action space A, state transition probability T defined by the environment (omitted for environment transition probability T requirements). The value function (cumulative reward value) of the car Agent can be expressed as The state-action-value function of the vehicle Agent can be expressed as Q(s,a)=R _θ (s,a)+γE _T(s,a,s′) [V(s′)]. In order to solve more complex real driving problems, the assumption of the reward function is no longer a simple linear combination, but a deep neural network R(s,a,θ)=g ₁ (g ₂ (…(g _n ( f(s,a),θ _n ),…),θ ₂ ),θ ₁ ), where f(s,a) represents the road feature information of driving at (s,a), θ _i represents the deep neural network The parameters of layer i.

At the same time, in order to meet more personalized and intelligent real driving scenarios, it is assumed that there are multiple reward functions R (objectives) existing at the same time, representing the different driving habits of the user driver. Suppose there are G reward functions, let the prior probability distribution of these G reward functions be ρ ₁ ,…,ρ _G , the reward weights are θ ¹ ,…,θ ^G , let Θ=(ρ ₁ ,…,ρ _G , θ ¹ ,…,θ ^G ), which represent the set of parameters of these G reward functions.

Please refer to Figure 2, under the condition that there is a hypothetical reward function (obtained by initialization or iteration), we can describe the problem as a complete Markov decision process MDP at this time. At this time, under the complete Markov decision process MDP, according to the knowledge of reinforcement learning, use the reward function R(s,a,θ)=g ₁ (g ₂ (…(g _n (f,θ _n ),…) ,θ ₂ ),θ ₁ ), we can evaluate the V value and Q value. For the evaluation algorithm of reinforcement learning, a new soft maximization method (MellowMax) is adopted to estimate the expected value of V value. The generator for MellowMax is defined as: MellowMax is a more optimized algorithm, which can guarantee that the estimation of V value can converge to a unique point. At the same time, MellowMax has special features: scientific probability distribution mechanism and expectation estimation method. In this embodiment, the reinforcement learning algorithm combined with MellowMax will be more reasonable in the exploration and utilization of the environment during the automatic driving process. This ensures that when the reinforcement learning process converges, the autopilot system has learned enough about various scenarios and can produce a more scientific assessment of the current state.

In this embodiment, according to reinforcement learning combined with a soft maximization algorithm MellowMax, a more scientific evaluation of the expected value of the feature of the state can be obtained. Using MellowMax, the probability distribution of action selection can be obtained as Under the rule of soft maximization action selection, using the iterative process of reinforcement learning, the expected value μ of the features that can be obtained in the reward function composed of the parameters of the current deep neural network as θ can be obtained. μ can be understood as the cumulative expectation of features.

In this embodiment, the EM algorithm is used to solve the above multi-objective inverse reinforcement learning problem with hidden variables. The EM algorithm can be divided into E step and M step according to the steps. Through the continuous iteration of E step and M step, the maximum value of the likelihood estimate is approximated.

Step E: first calculate where Z is a regular term. z _ij represents the probability that the i-th driving trajectory belongs to driving habit (reward function) j.

Let y _i =j indicate that the i-th driving trajectory belongs to driving habit j, and use the set of y=(y ₁ ,...,y _N ) to represent the subordinate set of N driving trajectories.

Calculate the likelihood estimate Q(Θ,Θ ^t )=∑ _y L(Θ|D,y)Pr(y|D,Θ ^t ) (the Q function Q(Θ,Θ ^t ) referred to here is the EM algorithm Update the objective function, pay attention to the difference from the Q action state value function in reinforcement learning), and obtain the likelihood estimate after calculation

Step M: Select an appropriate multi-driving habit parameter set Θ(ρ _l and θ _l ) to maximize the likelihood estimate Q(Θ,Θ ^t ) in step E. Due to the mutual independence of ρ _l and θ _l , their maximization can be obtained separately. can get second half

For the update objective that maximizes the second half of Q(Θ,Θ ^t ): can be understood as is the set of observed trajectories under the condition that the parameter of the l-th cluster target is θ _l The maximum likelihood equation can be obtained. We can use the knowledge of relative entropy deep inverse reinforcement learning to solve this maximum likelihood equation. The solution formula of relative entropy can be naturally applied to the backpropagation update of deep neural network parameters while meeting the maximum likelihood update objective. Let the maximization objective function of the deep neural network be L(θ)=logP(D,θ|r), and according to the decomposition formula of the joint likelihood function, L(θ)=logP(D,θ|r)=logP can be obtained (D|r)+logP(θ). The partial derivative of the joint likelihood objective function can be obtained For the first half of the partial derivative, it can be further decomposed, expressed as

in According to the knowledge of relative entropy inverse reinforcement learning, the solution result can be obtained as the difference between the feature expectation value under the current reward function and the expert feature value Among them, using importance sampling, Among them, π is a given strategy, and according to this strategy π sampling is obtained track. in where τ=s ₁ a ₁ ,...,s _H a _H . further, in Represented as gradients computed by the backpropagation algorithm when updating hidden layer parameters in a deep neural network.

The completion of the gradient update marks the completion of an iterative update of relative entropy deep inverse reinforcement learning. The new deep network reward function that has completed the parameter update is used to generate a new strategy π for a new iteration.

E-step and M-step calculations are iteratively performed until the likelihood estimate Q(Θ,Θ ^t ) converges to the maximum value. The parameter set Θ=(ρ ₁ ,...,ρ _G ,θ ¹ ,...,θ ^G ) obtained at this time is the prior distribution and weight of the reward function representing multiple driving habits that we want to solve.

In this embodiment, according to this parameter set Θ, the driving strategy π for each driving habit R is obtained through reinforcement learning RL calculation. Output multiple driving strategies and save them in the driving strategy library in the cloud. The user can choose a personalized and intelligent driving strategy in the client.

The present invention also provides a method for automatic driving based on relative entropy deep inverse reinforcement learning, said method comprising the following steps:

S1: collect road information and transmit the road information to the client and the storage module;

S2: The storage module receives the road information, analyzes and calculates and simulates various driving strategies according to the road information, and transmits the driving strategies to the client;

S3: The client receives the road information and driving strategy, and implements automatic driving according to the personalized driving strategy and road information selected by the user.

To sum up: by setting the driving basic data acquisition subsystem 2 in the system, road information is collected in real time, and the road information is transmitted to the storage module 3 and the client 1. After receiving the road information, the storage module 3 simulates driving according to the historical driving trajectory strategies to realize personalized and intelligent autonomous driving.

In the automatic driving based on this method, the driving strategy is calculated in the cloud 3 instead of running the calculation process in the client 1 . When the user needs to drive automatically, all driving strategies have been completed on the cloud 3. Users only need to choose to download the driving strategy they need, and the car body can perform real-time automatic driving according to the driving strategy selected by the user and real-time road information. Simultaneously, after completing any driving, a large amount of road information is uploaded to the cloud 3 and stored as historical driving tracks. Utilize the big data of stored historical driving trajectories to update the driving strategy library. Using the big data of trajectory information, this system will realize automatic driving that is closer to the needs of users.

The various technical features of the above-mentioned embodiments can be combined arbitrarily. To make the description concise, all possible combinations of the various technical features in the above-mentioned embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, should be considered as within the scope of this specification.

The above-mentioned embodiments only express several implementation modes of the present invention, and the descriptions thereof are relatively specific and detailed, but should not be construed as limiting the patent scope of the invention. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent for the present invention should be based on the appended claims.

Claims (9)

1. a kind of automated driving system based on relative entropy depth against intensified learning, it is characterised in that the system includes：

Client：Show driving strategy；

Drive basic data acquisition subsystem：Gather road information；

Memory module：It is connected with the client and driving basic data acquisition subsystem and stores the driving basic data and is adopted The road information that subsystem is collected；

Wherein, the driving basic data acquisition subsystem gathers road information and the road information is transferred into the client End and memory module, the memory module receives the road information, and one section of lasting road information is stored as into history rail Mark, analysis calculating is carried out according to the historical track and simulates driving strategy, the memory module transmits the driving strategy To client for selection by the user, the driving that the client receives and selected according to the road information and user personality Strategy implement automatic Pilot.

2. the automated driving system based on relative entropy depth against intensified learning as claimed in claim 1, it is characterised in that described Memory module is including being used to store the driving locus storehouse of history driving locus, calculating and simulate according to driving locus and driving habit Go out the trace information processing subsystem of driving strategy and the driving strategy storehouse of memory of driving strategy；The driving locus storehouse will drive Track data is transferred to the trace information processing subsystem, and the trace information processing subsystem is according to the driving locus number Calculated according to analysis and simulate driving strategy and be transferred to the driving strategy storehouse, the driving strategy storehouse receives and stored described Driving strategy.

3. the automated driving system based on relative entropy depth against intensified learning as claimed in claim 2, it is characterised in that described Trace information processing subsystem is calculated using the relative entropy depth of multiple target against nitrification enhancement and drive simulating strategy.

4. the automated driving system based on relative entropy depth against intensified learning as claimed in claim 3, it is characterised in that described The inverse nitrification enhancement of multiple target calculates more reward functions using EM algorithm frames nesting relative entropy depth against intensified learning Parameter.

5. the personalized automated driving system based on relative entropy depth against intensified learning, its feature exist as claimed in claim 1 In the basic data acquisition subsystem that drives includes being used for the sensor for gathering road information.

A kind of 6. method based on relative entropy depth against the automatic Pilot of intensified learning, it is characterised in that methods described is included such as Lower step：

S1：The road information is simultaneously transferred to client and memory module by collection road information；

S2：The memory module receives the road information and one section of lasting road information is stored as into historical track, according to The historical track analysis is calculated and simulates a variety of driving strategies, and the driving strategy is passed into the client；

S3：The client receives the road information and driving strategy, and the individual character driving strategy and road selected according to user Road information implements automatic Pilot.

7. the method based on relative entropy depth against the automatic Pilot of intensified learning as claimed in claim 6, it is characterised in that institute Memory module is stated including being used to store the driving locus storehouse of history driving locus, calculating simultaneously mould according to driving planning and driving habit Draw up the trace information processing subsystem of driving strategy and the driving strategy storehouse of memory of driving strategy；The driving locus storehouse will drive Sail track data and be transferred to the trace information processing subsystem, the trace information processing subsystem is according to the driving locus Data analysis calculates and simulates driving strategy and be transferred to the driving strategy storehouse, and the driving strategy storehouse receives and stores institute State driving strategy.

8. the method based on relative entropy depth against the automatic Pilot of intensified learning as claimed in claim 7, it is characterised in that institute State trace information processing subsystem and simultaneously drive simulating strategy is calculated against nitrification enhancement using the relative entropy depth of multiple target.

9. the method based on relative entropy depth against the automatic Pilot of intensified learning as claimed in claim 8, it is characterised in that institute The inverse nitrification enhancement for stating multiple target calculates more reward functions using EM algorithm frames nesting relative entropy depth against intensified learning Parameter.