CN114580539A - A vehicle driving strategy processing method and device - Google Patents

Abstract

The invention discloses a vehicle driving strategy processing method and device, and relates to the field of intelligent driving. A specific implementation of the method includes: collecting the travel trajectories of a plurality of vehicles in different states, grouping the travel trajectories according to the strategy used in the first state in each travel trajectories, and obtaining multiple travel trajectories data sets; The driving trajectory data set constructs a value function to calculate the confidence of each value function, and then calculates the probability activation threshold based on the confidence of each value function; the probability that the function value of each value function is greater than the preset value is used as the evaluation error, Based on the evaluation errors and probability activation thresholds of all value functions, a probability activation function is constructed; based on multiple driving trajectory data sets and probability activation functions, the network parameters of the decision model are adjusted to obtain an adjusted decision model. In this embodiment, the driving trajectory is segmented, and a probability activation function is constructed to ensure the reliability of the decision-making model under the condition of limited data.

Description

A vehicle driving strategy processing method and device

technical field

The present invention relates to the field of intelligent driving, in particular to a method and device for processing vehicle driving strategies.

Background technique

The self-learning decision-making method evaluates the unmanned strategy based on the feedback obtained by the unmanned vehicle in different states and actions in the historical data, and adjusts the strategy based on the evaluation to adapt to the environment. Therefore, the uncertainty of the environmental state will directly affect the strategy return to realize strategy adjustment. This scheme does not make a priori assumptions on the environmental model, and the solution process itself does not require the state space dimension, so it is hopeful to be applied to high-dimensional Solving unmanned driving strategies in deterministic scenarios.

In the process of implementing the present invention, the inventor found that insufficient training data would lead to unreliable self-learning strategies. Existing methods usually assume that the training data can cover all the scenes, and in a high-dimensional uncertain environment, the complete coverage of the scene requires a large amount of training data. However, the performance of self-learning decision-making methods that are not fully trained cannot be guaranteed when encountering new scenarios, so it is difficult to implement self-learning decision-making methods in unmanned driving.

SUMMARY OF THE INVENTION

In view of this, the embodiments of the present invention provide a vehicle driving strategy processing method and device, which can at least solve the problem of insufficient training data in the prior art, resulting in insufficient training of the self-learning decision-making method, and thus unsuitable for unmanned driving scenarios.

To achieve the above object, according to an aspect of the embodiments of the present invention, a method for processing a vehicle driving strategy is provided, including:

Collect the driving trajectories of multiple vehicles in different states, group the driving trajectories according to the strategy used in the first state of each driving trajectory, and obtain multiple driving trajectory data sets;

constructing a value function for each driving trajectory data set to calculate the confidence of each value function, and then calculate the probability activation threshold based on the confidence of each value function;

The probability that the function value of each value function is greater than the preset value is used as the evaluation error, and the probability activation function is constructed based on the evaluation errors of all value functions and the probability activation threshold;

Based on the multiple driving trajectory data sets and the probability activation function, the network parameters of the decision-making model are adjusted to obtain an adjusted decision-making model; wherein, the adjusted decision-making model is used for, during the driving process of the vehicle, according to the current state of the vehicle. state and driving trajectory for driving strategy planning.

Optionally, before the grouping of the driving trajectories according to the strategy used in the first state in each driving trajectory, the method further includes:

Using a sliding window of preset length, each travel trajectory is divided into a plurality of travel trajectories of preset length.

Optionally, it also includes: counting the number of driving trajectories in each driving trajectory data set, determining a target driving trajectory data set whose number of driving trajectories is less than or equal to a preset threshold, and setting the evaluation error of the target driving trajectory data set as 0 .

Optionally, the strategy includes one of a rule strategy and a self-learning strategy, and the probability activation threshold is calculated based on the confidence of each value function, including:

Calculate the probability value when the performance expectation of the self-learning policy relative to the rule policy is the largest, and use the probability value as the value range of the probability activation threshold; wherein, the performance expectation represents the confidence difference.

Optionally, the planning of the driving strategy according to the current state of the vehicle and the driving trajectory includes:

In response to the fact that the current state of the vehicle does not exist in the state space, determine that the vehicle driving strategy is the rule strategy;

Randomly generate a variable, in the process of the vehicle continuing to drive, by collecting the driving trajectory of the vehicle after using the rule policy to estimate the performance value of the rule policy, if it is detected that the variable is greater than the cumulative value of the performance value of the rule policy and 1, then Triggers a policy update operation.

In order to achieve the above object, according to another aspect of the embodiments of the present invention, a vehicle driving strategy processing device is provided, including:

The collection and grouping module is used to collect the driving trajectories of multiple vehicles in different states, and group the driving trajectories according to the strategy used in the first state in each driving trajectory to obtain multiple driving trajectory data sets;

A confidence level calculation module, used for constructing a value function for each driving trajectory data set, to calculate the confidence level of each value function, and then calculating a probability activation threshold based on the confidence level of each value function;

The function building module is used to take the probability that the function value of each value function is greater than the preset value as the evaluation error, and build the probability activation function based on the evaluation errors of all value functions and the probability activation threshold;

A parameter adjustment module for adjusting the network parameters of the decision-making model based on the plurality of travel trajectory data sets and the probability activation function to obtain an adjusted decision-making model; wherein, the adjusted decision-making model is used for driving the vehicle During the process, the driving strategy planning is carried out according to the current state of the vehicle and the driving trajectory.

Optionally, the collection and grouping module is also used for:

Using a sliding window of preset length, each travel trajectory is divided into a plurality of travel trajectories of preset length.

Optionally, the function builds a module for:

The number of driving trajectories in each driving trajectory data set is counted, a target driving trajectory data set whose number of driving trajectories is less than or equal to a preset threshold is determined, and the evaluation error of the target driving trajectory data set is set to 0.

Optionally, the strategy includes one of a rule strategy and a self-learning strategy, and the confidence calculation module is used for:

Calculate the probability value when the performance expectation of the self-learning policy relative to the rule policy is the largest, and use the probability value as the value range of the probability activation threshold; wherein, the performance expectation represents the confidence difference.

Optionally, also includes a strategic planning module for:

In response to the fact that the current state of the vehicle does not exist in the state space, determine that the vehicle driving strategy is the rule strategy;

Randomly generate a variable, in the process of the vehicle continuing to drive, by collecting the driving trajectory of the vehicle after using the rule policy to estimate the performance value of the rule policy, if it is detected that the variable is greater than the cumulative value of the performance value of the rule policy and 1, then Triggers a policy update operation.

To achieve the above object, according to yet another aspect of the embodiments of the present invention, an electronic device for processing a vehicle driving strategy is provided.

An electronic device according to an embodiment of the present invention includes: one or more processors; and a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more programs cause the One or more processors implement any of the above-described vehicle driving strategy processing methods.

In order to achieve the above object, according to another aspect of the embodiments of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, any one of the above-mentioned vehicle driving strategy processing is implemented. method.

According to the solution provided by the present invention, one of the above-mentioned embodiments of the present invention has the following advantages or beneficial effects: segmenting each driving track to increase the number of training data as much as possible; constructing a probability activation function to ensure activation of the self-learning strategy It can improve the reliability of the performance of the rules and policies, so as to ensure the reliability of the decision-making model under the condition of limited data.

Further effects of the above non-conventional alternatives will be described below in conjunction with specific embodiments.

Description of drawings

The accompanying drawings are used for better understanding of the present invention and do not constitute an improper limitation of the present invention. in:

FIG. 1 is a schematic flowchart of a main flow of a method for processing a vehicle driving strategy according to an embodiment of the present invention;

Figure 2 is a schematic diagram of the relationship between state, strategy and behavior;

Fig. 3 is a schematic diagram of the influence of probability activation threshold on mixed decision-making;

4 is a schematic diagram of main modules of a vehicle driving strategy processing device according to an embodiment of the present invention;

5 is an exemplary system architecture diagram to which an embodiment of the present invention may be applied;

FIG. 6 is a schematic structural diagram of a computer system suitable for implementing a mobile device or a server according to an embodiment of the present invention.

Detailed ways

Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, which include various details of the embodiments of the present invention to facilitate understanding and should be considered as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.

Referring to FIG. 1, it shows a main flowchart of a vehicle driving strategy processing method provided by an embodiment of the present invention, including the following steps:

S101: Collecting multiple vehicle driving trajectories in different states, grouping the driving trajectories according to the strategy used in the first state in each driving trajectory, and obtaining multiple driving trajectory data sets;

S102: constructing a value function for each driving trajectory data set, to calculate the confidence of each value function, and then calculating a probability activation threshold based on the confidence of each value function;

S103: Use the probability that the function value of each value function is greater than the preset value as an evaluation error, and construct a probability activation function based on the evaluation errors of all value functions and the probability activation threshold;

S104: Based on the multiple driving trajectory data sets and the probability activation function, adjust the network parameters of the decision-making model to obtain an adjusted decision-making model; The current state of the vehicle and the driving trajectory are used for driving strategy planning.

In the above embodiment, for step S101, the real world scene has the characteristics of high dimension and strong uncertainty, so it is very difficult for the unmanned vehicle to fully sample all the scenes. According to statistics, it takes about 14 billion kilometers of driving data to verify only one driving strategy in a real driving scenario, while the amount of data required for strategy training is even greater.

Due to the interaction between the vehicle and the surrounding vehicles and roads, the driving strategy decision-making of the vehicle needs to comprehensively consider the movement state of the ego vehicle, the movement state of the surrounding vehicles and the passability of the lane. The state is the driving state of the vehicle, including 1) the current motion state information of the own vehicle, such as vehicle speed and acceleration, 2) the current motion state information of the surrounding vehicles, such as the longitudinal distance, lateral distance and relative vehicle speed from the vehicle, if there are no surrounding vehicles , the distance is set to infinity, and the relative speed is set to 0. 3) Lane trafficability information, such as left lane marking information and right lane marking information, the passable mark is 1, and the passable mark (such as the opposite lane) , non-motorized lanes, curbs) is 0.

For example, when walking in a maze, you always like to go left. This is a behavioral pattern. Vehicles will also follow certain behavioral patterns during driving, such as U-turn, changing lanes, turning left, turning right, going straight, etc., corresponding to different strategies. is denoted by π, and the value function is also closely related to it. When the vehicle has been running with policy π, the value function indicates that there is a relationship between the expected reward value that can be obtained in the current state s and its successor state.

For example, as shown in Figure 2, the hollow circle represents the state, and the solid circle represents the behavior. In state s, the vehicle faces three choices, and the policy π is what prompts the vehicle to choose among the three behaviors, which represents a probability distribution, that is, from The probability of choosing a different action among all actions. After selecting the behavior a, it will be converted to a new state s' from different possibilities, the conversion probability is p, and the corresponding reward of the environment will be obtained in the process of conversion, that is, r.

In the process of real road driving, the length of the driving trajectory of the vehicle is usually not fixed, so the decision-making model needs to use a sliding window with a preset length to collect the required trajectory τ _π (s), so as to obtain the driving trajectory in different states. , as shown in the following formula:

τ _π (s _i ):={s _i ,a _i ,...,s _k , _ak }

k=min(i+H-1,n)

In the formula, s is the state, Z _s is the set of termination states, and ω _π is the trajectory of the vehicle using policy π. According to the description in Figure 2, each trajectory may be composed of multiple states and strategies. Therefore, it is divided according to the action (i.e. strategy) used in the first state in the trajectory to obtain two sub-datasets (rule-policy dataset and self-learning). policy dataset), as follows:

D _b (s):={τ(s ₁ =s,a ₁ = _πb (s ₁ ))}

D _r1 (s):={τ(s ₁ =s,a ₁ =π _r1 (s ₁ ))}

In the formula, D _b (s) contains the trajectory data of the vehicle in the running state s, and the first action uses the rule strategy; D _r1 (s) contains the trajectory of the vehicle in the running state s, the first action uses the self-learning strategy data. For example, if there are 2 states and 2 strategies in total, 4 driving trajectory data sets D(s, a) will be obtained, including driving trajectory data set 1-state 1 and strategy 1, driving trajectory data set 2-state 1 and strategy 2 , driving trajectory data set 3 - state 1 and strategy 2, driving trajectory data set 4 - state 2 and strategy 2.

For step S102, after generating the driving trajectory data set D(s, a), start to calculate the evaluation error of each driving trajectory data set; wherein, the evaluation error represents the deviation between the estimated value and the actual value, such as the estimated value The probability is 90%, but the true value is 1, then the bias is 10%.

Define the value function Q _π (s,u _π ), which represents the expected performance of the vehicle using state s and policy π in the future. However, in the case of limited training data, there is an error in the estimation of the value function. If the value function containing the error is directly used to adjust and train the driving strategy model in the decision-making model, it may lead to the problem of incorrect use of the strategy. In fact, the real value function exists only in the concept and cannot be directly observed. Considering that when the vehicle motion state can be sufficiently sampled, the deviation between the estimated value of the value function and the real value is small, so a function can be constructed to express the deviation. The estimation method based on Monte Carlo sampling here needs to fully sample the trajectory under each state and each strategy. However, due to the strong uncertainty of the environment, the high-dimensional characteristics of the environment, and the lack of data volume, it is impossible to guarantee that the trajectories are uniform. To meet the requirements of sufficient sampling, it is necessary to use this method to obtain the confidence of the value function.

This scheme preferably uses the Lindbergh-Feller theorem to estimate the confidence of the value function of each driving trajectory data set. The formula is as follows:

σ ² =Var(G _π (s))

使用

表示真实价值函数

大于λ的概率，将

作为评估误差。此处基于规则策略数据集D(s_t,a_b)计算

以及基于自学习策略数据集D(s_t,a_r1)计算

where Φ(·) represents the cumulative distribution function of the standard normal distribution, and G _π (s) represents the discounted return value of the trajectory. Through the above formula, the true value distribution of the value function is described as a cumulative distribution function, and as the number of sampling increases, its estimated value is gradually close to the true value. This formula can also obtain the confidence of the value function through integration. Define the true value function

use

represents the true value function

The probability of being greater than λ, will be

as an evaluation error. Here, it is calculated based on the rule policy dataset D(s _t , a _b )

And based on the self-learning strategy dataset D(s _t , a _r1 ) calculation

For step S103, after generating the evaluation error of each driving trajectory data set, it is necessary to construct an activation function considering the error of the value function, and this solution is called a probability activation function. Considering that in the actual application process, the self-learning strategy will be iteratively updated during the training process, so based on the trajectory data obtained by the self-learning strategy, the true value of the self-learning value function is estimated conservatively, that is, the calculated value of the probability activation function is higher than the true value. It is too large, but does not undermine the reliability of mixed decision-making to improve policy performance.

可能不等于自学习模型估计的价值函数Q(s,u_π)，无人驾驶车辆行驶在部分场景中时，存在如下所示的情况：Design of Probabilistic Activation Functions for Hybrid Decisions: Policy Performance Expectations in Limited Data

It may not be equal to the value function Q(s,u _π ) estimated by the self-learning model. When the unmanned vehicle is driving in some scenes, the following situations exist:

In the above formula, for the rule strategy π _b and the self-learning strategy π _r1 , when the probability activation function is activated according to the self-learning strategy value function Q(s, u _r1 ) and the regular strategy value function Q(s, u _b ), The self-learning strategy performs worse than the regular strategy, and the self-learning strategy is misactivated at this time. Since the value function cannot be estimated accurately, the performance of the self-learning policy is expected to outperform the performance of the regular policy, denoting the probabilistic activation function:

In the formula, P(·) represents the probability of event occurrence, and c _thres is the activation threshold of the probability of mixed decision, which needs to be calculated by theoretical calculation of confidence. Based on the above formulas (1) and (2), the relationship between the probability activation function and the sampled data set is as follows:

In the formula, when the sampling times are insufficient, that is, when n _d ≤n _thres =30, let κ(Q(s,u),λ)=0. The design principle of the probability activation threshold c _thres is to ensure that according to the current observation and training data, the self-learning strategy has the greatest expectation of improving the performance of the rule strategy, namely (4):

Furthermore, according to the above formulas (1) and (3), we can get:

Ideally, the expected performance of the two strategies is a constant value. However, due to the estimation error, the expected performance of the two cannot be accurately estimated. Therefore, the probability activation function is designed in the form of probability, which can be introduced into the activation function in the strategy evaluation process. , to solve the evaluation error caused by insufficient training, avoid the false activation of the self-learning strategy caused by insufficient sampling due to limited data, and achieve high reliability decision-making under limited data conditions.

Referring to Fig. 3, the probability activation threshold c _thres can realize the transition from a regular strategy to a fully self-learning strategy. Assuming that the probability activation threshold value ranges from 0 to 1, when the threshold value is 0, the probability activation function is always established, so the self-learning strategy will always be used to drive the vehicle. When the threshold is 1, the probability activation function does not hold, so the rule-based strategy will always be used to drive the vehicle. In the values from 0 to 1, the hybrid decision-making framework is used for driving. When the threshold is 0.5, the expectation of performance improvement is the highest, so the reliability of the hybrid strategy to the improvement of the rule strategy can be guaranteed. Higher thresholds make policy promotion conservative, while lower thresholds can make policy promotion unreliable.

For step S104, after the probability activation threshold c _thres is determined under the condition of limited data, the decision model is designed based on the probability activation threshold c _thres . This scheme refers to the deep Q-learning algorithm to establish a decision model. The iterative calculation of the Q value uses the Bellman equation, and its update principle is shown in the following formula, where α is the learning rate:

Q(s _k , _ak )←

Q(s _k , _ak )+α[r(s _k+1 )+γmax _a Q(s _k+1 ,a)-Q(s _k , _ak )]

In the above formula, the Q function (also known as the right tail function of the standard normal distribution) needs to be updated according to the state and the reward generated by the action strategy. However, in the definition of the driverless decision problem, the state space is usually continuous, so it cannot be repeated Access a specific state. For this reason, the storage and update of the Q function will be completed based on a neural network. This framework is called a deep Q learning method. The update principle is as follows:

Q ⁺ (s _k , _ak )=r(s _k+1 )+γmax _a Q(s _k+1 ,a,θ ⁻ )

In the formula, θ and θ ^- represent the parameters of the Q network being adjusted and the parameters of the Q network stored in the history, respectively. θ ^- will update θ after a certain number of iterations n _up = 100 times, this setting makes the network update more stable. (Q(s _k , _ak ,θ _j )-Q ⁺ (s _k , _ak )) ² represents the training error, and the value of Q ⁺ (s _k , _ak ) is calculated by the Bellman equation. Since the value of the Q function is updated based on the dataset D(s,a), the data sampling determines the quality of the policy update. In this algorithm, the self-learning strategy update is divided into two stages: the rule-policy verification stage and the self-learning exploration stage.

表示基于动态更新的数据集D(s₁,a_b)所估算的策略表现，n_d(D(s,a_b))表示数据集中轨迹的数量：The algorithm here describes the self-learning exploration and policy generation process,

represents the estimated policy performance based on the dynamically updated dataset D(s ₁ , a _b ), and n _d (D(s, a _b )) represents the number of trajectories in the dataset:

i = 0;

ω=0;

D(s, a _b )=0;

D(s, a) = 0;

while i≤total training times do

i=1+1;

if s _t ∈ Z _s , then

Random sampling ξ←U(0,1);

thenIf n _d (D(s,a _b ))≥n _thres and

then

Generate an exploration action a _explore ;

use the action a _explore to interact with the environment;

else

interact with the environment using actions a _b ;

Intercept τ={s ₁ , a ₁ , r ₁ ,..., s _H , a _H , r _H } from ω;

if a ₁ =π _b (s ₁ )then

Add the trajectory τ to D(s ₁ , a _b );

else

Add the trajectory τ to D(s ₁ , a=a ₁ );

remove (s ₁ ,a ₁ ) from ω

else

for i=l to n _ω do

Intercept τ={s _i ,a _i ,...,s _nω ,a _nω } from ω;

if a _i =π _b (s)then

Add the trajectory τ to D(s ₁ , a _b );

else

Add the trajectory τ to D(s ₁ , a= _ai );

ω=θ;

for i=l to H do

Interact with the environment using actions a _b

Specifically, in the first stage, the driverless vehicle will only drive using the regular policy to obtain an update of Q(s, a _b ); while in the second stage, the self-learning policy actively tries different actions from the regular policy , for a better strategy. The switching of the two stages is realized by designing the exploration scope of inhibition self-learning. In the algorithm framework proposed in this chapter, there are two inhibition conditions that limit the scope of exploration. 1) When the rule strategy has not been verified for a long enough time, That is, n _d (D(s, a _b ))<n _thres , policy exploration will not be performed; 2) When the performance of the rule policy is good enough, policy exploration will not be performed. In contrast, the exploration scheme of traditional self-learning algorithms is usually independent of other strategies.

Since the exploration range usually determines the strategy exploration efficiency, the larger the exploration range, the lower the strategy exploration efficiency. The method proposed in this scheme can also help improve the exploration efficiency. In addition, this scheme preferably uses the e-greedy method for strategy exploration, and other strategy exploration methods are also feasible in this scheme.

时，系统会主动探索。The first reason for setting the inhibition condition is that the probability activation function needs to first estimate the value function of the rule strategy. When the rule strategy does not obtain the confidence of the value function estimation, the system will not be able to judge whether the current self-learning strategy can be effectively improved. Therefore, when encountering an unfamiliar state, it is necessary to give priority to driving based on a rule-based strategy to ensure driving reliability. The second method of setting the suppression condition is to randomly generate a variable ξ～U(0,1) when the first condition is satisfied, that is, ξ obeys the average distribution between 0-1. When the variable satisfies the constraint such that

, the system will actively explore.

In this way, the self-learning exploration probability is -Q(s, a _b ), which is related to the rule policy value function. The larger the rule policy value function, the weaker the exploration motivation. Under the condition of the probability constraint-oriented reward function, the Q value represents the probability that the policy satisfies the constraint, so this method means that the more likely the rule-based policy is to be dangerous, the more motivated the system is to actively explore. This method can improve the exploration efficiency on the premise of ensuring reliability.

In addition, because the state space is continuous, the system uses a fixed sampling window method to obtain a data set corresponding to a certain state, and calculates the corresponding sampling times. The scale of the sampling window is b(max(D(s))-min(D (s))), b∈(0,1), the value of b needs to be designed according to experience, and 0.1 is preferred in this scheme.

When a certain state s _t is accessed during the driving process of the driverless vehicle, a decision-making model under the condition of limited data is generated according to the probability activation function:

This decision model can be used to plan driving strategies based on the current state of the vehicle and the driving trajectory.

The method provided by the embodiment of the present invention can still ensure the effective determination of the strategy under the condition of limited training data:

1. Use a sliding window of preset length to divide each driving trajectory into multiple driving trajectories of preset length to increase the amount of training data as much as possible;

2. Construct a probability activation function to ensure the reliability of the performance improvement of the rule strategy when the self-learning strategy is activated, so as to achieve the reliability of the decision-making model in the case of limited data;

3. Design a self-learning decision-making system based on the deep Q-learning framework. While considering the rule strategy for strategy search, the requirement of the probability activation function for the confidence of the strategy value function is also considered, so that the confidence of the rule strategy value function does not meet the requirements. Or when the performance is better, the policy exploration of the self-learning model can be suppressed.

Referring to FIG. 4 , a schematic diagram of main modules of a vehicle driving strategy processing device 400 provided by an embodiment of the present invention is shown, including:

The collecting and grouping module 401 is used for collecting the driving trajectories of multiple vehicles in different states, and grouping the driving trajectories according to the strategy used in the first state in each driving trajectory to obtain a plurality of driving trajectory data sets;

A confidence level calculation module 402, configured to construct a value function for each driving trajectory data set, to calculate the confidence level of each value function, and then calculate a probability activation threshold based on the confidence level of each value function;

The function building module 403 is used to use the probability that the function value of each value function is greater than the preset value as the evaluation error, and build the probability activation function based on the evaluation errors of all value functions and the probability activation threshold;

The parameter adjustment module 404 is configured to adjust the network parameters of the decision-making model based on the plurality of driving trajectory data sets and the probability activation function to obtain an adjusted decision-making model; wherein, the adjusted decision-making model is used in the vehicle During the driving process, the driving strategy planning is carried out according to the current state of the vehicle and the driving trajectory.

In the implementation device of the present invention, the collecting and grouping module 401 is further configured to:

Using a sliding window of preset length, each travel trajectory is divided into a plurality of travel trajectories of preset length.

In the implementation device of the present invention, the function building module 403 is used for:

The number of driving trajectories in each driving trajectory data set is counted, a target driving trajectory data set whose number of driving trajectories is less than or equal to a preset threshold is determined, and the evaluation error of the target driving trajectory data set is set to 0.

In the implementation device of the present invention, the strategy includes one of a rule strategy and a self-learning strategy, and the confidence calculation module 402 is used for:

Calculate the probability value when the performance expectation of the self-learning policy relative to the rule policy is the largest, and use the probability value as the value range of the probability activation threshold; wherein, the performance expectation represents the confidence difference.

The implementation device of the present invention also includes a strategy planning module for:

In response to the fact that the current state of the vehicle does not exist in the state space, determine that the vehicle driving strategy is the rule strategy;

Randomly generate a variable, in the process of the vehicle continuing to drive, by collecting the driving trajectory of the vehicle after using the rule policy to estimate the performance value of the rule policy, if it is detected that the variable is greater than the cumulative value of the performance value of the rule policy and 1, then Triggers a policy update operation.

In addition, the specific implementation content of the device in the embodiment of the present invention has been described in detail in the above-mentioned method, so the repeated content will not be described here.

Figure 5 shows an exemplary system architecture 500 to which embodiments of the present invention may be applied, including terminal devices 501, 502, 503, a network 504, and a server 505 (just an example).

The terminal devices 501, 502, 503 can be various electronic devices with display screens and support web browsing, and various communication client applications are installed. Users can use the terminal devices 501, 502, 503 to interact with the server 505 through the network 504 to receive or send messages, etc.

The network 504 is a medium used to provide a communication link between the terminal devices 501 , 502 , 503 and the server 505 . Network 504 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

The server 505 may be a server that provides various services. It should be noted that the methods provided in the embodiments of the present invention are generally executed by the server 505 , and accordingly, the apparatus is generally set in the server 505 . It should be understood that the numbers of terminal devices, networks and servers in FIG. 5 are only illustrative. There can be any number of terminal devices, networks and servers according to implementation needs.

Referring to FIG. 6 below, it shows a schematic structural diagram of a computer system 600 suitable for implementing a terminal device according to an embodiment of the present invention. The terminal device shown in FIG. 6 is only an example, and should not impose any limitations on the functions and scope of use of the embodiments of the present invention.

As shown in FIG. 6, a computer system 600 includes a central processing unit (CPU) 601, which can be loaded into a random access memory (RAM) 603 according to a program stored in a read only memory (ROM) 602 or a program from a storage section 608 Instead, various appropriate actions and processes are performed. In the RAM 603, various programs and data necessary for the operation of the system 600 are also stored. The CPU 601 , the ROM 602 , and the RAM 603 are connected to each other through a bus 604 . An input/output (I/O) interface 605 is also connected to bus 604 .

The following components are connected to the I/O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc. ; and a communication section 609 including a network interface card such as a LAN card, a modem, and the like. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I/O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the drive 610 as needed so that a computer program read therefrom is installed into the storage section 608 as needed.

In particular, the processes described above with reference to the flowcharts may be implemented as computer software programs in accordance with the disclosed embodiments of the present invention. For example, embodiments disclosed herein include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart. In such an embodiment, the computer program may be downloaded and installed from the network via the communication portion 609 and/or installed from the removable medium 611 . When the computer program is executed by the central processing unit (CPU) 601, the above-described functions defined in the system of the present invention are performed.

It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two. The computer readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), fiber optics, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In the present invention, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device . Program code embodied on a computer readable medium may be transmitted using any suitable medium including, but not limited to, wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more logical functions for implementing the specified functions executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams or flowchart illustrations, and combinations of blocks in the block diagrams or flowchart illustrations, can be implemented in special purpose hardware-based systems that perform the specified functions or operations, or can be implemented using A combination of dedicated hardware and computer instructions is implemented.

The modules involved in the embodiments of the present invention may be implemented in a software manner, and may also be implemented in a hardware manner. The described modules can also be set in the processor, for example, it can be described as: a processor includes a collection grouping module, a confidence calculation module, a function construction module, and a parameter adjustment module. Among them, the names of these modules do not constitute a limitation of the module itself in some cases, for example, the parameter adjustment module can also be described as "adjusting parameters and using model module".

As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or may exist alone without being assembled into the device. The above computer-readable medium carries one or more programs, and when the one or more programs are executed by a device, the device causes the device to execute the vehicle driving strategy processing method of the present solution.

The above-mentioned specific embodiments do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims (10)

1. A vehicle driving strategy processing method, characterized by comprising:

collecting running tracks of a plurality of vehicles in different states, and grouping the running tracks according to a strategy used in a first state in each running track to obtain a plurality of running track data sets;

constructing a cost function for each driving track data set to calculate the confidence coefficient of each cost function, and further calculating a probability activation threshold value based on the confidence coefficient of each cost function;

taking the probability that the function value of each cost function is larger than a preset value as an evaluation error, and constructing a probability activation function based on the evaluation errors of all the cost functions and the probability activation threshold;

adjusting network parameters of a decision model based on the plurality of running track data sets and the probability activation function to obtain an adjusted decision model; and the adjusted decision model is used for planning the driving strategy according to the current state and the driving track of the vehicle in the driving process of the vehicle.

2. The method of claim 1, further comprising, prior to said grouping the travel tracks according to the strategy used in accordance with the first state in each travel track:

and dividing each running track into a plurality of running tracks with preset lengths by using a sliding window with preset lengths.

3. The method of claim 1 or 2, further comprising:

counting the number of the running tracks in each running track data set, determining a target running track data set with the number of the running tracks less than or equal to a preset threshold value, and setting the evaluation error of the target running track data set to be 0.

4. The method of claim 1 or 2, wherein the policy comprises one of a rule policy and a self-learning policy, and wherein calculating the probability activation threshold based on the confidence level of each cost function comprises:

calculating a probability value when the performance expectation of the self-learning strategy relative to the rule strategy is maximum, and taking the probability value as a value range of a probability activation threshold; where the performance expectation represents a confidence difference.

5. The method of claim 1, wherein the driving strategy planning based on the current state and the driving trajectory of the vehicle comprises:

determining that the vehicle driving strategy is a rule strategy in response to the condition that the current state of the vehicle does not exist in the state space;

randomly generating a variable, estimating a rule strategy representation value by collecting the driving track of the vehicle using the rule strategy in the process of continuously driving the vehicle, and triggering to update the strategy if the variable is detected to be larger than the rule strategy representation value and the accumulated value of 1.

6. A vehicle driving strategy processing apparatus characterized by comprising:

the collection grouping module is used for collecting the running tracks of a plurality of vehicles in different states, and grouping the running tracks according to a strategy used in a first state in each running track to obtain a plurality of running track data sets;

the confidence coefficient calculation module is used for constructing a value function for each driving track data set so as to calculate the confidence coefficient of each value function, and further calculating a probability activation threshold value based on the confidence coefficient of each value function;

the function construction module is used for taking the probability that the function value of each value function is larger than a preset value as an evaluation error, and constructing a probability activation function based on the evaluation errors of all the value functions and the probability activation threshold;

the parameter adjusting module is used for adjusting network parameters of the decision model based on the plurality of running track data sets and the probability activating function to obtain an adjusted decision model; and the adjusted decision model is used for planning the driving strategy according to the current state and the driving track of the vehicle in the driving process of the vehicle.

7. The apparatus of claim 6, wherein the function building module is configured to:

counting the number of the running tracks in each running track data set, determining a target running track data set with the number of the running tracks less than or equal to a preset threshold value, and setting the evaluation error of the target running track data set to be 0.

8. The apparatus of claim 5, wherein the policy comprises one of a rules policy and a self-learning policy, and wherein the confidence calculation module is configured to:

calculating a probability value when the performance expectation of the self-learning strategy relative to the rule strategy is maximum, and taking the probability value as a value range of a probability activation threshold; where the performance expectation represents a confidence difference.

9. An electronic device, comprising:

one or more processors;

a storage device for storing one or more programs,

when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-5.

10. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-5.

Images