CN114328547A - Vehicle-mounted high-definition map data source selection method and device - Google Patents

Abstract

The invention provides a method and device for selecting a vehicle high-definition map data source. The method combines a reinforcement learning method to construct an asynchronous data source selection framework. The framework is divided into offline training and online selection parts, and the offline part is responsible for using a deep reinforcement learning algorithm for neural network. In the training of the model, the online part uses the neural network parameters synchronized from the offline part to select the data source, and realizes the parallel execution of data source selection, experience trajectory collection and model training. Through the present invention, the problem of throughput reduction caused by the data source transmission process can be avoided, frequent data source switching can be avoided, and the optimal vehicle high-definition map data source can be effectively selected.

Description

A method and device for selecting a vehicle high-definition map data source

technical field

The invention relates to the technical field of deep learning, and in particular, to a method and device for selecting a vehicle high-definition map data source.

Background technique

With the widespread deployment of information infrastructure and the rapid development of in-vehicle sensing technology, autonomous driving has emerged as a promising direction to revolutionize current automotive technology. Autonomous driving is the future development trend of intelligent technology. It uses a large number of sensors to form a perception system to perceive environmental information around the vehicle. According to the road structure, vehicle position, obstacle status and other information obtained by the perception system, the automatic electronic control system is implemented to control the speed and direction of the vehicle, so that it can drive safely and reliably on the road. Unlike traditional electronic maps, autonomous vehicles require high-definition (HD) maps to support lane-level navigation. The high-definition map is a thematic map, which can be divided into three layers: the road model layer, the lane model layer and the positioning model layer. Specifically, the road model is used for navigation planning; the lane model is used for route planning based on sensing the current road and traffic conditions; the localization model is used to locate the vehicle in the map, and the lane model can only assist the vehicle when the vehicle is accurately positioned on the map perception. To achieve autonomous driving, high-definition maps are an essential component, but compared with traditional electronic maps, high-definition maps have a relatively large amount of data. Therefore, it is impractical to store all high-definition maps in the car, and the road information and traffic information are changing in real time, and the high-definition maps should be distributed in real time, with low latency and high reliability.

The traditional high-definition map selection and distribution method is to use the RTT indicator to determine the data source, but as the number of vehicles within the coverage increases, the traditional solution uses a communication model (Vehicle-to-Infrastructure (V2I) or Vehicle-to-Vehicle (V2V)) to select the data source, where the throughput will be significantly reduced; in addition, the traditional scheme selects the data source only by measuring the round trip time (RTT) between the data source and the vehicle, in this case, the vehicle state is changing in real time, Especially in complex mobile scenarios, since other types of vehicle information (such as speed, direction) are not considered, the measurement of RTT cannot guarantee the best data source selection results; moreover, due to mobility problems, the method of implementing traditional solutions , there will be frequent data source switching, resulting in frequent RTT updates and inefficient data transmission. In conclusion, the existing solutions cannot effectively judge the quality of the currently selected data source, and lead to inaccurate RTT measurement and inefficient data transmission as the vehicle moves frequently.

SUMMARY OF THE INVENTION

The present invention provides a method and device for selecting a vehicle high-definition map data source, aiming to select an optimal vehicle high-definition map data source for the automatic driving of the vehicle.

To this end, the first object of the present invention is to propose a method for selecting a vehicle high-definition map data source, including:

constructing a vehicle high-definition map data source selection network, wherein the vehicle high-definition map data source selection network includes an offline training network and an online selection network;

Using the state information of different existing vehicle-mounted high-definition map data sources as a training data set, the offline training network is trained, and after the training is completed, the network parameters of the offline training network are applied to the online selection network;

During the driving of the autonomous vehicle, the real-time received status information of multiple on-board high-definition map data sources is input into the online selection network after training, and the output result is the selection result of on-board high-definition map data sources.

Wherein, the offline training network and the online selection network of the vehicle high-definition map data source selection network adopt DDQN neural network; wherein, the steps of training the offline training network include:

Collect the vehicle high-definition map data source and extract state information through the collector of the offline training network, and divide the training set and the test set as the training data of the offline training network;

constructing the offline training network; the offline training network includes two reinforcement learning networks DQN, and the two reinforcement learning networks DQN are trained synchronously;

Input the training set into two reinforcement learning networks DQN for training, optimize and update the parameters of the first reinforcement learning network DQN, find the action with the largest Q value in the first reinforcement learning network DQN, and use the second reinforcement The learning network DQN calculates the Q value that meets the requirements;

Construct the loss function of the offline training network to characterize the difference between the Q value of real-time learning and the target Q value, determine that the offline training network is trained when the loss function is the smallest, and input the test set into the offline training network that has been trained for verification. training accuracy.

Wherein, after the offline training network training is completed, the network parameters of the offline training network are applied to the online selection network; after the online selection network performs data source selection on the vehicle high-definition map data source, experience information is generated and sent to the offline training network for the offline training network training iterations.

Wherein, the step of receiving the status information of multiple vehicle high-definition map data sources in real time includes:

The autonomous vehicle sends out probe interest packets to find potential in-vehicle high-definition map data sources and status information;

After receiving the vehicle high-definition map data source, it is filtered through the filter of the online selection network to obtain the vehicle high-definition map data source set, which is used as the input of the online selection network.

Among them, the action formula of the first reinforcement learning network DQN to find the maximum Q value is expressed as:

a ^max (s', ω)=argmax _a' Q(s', a, ω) (1)

Among them, s represents the state, a represents the action, and ω is the weight parameter;

Use the action a ^max (s', ω) to calculate in the second reinforcement learning network DQN to obtain the target Q value, and the formula is expressed as:

y=r+γQ′(s′, argmax _a′ Q(s′, a, ω), ω ⁻ ) (2)

where γ is the discount factor and r is the reward.

Among them, the loss function formula of offline training network is expressed as:

Among them, m represents the number of training times;

Use the balanced update method to update the weight of the second reinforcement learning network DQN, and the smooth update calculation is shown in formula (4):

ω ^- ←l*ω+(1-l)ω ^- (4)

Among them, l represents the update rate, l<<1, ω ^- is the weight parameter of the second reinforcement learning network DQN.

Wherein, before selecting the data source of the vehicle high-definition map, the step of triggering the selection of the vehicle high-definition map data source is included; judging that the reasons for triggering the selection of the vehicle high-definition map data source include at least: selecting a data source for the autonomous vehicle when the vehicle is initialized; or Choose a new data source for the autonomous vehicle when the quality of the connected link deteriorates or when the connection is disconnected;

When determining whether the quality of the connected link deteriorates or the connection is disconnected, calculate the packet loss rate of the current autonomous vehicle connection data source, use the current timestamp as the random seed, and use the random function to calculate the hit switching probability, which is expressed as:

，则表示概率命中执行切换数据源；公式表示为：Generate random numbers within 100 by the rand() function, if

, it means that the probability hits the execution switch data source; the formula is expressed as:

Among them, H represents the data source switching judgment flag, P _max represents the maximum packet loss rate, that is, when the link packet loss rate is greater than the maximum packet loss rate, the probability calculation method is no longer used for switching judgment, and the data source switching is performed directly.

Among them, after receiving the vehicle high-definition map data source, the steps of screening through the filter of the line selection network include:

The selector of the line selection network receives the state information s _{t of the i-th data source at time t, i} =(N _{t, i} , M _{t, i} , V _{t, i} , D _{t, i} ), where N _{t, i} represents the round-trip time RTT between the data source and the vehicle of the i-th data source at time t; M _{t, i} represents the time interval for the i-th data source to send data packets; V _{t, i} represents the i-th data source at t Driving speed at time; D _{t, i} represents the distance between the i-th data source and the vehicle sending the request at time t;

N _{t, i} represents the smoothed RTT value, the smaller the RTT, the smaller the round-trip delay of the data source, and the better the network performance; when the RTT value from the same data source is obtained multiple times, the RTT smoothing process is more reliable. High, you can use the smoothing method in the Jacobson/Karels algorithm, and the calculation formula is as follows:

N _t,i =u*N _t-1,i +e*(R _t,i -N _t-1,i ) (7)

Among them, R _{t, i} represents the currently observed instantaneous RTT value, u=1, e=0.125;

M _{t, i} represents the smoothing interval time. In the case of the same bandwidth, the larger the interval is, the more idle the data source is and the larger the remaining available bandwidth; on the contrary, the smaller the interval is, the smaller the remaining available bandwidth of the data source is. The calculation formula is: :

M _t,i =(1-σ)*M _t-1,i +σ*(Data _t,i -Data _t-1,i ) (8)

Among them, σ=0.5, Data _{t, i} represents the current state information data sending time, Data _{t-1, i} represents the last state information data sending time, the subtraction of the two is the time interval;

V _t,i indicates the speed of the vehicle, V _{t, i} > 0 indicates that the corresponding data source and the requesting automatic driving vehicle travel in the same direction, V _{t, i} < 0 indicates that the corresponding data source and the requesting automatic driving vehicle travel in the opposite direction; the lower the speed, the The more stable the data source;

D _t,i represents the distance between the i-th data source and the autonomous vehicle that sends the request at time t, D _{t, i} > 0 indicates that the corresponding data source is in front of the autonomous vehicle, and D _{t, i} < 0 indicates that the corresponding data source is in Behind the autonomous vehicle; the closer the distance, the higher the stability of the data source;

Use the different status information contained as a basis for sorting, and filter the data sources with the best status; each of the 4 statuses can select an optimal data source, and the 4 best data sources contain a total of 4 groups of 16 status information data. , as the online selection network input.

Wherein, in the steps of sorting the data sources with the best state by sequentially using the different state information contained as a basis, the steps include:

Calculate the optimal value for each state, i.e.

Compute the score for choosing a data source when performing action a _t :

Among them, Max{G _{t , i} } represents the highest score of the data source i selected when performing the at action in the s _t state;

By adjusting the network parameters of the online selection network, the state parameters are normalized, the value range of the action parameters is adjusted, the mapping relationship between the data source and the state parameter value is constructed, and the data source with the highest score is determined as the final selection result.

Wherein, after the step of generating the experience information and sending it to the offline training network for the training iteration of the offline training network, it also includes the step of setting the reward corresponding to the data source; the reward function is expressed as:

表示链路吞吐量，

表示链路持续时间，

表示当前连接数据源RTT值，通过N_t，i计算平滑RTT；其它指标系数取值范围为1＜＜ρ≤2、

0＜＜φ≤0.5。in,

represents the link throughput,

represents the link duration,

Indicates the RTT value of the currently connected data source, and calculates the smoothed RTT by N _{t, i} ; the value range of other index coefficients is 1<<ρ≤2,

0<<φ≤0.5.

The second object of the present invention is to propose a vehicle high-definition map data source selection device, including:

Network building module: used to construct a vehicle high-definition map data source selection network, and the vehicle high-definition map data source selection network includes an offline training network and an online selection network;

The network training module is used to use the state information of different existing vehicle high-definition map data sources as training data sets to train the offline training network, and after the training is completed, apply the network parameters of the offline training network to the online training network. select network;

The data source selection module is used to input the status information of multiple in-vehicle high-definition map data sources received in real time into the online selection network that has been trained during the driving process of the autonomous vehicle, and the output result is the data source of the in-vehicle high-definition map data source. Select the result.

Different from the prior art, the vehicle high-definition map data source selection method provided by the present invention utilizes the content distribution mechanism and forwarding strategy of the NDN architecture, and combines the reinforcement learning method to construct an asynchronous data source selection framework, which is divided into offline training and online selection. Part, the offline part is responsible for using deep reinforcement learning algorithm to train the neural network model, while the online part uses the neural network parameters synchronized from the offline part to select the data source to realize the parallel execution of data source selection, experience trajectory collection and model training . Through the present invention, the problem of throughput reduction caused by the data source transmission process can be avoided, frequent data source switching can be avoided, and the optimal vehicle high-definition map data source can be effectively selected.

Description of drawings

The above and/or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of embodiments taken in conjunction with the accompanying drawings, wherein:

FIG. 1 is a schematic flowchart of a method for selecting a vehicle high-definition map data source provided by the present invention.

2 is a schematic structural diagram of a vehicle high-definition map data source selection network in a method for selecting a vehicle high-definition map data source provided by the present invention.

3 is a schematic diagram of a package structure of an exploration interest package and a data package in a method for selecting a vehicle high-definition map data source provided by the present invention.

FIG. 4 is a schematic structural diagram of a vehicle high-definition map data source selection device provided by the present invention.

Detailed ways

Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary, and are intended to explain the present invention and should not be construed as limiting the present invention.

The invention proposes a method for selecting a vehicle high-definition map data source, which adopts a deep reinforcement learning algorithm to learn and train a neural network according to the collected experience data to generate a selection strategy for data source selection. The map selection network consists of four main parts, namely state information, action, policy and reward. To simulate the dynamics of vehicle scenarios, vehicle speed, vehicle driving direction, interval time, and smooth RTT are used to represent the state information of the data source. To evaluate the action performance of selected data sources, a reward function is defined that takes into account link throughput, link duration, and RTT. In order to run the in-vehicle high-definition map selection network, an offline training and online decision-making mechanism is proposed to find suitable data sources in the in-vehicle scene, meaning that the neural network training process is performed offline and iteratively updated by using the collected empirical data to assist online selection. . In the offline training and online decision-making mechanism, an algorithm based on asynchronous reinforcement learning is designed to decouple the trajectory acquisition and neural network training, and realize the parallel execution of data source selection, empirical trajectory acquisition and model training. details as follows:

FIG. 1 is a schematic flowchart of a method for selecting a vehicle high-definition map data source according to an embodiment of the present invention. The method includes the following steps:

Step 101 , constructing a vehicle high-definition map data source selection network, where the vehicle high-definition map data source selection network includes an offline training network and an online selection network.

In order to support real-time data source selection, the vehicle high-definition map selection network constructed by the present invention includes an offline training network and an online selection network, as shown in FIG. 2 . The main purpose is to decouple data collection and model training, so that online algorithms can perform data source selection and data collection in real time, and offline algorithms can synchronize model training and iteration. In the offline part, a collector is used to collect empirical information from the interaction between the selector and the environment, which is then stored in the replay buffer library. Furthermore, the trainer uses a deep Q-network to train the policy from the collected experience set. In the online part, the selector obtains the data source information as state by observing the environment, and takes action through the policy. After each iteration, the selector shares its experience with the offline part of the collector and synchronizes the weights of the policy neural network.

Specifically, the network model structure constructed by the present invention is as follows:

Agent: The agent used to perform data source selection and switching decisions on the autonomous vehicle that needs to update the map. After the selection is triggered, the state information of the data source will be collected and processed, and then the corresponding selection action will be performed after obtaining the output from the neural network, and the agent will learn by interacting with the environment.

Status (s _t ): Due to the heterogeneity of data sources (RSU or vehicle), status information of various data sources can be obtained. The patent of the present invention uses four parameters to represent the current status information of data sources. The agent uses the state s _t,i to represent the state of the i-th data source obtained when the data source selection is triggered at time t, that is, s _t,i =(N _t,i ,M _t,i ,V _t,i ,D _{t, i} ). Among them, N _t,i represents the RTT value of the i-th data source at time t; M _t,i represents the time interval for the i-th data source to send data packets; V _t,i represents the travel of the i-th data source at time t Speed; D _{t, i} represents the distance between the i-th data source and the vehicle sending the request at time t, and the specific calculation method will be developed later.

动作参数为离散集合，然后再将具体的动作映射为对应的数据源选择，具体计算方法后续将展开。Action(at) _: at represents the action performed by the agent at time _t . In the design of the present invention, at _does not directly select the specific data source, but provides a selection method for the agent, that is, the neural network needs to determine the specific action parameters

Action parameters are discrete sets, and then specific actions are mapped to corresponding data source selections. The specific calculation method will be expanded later.

称为γ折扣累积奖赏，折扣因子γ∈(0，1]，时间越靠后奖赏权重越低，

是全部随机变量的期望。累积奖赏能够判断某个策略的优劣，越优秀的策略累积奖赏越高，具体计算方法和奖赏函数定义后续将展开。Reward (r _t ): After the agent completes the action at _t , if the data source selection is triggered again at time t+1, and the current data source state s _t+1 is obtained at the same time, the agent can calculate according to the state of the link The reward r(s _t , at _t ) for the previous action. All the agent has to do is maximize the cumulative reward, the expectation

It is called γ discount cumulative reward, the discount factor γ∈(0,1], the later the time, the lower the reward weight,

is the expectation of all random variables. The cumulative reward can judge the pros and cons of a strategy. The better the strategy, the higher the cumulative reward. The specific calculation method and definition of the reward function will be expanded later.

Strategy (π): The agent needs to interact with the environment to learn a strategy π that achieves better results, and guide the agent to choose the next action. The quality of the strategy is judged using the jackpot mentioned above. In the present invention, a deterministic strategy is used as the strategy search method, that is, π(s _t )=at , which means that the agent will execute the action at when it recognizes the state _{s t , so} the process of data source selection can be expressed as a series of data sources The mapping relationship between states and actions, and then through these mapping relationship pairs, the agent can choose the best action in the corresponding state. The present invention uses a deep reinforcement learning algorithm, and does not need to build a mapping table, and uses the actions shown in Figure 1 to select a neural network representation strategy. Action selection neural networks can easily process input states and output action parameter sets.

Step 102: Use the status information of different existing vehicle high-definition map data sources as training data sets to train the offline training network, and apply the network parameters of the offline training network to the online selection network after the training is completed. .

The offline training network and the online selection network of the vehicle high-definition map data source selection network adopt DDQN neural network; wherein, the steps of training the offline training network include:

Collect the vehicle high-definition map data source and extract state information through the collector of the offline training network, and divide the training set and the test set as the training data of the offline training network;

constructing the offline training network; the offline training network includes two reinforcement learning networks DQN, and the two reinforcement learning networks DQN are trained synchronously;

Input the training set into two reinforcement learning networks DQN for training, optimize and update the parameters of the first reinforcement learning network DQN, find the action with the largest Q value in the first reinforcement learning network DQN, and use the second reinforcement The learning network DQN calculates the Q value that meets the requirements;

Construct the loss function of the offline training network to characterize the difference between the Q value of real-time learning and the target Q value, determine that the offline training network is trained when the loss function is the smallest, and input the test set into the offline training network that has been trained for verification. training accuracy.

In the online training part, the present invention uses the DDQN algorithm as a reinforcement learning method to train a model based on Q-learning, so as to reduce the phenomenon of overestimation. There are two cores of Q-learning: OffPolicy and Temporal Difference (TD). Different strategies means that the strategy for selecting an action and the strategy for updating the Q value are not the same strategy. The strategy for selecting an action is a greedy strategy, and the strategy for updating the Q value is a deterministic strategy, that is, the action with the largest Q value is selected. The time difference refers to using the TD target to update the current value function. The TD target is the sum of future gains with decay. First of all, in order to improve the convergence of the algorithm, the present invention designs two neural networks for synchronous training, uses the current neural network Q (weight parameter ω) to be responsible for updating the weight parameters of the model; and uses the target neural network Q' (weight parameter ω ⁻ ) is responsible for calculating the Q value. In addition, in order to reduce the overestimation phenomenon caused by value iteration or parameter update (that is, the estimated function value is larger than the true value function, which eventually leads to model deviation), the present invention first finds the action with the largest Q value in the current neural network Q , the calculation method is as follows:

a ^max (s', ω)=argmax _a' Q(s', a, ω) (1)

Then, use the action a ^max (s', ω) to calculate in the target neural network Q', and finally obtain the target Q value y that meets the requirements. The calculation method is as follows:

y=r+γQ'(s', argmax _a' Q(s', a, ω), ω ^- (2)

where γ is the discount factor and r is the reward.

记录了每次迭代的过去经验，这对于实时车载网络中的模型训练至关重要。对于每次迭代，当前网络通过从经验回放库中随机采样m组的经验来训练，并且这些m组之间没有时间相关性。损失函数(L)的目标是最小化学习到的Q值和目标Q值之间的差异，计算如下：In the offline training part, the experience playback library saves the past experience groups

The past experience for each iteration is recorded, which is crucial for model training in real-time in-vehicle networks. For each iteration, the current network is trained by randomly sampling m groups of experiences from the experience replay library, and there is no temporal correlation between these m groups. The objective of the loss function (L) is to minimize the difference between the learned Q-value and the target Q-value, calculated as:

In addition, the present invention uses the balanced update method to update the weight ω ^- of the target network Q' to improve the stability of the target network Q', and the smooth update is calculated as follows:

ω ^- ←l*ω+(1-l)ω ^- (4)

Among them, l represents the update rate, and l<<1.

In the online selection part, actions are taken by observing the environmental state in the real-world vehicle scene during the operation of the vehicle, and the experience is collected into the collector for offline model training. In addition, the network structure of the selection strategy is the same as the neural network structure of the offline part, the input environment state, and the output is the action value. Each time a data source acquisition and switching request is initiated, the selector first obtains the weight ω trained by the current neural network Q from the trainer of the application layer, and then synchronously selects the neural network A for the action (the weight parameter is ω ^A ). Once the data selection mechanism is triggered, the user first sends a probe interest packet to discover potential data sources and status information. The filter in the selector will filter the data sources and their status, and eliminate the data sources that do not meet the requirements to obtain the data. Source set DS={DS1, DS2, DS3, DS4}. For each action selection, after the selection strategy network A outputs the action value, it needs to use the ∈ ~ greedy algorithm to determine whether to conduct random exploration, that is, the probability of (1-∈) selects the action with the largest value function, and the probability of ∈ randomly selects action. In the initial stage, this method can explore the action and avoid getting stuck in local optima. As training and exploration progress, we no longer need to explore frequently, and reduce the number of explorations by using the delta factor (downscaling). The convergence of the offline training algorithm will be facilitated as the exploration rate ∈ decreases, where ∈ = ∈ - δ*∈. After the selector takes the action a _t in the state s _t , the selector can obtain the reward r _t and the next state s _t+1 , and combine the experience information {s _t , at _t , r _t , s _t+1 , end} Synchronized to the offline part of the experience playback library. In addition, the online selector also needs to perform specific data request actions and obtain corresponding rewards at the network layer. Offline training and online selection of asynchronous algorithm design can improve the efficiency of algorithm training.

S103: During the driving process of the autonomous driving vehicle, input the status information of multiple in-vehicle high-definition map data sources received in real time into the online selection network that has been trained, and the output result is the selection result of the in-vehicle high-definition map data source.

In the present invention, there are two reasons for triggering the vehicle to select the data source: 1) the vehicle must select the data source for the vehicle during initialization (the default is based on the minimum RTT); 2) when the link quality of the connection is deterioration or disconnection. The entire data transmission process is divided into continuous and equal length time periods, the period of which adopts the Beacon frame interval time of the IEEE802.11 protocol, that is, 100ms is a period. The packet loss rate of the link is the most direct way to judge the quality of the link, but if the data source switching occurs once the packet loss occurs, it will cause frequent switching of the data source and affect the link throughput. The periodic probability triggering scheme of the packet loss rate is to determine whether to switch the data source according to the packet loss rate, so that the vehicle can switch to a new data source when the link quality is poor, and at the same time avoid the frequent switching of the data source. extra cost. In each cycle, first calculate the packet loss rate P of the current vehicle to the connected data source, and then use the current timestamp of the UNIX system (Timestamp) as a random seed to calculate the hit switching probability using a random function

则表示概率命中执行切换数据源。Generate random numbers within 100 through the rand() function. if

It means that the probability hits the execution switch data source.

Among them, H represents the data source switching judgment flag, P _max represents the maximum packet loss rate (default is 30%), that is, when the link packet loss rate is greater than the maximum packet loss rate, the probability calculation method is no longer used for switching judgment, and directly Data source switching.

If the selection mechanism is triggered, the vehicle will send Probe Interest packets, and the selector will collect state information by receiving packets. This means that the data source selection mechanism is activated. The selector takes the collection state as the network input, the selected data source as the action output, and finally obtains and records the corresponding reward in the vehicle. For each cycle, the purpose of the selector is to select the best data source in the current environment. If the selection mechanism is not triggered, the selector will not select the new data source and continue to transfer the current data from the data source, which means that the selection method based on the data source will not be activated.

Whenever data source selection is successfully triggered, the requesting vehicle will first broadcast a Probe Interest. Data sources are stored in other vehicles or infrastructure set up along the traffic route. The original link was not interrupted during the sending of Probe Interests. However, once a new data source is selected, the original link is broken and switched to the new data source. When the data source receives a Probe Interest, the data source adds additional state information (i.e., single-hop label, interval time, distance, and speed) to the returned data packet. After the probe interest packet is sent, the requested vehicle sets a waiting timer (50ms by default), and when the timer expires, the selector will be triggered to select the data source. Therefore, the present invention expands some new states to collect information on the basis of the existing probe Interest packets and Data packets, as shown in FIG. 3 . For Probe Interests, the vehicle will send a Probe packet with additional information, i.e. location and direction of travel. When a data source responds to a vehicle, the data source first calculates its distance from the requesting vehicle and additional information in the returned packet, namely interval time and speed.

After receiving the vehicle high-definition map data source, the steps of selecting the network filter through the line include:

The selector of the line selection network receives the state information s _{t of the i-th data source at time t, i} =(N _{t, i} , M _{t, i} , V _{t, i} , D _{t, i} ), where N _{t, i} represents the round-trip time RTT between the data source and the vehicle of the i-th data source at time t; M _{t, i} represents the time interval for the i-th data source to send data packets; V _{t, i} represents the i-th data source at t Driving speed at time; D _{t, i} represents the distance between the i-th data source and the vehicle sending the request at time t;

N _{t, i} represents the smoothed RTT value, the smaller the RTT, the smaller the round-trip delay of the data source, and the better the network performance; when the RTT value from the same data source is obtained multiple times, the RTT smoothing process is more reliable. High, you can use the smoothing method in the Jacobson/Karels algorithm, and the calculation formula is as follows:

N _t,i =u*N _t-1,i +e*(R _t,i -N _t-1,i ) (7)

Among them, R _{t, i} represents the currently observed instantaneous RTT value, u=1, e=0.125;

M _{t, i} represents the smoothing interval time. In the case of the same bandwidth, the larger the interval is, the more idle the data source is and the larger the remaining available bandwidth; on the contrary, the smaller the interval is, the smaller the remaining available bandwidth of the data source is. The calculation formula is: :

M _t,i =(1-σ)*M _t-1,i +σ*(Data _t,i -Data _t-1,i ) (8)

Among them, σ=0.5, Data _{t, i} represents the current state information data sending time, Data _{t-1, i} represents the last state information data sending time, the subtraction of the two is the time interval;

V _t,i indicates the speed of the vehicle, V _{t, i} > 0 indicates that the corresponding data source and the requesting automatic driving vehicle travel in the same direction, V _{t, i} < 0 indicates that the corresponding data source and the requesting automatic driving vehicle travel in the opposite direction; the lower the speed, the The more stable the data source;

D _t,i represents the distance between the i-th data source and the autonomous vehicle that sends the request at time t, D _{t, i} > 0 indicates that the corresponding data source is in front of the autonomous vehicle, and D _{t, i} < 0 indicates that the corresponding data source is in Behind the autonomous vehicle; the closer the distance, the higher the stability of the data source;

The state of the data source is used as the input of the neural network, and the number of inputs must be fixed, but the number of data sources that may be detected each time is not fixed, resulting in the number of states of the data source is not fixed. The method adopted in the present invention is to screen the data sources once, and then use each state as a basis for sorting. The data source with the best state can be filtered (for example, the data source DS1 has the smallest RTT, passed the filtering), and each of the four states can select an optimal data source. The 4 best data sources contain 4 groups of 16 states in total, and the selector takes these 16 states as input. If the selected data sources are less than 4, the data sources will be randomly selected as input supplements. For example, if a car only gets one data source, the selector will copy 4 states of this data source 4 times as 16 states for input. The main purpose of using this method is to fix the number of input states.

Data source selection action design. Even though there are only two types of data sources (i.e., infrastructure RSUs and vehicles), the vehicle may cause dynamic changes when it is a data source due to the movement of the vehicle. This means that the action space is not fixed in different vehicle scenarios. However, the number of data sources is limited by the coverage of vehicles and RSUs for the duration of the link. Therefore, the present invention adopts the idea of sorting to define actions. Based on the previous state definition, we first calculate the optimal value of each state, namely

Then by calculating the score for selecting a data source when performing action a _t as follows:

β_t，θ_t，μ_t表示相应状态参数，即动作参数)。Among them, Max{G _{t , i} } represents the highest score of the data source i selected when performing the at action in the s _t state. To evaluate the action score G _t for selecting data source i, the state parameters are normalized by adjusting the corresponding parameter values (

β _t , θ _t , μ _t represent corresponding state parameters, namely action parameters).

β_t∈{0，0.2，0.8}和θ_t，μ_t∈{0，0.5}，在这种情况下，随着每个状态的动作参数的变化，这四个集合共有64个组合，这意味着有64个动作可供智能体选择。然后，通过选择神经网络的输出结果构建数据源和动作参数值之间的映射关系。最后，当车辆在s_t状态下执行动作a_t时，选择的数据源i是最高评分G_t，i。选择神经网络输出动作参数，然后计算可以选择数据源的评分G_t，i，并映射到数据源集合DS＝{DS1，DS2，DS3，DS4}，最终选择具体的数据源。In the present invention, the value range of the action parameter is set as

β _t ∈ {0, 0.2, 0.8} and θ _t , μ _t ∈ {0, 0.5}, in this case, there are a total of 64 combinations of these four sets as the action parameters of each state change, which This means that there are 64 actions for the agent to choose from. Then, a mapping relationship between data sources and action parameter values is constructed by selecting the output results of the neural network. Finally, when the vehicle performs action a _t in state _st , the selected data source i is the highest score G _t,i . Select the neural network output action parameters, then calculate the score G _t,i that can select the data source, and map it to the data source set DS={DS1, DS2, DS3, DS4}, and finally select the specific data source.

Reward function design: To ensure that the network can learn from past experience, a corresponding reward is returned after each action is performed, representing the overall benefit of the agent following the policy. To understand the rewards of data sources, the present invention considers the following design principles: 1) Increase throughput as much as possible. Throughput is the most basic indicator of map data transmission, which means that the vehicle can acquire map data quickly and efficiently; 2) Extend the duration of the link. The purpose is to avoid the extra overhead caused by the vehicle frequently switching data sources, and maintaining the link stability can increase the throughput; 3) reduce the transmission delay. In autonomous driving scenarios, high-definition map data distribution has higher requirements on transmission delay. Low latency means that data sources can quickly respond to vehicle requests, reducing packet queuing time. Therefore, the present invention designs the following reward function:

表示链路吞吐量，

表示链路持续时间，

表示当前连接数据源RTT值，通过N_t，i计算平滑RTT。其它指标系数取值范围为1＜＜ρ≤2、

0＜＜φ≤0.5。in,

represents the link throughput,

represents the link duration,

Indicates the RTT value of the currently connected data source, and the smoothed RTT is calculated by N _{t, i} . The value range of other index coefficients is 1<<ρ≤2,

0<<φ≤0.5.

称为γ折扣累积奖赏。折扣因子γ∈(0，1]，它决定了奖励的时间尺度。时间越靠后奖赏权重越低，

是全部随机变量的期望。The purpose of designing the reward function in the present invention is to maximize the expected cumulative discount reward, and the calculation method is the expected

Called the gamma discount jackpot. The discount factor γ∈(0, 1], which determines the time scale of the reward. The later the time, the lower the reward weight,

is the expectation of all random variables.

In order to realize the above embodiment, the present invention also proposes a vehicle high-definition map data source selection device, as shown in FIG. 4 , including:

a network construction module 310, configured to construct a vehicle high-definition map data source selection network, the vehicle high-definition map data source selection network includes an offline training network and an online selection network;

The network training module 320 is configured to use the status information of different existing vehicle high-definition map data sources as training data sets to train the offline training network, and apply the network parameters of the offline training network to the offline training network after the training is completed. online selection network;

The data source selection module 330 is used to input the status information of multiple vehicle high-definition map data sources received in real time into the online selection network that has been trained during the driving process of the autonomous vehicle, and the output result is the data source of the vehicle-mounted high-definition map. selection result.

In the description of this specification, description with reference to the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples", etc., mean specific features described in connection with the embodiment or example , structure, material or feature is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and combine the different embodiments or examples described in this specification, as well as the features of the different embodiments or examples, without conflicting each other.

In addition, the terms "first" and "second" are only used for descriptive purposes, and should not be construed as indicating or implying relative importance or implying the number of indicated technical features. Thus, a feature delimited with "first", "second" may expressly or implicitly include at least one of that feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise expressly and specifically defined.

Any process or method description in the flowcharts or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing custom logical functions or steps of the process , and the scope of the preferred embodiments of the invention includes alternative implementations in which the functions may be performed out of the order shown or discussed, including performing the functions substantially concurrently or in the reverse order depending upon the functions involved, which should It is understood by those skilled in the art to which the embodiments of the present invention belong.

The logic and/or steps represented in flowcharts or otherwise described herein, for example, may be considered an ordered listing of executable instructions for implementing the logical functions, may be embodied in any computer-readable medium, For use with, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from and execute instructions from an instruction execution system, apparatus, or apparatus) or equipment. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport the program for use by or in connection with an instruction execution system, apparatus, or apparatus. More specific examples (non-exhaustive list) of computer readable media include the following: electrical connections with one or more wiring (electronic devices), portable computer disk cartridges (magnetic devices), random access memory (RAM), Read Only Memory (ROM), Erasable Editable Read Only Memory (EPROM or Flash Memory), Fiber Optic Devices, and Portable Compact Disc Read Only Memory (CDROM). In addition, the computer readable medium may even be paper or other suitable medium on which the program may be printed, as the paper or other medium may be optically scanned, for example, followed by editing, interpretation, or other suitable medium as necessary process to obtain the program electronically and then store it in computer memory.

It should be understood that various parts of the present invention may be implemented in hardware, software, firmware or a combination thereof. In the above-described embodiments, various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented by any one of the following techniques known in the art, or a combination thereof: discrete with logic gates for implementing logic functions on data signals Logic circuits, application specific integrated circuits with suitable combinational logic gates, Programmable Gate Arrays (PGA), Field Programmable Gate Arrays (FPGA), etc.

Those skilled in the art can understand that all or part of the steps carried by the methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, and the program can be stored in a computer-readable storage medium. When executed, one or a combination of the steps of the method embodiment is included.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing module, or each unit may exist physically alone, or two or more units may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware, and can also be implemented in the form of software function modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

The above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, and the like. Although the embodiments of the present invention have been shown and described above, it should be understood that the above-mentioned embodiments are exemplary and should not be construed as limiting the present invention. Embodiments are subject to variations, modifications, substitutions and variations.

Claims (10)

1. a vehicle high-definition map data source selection method, is characterized in that, comprises: constructing a vehicle high-definition map data source selection network, wherein the vehicle high-definition map data source selection network includes an offline training network and an online selection network; Using the state information of different existing vehicle-mounted high-definition map data sources as a training data set, the offline training network is trained, and after the training is completed, the network parameters of the offline training network are applied to the online selection network; During the driving of the autonomous vehicle, the real-time received status information of multiple on-board high-definition map data sources is input into the online selection network after training, and the output result is the selection result of on-board high-definition map data sources. 2. The vehicle-mounted high-definition map data source selection method according to claim 1, wherein the offline training network and the online selection network of the vehicle-mounted high-definition map data source selection network adopt DDQN neural network; The steps for network training include: Collect the vehicle high-definition map data source and extract state information through the collector of the offline training network, and divide the training set and the test set as the training data of the offline training network; constructing the offline training network; the offline training network includes two reinforcement learning networks DQN, and the two reinforcement learning networks DQN are trained synchronously; Input the training set into two reinforcement learning networks DQN for training, optimize and update the parameters of the first reinforcement learning network DQN, find the action with the largest Q value in the first reinforcement learning network DQN, and use the second reinforcement The learning network DQN calculates the Q value that meets the requirements; Construct the loss function of the offline training network to characterize the difference between the Q value of real-time learning and the target Q value, determine that the offline training network is trained when the loss function is the smallest, and input the test set into the offline training network that has been trained for verification. training accuracy. 3. The method for selecting a vehicle high-definition map data source according to claim 2, wherein after the offline training network training is completed, the network parameters of the offline training network are applied to the online selection network; After the vehicle high-definition map data source selects the data source, the experience information is generated and sent to the offline training network for the training iteration of the offline training network. 4. The method for selecting a vehicle high-definition map data source according to claim 1, wherein the step of receiving the status information of a plurality of vehicle-mounted high-definition map data sources in real time comprises: The autonomous vehicle sends out probe interest packets to find potential in-vehicle high-definition map data sources and status information; After receiving the vehicle high-definition map data source, it is filtered through the filter of the online selection network to obtain the vehicle high-definition map data source set, which is used as the input of the online selection network. 5. vehicle high-definition map data source selection method according to claim 2, is characterized in that, the action formula that described first reinforcement learning network DQN searches for maximum Q value is expressed as: a ^max (s', ω)=argmax _a' Q(s', a, ω) (1) Among them, s represents the state, a represents the action, and ω is the weight parameter; Use the action a ^max (s', ω) to calculate in the second reinforcement learning network DQN to obtain the target Q value, and the formula is expressed as: y=r+γQ′(s′, argmax _a′ Q(s′, a, ω), ω ⁻ ) (2) where γ is the discount factor and r is the reward. 6. vehicle high-definition map data source selection method according to claim 2, is characterized in that, the loss function formula of offline training network is expressed as:

Among them, m represents the number of training times; Use the balanced update method to update the weight of the second reinforcement learning network DQN, and the smooth update calculation is shown in formula (4): ω ^- ←l*ω+(1-l)ω ^- (4) Among them, l represents the update rate, l<<1, ω ^- is the weight parameter of the second reinforcement learning network DQN. 7. The method for selecting a vehicle high-definition map data source according to claim 1, wherein before selecting the vehicle-mounted high-definition map data source, it comprises the step of triggering the vehicle-mounted high-definition map data source selection; and judging to trigger the vehicle-mounted high-definition map data source selection The reasons include at least: the vehicle selects a data source for the self-driving vehicle during initialization; or the quality of the connected link deteriorates or a new data source is selected for the self-driving vehicle when the connection is disconnected; When determining whether the quality of the connected link deteriorates or the connection is disconnected, calculate the packet loss rate of the current autonomous vehicle connection data source, use the current timestamp as the random seed, and use the random function to calculate the hit switching probability, which is expressed as:

Generate random numbers within 100 by the rand() function, if

It means that the probability hits the execution switch data source; the formula is expressed as:

Among them, H represents the data source switching judgment flag, P _max represents the maximum packet loss rate, that is, when the link packet loss rate is greater than the maximum packet loss rate, the probability calculation method is no longer used for switching judgment, and the data source switching is performed directly. 8. The method for selecting a vehicle high-definition map data source according to claim 4, wherein after receiving the vehicle-mounted high-definition map data source, in the step of screening by a filter of the line selection network, comprising: The selector of the line selection network receives the state information s _{t of the i-th data source at time t, i} =(N _{t, i} , M _{t, i} , V _{t, i} , D _{t, i} ), where N _{t, i} represents the round-trip time RTT between the data source and the vehicle of the i-th data source at time t; M _{t, i} represents the time interval for the i-th data source to send data packets; V _{t, i} represents the i-th data source at t Driving speed at time; D _{t, i} represents the distance between the i-th data source and the vehicle sending the request at time t; N _{t, i} represents the smoothed RTT value, the smaller the RTT, the smaller the round-trip delay of the data source, and the better the network performance; when the RTT value from the same data source is obtained multiple times, the RTT smoothing process is more reliable. High, you can use the smoothing method in the Jacobson/Karels algorithm, and the calculation formula is as follows: N _t,i =u*N _t-1,i +e*(R _t,i -N _t-1,i ) (7) Among them, R _{t, i} represents the currently observed instantaneous RTT value, u=1, e=0.125; M _{t, i} represents the smoothing interval time. In the case of the same bandwidth, the larger the interval is, the more idle the data source is and the larger the remaining available bandwidth; on the contrary, the smaller the interval is, the smaller the remaining available bandwidth of the data source is. The calculation formula is: : M _t,i =(1-σ)*M _t-1,i +σ*(Data _t,i -Data _t-1,i ) (8) Among them, σ=0.5, Data _{t, i} represents the current state information data sending time, Data _{t-1, i} represents the last state information data sending time, the subtraction of the two is the time interval; V _t,i indicates the speed of the vehicle, V _{t, i} > 0 indicates that the corresponding data source and the requesting automatic driving vehicle travel in the same direction, V _{t, i} < 0 indicates that the corresponding data source and the requesting automatic driving vehicle travel in the opposite direction; the lower the speed, the The more stable the data source; D _t,i represents the distance between the i-th data source and the autonomous vehicle that sends the request at time t, D _{t, i} > 0 indicates that the corresponding data source is in front of the autonomous vehicle, and D _{t, i} < 0 indicates that the corresponding data source is in Behind the autonomous vehicle; the closer the distance, the higher the stability of the data source; Use the different status information contained as a basis for sorting, and filter the data sources with the best status; each of the 4 statuses can select an optimal data source, and the 4 best data sources contain a total of 4 groups of 16 status information data. , as the online selection network input. 9. The method for selecting a vehicle high-definition map data source according to claim 8, characterized in that, in the step of using the different state information contained in sequence as a basis for sorting, and screening the data source with the best state, comprising: Calculate the optimal value for each state, i.e.

Compute the score for choosing a data source when performing action a _t :

Among them, Max{G _{t , i} } represents the highest score of the data source i selected when performing the at action in the s _t state; By adjusting the network parameters of the online selection network, the state parameters are normalized, the value range of the action parameters is adjusted, the mapping relationship between the data source and the state parameter value is constructed, and the data source with the highest score is determined as the final selection result. It also includes the step of setting the reward corresponding to the data source; the reward function is expressed as:

in,

represents the link throughput,

represents the link duration,

Indicates the RTT value of the currently connected data source, and calculates the smoothed RTT by N _{t, i} ; the value range of other index coefficients is 1<<ρ≤2,

0<<φ≤0.5. 10. A vehicle high-definition map data source selection device, characterized in that, comprising: Network building module: used to construct a vehicle high-definition map data source selection network, and the vehicle high-definition map data source selection network includes an offline training network and an online selection network; The network training module is used to use the state information of different existing vehicle high-definition map data sources as training data sets to train the offline training network, and after the training is completed, apply the network parameters of the offline training network to the online training network. select network; The data source selection module is used to input the status information of multiple in-vehicle high-definition map data sources received in real time into the online selection network that has been trained during the driving process of the autonomous vehicle, and the output result is the data source of the in-vehicle high-definition map data source. Select the result.

Images