CN115171388A - Multi-intersection travel time collaborative optimization method for intelligent internet vehicle - Google Patents

Abstract

The invention belongs to the field of artificial intelligence, and particularly relates to a multi-intersection travel time collaborative optimization method for an intelligent internet vehicle; the intelligent internet vehicle is combined with a reinforcement learning method, a new reward function is provided, the average speed of vehicles in the traffic system is used as a reward value by the reward function, and punishment is carried out on the condition that the deceleration of vehicle running in the traffic system is lower than a comfort level parameter. And the vehicle is configured with an IDM following model to simulate an artificial vehicle by utilizing SUMO software, the calculation of comfort level parameters in the IDM following model is improved, and the comfort level parameters of the IDM following model are calculated according to the current traffic flow, the lane length and the expected speed of the vehicle. The invention proves that the traffic flow rate is effectively improved and the traffic stability is improved by combining the reinforcement learning method with the ICV. And the ICV after reinforcement learning can effectively reduce the frequent change of the vehicle acceleration.

Description

A collaborative optimization method of multi-intersection travel time for intelligent networked vehicles

technical field

The invention belongs to the field of artificial intelligence, and in particular relates to a multi-intersection travel time collaborative optimization method for intelligent networked vehicles.

Background technique

The Internet of Vehicles fully embodies the concept of the Internet of Things in the field of transportation. With the help of modern electronic sensing, radio communication and control technology, the Internet of Vehicles connects the three information topics of people, vehicles and the environment through a network, and realizes intelligent control of vehicles under the support of big data. ICV is the general term for all connected cars in the mature stage of the development of IoV technology. In my country, the production of Tesla vehicles proves the fact that intelligent networked vehicles and artificial vehicles will coexist for a long time. This phenomenon increases the difficulty of ICV collaboration. The emergence of intelligent networked vehicles enables vehicles to obtain basic status information such as the position, speed, and acceleration of surrounding vehicles, and can even obtain information through a centralized processor to adjust their own status. At present, the multi-agent reinforcement learning method is generally used to solve the problems in the traffic system in the mixed traffic scene of intelligent networked vehicles and artificial vehicles.

Using deep reinforcement learning methods to allow machines to learn human behavior in virtual scenes, eventually produces an artificial agent that can learn to perform well in a variety of challenging tasks, but cannot solve DQN (deep reinforcement learning methods) method training Defects such as low efficiency and long time. In transportation systems, traffic congestion, which results in a lot of wasted time and slow traffic, is one of the main challenges that traffic authorities and traffic participants must overcome. Among the many traffic congestion problems, solving the problem of intersection congestion is the top priority. From this point of view, it is necessary to use a multi-agent reinforcement learning method to solve the problem of intersection congestion in mixed traffic flow.

There are generally two types of centralized processing methods and distributed processing methods to solve the problem of intersection congestion in the early days. The idea of the centralized processing method is to consider a cooperative environment and directly extend the single-agent algorithm to directly learn the output of a joint action, but it is not easy to give a single agent how to make decisions, resulting in vehicle acceleration. Frequent changes. The distributed processing method is that each agent learns its own reward function independently. For each agent, other agents are part of the environment, so it is often necessary to consider the non-stationary state of the environment. The prior art introduces that vehicles can send requests for intersections to the central processor one by one using a first-in-first-out strategy, which is then processed centrally by the central processor, and then sends confirmation requests to vehicles. Extending the scheme to a network of interconnected intersections in the sequel aims to explore the best routes to guide vehicles to the intersection to minimize their delay through the network. The idea of a reservation-based unsignalized intersection scheme is further developed and the FIFO (first in first out) queuing policy is relaxed. By relaxing the FIFO, it has better performance compared to the previous FIFO-based reservation-based scheme. A fuzzy control model derived from the FIFO queuing strategy. This model is that when a vehicle enters an intersection (or a certain intersection), the vehicle will send a pass request to the centralized processor, and the concentrator will also base on the vehicle information (position, vehicle size) at the intersection. etc.) to group the vehicles, the concentrator calculates the average waiting time of each group of vehicles according to the fuzzy rules, and then arranges the grouping sequence according to the rating level of each group. The vehicles are grouped according to fuzzy rules, and the vehicles send requests to the centralized controller to pass the request. However, this type of model actually imitates the traffic pattern of the signal lights, so that the vehicles can pass in an orderly manner. The problem of frequent changes, and at the same time weakens the effect of the car-following model on the vehicle.

SUMMARY OF THE INVENTION

Aiming at the deficiencies of the prior art, the present invention designs a multi-intersection travel time collaborative optimization method for intelligent networked vehicles.

A multi-intersection travel time collaborative optimization method for intelligent networked vehicles, which specifically includes the following steps:

Step 1: In SUMO, establish a traffic intersection scene where the main lane is three lanes, and the incoming and outgoing lanes are two lanes, and the vehicle speed limit and lane flow direction are set to simulate the real traffic intersection. Increase the instability parameter when driving in the lane to allow a sudden left turn when the car on the far right is approaching a left turn intersection;

Step 2: A traffic flow parameter needs to be set to control the number of vehicles entering the lane per hour. In the process of simulating the traffic intersection scene: add the car-following model and the lane-changing model to simulate human-driven vehicles in the real world, so that The vehicle judges whether its own vehicle needs to accelerate, decelerate, or change lanes according to the state of the vehicle ahead;

Step 2.1: Use the IDM car-following model, which uses the expected speed of the current vehicle and the distance from the leading vehicle as variables to calculate the optimal acceleration required by the current vehicle, specifically:

Among them, the desired speed v0, the desired distance s ^* , the time interval T, the minimum gap s ₀ , the acceleration index δ, the acceleration term α, the own speed v, and the comfort level b, the vehicle travel distance is the distance s from the vehicle ahead and the own vehicle Speed difference Δv compared with the preceding vehicle; a is the current vehicle acceleration;

And in order to improve the comfort of passengers riding in the vehicle, the comfort variable b is added; different b results in different expected distances s ^* , when the safety distance is exceeded, the larger expected distance will affect the efficiency of the entire traffic system; thus, in order to To maximize the optimal acceleration of the vehicle, choose to improve the comfort b, the formula is as follows:

F=Vρ

Among them, F is the traffic flow, V is the traffic speed, ρ is the traffic density, H is the lane length, h is the body length, and the comfort is b; the comfort is improved according to the traffic flow and traffic speed;

Step 2.2: Determine whether the current vehicle is changing lanes through the lane-changing model; four different lane-changing motives are clearly defined in the lane-changing model: Strategic change, Cooperative change, Tactical change, Obligatory change Obligation to change lanes;

Whether the lane change action can occur is determined by calculating the lane change demand according to the vehicle speed and the expected speed, and calculating the lane change urgency according to the vehicle speed, the expected speed, the distance to the leading vehicle and the lane occupancy rate;

Select the preferred alternative lane according to the lane change demand of the own vehicle; calculate the safe speed in the current lane, and combine the speed requirements of the alternative lane change; decide whether to change the lane according to the lane change demand of the own vehicle and the urgency of the lane change;

Step 3: Combine the vehicle as a multi-agent with the reinforcement learning method ppo, that is, an intelligent networked vehicle;

Take the vehicle speed, position, acceleration, and desired speed as the state space, and the acceleration as the action space; design in the ppo reward function; take the sum of the average speed of each vehicle as the reward value, and set the comfort as the condition, if the vehicle If the deceleration is less than the comfort level, a penalty will be added to the reward function; the specific formula is as follows:

The reward function re consists of three aspects: the average speed V _averge , the instantaneous braking deceleration A _real , and the collision penalty Z; w ₁ , w ₂ , w ₃ represent the average speed, the weight of the collision between the two vehicles and the excessive deceleration, A _max is the set maximum acceleration; under normal vehicle driving conditions, when the vehicle decelerates, the deceleration will increase slowly in a certain area, but the leading vehicle in front suddenly stops, and the own vehicle will be forced to brake, at this time, the acceleration is set to instantaneous Braking and deceleration; with the continuous integration of intelligent networked vehicles, the overall average speed is improved.

Beneficial technical effects of the present invention:

With the increase of urbanization rate, traffic congestion is concentrated at intersections. In order to improve traffic flow rate and improve traffic stability, it is necessary to solve the mixed traffic flow composed of intelligent networked vehicles and non-connected vehicles in multi-intersections. To solve the problem of traffic congestion, the present invention proposes a multi-intersection travel time collaborative optimization method for intelligent networked vehicles. Combining ICV with reinforcement learning method, a new reward function is proposed. The reward function takes the average speed of the vehicle in the traffic system as the reward value, and punishes the situation that the deceleration of the vehicle is lower than the comfort parameter in the traffic system. . And use SUMO software to configure the vehicle with IDM car following model to simulate artificial vehicles, and improve the calculation of comfort parameters in the IDM car following model. The invention proves that the combination of the reinforcement learning method and the ICV can effectively improve the traffic flow rate and improve the traffic stability. And it is verified that the ICV after reinforcement learning can effectively reduce the frequent change of vehicle acceleration.

Description of drawings

1 is a schematic diagram of the environment structure of a simulated intersection environment of a multi-intersection travel time collaborative optimization method for intelligent networked vehicles according to an embodiment of the present invention:

FIG. 2 is a flowchart of a method for collaborative optimization of multi-intersection travel time for an intelligent networked vehicle according to an embodiment of the present invention.

3 is a traffic scene of a continuous T-shaped intersection according to an embodiment of the present invention;

FIG. 4 is an experimental result of an artificially driven vehicle simulated by a car-following model and a lane-changing model according to an embodiment of the present invention;

FIG. 5 is the experimental result of combining the vehicle as a multi-agent with the reinforcement learning method ppo according to the embodiment of the present invention;

FIG. 6 is a schematic diagram of the relationship between traffic flow and speed according to an embodiment of the present invention.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings and embodiments;

The method of the invention is realized by using the Rllab reinforcement learning algorithm in combination with the SUMO software and the FLOW framework.

The experiment is simulated by SUMO simulation software and FLOW secondary open source framework. The experimental operation structure is shown in Figure 1. The simulation scene adopts the continuous T-intersection traffic scene as shown in Figure 3, and allows the vehicle to temporarily change the driving route when it is about to pass the intersection. . The simulation environment and parameter settings are introduced, and the algorithm analysis is carried out at the same time.

A multi-intersection travel time collaborative optimization method for intelligent networked vehicles, as shown in FIG. 2, specifically includes the following steps:

Step 1: Create a traffic intersection scene in SUMO where the main lane is three lanes, and the incoming and outgoing lanes are two lanes, and the vehicle speed limit and lane flow direction are set to simulate the real traffic intersection situation, for example "from ":"edge_{}".format(i),"to":"edge_{}".format(i+1). Added instability parameter when merging out of lanes to allow a sudden left turn when the car on the far right is approaching a left turn intersection;

Simulation environment and parameter settings:

In the experiment, the required traffic scene is generated by connecting the public interface in the FLOW framework with the Traci interface in the SUMO simulation software, and the instability parameter 0.1 is added when exiting the lane, allowing the rightmost car to suddenly turn left when approaching the left-turn intersection. Then use reinforcement learning to generate a control policy for ICV, train 100 iterations, 200 time slots in a round, each time slot is 0.2 seconds long, and optimize it with an optimizer. And record iteration-related metrics.

The hyperparameters of the experimental setup are shown in Table 1. The valid figures of the same type of data must be consistent.

Table 1 Hyperparameters set by the experiment;

Step 2: A traffic flow parameter needs to be set to control the number of vehicles entering the lane per hour. In the process of simulating the traffic intersection scene: add the car-following model and the lane-changing model to simulate human-driven vehicles in the real world, so that The vehicle judges whether its own vehicle needs to accelerate, decelerate, or change lanes according to the state of the vehicle ahead;

Step 2.1: Use the IDM car-following model, which uses the expected speed of the current vehicle and the distance from the leading vehicle as variables to calculate the optimal acceleration required by the current vehicle, specifically:

Among them, the desired speed v0, the desired distance s ^* , the time interval T, the minimum gap s ₀ , the acceleration index δ, the acceleration term α, the own speed v, and the comfort level b, the vehicle travel distance is the distance s from the vehicle ahead and the own vehicle Speed difference Δv compared with the preceding vehicle; a is the current vehicle acceleration;

And in order to improve the comfort of passengers riding in the vehicle, the comfort variable b is added; different b results in different expected distances s ^* , when the safety distance is exceeded, the larger expected distance will affect the efficiency of the entire traffic system; thus, in order to To maximize the optimal acceleration of the vehicle, choose to improve the comfort b, the formula is as follows:

F=Vρ

Among them, F is the traffic flow, V is the traffic speed, ρ is the traffic density, H is the lane length, h is the body length, and the comfort is b; the comfort is improved according to the traffic flow and traffic speed;

The appearance of comfort increases the stability of the transportation system to a certain extent. However, excessive comfort increases resource consumption for the transportation system. For the traffic system, there is a linear relationship between the traffic flow and the traffic speed as shown in Figure 6. Initially, the vehicles flow into the traffic system. As the traffic speed increases, the traffic density increases. When the increased traffic volume tends to be stable, the traffic flow The increase in speed will reduce the volume of traffic experienced by the traffic system. Therefore, the improved calculation comfort level will calculate the optimal comfort level according to the traffic flow and the speed of the traffic flow in the traffic system.

Step 2.2: Determine whether the current vehicle is changing lanes through the lane-changing model; four different lane-changing motives are clearly defined in the lane-changing model: Strategic change, Cooperative change, Tactical change, Obligatory change Obligation to change lanes;

The control of the vehicle is divided into longitudinal control and lateral control. The longitudinal control selects the IDM car-following model, and the lateral control selects the LC2013 lane changing model in SUMO. In a complex multi-vehicle road network, most vehicles need to change lanes in the same direction, which not only improves the efficiency of the entire traffic system, but also reduces the frequent changes in vehicle acceleration. The speed of the vehicle is mainly determined by the leading lane. When the current vehicle wants to change lanes, in order to prevent collisions with vehicles ahead and behind the target lane, it will only perform the lane-changing action when the target lane has enough physical space. During the simulation process,

Whether the lane change action can occur is determined by calculating the lane change demand according to the vehicle speed and the expected speed, and calculating the lane change urgency according to the vehicle speed, the expected speed, the distance to the leading vehicle and the lane occupancy rate;

Select the preferred alternative lane according to the lane change demand of the own vehicle; calculate the safe speed in the current lane, and combine the speed requirements of the alternative lane change; decide whether to change the lane according to the lane change demand of the own vehicle and the urgency of the lane change;

When the vehicle has to change lanes to make the next road on its driving path, it is called strategic lane change. For example, the traffic system is three lanes, and the vehicle is in the second lane at this time, but the vehicle needs to turn after unit time, then the vehicle even stops at this time. Also have to wait for a lane change. When the own vehicle is informed by other vehicles of the congestion ahead, it is called cooperative lane change. For example, the leading vehicle of the own vehicle needs to perform a strategic lane change to stop, and the own vehicle needs to change lanes according to the change of the obtained leading speed. , thereby changing lanes. The tactical lane change motivation arises from the slow speed of the leading vehicle that the ego vehicle wants to avoid following. The obligatory lane change motivation is the lane change that occurs when the own vehicle does not affect other faster vehicles.

Under the condition that the traffic volume threshold is not exceeded, the situation of lane change is more obvious. In a certain respect, the generation of the lane change model slows down the generation of congestion and reduces the congestion period. The premise of all vehicle lane changes in this simulation is not to affect the vehicle state after the lane change. And for the comfort parameter of the car following model calculated by the research, the compatibility of the comfort is increased.

Step 3: Combine the vehicle as a multi-agent with the reinforcement learning method ppo, that is, an intelligent networked vehicle;

Take the vehicle speed, position, acceleration, and desired speed as the state space, and the acceleration as the action space; design in the ppo reward function; take the sum of the average speed of each vehicle as the reward value, and set the comfort as the condition, if the vehicle If the deceleration is less than the comfort level, a penalty will be added to the reward function; the specific formula is as follows:

The reward function re consists of three aspects: the average speed V _averge , the instantaneous braking deceleration A real , and the collision penalty Z; _{w 1} , w ₂ , w ₃ represent the average speed, the weight of the collision between the two vehicles and the excessive deceleration, A _max is the maximum acceleration set; under normal vehicle driving conditions, when the vehicle decelerates, the deceleration will increase slowly in a certain area, but the leading vehicle in front suddenly stops, and the own vehicle will be forced to brake. At this time, the acceleration is set to Instantaneous braking deceleration; with the continuous integration of intelligent networked vehicles, the overall average speed is improved.

In the experiment, by adding different proportions of vehicles with reinforcement learning algorithm to the traffic system, the vehicles with reinforcement learning algorithm can significantly reduce the problem of frequent change of vehicle speed. And compare different reward functions.

The experimental results are shown in Figure 4 for vehicles without reinforcement learning methods, and Figure 5 for vehicles combined with reinforcement learning methods. First, compare the impact of reinforcement learning algorithms on the traffic system, as shown in Figure 4 and Figure 5. The image data counts the average speed of each vehicle from the beginning to the end of the entire driving process in the traffic system. It is obvious that the effect of Figure 5 is better than that of Figure 4. The overall average speed of the vehicle is increased by 2.5 times after combining with the reinforcement learning method. Figure 4, without the reinforcement learning method, the vehicle travels at a slower speed and the traffic congestion is serious. After combining the reinforcement learning method, as shown in Figure 5, the vehicle can effectively increase the traffic efficiency of the vehicle in the traffic system.

The experiment uses the SUMO simulation software to simulate the traffic state in the real scene, and studies the use of the deep reinforcement learning PPO algorithm to solve the problem of intersection congestion in the mixed traffic flow, and effectively improves the stability of the traffic system and the traffic flow. And the experiment added different proportions of intelligent networked vehicles into the transportation system, which proved the positive effect of the development of intelligent networked vehicles on smart cities.

Claims (3)

1. A multi-intersection travel time collaborative optimization method for an intelligent internet vehicle is characterized by specifically comprising the following steps:

step 1: establishing a traffic intersection scene with three main lanes and two merging lanes in the SUMO, and establishing vehicle limiting speed and lane flow direction to simulate the situation of a real traffic intersection, adding unstable parameters when merging the lanes, and allowing a vehicle on the rightmost side to suddenly turn left when approaching a left-turn intersection;

and 2, step: setting a traffic flow parameter for controlling the number of vehicle inrush lanes per hour, in the process of simulating traffic intersection scenes: adding a following model and a lane changing model to simulate a human-driven vehicle in the real world, so that the vehicle judges whether the vehicle needs to accelerate, decelerate and change lanes according to the state of the vehicle in front;

and step 3: combining a vehicle as a multi-agent with a reinforcement learning method ppo, namely, an intelligent networking vehicle; along with the continuous convergence of the intelligent network connection vehicle, the integral average speed is improved.

2. The method for collaborative optimization of the multi-intersection travel time of the intelligent networked vehicle according to claim 1, wherein the step 2 specifically comprises:

step 2.1: using an IDM following model, wherein the model takes the expected speed of the current vehicle and the distance between the current vehicle and a leading vehicle as variables to calculate the optimal acceleration required by the current vehicle, and specifically comprises the following steps:

wherein the desired velocity v0, the desired distance s ^* Time interval T, minimum gap s ₀ Acceleration index δ, acceleration term α, self-speed v, and comfort b, vehicle travel distance, i.e. distance s from the vehicle in front and speed difference Δ v of the self-vehicle compared to the vehicle in front; a is the current vehicle acceleration;

and increasing a comfort variable b in order to improve the comfort of passengers riding the vehicle; b are different so that different desired distances s result ^* The greater the expected distance beyond the safe vehicle distance will affect the efficiency of the overall traffic system; in order to maximize the optimal acceleration of the vehicle, the comfort level b is improved, and the formula is as follows:

F＝Vρ

wherein F is traffic flow, V is traffic flow speed, rho is traffic flow density, H is lane length, H is vehicle body length, and comfort level is b; comfort is improved according to traffic flow and traffic speed;

step 2.2: judging whether the current vehicle changes lanes or not through a lane changing model; four different lane changing motivations are clearly defined in the lane changing model: stratetic change Strategic lane change, cooperative change collaborative lane change, tactical change Tactical lane change, obblication change obligation lane change;

whether lane changing action can be carried out is determined by calculating lane changing requirements through the vehicle speed and the expected speed and calculating lane changing urgency according to the vehicle speed, the expected speed, the distance from the leading vehicle and the lane occupancy rate;

selecting a priority alternative lane according to the lane change requirement of the vehicle; calculating the safe speed of the current lane and combining the speed requirement of the alternative lane change; and determining whether to change lanes or not according to the lane changing requirement of the vehicle and the lane changing emergency degree.

3. The method for collaborative optimization of multi-intersection travel time of the intelligent networked vehicle as claimed in claim 1, wherein the step 3 combines the vehicle as a multi-agent with a reinforcement learning method ppo specifically as follows:

taking the vehicle speed, position, acceleration and expected speed as a state space and taking the acceleration as an action space; designing in a ppo reward function; taking the sum of the average speeds of each vehicle as reward value, setting comfort level as condition, and adding punishment to reward function if the deceleration of the vehicle is less than the comfort level; the specific formula is as follows:

the reward function re consists of three aspects: average velocity V _averge Instantaneous brake deceleration A _real A collision penalty Z; w is a ₁ ，w ₂ ，w ₃ Weight representing average speed, collision of two vehicles and over-rapid deceleration, A _max Is the set maximum acceleration; (ii) aUnder the condition of normal vehicle running, when the vehicle decelerates, the deceleration can be slowly increased in a certain area, but the front leader vehicle stops suddenly, the self vehicle brakes forcibly, and the acceleration is set as the instantaneous braking deceleration; along with the continuous convergence of the intelligent network connection vehicle, the integral average speed is improved.

Images