CN106097733A - A kind of traffic signal optimization control method based on Policy iteration and cluster - Google Patents

Abstract

The present invention proposes a kind of traffic signal optimization control method based on Policy iteration and cluster, the method relates to Intelligent Optimization Technique field, including step 1, selects control program, definition traffic behavior, control action, immediate yield andQValue；Step 2, sensing controls traffic light, records the traffic behavior of each sampling instant, control action and leave the vehicle number of stop line；Step 3, carries out pretreatment, then carries out k mean cluster traffic behavior；Step 4, uses Policy iteration method optimisation strategy in junction machine, and the barycenter obtained in the strategy that optimization is obtained and step 3 is saved in traffic signal control；Step 5, the control strategy using step 4 to obtain substitutes sensing and controls, and at the initial time in each sampling period, traffic signal control receives the traffic behavior that junction machine gathers, according to the discrete state inquiry control strategy that barycenter is corresponding, obtain control action and send to junction machine execution.

Description

A Traffic Signal Optimal Control Method Based on Policy Iteration and Clustering

technical field

The invention relates to the technical field of intelligent optimization.

Background technique

The optimal control of traffic signals is an important part of urban traffic management and control systems. The quality of traffic signal control strategies directly affects the transportation efficiency of the entire road network and people's travel experience. Therefore, various intelligent optimal control methods have been proposed and adopted. Attempts to apply to the optimization of traffic signal control strategies.

Dynamic programming is a method to solve the optimal control strategy, including value iteration and strategy iteration. Sampling the traffic state, phase and direct return of the strategy, and then using the samples to further optimize the control strategy, so it is very suitable for solving the problem of optimal control of traffic signals. When performing policy iteration on the traffic signal control problem, it is necessary to discretize continuous variables such as vehicle queue length. The traditional discretization method is to divide the entire state space evenly, but the actual states only gather in certain areas of the state space. Therefore, using k-means clustering to divide the area of state aggregation can use the same number Under the conditions of discrete states, higher discretization accuracy is guaranteed, thereby improving the optimization effect.

Contents of the invention

The purpose of the present invention is to use k-means clustering to discretize the traffic state to improve the optimization effect of strategy iteration and better optimize the control strategy of traffic lights at intersections. The ultimate goal is to increase the number of vehicles passing through the intersection per unit time, and reduce the number of stops and average delays caused by waiting for red lights.

The present invention firstly uses the induction control method to control the intersection traffic signal, and at intervals of a short unit time interval, the intersection machine records the vehicle queuing length of the current phase and the next phase, the number of vehicles leaving the stop line, and the traffic signal controller. Control the action. After the intersection machine collects enough samples, it performs k-means clustering on the vehicle queuing length in the samples to obtain discrete traffic states. Then use strategy iteration to optimize the strategy, and save the optimized strategy in the traffic signal controller. Every short unit time interval, the intersection machine sends the detected current phase and the vehicle queue length of the next phase to the traffic signal controller, and the traffic signal controller selects the appropriate The phase action for the intersection machine to execute.

The present invention proposes a traffic signal optimization control method based on strategy iteration and clustering, comprising the following steps:

Step 1. Select the signal control scheme to be optimized as fixed phase sequence control, define the traffic state as the vehicle queuing length of the current phase and the next phase, define the control action as maintaining the current phase or switch to the next phase, and define the direct return as a Variables related to the number of vehicles leaving the stop line in a single sampling period, define the state-action pair as a data vector composed of discrete traffic states and control actions, and define the Q value of each state-action pair to represent the corresponding discrete traffic state. The expected cumulative reward obtained after the control action defines the control strategy as the control action that should be executed for each discrete traffic state;

Step 2. The intersection machine sets the control strategy of the traffic signal controller as induction control. The minimum green light time and the maximum green light time are set as positive integer multiples of the sampling period. The unit green light extension time is the same as the sampling period. The phase action and the number of vehicles leaving the stop line are sampled and recorded. The sampling method is: record the traffic state, control action and the number of vehicles leaving the stop line in each sampling period at each sampling moment;

Step 3. After the intersection machine collects a specified number of samples, it discretizes the traffic state in the sample. The discretization method is: first normalize the traffic state obtained by sampling, and remove the traffic state whose distance exceeds the preset threshold , and then perform k-means clustering, number the obtained centroids, each centroid corresponds to a discrete traffic state, and represent the traffic state in the normalized sample with the number of the nearest centroid to obtain the corresponding discrete traffic state;

Step 4, the intersection machine uses strategy iteration to optimize the strategy, and saves the optimized strategy and the centroid obtained in step 3 in the traffic signal controller;

Step 5, the intersection machine sets the control strategy of the traffic signal controller as the control strategy obtained in step 4, and sets the decision-making cycle as the sampling cycle. At each decision-making moment, the traffic signal controller receives the traffic status detected by the intersection machine and performs Normalization, calculate the distance from the normalized traffic state to each centroid, find the nearest centroid, query the control strategy according to the discrete traffic state corresponding to the centroid, get the control action and send it to the intersection machine for execution.

The present invention has the advantage compared with prior art:

Before using strategy iteration to optimize the traffic signal control strategy, it is necessary to discretize the traffic state first—convert the continuous state space formed by the vehicle queuing lengths of the two phases into a discrete state space, and the accuracy of discretization will affect the optimization of strategy iteration Effect. In different typical time periods, the actual traffic state is not scattered throughout the state space, but concentrated in certain areas. The discrete traffic state obtained by using the k-means clustering algorithm only considers the area where the actual traffic state is concentrated, instead of taking into account the area where there is no actual traffic state like the traditional discretization method. Therefore, compared with the traditional method, after using the k-means clustering algorithm, the same number of discrete traffic states can be used to obtain higher discretization accuracy, thereby improving the optimization effect of strategy iteration.

Description of drawings

Figure 1 is a schematic diagram of traffic signal control at urban road intersections.

Fig. 2 is a flowchart of a traffic signal optimization control method based on strategy iteration and clustering.

1. The first geomagnetic vehicle detector; 2. The second geomagnetic vehicle detector; 3. The third geomagnetic vehicle detector; 4. The fourth geomagnetic vehicle detector; 5. The fifth geomagnetic vehicle detector; 6. The sixth geomagnetic vehicle detector Vehicle detector; 7. The seventh geomagnetic vehicle detector; 8. The eighth geomagnetic vehicle detector; 9. The ninth geomagnetic vehicle detector; 10. The tenth geomagnetic vehicle detector; 11. The eleventh geomagnetic vehicle detector; 12. The twelfth geomagnetic vehicle detector; 13. The thirteenth geomagnetic vehicle detector; 14. The fourteenth geomagnetic vehicle detector; 15. The fifteenth geomagnetic vehicle detector; 16. The sixteenth geomagnetic vehicle detector; 17. The seventeenth geomagnetic vehicle detector; 18. The eighteenth geomagnetic vehicle detector; 19. The nineteenth geomagnetic vehicle detector; 20. The twentieth geomagnetic vehicle detector; 21. The twenty-first geomagnetic vehicle detector ; 22, the twenty-second geomagnetic vehicle detector; 23, the twenty-third geomagnetic vehicle detector; 24, the twenty-fourth geomagnetic vehicle detector; 25, one lane; 26, two lanes; 27, three lanes; 28 , lane four; 29, lane five; 30, lane six; 31, lane seven; 32, lane eight; 33, lane nine; 34, lane ten; 35, lane eleven; 36, lane twelve.

detailed description

In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention will be described in further detail below with reference to the accompanying drawings.

Each lane needs to install two geomagnetic vehicle detectors, one geomagnetic vehicle detector is placed at the upstream of the stop line and close to the stop line to detect the number of vehicles passing the stop line, and the other geomagnetic vehicle detector is placed 120 meters upstream of the stop line At , the number of vehicles passing through the section at 120 meters upstream of the stop line is detected. Through these two geomagnetic vehicle detectors, the number of vehicles located between the stop line and the 120-meter section upstream of the stop line can be calculated at any time in the lane where it is located, and converted into the vehicle queue length. As shown in Figure 1, the first geomagnetic vehicle detector 1 and the second geomagnetic vehicle detector 2 are used to detect the vehicle queuing length of lane one 25, and the 3rd geomagnetic vehicle detector 3 and the 4th geomagnetic vehicle detector 4 are used for detecting The vehicle queue length of lane two 26, the fifth geomagnetic vehicle detector 5 and the sixth geomagnetic vehicle detector 6 are used to detect the vehicle queue length of lane three 27, the seventh geomagnetic vehicle detector 7 and the eighth geomagnetic vehicle detector 8 are used To detect the vehicle queuing length of lane four 28, the ninth geomagnetic vehicle detector 9 and the tenth geomagnetic vehicle detector 10 are used to detect the vehicle queuing length of lane five 29, the eleventh geomagnetic vehicle detector 11 and the twelfth geomagnetic vehicle The detector 12 is used to detect the vehicle queuing length of lane six 30, the thirteenth geomagnetic vehicle detector 13 and the fourteenth geomagnetic vehicle detector 14 are used to detect the vehicle queuing length of lane seven 31, and the fifteenth geomagnetic vehicle detector 15 And the sixteenth geomagnetic vehicle detector 16 is used to detect the vehicle queue length of lane eight 32, the seventeenth geomagnetic vehicle detector 17 and the eighteenth geomagnetic vehicle detector 18 are used to detect the vehicle queue length of lane nine 33, the tenth Nine geomagnetic vehicle detectors 19 and the twentieth geomagnetic vehicle detector 20 are used to detect the vehicle queuing length of lane ten 34, and the twenty-first geomagnetic vehicle detector 21 and the twenty-second geomagnetic vehicle detector 22 are used to detect lane ten The vehicle queuing length of -35, the twenty-third geomagnetic vehicle detector 23 and the twenty-fourth geomagnetic vehicle detector 24 are used to detect the vehicle queuing length of lane twelve 36.

The intersection machine receives the information sent by a total of twenty-four geomagnetic vehicle detectors from the first geomagnetic vehicle detector 1 to the twenty-fourth geomagnetic vehicle detector 24, and then forwards the information to the traffic signal controller. Every 10 seconds, the traffic signal controller determines the control action according to the received traffic status and the control strategy set by the intersection machine.

The flow chart of a traffic signal optimization control method based on policy iteration and clustering shown in Figure 2 includes the following steps:

Step 1, select the signal control scheme, define the traffic state, control action, direct return and Q value:

The signal control scheme to be optimized adopts a fixed phase sequence control scheme, and the control scheme is described below taking the case of four symmetrical phases as an example, but the present invention is not limited to the use of four phases, nor is it limited to the use of symmetrical phases. Phase 1: vehicles on lane one 25 and lane four 28 are allowed to go straight and turn right, and vehicles on lane two 26 and lane five 29 are allowed to go straight; phase 2: vehicles on lane three 27 and lane six 30 are allowed to turn left; phase 3: Allow vehicles on lane seven 31 and lane ten 34 to go straight and turn right, and allow vehicles on lane eight 32 and lane eleven 35 to go straight; Phase 4: Allow vehicles on lane nine 33 and lane twelve 36 to turn left. Traffic signals can only be in one of the four phases at each moment, and they are executed sequentially. Although the order of the phases is fixed, the duration of the green light for each phase does not have to be fixed. Define the control action as maintaining the current phase or switching to the next phase. If the current phase is phase 1, after 10 seconds, the traffic signal controller needs to make a decision control action: maintain phase 1, or switch to phase 2, if phase 2 is selected , after 10 seconds, you need to make another control action: keep phase 2, or switch to phase 3, if you choose phase 3, you need to make another control action after 10 seconds: keep phase 3, or switch to phase 4, if you choose Phase 4, after 10 seconds, another control action is required: maintain phase 4, or switch to phase 1... and so on. Define a minimum green time of 10 seconds and a maximum green time of 60 seconds for all phases.

The vehicle queuing length of each phase is defined as the maximum value of the vehicle queuing lengths of all lanes of the phase, and the vehicle queuing length of phase 1 is equal to the maximum of the vehicle queuing lengths of lane one 25, lane two 26, lane four 28 and lane five 29 value; the vehicle queuing length of phase 2 is equal to the maximum value among the vehicle queuing lengths of lane three 27 and lane six 30; the vehicle queuing length of phase 3 is equal to the vehicles of lane seven 31, lane eight 32, lane ten 34 and lane eleven 35 The maximum value in the queuing length; the vehicle queuing length of phase 4 is equal to the maximum value in the vehicle queuing lengths of lane nine 33 and lane twelve 36 .

The traffic state is defined as the vehicle queue length of the current phase and the next phase. For example, if the current phase is phase 1, the current traffic state is represented by the vector data consisting of two variables, the vehicle queue length of phase 1 and phase 2.

Define the starting moment of a sampling period as the moment when the traffic signal controller makes a decision-making control action, and the duration of the sampling period is equal to the duration of the decision-making period, which is 10 seconds; define the direct return as the number of vehicles leaving the stop line in a single sampling period Dimensionless quantity, which represents the direct benefits obtained after taking control actions in a traffic state; define a state-action pair as a data vector composed of a discrete traffic state and a control action; define the Q value of each state-action pair to be in the corresponding discrete The expected cumulative return obtained after taking the control action in the traffic state, that is, the expectation of the sum of the direct returns obtained in several sampling periods after the control action is taken, and the Q value represents the long-term benefit obtained after taking the control action in the discrete traffic state ;Define the control strategy as the control action that should be taken when the discrete traffic state is given;

The calculation formula of the direct return r is as follows:

In the above formula, n _p represents the number of vehicles passing the stop line within a sampling period, and the constants 6.5, 4.5 and -1.0 in the formula are used to maintain the direct return r between [-1,1]. The traffic signal controller calculates n _p according to the traffic status sent by two adjacent intersection machines, and then calculates according to the above formula to obtain the direct return r.

The Q value of a state-action pair is defined as follows:

$\mathrm{<span\; class="notranslate">\; Q</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; (</span>\underset{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; T</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; \&gamma;</span>}}^{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>$

s represents the discrete traffic state, a represents the control action performed in the traffic state s, Q(s,a) represents the Q value of the state-action pair s-a, E represents the expectation, r(s,a) represents the execution in the state s The direct reward obtained by control action a, γ is the discount factor, which is a real number between 0 and 1, and k represents the kth sampling period after encountering traffic state s, experiencing traffic state s and executing control action a , corresponding to k=1 after a sampling period, T means that the sampling ends at the Tth sampling period after encountering the traffic state s, that is, the calculation of the cumulative return only uses the direct return of T sampling periods.

Step 2, sampling the traffic state, the executed control action and the number of vehicles leaving the stop line.

Sampling is carried out for a period of time during the specified typical time period, such as the morning rush hour or evening rush hour. In the sampling stage, the intersection machine sets the control strategy of the traffic signal controller as induction control, and the minimum green light time, maximum green light time and set as the sampling period The positive integer multiple of , the unit green light extension time is the same as the sampling period, the minimum green light time of each phase is set to 10 seconds, the maximum green light time is 60 seconds, and the unit green light extension time is 10 seconds. Every second, the phase is determined according to the following method: when the green light time of the current phase is less than 10 seconds, keep the current phase; when the green light time of the current phase exceeds or is equal to 60 seconds, switch to the next phase; the green light time of the current phase is between 10 seconds and 60 seconds, if there is an incoming vehicle in the current phase, the green light time will be extended by 10 seconds, and if there is no incoming vehicle, it will directly switch to the next phase. Every 10 seconds, the intersection machine detects and stores the following information as a sample: the vehicle queuing length of the current phase and the next phase, the executed control action and the number of vehicles leaving the stop line in each sampling period. Set the number of samples to be collected to 9000.

Step 3, after the intersection machine collects 9000 samples, discretize the traffic status in the samples. Organize each sample into the form of data vector (l, a, l', r), l represents the traffic state at a certain sampling moment, a represents the control action executed when the traffic state is l, and l' represents the next The traffic state at the sampling time, r represents the direct return obtained during the sampling period when the traffic state shifts from l to l', you can use the number of vehicles leaving the stop line in each sampling period in the original sample, and directly return r according to step 1 calculated by the calculation formula.

Preprocess the traffic state in the sample, first normalize, and then remove the traffic state whose distance exceeds the preset threshold. Select the Euclidean distance as the distance, set the threshold to 0.1, first randomly select a normalized traffic state from the sample and add it to an empty data set, called the traffic state data set, and then use the remaining traffic states in the sample according to the following principles Add to the data set: If the distance between the traffic state in the sample and all the traffic states in the traffic state data set is greater than 0.1, then add the traffic state to the traffic state data set, otherwise it will not be added.

Carry out k-means clustering on the traffic state in the traffic state data set, define the cluster as a collection of similar traffic states, each cluster corresponds to a discrete traffic state, define the centroid as the centroid of all traffic states contained in the cluster, and set the number of centroids to 30 , and start clustering, the steps are as follows:

Step a, randomly select 30 different traffic states from the traffic state dataset as initial centroids;

Step b, calculate the distance from each traffic state to each centroid, and assign each traffic state to the nearest centroid, forming 30 clusters;

Step c, recalculate the centroid of each cluster;

Step d, calculate the change of the centroid, that is, the distance between the original centroid and the new centroid, if the centroids of all clusters no longer change, the k-means clustering ends, otherwise, perform step b.

After k-means clustering, assign l and l' in each sample (l, a, l', r) to the nearest centroid, that is, transform them into discrete traffic states s and s' respectively, and organize the samples is the data vector (s,a,s',r).

Step 4, arbitrarily initialize a traffic signal control strategy in the intersection computer, then use the strategy iteration method to optimize the strategy, and save the optimized strategy and the centroid obtained in step 3 in the traffic signal controller;

In the single-intersection traffic signal control optimization problem, there are 30 discrete traffic states, and each discrete traffic state has two control actions—— _{a 1} means to maintain the current phase, a ₂ means to switch to the next phase, the optimization of the strategy It is carried out in the intersection machine, and the strategy iteration method is used for optimization. The steps are as follows:

Step a, set the number of iterations to 1, initialize the Q value and control strategy, and calculate the state transition matrix and direct return matrix. Initialize the Q value of each state-action pair to zero, save it in the matrix Q, and estimate the direct return matrix R ₁ and R ₂ according to the sample (s, a, s', r), R ₁ and R ₂ are saved and executed respectively The expectation of direct reward obtained after controlling actions a ₁ and a ₂ , set i=1,2,...,30, j=1,2,...,30, k=1,2, Q, R ₁ and R2 are defined as follows _:

$\mathrm{<span\; class="notranslate">\; Q</span>}<span\; class="notranslate">\; =</span>\left[\begin{array}{c}\mathrm{<span\; class="notranslate">\; Q</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>\\ \mathrm{<span\; class="notranslate">\; Q</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>\\ \mathrm{<span\; class="notranslate">\; Q</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>\\ \mathrm{<span\; class="notranslate">\; Q</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>\\ <span\; class="notranslate">\; .</span>\\ <span\; class="notranslate">\; .</span>\\ <span\; class="notranslate">\; .</span>\\ \mathrm{<span\; class="notranslate">\; Q</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 30</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>\\ \mathrm{<span\; class="notranslate">\; Q</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 30</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>\end{array}\right]<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; =</span>\left[\begin{array}{cccc}\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; ...</span>}& \mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 30</span>}}<span\; class="notranslate">\; )</span>\\ \mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; ...</span>}& \mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 30</span>}}<span\; class="notranslate">\; )</span>\\ <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& & <span\; class="notranslate">\; .</span>\\ <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& & <span\; class="notranslate">\; .</span>\\ <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& & <span\; class="notranslate">\; .</span>\\ \mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 30</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 30</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; ...</span>}& \mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 30</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 30</span>}}<span\; class="notranslate">\; )</span>\end{array}\right]<span\; class="notranslate">\; ,</span>$

${\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; =</span>\left[\begin{array}{cccc}\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; ...</span>}& \mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 30</span>}}<span\; class="notranslate">\; )</span>\\ \mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; ...</span>}& \mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 30</span>}}<span\; class="notranslate">\; )</span>\\ <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& & <span\; class="notranslate">\; .</span>\\ <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& & <span\; class="notranslate">\; .</span>\\ <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& & <span\; class="notranslate">\; .</span>\\ \mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 30</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 30</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; ...</span>}& \mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 30</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 30</span>}}<span\; class="notranslate">\; )</span>\end{array}\right]<span\; class="notranslate">\; .</span>$

Among them, Q(s _i , a _k ) represents the Q value of the action-state pair s _i -a _k , r(s _i , a _k , s _j ) represents the discrete traffic state s _i , after executing the control action a _k , The immediate reward obtained when transitioning to the discrete traffic state _sj . Initialize a control strategy as an arbitrary strategy and store it in the matrix Π. The definition of Π is as follows:

$\mathrm{<span\; class="notranslate">\; \&Pi;</span>}<span\; class="notranslate">\; =</span>\left[\begin{array}{ccccccccc}\mathrm{<span\; class="notranslate">\; \&pi;</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; \&pi;</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; ...</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}\\ \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; \&pi;</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; \&pi;</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; ...</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}\\ \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; \&pi;</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; \&pi;</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; ...</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}\\ <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& & <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>\\ <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& & <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>\\ <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& & <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>\\ \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; ...</span>}& \mathrm{<span\; class="notranslate">\; \&pi;</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 30</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; \&pi;</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 30</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>\end{array}\right]<span\; class="notranslate">\; .</span>$

Among them, π(s _i , a _k ) represents the probability of executing action a _k in discrete state s _i , and the sum of elements in each row of Π is 1. Estimate the state transition matrix P according to the sample (s, a, s', r), defined as follows:

$\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; =</span>\left[\begin{array}{cccc}\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; ...</span>}& \mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 30</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>\\ \mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; ...</span>}& \mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 30</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>\\ \mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; ...</span>}& \mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 30</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>\\ \mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; ...</span>}& \mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 30</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>\\ <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& & <span\; class="notranslate">\; .</span>\\ <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& & <span\; class="notranslate">\; .</span>\\ <span\; class="notranslate">\; .</span>& <span\; class="notranslate">\; .</span>& & <span\; class="notranslate">\; .</span>\\ \mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 30</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 30</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; ...</span>}& \mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 30</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 30</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>\\ \mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 30</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 30</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; ...</span>}& \mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 30</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 30</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>\end{array}\right]$

Among them, the matrix element p(s _j |s _i , a _k ) is the conditional probability, which represents the probability of shifting to the discrete traffic state s _j at the next sampling moment after executing the control action a _k in the discrete traffic state s _i . Using the elements in R ₁ , R ₂ and P, the direct return matrix R can be obtained. The definition of R is as follows:

$\mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; =</span>\left[\begin{array}{c}\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>\\ \mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>\\ \mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>\\ \mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>\\ <span\; class="notranslate">\; .</span>\\ <span\; class="notranslate">\; .</span>\\ <span\; class="notranslate">\; .</span>\\ \mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 30</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>\\ \mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 30</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>\end{array}\right]<span\; class="notranslate">\; .</span>$

Among them, r(s _i , a _k ) represents the expectation of the direct reward obtained after executing the control action a _k in the discrete traffic state s _i , and the calculation formula is as follows:

$\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; 30</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; .</span>$

Step b, update the Q value, and update the matrix Q according to the following formula:

Q＝(I-γPΠ) ^-1 R

Among them, I represents the identity matrix, γ is the discount factor, set to 0.95, () ^-1 represents the inversion of the matrix;

Step c, update the control strategy according to the Q value, and update the elements in the matrix Π according to the following formula:

$\mathrm{<span\; class="notranslate">\; \&pi;</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}\mathrm{<span\; class="notranslate">\; 1</span>}& {\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; \&Element;</span><span\; class="notranslate">\; \{</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; \}</span>}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; max</span>}}\mathrm{<span\; class="notranslate">\; Q</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; )</span>\\ \mathrm{<span\; class="notranslate">\; 0</span>}& {\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; \&NotEqual;</span>\underset{\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; \&Element;</span><span\; class="notranslate">\; \{</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; \}</span>}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; max</span>}}\mathrm{<span\; class="notranslate">\; Q</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; )</span>\end{array}\right.$

Step d, if the number of iterations is 1, save the matrix Π to a matrix Π' of the same dimension, increase the number of iterations by 1, and return to step b, otherwise, solve the second norm of the difference between matrix Π and matrix Π':

D＝||Π-Π'||

If D is equal to 0, the strategy iteration ends. If D is not equal to 0, save matrix Π to matrix Π', increase the number of iterations by 1, and return to step b.

After the strategy iteration is over, the obtained control strategy is saved in the matrix Q, and the matrix Q and the centroid obtained in step 3 are saved in the traffic signal controller;

Step 5, the intersection machine sets the control strategy of the traffic signal controller to the control strategy obtained in step 4, every 10 seconds, the traffic signal controller receives the traffic status detected by the intersection machine, normalizes it, and calculates the normalized Calculate the distance from the transformed traffic state to each centroid, find the number of the nearest centroid, that is, the number i of the discrete traffic state s _i state, and then select the control action a ^* according to the following formula:

${\mathrm{<span\; class="notranslate">\; a</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; arg</span>}\underset{\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; \&Element;</span><span\; class="notranslate">\; \{</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; \}</span>}{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; x</span>}}\mathrm{<span\; class="notranslate">\; Q</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>$

The traffic signal controller sends the control action a ^* to the intersection machine for execution. If the value of a ^* is a ₁ , the current phase will be maintained, and if the value of a ^* is a ₂ , it will switch to the next phase.

Claims (2)

1. A traffic signal optimization control method based on policy iteration and clustering, comprising the following steps: Step 1. Select the signal control scheme to be optimized as fixed phase sequence control, define the traffic state as the vehicle queuing length of the current phase and the next phase, define the control action as maintaining the current phase or switch to the next phase, and define the direct return as a Variables related to the number of vehicles leaving the stop line in a single sampling period, define the state-action pair as a data vector composed of discrete traffic states and control actions, and define the Q value of each state-action pair to represent the corresponding discrete traffic state. The expected cumulative reward obtained after the control action defines the control strategy as the control action that should be executed for each discrete traffic state; Step 2. The intersection machine sets the control strategy of the traffic signal controller as induction control. The minimum green light time and the maximum green light time are set as positive integer multiples of the sampling period. The unit green light extension time is the same as the sampling period. The phase action and the number of vehicles leaving the stop line are sampled and recorded. The sampling method is: record the traffic state, control action and the number of vehicles leaving the stop line in each sampling period at each sampling moment; Step 3. After the intersection machine collects a specified number of samples, it discretizes the traffic state in the sample. The discretization method is: first normalize the traffic state obtained by sampling, and remove the traffic state whose distance exceeds the preset threshold , and then perform k-means clustering, number the obtained centroids, each centroid corresponds to a discrete traffic state, and represent the traffic state in the normalized sample with the number of the nearest centroid to obtain the corresponding discrete traffic state; Step 4, the intersection machine uses strategy iteration to optimize the strategy, and saves the optimized strategy and the centroid obtained in step 3 in the traffic signal controller; Step 5, the intersection machine sets the control strategy of the traffic signal controller as the control strategy obtained in step 4, and sets the decision-making cycle as the sampling cycle. At each decision-making moment, the traffic signal controller receives the traffic status detected by the intersection machine and performs Normalization, calculate the distance from the normalized traffic state to each centroid, find the nearest centroid, query the control strategy according to the discrete traffic state corresponding to the centroid, get the control action and send it to the intersection machine for execution. 2. The method as claimed in claim 1, wherein, after normalization and before k-means clustering, the traffic state whose distance exceeds a preset threshold is removed, the method is: first randomly select a The traffic state is added to an empty data set, and then the remaining traffic states in the normalized sample are added to the data set according to the following principles: If the Euclidean distance between the traffic state in the normalized sample and all the traffic states in the data set is greater than the preset If the threshold is set, the traffic state will be added to the data set, otherwise it will not be added.