Abstract
Recent efforts that applied Multi-Agent Reinforcement Learning (MARL) to the adaptive traffic signal control (ATSC) problem have shown remarkable progress. However, those methods assume that all agents in the cooperative games are isomorphic, which ignores the situation that different agents can play heterogeneous roles in the ATSC scenario. The tolerance of vehicles at different intersections in the same area is different, e.g., traffic congestion near hospitals or schools will affect the timely treatment of patients or the safety of children and definitely need to be paid more attention than ordinary congestions. Motivated by the human wisdom in cooperative behaviours (e.g. team members will execute the action according to the strategy implemented by the team leader), we present a leader-follower paradigm based Markov game model which taking into account both the overall and special intersections. Specifically, the leader-follower paradigm control intersections in a traffic scenario by two kinds of agents, i.e., leader agent controlling intersections that need special attention, and follower agents controlling ordinary intersections. Then a multi-agent reinforcement learning framework, named Breadth First Sort Hysteretic DQN (BFS-HDQN) is proposed to train the optimal control policy of the proposed ATSC model. BFS-HDQN consists of two parts, an independent MARL algorithm (here we use Hysteretic DQN as the base algorithm) to train different kinds of agents, and a communication mechanism based on Breadth First Sort (BFS) to generate observation information of each agent. We evaluate our methods empirically in two synthetic and one real-world traffic scenarios. Experimental results show that, compared with the state-of-the-art methods, BFS-HDQN can not only ensure the optimal overall performance, but also obtain better performance at special intersections, in almost all metrics commonly used in ATSC.
The work is supported by the National Natural Science Foundation of China (Grant Nos.: 61906027, 61906135), China Postdoctoral Science Foundation Funded Project (Grant No.: 2019M661080).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
MA2C, Colight: https://traffic-signal-control.github.io/.
References
Chen, C., et al.: Toward a thousand lights: decentralized deep reinforcement learning for large-scale traffic signal control. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 3414–3421 (2020)
Chu, T., Chinchali, S., Katti, S.: Multi-agent reinforcement learning for networked system control. In: International Conference on Learning Representations (2019)
Chu, T., Wang, J., Codeca, L., Li, Z.: Multi-agent deep reinforcement learning for large-scale traffic signal control. IEEE Trans. Intell. Transp. Syst. 21(3), 1086–1095 (2020)
Cools, S., Gershenson, C., Dhooghe, B.: Self-organizing traffic lights: a realistic simulation. \(\text{arXiv}\): Adaptation and Self-Organizing Systems (2006)
Gartner, N.H., Assmann, S.F., Lasaga, F., Hous, D.L.: Multiband-a variable-bandwidth arterial progression scheme. Transportation Research Record (1287) (1990)
Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., Mordatch, I.: Multi-agent actor-critic for mixed cooperative-competitive environments. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS 2017, pp. 6382–6393 (2017)
Luk, J.Y.: Two traffic-responsive area traffic control methods: SCAT and SCOOT. Traffic Eng. Control 25(1), 14–22 (1984)
Ma, J., Wu, F.: Feudal multi-agent deep reinforcement learning for traffic signal control. In: Proceedings of the 19th International Conference on Autonomous Agents and Multiagent Systems (AAMAS) (2020)
Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529 (2015)
Omidshafiei, S., Pazis, J., Amato, C., How, J.P., Vian, J.: Deep decentralized multi-task multi-agent reinforcement learning under partial observability. In: Proceedings of the 34th International Conference on Machine Learning, vol. 70, pp. 2681–2690. JMLR. org (2017)
Van der Pol, E., Oliehoek, F.A.: Coordinated deep reinforcement learners for traffic light control. In: Advances in Neural Information Processing Systems (2016)
Son, K., Kim, D., Kang, W.J., Hostallero, D.E., Yi, Y.: QTRAN: learning to factorize with transformation for cooperative multi-agent reinforcement learning. In: Proceedings of the 36th International Conference on Machine Learning, vol. 97, pp. 5887–5896 (2019)
Sunehag, P., et al.: Value-decomposition networks for cooperative multi-agent learning. \(\text{ arXiv }\): Artificial Intelligence (2017)
Tan, T., Bao, F., Deng, Y., Jin, A., Dai, Q., Wang, J.: Cooperative deep reinforcement learning for large-scale traffic grid signal control. IEEE Trans. Cybernet. 50(6), 2687–2700 (2019)
Wei, H., et al.: PressLight: learning max pressure control to coordinate traffic signals in arterial network. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1290–1298 (2019)
Wei, H., et al.: CoLight: learning network-level cooperation for traffic signal control. In: Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pp. 1913–1922 (2019)
Wei, H., Zheng, G., Gayah, V., Li, Z.: Recent advances in reinforcement learning for traffic signal control: a survey of models and evaluation. SIGKDD Explor. 22(2), 12–18 (2021)
Zhang, C., Jin, S., Xue, W., Xie, X., Chen, S., Chen, R.: Independent reinforcement learning for weakly cooperative multiagent traffic control problem. IEEE Trans. Veh. Technol., 1 (2021). https://doi.org/10.1109/TVT.2021.3090796
Zhu, F., Aziz, H.M.A., Qian, X., Ukkusuri, S.V.: A junction-tree based learning algorithm to optimize network wide traffic control: a coordinated multi-agent framework. Transp. Res. Part C Emerg. Technol. 58, 487–501 (2015)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Luan, L. et al. (2022). MARL for Traffic Signal Control in Scenarios with Different Intersection Importance. In: Chen, J., Lang, J., Amato, C., Zhao, D. (eds) Distributed Artificial Intelligence. DAI 2021. Lecture Notes in Computer Science(), vol 13170. Springer, Cham. https://doi.org/10.1007/978-3-030-94662-3_7
Download citation
DOI: https://doi.org/10.1007/978-3-030-94662-3_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-94661-6
Online ISBN: 978-3-030-94662-3
eBook Packages: Computer ScienceComputer Science (R0)