Abstract
Traffic signal control (TSC) plays an important role in intelligent transportation system. It is helpful to improve the efficiency of urban transportation by controlling the traffic signal intelligently. Recently, various deep reinforcement learning methods have been proposed to solve TSC. However, most of these methods ignore the fusion of spatial and temporal features in traffic roadnets. Besides, these methods pay no attention to the correlations of the intersections in several local areas. This paper proposes a novel multi-agent deep reinforcement learning method with spatio-temporal feature fusion to solve TSC. The proposed method firstly calculates the correlations among different time steps to capture their temporal dependencies. Secondly, the proposed method constructs connected subnetworks to capture interactive relations among intersections in the subnetwork. Experimental results demonstrate that our method achieves state-of-the-art performance on synthetic and real-world datasets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Auer, P., Cesa-Bianchi, N., Fischer, P.: Finite-time analysis of the multiarmed bandit problem. Mach. Learn. 47(2–3), 235–256 (2002). https://doi.org/10.1023/A:1013689704352
Bai, S., Kolter, J.Z., Koltun, V.: An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271 (2018)
Chen, C., et al.: Toward a thousand lights: decentralized deep reinforcement learning for large-scale traffic signal control. In: AAAI 2020, vol. 34, no. 4, pp. 3414–3421 (2020)
Choe, C., Baek, S., Woon, B., Kong, S.H.: Deep Q learning with LSTM for traffic light control. In: 2018 24th Asia-Pacific Conference on Communications (APCC), pp. 331–336 (2018)
Chu, T., Wang, J., Codec, L., Li, Z.: Multi-agent deep reinforcement learning for large-scale traffic signal control. IEEE Trans. Intell. Transp. Syst. 21(3), 1086–1095 (2020)
Gao, K., et al.: Solving traffic signal scheduling problems in heterogeneous traffic network by using meta-heuristics. IEEE Trans. Intell. Transp. Syst. 20(9), 3272–3282 (2019)
Garg, D., Chli, M., Vogiatzis, G.: Deep reinforcement learning for autonomous traffic light control. In: 2018 3rd IEEE International Conference on Intelligent Transportation Engineering (ICITE), pp. 214–218 (2018)
Gottlich, S., Herty, M., Ziegler, U.: Modeling and optimizing traffic light settings in road networks. Comput. Oper. Res. 55, 36–51 (2015)
Haydari, A., Yilmaz, Y.: Deep reinforcement learning for intelligent transportation systems: a survey. IEEE Trans. Intell. Transp. Syst., 1–22 (2020, in press). https://doi.org/10.1109/TITS.2020.3008612
Hsu, J.: Alibaba cloud launched ‘ET City Brain 2.0’ in Hangzhou (2018)
Hu, H.C., Smith, S.F., Goldstein, R.: Cooperative schedule-driven intersection control with connected and autonomous vehicles. In: IROS 2019, pp. 1668–1673 (2019)
Hu, H.C., Smith, S.F.: Learning model parameters for decentralized schedule-driven traffic control. In: ICAPS 2020, pp. 531–539 (2020)
Koonce, P., Rodegerdts, L.: Traffic signal timing manual. Technical report, United States. Federal Highway Administration (2008)
Li, L., Lv, Y., Wang, F.: Traffic signal timing via deep reinforcement learning. IEEE/CAA J. Autom. Sin. 3(3), 247–254 (2016)
Liang, X., Du, X., Wang, G., Han, Z.: A deep reinforcement learning network for traffic light cycle control. IEEE Trans. Veh. Technol. 68(2), 1243–1253 (2019)
Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)
Mousavi, S.S., Schukat, M., Howley, E.: Traffic light control using deep policy-gradient and value-function-based reinforcement learning. IET Intell. Transp. Syst. 11(7), 417–423 (2017)
Nishi, T., Otaki, K., Hayakawa, K., Yoshimura, T.: Traffic signal control based on reinforcement learning with graph convolutional neural nets. In: 2018 21st International Conference on Intelligent Transportation Systems, pp. 877–883 (2018)
Pol, E.V.D., Oliehoek, F.A.: Coordinated deep reinforcement learners for traffic light control. In: NeurIPS 2016 (2016)
Schaul, T., Quan, J., Antonoglou, I., Silver, D.: Prioritized experience replay. In: ICLR 2016 (2016)
Tan, T., et al.: Cooperative deep reinforcement learning for large-scale traffic grid signal control. IEEE Trans. Cybern. 50(6), 2687–2700 (2020)
Varaiya, P.: The max-pressure controller for arbitrary networks of signalized intersections. In: Ukkusuri, S., Ozbay, K. (eds.) Advances in Dynamic Network Modeling in Complex Transportation Systems. Complex Networks and Dynamic Systems, vol. 2, pp. 27–66. Springer, New York (2013). https://doi.org/10.1007/978-1-4614-6243-9_2
Veres, M., Moussa, M.: Deep learning for intelligent transportation systems: A survey of emerging trends. IEEE Trans. Intell. Transp. Syst. 21(8), 3152–3168 (2020)
Wang, X., Ke, L., Qiao, Z., Chai, X.: Large-scale traffic signal control using a novel multiagent reinforcement learning. IEEE Trans. Cybern. 21(3), 1086–1095 (2020)
Wang, Y., et al.: STMARL: A spatio-temporal multi-agent reinforcement learning approach for cooperative traffic light control. IEEE Trans. Mob. Comput., 1–15 (2020, in press). https://doi.org/10.1109/TMC.2020.3033782
Wei, H., Zheng, G., Gayah, V., Li, Z.: A survey on traffic signal control methods. arXiv preprint arXiv:1904.08117 (2019)
Wei, H., Zheng, G., Gayah, V., Li, Z.: Recent advances in reinforcement learning for traffic signal control: A survey of models and evaluation. ACM SIGKDD Explor. Newsl. 22(2), 12–18 (2020)
Wei, H., Zheng, G., Yao, H., Li, Z.: IntelliLight: A reinforcement learning approach for intelligent traffic light control. In: KDD 2018, pp. 2496–2505 (2018)
Wei, H., et al.: CoLight: Learning network-level cooperation for traffic signal control. In: CIKM 2019, pp. 1913–1922 (2019)
Wei, H., et al.: PressLight: Learning max pressure control to coordinate traffic signals in arterial network. In: KDD 2019, pp. 1290–1298 (2019)
Ye, J., Zhao, J., Ye, K., Xu, C.: How to build a graph-based deep learning architecture in traffic domain: A survey. IEEE Trans. Intell. Transp. Syst., 1–21 (2020, in press). https://doi.org/10.1109/TITS.2020.3043250
Zhang, H., et al.: CityFlow: A multi-agent reinforcement learning environment for large scale city traffic scenario. In: WWW 2019, pp. 3620–3624 (2019)
Zheng, G., et al.: Diagnosing reinforcement learning for traffic signal control. arXiv preprint arXiv:1905.04716 (2019)
Zheng, G., et al.: Learning phase competition for traffic signal control. In: CIKM 2019, pp. 1963–1972 (2019)
Acknowledgements
This work is supported by the National Key R&D Program of China (2018AAA0101203), and the National Natural Science Foundation of China (62072483).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Du, X., Wang, J., Chen, S., Liu, Z. (2021). Multi-agent Deep Reinforcement Learning with Spatio-Temporal Feature Fusion for Traffic Signal Control. In: Dong, Y., Kourtellis, N., Hammer, B., Lozano, J.A. (eds) Machine Learning and Knowledge Discovery in Databases. Applied Data Science Track. ECML PKDD 2021. Lecture Notes in Computer Science(), vol 12978. Springer, Cham. https://doi.org/10.1007/978-3-030-86514-6_29
Download citation
DOI: https://doi.org/10.1007/978-3-030-86514-6_29
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-86513-9
Online ISBN: 978-3-030-86514-6
eBook Packages: Computer ScienceComputer Science (R0)
