Abstract
The key to lowering the threshold of the application of reinforcement learning is the simplicity and convenience of reward function design. At present, reinforcement learning with good performance mostly adopts complex rewards of artificial trial and error, or adopts supervised learning to track the artificial trajectory, but these methods increase the workload. Assuming that the basic mathematical elements (operators, operands) can be used to automatically accomplish the combinatorial search process, it is possible to search for a compact, concise and informative reward model. Starting from this idea, this paper explores the reward function of reinforcement learning, which can find the optimal or suboptimal solution that can meet the multi-optimization index through operator search without clear prior knowledge. Based on AutoML-zero, the automatic search method of operator-level reward function based on evolutionary search is realized, and the reward function algorithm which can satisfy the constraint conditions is found to be equal to or better than human design.
This work is supported by the National Natural Science Foundation of China under Grant U21B2028.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Yang, R., Yan, J., Li, X.: Survey of sparse reward algorithms in reinforcement learning—theory and experiment. CAAI Trans. Intell. Syst. 15(05), 888–899 (2020)
Ng, A.Y.: Shaping and policy search in reinforcement learning. Ph.D. thesis, University of California, Berkeley (2003)
Sutton, R.S., Barto, A.G.: Reinforcement Learning in Feedback Control—Challenges and Benchmarks from Technical Process Control. MIT Press, Cambridge (1998)
Li, Y., Shao, Z., Zhao, Z., et al.: Design of reward function in deep reinforcement learning for trajectory planning. Comput. Eng. Appl. 56(2), 226–232 (2020)
Knox, W., Stone, P.: Framing reinforcement learning from human reward: reward positivity, temporal discounting, episodicity, and performance. Artif. Intell. 225(C), 24–50 (2015)
Abbeel, P., Ng, A.Y.: Apprenticeship learning via inverse reinforcement learning. In: Proceedings of the 21st International Conference on Machine Learning, Banff, pp. 1–8 (2004)
Ziebart, B.D., Maas, A.L., Bagnell, J.A., et al.: Maximum entropy inverse reinforcement learning. In: Proceedings of the 23rd AAAI Conference on Artificial Intelligence, Illinois, pp. 1433–1438 (2008)
Ho, J., Ermon, S.: Generative adversarial imitation learning. In: Proceedings of the 30th Conference and Workshop on Neural Information Processing Systems, Barcelona, pp. 4565–4573 (2016)
Wu, Y., Mozifian, M., Shkurti, F.: Shaping rewards for reinforcement learning with imperfect demonstrations using generative models. In: The 2021 International Conference on Robotics and Automation, Xi’an, pp. 6628–6634 (2021)
Pathak, D., Agrawal, P., Efros, A.A., et al.: Curiosity-driven exploration by self-supervised prediction. In: Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops, Hawaii, pp. 488–489 (2017)
Jaderberg, M., Czarnecki, W.M., Dunning, I., et al.: Human level performance in first-person multiplayer games with population-based deep reinforcement learning. arXiv (2018)
Zou, H., Ren, T., Dong, Y., et al.: Learning task-distribution reward shaping with meta-learning. In: The 35th AAAI Conference on Artificial Intelligence, New York (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, L., Wang, Z., Gong, Q. (2022). Bi-level Optimization Method for Automatic Reward Shaping of Reinforcement Learning. In: Pimenidis, E., Angelov, P., Jayne, C., Papaleonidas, A., Aydin, M. (eds) Artificial Neural Networks and Machine Learning – ICANN 2022. ICANN 2022. Lecture Notes in Computer Science, vol 13531. Springer, Cham. https://doi.org/10.1007/978-3-031-15934-3_32
Download citation
DOI: https://doi.org/10.1007/978-3-031-15934-3_32
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-15933-6
Online ISBN: 978-3-031-15934-3
eBook Packages: Computer ScienceComputer Science (R0)