Raven: Scheduling Virtual Machine Migration During Datacenter Upgrades with Reinforcement Learning

Abstract

Physical machines in modern datacenters are routinely upgraded due to their maintenance requirements, which involve migrating all the virtual machines they currently host to alternative physical machines. For this kind of datacenter upgrades, it is critical to minimize the time it takes to upgrade all the physical machines in the datacenter, so as to reduce disruptions to cloud services. To minimize the upgrade time, it is essential to carefully schedule the migration of virtual machines on each physical machine during its upgrade, without violating any constraints imposed by virtual machines that are currently running. Rather than resorting to heuristic algorithms as existing work, we propose a new scheduler, Raven, that uses an experience-driven approach with deep reinforcement learning to schedule the virtual machine migration. With our design of the state space, action space and reward function, Raven trains a fully-connected neural network using the cross-entropy method to approximate the policy of choosing a destination physical machine for each virtual machine before its migration. We compare Raven with state-of-the-art algorithms in the literature, and our results show that Raven can effectively shorten the time to complete the datacenter upgrade under different datacenter settings.