New timers are scheduled relative to a local time kept in the timers class, which is only updated when Timers::tick() is run. However, when there are no scheduled tick callbacks, StreamSelectLoop will wait until the next scheduled timer (or infinite, if there are none) for stream activity, which prevents Timers::tick() from being called regularly and updating its local time. If a new timer is added in response to a stream event, it will be shortened by whatever period of time was spent waiting for stream activity (or, if the new timer's length is shorter than was spent waiting, the timer will be resolved immediately).

I believe a potential solution is:

diff --git a/src/Timer/Timers.php b/src/Timer/Timers.php
index c183a63..bebe552 100644
--- a/src/Timer/Timers.php
+++ b/src/Timer/Timers.php
@@ -30,7 +30,7 @@ class Timers
     public function add(TimerInterface $timer)
     {
         $interval = $timer->getInterval();
-        $scheduledAt = $interval + $this->getTime();
+        $scheduledAt = $interval + microtime(true);

         $this->timers->attach($timer, $scheduledAt);
         $this->scheduler->insert($timer, -$scheduledAt);

I can provide example reproducing scripts if preferred.