Recursive least-squares temporal difference learning for adaptive traffic signal control at intersection

Abstract

This paper presents a new method to solve the scheduling problem of adaptive traffic signal control at intersection. The method involves recursive least-squares temporal difference (RLS-TD(λ)) learning that is integrated into approximate dynamic programming. The learning mechanism of RLS-TD(λ) is to make an adaptation of linear function approximation by updating its parameters based on environmental feedback. This study investigates the method implementation after modeling a traffic dynamic system at intersection in discrete time. In the model, different traffic control schemes regarding signal phase sequence are considered, especially the defined adaptive phase sequence (APS). By simulating traffic scenarios, RLS-TD(λ) is superior to TD(λ) for updating functional parameters in the approximation, and APS outperforms other conventional control schemes on reducing traffic delay. By comparing with other traffic signal control algorithms, the proposed algorithm yields satisfying results in terms of traffic delay and computation time.

Appendix: Derivation of RLS-TD(λ) in M-step planning

In M-step planning, the objective function Eq. (11) is modified as:

$$O(\theta_{t} ) = \frac{1}{t}\sum\limits_{i = 1}^{t} {\left( {\sum\limits_{k = i}^{i + M - 1} {\gamma^{k - i} } r_{k} - (\phi_{i} - \gamma^{M} \phi_{i + M} )^{\text{T}} \theta_{t} } \right)^{2} } .$$

(32)

According to related theories [14, 47], we can rewrite Eq. (12) by using $\phi_{t}$ as the instrumental variable in LS-TD. That is,

$$\theta_{t} = \left( {\frac{1}{t}\sum\limits_{i = 1}^{t} {\phi_{i} (\phi_{i} - \gamma^{M} \phi_{i + M} )^{\text{T}} } } \right)^{ - 1} \left( {\frac{1}{t}\sum\limits_{i = 1}^{t} {\phi_{i} \sum\limits_{k = i}^{i + M - 1} {\gamma^{k - i} } r_{k} } } \right) .$$

(33)

In LS-TD(λ), $\theta_{t}$ can be estimated as

$$\begin{aligned} \theta_{t} = \left( {\frac{1}{t}\sum\limits_{i = 1}^{t} {z_{i} (\phi_{i} - \gamma^{M} \phi_{i + M} )^{\text{T}} } } \right)^{ - 1} \left( {\frac{1}{t}\sum\limits_{i = 1}^{t} {z_{i} \sum\limits_{k = i}^{i + M - 1} {\gamma^{k - i} } r_{k} } } \right) \hfill \\ \;\;\;\; \approx \left( {\sum\limits_{i = 1}^{t} {z_{i} (\phi_{i} - \gamma^{M} \phi_{i + M} )^{\text{T}} } } \right)^{ - 1} \left( {\sum\limits_{i = 1}^{t} {z_{i} \sum\limits_{k = i}^{i + M - 1} {\gamma^{k - i} } r_{k} } } \right) \hfill \\ \end{aligned}$$

(34)

where using the eligibility vector $z_{t}$ in Eq. (10) substitutes the variable $\phi_{t}$. According to matrix inverse lemma and RLS-TD(λ) [12], the parameter vector $\theta_{t}$ updated by RLS-TD(λ) in M-step planning (in Eqs. (27), (28), and (29)) can be guaranteed.