Lecture 2: Making Sequences of Good Decisions Given

a Model of the World

Emma Brunskill

CS234 Reinforcement Learning

Spring 2024

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 1 / 65
 L2N1 Quick Check Your Understanding 1. Participation
Poll

In a Markov decision process, a large discount factor γ means that short

term rewards are much more influential than long term rewards. [Enter
your answer in participation poll ]
True
False
Don’t know
Question for today’s lecture (not for poll): Can we construct algorithms
for computing decision policies so that we can guarantee with additional
computation / iterations, we monotonically improve the decision policy?
Do all algorithms satisfy this property?

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 2 / 65
 L2N1 Quick Check Your Understanding 1. Participation
Poll
In a Markov decision process, a large discount factor γ means that short
term rewards are much more influential than long term rewards. [Enter
your answer in the poll]
True
False
Don’t know
False. A large γ implies we weigh delayed / long term rewards more.
γ = 0 only values immediate rewards

Question for today’s lecture (not for poll): Can we construct algorithms
for computing decision policies so that we can guarantee with additional
computation / iterations, we monotonically improve the decision policy?
Do all algorithms satisfy this property?
Yes it is possible! We will see this today. Not all of them do.

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 3 / 65
 Class Tasks and Updates

Homework 1 out shortly. Due Friday April 12 at 6pm.

Office hours will start Friday. See Ed for days, times of group and 1:1
office hours, and we will update the calendar on the website with
locations shortly.

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 4 / 65
 Today’s Plan

Last Time:
Introduction
Components of an agent: model, value, policy
This Time:
Making good decisions given a Markov decision process
Next Time:
Policy evaluation when don’t have a model of how the world works

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 5 / 65
 Today: Given a model of the world

Markov Processes (last time)

Markov Reward Processes (MRPs) (continue from last time)
Markov Decision Processes (MDPs)
Evaluation and Control in MDPs

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 6 / 65
 Return & Value Function

Definition of Horizon (H)

Number of time steps in each episode
Can be infinite
Otherwise called finite Markov reward process
Definition of Return, Gt (for a MRP)
Discounted sum of rewards from time step t to horizon H

Gt = rt + γrt+1 + γ 2 rt+2 + · · · + γ H−1 rt+H−1

Definition of State Value Function, V (s) (for a MRP)

Expected return from starting in state s

V (s) = E[Gt |st = s] = E[rt + γrt+1 + γ 2 rt+2 + · · · + γ H−1 rt+H−1 |st = s]

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 7 / 65
 Computing the Value of an Infinite Horizon Markov
Reward Process

Markov property provides structure

MRP value function satisfies
X
V (s) = R(s) + γ P(s ′ |s)V (s ′ )
s ′ ∈S
|{z}
Immediate reward | {z }
Discounted sum of future rewards

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 8 / 65
 Matrix Form of Bellman Equation for MRP

For finite state MRP, we can express V (s) using a matrix equation
 
  P(s1 |s1 ) · · ·
  P(sN |s1 )  
V (s1 ) R(s1 )  P(s1 |s2 ) · · · V (s1 )
 ..   ..  P(sN |s2 )   . 

 . = . +γ   .. 

.. .. ..
 . . . 
V (sN ) R(sN ) V (sN )
P(s1 |sN ) · · · P(sN |sN )
V = R + γPV

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 9 / 65
 Analytic Solution for Value of MRP

For finite state MRP, we can express V (s) using a matrix equation
 
    P(s1 |s1 ) · · · P(sN |s1 )  
V (s1 ) R(s1 )  P(s1 |s2 ) · · · V (s1 )
 ..   ..  P(sN |s2 )   . 

 . = . +γ   .. 

.. .. ..
 . . . 
V (sN ) R(sN ) V (sN )
P(s1 |sN ) · · · P(sN |sN )
V = R + γPV
V − γPV = R
(I − γP)V = R
V = (I − γP)−1 R

Solving directly requires taking a matrix inverse ∼ O(N 3 )

Requires that (I − γP) is invertible
Emma Brunskill (CS234 Reinforcement Learning)
Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 10 / 65
 Iterative Algorithm for Computing Value of a MRP

Dynamic programming
Initialize V0 (s) = 0 for all s
For k = 1 until convergence
For all s in S
X
Vk (s) = R(s) + γ P(s ′ |s)Vk−1 (s ′ )
s ′ ∈S

Computational complexity: O(|S|2 ) for each iteration (|S| = N)

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 11 / 65
 Markov Decision Process (MDP)

Markov Decision Process is Markov Reward Process + actions

Definition of MDP
S is a (finite) set of Markov states s ∈ S
A is a (finite) set of actions a ∈ A
P is dynamics/transition model for each action, that specifies
P(st+1 = s ′ |st = s, at = a)
R is a reward function1

R(st = s, at = a) = E[rt |st = s, at = a]

Discount factor γ ∈ [0, 1]

MDP is a tuple: (S, A, P, R, γ)

1
Reward is sometimes defined as a function of the current state, or as a function of
the (state, action, next state) tuple. Most frequently in this class, we will assume reward
is a function of state and action
Emma Brunskill (CS234 Reinforcement Learning)
Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 12 / 65
 Example: Mars Rover MDP

!" !# !$ !% !& !' !(

   
1 0 0 0 0 0 0 0 1 0 0 0 0 0
1 0 0 0 0 0 0 0 0 1 0 0 0 0
   
0 1 0 0 0 0 0 0 0 0 1 0 0 0
′ ′
   
P(s |s, a1 ) = 
0 0 1 0 0 0 0 P(s |s, a2 ) = 

0 0 0 0 1 0 0
0 0 0 1 0 0 0 0 0 0 0 0 1 0
   
0 0 0 0 1 0 0  0 0 0 0 0 0 1
0 0 0 0 0 1 0 0 0 0 0 0 0 1

2 deterministic actions
Emma Brunskill (CS234 Reinforcement Learning)
Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 13 / 65
 MDP Policies

Policy specifies what action to take in each state

Can be deterministic or stochastic
For generality, consider as a conditional distribution
Given a state, specifies a distribution over actions
Policy: π(a|s) = P(at = a|st = s)

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 14 / 65
 MDP + Policy

MDP + π(a|s) = Markov Reward Process

Precisely, it is the MRP (S, R π , P π , γ), where
X
R π (s) = π(a|s)R(s, a)
a∈A
X
′
π
P (s |s) = π(a|s)P(s ′ |s, a)
a∈A

Implies we can use same techniques to evaluate the value of a policy

for a MDP as we could to compute the value of a MRP, by defining a
MRP with R π and P π

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 15 / 65
 MDP Policy Evaluation, Iterative Algorithm

Initialize V0 (s) = 0 for all s

For k = 1 until convergence
For all s in S
" #
X X
Vkπ (s) = π(a|s) R(s, a) + γ p(s ′ |s, a)Vk−1
π
(s ′ )
a s ′ ∈S

This is a Bellman backup for a particular policy

Note that if the policy is deterministic then the above update
simplifies to
X
Vkπ (s) = R(s, π(s)) + γ p(s ′ |s, π(s))Vk−1
π
(s ′ )
s ′ ∈S

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 16 / 65
 Exercise L2E1: MDP 1 Iteration of Policy Evaluation, Mars
Rover Example

Dynamics: p(s6 |s6 , a1 ) = 0.5, p(s7 |s6 , a1 ) = 0.5, . . .

Reward: for all actions, +1 in state s1 , +10 in state s7 , 0 otherwise
Let π(s) = a1 ∀s, assume Vk =[1 0 0 0 0 0 10] and k = 1, γ = 0.5
Compute Vk+1 (s6 )
See answer at the end of the slide deck. If you’d like practice, work this
out and then check your answers.

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 17 / 65
 Check Your Understanding Poll L2N2

!" !# !$ !% !& !' !(

We will shortly be interested in not just evaluating the value of a

single policy, but finding an optimal policy. Given this it is informative
to think about properties of the potential policy space.
First for the Mars rover example [ 7 discrete states (location of
rover); 2 actions: Left or Right]
How many deterministic policies are there?
Select answer on the participation poll: 2 / 14 / 72 / 27 / Not sure
Is the optimal policy (one with highest value) for a MDP unique?
Select answer on the participation poll: Yes / No / Not sure
Emma Brunskill (CS234 Reinforcement Learning)
Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 18 / 65
 Check Your Understanding L2N2

!" !# !$ !% !& !' !(

7 discrete states (location of rover)

2 actions: Left or Right
How many deterministic policies are there?
27

Is the highest reward policy for a MDP always unique?

No, there may be two policies with the same (maximal) value
function.

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 19 / 65
 MDP Control

Compute the optimal policy

π ∗ (s) = arg max V π (s)

π

There exists a unique optimal value function

Optimal policy for a MDP in an infinite horizon problem is
deterministic

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 20 / 65
 MDP Control

Compute the optimal policy

π ∗ (s) = arg max V π (s)

π

There exists a unique optimal value function

Optimal policy for a MDP in an infinite horizon problem (agent acts
forever is
Deterministic
Stationary (does not depend on time step)
Unique? Not necessarily, may have two policies with identical (optimal)
values

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 21 / 65
 Policy Search

One option is searching to compute best policy

Number of deterministic policies is |A||S|
Policy iteration is generally more efficient than enumeration

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 22 / 65
 MDP Policy Iteration (PI)

Set i = 0
Initialize π0 (s) randomly for all states s
While i == 0 or ∥πi − πi−1 ∥1 > 0 (L1-norm, measures if the policy
changed for any state):
V πi ← MDP V function policy evaluation of πi
πi+1 ← Policy improvement
i =i +1

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 23 / 65
 New Definition: State-Action Value Q

State-action value of a policy

X
Q π (s, a) = R(s, a) + γ P(s ′ |s, a)V π (s ′ )
s ′ ∈S

Take action a, then follow the policy π

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 24 / 65
 Policy Improvement

Compute state-action value of a policy πi

For s in S and a in A:
X
Q πi (s, a) = R(s, a) + γ P(s ′ |s, a)V πi (s ′ )
s ′ ∈S

Compute new policy πi+1 , for all s ∈ S

πi+1 (s) = arg max Q πi (s, a) ∀s ∈ S

a

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 25 / 65
 MDP Policy Iteration (PI)

Set i = 0
Initialize π0 (s) randomly for all states s
While i == 0 or ∥πi − πi−1 ∥1 > 0 (L1-norm, measures if the policy
changed for any state):
V πi ← MDP V function policy evaluation of πi
πi+1 ← Policy improvement
i =i +1

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 26 / 65
 Delving Deeper Into Policy Improvement Step

X
Q πi (s, a) = R(s, a) + γ P(s ′ |s, a)V πi (s ′ )
s ′ ∈S

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 27 / 65
 Delving Deeper Into Policy Improvement Step

X
Q πi (s, a) = R(s, a) + γ P(s ′ |s, a)V πi (s ′ )
s ′ ∈S
X
max Q πi (s, a) ≥ R(s, πi (s)) + γ P(s ′ |s, πi (s))V πi (s ′ ) = V πi (s)
a
s ′ ∈S

πi+1 (s) = arg max Q πi (s, a)

a

Suppose we take πi+1 (s) for one action, then follow πi forever
Our expected sum of rewards is at least as good as if we had always
followed πi
But new proposed policy is to always follow πi+1 ...

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 28 / 65
 Monotonic Improvement in Policy

Definition
V π1 ≥ V π2 : V π1 (s) ≥ V π2 (s), ∀s ∈ S
Proposition: V πi+1 ≥ V πi with strict inequality if πi is suboptimal,
where πi+1 is the new policy we get from policy improvement on πi

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 29 / 65
 Proof: Monotonic Improvement in Policy

V πi (s) ≤ max Q πi (s, a)

a
X
= max R(s, a) + γ P(s ′ |s, a)V πi (s ′ )
a
s ′ ∈S

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 30 / 65
 Proof: Monotonic Improvement in Policy

V πi (s) ≤ max Q πi (s, a)

a
X
= max R(s, a) + γ P(s ′ |s, a)V πi (s ′ )
a
s ′ ∈S
X
=R(s, πi+1 (s)) + γ P(s ′ |s, πi+1 (s))V πi (s ′ ) //by the definition of πi+1
s ′ ∈S
X  
≤R(s, πi+1 (s)) + γ P(s ′ |s, πi+1 (s)) max
′
Q πi ′ ′
(s , a )
a
s ′ ∈S
X
=R(s, πi+1 (s)) + γ P(s ′ |s, πi+1 (s))
s ′ ∈S
!
X
R(s ′ , πi+1 (s ′ )) + γ P(s ′′ |s ′ , πi+1 (s ′ ))V πi (s ′′ )
s ′′ ∈S
..
.
=V πi+1 (s)

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 31 / 65
 Check Your Understanding L2N3: Policy Iteration (PI)

Note: all the below is for finite state-action spaces

Set i = 0
Initialize π0 (s) randomly for all states s
While i == 0 or ∥πi − πi−1 ∥1 > 0 (L1-norm, measures if the policy
changed for any state):
V πi ← MDP V function policy evaluation of πi
πi+1 ← Policy improvement
i =i +1
If policy doesn’t change, can it ever change again?
Select on participation poll: Yes / No / Not sure
Is there a maximum number of iterations of policy iteration?
Select on participation poll: Yes / No / Not sure

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 32 / 65
 Lecture Break after Policy Iteration

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 33 / 65
 Results for Check Your Understanding L2N3 Policy
Iteration
Note: all the below is for finite state-action spaces
Set i = 0
Initialize π0 (s) randomly for all states s
While i == 0 or ∥πi − πi−1 ∥1 > 0 (L1-norm, measures if the policy
changed for any state):
V πi ← MDP V function policy evaluation of πi
πi+1 ← Policy improvement
i =i +1
If policy doesn’t change, can it ever change again?
No

Is there a maximum number of iterations of policy iteration?

|A||S| since that is the maximum number of policies, and as the policy
improvement step is monotonically improving, each policy can only
appear in one round of policy iteration unless it is an optimal policy.

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 34 / 65
 Check Your Understanding Explanation of Policy Not
Changing

Suppose for all s ∈ S, πi+1 (s) = πi (s)

Then for all s ∈ S, Q πi+1 (s, a) = Q πi (s, a)
Recall policy improvement step
X
Q πi (s, a) = R(s, a) + γ P(s ′ |s, a)V πi (s ′ )
s ′ ∈S

πi+1 (s) = arg max Q πi (s, a)

a
πi+1
πi+2 (s) = arg max Q (s, a) = arg max Q πi (s, a)
a a

Therefore policy cannot ever change again

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 35 / 65
 MDP: Computing Optimal Policy and Optimal Value

Policy iteration computes infinite horizon value of a policy and then

improves that policy
Value iteration is another technique
Idea: Maintain optimal value of starting in a state s if have a finite
number of steps k left in the episode
Iterate to consider longer and longer episodes

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 36 / 65
 Bellman Equation and Bellman Backup Operators

Value function of a policy must satisfy the Bellman equation

X
V π (s) = R π (s) + γ P π (s ′ |s)V π (s ′ )
s ′ ∈S

Bellman backup operator

Applied to a value function
Returns a new value function
Improves the value if possible
" #
X
′ ′
BV (s) = max R(s, a) + γ p(s |s, a)V (s )
a
s ′ ∈S

BV yields a value function over all states s

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 37 / 65
 Value Iteration (VI)

Set k = 1
Initialize V0 (s) = 0 for all states s
Loop until convergence: (for ex. ||Vk+1 − Vk ||∞ ≤ ϵ)
For each state s
" #
X
′ ′
Vk+1 (s) = max R(s, a) + γ P(s |s, a)Vk (s )
a
s ′ ∈S

View as Bellman backup on value function

Vk+1 = BVk
" #
X
′ ′
πk+1 (s) = arg max R(s, a) + γ P(s |s, a)Vk (s )
a
s ′ ∈S

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 38 / 65
 Policy Iteration as Bellman Operations

Bellman backup operator B π for a particular policy is defined as

X
B π V (s) = R π (s) + γ P π (s ′ |s)V (s ′ )
s ′ ∈S

Policy evaluation amounts to computing the fixed point of B π

To do policy evaluation, repeatedly apply operator until V stops
changing
V π = BπBπ · · · BπV

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 39 / 65
 Policy Iteration as Bellman Operations

Bellman backup operator B π for a particular policy is defined as

X
B π V (s) = R π (s) + γ P π (s ′ |s)V (s ′ )
s ′ ∈S

To do policy improvement
" #
X
πk+1 (s) = arg max R(s, a) + γ P(s ′ |s, a)V πk (s ′ )
a
s ′ ∈S

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 40 / 65
 Going Back to Value Iteration (VI)
Set k = 1
Initialize V0 (s) = 0 for all states s
Loop until convergence: (for ex. ||Vk+1 − Vk ||∞ ≤ ϵ)
For each state s
" #
X
′ ′
Vk+1 (s) = max R(s, a) + γ P(s |s, a)Vk (s )
a
s ′ ∈S

Equivalently, in Bellman backup notation

Vk+1 = BVk

To extract optimal policy if can act for k + 1 more steps,

" #
X
′ ′
π(s) = arg max R(s, a) + γ P(s |s, a)Vk+1 (s )
a
s ′ ∈S

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 41 / 65
 Contraction Operator

Let O be an operator,and |x| denote (any) norm of x

If |OV − OV ′ | ≤ |V − V ′ |, then O is a contraction operator

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 42 / 65
 Will Value Iteration Converge?

Yes, if discount factor γ < 1, or end up in a terminal state with

probability 1
Bellman backup is a contraction if discount factor, γ < 1
If apply it to two different value functions, distance between value
functions shrinks after applying Bellman equation to each

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 43 / 65
 Proof: Bellman Backup is a Contraction on V for γ < 1

Let ∥V − V ′ ∥ = maxs |V (s) − V ′ (s)| be the infinity norm

   
′ ′ ′ ′ ′ ′
X X
∥BVk − BVj ∥ = max R(s, a) + γ P(s |s, a)Vk (s ) − max R(s, a ) + γ P(s |s, a )Vj (s )
a a′
s ′ ∈S s ′ ∈S

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 44 / 65
 Proof: Bellman Backup is a Contraction on V for γ < 1

Let ∥V − V ′ ∥ = maxs |V (s) − V ′ (s)| be the infinity norm

   
′ ′ ′ ′ ′ ′
X X
∥BVk − BVj ∥ = max R(s, a) + γ P(s |s, a)Vk (s ) − max R(s, a ) + γ P(s |s, a )Vj (s )
a a′
s ′ ∈S s ′ ∈S
 
′ ′ ′ ′
X X
≤ max R(s, a) + γ P(s |s, a)Vk (s ) − R(s, a) − γ P(s |s, a)Vj (s )
a
s ′ ∈S s ′ ∈S

′ ′ ′
X
= max γ P(s |s, a)(Vk (s ) − Vj (s ))
a
s ′ ∈S

′
X
≤ max γ P(s |s, a)∥Vk − Vj ∥)
a
s ′ ∈S

′
X
= max γ∥Vk − Vj ∥ P(s |s, a))
a
s ′ ∈S

= γ∥Vk − Vj ∥

Note: Even if all inequalities are equalities, this is still a contraction if γ < 1

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 45 / 65
 Opportunities for Out-of-Class Practice

Prove value iteration converges to a unique solution for discrete state

and action spaces with γ < 1
Does the initialization of values in value iteration impact anything?
Is the value of the policy extracted from value iteration at each round
guaranteed to monotically improve (if executed in the real infinite
horizon problem), like policy iteartion?
[GPT]
- No, the value of the policy extracted from value iteration at each round is not guaranteed to
monotonically improve in the real infinite-horizon problem. This contrasts with policy iteration,
where the policy's value is guaranteed to improve or remain the same at each step.
- Reason: The value function Vk(s) in value iteration is only an intermediate estimate and does
not necessarily represent the value of a valid policy.
- Greedily extracting a policy at intermediate steps can sometimes lead to policies that perform
worse in the real infinite-horizon problem compared to those extracted at earlier iterations.

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 46 / 65
 Value Iteration for Finite Horizon H

Vk = optimal value if making k more decisions

πk = optimal policy if making k more decisions
Initialize V0 (s) = 0 for all states s
For k = 1 : H
For each state s
" #
X
′ ′
Vk+1 (s) = max R(s, a) + γ P(s |s, a)Vk (s )
a
s ′ ∈S
" #
X
′ ′
πk+1 (s) = arg max R(s, a) + γ P(s |s, a)Vk (s )
a
s ′ ∈S

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 47 / 65
 Computing the Value of a Policy in a Finite Horizon

Alternatively can estimate by simulation

Generate a large number of episodes
Average returns
Concentration inequalities bound how quickly average concentrates to
expected value Concentration inequalities: e.g. Hoeffding's inequality, Bernstein's inequality

Requires no assumption of Markov structure

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 48 / 65
 Example: Mars Rover

!" !# !$ !% !& !' !(

0.4 0.4 0.4 0.4 0.4 0.4

0.4 0.4 0.4 0.4 0.4 0.4

0.6 0.2 0.2 0.2 0.2 0.2 0.6

Reward: +1 in s1 , +10 in s7 , 0 in all other states

Sample returns for sample 4-step (H=4) episodes, γ = 1/2
1 1 1
s4 , s5 , s6 , s7 : 0 + 2 ×0+ 4 ×0+ 8 × 10 = 1.25

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 49 / 65
 Example: Mars Rover

!" !# !$ !% !& !' !(

0.4 0.4 0.4 0.4 0.4 0.4

0.4 0.4 0.4 0.4 0.4 0.4

0.6 0.2 0.2 0.2 0.2 0.2 0.6

Reward: +1 in s1 , +10 in s7 , 0 in all other states

Sample returns for sample 4-step (H=4) episodes, start state s4 ,
γ = 1/2
1 1 1
s4 , s5 , s6 , s7 : 0 + 2 ×0+ 4 ×0+ 8 × 10 = 1.25
1 1 1
s4 , s4 , s5 , s4 : 0 + 2 ×0+ 4 ×0+ 8 ×0=0
1 1 1
s4 , s3 , s2 , s1 : 0 + 2 ×0+ 4 ×0+ 8 × 1 = 0.125

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 50 / 65
 Question: Finite Horizon Policies

Set k = 1
Initialize V0 (s) = 0 for all states s
Loop until k == H:
For each state s
X
Vk+1 (s) = max R(s, a) + γ P(s ′ |s, a)Vk (s ′ )
a
s ′ ∈S
X
πk+1 (s) = arg max R(s, a) + γ P(s ′ |s, a)Vk (s ′ )
a
s ′ ∈S

Is optimal policy stationary (independent of time step) in finite horizon

tasks?

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 51 / 65
 Question: Finite Horizon Policies

Set k = 1
Initialize V0 (s) = 0 for all states s
Loop until k == H:
For each state s
X
Vk+1 (s) = max R(s, a) + γ P(s ′ |s, a)Vk (s ′ )
a
s ′ ∈S
X
πk+1 (s) = arg max R(s, a) + γ P(s ′ |s, a)Vk (s ′ )
a
s ′ ∈S

Is optimal policy stationary (independent of time step) in finite horizon

tasks?
In general no.

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 52 / 65
 Value vs Policy Iteration

Value iteration:
Compute optimal value for horizon = k
Note this can be used to compute optimal policy if horizon = k
Increment k
Policy iteration
Compute infinite horizon value of a policy
Use to select another (better) policy
Closely related to a very popular method in RL: policy gradient

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 53 / 65
 RL Terminology: Models, Policies, Values

Model: Mathematical models of dynamics and reward

Policy: Function mapping states to actions
Value function: future rewards from being in a state and/or action
when following a particular policy

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 54 / 65
 What You Should Know

Define MP, MRP, MDP, Bellman operator, contraction, model,

Q-value, policy
Be able to implement
Value Iteration
Policy Iteration
Give pros and cons of different policy evaluation approaches
Be able to prove contraction properties
Limitations of presented approaches and Markov assumptions
Which policy evaluation methods require the Markov assumption?

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 55 / 65
 Where We Are

Last Time:
Introduction
Components of an agent: model, value, policy
This Time:
Making good decisions given a Markov decision process
Next Time:
Policy evaluation when don’t have a model of how the world works

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 56 / 65
 Exercise L2E1: MDP 1 Iteration of Policy Evaluation, Mars
Rover Example, Answer

Dynamics: p(s6 |s6 , a1 ) = 0.5, p(s7 |s6 , a1 ) = 0.5, . . .

Reward: for all actions, +1 in state s1 , +10 in state s7 , 0 otherwise
Let π(s) = a1 ∀s, assume Vk =[1 0 0 0 0 0 10] and k = 1, γ = 0.5
Compute Vk+1 (s6 )
X
Vk+1 (s6 ) = r (s6 ) + γ p(s ′ |s6 , a1 )Vk (s ′ ) (1)
s′
= 0 + 0.5 ∗ (0.5 ∗ 10 + 0.5 ∗ 0) (2)
= 2.5 (3)

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 57 / 65
 Check Your Understanding L2N1: MDP 1 Iteration of
Policy Evaluation, Mars Rover Example
Dynamics: p(s6 |s6 , a1 ) = 0.5, p(s7 |s6 , a1 ) = 0.5, . . .
Reward: for all actions, +1 in state s1 , +10 in state s7 , 0 otherwise
Let π(s) = a1 ∀s, assume Vk =[1 0 0 0 0 0 10] and k = 1, γ = 0.5
X
Vkπ (s) = r (s, π(s)) + γ p(s ′ |s, π(s))Vk−1
π
(s ′ )
s ′ ∈S
Vk+1 (s6 ) = r (s6 , a1 ) + γ ∗ 0.5 ∗ Vk (s6 ) + γ ∗ 0.5 ∗ Vk (s7 )

Vk+1 (s6 ) = 0 + 0.5 ∗ 0.5 ∗ 0 + .5 ∗ 0.5 ∗ 10

Vk+1 (s6 ) = 2.5

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 58 / 65
 Full Observability: Markov Decision Process (MDP)

MDPs can model a huge number of interesting problems and settings

Bandits: single state MDP
Optimal control mostly about continuous-state MDPs
Partially observable MDPs = MDP where state is history

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 59 / 65
 Recall: Markov Property

Information state: sufficient statistic of history

State st is Markov if and only if:

p(st+1 |st , at ) = p(st+1 |ht , at )

Future is independent of past given present

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 60 / 65
 Markov Process or Markov Chain

Memoryless random process

Sequence of random states with Markov property
Definition of Markov Process
S is a (finite) set of states (s ∈ S)
P is dynamics/transition model that specifices p(st+1 = s ′ |st = s)
Note: no rewards, no actions
If finite number (N) of states, can express P as a matrix
 
P(s1 |s1 ) P(s2 |s1 ) · · · P(sN |s1 )
 P(s1 |s2 ) P(s2 |s2 ) · · · P(sN |s2 ) 
P=
 
.. .. . . .. 
 . . . . 
P(s1 |sN ) P(s2 |sN ) · · · P(sN |sN )

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 61 / 65
 Example: Mars Rover Markov Chain Transition Matrix, P

!" !# !$ !% !& !' !(

0.4 0.4 0.4 0.4 0.4 0.4

0.4 0.4 0.4 0.4 0.4 0.4

0.6 0.2 0.2 0.2 0.2 0.2 0.6

 
0.6 0.4 0 0 0 0 0
0.4 0.2 0.4 0 0 0 0
 
 0 0.4 0.2 0.4 0 0 0
 
0
P= 0 0.4 0.2 0.4 0 0
0 0 0 0.4 0.2 0.4 0 
 
0 0 0 0 0.4 0.2 0.4
0 0 0 0 0 0.4 0.6

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 62 / 65
 Example: Mars Rover Markov Chain Episodes

!" !# !$ !% !& !' !(

0.4 0.4 0.4 0.4 0.4 0.4

0.4 0.4 0.4 0.4 0.4 0.4

0.6 0.2 0.2 0.2 0.2 0.2 0.6

Example: Sample episodes starting from S4

s4 , s5 , s6 , s7 , s7 , s7 , . . .
s4 , s4 , s5 , s4 , s5 , s6 , . . .
s4 , s3 , s2 , s1 , . . .

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 63 / 65
 Markov Reward Process (MRP)

Markov Reward Process is a Markov Chain + rewards

Definition of Markov Reward Process (MRP)
S is a (finite) set of states (s ∈ S)
P is dynamics/transition model that specifices P(st+1 = s ′ |st = s)
R is a reward function R(st = s) = E[rt |st = s]
Discount factor γ ∈ [0, 1]
Note: no actions
If finite number (N) of states, can express R as a vector

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 64 / 65
 Example: Mars Rover MRP

!" !# !$ !% !& !' !(

0.4 0.4 0.4 0.4 0.4 0.4

0.4 0.4 0.4 0.4 0.4 0.4

0.6 0.2 0.2 0.2 0.2 0.2 0.6

Reward: +1 in s1 , +10 in s7 , 0 in all other states

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model
Spring of
2024
the World 65 / 65