Reinforcement Learning

Geoff Hulten
 Reinforcement Learning
• Learning to interact with an environment
• Robots, games, process control
• With limited human training
• Where the ‘right thing’ isn’t obvious Agent

Reward

Action
State
• Supervised Learning:
• Goal:
• Data:
Environment

• Reinforcement Learning:
• Goal:
Maximize

• Data:
TD-Gammon – Tesauro ~1995 State: Board State
Actions: Valid Moves
Reward: Win or Lose

• Net with 80 hidden units,

initialize to random weights

• Select move based on network

estimate & shallow search
P(win)

• Learn by playing against itself

• 1.5 million games of training

-> competitive with world class players
 Atari 2600 games

State: Raw Pixels

Actions: Valid Moves
Reward: Game Score

• Same model/parameters for

~50 games

https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf
Robotics and
Locomotion

State:
Joint States/Velocities
Accelerometer/Gyroscope
Terrain
Actions: Apply Torque to Joints
Reward: Velocity – { stuff }

https://youtu.be/hx_bgoTF7bs

2017 paper https://arxiv.org/pdf/1707.02286.pdf

 Alpha Go State: Board State
Actions: Valid Moves
Reward: Win or Lose
• Learning how to beat humans at ‘hard’ games (search
space too big)

• Far surpasses (Human) Supervised learning

• Algorithm learned to outplay humans at chess in 24 hours

https://deepmind.com/documents/119/agz_unformatted_nature.pdf
How Reinforcement Learning is Different
• Delayed Reward

• Agent chooses training data

• Explore vs Exploit (Life long learning)

• Very different terminology (can be confusing)

Setup for Reinforcement Learning
Markov Decision Process Policy
(environment) (agent’s behavior)
• Discrete-time stochastic control process • – The action to take in state

• Each time step, :

• Agent chooses action from set • Goal maximize:
• Moves to new state with probability:
Probability of moving to each state
• Receives reward:
• – Tradeoff immediate vs future

• Every outcome depends on and

• Nothing depends on previous states/actions

Reward for making that move Value of being in that state

 Simple Example of Agent in an Environment
State:
Map Locations
Score: 100
0
0, 0 1, 0 2, 0
100
Actions:
Move within map
Reaching chest ends episode 0, 1 1, 1 2, 1

0, 2 1, 2 2, 2
…

Reward:
100 at chest
0 for others
Policies
Policy Evaluating Policies

0, 0 1, 0 2, 0

12.5 100

0, 1 1, 1 2, 1

50

0, 2 1, 2 2, 2

Move to <1,1>
Move to <0,1> Move to <1,0>
Move to <2,0>

𝜋
Policy could be better
𝑉 ¿
Q learning
Learn a policy that optimizes for all states, using:
• No prior knowledge of state transition probabilities:
• No prior knowledge of the reward function:

Approach:
• Initialize estimate of discounted reward for every state/action pair:
• Repeat (for a while):
• Take a random action from
• Receive and from environment
• Update = +
• Random restart if in terminal state

1 Exploration Policy:
∝𝑣 =
1+𝑣𝑖𝑠𝑖𝑡𝑠( 𝑠 , 𝑎)
 Example of Q learning
(round 1)

• Initialize to 0
• Random initial state =
0, 0 1, 0 2, 0
0
• Random action from 0

0
0 0
0 0 100
0
0, 1 1, 1 2, 1
0 0

• Update
0 0
0 0 0
• Random action from 0, 2
0
1, 2
0
2, 2
0

0 0

0 0

• Update

• No more moves possible, start again…

 Example of Q learning
(round 2)

• Round 2: Random initial state =

• Random action from 0, 0 1, 0 2, 0
0 0

0
0 0
0 0 100
0, 1 1, 1 2, 1
• Update + * 100 0 0

0 0
0 0 0
• Random action from 0, 2
0
1, 2
0
2, 2
0
50

0 0

0 0

• Update

• No more moves possible, start again…

𝛾=0.5
 Example of Q learning
(some acceleration…)
𝛾=0.5

• Random Initial State 0, 0

0
1, 0
0
2, 0

0
0 0
• Update 0, 1
0
1, 1
0 100
2, 1
0 0
50

0 0
• Update 0
0
0
0
50
0

0, 2 1, 2 2, 2
0 25
0

0 0
 Example of Q learning
(some acceleration…)
𝛾=0.5

• Random Initial State 0, 0

0
1, 0
100
0
2, 0

0
0 0
• Update 0, 1
0
1, 1
0 100
2, 1
25
0 50

0 0
• Update 0
0
0
0
50
0

0, 2 1, 2 2, 2
0 25

0 0
 Example of Q learning
( after many, many runs…)

0, 0 1, 0 2, 0
50 100
• converged
25
12.5 25
25 50 100
0, 1 1, 1 2, 1
• Policy is: 25 50

12.5 25
6.25 12.5 25
12.5 25 50
0, 2 1, 2 2, 2
12.5 25

6.25 12.5
 Challenges for Reinforcement Learning
• When there are many
states and actions Turns Remaining: 15

• When the episode can

end without reward

• When there is a
‘narrow’ path to Each stepexploring
Random ~50% probability of of
will fall off going
ropewrong
~97%way – P(reaching
of the time goal) ~ 0.01%
reward
 Reward Shaping

• Hand craft intermediate

objectives that yield
reward

• Encourage the right type

of exploration

• Requires custom human

work

• Risks of learning to game

the rewards
Memory
• Retrain on previous explorations
0, 0 1, 0 2, 0
0
50 100

• Maintain samples of:

0
0 0
25
0 0
50 100
0
0, 1 1, 1 2, 1
25
0 0
50

0 0
0 0 0
25
0
25 0 50
0
0, 2 1, 2 2, 2
• Useful when 0 25
0

• It is cheaper to use some RAM/CPU than to run 0 12.5

0
more simulations
Replay
Replay
Do itana exploration
bunchexploration
a different of times
• It is hard to get to reward so you want to
leverage it for as much as possible when it
happens
Gym – toolkit for reinforcement learning
CartPole
import gym

env = gym.make('CartPole-v0')

import random
import QLearning # Your implementation goes here...
import Assignment7Support

trainingIterations = 20000

qlearner = QLearning.QLearning(<Parameters>)

for trialNumber in range(trainingIterations):

observation = env.reset()
reward = 0
for i in range(300):
Reward +1 per step the pole remains up env.render() # Comment out to make much faster...

currentState = ObservationToStateSpace(observation)
action = qlearner.GetAction(currentState, <Parameters>)
MountainCar
oldState = ObservationToStateSpace(observation)
observation, reward, isDone, info = env.step(action)
newState = ObservationToStateSpace(observation)

qlearner.ObserveAction(oldState, action, newState, reward, …)

if isDone:
if(trialNumber%1000) == 0:
print(trialNumber, i, reward)
break

# Now you have a policy in qlearner – use it...

Reward 200 at flag -1 per step

https://gym.openai.com/docs/
Some Problems with QLearning
• State space is continuous
• Must approximate by discretizing

• Treats states as identities print(env.observation_space.high)

#> array([ 2.4 , inf, 0.20943951, inf])
• No knowledge of how states relate print(env.observation_space.low)
• Requires many iterations to fill in #> array([-2.4 , -inf, -0.20943951, -inf])

• Converging can be difficult with

randomized transitions/rewards
 Policy Gradients

• Q-learning -> learn a value function

• = an estimate of the expected
discounted reward of taking from
• Performance time: take the action
that has the highest estimated value

• Policy Gradient -> learn policy

directly
• Probability distribution over
• Performance time: choose action
according to distribution

Example from: https://www.youtube.com/watch?v=tqrcjHuNdmQ

 Policy Gradients

• Receive a frame
• Forward propagate to get
• Select by sampling from
• Find the gradient that makes more likely –
store it
• Play the rest of the game
• If won, take a step in direction
• If lost, take a step in direction

One per action

Sum and step
in correct direction
Policy Gradients – reward shaping
Not relevant to outcome(?)

Less important to outcome More important to outcome

 Summary Agent
Reinforcement Learning:

Reward

Action
• Goal: Maximize

State
• Data:

Environment

Many (awesome) recent successes:

• Robotics
• Surpassing humans at difficult games
• Doing it with (essentially) zero human knowledge (Simple) Approaches:
• Q-Learning -> discounted reward of action
• Policy Gradients -> Probability distribution over
Challenges: • Reward Shaping
• When the episode can end without reward • Memory
• When there is a ‘narrow’ path to reward • Lots of parameter tweaking…
• When there are many states and actions