10 QUESTION FOR MID-II ANSWER KEY

Qno Question
1 Design a simple MDP for a robotic agent navigating through a grid-based
environment With rewards and penalties.
A simple Markov Decision Process (MDP) for a robotic agent navigating through a
grid-based environment. In this scenario, the robot moves on a grid, and each cell
represents a state. Here are the key components of our MDP:
State Space (S):
The state space consists of all possible grid cells. Each cell represents a unique
state.
For example, if we have a 5x5 grid, there will be 25 states.
Action Space (A):
The robot can take actions to move from one cell to another.
Common actions might include moving up, down, left, or right.
Let’s define (A = {{up}, {down},{left}, {right}})
Transition Probabilities (P):
Given the current state and action, we need to define the transition probabilities to
other states.
For example:
If the robot is in state (s) and takes the action “up,” it might move to state (s’) with
a certain probability.
We can represent this as (P(s’ | s, {up})).
Rewards (r):
 Assign rewards to each state-action pair.
 Common rewards might include:
 Positive reward for reaching a goal state (e.g., +10).
 Negative reward for hitting obstacles (e.g., -5).
 Small negative reward for each step (e.g., -1).
 Define the reward function (R(s, a, s’)).
Goal State:
Identify one or more goal states where the robot receives a high positive reward.
Reaching the goal(s) is the objective of the robot.
Obstacles:
Introduce obstacles (blocked cells) with negative rewards.
The robot should avoid these cells.
Discount Factor (γ):
 Set a discount factor ((0 < 1)) to balance immediate vs. future rewards.
It affects the value function.
Value Function (V):
The value function (V(s)) represents the expected cumulative reward from state (s)
following a given policy.
We can compute it using dynamic programming methods.
Policy:
A policy (π) specifies the action to take in each state.
The optimal policy maximizes the expected cumulative reward.
Example:
Suppose we have a 3x3 grid:
S -1 G
-1 -1 -1
R -1 S

 (S) represents the start state.

 (G) is the goal state with a reward of +10.
 (R) is an obstacle with a reward of -5.
 Other cells have a small negative reward (-1) for each step.
 The robot’s task is to reach the goal while avoiding obstacles.
Remember that this is a simplified example, and real-world environments may
have more complex dynamics and additional features. You can expand this MDP by
adding more states, actions, and customizing rewards based on your specific use
case.
2 Compare and contrast value iteration and policy iteration methods for solving
MDPs in RL.
The differences between Value Iteration and Policy Iteration methods for solving
Markov Decision Processes (MDPs) in Reinforcement Learning (RL):
Policy Iteration:
Objective: Policy iteration aims to find an optimal policy by iteratively improving
an initial policy.
Phases:
1. Policy Evaluation: Evaluate the current policy by calculating the state value
function (V_π(s)).
 Compute the expected cumulative reward from each state using the Bellman
equation.
2. Policy Improvement: Update the policy by selecting actions that maximize
the expected value.
 Replace the initial policy with a new policy based on one-step look-ahead.
 3. Iteration: Repeat policy evaluation and policy improvement until
convergence.
Advantages:
 Guarantees convergence to an optimal policy and value function.
 Produces a sequence of policies, each improving over the previous one.
Disadvantages:
 Requires solving possibly large linear systems during policy evaluation.
 Each iteration takes (O({card}(S)^3)) time.
 May be computationally expensive for large state spaces12.

Value Iteration:
Objective: Value iteration computes the optimal state value function directly.
Algorithm:
1. Start with an initial value function (V_0(s)).
2. Iteratively update the value function:
V_{k+1}(s) =max_a (R(s, a) +gamma sum_{s’} P(s’|s, a) V_k(s’)) REFER PPT
 The update step looks ahead by considering all possible rewards.
Advantages:
 Converges to the optimal values in a single step.
 Requires only (O({{card}}(S) dot {{card}}(A))) time per iteration.
Disadvantages:
 Does not explicitly maintain a policy.
 May require more iterations to converge compared to policy iteration.
 Assumes a perfect model of the environment.
Comparison:
 Both methods implicitly update the policy and state value function.
 Policy iteration focuses on policy improvement, while value iteration
directly updates the value function.
 Value iteration is guaranteed to converge to the optimal values, whereas
policy iteration converges to an optimal policy and value function.
 Policy iteration involves two phases (evaluation and improvement), while
value iteration combines both steps in a single iteration.
In summary, policy iteration is more intuitive in terms of policy improvement,
while value iteration is computationally efficient and directly finds optimal values.
The choice between them depends on the specific problem and computational
resources available.
3 Analyze how the Bellman Optimality equation changes when the environment has
stochastic transitions and rewards.
The Bellman Optimality equation changes when the environment has stochastic
transitions and rewards in the context of Reinforcement Learning (RL).
Bellman Optimality Equation:
The Bellman Optimality equation expresses the optimal value function (V^(s)) for
a given state (s):
V^(s) = max_a (R(s, a) + gamma sum_{s’} P(s’|s, a) V^*(s’)) REFER PPT
where:
 (R(s, a)) is the immediate reward when taking action (a) in state (s).
 (P(s’|s, a)) represents the transition probability from state (s) to state (s’)
when action (a) is taken.
 (gamma) is the discount factor (0 ≤ (\gamma) < 1).
Stochastic Transitions:
 In a stochastic environment, transitions are uncertain.
 Instead of deterministic transitions, we have transition probabilities (P(s’|s,
a)).
 The agent doesn’t always end up in the same state after taking an action;
there’s randomness.
Stochastic Rewards:
 Similarly, rewards can be stochastic.
 Instead of fixed rewards, we have reward distributions (R(s, a)).
 The actual reward received might vary due to randomness or uncertainty.
Modified Bellman Optimality Equation:

When both transitions and rewards are stochastic, the Bellman Optimality equation
becomes:
V^(s) = max_a (sum_{s’} P(s’|s, a) [R(s, a, s’) + V^(s’)
 The sum accounts for all possible next states (s’).
 The expected value considers both transition probabilities and stochastic
rewards.
Interpretation:
 The optimal value function (V^*(s)) now reflects the expected cumulative
reward considering the randomness in transitions and rewards.
 The agent must balance exploration (to learn about uncertain transitions) and
exploitation (to maximize expected rewards).
Policy Implications:
 The optimal policy derived from (V^*(s)) will adapt to the stochastic nature
of the environment.
 The agent may explore different actions to learn about uncertain outcomes.
In summary, when dealing with stochastic transitions and rewards, the Bellman
Optimality equation accounts for the inherent uncertainty in RL environments. The
 agent’s decisions are influenced by both expected rewards and the likelihood of
transitioning to different states.
4 Apply Temporal Difference learning to update the value function for a specific
state in an RL task.
Temporal Difference (TD) learning updates the value function for a specific state in
a Reinforcement Learning (RL) task.
Temporal Difference Learning:
 TD learning is a model-free RL method that combines elements of both
Monte Carlo and Dynamic Programming approaches.
 Unlike Monte Carlo methods (which wait until the end of an episode), TD
methods update state values in the next time step.
 TD estimates the value function by learning from experience, making it
suitable for online learning scenarios.
TD(0) Update:
Consider a specific state (s_t) in our RL environment.
Suppose the agent takes action (a_t) and receives a reward (r_t).
The TD(0) update for the estimated value of state
(V(s_t)) is as follows:
V(s_t) = V(s_t) + alpha[r_{t+1} +V(s_{t+1}) - V(s_t)
Here:
 (alpha) is a constant step-size parameter (usually small) that controls how
quickly the TD algorithm learns.
 (r_{t+1}) is the reward received after transitioning from state (s_t) to state
(s_{t+1}).
 (gamma) is the discount factor (0 ≤ (\gamma) < 1).
 (V(s_{t+1})) is the estimated value of the next state.
Interpretation:
 The TD error ([r_{t+1} + V(s_{t+1}) - V(s_t)) represents the difference
between the current estimate for (V(s_t)) and the actual reward gained from
transitioning between states.
 By updating (V(s_t)) using this error, we correct the estimate over many
iterations.
Advantages of TD Learning:
 TD methods combine the benefits of both Monte Carlo and DP approaches.
 They allow for online updates without waiting for full episodes.
 TD learning converges to the true value function under certain conditions.
Example:
Imagine a robot navigating a grid-based environment.
At each time step, the robot observes its current state, takes an action, receives a
 reward, and transitions to the next state.
TD learning helps the robot estimate the value of each state based on the observed
rewards and future state values.
In summary, TD learning provides a way to iteratively update value estimates for
specific states, allowing RL agents to learn from their interactions with the
environment.
5 Design an RL agent to navigate a grid world using Fitted Q-learning with function
approximation.
To design an RL agent to navigate a grid world using Fitted Q-learning with function
approximation, you can follow these steps:
Step 1: Initialize Parameters
Initialize the parameters (weights and biases) of the Q-function approximation model. For
simplicity, we'll use a table to represent the Q-function.
Step 2: Interaction with the Environment
The agent interacts with the grid world environment, moving from one state to another and taking
actions according to its current policy.
Step 3: Observe State and Take Action
At each time step t, the agent observes the current state s_t (a grid cell in the grid world). It
selects an action a_t (e.g., up, down, left, right) based on its current Q-function approximation.
Step 4: Receive Reward and Next State
After taking action a_t in state s_t, the agent receives a reward r_t based on the environment's
rules (e.g., +10 for reaching the goal, -1 for hitting an obstacle).
The agent transitions to the next state s_{t+1} based on the action a_t and the environment
dynamics.
Step 5: TD Target Calculation
Calculate the Temporal Difference (TD) target, which represents the target Q-value the Q-
function approximation should aim to approximate.
In Fitted Q-learning, the TD target is the observed reward plus the estimated maximum Q-value
of the next state-action pairs using the Q-function approximation: TD_target = r_t + γ * max_a
Q(s_{t+1}, a)
Step 6: Collect Data for Q-function Approximation
Collect a dataset (D) of state-action pairs (s, a) and their corresponding TD targets (TD_target)
over multiple time steps or episodes.
Step 7: Fitted Q-function Update
Use the collected dataset (D) to update the Q-function approximation model (table) by fitting it to
the TD targets. For each state-action pair (s, a) in the dataset, update the Q-Value in the table to
minimize the Mean Squared Error (MSE) between the predicted Qvalue and the TD target: Q(s,
a) = Q(s, a) + α * (TD_target - Q(s, a))
where α is the learning rate, controlling the step size of the update.
Step 8: Update Parameters
As we are using a table as the Q-function approximation model, there are no parameters
(weights) to update.
Step 9: Repeat
Repeat Steps 2 to 8 for multiple time steps or episodes, allowing the agent to learn and
update the Q-function approximation based on its interactions with the environment.
Step 10: Convergence and Evaluation
Monitor the performance of the Fitted Q-learning algorithm and the convergence of the Q-
function approximation.
Evaluate the learned Q-function or the corresponding policy on test scenarios to assess
the agent's performance in the grid world environment.
In this example, Fitted Q-learning would involve updating the Q-values in the table based on the
observed rewards and the estimated maximum Q-values of the next states.
The agent would continue exploring the grid world environment, gradually improving its
Qfunction approximation to make better decisions and navigate to the goal state while avoiding
obstacles efficiently.
Here is an example implementation of Fitted Q-learning with function approximation in
Python:
import numpy as np
from sklearn.linear_model import SGDRegressor
class FittedQAgent:
def __init__(self, env, n_episodes=1000, gamma=0.99, epsilon=0.1, alpha=0.01):
self.env = env
self.n_episodes = n_episodes
self.gamma = gamma
self.epsilon = epsilon
self.alpha = alpha
self.Q = SGDRegressor(learning_rate='constant')
self.Q.partial_fit(np.zeros((env.observation_space.n,env.action_space.n)),
np.zeros((env.observation_space.n,)))

def fit(self):
for i in range(self.n_episodes):
s = self.env.reset()
done = False
while not done:
if np.random.rand() < self.epsilon:
a = self.env.action_space.sample()
else:
a = np.argmax(self.Q.predict([s])[0])
s_prime, r, done, _ = self.env.step(a)
self.Q.partial_fit([s], [self.Q.predict([s])[0] + self.alpha * (r + self.gamma *
np.max(self.Q.predict([s_prime])[0])-self.Q.predict([s])[0][a])*
np.eye(self.env.action_space.n)[a]])
s = s_prime

def predict(self, s):

return np.argmax(self.Q.predict([s])[0])
This implementation uses stochastic gradient descent to fit a linear model to the Q-function. The
fit method trains the agent using Fitted Q-learning with function approximation, and the predict
method returns the action with the highest Q-value for a given state.
6 Assess the effectiveness of using Eligibility Traces for updating Q-values in a
dynamic environment.
 Eligibility traces are a technique used in reinforcement In Reinforcement Learning (RL),
Eligibility Traces are a mechanism used to update the value function or policy more
efficiently, especially in Temporal Difference (TD) methods.
 They help in handling the credit assignment problem by giving credit to past states and
actions that contributed to the observed rewards and encouraging learning from both
recent and distant experiences.
 The main idea behind Eligibility Traces is to maintain a trace of the states and actions
visited during the agent's interaction with the environment.
 These traces act as a record of "eligibility" for each state-action pair, indicating how much
they contributed to the observed rewards.
 There are different types of Eligibility Traces, such as Accumulating Traces, Replacing
Traces, and Dutch Traces, each with its specific characteristics.
 In Accumulating Traces, a trace is accumulated over time whenever a state-action pair is
visited.
 The trace value increases with each visit, decaying at a specific rate over time.
 When a TD update is performed, the accumulated trace is used to update the value
function or policy.
 The update for a state-action value (Q-value) using Accumulating Traces can be
expressed as:

 Eligibility traces are a technique used in reinforcement learning to update Q-values in a

dynamic environment.
 They are used to keep track of the history of state-action pairs that have been visited and
to update the Q-values of these pairs based on their frequency of occurrence.
 The effectiveness of using eligibility traces for updating Q-values in a dynamic
environment depends on several factors, such as the learning rate, the discount factor, and
the trace decay parameter.
  The eligibility traces can be used to speed up learning in dynamic environments by
propagating knowledge back over time-steps in a single update.
 This can be useful in situations where the environment is constantly changing and the
agent needs to adapt quickly to new conditions.

However, eligibility traces can also introduce additional complexity into the learning process and
may require more computational resources than other methods. Eligibility traces can be difficult
to implement and may not always lead to better performance than other methods.

In conclusion, the effectiveness of using eligibility traces for updating Q-values in a dynamic
environment depends on several factors and may not always lead to better performance than
other methods. However, they can be useful in certain situations and can help speed up learning
in dynamic environments.
7 Develop a Policy Gradient algorithm to train a robotic arm to reach a target in a
simulated environment.
To develop a Policy Gradient algorithm to train a robotic arm to reach a target in a
simulated environment, you can follow these steps:
Introduction:
Policy Gradient algorithms are a family of model-free reinforcement learning (RL)
methods that directly optimize the policy of an agent to find the best actions to take
in different states.
Unlike Q-learning, which approximates the value function and then derives the
policy, Policy Gradient methods focus on directly learning the policy function and
updating it to maximize the expected cumulative reward.
Steps:
Step 1: Initialize Policy Network:
Initialize a parameterized policy network, such as a neural network, with random
weights.
This network takes the state as input and outputs a probability distribution over
actions.
Step 2:Interaction with the Environment:
The agent interacts with the environment and takes actions based on its current
policy.
Step 3. Observe State and Sample Action:
At each time step t, the agent observes the current state s_t and samples an action
a_t from the policy network's output probability distribution.
Step 4: Receive Reward and Next State:
After taking action a_t in state s_t, the agent receives a reward r_t from the
environment and transitions to the next state s_{t+1}.
Step 5: Calculate Policy Gradient:
Calculate the gradient of the policy with respect to its parameters, indicating how
 much the policy should change to improve the expected cumulative reward.
Step 6: Update Policy Parameters:
Use the policy gradient to update the policy network's parameters in the direction
that improves the expected cumulative reward.
This can be done through gradient ascent: θ = θ + α * ∇θ J(θ)
where θ represents the policy network's parameters, J(θ) is the objective function to
maximize (e.g., expected cumulative reward), α is the learning rate, and ∇θ J(θ) is
the policy gradient.
Step 7: Repeat:
Repeat Steps 2 to 6 for multiple time steps or episodes, allowing the agent to learn
and update the policy based on its interactions with the environment.
Step 8: Convergence and Evaluation:
Monitor the performance of the Policy Gradient algorithm and evaluate the learned
policy on test scenarios to assess the agent's performance in the environment.
8 Assess the effectiveness of Dynamic Programming methods for solving large-scale
RL problems compared to other approaches, such as Monte Carlo methods.
The effectiveness of Dynamic Programming (DP) methods for solving large-scale
Reinforcement Learning (RL) problems and compare them to other approaches,
such as Monte Carlo (MC) methods.
Dynamic Programming (DP):
Definition: DP algorithms can solve an MDP (Markov Decision Process)
reinforcement learning task given the model of the environment (including state-
transition probabilities and the reward function)

Key Characteristics:
 DP assumes that we have a perfect model of the environment, meaning we
know the probability distributions of any changes in the problem setup.
 It works well when the environment specifics are known, and the agent can
only take discrete actions.
 DP essentially solves a planning problem, focusing on finding optimal
solutions.
Applications:
 DP is useful for tasks where the environment is well-defined, such as
inventory management, resource allocation, and scheduling problems.
 It provides a foundation for understanding more complex RL algorithms.
 Monte Carlo (MC) Methods:
Definition: MC methods estimate value functions by sampling from the
environment without relying on a model.
Key Characteristics:
 MC methods do not require a model of the environment; they learn directly
from interaction.
 They work well for episodic tasks (where episodes have a finite length).
 MC methods are model-free, making them more flexible.
Applications:
 MC methods are commonly used for policy evaluation and prediction tasks.
 They are suitable for scenarios where the environment dynamics are
unknown.
Comparison:
Scalability:
DP: DP can be computationally expensive for large-scale problems due to the need
to evaluate policies at every step.
MC: MC methods are more scalable because they do not rely on a model and can
handle larger state spaces.
Convergence:
DP: Converges faster since it combines policy evaluation and improvement in each
step.
MC: Slower convergence due to the need for more samples.
Assumptions:
DP: Assumes a known model.
MC: Works without a model.
Trade-offs:
DP: Requires a model but provides more accurate value estimates.
MC: Model-free but may have higher variance in value estimates.
In summary, DP is effective for planning problems with a known environment,
while MC methods are more versatile for model-free RL tasks. The choice between
them depends on the problem specifics and computational constraints.
9 Devise a novel function approximation method for handling continuous state
spaces in RL.
A novel function approximation method for handling continuous state spaces in
Reinforcement Learning (RL). One promising approach is the use of Radial Basis
Function Networks (RBFNs).
Radial Basis Function Networks (RBFNs):
Overview:
 RBFNs are a type of non-linear function approximators.
  They are particularly useful for approximating value functions in continuous
state spaces.
 RBFNs combine the benefits of both local and global approximation
methods.
Key Components:
Radial Basis Functions:
 These are kernel functions that measure the similarity between a given state
and a set of reference points (centers).
 Common choices include Gaussian functions.
Centers:
 The centers represent specific points in the state space.
 They can be randomly initialized or learned during training.
Weights:
 Each center has an associated weight.
 The weights determine the contribution of each center to the overall
approximation.
Training Process:
Initialization:
 Randomly initialize the centers and weights.
Sample Data:
 Collect samples from the environment by interacting with it.
Feature Extraction:
 Compute the activation of each center for each state using the radial basis
functions.
Weight Update:
 Use gradient-based optimization (e.g., stochastic gradient descent) to update
the weights.
 Minimize the difference between the predicted value and the true value (e.g.,
using TD error).
Generalization:
 RBFNs generalize well to unseen states due to their local nature.
 They adapt to the underlying structure of the state space.
Advantages:
 Local Approximation:
 RBFNs capture local patterns effectively.
 They adapt well to variations in the state space.
Scalability:
 RBFNs can handle high-dimensional continuous state spaces.
 The number of centers can be adjusted to control the trade-off between
 accuracy and computational cost.
Interpretability:
 The centers provide insights into critical regions of the state space.
Challenges:
Center Selection:
Choosing appropriate centers is crucial.
Clustering techniques or heuristics can be used.
Hyperparameter Tuning:
Parameters such as the width of the radial basis functions need careful tuning.
Overfitting:
Regularization techniques are essential to prevent overfitting.
Applications:
RBFNs have been successfully applied to RL tasks such as robotic control,
financial modeling, and game playing.
In summary, RBFNs offer a promising way to approximate value functions in
continuous state spaces. Their ability to capture local patterns while maintaining
scalability makes them a valuable addition to the RL toolbox.
10 Compare the exploration-exploitation dilemma in Temporal Difference learning
with the concept of "horizon" in Dynamic Programming. How do these two aspects
impact the learning process and decision-making in RL?
The exploration-exploitation dilemma in Temporal Difference (TD) learning and
compare it with the concept of “horizon” in Dynamic Programming (DP). These
aspects play crucial roles in the learning process and decision-making in
Reinforcement Learning (RL).
Exploration-Exploitation Dilemma in TD Learning:
Definition:
 The exploration-exploitation dilemma refers to the trade-off between
exploring new actions (to gain information) and exploiting known actions (to
maximize immediate rewards).
 In TD learning, this dilemma arises when updating value estimates based on
observed rewards and transitions.
Impact on Learning Process and Decision-Making:
Exploration:
Pros:
 Exploration allows the agent to discover new states and actions.
 It helps prevent premature convergence to suboptimal policies.
Cons:
 Excessive exploration can lead to slow learning and inefficient use of
resources.
  It may delay the agent from exploiting the best-known actions.
Exploitation:
Pros:
 Exploitation focuses on maximizing immediate rewards.
 It leads to efficient decision-making based on current knowledge.
Cons:
 Over-reliance on exploitation may cause the agent to miss out on better
options.
 It can lead to suboptimal policies if the agent prematurely settles on a
suboptimal action.
Balancing Act:
 TD algorithms (e.g., Q-learning, SARSA) use an exploration policy (e.g., ε-
greedy, softmax) to balance exploration and exploitation.
 The choice of exploration policy impacts the learning rate and convergence
speed.
Horizon in Dynamic Programming:
Definition:
The horizon in DP refers to the time horizon or the number of steps into the future
that an agent considers when making decisions.
It determines how far ahead the agent plans.
Impact on Learning Process and Decision-Making:
Short Horizon:
Pros:
 Faster computation since only a limited number of steps are considered.
 Suitable for problems with immediate rewards.
Cons:
 May lead to myopic decisions that ignore long-term consequences.
 Suboptimal policies if the true optimal policy requires longer planning.
Long Horizon:
Pros:
 Considers long-term effects and global optimization.
 Better handling of delayed rewards and complex interactions.
Cons:
 Computationally expensive due to deeper planning.
 Prone to overfitting if the environment is stochastic.
Trade-offs:
 DP methods (e.g., value iteration, policy iteration) allow us to explicitly set
the horizon.
 Balancing computational cost and planning depth is essential.
In practice, we often use finite horizons or discount factors (γ) to balance short-
term and long-term rewards.
Overall Impact:
TD Learning:
 Balances exploration and exploitation during learning.
 Adapts online to changing environments.
DP:
 Considers the entire planning horizon.
 Provides optimal solutions but assumes a known model.
 Used for offline planning and policy evaluation.
In summary, TD learning handles exploration-exploitation trade-offs, while DP
considers the planning horizon. Both aspects impact the agent’s learning efficiency,
decision-making, and ability to find optimal policies in RL tasks