School of Mathematics and Statistics

UNSW Sydney

Introduction to Probability and Stochastic Processes

OPEN LEARNING
Chapter 8 Markov Chains

2 / 101
Outline:

8.1 Introduction to Stochastic Processes

① Four Main Types of Stochastic Processes
② Applications of Stochastic Processes
8.2 Markov Chains
① Chapman-Kolmogorov Equations
② Classification of States
③ Long-Run Properties
④ Stationary Probabilities
8.3 Supplementary Material

3 / 101
The source of this chapter is Ross, S.M. (2023) Introduction to Probability
Models, 13th Edition, Academic Press (Elsevier), California, U.S.A.

4 / 101
8.1 Introduction to Stochastic
Processes

5 / 101
Definition
A stochastic (or random) process {X(t), t ≥ 0} is a family (or
collection of random variables.

That is, for all t contained in the index set T , X(t) is a random variable.

The parameter t is usually interpreted as time and

X(t) represents the state of the process at time t.

Notation. X(t) will also be denoted as Xt . We will interchange using the

notation X(t) ≡ Xt .

6 / 101
You can think of X(t) as
the position of a particle at time t
the number of people going to the library at time t
the amount of fish in the ocean at time t, etc. .

The set T is called the index set of the stochastic process.

If T is a countable set, for example, T = {0, 1, 2, 3, . . . , }, then the

stochastic process is said to be a discrete-time process.

If T is an open or closed interval of the real line, for example,

T ∈ [0, ∞), we say that the process is a continuous-time process.

The set of possible values which the random variable X(t), t ∈ T , may
take is called the state space of the process.

7 / 101
A continuous-time stochastic process {X(t), t ≥ 0} has
independent increments if for all choices

t0 < t1 < t2 < · · · < tn

of your interval, the n random variables

X(t1 ) − X(t0 ), X(t2 ) − X(t1 ), . . . , X(tn ) − X(tn−1 )

are independent;

stationary increments if X(t2 + s) − X(t1 + s) has the same

distribution as X(t2 ) − X(t1 ) for all t1 , t2 ∈ T , s > 0, i.e.,
D
X(t2 + s) − X(t1 + s) −
→ X(t2 ) − X(t1 ), ∀t1 , t2 ∈ T, s > 0.

8 / 101
Example

Let Y1 , Y2 , . . . be a sequence of independent and identically distributed

random variables and let Xn = ni=1 Yi . The stochastic process
P

{Xn , n = 0, 1, 2, 3, . . . }

is called a general random walk process.

For example, if Yi , i = 1, 2, 3, . . . , n, represents the number of shoes sold by

David Jones during the i week, then Xn would be the total number of shoes
sold during the first n weeks.

9 / 101
Example
A stochastic process {X(t), t ≥ 0} is said to be a Brownian motion
(BM) if

X(0) = 0

{X(t), t ≥ 0} has stationary independent increments

for every t > 0, X(t) is normally distributed with mean zero and
variance σ 2 t.

We will discuss this stochastic process later in the course.

10 / 101
The general random walk and Brownian motion are examples of a class of
stochastic processes known as Markov processes.

Definition
The process {X(t), t ≥ 0} is a Markov Process if



P X(t) ≤ x X(t1 ) = x1 , X(t2 ) = x2 , . . . , X(tn ) = xn = P X(t) ≤ x X(tn ) = xn

whenever t1 < t2 < · · · < tn < t.

In other words, a Markov process is a stochastic process with the property

that the conditional distribution of the future values of the process, for
example, X(t + s) given the present value X(t) and the past X(u), u < t is
independent of the past.

11 / 101
8.1.1 Four Main Types of Stochastic Processes

12 / 101
discrete-time discrete-state stochastic processes, for example,
discrete-time Markov chains, Branching processes

continuous-time discrete-state stochastic processes, for example, Poisson

processes, continuous-time Markov chains, birth and death processes

discrete-time continuous-state stochastic processes, for example,

Autoregressive processes

continuous-time continuous-state stochastic processes, for example,

Brownian Motion, Gaussian processes

13 / 101
8.1.2 Applications of Stochastic Processes

14 / 101
Finance: stock price movements ↭ Brownian motion, Itô’s formula;
volatility ↭ Gaussian processes; options ↭ Levy processes

Biology/Ecology: migration of species ↭ birth and death processes;

survival of endangered species ↭ birth and death processes, branching
processes

Genetics: chromosome interchanges subjected to X-ray radiation ↭

Poisson process; location of harmful DNA mutations ↭ Poisson
process, Markov process; chromatin states predication ↭ Markov
process, semi-Markov process

15 / 101
Medicine: the spread of diseases ↭ birth and death process; the
number of bacteria and/or red blood cells in a drop of blood ↭ Poisson
process

Physics/Chemistry: emissions of a particle due to radioactive decay

by an unstable substance ↭ Poisson process; movement of particles ↭
diffusion process, heat equation; photons landing on a photo-diode in
low light environments ↭ Poisson process

Electrical Engineering: internet pockets at a router ↭ Poisson

process, speech processing ↭ Markov process, hidden Markov process;
image process ↭ Markov random field

16 / 101
8.2 Markov Chains

17 / 101
Definition
A stochastic process {Xn , n = 0, 1, . . . } with a finite or countable state is a
Markov chain if for all states i0 , i1 , . . . , in−1 , i, j and all n ≥ 0

P(Xn+1 = j|X0 = i0 , X1 = i1 , . . . , Xn−1 = in−1 , Xn = i) = P(Xn+1 = j|Xn = i).

(1)

P(Xn+1 = j|Xn = i) are known as one step transition probabilities.

If P(Xn+1 = j|Xn = i) is independent of n, then the Markov chain is said

to have stationary transition probabilities.

18 / 101
This course will focus on Markov chains, which have stationary transition
probabilities.

Let Pij = P(Xn+1 = j|Xn = i), one step transition probability that one has
moved to state j at time n + 1 given that one was in state i at time n.

Notation: We will label the state space of the process by the positive
integers {0, 1, 2, . . . , } unless otherwise stated.

In addition, we will say that the process is in state j at time n if Xn = j.

19 / 101
Write P to be the matrix of one-step transition probabilities Pij :

P00 P01 P02 · · ·

 
 P10 P11 P12 · · · 
 . .. .. 
P =  .. . . ··· .
 
 Pn0 Pn1 Pn2 · · · 
 
.. .. ..
. . . ···

20 / 101
Since we consider the Pij ’s as probabilities, we require that

Pij ≥ 0, i, j = 0, 1, 2, . . .
and

∞
X
Pij = 1 for i = 0, 1, 2, . . . .
j=0

It turns out the process X is completely specified once P and the initial
distributions of X are known.

21 / 101
To see this, consider the following:

P(X0 = i0 , X1 = i1 , . . . , Xn = in )
= P(Xn = in |X0 = i0 , . . . , Xn−1 = in−1 )P(X0 = i0 , . . . , Xn−1 = in−1 )
= Pin−1 ,in P(X0 = i0 , . . . , Xn−1 = in−1 ) by the Markov property
= Pin−1 ,in Pin−2 ,in−1 P(X0 = i0 , . . . , Xn−1 = in−2 )
..
.
= Pin−1 ,in Pin−2 ,in−1 · · · Pi0 ,i1 P(X0 = i0 ),

where P(X0 = i0 ) is the initial distribution of the Markov chain, X.

22 / 101
Example
Random Walk Model

A Markov chain with state space given by integers i = 0, ±1, ±2, . . . is a

random walk if, for some p ∈ (0, 1), the one-step transition probabilities are

Pi,i+1 = p = 1 − Pi,i−1 i = 0, ±1, ±2, . . . .

p p p p p p

··· −2 −1 0 1 2 ···

1−p 1−p 1−p 1−p 1−p 1−p

(
1
with probability p
You can think that Yi = , where the Yi , i = 0, ±1, ±2, . . . are independent and identically
−1
with probability 1 − p
Pn
distributed. Hence the random walk is Xn = i=1 Yi .

23 / 101
Example
Gambling Model

Suppose a gambler who at each play of a game either wins $1 with

probability p and loses $1 with probability 1 − p = q.

Further, suppose that this gambler stops playing when she has no money left,
or she has $N. Then the gambler’s fortune is a Markov chain with transition
probabilities given by

Pi,i+1 = p = 1 − Pi,i−1 , i = 1, 2, . . . , N − 1, and

P00 = PN N = 1.

24 / 101
Example

The states 0 and N are known as absorbing states because once you reach
that state, you can’t leave (i.e. you are fixed there for life.).
Hence, the gambler’s fortune is a finite state random variable with absorbing
states 0 and N . We will discuss absorbing states in detail later on.

p p p p

1 0 1 2 ··· N −1 N 1
1−p 1−p 1−p 1−p

25 / 101
8.2.1 Chapman-Kolmogorov Equations

26 / 101
Recall that we established

P(X0 = i0 , X1 = i1 , . . . , Xn = in ) = Pin−1 ,in Pin−2 ,in−1 · · · Pi0 ,i1 P(X0 = i0 ).

Write Pijn as the probability of the process going from state i to state j in n
steps. That is,

Pijn = P(Xn+m = j|Xm = i).

This is known as the n step transition probabilities. The next proposition is

known as the Chapman-Kolmogorov equation, and it is a method of
calculating n transitional probabilities.

27 / 101
Chapman-Kolmogorov Equations
For any r ≤ n,
∞
X
n−r
Pijn = Pikr Pkj . (2)
k=0

28 / 101
If we let P(n) denote the matrix of n- step transition probability Pijn , then

P(n) = P(r) · P(n−r) , for any r ≤ n.

In particular, P(2) = P(1) · P(1) = P · P = P2 .

By induction, P(n) = P(n−1) · P(1) = P(1) · P(n−1) = Pn .

That is, the n-step transition matrix is the nth power of the one-step
transition matrix.

Note: P(0) = P0 = I, where I is the identity matrix.

29 / 101
Example
Ticks of Stocks

Traders of a stock pay close attention to the ticks of a stock. A stock can
trade on an uptick, even tick or downtick, if the price is higher, the same or
lower than the previous price, respectively.

For a particular stock, traders observed that the ticks could be modelled
using a Markov chain.

For example, they observed that following an even tick, the next trade price
was an even tick with a probability of 0.6, an uptick with a probability of 0.2
and a downtick with a probability of 0.2.

After a downtick, another downtick occurs with a probability of 0.4, while an

even tick has a probability of 0.6.

30 / 101
Example

Ticks of Stocks - continued

After an uptick, another uptick occurs with a probability of 0.4, while an

even tick occurs with a probability of 0.6.

Suppose that the states are 0,1,2 if the previous trade price is a downtick, an
even tick and an uptick, respectively.

Find the state transition matrix P , the 2-step transition matrix, and the n
step transition matrix, when n = 20, for example.

31 / 101
Example

Solution:
First we will state the states of the Markov chain:
state 0 will be downtick
state 1 will be an even tick
state 2 will be an uptick.

The graphical representation of this Markov chain is

0.6 0.2
0.4 0 1 2 0.4
0.2 0.6
0.6
32 / 101
Example

Solution - continued:
The one-step transition matrix is
 
0.4 0.6 0
P =  0.2 0.6 0.2  .
0 0.6 0.4
The second-step transition matrix is
P(2)  ·P
= P   
0.4 0.6 0 0.4 0.6 0
=  0.2 0.6 0.2  ·  0.2 0.6 0.2 
0 0.6 0.4 0 0.6 0.4
 
0.28 0.6 0.12
=  0.2 0.6 0.2  .
0.12 0.6 0.28
33 / 101
Example

Solution - continued:
The twenty-step transition matrix is
 
0.2 0.6 0.2
P(20) = P20 =  0.2 0.6 0.2  .
0.2 0.6 0.2

34 / 101
Notation

Let αj be the probability that the process is initially in state j.

That is, P(X0 = j) = αj , j ≥ 0 and ∞

P
j=0 αj = 1.

We can also determine the probability of the process being in state j at time
time n by
∞
X
n
P(Xn = j) = αk Pkj .
k=0

35 / 101
Example
Suppose a Markov chain X0 , X1 , . . . has the transition probability matrix
 
0.3 0.2 0.5
P =  0.5 0.1 0.4 
0.5 0.2 0.3
with initial distribution α0 = 0.5 and α1 = 0.5. Determine P(X2 = 0) and
P(X3 = 0).

36 / 101
Example

Solution:
Firstly, the states of this Markov chain are 0, 1, 2.

We are given the initial distribution of the Markov chain to be

α0 = P(X0 = 0) = 0.5 , α1 = P(X0 = 1) = 0.5, and α2 = P(X0 = 2) = 0.
The second-step transition matrix is
P(2)  ·P
= P   
0.3 0.2 0.5 0.3 0.2 0.5
=  0.5 0.1 0.4  ·  0.5 0.1 0.4 
0.5 0.2 0.3 0.5 0.2 0.3
 
0.44 0.18 0.38
=  0.40 0.19 0.41  .
0.40 0.18 0.42

37 / 101
Example

Solution - continued:
The third-step transition matrix is
P(3) = P (2)
 ·P   
0.44 0.18 0.38 0.3 0.2 0.5
=  0.40 0.19 0.41  ·  0.5 0.1 0.4 
0.40 0.18 0.42 0.5 0.2 0.3
 
0.412 0.182 0.406
=  0.42 0.181 0.399  .
0.42 0.182 0.398

38 / 101
Example

Solution - continued:

P(X2 = 0) = P(X2 = 0 X0 = 0) P(X0 = 0) + P(X2 = 0 X0 = 1) P(X0 = 1)

+ P(X2 = 0 X0 = 2) P(X0 = 2)
2 2 2
= P00 · α0 + P10 · α1 + P20 · α2
= 0.44 × 0.5 + 0.40 × 0.5 + 0.42 × 0
= 0.42

P(X3 = 0) = P(X3 = 0 X0 = 0) P(X0 = 0) + P(X3 = 0 X0 = 1) P(X0 = 1)

+ P(X3 = 0 X0 = 2) P(X0 = 2)
3 3 3
= P00 · α0 + P10 · α1 + P20 · α2
= 0.412 × 0.5 + 0.42 × 0.5 + 0.42 × 0
= 0.416.
39 / 101
8.2.2 Classification of States

40 / 101
State j is said to be accessible from state i if for some n ≥ 0, Pijn > 0.

We write i → j if state j is accessible from state i.

This implies that state j is accessible from state i if and only if, starting in
state i, it is possible that the process will eventually enter state j.

Two states i and j accessible to each other, are said to communicate, and
we write i ↔ j.

41 / 101
Proposition
Communication is an equivalence relation. That is, for any states i, j and k,
we have
(i) i ↔ i
(ii) if i ↔ j, then j ↔ i
(iii) if i ↔ j and j ↔ k, then i ↔ k.

If two states communicate with each other, then they are in the same class.

Then, by the above proposition, any two classes are either disjoint or
identical.

A Markov chain is irreducible if there is only one class, meaning all

the states communicate.

42 / 101
Example
Consider the following Markov chain consisting of the three states 0, 1, and 2
and having a transition matrix
 
1/2 1/2 0
P =  1/2 1/4 1/4  .
0 1/3 2/3

Is this Markov chain irreducible? Determine the classes of this Markov chain.

43 / 101
Example

Solution:
This Markov chain has 3 states: 0,1,2. A graphical representation of this Markov chain is
given by
1/2 1/4

1/2 0 1 2 2/3

1/2 1/3
1/4

We need to check if the states are communicating with each other. That is, we see
0 → 1 and 1 → 0 so states 0 and 1 communicate with each other (0 ↔ 1).
1 → 2 and 2 → 1 so states 1 and 2 communicate with each other (1 ↔ 2).
0 → 2 via 0 → 1 and 1 → 2; and 2 → 0 via 2 → 1 and 1 → 0.

Therefore, the Markov chain is irreducible since all the states communicate with each
other, and only one class contains all the states. 44 / 101
Example
Consider the following Markov chain consisting of four states 0, 1, 2, and 3
and having a transition matrix
 
1/2 1/2 0 0
 1/2 1/2 0 0 
P = 1/4 1/4 1/4 1/4  .


0 0 0 1

Is this Markov chain irreducible? Determine the classes of this Markov chain.

45 / 101
Example

Solution:
This Markov chain has 3 states: 0,1,2,3. A graphical representation of this Markov chain is
given by
1/2 1/4
1/2 1/4

1/2 0 1 2 3 1
1/2 1/4

1/4

We need to check if the states are communicating with each other. That is, we see
0 → 1 and 1 → 0 so states 0 and 1 communicate with each other (0 ↔ 1).

2 → 1, but 1 ↛ 2, so 1 and 2 do not communicate with each other

2 → 3, but 3 ↛ 2 so 2 and 3 do not communicate with each other .

Hence the classes are {0, 1}, {2}, and {3}. This Markov chain is not irreducible. Moreover,
{3} is an absorbing state.
46 / 101
Definition
State i has period d if Piin = 0 whenever n is not divisible by d and d is the
largest integer with this property.

State i has period d if d is the greatest common divisor (g.c.d) of the number
of transitions by which state i can be reached, starting in state i. That is,

d = g.c.d( {n : Piin > 0} ).

If Piin = 0 for all n, then the period of state i is zero.

A state with period one is said to be aperiodic.

47 / 101
Let d(i) be the period of state i. The next proposition tells us that
periodicity is a class property.

Proposition
If i ↔ j, then d(i) = d(j).

48 / 101
Example
Consider the Markov chain with a transition probability matrix
 
0 1 0 0
 0 0 1 0 
P =  0
.
0 0 1 
1/2 0 1/2 0
What is the period of state 0? Is this chain irreducible?

49 / 101
Example

Solution:
This Markov chain has 3 states: 0,1,2,3. A graphical representation of this Markov chain is
given by
1 1 1

0 1 2 3
1/2

1/2

n
The period of 0 is determined by finding d(0) = g.c.d({n : P00 > 0}). We see that

1 2 3 4
P00 =0 P00 =0 P00 =0 P00 >0
5 6
P00 =0 P00 = (1/2)2 7
P00 =0 8
P00 >0

n
For n = 4, 6, 8, . . . , we have P00 > 0, so d(0) = g.c.d({4, 6, 8, . . . }) = 2.

That is, the period of state 0 is 2. 50 / 101

Example

Solution - continued:
We need to check if the states are communicating with each other. That is, we see
0 → 1 and 1 → 0 so states 0 and 1 communicate with each other (0 ↔ 1).
1 → 2 and 2 → 1, so 1 and 2 communicate with each other (1 ↔ 2).
2 → 3 and 3 → 2, so 2 and 3 communicate with each other (2 ↔ 3).

All the states communicate with each other, so there is only one class, {0, 1, 2, 3}.

Therefore, the Markov chain is irreducible.

51 / 101
For any state i, let fi be the probability that starting in i, the process will
ever re-enter state i.

Definition
State i is said to be recurrent if fi = 1 and transient if fi < 1.

Suppose the process starts in state i and i is recurrent. Then

with probability one, the process will eventually reenter state i.
by the Markov property, the process will start over again.
state i will eventually be visited again.

Continual repetition of this argument leads to the conclusion that if state i is

recurrent, starting in state i, the process will reenter state i again and again,
infinitely often.

52 / 101
Suppose the process starts in state i and i is transient. Then
with probability (1 − fi ) > 0, the process will never reenter state i.
starting in state i, the probability that the process will be in state i for
exactly n times equals fin−1 (1 − fi ), n ≥ 1.

If state i is transient, starting in state i, the process will only visit a finite
number of times (with probability one); hence the name transient.

This leads to the conclusion that in a finite-state Markov chain, not all states
can be transient.

53 / 101
Let Ni be the number of times state i is visited.

If state i is transient, then, starting in state i, Ni has a geometric

1
distribution with finite mean 1−f i
< ∞.

1
That is Ni X0 = i ∼ Geometric(1 − fi ) and E[Ni X0 = i] = 1−fi
< ∞ if
state i is transient.

It follows that state i is recurrent if and only if, starting in state i,

the expected number of time periods that the process is in state i
is infinite (i.e., E[Ni X0 = i] = ∞).

54 / 101
We write ∞
X
Ni = 1{Xn =i} ,
n=0

where (
1, if Xn = i
1{Xn =i} =
0, if Xn ̸= i.

Note that Ni and Xn are random variables.

55 / 101
Hence
∞
hX i
E(Ni X0 = i) = E 1{Xn =i} X0 = i
n=0
∞
X h i
= E 1{Xn =i} X0 = i
n=0
X∞ h i
= P X n = i X0 = i
n=0
∞
X
Piin .
n=0

56 / 101
Proposition
State i is recurrent if ∞
X
Piin = ∞.
n=1

State i is transient if ∞
X
Piin < ∞.
n=1

Corollary
If state i is transient, then Piin → 0 as n → ∞.

57 / 101
Next, we learn that recurrence/transience state is a class property.

Corollary
If State i is recurrent and state i communicates with state j , then state j
is recurrent.

It follows that if state i is transient and communicates with state j, then

state j must also be transient.
If state j was recurrent, then, by the above Corollary, state i would also be
recurrent and hence could not be transient.

Corollary
In a Markov chain, all states in a given class are recurrent or transient. We
refer to the class itself as being recurrent or transient.

58 / 101
Example
Consider the Markov Chain having states 0, 1, 2, 3 and has a transition
probability matrix

0 0 12 12
 
1 0 0 0 
P = 0 1 0 0  .


0 1 0 0

Determine which states are transient and which are recurrent

59 / 101
Example

Solution:
Here we have four states: 0,1,2,3. The graphical representation of this
Markov chain is
1/2

1/2

0 1 2 3
1 1

60 / 101
Example

Solution- continued:
From this graphic, we see that
0 → 1 via 0 → 2, 2 → 1 1→0 =⇒ 0 ↔ 1

1 → 2 via 1 → 0, 0 → 2 2→1 =⇒ 1 ↔ 2

2 → 3 via 2 → 1, 1 → 0 , 0 → 3 3 → 2 via 3 → 1, 1 → 0, 0 → 2 =⇒ 2 ↔ 3.

We observe that all the states communicate with each other; hence, there is
only one class {0, 1, 2, 3}. The Markov chain is irreducible . All the states
are recurrent

61 / 101
Example
Consider the Markov Chain having states 0, 1, 2, 3, 4 and has a transition
probability matrix
 
1/2 1/2 0 0 0
1/2 1/2 0 0 0 
 
P = 0 0 1/2 1/2 0 .

 0 0 1/2 1/2 0 
1/4 1/4 0 0 1/2

Determine which states are transient and which are recurrent.

62 / 101
Example

Solution:
This Markov chain has 5 states: 0,1,2,3,4. A graphical representation of this Markov chain is given by

1/2
1/2

1/2 0 1 4 1/2

1/2 1/4

1/4

1/2

1/2 2 3 1/2

1/2

63 / 101
Example

Solution- continued:
From this graphic, we see that
0→1 1→0 =⇒ 0 ↔ 1

1↛2 2↛1 =⇒ 1 ↮ 2

4→1 1↛4 =⇒ 1 ↮ 4

2→3 3→2 =⇒ 2 ↔ 3

We observe that there are 3 classes

{0, 1} recurrent
{2, 3} recurrent
{4} transient.
64 / 101
Example
Gambling Model

Suppose a gambler who at each play of a game either wins $1 with

probability p and loses $1 with probability 1 − p = q.

Further, suppose that this gambler stops playing when she has no money left,
or she has $N. Then the gambler’s fortune is a Markov chain with transition
probabilities given by

Pi,i+1 = p = 1 − Pi,i−1 i = 1, 2, . . . , N − 1

P00 = PN N = 1.

65 / 101
Example

The states 0 and N are known as absorbing states because once you reach
that state, you can’t leave (i.e. you are fixed there for life.).

Hence, the gambler’s fortune is a finite state random variable with absorbing
states 0 and N . We will discuss absorbing states in detail later on.

p p p p

1 0 1 2 ··· N −1 N 1
1−p 1−p 1−p 1−p

There are three classes: {0} - recurrent; {1, 2, . . . , N − 1} - transient; and {N } - recurrent.

66 / 101
Example
Random Walk Model

A Markov chain with state space given by integers i = 0, ±1, ±2, . . . is a

random walk if, for some p ∈ (0, 1), the one-step transition probabilities are

Pi,i+1 = p = 1 − Pi,i−1 i = 0, ±1, ±2, . . . .

The graphic representation of this Markov chain is

p p p p p p
··· −2 −1 0 1 2 ···
1−p 1−p 1−p 1−p 1−p 1−p

67 / 101
Example
From this graphic representation, we observe that
The chain is irreducible, and all states are either recurrent or transient.
P∞ n
Let us consider state 0 and attempt to determine if n=0 P00 is finite or
infinite.

2n−1
We also observe for n = 1, 2, . . . , P00 = 0 and

(4 p (1 − p))n
 
2n 2n n 2n! n
P00 = p (1 − p)n = p (1 − p)n ≈ √
n n! n! πn
√
(by Stirling’s approximation n! = nn+1/2 e−n 2 π. )

68 / 101
Example
In particular,
When p = 1/2, then (4 p (1 − p) )n = 1. Hence,
∞ ∞
n ∞
X
n
X 4 p (1 − p) X 1
P00 ≈ √ = √ = ∞.
n=0 n=0
πn n=0
πn

When p ̸= 1/2, then (4 p (1 − p) )n < 1. Hence,

∞ ∞
n ∞
X
n
X 4 p (1 − p) X 1
P00 ≈ √ < √ = ∞.
n=0 n=0
πn n=0
πn

The Markov chain is recurrent when p = 1/2 (symmetric random walk)

and transient otherwise.

69 / 101
8.2.3 Long-Run Properties

70 / 101
For pairs of states i ̸= j, let fij denote the probability that the Markov
chain, starting in state i, will ever transition into state j. That is,

fij = P (Xn = j for some n > 0 X0 = i).

Proposition
If i is recurrent and i communicates with j, then fij =1.

71 / 101
If state j is recurrent, let mj denote the expected number of
transitions that it takes the Markov chain when starting in state j
to return to back to state j. That is, we see that mj , with

Nj = min{ n > 0 : Xn = j}

equal to the number of transitions until the Markov chain makes a transition
into state j,
mj = E(Nj X0 = j).

72 / 101
Definition
We say that the recurrent state j is
positive recurrent if mj < ∞

null recurrent if mj = ∞.

Positive recurrent and null recurrent are class properties

In a finite-state Markov chain, all recurrent stats are positive recurrent.

73 / 101
Suppose that the Markov chain is irreducible and recurrent.

Let πj denote the long-run proportion of time that the Markov chain
is in state j.

Proposition
If the finite-state Markov chain is irreducible and recurrent, then for
any initial state
πj = 1/mj .

74 / 101
Next we need to determine the long-run proportions {πj , j ≥ 1}.

Note that because πi is the long-run proportion of transitions that come from
state i, we observe that

πi Pij = long-run proportion of transitions that go from state i to state j.

Summing up the preceding for all i yields

X
πj = πi Pij .
i

75 / 101
Theorem
Consider an irreducible Markov chain. If the chain is positive recurrent, then
the long-run proportions are the unique solution of the equation
X
πj = πi Pij , j ≥ 0
i
X
πj = 1.
j

Moreover, if there is no solution to the preceding linear equations, then the

Markov chain is either transient or null recurrent and all πj = 0.

76 / 101
Example
Forecasting the Weather
This example comes from Ross (2023).

Calculate the long-run proportion of rain when

P (rain today|rain yesterday) = α = 0.7
P (rain today|no rain yesterday) = β = 0.4.

Let state R represent rain and N R represent no rain.

77 / 101
Example
The transition probability matrix is
 
α 1−α
P = .
β 1−β

The graphical representation of this Markov chain is

1−α
α R NR 1−β
β

78 / 101
Example
We have an irreducible Markov chain, and the chain is positive recurrent.

Let π = (πR , πN R ), where π = π P and πR + πN R = 1.

That is,
 
α 1−α
(πR , πN R ) = (πR , πN R ) = (α πR + β πN R , (1 − α) πR + (1 − β) πN R .
β 1−β

This gives the following linear equations

πR = α π R + β π N R
πN R = (1 − α) πR + (1 − β) πN R
πR + πN R = 1.

79 / 101
Example
From this, we see that
β
πR =
1−α+β
1−α
πN R = 1 − πR = .
1−α+β

Using the values α = 0.7 and β = 0.4, we see that the long-run proportion of
0.4
rain, πR = 1−0.7+0.4 = 47 .

80 / 101
Example
Consider a Markov chain with 3 states: 0, 1, 2 and a transition probability
matrix  
0.5 0.4 0.1
P = 0.3 0.4 0.3 .
0.2 0.3 0.5

We see that we have a finite-state irreducible Markov chain.

In the long-run, what proportion of time is the process in each of

the three states?

81 / 101
Example
P2
We need to solve for π = (π0 , π1 , π2 ) when π = π P and i=0 πi = 1.

This gives the following linear equations

π0 = 0.5π0 + 0.3π1 + 0.2π2

π1 = 0.4π0 + 0.4π1 + 0.3π2
π2 = 0.1π0 + 0.3π1 + 0.5π2
π0 + π1 + π2 = 1.

From this, we see

21 23 18
π0 = , π1 = , π2 = .
62 62 62

82 / 101
8.2.4 Stationary Probabilities

83 / 101
The long-run proportion {πj , j ≥ 0}, are often called stationary
probabilities.

The reason is that if the initial state is chosen according to the probabilities
πj , j ≤ 0, then the probability of being in state j at any time n is also equal
to πj . That is,
P (X0 = j) = πj ,
then
P (Xn = j) = πj for all n.

84 / 101
The preceding is easily proven by induction. Assume that it is true when
n = 0, and if we suppose it is true for n − 1, then by writing
X
P (Xn = j) = P (Xn = j Xn−1 = i)P (Xn−1 = i)
i
X
= Pij πi
i
= πj .

85 / 101
Definition
A state is said to be ergodic if it is positive recurrent and aperiodic.

A class of ergodic states is an ergodic class.

An irreducible Markov chain consisting of one ergodic class is an

ergodic chain.

86 / 101
Example
Recall the forecasting of the weather problem from Ross (2023) .

We are given the transition probability matrix

   
α 1−α 0.7 0.3
P = = .
β 1−β 0.4 0.6

Calculate P4 , P8 , P16 .

87 / 101
Example
We have
     
2 0.7 0.3 0.7 0.3 0.61 0.39
P = P ·P = · =
0.4 0.6 0.4 0.6 0.52 0.48
     
0.61 0.39 0.61 0.39 0.5749 0.4251
P4 = P2 · P2 = · =
0.52 0.48 0.52 0.48 0.5668 0.4332
     
0.5749 0.4251 0.5749 0.4251 0.5715 0.4285
P8 4
= P ·P =4
· =
0.5668 0.4332 0.5668 0.4332 0.5714 0.4286
     
0.5715 0.4285 0.5715 0.4285 0.5714 0.4286
P 16 8
= P ·P =8
· = .
0.5714 0.4286 0.5714 0.4286 0.5714 0.4286

Recall from our earlier calculation , we found that πR = 47 ≈ 0.5714 and πN R = 3

7
≈ 0.4286.
Intuitively, in the long-run, the starting time does not matter.
Long-ago weather does not influence today’s weather.

88 / 101
It seems that Pijn is converging to some value as n → ∞, with this value not
depending on state i.

It appears that the long-run proportions may also be limiting probabilities.

It is not always true that the long-run proportions are also the limit
probabilities.

89 / 101
Theorem
For an ergodic Markov chain, limn→∞ Pijn exists for all state j, and is
independent of state i.

Further, letting
πj = lim Pijn ,
n→∞

then π = (π0 , π1 , π2 , . . . , ) is the unique nonnegative solution of the

system of equations

X
πj = πi Pijn , j ≥ 0,
i
X
πj = 1.
j

90 / 101
Example
Application of the Gambler’s ruin problem

We will now look at the gambler’s ruin problem again.

Suppose we have a gambler who at each play of the game has probability p of
winning one dollar and probability q = 1 − p of losing one dollar.

Let successive plays of this game be independent of each other.

What is the probability that starting with i units, the gambler’s

fortune will reach N before he goes broke?

91 / 101
Example
The graphical representation of the gambler’s ruin problem is

p p p p

1 0 1 2 ··· N −1 N 1
1−p 1−p 1−p 1−p

Let Pi , i = 0, 1, . . . , N , be the probability that starting with i, the gambler

will eventually reach N .

92 / 101
Example

Result
 1−(q/p)i
 1−(q/p)N , if p ̸= 1/2

Pi =

i
N
, if p = 1/2.

See derivation of this result .

As N → ∞, we observe

i
1 − (q/p) , if p > 1/2

Pi =

0, if p = 1/2.

93 / 101
8.3 Supplementary Material

94 / 101
Supplementary Material - Gambler’s Ruin Problem

Gambler’s Ruin Problem

We have that
There is a possibility of an infinite sequence of independent identically
distributed games.
P (win one unit) = p, P (lose one unit) = q = 1 − p.
Xn is the player’s fortune after n games, n = 0, 1, 2, 3, . . . .
The state space is S = {0, 1, 2, . . . , N } and it represents the player’s
fortune.
Transitional probabilities are
P00 = PN N = 1,
Pi,i+1 = p, i = 1, 2, . . . N − 1,
Pi,i−1 = q, i = 1, 2, . . . N − 1.
The classes are {0} (recurrent), {1, 2, . . . , N − 1} (transient), and {N }
(recurrent). 95 / 101
Supplementary Material - Gambler’s Ruin Problem -continued
S 
∞
Let Pi ≡ P n=0 {Xn = N } X0 = i , i − 0, 1, 2, . . . , N be the probability
that starting with i, the gambler’s fortune will eventually reach N .
By conditioning on X1 , we get
Pi = p Pi+1 + q Pi−1 , i = 1, 2, . . . N − 1.
We can rewrite this as
p Pi + q Pi = p Pi+1 + q Pi−1 , i = 1, 2, . . . N − 1
since 1 = p + q, or
q Pi − q Pi−1 = p Pi+1 − p Pi , i = 1, 2, . . . N − 1.
That is,
q
Pi+1 − Pi = (Pi − Pi−1 ), i = 1, 2, . . . N − 1.
p

96 / 101
Supplementary Material - Gambler’s Ruin Problem -continued

We know that P0 = 0, so we have

q q
P 2 − P1 = (P1 − P0 ) = P1
p p
 2
q q
P 3 − P2 = (P2 − P1 ) = P1
p p
..
.
 i−1
q q
Pi − Pi−1 = (Pi−1 − Pi−2 ) = P1
p p
..
.
 N −1
q q
PN − PN −1 = (PN −1 − PN −2 ) = P1 .
p p

97 / 101
Supplementary Material - Gambler’s Ruin Problem -continued

Now add the first (i − 1) equations. We get

(P2 − P1 ) + (P3 − P2 ) + · · · + (Pi − Pi−1 ) = Pi − P1
   2  i−1 !
q q q
= + + ··· + P1 ,
p p p
or
   2  i−1 !
q q q
Pi = 1+ + + ··· + P1
p p p
 1−(q/p)i q
 1−(q/p) P1 , if p ̸= 1,

= (3)

i P1 if pq = 1.


98 / 101
Supplementary Material - Gambler’s Ruin Problem -continued

Now, q/p ̸= 1 if and only if p ̸= 12 , and PN = 1.

1
Case 1: p ̸= 2

1 − (q/p)N
PN = 1 = P1 .
1 − (q/p)
1 − (q/p)
∴ P1 = .
1 − (q/p)N
1
Case 2: p = 2
PN = 1 = N P 1 .
1
∴ P1 = .
N

99 / 101
Supplementary Material - Gambler’s Ruin Problem -continued

Insert these values in Equation (3), and we have

 1−(q/p)i q
 1−(q/p) P1 , if p =
 ̸ 1,
Pi =

i P1 if pq = 1.


 1−(q/p)i
1
 1−(q/p)N P1 , if p > 2 ,

=

i
N
if p = 21 .

100 / 101
Supplementary Material - Gambler’s Ruin Problem -continued

Note that if p > 12 , then (q/p) < 1, and hence (q/p)N → 0 as N → ∞.

Similarly, if p < 21 , then (q/p) > 1, and hence (q/p)N → ∞ as N → ∞.

Therefore, (
1 − (q/p)i , if p > 12 ,
lim Pi =
N →∞ 0, if p ≤ 21 .

101 / 101