Discrete Time, Finite, Markov Chain

• A stochastic process X = {X (t) : t ∈ T } is a collection of

random variables.
• X (t) = the state of the process at time t = Xt .
• X is a discrete (finite) space process if for all t, Xt assumes
values from a countably infinite (finite) set.
• If T is a countably infinite set we say that X is a discrete time
process.
Definition
A discrete time stochastic process X0 , X1 , X2 , . . . is a Markov chain
if

Pr(Xt = at |Xt−1 = at−1 , Xt−2 = at−2 , . . . , X0 = a0 )

= Pr(Xt = at |Xt−1 = at−1 ) = Pat−1 ,at .

Transition probability: Pi,j = Pr(Xt = j | Xt−1 = i)

Transition matrix:

 
P0,0 P0,1 ··· P0,j ···
 P1,0
 P1,1 ··· P1,j ··· 

 .. .. .. .. .. 
 .
P=  . . . . .

 Pi,0
 Pi,1 ··· Pi,j ··· 
.. .. .. .. ..
. . . . .
Probability distribution for a given time t:

p̄(t) = (p0 (t), p1 (t), p2 (t), . . .)

X
pi (t) = pj (t − 1)Pj,i
j≥0

p̄(t) = p̄(t − 1)P.

For any n ≥ 0 we define the n-step transition probability

n
Pi,j = Pr(Xt+n = j | Xt = i)
Conditioning on the first transition from i we have
X
n n−1
Pi,j = Pi,k Pk,j . (1)
k≥0

Let P(n) be the matrix whose entries are the n-step transition
n
probabilities, so that the entry in the ith row and jth column is Pi,j
Then we have
P(n) = P · P(n−1) ,
and by induction on n
P(n) = Pn .

Thus, for any t ≥ 0 and n ≥ 1,

p̄(t + n) = p̄(t)Pn .
 Example

Consider a system with a total of m balls in two containers.

We start with all balls in the first container.
At each step we choose a ball uniformly at random from all the
balls and with probability 1/2 move it to the other container.
Let Xi denote the number of balls in the first container at time i.
X0 , X1 , X2 , .... defines a Markov chain with the following transition
matrix:
 m−i

 2m j =i +1



i

j =i −1


 2m
pi,j =
1
j =i




 2



|i − j| > 1

0
 Randomized 2-SAT Algorithm

Given a formula with up to two variables per clause, find a Boolean

assignment that satisfies all clauses.
Algorithm:
1 Start with an arbitrary assignment.
2 Repeat till all clauses are satisfied:
1 Pick an unsatisfied clause.
2 If the clause has one variable change the value of that variable.
3 If the clause has two variable choose one uniformly at random
and change its value.
What the is the expected run-time of this algorithm?
W.l.o.g. assume that all clause have two variables.
Assume that the formula has a satisfying assignment. Pick one
such assignment S.
Let Xi be the number of variables with the correct assignment
according to the assignment S after iteration i of the algorithm.
Let n be the number of variable.

Pr (Xi = 1 | Xi−1 = 0) = 1
For 1 ≤ t ≤ n − 1,

Prob(Xi = t + 1 | Xi−1 = t) ≥ 1/2

Prob(Xi = t − 1 | Xi−1 = t) ≤ 1/2

Assume
Pr (Xi = 1 | Xi−1 = 0) = 1
for 1 ≤ t ≤ n − 1,

Prob(Xi = t + 1 | Xi−1 = t) = 1/2

Prob(Xi = t − 1 | Xi−1 = t) = 1/2

Let Dt be the expected number of steps to termination when we
have t incorrect variable assignments.
Dn = 1 + Dn−1 .
Dt = 1 + 21 Dt+1 + 12 Dt−1
We “guess”
Dt = t(2n − t)
.
D0 = 0.
 1 1
Dt = 1 + (t + 1)(2n − t − 1) + (t − 1)(2n − t + 1) =
2 2

1
1 + (2nt + 2n − t 2 − t − t − 1 + 2nt − 2n − t 2 + t + t − 1) =
2

1 + 2nt − t 2 − 1 = t(2n − t).

Dn = 1 + Dn−1 = 1 + (n − 1)(2n − n + 1) = n2 .
Theorem
Assuming that the formula has a satisfying assignment the
expected run-time to find that assignment is O(n2 ).

Theorem
There is a one-sides error randomized algorithm for the 2-SAT
problem that terminates in O(n2 log n) time, with high probability
returns an assignment when the formula is satisfiable, and always
returns “UNSATISFIABLE” when no assignment exists.

Proof.
The probability that the algorithm does not find an assignment
when exists in 2n2 steps is bounded by 12 .
 A randomized 3-SAT algorithm

Given a formula with exactly 3 variables per clause, find a Boolean

assignment that satisfies all clauses.
Algorithm:
1 Start with an arbitrary assignment.
2 Repeat till all clauses are satisfied:
1 Pick an unsatisfied clause C .
2 Pick a random litteral in the clause C and switch its value.
NB: 3-SAT is NPC so we should not expect our algorithm to be
polynomial, even if there is a valid truth assignment.
We first try to analyze in the same way as for the 2-SAT algorithm:
Assume the formula is satisfiable and that S is a fized satisfying
assignment.Let Ai the assignment after i steps and let Xi denote
the number of variables whose values is the same in Ai as in S.

1
Pr (Xi+1 = j + 1|Xi = j) ≥
3
2
Pr (Xi+1 = j − 1|Xi = j) ≤
3
If we assume equalities above and solve as for 2-SAT, we will get
(with Dt being the expected number of steps to termination when
we have t incorrect variable assignments), then we get

Dt = 2n+2 − 2t+2 − 3(n − t)

D0 = Θ(2n )

This is not good since there are only 2n truth assignments to try.
The problem is that the number of variables that agree with S
becomes smaller over time with high probability.
• If we start from a random truth assignment, then w.h.p. this
agrees with S in n/2 variables (this is the expectation).
• Once we start the algorithm, we tend to move towards 0
rather than n correct variables. Hence we are better off
restarting the process several times and taking only a small
number of steps (3n works as we shall see) before restarting.
 A modified algorithm

Modified 3-SAT Algorithm:

1 Repeat m times (alternatively: till all clauses are satisfied):
2 Start with uniformly random assignment.
3 Repeat up to 3n times, terminating if a satisfying assignment
is found:
1 Pick an unsatisfied clause C .
2 Pick a random litteral in the clause C and switch its value.
 Analogy to a particle move on the integer line

Consider a particle moving on the integer line:

with probability 13 it moves up by one and with probability 23 it
moves down by one. Then the probability of exatly k moves down
and j + k moves up is:
 
j + 2k 2
k 1
j+k
k 3 3

Let qj denote the probability that the algorithm reaches a

satisfying assignment within 3n steps when the initial (random)
assignment agreed with S on j variables.
 
j + 2k 2
k 1
j+k
qj ≥ max
k=0,...,j k 3 3
In particular, with k = j we have:
 
3j 2
j 1
2j
qj ≥
j 3 3

Using Stirlings formula, one can get (when j > 0)

 
3j c 27
j
≥√
j j 4

 
3j 2
j 1
2j
qj ≥
j 3 3
c 27
j 2
j 1
2j
≥ √ (
j 4 3 3
c 1
≥ √ j
j2
Let q be the probability of reaching a satisfying assignment within
3n steps starting from the random initial assignment (in one
round):
n
X
q ≥ Pr(start with j mismatches with S)qj
j=0
n  
1 X n 1
n c 1
≥ + √ j
2n j 2 j2
j=1
n  
c 1
n X n 1
j n−j
≥ √ ( (1)
n 2 j 2
j=1
c 1
n 3
n
= √ ( (
n 2 2
c 3
n
= √ (
n 4
If S exists, the number of times we have to repeat the initial
random assignments is a geometric random variable with
parameter q. The expected number of repetitions is q1 .
√
n
Hence, the expected number of repetitions is cn ( 43 so the
3
n
expected number of steps until a solution is found is O(n 2 ( 43 )
If a denotes the expected number of steps above, then, by
Markov’s inequality, the probability that we need more than 2a
steps is at most 21 so if we repeat the outer loop (picking a new
random assignment) 2ab times, then the probability that no
solution is found when one exists is at most 2−b .
 Classification of States

Definition
State j is accessible from state i if for some integer n ≥ 0,
n > 0. If two states i and j are accessible from each other we
Pi,j
say that they communicate, and we write i ↔ j.

In the graph representation i ↔ j if and only if there are directed

paths connecting i to j and j to i.
The communicating relation defines an equivalence relation. That
is, the relation is
1 Reflexive: for any state i, i ↔ i;
2 Symmetric: if i ↔ j then j ↔ i; and
3 Transitive: if i ↔ j and j ↔ k, then i ↔ k.
Definition
A Markov chain is irreducible if all states belong to one
communicating class.

Lemma
A finite Markov chain is irreducible if and only if its graph
representation is a strongly connected graph.
 t = the probability that starting at state i the first transition to
ri,j
state j occurred at time t, that is,
t
ri,j = Pr(Xt = j and for 1 ≤ s ≤ t − 1, Xs 6= j | X0 = i).

Definition
t
P
A state is recurrent if t≥1 ri,i = 1, and it is transient if
t
P
t≥1 ri,i < 1. A Markov chain is recurrent if every state in the
chain is recurrent.
The expected time to return to state i when starting at state j:
X
t
hj,i = t · rj,i
t≥1

Definition
A recurrent state i is positive recurrent if hi,i < ∞. Otherwise, it is
null recurrent.
 Example - null recurrent states

States are the positive numbers.

 i
 i+1 j =i +1
i
Pi,j = 1 − i+1 j =1

0 otherwise

The probability of not having returned to state 1 within the first t

steps is
t
Y j 1
= .
j +1 t +1
j=1

The probability of never returning to state 1 from state 1 is 0, and

state 1 is recurrent.
 t 1
r1,1 = .
t(t + 1)

∞ ∞
X
t
X 1
h1,1 = t · r1,1 = =∞
t +1
t=1 t=1

State 1 is null recurrent.

Lemma
In a finite Markov chain,
1 At least one state is recurrent;
2 All recurrent states are positive recurrent.
Definition
A state j in a discrete time Markov chain is periodic if there exists
an integer ∆ > 1 such that Pr(Xt+s = j | Xt = j) = 0 unless s is
divisible by ∆. A discrete time Markov chain is periodic if any
state in the chain is periodic. A state or chain that is not periodic
is aperiodic.

Definition
An aperiodic, positive recurrent state is an ergodic state. A
Markov chain is ergodic if all its states are ergodic.

Corollary
Any finite, irreducible, and aperiodic Markov chain is an ergodic
chain.
 Example: The Gambler’s Ruin
• Consider a sequence of independent, two players, fair
gambling games.
• In each round a player wins a dollar with probability 1/2 or
loses a dollar with probability 1/2.
• W t = the number of dollars won by player 1 up to (including)
step t.
• If player 1 has lost money, this number is negative.
• W 0 = 0. For any t, E[W t ] = 0.
• Player 1 must end the game if she loses `1 dollars
(W t = −`1 ); player 2 must terminate when she loses `2
dollars (W t = `2 ).
• Let q be the probability that the game ends with player 1
winning `2 dollars.
• If `2 = `1 , then by symmetry q = 1/2. What is q when
`2 6= `1 ?
−`1 and `2 are recurrent states. All other states are transient. Let

Pit be the probability that after t steps the chain is at state i.

For −`1 < i < `2 ,limt→∞ Pit = 0.

lim P`t2 = q.
t→∞

lim P`t1 = 1 − q.
t→∞
 `2
X
t
E[W ] = iPit = 0
i=−`1

lim E[W t ] = `2 q − `1 (1 − q) = 0.
t→∞

`1
q = .
`1 + `2