m3f33chIV

Chapter IV: STOCHASTIC PROCESSES IN DISCRETE TIME

§1. Filtrations.

The Kolmogorov triples (Ω, F, P ), and the Kolmogorov conditional ex-

pectations E(X|B), give us all the machinery we need to handle static situ-
ations involving randomness. To handle dynamic situations, involving ran-
domness which unfolds with time, we need further structure.
We may take the initial, or starting, time as t = 0. Time may evolve dis-
cretely, or continuously. We postpone the continuous case to Ch. VI; in the
discrete case, we may suppose time evolves in integer steps, t = 0, 1, 2, · · ·
(say, stock-market quotations daily, or tick data by the second). There may
be a final time T , or time horizon, or we may have an infinite time horizon
(in the context of option pricing, the time horizon T is the expiry time).
We wish to model a situation involving randomness unfolding with time.
We suppose, for simplicity, that information is never lost (or forgotten): thus,
as time increases we learn more. Recall that σ-fields represent information
or knowledge. We thus need a sequence of σ-fields {Fn : n = 0, 1, 2, · · · },
which are increasing:

Fn ⊂ Fn+1 (n = 0, 1, 2, · · · ),

with Fn representing the information, or knowledge, available to us at time

n. We shall always suppose all σ-fields to be complete (this can be avoided,
and is not always appropriate, but it simplifies matters and suffices for our
purposes). Thus F0 represents the initial information (if there is none, F0 =
{∅, Ω}, the trivial σ-field). On the other hand,

F∞ := limn→∞ Fn

represents all we ever will know (the ‘Doomsday σ-field’). Often, F∞ will be
F (the σ-field from Ch. II, representing ‘knowing everything’. But this will
not always be so; see e.g. [W], §15.8 for an interesting example.
Such a family {Fn : n = 0, 1, 2, · · · } is called a filtration; a probabil-
ity space endowed with such a filtration, {Ω, {Fn }, F, P } is called a filtered
probability space. (These definitions are due to P.- A. MEYER of Strasbourg;
Meyer and the Strasbourg (and more generally, French) school of probabilists

1
have been responsible for the ‘general theory of [stochastic] processes’, and
for much of the progress in stochastic integration, since the 1960s.) Since
the filtration is so basic to the definition of a stochastic process, the more
modern term for a filtered probability space is a stochastic basis.

§2. Discrete-Parameter Stochastic Processes.

A stochastic process X = {Xt : t ∈ I} is a family of random variables,

defined on some common probability space, indexed by an index-set I. Usu-
ally (always in this course), I represents time (sometimes I represents space,
and one calls X a spatial process). Here, I = {0, 1, 2, · · · , T } (finite horizon)
or I = {0, 1, 2, · · · } (infinite horizon – as in VII.6, Real/Investment options).
The (stochastic) process X = (Xn )∞ n=0 is said to be adapted to the filtra-
∞
tion (Fn )n=0 if
Xn is Fn − measurable.
So if X is adapted, we will know the value of Xn at time n. If

Fn = σ(X0 , X1 , · · · , Xn )

we call (Fn ) the natural filtration of X. Thus a process is always adapted to

its natural filtration. A typical situation is that

Fn = σ(W0 , W1 , · · · , Wn )

is the natural filtration of some process W = (Wn ). Then X is adapted to

(Fn ), i.e. each Xn is Fn - (or σ(W0 , · · · , Wn )-) measurable, iff

Xn = fn (W0 , W1 , · · · , Wn )

for some measurable function fn (non-random) of n + 1 variables.

Notation.
For a random variable X on (Ω, F, P ), X(ω) is the value X takes on ω
(ω represents the randomness).R Often, to simplify notation,R ω is suppressed
- e.g., we may write E[X] := Ω XdP instead of E[X] := Ω X(ω)dP (ω).
For a stochastic process X = (Xn ), it is convenient (e.g., if using suffices,
ni say) to use Xn , X(n) interchangeably, and we shall feel free to do this.
With ω displayed, these become Xn (ω), X(n, ω), etc.

2
§3. Discrete-Parameter Martingales.

We summarise what we need; for details, see [W], or e.g. [N]

Definition.
A process X = (Xn ) is called a martingale (mg for short) relative to
((Fn ), P ) if
(i) X is adapted (to (Fn )),
(ii) E[|Xn |] < ∞ for all n,
(iii) E[Xn |Fn−1 ] = Xn−1 P − a.s. (n ≥ 1);
X is a supermartingale if in place of (iii)

E[Xn |Fn−1 ] ≤ Xn−1 P − a.s. (n ≥ 1);

X is a submartingale if in place of (iii)

E[Xn |Fn−1 ] ≥ Xn−1 P − a.s. (n ≥ 1).

Thus: a mg is ‘constant on average’, and models a fair game;

a supermg is ‘decreasing on average’, and models an unfavourable game;
a submg is ‘increasing on average’, and models a favourable game.
Note. 1. Martingales have many connections with harmonic functions in
probabilistic potential theory. The terminology in the inequalities above
comes from this: supermartingales correspond to superharmonic functions,
submartingales to subharmonic functions.
2. X is a submg [supermg] iff −X is a supermg [submg]; X is a mg iff it is
both a submg and a supermg.
3. (Xn ) is a mg iff (Xn − X0 ) is a mg. So we may without loss of generality
take X0 = 0 when convenient.
4. If X is a mg, then for m < n

E[Xn |Fm ] = E[E(Xn |Fn−1 )|Fm ] (iterated conditional expectations)

= E[Xn−1 |Fm ] a.s. (martingale property)

= · · · = E[Xm |Fm ] a.s. (induction on n),
= Xm (Xm is Fm -measurable)
and similarly for submartingales, supermartingales.
5. Examples of a mg include: sums of independent, integrable zero-mean
random variables [submg: positive mean; supermg: negative mean].

3
From the OED: martingale (etymology unknown)
1. 1589. An article of harness, to control a horse’s head.
2. Naut. A rope for guying down the jib-boom to the dolphin-striker.
3. A system of gambling which consists in doubling the stake when losing in
order to recoup oneself (1815).
Thackeray: ‘You have not played as yet? Do not do so; above all avoid a
martingale if you do.’
Problem. Analyse this strategy.

Gambling games have been studied since time immemorial - indeed, the
Pascal-Fermat correspondence of 1654 which started the subject was on a
problem (de Méré’s problem) related to gambling.
The doubling strategy above has been known at least since 1815.
The term ‘mg’ in our sense is due to J. VILLE (1939). Martingales were
studied by Paul LÉVY (1886-1971) from 1934 on [see obituary, Annals of
Probability 1 (1973), 5-6] and by J. L. DOOB (1910-2004) from 1940 on.
The first systematic exposition was Doob’s book [D], Ch. VII.
Example: Accumulating data about a random variable ([W], 96, 166-167).
If ξ ∈ L1 (Ω, F, P ), Mn := E(ξ|Fn ) (so Mn represents our best estimate of ξ
based on knowledge at time n), then
E[Mn |Fn−1 ] = E[E(ξ|Fn )|Fn−1 ]
= E[ξ|Fn−1 ] (iterated conditional expectations)
= Mn−1 ,
so (Mn ) is a mg. One has the convergence (see IV.4 below)
Mn → M∞ := E[ξ|F∞ ] a.s. and in L1 .
§4. Martingale Convergence.

A supermartingale is ‘decreasing on average’. Recall that a decreasing

sequence [of real numbers] that is bounded below converges (decreases to
its greatest lower bound or infimum). This suggests that a supermartingale
which is bounded below converges a.s. This is so [Doob’s Forward Conver-
gence Theorem: [W], §§11.5, 11.7].
More is true. Call X L1 -bounded if
sup E[|Xn |] < ∞.
n

4
Theorem (Doob). An L1 -bounded supermartingale is a.s. convergent:
there exists X∞ finite such that

X n → X∞ (n → ∞) a.s.

In particular, we have

Doob’s Martingale Convergence Theorem [W, §11.5]. An L1 -bounded

martingale converges a.s.

We say that
Xn → X∞ in L1
if
E[ |Xn − X∞ | ] → 0 (n → ∞).
For a class of martingales, one gets convergence in L1 as well as almost
surely [= with probability one]. Such martingales are called uniformly in-
tegrable (UI) [W], or regular [N], or closed (see below), They are ”the nice
ones”. Fortunately, they are the ones we need.
The following result is in [N], IV.2, [W], Ch. 14; cf. SP L18-19, SA L6.

Theorem (UI Martingale Convergence Theorem). The following are

equivalent for martingales X = (Xn ):
(i) Xn converges in L1 ,
(ii) Xn is L1 -bounded, and its a.s. limit X∞ (which exists, by above) satisfies

Xn = E[X∞ |Fn ],

(iii) There exists an integrable random variable X with

Xn = E[X|Fn ].

The random variable X∞ above serves to ”close” the martingale, by giv-

ing Xn a value at ”n = ∞”; then {Xn : n = 1, 2, . . . , ∞} is again a martingale
– which we may accordingly call a closed mg. The terms closed, regular and
UI are used interchangeably here.
Notice that all the randomness in a closed mg is in the closing value
X∞ (so, although a stochastic process is an infinite-dimensional object, the
randomness in a closed mg is one-dimensional). As time progresses, more

5
is revealed, by ”progressive revelation” – as in (choose your metaphor) a
striptease, or the ”Day of Judgement” (when all will be revealed).
As we shall see (Risk-Neutral Valuation Formula): closed mgs are vital
in mathematical finance, and the closing value corresponds to the payoff of
an option.

§5. Martingale Transforms.

Now think of a gambling game, or series of speculative investments, in

discrete time. There is no play at time 0; there are plays at times n = 1, 2, · · · ,
and
∆Xn := Xn − Xn−1
represents our net winnings per unit stake at play n. Thus if Xn is a mar-
tingale, the game is ‘fair on average’.
Call a process C = (Cn )∞n=1 previsible (or predictable) if

Cn is Fn−1 − measurable for all n ≥ 1.

Think of Cn as your stake on play n (C0 is not defined, as there is no play at

time 0). Previsibility says that you have to decide how much to stake on play
n based on the history before time n (i.e., up to and including play n − 1).
Your winnings on game n are Cn ∆Xn = Cn (Xn − Xn−1 ). Your total (net)
winnings up to time n are
n
X n
X
Yn = Ck ∆Xk = Ck (Xk − Xk−1 ).
k=1 k=1

We write

Y = C • X, Yn = (C • X)n , ∆Yn = Cn ∆Xn

((C • X)0 = 0 as 01 is empty), and call C • X the martingale transform of

P
X by C.

Theorem. (i) If C is a bounded non-negative previsible process and X is a

supermartingale, C • X is a supermartingale null at zero.
(ii) If C is bounded and previsible and X is a martingale, C • X is a martin-
gale null at zero.

6
Proof.
With Y = C • X as above,

E[Yn − Yn−1 |Fn−1 ] = E[Cn (Xn − Xn−1 )|Fn−1 ]

= Cn E[(Xn − Xn−1 )|Fn−1 ]

(as Cn is bounded, so integrable, and Fn−1 -measurable, so can be taken out)

≤0

in case (i), as C ≥ 0 and X is a supermartingale,

=0

in case (ii), as X is a martingale. //

Interpretation. You can’t beat the system!

In the martingale case, previsibility of C means we can’t foresee the future
(which is realistic and fair). So we expect to gain nothing – as we should.
Note. 1. Martingale transforms were introduced and studied by Donald L.
BURKHOLDER (1927 - 2013) in 1966 [Ann. Math. Statist. 37, 1494-1504].
For a textbook account, see e.g. [N], VIII.4.
2. Martingale transforms are the discrete analogues of stochastic integrals.
They dominate the mathematical theory of finance in discrete time, just as
stochastic integrals dominate the theory in continuous time.
3. In mathematical finance, X plays the role of a price process, C plays the
role of our trading strategy, and the mg transform C • X plays the role of
our gains (or losses!) from trading.

Proposition (Martingale Transform Lemma). An adapted sequence of

real integrable random variables (Mn ) is a martingale iff for any bounded
previsible sequence (Hn ),
n
X
E[ Hr ∆Mr ] = 0 (n = 1, 2, · · · ).
r=1

Proof.
If (Mn ) is a martingale, X defined by X0 = 0, Xn = n1 Hr ∆Mr
P
(n ≥
1) is the martingale transform H • M , so is a martingale.

7
 Conversely, if the condition of the Proposition holds, choose j, and for
any Fj -measurable set A write Hn = 0 for n 6= j + 1, HP j+1 = IA . Then
(Hn ) is previsible, so the condition of the Proposition, E[ n1 Hr ∆Mr ] = 0,
becomes
E[IA (Mj+1 − Mj )] = 0.
As this holds for every A ∈ Fj , the definition of conditional expectation gives

E[Mj+1 |Fj ] = Mj .

Since this holds for every j, (Mn ) is a martingale. //

§6. Stopping Times and Optional Stopping.

A random variable T taking values in {0, 1, 2, · · · ; +∞} is called a stop-

ping time (or optional time) if

{T ≤ n} = {ω : T (ω) ≤ n} ∈ Fn ∀n ≤ ∞.

Equivalently,
{T = n} ∈ Fn n ≤ ∞.
Think of T as a time at which you decide to quit a gambling game: whether
or not you quit at time n depends only on the history up to and including
time n – NOT the future. [Elsewhere, T denotes the expiry time of an option.
If we mean T to be a stopping time, we will say so.]
The following important classical theorem is discussed in [W], 10.10.

Theorem (Doob’s Optional Stopping Theorem, OST). Let T be a

stopping time, X = (Xn ) be a supermartingale, and assume that one of the
following holds:
(i) T is bounded [T (ω) ≤ K for some constant K and all ω ∈ Ω];
(ii) X = (Xn ) is bounded [|Xn (ω)| ≤ K for some K and all n, ω];
(iii) E[T ] < ∞ and (Xn − Xn−1 ) is bounded.
Then XT is integrable, and

E[XT ] ≤ E[X0 ].

If here X is a martingale, then

E[XT ] = E[X0 ].

8
 The OST is important in many areas, such as sequential analysis in
statistics. We turn in the next section to related ideas specific to the gam-
bling/financial context.
Write XnT := Xn∧T for the sequence (Xn ) stopped at time T .

Proposition. (i) If (Xn ) is adapted and T is a stopping-time, the stopped

sequence (Xn∧T ) is adapted.
(ii) If (Xn ) is a martingale [supermartingale] and T is a stopping time, (XnT )
is a martingale [supermartingale].

Proof. If φj := I{j ≤ T },
n
X
XT ∧n = X0 + φj (Xj − Xj−1 ).
1

Since {j ≤ T } is the complement of {T < j} = {T ≤ j − 1} ∈ Fj−1 ,

φj = I{j ≤ T } ∈ Fj−1 , so (φn ) is previsible. So (XnT ) is adapted.
If (Xn ) is a martingale, so is (XnT ) as it is the martingale transform of
(Xn ) by (φn ). Since by previsibility of (φn )
n−1
X
E[XT ∧n |Fn−1 ] = X0 + φj (Xj − Xj−1 ) + φn (E[Xn |Fn−1 ] − Xn−1 ), i.e.
1

E[XT ∧n |Fn−1 ] − XT ∧(n−1) = φn (E[Xn |Fn−1 ] − Xn−1 ),

φn ≥ 0 shows that if (Xn ) is a supermg [submg], so is (XT ∧n ). //

§7. The Snell Envelope and Optimal Stopping.

Definition. If Z = (Zn )N
n=0 is a sequence adapted to a filtration (Fn ), the
sequence U = (Un )N
n=0 defined by backward recursion by

UN := ZN , Un := max(Zn , E[Un+1 |Fn ]) (n ≤ N − 1)

is called the Snell envelope of Z (J. L. Snell in 1952; [N] Ch. 6). U is adapted,
i.e. Un ∈ Fn for all n. For, Z is adapted, so Zn ∈ Fn . Also E[Un+1 |Fn ] ∈ Fn
(definition of conditional expectation). Combining, Un ∈ Fn , as required.
The Snell envelope (see IV.8 L20) is exactly the tool needed in pricing
American options. It is the least supermg majorant (also called the réduite

9
or reduced function – crucial in the mathematics of gambling):

Theorem. The Snell envelope (Un ) of (Zn ) is a supermartingale, and is the

smallest supermartingale dominating (Zn ) (that is, with Un ≥ Zn for all n).

Proof.
First, Un ≥ E[Un+1 |Fn ], so U is a supermartingale, and Un ≥ Zn , so U
dominates Z.
Next, let T = (Tn ) be any other supermartingale dominating Z; we must
show T dominates U also. First, since UN = ZN and T dominates Z, TN ≥
UN . Assume inductively that Tn ≥ Un . Then

Tn−1 ≥ E[Tn |Fn−1 ] (as T is a supermartingale)

≥ E[Un |Fn−1 ] (by the induction hypothesis)

and
Tn−1 ≥ Zn−1 (as T dominates Z).
Combining,
Tn−1 ≥ max(Zn−1 , E[Un |Fn−1 ]) = Un−1 .
By backward induction, Tn ≥ Un for all n, as required. //

Note. It is no accident that we are using induction here backwards in time.

We will use the same method – also known as dynamic programming (DP) –
in Ch. IV below when we come to pricing American options.

Proposition. T0 := min{n ≥ 0 : Un = Zn } is a stopping time, and the

stopped sequence (UnT0 ) is a martingale.

We omit the proof (not hard, but fiddly – for details, see e.g. L13, 2014).
Because U is a supermartingale, we knew that stopping it would give a su-
permartingale, by the Proposition of §6. The point is that, using the special
properties of the Snell envelope, we actually get a martingale.
Write Tn,N for the set of stopping times taking values in {n, n+1, · · · , N }
(a finite set, as Ω is finite). We next see that the Snell envelope solves the
optimal stopping problem: it maximises the expectation of our final value of
Z – the value when we choose to quit – conditional on our present (publicly

10
available) information. This is the best we can hope to do in practice (with-
out cheating – insider trading, etc.)

Theorem. T0 solves the optimal stopping problem for Z:

U0 = E[ZT0 |F0 ] = max{E[ZT |F0 ] : T ∈ T0,N }.

Proof. As (UnT0 ) is a martingale (above),

U0 = U0T0 (since 0 = 0 ∧ T0 )
T0
= E[UN |F0 ] (by the martingale property)
= E[UT0 |F0 ] (since T0 = T0 ∧ N )
= E[ZT0 |F0 ] (since UT0 = ZT0 ),

proving the first statement. Now for any stopping time T ∈ T0,N , since U is
a supermartingale (above), so is the stopped process (UnT ) (§6). So

U0 = U0T (0 = 0 ∧ T , as above)
≥ E[UNT |F0 ] ((UnT ) a supermartingale)
= E[UT |F0 ] (T = T ∧ N )
≥ E[ZT |F0 ] ((Un ) dominates (Zn )),

and this completes the proof. //

The same argument, starting at time n rather than time 0, gives an ap-
parently more general version:

Theorem. If Tn := min{j ≥ n : Uj = Zj },

Un = E[ZTn |Fn ] = sup{E[ZT |Fn ] : T ∈ Tn,N }.

To recapitulate: as we are attempting to maximise our payoff by stopping

Z = (Zn ) at the most advantageous time, the Theorem shows that Tn gives
the best stopping-time that is realistic: it maximises our expected payoff given
only information currently available (it is easy, but irrelevant, to maximise
things with hindsight!). We thus call T0 (or Tn , starting from time n) the
optimal stopping time for the problem.

11
§8. Doob Decomposition.
Theorem. Let X = (Xn ) be an adapted process with each Xn ∈ L1 . Then
X has an (essentially unique) Doob decomposition
X = X0 + M + A : Xn = X0 + Mn + An ∀n (D)
with M a mg null at zero, A a previsible process null at zero. If also X is a
submg (‘increasing on average’), A is increasing: An ≤ An+1 for all n, a.s.

The proof in discrete time is quite easy (see L13, 2014). It is hard in
continuous time – but more important there (see Ch. V: quadratic variation
(QV) and the Itô integral). This illustrates the contrasts between the theo-
ries of stochastic processes in discrete and continuous time.

§9. Examples. Pn
1. Simple random walk. Recall the simple random walk: Sn := 1 Xk ,
where the Xn are independent tosses of a fair coin, taking values ±1 with
equal probability 1/2. We decide to bet until our net gain is first +1, then
quit – at time T , a stopping time. This has been analysed in detail; see e.g.
[GS] GRIMMETT, G. R. & STIRZAKER, D.: Probability and random pro-
cesses, OUP, 3rd ed., 2001 [2nd ed. 1992, 1st ed. 1982], §5.2:
(i) T < ∞ a.s.: the gambler will certainly achieve a net gain of +1 eventually;
(ii) E[T ] = +∞: the mean waiting-time till this happens is infinity. So:
(iii) No bound can be imposed on the gambler’s maximum net loss before his
net gain first becomes +1.
At first sight, this looks like a foolproof way to make money out of noth-
ing: just bet till you get ahead (which happens eventually, by (i)), then quit.
But as a gambling strategy, this is hopelessly impractical: because of (ii),
you need unlimited time, and because of (iii), you need unlimited capital!
Notice that the Optional Stopping Theorem fails here: we start at zero,
so S0 = 0, E[S0 ] = 0; but ST = 1, so E[ST ] = 1. This shows two things:
(a) The Optional Stopping Theorem does indeed need conditions, as the con-
clusion may fail otherwise [none of the conditions (i) - (iii) in the OST are
satisfied in the example above],
(b) Any practical gambling (or trading) strategy needs to have some inte-
grability or boundedness restrictions to eliminate such theoretically possible
but practically ridiculous cases.
2. The doubling strategy. Similarly for the doubling strategy (§3).

12