Chapter 5

An Introduction to Discrete Probability

5.1 Sample Space, Outcomes, Events, Probability

Roughly speaking, probability theory deals with experiments whose outcome are
not predictable with certainty. We often call such experiments random experiments.
They are subject to chance. Using a mathematical theory of probability, we may be
able to calculate the likelihood of some event.
In the introduction to his classical book [1] (first published in 1888), Joseph
Bertrand (1822–1900) writes (translated from French to English):

“How dare we talk about the laws of chance (in French: le hasard)? Isn’t chance
the antithesis of any law? In rejecting this definition, I will not propose any
alternative. On a vaguely defined subject, one can reason with authority. ...”
Of course, Bertrand’s words are supposed to provoke the reader. But it does seem
paradoxical that anyone could claim to have a precise theory about chance! It is not
my intention to engage in a philosophical discussion about the nature of chance.
Instead, I will try to explain how it is possible to build some mathematical tools that
can be used to reason rigorously about phenomema that are subject to chance. These
tools belong to probability theory. These days, many fields in computer science
such as machine learning, cryptography, computational linguistics, computer vision,
robotics, and of course algorithms, rely a lot on probability theory. These fields are
also a great source of new problems that stimulate the discovery of new methods
and new theories in probability theory.
Although this is an oversimplification that ignores many important contributors,
one might say that the development of probability theory has gone through four eras
whose key figures are: Pierre de Fermat and Blaise Pascal, Pierre–Simon Laplace,
and Andrey Kolmogorov. Of course, Gauss should be added to the list; he made
major contributions to nearly every area of mathematics and physics during his life-
time. To be fair, Jacob Bernoulli, Abraham de Moivre, Pafnuty Chebyshev, Alek-
sandr Lyapunov, Andrei Markov, Emile Borel, and Paul Lévy should also be added
to the list.

269
270 5 An Introduction to Discrete Probability

Fig. 5.1 Pierre de Fermat (1601–1665) (left), Blaise Pascal (1623–1662) (middle left), Pierre–
Simon Laplace (1749–1827) (middle right), Andrey Nikolaevich Kolmogorov (1903–1987) (right).

Before Kolmogorov, probability theory was a subject that still lacked precise def-
initions. In1933, Kolmogorov provided a precise axiomatic approach to probability
theory which made it into a rigorous branch of mathematics with even more appli-
cations than before!
The first basic assumption of probability theory is that even if the outcome of an
experiment is not known in advance, the set of all possible outcomes of an experi-
ment is known. This set is called the sample space or probability space. Let us begin
with a few examples.

Example 5.1. If the experiment consists of flipping a coin twice, then the sample
space consists of all four strings

W = {HH, HT, TH, TT},

where H stands for heads and T stands for tails.

If the experiment consists in flipping a coin five times, then the sample space
W is the set of all strings of length five over the alphabet {H, T}, a set of 25 = 32
strings,

W = {HHHHH, THHHH, HTHHH, TTHHH, . . . , TTTTT}.

Example 5.2. If the experiment consists in rolling a pair of dice, then the sample
space W consists of the 36 pairs in the set

W = D⇥D

with
D = {1, 2, 3, 4, 5, 6},
where the integer i 2 D corresponds to the number (indicated by dots) on the face of
the dice facing up, as shown in Figure 5.2. Here we assume that one dice is rolled
first and then another dice is rolled second.

Example 5.3. In the game of bridge, the deck has 52 cards and each player receives
a hand of 13 cards. Let W be the sample space of all possible hands. This time it is
not possible to enumerate the sample space explicitly. Indeed, there are
5.1 Sample Space, Outcomes, Events, Probability 271

Fig. 5.2 Two dice.

✓ ◆
52 52! 52 · 51 · 50 · · · 40
= = = 635, 013, 559, 600
13 13! · 39! 13 · 12 · · · · 2 · 1

different hands, a huge number.

Each member of a sample space is called an outcome or an elementary event.

Typically, we are interested in experiments consisting of a set of outcomes. For
example, in Example 5.1 where we flip a coin five times, the event that exactly one
of the coins shows heads is

A = {HTTTT, THTTT, TTHTT, TTTHT, TTTTH}.

The event A consists of five outcomes. In Example 5.2, the event that we get “dou-
bles” when we roll two dice, namely that each dice shows the same value is,

B = {(1, 1), (2, 2), (3, 3), (4, 4), (5, 5), (6, 6)},

an event consisting of 6 outcomes.

The second basic assumption of probability theory is that every outcome w of
a sample space W is assigned some probability Pr(w). Intuitively, Pr(w) is the
probabilty that the outcome w may occur. It is convenient to normalize probabilites,
so we require that
0  Pr(w)  1.
If W is finite, we also require that

 Pr(w) = 1.
w2W

The function Pr is often called a probability measure or probability distribution on

W . Indeed, it distributes the probability of 1 among the outcomes w.
In many cases, we assume that the probably distribution is uniform, which means
that every outcome has the same probability.
For example, if we assume that our coins are “fair,” then when we flip a coin five
times as in Example 5.1, since each outcome in W is equally likely, the probability
of each outcome w 2 W is
1
Pr(w) = .
32
272 5 An Introduction to Discrete Probability

If we assume in Example 5.2, that our dice are “fair,” namely that each of the six
possibilities for a particular dice has probability 1/6 , then each of the 36 rolls w 2 W
has probability
1
Pr(w) = .
36
We can also consider “loaded dice” in which there is a different distribution of
probabilities. For example, let
1
Pr1 (1) = Pr1 (6) =
4
1
Pr1 (2) = Pr1 (3) = Pr1 (4) = Pr1 (5) = .
8
These probabilities add up to 1, so Pr1 is a probability distribution on D. We can
assign probabilities to the elements of W = D ⇥ D by the rule

Pr11 (d, d 0 ) = Pr1 (d)Pr1 (d 0 ).

We can easily check that

 Pr11 (w) = 1,
w2W

so Pr11 is indeed a probability distribution on W . For example, we get

1 1 1
Pr11 (6, 3) = Pr1 (6)Pr1 (3) = · = .
4 8 32
Let us summarize all this with the following definition.

Definition 5.1. A finite discrete probability space (or finite discrete sample space)
is a finite set W of outcomes or elementary events w 2 W , together with a function
Pr : W ! R, called probability measure (or probability distribution) satisfying the
following properties:

0  Pr(w)  1 for all w 2 W .

 Pr(w) = 1.
w2W

The uniform probability distribution on W is the probability measure given by

Pr(w) = 1/|W | for all w 2 W . An event is any subset A of W . The probability of an
event A is defined as
Pr(A) = Â Pr(w).
w2A

Definition 5.1 immediately implies that

Pr(0)
/ =0
Pr(W ) = 1.
5.1 Sample Space, Outcomes, Events, Probability 273

The event W is called the certain event. In general there are other events A such that
Pr(A) = 1.
Remark: Even though the term probability distribution is commonly used, this is
not a good practice because there is also a notion of (cumulative) distribution func-
tion of a random variable (see Section 5.2, Definition 5.4), and this is a very different
object (the domain of the distribution function of a random variable is R, not W ).
For another example, if we consider the event

A = {HTTTT, THTTT, TTHTT, TTTHT, TTTTH}

that in flipping a coin five times, heads turns up exactly once, the probability of this
event is
5
Pr(A) = .
32
If we use the probability measure Pr on the sample space W of pairs of dice, the
probability of the event of having doubles

B = {(1, 1), (2, 2), (3, 3), (4, 4), (5, 5), (6, 6)},

is
1 1
Pr(B) = 6 ·= .
36 6
However, using the probability measure Pr11 , we obtain
1 1 1 1 1 1 3 1
Pr11 (B) = + + + + + = > .
16 64 64 64 64 16 16 6
Loading the dice makes the event “having doubles” more probable.
It should be noted that a definition slightly more general than Definition 5.1 is
needed if we want to allow W to be infinite. In this case, the following definition is
used.
Definition 5.2. A discrete probability space (or discrete sample space) is a triple
(W , F , Pr) consisting of:
1. A nonempty countably infinite set W of outcomes or elementary events.
2. The set F of all subsets of W , called the set of events.
3. A function Pr : F ! R, called probability measure (or probability distribution)
satisfying the following properties:
a. (positivity)
0  Pr(A)  1 for all A 2 F .
b. (normalization)
Pr(W ) = 1.
c. (additivity and continuity)
For any sequence of pairwise disjoint events E1 , E2 , . . . , Ei , . . . in F (which
means that Ei \ E j = 0/ for all i 6= j), we have
274 5 An Introduction to Discrete Probability
!
•
[ •
Pr Ei = Â Pr(Ei ).
i=1 i=1

The main thing to observe is that Pr is now defined directly on events, since
events may be infinite. The third axiom of a probability measure implies that

Pr(0)
/ = 0.

The notion of a discrete probability space is sufficient to deal with most problems
that a computer scientist or an engineer will ever encounter. However, there are
certain problems for which it is necessary to assume that the family F of events
is a proper subset of the power set of W . In this case, F is called the family of
measurable events, and F has certain closure properties that make it a s -algebra
(also called a s -field). Some problems even require W to be uncountably infinite. In
this case, we drop the word discrete from discrete probability space.
Remark: A s -algebra is a nonempty family F of subsets of W satisfying the fol-
lowing properties:
1. 0/ 2 F .
2. For every subset A ✓ W , if A 2 F then A 2 F .
S
3. For every countable family (Ai )i 1 of subsets Ai 2 F , we have i 1 Ai 2 F.
Note that every s -algebra is a Boolean algebra (see Section 6.11, Definition 6.14),
but the closure property (3) is very strong and adds spice to the story.
In this chapter we deal mostly with finite discrete probability spaces, and occa-
sionally with discrete probability spaces with a countably infinite sample space. In
this latter case, we always assume that F = 2W , and for notational simplicity we
omit F (that is, we write (W , Pr) instead of (W , F , Pr)).
Because events are subsets of the sample space W , they can be combined using
the set operations, union, intersection, and complementation. If the sample space
W is finite, the definition for the probability Pr(A) of an event A ✓ W given in
Definition 5.1 shows that if A, B are two disjoint events (this means that A \ B = 0),
/
then
Pr(A [ B) = Pr(A) + Pr(B).
More generally, if A1 , . . . , An are any pairwise disjoint events, then

Pr(A1 [ · · · [ An ) = Pr(A1 ) + · · · + Pr(An ).

It is natural to ask whether the probabilities Pr(A [ B), Pr(A \ B) and Pr(A) can
be expressed in terms of Pr(A) and Pr(B), for any two events A, B 2 W . In the first
and the third case, we have the following simple answer.

Proposition 5.1. Given any (finite) discrete probability space (W , Pr), for any two
events A, B ✓ W , we have
5.1 Sample Space, Outcomes, Events, Probability 275

Pr(A [ B) = Pr(A) + Pr(B) Pr(A \ B)

Pr(A) = 1 Pr(A).

Furthermore, if A ✓ B, then Pr(A)  Pr(B).

Proof. Observe that we can write A [ B as the following union of pairwise disjoint
subsets:
A [ B = (A \ B) [ (A B) [ (B A).
Then using the observation made just before Proposition 5.1, since we have the dis-
joint unions A = (A \ B) [ (A B) and B = (A \ B) [ (B A), using the disjointness
of the various subsets, we have

Pr(A [ B) = Pr(A \ B) + Pr(A B) + Pr(B A)

Pr(A) = Pr(A \ B) + Pr(A B)
Pr(B) = Pr(A \ B) + Pr(B A),

and from these we obtain

Pr(A [ B) = Pr(A) + Pr(B) Pr(A \ B).

The equation Pr(A) = 1 Pr(A) follows from the fact that A \ A = 0/ and A [ A = W ,
so
1 = Pr(W ) = Pr(A) + Pr(A).
If A ✓ B, then A \ B = A, so B = (A \ B) [ (B A) = A [ (B A), and since A and
B A are disjoint, we get

Pr(B) = Pr(A) + Pr(B A).

Since probabilities are nonegative, the above implies that Pr(A)  Pr(B). t
u

Remark: Proposition 5.1 still holds when W is infinite as a consequence of axioms

(a)–(c) of a probability measure. Also, the equation

Pr(A [ B) = Pr(A) + Pr(B) Pr(A \ B)

can be generalized to any sequence of n events. In fact, we already showed this as

the Principle of Inclusion–Exclusion, Version 2 (Theorem 4.2).
The following proposition expresses a certain form of continuity of the function
Pr.

Proposition 5.2. Given any probability space (W , F , Pr) (discrete or not), for any
sequence of events (Ai )i 1 , if Ai ✓ Ai+1 for all i 1, then
✓ • ◆
[
Pr Ai = lim Pr(An ).
n7!•
i=1
276 5 An Introduction to Discrete Probability
S•
Proof. The trick is to express i=1 Ai as a union of pairwise disjoint events. Indeed,
we have
•
[
Ai = A1 [ (A2 A1 ) [ (A3 A2 ) [ · · · [ (Ai+1 Ai ) [ · · · ,
i=1

so by property (c) of a probability measure

✓• ◆ ✓ • ◆
[ [
Pr Ai = Pr A1 [ (Ai+1 Ai )
i=1 i=1
•
= Pr(A1 ) + Â Pr(Ai+1 Ai )
i=1
n 1
= Pr(A1 ) + lim
n7!•
 Pr(Ai+1 Ai )
i=1
= lim Pr(An ),
n7!•

as claimed. t
u
We leave it as an exercise to prove that if Ai+1 ✓ Ai for all i 1, then
✓• ◆
\
Pr Ai = lim Pr(An ).
n7!•
i=1

In general, the probability Pr(A \ B) of the event A \ B cannot be expressed in a

simple way in terms of Pr(A) and Pr(B). However, in many cases we observe that
Pr(A \ B) = Pr(A)Pr(B). If this holds, we say that A and B are independent.
Definition 5.3. Given a discrete probability space (W , Pr), two events A and B are
independent if
Pr(A \ B) = Pr(A)Pr(B).
Two events are dependent if they are not independent.
For example, in the sample space of 5 coin flips, we have the events

A = {HHw | w 2 {H, T}3 } [ {HTw | w 2 {H, T}3 },

the event in which the first flip is H, and

B = {HHw | w 2 {H, T}3 } [ {THw | w 2 {H,T}3 },

the event in which the second flip is H. Since A and B contain 16 outcomes, we have
16 1
Pr(A) = Pr(B) = = .
32 2
The intersection of A and B is
5.1 Sample Space, Outcomes, Events, Probability 277

A \ B = {HHw | w 2 {H, T}3 },

the event in which the first two flips are H, and since A \ B contains 8 outcomes, we
have
8 1
Pr(A \ B) = = .
32 4
Since
1
Pr(A \ B) =
4
and
1 1 1
Pr(A)Pr(B) = · = ,
2 2 4
we see that A and B are independent events. On the other hand, if we consider the
events
A = {TTTTT, HHTTT}
and
B = {TTTTT, HTTTT},
we have
2 1
Pr(A) = Pr(B) = = ,
32 16
and since
A \ B = {TTTTT},
we have
1
Pr(A \ B) = .
32
It follows that
1 1 1
Pr(A)Pr(B) = · = ,
16 16 256
but
1
Pr(A \ B) = ,
32
so A and B are not independent.
Example 5.4. We close this section with a classical problem in probability known as
the birthday problem. Consider n < 365 individuals and assume for simplicity that
nobody was born on February 29. In this problem, the sample space is the set of
all 365n possible choices of birthdays for n individuals, and let us assume that they
are all equally likely. This is equivalent to assuming that each of the 365 days of
the year is an equally likely birthday for each individual, and that the assignments
of birthdays to distinct people are independent. Note that this does not take twins
into account! What is the probability that two (or more) individuals have the same
birthday?
To solve this problem, it is easier to compute the probability that no two individ-
uals have the same birthday. We can choose n distinct birthdays in 365 n ways, and
these can be assigned to n people in n! ways, so there are
278 5 An Introduction to Discrete Probability
✓ ◆
365
n! = 365 · 364 · · · (365 n + 1)
n

configurations where no two people have the same birthday. There are 365n possible
choices of birthdays, so the probabilty that no two people have the same birthday is
✓ ◆✓ ◆ ✓ ◆
365 · 364 · · · (365 n + 1) 1 2 n 1
q= = 1 1 ··· 1 ,
365n 365 365 365

and thus, the probability that two people have the same birthday is
✓ ◆✓ ◆ ✓ ◆
1 2 n 1
p=1 q=1 1 1 ··· 1 .
365 365 365

In the proof of Proposition 4.15, we showed that x  ex 1 for all x 2 R, so 1 x  e x

for all x 2 R, and we can bound q as follows:

n 1✓ ◆
i
q=’ 1
i=1 365
n 1
q ’e i/365
i=1

=e Âni=11 365
i

n(n 1)
e 2·365 .

If we want the probability q that no two people have the same birthday to be at most
1/2, it suffices to require
n(n 1) 1
e 2·365  ,
2
that is, n(n 1)/(2 · 365)  ln(1/2), which can be written as

n(n 1) 2 · 365 ln 2.

The roots of the quadratic equation

n2 n 2 · 365 ln 2 = 0

are p
1 ± 1 + 8 · 365 ln 2
m= ,
2
and we find that the positive root is approximately m = 23. In fact, we find that if
n = 23, then p = 50.7%. If n = 30, we calculate that p ⇡ 71%.
What if we want at least three people to share the same birthday? Then n = 88
does it, but this is harder to prove! See Ross [12], Section 3.4.
5.2 Random Variables and their Distributions 279

Next, we define what is perhaps the most important concept in probability: that
of a random variable.

5.2 Random Variables and their Distributions

In many situations, given some probability space (W , Pr), we are more interested
in the behavior of functions X : W ! R defined on the sample space W than in the
probability space itself. Such functions are traditionally called random variables, a
somewhat unfortunate terminology since these are functions. Now, given any real
number a, the inverse image of a
1
X (a) = {w 2 W | X(w) = a},

is a subset of W , thus an event, so we may consider the probability Pr(X 1 (a)),

denoted (somewhat improperly) by

Pr(X = a).

This function of a is of great interest, and in many cases it is the function that we
wish to study. Let us give a few examples.

Example 5.5. Consider the sample space of 5 coin flips, with the uniform probability
measure (every outcome has the same probability 1/32). Then the number of times
X(w) that H appears in the sequence w is a random variable. We determine that

1 5 10
Pr(X = 0) = Pr(X = 1) = Pr(X = 2) =
32 32 32
10 5 1
Pr(X = 3) = Pr(X = 4) = Pr(X = 5) = .
32 32 32
The function defined Y such that Y (w) = 1 iff H appears in w, and Y (w) = 0
otherwise, is a random variable. We have
1
Pr(Y = 0) =
32
31
Pr(Y = 1) = .
32
Example 5.6. Let W = D ⇥ D be the sample space of dice rolls, with the uniform
probability measure Pr (every outcome has the same probability 1/36). The sum
S(w) of the numbers on the two dice is a random variable. For example,

S(2, 5) = 7.

The value of S is any integer between 2 and 12, and if we compute Pr(S = s) for
s = 2, . . . , 12, we find the following table.
280 5 An Introduction to Discrete Probability

s 2 3 4 5 6 7 8 9 10 11 12
1 2 3 4 5 6 5 4 3 2 1
Pr(S = s) 36 36 36 36 36 36 36 36 36 36 36

Here is a “real” example from computer science.

Example 5.7. Our goal is to sort of a sequence S = (x1 , . . . , xn ) of n distinct real

numbers in increasing order. We use a recursive method known as quicksort which
proceeds as follows:
1. If S has one or zero elements return S.
2. Pick some element x = xi in S called the pivot.
3. Reorder S in such a way that for every number x j 6= x in S, if x j < x, then x j is
moved to a list S1 , else if x j > x then x j is moved to a list S2 .
4. Apply this algorithm recursively to the list of elements in S1 and to the list of
elements in S2 .
5. Return the sorted list S1 , x, S2 .
Let us run the algorithm on the input list

S = (1, 5, 9, 2, 3, 8, 7, 14, 12, 10).

We can represent the choice of pivots and the steps of the algorithm by an ordered
binary tree as shown in Figure 5.3. Except for the root node, every node corresponds

(1, 5, 9, 2, 3, 8, 7, 14, 12, 10)

1/10

(1, 2) 3 (5, 9, 8, 7, 14, 12, 10)

1/2 1/7

() 1 (2) (5, 7) 8 (9, 14, 12, 10)

1/2 1/4

(5) 7 () (9) 10 (14, 12)

1/2

() 12 (14)

Fig. 5.3 A tree representation of a run of quicksort.

to the choice of a pivot, say x. The list S1 is shown as a label on the left of node x,
5.2 Random Variables and their Distributions 281

and the list S2 is shown as a label on the right of node x. A leaf node is a node such
that |S1 |  1 and |S2 |  1. If |S1 | 2, then x has a left child, and if |S2 | 2, then x
has a right child. Let us call such a tree a computation tree. Observe that except for
minor cosmetic differences, it is a binary search tree. The sorted list can be retrieved
by a suitable traversal of the computation tree and is

(1, 2, 3, 5, 7, 8, 9, 10, 12, 14).

If you run this algorithm on a few more examples, you will realize that the choice
of pivots greatly influences how many comparisons are needed. If the pivot is chosen
at each step so that the size of the lists S1 and S2 is roughly the same, then the number
of comparisons is small compared to n, in fact O(n ln n). On the other hand, with a
poor choice of pivot, the number of comparisons can be as bad as n(n 1)/2.
In order to have a good “average performance,” one can randomize this algorithm
by assuming that each pivot is chosen at random. What this means is that whenever
it is necessary to pick a pivot from some list Y , some procedure is called and this
procedure returns some element chosen at random from Y . How exactly this done is
an interesting topic in itself but we will not go into this. Let us just say that the pivot
can be produced by a random number generator, or by spinning a wheel containing
the numbers in Y on it, or by rolling a dice with as many faces as the numbers in Y .
What we do assume is that the probability measure that a number is chosen from a
list Y is uniform, and that successive choices of pivots are independent. How do we
model this as a probability space?
Here is a way to do it. Use the computation trees defined above! Simply add
to every edge the probability that one of the element of the corresponding list, say
Y , was chosen uniformly, namely 1/|Y |. So given an input list S of length n, the
sample space W is the set of all computation trees T with root label S. We assign
a probability to the trees T in W as follows: If n = 0, 1, then there is a single tree
and its probability is 1. If n 2, for every leaf of T , multiply the probabilities along
the path from the root to that leaf and then add up the probabilities assigned to
these leaves. This is Pr(T ). We leave it as an exercise to prove that the sum of the
probabilities of all the trees in W is equal to 1.
A random variable of great interest on (W , Pr) is the number X of comparisons
performed by the algorithm. To analyze the average running time of this algorithm,
it is necessary to determine when the first (or the last) element of a sequence

Y = (yi , . . . , y j )

is chosen as a pivot. To carry out the analysis further requires the notion of expecta-
tion that has not yet been defined. See Example 5.23 for a complete analysis.

Let us now give an official definition of a random variable.

Definition 5.4. Given a (finite) discrete probability space (W , Pr), a random vari-
able is any function X : W ! R. For any real number a 2 R, we define Pr(X = a)
as the probability
282 5 An Introduction to Discrete Probability

1
Pr(X = a) = Pr(X (a)) = Pr({w 2 W | X(w) = a}),

and Pr(X  a) as the probability

1
Pr(X  a) = Pr(X (( •, a])) = Pr({w 2 W | X(w)  a}).

The function f : R ! [0, 1] given by

f (a) = Pr(X = a), a2R

is the probability mass function of X, and the function F : R ! [0, 1] given by

F(a) = Pr(X  a), a2R

is the cumulative distribution function of X.

The term probability mass function is abbreviated as p.m.f , and cumulative dis-
tribution function is abbreviated as c.d.f . It is unfortunate and confusing that both
the probability mass function and the cumulative distribution function are often ab-
breviated as distribution function.
The probability mass function f for the sum S of the numbers on two dice from
Example 5.6 is shown in Figure 5.4, and the corresponding cumulative distribution
function F is shown in Figure 5.5.

Fig. 5.4 The probability mass function for the sum of the numbers on two dice.

If W is finite, then f only takes finitely many nonzero values; it is very discontin-
uous! The c.d.f F of S shown in Figure 5.5 has jumps (steps). Observe that the size
of the jump at every value a is equal to f (a) = Pr(S = a).
The cumulative distribution function F has the following properties:
1. We have
lim F(x) = 0, lim F(x) = 1.
x7! • x7!•

2. It is monotonic nondecreasing, which means that if a  b, then F(a)  F(b).

5.2 Random Variables and their Distributions 283

F (a)

1.0

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0 a
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Fig. 5.5 The cumulative distribution function for the sum of the numbers on two dice.

3. It is piecewise constant with jumps, but it is right-continuous, which means that

limh>0,h7!0 F(a + h) = F(a).
For any a 2 R, because F is nondecreasing, we can define F(a ) by

F(a ) = lim F(a h) = lim F(a h).

h#0 h>0,h7!0

These properties are clearly illustrated by the c.d.f on Figure 5.5.

The functions f and F determine each other, because given the probability mass
function f , the function F is defined by

F(a) = Â f (x),
xa

and given the cumulative distribution function F, the function f is defined by

f (a) = F(a) F(a ).

If the sample space W is countably infinite, then f and F are still defined as above
but in
F(a) = Â f (x),
xa

the expression on the righthand side is the limit of an infinite sum (of positive terms).
284 5 An Introduction to Discrete Probability

Remark: If W is not countably infinite, then we are dealing with a probability

space (W , F , Pr) where F may be a proper subset of 2W , and in Definition 5.4,
we need the extra condition that a random variable is a function X : W ! R such
that X 1 (a) 2 F for all a 2 R. (The function X needs to be F -measurable.) In this
more general situation, it is still true that

f (a) = Pr(X = a) = F(a) F(a ),

but F cannot generally be recovered from f . If the c.d.f F of a random variable X

can be expressed as Z x
F(x) = f (y)dy,
•
for some nonnegative (Lebesgue) integrable function f , then we say that F and X
are absolutely continuous (please, don’t ask me what type of integral!). The function
f is called a probability density function of X (for short, p.d.f ).
In this case, F is continuous, but more is true. The function F is uniformly con-
tinuous, and it is differentiable almost everywhere, which means that the set of input
values for which it is not differentiable is a set of (Lebesgue) measure zero. Further-
more, F 0 = f almost everywhere.
Random variables whose distributions can be expressed as above in terms of a
density function are often called continuous random variables. In contrast with the
discrete case, if X is a continuous random variable, then

Pr(X = x) = 0 for all x 2 R.

As a consequence, some of the definitions given in the discrete case in terms of the
probabilities Pr(X = x), for example Definition 5.7, become trivial. These defini-
tions need to be modifed; replacing Pr(X = x) by Pr(X  x) usually works.
In the general case where the cdf F of a random variable X has discontinuities,
we say that X is a discrete random variable if X(w) 6= 0 for at most countably many
w 2 W . Equivalently, the image of X is finite or countably infinite. In this case, the
mass function of X is well defined, and it can be viewed as a discrete version of a
density function.
In the discrete setting where the sample space W is finite, it is usually more
convenient to use the probability mass function f , and to abuse language and call it
the distribution of X.
Example 5.8. Suppose we flip a coin n times, but this time, the coin is not necessarily
fair, so the probability of landing heads is p and the probability of landing tails
is 1 p. The sample space W is the set of strings of length n over the alphabet
{H, T}. Assume that the coin flips are independent, so that the probability of an
event w 2 W is obtained by replacing H by p and T by 1 p in w. Then let X be the
random variable defined such that X(w) is the number of heads in w. For any i with
0  i  n, since there are ni subsets with i elements, and since the probability of a
sequence w with i occurrences of H is pi (1 p)n i , we see that the distribution of
X (mass function) is given by
5.2 Random Variables and their Distributions 285
✓ ◆
n i
f (i) = p (1 p)n i , i = 0, . . . , n,
i

and 0 otherwise. This is an example of a binomial distribution.

Example 5.9. As in Example 5.8, assume that we flip a biased coin, where the prob-
ability of landing heads is p and the probability of landing tails is 1 p. However,
this time, we flip our coin any finite number of times (not a fixed number), and we
are interested in the event that heads first turns up. The sample space W is the infinite
set of strings over the alphabet {H, T} of the form

W = {H, TH, TTH, . . . , Tn H, . . . , }.

Assume that the coin flips are independent, so that the probability of an event w 2 W
is obtained by replacing H by p and T by 1 p in w. Then let X be the random
variable defined such that X(w) = n iff |w| = n, else 0. In other words, X is the
number of trials until we obtain a success. Then it is clear that

f (n) = (1 p)n 1
p, n 1.

and 0 otherwise. This is an example of a geometric distribution.

The process in which we flip a coin n times is an example of a process in which

we perform n independent trials, each of which results in success or failure (such
trials that result exactly two outcomes, success or failure, are known as Bernoulli tri-
als). Such processes are named after Jacob Bernoulli, a very significant contributor
to probability theory after Fermat and Pascal.

Fig. 5.6 Jacob (Jacques) Bernoulli (1654–1705).

Example 5.10. Let us go back to Example 5.8, but assume that n is large and that
the probability p of success is small, which means that we can write np = l with l
of “moderate” size. Let us show that we can approximate the distribution f of X in
an interesting way. Indeed, for every nonnegative integer i, we can write
286 5 An Introduction to Discrete Probability
✓ ◆
n i
f (i) = p (1 p)n i
i
✓ ◆i ✓ ◆n i
n! l l
= 1
i!(n i)! n n
✓ ◆n ✓ ◆ i
n(n 1) · · · (n i + 1) l i l l
= 1 1 .
ni i! n n

Now for n large and l moderate, we have

✓ ◆n ✓ ◆ i
l l l n(n 1) · · · (n i + 1)
1 ⇡e 1 ⇡1 ⇡ 1,
n n ni

so we obtain
li l
f (i) ⇡ e
, i 2 N.
i!
The above is called a Poisson distribution with parameter l . It is named after the
French mathematician Simeon Denis Poisson.

Fig. 5.7 Siméon Denis Poisson (1781–1840).

It turns out that quite a few random variables occurring in real life obey the
Poisson probability law (by this, we mean that their distribution is the Poisson dis-
tribution). Here are a few examples:
1. The number of misprints on a page (or a group of pages) in a book.
2. The number of people in a community whose age is over a hundred.
3. The number of wrong telephone numbers that are dialed in a day.
4. The number of customers entering a post office each day.
5. The number of vacancies occurring in a year in the federal judicial system.
As we will see later on, the Poisson distribution has some nice mathematical
properties, and the so-called Poisson paradigm which consists in approximating the
distribution of some process by a Poisson distribution is quite useful.
5.3 Conditional Probability and Independence 287

5.3 Conditional Probability and Independence

In general, the occurrence of some event B changes the probability that another
event A occurs. It is then natural to consider the probability denoted Pr(A | B) that
if an event B occurs, then A occurs. As in logic, if B does not occur not much can be
said, so we assume that Pr(B) 6= 0.

Definition 5.5. Given a discrete probability space (W , Pr), for any two events A and
B, if Pr(B) 6= 0, then we define the conditional probability Pr(A | B) that A occurs
given that B occurs as
Pr(A \ B)
Pr(A | B) = .
Pr(B)

Example 5.11. Suppose we roll two fair dice. What is the conditional probability
that the sum of the numbers on the dice exceeds 6, given that the first shows 3? To
solve this problem, let
B = {(3, j) | 1  j  6}
be the event that the first dice shows 3, and

A = {(i, j) | i + j 7, 1  i, j  6}

be the event that the total exceeds 6. We have

A \ B = {(3, 4), (3, 5), (3, 6)},

so we get
Pr(A \ B) 3 6 1
Pr(A | B) = = = .
Pr(B) 36 36 2
The next example is perhaps a little more surprising.
Example 5.12. A family has two children. What is the probability that both are boys,
given at least one is a boy?
There are four possible combinations of sexes, so the sample space is

W = {GG, GB, BG, BB},

and we assume a uniform probability measure (each outcome has probability 1/4).
Introduce the events
B = {GB, BG, BB}
of having at least one boy, and
A = {BB}
of having two boys. We get
A \ B = {BB},
and so
288 5 An Introduction to Discrete Probability

Pr(A \ B) 1 3 1
Pr(A | B) = = = .
Pr(B) 4 4 3
Contrary to the popular belief that Pr(A | B) = 1/2, it is actually equal to 1/3. Now,
consider the question: what is the probability that both are boys given that the first
child is a boy? The answer to this question is indeed 1/2.
The next example is known as the “Monty Hall Problem,” a standard example of
every introduction to probability theory.

Example 5.13. On the television game Let’s Make a Deal, a contestant is presented
with a choice of three (closed) doors. Behind exactly one door is a terrific prize. The
other doors conceal cheap items. First, the contestant is asked to choose a door. Then
the host of the show (Monty Hall) shows the contestant one of the worthless prizes
behind one of the other doors. At this point, there are two closed doors, and the
contestant is given the opportunity to switch from his original choice to the other
closed door. The question is, is it better for the contestant to stick to his original
choice or to switch doors?
We can analyze this problem using conditional probabilities. Without loss of gen-
erality, assume that the contestant chooses door 1. If the prize is actually behind door
1, then the host will show door 2 or door 3 with equal probability 1/2. However, if
the prize is behind door 2, then the host will open door 3 with probability 1, and if
the prize is behind door 3, then the host will open door 2 with probability 1. Write
Pi for “the prize is behind door i,” with i = 1, 2, 3, and D j for “the host opens door
D j, ” for j = 2, 3. Here it is not necessary to consider the choice D1 since a sensible
host will never open door 1. We can represent the sequences of choices occurring in
the game by a tree known as probability tree or tree of possibilities, shown in Figure
5.8.
Every leaf corresponds to a path associated with an outcome, so the sample space
is
W = {P1; D2, P1; D3, P2; D3, P3; D2}.
The probability of an outcome is obtained by multiplying the probabilities along the
corresponding path, so we have
1 1 1 1
Pr(P1; D2) = Pr(P1; D3) = Pr(P2; D3) = Pr(P3; D2) = .
6 6 3 3
Suppose that the host reveals door 2. What should the contestant do?
The events of interest are:
1. The prize is behind door 1; that is, A = {P1; D2, P1; D3}.
2. The prize is behind door 3; that is, B = {P3; D2}.
3. The host reveals door 2; that is, C = {P1; D2, P3; D2}.
Whether or not the contestant should switch doors depends on the values of the
conditional probabilities
1. Pr(A | C): the prize is behind door 1, given that the host reveals door 2.
5.3 Conditional Probability and Independence 289

1/2 D2 Pr(P 1; D2) = 1/6

P1

1/2 D3 Pr(P 1; D3) = 1/6

1/3

1/3 1
P2 D3 Pr(P 2; D3) = 1/3

1/3
1
P3 D2 Pr(P 3; D2) = 1/3

Fig. 5.8 The tree of possibilities in the Monty Hall problem.

2. Pr(B | C): the prize is behind door 3, given that the host reveals door 2.
We have A \C = {P1; D2}, so

Pr(A \C) = 1/6,

and
1 1 1
Pr(C) = Pr({P1; D2, P3; D2}) = + = ,
6 3 2
so
Pr(A \C) 1 1 1
Pr(A | C) = = = .
Pr(C) 6 2 3
We also have B \C = {P3; D2}, so

Pr(B \C) = 1/3,

and
Pr(B \C) 1 1 2
Pr(B | C) = = = .
Pr(C) 3 2 3
Since 2/3 > 1/3, the contestant has a greater chance (twice as big) to win the bigger
prize by switching doors. The same probabilities are derived if the host had revealed
door 3.
A careful analysis showed that the contestant has a greater chance (twice as large)
of winning big if she/he decides to switch doors. Most people say “on intuition” that
it is preferable to stick to the original choice, because once one door is revealed,
the probability that the valuable prize is behind either of two remaining doors is
290 5 An Introduction to Discrete Probability

1/2. This is incorrect because the door the host opens depends on which door the
contestant orginally chose.
Let us conclude by stressing that probability trees (trees of possibilities) are very
useful in analyzing problems in which sequences of choices involving various prob-
abilities are made.

The next proposition shows various useful formulae due to Bayes.

Proposition 5.3. (Bayes’ Rules) For any two events A, B with Pr(A) > 0 and Pr(B) >
0, we have the following formulae:
1. (Bayes’ rule of retrodiction)

Pr(A | B)Pr(B)
Pr(B | A) = .
Pr(A)

2. (Bayes’ rule of exclusive and exhaustive clauses) If we also have Pr(A) < 1 and
Pr(B) < 1, then

Pr(A) = Pr(A | B)Pr(B) + Pr(A | B)Pr(B).

More generally, if B1 , . . . , Bn form a partition of W with Pr(Bi ) > 0 (n 2), then

n
Pr(A) = Â Pr(A | Bi )Pr(Bi ).
i=1

3. (Bayes’ sequential formula) For any sequence of events A1 , . . . , An , we have

! !
n
\ n\1
Pr Ai = Pr(A1 )Pr(A2 | A1 )Pr(A3 | A1 \ A2 ) · · · Pr An | Ai .
i=1 i=1

4. (Bayes’ law)

Pr(A | B)Pr(B)
Pr(B | A) = .
Pr(A | B)Pr(B) + Pr(A | B)Pr(B)

Proof. By definition of a conditional probability we have

Pr(A | B)Pr(B) = Pr(A \ B) = Pr(B \ A) = Pr(B | A)Pr(A),

which shows the first formula. For the second formula, observe that we have the
disjoint union
A = (A \ B) [ (A \ B),
so

Pr(A) = Pr(A \ B) [ Pr(A \ B)

= Pr(A | B)Pr(B) [ Pr(A | B)Pr(B).
5.3 Conditional Probability and Independence 291

We leave the more general rule as an exercise, and the third rule follows by unfolding
definitions. The fourth rule is obtained by combining (1) and (2). t u

Bayes’ rule of retrodiction is at the heart of the so-called Bayesian framework. In

this framework, one thinks of B as an event describing some state (such as having
a certain disease) and of A an an event describing some measurement or test (such
as having high blood pressure). One wishes to infer the a posteriori probability
Pr(B | A) of the state B given the test A, in terms of the prior probability Pr(B) and
the likelihood function Pr(A | B). The likelihood function Pr(A | B) is a measure of
the likelihood of the test A given that we know the state B, and Pr(B) is a measure
of our prior knowledge about the state; for example, having a certain disease. The
probability Pr(A) is usually obtained using Bayes’s second rule because we also
know Pr(A | B).

Example 5.14. Doctors apply a medical test for a certain rare disease that has the
property that if the patient is affected by the disease, then the test is positive in 99%
of the cases. However, it happens in 2% of the cases that a healthy patient tests
positive. Statistical data shows that one person out of 1000 has the disease. What is
the probability for a patient with a positive test to be affected by the disease?
Let S be the event that the patient has the disease, and + and the events that
the test is positive or negative. We know that

Pr(S) = 0.001
Pr(+ | S) = 0.99
Pr(+ | S) = 0.02,

and we have to compute Pr(S | +). We use the rule

Pr(+ | S)Pr(S)
Pr(S | +) = .
Pr(+)

We also have
Pr(+) = Pr(+ | S)Pr(S) + Pr(+ | S)Pr(S),
so we obtain
0.99 ⇥ 0.001 1
Pr(S | +) = ⇡ = 5%.
0.99 ⇥ 0.001 + 0.02 ⇥ 0.999 20
Since this probability is small, one is led to question the reliability of the test! The
solution is to apply a better test, but only to all positive patients. Only a small portion
of the population will be given that second test because

Pr(+) = 0.99 ⇥ 0.001 + 0.02 ⇥ 0.999 ⇡ 0.003.

Redo the calculations with the new data

292 5 An Introduction to Discrete Probability

Pr(S) = 0.00001
Pr(+ | S) = 0.99
Pr(+ | S) = 0.01.

You will find that the probability Pr(S | +) is approximately 0.000099, so the chance
of being sick is rather small, and it is more likely that the test was incorrect.
Recall that in Definition 5.3, we defined two events as being independent if

Pr(A \ B) = Pr(A)Pr(B).

Asuming that Pr(A) 6= 0 and Pr(B) 6= 0, we have

Pr(A \ B) = Pr(A | B)Pr(B) = Pr(B | A)Pr(A),

so we get the following proposition.

Proposition 5.4. For any two events A, B such that Pr(A) 6= 0 and Pr(B) 6= 0, the
following statements are equivalent:
1. Pr(A \ B) = Pr(A)Pr(B); that is, A and B are independent.
2. Pr(B | A) = Pr(B).
3. Pr(A | B) = Pr(A).

Remark: For a fixed event B with Pr(B) > 0, the function A 7! Pr(A | B) satisfies
the axioms of a probability measure stated in Definition 5.2. This is shown in Ross
[11] (Section 3.5), among other references.
The examples where we flip a coin n times or roll two dice n times are examples
of indendent repeated trials. They suggest the following definition.
Definition 5.6. Given two discrete probability spaces (W1 , Pr1 ) and (W2 , Pr2 ), we
define their product space as the probability space (W1 ⇥ W2 , Pr), where Pr is given
by
Pr(w1 , w2 ) = Pr1 (w1 )Pr2 (W2 ), w1 2 W1 , w2 2 W2 .
There is an obvious generalization for n discrete probability spaces. In particular, for
any discrete probability space (W , Pr) and any integer n 1, we define the product
space (W n , Pr), with

Pr(w1 , . . . , wn ) = Pr(w1 ) · · · Pr(wn ), wi 2 W , i = 1, . . . , n.

The fact that the probability measure on the product space is defined as a prod-
uct of the probability measures of its components captures the independence of the
trials.
Remark: The product of two probability spaces (W1 , F1 , Pr1 ) and (W2 , F2 , Pr2 )
can also be defined, but F1 ⇥ F2 is not a s -algebra in general, so some serious
work needs to be done.
5.3 Conditional Probability and Independence 293

The notion of independence also applies to random variables. Given two random
variables X and Y on the same (discrete) probability space, it is useful to consider
their joint distribution (really joint mass function) fX,Y given by

fX,Y (a, b) = Pr(X = a and Y = b) = Pr({w 2 W | (X(w) = a) ^ (Y (w) = b)}),

for any two reals a, b 2 R.

Definition 5.7. Two random variables X and Y defined on the same discrete proba-
bility space are independent if

Pr(X = a and Y = b) = Pr(X = a)Pr(Y = b), for all a, b 2 R.

Remark: If X and Y are two continuous random variables, we say that X and Y are
independent if

Pr(X  a and Y  b) = Pr(X  a)Pr(Y  b), for all a, b 2 R.

It is easy to verify that if X and Y are discrete random variables, then the above
condition is equivalent to the condition of Definition 5.7.

Example 5.15. If we consider the probability space of Example 5.2 (rolling two
dice), then we can define two random variables S1 and S2 , where S1 is the value
on the first dice and S2 is the value on the second dice. Then the total of the two
values is the random variable S = S1 + S2 of Example 5.6. Since
1 1 1
Pr(S1 = a and S2 = b) = = · = Pr(S1 = a)Pr(S2 = b),
36 6 6
the random variables S1 and S2 are independent.

Example 5.16. Suppose we flip a biased coin (with probability p of success) once.
Let X be the number of heads observed and let Y be the number of tails observed.
The variables X and Y are not independent. For example

Pr(X = 1 and Y = 1) = 0,

yet
Pr(X = 1)Pr(Y = 1) = p(1 p).
Now, if we flip the coin N times, where N has the Poisson distribution with parame-
ter l , it is remarkable that X and Y are independent; see Grimmett and Stirzaker [6]
(Section 3.2).

The following characterization of independence for two random variables is left

as an exercise.

Proposition 5.5. If X and Y are two random variables on a discrete probability

space (W , Pr) and if fX,Y is the joint distribution (mass function) of X and Y , fX is
294 5 An Introduction to Discrete Probability

the distribution (mass function) of X and fY is the distribution (mass function) of Y ,

then X and Y are independent iff

fX,Y (x, y) = fX (x) fY (y) for all x, y 2 R.

Given the joint mass function fX,Y of two random variables X and Y , the mass
functions fX of X and fY of Y are called marginal mass functions, and they are
obtained from fX,Y by the formulae

fX (x) = Â fX,Y (x, y), fY (y) = Â fX,Y (x, y).

y x

Remark: To deal with the continuous case, it is useful to consider the joint distri-
bution FX,Y of X and Y given by

FX,Y (a, b) = Pr(X  a and Y  b) = Pr({w 2 W | (X(w)  a) ^ (Y (w)  b)}),

for any two reals a, b 2 R. We say that X and Y are jointly continuous with joint
density function fX,Y if
Z x Z y
FX,Y (x, y) = fX,Y (u, v)du dv, for all x, y 2 R
• •

for some nonnegative integrable function fX,Y . The marginal density functions fX
of X and fY of Y are defined by
Z • Z •
fX (x) = fX,Y (x, y)dy, fY (y) = fX,Y (x, y)dx.
• •

They correspond to the marginal distribution functions FX of X and FY of Y given

by

FX (x) = Pr(X  x) = FX,Y (x, •), FY (y) = Pr(Y  y) = FX,Y (•, y).

For example, if X and Y are two random variables with joint density function
fX,Y given by
1 x
fX,Y (x, y) = e y y , 0 < x, y < •,
y
then the marginal density function fY of Y is given by
Z +• Z •
1 y xy
fY (y) = fX,Y (x, y) dx = e dx = e y , y > 0.
• 0 y
It can be shown that X and Y are independent iff

FX,Y (x, y) = FX (x)FY (y) for all x, y 2 R,

which, for continuous variables, is equivalent to

5.4 Expectation of a Random Variable 295

fX,Y (x, y) = fX (x) fY (y) for all x, y 2 R.

We now turn to one of the most important concepts about random variables, their
mean (or expectation).

5.4 Expectation of a Random Variable

In order to understand the behavior of a random variable, we may want to look at

its “average” value. But the notion of average in ambiguous, as there are different
kinds of averages that we might want to consider. Among these, we have
1. the mean: the sum of the values divided by the number of values.
2. the median: the middle value (numerically).
3. the mode: the value that occurs most often.
For example, the mean of the sequence (3, 1, 4, 1, 5) is 2.8; the median is 3, and the
mode is 1.
Given a random variable X, if we consider a sequence of values X(w1 ), X(w2 ), . . .,
X(wn ), each value X(w j ) = a j has a certain probability Pr(X = a j ) of occurring
which may differ depending on j, so the usual mean

X(w1 ) + X(w2 ) + · · · + X(wn ) a1 + · · · + an

=
n n
may not capture well the “average” of the random variable X. A better solution is to
use a weighted average, where the weights are probabilities. If we write a j = X(w j ),
we can define the mean of X as the quantity

a1 Pr(X = a1 ) + a2 Pr(X = a2 ) + · · · + an Pr(X = an ).

Definition 5.8. Given a finite discrete probability space (W , Pr), for any random
variable X, the mean value or expected value or expectation1 of X is the number
E(X) defined as

E(X) = Â x · Pr(X = x) = Â x f (x),

x2X(W ) x| f (x)>0

where X(W ) denotes the image of the function X and where f is the probability
mass function of X. Because W is finite, we can also write

E(X) = Â X(w)Pr(w).
w2W

1 It is amusing that in French, the word for expectation is espérance mathématique. There is hope
for mathematics!
296 5 An Introduction to Discrete Probability

In this setting, the median of X is defined as the set of elements x 2 X(W ) such
that
1 1
Pr(X  x) and Pr(X x) .
2 2

Remark: If W is countably infinite, then the expectation E(X), if it exists, is given

by
E(X) = Â x f (x),
x| f (x)>0

provided that the above sum converges absolutely (that is, the partial sums of abso-
lute values converge). If we have a probability space (X, F , Pr) with W uncountable
and if X is absolutely continuous so that it has a density function f , then the expec-
tation of X is given by the integral
Z +•
E(X) = x f (x)dx.
•

It is even possible to define the expectation of a random variable that is not neces-
sarily absolutely continuous using its cumulative density function F as
Z +•
E(X) = xdF(x),
•

where the above integral is the Lebesgue–Stieljes integal, but this is way beyond the
scope of this book.
Observe that if X is a constant random variable (that is, X(w) = c for all w 2 W
for some constant c), then

E(X) =  X(w)Pr(w) = c  Pr(w) = cPr(W ) = c,

w2W w2W

since Pr(W ) = 1. The mean of a constant random variable is itself (as it should be!).

Example 5.17. Consider the sum S of the values on the dice from Example 5.6. The
expectation of S is
1 2 5 6 5 1
E(S) = 2 · +3· +···+6· +7· +8· + · · · + 12 · = 7.
36 36 36 36 36 36
Example 5.18. Suppose we flip a biased coin once (with probability p of landing
heads). If X is the random variable given by X(H) = 1 and X(T) = 0, the expectation
of X is

E(X) = 1 · Pr(X = 1) + 0 · Pr(X = 0) = 1 · p + 0 · (1 p) = p.

Example 5.19. Consider the binomial distribution of Example 5.8, where the ran-
dom variable X counts the number of heads (success) in a sequence of n trials. Let
us compute E(X). Since the mass function is given by
5.4 Expectation of a Random Variable 297
✓ ◆
n i
f (i) = p (1 p)n i , i = 0, . . . , n,
i

we have
n n ✓ ◆
n i
E(X) = Â i f (i) = Â i p (1 p)n i .
i=0 i=0 i
We use a trick from analysis to compute this sum. Recall from the binomial theorem
that
n ✓ ◆
n i
(1 + x)n = Â x.
i=0 i

If we take derivatives on both sides, we get

n ✓ ◆
n i 1
n(1 + x) n 1
= Âi x ,
i=0 i

and by multiplying both sides by x,

✓ ◆
n
n i
nx(1 + x) n 1
= Âi x. (⇤)
i=0 i

Let q = 1 p. Now if we set x = p/q, since p + q = 1, we have

✓ ◆ ✓ ◆n
n 1 p p n 1
p p+q 1
p
x(1 + x) = 1+ = = ,
q q q q qn

so (⇤) becomes
n ✓ ◆ i
np n p
= Â i i qi ,
qn i=0
and multiplying both sides by qn and using the fact that q = 1 p, we get
n ✓ ◆
n
 i i pi (1 p)n i = np,
i=0

and so
E(X) = np.
It should be observed that the expectation of a random variable may be infinite.
For example, if X is a random variable whose probability mass function f is given
by
1
f (k) = , k = 1, 2, . . . ,
k(k + 1)
then Âk2N {0} f (k) = 1, since
• • ✓ ◆ ✓ ◆
1 1 1 1
 k(k + 1)  k
=
k+1
= lim 1
k7!• k+1
= 1,
k=1 k=1
298 5 An Introduction to Discrete Probability

but
1
E(X) = Â k f (k) = Â k + 1
= •.
k2N {0} k2N {0}

Example 5.19 illustrates the fact that computing the expectation of a random
variable X can be quite difficult due the complicated nature of the mass function f .
Therefore it is desirable to know about properties of the expectation that make its
computation simpler. A crucial property of expectation that often allows simplifica-
tions in computing the expectation of a random variable is its linearity.

Proposition 5.6. (Linearity of Expectation) Given two random variables on a dis-

crete probability space, for any real number l , we have

E(X +Y ) = E(X) + E(Y )

E(l X) = l E(X).

Proof. We have

E(X +Y ) = Â z · Pr(X +Y = z)
z
= Â Â(x + y) · Pr(X = x and Y = y)
x y

= Â Â x · Pr(X = x and Y = y) + Â Â y · Pr(X = x and Y = y)

x y x y

= Â Â x · Pr(X = x and Y = y) + Â Â y · Pr(X = x and Y = y)

x y y x

=  x  Pr(X = x and Y = y) +  y  Pr(X = x and Y = y).

x y y x

Now the events Ax = {x | X = x} form a partition of W , which implies that

 Pr(X = x and Y = y) = Pr(X = x).

y

Similarly the events By = {y | Y = y} form a partition of W , which implies that

 Pr(X = x and Y = y) = Pr(Y = y).

x

By substitution, we obtain

E(X +Y ) = Â x · Pr(X = x) + Â y · Pr(Y = y),

x y

proving that E(X + Y ) = E(X) + E(Y ). When W is countably infinite, we can per-
mute the indices x and y due to absolute convergence.
For the second equation, if l 6= 0, we have
5.4 Expectation of a Random Variable 299

E(l X) = Â x · Pr(l X = x)
x
x
=l · Pr(X = x/l )
x l
= l  y · Pr(X = y)
y

= l E(X).

as claimed. If l = 0, the equation is trivial. t

u
By a trivial induction, we obtain that for any finite number of random variables
X1 , . . . , Xn , we have ✓ ◆
n n
E Â Xi = Â E(Xi ).
I=1 I=1

It is also important to realize that the above equation holds even if the Xi are not
independent.
Here is an example showing how the linearity of expectation can simplify calcu-
lations. Let us go back to Example 5.19. Define n random variables X1 , . . . , Xn such
that Xi (w) = 1 iff the ith flip yields heads, otherwise Xi (w) = 0. Clearly, the number
X of heads in the sequence is

X = X1 + · · · + Xn .

However, we saw in Example 5.18 that E(Xi ) = p, and since

E(X) = E(X1 ) + · · · + E(Xn ),

we get
E(X) = np.
The above example suggests the definition of indicator function, which turns out
to be quite handy.
Definition 5.9. Given a discrete probability space with sample space W , for any
event A, the indicator function (or indicator variable) of A is the random variable IA
defined such that n
1 if w 2 A
IA (w) =
0 if w 2 / A.
Here is the main property of the indicator function.
Proposition 5.7. The expectation E(IA ) of the indicator function IA is equal to the
probabilty Pr(A) of the event A.
Proof. We have

E(IA ) = Â IA (w)Pr(w) = Â Pr(w) = Pr(A),

w2W w2A

as claimed t
u
300 5 An Introduction to Discrete Probability

This fact with the linearity of expectation is often used to compute the expectation
of a random variable, by expressing it as a sum of indicator variables. We will see
how this method is used to compute the expectation of the number of comparisons in
quicksort. But first, we use this method to find the expected number of fixed points
of a random permutation.

Example 5.20. For any integer n 1, let W be the set of all n! permutations of
{1, . . . , n}, and give W the uniform probabilty measure; that is, for every permutation
p, let
1
Pr(p) = .
n!
We say that these are random permutations. A fixed point of a permutation p is any
integer k such that p(k) = k. Let X be the random variable such that X(p) is the
number of fixed points of the permutation p. Let us find the expectation of X. To do
this, for every k, let Xk be the random variable defined so that Xk (p) = 1 iff p(k) = k,
and 0 otherwise. Clearly,
X = X1 + · · · + Xn ,
and since
E(X) = E(X1 ) + · · · + E(Xn ),
we just have to compute E(Xk ). But, Xk is an indicator variable, so

E(Xk ) = Pr(Xk = 1).

Now there are (n 1)! permutations that leave k fixed, so Pr(X = 1) = 1/n. There-
fore,
1
E(X) = E(X1 ) + · · · + E(Xn ) = n · = 1.
n
On average, a random permutation has one fixed point.

Definition 5.10. If X is a random variable on a discrete probability space W (pos-

sibly countably infinite), for any function g : R ! R, the composition g X is a
random variable defined by

(g X)(w) = g(X(w)), w 2 W.

This random variable is usually denoted by g(X).

Given two random variables X and Y , if j and y are two functions, we leave it
as an exercise to prove that if X and Y are independent, then so are j(X) and y(Y ).
Although computing the mass function of g in terms of the mass function f of X
can be very difficult, there is a nice way to compute its expectation. Here is a second
tool that makes it easier to compute an expectation.

Proposition 5.8. If X is a random variable on a discrete probability space W , for

any function g : R ! R, the expectation E(g(X)) of g(X) (if it exists) is given by
5.4 Expectation of a Random Variable 301

E(g(X)) = Â g(x) f (x),

x

where f is the mass function of X.

Proof. We have

E(g(X)) = Â y · Pr(g X = y)
y

= Â y · Pr({w 2 W | g(X(w)) = y})

y

=  y  Pr({w 2 W | g(x) = y, X(w) = x})

y x

=Â Â y · Pr({w 2 W , | X(w) = x})

y x,g(x)=y

=Â Â g(x) · Pr(X = x)
y x,g(x)=y

= Â g(x) · Pr(X = x)
x
= Â g(x) f (x),
x

as claimed. t
u

The cases g(X) = X k , g(X) = zX , and g(X) = etX (for some given reals z and t)
are of particular interest.
Given two random variables X and Y on a discrete probability space W , for any
function g : R ⇥ R ! R, then g(X,Y ) is a random variable and it is easy to show
that E(g(X,Y )) (if it exists) is given by

E(g(X,Y )) = Â g(x, y) fX,Y (x, y),

x,y

where fX,Y is the joint mass function of X and Y .

Example 5.21. Consider the random variable X of Example 5.19 counting the num-
ber of heads in a sequence of coin flips of length n, but this time, let us try to compute
E(X k ), for k 2. By Proposition 5.8, we have
n
E(X k ) = Â ik f (i)
i=0
n ✓ ◆
n i
= Â ik p (1 p)n i
i=0 i
n ✓ ◆
k n
= Âi pi (1 p)n i .
i=1 i

Recall that
302 5 An Introduction to Discrete Probability
✓ ◆ ✓ ◆
n n 1
i =n .
i i 1
Using this, we get
n ✓ ◆
n i
E(X k ) = Â ik p (1 p)n i
i=1 i
n ✓ ◆
k 1 n 1
= np  i pi 1 (1 p)n i (let j = i 1)
i=1 i 1
n 1 ✓ ◆
k 1 n 1
= np  ( j + 1) p j (1 p)n 1 j
j=0 j
= npE((Y + 1)k 1
)

using Proposition 5.8 to establish the last equation, where Y is a random variable
with binomial distribution on sequences of length n 1 and with the same probabil-
ity p of success. Thus, we obtain an inductive method to compute E(X k ). For k = 2,
we get
E(X 2 ) = npE(Y + 1) = np((n 1)p + 1).
Here is a third tool to compute expectation. If X only takes nonnegative integer
values, then the following result may be useful for computing E(X).
Proposition 5.9. If X is a random variable that takes on only nonnegative integers,
then its expectation E(X) (if it exists) is given by
•
E(X) = Â Pr(X i).
i=1

Proof. For any integer n 1, we have

n n j n n n
 jPr(X = j) =   Pr(X = j) =   Pr(X = j) =  Pr(n X i).
j=1 j=1 i=1 i=1 j=i i=1

Then if we let n go to infinity, we get

• • • • j •
 Pr(X i) =   Pr(X = j) =   Pr(X = j) =  jPr(X = j) = E(X),
i=1 i=1 j=i j=1 i=1 j=1

as claimed. t
u
Proposition 5.9 has the following intuitive geometric interpretation: E(X) is the
area above the graph of the cumulative distribution function F(i) = Pr(X  i) of X
and below the horizontal line F = 1. Here is an application of Proposition 5.9.
Example 5.22. In Example 5.9, we consider finite sequences of flips of a biased
coin, and the random variable of interest is the first occurrence of tails (success).
5.4 Expectation of a Random Variable 303

The distribution of this random variable is the geometric distribution,

f (n) = (1 p)n 1
p, n 1.

To compute its expectation, let us use Proposition 5.9. We have

•
Pr(X i) = Â (1 p)i 1
p
j=i
•
= p(1 p)i 1
 (1 p) j
j=0
1
= p(1 p)i 1
1 (1 p)
i 1
= (1 p) .

Then we have
•
E(X) = Â Pr(X i)
i=1
•
= Â (1 p)i 1 .
i=1
1 1
= = .
1 (1 p) p

Therefore,
1
E(X) = ,
p
which means that on the average, it takes 1/p flips until heads turns up.
Let us now compute E(X 2 ). By Proposition 5.8, we have
•
E(X 2 ) = Â i2 (1 p)i 1
p
i=1
•
= Â (i 1 + 1)2 (1 p)i 1
p
i=1
• • •
= Â (i 1)2 (1 p)i 1
p + Â 2(i 1)(1 p)i 1
p + Â (1 p)i 1
p
i=1 i=1 i=1
• •
= Â j2 (1 p) j p + 2 Â j(1 p) j p + 1 (let j = i 1)
j=0 j=1

= (1 p)E(X 2 ) + 2(1 p)E(X) + 1.

Since E(X) = 1/p, we obtain

304 5 An Introduction to Discrete Probability

2(1 p)
pE(X 2 ) = +1
p
2 p
= ,
p
so
2 p
E(X 2 ) = .
p2
By the way, the trick of writing i = i 1 + 1 can be used to compute E(X). Try to
recompute E(X) this way. The expectation E(X) can also be computed using the
derivative technique of Example 5.19, since (d/dt)(1 p)i = i(p 1)i 1 .
Example 5.23. Let us compute the expectation of the number X of comparisons
needed when running the randomized version of quicksort presented in Example
5.7. Recall that the input is a sequence S = (x1 , . . . , xn ) of distinct elements, and that
(y1 , . . . , yn ) has the same elements sorted in increasing order. In order to compute
E(X), we decompose X as a sum of indicator variables Xi, j , with Xi, j = 1 iff yi and
y j are ever compared, and Xi, j = 0 otherwise. Then it is clear that

n j 1
X= Â Â Xi, j ,
j=2 i=1

and
n j 1
E(X) = Â Â E(Xi, j ).
j=2 i=1

Furthermore, since Xi, j is an indicator variable, we have

E(Xi, j ) = Pr(yi and y j are ever compared).

The crucial observation is that yi and y j are ever compared iff either yi or y j is chosen
as the pivot when {yi , yi+1 , . . . , y j } is a subset of the set of elements of the (left or
right) sublist considered for the choice of a pivot.
This is because if the next pivot y is larger than y j , then all the elements in
(yi , yi+1 , . . . , y j ) are placed in the list to the left of y, and if y is smaller than yi ,
then all the elements in (yi , yi+1 , . . . , y j ) are placed in the list to the right of y. Conse-
quently, if yi and y j are ever compared, some pivot y must belong to (yi , yi+1 , . . . , y j ),
and every yk 6= y in the list will be compared with y. But, if the pivot y is distinct
from yi and y j , then yi is placed in the left sublist and y j in the right sublist, so yi
and y j will never be compared.
It remains to compute the probability that the next pivot chosen in the sublist
Yi, j = (yi , yi+1 , . . . , y j ) is yi (or that the next pivot chosen is y j , but the two proba-
bilities are equal). Since the pivot is one of the values in (yi , yi+1 , . . . , y j ) and since
each of these is equally likely to be chosen (by hypothesis), we have
1
Pr(yi is chosen as the next pivot in Yi, j ) = .
j i+1
5.4 Expectation of a Random Variable 305

Consequently, since yi and y j are ever compared iff either yi is chosen as a pivot or
y j is chosen as a pivot, and since these two events are mutally exclusive, we have

2
E(Xi, j ) = Pr(yi and y j are ever compared) = .
j i+1
It follows that
n j 1
E(X) = Â Â E(Xi, j )
j=2 i=1
n j
1
=2Â Âk (set k = j i + 1)
j=2 k=2
n n
1
=2Â Âk
k=2 j=k
n
n k+1
=2Â
k=2 k
n
1
= 2(n + 1) Â 4n.
k=1 k

At this stage, we use the result of Problem 4.32. Indeed,

n
1
 k = Hn
k=1

is a harmonic number, and it is shown that

1
ln(n) +  Hn  ln n + 1.
n
Therefore, Hn = ln n +Q (1), which shows that

E(X) = 2n ln n +Q (n).

Therefore, the expected number of comparisons made by the randomized version of

quicksort is 2n ln n +Q (n).

Example 5.24. If X is a random variable with Poisson distribution with parameter l

(see Example 5.10), let us show that its expectation is

E(X) = l .

Recall that a Poisson distribution is given by

l li
f (i) = e , i 2 N,
i!
306 5 An Introduction to Discrete Probability

so we have
•
li
E(X) = Â ie l
i=0 i!
•
li 1
= le l
 (i 1)!
i=1
• j
l
= le l
 (let j = i 1)
j=0 j!
l l
= le e = l,

as claimed. This is consistent with the fact that the expectation of a random variable
with a binomial distribution is np, under the Poisson approximation where l = np.
We leave it as an exercise to prove that

E(X 2 ) = l (l + 1).

Although in general E(XY ) 6= E(X)E(Y ), this is true for independent random

variables.
Proposition 5.10. If two random variables X and Y on the same discrete probability
space are independent, then

E(XY ) = E(X)E(Y ).

Proof. We have

E(XY ) = Â X(w)Y (w)Pr(w)

w2W
= Â Â xy · Pr(X = x and Y = y)
x y

= Â Â xy · Pr(X = x)Pr(Y = y)
x y
✓ ◆✓ ◆
= Â x · Pr(X = x) Â y · Pr(Y = y)
x y

= E(X)E(Y ),

as claimed. Note that the independence of X and Y was used in going from line 2 to
line 3. t
u
In Example 5.15 (rolling two dice), we defined the random variables S1 and S2 ,
where S1 is the value on the first dice and S2 is the value on the second dice. We
also showed that S1 and S2 are independent. If we consider the random variable
P = S1 S2 , then we have
7 7 49
E(P) = E(S1 )E(S2 ) = · = ,
2 2 4
5.5 Variance, Standard Deviation, Chebyshev’s Inequality 307

since E(S1 ) = E(S2 ) = 7/2, as we easily determine since all probabilities are equal
to 1/6. On the other hand, S and P are not independent (check it).

5.5 Variance, Standard Deviation, Chebyshev’s Inequality

The mean (expectation) E(X) of a random variable X gives some useful information
about it, but it does not say how X is spread. Another quantity, the variance Var(X),
measure the spread of the distribution by finding the “average” of the square differ-
ence (X E(X))2 , namely

Var(X) = E(X E(X))2 .

Note that computing E(X E(X)) yields no information since

E(X E(X)) = E(X) E(E(X)) = E(X) E(X) = 0.

Definition 5.11. Given a discrete probability space (W , Pr), for any random variable
X, the variance Var(X) of X (if it exists) is defined as

Var(X) = E(X E(X))2 .

The expectation E(X) of a random variable X is often denoted by µ. The variance

is also denoted V(X), for instance, in Graham, Knuth and Patashnik [5]).
Since the variance Var(X) involves a square, it can be quite large, so it is conve-
nient to take its square root and to define the standard deviation of s X as
p
s = Var(X).

The following result shows that the variance Var(X) can be computed using E(X 2 )
and E(X).
Proposition 5.11. Given a discrete probability space (W , Pr), for any random vari-
able X, the variance Var(X) of X is given by

Var(X) = E(X 2 ) (E(X))2 .

Consequently, Var(X)  E(X 2 ).

Proof. Using the linearity of expectation and the fact that the expectation of a con-
stant is itself, we have

Var(X) = E(X E(X))2

= E(X 2 2XE(X) + (E(X))2 )
= E(X 2 ) 2E(X)E(X) + (E(X))2
= E(X 2 ) (E(X))2
308 5 An Introduction to Discrete Probability

as claimed. t
u

For example, if we roll a fair dice, we know that the number S1 on the dice has
expectation E(S1 ) = 7/2. We also have

1 91
E(S12 ) = (12 + 22 + 32 + 42 + 52 + 62 ) = ,
6 6
so the variance of S1 is
✓ ◆2
91 7 35
Var(S1 ) = E(S12 ) (E(S1 ))2 = = .
6 2 12

The quantity E(X 2 ) is called the second moment of X. More generally, we have
the following definition.

Definition 5.12. Given a random variable X on a discrete probability space (W , Pr),

for any integer k 1, the kth moment µk of X is given by µk = E(X k ), and the kth
central moment sk of X is defined by sk = E((X µ1 )k ).
p
Typically, only µ = µ1 and s2 are of interest. As before, s = s2 . However,
s3 and s4 give rise to quantities with exotic names: the skewness (s3 /s 3 ) and the
kurtosis (s4 /s 4 3).
We can easily compute the variance of a random variable for the binomial distri-
bution and the geometric distribution, since we already computed E(X 2 ).

Example 5.25. In Example 5.21, the case of a binomial distribution, we found that

E(X 2 ) = npE(Y + 1) = np((n 1)p + 1).

We also found earlier (Example 5.19) that E(X) = np. Therefore, we have

Var(X) = E(X 2 ) (E(X))2

= np((n 1)p + 1) (np)2
= np(1 p).

Therefore,
Var(X) = np(1 p).

Example 5.26. In Example 5.22, the case of a geometric distribution, we found that
1
E(X) =
p
2 p
E(X 2 ) = .
p2
It follows that
5.5 Variance, Standard Deviation, Chebyshev’s Inequality 309

Var(X) = E(X 2 ) (E(X))2

2 p 1
= 2
p p2
1 p
= 2 .
p
Therefore,
1 p
Var(X) = .
p2
Example 5.27. In Example 5.24, the case of a Poisson distribution with parameter
l , we found that

E(X) = l
E(X 2 ) = l (l + 1).

It follows that

Var(X) = E(X 2 ) (E(X))2 = l (l + 1) l2 = l.

Therefore, a random variable with a Poisson distribution has the same value for its
expectation and its variance,

E(X) = Var(X) = l .

In general, if X and Y are not independent variables, Var(X + Y ) 6= Var(X) +

Var(Y ). However, if they are, things are great!
Proposition 5.12. Given a discrete probability space (W , Pr), for any random vari-
able X and Y , if X and Y are independent, then

Var(X +Y ) = Var(X) + Var(Y ).

Proof. Recall from Proposition 5.10 that if X and Y are independent, then E(XY ) =
E(X)E(Y ). Then, we have

E((X +Y )2 ) = E(X 2 + 2XY +Y 2 )

= E(X 2 ) + 2E(XY ) + E(Y 2 )
= E(X 2 ) + 2E(X)E(Y ) + E(Y 2 ).

Using this, we get

Var(X +Y ) = E((X +Y )2 ) (E(X +Y ))2

= E(X 2 ) + 2E(X)E(Y ) + E(Y 2 ) ((E(X))2 + 2E(X)E(Y ) + (E(Y ))2 )
= E(X 2 ) (E(X))2 + E(Y 2 ) (E(Y ))2
= Var(X) + Var(Y ),
310 5 An Introduction to Discrete Probability

as claimed. t
u
The following proposition is also useful.
Proposition 5.13. Given a discrete probability space (W , Pr), for any random vari-
able X, the following properties hold:
1. If X 0, then E(X) 0.
2. If X is a random variable with constant value l , then E(X) = l .
3. For any two random variables X and Y defined on the probability space (W , Pr),
if X  Y , which means that X(w)  Y (w) for all w 2 W , then E(X)  E(Y )
(monotonicity of expectation).
4. For any scalar l 2 R, we have

Var(l X) = l 2 Var(X).

Proof. Properties (1) and (2) are obvious. For (3), X  Y iff Y X 0, so by (1) we
have E(Y X) 0, and by linearity of expectation, E(Y ) E(X). For (4), we have

Var(l X) = E (l X E(l X))2

= E l 2 (X E(X))2
= l 2 E (X E(X))2 = l 2 Var(X),

as claimed. t
u
Property (4) shows that unlike expectation, the variance is not linear (although
for independent random variables, Var(X +Y ) = Var(X) + Var(Y ). This also holds
in the more general case of uncorrelated random variables; see Proposition 5.14
below).
Example 5.28. As an application of Proposition 5.12, if we consider the event of
rolling two dice, since we showed that the random variables S1 and S2 are indepen-
dent, we can compute the variance of their sum S = S1 + S2 and we get
35 35 35
Var(S) = Var(S1 ) + Var(S2 ) = + = .
12 12 6
Recall from Example 5.17 that E(S) = 7.
Here is an application of geometrically distributed random variables.
Example 5.29. Suppose there are m different types of coupons (or perhaps, the kinds
of cards that kids like to collect), and that each time one obtains a coupon, it is
equally likely to be any of these types. Let X denote the number of coupons one
needs to collect in order to have at least one of each type. What is the expected
value E(X) of X? This problem is usually called a coupon collecting problem.
The trick is to introduce the random variables Xi , where Xi is the number of
additional coupons needed, after i distinct types have been collected, until another
new type is obtained, for i = 0, 1, . . . , m 1. Clearly,
5.5 Variance, Standard Deviation, Chebyshev’s Inequality 311

m 1
X= Â Xi ,
i=0

and each Xi has a geometric distribution, where each trial has probability of success
pi = (m i)/m. We know (see Example 5.22,) that

1 m
E(Xi ) = = .
pi m i
Consequently,
m 1 m 1 m
m 1
E(X) = Â E(Xi ) = Â m i
=m .
i=0 i=0 i=1 i

Once again, the harmonic number

m
1
Hm = Âk
k=1

shows up! Since Hn = ln n +Q (1), we obtain

E(X) = m ln m +Q (m).

For example, if m = 50, then ln 50 = 3.912, and m ln m ⇡ 196. If m = 100, then

ln 100 = 4.6052, and m ln m ⇡ 461. If the coupons are expensive, one begins to see
why the company makes money!
It turns out that using a little bit of analysis, we can compute the variance of X.
This is because it is easy to check that the Xi are independent, so
m 1
Var(X) = Â Var(Xi ).
i=0

From Example 5.26, we have

✓ ◆
1 pi m i m2 mi
Var(Xi ) = = 1 = .
p2i m (m i) 2 (m i)2

It follows that
m 1 m 1
i
Var(X) =  Var(Xi ) = m  (m i)2
.
i=0 i=0

To compute this sum, write

312 5 An Introduction to Discrete Probability

m 1 m 1 m 1
i m m i
 (m i)2
= Â (m i)2 Â (m i)2
i=0 i=0 i=0
m 1 m 1
m 1
= Â (m i)2 Â (m i)
i=0 i=0
m m
1 1
=m  .
j=1 j2 j=1 j

Now, it is well known from analysis that

m
1 p2
lim
m7!•
 j2
=
6
,
j=1

so we get
m2 p 2
Var(X) = +Q (m ln m).
6
Let us go back to the example about fixed points of random permutations (Ex-
ample 5.20). We found that the expectation of the number of fixed points is µ = 1.
The reader should compute the standard deviation. The difficulty is that the ran-
dom variables Xk are not independent, (for every permutation p, we have Xk (p) = 1
iff p(k) = k, and 0 otherwise). You will find that s = 1. If you get stuck, look at
Graham, Knuth and Patashnik [5], Chapter 8.
If X and Y are not independent, we still have

E((X +Y )2 ) = E(X 2 + 2XY +Y 2 )

= E(X 2 ) + 2E(XY ) + E(Y 2 ),

and we get

Var(X +Y ) = E((X +Y )2 ) (E(X +Y ))2

= E(X 2 ) + 2E(XY ) + E(Y 2 ) ((E(X))2 + 2E(X)E(Y ) + (E(Y ))2 )
= E(X 2 ) (E(X))2 + E(Y 2 ) (E(Y ))2 + 2(E(XY ) E(X)E(Y ))
= Var(X) + Var(Y ) + 2(E(XY ) E(X)E(Y )).

The term E(XY ) E(X)E(Y ) has a more convenient form. Indeed, we have

E (X E(X))(Y E(Y )) = E XY XE(Y ) E(X)Y + E(X)E(Y )

= E(XY ) E(X)E(Y ) E(X)E(Y ) + E(X)E(Y )
= E(XY ) E(X)E(Y ).

In summary we proved that

Var(X +Y ) = Var(X) + Var(Y ) + 2E (X E(X))(Y E(Y )) .

5.5 Variance, Standard Deviation, Chebyshev’s Inequality 313

The quantity E (X E(X))(Y E(Y )) is well known in probability theory.

Definition 5.13. Given two random variables X and Y , their covariance Cov(X,Y )
is defined by

Cov(X,Y ) = E (X E(X))(Y E(Y )) = E(XY ) E(X)E(Y ).

If Cov(X,Y ) = 0 (equivalently if E(XY ) = E(X)E(Y )) we say that X and Y are

uncorrelated.

Observe that the variance of X is expressed in terms of the covariance of X by

Var(X) = Cov(X, X).

Let us recap the result of our computation of Var(X + Y ) in terms of Cov(X,Y ) as

the following proposition.

Proposition 5.14. Given two random variables X and Y , we have

Var(X +Y ) = Var(X) + Var(Y ) + 2Cov(X,Y ).

Therefore, if X an Y are uncorrelated (Cov(X,Y ) = 0), then

Var(X +Y ) = Var(X) + Var(Y ).

In particular, if X and Y are independent, then X and Y are uncorrelated because

Cov(X,Y ) = E(XY ) E(X)E(Y ) = E(X)E(Y ) E(X)E(Y ) = 0.

This yields another proof of Proposition 5.12.

However, beware that Cov(X,Y ) = 0 does not necessarily imply that X and Y are
independent. For example, let X and Y be the random variables defined on { 1, 0, 1}
by
1
Pr(X = 0) = Pr(X = 1) = Pr(X = 1) = ,
3
and n
0 if X 6= 0
Y=
1 if X = 0 .
Since XY = 0, we have E(XY ) = 0, and since we also have E(X) = 0, we have

Cov(X,Y ) = E(XY ) E(X)E(Y ) = 0.

However, the reader will check easily that X and Y are not independent.
A better measure of independence is given by the correlation coefficient r(X,Y )
of X and Y , given by
Cov(X,Y )
r(X,Y ) = p p ,
Var(X) Var(Y )
314 5 An Introduction to Discrete Probability

provided that Var(X) 6= 0 and Var(Y ) 6= 0. It turns out that |r(X,Y )|  1, which is
shown using the Cauchy–Schwarz inequality.

Proposition 5.15. (Cauchy–Schwarz inequality) For any two random variables X

and Y on a discrete probability space W , we have
q q
|E(XY )|  E(X 2 ) E(Y 2 ).

Equality is achieved if and only if there exist some a, b 2 R (not both zero) such
that E((aX + bY )2 ) = 0.

Proof. This is a standard argument involving a quadratic equation. For any l 2 R,

define the function T (l ) by

T (l ) = E((X + lY )2 ).

We get

T (l ) = E(X 2 + 2l XY + l 2Y 2 )
= E(X 2 ) + 2l E(XY ) + l 2 E(Y 2 ).

Since E((X +lY )2 ) 0, we have T (l ) 0 for all l 2 R. If E(Y 2 ) = 0, then we must

have E(XY ) = 0, since otherwise we could choose l so that E(X 2 ) + 2l E(XY ) < 0.
In this case, the inequality is trivial. If E(Y 2 ) > 0, then for T (l ) to be nonnegative
the quadratic equation

E(X 2 ) + 2l E(XY ) + l 2 E(Y 2 ) = 0

should have at most one real root, which is equivalent to the well-known condition

4(E(XY ))2 4E(X 2 )E(Y 2 )  0,

which is equivalent to q q
|E(XY )|  E(X 2 ) E(Y 2 ),
as claimed.
If (E(XY ))2 = E(X 2 )E(Y 2 ), then either E(Y 2 ) = 0, and then with a = 0, b = 1,
we have E((aX + bY )2 ) = 0, or E(Y 2 ) > 0, in which case the quadratic equation

E(X 2 ) + 2l E(XY ) + l 2 E(Y 2 ) = 0

has a unique real root l0 , so we have E((X + l0Y )2 ) = 0.

Conversely, if E((aX + bY )2 ) = 0 for some a, b 2 R, then either E(Y 2 ) = 0, in
which case we showed that we also have E(XY ) = 0, or the quadratic equation has
some real root, so we must have (E(XY ))2 E(X 2 )E(Y 2 ) = 0. In both cases, we
have (E(XY ))2 = E(X 2 )E(Y 2 ). t u
5.5 Variance, Standard Deviation, Chebyshev’s Inequality 315

It can be shown that for any random variable Z, if E(Z 2 ) = 0, then Pr(Z = 0) =
1; see Grimmett and Stirzaker [6] (Chapter 3, Problem 3.11.2). In fact, this is a
consequence of Proposition 5.2 and Chebyshev’s Inequality (see below), as shown
in Ross [11] (Section 8.2, Proposition 2.3). It follows that if equality is achieved in
the Cauchy–Schwarz inequality, then there are some reals a, b (not both zero) such
that Pr(aX + bY = 0) = 1; in other words, X and Y are dependent with probability
1. If we apply the Cauchy-Schwarz inequality to the random variables X E(X) and
Y E(Y ), we obtain the following result.
Proposition 5.16. For any two random variables X and Y on a discrete probability
space, we have
|r(X,Y )|  1,
with equality iff there are some real numbers a, b , g (with a, b not both zero) such
that Pr(aX + bY = g) = 1.
As emphasized by Graham, Knuth and Patashnik [5], the variance plays a key
role in an inquality due to Chebyshev (published in 1867) that tells us that a random
variable will rarely be far from its mean E(X) if its variance Var(X) is small.

Fig. 5.9 Pafnuty Lvovich Chebyshev (1821–1894).

Proposition 5.17. (Chebyshev’s Inequality) If X is any random variable, for every

a > 0, we have
Var(X)
Pr (X E(X))2 a 
a
Proof. We follow Knuth. We have

Var(X) = Â (X(w) E(X))2 Pr(w)

w2W

 (X(w) E(X))2 Pr(w)

w2W
(X(w) E(X))2 a

 aPr(w)
w2W
(X(w) E(X))2 a

= aPr (X E(X))2 a ,
316 5 An Introduction to Discrete Probability

which yields the desired inequality. t

u

The French know this inequality as the Bienaymé–Chebyshev’s Inequality. Bien-

aymé proved this inequality in 1853, before Chebyshev who published it in 1867.
However, it was Chebyshev who recognized its significance.2 Note that Chebyshev’s
Inequality can also be stated as

Var(X)
Pr |X E(X)| a  .
a2
It is also convenient
p to restate the Chebyshev’s Inequality in terms of the standard
deviation s = Var(X) of X, to write E(X) = µ, and to replace a 2 by c2 Var(X),
and we get: For every c > 0,
1
Pr(|X µ| cs )  ;
c2
equivalently
1
Pr(|X µ| < cs ) . 1
c2
This last inequality says that a random variable will lie within cs of its mean with
probability at least 1 1/c2 . If c = 10, the random variable will lie between µ 10s
and µ + 10s at least 99% of the time.
We can apply the Chebyshev Inequality to the experiment of Example 5.28 where
we roll two fair dice. We found that µ = 7 and s 2 = 35/6 (for one roll). If we
assume that we perform n independent trials, then the total value of the n rolls has
expecation 7n and the variance if 35n/6. It follows that the sum will be between
r r
35n 35n
7n 10 and 7n + 10
6 6

at least 99% of the time. If n = 106 (a million rolls), then the total value will be
between 6.976 million and 7.024 million more than 99% of the time.
Another interesting consequence of the Chebyshev’s Inequality is this. Suppose
we have a random variable X on some discrete probability space (W , Pr). For any n,
we can form the product space (W n , Pr) as explained in Definition 5.6, with

Pr(w1 , . . . , wn ) = Pr(w1 ) · · · Pr(wn ), wi 2 W , i = 1, . . . , n.

Then, we define the random variable Xk on the product space by

Xk (w1 , . . . , wn ) = X(wk ).

It is easy to see that the Xk are independent. Consider the random variable

S = X1 + · · · + Xn .

2 Still, Bienaymé is well loved!

5.6 Limit Theorems; A Glimpse 317

We can think of S as taking n independent “samples” from W and adding thempto-

gether. By our previous discussion, S has mean nµ and standard deviation s n,
where µ is the mean of X and s is its standard deviation. The Chebyshev’s Inequal-
ity implies that the average
X1 + · · · + Xn
n
p p
will lie between s 10s / n and s + 10s / n at least 99% of the time. This im-
plies that if we choose n large enough, then the average of n samples will almost
always be very near the expected value µ = E(X).
This concludes our elementary introduction to discrete probability. The reader
should now be well prepared to move on to Grimmett and Stirzaker [6] or Venkatesh
[14]. Among the references listed at the end of this chapter, let us mention the clas-
sical volumes by Feller [3, 4], and Shiryaev [13].
The next three sections are devoted to more advanced topics and are optional.

5.6 Limit Theorems; A Glimpse

The behavior of the average sum of n independent samples described at the end of
Section 5.5 is an example of a weak law of large numbers. A precise formulation
of such a result is shown below. A version of this result was first shown by Jacob
Bernoulli and was published by his nephew Nicholas in 1713. Bernoulli did not have
Chebyshev’s Inequality at this disposal (since Chebyshev Inequality was proved in
1867), and he had to resort to a very ingenious proof.

Fig. 5.10 Jacob (Jacques) Bernoulli (1654–1705).

Theorem 5.1. (Weak Law of Large Numbers (“Bernoulli’s Theorem”)) Let X1 , X2 ,

. . . , Xn , . . . be a sequence of random variables. Assume that they are independent,
that they all have the same distribution, and let µ be their common mean and s 2 be
their common variance (we assume that both exist). Then, for every e > 0,
✓ ◆
X1 + · · · + Xn
lim Pr µ e = 0.
n7!• n
318 5 An Introduction to Discrete Probability

Proof. As earlier, ✓ ◆
X1 + · · · + Xn
E =µ
n
and because the Xi are independent,
✓ ◆
X1 + · · · + Xn s2
Var = .
n n

Then, we apply Chebyshev’s Inequality and we obtain

✓ ◆
X1 + · · · + Xn s2
Pr µ e  2,
n ne

which proves the result. t

u

The locution independent and identically distributed random variables is often

used to say that some random variables are independent and have the same distribu-
tion. This locution is abbreviated as i.i.d. Probability books are replete with i.i.d.’s

Another remarkable limit theorem has to do with the limit of the distribution of
the random variable
X1 + · · · + Xn nµ
p ,
s n
where the Xi are i.i.d random variables with mean µpand variance s . Observe that
the mean of X1 + · · · + Xn is nµ and its variance is s n, since the Xi are assumed to
be i.i.d.
We have not discussed a famous distribution, the normal or Gaussian distribution,
only because it is a continuous distribution. The standard normal distribution is the
cumulative distribution function F whose density function is given by
1 1 x2
f (x) = p e 2 ;
2p
that is, Z x
1 1 y2
F(x) = p e 2 dy.
2p •

The function f (x) decays to zero very quickly and its graph has a bell–shape. More
generally, we say that a random variable X is normally distributed with parameters
µ and s 2 (and that X has a normal distribution) if its density function is the function

1 (x µ)2
f (x) = p e 2s 2 .
2ps
Figure 5.11 shows some examples of normal distributions.
Using a little bit of calculus, it is not hard to show that if a random variable X
is normally distributed with parameters µ and s 2 , then its mean and variance are
5.6 Limit Theorems; A Glimpse 319

Fig. 5.11 Examples of normal distributions.

given by

E(X) = µ,
Var(X) = s 2 .

The normal distribution with parameters µ and s 2 is often denoted by N (µ, s 2 ).

The standard case corresponds to µ = 0 and s = 1.
The following theorem was first proved by de Moivre in 1733 and generalized by
Laplace in 1812. De Moivre introduced the normal distribution in 1733. However, it
was Gauss who showed in 1809 how important the normal distribution (alternatively
Gaussian distribution) really is.

Fig. 5.12 Abraham de Moivre (1667–1754) (left), Pierre–Simon Laplace (1749–1827) (middle),
Johann Carl Friedrich Gauss (1777–1855) (right).

Theorem 5.2. (de Moivre–Laplace Limit Theorem) Consider n repeated indepen-

dent Bernoulli trials (coin flips) Xi , where the probability of success is p. Then, for
all a < b,
✓ ◆
X1 + · · · + Xn np
lim Pr a  p  b = F(b) F(a).
n7!• np(1 p)
320 5 An Introduction to Discrete Probability

Observe that now, we have two approximations for the distribution of a random
variable X = X1 + · · · + Xn with a binomial distribution. When n is large and p is
small, we have the Poisson approximation. When np(1 p) is large, the normal
approximation can be shown to be quite good.
Theorem 5.2 is a special case of the following important theorem known as cen-
tral limit theorem.

Theorem 5.3. (Central Limit Theorem) Let X1 , X2 , . . . , Xn , . . . be a sequence of ran-

dom variables. Assume that they are independent, that they all have the same dis-
tribution, and let µ be their common mean and s 2 be their common variance (we
assume that both exist). Then, the distribution of the random variable
X1 + · · · + Xn nµ
p
s n

tends to the standard normal distribution as n goes to infinity. This means that for
every real a,
✓ ◆ Z a
X1 + · · · + Xn nµ 1 1 2
lim Pr p a = p e 2x .
n7!• s n 2p •
We lack the machinery to prove this theorem. This machinery involves character-
istic functions and various limit theorems. We refer the interested reader to Ross [11]
(Chapter 8), Grimmett and Stirzaker [6] (Chapter 5), Venkatesh [14], and Shiryaev
[13] (Chapter III).
The central limit theorem was originally stated and proved by Laplace but
Laplace’s proof was not entirely rigorous. Laplace expanded a great deal of efforts
in estimating sums of the form
✓ ◆
n k
 k
p (1 p)n k
pk
knp+x np(1 p)

using Stirling’s formula.

Reading Laplace’s classical treatise [7, 8] is an amazing experience. The intro-
duction to Volume I is 164 pages long! Among other things, it contains some inter-
esting philosophical remarks about the role of probability theory, for example on the
reliability of the testimony of witnesses. It is definitely worth reading. The second
part of Volume I is devoted to the theory of generating functions, and Volume II
to probability theory proper. Laplace’s treatise was written before 1812, and even
though the factorial notation was introduced in 1808, Laplace does not use it, which
makes for complicated expressions. The exposition is clear, but it is difficult to read
this treatise because definitions and theorems are not clearly delineated. A version
of the central limit theorem is proved in Volume II, Chapter III; page 306 contains
a key formula involving the Gaussian distribution, although Laplace does not refer
to it by any name (not even as normal distribution). Anybody will be struck by the
5.6 Limit Theorems; A Glimpse 321

elegance and beauty of the typesetting. Lyapunov gave the first rigorous proof of the
central limit theorem around 1901.

Fig. 5.13 Pierre–Simon Laplace (1749–1827) (left), Aleksandr Mikhailovich Lyapunov (1857-
1918) (right).

The following example from Ross [11] illustrates how the central limit theorem
can be used.

Example 5.30. An astronomer is interested in measuring the distance, in light-years,

from his observatory to a distant star. Although the astronomer has a measuring tech-
nique, he knows that, because of changing atmospheric conditions and normal error,
each time a measurement is made it will not be the exact distance, but merely an ap-
proximation. As a result, the astronomer plans to make a series of measurements
and then use the average value of these measurements as his estimated value of the
actual distance.
If the astronomer believes that the values of the measurements are independent
and identically distributed random variables having a common mean d and a com-
mon variance 4 (light-years), how many measurements need he make to be reason-
ably sure that his estimated distance is accurrate to within ±0.5 light-years?
Suppose that the astronomer makes n observations, and let X1 , . . . , Xn be the n
measurements. By the central limit theorem, the random variable

X1 + · · · + Xn nd
Zn = p
2 n

has approximately a normal distribution. Hence,

✓ ◆ ✓ p p ◆
1 X1 + · · · + Xn 1 1 n 1 n
Pr   = Pr  Zn 
2 n 2 2 2 2 2
✓p ◆ ✓ p ◆
n n
⇡F F
4 4
✓p ◆
n
= 2F 1.
4

If the astronomer wants to be 95% certain that his estimated value is accurrate to
within 0.5 light year, he should make n⇤ measurements, where n⇤ is given by
322 5 An Introduction to Discrete Probability
✓p ◆
n⇤
2F 1 = 0.95,
4

that is, ✓p ◆
n⇤
F = 0.975.
4
Using tables for the values of the function F, we find that
p
n⇤
= 1.96,
4
which yields
n⇤ ⇡ 61.47.
Since n should be an integer, the astronomer should make 62 observations.
The above analysis relies on the assumption that the distribution of Zn is well
approximated by the normal distribution. If we are concerned about this point, we
can use Chebyshev’s inequality. If we write

X1 + · · · + Xn nd
Sn = ,
2
we have
4
E(Sn ) = d and Var(Sn ) = ,
n
so by Chebyshev’s inequality, we have
✓ ◆
1 4 16
Pr |Sn d| >  2
= .
2 n(1/2) n

Hence, if we make n = 16/0.05 = 320 observations, we are 95% certain that the
estimate will be accurate to within 0.5 light year.
The method of making repeated measurements in order to “average” errors is
applicable to many different situations (geodesy, astronomy, etc.).

There are generalizations of the central limit theorem to independent but not
necessarily identically distributed random variables. Again, the reader is referred to
Ross [11] (Chapter 8), Grimmett and Stirzaker [6] (Chapter 5), and Shiryaev [13]
(Chapter III).
There is also the famous strong law of large numbers due to Andrey Kolmogorov
proved in 1933 (with an earlier version proved in 1909 by Émile Borel). In order to
state the strong law of large numbers, it is convenient to define various notions of
convergence for random variables.

Definition 5.14. Given a sequence of random variable X1 , X2 , . . . , Xn , . . ., and some

random variable X (on the same probability space (W , Pr)), we have the following
definitions:
5.6 Limit Theorems; A Glimpse 323

Fig. 5.14 Félix Edouard Justin Émile Borel (1871–1956) (left), Andrey Nikolaevich Kolmogorov
(1903–1987) (right).

1. We say that Xn converges to X almost surely (abbreviated a.s.), denoted by

a.s.
Xn ! X, if
Pr({w 2 W | lim Xn (w) = X(w)}) = 1.
n7!•
r
2. We say that Xn converges to X in rth mean, with r 1, denoted Xn ! X, if
E(|Xnr |) is finite for all n and if

lim E |Xn X|r = 0.

n7!•

P
3. We say that Xn converges to X in probability, denoted Xn ! X, if for every
e > 0,
lim Pr(|Xn X| > e) = 0.
n7!•
D
4. We say that Xn converges to X in distribution, denoted Xn ! X, if

lim Pr(Xn  x) = Pr(X  x),

n7!•

for every x 2 R for which F(x) = Pr(X  x) is continuous.

Convergence of type (1) is also called convergence almost everywhere or conver-

gence with probability 1. Almost sure convergence can be stated as the fact that the
set
{w 2 W | Xn (w) does not converge to X(w)}
of outcomes for which convergence fails has probability 0.
It can be shown that both convergence almost surely and convergence in rth
mean imply convergence in probability, which implies convergence in distribution.
All converses are false. Neither convergence almost surely nor convergence in rth
mean imply the other. For proofs, Interested readers should consult Grimmett and
Stirzaker [6] (Chapter 7) and Shiryaev [13] (Chapter III).
Observe that the convergence of the weak law of large numbers is convergence
in probability, and the convergence of the central limit theorem is convergence in
distribution.
The following beautiful result was obtained by Kolmogorov (1933).
324 5 An Introduction to Discrete Probability

Theorem 5.4. (Strong Law of Large Numbers, Kolmogorov) Let X1 , X2 , . . . , Xn , . . .

be a sequence of random variables. Assume that they are independent, that they all
have the same distribution, and let µ be their common mean and E(X12 ) be their
common second moment (we assume that both exist). Then,
X1 + · · · + Xn
n
converges almost surely and in mean square to µ = E(X1 ).

The proof is beyond the scope of this book. Interested readers should consult
Grimmett and Stirzaker [6] (Chapter 7), Venkatesh [14], and Shiryaev [13] (Chapter
III). Fairly accessible proofs under the additional assumption that E(X14 ) exists can
be found in Brémaud [2], and Ross [11].
Actually, for almost sure convergence, the assumption that E(X12 ) exists is re-
dundant provided that E(|X1 |) exists, in which case µ = E(|X1 |), but the proof takes
some work; see Brémaud [2] (Chapter 1, Section 8.4) and Grimmett and Stirzaker
[6] (Chapter 7). There are generalizations of the strong law of large numbers where
the independence assumption on the Xn is relaxed, but again, this is beyond the
scope of this book.

5.7 Generating Functions; A Glimpse

If a random variables X on some discrete probability space (W , Pr) takes nonnega-

tive integer values, then we can define its probability generating function (for short
pgf ) GX (z) as
GX (z) = Â Pr(X = k)zk ,
k 0

which can also be expressed as

GX (z) = Â Pr(w)zX(w) = E(zX ).

w2W

Therefore
GX (z) = E(zX ).
Note that
GX (1) = Â Pr(w) = 1,
w2W

so the radius of convergence of the power series GX (z) is at least 1. The nicest
property about pgf’s is that they usually simplify the computation of the mean and
variance. For example, we have
5.7 Generating Functions; A Glimpse 325

E(X) = Â kPr(X = k)
k 0

= Â Pr(X = k) · kzk 1
z=1
k 0
= G0X (1).

Similarly,

E(X 2 ) = Â k2 Pr(X = k)
k 0

= Â Pr(X = k) · (k(k 1)zk 2

+ kzk 1
) z=1
k 0
= G00X (1) + G0X (1).

Therefore, we have

E(X) = G0X (1)

Var(X) = G00X (1) + G0X (1) (G01 (1))2 .

Remark: The above results assume that G0X (1) and G00X (1) are well defined, which
is the case if the radius of convergence of the power series GX (z) is greater than 1.
If the radius of convergence of GX (z) is equal to 1 and if limz"1 G0X (z) exists, then

E(X) = lim G0X (z),

z"1

and similarly if limz"1 G00X (z) exists, then

E(X 2 ) = lim G00X (z).

z"1

The above facts follow from Abel’s theorem, a result due to N. Abel. Abel’s theorem

Fig. 5.15 Niels Henrik Abel (1802–1829).

states that if G(x) = • n

n=0 an z is a real power series with radius of convergence
•
R = 1 and if the sum Ân=0 an exists, which means that
326 5 An Introduction to Discrete Probability
n
lim
n7!•
 ai = a
i=0

for some a 2 R, then G(z) can be extended to a uniformly convergent series on

[0, 1] such that limz7!1 GX (z) = a. For details, the reader is referred to Grimmett and
Stirzaker [6] (Chapter 5) and Brémaud [2] (Appendix, Section 1.2).
However, as explained in Graham, Knuth and Patashnik [5], we may run into
unexpected problems in using a closed form formula for GX (z). For example, if X
is a random variable with the uniform distribution of order n , which means that X
takes any value in {0, 1, . . . , n 1} with equal probability 1/n, then the pgf of X is

1 1 zn
Un = (1 + z + · · · + zn 1 ) = .
n n(1 z)

If we set z = 1 in the above closed-form expression, we get 0/0. The computations

of the derivatives UX0 (1) and UX00 (1) will also be problematic (although we can resort
to L’Hospital’s rule).
Fortunately, there is an easy fix. If G(z) = Ân 0 an zn is a power series that con-
verges for some z with |z| > 1, then G0 (z) = Ân 0 nan zn 1 also has that property,
and by Taylor’s theorem, we can write

G0 (1) G00 (1) 2 G000 (1) 3

G(1 + x) = G(1) + x+ x + x +··· .
1! 2! 3!
It follows that all derivatives of G(z) at z = 1 appear as coefficients when G(1 + x)
is expanded in powers of x. For example, we have

(1 + x)n 1
Un (1 + x) =
✓ ◆ nx ✓ ◆ ✓ ◆ ✓ ◆
1 n 1 n 1 n 2 1 n n 1
= + x+ x +···+ x .
n 1 n 2 n 3 n n

It follows that
n 1 (n 1)(n 2)
Un (1) = 1; Un0 (1) = ; Un00 (1) = ;
2 3
Then, we find that the mean is given by
n 1
µ=
2
and the variance by

n2 1
s 2 = Un00 (1) +Un0 (1) (Un0 (1))2 = .
12
5.7 Generating Functions; A Glimpse 327

Another nice fact about pgf’s is that the pdf of the sum X + Y of two indepen-
dent variables X and Y is the product their pgf’s. This is because if X and Y are
independent, then
n
Pr(X +Y = n) = Â Pr(X = k and Y = n k)
k=0
n
= Â Pr(X = k)Pr(Y = n k),
k=0

a convolution! Therefore, if X and Y are independent, then

GX+Y (z) = GX (z)GY (z).

If we flip a biased coin where the probability of tails is p, then the pgf for the
number of heads after one flip is

H(z) = 1 p + pz.

If we make n independent flips, then the pgf of the number of heads is

H(z)n = (1 p + pz)n .

This allows us to rederive the formulae for the mean and the variance. We get

µ = (H n (z)0 (1) = nH 0 (1) = np,

and
s 2 = n(H 00 (1) + H 0 (1) (H 0 (1))2 ) = n(0 + p p2 ) = np(1 p).
If we flip a biased coin repeatedly until heads first turns up, we saw that the
random variable X that gives the number of trials n until the first occurrence of tails
has the geometric distribution f (n) = (1 p)n 1 p. It follows that the pgf of X is
pz
GX (z) = pz + (1 p)pz2 + · · · + (1 p)n 1
pzn + · · · = .
1 (1 p)z

Since we are assuming that these trials are independent, the random variables that
tell us that m heads are obtained has pgf
✓ ◆m
pz
GX (z) =
1 (1 p)z
✓ ◆
m+k 1
= pm zm  ((1 p)z)k
k k
✓ ◆
j 1 m
=Â p (1 p) j m zk .
j j m
328 5 An Introduction to Discrete Probability

An an exercise, the reader should check that the pgf of a Poisson distribution with
parameter l is
GX (z) = el (z 1) .
More examples of the use of pgf can be found in Graham, Knuth and Patashnik
[5].
Another interesting generating function is the moment generating function MX (t).
It is defined as follows: for any t 2 R,

MX (t) = E(etX ) = Â etx f (x),

x

where f (x) is the mass function of X. If X is a continuous random variable with

density function f , then Z •
MX (t) = etx f (x)dx.
•
The main problem with the moment generating function is that it is not always
defined for all t 2 R. If MX (t) converges absolutely on some open interval ( r, r)
with r > 0, then its nth derivative for t = 0 is given by

M (n) (0) = Â xn etx f (x) t=0

= Â xn f (x) = E(X n ).
x x

Therefore, the moments of X are all defined and given by

E(X n ) = M (n) (0).

Within the radius of convergence of MX (t), we have the Taylor expansion

•
E(X k ) k
MX (t) = Â t .
k=0 k!

This shows that MX (t) is the exponential generating function of the sequence of mo-
ments (E(X n )); see Graham, Knuth and Patashnik [5]. If X is a continuous random
variable, then the function MX ( t) is the Laplace transform of the density function
f.
Furthermore, if X and Y are independent, then E(XY ) = E(X)E(Y ), so we have
n ✓ ◆ n ✓ ◆
n n
E (X +Y )n = Â E(X kY n k ) = Â E(X)k E(Y )n k ,
k=0 k k=0 k

and since
5.7 Generating Functions; A Glimpse 329

E (X +Y )n n
MX+Y (t) = Â t
n n!
!
n ✓ ◆
1 n
= Â k E(X)k E(Y )n k t n
n! k=0
E(X)k E(Y )n k n
=Â t
n k! (n k)!
E(X k ) E(Y n k ) n
=Â t .
n k! (n k)!

But, this last term is the coefficient of t n in MX (t)MY (t). Therefore, as in the case of
pgf’s, if X and Y are independent, then

MX+Y (t) = MX (t)MY (t).

Another way to prove the above equation is to use the fact that if X and Y are
independent random variables, then so are etX and etY for any fixed real t. Then,

E(et(X+Y ) ) = E(etX etY ) = E(etX )E(etY ).

Remark: If the random variable X takes nonnegative integer values, then it is easy
to see that
MX (t) = GX (et ),
where GX is the generating function of X, so MX is defined over some open
interval ( r, r) with r > 0 and MX (t) > 0 on this interval. Then, the function
KX (t) = ln MX (t) is well defined, and it has a Taylor expansion
k1 k2 k3 kn
KX (t) = t + t2 + t3 + · · · + tn + · · · . (⇤)
1! 2! 3! n!
The numbers kn are called the cumulants of X. Since
•
µn n
MX (t) = Â t ,
n=0 n!

where µn = E(E n ) is the nth moment of X, by taking exponentials on both sides of

(⇤), we get relations between the cumulants and the moments, namely:

k1 = µ1
k2 = µ2 µ12
k3 = µ3 3µ1 µ2 + 2µ13
k4 = µ4 4µ1 µ4 + 12µ12 µ2 3µ22 6µ14
..
.
330 5 An Introduction to Discrete Probability

Notice that k1 is the mean and k2 is the variance of X. Thus, it appears that the
cumulants are the natural generalization of the mean and variance. Furthermore, be-
cause logs are taken, all cumulants of the sum of two independent random variables
are additive, just as the mean and variance. This property makes cumulants more
important than moments.
The third generating function associtaed with a random variable X, and the most
important, is the characteristic function jX (t), defined by

jX (t) = E(eitX ) = E(costX) + iE(sintX),

for all t 2 R. If f is the mass function of X, we have

jX (t) =  eitx f (x) =  cos(tx) f (x) + i  sin(tx) f (x),

x x x

a complex function of the real variable t. The “innocent” insertion of i in the expo-
nent has the effect that |eitX | = 1, so jX (t) is defined for all t 2 R.
If X is a continuous random variable with density function f , then
Z •
jX (t) = eitx f (x)dx.
•

Up to sign and to a change of variable, jX (t) is basically the Fourier transform of

f . Traditionally the Fourier transform fb of f is given by
Z •
2pitx
fb(t) = e f (x)dx.
•

Next, we summarize some of the most important properties of jX without proofs.

Details can be found in Grimmett and Stirzaker [6] (Chapter 5).
The characteristic function jX of a random variable satisfies the following prop-
erties:
1. jX (0) = 1, |jX (t)|  1.
2. jX is uniformly continuous on R.
3. If j (n) exists, then E(|E k |) is finite if k is even, and E(|E k 1 |) is finite if k is
odd.
4. If X and Y are independent, then

jX+Y (t) = jX (t)jY (t).

The proof is essentially the same as the one we gave for the moment generating
function, modulo powers of i.
5. If X is a random variable, for any two reals a, b,

jaX+b (t) = eitb jX (at).

Given two random variables X and Y , their joint characteristic function jX,Y (x, y)
is defined by
5.8 Chernoff Bounds 331

jX,Y (x, y) = E(eixX eiyY ).

Then, X and Y are independent iff

jX,Y (x, y) = jX (x)jY (y) for all x, y 2 R.

In general, if all the moments µn = E(X n ) of a random variable X are defined,

these moments do not uniquely define the distibution F of X. There are examples of
distinct distributions F (for X) and G (for Y ) such that E(X n ) = E(Y n ) for all n; see
Grimmett and Stirzaker [6] (Chapter 5).
However, if the moment generating function of X is defined on some open inter-
val ( r, r) with r > 0, then MX (t) defines the distribution F of X uniquely.
The reason is that in this case, the characteristic function jX is holomorphic
on the strip |Im(z)| < r, and then MX can extended to that strip to a holomorphic
function such that jX (t) = MX (it). Furthermore, the characteristic function jX de-
termines the distribution F of X uniquely. This is a rather deep result which is ba-
sically a version of Fourier inversion. If X is a continuous random variable with
density function f , then
Z •
1 itx
f (x) = e jX (t)dt,
2p •

for every x for which f is differentiable.

If the distribution F is not given as above, it is still possible to prove the following
result (see Grimmett and Stirzaker [6] (Chapter 5)):

Theorem 5.5. Two random variables X and Y have the same characteristic function
iff they have the same distribution.

As a corollary, if the moment generating functions MX and MY are defined on

some interval ( r, r) with r > 0 and if MX = MY , then X and Y have the same
distribution. In computer science, this condition seems to be always satisfied.
If X is a discrete random variable that takes integer values, then
Z p
1 itk
f (k) = e jX (t)dt;
2p p

see Grimmett and Stirzaker [6] (Chapter 5, Exercise 4).

There are also some useful continuity theorems which can be found in Grimmett
and Stirzaker [6] (Chapter 5). In the next section, we use the moment generating
function to obtain bounds on tail distributions.

5.8 Chernoff Bounds

Given a random variable X, it is often desirable to have information about proba-

bilities of the form Pr(X a) (for some real a). In particular, it may be useful to
332 5 An Introduction to Discrete Probability

know how quickly such a probability goes to zero as a becomes large (in absolute
value). Such probabilities are called tail distributions. It turns out that the moment
generating function MX (if it exists) yields some useful bounds by applying a very
simple inequality to MX known as Markov’s inequality and due to the mathematician
Andrei Markov, a major contributor to probability theory (the inventor of Markov
chains).

Fig. 5.16 Andrei Andreyevich Markov (1856–1922).

Proposition 5.18. (Markov’s Inequality) Let X be a random variable and assume

that X is nonnegative. Then, for every a > 0, we have

E(X)
Pr(X a)  .
a
Proof. Let Ia be the random variable defined so that
n
1 if X a
Ia =
0 otherwise.
Since X 0, we have
X
Ia  . (⇤)
a
Also, since Ia takes only the values 0 and 1, E(Ia ) = Pr(X a). By taking expecta-
tions in (⇤), we get
E(X)
E(Ia )  ,
a
which is the desired inequality since E(Ia ) = Pr(X a). t u

If we apply Markov’s inequality to the moment generating function MX = E(E tX )

we obtain exponential bounds known as Chernoff bounds, after Herman Chernoff.

Proposition 5.19. (Chernoff Bounds) Let X be a random variable and assume that
the moment generating function MX = E(etX ) is defined. Then, for every a > 0, we
have
5.8 Chernoff Bounds 333

Fig. 5.17 Herman Chernoff (1923–).

ta
Pr(X a)  min e MX (t)
t>0
ta
Pr(X  a)  min e MX (t).
t<0

Proof. If t > 0, by Markov’s inequality applied to MX (t) = E(etX ), we get

Pr(X a) = Pr(etX eta )

 E(etX )e ta
,

and if t < 0, we get

Pr(X  a) = Pr(etX  eta )

 E(etX )e ta
,

which imply both inequalities of the proposition. t

u

In order to make good use of the Chernoff bounds, one needs to find for which
values of t the function e ta MX (t) is minimum. Let us give a few examples.

Example 5.31. If X has a standard normal distribution, then it is not hard to show
that
2
M(t) = et /2 .
Consequently, for any a > 0 and all t > 0, we get
ta t 2 /2
Pr(X a)  e e .
2
The value t that minimizes et /2 ta is the value that minimizes t 2 /2 ta, namely
t = a. Thus, for a > 0, we have
a2 /2
Pr(X a)  e .

Similarly, for a < 0, we obtain

a2 /2
Pr(X  a)  e .
334 5 An Introduction to Discrete Probability

The function on the right hand side decays to zero very quickly.

Example 5.32. Let us now consider a random variable X with a Poisson distribution
with parameter l . It is not hard to show that
t
M(t) = el (e 1)
.

Applying the Chernoff bound, for any nonnegative integer k and all t > 0, we get
t
Pr(X k)  el (e 1)
e kt
.

Using calculus, we can show that the function on the right hand side has a minimum
when l (et 1) kt is minimum, and this is when et = k/l . If k > l and if we let
et = k/l in the Chernoff bound, we obtain
✓ ◆k
l
Pr(X k)  el (k/l 1)
,
k

which is equivalent to
e l (el )k
Pr(X k)  .
kk
Our third example is taken from Mitzenmacher and Upfal [10] (Chapter 4).

Example 5.33. Suppose we have a sequence of n random variables X1 , X2 , . . . , Xn ,

such that each Xi is a Bernoulli variable (with value 0 or 1) with probability of
success pi , and assume that these variables are independent. Such sequences are
often called Poisson trials. We wish to apply the Chernoff bounds to the random
variable
X = X1 + · · · + Xn .
We have
n n
µ = E(X) = Â E(Xi ) = Â pi .
i=1 i=1

The moment generating function of Xi is given by

MXi (t) = E(etXi )

= pi et + (1 pi )
t
= 1 + pi (e 1).

Using the fact that 1 + x  ex for all x 2 R, we obtain the bound

t
MXi (t)  e pi (e 1)
.

Since the Xi are independent, we know from Section 5.7 that

5.8 Chernoff Bounds 335
n
MX (t) = ’ MXi (t)
i=1
n
 ’ e pi (e
t 1)
i=1
n t
= eÂi=1 pi (e 1)

t
= eµ(e 1)
.

Therefore,
t
MX (t)  eµ(e 1)
for all t.
The next step is to apply the Chernoff bounds. Using a little bit of calculus, we
obtain the following result proved in Mitzenmacher and Upfal [10] (Chapter 4).

Proposition 5.20. Given n independent Bernoulli variables X1 , . . . , Xn with success

probability pi , if we let µ = Âni=1 pi and X = X1 + · · · + Xn , then for any d such that
0 < d < 1, we have
µd 2
Pr(|X µ| d µ)  2e 3 .

An an application, if the Xi are independent flips of a fair coin (pi = 1/2), then
µ = n/2, and by picking d = 6 lnn n , it is easy to show that
✓ ◆
n 1p µd 2 2
Pr X 6n ln n  2e 3 = .
2 2 n

This shows that the concentrations of the number of heads around the mean n/2
is very
p tight. Most of the time, the deviations from the mean are of the order
O( n ln n). Another simple calculation using the Chernoff bounds shows that
✓ ◆
n n n
Pr X  2e 24 .
2 4

This is a much better bound than the bound provided by the Chebyshev inequality:
✓ ◆
n n 4
Pr X  .
2 4 n

Ross [11] and Mitzenmacher and Upfal [10] consider the situation where a gam-
bler is equally likely to win or lose one unit on every play. Assuming that these
random variables Xi are independent, and that
1
Pr(Xi = 1) = Pr(Xi = 1) = ,
2
let Sn = Âni=1 Xi be the gamblers’s winning after n plays. It is easy that to see that
the moment generating function of Xi is
336 5 An Introduction to Discrete Probability

et + e t
MXi (t) = .
2
Using a little bit of calculus, one finds that
t2
MXi (t)  e 2 .

Since the Xi are independent, we obtain

n nt 2
MSn (t) = ’ MXi (t) = (MXi (t))n  e 2 , t > 0.
i=1

The Chernoff bound yields

nt 2 ta
Pr(Sn a)  e 2 , t > 0.

The minimum is achieved for t = a/n, and assuming that a > 0, we get
a2
P(Sn a)  e 2n , a > 0.

For example, if a = 6, we get

36
Pr(S10 6)  e 20 ⇡ 0.1653.

We leave it as exercise to prove that

56
Pr(Sn 6) = Pr(gambler wins at least 8 of the first 10 games) = ⇡ 0.0547.
1024
Other examples of the use of Chernoff bounds can be found in Mitzenmacher
and Upfal [10] and Ross [12]. There are also inequalities giving a lower bound on
the probability Pr(X > 0), where X is a nonnegative random variable; see Ross [12]
(Chapter 3), which contains other techniques to find bounds on probabilities, and
the Poisson paradigm. Probabilistic methods also play a major role in Motwani and
Raghavan [9].

5.9 Summary

This chapter provides an introduction to discrete probability theory. We define prob-

ability spaces (finite and countably infinite) and quickly get to random variables. We
emphasize that random variables are more important that their underlying probabil-
ity spaces. Notions such as expectation and variance help us to analyze the behavior
of random variables even if their distributions are not known precisely. We give a
5.9 Summary 337

number of examples of computations of expectations, including the coupon collec-

tor problem and a randomized version of quicksort.
The last three sections of this chapter contain more advanced material and are
optional. The topics of these optional sections are generating functions (including
the moment generating function and the characteristic function), the limit theorems
(weak law of lage numbers, central limit theorem, and strong law of large numbers),
and Chernoff bounds.
• We define: a finite discrete probability space (or finite discrete sample space),
outcomes (or elementary events), and events.
• a probability measure (or probability distribution) on a sample space.
• a discrete probability space.
• a s -algebra.
• independent events.
• We discuss the birthday problem.
• We give examples of random variables.
• We present a randomized version of the quicksort algorithm.
• We define: random variables, and their probability mass functions and cumula-
tive distribution functions.
• absolutely continuous random variables and their probability density functions.
• We give examples of: the binomial distribution.
• the geometric distribution.
• We show how the Poisson distribution arises as the limit of a binomial distribu-
tion when n is large and p is small.
• We define a conditional probability.
• We present the “Monty Hall Problem.”
• We introduce probability trees (or trees of possibilities).
• We prove several of Bayes’ rules.
• We define: the product of probability spaces.
• independent random variables.
• the joint mass function of two random variables, and the marginal mass func-
tions.
• the expectation (or expected value, or mean) E(X) = µ of a random variable X.
• We prove the linearity of expectation.
• We introduce indicator functions (indicator variables).
• We define functions of a random variables.
• We compute the expected value of the number of comparsions in the random-
ized version of quicksort.
• We define the variance
p Var(X) of a random variable X and the standard devia-
tion s of X by s = Var(X).
• We prove that Var(X) = E(X 2 ) (E(X))2 .
• We define the moments and the central moments of a random variable.
• We prove that if X and Y are uncorrelated random variables, then Var(X +Y ) =
Var(X) + Var(Y ); in particular, this equation holds if X and Y are independent.
• We prove: the Cauchy-Schwarz inequality for discrete random variables.
• Cheybyshev’s inequality and give some of its applications.
338 5 An Introduction to Discrete Probability

The next three sections are optional.

• We state the weak law of large numbers (Bernoulli’s theorem).
• We define the normal distribution (or Gaussian distribution).
• We sate the central limit theorem and present an application.
• We define various notions of convergence, including almost sure convergence
and convergence in probability.
• We state Kolmogorov’s strong law of large numbers.
• For a random variable that takes nonnegative integer values, we define the prob-
ability generating function, GX (z) = E(zX ). We show how the derivatives of GX
at z = 1 can be used to compute the mean µ and the variance of X.
• If X and Y are independent random variables, then GX+Y = GX GY .
• We define the moment generating function MX (t) = E(etX ) and show that
(n)
MX (0) = E(X n ).
• If X and Y are independent random variables, then MX+Y = MX MY .
• We define: the cumulants of X.
• the characteristic function jX (t) = E(eitX ) of X and discuss some of its proper-
ties. Unlike the moment generating function, jX is defined for all t 2 R.
• If X and Y are independent random variables, then jX+Y = jX jY . The distribu-
tion of a random variable is uniquely determined by its characteristic function.
• We prove: Markov’s inequality.
• the general Chernoff bounds in terms of the moment generating function.
• We compute Chernoff bound for various distributions, including normal and
Poisson.
• We obtain Chernoff bounds for Poisson trials (independent Bernoulli trials with
success probability pi ).

Problems

References

1. Joseph Bertrand. Calcul des Probabilités. New York, NY: Chelsea Publishing Company,
third edition, 1907.
2. Pierre Brémaud. Markov Chains, Gibbs Fields, Monte Carlo Simulations, and Queues. TAM
No. 31. New York, NY: Springer, third edition, 2001.
3. William Feller. An Introduction to Probability Theory and its Applications, Vol. 1 . New
York, NY: Wiley, third edition, 1968.
4. William Feller. An Introduction to Probability Theory and its Applications, Vol. 2 . New
York, NY: Wiley, second edition, 1971.
5. Ronald L. Graham, Donald E. Knuth, and Oren Patashnik. Concrete Mathematics: A Foun-
dation For Computer Science. Reading, MA: Addison Wesley, second edition, 1994.
6. Geoffrey Grimmett and David Stirzaker. Probability and Random Processes. Oxford, UK:
Oxford University Press, third edition, 2001.
7. Pierre–Simon Laplace. Théorie Analytique des Probabilités, Volume I. Paris, France: Editions
Jacques Gabay, third edition, 1820.
References 339

8. Pierre–Simon Laplace. Théorie Analytique des Probabilités, Volume II. Paris, France: Edi-
tions Jacques Gabay, third edition, 1820.
9. Rajeev Motwani and Prabhakar Raghavan. Randomized Algorithms. Cambridge, UK: Cam-
bridge University Press, first edition, 1995.
10. Michael Mitzenmacher and Eli Upfal. Probability and Computing. Randomized Algorithms
and Probabilistic Analysis. Cambridge, UK: Cambridge University Press, first edition, 2005.
11. Sheldon Ross. A First Course in Probability. Upper Saddle River, NJ: Pearson Prentice Hall,
eighth edition, 2010.
12. Sheldon Ross. Probability Models for Computer Science. San Diego, CA: Har-
court/Academic Press, first edition, 2002.
13. Albert Nikolaevich Shiryaev. Probability. GTM No. 95 . New York, NY: Springer, second
edition, 1995.
14. Santosh S. Venkatesh. The Theory of Probability: Explorations and Applications. Cambridge,
UK: Cambridge University Press, first edition, 2012.
U.C. Berkeley — CS276: Cryptography Handout 0.2
Luca Trevisan January, 2009

Notes on Discrete Probability

The following notes cover, mostly without proofs, some basic notions and results of
discrete probability. They were written for an undergraduate class, so you may find
them a bit slow.

1 Basic Definitions
In cryptography we typically want to prove that an adversary that tries to break a
certain protocol has only minuscule (technically, we say “negligible”) probability of
succeeding. In order to prove such results, we need some formalism to talk about the
probability that certain events happen, and also some techniques to make computa-
tions about such probabilities.
In order to model a probabilistic system we are interested in, we have to define
a sample space and a probability distribution. The sample space is the set of all
possible elementary events, i.e. things that can happen. A probability distribution is
a function that assigns a non-negative number to each elementary event, this number
being the probability that the event happen. We want probabilities to sum to 1.
Formally,

Definition 1 For a finite sample space ⌦ and a function P : ⌦ ! R, we say that P

is a probability distribution if

1. P(a) 0 for every a 2 ⌦;

P
2. a2⌦ P(a) = 1.

For example, if we want to model a sequence of three coin flips, our sample space
will be {Head, T ail}3 (or, equivalently, {0, 1}3 ) and the probability distribution will
assign 1/8 to each element of the sample space (since each outcome is equally likely).
If we model an algorithm that first chooses at random a number in the range 1, . . . , 10200
and then does some computation, our sample space will be the set {1, 2, . . . , 10200 },
and each element of the sample space will have probability 1/10200 .
We will always restrict ourselves to finite sample spaces, so we will not remark it
each time. Discrete probability is the restriction of probability theory to finite sample
spaces. Things are much more complicated when the sample space can be infinite.

1
An event is a subset A ✓ ⌦ of the sample space. The probability of an event is defined
in the intuitive way
X
P[A] = P(a)
a2A

(Conventially, we set P[;] = 0.)

We use square brackets to remind us that now we are considering a di↵erent function:
while P(·) is a function whose inputs are elements of the sample space, P[·] is a
function whose inputs are subsets of the sample space.
For example, suppose that we want to ask what is the probability that, when flipping
three coins, we get two heads. Then ⌦ = {0, 1}3 , P(a) = 1/8 for every a 2 ⌦,
we define A as the subset of {0, 1}3 containing strings with exactly two 1s, and we
ask what is P[A]. As it turns out, A has 3 elements, that is 011, 101, 110, and so
P[A] = 3/8. Very often, as in this example, computing the probability of an event
reduces to counting the number of elements of a set.
When P(·) assigns the same value 1/|⌦| to all the elements of the sample space, then
it is called the uniform distribution over ⌦.
The following distribution arises very often, and it is good to know about it. Consider
the situation where you flip n biased coins, and the probability that each coin turns
out head is p (where 0  p  1 is some fixed parameter). The outcome of each flip
is independent of all the other outcomes. Then the sample space is ⌦ = {0, 1}n ,
identifying heads with 1s; the probability distribution is

P(a) = pk (1 p)n k
where k is the number of 1s in a

When p = 1/2 then we have the uniform distribution over {0, 1}n . If p = 0 then all a
have probability zero, except 00 · · · 0, which has probability one. (Similarly if p = 1.)
The other cases are more interesting.
These distributions are called Bernoulli distributions or binomial distributions.
If we have a binomial distribution with parameter p, and we ask what is the probability
of the event Ak that we get a string with k ones, then such a probability is
✓ ◆
n k
P[Ak ] = p (1 p)n k
k

2 Random Variables and Expectation

Very often, when studying a probabilistic system (say, a randomized algorithm) we are
interested in some values that depend on the elementary event that takes place. For

2
example, when we play dice, we are interested in the probabilistic system where two
dice are rolled, and the sample space is {1, 2, . . . , 6}2 , with the uniform distribution
over the 36 elements of the sample space, and we are interested in the sum of the
outcomes of the two dice. Similarly, when we study a randomized algorithm that
makes some internal random choices, we are interested in the running time of the
algorithm, or in its output. The notion of a random variable gives a tool to formalize
questions of this kind.
A random variable X is a function X : ⌦ ! V where ⌦ is a sample space and V is
some arbitrary set (V is called the range of the random variable). One should think
of a random variable as an algorithm that on input an elementary event returns some
output. Typically, V will either be a subset of the set of real numbers or of the set
of binary strings of a certain length.
Let ⌦ be a sample space, P a probability distribution on ⌦ and X be a random
variable on ⌦. If v is in the range of X, then the expression X = v denotes an event,
namely the event {a 2 ⌦ : X(a) = v}, and thus the expression P[X = v] is well
defined, and it is something interesting to try to compute.
Let’s look at the example of dice. In that case, ⌦ = {1, . . . , 6}2 , for every (a, b) 2 ⌦
we have P(a, b) = 1/36. Let us define X as the random variable that associates a + b
to an elementary event (a, b). Then the range of X is {2, 3, . . . , 12}. For every element
of the range we can compute the probability that X take such value. By counting
the number of elementary events in each event we get

P[X = 2] = 1/36 , P[X = 3] = 2/36 , P[X = 4] = 3/36

P[X = 5] = 4/36 , P[X = 6] = 5/36 , P[X = 7] = 6/36
and the other probabilities can be computed by observing that
P[X = v] = P[X = 14 v]

It is possible to define more than one random variable over the same sample space,
and consider expressions more complicated than equalities.
When the range of a random variable X is a subset of the real numbers (e.g. if X
is the running time of an algorithm — in which case the range is even a subset of
the integers) then we can define the expectation of X. The expectation of a random
variable is a number defined as follows.
X
E[X] = vP[X = v]
v2V

where V is the range of X. We can assume without loss of generality that V is finite,
so that the expression above is well defined (if it were an infinite series, it could
diverge or even be undefined).

3
Expectations can be understood in terms of betting. Say that I am playing some
game where I have a probability 2/3 of winning, a probability 1/6 of losing and a
probability 1/6 of a draw. If I win, I win $ 1; if I lose I lose $ 2; if there is a draw I
do not win or lose anything. We can model this situation by having a sample space
{L, D, W } with probabilities defined as above, and a random variable X that specifies
my wins/losses. Specifically X(L) = 2, X(D) = 0 and X(W ) = 1. The expectation
of X is

1 1 2 1
E[X] = · ( 2) + · 0 + · 1 =
6 6 3 3
so if I play this game I “expect” to win $ 1/3. The game is more than fair on my
side.
When we analyze a randomized algorithm, the running time of the algorithm typically
depends on its internal random choices. A complete analysis of the algorithm would
be a specification of the running time of the algorithm for each possible sequence of
internal choices. This is clearly impractical. If we can at least analyse the expected
running time of the algorithm, then this will be just a single value, and it will give
useful information about the typical behavior of the algorithm (see Section 4 below).
Here is a very useful property of expectation.

Theorem 2 (Linearity of Expectation) Let X be a random variable and a be

real; then E[aX] = aE[X]. Let X1 , . . . , Xn be random variables over the same sample
space; then E[X1 + · · · + Xn ] = E[X1 ] + · · · E[Xn ].

Example 3 Consider the following question: if we flip a coin n times, what is the
expected number of heads? If we try to answer this question without using the linearity
of expectation we have to do a lot of work. Define ⌦ = {0, 1}n and let P be the uniform
distribution; let X be the random variable such that X(a) = the number of 1s in a 2 ⌦.
Then we have, as a special case of Bernoulli distribution, that
✓ ◆
n
P[X = k] = 2 n
k
In order to compute the average of X, we have to compute the sum

n ✓ ◆
X n n
k2 (1)
k=0
k
which requires quite a bit of ingenuity. We now show how to solve Expression (1) just
to see how much work can be saved by using the linearity of expectation. An inspection
of Expression (1) shows that it looks a bit like the expressions that one gets out of the

4
Binomial Theorem, except for the presence of k. In fact it looks pretty much like the
derivative of an expression coming from the Binomial Theorem (this is a standard
trick). Consider (1/2 + x)n (we have in mind to substitute x = 1/2 at some later
point), then we have
✓ ◆n X n ✓ ◆
1 n
+x = 2 (n k) xk
2 k=0
k
and then n ✓ ◆
d((1/2 + x)n ) X n (n k)
= 2 kxk 1
dx k=0
k
but also ✓ ◆n 1
d((1/2 + x)n ) 1
=n +x
dx 2
and putting together
n ✓ ◆
X ✓ ◆n 1
n (n k) k 1 1
2 kx =n +x .
k=0
k 2

Now we substitute x = 1/2, and we have

Xn ✓ ◆
n
2 (n k)
k2 (k 1)
=n.
k=0
k

Here we are: dividing by 2 we get

n ✓ ◆
X n n n
k2 = .
k=0
k 2

So much for the definition of average. Here is a better route: we can view X as the
sum of n random variables X1 , . . . , Xn , where Xi is 1 if the i-th coin flip is 1 and Xi
is 0 otherwise. Clearly, for every i, E[Xi ] = 12 · 0 + 12 · 1 = 12 , and so
n
E[X] = E[X1 + · · · + Xn ] = E[X1 ] + · · · + E[Xn ] = .
2

3 Independence

3.1 Conditioning and Mutual Independence

Suppose I toss two coins, without letting you see the outcome, and I tell you that
at least one of the coins came up heads, what is the probability that both coin are
heads?

5
In order to answer to this question (I will give it away that the answer is 1/3), one
needs some tools to reason about the probability that a certain event holds given (or
conditioned on the fact) that a certain other event holds.
Fix a sample space ⌦ and a probability distribution P. Suppose we are given that a
certain event A ✓ ⌦ holds. Then the probability of an elementary event a given the
fact that A holds (written P(a|A)) is defined as follows: if a 62 A, then it is impossible
that a holds, and so P(a|A) = 0; otherwise, if a 2 A, then P(a|A) has a value that is
proportional to P(a). One realizes that the factor of proportionality has to be 1/P[A],
so that probabilities sum to 1 again. Our definition of conditional probability of an
elementary event is then
(
0 If a 62 A
P(a|A) = P(a)
P[A]
Otherwise

The above formula already lets us solve the question asked at the beginning of this
section. Notice that probabilities conditioned on an event A such that P[A] = 0 are
undefined.
Then we extend the definition to arbitrary events, and we say that for an event B
X
P[B|A] = P(b|A)
b2B

One should check that the following (more standard) definition is equivalent

P[A \ B]
P[B|A] =
P[A]

Definition 4 Two events A and B are independent if

P[A \ B] = P[A] · P[B]

If A and B are independent, and P[A] > 0, then we have P[B|A] = P[B]. Similarly, if
A and B are independent, and P[B] > 0, then we have P[A|B] = P[A]. This motivates
the use of the term “independence.” If A and B are independent, then whether A
holds or not is not influenced by the knowledge that B holds or not.
When we have several events, we can define a generalized notion of independence.

Definition 5 Let A1 , . . . , An ✓ ⌦ be events is a sample space ⌦; we say that such

events are mutually independent if for every subset of indices I ✓ {1, . . . n}, I 6= ;,
we have

6
 \ Y
P[ Ai ] = P[Ai ]
i2I i2I

All this stu↵ was just to prepare for the definition of independence for random vari-
ables, which is a very important and useful notion.

Definition 6 If X and Y are random variables over the same sample space, then we
say that X and Y are independent if for any two values v, w, the event (X = v) and
(Y = w) are independent.

Therefore, if X and Y are independent, knowing the value of X, no matter which

value it is, does not tell us noting about the distribution of Y (and vice versa).

Theorem 7 If X and Y are independent, then E[XY ] = E[X]E[Y ].

This generalizes to several random variables

Definition 8 Let X1 , . . . , Xn be random variables over the same sample space, then
we say that they are mutually independent if for any sequence of values v1 , . . . , vn ,
the events (X1 = v1 ), . . . , (Xn = vn ) are mutually independent.

Theorem 9 If X1 , . . . , Xn are mutually independent random variables, then

E[X1 · X2 · · · Xn ] = E[X1 ] · E[X2 ] · · · E[Xn ]

3.2 Pairwise Independence

It is also possible to define a weaker notion of independence.

Definition 10 Let X1 , . . . , Xn be random variables over the same sample space, then
we say that they are pairwise independent if for every i, j 2 {1, . . . , n}, i 6= j, we have
that Xi and Xj are independent.

It is important to note that a collection of random variables can be pairwise indepen-

dent without being mutually independent. (But a collection of mutually independent
random variables is always pairwise independent for a stronger reason).

7
Example 11 Consider the following probabilistic system: we toss 2 coins, and we let
the random vaiables X, Y, Z be, respectively, the outcome of the first coin, the outcome
of the second coin, and the XOR of the outcomes of the two coins (as usual, we inter-
pret outcomes of coins as 0/1 values). Then X, Y, Z are not mutually independent,
for example

P[Z = 0|X = 0, Y = 0] = 1

while

P[Z = 0] = 1/2

in fact, intuitively, since the value of Z is totally determined by the values of X

and Y , the three variables cannot be mutually independent. On the other hand, we
will now show that X, Y, Z are pairwise independent. By definition, X and Y are
independent, so we have to focus of X and Z and on Y and Z. Let us prove that X
and Z are independent (the proof for Y and Z is identical). We have to show that
for each choice of two values v, w 2 {0, 1}, we have

1
P[X = v, Z = w] = P[X = v]P[Z = w] =
4
and this is true, since, in order to have Z = w and X = v, we must have Y = w v,
and the event that X = v and Y = w v happens with probability 1/4.

Let us see two additional, more involved, examples.

Example 12 Suppose we flip k coins, whose outcomes be a1 , . . . , an 2 {0, 1}k . Then

k
L every non-empty subset I ✓ {0, 1} we define a random variable XI , whose value is
for
i2I ai . It is possible to show that {XI }I✓{0,1}k ,I6=; is a pairwise independent collection
of random variables. Notice that we have 2k 1 random variables defined over a sample
space of only 2k points.

Example 13 Let p be a prime number; suppose we pick at random two elements

a, b 2 zp — that is, our sample space is the set of pairs (a, b) 2 zp ⇥ zp = ⌦, and we
consider the uniform distribution over this sample space. For every z 2 zp , we define
one random variable Xz whose value is az + b (mod p). Thus we have a collection
of p random variables. It is possible to show that such random variables are pairwise
independent.

8
4 Deviation from the Expectation

4.1 Markov’s Inequality

Say that X is the random variable expressing the running time in seconds of an
algorithm on inputs of a certain size, and that we computed E[X] = 10. Since
this is the order of magnitude of the time that we expect to spend while running the
algorithm, it would be devastating if it happened that, say, X 1, 000, 000 (i.e. more
than 11 days) with large probability. However, we quickly realize that if E[X] = 10,
then it must be P[X 1, 000, 000]  1/100, 000, as otherwise the contribution to the
expectation of the only events where X 1, 000, 000 would already exceed the value
10. This reasoning can be generalized as follows.

Theorem 14 (Markov’s Inequality) If X is a non-negative random variable then

E[X]
P[X k] 
k

Sometimes the bound given by Markov’s inequality are extremely bad, but the bound
is as strong as possible if the only information that we have is the expectation of X.
For example, suppose that X counts the number of heads in a sequence of n coin
flips. Formally, ⌦ is {0, 1}n with the uniform distribution, and X is the number of
ones in the string. Then E[X] = n/2. Suppose we want to get an upper bound on
P[X n] using Markov. Then we get

E[X] 1
P[X n]  =
n 2
This is ridiculous! The right value is 2 n , and the upper bound given by Markov’s
inequality is totally o↵, and it does not even depend on n.
However, consider now the experiment where we flip n coins that are glued together,
so that the only possible outcomes are n heads (with probability 1/2) and n tails
(with probability 1/2). Define X again as the number of heads. We still have that
E[X] = n/2, and we can apply Markov’s inequality as before to get

E[X] 1
P[X n]  =
n 2
But, now, the above inequality is tight, becuase P[X n] is precisely 1/2.
The moral is that Markov’s inequality is very useful because it applies to every non-
negative random variables having a certain expectation, so we can use it without
having to study our random variable too much. On the other hand, the inequality

9
will be accurate when applied to a random variable that typically deviates a lot from
its expectation (say, the number of heads that we get when we toss n glued coins)
and the inequality will be very bad when we apply it to a random variable that
is concentrated around its expectation (say, the number of heads that we get in n
independent coin tosses). In the latter case, if we want accurate estimations we have
to use more powerful methods. One such method is described below.

4.2 Variance
For a random variable X, the random variable

X 0 = |X E[X]|
gives all the information that we need in order to decide whether X is likely to deviate
a lot from its expectation or not. All we need to do is to prove that X 0 is typically
small. However this idea does not lead us very far (analysing X 0 does not seem to be
any easier than analysing X).
Here is a better tool. Consider

(X E[X])2
This is again a random variable that tells us how much X deviates from its expec-
tation. In particular, if the expectation of such an auxliary random variable is small,
then we expect X to be typically close to its expectation. The variance of X is defined
as

Var(X) = E[(X E[X])2 ]

Here is an equivalent epxression (we use linearity of expectation in the derivation of

the final result):

Var(X) = E[(X E[X])2 ]

= E[X 2 2XE[X] + (E[X])2 ]
= E[X 2 ] 2E[XE[X]] + (E[X])2
= E[X 2 ] 2E[X]E[X] + (E[X])2
= E[X 2 ] (E[X])2

The variance is a useful notion for two reasons: it is often easy to compute and it
gives rise to sometimes strong estimations on the probability that a random variable
deviates from its expectation.

10
Theorem 15 (Chebyshev’s Inequality)

Var(X)
P[|X E[X]| k] 
k2

The proof uses Markov’s inequality and a bit of ingenuity.

P[|X E[X]| k] = P[(X E[X])2 k 2 ]

E[(X E[X])2 ]

k2
Var(X)
=
k2

The nice idea is in the first step. The second step is just an application of Markov’s
inequality and the last step uses the definition of variance.
p
The value (X) = Var(X) is called the standard deviation of X. One expects the
value of a random variable X to be around the interval E[X] ± (X). We can restate
Chebyshev’s Inequality in terms of standard deviation

Theorem 16 (Chebyshev’s Inequality, Alternative Form)

1
P[|X E[X]| c · (X)] 
c2

Let Y be a random variable that is equal to 0 with probability 1/2 and to 1 with
probability 1/2. Then E[Y ] = 1/2, Y = Y 2 , and

1 1 1
Var(Y ) = E[Y 2 ] (E[Y ])2 = =
2 4 4
Let X the random variable that counts the number of heads in a sequence of n
independent coin flips. We have seen that E[X] = n/2. Computing the variance
according to the definition would be painful. We are fortunate that the following
result holds.

Lemma 17 (Tools to Compute Variance)

1. Let X be a random variable, a, b be reals, then

Var(aX + b) = a2 Var(X)

11
 2. Let X1 , . . . , Xn be pairwise independent random variables on the same sample
space. Then

Var(X1 + · · · + Xn ) = Var(X1 ) + · · · + Var(Xn )

Then we can view X as X1 + · · · + Xn where Xi are mutually independent random

variables such that for each i Xi takes value 1 with probability 1/2 and value 0 with
probability 1/2. As computed
p before, Var(Xi ) = 1/4. Therefore Var(X) = n/4 and
the standard deviation
p is n/2. This means that when we flip n coins we expect to
get about n ± n heads.
Let us test Chebyshev’s inequality on the same example of the previous subsection.
Let X be a random variable defined over ⌦ = {0, 1}n , where P is uniform, and
X counts the number of 1s in the elementary event: suppose we want to compute
P[X n]. As computed above, Var(X) = n/4, so

1
P[X n]  P[|X E[X]| n/2] 
n
This is still much less than the correct value 2 n , but at least it is a value that
decreases with n. It is also possible to show that Chebyshev’s inequality is as strong
as possible given its assumption.
Let n = 2k 1 for some integer k and let X1 , . . . , Xn be the collection of pairwise
independent random variables as defined in Example 12. Let X = X1 + . . . + Xn .
Suppose we want to compute P[X = 0]. Since each Xi has variance 1/4, we have that
X has variance n/4, and so

1
P[X = 0]  P[|X E[X]| n/2] 
n
k
which is almost the right value: the right value is 2 = 1/(n + 1).

A Appendix

A.1 Some Combinatorial Facts

Consider a set ⌦ with n elements. ⌦ has 2n subsets (including the empty set and ⌦
itself).
n n
For every 0  k  n, ⌦ has k
subsets of k elements. The symbol k
is read “n
choose k” and is defined as

12
 ✓ ◆
n n!
=
k k!(n k)!

Then we must have

n ✓ ◆
X n
= 2n (2)
k=0
k

which is a special case of the following result

Theorem 18 (Binomial Theorem) For every two reals a, b and non-negative in-
teger n,

n ✓ ◆
X
n n
(a + b) = ak b n k

k=0
k

We can see that Equation (2) follows from the Binomial Theorem by simply substi-
tuting a = 1 and b = 1.
Sometimes we have to deal
Pn with summations of the form 1 + 1/2 + 1/3 + . . . + 1/n.
It’s good to know that k=1 1/k ⇡ ln n. More precisely
Pn
1/k
Theorem 19 limn!1 k=1
ln n
= 1.
Pn Pn
In particular, k=1 1/k  1 + ln n for every n, and k=1 1/k ln n for sufficiently
large n.
The following inequality is exceedingly useful in computing upper bounds of proba-
bilities of events:

1 + x  ex (3)

This is easy to prove by looking at the Taylor series of ex :

1 1
ex = 1 + x + x2 + . . . + xk + . . .
2 k!
Observe that Equation (3) is true for every real x, not necessarily positive (but it
becomes trivial for x < 1).
Here is a typical application of Equation (3). We have a randomized algorithm that
has a probability ✏ over its internal coin tosses of succeeding in doing something (and

13
when it succeeds, we notice that it does, say because the algorithm is trying to invert
a one-way function, and when it succeeds then we can check it efficiently); how many
times do we have to run the algorithm before we have probability at least 3/4 that
the algorithm succeeds?
The probability that it never succeeds in k runs is

(1 ✏)k  e ✏k

2
If we choose k = 2/✏, the probability of k consecutive failures is less than e < 1/4,
and so the probability of succeeding (at least once) is at least 3/4.

A.2 Examples of Analysis of Error Probability of Algorithms

Example 20 Suppose that we have an algorithm whose worst-case running time (on
inputs of a certain length) is bounded by a random variable T (whose sample space is
the set of random choices made by the algorithm). For concreteness, suppose that we
are considering the randomized algorithm that given a prime p and an element a 2 Z⇤p
decides whether a is a quadratic residue or not. Suppose that we are given t = E[T ]
but no additional information on the algorithm, and we would like to know how much
time we have to wait in oder to have a probability at least 1 10 6 that the algorithm
terminates. If we only know E[T ], then we can just use Markov’s inequality and say
that

1
P[T kt] 
k
and if we choose k = 106 we have that

P[T 106 t]  10 6
.

However there is a much faster way of guaranteeing termination with high probability.
We let the program run for 2t time. There is a probability 1/2 that the algorithm will
stop before that time. If so we are happy. If not, we terminate the computation, and
start it over (in the second iteration, we let the algorithm use independent random
bits). If the second computation does not terminate within 2t time, we reset it once
more, and so on. Let T 0 be the random variable that gives the time taken by this
new version of the algorithm (with the stop and reset actions). Now we have that
the probability that we use more than 2kt time is equal to the probability that for k
consecutive (independent) times the algorithm takes more than 2t time. Each of these
events happen with probability at most 1/2, and so

P[T 0 2kt]  2 k

14
and if take k = 20, the probability is less than 10 6 , and the time is only 40t rather
than 1, 000, 000t.
Suppose that t = t(n) is the average running time of our algorithm on inputs of
length n, and that we want to find another algorithm that finishes always in time
t0 (n) and that reports a failure only with negligible probability, say with probability at
most n log n . How large do we have choose t0 , and what the new algorithm should be
like?
If we just put a timeout t0 on the original algorithm, then we can use Markov’s in-
equality to say that t0 (n) = nlog n t(n) will suffice, but now t0 is not polynomial in
n (even if t was). Using the second method, we can put a timeout 2t and repeat
the algorithm (log n)2 times. Then the failure probability will be as requested and
t0 (n) = 2(log n)2 t(n).

If we know how the algorithm works, then we can make a more direct analysis.

Example 21 Suppose that our goal is, given n, to find a number 2  a  n 1 such
that gcd(a, n) = 1. To simplify notation, let l = ||n|| ⇡ log n be the number of digits
of n in binary notation (in a concrete application, l would be a few hundreds). Our
algorithm will be as follows:

• Repeat no more than k times:

1. Pick uniformly at random a 2 {2, . . . , n 1};

2. Use Euclid’s algorithm to test whether gcd(a, n) = 1.
3. If gcd(a, n) = 1 then output a and halt.

• Output “failure”.

We would like to find a value of k such that the probability that the algorithm reports
a failure is negligible in the size of the input (i.e. in l).
At each iteration, the probability that algorithm finds an element that is coprime with
n is

(n) 1 1
=
n 2 6 log log n 6 log l
So the probability that there is a failure in one iteration is at most
✓ ◆
1
1
6 log l

15
and the probability of k consecutive independent failures is at most
✓ ◆k
1 k/6 log l
1 e
6 log l

if we set k = (log l)3 then the probability of k consecutive failures is at most

(log l)3 /6 log l (log l)/6

e =l
that is negligible in l.

16