1

CSCI 104
Discrete Probability

Aaron Cote
Mark Redekopp'
 2

Key Ideas In this Unit

• Basic probability calculation
• Conditional probability and its definition
– Law of total probability
– Definition of (mutual and pairwise) independence
• Bernoulli trials and binomial distribution
• Bayes Theorem
• Random Variables and Expected Value
• Linearity of Expectation
• Geometric Distribution
 3

Definitions
• A trial is a procedure that yields one of a set of possible outcomes. A
sequence of trials is an experiment.
– Flipping a coin is a trial.
• The sample space is the set of possible outcomes
– {heads, tails, on edge}
• An event is a subset of the sample space
– {heads, on edge}
• You roll two dice and are wondering if you’ll get snake eyes (two 1’s).
• What is the trial?

• What is the sample space? Sample Space

E1 E2
• What is the event?

• What is the probability of the event?

E3
 4

Definitions
• A trial is a procedure that yields one of a set of possible outcomes. A
sequence of trials is an experiment.
– Flipping a coin is a trial.
• The sample space is the set of possible outcomes
– {heads, tails, on edge}
• An event is a subset of the sample space
– {heads, on edge}
• You roll two dice and are wondering if you’ll get snake eyes (two 1’s).
• What is the trial?
– Rolling two dice
• What is the sample space?
– {1,2,3,4,5,6} x {1,2,3,4,5,6}
• What is the event?
– Snake eyes (1,1)
• What is the probability of the event?
– 1/36
 5

The Uniform Distribution

• In the previous problem, we assumed something.
1
• The uniform distribution assigns the probability
𝑛
to each of the n elements in the sample space.
– An unbiased coin assigns 1Τ2 to both heads and tails
– An unbiased die assigns 1Τ6 to each side.
– Two unbiased dice assigns 1Τ36 to each result.
• If S is a sample space of equally likely outcomes,
and E is an event of S, then the probability of E is
|𝐸|
–𝑝 𝐸 =
|𝑆|
– To compute 𝑝 𝐸 use counting techniques to find |E|
and |S|
 6

Practice 1
• What is the probability that the sum of two dice rolls is 7?

• How many 5-card Poker hands are there?

• How many 5-card Poker hands contain a 4-of-a-kind?

• What is the probability of a 4-of-a-kind?

• What is the probability of a full house?

 7

Practice 1 Solution
• What is the probability that the sum of two dice rolls is 7?
6 1
➢ =
36 6
• How many 5-card Poker hands are there?
52
➢ 5
• How many 5-card Poker hands contain a 4-of-a-kind?
➢ 13 ∙ 48
• What is the probability of a 4-of-a-kind?
13 ∙ 48
➢ 52 = 0.024%
5
• What is the probability of a full house?
4 4
𝑃 ∙
13 2 3 2
➢ 52
5
 8

Inverting Probabilities
• You have a sequence of 10 bits, generated
by the uniform distribution. What is the
probability that at least one bit is a 0?
E

• Given event E in sample space S, the

probability of the complement of E is:

𝑝 𝐸ത = 1 − 𝑝(𝐸)

E
 9

Inverting Probabilities
• You have a sequence of 10 bits, generated by
the uniform distribution. What is the
probability that at least one bit is a 0?

➢1 − P all bits are 1 = 1 − 1Τ210

 10

The Subtraction Rule for Discrete Probabilities

• There is a line of 100 people.

• Everyone at an even position is a pirate.
• Everyone at a position divisible by 5 is a ninja
• You choose a person in the line according to the uniform
distribution.
• What is the probability they are a pirate or ninja (or both)?

➢ ______________________________

• p(E1  E2) = p(E1) + p(E2) – p(E1  E2)

• This is also known as the principle of inclusion-exclusion (aka PIE)

 11

The Subtraction Rule for Discrete Probabilities

• There is a line of 100 people.

• Everyone at an even position is a pirate.
• Everyone at a position divisible by 5 is a ninja
• You choose a person in the line according to the uniform
distribution.
• What is the probability they are a pirate or ninja (or both)?

1 1 1 3
➢ + − =
2 5 10 5

• p(E1  E2) = p(E1) + p(E2) – p(E1  E2)

• This is also known as the principle of inclusion-exclusion (aka PIE)

 12

Non-Uniform Distributions
• A biased coin is twice as likely to come up
heads as tails. What is the probability it
comes up heads?
➢2Τ3

• If S = {x1, …, xn} is the sample space, then

𝑛

෍ 𝑝(𝑥𝑖 ) = 1 and ∀𝑖: 0 ≤ 𝑝(𝑥𝑖 ) ≤ 1

𝑖=1
 13

The Sum Rule for Probability

• A biased die has a 3 appear twice as often as
any other individual number. What is the
probability an odd number is rolled?
➢𝑃 1 + 𝑃 3 + 𝑃 5 = 1Τ7 + 2Τ7 + 1Τ7 = 4Τ7

• 𝑝 𝐸 = σ𝑠∈𝐸 𝑝(𝑠)
– provided all s are disjoint (mutually exclusive)
 14

CONDITIONAL PROBABILITY
 15

Conditional Probability
• For the same biased die, we know that an odd
number was rolled. What is the probability a 3
was rolled?
2Τ
➢ 4Τ
7
= 1Τ2 E|F
7
F E

• The conditional probability of E, given that F

occurred, is:
𝑝(𝐸 ∩ 𝐹)
𝑝 𝐸𝐹 =
𝑃(𝐹)
 16

Practice 2
There are 4 people in line. Each person is either
a pirate or a ninja (but not both), assigned by
the uniform distribution. What is the probability
there are two consecutive ninjas, given the first
person in line is a ninja?
𝑝(NN ∩ 1N)
𝑝 NN 1𝑁 =
𝑃(1N)
 17

Practice 2 Solution
There are 4 people in line. Each person is either
a pirate or a ninja (but not both), assigned by
the uniform distribution. What is the probability
there are two consecutive ninja, given the first
person in line is a ninja?
𝑝(NN ∩ 1N) Compute 𝑝 NN ∩ 1N :
𝑝 NN 1𝑁 =
𝑃(1N) NNPP
NNPN
𝑝(NN∩1N) NNNP
= 1
2 NNNN
NPNN
=
5 𝑝 NN ∩ 1N = 5ൗ16
8
 18

Practice 3

• A family has two children, the gender is generated u.a.r. (uniformly at

random). What is the probability of two boys, given the youngest is a boy?
➢ 1Τ2

• A family has two children, the gender is generated u.a.r. (uniformly at

random). What is the probability of two boys, given at least one boy?

• Intuition A: This is just the same problem, right? So it is 1Τ2?

• Intuition B: There are 3 possible events for "at least one boy": BG, GB, BB.
So it is 1Τ3 ?

• Intuition B is correct.
 19

For the Unconvinced

• Gather data from all U.S. families with exactly 2
children (lets say there are 10 million).
– 2.5 million families have two girls.
– 2.5 million families have two boys.
– 2.5 million families have BG.
– 2.5 million families have GB.
• Given there is at least one boy, there are 7.5 million
families we might have selected from. Of them, 1Τ3
of them have two boys.
 20

The Law of Total Probability

• For any partition of the sample space into disjoint events
𝐹1, … , 𝐹𝑘, where 𝐹1 ∪ … ∪ 𝐹𝑘 = 𝑆 and 𝐹1 ∩ … ∩ 𝐹𝑘 = ∅:

𝑝(𝐸) = 𝑝(𝐸|𝐹1) ∙ 𝑝(𝐹1) + … + 𝑝(𝐸|𝐹𝑘) ∙ 𝑝(𝐹𝑘)

• Because:
𝑝(𝐸∩𝐹1) 𝑝 𝐸∩𝐹𝑘
𝑝 𝐸 = ∙ 𝑝 𝐹1 + … + ∙ 𝑝 𝐹𝑘
𝑝(𝐹1) 𝑝 𝐹𝑘
= 𝑝 𝐸 ∩ 𝐹1 + … + 𝑝(𝐸 ∩ 𝐹𝑘)
F1 F2

F3
 21

The Monty Hall Three-Door Puzzle

• There are three doors.
– Behind one is a car.
– Behind each of the other two doors is a goat.
• You choose one door, but don’t open it.
• The gameshow host knows where the car is.
• The gameshow host intentionally chooses a door which
you did not choose, which does not have the car, and
opens it, revealing a goat.
• You can now open the door you originally chose, or the
other closed door.
• What should you do?
 22

Monty Hall
• Intuition A: there are two doors left, and each is
equally likely, so it doesn’t matter which door you
open.

• Intuition B: when you guessed which door to

open, you had a 1Τ3 chance of it having a car.
That is still the case, so the other doors have a 2Τ3
probability of a car. So, you should switch doors.

• Intuition B is correct. Why?

 23

Monty Hall Rephrased

• There are three doors.
• Behind one is a car.
• Behind each of the other two doors is a goat.
• You choose one door (door A), but don’t open it.
• The gameshow host knows where the car is.
• The gameshow host decides to give you a hint. He tells
you that if the car is behind door B or C, then it is not
behind door C. To drive the point home, he opens
door C to reveal a goat.
• By switching to door B, you effectively get to open two
doors and take the better of the prizes.
 24

Monty Hall for the unconvinced

• There are 100 doors.
• Behind one is a car.
• Behind each of the other 99 doors is a goat.
• You choose one door (door 1), but don’t open it.
• The gameshow host knows where the car is.
• The gameshow host opens 98 other doors, not
including your door, and not including the door with
the car.
• Does the probability you originally chose the correct
door magically increase from 1Τ100 to 1Τ2? (No)
 25

XKCD #1282: Monty Hall

 26

Indepedence
• Two events E and F are independent exactly
when:
➢𝑃(𝐸|𝐹) = 𝑃(𝐸) or equivalently when
➢𝑃 𝐸 ∩ 𝐹 = 𝑃 𝐸 ∙ 𝑃(𝐹)

𝑃(𝐸∩𝐹) 𝑃(𝐸∩𝐹)
• 𝑃 𝐸𝐹 = and = P(𝐸) only if
𝑃(𝐹) 𝑃(𝐹)
𝑃 𝐸 ∩ 𝐹 = 𝑃 𝐸 ∙ 𝑃(𝐹) to make
𝑃(𝐸 ∩ 𝐹) 𝑃 𝐸 P(𝐹)
= = 𝑃(𝐸)
𝑃(𝐹) 𝑃(𝐹)
 27

Practice 4
• A family has 2 children, the gender is determined u.a.r. The event E is when
there are two girls. F is when there is at least one girl. Are E and F
independent?

• The family has 3 children. E is when there is a boy and a girl. F is when there is
at most 1 boy. Are E and F independent?

• There are 4 people in line. Each person is a pirate or ninja (not both),
generated u.a.r. E is when the first person is a ninja. F is when there are an
even number of ninjas. Are E and F independent?
 28

Practice 4 Solution
• A family has 2 children, the gender is determined u.a.r. The event E is when
there are two girls. F is when there is at least one girl. Are E and F
independent?
1
– No. p E = 1Τ4 𝑝 𝐹 = 3Τ4 𝑝 𝐸 ∙ 𝑝 𝐹 = 3Τ16 𝑝 𝐸∩𝐹 =
4

• The family has 3 children. E is when there is a boy and a girl. F is when there is
at most 1 boy. Are E and F independent?
3
– Yes. p E = 3Τ4 𝑝 𝐹 = 1Τ2 𝑝 𝐸 ∙ 𝑝 𝐹 = 3Τ8 𝑝 𝐸∩𝐹 =
8

• There are 4 people in line. Each person is a pirate or ninja (not both),
generated u.a.r. E is when the first person is a ninja. F is when there are an
even number of ninjas. Are E and F independent?
1
– Yes. p E = 1Τ2 𝑝 𝐹 = 1Τ2 𝑝 𝐸 ∙ 𝑝 𝐹 = 1Τ4 𝑝 𝐸∩𝐹 =
4
 29

Practice 5
• You have a jar with 6 red marbles and 4 blue marbles.
You draw 3 of them from the jar, u.a.r.
➢ Let E be the event where not all marbles are the same
color.
➢ Let F be the event where the first marble drawn is red.
• Are E and F independent?
➢𝑝 𝐹 =

➢ 𝑝 𝐸ത = 𝑝 𝐸 =

➢𝑝 𝐸 ∩ 𝐹 =

• Not independent!
 30

Practice 5 Solution
• You have a jar with 6 red marbles and 4 blue
marbles. You draw 3 of them from the jar, u.a.r.
➢Let E be the event where not all marbles are the same
color.
➢Let F be the event where the first marble drawn is red.
• Are E and F independent?
➢𝑝 𝐹 = 6Τ10
6
+ 43
➢𝑝 𝐸ത = 3
10 = 1Τ5 𝑝 𝐸 = 4Τ5
3
➢𝑝 𝐸 ∩ 𝐹 = 3Τ5 ∙ 1 − 5Τ9 ∙ 4Τ8 = 13Τ30
• Not independent!
 31

Indendence of many events

• Events E1, … En are pairwise independent if
every pair of events are independent.
• They are mutually independent if, for any
subset of events, they are independent.
• Events F1, …, Fk are independent if
𝑝 𝐹1 ∩ … ∩ 𝐹𝑘 = 𝑝 𝐹1 ∙ . . . ∙ 𝑝(𝐹𝑘 )
 32

The Birthday Paradox

• What are the odds that two people watching this lecture share the
same birthday? Is it above or below 50%?
• Assume that any of the 366 possible birthdays are equally likely.
• The probability that among n people, two have the same birthday,
is:
366 365 364 367−𝑛
• 1 − ∙ ∙ ∙ …∙
366 366 366 366

• n = 10: 11.7%
• n = 23: 50.7%
• n = 30: 70.6%
• n = 50: 97%
• n = 70: 99.9%
• n=367: 100%
 33

Practice 6
• Consider 2 fair coin flips.
– E1 is the event where the first flip is heads.
– E2 is the event where the second flip is heads.
– E3 is the event where exactly one flip is heads.
• Are these pairwise independent? Mutually
independent?
– p(E1) = p(E2) = p(E3) = 1Τ2
– p(E1 ∩ E2) = p(E1 ∩ E3) = p(E2 ∩ E3) = 1Τ4
– p(E1 ∩ E2 ∩ E3) = 0
• Pairwise, but not mutual.
 34

Practice 7
• An integer between 1 and 8 inclusive is chosen
u.a.r.
– E1 is the event that the number is ≤ 4.
– E2 is the event that the number is odd.
– E3 is the event that the number is prime.
• Pairwise independent? Mutually independent?
– p(E1) = p(E2) = p(E3) =
– p(E1 ∩ E2 ∩ E3) =
– p(E2 ∩ E3) =
• Neither!
 35

Practice 7 Solution
• An integer between 1 and 8 inclusive is chosen
u.a.r.
– E1 is the event that the number is ≤ 4.
– E2 is the event that the number is odd.
– E3 is the event that the number is prime.
• Pairwise independent? Mutually independent?
– p(E1) = p(E2) = p(E3) = 1Τ2
– p(E1 ∩ E2 ∩ E3) = 1Τ8
– p(E2 ∩ E3) = 3Τ8
• Neither!
 36

Bernoulli Trials
• A Bernoulli Trial is a trial with 2 outcomes that
may or may not have equal probability. One
outcome is the success with probability p, the
other is the failure with probability q = 1-p.
• If you have an experiment of n identical
Bernoulli trials, the probability of getting
exactly k successes is:
𝒏
➢ 𝒌
𝒑𝒌 𝒒𝒏−𝒌
• This looks like the Binomial Theorem!
 37

Practice 8
• A biased coin has probability of heads 2Τ3.
Given 4 flips, what is the probability of exactly
2 heads?
➢

• There are 10 people in line. Each person has a

90% chance of being a pirate, and is otherwise
a ninja. What is the probability of exactly 8
pirates?
➢
 38

Practice 8 Solution
• A biased coin has probability of heads 2Τ3.
Given 4 flips, what is the probability of exactly
2 heads?
4 2 2
➢ 2
∙ 2Τ3 ∙ 1Τ3 = 30%
• There are 10 people in line. Each person has a
90% chance of being a pirate, and is otherwise
a ninja. What is the probability of exactly 8
pirates?
10
➢ 2
∙ (1Τ10)2 ∙ (9Τ10)8 = 19%
 39

Randomized Algorithms
• Suppose a group of size n got together yesterday, and they’re worried that they
contracted coronavirus during this gathering.
• Let’s say that enough social mingling occurred that if even one person had it, then
each person now has a 10% of having it now.
• You could test everyone from the gathering, but that may not be feasible if testing is
limited.
• Instead, you can test k of the members (chosen u.a.r.), and assert that if no one tests
positive, then no one contracted the virus at the gathering.
• If no one actually contracted the virus, what is the probability you will assert as
much?
– Define success as positive test: P(success)=0
– 𝑝 0 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑜𝑢𝑡 𝑜𝑓 𝑘 𝑡𝑒𝑠𝑡𝑠 = 𝑘0 00 (1)𝑘 = 1 (i.e. 100%)
• If someone did contract the virus, what is the probability you will incorrectly assert
that no one did?
1
– Now P(success)=
10
𝑘 1 9
– 𝑝 0 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑜𝑢𝑡 𝑜𝑓 𝑘 𝑡𝑒𝑠𝑡𝑠 = 𝑘
( )0 ( )𝑘
10 10
• If k = 66, this is < 0.1%. If k = 132, this is < 0.0001%
 40

Randomized Algorithms
• This was an example of a Monte Carlo
randomized algorithm: such an algorithm is
always fast, but sometimes returns the wrong
answer.
– When you get to them in CSCI 104, you will see that a
Bloom Filter is a Monte Carlo data structure.

• The other type of randomized algorithm is a Las

Vegas algorithm: it is always right, but sometimes
takes a long time.
– QuickSort, and HashTables, are examples of this type.
 41

The Prosecutor’s Fallacy

• You are accused of speeding.
• The prosecutor notes that you have a radar detector (it
detects when police are using their radar system which
identifies speeders).
• The prosecutor also notes that 80% of speeders have radar
detectors, so you are probably guilty.
• Is there any flaw in his reasoning?
• It could be that 1% of the population speeds, but 80% of all
drivers have radar detectors, for example. Now the same
argument indicates you probably aren’t guilty!
• The fallacy is mistaking p(E|F) for p(F|E)
– 80% of speeders have radar detectors => P(D|S)
– But the real question we need to answer is P(S|D)
 42

Simpson’s Paradox
• Alice and Bob are studying for the CSCI 104 midterm. They each do
25 practice problems, split over 2 types (graphs and probability).
• Bob does 5 graph problems, and gets 3 of them right.
• Alice does 20 graph problems, and gets 13 of them right.
• Bob does 20 probability problems and gets 15 of them right.
• Alice does 5 probability problems and gets 4 of them right.
• Bob got a 60% on graphs, Alice got a 65%
(Alice did better).
• Bob got a 75% on probability, Alice got an 80%
(Alice did better).
• Bob got 18 out of 25 problems right, Alice got 17 out of 25 right
(Bob did better overall).
 43

Simpson’s Paradox
• A new study promotes the benefits of a specific diet.
• 3 out of 8 of the residents of Roshar survive to age 70
on diet 1.
• 40 out of 90 of Roshar residents survive on diet 2.
• 60 out of 92 of the residents of Tyrea survive to age 70
on diet 1.
• 7 out of 10 of Tyrea residents survive on diet 2.
• 63% of people on diet 1 survive, vs. 47% on diet 2.
• This is not enough information to extol the benefits of
diet 1!
 44

Simpson’s Paradox
• 38% of the residents of Roshar survive to age 70
on diet 1.
• 44% of Roshar residents survive on the new diet
(diet 2).
• 65% of the residents of Tyrea survive to age 70 on
diet 1.
• 70% of Tyrea residents survive on the new diet.
• It looks like diet 2 is the better diet! The natural
longevity of Tyreans made diet 1 look better than
it was.
 45

BAYE'S THEOREM
 46

A motivating problem
• There are two boxes.
• Box 1 contains 2 gold coins and 7 cardinal coins
• Box 2 contains 4 gold coins and 3 cardinal coins
• You choose a box u.a.r.
• From your box, you choose a coin u.a.r.
• You draw a cardinal coin
• What is the probability the coin came from the first box?
 47

A motivating problem
• E is the event that you draw a cardinal coin.
• F is the event that you draw from box 1.
• Calculating p(E|F) is easy (7Τ9)
• We want to calculate p(F|E) (much harder!)
 48

Bayes Theorem
• Consider:
𝑝(𝐸∩𝐹) 𝑝(𝐹∩𝐸)
–𝑝 𝐹𝐸 = but also 𝑝 𝐸 𝐹 =
𝑝(𝐸) 𝑝(𝐹)
– 𝑝 𝐹 𝐸 ∙𝑝 𝐸 =𝑝 𝐸∩𝐹 =𝑝 𝐸 𝐹 ∙𝑝 𝐹
𝑝(𝐸|𝐹)∙𝑝(𝐹)
–𝑝 𝐹𝐸 =
𝑝(𝐸)

• Bayes’ Theorem:
𝑝(𝐸|𝐹)∙𝑝(𝐹)
–𝑝 𝐹𝐸 =
𝑝(𝐸)

– And often we will calculate E through disjoint sets to get:

𝑝(𝐸|𝐹)∙𝑝(𝐹)
–𝑝 𝐹𝐸 = ത ത
𝑝(𝐸|𝐹)∙𝑝 𝐹 +𝑝(𝐸|𝐹)∙𝑝( 𝐹)
 49

The Cardinal Coin Problem

• 𝑝 𝐹 = 𝑝 𝐹ത = 1Τ2
• 𝑝 𝐸 𝐹 = 7Τ9
• 𝑝 𝐸 𝐹ത = 3Τ7
7Τ ∙1Τ
• 𝑝 𝐹𝐸 = 9 2
7Τ ∙1Τ +3Τ ∙1Τ = 64%
9 2 7 2
 50

Practice 9
• Say that 10% of the population has coronavirus.
• 85% of those infected display a fever.
• 5% of those who are not infected display a fever
(from other causes)
• You display a fever. What is the probability you
have coronavirus?
• E = you display a fever. F = you have coronavirus.
• p E F = , 𝑝 𝐸 𝐹ത = ,𝑝 𝐹 =
• 𝑝 𝐹𝐸 =
 51

Practice 9 Solution
• Say that 10% of the population has coronavirus.
• 85% of those infected display a fever.
• 5% of those who are not infected display a fever
(from other causes)
• You display a fever. What is the probability you
have coronavirus?
• E = you display a fever. F = you have coronavirus.
• p E F = 85%, 𝑝 𝐸 𝐹ത = 5%, 𝑝 𝐹 = 10%
85%∙10%
• 𝑝 𝐹𝐸 = = 65%
85%∙10%+5%∙90%
 52

Oddities from Bayes’ Theorem

• Suppose there is a much rarer disease that infects 1 out of
100,000 people. You have a very accurate test for the
disease:
• If you have the disease, it correctly says so 99% of the time.
• If you don’t, it correctly says so 99.5% of the time.
• You take the test, and it comes back positive. What is the
probability you have the disease?
• E = test comes back positive, F = you have the disease.
• 𝑝 𝐸 𝐹 = 99%, 𝑝 𝐸 𝐹ത = 0.5%, 𝑝 𝐹 = 0.001%
99%∙0.001%
• 𝑝 𝐹𝐸 = = 0.2%? ? ?
99%∙0.001%+0.5%∙99.999%
 53

XKCD #1132
Frequentists vs.
Bayesians
 54

Bayes’ Theorem: Other Versions

• A simpler version:
𝑝(𝐸|𝐹)∙𝑝(𝐹)
• 𝑝 𝐹𝐸 =
𝑝(𝐸)
• A more complex version, for any partition of
the sample space
into distinct sets F1, …, Fk :
𝑝(𝐸|𝐹)∙𝑝(𝐹)
• 𝑝 𝐹𝐸 =
𝑝(𝐸|𝐹1)∙𝑝 𝐹1 + … +𝑝(𝐸|𝐹𝑘)∙𝑝 𝐹𝑘
 55

Monty Hall Revisited

• There are 3 doors. You choose door 1. The host then
opens door 2 to reveal a goat. What is the probability
the car is behind door 3?
• Fi = the car is behind door i. E = the host opens door 2.
• 𝑝 𝐸 𝐹3 = 1, 𝑝 𝐸 𝐹2 = 0, because the host never
opens the door with the car.
• 𝑝 𝐸 𝐹1 = 1Τ2, because the host can open door 2 or 3.
• p F𝑖 = 1Τ3
1∙1Τ3
• 𝑝 𝐹3 𝐸 = 1Τ ∙1Τ +0∙1Τ +1∙1Τ = 2Τ3
2 3 3 3
 56

RANDOM VARIABLES, EXPECTED

VALUE AND VARIANCE
 57

Definitions
• A Random Variable is a mapping from the sample space to the set of real
numbers.
• That is, every single outcome is assigned a number.
• If your trial is to flip 3 coins, there are 8 possible outcomes. You could
have a random variable X that calculates the number of heads for each
outcome: X(HHT) = 2, for example
• The distribution of a random variable is the probability of each possible
number.
• So for the above trial:

• 𝑝 𝑋 = 0 = 1Τ8 = 𝑝 X = 3
• 𝑝 X = 1 = 3Τ8 = 𝑝(X = 2)
 58

A Dice Problem
• You roll two fair dice. X(t) is a random variable that
outputs the sum of the dice. What is the distribution
of X?

• 𝑝 2 = 1Τ36 = 𝑝(12)
• 𝑝 3 = 2Τ36 = 𝑝 11
• 𝑝 4 = 3Τ36 = 𝑝(10)
• 𝑝 5 = 4Τ36 = 𝑝 9
• 𝑝 6 = 5Τ36 = 𝑝 8
• 𝑝 7 = 6Τ36
 59

Random Variables: Origins

• In 1940, two mathematicians were planning to
publish papers about random variables, and
they were trying to decide what to call them
(random variable or chance variable, each
preferred by one of the mathematicians).
• Given the subject matter, how do you suppose
they decided?
• They flipped a coin, and we’ve been stuck with
random variable ever since.
 60

Expectation
• The Expected Value E(X) of a random variable X is the
average value outputted by X. That is:
➢ 𝐸 𝑋 = σ𝑠∈𝑆 𝑝(𝑠) ∙ 𝑋(𝑠)
• What is the expected value of a fair die roll?

➢ 1Τ6 ∙ 1 + 1Τ6 ∙ 2 + 1Τ6 ∙ 3 + 1Τ6 ∙ 4 + 1Τ6 ∙ 5 + 1Τ6 ∙ 6 = 3.5

• Given n Bernoulli trials, each with probability of

success p, what is the expected number of successes?
➢𝑛 ∙ 𝑝
• Wait… why could we do that?
 61

Linearity of Expectations
• 𝐸 𝑋1 + ⋯ + 𝑋𝑛 = 𝐸 𝑋1 + ⋯ + 𝐸(𝑋𝑛)
• 𝐸 𝑎∙𝑋+𝑏 =𝑎∙𝐸 𝑋 +𝑏
• That is, if you want to calculate the expected number of successes
over n trials, you can calculate the expected number of successes
for the 1st trial, and the 2nd trial, and the 3rd trial, etc, and add
them all up.
• What is the expected value of the sum of 3 fair dice?
➢ 3 ∙ 3.5 = 10.5
• Professor Slacker hates grading, and just assigned each student a
grade (A, B, C, or D) u.a.r. If he had actually bothered to grade, he
would have given 25% of students each grade. What is the
expected percentage of students that got the correct grade?
➢ 25%
 62

Practice 10
• The ordered pair i,j is an inversion in a permutation of the
first n positive integers if i < j, but j precedes i in the
permutation.
• There are 6 inversions in 3, 5, 1, 4, 2
• 1,3, 1,5, 2,3, 2,4, 2,5, 4,5
• What is the max number of inversions in a list of n positive
integers?
𝑛∙(𝑛−1)
• 𝑛2 =
2
• What is the expected number of inversions in a random
permutation of the first n positive integers?
𝑛∙(𝑛−1)
•
4
 63

Expectation on Non-Independent Random

Variables
• You have two coins. One has heads on both
sides, the other has tails on both sides.
• You choose a coin u.a.r., and then you flip it
twice. What is the expected number of
heads?
➢1
• Linearity of Expectations holds, even if the
variables aren’t independent!
 64

Insertion Sort
• What is the expected number of comparisons made by Insertion
Sort?
• Let Xi be the number of comparisons to insert the i-th element.
• We want to calculate E(𝑋1 + … + 𝑋𝑛 )
• By Linearity of Expectations, we can just calculate E(𝑋1 ), …, E(𝑋𝑛 )
• E(𝑋1 ) = 0, because the first item is already in the correct position.
• 𝑋2 will always output 1, because we need to compare a1 and a2.
E(𝑋2 ) = 1
• 𝑋3 will always output at least 1.
– If we find it to be the largest element so far, it will only output 1.
– Otherwise we will do another comparison with the smallest element,
and it will output 2.
– There are 2 cases where we do 2 comparisons (smallest and 2nd
smallest)
– E(𝑋3 ) = 2Τ3 ∙ 2 + 1Τ3 ∙ 1 = 5Τ3
 65

Insertion Sort, Continued

• 𝐸 𝑋4 = 1Τ4 ∙ 1 + 1Τ4 ∙ 2 + 2Τ4 ∙ 3 = 9Τ4
• 𝐸 𝑋5 = 1Τ5 ∙ 1 + 1Τ5 ∙ 2 + 1Τ5 ∙ 3 + 2Τ5 ∙ 4 = 14Τ5
• 𝐸 𝑋6 = 20Τ6

k∙ k+1 −2
• 𝐸 𝑋𝑘 =
2k

𝑛 k+1 1
• σ𝑛k=1 𝐸(𝑋𝑘 ) = σk=1 − = 𝜃(𝑛2 )
2 2k
 66

Hirings
• n applicants are interviewed for a single job
opening, 1 applicant per day.
• The first applicant is automatically hired.
• For every day thereafter, if the current
applicant is better than the current employee,
then the applicant is hired and the employee
is fired.
• What is the expected number of hirings?
 67

Hirings Solution
• Let 𝐼𝑘 return 1 if the kth arriving person is hired, and 0
otherwise. We want:
• 𝐸(σ𝑛𝑘=1 𝐼𝑘 ) = σ𝑛𝑘=1 𝐸(𝐼𝑘 )
• E(𝐼1 ) = ___ (the first person is _____________).
• E(𝐼2 ) = ____ (there are __ Ways to (relatively) order the first
2 arrivers and only ____ of them has the best arriving
second)
• E(𝐼3 ) = ___ (there are ___ Ways to (relatively) order the first
3 arrivers and only __ of them have the best one last )
• …
1
• σ𝑛𝑘=1 𝐸(𝐼𝑘 ) = σ𝑛𝑘=1 = (log 𝑛)
𝑘
 68

Hirings Solution
• Let 𝐼𝑘 return 1 if the kth arriving person is hired, and 0
otherwise. We want:
• 𝐸(σ𝑛𝑘=1 𝐼𝑘) = σ𝑛𝑘=1 𝐸(𝐼𝑘 )
• E(𝐼1 ) = 1 (the first person is always hired).
1
• E(𝐼2 ) = (there are 2! Ways to (relatively) order the first 2
2
arrivers and only one of them has the best arriving second)
1
• E(𝐼3 ) = (there are 3! Ways to (relatively) order the first 3
3
arrivers and only 2! of them have the best one last )
• …
1
• σ𝑛𝑘=1 𝐸(𝐼𝑘 ) = σ𝑛𝑘=1 = (log 𝑛)
𝑘
 69

Hash Tables
• A hash table has k buckets and n items.
• Each item is distributed to buckets u.a.r., and independently from
each other.
• What is the expected number of empty buckets?
• We can figure this out by determining the probability a specific
bucket is empty.
• Let Bi be the event where bucket i is empty.
𝑘−1 𝑛
• p(Bi) = E(Bi) =
𝑘
• We want E(B1 + … + Bk)
𝑘−1 𝑛
• By Linearity of Expectations, this is k ∙
𝑘
• If n = k = 1000, this is about 368. If n =1000 and k = 10000, this is
about 9048.
 70

The Geometric Distribution

• A coin comes up heads with probability p. We will flip
the coin until it comes up heads. What is the expected
number of flips?
• pr(X=1) = p
• pr(X=2) = (1 − 𝑝) ∙ 𝑝
• pr(X=3) = 1 − 𝑝 2 ∙ 𝑝
𝑖−1 1
• 𝐸 𝑋 = σ𝑖=1 𝑖 ∙ 1 − 𝑝 ∙𝑝=
𝑝
• Because for |x| < 1:
1
• σ∞
𝑘=1 𝑘𝑥 𝑘−1 =
(1−𝑥)2
 71

The Geometric Distribution

• A random variable has the geometric distribution with parameter
𝑝 ∶ 0 ≤ 𝑝 ≤ 1 if 𝑝𝑟 𝑋 = 𝑘 = (1 − 𝑝)𝑘−1 ∙ 𝑝, for all positive
integers k. It has expected value 1Τ𝑝.
• The probability that a randomly chosen 1000 digit number is prime
is approximately 1Τ2302. We choose a 1000 digit number u.a.r., and
check if its prime. If it is, we’re done, otherwise we randomly select
another (possibly different) number and repeat. What is the
expected number of times we select a number?
– 2302
• A doctor is treating coronavirus victims. On each day, she has a 1Τ4
probability of catching coronavirus, a 1Τ8 probability of finishing
treatment for the last victim, and otherwise she comes back to
work the next day. What is the expected number of days she comes
in to work?
– 8Τ3
 72

Multiplying Expectations
• If X and Y are independent, then E(X ∙ Y) = E(X) ∙ E(Y).
• If we roll two fair dice and multiply their values, what is
the expected result?
• 3.52 = 12.25

• If we roll one fair die, and square its value, what is the
expected result?
• E(X ∙ X) = E(𝑋 2 )  E(X) ∙ E(X), because X is not
independent from X.

1 2 1 2 1 2 1 2 1 2 1 2
• ∙1 + ∙2 + ∙3 + ∙4 + ∙5 + ∙6  15.17
6 6 6 6 6 6
 73

Variance
• The expected value of a random variable tells us the average, but
nothing about how widely its values are distributed.
• If the midterm has an average of 33, it’s possible that everyone got
33, and it’s also possible that half the class got 50 while the other
half got 16. Variance is a metric that measures how widely values
are distributed.
• The variance of X is
2
𝑉 𝑋 =෍ 𝑋 𝑠 −𝐸 𝑋 ∙ 𝑝(𝑠)
𝑠∈𝑆
• 𝑋 𝑠 − 𝐸 𝑋 measures how much individual values vary from the
average.
• Squaring it heavily weights outliers: a large value will be produced if
there are a lot of scores very far from the average.
• An easier way to calculate variance is V(X) = E(X2) - E(X)2
 74

Practice 11
• A gambler’s coin has heads on both sides. What is the
variance on the number of heads after n flips?
• X is always n, X2 is always n2, so E(X2) – E(X)2 = n2 – n2 =
0
• That’s what you’d expect.
• What is the variance on the number of successes for a
single Bernoulli trial?
• pr(X = 1) = p
• E(X) = p ∙ 1 + (1 - p) ∙ 0 = p
• E(X2) = p ∙ 12 + (1 - p) ∙ 02 = p
• V(X) = E(X2) – E(X)2 = p - p2 = p ∙ (1 - p) = p ∙ q
 75

Practice 12
• What is the variance on the result of a fair die roll?
𝐸 𝑋 = 3.5
1
𝐸 𝑋 = ∙ (12 +22 +32 +42 +52 +62 )  15.17
2
6
V(X) = 15.17 − 3.52  2.92

• What is the variance of the number of heads for 1 coin flip?

𝑝𝑞 = 1ൗ4
• What is the variance of the number of heads for 3 coin
flips?
3𝑝𝑞 = 3ൗ4
• Wait, why did that work?
 76

Bienayme’s Formula
• If X1, …, Xn are pairwise independent random
variables, then:
•
V(X1 + … + Xn) = V(X1) + … + V(Xn)

• What is the variance on the number of

successes over n Bernoulli trials?

• npq
 77

Standard Deviation
• The standard deviation of a random variable is
the square root of its variance.
• This is used for test statistics quite frequently.
• If you assume that test scores follow a specific
type of distribution known as the bell curve
(which is often the case for well-constructed
tests), then the standard deviation tells you how
many students fall within a specific range of
scores.
• 94% will fall within +/- 2 standard deviations
• 68% will fall within +/- 1 standard deviations
 78

XKCD #221
 79

BACKUP
 80

Hirings
• Let 𝐼𝑘 return 1 if the kth best employee is hired, and 0
otherwise. We want:
• 𝐸(σ𝑛𝑘=1 𝐼𝑘) = σ𝑛𝑘=1 𝐸(𝐼𝑘 )
• E(𝐼1 ) = ___ (the best employee is always hired).
• E(𝐼2 ) = ____ (there is a ___% chance the best employee
shows up before the 2nd best employee)
• E(𝐼3 ) = ___ (there is a ___% chance the 3rd best
employee shows up
before the best two employees)
• σ𝑛𝑘=1 𝐸(𝐼𝑘 ) =
 81

Hirings Solution
• Let 𝐼𝑘 return 1 if the kth best employee is hired, and 0
otherwise. We want:
• 𝐸(σ𝑛𝑘=1 𝐼𝑘) = σ𝑛𝑘=1 𝐸(𝐼𝑘 )
• E(𝐼1 ) = 1 (the best employee is always hired).
1
• E(𝐼2 ) = (there is a 50% chance the best employee
2
shows up before the 2nd best employee)
1
• E(𝐼3 ) = (there is a 33% chance the 3rd best employee
3
shows up
before the best two employees)
1
• σ𝑛𝑘=1 𝐸(𝐼𝑘 ) = 𝑛
σ𝑘=1 = (log 𝑛)
𝑘