Master Program in Data Science and Business Informatics

Statistics for Data Science

Lesson 01 - Probabilities and independence

Salvatore Ruggieri
Department of Computer Science
University of Pisa, Italy
salvatore.ruggieri@unipi.it

1 / 18
Why Statistics in Data Science
We need grounded means for reasoning about data generated from real world with some
degree of randomness.

What will you learn?

• Probability: properties of data generated by a known/assumed randomness model
• Statistics: properties of a randomness model that could have generated given data
• Simulation and R

2 / 18
Sample spaces and events
• An experiment is a measurement of a random process
• The outcome of a measurement takes values in some set Ω, called the sample space.
Examples:
▶ Tossing a coin: Ω = {H, T} [Finite]
▶ Month of birthdays Ω = {Jan, . . . , Dec} [Finite]
▶ Population of a city Ω = N = {0, 1, 2, . . . , } [Countably infinite]
▶ Length of a street Ω = R+ = (0, ∞) [Uncountably infinite]
▶ Tossing a coin twice: Ω = {H, T} × {H, T} = {(H, H),(H, T),(T, H),(T, T)}
▶ Testing for Covid-19 (univariate): Ω = {+, −}
▶ Testing for Covid-19 (multivariate): Ω = {f, m} × N × {+, −}, e..g, (f, 25, −) ∈ Ω
Look at seeing-theory.brown.edu
• An event is some subset of A ⊆ Ω of possible outcomes of an experiment.
▶ L = { Jan, March, May, July, August, October, December } a long month with 31 days
• We say that an event A occurs if the outcome of the experiment lies in the set A.
▶ If the outcome is Jan then L occurs
3 / 18
Probability functions on finite sample space
A probability function is a mapping from events to real numbers that satisfies certain
axioms. Intuition: how likely is an event to occur.

• Fact: P({a1 , . . . , an }) = P({a1 }) + . . . + P({an }) [Generalized additivity]

• Examples:
▶ P({H}) = P({T}) = 1/2
▶ P({Jan}) = 31/365, P({Feb}) = 28/365, . . . P({Dec}) = 31/365
▶ P(L) = 7/12 or 31·7/365?
• P({a}) often abbreviated as P(a), e.g., P(H) instead of P({H})
4 / 18
Properties of probability functions

• P(Ac ) = 1 − P(A)
• P(∅) = 0 [Impossible event]
• A ⊆ B ⇒ P(A) ≤ P(B)
• P(A ∪ B) = P(A) + P(B) − P(A ∩ B) [Inclusion-exclusion principle]
• P(A ∪ B) = P(A) + P(B \ A)
• probability that at least one coin toss over two lands head?

5 / 18
Defining probability functions

Assigning probability is NOT an easy task: a prob. function can be an approximation of reality
• Frequentist interpretation: probability measures a “proportion of outcomes”.
▶ A fair coin lands on heads 50% of times
▶ P(A) = |A|/Ω [Counting]
▶ P({ at least one H in two coin tosses}) = |{(H, H), (H, T ), (T , H)}|/4 = 3/4
• Bayesian (or epistemological) interpretation: probability measures a “degree of belief ”.
▶ Iliad and Odissey were composed by the same person at 90%

6 / 18
The Monty Hall problem

https://math.andyou.com/tools/montyhallsimulator/montysim.htm
(See also Exercise 2.14 of textbook [T])

Tree-based sequential description of probability function

7 / 18
Probability functions on countably infinite sample space

• (ii) is equivalent to countable additivity (or σ-additivity): for A1 ⊆ A2 ⊆ . . .

P( lim Ai ) = lim P(Ai )
n→∞ n→∞
• Example
▶ Experiment: we toss a coin repeatedly until H turns up.
▶ Outcome: the number of tosses needed.
▶ Ω = {1, 2, . . .} = N+
▶ Suppose: P(H) = p. Then: P(n) = (1 − p)n−1 p
▶ Is it a probability function? P(Ω) = . . .
8 / 18
Conditional probability
• Long months and months with ‘r’
▶ L = { Jan, Mar, May, July, Aug, Oct, Dec } a long month with 31 days
▶ R = { Jan, Feb, Mar, Apr, Sep, Oct, Nov, Dec } a month with ‘r’
▶ P(L) = 7/12 P(R) = 8/12
• Anna is born in a long month. What is the probability she is born in a month with ‘r’ ?
P(L ∩ R) P({Jan, Mar, Oct, Dec}) 4/12 4
P(R|L) = = =7 =
P(L) P(L) /12 7
• Intuition: probability of an event in the restricted sample space Ω ∩ L
▶ a-priori probability P(R) = 8/12
▶ a-posteriori probability P(R|L) = 4/7 < 8/12
• Example (classification): probab. of Covid given gender=f and age≥ 60: P(C |G ∩ A)
▶ Ω = {f, m} × N × {+, −}
▶ C = {( , , +) ∈ Ω} G = {(f , , ) ∈ Ω} A = {( , a, ) ∈ Ω | a ≥ 60}
▶ naming triples with features (gender, age, covid): P(covid= +|gender=f, age≥ 60)
Another example at seeing-theory.brown.edu
9 / 18
Conditional probability

Properties:
• P(A|C ) ̸= P(C |A), in general
• P(Ω|C ) = 1
• if A ∩ B = ∅ then P(A ∪ B|C ) = P(A|C ) + P(B|C ) P(·|C ) is a probability function

More generally, the Chain Rule:

P(A1 ∩ A2 ∩ A3 . . . ∩ An ) = P(A1 ) · P(A2 |A1 ) · P(A3 |A1 ∩ A2 ) · . . . · P(An | ∩n−1
i=1 Ai ) 10 / 18
Example: no coincident birthdays
• Bn = {n different birthdays}
• For n = 1, P(B1 ) = 1
• For n > 1,
P(Bn ) = P(Bn−1 ) · P({the n-th person’s birthday differs from the other n − 1}|Bn−1 )
n−1
n−1 Y i
= P(Bn−1 ) · (1 − ) = ... = (1 − )
365 365
i=1

11 / 18
Example: case-based reasoning
Factory 1’s light bulbs work for over 5000 hours in 99% of cases.
Factory 2’s bulbs work for over 5000 hours in 95% of cases.
Factory 1 supplies 60% of the total bulbs on the market and Factory 2 supplies 40% of it.
What is the chance that a purchased bulb will work for longer than 5000 hours?
• A = {bulbs working for longer than 5000 hours}
• C = {bulbs made by Factory 1}, hence C c = {bulbs made by Factory 2}
• Since A = (A ∩ C ) ∪ (A ∩ C c ) with (A ∩ C ) and (A ∩ C c ) disjoint:

P(A) = P(A ∩ C ) + P(A ∩ C c )

• and then by the multiplication rule:

P(A) = P(A|C ) · P(C ) + P(A|C c ) · P(C c )

Answer: P(A) = 0.99 · 0.6 + 0.95 · 0.4 = 0.974

12 / 18
The law of total probability

• Intuition: case-based reasoning

13 / 18
Exercise: Prisoners and guard dilemma

3 prisoners, 2 of which will be released.

You are one of the prisoners:
• at your’s present state of knowledge, the probability of being released is 2/3
▶ P(A1) = P(A1 ∩ A2c ∩ A3) + P(A1 ∩ A2 ∩ A3c ) = 2/3
• if you ask a friendly guard to tell you who is the prisoner other than yourself that will be
released, your probability of being released will become 1/2
▶ P(A1|A2) = P(A1 ∩ A2)/P(A2) = (1/3)/(2/3) = 1/2
▶ P(A1) = P(A1|A2)P(A2) + P(A1|A3)P(A3) − P(A1|A2 ∩ A3)P(A2 ∩ A3) =
1/2 · 2/3 + 1/2 · 2/3 − 0 · 1/3 = 2/3
What is wrong with this line of reasoning?
• A1 = { Prisoner 1 is released }, A2 = { Prisoner 2 is released }, A3 = { Prisoner 3 is released }

P(A1c ∩ A2 ∩ A3) = P(A1 ∩ A2c ∩ A3) = P(A1 ∩ A2 ∩ A3c ) = 1/3

14 / 18
Independence of events
Intuition: whether one event provides any information about another.
Independence
An event A is independent of B, if P(B) = 0 or

P(A|B) = P(A)
• For P(R|L) = 4/7 ̸= 8/12 = PR(R) - knowing Anna was born in a long month change the
probability she was born in a month with ’r’ !
• Tossing 2 coins:
▶ A1 is “H on toss 1” and A2 is “H on toss 2”
▶ P(A1 ) = P(A2 ) = 1/2
▶ P(A2 |A1 ) = P(A2 ∩ A1 )/P(A1 ) = 1/4/1/2 = 1/2 = P(A1 )
• Physical and stochastic independence
• Properties:
▶ A independent of B iff P(A ∩ B) = P(A) · P(B)
▶ A independent of B iff B independent of A [Symmetry]
▶ A independent of B iff Ac independent of B 15 / 18
Conditional independence of events
Intuition: whether one event provides any information about another given a third event
occurred. Technically, consider P(·|C ) in independence.

Conditional independence
An event A is conditionally independent of B given C such
that P(C ) > 0, if P(B|C ) = 0 or

P(A|B ∩ C ) = P(A|C )

• Properties:
▶ A conditionally independent of B iff P(A ∩ B|C ) = P(A|C ) · P(B|C )
▶ A conditionally independent of B iff B conditionally independent of A [Symmetry]
• Exercise at home. Prove or disprove:
▶ If A is independent of B then A is conditionally independent of B given C

16 / 18
Independence of two or more events

Alternative definition
Events A1 , A2 , . . . , Am are called independent if for every J ⊆ {1, . . . , m}:
\ Y
P( Ai ) = P(Ai )
i∈J i∈J

• Exercise at home: show the two definitions are equivalent

17 / 18
Independence of two or more events

Alternative definition
Events A1 , A2 , . . . , Am are called independent if for every J ⊆ {1, . . . , m}:
\ Y
P( Ai ) = P(Ai )
i∈J i∈J

• It is stronger than pairwise independence

P(Ai ∩ Aj ) = P(Ai ) · P(Aj ) for i ̸= j ∈ {1, . . . , m}

• Example: what is the probability of at least one head in the first 10 tosses of a coin?
Ai = {head in i-th toss}
[10 \10 Y10 Y10
P( Ai ) = 1 − P( Aci ) = 1 − P(Aci ) = 1 − (1 − P(Ai ))
i=1 i=1 i=1 i=1

18 / 18