CHAPTER ONE

Introduction
Experiment, Sample Space and Events
Definitions of
Probability Elementary
Set Theory Axioms of
Probability Conditional
Probability
Independent Events Total Probability

Eya Z.
2012AMU
 Introduction to Probability Theory

Phenomena in nature “seem to be” random because of lack of

awareness of underlying phenomena and / or because of
“inaccurate” measuring instruments.

Notwithstanding the “perceived” randomness, there is some

structure in the randomness, which the human mind has come to
grasp.

Consider two options which I give to my friend:

– Option 1: Here is a dice. Roll it once. If you get a ‘6’, then I

will give you $1M

– Option 2: Here is a coin. Toss it once. If you get a ‘Head’, I will

give you $1M

Second option would be a ‘wiser’ decision as the ‘chances’ of my

friend getting a ‘Head’ on a ‘Coin-throw’ are more than getting a ‘6’
on a ‘Dice-throw’.

Thus, there is some structure, or in other words there is some

underlying quality, which differentiates the two experiments.
Probability Theory is an attempt to ‘quantify’ / ‘characterize’ random
phenomena and thereby take ‘wiser’ decisions in this ‘seemingly’ random
world.
 • As already mentioned, in reality, no phenomenon is random.
Phenomena ‘seem’ to be random because of the limitations of
our measuring equipment and / or because of our ignorance
about the existence of certain factors / phenomena / laws.

• Probability Theory is a tool employed by the ingenious man to

take ‘wiser’ decisions in spite of his ‘primitive’ measuring
equipment and/ or his incomplete knowledge of the laws of nature.

Need for Probability Theory for the communication

Engineer/ Signal Processing Community

Probability theory provides powerful tools to explain, model,

analyze, and design technology by engineers.

Examples

1. Wireless communication systems

▪ In order to enhance weak signals and maximize the range of

communication systems, it is necessary to use amplifiers.
Unfortunately, amplifiers always generate thermal noise,
which is added to the desired signal.
2. Signal processing

▪ Signals generated / encountered in a system are random

 ▪ Typical examples:

– Recovery of data coming over a noisy communication channel

– Estimation of the location of an aircraft from noisy radar
data.

– Estimation of a state variable in a control system based

on noisy measurements.

▪ Signals of this types can’t be described with explicitly

mathematical functions like sine, cos, step functions. their
description must be put in probabilistic terms

▪ Based on the signal / noise modeling (and succeeding

simulations) and the mathematical treatment we can design
optimum receivers / based on optimum algorithms.

Definition of Probability

▪ Probability theory is based on the paradigm of a random

experiment
1.Experiment is an experiment in which the following hold:

(a) The experiment can be repeated under identical

conditions.

(b) We have knowledge of all possible outcomes in advance.

(c) Any trial of the experiment results in an outcome
that is not known in advance.
 ▪ The results of an observation are called the
outcomes of the experiment.
▪ Typical examples of a random experiment are
• The roll of a die,
• The toss of a coin & drawing a card from a deck, or
• Selecting a message form transmission of several
messages.

2. Sample Space: The set of all possible outcomes of a

random experiment is called the sample
space (or universal set), and it is
denoted by S.

▪ An element in S is called a sample point.

▪ Each outcome of a random experiment corresponds to a
sample point

Example Find the sample space for the experiment of tossing a coin
twice.

Example Consider an experiment of drawing two cards at random

from a bag containing four cards marked with the
integers 1 through 4.
a. Find the sample space S , of the experiment if
the first card is replaced before the second is
drawn ?
b. Find the sample space S, of the experiment if the
first card is not replaced?
Example Consider the experiment of tossing a coin repeatedly and
of counting the number of tosses required until the first head
appears.

a. Let A be the event that the number of tosses

required until the first head appears is even.
b. Let B be the event that the number of tosses
required until the first head appears is odd.

c. Let C be the event that the number of tosses

required until the first head appears is less
than 5

Classical Definition of Probability

The probability P(A) of an event A is found without

experimentation. This is done by counting the total number, N of
possible ‘atomic’ outcomes of the experiment. If NA is the total
number of ‘atomic’ outcomes belonging to the event A, then P(A) is
defined to be

▪ Probability of an even number on the top face, when a dice is

rolled, would be 3/6. This answer is correct only when the
dice is not biased. Thus, there is a limitation in this definition
in that it assumes that there is no ‘bias’.

Relative Frequency Definition of Probability

 ▪ This definition overcomes the limitation of the Classical
definition. Here the experiment is actually performed a ‘large’
number of times, n and if the event A occurs nA times, then its

probability, P(A) is defined as

▪ An experiment by Willard H Longcor in which he tossed a

wooden dice a million times clearly shows the difference

between the two probabilities. Incidentally, the dice happened

to be biased because the numbers were indicated by drilling

small holes on the respective faces.

Face on 1 2 3 4 5 6
top

Rel freq 0.155 0.159 0.164 0.169 0.174 0.179

defn

Classical 1/6 1/6 1/6 1/6 1/6 1/6

defn
 ▪ Where n(A)/n is called the relative frequency of event A.
▪ For any event A, the relative frequency of A will have the
following properties

1. 0 ≤ n(A)/n ≤ 1,
where n(A)/n = 0 if A occurs in none of the n repeated trials and
n(A)/n = 1 if A occurs in all of the n repeated trials
2. If A and B are mutually exclusive events, then

Axiomatic Definition: Let S be a finite sample space and A be an

event in S. Then in the axiomatic definition, the, probability P(A) of
the event A is a real number assigned to A which satisfies the
following three axioms:
I. Axiom 1: P(A) ≥ 0
II. Axiom 2: P(S) =1
III. Axiom 3: P(AUB) = P(A) + P(B), if A n B = Ø
If the sample space S is not finite, then axiom 3 must be modified as
follows.
Axiom 3': If A1, A2, . . . is an infinite sequence of mutually
exclusive events in S (Ai n Aj = 0 for i ‡ j),
then
 Elementary Properties of Probability

▪ By using the above axioms, the following useful

properties of probability can be obtained:
1 : P(Ā) = 1-P(A)
2 : P(Ø) = 0
3: P(A) ≤ P(B), if A c B
4: P(A) ≤ P(S) = 1
5: P(A u B) = P(A) + P(B) – P(A n B)

Example

What is the probability of event B?

 Conditional Probability

• Quite often, the probability of an event depends on whether

another event has occurred or not.

• The conditional probability of an event A given event B, denoted

by P(A/ B), is defined as

where P(A n B) is the joint probability of A and B. Similarly,

Example A box contains 2Capacitor and 2 resistors. Two components

are drawn one by one sequentially without replacement.

a. P (drawing a C in 2nd draw given 1st draw was

a R)?

b. P (drawing a C in 2nd draw given 1st draw was

a C) ?

Example:- A company producing electric relays has three manufacturing

plants producing 50, 30, and 20 percent, respectively, of its product.
Suppose that the probabilities that a relay manufactured by these
plants is defective are 0.02,0.05, and 0.01, respectively.
 a) If a relay is selected at random from the output of the
company, what is the probability that it is defective?

b) If a relay selected at random is found to be defective,

what is the probability that it was manufactured by
plant 2?

Example: - In a box there are 100 resistors having resistance and

tolerance as shown in table below. Let a resistor be selected from
box and assume each resistor has the same likelihood of being
chosen. Define three events A as draw a 47Ω resistor, B as draw a
resistor with 5% tolerance and C as draw a 100Ω resistor. Find
P(A/B) , PA/C) and P(B/C)
 Independent events

Statistical Independence: Suppose that Ai and Bj are events

associated with the outcomes of two experiments. Suppose that
the occurrence of Ai does not affect the occurrence of Bj and vice
versa, then we say that the events are statistically independent.
• Two events A and B are said to be (statistically) independent if and
only if

• It follows immediately that if A and B are independent, then

Example Let A and B be events in a sample space S. Show that if

A and B are independent, then so are
(a) A’ and B’
(b) A’ and B