Chapter 1 Probability LI INTRODUCTION The study of probability stems from the analysis of certain games of chance, and it has found applications in most branches of science and engineering. In this chapter the basic concepts of prob- ability theory are presented, 12 SAMPLE SPACE AND EVENTS ‘A, Random Experiments: In the study of probability, any process of observation is referred to as an experiment. The results of an observation are called the outcomes of the experiment, An experiment is called a random experi- ‘ment if its outcome cannot be predicted. Typical examples of a random experiment are the roll of @ die, the toss of a coin, drawing a card from a deck, or selecting a message signal for transmission from several messages. B, Sample Space The set of all possible outcomes of a random experiment is called the sample space (or universal set), and it is denoted by S. An clement in $ is called a sample point. Each outcome of a random experiment corresponds to a sample point, EXAMPLE 1.1. Find the sample space for the experiment of tossing a coin (a) once and (b) twice. (a) There are two possible outcomes, heads or tails. Thus S=(H,T} where H and T represent head and tail, respectively. (6) There are four possible outcomes. They are pairs of heads and tails. Thus $= (HH, HT, TH, TT) EXAMPLE 1.2 Find the sample space for the experiment of tossing a coin repeatedly and of counting the number of tosses required until the first head appears. Clearly all possible outcomes for this experiment are the terms of the sequence 1, 2, 3... Thus $= 112,304) Note that there ate an infinite number of outcomes. EXAMPLE 1.3 Find the sample space for the experiment of measuring (in hours) the lifetime of a transistor. Clearly all possible outcomes are all nonnegative real numbers. That is, S=(c0sts 0} where « represents the life ofa transistor in hours. Note that any particular experiment can often have many different sample spaces depending on the observ ation of interest (Probs. 1.1 and 1.2). A sample space S is said to be discrete i it consists of a finite number of i2 PROBABILITY [CHAP 1 sample points (as in Example 1.1} or countably infinite sample points (as in Example 1.2). A set is called countable if its clements can be placed in a one-to-one correspondence with the positive integers. A sample space S is said to be continuous ifthe sample points constitute a continuum (as in Example 1.3) C. Events: Since we have identified a sample space S as the set of all possible outcomes of a random experi- ment, we will review some set notations in the following. If is an element of S (or belongs to S), then we write tes IFS is not an element of S (or does not belong to S), then we write ces Asset A is called a subset of B, denoted by . AcB if every element of A is also an element of B. Any subset of the sample space S is called an event. A sample point of S is often referred to as an elementary event. Note that the sample space S is the subset of itself, that is, $< S. Since S is the set of all possible outcomes, it is often called the certain ‘event. EXAMPLE 1.4 Consider the experiment of Example 1.2. Let A be the event that the number of tosses required ‘until the first head appears is even. Let B be the event that the number of tosses required until the first head appears is odd. Let C be the event that the number of tosses required until the first head appears is less than 5. Express events 4, B, and C. 212460} B= (13,5...) 1,2,3,4} 13 ALGEBRA OF SETS ‘A. Set Operations: J. Equality: Two sets A and Bare equal, denoted A = B, 2 Complementation: Suppose A < S. The complement of set A, denoted A, is the set containing all elements in S but notin A. and only if Ac Band Bo A. = {Oe Sand ¢¢ A) 3. Union: The union of sets A and B, denoted 4 vB, is the set containing all elements in either A or Bor both. Av B={0:CeAorle B} 4 Intersection: The intersection of sets A and B, denoted A © B, is the set containing all elements in both A and B. An B= {Ce and le B}CHAP. 1 PROBABILITY 3 S. Null Set: ‘The set containing no clement is called the null set, denoted @. Note that g=5 6. Disjoint Sets: ‘Two sets A and B are called disjoint or mutually exclusive if they contain no common element, thatis,if Ao B= 2. ‘The definitions of the union and intersection of two sets can be extended to any finite number of sets as follows: = UE: Ce Aj ole Ay or CE Ay} Aaa An And, ={0:Ce A, and f€ A, and --- 0 ,} ‘Note that these definitions can be extended (o an infinite number of sets: Uae au ayy Aye Apa ALO ALO ASO In our definition of event, we state that every subset of S is an event, including S and the null set @. Then S = the certain event © = the impossible event Wf A and B are events in S, then A = the event that A did not occur AU B= the event that either 4 or B or both occurred Ao B= the event that both A and B occurred Similarly, if A,, A2,..., A, are a sequence of events in S, then the event that at least one of the A, occurred; he event that all of the A, occurred. B. Venn Diagram: ‘A graphical representation that is very useful for illustrating set operation is the Venn diagram. For instance, in the three Venn diagrams shown in Fig. I-L, the shaded areas represent, respectively, the events A U B, Am B, and A. The Venn diagram in Fig. 1-2 indicates that B< A and the event A Bis shown as the shaded area.4 PROBABILITY (CHAP 1 te) Shinde regio A Fig. tt Bea Shade ion A.B Fig, 12 C. Identities: By the above set definitions or reference to Fig, 1-1, we obtain the following identities S=@ ay B=s (12) AnA 3) SuAss us) SAAaA 5) Avud=s 6) And=@ (17) The union and intersection operations also satisfy the following laws: Commutative Laws: AUB=BUA (18) ANB=BNA 9) Associative Laws: Av (Bu C=(Au BUC (1.10) AN(BAC=(AN BOC aa,CHAP. 1) PROBABILITY 5 Distributive Laws: AO(BUOQ=(ANB)U(ANO) (LR) AU(BOO=(AU BN (AUG (1.13) De Morgan’s Laws: AGB=A0B 4) AnB=40B8 (1.15) ‘These relations are verified by showing that any element that is contained in the set on the left side of the equality sign is also contained in the set on the right side, and vice versa. One way of showing this is by means of a Venn diagram (Prob. 1.13). The distributive laws can be extended as follows: An (0 ) Ua 0 B) (1.16) A »(Aa)= Arava wan Similarly, De Morgan’s laws also can be extended as follows (Prob. 1.17): (18) (1.19) 14 THE NOTION AND AXIOMS OF PROBABILITY An assignment of real numbers to the events defined in a sample space S is known as the prob- ability measure. Consider a tandom experiment with a sample space S, and let A be a particular event defined in S. ‘A. Relative Frequency Definition: Suppose that the random experiment is repeated m times. If event occurs n(A) times, then the probability of event A, denoted P(A), is defined as (4) = tit © where n(4)/n is called the relative frequency of event A. Note that this limit may not exist, and in addition, there are many situations in which the concepts of repeatability may not be valid, It is clear that for any event A, the relative frequency of A will have the following properties (1.20) 1. 0< mfAl/n < 1, where m(A)/n =0 if A occurs in none of the m repeated trials and n(A)jn = 1 if A occurs in all of the m repeated trials. 2. ICA and B ate mutually exclusive events, then mA UB) = nA) + (B)6 PROBABILITY (CHAP 1 and B. Axiomatic Definit Let S be a finite sample space and A be an event in S. Then in the axiomatic definition, the probability P(4) of the event 4 is a real number assigned to A which satisfies the following three axioms: Axiom 1: P(A) 20 (2n Axiom 2: P(S) (1.22) Axiom 3: P(A U B)= P(A) + PIB) if AN B= @ (1.23) If the sample space S is not finite, then axiom 3 must be modified as follows Axiom 3: If Ay, Aa, ... is an infinite sequence of mutually exclusive events in S (A; 0 A; = @ for i #,), then (Ua) = Era) (1.24) ‘These axioms satisfy our intuitive notion of probability measure obtained from the notion of relative frequency. C. Elementary Properties of Probability By using the above axioms, the following useful properties of probability can be obtained 1. P= 1-PtA) (128) 2 M)=0 (126) 3 PAS PB) ifdc® (127 4 AAVEI (1.28) 5. PAL B)= Pla) + PIB) — PLA 9 B) 129) 6 Ag Ags soe Ay are m arbitrary events in S, then ( ba) Endy Daan Aye FAA An A (DP PA, a Ay nen A) (1.30) where the sum of the second term is over all distinct pairs of events, that of the third term is over all distinct triples of events, and so forth TW Ag, Aas voy Anis & finite sequence of mutually exclusive events in $ (A, 0 A; = @5 for #), then (Ua) = SMA) (3th) and a similar equality holds for any subcollection of the events. Note that property 4 can be easily derived from axiom 2 and property 3. Since A c S, we have P(A) < P(S)CHAP. 1) PROBABILITY 7 ‘Thus, combining with axiom 1, we obtain Os PA)S1 (132) Property $ implies that P(A w B) s P(A) + PCB) (1.33) since P(A © B) = 0 by axiom 1 LS EQUALLY LIKELY EVENTS A. Finite Sample Space: Consider a finite sample space $ with n finite elements 8% (irtaveet where {5 are elementary events. Let P({,) = p,. Then 1 OspS1 i=12... (134 2 Epartm to teal (135) 3. If A= Ut, where 1 is a collection of subscripts, then MAy= FP= Ta (136) ea fro B. Equally Likely Events: ‘When all elementary events ¢, (i= 1, 2,...,m) are equally likely, that is, P= P2=""= Pa then from Eq. (1.35), we have i=1,2, and P(Ay= a (1.38) B n (37 where n(A) is the number of outcomes belonging to event A and a is the number of sample points ins. 1.6 CONDITIONAL PROBABILITY A. Definition: The conditional probability of an event A given event B, denoted by P(A |B), is defined as P(A 4B) Palpy= POR” Piao (139) where P(A 7 B)is the joint probability of 4 and B, Similarly, Pipjay= 2428) ways (140) P(A)8 PROBABILITY [CHAP 1 is the conditional probability of an event B given event 4. From Eqs. (1.39) and (1.40), we have P(A © B) = P(A|B)P(B) = P(BLA)PLAY (4D) Equation (1.41) is often quite useful in computing the joint probability of events. B. Bayes’ Rule: From Eq. (1.41) we can obtain the following Bayes’ rule: P(B\ A)P(A) P(A|B) =—— (1.42) (418) = Fm (142) 1.7 TOTAL PROBABILITY The events A, 4,..., A, are called mutually exclusive and exhaustive if UA=4,UA,U5U AS and A, ASH BW iAj (1.43) Let B be any event in S. Then PB) = PB A)= ¥ PB) APA) (1.44) which is known as the total probability of event B (Prob. 1.47). Let A = 4, in Eq, (1.42); then, using Eq. (1.44), we obtain (BL A)P(A) P(4,| 5) = —PBLADPAY 4s) LPBLAIMAd Note that the terms on the right-hand side are all conditioned on events 4,, while the term on the left is conditioned on B. Equation (1.45) is sometimes referred to as Bayes’ theorem. 18 INDEPENDENT EVENTS Two events A and B are said to be (statistically) independent if and only if P(A 9 B) = PlA)PB) (46) It follows immediately that if A and B are independent, then by Eqs. (1.39) and (1.40), P(ALB)= P(A) and PBI A) = PLB) a7) Iftwo events A and B are independent, then it can be shown that A and B are also independent; that Prob. 1.53), P(A o B) = P(A)P(B) (1.48) PA B)_ Then PAB) = EO ma) (49) ‘Thus, if 4 is independent of B, then the probability of 4’s occurrence is unchanged by information as to whether or not B has occurred. Three events A, B, C are said to be independent if and only if P(A 0 BC) = PAPC) PUA 9 B) = PA)P(B) PAN PAPO) PB 9 ©) = BPO) (1.50)CHAP. 1) PROBABILITY 9 ‘We may also extend the definition of independence to more than three events. The events Ay, Aa, --., A, are independent if and only if for every subset {4j,, Aj,,.--, Ay) (2 k n) of these events, PUA, 0 Aig 07 0 Ay) = P(A; )P(Aa) +77 PlAQ) (si) Finally, we define an infinite set of events to be independent if and only if every finite subset of these events is independent. ‘To distinguish between the mutual exclusiveness (or disjointness) and independence of a collec- tion of events we summarize as follows: I {Aj 1 = 1,2,...,m) isa sequence of mutually exclusive events, then (0, 4) EPA) (1.52) 2. I{A,, i= 1.2,....m} isa sequence of independent events, then ((4)= Tran (1.53) and a similar equality holds for any subcollection of the events, Solved Problems SAMPLE SPACE AND EVENTS Ll. 12. Consider a random experiment of tossing a coin three times. (@) Find the sample space S, if we wish to observe the exact sequences of heads and tails obtained. (b) Find the sample space S, if we wish to observe the number of heads in the three tosses. (a) The sampling space 5, is given by S\ = (HHA, HHT, HTH, THH, HTT, THT, TTH, TTT) ‘where, for example, HTH indicates a head on the first and third throws and a tail on the second throw. There are eight sample points in 5, (6) The sampling space Sis given by 2+ 10,1,2,3) where, for example, the outcome 2 indicates that two heads were obtained in the three tosses. The sample space S, contains four sample points. Consider an experiment of drawing two cards at random from a bag containing four cards ‘marked with the integers 1 through 4 (a) Find the sample space S, of the experiment if the first card is replaced before the second is drawn. (b) Find the sample space S, ofthe experiment ifthe first card is not replaced. (a) The sample space S, contains 16 ordered pairs (i,j, 1SiS4 I<) <4, where the frst number indicate the fist mamber drawn. Thus, fan aa a9 aa J20 22 2H AA) ]a0 G2 G9 G4 40 G2 9 Ga) Sm