Chapter 17 Markov ChainsDescription Sometimes we are interested in how a random variable changes over time. The study of how a random variable evolves over time includes stochastic processes. An explanation of stochastic processes — in particular, a type of stochastic process known as a Markov chain is included. We begin by defining the concept of a stochastic process.5.1 What is a Stochastic Process? Suppose we observe some characteristic of a system at discrete points in time. Let X, be the value of the system characteristic at time t. In most situations, X, is not known with certainty before time ¢ and may be viewed as a random variable. A discrete-time stochastic process is simply a description of the relation between the random variables X, X, X, ..... Example: Observing the price of a share of Intel at the beginning of each day Application areas: education, marketing, health services, finance, accounting, and production* A continuous —time stochastic process is simply the stochastic process in which the state of the system can be viewed at any time, not just at discrete instants in time. * For example, the number of people in a supermarket t minutes after the store opens for business may be viewed as a continuous-time stochastic process.5.2 What is a Markov Chain? * One special type of discrete-time is called a Markov Chain. Definition: A discrete-time stochastic process is a Markov chain if, for f= 0,1,2... and all states P(X,,; = i,, |X, = i, X=. " X =i, X =i) =PUX,.=H.:1X,= i) Essentially this says that the probability distribution of the state at time ¢+1 depends on the state at time t(i,) and does not depend on the states the chain passed through on the way to i, at time f.* In our study of Markov chains, we make further assumption that for all states 7 andj and all t, P(X,,, = j|X, = 1) is independent of ¢. * This assumption allows us to write P(X,,, = j|X, =) = p, where p,, is the probability that given the system is in state i at time ¢, it will be in a state / at time (+1. * Ifthe system moves from state 7 during one period to state j during the next period, we call that a transition from i to / has occurred.The p,’s are often referred to as the transition probabilities for the Markov chain. This equation implies that the probability law relating the next period’s state to the current state does not change over time. It is often called the Stationary Assumption and any Markov chain that satisfies it is called a stationary Markov chain. We also must define g, to be the probability that the chain is in state 7 at the time 0; in other words, P(X=1) = 4;+ We call the vector q= [q,, 7,,...q,] the initial probability distribution for the Markov chain. * In most applications, the transition probabilities are displayed as an s x 5 transition probability matrix P. The transition probability matrix P may be written as Pu Prov Pas Pn Px + Pas Pa Psx + Piss* For each J dpa! * We also know that each entry in the P matrix must be nonnegative. * Hence, all entries in the transition probability matrix are nonnegative, and the entries in each row must sum to |.The Gambler’s Ruin Problem * At time 0, I have $2. At times 1, 2, ..., I play a game in which I bet $1, with probabilities p, I win the game, and with probability | — p, I lose the game. My goal is to increase my capital to $4, and as soon as I do, the game is over. The game is also over if my capital is reduced to 0. — Let X, represent my capital position after the time t game (if any) is played — X,, X,, X,, .... May be viewed as a discrete-time stochastic processThe Gambler’s Ruin Problem $0 $I $2 $3 $4 1 0 0 0 0 l-p 0 p 0 0 P+ 0 l-p 0 pO 0 0 l-p 0 p 0 0 0 015.3 n-Step Transition Probabilities A question of interest when studying a Markov chain is: If a Markov chain is in a state i at time m, what is the probability that 7 periods later the Markov chain will be in state /? This probability will be independent of m, so we may write PX 4, =X, =i) = P(X, SIX, =i) = Pn) where P,,(n) is called the n-step probability of a transition from state i to state j. For n> 1, P,(n) = ijth element of P" P,,(2) is the (i, j)th element of matrix P?= P! P! P,(n) is the (i, j)th element of matrix P*= P! P™!The Cola Example * Suppose the entire cola industry produces only two colas. * Given that a person last purchased cola 1, there is a 90% chance that their next purchase will be cola 1. * Given that a person last purchased cola 2, there is an 80% chance that their next purchase will be cola 2. 1. Ifa person is currently a cola 2 purchaser, what is the probability that they will purchase cola | two purchases from now? 2. Ifa person is currently a cola | a purchaser, what is the probability that they will purchase cola 1 three purchases from now?The Cola Example * We view each person’s purchases as a Markov chain with the state at any given time being the type of cola the person last purchased. ¢ Hence, each person’s cola purchases may be represented by a two-state Markov chain, where — State 1 = person has last purchased cola 1 — State 2 = person has last purchased cola 2 + Ifwe define X, to be the type of cola purchased by a person on her mth future cola purchase, then X), X,, ... may be described as the Markov chain with the following transition matrix:The Cola Example Colal Cola2 _ Cola} 90.10 ~ Cola2| 20.80 We can now answer questions | and 2. 2. We seek P(X, = 1|K, = 2) = P,,(2) = element 21 of Pr p> [90 107/90 .10)_[-83..17 ~|.20 .80|].20 .80| |.34 .66The Cola Example — Hence, P,,(2) =.34. This means that the probability is .34 that two purchases in the future a cola 2 drinker will purchase cola 1. 1. We seek P,,(3) = element 11 of P*: P* = Pce?)—|-2° -10]/-83 .17]_[.781 .219 a — |.20 .80],.34 .66] |.438 .562 Therefore, P, (3) =.781Many times we do not know the state of the Markov chain at time 0. Then we can determine the probability that the system is in state i at time 7 by using the reasoning. Probability of being in state j at time n =¥4,P,(n) isl where q=[q,, 42> --- 43]- Hence, g* = q°p" = q™'!p Example, q’ = (4,.6) a (4,6) [> ‘| q' = (.48,.52)* To illustrate the behavior of the n-step transition probabilities for large values of n, we have computed several of the n-step transition probabilities for the Cola example. * This means that for large n, no matter what the initial state, there is a .67 chance that a person will be a cola 1 purchaser.5.4 Classification of States in a Markov Chain * To understand the n-step transition in more detail, we need to study how mathematicians classify the states of a Markov chain. * The following transition matrix illustrates most of the following definitions. A graphical representation is shown in the book (State-Transition diagram) 4A 6 ococou 0 Suns 0 ah 4 8 0 Buc se 020 Definition: Given two states of i and/, a path from i to/ is a sequence of transitions that begins in i and ends in, such that each transition in the sequence has a positive probability of occurring. Definition: A state j is reachable from state i if there is a path leading from i to /. Definition: Two states i and j are said to communicate if j is reachable from i, and i is reachable from j. Definition: A set of states S in a Markov chain is a closed set if no state outside of S is reachable from any state in S.21 Definition: A state i is an absorbing state if p,; Definition: A state i is a transient state if there exists a state j that is reachable from /, but the state i is not reachable from state /. Definition:If a state is not transient, it is called a recurrent state. Definition:A state i is periodic with period k > | if k is the smallest number such that all paths leading from state i back to state i have a length that is a multiple of k. Ifa recurrent state is not periodic, it is referred to as aperiodic. If all states in a chain are recurrent, aperiodic, and communicate with each other, the chain is said to be ergodic. The importance of these concepts will become clear after the next two sections.5.5 Steady-State Probabilities and Mean First Passage Times * Steady-state probabilities are used to describe the long-run behavior of a Markov chain. * Theorem 1: Let P be the transition matrix for an s-state ergodic chain. Then there exists a vector m=([n,7, ... 2] such that T WH, + 1 . n | 7 lim P” =| . nn TM My, ot Fy 2223 * Theorem | tells us that for any initial state i, lim P(nj=a, + The vector m=[z, 2, ... 2,] is often called the steady-state distribution, or equilibrium distribution, for the Markov chain. Hence, they are independent of the initial probability distribution defined over the states24 Transient Analysis & Intuitive Interpretation * The behavior of a Markov chain before the steady state is reached is often call transient (or short-run) behavior. * An interpretation can be given to the steady-state probability equations. a (-p)= Yt Py kej ¢ This equation may be viewed as saying that in the steady-state, the “flow” of probability into each state must equal the flow of probability out of each state.Steady-State Probabilities * The vector IT = [7,, 7,, .... , 2, ] is often known as the steady-state distribution for the Markov chain * For large n and all i, + P(ntl) = P,(n) > x, k=s Py a+ I= kK Vik MP Ss w= 2 AKPK I ey 7] * In matrix form IT = TTP. * For any n and any i, + Pi (n)+P(n) +... +P.(n)=1 + As noo, we have m, +7, +....+a,=1 25An Intuitive Interpretation of Steady- State Probabilities * Consider Ss j= > EPR pO RE + Subtracting mp, from both sides of the above equation, we have 7, 0- Py) = Py Foy] * Probability that a particular transition enters state j = probability that a particular transition leaves state j 2627 . . Use of Steady-State Probabilities in Decision Making In the Cola Example, suppose that each customer makes one purchase of cola during any week. Suppose there are 100 million cola customers. One selling unit of cola costs the company $1 to produce and is sold for $2. For $500 million/year, an advertising firm guarantees to decrease from 10% to 5% the fraction of cola 1 customers who switch after a purchase. Should the company that makes cola 1 hire the firm?¢ At present, a fraction 2, =% of all purchases are cola 1 purchases, since: m, = .90n,+.207, n, = .102,+.807, and using the following equation by 2, + 2, = 1 * Each purchase of cola 1 earns the company a $1 profit. We can calculate the annual profit as $3,466,666,667 [2/3(100 million)(52 weeks)$1]. * The advertising firm is offering to change the P matrix to 95.05 P= 20 .80 2829 . . For P1, the steady-state equations become T, = .952,+.202, Tt, = .052,+.807, Replacing the second equation by 2, + 2, = 1 and solving, we obtain 2,=.8 and 7, = .2. Now the cola 1 company’s annual profit will be $3,660,000,000 [.8(100 million)(52 weeks)$1-($500 million)]. Hence, the cola 1 company should hire the ad agency.30 Inventory Example * A camera store stocks a particular model camera that can be ordered weekly. Let D,, D,, ... represent the demand for this camera (the number of units that would be sold if the inventory is not depleted) during the first week, second week, ..., respectively. It is assumed that the D,’s are independent and identically distributed random variables having a Poisson distribution with a mean of 1. Let X, represent the number of cameras on hand at the outset, X, the number of cameras on hand at the end of week 1, X, the number of cameras on hand at the end of week 2, and so on. — Assume that X, = 3. — On Saturday night the store places an order that is delivered in time for the next opening of the store on Monday. ~ The store using the following order policy: If there are no cameras in stock, 3 cameras are ordered. Otherwise, no order is placed. — Sales are lost when demand exceeds the inventory on hand31 Inventory Example X, is the number of Cameras in stock at the end of week t (as defined earlier), where X, represents the state of the system at time t Given that X, = i, X,,, depends only on D,,, and X, (Markovian property) D, has a Poisson distribution with mean equal to one. This means that P(D,,, =n) =e'l"/n! forn=0, 1, ... P(D, = 0) =e! = 0.368 P(D,= 1) =e1 = 0.368 P(D, = 2) = (1/2)e" = 0.184 P(D,2 3) =1—P(D, <2) = 1 — (.368 + .368 + .184) = 0.08 X,,, = max(3-D,.,, 0) if X,=0 and X,,, = max(X, — D,,,, 0) if X, > 1, for t= 1 QeseneeInventory Example: (One-Step) Transition Matrix + P,; = P(D,,, = 0) = 0.368 +P, =P(D,,, = 1) = 0.368 + P,, =P(D,,, =2) = 0.184 + Py = P(D,,, = 3) = 0.080 0 1 2 3 PO! P02 P03 P04 Pll P12) P22) P23 P31 P32 P33 P34 32 P41 P42 P43 P44 WwW NF Oo33 Inventory Example: Transition DiagramInventory Example: (One-Step) Transition Matrix wo nN & .080 .632 .264 .080 184 368 368 184 2 3 368 368 0 0 368 60 368 368Transition Matrix: Two-Step * po=pp pO) — Ww NY - © .249 .283 351 249 .286 252 319 .286 300 .233 .233 300 .165 .233 097 .165Transition Matrix: Four-Step © pa) = pape 289 286 .261 .164 282.285 .268 .166 284.283 .263 171 .289 .286 261 .164 WN rFe © pt)37 Transition Matrix: Eight-Step «© Pe) = papa 286 .286 286 286 285 285 .285 285 264 264 264 264 .166 .166 .166 .16638 Steady-State Probabilities * The steady-state probabilities uniquely satisfy the following steady-state equations -=| s xy aj J=0 VY My = ToPoo + MiPio + TP20+ MP0 VT = MpPor + MP + TP > TPs VT = MPo2 + MPis + MP2 + TPs2 Vs = MpPos + MiPis + TPs TPs © l=n, +0, +7,+7,39 Steady-State Probabilities: Inventory Example V 1, = .080n, + .632n, + .2647,+ .0807, V m, = .184n,) + 3687, + 3682, + 1847, V 1, = 3682, + 3682, + 3687, V 1; = 3682, + 3687, © | =nytaytayt ny, VT) = .286, m, =.285, 1,=.263, 2, = .166 * The numbers in each row of matrix P® match the corresponding steady-state probability40 Mean First Passage Times + For an ergodic chain, let m,,= expected number of transitions before we first reach state j, given that we are currently in state i; m, is called the mean first passage time from state / to state. * In the example, we assume we are currently in state /. Then with probability p,, it will take one transition to go from state i to state j. For k #j, we next go with probability p,, to state k. In this case, it will take an average of | + m,, transitions to go from i and j.* This reasoning implies m, = 1 +> Py My kaj * By solving the linear equations of the equation above, we find all the mean first passage times. It can be shown that At42 + For the cola example, 2,=2/3 and ,= 1/3 « Hence, m,,= 1.5 andm,,=3 * m),=1+p)m,,=1+.9m,, m,,=1+p,,m,,=1+ 8m,, Solving these two equations yields, * m,,=10andm,,=5Solving for Steady-State Probabilities and Mean First Passage Times on the Computer * Since we solve steady-state probabilities and mean first passage times by solving a system of linear equations, we may use LINDO to determine them. * Simply type in an objective function of 0, and type the equations you need to solve as your constraints. 435.6 Absorbing Chains * Many interesting applications of Markov chains involve chains in which some of the states are absorbing and the rest are transient states. * This type of chain is called an absorbing chain. * To see why we are interested in absorbing chains we consider the following accounts receivable example. 4445 Accounts Receivable Example ¢ The accounts receivable situation of a firm is often modeled as an absorbing Markov chain. * Suppose a firm assumes that an account is uncollected if the account is more than three months overdue. * Then at the beginning of each month, each account may be classified into one of the following states: State 1 New account State 2 Payment on account is one month overdue State 3 Payment on account is two months overdue State 4 Payment on account is three months overdue State 5 Account has been paid State 6 Account is written off as bad debt* Suppose that past data indicate that the following Markov chain describes how the status of an account changes from one month to the next month: New | month 2 months 3 months Paid Bad Debt New Oo 6 O O 4 0 1month | O oO 2S Oo Ss Oo 2 months | O oO oO 4 .6 Oo 3 months | O oO oO oO -7 as Paid oO oO oO oO 1 oO Bad Debt | O oO oO Oo Oo 1 4647 * To simplify our example, we assume that after three months, a debt is either collected or written off as a bad debt. * Once a debt is paid up or written off as a bad debt, the account if closed, and no further transitions occur. * Hence, Paid or Bad Debt are absorbing states. Since every account will eventually be paid or written off as a bad debt, New, | month, 2 months, and 3 months are transient states.48 * A typical new account will be absorbed as either a collected debt or a bad debt. * What is the probability that a new account will eventually be collected? * To answer this questions we must write a transition matrix. We assume s — m transient states and m absorbing states. The transition matrix is written in the form of ——s columns columns s-m rows R pe Q m rows 0/1* The transition matrix for this example is New 1 month 2 months 3 months Paid Bad Debt New 1 month 2 months 3 months Paid Bad Debt ¢ Then s =6, m =2, and Q and R are as shown. 4950 What is the probability that a new account will eventually be collected? (.964) What is the probability that a one-month overdue account will eventually become a bad debt? (.06) If the firm’s sales average $100,000 per month, how much money per year will go uncollected? From answer 1, only 3.6% of all debts are uncollected. Since yearly accounts payable are $1,200,000 on the average, (0.036)(1,200,000) = $43,200 per year will be uncollected.