Examples of Dynamic Programming Problems
Uploaded by
seanwu95Examples of Dynamic Programming Problems
Uploaded by
seanwu95M.I.T. 15.
450-Fall 2010
Sloan School of Management Professor Leonid Kogan
Examples of Dynamic Programming Problems
Problem 1 A given quantity X of a single resource is to be allocated optimally among N
production processes. Each process produces an output of the same good in the amount
x,
where x is the amount of input (x has to be nonnegative). Use dynamic programming to
determine the allocation of the resource x
n
, n= 1, . . . , N, among the production processes
that maximizes the aggregate output.
1. Enumeratetheproductionprocesses1ton. Supposethatcertainamountoftheresource
hasbeenalreadyallocatedamongtherstn1processes. Lettheremainingamountbe
X
n
. Let J
n
(X
n
) be the aggregate output of the remaining N n production processes,
given that the input X
n
is distributed optimally among them. Show by induction that
the value function J
n
(X
n
) has the form J
n
(X
n
) = c
n
X
n
. Calculate the constants c
n
for n= 1, . . . , N.
2. Use the fact that X
1
=X to nd x
n
, n= 1, . . . , N, and the optimal aggregate output
J
1
.
Problem 2 Consider the following dynamic investment problem. The market consists of
two assets: the riskless asset and the risky asset. Both assets are traded periodically at time
periods t = 0,1, . . . , T. The net simple return over a single holding period on the riskless
asset is denoted by R
ft
(e.g., R
ft
= 5%), while the net simple return on the risky asset is
denoted by R
t
. It is assumed that the distribution of the returns on the risky asset is given
by
R
t
=+
t
,
where
t
, t = 0,1, . . . , T 1 are independently and identically distributed standard normal
random variables, i.e.,
t
N(0,1).
Investor seeks to maximize the expected utility of wealth at time T. Her utility function
isexponential: U(x) =e
x
. TheinitialwealthisdenotedbyW
0
. Therearenoconstraints
on short-sales and borrowing and there are no transactions costs.
1. Letthecontrolvariablebex
t
theamountofwealthinvestedintheriskyassetattime
t. Express the wealth W
t+1
at time t+ 1 as a function of the wealth W
t
at time t and
x
t
.
2. LetJ
t
(W
t
)denotethevaluefunction(theindirectutilityfunction)ofwealthW
t
attime
t. Show using induction that the value function has the functional form
J
t
(x) =a
t
e
btx
.
1
x ,
t
Show that the optimal amount of wealth allocated into the risky asset, does not
depend on the current level of wealth W
t
.
t
, t= 3. Findarecursiverelationfora
t
andb
t
. Whatistheoptimalinvestmentstrategyx
0,1, . . . , T 1? How does it depend on the risk-aversion parameter , the mean and
the variance of the returns on the risky asset and on the investment horizon T t?
Solution of Problem 1
1. Whenn=N,alltheremainingresourceX
N
shouldbeallocatedintoasingleremaining
production process N, i.e.,
J
N
(X
N
) = X
N
.
Thus, c
N
=1.
Lets assume that for n k +1, J
n
(X
n
) = c
n
X
n
. Note that X
k+1
= X
k
x
k
.
Therefore, according to the Bellman optimality principle,
J
k
(X
k
)=max(
x
k
+J
k+1
(X
k
x
k
)), s.t. x
k
X
k
. (1)
x
k
k
can be found from the rst-order condition The solution of this problem x
1 c
k+1
+ = 0. (2)
x
k
2 X
k
x
2
k
From (2) we nd that
x
k
(X
k
) =
X
k
.
1 +c
k
2
+1
(3)
Then, according to (1),
J
k
(X
k
) = 1 +c
k
2
+1
X
k
.
Thus, c
k
= 1 +c
2
k+1
and we conclude the induction argument.
In order to nd all constants explicitly, note that = 1 and c
2
= 1 +c
2
2
Therefore, c =N n+1 and
c
c
n
n
=
N n+1.
c
N
n n+1
.
n
n
x
2. Using the relation X =X (X ) and 3, we conclude that
n+1 n n
2
N n c
n+1
=
X X X = .
n+1
2
n
1 +c N n+ 1
n+1
Since X
1
=X,
N 1N 2 N n+ 1
X =
N n+ 1
X X. =
N N 1
N n+ 2 N
2
n
We use (3) again to conclude that
x
=
1 N k+ 1
X =
X
.
n
1 +N k N N
Thus,theresourcehastobeallocatedevenlyamongallN productionprocesses. Also
J
1
=c
1
X
1
X.
Solution of Problem 2
Let Z
t
denote the excess return on the risky asset, i.e.,
Z
t
=R
t
R
ft
.
1. Ifx
t
istheamountofwealthallocatedintotheriskyasset,W
t
x
t
mustbeallocated
into the riskless asset. Then
W
t+1
=x
t
(1+R
t
) + (W
t
x
t
)(1+R
ft
) =W
t
(1+R
ft
) +x
t
Z
t
. (4)
2. When t=T, J
T
=e
W
T
. Thus, a
T
=1 and b
t
=.
Letsassumethatfornt+1,J
n
(W
n
) =a
n
e
bnWn
.Then,accordingtotheBellman
optimality principle and (4),
J
t
(W
t
)=maxE[a
t
exp(b
t
W
t
(1+R
ft
)b
t
x
t
Z
t
)].
xt
Next, note that
E[exp(b
t
x
t
Z
t
)]=E[exp(b
t
x
t
(R
ft
+
t
))]=exp b
t
x
t
(R
ft
) +
2
1
2
b
2
t
x
t
2
.
Thus,
1
2
J
t
(W
t
)=maxE a
t
exp b
t
W
t
(1+R
ft
)b
t
x
t
(R
ft
) +
2
b
2
x .
xt 2
t t
The rst-order condition for the maximization problem is equivalent to
b
t
(R
ft
) +
2
b
2
x
= 0,
t t
which implies that
x
t
(W
t
) =
2
b
R
t
ft
. (5)
Thus,theoptimalamountofwealthallocatedintotheriskyassetx
t
doesnotdepend
on the current level of wealth W
t
.
We nd that
J
t
(W
t
) =a
t
exp b
t
W
t
(1+R
ft
)
1 (R
ft
)
2
, (6)
2
2
which concludes the step of induction.
3
3. From (6) we observe that
1 (R
ft
)
2
a
t
=a
t+1
exp , b
t
=b
t+1
(1+R
ft
).
2
2
We can solve these recursive equations using the terminal conditions
a
T
= 1, b
T
=
to obtain
T t(R
ft
)
2
a
t
=exp
2
2
, b
t
=(1 +R
ft
)
Tt
.
We now combine this with (5) to obtain
x
t
=
1
2
R
ft
(1+R
ft
)
(Tt)
.
Fromthisweconcludethatx
t
ishigherwhentherisk-aversionparameter islower;it
increases linearly with the mean excess return on the riskless asset and it is inversely
proportional to the variance of the returns; it is lower for longer investment horizons
T t.
4
MIT OpenCourseWare
http://ocw.mit.edu
15.450 Analytics of Finance
Fall 2010
For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms.