EC400: Mathematical Methods for Macroeconomics Fall 2023

Dynamic Optimization
Dmitry Mukhin
d.mukhin@lse.ac.uk

Introduction This part of the course studies how to solve dynamic models. Although such
problems are particularly relevant in macroeconomics, they are also common in other fields.
There are two main approaches to solving dynamic optimization problems — sequential and
recursive ones. Within each approach, there are methods to solve (i) deterministic and stochas-
tic problems in (ii) discrete and continuous time. These methods can be used to (a) solve the
model in closed form, (b) obtain numerical solution, and (c) prove analytically some proper-
ties of the optimal solution. The choice between the approaches depends on the particular
application and the goal of the analysis.

discrete time continuous time

sequential approach Lagrangian Hamiltonian
recursive approach Bellman equation HJB equation

This part of the course is closer to the cookbook — the main focus is on the mathematical
tools that can be used to solve dynamic problems, the proofs are mostly heuristic, and simple
macroeconomic models are used as examples.

Savings problem Consider a simple savings problem of a household living for T periods:

T
X
max β t u(Ct )
{Ct ,Bt+1 }
t=0

s.t. Bt+1 = RBt + Yt − Ct (1)

BT +1 ≥ 0, (2)

where Ct and Bt+1 are consumption and savings in period t, R ≡ 1 + r denotes the gross
returns on assets, and Yt is an exogenous income. Equation (1) is the flow budget constraint of
the household, while equation (2) restricts the agent to pay back any debt before her death.
We make the standard assumptions about preferences:

1. discount factor β ∈ [0, 1),

1
 2. utility function u(·) is increasing and concave,

3. lim u0 (C) = ∞.
C→0

Optimality conditions Perhaps the simplest way to approach this problem is to apply the
same optimization methods that are used to solve the static problem. The dynamic prob-
lem is indeed not that different from a static one given that consumption in different periods
C0 , C1 , . . . , CT can be interpreted as consumption of different goods. To this end, write down
the Lagrangian

T
X T
X  
t
L= β u(Ct ) + λt RBt + Yt − Ct − Bt+1 + µBT +1 ,
t=0 t=0

where λt , µ ≥ 0, and take the first-order conditions (FOC):

wrt Ct : β t u0 (Ct ) = λt ,
wrt Bt+1 : λt = Rλt+1 , t < T,
wrt BT +1 : λT − µ = 0.

As usual, the Lagrange multiplier λt reflects the shadow price of having an additional unit of
good in period t and is equal to the margin utility of consumption in that period. The economic
interpretation of the Lagrangian is that there are markets, in which the agent can exchange
a unit of period-t good for λt units of utility (and vice versa). There is also a punishment (in
units of utility) for leaving an unpaid debt when dying. In this sense, the optimization problem
is fully static: all decisions are made in period zero and it only remains to execute the signed
contracts in the following periods. This interpretation will prove useful below when we discuss
the continuous-time version of the model. Combining the first two equations for t < T , we
get the intertemporal optimality conditions, which are often called the Euler equations (EE):

u0 (Ct ) = βR u0 (Ct+1 ), (3)

for ∀t < T . Several things to notice about this equation:

1. What is the mathematical intuition for the EE? The trajectory {Ct , Bt+1 } is optimal if
any small achievable deviation from it has zero first-order effect on utility. E.g. consider
a deviation in period t returning back to the trajectory in period t + 2, i.e. increase
Ct by ∆, which lowers savings Bt+1 by ∆ and the next-period wealth by RBt+1 . To
get back to the optimal trajectory, Bt+2 needs to be unchanged and hence, from (1),
we need to decrease Ct+1 by RBt+1 . The net effect on welfare of this perturbation is
equal dU = β t u0 (Ct )dCt + β t+1 u0 (Ct+1 )dCt+1 = β t u0 (Ct )∆ − β t+1 u0 (Ct+1 )R∆. Thus,

2
 the trajectory is optimal and the deviation results in zero first-order effects if the Euler
equation (3) is satisfied.

2. What is the economic intuition for the EE? It shows two saving motives: (i) consumption
smoothing, (ii) intertemporal substitution βR — in particular, consumption increases in
time when βR > 1.

3. Notice this is only one possible deviation and one optimality condition. One could in-
stead focus on a two-period deviation and obtain u0 (Ct ) = (βR)2 u0 (Ct+2 ). Things only
get more complicated when there are several choice variables, e.g. consumption and la-
bor. The reason most of the literature focuses on equation (3) is because of its simplicity
and intuitiveness.

The last period requires special treatment because of the complementary slackness condition

µBT +1 = 0,

i.e. either (i) constraint (2) binds and BT +1 = 0 or (ii) constraint (2) does not bind and µ = 0.
Combining this expression with the first-order conditions, we get the transversality condi-
tion (TVC)
β T u0 (CT )BT +1 = 0, (4)

which implies that the optimal bequest is BT +1 = 0. The economic intuition for the TVC
is straightforward: either consumption should have zero marginal utility in the last period
(which cannot be true under the assumed preferences), or it is optimal to leave zero savings
and consume all funds before dying.
In sum, the optimal path {Ct , Bt+1 } with 2 × (T + 1) unknowns is determined by T + 1
budget constraints (1), T Euler equations (3) and one transversality condition (4).

Example To solve the model in closed form, assume, for example, a constant relative risk
1−σ
aversion (CRRA) utility u(C) = C1−σ and follow these steps:

1. Iterate forward the budget constraint

B0 = R−1 C0 − Y0 + B1 = R−1 C0 − Y0 + R−2 C1 − Y1 + B2 = . . .

to obtain
T
X
R−t Ct − Yt + R−T BT +1 .


RB0 =
t=0

3
 2. Impose the terminal condition BT +1 = 0 to obtain the intertemporal budget constraint:

T
X T
X
R−t Ct = RB0 + R−t Yt .
t=0 t=0

3. Express the Euler equation as

1
Ct+1 = (βR) σ Ct

and substitute into the budget constraint to solve for

PT −t
RB0 + t=0 R Yt
C0 = PT t .
t=0 R−t (βR) σ

Infinite horizon So far, we assumed a finite T . However, it is often more convenient to

focus on infinite horizon. This approach can also provide an accurate description of the real
world if individual people care about the wellbeing of their descendants. Although the formal
proofs are more complicated for infinite horizon, most results can be derived heuristically by
simply taking the limit T → ∞ in the expressions above. In particular, the household problem
remains almost unchanged, except for constraint (2):
∞
X
max β t u(Ct )
{Ct ,Bt+1 }
t=0

s.t. Bt+1 = RBt + Yt − Ct (1’)

Bt ≥ −B̄, (2’)

where B̄ is a sufficiently high borrowing limit. The latter constraint is called the no-Ponzi
game condition (NPGC) and does not allow the agent to choose an infinitely high consump-
tion by borrowing more and more each period. Often, this restriction does not appear in the
optimization problem and is only implicitly assumed. Write down the Lagrangian assuming
that constraint (2’) does not bind
∞
X ∞
X  
t
L= β u(Ct ) + λt RBt + Yt − Ct − Bt+1 .
t=0 t=0

Taking the optimality conditions, it is straightforward to show that the Euler equations (3)
remain unchanged and hold for any period t. Remarkably, the transversality condition is also
similar to the one above:
lim β T u0 (CT )BT +1 = 0. (4’)
T →∞

4
 Following the same steps as above, derive the intertemporal budget constraint
∞
X ∞
X
−j
R Ct+j = RBt + R−j Yt+j
j=0 j=0

and solve for the optimal consumption under the CRRA utility:
∞
1 1
−1

 X
−j

Ct = 1 − β R σ σ RBt + R Yt+j .
j=0

More generally, however, it is often not possible to solve the problem in closed form. For
example, a model with stochastic shocks — a case that goes beyond the scope of this course,
but will be discussed in detail in macro courses — leads to the following Euler equation

Ct−σ = βR Et Ct+1
−σ
.

It is not possible to express Ct+1 from this condition and to substitute it into the budget con-
straint. Neither a finite horizon nor a continuous-time limit help to solve this problem. Instead,
macroeconomists usually employ one of the following approaches (or their combination):

1. approximate solution based on the linearized equilibrium conditions,

2. recursive formulation of the problem and dynamic programming.

The next two lectures discuss in detail these two methods.

Continuous time Similarly to the infinite horizon, assuming that time is continuous rather
than discrete can substantially simplify the analysis. This is especially true for the optimal
stopping time problems, in which agents choose when to change price, buy/sell an asset, invest
capital, etc. We start again with the baseline saving model, heuristically derive the continuous-
time version of the problem and the optimality conditions.
The basic idea is to rewrite all equations assuming an arbitrary length of period ∆ (instead
of normalizing it to one) and then to take the limit of ∆ → 0. To do this correctly, recall
that macroeconomic variables can be divided into (i) stocks measured at a given point in time
(e.g. capital, debt, prices) and (ii) flows measured over a period of time (e.g. inflation, interest,
consumption, GDP). It follows that the former variables are independent of ∆, while the latter
ones are proportional to the length of period. Applying this principle to the household budget
constraint, we get
Bt+∆ = (1 + r∆)Bt + Yt ∆ − Ct ∆,

5
where r = R − 1. Rewrite the equation in terms of growth rates

Bt+∆ − Bt
= rBt + Yt − Ct
∆

and take the continuous-time limit ∆ → 0:

Ḃt = rBt + Yt − Ct .

The left hand side of the equation Ḃt ≡ lim Bt+∆∆−Bt corresponds the instantaneous change
∆→0
in wealth Bt in period t. To express the objective function in continuous time, it is convenient
to define the discount factor ρ > 0 such that β ≡ e−ρ∆ . The discounted utility is then equal to
∞
X Z ∞
−ρt
e u(Ct )∆ −→ e−ρt u(Ct )dt.
t=0 0

Hamiltonian Consider next a generalized optimal control problem:

Z ∞
max e−ρt u(Ct )dt
{Ct ,Bt } 0

s.t. Ḃt = g(Bt , Ct ),

where Ct is a flow variable and Bt is a stock variable. Define the Hamiltonian as

Ht ≡ e−ρt u(Ct ) + λt g(Bt , Ct ),

where λt is called a co-state variable. The Lagrangian of the discrete-time version of the
problem can then be written as
∞ h
X
i
L= Ht ∆ − λt Bt+∆ − Bt .
t=0

Taking the first-order conditions and the limit ∆ → 0, we get

∂Ht ∂Ht
∆=0 ⇒ =0
∂Ct ∂Ct
∂Ht+∆ ∂Ht
∆ − λt + λt+1 = 0 ⇒ = −λ̇t
∂Bt+∆ ∂Bt

The intuition for these conditions is virtually the same as in the discrete time. The agent is free
to sell and buy period-t goods at price λt in units of utility. The first optimality condition says
that for each period (market) t, the household chooses consumption to maximize total utility.
The second optimality condition is the no-arbitrage condition, which says that the price of a

6
 ∂Ht
period-t good is equalized with its returns λt = λt+∆ + ∂Bt
∆. As before, the transversality
condition states
lim λt Bt = 0.
t→∞

The maximum principle due to Pontryagin says that these three equations are the necessary
conditions for optimality.

Example Consider again the savings problem with CRRA preferences and assume constant
income Yt = Y . The household problem in continuous time is
∞
Ct1−σ
Z
−ρt
max e dt
{Ct ,Bt }∞
t=0 0 1−σ

s.t. Ḃt = rBt + Y − Ct .

The Hamiltonian is given by

Ct1−σ
Ht ≡ e−ρt


+ λt rBt + Y − Ct
1−σ

and therefore, the optimality conditions are

∂Ht
0= = e−ρt Ct−σ − λt
∂Ct
∂Ht
−λ̇t = = rλt
∂Bt

Combine these equation to get the Euler equation:

Ċt 1
= (r − ρ).
Ct σ

Solve this differential equation:

r−ρ
t
Ct = C0 e σ ,

where C0 is the (unknown) level of consumption in the initial period. This constant is deter-
mined by the intertemporal budget constraint. To obtain the latter, multiply both sides of the
(flow) budget constraint by e−rt and express it as

e−rt Ḃt − re−rt Bt = (Y − Ct )e−rt ,

or equivalently as
d(e−rt Bt ) = (Y − Ct )e−rt dt.

7
Integrate both sides from the initial period to period t:
Z t
−rt
e Bt = B0 + (Y − Cs )e−rs ds.
0

Next, use the transversality condition

lim λt Bt = 0,
t→∞

which combined with the expression for λt and Ct implies e−rt Bt → 0 as t → ∞. Applying
this limit to the integrated budget constraint, obtain
Z ∞
B0 + (Y − Cs )e−rs ds = 0.
0

Finally, substitute the process for consumption

Z ∞
r−ρ
B0 + (Y − C0 e σ
s
)e−rs ds = 0
0

and assuming r is not too high, solve

ρ − r(1 − σ)  1 
C0 = B0 + Y .
σ r

Notice that the same expression can be obtained from the solution of the discrete-time model
taking the limit ∆ → 0.