25 Optimal Control Theory (Part I)

Dr. A. M. Tanvir Hussain

ECO511: Mathematical Economics

Department of Economics
East West University
Spring 2020
Chapter 25

Introduction

The undiscounted problem

The discounted problem

Alternative terminal conditions

2
25.1 Introduction
25.1.1 Optimization: static and dynamic

I Optimization: fundamentally important in economics

doing the best we can under the circumstances

I The planning horizon/“timespan” for an optimization problem may be

... static (a “one-shot” problem)

... dynamic (a problem to be solved for a number of years/periods)

I Solution to a dynamic optimization problem should be an optimal time

path – gives the optimum for the choice variable(s) for each period

optimal control theory provides one solution approach

3
25.1 Introduction
25.1.2 Optimization: static and dynamic

I Classical approach known as Calculus of variations dates back to work

done by Newton and Bernoulli

I More recent development during 1950s and 1960s based mainly on work
done by Bellman and Pontryagin led to optimal control theory

optimal control more powerful than variational calculus, e.g., dealing with
corner solutions

In optimal control, aim is to find the optimal time path for a “control”
variable which determines optimal time path for the state variable

control variable is “flow” (or “instrument”) while state variable is “stock”

some fundamental relationship exists between stock and flow variables

4
25.1 Introduction
25.1.3 Planning or time horizon

I Planning (or time) horizon for a dynamic optimization problem can be

... finite, spanning over the interval [0, T ]

... infinite, spanning over the interval [0, ∞)

I Consider only continuous time problems (⇒ differential equations)

I Notation: start at period is 0 and end at period T (finite horizon)

the world ends beyond T (important to remember)

find optimal at every instant in time in the interval [0, T ]

objective: maximize profit, minimize cost over interval [0, T ]

5
 oM@~iOKMoaeno~ geOco~
5o~ oM@~@c?~ eD~ np5K@~ ’ 4M@~eoM@i~geOcon~ o *~ х ~!xUx
nMew~ oM@~ v5iOesn~ Oco@ia@?O5o@~ np5o@n~ Ocoe~ wMO9M~ oM@~ ns7no6d9@~ _5y~ 7@~
oi5cnDei_@?~ ?sjOcK~ oM@~ gie9@nn~ 4M@n@~ geOcon~'~ o 7 7&7 5i@~ i@D@ii@?~ O੩
25.1 Introduction
5n~ ̭7Ē
1 7z@͌ 4e~ Oc?Q95o@~ oM@~ gennO7OYOoy~ eD~ pi5cnDeia5oOec~ Diea~ no5o@~ '~ oe~
25.1.4 Multistage
no5o@~decisionmaking
o w@~?i5w~5c~ Ē͌ Die_~geOco~'~oe~geOco~ o 4M@~eoM@i~6k9~'*~ nMewn~

!qČ

:੩ ɣɤɥɦɧɨɩɪɫɬɭ ‫߇ߧۏ‬੩
%p “੩ . . rǶ

ͽÓÔ×
ÕÖ ØÙÚÛÜÝÞ;
!Č %p !Č “੩ !Č. !ĀČ. !Č0­

9?;G8­ ((­
Figure 25.1: A discrete problem

6
 -
Ķ
,
ĶĶ24Ķ œk œ͌Ķ 


ī
%ĶĶĶ


Ķ 
Ķ$


›6
ĶĶ


Ķ$
1ĶV'Ķ
25.1 Introduction 
Ķ 
Ķ$

ĶĶ
Ķ Ķ 6


Ķ ĶĶĶî
Ķ
:Ķ ̹<'͌o 1 k 7&7JĶ 

Ķ$

Ķ

Ķ
˅ѵ

25.1.4 Multistage decisionmaking

%r,s6†

Ġ
‫ط‬੩
੩
੩
† ੩
੩
੩
੩
੩
‫ظ‬੩
࣎੖੗੘ਖ਼ਗ਼ਜ਼ੜ੝ਫ਼੟੠੡੢πρԎ੒੩
Lƪ ¢Ň¯੩ŽĂė੩

Figure 25.2: A continuous problem

7
 
ĶĶ͌ Ķ]
%ĶĶ% 
ĶĶ
#Ķ
/ĶĶĶì,
%MĶ%Ķ
#Ķ Ȭࡪ
'Ķ a
͌#Ķ
ĶĶ
Ķ#Ķ
͌#Ķ
Ķ
#Ķ
EĶ5Ķ #MĶ Ķ
25.1 Introduction 
Ķ Ķ /Ķ 
Ķ
͌ ěĶ 

͌Ķ Ķ 
%Ķ Ķ 
0 Ķ $
1Ķ ÕĶ
Ķ
'Ķ 
Ķ 
Ķ Ķ Ķ #
Ķ 
Ķ 
éþ Ķ 
Ķ 
#$Ķ $$
%Ķ Ķ Ķ
25.1.5 Objective functional

%7r†ÓČ-4[CmmD0V6† c,tAm† &7r† ljߗ੩c,rB† |,Vy7m†

3xe{7m† f7.W† XE\7†

ŃŎŇ¯੩ ēуԍ੩

Figure 25.3: A continuous problem

8
25.1 Introduction
25.1.5 Objective functional

I Imagine our objective is to maximize V

our choice variable is y which depends on time

choose the path (formally: time path) that gives maximum V value

choose the time path yI (t) ⇒ optimal time path

I Objective functional: mapping from time paths (y(t) curves) resulting in

path values (objective values V )

I Although we will write V (y(t)), we should not think of a direct

dependence in the form of V (t)

9
25.2 The undiscounted problem
25.2.1 Problem statement and preliminaries

I The general undiscounted optimal control problem is

Z T
Maximize: J = f [x(t), y(t), t]dt (25.1)
0
subject to: ẋ = g[x(t), y(t), t]
x(0) = x0 > 0 given, x(T ) free

where J is the value of functional to be maximized, x(t) is state variable,

y(t) is control variable and [0, T ] is finite time horizon

ẋ(t) is the equation of motion of state variable (or the state equation)

finite end period T , free endpoint (or vertical terminal line) problem
⇒ x(T ) is free to be chosen optimally

10
25.2 The undiscounted problem
25.2.2 Solution process

I Objective: maximize aggregated function value over [0, T ]

e.g., maximize aggregate (“lifetime”) utility or net benefit or profit in

economic models

warning: the integrand function in (25.1) seems non-autonomous

I Step 1: form the Hamiltonian function

H[x(t), y(t), λ(t), t] = f [x(t), y(t), t] + λ(t)g[x(t), y(t), t] (25.2)

I λ is the costate variable (similar to Lagrange multipliers)

λ(t) function to be discussed later

11
25.2 The undiscounted problem
25.2.2 Solution process

Step 2: Maximum principle necessary conditions1 must be satisfied by y ? (t)

1 choose only y to maximize H for [0, T ]

∂H
= 0 (25.3)
∂y

2 time paths for x and λ are given by the solution to

∂H
λ̇ = − and ẋ = g[x(t), y(t), t] (25.4)
∂x

Step 3: Add relevant boundary conditions2

x(0) = x0 and λ(T ) = 0 (25.5)

1
These conditions were developed by L. S. Pontryagin and his associates
2
See Theorem 25.1 (H-L) for maximum principle and boundary conditions
12
25.2 The undiscounted problem
25.2.2 Solution process
1&dŤ ‫ִڠ‬੩Ğԭ੩Ş‫ڧ‬੩‫ٰږڕ‬੩ۛŧ੩

`੩

Figure 25.4: Plotting an

ࡋࡪΏࡪΐԍӷࡪ
°ѵ
H function over control u
for a specific period t and
specific values of x and λ;
interior optimum at u = b
for concave H (Curve 1),
‫؜‬੩ corner optimum at u = a
‫آ‬੩ for constantly decreasing
੩
੩ Cß Ő੩
H (Curve 3) and corner
optimum at u = c for
ƄƅͭƆeƇƈƉƊƋƌƍƎƏ ƐƑeƒƓȮ͘Ɣ Ǝ ƗƕʆƖ ʹ; constantly increasing H
‫ࡪی‬ zƇ ‡Ƈ ‫ࡪۺ‬
(Curve 2)
½À¿É(Ĵ $O

Ā Ā

Ā 
Ā ;

Ā
Ā dĀ -Ā 
MĀ 
Ā
Ā 
Ā
ࣄ8Jࣃ ࡪ ੩ 8 Ā Ā
ĀĀ Ā
ĀQ „
òĀ ĀÛ
bĀ
13

„
Ā 
Ā
Ā ࡪ Ā

Ā Ā 
Ā 

&Ā
Ā
Ā

Ā
Ā

Ā
25.2 The undiscounted problem
25.2.2 Solution process

I If Hamiltonian is not differentiable, then replace (25.3) with

H[x(t), y ? (t), λ(t), t] ≥ H[x(t), y(t), λ(t), t] ∀t ∈ [0, T ] (25.6)

I Sufficiency conditions for a maximum ⇒ see Theorem 25.2 (H-L)

I For minimization problems, multiply f (·) by −1 and proceed

I λ(T ) = 0 known as the transversality condition (more on this later)

intuitively: it explains how we transverse (or “cross over”) to the world

beyond period T

practically: it gives us a much needed boundary condition

14
25.2 The undiscounted problem
25.2.2 Solution process

Q2
I Solutions: combine (25.3) and (25.4) to find y ? (t), x? (t) and λ? (t); use
(25.5) to find definite solutions Overall efficien
2

M RS A = M RS B = M
x*(T)
x*(t)
PPF
x0
Figure 25.5:
Hypothetical solution
λ*(t) paths for our free E
endpoint (or vertical
terminal line) problem

y*(t)
λ*(T) = 0
0 T time, t

15
25.2 The undiscounted problem
25.2.3 Interpreting the λ? (t) function

I We have a sequence (or path) of costate variables

I It can be formally proven that λ? (0) gives the amount of increase in

maximum function value for a marginal increase in x(0)

λ? (0) is the shadow value (or marginal willingness to pay) for a marginal
addition to x(0)

λ? (T ) gives the loss (or sacrifice) of maximum function value incurred due
to having the marginal unit of x(T ) remaining at the end

I Therefore, λ? (t) it gives the shadow price (or imputed value) of an

associated state variable at time t

16
25.2 The undiscounted problem
25.2.4 An example

I Consider the optimal control problem

Z T
1
Maximize: −(1 + y 2 ) 2 dt (25.7)
0
subject to: ẋ = y
x(0) = x0 > 0 given, X(T ) free

I Also consider solved example 25.1 (H-L)

17
25.3 The discounted problem
25.3.1 Fundamentals

I Discounting: key part of dynamic optimization models in economics

stems from the behavioral feature of time preference

people are impatient and prefer the present over the future

when aggregating, future welfare gets ever-decreasing weights

I Notation: ρ will be the the “discount rate” and e−ρt will be the the
“discount factor”

discount rate remains constant while discount factor increases over [0, T ]

in economic models: objective will be to maximize discounted sum of a

stream of benefits over [0, T ]

18
25.3 The discounted problem
25.3.2 The general problem statement

I The general undiscounted optimal control problem is

Z T
Maximize: J = F [x(t), y(t)]e−ρt dt (25.8)
0
subject to: ẋ = G[x(t), y(t)]
x(0) = x0 > 0 given, x(T ) free

where the problem is non-autonomous, but t enters the problem only

through the discount factor

19
25.3 The discounted problem
25.3.3 Solution process

I Step 1: form the current-valued Hamiltonian function

Hc [x(t), y(t), µ(t)] = F [x(t), y(t)] + µ(t)G[x(t), y(t)] (25.9)

where

Hc = He−ρt (25.10)
−ρt
µ = λe (25.11)

I Step 2: Maximum principle necessary conditions starting with

1 choose y(t) to maximize Hc

∂Hc
= 0 (25.12)
∂y

20
25.3 The discounted problem
25.3.3 Solution process

I Continuing with the Maximum principle conditions

2 time paths for x(t) and µ(t) are given by the solution to

∂Hc
µ̇ − ρµ = − and ẋ = g[x(t), y(t), t] (25.13)
∂x

I Step 3: Add relevant boundary conditions

x(0) = x0 and µ(T )e−ρT = 0 (25.14)

21
25.3 The discounted problem
25.3.4 An example

I Consider the following problem (Example 25.2 in H-L)

Z T
Maximize: e−ρt (ax − bx2 − cy 2 )dt (25.15)
0
subject to: ẋ = y − αx
x(0) = x0 > 0 given

22
 Overall effic
2

25.4 Alternative terminal conditions M RS A = M RS B =

25.4.1 Fixed terminal point
PP

x2(t) Figure 25.6: An

optimization problem
x0 E
with given terminal
x1(t) x(T) given
condition x(T ) = xT
obviates the need for any
transversality condition;
x3(t) look at Theorem 25.3 λ*(t)
(H-L)
0 T time, t λ*(T) =

I In our original fixed terminal line problem, we would choose x(T ) optimally,
i.e., any change in x(T ) will not change maximum value of the objective

In this fixed terminal point problem, we are not free to do that and therefore
it’s not certain that λ(T ) = 0 will apply; see example 25.3 (H-L)

23
 Q2

Overall efficienc
25.4 Alternative terminal conditions 2

25.4.2 Truncated vertical terminal line M RS A = M RS B = M R

x1(t) PPF
Figure 25.7: An
optimization problem
x2(t) with given terminal
x0 condition x(T ) ≥ b
E
x3(t) x(T) = b divides all x(t) paths into
permissible (those with
x(T ) ≥ b) and not
x4(t) permissible (those with
λ*(t)
x(T ) < b)
0 T time, t λ*(T) = 0

I Practical approach: start as problem (25.1) and, if x? (T ) > b, continue to

use λ? (T ) = 0 as the transversality condition; but if x? (T ) < b then we
must settle for x? (T ) = b knowing that λ? (T ) = 0 is most likely not valid

“inequality constrained endpoint” problems, see example 25.5 (H-L)

24
 x3(t)
Q2

25.4 Alternative terminal conditions x1(t)

Overall effi
2
25.4.3 Horizontal terminal line x2(t)
M RS A = M RS B =

x4(t)
x(T) = b PP

Figure 25.8: An
optimization problem E
with given terminal
condition x(T ) = b where
T is finite but free
x0
λ*(t)

0 T1 T2 T3 time, t λ*(T) =

I Terminal point is fixed at b, but, the time horizon (or “speed”) of reaching
that target is free; in this case, transversality condition is H(T ) = 0

25