Advanced Optimization:

Techniques and Industrial

Applications

Curriculum Development
of Master’s Degree Program in
Industrial Engineering for Thailand Sustainable Smart Industry
 Session 1.5:
Dynamic Programming

Curriculum Development
of Master’s Degree Program in
Industrial Engineering for Thailand Sustainable Smart Industry
 Introduction

• A specific type of mathematical programming in which the optimal solution of the

original problem is found by solving a chain of subproblems.

• In dynamic programming, the optimal solution of one subproblem will be used as

input to the next subproblem. When the last subproblem is solved, the optimal
solution for the entire problem is achieved, which includes the solution of the
original problem.

• Linkage between the stages of a DP problem is performed through recursive

computations. Depending on the nature of the problem at hand, forward recursive
equation or backward recursive equation will be developed for finding solution.
 Introduction

Example 1: Shortage Path Problem

Find the shortage path from node 1 to node 9 of the network:

12 7
2 5 7

1 4 15 3
6

7 10
1 4 8 9

3 3 7
2 15
3 6
4

Define:
: minimum total travel time from node i to node 9.
: travel time through the directed arc .
 Introduction

On an arbitrary arc , it can be seen that:

+
Hence:

However, the shortage path from node i to node 9 should include some intermediate node j (if these
intermediate nodes exist). Then,

The above equation is the recursive equation (or functional equation) of the shortage path problem
in the backward form.
 Introduction

Based on the recursive equation, the optimal solution can be

found as follows:

4
 Introduction

Shortage path from node 1 to node 9: 1-3-4-5-7-9 with total

travel time = 19.
 Introduction

It is noted that each subproblem is associated with a network’s node and

when the shortage path from node 1 to node 9 is determined, we also know
the shortage paths from every nodes of the network to node 9.

7
2 5 7

4 3
6
10
1 4 8 9

3
2 15
3 6
 Introduction

The above problem can also be solved by use of forward recursive equation as presented below
Define:
: minimum total travel time from node 1 to node j.
: travel time through the directed arc .

On an arbitrary arc , it can be seen that:

+
Hence:

However, the shortage path from node 1 to node j should include some intermediate node i (if these
intermediate nodes exist). Then,
 Introduction

Solution can be found recursively as follows:

 Introduction

The solution from forward recursive equation gives the shortage

paths from node 1 to every other nodes of the network, not only
to node 9.
 Introduction

Note:

Not all DP programs can be solved by both forward and

backward recursive techniques. The use of backward recursion
or forward recursion depends on the specific structure of the
problem under consideration.
 Introduction
Bellman’s Principle of Optimality

An optimal policy has the property that whatever the initial state and the
initial decision are, the remaining decisions must constitute an optimal
policy with regard to the state resulting from the first decision

The basic DP approach can be illustrated by the following diagram:

Stage n Stage n  1

sn sn+1

state  state 
rn sn , xn 
 Introduction
Bellman’s Principle of Optimality

• At state in stage , if decision is taken the current state will be

transferred to a new state in stage .

• A revenue will be obtained by decision taken at state

• The new state is also a function of and , and can be

expressed in form of a transformation function: .
 Introduction
Bellman’s Principle of Optimality

In case of maximization problem and backward recursive

technique is employed, if we denote as the maximum total
revenue obtained when the system moves from stage to stage
(the last stage), given the observed state at stage is , then:

 
f n sn   max rn sn , xn   f n 1 tn sn , xn 
xn Dsn 

In which is the set of all possible decisions of a given state at

stage (decision set).
 Introduction
Bellman’s Principle of Optimality

Similarly, when the forward recursive technique is employed, if

we denote as the maximum total revenue when the system
move from stage 1 to stage , given the observed state at stage
is , then:

 
f n1 sn1   max rn sn , xn   f n sn 
xn Dsn 1 

In which is the set of all possible decisions at stage such that

these decisions will help to transfer the states ’s at stage to a
predefined state in stage , i.e., .
 Introduction
Bellman’s Principle of Optimality

The Bellman’ optimality principle can help to establish recursive

equations when the structure of the problem can be arranged in
stages.
 Introduction
Bellman’s Principle of Optimality

Example 2: Find the shortage path from node 1 to node 10 of

the network
2
7 5 1
4
2 6 4
8 3
4 3 6
1 3 2 6 10
4 3
9 4
3 4 3
1
4 7 3
5
Stage 1 Stage 2 Stage 3 Stage 4 Stage 5
 x1   x2   x3   x4   x5 
 Introduction
Bellman’s Principle of Optimality

Applying backward recursive approach, the solution can be found as

follows:
Stage 4:
x4 r4 ( s4 , x4 )  f5 (t4 ( s4 , x4 )) f 4 ( s4 ) x4*
s4 10
Node 8 3 3 10
Node 9 4 4 10

In stage 4, the state can be node 8 or node 9, and the only decision
(i.e., ) that can be taken is go to node 10.
 Introduction
Bellman’s Principle of Optimality

When the state is node 8:

When the state is node 9:

Due to the fact that there exists only one possible decision, that
decision is the optimal decision.
rsxf3333*((ss33,)x3 )  f 4 (t3 ( s3 , x3 ))

Introduction
Bellman’s Principle of Optimality

Stage 3:

x3 r3 ( s3 , x3 )  f 4 (t3 ( s3 , x3 )) f3 ( s3 ) x3*
s3 8 9
Node 5 1+3=4 4+4=8 4 8
Node 6 6+3=9 3+4=7 7 9
Node 7 3+3=6 3+4=7 6 8

In this stage, the state can be node 5, node 6 or node 7. The

decision can be go to node 8 or go to node 9.
 Introduction
Bellman’s Principle of Optimality

When the state is node 5:

If the decision is go to node 8:

If the decision is go to node 9:

Optimal decision: go to node 8

 Introduction
Bellman’s Principle of Optimality

When the state is node 6:

If the decision is go to node 8:

If the decision is go to node 9:

Optimal decision: go to node 9

 Introduction
Bellman’s Principle of Optimality

When the state is node 7:

If the decision is go to node 8:

If the decision is go to node 9:

Optimal decision: go to node 8

 Introduction
Bellman’s Principle of Optimality

Stage 2:

x2 r2 ( s2 , x2 )  f3 (t2 ( s2 , x2 )) Min x2*

s2 5 6 7 f 2 ( s2 )
Node 2 7 + 4 = 11 4 + 7 = 11 6 + 6 = 12 11 5 or 6
Node 3 3 + 4 = 7 2 + 7 = 9 4 + 6 = 10 7 5
Node 4 4 + 4 = 8 1 + 7 = 8 5 + 6 = 11 8 5 or 6

In this stage, the state can be node 2, node 3 or node 4. The

decision can be go to node 5, go to node 6, or go to node 7.
 Introduction
Bellman’s Principle of Optimality

When the state is node 2:

If the decision is go to node 5:

If the decision is go to node 6:

If the decision is go to node 7:

Optimal decision: go to node 5 or go to node 6

 Introduction
Bellman’s Principle of Optimality

When the state is node 3:

If the decision is go to node 5:

If the decision is go to node 6:

If the decision is go to node 7:

Optimal decision: go to node 5

 Introduction
Bellman’s Principle of Optimality

When the state is node 4:

If the decision is go to node 5:

If the decision is go to node 6:

1
If the decision is go to node 7:

Optimal decision: go to node 5 or go to node 6

 Introduction
Bellman’s Principle of Optimality

Stage 1:

x1 r1 ( s1 , x1 )  f 2 (t1 ( s1 , x1 )) f1 ( s1 ) x1*
s1 2 3 4
Node 1 2 + 11 = 13 4 + 7 = 11 3 + 8 = 11 11 3 or 4

In this stage, the state is node 1. The decisions can be go to

node 2, go to node 3, or go to node 4.
 Introduction
Bellman’s Principle of Optimality

If the decision is go to node 2:

If the decision is go to node 3:

If the decision is go to node 7:

Optimal decision: go to node 3 or go to node 4

 Introduction
Bellman’s Principle of Optimality

The optimal solution has been found. There exist three shortage
paths from node 1 to node 10:

1-3-5-8-10, 1-4-5-8-10, 1-4-6-9-10

 Introduction
Bellman’s Principle of Optimality

Example 3:
No. of Additional person-years life
Five medical teams will be Teams (in 1000 units)
dispatched to 3 regions to help
Region 1 Region 2 Region 3
improve medical care. The
performance is measured by the 0 0 0 0
expected additional person-years of 1 45 20 50
2 70 45 70
life. The estimated performance 3 90 75 80
measures are given in the table: 4 105 110 100
5 120 150 130
 Introduction
Bellman’s Principle of Optimality

The problem is to allocate the medical teams so that the total

additional person-years of life can be maximized.

Denote:
: number of teams to be allocated to region n (n = 1, 2, 3).
: number of teams available for allocation to the regions
n, n +1,…, 3.
: the measure of performance from allocation teams to region n.
: Total maximum performance obtained when teams are
allocated to regions n, n +1,…, 3.
 Introduction
Bellman’s Principle of Optimality

The problem can be formulated as follows:

s.t.
and integer

The above problem can be considered as embedded in the following chains of

subproblems:

s.t.
and integer
 Introduction
Bellman’s Principle of Optimality

The backward recursive equation can be developed as:

f n sn   max
0xn sn
 pn  xn   f n1 sn  xn 
xn integer

Noting that , the solution can be obtained as follows:

 Introduction
Bellman’s Principle of Optimality

No. of p3 ( x3 )  f 4 ( s4 )  p3 ( x3 )  f 4 ( s3  x3 ) Max x3*

teams x 0 x 1 x 2 x 3 x 4 x 5 f3 ( s3 )
3 3 3 3 3 3
s3 0 0 - - - - - 0 0
s3 1 0 50 - - - - 50 1
s3 2 0 50 70 - - - 70 2
s3 3 0 50 70 80 - - 80 3
s3 4 0 50 70 80 100 - 100 4
s3 5 0 50 70 80 100 130 130 5
 Introduction
Bellman’s Principle of Optimality

No. of p2 ( x2 )  f3 ( s3 )  p2 ( x2 )  f3 ( s2  x2 ) Max x2*

teams x 0 x 1 x 2 x 3 x 4 x 5 f 2 ( s2 )
2 2 2 2 2 2
s2 0 0+0 - - - - - 0 0
=0
s2 1 0+50 20+0 - - - - 50 0
= 50 = 20
s2 2 0+70 20+50 45+0 - - - 70 0-1
= 70 = 70 = 45
s2 3 0+80 20+70 45+50 75+0 - - 95 2
= 80 = 90 = 95 = 75
s2 4 0+100 20+80 45+70 75+50 110+0 - 125 3
= 100 = 100 = 115 = 125 = 110
s2 5 0+130 20+100 45+80 75+70 110+50 150+0 160 4
= 130 = 120 = 125 = 145 = 160 = 150
 Introduction
Bellman’s Principle of Optimality

No. of p1 ( x1 )  f 2 ( s2 )  p1 ( x1 )  f 2 ( s1  x1 ) Max x1*

teams x 0 x 1 x 2 x 3 x 4 x 5 f1 ( s1 )
1 1 1 1 1 1
s1 0 * - - - - - * *
s1 1 * * - - - - * *
s1 2 * * * - - - * *
s1 3 * * * * - - * *
s1 4 * * * * * - * *
s1 5 0+160 45+125 70+95 90+70 115+50 120+0 170 1
= 160 = 170 = 165 = 160 = 165 = 120

(*: no need to determine those values)

Optimal solution: ; optimal objective function 170.
 Introduction
Bellman’s Principle of Optimality

Example 4: Resource Allocation Problem

Consider the single resource allocation problem to produce N products:

Max
s.t.

In which:
: profit obtained by producing units of product j.
: units of the resource consumed for producing units of product j.
: the set of possible production levels for product j.
The above problem can be solved by dynamic programming
 Introduction
Bellman’s Principle of Optimality

The embedded problem in backward recursive form

Define:
• : state - y units of resource are allocated to produce products
from n through N.
• : maximum total profit obtained from products n through N,
when y units of resource are allocated to them.
• : the optimal value to be determined.
 Introduction
Bellman’s Principle of Optimality

Notes:
1. can be expressed as:

Max
s.t.

The problem is embedded in the above problems: for and

 Introduction
Bellman’s Principle of Optimality

2. Boundary conditions:

Max
s.t.
 Introduction
Bellman’s Principle of Optimality

Backward recursive equation:

In this case, we have:

; ; and
 Introduction
Bellman’s Principle of Optimality

The embedded problem in forward recursive form

Define:
• : state - y units of resource are allocated to produce products
from 1 to n.
• : maximum total profit obtained from products 1 through n,
when y units of resource are allocated to them.
• : the optimal value to be determined.
 Introduction
Bellman’s Principle of Optimality

Forward recursive equation:

In this case, we still have:

; ; and
.