Optimization Methods: Dynamic Programming - Introduction

Module 5 Lecture Notes 1 Introduction Introduction In some complex problems, it will be advisable to approach the problem in a sequential manner in order to find the solution quickly. The solution is found out in multi stages. This is the basic approach behind dynamic programming. It works in a divide and conquer manner. The word "programming" in "dynamic programming" has no particular connection to computer programming at all. A program is, instead, the plan for action that is produced. In this lecture, the multistage decision process, its representation, various types and the concept of sub-optimization and principle of optimality are discussed. Sequential optimization In sequential optimization, a problem is approached by dividing it into smaller subproblems and optimization is done for these subproblems without losing the integrity of the original problem. Sequential decision problems are those in which decisions are made in multiple stages. These are also called multistage decision problems since decisions are made at a number of stages. In multistage decision problems, an N variable problem is represented by N single variable problems. These problems are solved successively such that the optimal value of the original problem can be obtained from the optimal solutions of these N single variable problems. The N single variable problems are connected in series so that the output of one stage will be the input to the succeeding stage. This type of problem is called serial multistage decision process. For example, consider a water allocation problem to N users. The objective function is to maximize the total net benefit from all users. This problem can be solved by considering each user separately and optimizing the individual net benefits, subject to constraints and then adding up the benefits from all users to get the total optimal benefit.

D Nagesh Kumar, IISc, Bangalore

M5L1

Optimization Methods: Dynamic Programming - Introduction Representation of multistage decision process Consider a single stage decision process as shown in the figure below. Net Benefits, NB1

Input S1

Stage 1

Output S2

Decision variable, X1 Fig 1. Let S1 be the input state variable, S2 be the output state variable, X1 be the decision variable and NB1 be the net benefits. The input and output are related by a transformation function expressed as, S2 = g(X1, S1) Also since the net benefits are influenced by the decision variables and also the input variable, the benefit function can be expressed as NB1 = h(X1, S1) Now, consider a serial multistage decision process consisting of T stages as shown in the figure below.

NB1 S1 S2 St

NBt St+1 ST

NBT ST+1

Stage 1

Stage t

Stage T

X1

Xt

XT

Fig 2.

D Nagesh Kumar, IISc, Bangalore

M5L1

Optimization Methods: Dynamic Programming - Introduction Here, for the tth stage the, state transformation and the benefit functions are written as, St+1 = g(Xt, St) NBt = h(Xt, St)

The objective of this multistage problem is to find the optimum values of all decision variables X1, X2,, XT such that the individual net benefits of each stage that is expressed by some objective function, f(NBt) and the total net benefit which is expressed by f(NB1, NB2,, NBT) should be maximized. The application of dynamic programming to a multistage problem depends on the nature of this objective function i.e., the objective function should be separable and monotonic. An objective function is separable, if it can be decomposed and expressed as a sum or product of individual net benefits of each stage, i.e., either f = NBt = h( X t , S t )
t =1 t =1 T T

or

f = NBt = h( X t , S t )
t =1 t =1

An objective function is monotonic if for all values of a and b for which the value of the benefit function is h(xt = a, St ) h( xt = b, St ) , then
f (x1 , x2, ..., xt = a,..., xT , S t +1 ) f (x1 , x2, ..., xt = b,..., xT , S t +1 )

should be satisfied.

D Nagesh Kumar, IISc, Bangalore

M5L1

Optimization Methods: Dynamic Programming - Introduction Types of multistage decision problems

A serial multistage problem such as shown, can be classified into three categories as initial value problem, final value problem and boundary value problem. 1. Initial value problem: In this type, the value of the initial state variable, S1 is given. 2. Final value problem: In this, the value of the final state variable, ST is given. A final value problem can be transformed into an initial value problem by reversing the procedure of computation of the state variable, St. 3. Boundary value problem: In this, the values of both the initial and final state variables, S1 and ST are given. Concept of sub-optimization and principle of optimality Bellman (1957) stated the principle of optimality which explains the process of suboptimality as: An optimal policy (or a set of decisions) has the property that whatever the initial state and initial decision are, the remaining decisions must constitute an optimal policy with regard to the state resulting from the first decision. Consider the objective function consisting of T decision variables x1, x2, , xT,
f = NBt = h( X t , S t )
t =1 t =1 T T

and satisfying the equations, St+1 = g(Xt, St) NBt = h(Xt, St) for t = 1,2,,T

The concepts of suboptimization and principle of optimality are used to solve this problem through dynamic programming. To explain these concepts, consider the design of a water tank in which the cost of construction is to be minimized. The capacity of the tank to be designed is given as K.

D Nagesh Kumar, IISc, Bangalore

M5L1

Optimization Methods: Dynamic Programming - Introduction

The main components of a water tank include (i) tank (ii) columns to support the tank and (iii) the foundation. While optimizing this problem to minimize the cost, it would be advisable to break this system into individual parts and optimizing each part separately instead of considering the system as a whole together. However, while breaking and doing suboptimization, a logical procedure should be used; otherwise this approach can lead to a poor solution. For example, consider the suboptimization of columns without considering the other two components. In order to reduce the construction cost of columns, one may use heavy concrete columns with less reinforcement, since the cost of steel is high. But while considering the suboptimization of foundation component, the cost becomes higher as the foundation should be strong enough to carry these heavy columns. Thus, the suboptimization of columns before considering the suboptimization of foundation will adversely affect the overall design. In most of the serial systems as discussed above, since the suboptimization of last component does not influence the other components, it can be suboptimized independently. For the above problem, foundation can thus be suboptimized independently. Then the last two components (columns and foundation) are considered as a single component and suboptimization is done without affecting other components. This process can be repeated for any number of end components. The process of suboptimization for the above problem is shown in the next page.

D Nagesh Kumar, IISc, Bangalore

M5L1

Optimization Methods: Dynamic Programming - Introduction

Tank

Columns

Foundation

Original System

Tank

Columns

Foundation

Suboptimize design of Foundation component Tank Columns Foundation

Suboptimize design of Foundation & Columns together

Tank

Columns

Foundation

Optimize complete system

Fig 3.

D Nagesh Kumar, IISc, Bangalore

M5L1