OPTIMIZATION

TECHNIQUES
Prof. Shibdas Dutta,
Associate Professor,
DCG DATA CORE SYSTEMS INDIA PVT LTD
Kolkata
Objective Function,
Maxima,
Minima and
Saddle Points,
Convexity and
Concavity
Objectives
 Study the basic components of an optimization problem.

 Formulation of design problems as mathematical programming

problems.

 Define stationary points

 Necessary and sufficient conditions for the relative maximum of a

function of a single variable and for a function of two variables.

 Define the global optimum in comparison to the relative or local

optimum

 Determine the convexity or concavity of functions

Introduction - Preliminaries

Basic components of an optimization problem :

 An objective function expresses the main aim of the model which is
either to be minimized or maximized.
 Set of unknowns or variables which control the value of the objective
function.
 Set of constraints that allow the unknowns to take on certain values
but exclude others.

Optimization problem is then to:

Find values of the variables that minimize or maximize the objective

function while satisfying the constraints.
Objective Function

• Many optimization problems have a single objective function

• The two interesting exceptions are:

• No objective function: User does not particularly want to optimize

anything so there is no reason to define an objective function. Usually
called a feasibility problem.

• Multiple objective functions. In practice, problems with multiple objectives

are reformulated as single-objective problems by either forming a weighted
combination of the different objectives or by treating some of the objectives
by constraints.
Statement of an optimization problem
 x1 
 
 x2 
 . 
To find X=   which maximizes f(X)
 . 
x 
 n

Subject to the constraints

gi(X) <= 0 , i = 1, 2,….,m

lj(X) = 0 , j = 1, 2,….,p
Statement of an optimization problem…

where
 X is an n-dimensional vector called the design vector
 f(X) is called the objective function
 gi(X) and lj(X) are inequality and equality constraints,
respectively.
This type of problem is called a constrained optimization problem.

Optimization problems can be defined without any constraints as well

Unconstrained optimization problems.
Objective Function Surface

• If the locus of all points satisfying f(X) = a constant c is considered, it can

form a family of surfaces in the design space called the objective function
surfaces.

• When drawn with the constraint surfaces as shown in the figure we can
identify the optimum point (maxima).

• Possible graphically only when the number of design variable is two.

• When we have three or more design variables because of complexity in the

objective function surface we have to solve the problem as a mathematical
problem and this visualization is not possible.
Objective function surfaces to find the
optimum point (maxima)
Variables and Constraints

 Variables

 These are essential. If there are no variables, we cannot define the

objective function and the problem constraints.

 Constraints

 Even though constraints are not essential, it has been argued that almost all

problems really do have constraints.

 In many practical problems, one cannot choose the design variable

arbitrarily. Design constraints are restrictions that must be satisfied to

produce an acceptable design.
Constraints (contd.)

• Constraints can be broadly classified as :

• Behavioral or Functional constraints : Represent limitations on the

behavior and performance of the system.

• Geometric or Side constraints : Represent physical limitations on

design variables such as availability, fabricability, and transportability.
Constraint Surfaces

 Consider the optimization problem presented earlier with only inequality constraints

gi(X) . Set of values of X that satisfy the equation gi(X) forms a boundary surface in the

design space called a constraint surface.

 Constraint surface divides the design space into two regions: one with gi(X) < 0 (feasible

region) and the other in which gi(X) > 0 (infeasible region). The points lying on the hyper

surface will satisfy gi(X) =0.

The figure shows a hypothetical two-dimensional
design space where the feasible region is
denoted by hatched lines

Behavior
Infeasible constraint
region g2  0

Side
constraint Feasible region
g3 ≥ 0 Behavior
.
constraint
. g 1 0
Bound
acceptable point.

Free acceptable point

Free unacceptable Bound
point unacceptable
point.
Stationary Points: Functions of
Single and Two Variables
Stationary points

 For a continuous and differentiable function f(x) a stationary

point x* is a point at which the function vanishes,

i.e. f ’(x) = 0 at x = x*. x* belongs to its domain of definition.

 Stationary point may be a minimum, maximum or an
inflection (saddle) point
 Stationary points…

f x  0
f x   0 ,
f x   0 f x   0
f x   0

f x   0 ,
f x  0 , f x   0 f x   0 f x   0
f x  0 f x  0

(a) Inflection point (b) Minimum (c) Maximum

Relative and Global Optimum
• A function is said to have a relative or local minimum at x = x* iff ( x* )  f ( x  h )
for all sufficiently small positive and negative values of h, i.e. in the near
vicinity of the point x.

• A point x* is called a relative or local maximum if

f ( x* )  f ( x  h)for all values of h sufficiently close to zero.

*
f ( x )  f ( x)
• A function is said to have a global or absolute minimum at x = x if *
for all x in
the domain over which f(x) is defined.
f ( x* )  f ( x )
• A function is said to have a global or absolute maximum at x = x* if for all x in the
domain over which f (x) is defined.
Relative and Global Optimum…
A1, A2, A3 = Relative maxima
A2 = Global maximum
B1, B2 = Relative minima

.
B1 = Global minimum Relative minimum is
A2 also global optimum
f(x)

. .
A3
f(x)

A1
.
B2

. .
B1
x
a b x a b
Functions of a single variable

 Consider the function f(x) defined for a  x b

 [ a, b]
 To find the value of x *
such that x* maximizes f(x) we need to

solve a single-variable optimization problem.

Functions of a single variable …
 Necessary condition : For a single variable function f(x) defined for
x  [a, b] which has a relative maximum at x = x* , x*[a, b] if the
derivative f ‘(x) = df(x)/dx exists as a finite number at x = x* then
f ‘(x*) = 0.
 Above theorem holds good for relative minimum as well.
 Theorem only considers a domain where the function is continuous and
derivative.
 It does not indicate the outcome if a maxima or minima exists at a point
where the derivative fails to exist. This scenario is shown in the figure
below, where the slopes m1 and m2 at the point of a maxima are unequal,
hence cannot be found as depicted by the theorem.
Functions of a single variable ….
Salient features:
 Theorem does not consider if the
maxima or minima occurs at the
end point of the interval of
definition.
 Theorem does not say that the
function will have a maximum or
minimum at every point where
f ’(x) = 0, since this condition
f ’(x) = 0 is for stationary points
which include inflection points
which do not mean a maxima or
a minima.
Sufficient condition

• For the same function stated above let f ’(x*) = f ”(x*) = . . . = f (n-1)(x*) =

0, but f (n)(x*) ≠ 0, then it can be said that f (x*) is

• (a) a minimum value of f (x) if f (n)(x*) > 0 and n is even

• (b) a maximum value of f (x) if f (n)(x*) < 0 and n is even

• (c) neither a maximum or a minimum if n is odd

Example 1
Find the optimum value of the function f ( x) x 2  3x  5
and also state if the function attains a maximum or a
minimum.
Solution
f '( x) 2 x  3 0
For maxima or minima,

or x* = -3/2
f ''( x*) 2
is positive .

Hence the point x* = -3/2 is a point of minima and the function attains a
minimum value of -29/4 at this point.
Example 2
4
Find the optimum value of the function f ( x) ( x  2) and also state if
the function attains a maximum or a minimum

Solution:
f '( x) 4( x  2)3 0 or x = x* = 2 for maxima or minima.

f ''( x*) 12( x *  2) 2 0 at x* = 2

f '''( x*) 24( x *  2) 0 at x* = 2

f x *  24 at x* = 2

Hence fn(x) is positive and n is even hence the point x = x* = 2 is a point of

minimum and the function attains a minimum value of 0 at this point.
More examples can be found in the lecture notes
Functions of two variables
 Concept discussed for one variable functions may be easily extended to functions of
multiple variables.
 Functions of two variables are best illustrated by contour maps, analogous to
geographical maps.
 A contour is a line

representing a constant value

of f(x) as shown in the
following figure. From this
we can identify maxima,
minima and points of
inflection.

A contour plot
Necessary conditions
 From the above contour map, perturbations from points of local minima in any
direction result in an increase in the response function f(x), i.e. the slope of the
function is zero at this point of local minima.

 Similarly, at maxima and points of inflection as the slope is zero, the first derivative
of the function with respect to the variables are zero.
 Which gives us f 0; f 0 at the stationary points. i.e. the
x1 x2
x f
gradient vector of f(X), at X = X* = [x1 , x2] defined as follows, must equal
zero:  f 
 x ( *) 
 x f   0
1

 f 
 x (  *) 
 2 
Sufficient conditions
 Consider the following second order derivatives:

2 f 2 f  2 f
2
; 2;
x1 x2 x1x2

 Hessian matrix defined by H is formed using the above second order

derivatives.  2 f 2 f 
 2 
 x x1x2 
H  2
1
  f 2 f 
 2 
 x1x2 x2  [ x , x ]
1 2
Sufficient conditions …
The value of determinant of the H is calculated and
 if H is positive definite then the point X = [x1, x2] is a point of local

minima.

 if H is negative definite then the point X = [x1, x2] is a point of local

maxima.

 if H is neither then the point X = [x1, x2] is neither a point of maxima nor

minima.
Example
Locate the stationary points of f(X) and classify them as
relative maxima, relative minima or neither based on the
rules discussed in the lecture.
f ( X) 2 x13 / 3  2 x1 x2  5 x1  2 x22  4 x2  5

Solution
Example …
f
From (X) 0 ,
x1
22 x2  2   2 x2  5 0
2

8 x22  14 x2  3 0
(2 x2  3)(4 x2  1) 0
x2  3 / 2 or x2  1/ 4

So the two stationary points are

X1 = [-1,-3/2] and X2 = [3/2,-1/4]

Example …
2 f 2 f 2 f 2 f
The Hessian of f(X) is 2
4 x1 ; 2 4;   2
x1 x2 x1x2 x2 x1

 4 x  2
H  1 
2 4 

  4 x1 2
I - H 
2  4
At X1 = [-1,-3/2] ,  4 2
I - H  (  4)(  4)  4 0
2  4

 2  16  4 0
Since one eigen value is positive and
1  12 2  12 one negative, X1 is neither a relative
maximum nor a relative minimum
Example …

At X2 = [3/2,-1/4]

 6 2
I - H  (  6)(  4)  4 0
2  4

1 5  5 2 5  5

Since both the eigen values are positive, X­2 is a local minimum.

Minimum value of f(x) is -0.375

More examples can be found in the lecture notes
Convex and Concave Functions
Convex Function (Function of one
variable)
 A real-valued function f defined on an interval (or on any convex subset C
of some vector space) is called convex, if for any two points x and y in its
domain C and any t in [0,1], we have

f (ta  (1  t )b) tf (a )  (1  t ) f (b))

 A function is convex if and only if its epigraph (the set of points lying on or
above the graph) is a convex set. A function is also said to be strictly convex
if
f (ta  (1  t )b)  tf (a )  (1  t ) f (b)
for any t in (0,1) and a line connecting any two points on the function lies
completely above the function.
A convex function
Testing for convexity of a single variable
function

 A function is convex if its slope is non decreasing or

2 f / x 2 

0
2 2
 It isstrictly
f / x convex
 if its slope is continually increasing
or 0 throughout the function.
Properties of convex functions

• A convex function f, defined on some convex open interval C, is

continuous on C and differentiable at all or at many points. If C is closed,

then f may fail to be continuous at the end points of C.

• A continuous function on an interval C is convex if and only if

 a  b  f (a )  f (b)
f 
 2  2 for all a and b in C.

• A differentiable function of one variable is convex on an interval if and

only if its derivative is monotonically non-decreasing on that interval.
Properties of convex functions …
• A continuously differentiable function of one variable is convex on an interval if and
only if the function lies above all of its tangents: for all a and b in the interval.

• A twice differentiable function of one variable is convex on an interval if and only if

its second derivative is non-negative in that interval; this gives a practical test for
convexity.

• A continuous, twice differentiable function of several variables is convex on a

convex set if and only if its Hessian matrix is positive semi definite on the interior of
the convex set.

• If two functions f and g are convex, then so is any weighted combination af + b g

with non-negative coefficients a and b. Likewise, if f and g are convex, then the
function max{f, g} is convex.
Concave function (Function of one
variable)
• A differentiable function f is concave on an interval if its derivative
function f ′ is decreasing on that interval: a concave function has a
decreasing slope.

• A function f(x) is said to be concave on an interval if, for all a and b in

that interval,
t  [0,1], f (ta  (1  t )b) tf (a )  (1  t ) f (b)

• Additionally, f(x) is strictly concave if

t  [0,1], f (ta  (1  t )b)  tf (a )  (1  t ) f (b)
A concave function
Testing for concavity of a single variable
function

• A function is convex if its slope is non increasing or

2 f / x 2  0.

• It is strictly concave if its slope is continually

2 f / x 2 

decreasing or 0 throughout the function.

Properties of concave functions
• A continuous function on C is concave if and only if

 a  b  f (a )  f (b) for any a and b in C.

f 
 2  2

• Equivalently, f(x) is concave on [a, b] if and only if the function −f(x) is convex
on every subinterval of [a, b].

• If f(x) is twice-differentiable, then f(x) is concave if and only if

f ′′(x) is non-positive. If its second derivative is negative then it is strictly concave,

but the opposite is not true, as shown by f(x) = -x4.
Example
5 4 3
Locate the stationary points of f ( x) 12 x  45 x  40 x  5 and find out if
the function is convex, concave or neither at the points of optima based on the
testing rules discussed above.
Solution:

f '( x) 60 x 4  180 x 3  120 x 2 0

 x 4  3 x 3  2 x 2 0
or x 0,1, 2
Consider the point x = x* = 0
f '' ( x* ) 240( x* )3  540( x* ) 2  240 x* 0 at x * = 0

f ''' ( x* ) 720( x* )2  1080 x*  240 240 at x * = 0

Example …
Since the third derivative is non-zero, x = x* = 0 is neither a point of
maximum or minimum , but it is a point of inflection. Hence the function is
neither convex nor concave at this point.

Consider the point x = x* = 1

f x*  240x*   540x*   240x*   60
3 2
at x * = 1

Since the second derivative is negative, the point x = x* = 1 is a point of local

maxima with a maximum value of f(x) = 12 – 45 + 40 +5 = 12. At the point the
2 f
function is concave since 0
x 2
Example …
Consider the point x = x* = 2

f x*  240x*   540x*   240x*  240

3 2
at x * = 2

Since the second derivative is positive, the point x = x* = 2 is a point of local

minima with a minimum value of f(x) = -11. At the point the function is
2 f
concave since 2
0
x
Functions of two variables

• A function of two variables, f(X) where X is a vector = [x1,x2], is

strictly convex if

f (t 1  (1  t )  2 )  tf ( 1 )  (1  t ) f (  2 )

• where X1 and X2 are points located by the coordinates given in their

respective vectors. Similarly a two variable function is strictly concave
if f (t 1  (1  t )  2 )  tf ( 1 )  (1  t ) f (  2 )
Contour plot of a convex function
Contour plot of a concave function
Sufficient conditions

• To determine convexity or concavity of a function of multiple variables,

the eigen values of its Hessian matrix is examined and the following
rules apply.
• If all eigen values of the Hessian are positive the function is strictly
convex.
• If all eigen values of the Hessian are negative the function is
strictly concave.
• If some eigen values are positive and some are negative, or if some
are zero, the function is neither strictly concave nor strictly convex.
Example
Locate the stationary points of
f ( X) 2 x13 / 3  2 x1 x2  5 x1  2 x22  4 x2  5
and find out if the function is convex, concave or neither at the points of
optima based on the rules discussed in this lecture.

Solution:
 f 
 x ( *)   2 x 2  2 x  5   0
 x f    1
1 2
  
 f    2 x1  4 x2  4   0 
 x (  *) 
 2 

Solving the above the two stationary points are

X1 = [-1,-3/2] and X2 = [3/2,-1/4]
Example…
Example…
• Since one eigen value is positive and one negative, X 1 is neither a relative

maximum nor a relative minimum. Hence at X1 the function is neither convex nor
concave.
• At X2 = [3/2,-1/4]

• Since both the eigen values are positive, X­2 is a local minimum
• Function is convex at this point as both the eigen values are positive.
Lagrange Multipliers and
Kuhn-tucker Conditions
Objectives

 To study the optimization with multiple decision variables

and equality constraint : Lagrange Multipliers.

 To study the optimization with multiple decision variables

and inequality constraint : Kuhn-Tucker (KT) conditions

Constrained optimization with equality
constraints

A function of multiple variables, f(x), is to be optimized subject to one or

more equality constraints of many variables. These equality constraints, gj(x),

may or may not be linear. The problem statement is as follows:

Maximize (or minimize) f(X), subject to gj(X) = 0, j = 1, 2, … , m

 x1 
where x 
 
X  2 
(1)
 
 xn 
Constrained optimization…

 With the condition that or 

;m n if m > n then the problem
else

becomes an over defined one and there will be no solution. Of the

many available methods, the method of constrained variation and the

method of using Lagrange multipliers are discussed.

 Solution by method of Lagrange multipliers
• Continuing with the same specific case of the optimization problem with n = 2 and m = 1 we define a
quantity λ, called the Lagrange multiplier as
f / x2
  (2)
g / x2 (x1* , x 2* )

• Using this in the constrained variation of f [ given in the previous lecture]

 f g / x1 f 
df    dx1 0
 x1 g / x2 x2  (x *, x *) 1 2

 f g 
    0
And (2) written as 
 1x x1  (x * , x * ) (3)
1 2

 f g 
    0
 x2 x2  (x * , x * )
1 2
(4)
Solution by method of Lagrange
multipliers…

• Also, the constraint equation has to be satisfied at the extreme point

g ( x1 , x2 ) ( x * , x * )  0 (5)
1 2

• Hence equations (2) to (5) represent the necessary conditions for the point
[x1*, x2*] to be an extreme point.

g / x1 g / x1

• λ could be expressed in terms of as well has to be non-zero.

• These necessary conditions require that at least one of the partial derivatives
of g(x1 , x2) be non-zero at an extreme point.
Solution by method of Lagrange
multipliers…
• The conditions given by equations (2) to (5) can also be generated by
constructing a functions L, known as the Lagrangian function, as

(6)
L( x1 , x2 ,  )  f ( x1 , x2 )   g ( x1 , x2 )
• Alternatively, treating L as a function of x1,x2 and , the necessary
conditions for its extremum are given by

L f g
( x1 , x2 ,  )  ( x1 , x2 )   ( x1 , x2 ) 0
x1 x1 x1
L f g
( x1 , x2 ,  )  ( x1 , x2 )   ( x1 , x2 ) 0
x2 x2 x2
(7)
L
( x1 , x2 ,  )  g ( x1 , x2 ) 0

Necessary conditions for a general problem
• For a general problem with n variables and m equality constraints the
problem is defined as shown earlier

• Maximize (or minimize) f(X), subject to gj(X) = 0, j = 1, 2, … , m

 x1 
x 
 
X  2 
where  
 xn 

• In this case the Lagrange function, L, will have one Lagrange multiplier j
for Leach
( x1 , xconstraint as
2 ,..., xn , 1 , 2 ,..., m )  f ( X)  1 g1 ( X)  2 g 2 ( X)  ...   m g m ( X)

(8)
Necessary conditions for a general problem…
• L is now a function of n + m unknowns, x1 , x2 ,..., xn , 1 , 2 ,...,
, and mthe

necessary conditions for the problem defined above are given by

L f m g j
 ( X)    j ( X) 0, i 1, 2,..., n; j 1, 2,..., m
xi xi j 1 xi

L (9)
 g j ( X) 0, j 1, 2,..., m
 j

• which represent n + m equations in terms of the n + m unknowns, xi and j. The

solution to this set of equations gives us
 x1*   1* 
 *  *
 x2   2 
X    
*

and  (10)
   
 *   *
 m 
x
 n 
Sufficient conditions for a general problem

• A sufficient condition for f(X) to have a relative minimum at X* is that each

root of the polynomial in Є, defined by the following determinant equation
be positive.
L11   L12  L1n g11 g 21  g m1
L21 L22   L2 n g12 g 22 gm2
     
Ln1 Ln 2  Lnn   g1n g 2 n  g mn
0
g11 g12  g1n 0   0
(11)
g 21 g 22 g2n   
    
g m1 gm2  g mn 0   0
Sufficient conditions for a general problem…

where 2 L
Lij  ( X* ,  * ), for i 1, 2,..., n and j 1, 2,..., m
xi x j (12)
g p
g pq  ( X* ), where p 1, 2,..., m and q 1,2,..., n
xq

Similarly, a sufficient condition for f(X) to have a relative maximum at X* is

that each root of the polynomial in Є, defined by equation (11) be negative.

• If equation (11), on solving yields roots, some of which are positive and
others negative, then the point X* is neither a maximum nor a minimum.
Example
Minimize f ( X)  3x12  6 x1 x2  5 x22  7, x1  5 x2
Subject to x1  x2 5

Solution
g1 ( X)  x1  x2  5 0
L( x1 , x2 ,..., xn , 1 , 2 ,..., m )  f ( X)  1 g1 ( X)  2 g 2 ( X)  ...  m g m ( X)
with n = 2 and m = 1
 3 x  6 x1 x2  5 x  7 x1  5 x2  1 ( x1  x2  5)
2 2
L= 1 2

L
 6 x1  6 x2  7  1 0
x1
1
 x1  x2  (7  1 )
6
1
 5  (7  1 ) 1 23
6 or
Example… L
 6 x1  10 x2  5  1 0
x2
1
 3 x1  5 x2  (5  1 )
2
1
 3( x1  x2 )  2 x2  (5  1 )
2
1 11
x2  x1 
2 2

 11  1 
Hence, X*  ,  ; λ*  23
2 2

 L11   L12 g11 

 
 21L L22   g 21  0
 g g12 0 
 11
Example…
2 L 2 L 2 L
L11  2  6 L12 L21   6 L22  2  10
x1 ( X*,λ*) x1x2 ( X*,λ* )
x2 ( X*,λ*)

g1
g11  1
x1 ( X*,λ* )

g1
g12  g 21  1
x2 ( X*,λ* )

The determinant becomes   6  6 1

 
 6  10  1  0
 1 1 0 

or ( 6 )[ 1]  ( 6)[ 1]  1[ 6  10 ] 0
 2
Since  is negative, X*, λ * correspond to a maximum
Kuhn – Tucker Conditions

 KT condition: Both necessary and sufficient if the objective function

is concave and each constraint is linear or each constraint function is

concave, i.e., the problems belongs to a class called the convex

programming problems.
Kuhn-Tucker Conditions: Optimization
Model
Consider the following optimization problem
Minimize f(X)

subject to
gj(X) ≤ 0 for j=1,2,…,p

where the decision variable vector

X=[x1,x2,…,xn]
Kuhn-Tucker Conditions
Kuhn-Tucker conditions for X* = [x1* , x2* , . . . xn*] to be a local minimum are

f m
g
  j 0 i 1, 2,..., n
xi j 1 xi
 j g j 0 j 1, 2,..., m
g j 0 j 1, 2,..., m
 j 0 j 1, 2,..., m
Kuhn Tucker Conditions …

 In case of minimization problems, if the constraints are of

the form gj(X) ≥ 0, then λj have to be non-positive

 If the problem is one of maximization with the constraints

in the form gj(X) ≥ 0, then λj have to be nonnegative.

Example 1

Minimize
2 2 2
f x  2 x  3x
1 2 3

subject to

g1  x1  x2  2 x3 12
g 2  x1  2 x2  3 x3 8
Example 1…
Kuhn – Tucker Conditions
f g g 2 x1  1  2 0 (14)
 1 1  2 2 0 4 x2  1  22 0 (15)
xi xi xi
6 x3  21  32 0 (16)

 j g j 0 1 ( x1  x2  2 x3  12) 0 (17)
2 ( x1  2 x2  3x3  8) 0 (18)

x1  x2  2 x3  12 0 (19)
g j 0
x1  2 x2  3 x3  8 0 (20)

1 0 (21)
 j 0 2 0 (22)
Example 1…
From (17) either 1 = 0 or x1  x2  2 x3  12 0 ,
1
Case 1: =0
 2 / 2 2 / 2
From (14), (15) and (16) we have x1 = x2 = and x3 =
  82 0,  2 0 or  8
2
2
Using these in (18) we get
2 0 2
From (22), , therefore, =0,

Therefore, X* = [ 0, 0, 0 ]

This solution set satisfies all of (18) to (21)

Example 1…
Case 2: x1  x2  2 x3  12 0
 1  2 1  22 21  32
Using (14), (15) and (16), we have    12 0
2 4 3
or 171  122  144

 But conditions (21) and (22) give us 1 0 and 2 0

17which
simultaneously, 1  122  144
cannot be
possible with

Hence the solution set for this optimization problem is

X* = [ 0 0 0 ]
Example 2

Minimize

f  x12  x22  60 x1
subject to

g1  x1  80 0
g 2  x1  x2  120 0
Example 2…
Kuhn – Tucker Conditions

f g g 2 x1  60  1  2 0 (23)
 1 1  2 2 0
xi xi xi 2 x2  2 0 (24)

1 ( x1  80) 0 (25)
 j g j 0 2 ( x1  x2  120) 0 (26)

x1  80 0 (27)
g j 0 x1  x2  120 0 (28)

1 0 (29)
 j 0
2 0 (30)
Example 2…

From (25) either 1 = 0 or ( x1  80) 0 ,

Case 1
2 x2 
2
x1   30 2
From (23) and (24) we have 2 and
2 2  150  0
Using these in (26) we get
 2 0 or  150
2 0
Considering , X* = [ 30, 0]. But this solution set violates (27)

and (28)
  150
2

For , X* = [ 45, 75]. But this solution set violates (27)

Example 2…

Case 2: ( x1  80) 0

Using x1 80 in (23) and (24), we have

2  2 x2
1 2 x2  220 (31)

Substitute (31) in (26), we have 2 x  x2  40  0

2

For this to be true, either x 0 or x2  40 0

2
Example 2…

Forx2 0 , 1  220

This solution set violates (27) and (28)

Forx  40 0 ,  140 and 2  80

2 1

This solution set is satisfying all equations from (27) to (31) and hence

the desired

Thus, the solution set for this optimization problem is

X* = [ 80, 40 ]
BIBLIOGRAPHY / FURTHER READING

1. Rao S.S., Engineering Optimization – Theory and Practice, Fourth

Edition, John Wiley and Sons, 2009.
2. Ravindran A., D.T. Phillips and J.J. Solberg, Operations Research –
Principles and Practice, John Wiley & Sons, New York, 2001.
3. Taha H.A., Operations Research – An Introduction, 8th edition, Pearson
Education India, 2008.
4. Vedula S., and P.P. Mujumdar, Water Resources Systems: Modelling
Techniques and Analysis, Tata McGraw Hill, New Delhi, 2005.
(ii) Constrained and Unconstrained
Optimization
Objectives

 To study the optimization of functions of multiple variables without

optimization.

 To study the above with the aid of the gradient vector and the Hessian matrix.

 To discuss the implementation of the technique through examples

 To study the optimization of functions of multiple variables subjected to

equality constraints using

 the method of constrained variation

Unconstrained optimization

If a convex function is to be minimized, the stationary point is

the global minimum and analysis is relatively straightforward

A similar situation exists for maximizing a concave variable

function.

Necessary and sufficient conditions for the optimization of

unconstrained function of several variables
Necessary condition

• In case of multivariable functions a necessary condition for a stationary

point of the function f(X) is that each partial derivative is equal to zero.

• Each element of the gradient vector defined below must be equal to

x f
zero. i.e. the gradient vector of f(X), at X=X*, defined as follows,
 f * 
must be equal to zero:  x (  ) 
 1

 f * 
 x (  )
 x f   0
2

  
 
  
 f *

 ( )
 dxn 
Sufficient condition

 For a stationary point X* to be an extreme point, the matrix of second

partial derivatives (Hessian matrix) of f(X) evaluated at X* must be:
 positive definite when X* is a point of relative minimum, and
 negative definite when X* is a relative maximum point.

 When all eigen values are negative for all possible values of X, then X*
is a global maximum, and when all eigen values are positive for all
possible values of X, then X* is a global minimum.

 If some of the eigen values of the Hessian at X* are positive and some
negative, or if some are zero, the stationary point, X*, is neither a local
maximum nor a local minimum.
Example
Analyze the function f ( x)  x12  x22  x32  2 x1 x2  2 x1 x3  4 x1  5 x3  2

and classify the stationary points as maxima, minima and points of inflection

Solution
 f * 
 ( ) 
 x1    2 x1  2 x2  2 x3  4   0
 f * 
 x f  (  )    2 x2  2 x1 
  0 
 x2    2 x3  2 x1  5   0
 f * 
 ( ) 
 x3 
Example …
Solving these simultaneous equations we get

X*=[1/2, 1/2, -2]

2 f 2 f 2 f
 2; 2  2; 2  2
x12 x2 x3
2 f 2 f
 2
x1x2 x2x1

2 f 2 f
 0
x2 x3 x3x2
2 f 2 f
 2
x3x1 x1x3
Example …

Hessian of f(X) is
 2 f 
H  
x x
 i j 

 2 2 2 
H  2  2 0 
 2 0  2 

 2 2 2
I - H   2  2 0 0
2 0  2
Example …

or (  2)(  2)(  2)  2(  2)(2)  2(2)(  2) 0

(  2)[ 2  4  4  4  4] 0

(  2)3 0

or 1 2 3  2
Since all eigen values are negative the function attains a maximum at the point

X*=[1/2, 1/2, -2]

Constrained optimization

A function of multiple variables, f(x), is to be optimized subject to one or

more equality constraints of many variables. These equality constraints,

gj(x), may or may not be linear. The problem statement is as follows:

Maximize (or minimize) f(X), subject to gj(X) = 0, j = 1, 2, … , m

 x1 
where x 
 
X  2 
 
 xn 
Constrained optimization…

 With the condition that m n; or else if m > n then the problem

becomes an over defined one and there will be no solution. Of the many

available methods, the method of constrained variation is discussed

 Another method using Lagrange multipliers will be discussed in the next

lecture.
Solution by Method of Constrained
Variation
• For the optimization problem defined above, consider a specific case with n
= 2 and m = 1

• The problem statement is as follows:

Minimize f(x1,x2), subject to g(x1,x2) = 0

• For f(x1,x2) to have a minimum at a point X* = [x1*,x2*], a necessary

condition is that the total derivative of f(x1,x2) must be zero at [x1*,x2*].

f f
df  dx1  dx2 0
x1 x2
(1)
Method of Constrained Variation…
• Since g(x1*,x2*) = 0 at the minimum point, variations dx1 and dx2 about the

point [x1*, x2*] must be admissible variations, i.e. the point lies on the

constraint:

g(x1* + dx1 , x2* + dx2) = 0 (2)

assuming dx1 and dx2 are small the Taylor’s series expansion of this gives
g ( x1 * dx1 , x2*  dx2 )
us
g * * g * *
g ( x1* , x2* )  (x1 ,x 2 ) dx1  (x1 ,x 2 ) dx2 0
x1 x2

(3)
Method of Constrained Variation…

g g
or dg  dx1  dx2 0 at [x1*,x2*] (4)
x1 x2

which is the condition that must be satisfied for all admissible

variations.
g / x2 0
Assuming , (4) can be rewritten as

g / x1 * *
dx2  ( x1 , x2 )dx1
g / x2
(5)
Method of Constrained Variation…
(5) indicates that once variation along x1 (dx1) is chosen arbitrarily, the variation along x2 (dx2) is

decided automatically to satisfy the condition for the admissible variation. Substituting equation
(5) in (1) we have:
 f g / x1 f 
df    dx1 0
 x1 g / x2 x2  (x1* , x 2* )
(6)

The equation on the left hand side is called the constrained variation of f. Equation (5) has to be

satisfied for all dx1, hence we have

 f g f g 
   0
 x1 x2 x2 x1  (x1* , x 2* ) (7)

This gives us the necessary condition to have [x1*, x2*] as an extreme point (maximum or minimum)