Introduction to Optimization

Lee T. Ostrom, Ph.D., CSP, CPE, CTM

Ostrom@uidaho.edu
208 7575427
 Optimization literature
Textbook:

Introduction to Linear Optimization

Bertsimas and Tsitsiklis
Athena Scientific
ISBN 978-1-886529-19-9

Other materials supplied on BBLearn

2
 Optimization literature
Journals:

1. Engineering Optimization
2. ASME Journal of Mechnical Design
3. AIAA Journal
4. ASCE Journal of Structural Engineering
5. Computers and Structures
6. International Journal for Numerical Methods in Engineering
7. Structural Optimization
8. Journal of Optimization Theory and Applications
9. Computers and Operations Research
10. Operations Research and Management Science

3
 BBLearn Website
• A quick look at the BBLearn website
 Course
Module Topics Readings Assignments Due Date
Objectives
Introduction to
Optimization and Problems 1.14, September
1 1, 2 and 3 Chapter 1
the mathematics 1.15, 1.17 8, 2016
of Optimization
Problems 3.12,
September
2 Simplex Method 1, 2 and 3 Chapter 3 3.18, 3.19, and
29, 2016
3.20
Problems 4.1, October 13,
3 Duality Theory 1 Chapter 4
4.3, 4.5, and 4.9 2016
Sensitivity Problems 5.1, October 20,
4 1, 2, and 3 Chapter 5
Analysis 5.5, and 5.6 2016
Midter Week of
  Covering Chapters 1 and 3 to 5
m October 24
Large Scale October 27,
5 1, 2, and 3 Chapter 6 Problem 6.3
Optimization 2016
Problems 7.1, November
6 Network Flows 1, 2, 3, and 4 Chapter 7
7.3, 7.8 and 7.11 17, 2016
The Ellipsoid Chapter 8.1 Demonstration in
7 1 NA
method to 8.4 class
Write a 3-4 page
paper on the
practical aspects
Optimization and
Materials of optimization December
8 Engineering 4
supplied and how it can be 9, 2016
Management
used in
Engineering
management
Final Project

Find an activity, collect data on how the activity is

performed, use one of the optimization methods
you learned during the course and see if you can
optimize it. You will need to describe the activity. December
  Final Project There is not right or wrong answer. I am estimating 16, 2016
the paper will be approximately 10 pages. One
example would be optimizing shopping in a grocery
store.

 
Week of
Final
  Primarily covering chapters 6 and 7 December
Exam
12th

5
 Optimization
• The final grade will be calculated as
follows:
• Graded homework will be worth 40% of the
grade
• Midterm exam will be worth 20%
• Final Exam 20%
• Final paper will be worth 20%

6
 Schedule for the Semester
• Week of September 6th – Materials supplied
on BBLearn
•  
• For October 17th and October 31st I will pre-
record lectures
•  
• The week of October 24th there will be an
exam.
 Questions
• If you have a technical or logistical
question I will prefer you provide those at
the end of class and I will provide the
question and answer on the BBLearn
website and during the next week’s
classes.
 Optimization

• Towards the end of the class we will

demonstrate how to use Matlab for doing
optimization.

9
 Optimization
• Optimization is the act of obtaining the best result under given
circumstances.

• Optimization can be defined as the process of finding the conditions that

give the maximum or minimum of a function.

• The optimum seeking methods are also known as mathematical

programming techniques and are generally studied as a part of operations
research.

• Operations research is a branch of mathematics concerned with the

application of scientific methods and techniques to decision making
problems and with establishing the best or optimal solutions.
10
 Optimization
• Operations research or operational research (OR) is an
interdisciplinary branch of mathematics which uses methods like:

– mathematical modeling
– statistics
– algorithms to arrive at optimal or good decisions in complex problems
which are concerned with optimizing the maxima (profit, faster
assembly line, greater crop yield, higher bandwidth, etc) or minima
(cost loss, lowering of risk, etc) of some objective function.

• The eventual intention behind using operations research is to elicit

a best possible solution to a problem mathematically, which
improves or optimizes the performance of the system.

11
Optimization

12
 Optimization
Historical development

• Isaac Newton (1642-1727)

(The development of differential calculus
methods of optimization)

• Joseph-Louis Lagrange (1736-1813)

(Calculus of variations, minimization of functionals,
method of optimization for constrained problems)

• Augustin-Louis Cauchy (1789-1857)

(Solution by direct substitution, steepest
descent method for unconstrained optimization)

13
 Optimization
Historical development

• Leonhard Euler (1707-1783)

(Calculus of variations, minimization of
functionals)

• Gottfried Leibnitz (1646-1716)

(Differential calculus methods
of optimization)

14
 Optimization
Historical development

• George Bernard Dantzig (1914-2005)

(Linear programming and Simplex method (1947))

• Richard Bellman (1920-1984)

(Principle of optimality in dynamic
programming problems)

• Harold William Kuhn (1925-)

(Necessary and sufficient conditions for the optimal solution of
programming problems, game theory)

15
 Optimization
Historical development

• Albert William Tucker (1905-1995)

(Necessary and sufficient conditions
for the optimal solution of programming
problems, nonlinear programming, game
theory: his PhD student
was John Nash)

• Von Neumann (1903-1957)

(game theory)

16
 Why is Optimization Important
• Here are a couple examples as to why
optimizing processes is important:

https://flightaware.com/live/
Food Manufacturer
 Food Manufacturer
• Manufacture 9 products in two different
packaging types
• Use two processing lines
• Use a Kanban system using Kaizen TQM
principles
• Want to keep number of employees low
Food Manufacturer
Food Manufacturer
 Decision Making

Examples:

• determining which ingredients and in what quantities to add

to a mixture being made so that it will meet specifications on
its composition
• allocating available funds among various competing agencies
• deciding which route to take to go to a new location in the
city

• Decision making always involves making a choice between

various possible alternatives
 Categories of Decision Making Problems

Category 1:
• The set of possible alternatives for the decision is a finite
discrete set typically consisting of a small number of
elements.
– Example: “A teenage girl knows four boys all of whom she likes, and
has to decide who among them to go steady with.”
• Solution: scoring methods

Category 2:
• The number of possible alternatives is either infinite, or finite
but very large, and the decision may be required to satisfy
some restrictions and constraints
• Solution: unconstrained and constrained optimization
methods
 The Scoring Method – an example
Rita has been dating 4 boys off and on over the last 3 years,
and has come to know each of them very well. Who among
the four boys would be her best choice?
Category 2 Decision problems

1. Get a precise definition of the problem, all relevant data

and information on it.
• Uncontrollable factors (random variables)
• Controllable inputs (decision variables)

2. Construct a mathematical (optimization) model of the

problem.
• Build objective functions and constraints

3. Solve the model

• Apply the most appropriate algorithms for the given problem

4. Implement the solution

 Optimization
• Mathematical optimization problem:
minimize f 0 ( x)
subject to g i ( x)  bi , i  1,...., m
• f0 : Rn R: objective function
• x=(x1,…..,xn): design variables (unknowns of the problem,
they must be linearly independent)
• gi : Rn R: (i=1,…,m): inequality constraints

26
• The problem is a constrained optimization problem
 Optimization
• If a point x* corresponds to the minimum value of the function f (x), the
same point also corresponds to the maximum value of the negative of
the function, -f (x). Thus optimization can be taken to mean
minimization since the maximum of a function can be found by seeking
the minimum of the negative of the same function.

27
 Optimization
Constraints

• Behaviour constraints: Constraints that represent limitations on the

behaviour or performance of the system are termed behaviour or
functional constraints.

• Side constraints: Constraints that represent physical limitations on

design variables such as manufacturing limitations.

28
 Optimization
Constraint Surface
• For illustration purposes, consider an optimization problem with only
inequality constraints gj (X)  0. The set of values of X that satisfy the
equation gj (X) =0 forms a hypersurface in the design space and is
called a constraint surface.

29
 Optimization
Constraint Surface
• Note that this is a (n-1) dimensional subspace, where n is the number
of design variables. The constraint surface divides the design space
into two regions: one in which gj (X)  0and the other in which gj (X)
0.

30
 Optimization
Constraint Surface
• Thus the points lying on the hypersurface will satisfy the constraint
gj (X) critically whereas the points lying in the region where gj (X) >0
are infeasible or unacceptable, and the points lying in the region
where gj (X) < 0 are feasible or acceptable.

31
 Optimization
Constraint Surface
• In the below figure, a hypothetical two dimensional design space is
depicted where the infeasible region is indicated by hatched lines. A
design point that lies on one or more than one constraint surface is
called a bound point, and the associated constraint is called an active
constraint.

32
 Optimization
Constraint Surface
• Design points that do not lie on any constraint surface are known as
free points.

33
 Optimization
Constraint Surface

Depending on whether a particular

design point belongs to the
acceptable or unacceptable
regions, it can be identified as one
of the following four types:

• Free and acceptable point

• Free and unacceptable point

• Bound and acceptable point

34
• Bound and unacceptable point
 Optimization
• The conventional design procedures aim at finding an acceptable or
adequate design which merely satisfies the functional and other
requirements of the problem.

• In general, there will be more than one acceptable design, and the
purpose of optimization is to choose the best one of the many
acceptable designs available.

• Thus a criterion has to be chosen for comparing the different

alternative acceptable designs and for selecting the best one.

• The criterion with respect to which the design is optimized, when

expressed as a function of the design variables, is known as the
objective function.

35
 Optimization
• In civil engineering, the objective is usually taken as the minimization
of the cost.

• In mechanical engineering, the maximization of the mechanical

efficiency is the obvious choice of an objective function.

• In aerospace structural design problems, the objective function for

minimization is generally taken as weight.

• In some situations, there may be more than one criterion to be

satisfied simultaneously. An optimization problem involving multiple
objective functions is known as a multiobjective programming
problem.

36
 Optimization
• With multiple objectives there arises a possibility of conflict, and one
simple way to handle the problem is to construct an overall objective
function as a linear combination of the conflicting multiple objective
functions.

• Thus, if f1 (X) and f2 (X) denote two objective functions, construct a new
(overall) objective function for optimization as:

f ( X)   1 f 1 ( X)   2 f 2 ( X)

where 1 and 2 are constants whose values indicate the relative

importance of one objective function to the other.
37
 Optimization
• The locus of all points satisfying f (X) = c = constant forms a hypersurface
in the design space, and for each value of c there corresponds a different
member of a family of surfaces. These surfaces, called objective function
surfaces, are shown in a hypothetical two-dimensional design space in the
figure below.

38
 Optimization
• Once the objective function surfaces are drawn along with the constraint
surfaces, the optimum point can be determined without much difficulty.
• But the main problem is that as the number of design variables exceeds
two or three, the constraint and objective function surfaces become
complex even for visualization and the problem has to be solved purely as a
mathematical problem.

39
Optimizing Giapetto Problem

Graphical Method
 Example: Giapetto woodcarving Inc.,

• Giapetto Woodcarving, Inc., manufactures two types

of wooden toys: soldiers and trains. A soldier sells
for $27 and uses $10 worth of raw materials. Each
soldier that is manufactured increases Giapetto’s
variable labor and overhead cost by $14. A train sells
for $21 and uses $9 worth of raw materials. Each
train built increases Giapetto’s variable labor and
overhead cost by $10.

41
 Example: Giapetto woodcarving Inc.,
• The manufacture of wooden soldiers and trains
requires two types of skilled labor: carpentry and
finishing. A soldier requires 2 hours of finishing labor
and 1 hour of carpentry labor. A train requires 1
hour of finishing and 1 hour of carpentry labor. Each
week, Giapetto can obtain all the needed raw
material but only 100 finishing hours and 80
carpentry hours.
• Demand for trains is unlimited, but at most 40
soldiers are bought each week. Giapetto wants to
maximize weekly profit. Formulate a linear
programming model of Giapetto’s situation that can
be used to maximize Giapetto’s weekly profit.

42
 Solution: Giapetto woodcarving Inc.,
• Step 1: Model formulation
1. Decision variables: we begin by finding the
decision variables. In any LP, the decision
variables should completely describe the decisions
to be made. Clearly, Giapetto must decide how
many soldiers and trains should be manufactured
each week. With this in mind, we define:
X1 = number of soldiers produced each week
X2 = number of trains produced each week

43
 Solution: Giapetto woodcarving Inc.,
2. Objective function: in any LP, the decision maker
wants to maximize (usually revenue or profit) or
minimize (usually costs) some function of the
decision variables. The function to be maximized or
minimized is called the objective function. For the
Giapetto problem, we will maximize the net profit
(weekly revenues – raw materials cost – labor and
overhead costs).
Weekly revenues and costs can be expressed in terms
of the decision variables, X1 and X2 as following:

44
 Solution: Giapetto woodcarving Inc.,
• Weekly revenues = weekly revenues from soldiers +
weekly revenues from trains
= 27 X1 + 21 X2
Also,
Weekly raw materials costs = 10 X1 + 9 X2
Other weekly variable costs = 14 X1 + 10 X2
Therefore, the Giapetto wants to maximize:
(27 X1 + 21 X2) – (10 X1 + 9 X2) – (14 X1 + 10 X2) = 3 X1 + 2
X2
Hence, the objective function is:
Maximize Z = 3 X1 + 2 X2 45
 Solution: Giapetto woodcarving Inc.,
3. Constraints: as X1 and X2 increase, Giapetto’s
objective function grows larger. This means that if
Giapetto were free to choose any values of X1 and
X2, the company could make an arbitrarily large
profit by choosing X1 and X2 to be very large.
Unfortunately, the values of X1 and X2 are limited
by the following three restrictions (often called
constraints):
Constraint 1: each week, no more than 100 hours of
finishing time may be used.
Constraint 2: each week, no more than 80 hours of
carpentry time may be used.
Constraint 3: because of limited demand, at most 40
soldiers should be produced.
46
 Solution: Giapetto woodcarving Inc.,
• The three constraints can be expressed in terms of
the decision variables X1 and X2 as follows:
Constraint 1: 2 X1 + X2  100
Constraint 2: X1 + X2  80
Constraint 3: X1  40
Note:
The coefficients of the decision variables in the
constraints are called technological coefficients.
This is because its often reflect the technology used
to produce different products. The number on the
right-hand side of each constraint is called Right-
Hand Side (RHS). The RHS often represents the
quantity of a resource that is available.
47
 Solution: Giapetto woodcarving Inc.,
• Sign restrictions: to complete the formulation of the
LP problem, the following question must be
answered for each decision variable: can the
decision variable only assume non-negative values,
or it is allowed to assume both negative and positive
values?
If a decision variable Xi can only assume a nonnegative
values, we add the sign restriction (called non-
negativity constraints)
Xi  0.
If a variable Xi can assume both positive and negative
values (or zero), we say that Xi is unrestricted in sign
(urs).
In our example the two variables are restricted in sign,
i.e., X1  0 and X2  0 48
 Solution: Giapetto woodcarving Inc.,
• Combining the non-negativity constraints
with the objective function and the
structural constraints yield the following
optimization model (usually called LP
model):
Max Z = 3 X1 + 2 X2 (objective function)
subject to (st)
2 X1 + X2  100 (finishing constraint)
X1 + X2  80 (carpentry constraint)
X1  40 (soldier demand constraint)
X1  0 and X2  0 (non-negativity constraint)
The optimal solution to this problem is :
X1 = 20, and X2 = 60, Z = 180 49
 First Graph Linear Equations
• Graph the 3 equations

2X1 + X2 ≤ 100
X1 + X2 ≤ 80
X1 ≤ 40
Each square = 10

Line for
2X1 + X2 ≤ 100
Each square = 10
Region for
2X1 + X2 ≤ 100
Each square = 10
Line for
X1 + X2 ≤ 80
Each square = 10
Region for
X1 + X2 ≤ 80
Each square = 10
Line for
X1 ≤ 40
Each square = 10
Region for
X1 ≤ 40
 The 4 points
Each square = 10 that represent
the corners of
the region are:
X1 = 0
X2 = 80

X1 = 20
X2 = 60
X1 = 40
X2 = 0

X1 = 40
X2 = 20
 Testing
• We next test the points in the objective function
to see which maximizes the process:

3 X1 + 2 X2 = Z
3 (0) + 2(80) = 160
3(20) + 2(60) = 180
3(40) + 2(20) = 140
3(40) + 2(0) = 120
 Testing
• We see that the points X1 = 20 and X2 = 60
maximizes the objective function:

3 X1 + 2 X2 = Z
3 (0) + 2(80) = 160
3(20) + 2(60) = 180
3(40) + 2(20) = 140
3(40) + 2(0) = 120
 Solution
• Therefore the optimal solution to this
problem is to produce:

20 soldiers and 60 trains per week

 Examples

Design of civil engineering structures

• variables: width and height of member cross-sections
• constraints: limit stresses, maximum and minimum dimensions
• objective: minimum cost or minimum weight

Analysis of statistical data and building empirical models from

measurements
• variables: model parameters
• Constraints: physical upper and lower bounds for model parameters
• Objective: prediction error

61
 Classification of optimization problems

Classification based on:

• Constraints
– Constrained optimization problem
– Unconstrained optimization problem

• Nature of the design variables

– Static optimization problems
– Dynamic optimization problems

62
 Classification of optimization problems

Classification based on:

• Physical structure of the problem

– Optimal control problems
– Non-optimal control problems

• Nature of the equations involved

– Nonlinear programming problem
– Geometric programming problem
– Quadratic programming problem
– Linear programming problem

63
 Classification of optimization problems

Classification based on:

• Permissable values of the design variables

– Integer programming problems
– Real valued programming problems

• Deterministic nature of the variables

– Stochastic programming problem
– Deterministic programming problem

64
 Classification of optimization problems

Classification based on:

• Separability of the functions

– Separable programming problems
– Non-separable programming problems

• Number of the objective functions

– Single objective programming problem
– Multiobjective programming problem

65
 Geometric Programming
• A geometric programming problem (GMP)
is one in which the objective function and
constraints are expressed as posynomials in
X.

66
67
 Quadratic Programming Problem
• A quadratic programming problem is a nonlinear programming problem with a quadratic
objective function and linear constraints. It is usually formulated as follows:

n n n
F (X)  c   qi xi   Q ij xi x j
subject to i 1 i 1 j 1

a
i 1
ij xi  b j , j  1,2,, m

xi  0, i  1,2,, n
where c, qi,Qij, aij, and bj are constants.

68
 Optimal Control Problem
• An optimal control (OC) problem is
a mathematical programming
problem involving a number of
stages, where each stage evolves
from the preceding stage in a
prescribed manner.
• It is usually described by two types
of variables: the control (design)
and the state variables. The control
variables define the system and
govern the evolution of the system
from one stage to the next, and the
state variables describe the
behaviour or status of the system in
any stage. 69
 Optimal Control Problem
• The problem is to find a set of control or design
variables such that the total objective function
(also known as the performance index) over all
stages is minimized subject to a set of constraints
on the control and state variables.
• An OC problem can be stated as follows:
l
Find X which minimizes f ( X)   f i ( xi , yi )
subject to the constraints i 1

qi ( xi , yi )  y i  y i 1 , i  1,2, , l
g j ( x j )  0, j  1,2, , l
h k ( y k )  0, k  1,2,, l
where xi is the ith control variable, yi is the ith
control variable, and fi is the contribution of the
70
ith stage to the total objective function; gj, hk and
qi are functions of xj, yk and xi and yi,
respectively, and l is the total number of stages.
 Integer Programming Problem
• If some or all of the design variables x1,x2,..,xn of
an optimization problem are restricted to take
on only integer (or discrete) values, the problem
is called an integer programming problem.

• If all the design variables are permitted to take

any real value, the optimization problem is
called a real-valued programming problem.
71
 Stochastic Programming Problem
• A stochastic programming problem is an
optimization problem in which some or all of the
parameters (design variables and/or preassigned
parameters) are probabilistic (nondeterministic
or stochastic).
• In other words, stochastic programming deals
with the solution of the optimization problems in
which some of the variables are described by
probability distributions. 72
 Separable Programming Problem
• A function f (x) is said to be separable if it can be expressed as the
sum of n single variable functions, f1(x1), f2(x2),….,fn(xn), that is,
n
f (X)   f i xi
i 1

• A separable programming problem is one in which the objective function

and the constraints are separable and can be expressed in standard form as:
Find X which minimizes n

subject to f ( X)   f i ( xi )
i 1

n
g j ( X)   g ij ( xi )  b j , j  1,2,, m
i 1
where bj is constant.
73
 Multiobjective Programming Problem
• A multiobjective programming problem can be stated as follows:

Find X which minimizes f1 (X), f2 (X),…., fk (X)

subject to
g j ( X)  0, j  1,2,..., m

where f1 , f2,…., fk denote the objective functions to be minimized

simultaneously.

74
 Review of mathematics
Concepts from linear algebra:
Positive definiteness
• Test 1: A matrix A will be positive definite if all its
eigenvalues are positive; that is, all the values of  that satisfy
the determinental equation
A  I  0
should be positive. Similarly, the matrix A will be negative
definite if its eigenvalues are negative.

75
 Review of mathematics
Positive definiteness
• Test 2: Another test that can be used to find the positive definiteness
of a matrix A of order n involves evaluation of the determinants
A  a11
a11 a12 a13  a1n
a11 a12 a21 a22 a23  a2 n
A2 
a21 a22 A3  a31 a32 a33  a3n
a11 a12 a13 
A3  a21 a22 a23 an1 an 2 an 3  ann
a31 a32 a33

• The matrix A will be positive definite if and only if all the values A1,
A2, A3,An are positive
• The matrix A will be negative definite if and only if the sign of Aj is
(-1)j for j=1,2,,n
• If some of the Aj are positive and the remaining Aj are zero, the matrix
A will be positive semidefinite
76
 Review of mathematics

Negative definiteness

• Equivalently, a matrix is negative-definite if all its

eigenvalues are negative

• It is positive-semidefinite if all its eigenvalues are all

greater than or equal to zero

• It is negative-semidefinite if all its eigenvalues are all

less than or equal to zero

77
 Review of mathematics

Concepts from linear algebra:

Nonsingular matrix: The determinant of the matrix is not

zero.

Rank: The rank of a matrix A is the order of the largest

nonsingular square submatrix of A, that is, the largest
submatrix with a determinant other than zero.

78
 Review of mathematics

Solutions of a linear problem

Minimize f(x)=cTx
Subject to g(x): Ax=b
Side constraints: x ≥0
• The existence of a solution to this problem depends on the rows of
A.

• If the rows of A are linearly independent, then there is a unique

solution to the system of equations.

• If det(A) is zero, that is, matrix A is singular, there are either no

solutions or infinite solutions.

79
 Review of mathematics

Suppose

1 1 1 3  1 1 1 3 
A  b  A*   
 1 1 0.5 1.5 - 1 1 0.5 1.5
The new matrix A* is called the augmented matrix- the columns of b are added to A.
According to the theorems of linear algebra:

• If the augmented matrix A* and the matrix of coefficients A have the same rank r
which is less than the number of design variables n: (r < n), then there are many
solutions.

• If the augmented matrix A* and the matrix of coefficients A do not have the same
rank, a solution does not exist.

• If the augmented matrix A* and the matrix of coefficients A have the same rank 80 r=n,
where the number of constraints is equal to the number of design variables, then there
is a unique solution.
 Review of mathematics

In the example

1 1 1 3  1 1 1 3 
A  b  A*   
  1 1 0.5 1.5  - 1 1 0.5 1.5 
The largest square submatrix is a 2 x 2 matrix (since m = 2 and
m < n). Taking the submatrix which includes the first two
columns of A, the determinant has a value of 2 and therefore is
nonsingular. Thus the rank of A is 2 (r = 2). The same columns
appear in A* making its rank also 2. Since
r < n, infinitely many solutions exist. 81
 Review of mathematics

In the example

1 1 1 3  1 1 1 3 
A  b  A*   
  1 1 0. 5 1.5  - 1 1 0.5 1.5 
One way to determine the solutions is to assign ( n-r) variables arbitrary
values and use them to determine values for the remaining r variables. The
value n-r is often identified as the degree of freedom for the system of
equations.

In this example, the degree of freedom is 1 (i.e., 3-2). For instance x3 can be
assigned a value of 1 in which case x1=0.5 and x2=1.5 82