JohnRasmussen, Insti t ut e of Mec hanical Engineeri ng, Aalborg Univ ersity, 2001

Engineering design
optimization
Lecture 6: Algori t hms for 1-D opti mi zation
PDefi ni ti ons of t er ms
POpti mal it y co ndit i ons
PCon vexit y
P1- D algorit h ms
<Golden section search
<Bisecti on search
<Pol ynomial i nter pol ati on
PThe skel eto n of an alg orit h m
Lectur e 6
1-D algori thms
To read:
PArora: chapter 3 + [5-5.3.4]
Definitions
Global mini mum
x
f (x )
x
f (x )
Gl obal mini mum: f(x*) #
f( x) f or all x.
We can have manyglobal
mini ma, but t hey must all
be equall y good.
Strict global mini mum:
f( x*) < f( x) for all x.
Local mi ni mu m: f( x* )
#f( x+, ) wher e , i s a
s mall nu mber .
Definitions
Local mini mum
x
f(x)
x*
,
If f(x) i s continuou s in a clo sed and limited set, S,
then f ha s a global minimum in S.
Thi s sounds trivial, but it i s important. It give s u s
hope. We know there i s somethi ng to look for.
It i s al so not a s simple a s it sound s. Remember that
the set must be closed. Thi s can lead to many
strange problems in practi cal optimization.
Topology optimization i s a good example.
Existence of the solution
Weierstrass theorem
A cl osed set: L
j
# x
j
# U
j
A n op en s et: L
j
< x
j
< U
j
Definitions
Open and closed set s
f(x) = -1/x.
If 0 < x #2 then the interval i s
open
If 0 #x #2 then the interval i s
closed, but f i s undefined for
x=0.
JohnRasmussen, Insti t ut e of Mec hanical Engineeri ng, Aalborg Univ ersity, 2001
Necessary condition:
for

f
x
j n
j
0 1 , ..
Sufficient condition: The Hessian
must be positive definite.
H
f
x
f
x x
f
x x
f
x
symm
f
x
n
n

1
]
1
1
1
1
1
1
1
1
1
1

2
1
2
2
1 2
2
1
2
2
2
2
2
. .
.
.
.
Necessar y con dit i on:
f( x) = 0
S uffi ci e nt c ondi ti o n:
f( x) > 0
Thi s will i d entif y onl y l oc al opti ma.
For g e ner al f unct i ons, t her e ar e no
con diti o ns t hat wil l ens ur e a gl ob al
opti mu m.
Optimality conditions
1-D problems
x
f(x)
Optimality conditions
Multi-di mensional problemswi thout const raints
x
x
1
2
P Wh en t h e st ati on ar y
p oi nt i s a maxi mu m.
P Wh en t h e st ati on ar y
p oi nt i s a sa ddl e poi nt .
P Wh en t h e st ati on ar y
p oi nt i s a l ocal opti mu m
The necessary condition(s)
- identify stationary point s. When i s a stationary point not
the solution we are looking for?
x
f(x)
A con ve x f u ncti on i s one
t hat has p osi ti ve defi ni t e
Hessi an ever ywh er e.
A con ve x f u ncti on has onl y
on e mi ni mu m - t h e gl obal
on e.
I n 1D a con ve x f uncti on i s
on e t h at e ver ywh er e has
posi ti ve seco nd deri vati ve.
Convexity
When can we be sure to find an opti mum?
x
f(x)
Convex
I n 2D, a con ve x f u ncti on f or ms a s urf ace t hat cur ves upwar ds
ever ywher e.
Convexity (contd)
In two dimensions
Con vex
Non-convex
For an y pair of poi nt s, ( P, Q) bel ongi ng t o S, a str aight li ne
conn ecti ng P and Q wi ll be co mpl etel y cont ai n ed i n S.
Thi s appli es i n an y nu mber of di mensi ons.
Convex sets
For a convex set, S, the following i st rue:
P
Q
Convex
P Q
Non-convex
JohnRasmussen, Insti t ut e of Mec hanical Engineeri ng, Aalborg Univ ersity, 2001
PIf the obj ecti ve function is conve x, and the feasible domain is a convex set,
t hen the opti mization probl em is convex.
PIf all the constr aint f uncti ons are convex, then the feasible domain is
convex.
PConve xopti mization problems have onl y one opti mum - the global one.
This is ver y alg orit hmicall y convenient. I f we have found a stationar y poi nt,
t hen we know that it is the global sol ution. The necessar y conditions ar e
also suf ficient.
PTher e are no g ood alg orit hms for treatment of non-convex problems. Most
alg orit hms assume that the probl em is convex. Many pr oblems are not, so
beware!
PIt is usuall y ver y difficult to check if a function is convex. If the function i s
implicit, then it is impossible. A good underst anding of the physical natur e
of the probl em is usuall y ver y hel pful .
PLinear probl ems are al ways convex.
Convex optimization problems
A f uncti on can c ur ve
up war ds an d still have no
l ocal mi ni ma.
Thi s will not conf use an
opti mi zati on al g orit h m
unl ess i t r eli es on 2nd or d er
inf or mati on.
Convexity
Non-convex functions do not necessarily cause problems
x
f(x)
Optimization algorithms
We shall develop an algorithm for general const rained
optimization in multiple di mensions.
Constrai ned pr oblems i n mul ti ple di mensions
Unconstr ained probl ems in multipl e di mensi ons
Algorithms f or 1-D mini mizati on
Today, we develop and i mpl ement a 1-D algorithm. It is important that you
fi nish t he wor k for ever y l ecture. The algorithm of each new lecture is built on
t op of t he previous.
PGolden section search: 0th order
PBi section: 1 st order
PBrent , pol ynomial i nterp olation: 2nd order
1D algorithms
Categoriztion by order
We assu me t hat a f u ncti o n
f( x) is gi ven, and we wa nt t o
fi nd i t s mi ni mu m i n t h e
i nt er val [ A, B].
We al s o assu me t hat is i s
e xpe nsi ve t o c omput e f( x) .
S o we must fi n d t he
mi ni mu m wi t h t he l east
possi bl e n u mber of f uncti on
eval uati ons.
The f uncti on i s i mpli cit - we
do n t kno w wh at t he gr a ph
l o oks li ke.
1-D minimization problem
- defini tion
x
f(x)
A B
The i d ea b ehi n d g ol den
secti o n is t o successi vel y
pr une t he i nt er val f or p arts
t hat do no c ont ai n t he
mi ni mu m.
Thi s wa y, t he r e mai ni ng
i nt er val shri n ks unti l i t is so
s mall t hat we ha ve
det er mi ned t he l ocati on of
t he mi ni mu m wi t h suf fi ci ent
accur ac y.
Golden Section Search
-a 0th order algorithm
x
f(x)
A B
JohnRasmussen, Insti t ut e of Mec hanical Engineeri ng, Aalborg Univ ersity, 2001

I I
I I
I I I
k k
k k
k k k
( ) ( )
( ) ( )
( ) ( ) ( )
( )
.
.
+
+

'

1
1
2
2
1
1 0
0 618
1
1618
The Golden Section
It t ur ns out t hat , if we don t
ha ve g r adi ent i nf or mati on,
t hen we nee d t wo f uncti on
e val uati ons b ef or e we can
i d entif y an i nt er val t hat d oes
not cont ai n t he mi ni mu m.
Golden Section Search
- computing function values
x
f(x)
A B
" $
We don t kno w t he gr aph of
t he f uncti on, b ut based o n
t he f uncti on val ues i n " and
$, and t he assu mpti on t hat
we ha ve onl y o ne mi ni mu m
i n t he i nt er val , we c an
de duce t h at t he mi ni mu m
cann ot be t o t h e ri g ht of $.
S o we pr u ne t hat par t .
Golden Section Search (contd)
- pruning
x
f(x)
A B " $
We coul d co nti nue li ke t hi s,
pr uni ng t he i nt er val until it
g et s s mall enoug h.
We woul d have t o co mput e
t wo new f uncti on val ues f or
eac h pr uni ng.
Is t her e a wa y t o save so me
of t hes e f u ncti on
eval uati ons?
Golden Section Search (contd)
- development
x
f(x)
A B
" $
If we positi on " and $ car ef ull y,
t hen we ca n make sur e t hat t h e "
of one i t er ati on bec omes t he $ of
t he ne xt a nd vi ce vers a.
Golden Section Search (contd)
- re-using function values
"
(k )
$
(k )
"
(k +1)
$
(k +1)
$
(k +2)
I
(k )
I
(k +1)
= J I
(k )
I
(k +1)
= J
2
I
(k )
PThe i d ea of t he g ol d en secti o n d at es bac k t o P yt hag or as an d
l at er It ali an mat he mati ci an Fi bo nacchi ( 1202).
PI n t h e mi ddl e ag es, t h e g ol de n secti on beca me a measur e of
a est heti cs a dobt ed b y ma ny hu mani st s an d archit ect s. I t
d efi n es a r h yt h m of sh ape t h at pl eas es t he e ye.
PCo mp osi ti o n of pai nti ng s, sc ul pt ur es and buil di ng s was done
accor di ng t o t he g ol d en secti on.
PThe g ol den secti o n pl a ye d an i mportant r ol e also t o t h e
cubi sts i n t he beg i nni ng og t he 20t h cent ur y.
The Golden Section
- the idea
PS et a s ear ch i nt er val, a nd i nit i aliz e " and $. Co mput e f(" )
a nd f($).
PChoose t he rig ht or l eft i nter val t o pr une.
PIf t he rig ht ha nd i nt er val i s r e mo ved, set $:= " and f ( $) := f( " ).
Co mp ute n ew " an d f( " ) .
PIf t he l eft h and i nt er val is r e move d, set " : =$ an d f( " ):=f( $) .
Co mp ute n ew $ and f( $) .
PIf t he i nt er val is s mall en oug h, st op t he alg orit h m a nd r et ur n
t he happ y ne ws.
PRepe at t he i t er ati on.
Golden section (contd)
- the algori thm
JohnRasmussen, Insti t ut e of Mec hanical Engineeri ng, Aalborg Univ ersity, 2001
PEach itera tion remove s 1 - 0.618 = 38% of t he
interval .
PAfter n i t eration s, the i nterval i s reduced to 0.618
n
ti mes i t s origi nal si ze.
PIf n i s 10, less than 1% of the original interval
remains. If n=15, le ss than 1 remain s.
PThe algorithm i s rock- solid stable. I t re moves a
cert ain f raction of the interval each time, and i t
require s only that t he function i s uni modal (ha s one
mini mum) i n the interval .
Golden section (contd)
- properties
If we ha ve gr adi e nt
i nf or mati on, t he n we know t o
whi ch si de of a c o mput ed
f uncti on val ue, t h e f uncti on
decr eas es.
I n t h at case, we ca n cut t he
i nt er val i n hal f each t i me and
obt ai n f ast er c on vergence.
The bisection method
- cutting the interval in hal f
x
f(x)
A B "
PFast er co nvergence. Af t er 1 0 i t er ati ons,
a bout 1 of t he i nt er val i s l eft .
P We n eed gr adi ent i nf or mati on. I t us uall y
r eq uir es mor e co mput ati on, but it can
so meti mes co me ver y chea p. M or e abo ut
t hat i n a l at er lect ur e.
P Wh en we r el y on g r adi e nts, t hen we al so
assu me t hat t he f uncti on i s dif f er enti abl e.
Gol den sect i on does not h ave t hi s
req uir e ment.
PThi s met hod is l ess r ob ust t h an g ol de n
secti on.
The bisection method
- pros and cons
x
f(x)
A B "
We co mpute t he f uncti on
val ues i n t he end poi nt s and
a poi nt i n t he mi ddl e.
We fi t a par abol a t hr oug h t he
t hr ee p oi nts.
We anal yti cal l y det er mi n e
t he mi ni mu m of t he
par abol a.
We l et t he ne w poi nt r epl ace
t he worst of t he pr evi ous
on es an d r epeat until
con verg ence.
Polynomial Interpolation
- fast and deli cate li ke an old sport s car
x
f(x)
A B "
PConvergence i s ver y fast i f the function
behaves nicel y.
PNo gr adients r equired.
POnl yone function evaluati on for each new
iterati on
PThe alg orithm is ver y sensitive to non-convex
functions.
PThe alg orithm r equir es 2nd or der
differenti abilit y.
PThe alg orithm may di verge compl etel y.
Polynomial Interpolation
- pros and cons
x
f(x)
A B "
Not advisable for use in general algorithms, but ver y usef ul f or special
applicati ons.
PChoose a l a ng uag e a nd fi n d a co mpil er
<C (recommended).
Mi cr os oft Visual C++ is f i ne but ex pensive
Bor l and C is av ai l abl e f or fr ee downl oad fr om
htt p:// www. borl and. c om/downl oads/
Gnu C i s fr ee and av ai l abl e many pl aces ont he net
<For tran
Vi s ual Fortr an is good but ex pensive
Other c ompil ers I know of ar e t err ibl e ol d-f ashi oned c ommand l i ne v er si ons.
<Pascal
Any ol d v er si on of T ur bo or Borl and Pasc al wil l do f i ne. Delphi wi ll work t oo.
<Java
Vi s ual Caf or any other c ommer c i al co mpi ler wi ll work f i ne
Sunhas a fr ee co mpil er f or downl oad
<Visual Basic
<Matlab
Wil l work, but I dont k now much about it.
Assignment
Program your own golden section algorithm
JohnRasmussen, Insti t ut e of Mec hanical Engineeri ng, Aalborg Univ ersity, 2001
PDownload the skeleton C code f rom
www.i me.auc.dk/~jr/lecturenotes.htm
PIf you use C, start programming your own golden section
function into the program
PIf you use any other language, translate the code by hand
and start developing the golden section function
PCompi le and test wi th different 1-D functions
Assignment (contd)
Program your own golden section algorithm