ECEN 629 Shuguang Cui

Lecture 5: Duality and KKT Conditions

• Lagrange dual function

• Lagrange dual problem

• strong duality and Slater’s condition

• KKT optimality conditions

• sensitivity analysis

• generalized inequalities

1
ECEN 629 Shuguang Cui

Lagrangian
standard form problem, (for now) we don’t assume convexity

minimize f0 (x)
subject to fi (x) ≤ 0, i = 1, . . . , m
hi(x) = 0, i = 1, . . . , p

• optimal value p⋆, domain D

• called primal problem (in context of duality)

Lagrangian L : Rn+m → R
m
X p
X
L(x, λ, ν) = f0 (x) + λifi (x) + νihi(x)
i=1 i=1

• λi ≥ 0 and νi called Lagrange multipliers or dual variables

• objective is augmented with weighted sum of constraint functions

2
ECEN 629 Shuguang Cui

Lagrange dual function

(Lagrange) dual function g : Rm → R ∪ {−∞}
m p
!
X X
g(λ, ν) = inf L(x, λ, ν) = inf f0 (x) + λi fi (x) + νihi (x)
x x
i=1 i=1

• minimum of augmented cost as function of weights

• can be −∞ for some λ and ν
• g is concave (even if fi not convex!)

example: LP
minimize cT x
subject to aTi x − bi ≤ 0, i = 1, . . . , m
m
T
λi(aTi x−bi ) = −bT λ+ (AT λ+ c)T x
X
Note that L(x, λ) = c x+
i=1
−bT λ T

if A λ + c = 0
hence g(λ) =
−∞ otherwise

3
 ECEN 629 Shuguang Cui

Lower bound property

if x is primal feasible, then

g(λ, ν) ≤ f0 (x)
proof: if fi (x) ≤ 0 and λi ≥ 0,
!
X X X X
f0 (x) ≥ f0(x)+ λifi (x)+ νihi(x) ≥ inf f0 (z) + λi fi (z) + νi hi(z) = g(λ, ν)
z
i i i i

f0 (x) − g(λ, ν) is called the duality gap

minimize over primal feasible x to get, for any λ  0 and ν ,

⋆
g(λ, ν) ≤ p

λ ∈ Rm and ν ∈ Rp are dual feasible if λ  0 and g(λ, ν) > −∞

dual feasible points yield lower bounds on optimal value!

4
ECEN 629 Shuguang Cui

Lagrange dual problem

let’s find best lower bound on p⋆:

maximize g(λ, ν)
subject to λ0

• called (Lagrange) dual problem

(associated with primal problem)
• always a convex problem, even if primal isn’t!
• optimal value denoted d⋆
• we always have d⋆ ≤ p⋆ (called weak duality )
• p⋆ − d⋆ is optimal duality gap

5
ECEN 629 Shuguang Cui

Strong duality

for convex problems, we (usually) have strong duality:

d ⋆ = p⋆

when strong duality holds, dual optimal λ⋆ serves as certificate of optimality for primal
optimal point x⋆

many conditions or constraint qualifications guarantee strong duality for convex problems

Slater’s condition: if primal problem is strictly feasible (and convex), i.e., there exists
x ∈ relint D with
fi (x) < 0, i = 1, . . . , m
hi(x) = 0, i = 1, . . . , p
then we have p⋆ = d⋆

6
ECEN 629 Shuguang Cui

Dual of linear program

(primal) LP
minimize cT x
subject to Ax  b

• n variables, m inequality constraints

dual of LP is (after making implicit equality constraints explicit)

maximize −bT λ
subject to AT λ + c = 0
λ0

• dual of LP is also an LP (indeed, in std LP format)

• m variables, n equality constraints, m nonnegativity contraints

for LP we have strong duality except in one (pathological) case: primal and dual both
infeasible (p⋆ = +∞, d⋆ = −∞)

7
ECEN 629 Shuguang Cui

Dual of quadratic program

(primal) QP
minimize xT P x
subject to Ax  b
T T
we assume P ≻ 0 for simplicity Lagrangian is L(x, λ) = x P x + λ (Ax − b)
∇xL(x, λ) = 0 yields x = −(1/2)P −1 AT λ, hence dual function is
T −1 T T
g(λ) = −(1/4)λ AP A λ−b λ

• concave quadratic function

• all λ  0 are dual feasible

dual of QP is
maximize −(1/4)λT AP −1AT λ − bT λ
subject to λ0
. . . another QP

8
ECEN 629 Shuguang Cui

Equality constrained least-squares

minimize xT x
subject to Ax = b
A is fat, full rank (solution is x⋆ = AT (AAT )−1 b)

dual function is
“ ” 1
g(ν) = inf x x + ν (Ax − b) = − ν T AAT ν − bT ν
T T
x 4

dual problem is
maximize − 14 ν T AAT ν − bT ν
solution: ν ⋆ = −2(AAT )−1b

can check d⋆ = p⋆

9
ECEN 629 Shuguang Cui

Introducing equality constraints

idea: simple transformation of primal problem can lead to very different dual

example: unconstrained geometric programming

primal problem:
m
T
X
minimize log exp(ai x − bi )
i=1
⋆
dual function is constant g = p (we have strong duality, but it’s useless)

now rewrite primal problem as

m
X
minimize log exp yi
i=1
subject to y = Ax − b

10
ECEN 629 Shuguang Cui

let us introduce
• m new variables y1, . . . , ym
• m new equality constraints y = Ax − b

dual function
m
!
exp yi + ν T (Ax − b − y)
X
g(ν) = inf log
x,y
i=1

• infimum is −∞ if AT ν 6= 0
• assuming AT ν = 0, let’s minimize over y :
eyi
Pm yj = νi
j=1 e

solvable iff νi > 0, 1T ν = 1

νi log νi − bT ν
X
g(ν) = −
i

11
ECEN 629 Shuguang Cui

• same expression if ν  0, 1T ν = 1 (0 log 0 = 0)

dual problem
T
X
maximize −b ν − νi log νi
i
T
subject to 1 ν = 1, (ν  0)
AT ν = 0
moral: trivial reformulation can yield different dual

12
ECEN 629 Shuguang Cui

Duality in algorithms

many algorithms produce at iteration k

• a primal feasible x(k)

• a dual feasible λ(k) and ν (k)

with f0 (x(k) ) − g(λ(k) , ν (k)) → 0 as k → ∞

h i
⋆ (k) (k) (k)
hence at iteration k we know p ∈ g(λ , ν ), f0 (x )

• useful for stopping criteria

• algorithms that use dual solution are often more efficient (e.g., LP)

13
 ECEN 629 Shuguang Cui

Nonheuristic stopping criteria

absolute error = f0(x(k) ) − p⋆ ≤ ǫ
“ ”
(k) (k) (k)
stopping criterion: until f0 (x ) − g(λ , ν ) ≤ ǫ

f0 (x(k) ) − p⋆
relative error = ≤ǫ
|p⋆|
stopping criterion:
„ « „ «
f0 (x(k) )−g(λ(k) ,ν (k) ) f0 (x(k) )−g(λ(k) ,ν (k) )
until g(λ(k) , ν (k)) > 0 & ≤ǫ or f0 (x(k) ) < 0 & ≤ǫ
g(λ(k) ,ν (k) ) −f0 (x(k) )

achieve target value ℓ or, prove ℓ is unachievable

(i.e., determine either p⋆ ≤ ℓ or p⋆ > ℓ)
“ ”
(k) (k) (k)
stopping criterion: until f0 (x ) ≤ ℓ or g(λ , ν ) > ℓ

14
ECEN 629 Shuguang Cui

Complementary slackness

suppose x⋆, λ⋆, and ν ⋆ are primal, dual feasible with zero duality gap (hence, they are
primal, dual optimal)

m p m
!
⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆ ⋆
X X X
f0 (x ) = g(λ , ν ) = inf f0(x) + λi fi (x) + νi hi (x) ≤ f0(x )+ λi fi (x )
x
i=1 i=1 i=1

Pm
hence we have i=1 λ⋆ifi (x⋆ ) = 0, and so

⋆ ⋆
λi fi (x ) = 0, i = 1, . . . , m

• called complementary slackness condition

• ith constraint inactive at optimum =⇒ λi = 0
• λ⋆i > 0 at optimum =⇒ ith constraint active at optimum

15
ECEN 629 Shuguang Cui

KKT optimality conditions

suppose

• fi are differentiable
• x⋆, λ⋆ are (primal, dual) optimal, with zero duality gap

by complementary slackness we have

!
f0 (x⋆ ) + λ⋆ifi (x⋆ ) = inf λ⋆ifi (x) + νi⋆fi (x)
X X X
f0(x) +
x
i i i

i.e., x⋆ minimizes L(x, λ⋆, ν ⋆)

therefore
⋆ ⋆ ⋆ ⋆ ⋆
X X
∇f0 (x ) + λi ∇fi (x ) + νi ∇hi(x ) = 0
i i

16
ECEN 629 Shuguang Cui

so if x⋆ , λ⋆, and ν ⋆ are (primal, dual) optimal, with zero duality gap, they satisfy

fi (x⋆ ) ≤ 0
hi (x⋆ ) = 0
λ⋆i ≥ 0
λ⋆ifi (x⋆ ) = 0
∇f0(x⋆ ) + ⋆ ⋆
νi⋆∇hi(x⋆ ) = 0
P P
i λi ∇fi (x ) + i

the Karush-Kuhn-Tucker (KKT) optimality conditions

conversely, if the problem is convex and x⋆, λ⋆ satisfy KKT, then they are (primal, dual)
optimal

17
ECEN 629 Shuguang Cui

Geometric interpretation of duality

consider set
m+1
A = { (u, t) ∈ R | ∃x fi (x) ≤ ui , f0 (x) ≤ t }

• A is convex if fi are
• for λ  0, ( » –T » – ˛˛ » – )
λ u u
g(λ) = inf ∈A
˛
1 t t
˛
˛
t
A

T
t + λ u = g(λ)
g(λ)
» –
λ
1
u

18
ECEN 629 Shuguang Cui

(Idea of) proof of Slater’s theorem

problem convex, strictly feasible =⇒ strong duality
t

p⋆ A

λ⋆
» –

1 u

• (0, p⋆) ∈ ∂A ⇒ ∃ supporting hyperplane at (0, p⋆):

⋆ T
(u, t) ∈ A =⇒ µ0(t − p ) + µ u ≥ 0
• µ0 ≥ 0, µ  0, (µ, µ0 ) 6= 0
• strong duality ⇔ ∃ supporting hyperplane with µ0 > 0: for λ⋆ = µ/µ0 , we have
T
p⋆ ≤ t + λ⋆ u ∀(t, u) ∈ A, p⋆ ≤ g(λ⋆ )
• Slater’s condition: there exists (u, t) ∈ A with u ≺ 0; implies that all supporting
hyperplanes at (0, p⋆) are non-vertical (µ0 > 0)

19
ECEN 629 Shuguang Cui

Sensitivity analysis via duality

define p⋆(u) as the optimal value of
minimize f0 (x), subject to fi (x) ≤ ui , i = 1, . . . , m

epi p⋆

p⋆ (u)
p⋆ (0) − λ⋆T u
0

u
Pm
λ⋆ gives lower bound on p⋆(u): p⋆(u) ≥ p⋆ − i=1 λ⋆iui

• if λ⋆i large: ui < 0 greatly increases p⋆

• if λ⋆i small: ui > 0 does not decrease p⋆ too much

⋆ ⋆ ∂p⋆(0) ⋆
if p (u) is differentiable, λi =− , λi is sensitivity of p⋆ w.r.t. ith constraint
∂ui

20
ECEN 629 Shuguang Cui

Generalized inequalities

minimize f0 (x)
subject to fi (x) Ki 0, i = 1, . . . , L

• Ki are generalized inequalities on Rmi

• fi : Rn → Rmi are Ki-convex

Lagrangian L : Rn × Rm1 × · · · × RmL → R,

T T
L(x, λ1, . . . , λL) = f0 (x) + λ1 f1 (x) + · · · + λL fL(x)

dual function
“ ”
g(λ1 , . . . , λL) = inf f0 (x) + λT1 f1 (x) + ··· + λTL fL(x)
x

λi dual feasible if λi K ⋆ 0, g(λ1 , . . . , λL) > −∞

i

21
ECEN 629 Shuguang Cui

lower bound property: if x primal feasible and

(λ1 , . . . , λL) is dual feasible, then

g(λ1 , . . . , λL) ≤ f0 (x)

(hence, g(λ1 , . . . , λL) ≤ p⋆)

dual problem
maximize g(λ1 , . . . , λL)
subject to λi K ⋆ 0, i = 1, . . . , L
i

weak duality: d⋆ ≤ p⋆ always

strong duality: d⋆ = p⋆ usually

Slater condition: if primal is strictly feasible, i.e.,

∃x ∈ relint D : fi (x) ≺Ki 0, i = 1, . . . , L

then d⋆ = p⋆

22
ECEN 629 Shuguang Cui

Example: semidefinite programming

minimize cT x
subject to F0 + x1 F1 + · · · + xnFn  0
Lagrangian (multiplier Z  0)
T
L(x, Z) = c x + Tr Z(F0 + x1F1 + · · · + xnFn)
dual function
“ ”
T
g(Z) = inf c x + Tr Z(F0 + x1F1 + · · · + xnFn)
x

Tr F0Z if Tr FiZ + ci = 0, i = 1, . . . , n
=
−∞ otherwise

dual problem
maximize Tr F0Z
subject to Tr FiZ + ci = 0, i = 1, . . . , n
Z = ZT  0
strong duality holds if there exists x with F0 + x1 F1 + · · · + xnFn ≺ 0

23