C&O 355

Lecture 2
N. Harvey
http://www.math.uwaterloo.ca/~harvey/
 Outline
• LP definition & some equivalent forms
• Example in 2D
– Possible outcomes
• Examples
– Linear regression, bipartite matching, indep set
• Feasible Region, Convex Sets
• Corner solutions & certificates
• Local-Search Algorithm
 Linear Program
• General definition
n
– Parameters: c, a1,…,am2R , b1,…,bm2R
– Variables: x2Rn
Objective function
Constraints

• Terminology
– Feasible point: any x satisfying constraints
– Optimal point: any feasible x that minimizes obj. func
– Optimal value: value of obj. func for any optimal point
 Linear Program
• General definition
n
– Parameters: c, a1,…,am2R , b1,…,bm2R
– Variables: x2Rn

• Matrix form

n
• Parameters: c2R , A2Rm£n, b2Rm
• Variables: x2Rn
 Simple LP Manipulations
• “max” instead of “min”
max cT x ´ min –cT x

• “¸” instead of “·”

aTx ¸ b , -aTx · -b

• “=” instead of “·”

aTx=b , aTx · b and aTx ¸ b
 2D Example
(Textbook, Ch 1)
Constraint

x2 x2 - x1 · 1 Optimal
point
(3,2)

x1 + 6x2 · 15

Feasible region
(1,1)
x2¸0
x1
(0,0)
x1¸0
4x1 - x2 · 10

Unique optimal solution exists

 2D Example
Constraint

x2 x2 - x1 · 1 Optimal
points

x1 + 6x2 · 15

(1/6,1) Feasible region

x2¸0
x1
(0,0)
x1¸0
4x1 - x2 · 10

Optimal solutions exist:

Infinitely many!
 2D Example
(Textbook, Ch 1)
Constraint

x2 x2 - x1 ¸ 1

x1 + 6x2 · 15

No feasible region
(1,1)
x2¸0
x1
(0,0)
x1¸0
4x1 - x2 ¸ 10

Infeasible
No feasible solutions
 2D Example
(Textbook, Ch 1)
Constraint
x2 x2 - x1 · 1

(1,1)
x2¸0
x1
(0,0)
x1¸0 Feasible region
Unbounded
Feasible solutions, but no optimal solution
(Optimal value = 1)
 “Fundamental Theorem” of LP
• Theorem: For any LP, the outcome is either:
– Optimal solution (unique or infinitely many)
– Infeasible
– Unbounded (optimal value is 1 or -1)

• Proof: Later in the course!

 Example: Linear regression
• Given data (x1, y1), …, (xn, yn) in R2
• Find a line y = ax + b that fits the points
Usual setup Our setup
Pn 2
Pn
min i=1 (axi + b ¡ yi ) min i=1 jaxi + b ¡ yi j
Easy: differentiate, set to zero Not differentiable!

• Absolute value trick: jwj ´

• Our setup can be written as an LP

 Example: Bipartite Matching
• Given bipartite graph G=(V, E)
• Find a maximum size matching
– A set M µ E s.t. every vertex has at most one incident edge in M
• Write an integer program

(IP)

• But we don’t know how to solve IPs. Try an LP instead.

(LP)

• Theorem: (IP) and (LP) have the same solution!

• Proof: Later in the course!
• Corollary: Bipartite matching can be solved efficiently (it’s in P).
 Example: Independent Set
• Given graph G=(V, E)
• Find a maximum size independent set
– A set UµV s.t. {u,v}  E for every distinct u,v 2 U
• Write an integer program

(IP)

• But we don’t know how to solve IPs. Try an LP instead.

(LP)

• Unfortunately (IP) and (LP) are extremely different.

• Fact: There are graphs for which OPTLP / OPTIP ¸ |V|/2.
 Feasible Region
• For any a2Rn, b2R, define
Hyperplane

Halfspaces

• So feasible region of
\m
is P = Ha¡ ;b
i i
i=1

• Intersection of finitely many halfspaces is polyhedron

• A bounded polyhedron is a polytope,
i.e., P µ { x : -M · xi · M 8i } for some M
 Convex Sets
• C µ Rn is convex if for every x,y 2 C,
C contains line segment between x and y.
i.e., 8®2[0,1], we have ®x + (1-®)y 2 C.

y y Such a point is called a

convex combination of x and y
x x
Convex Not convex

• Claim 1: Any halfspace is convex.

• Claim 2: The intersection of any number of
convex sets is convex.
• Corollary: Any polyhedron is convex.
Where are optimal solutions?
x2
x2 - x1 · 1

x1 + 6x2 · 15

x1

4x1 - x2 · 10
Where are optimal solutions?
x2
x2 - x1 · 1

x1 + 6x2 · 15

x1

4x1 - x2 · 10
 Where are optimal solutions?
x2
x2 - x1 · 1

x1 + 6x2 · 15

x1
4x1 - x2 · 10

Lemma: For any objective function, a “corner point” is an

optimal solution.
(Assuming an optimal solution exists and some corner point exists).
Proof: Later in the course!
 Proving optimality
• Question: What is optimal point in direction c = (-7,14)?
• Solution: Optimal point is x=(9/7,16/7), optimal value is 23.
• How can I be sure?
– Every feasible point satisfies x1+6x2 · 15
– Every feasible point satisfies -x1+x2 · 1 ) -8x1+8x2 · 8
– Every feasible point satisfies their sum: -7x1+14x2 · 23
x2 This is the objective function!
(9/7,16/7) -x1+x2 · 1
x1 + 6x2 · 15

x1
4x1 - x2 · 10
 Proving optimality
• Question: What is optimal point in direction c = (-7,14)?
• Solution: Optimal point is x=(9/7,16/7), optimal value is 23.
• How can I be sure?
– Every feasible point satisfies x1+6x2 · 15
– Every feasible point satisfies -x1+x2 · 1 ) -8x1+8x2 · 8
– Every feasible point satisfies their sum: -7x1+14x2 · 23
This is the objective function!
• Certificates
• To convince you that optimal value is ¸ k, I can find x such
that cT x ¸ k.
• To convince you that optimal value is · k, I can find a linear
combination of the constraints which proves that cT x · k.
• Theorem: Such certificates always exists.
• Proof: Later in the course!
 Local-Search Algorithm
(The “Simplex Method”)

• “The obvious idea of moving along edges from one vertex of a

convex polygon to the next” [Dantzig, 1963]
Algorithm
Let x be any corner point
For each neighbor y of x
If cTy>cTx then set x=y
Halt

• In practice, very efficient

• Its analysis proves all the theorems mentioned earlier
• Every known variant of this algorithm takes exponential time
• Open problem: does some variant run in polynomial time?
 Pitfalls and missing details
Algorithm
Let x be any corner point
For each neighbor y of x
If cTy>cTx then set x=y
Halt

1. What is a corner point?

2. What if there are no corner points?
3. What are the “neighboring” corner points?
4. What if there are no neighboring corner points?
5. How can I find a starting corner point?
6. Does the algorithm terminate?
7. Does it produce the right answer?