Lecture 2

Design Patterns for Optimization Problems

Greedy Algorithms
 Algorithmic Paradigm Context

Solve subproblem(s), Make choice, then solve

Subproblem solution order then make choice subproblem(s)
Greedy Algorithms
 What is a Greedy Algorithm?
• Solves an optimization problem
• Optimal Substructure:
• optimal solution contains in it optimal solutions to
subproblems
• Greedy Strategy:
• At each decision point, do what looks best “locally”
• Choice does not depend on evaluating potential future choices
or presolving overlapping subproblems
• Top-down algorithmic structure
• With each step, reduce problem to a smaller problem
• Greedy Choice Property:
• “locally best” = globally best
 A Greedy Strategy Approach
source: 91.503 textbook Cormen, et al.

1. Determine the optimal substructure of the

problem.
2. Develop a recursive solution.
3. Prove that, at any stage of the recursion, one of
the optimal choices is the greedy choice.
4. Show that all but one of the subproblems caused
by making the greedy choice are empty.
5. Develop a recursive greedy algorithm.
6. Convert it to an iterative algorithm.
With experience, it is also possible to directly devise a
(possibly iterative) greedy strategy.
 Examples of Greedy Algorithms

• Activity Selection
• Minimum Spanning Tree
• Dijkstra Shortest Path
• Huffman Codes
• Fractional Knapsack
Activity Selection
 Activity Selection
Optimization Problem
• Problem Instance:
• Set S = {a1, a2,..., an} of n activities
• Each activity ai has:
• start time: si
• finish time: fi
• Activities i, j are compatible iff non-overlapping:
and and
• Objective:
• select a maximum-sized set of mutually compatible activities

source: 91.503 textbook Cormen, et al.

 Activity Selection

Activity Time Duration

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Activity 1
Number
2

8
 Algorithmic Progression

• “Brute-Force”
• Check all possible solutions
• Exponential number of subproblems!
• Dynamic Programming
• Quadratic number of subproblems
• Greedy Algorithm
• Linear number of subproblems
 Activity Selection: Dynamic
Programming Formulation

Solution to Sij including ak produces 2 subproblems:

1) Sik (start after ai finishes; finish before ak starts)
2) Skj (start after ak finishes; finish before aj starts)

c[i,j]=size of maximum-size subset of

mutually compatible activities in Sij.

source: 91.404 textbook Cormen, et al.

 Activity Selection:
Recursive Greedy Algorithm

Only 1 subproblem to solve at each “level”.

High-level call: REC-ACTIVITY-SELECTOR(s,f,0,n)

Returns an optimal solution for

source: 91.503 textbook Cormen, et al. (3rd edition)

source: web site accompanying 91.503 textbook Cormen, et al.
 Activity Selection: Iterative Greedy
source: 91.503 textbook Cormen, et al.

• Algorithm:
• Presort activities in S by non-decreasing finish time and renumber.

Running time?
source: 91.503 textbook Cormen, et al. (3rd edition)
 Streamlined Greedy Strategy
Approach

1. View optimization problem as one in which

making choice leaves one subproblem to
solve.
2. Prove there always exists an optimal solution
that makes the greedy choice.
3. Show that greedy choice + optimal solution
to subproblem optimal solution to
problem.
Greedy Choice Property: “locally best” = globally best
source: 91.503 textbook Cormen, et al.
Minimum Spanning Tree
 Minimum Spanning Tree
Time: Invariant: Minimum weight
O(|E|lg|E|) spanning forest
Produces minimum weight tree of
given fast edges that includes every vertex.
FIND-SET,
UNION 2
A B
4
Becomes single 3
tree at end
G
6
Where are the greedy choices made? 5
Time: 1 E 1
O(|E|lg|V|) Invariant: Minimum 7
= weight tree 6 F
8
O(|E|lg|E|) 4
slightly D C
faster with
Spans all 2
vertices at end for Undirected, Connected,
fast priority
queue Weighted Graph
G=(V,E)

source: 91.503 textbook Cormen et al. (Ch 23)

Dijkstra Shortest Path
 Single Source Shortest Paths:
Dijkstra’s Algorithm
for (nonnegative) weighted, directed graph G=(V,E)

A 2
B
4
3 G
6
5
1 E 1
7
6 F
8
D 4
C
2

We will revisit this algorithm when we study shortest paths.

source: 91.503 textbook Cormen et al. (Ch 24)

Huffman Codes
 Huffman Code Motivation

Prefix Code: No code is a prefix of any other.

source: web site accompanying 91.503 textbook Cormen, et al.

 Huffman Code Tree Example

Goal: Minimize number of bits

required to encode a file.

length of codeword for character c

source: 91.503 textbook Cormen, et al.

 Code Tree Comparison

Fixed-length code: not optimal Huffman prefix code: optimal

source: web site accompanying 91.503 textbook Cormen, et al.

Example of Huffman Code Steps

source: web site accompanying 91.503 textbook Cormen, et al.

Huffman Code Greedy Algorithm

Running Time?

source: web site accompanying 91.503 textbook Cormen, et al.

 Huffman Correctness: A Key Idea
Lemma 16.2: Let C be an alphabet in which each character c in C has
frequency c.freq. Let x and y be 2 characters in C with lowest frequencies.
Then there exists an optimal prefix code for C in which codewords x and y
have the same length and differ only in the last bit.

(T represents arbitrary optimal prefix code.) (Claim: T’’ also has optimal cost.)

source: web site accompanying 91.503 textbook Cormen, et al.

Fractional Knapsack
 Knapsack
Each item has value and weight.
Goal: maximize total value of items chosen,
subject to weight limit.
fractional: can take part of an item
0-1: take all or none of an item 50

30
20
10

item1 item2 item3 “knapsack”

Value: $60 $100 $120

source: web site accompanying 91.503 textbook Cormen, et al.
 Greedy Heuristic

• If optimization problem does not have

“greedy choice property”, greedy approach
may still be useful as a heuristic in
bounding the optimal solution
• Example: minimization problem
Upper Bound (heuristic)
Solution Optimal (unknown value)
Values
Lower Bound