CS161 Lecture 14 Greedy Algorithms

Scribe: Virginia Williams, Sam Kim (2016), Mary Wootters (2017) Date: May 22, 2017

1 Greedy Algorithms
Suppose we want to solve a problem, and we’re able to come up with some recursive formulation of the
problem that would give us a nice dynamic programming algorithm. But then, upon further inspection, we
notice that any optimal solution only depends on looking up the optimal solution to one other subproblem.
A greedy algorithm is an algorithm which exploits such a structure, ignoring other possible choices. Greedy
algorithms can be seen as a refinement of dynamic programming; in order to prove that a greedy algorithm
is correct, we must prove that to compute an entry in our table, it is sufficient to consider at most one
other table entry; that is, at each point in the algorithm, we can make a “greedy”, locally-optimal choice,
and guarantee that a globally-optimal solution still exists. Instead of considering multiple choices to solve a
subproblem, greedy algorithms only consider a single subproblem, so they run extremely quickly – generally,
linear or close-to-linear in the problem size.
Unfortunately, greedy algorithms do not always give the optimal solution, but they frequently give good
(approximate) solutions. To give a correct greedy algorithm one must first identify optimal substructure (as
in dynamic programming), and then argue that at each step, you only need to consider one subproblem. That
is, even though there may be many possible subproblems to recurse on, given our selection of subproblem,
there is always an optimal solution that contains the optimal solution to the selected subproblem.

1.1 Activity Selection Problem

One problem, which has a very nice (correct) greedy algorithm, is the Activity Selection Problem. In this
problem, we have a number of activities. Your goal is to choose a subset of the activies to participate in.
Each activity has a start time and end time, and you can’t participate in multiple activities at once. Thus,
given n activities a1 , a2 , ..., an where ai has start time si and finish time fi , we want to find a maximum set
of non-conflicting activities.
The activity selection problem has many applications, most notably in scheduling jobs to run on a single
machine.

1.1.1 Optimal Substructure

Let’s start by considering a subset of the activities. In particular, we’ll be interested in considering the
set of activities Si,j that start after activity ai finishes and end before activity aj starts. That is, Si,j =
{ak |fi ≤ sk , fk ≤ sj }. We can participate in these activities between ai and aj . Let Ai,j be a maximum
subset of non-conflicting activities from the subset Si,j . Our first intuition would be to approach this by
using dynamic programming. Suppose some ak ∈ Ai,j , then we can break down the optimal subsolution Ai,j
as follows
|Ai,j | = 1 + |Ai,k | + |Ak,j |
where Ai,k is the best set for Si,k (before ak ), and Ak,j is the best set for after ak . Another way of interpreting
this expression is to say “once we place ak in our optimal set, we can only consider optimal solutions to
subproblems that do not conflict with ak .”
Thus, we can immediately come up with a recurrence that allows us to come up with a dynamic pro-
gramming algorithm to solve the problem.

|Ai,j | = max 1 + |Ai,k | + |Ak,j |

ak ∈Si,j

1
This problem requires us to fill in a table of size n2 , so the dynamic programming algorithm will run in
Ω(n2 ) time. The actual runtime is O(n3 ) since filling in a single entry might take O(n) time.
But we can do better! We will show that we only need to consider the ak with the smallest finishing
time, which immediately allows us to search for the optimal activity selection in linear time.
Claim 1. For each Si,j , there is an optimal solution Ai,j containing ak ∈ Si,j of minimum finishing time
fk .
Note that if the claim is true, when fk is minimum, then Ai,k is empty, as no activities can finish before
ak ; thus, our optimal solution only depends on one other subproblem Ak,j (giving us a linear time algorithm).
Here, we prove the claim.
Proof. Let ak be the activity of minimum finishing time in Si,j . Let Ai,j be some maximum set of non-
conflicting activities. Consider A0i,j = Ai,j \ {al } ∪ {ak } where al is the activity of minimum finishing time
in Ai,j . It’s clear that |A0i,j | = |Ai,j |. We need to show that A0i,j does not have conflicting activities. We
know al ∈ Ai,j ⊂ Si,j . This implies fl ≥ fk , since ak has the minimum finishing time in Si,j .
All at ∈ Ai,j \ {al } don’t conflict with al , which means that st ≥ fl , which means that st ≥ fk , so this
means that no activity in Ai,j \ {al } can conflict with ak . Thus, A0i,j is an optimal solution. 

Due to the above claim, the expression for Ai,j from before simplifies to the following expression in terms
of ak ⊆ Si,j , the activity with minimum finishing time fk .

|Ai,j | = 1 + |Ak,j |
Ai,j = Ak,j ∪ {ak }

Algorithm Greedy-AS assumes that the activities are presorted in nondecreasing order of their finishing
time, so that if i < j, fi ≤ fj .

Algorithm 1: Greedy-AS(a)
A ← {a1 } // activity of min fi
k←1
for m = 2 → n do
if sm ≥ fk then
//am starts after last acitivity in A
A ← A ∪ {am }
k←m
return A

By the above claim, this algorithm will produce a legal, optimal solution via a greedy selection of activities.
There may be multiple optimal solutions, but there always exists a solution that includes ak with the
minimum finishing time. The algorithm does a single pass over the activities, and thus only requires O(n)
time – a dramatic improvement from the trivial dynamic programming solution. If the algorithm also needed
to sort the activities by fi , then its runtime would be O(n log n) which is still better than the original dynamic
programming solution.

1.2 Scheduling
Consider another problem that can be solved greedily. We are given n jobs which all need a common resource.
Let wj be the weight (or importance) and lj be the length (time required) of job j. Our output is an ordering
of jobs. We define the completion time cj of job j to be the sum of the lengths of jobs in the ordering up to
and including
P lj . Our goal is to output an ordering of jobs that minimizes the weighted sum of completion
times j wj cj .

2
1.2.1 Intuition
Our intuition tells us that if all jobs have the same length, then we prefer larger weighted jobs to appear
earlier in the order. If jobs all have equal weights, then we prefer shorter length jobs in the order.

1 2 3

vs
3 2 1

P3
In the first case, assuming they all have equal weights of 1, i=1 wi ci = 1 + 3 + 6 = 10. In the second case,
P3
i=1 wi ci = 3 + 5 + 6 = 14.

1.2.2 Optimal Substructure

What do we do in the cases where li < lj and wi < wj ? Consider the optimal ordering of jobs. Suppose we
have a job i that is followed by job j in the optimal order. Consider swapping jobs i and j. The example
below swaps jobs 1 and 2.

l1 l2 l2 l1

1 2 → 2 1

Note that swapping jobs i and j does not alter the completion times for every other job and only changes
the completion
P times for i and j. ci increases by lj and cj decreases by li . This means that our objective
function i wi ci changes by wi lj − wj li . Since we assumed our order was optimal originally, our objective
function cannot decrease after swapping the jobs. This means,

wi lj − wj li ≥ 0

which implies,
lj li
≥
wj wi
Therefore, we want to process jobs in increasing order of wlii , the ratio of the length to the weight of
each job. The algorithm also does a single pass over jobs, and thus only requires O(n) time, assuming the
jobs were ordered by wlii . Like previously, if the algorithm also needed to sort the jobs based on the ratio of
length to weight, then its runtime would be O(n log n).

1.3 Optimal Codes

Our third example comes from the field of information theory. In ASCII, there is a fixed 8 bit code for
each character. Suppose we want to incorporate information about frequencies of characters to obtain
shorter encodings. What if we want to represent characters by codes of different lengths depending on each
character’s frequencies? We explore a greedy solution to find the optimal encoding of characters.
To create optimal codes, we want a way to encode and decode our sequence. To encode the sequence, we
would just have to concatenate the code of each character together. How about for decoding? Consider the
following codes of characters: a → 0, b → 1, c → 01. However, when decoding, when we encounter 01, this
could be decoded as ”ab” or ”c”. Therefore, our codes need to be prefix free: no codeword is a prefix of
another.

3
1.3.1 Tree Representation
We may think of representing our codes in a tree structure, where the codewords represent the leaves of our
tree. An example is shown below:

1
1
0
a: .45 .55
1
0
.25 .30
1 1
0 0
c: .12 b: .13 .14 d:.16
1
0
f:.05 e:.09

Above, in addition to the characters {a, b, c, d, e, f }, we’ve included frequency information. That is, f (a) =
0.45 means that the probability of a random character in this language being equal to a is .45.
The code for each character can be found by concatenating the bits of the path from the root to the
leaves. By convention, every left branch is given the bit 0 and every right branch is given the bit 1.
As long as the characters are on the leaves of this tree, the corresponding code will be prefix-free. This
is because one string is a prefix of another if and only if the node corresponding to the first is an ancestor
of the node corresponding to the second. No leaf is an ancestor of any other leaf, so the code is prefix-free.

1.3.2 How good is a code?

P
Suppose we have a set of characters C with frequencies f (c) so that c∈C f (c) = 1. That is, f (c) can be
thought of as the probability of using a letter c in this language. The cost, in terms of bits, of a character
c ∈ C when using the coding scheme represented by a tree T is just the depth in the tree T : cost(c) = dT (c).
For example, in the tree above, e has depth 4 in the tree, and requires 4 bits to represent. The average cost
of the tree is X
B(T ) = Ec∈C dT (c) = f (c)dT (c).
c∈C

We say that a tree T is optimal if this expected cost B(T ) is as small as possible.

1.3.3 Huffman Codes

In 1951, David A. Huffman, in his MIT information theory class, was given the choice of a term paper or
final exam. Huffman chose to do the term paper rather than take the final exam. He found greedy algorithm
to find the most efficient binary code, which we know today as Huffman codes.
The basic idea is this: build subtrees for subsets of characters and merge them from the bottom up,
combining the two trees with the characters of minimum total frequency.

4
Input: Set of characters C = {c1 , c2 , . . . , cn } of size n, and F = {f (c1 ), f (c2 ), . . . , f (cn )}, a set of frequencies.
• Create nodes Nk for each character ck , with key f (ck ).

• Let current denote the set {N1 , . . . , Nn } of nodes.

• while current has length more than one:
– Find the two nodes Ni and Nj in current with the minimum frequencies and create a new
intermediate node I with Ni and Nj as its children, so that I.key = Ni .key + Nj .key.
– Add I to current and remove Ni , Nj .
• Return the only entry of current, which is the root of the tree.

Figure 1: A high-level version of theHuffman Coding algorithm.

The tree shown above results from running this algorithm on the letters with those frequencies; see the
slides for an illustration of this process.

1.3.4 Proof of Correctness

This algorithm works, but at first it’s not at all obvious why. For a rigorous proof, refer to Lemmas 16.2
and 16.3 in CLRS. However, we’ll sketch the idea below. Formally, the proof goes by induction. Recall that
after iteration t in Algorithm 1, we have a list current, which contains the roots of subtrees that we still
need to merge up. We will maintain the following inductive hypothesis:
• Inductive Hypothesis: Suppose we have completed t iterations of the loop in Algorithm 1. Then
there exists a way to merge the subtrees in current that is optimal.
• For the base case, we observe that when t = 0, current is just the set of all characters, and defini-
tionally there exists an optimal tree made out of these nodes.
• For the inductive step, we need to show that if the inductive hypothesis holds at step t − 1, then it
holds at step t. We’ll sketch this later.
• Finally, to conclude the argument, we see that at the end of the algorithm, there is only one element
in ucrrent, and in this case the inductive hypothesis reads that there is a way to merge this single
subtree to obtain an optimal subtree. That’s just a convoluted way of saying that the single tree we
return is optimal, and so we are done.
All that remains to show is the inductive step. We first observe the following claim:
Claim 2. We are given a set of characters C and a set of its associated frequencies F where f (c) is the
frequency of character c. Let x and y be the characters with the two smallest frequencies. There exists an
optimal coding tree for C such that x, y are sibling leaves.
Proof. Let T be the optimal coding tree for C. The optimal coding tree must be a full binary tree, that
is, every non-leaf node must have two children. Let a, b be characters that are sibling leaves of maximum
depth. We P define the number of bits to encode c as dT (c) and the number of bits needed for the coding tree
as B(T ) = c f (c)dT (c).
We can replace a, b by x, y without increasing the total number of bits needed for the coding tree.1 If we
swap x and a, the change in cost becomes

f (x)dT (a) + f (a)dT (x) − f (x)dT (x) − f (a)dT (a) = (f (x) − f (a))(dT (a) − dT (x)) ≤ 0
1 For simplicity, we ignore the case where a, b, x, y are not distinct. For more details, see Lemma 16.2 in CLRS.

5
 Therefore, there swapping a, b with x, y will not increase our objective function B(T ). Hence, there exists
an optimal coding tree where x, y are siblings in the tree. 
Claim 2 shows that there exists an optimal coding tree where x and y are sibling leaves, that is, there
is an optimal code that makes the same greedy choice as the algorithm. However, this is only immediately
helpful for the first iteration of the inductive step, when all of the elements of current are indeed leaves. In
order to make this idea work for all t, we need one more claim.
Claim 3. Let C be a set of characters, and let T be a coding tree for C. Imagine creating C 0 from C by
collapsing all the characters in a subtree rooted at a node N with key k = N.key into a single character c0
with frequency k. Then the corresponding tree T 0 is optimal for C 0 .
Conversely, suppose that a tree T 0 that is an optimal coding tree for an alphabet CP 0
. Let c0 ∈ C 0 be a
r
character with frequency f (c ). Introduce new characters c1 , . . . , cr with total frequency i=1 f (c00i ) = f (c0 ).
0 00 00

Let T be an optimal coding tree on c1 , . . . , cr . Then the tree T on the alphabet C = (C \ {c0 }) ∪ {c001 , . . . , c00r }
00 00 00 0

that has the leaf c0 replaced with the subtree T 00 is optimal.

Proof. Let T and T 0 be the two trees described in the lemma, and consider the difference of their costs.
X X
B(T ) − B(T 0 ) = f (c) · dT (c) − f (c) · dT 0 (c)
c∈C c∈C 0
r
!
X
= f (c00i )dT (c00i ) − f (c0 )dT 0 (c0 )
i=1
r
!
X
= f (c00i )(dT 00 (c00i ) 0
+ dT 0 (c ) − f (c0 )dT 0 (c0 )
i=1
r
X r
X
= f (c00i )dT 00 (c00i ) + dT 0 (c0 ) f (c00i ) − f (c0 )dT 0 (c0 )
i=1 i=1
t
X
= f (c00i )dT 00 (c00i )
i=1
Pr
where the last line used the fact that i=1 f (c00i ) = f (c0 ), and so the last two terms cancelled. This means
that the difference in the cost between these two trees only depends on T 00 , it doesn’t depend at all about
the structure of T . Thus, T is optimal if and only if T 0 is optimal. 
The two Claims together prove the inductive step, because the second claim implies that the logic of the
first claim holds, even for newly created intermediate nodes I.
Note: The proof in CLRS has the same basic steps (Lemmas 16.2 and 16.3 instead of the claims above),
although phrased slightly differently. The sketch above is pretty sketchy, so if the above is hard to follow,
please check out CLRS for a more detailed version.