CS341: ALGORITHMS (F23)

Lecture 1

Trevor Brown
https://student.cs.uwaterloo.ca/~cs341
trevor.brown@uwaterloo.ca
1
TABLE OF CONTENTS
• Course mechanics
• Models of computation
• Worked example: Bentley’s problem
• Multiple solutions,
demonstrating different algorithm design techniques
• Analyzed in different models of computation

2
COURSE MECHANICS

3
COURSE MECHANICS
• In person
• Lectures
• “Lab” section is for tutorials
• Course website: https://student.cs.uwaterloo.ca/~cs341/
• Syllabus, calendar, policies, slides, assignments…
• Read this and mark important dates.
• Keep up with the lectures: Material builds over time…
• Piazza: For questions and announcements.

4
ASSESSMENTS
• All sections have same assignments, midterm and final
• Sections are roughly synchronized to ensure necessary content is
taught
• Tentative plan is 5 assignments, midterm, final
• See website for grading scheme, etc.

5
TEXTBOOK
• Available for free via library website!
• You are expected to know
• entire textbook sections,
as listed on course website
• all the material presented in lectures
(unless we explicitly say you aren’t
responsible for it)
• Some other textbooks cover some material better… see www

6
ACADEMIC OFFENSES
• Beware plagiarism
• High level discussion
about solutions with individual students
is OK
• Don’t take written notes away from
such discussions
• Class-wide discussion of solutions is
not OK (until deadline+2 days)

7
WHY IS CS341 IMPORTANT FOR YOU?
• Algorithms is the heart of CS
• It appears often in later courses
• It dominates technical interviews
• Master this material…
make your interviews easy!
• Designing algorithms is creative work
• Useful for some of the more interesting
jobs out there
• And, you want to graduate…

8
MODELS OF COMPUTATION

9
WHAT IS A COMPUTATIONAL PROBLEM?

• Informally: A description of input,

and the desired output

WHAT IS AN ALGORITHM?
• Informally: A well-defined
procedure (sequence of steps)
to solve a computational problem Correctness?

10
ANALYSIS OF ALGORITHMS
• Every program uses resources
• CPU instructions / cycles  time
• Memory (RAM)  space
• Others: I/O, network bandwidth/messages, locks…
(not covered in this course)
• Analysis is the study of how many resources an algorithm uses
• Usually using big-O notation (to ignore constant factors)

11
But how do we know
how much time will
take on input ?

Depends on the
model of
computation

12
MODELS OF COMPUTATION
• Make analysis possible
• Ones covered in this course
• Unit cost model
• Word RAM model
• Bit complexity model

13
UNIT COST MODEL
• Each variable (or array entry) is a word
• Words can contain unlimited bits
• Basic operations on words take O(1) time
• Read/write a word in O(1)
• Add two words in O(1)
• Multiply two words in O(1)
• Space complexity is the number of words used
(excluding the input)

14
 BUT SOMETIMES WE CARE ABOUT WORD SIZE
n in decimal n in binary
• Suppose we want to limit the size
1 1
of words 2 10
• Must consider how many bits are 3 11
4 100
needed to represent a number
5 101
6 110
7 111
Need bits to store 8 1000
9 1001
10 1010
i.e., bits
11 1011
12 1100

15
WORD RAM MODEL
• Key difference: we care about the size of words
• Words can contain O(lg n) bits,
where n is the number of words in the input
• Word size depends on input size!
• Intuition: if the input is an array of n words,
a word is large enough to store an array index
• Basic operations on words still take O(1) time
• (but the values they can contain are limited)

16
BIT COMPLEXITY MODEL
• Each variable (or array entry) is a bit string
• Size of a variable x is the number of bits it needs
• It takes O(log v) bits to represent a value v
• So if v is stored in x, the size of x must be bits
• Basic operations are performed on individual bits
• Read/write a bit in O(1)
• Add/multiply two bits in O(1)
• Space complexity is the total number of bits used
(excluding the input)

17
 BENTLEY’S PROBLEM
A worked example to demonstrate algorithm design & analysis

18
Example 1 1 7 4 0 2 1 3 1 Solution: 19
Array index 1 2 3 4 5 6 7 8 (take all of A[1..8])

Example 2 -1 -7 -4 -1 -2 -1 -3 -1 Solution: 0
(take no elements of A)
Index 1 2 3 4 5 6 7 8

Example 3 1 -7 4 0 2 -1 3 -1 Solution: 8
Index 1 2 3 4 5 6 7 8 (take A[3..7])

19
 Design: brute force

Try all combinations of

And for each combination,
sum over

𝒌
𝒊 𝒋
1 -7 4 0 2 -1 3 -1

in unit cost model?

20
 Design: slightly better
brute force

Avoid repeatedly summing over

1 -7 4 0 2 -1 3 -1

𝒊= 𝒋 𝒋 𝒋 𝒋

in unit cost model?

21
Case 1: optimal sol’n
A 9 -3 4 -5 -2 -5 3 -1

is entirely in L L 9 -3 4 -5 R -2 -5 3 -1

Case 2: optimal sol’n

is entirely in R

Case 3: optimal sol’n

A 1 -7 4 0 2 -1 3 0

crosses the partition L 1 -7 4 0 R 2 -1 3 0

22
Find: maximum subarray
going over the middle A 1 -7 4 0 2 -1 3 0
Index 1 2 … n/2 n/2+1 … n
partition

𝒊 𝒋
Find that maximizes the
We can prove is the
sum over
Find that maximizes the sum maximum subarray going
over over the middle partition!

23
WHY IS MAXIMAL
• Suppose not for contradiction
• Then some that crosses the partition
has a larger sum
A
But both are
𝐿 𝑅
impossible!
𝑖 𝑗
𝐿′ 𝑅′ This sum is bigger
𝑖′ 𝑗′ So either
or

24
 A 9 -3 4 -5 -2 -5 3 -1

L 9 -3 4 -5 R -2 -5 3 -1

maxL = 10 maxR = 3

maxM = maxI + maxJ = 5

A 9 -3 4 -5 -2 -5 3 -1
Index 1 2 … n/2 … n

maxI = 5 maxJ = 0

Return max( 10, 3, 5 ) = 10

25
 A 1 -7 4 0 2 -1 3 0

L 1 -7 4 0 R 2 -1 3 0

maxL = 4 maxR = 4

maxM = maxI + maxJ = 8

A 1 -7 4 0 2 -1 3 0
Index 1 2 … n/2 … n

maxI = 4
maxJ = 4

Return max( 4, 4, 8 ) = 8
26
 (in unit cost model)

How do we analyze this running time?

Need new mathematical techniques!

Recurrence relations, recursion tree methods,

master theorem…

This result is really quite good…

but can we do asymptotically better?

27
ANALYSIS IN THE BIT COMPLEXITY MODEL
• Revisiting Solution 1 Can only add a pair of bits in
O(1) time. How many bits are
added here?

bits.

???

𝐬𝐮𝐦=𝐀 [ 𝒊 ]+ …+𝐀[𝒌−𝟏]

so bits

How to simplify?

28
 COMPLEXITY OF ADDITION
Adding two numbers x+y takes O(max{size(x),
Fun fact: the size of x+y can be 1 bit
size(y)}) bit operations
larger than either x or y
(multiplication can double #bits)
This can be rewritten O(size(x)+size(y))
= O(lg x + lg y)

Let

bits
bits
Optional: simplify to

29
ADDING SUM AND A[K]
• Recall , bits
• Adding them takes
bit operations
• And since we get:

• And the first term asymptotically dominates:

30
ZOOMING OUT TO THE K LOOP
• The addition happens for all values of
Careful to check this does
• Total time for the loop is at most not affect the complexity
(much).
• Complicated to sum for (Check by finding similar
so get an upper bound with result.)

And similarly
• for this…

31
ACCOUNTING FOR THE OUTER LOOPS
• loop is repeated
at most times
• Each time taking at most
time
• So total runtime is
time

Difference is due to
Compare to unit cost model:
(1) growth in variable sizes and
time
(2) cost of bitwise addition
log-factor difference is common… 32
HOW ABOUT WORD RAM?
• If each variable fits in a single word,
the analysis (and result) is as in the unit cost model
• Since there are input words,
each will fit in one word only if
• i.e., if
• If a variable is too big to fit in a word,
it is stored in multiple words,
and analysis looks more like bit complexity model

33
BENTLEY’S SOLUTIONS: RUNTIME IN PRACTICE

• Consider solutions
implemented in C n Sol.4 O(n) Sol.3 O(n lg n) Sol.2 O(n2) Sol.1 O(n3)
100 0 0 0 0
• Some values 1,000 0 0 0 0.12
10,000 0 0 0.036 2 minutes
measured on a 100,000 0 0.002 3.582 33 hours
Threadripper 3970x 1M 0.001 0.017 6 minutes 4 years
10M 0.012 0.195 12 hours 3700 years
• Red values 100M 0.112 2.168 50 days 3.7M years
1 billion 1.124 24.57 1.5 years > age of life
extrapolated from 10 billion 19.15 5 minutes 150 years > age of universe
measurements
• 0 represents time
under 0.01s
34
HOMEWORK: BIG-O REVIEW & EXERCISES

35
𝑓 (𝑛 ) ∈𝑂(𝑔 ( 𝑛 ))

36
 𝑓 (𝑛 )

𝑓 (𝑛 ) ∈Ω(𝑔 ( 𝑛)) 𝑔 ( 𝑛)

37
𝑓 (𝑛 ) ∈Θ(𝑔 ( 𝑛 )) 𝑓 (𝑛 ) ∈O(𝑔 ( 𝑛 ))
𝑔 ( 𝑛) ∈Θ( 𝑓 ( 𝑛 )) 𝑓 (𝑛 ) ∈Ω(𝑔 ( 𝑛))
𝑂+Ω=Θ
38
 But NOT
implies
vice versa

But NOT
implies
vice versa

39
EXERCISE
• Which of the following are true?

40
EXERCISE
• Which of the following are true?
• YES
• YES
• NO
• YES
• YES
• NO
• NO

41
COMPARING GROWTH RATES

42
43
LIMIT TECHNIQUE
FOR COMPARING GROWTH RATES

44
LIMIT RULES 1/3

All of the identities shown hold

only if the limits exist

45
LIMIT RULES 2/3

46
LIMIT RULES 3/3

Limit of an Exponential Function

Limit of a Logarithm of a Function

(Where base )

47
L’HOSPITAL’S RULE
• Often we take the limit of where
both and tend to , or
both and tend to
• Such limits require L’Hospital’s rule
• This rule says the limit of in this case
is the same as the limit of the derivative
• In other words,

48
USING THE LIMIT METHOD: EXERCISE 1
• Compare growth rate of and

• So

49
USING THE LIMIT METHOD: EXERCISE 2
• Compare growth rate of and

50
USING THE LIMIT METHOD: EXERCISE 2
• Compare growth rate of and

• So,

51
Try these at home…

52
 SUMMATIONS
AND SEQUENCES

53
This is included for
your notes

54
55
SEQUENCES

56
SEQUENCES CONTINUED This is included for
your notes

57
This is included for
your notes

58
LOGARITHM RULES

59
60
BASE OF LOGARITHM DOES NOT MATTER!

• Big-O notation does not distinguish between log bases

• Proof:
• Fix two constant logarithm bases b and c
• From log rules, we can change from to
by using formula:
We typically omit the base, and
• But is a constant! just write
• So for this reason

61
LOOP ANALYSIS

62
META-ALGORITHM FOR ANALYZING LOOPS
• Identify operations that require only constant time
• The complexity of a loop is the sum of
the complexities of all iterations
• Analyze independent loops separately and add the results
• If loops are nested, it often helps to start at the innermost,
and proceed outward… but,
• sometimes you must express several nested loops together in a single
equation (using nested summations),
• and actually evaluate the nested summations… (can be hard)

63
TWO BIG-O ANALYSIS STRATEGIES
• Strategy 1
• Prove a O-bound and a matching -bound
separately to get a -bound. Often easier
• Strategy 2 (but not always)

• Use -bounds throughout the analysis and thereby obtain a -bound for
the complexity of the algorithm

64
EXAMPLE 1

65
Strategy 1: big-O and big- bounds

66
Strategy 2: use -bounds throughout the analysis

67
EXAMPLE 2
𝑂 (1) Consider this loop alone…
number of loop iterations?

𝑂 (1) starts at and is repeatedly divided by 2… this

can happen only times

𝑂 (1) So inner loop has runtime

𝑂 (1)
And the entire inner loop is executed
for
𝑂 (1)
So, we have

( ) ( ) ( ) ( )
𝑛 𝑛 𝑛 𝑛
𝑛
𝑇 ( 𝑛 ) ∈O ∑ log 𝑖 ⊆O ∑ log 𝑛 ⊆ 𝑶(𝒏 𝐥𝐨𝐠 𝒏) 𝑇 ( 𝑛) ∈ Ω ∑ log 𝑖 ⊆ Ω ∑ log 2
⊆ 𝜴 (𝒏 𝐥𝐨𝐠 𝒏)
𝑖=1 𝒏
𝑖=
𝑖=1 𝑖=1 𝟐

68
… ANOTHER EXERCISE
IN LOOP ANALYSIS?

69
EXAMPLE 3 (BENTLEY’S PROBLEM, SOLUTION 1)

Try to analyze this yourself!

One possible solution is given in
these slides…

70
 𝑛
𝑂 (1)
Strategy 1: big-O and big- bounds ∑…
𝒊=1

𝑂 (1)

𝑂 (1)

𝑂 (1)

This is the maximum number of

iterations that could be performed in
this loop

71
 Proving a big- bound…

Recall:

Intuition: is in some iterations. How many

iterations? Lots?

To get a good -bound, we ask questions like:

When do our loops have many iterations?
When is our dominant term large?

Many iterations: when our loop does iterations!

For example, when …

Large dominant term: when is much larger than

(i.e., by a factor of n)
72
 Proving a big- bound… continued

Recall:

Smallest possible value of

for these bounds on

We will perform at least this much work

in every iteration!

This term does not depend on the loop

indexes, so just multiply by the total
number of loop iterations…

Since we have and , we have proved

73
74
BONUS
• Study-song of the day
• Tool - Descending
• youtu.be/PcSoLwFisaw

75