BIG OH and THETA

(download slides and .py files to follow along)

6.100L Lecture 22
Ana Bell

1
TIMING

6.100L Lecture 22
TIMING A PROGRAM

 Use time module

 Importing means import time
bringing collection
of functions into def convert_to_km(m):
your own file return m * 1.609
 Start clock t0 = time.perf_counter()
 Call function convert_to_km(100000)
 Stop clock dt = time.perf_counter() - t0
print("t =", dt, "s,")

6.100L
6.0001Lecture
LECTURE 822
EXAMPLE: convert_to_km, compound

def convert_to_km(m):
return m * 1.609

def compound(invest, interest, n_months):

total=0
for i in range(n_months):
total = total * interest + invest
return total

 How long does it take to compute these functions?

 Does the time depend on the input parameters?
 Are the times noticeably different for these two
functions?
4

6.100L
6.0001Lecture
LECTURE 922
 CREATING AN INPUT LIST

L_N = [1]
for i in range(7):
L_N.append(L_N[-1]*10)

for N in L_N:
t = time.perf_counter()
km = convert_to_km(N)
dt = time.perf_counter()-t
print(f"convert_to_km({N}) took {dt} seconds ({1/dt}/sec)")

6.100L
6.0001Lecture
LECTURE 922
RUN IT!
convert_to_km OBSERVATIONS

Observation: average time seems independent of size of argument

6.100L Lecture 22
 MEASURE TIME:
compound with a variable number of months

def compound(invest, interest, n_months):

total=0
for i in range(n_months):
total = total * interest + invest
return total

Observation 1: Time grows with

the input only when n_months
changes
compound(1) took 2.26e-06 seconds (441,696.12/sec)
compound(10) took 2.31e-06 seconds (433,839.48/sec) Observation 2: average time
compound(100) took 6.59e-06 seconds (151,676.02/sec) seems to increase by 10 as size of
compound(1000) took 5.02e-05 seconds (19,938.59/sec) argument increases by 10
compound(10000) took 5.10e-04 seconds (1,961.80/sec)
compound(100000) took 5.14e-03 seconds (194.46/sec) Observation 3: relationship
compound(1000000) took 4.79e-02 seconds (20.86/sec) between size and time only
compound(10000000) took 4.46e-01 seconds (2.24/sec) predictable for large sizes
7

6.100L
6.0001Lecture
LECTURE 922
 MEASURE TIME: sum over L Observation 1: Size of the input is
now the length of the list, not
how big the element numbers are.
def sum_of(L):
total = 0.0 Observation 2: average time
for elt in L: seems to increase by 10 as size of
total = total + elt argument increases by 10
return total
Observation 3: relationship
L_N = [1] between size and time only
for i in range(7): predictable for large sizes
L_N.append(L_N[-1]*10)
Observation 4: Time seems
for N in L_N: comparable to computation of
L = list(range(N)) compound
t = time.perf_counter()
s = sum_of(L)
dt = time.perf_counter()-t
print(f"sum_of({N}) took {dt} seconds ({1/dt}/sec)")

6.100L
6.0001Lecture
LECTURE 922
MEASURE TIME: find element in a list
# search each element one-by-one
def is_in(L, x):
for elt in L:
if elt==x:
return True
return False

# search by bisecting the list (list should be sorted!)

def binary_search(L, x):
lo = 0
hi = len(L)
while hi-lo > 1:
mid = (hi+lo) // 2
if L[mid] <= x:
lo = mid
else:
hi = mid
return L[lo] == x

# search using built-in operator

x in L 9

6.100L
6.0001Lecture
LECTURE 922
MEASURE TIME: find element in a list

is_in(10000000) took 1.62e-01 seconds (6.16/sec)

9.57 times more than for 10 times fewer elements
binary(10000000) took 9.37e-06 seconds (106,761.64/sec)
1.40 times more than for 10 times fewer elements
builtin(10000000) took 5.64e-02 seconds (17.72/sec)
9.63 times more than for 10 times fewer elements

is_in(100000000) took 1.64e+00 seconds (0.61/sec)

10.12 times more than for 10 times fewer elements
binary(100000000) took 1.18e-05 seconds (84,507.09/sec)
1.26 times more than for 10 times fewer elements
builtin(100000000) took 5.70e-01 seconds (1.75/sec)
10.11 times more than for 10 times fewer elements

Observation 1: searching one-by-one grows by factor of 10, when L increases by 10

10

9/28/20 6.100L
6.0001Lecture
LECTURE 922
MEASURE TIME: find element in a list

is_in(10000000) took 1.62e-01 seconds (6.16/sec)

9.57 times more than for 10 times fewer elements
binary(10000000) took 9.37e-06 seconds (106,761.64/sec)
1.40 times more than for 10 times fewer elements
builtin(10000000) took 5.64e-02 seconds (17.72/sec)
9.63 times more than for 10 times fewer elements

is_in(100000000) took 1.64e+00 seconds (0.61/sec)

10.12 times more than for 10 times fewer elements
binary(100000000) took 1.18e-05 seconds (84,507.09/sec)
1.26 times more than for 10 times fewer elements
builtin(100000000) took 5.70e-01 seconds (1.75/sec)
10.11 times more than for 10 times fewer elements

Observation 1: searching one-by-one grows by factor of 10, when L increases by 10

Observation 2: built-in function grows by factor of 10, when L increases by 10

11

9/28/20 6.100L
6.0001Lecture
LECTURE 922
MEASURE TIME: find element in a list

is_in(10000000) took 1.62e-01 seconds (6.16/sec)

9.57 times more than for 10 times fewer elements
binary(10000000) took 9.37e-06 seconds (106,761.64/sec)
1.40 times more than for 10 times fewer elements
builtin(10000000) took 5.64e-02 seconds (17.72/sec)
9.63 times more than for 10 times fewer elements

is_in(100000000) took 1.64e+00 seconds (0.61/sec)

10.12 times more than for 10 times fewer elements
binary(100000000) took 1.18e-05 seconds (84,507.09/sec)
1.26 times more than for 10 times fewer elements
builtin(100000000) took 5.70e-01 seconds (1.75/sec)
10.11 times more than for 10 times fewer elements

Observation 1: searching one-by-one grows by factor of 10, when L increases by 10

Observation 2: built-in function grows by factor of 10, when L increases by 10
Observation 3: binary search time seems almost independent of size

12

9/28/20 6.100L
6.0001Lecture
LECTURE 922
MEASURE TIME: find element in a list

is_in(10000000) took 1.62e-01 seconds (6.16/sec)

9.57 times more than for 10 times fewer elements
binary(10000000) took 9.37e-06 seconds (106,761.64/sec)
1.40 times more than for 10 times fewer elements
builtin(10000000) took 5.64e-02 seconds (17.72/sec)
9.63 times more than for 10 times fewer elements

is_in(100000000) took 1.64e+00 seconds (0.61/sec)

10.12 times more than for 10 times fewer elements
binary(100000000) took 1.18e-05 seconds (84,507.09/sec)
1.26 times more than for 10 times fewer elements
builtin(100000000) took 5.70e-01 seconds (1.75/sec)
10.11 times more than for 10 times fewer elements

Observation 1: searching one-by-one grows by factor of 10, when L increases by 10

Observation 2: built-in function grows by factor of 10, when L increases by 10
Observation 3: binary search time seems almost independent of size
Observation 4: binary search much faster than is_in, especially on larger problems
13

9/28/20 6.100L
6.0001Lecture
LECTURE 922
MEASURE TIME: find element in a list

is_in(10000000) took 1.62e-01 seconds (6.16/sec)

9.57 times more than for 10 times fewer elements
binary(10000000) took 9.37e-06 seconds (106,761.64/sec)
1.40 times more than for 10 times fewer elements
builtin(10000000) took 5.64e-02 seconds (17.72/sec)
9.63 times more than for 10 times fewer elements

is_in(100000000) took 1.64e+00 seconds (0.61/sec)

10.12 times more than for 10 times fewer elements
binary(100000000) took 1.18e-05 seconds (84,507.09/sec)
1.26 times more than for 10 times fewer elements
builtin(100000000) took 5.70e-01 seconds (1.75/sec)
10.11 times more than for 10 times fewer elements

Observation 1: searching one-by-one grows by factor of 10, when L increases by 10

Observation 2: built-in function grows by factor of 10, when L increases by 10
Observation 3: binary search time seems almost independent of size
Observation 4: binary search much faster than is_in, especially on larger problems
Observation 5: is_in is slightly slower than
14 using Python’s “in” capability
9/28/20 6.100L
6.0001Lecture
LECTURE 922
MEASURE TIME: find element in a list
def is_in(L, x):
for elt in L:
if elt==x:
return True So we have seen
return False computations where
time seems very
def binary_search(L, x): different
lo = 0 • Constant time
hi = len(L)
while hi-lo > 1:
• Linear in size of
mid = (hi+lo) // 2 argument
if L[mid] <= x: • Something less than
lo = mid
else:
linear?
hi = mid
return L[lo] == x
15

6.100L
6.0001Lecture
LECTURE 922
 MEASURE TIME: diameter function
L=[(cos(0), sin(0)),
(cos(1), sin(1)),
(cos(2), sin(2)), ... ] #example numbers
def diameter(L):
farthest_dist = 0
for i in range(len(L)):
for j in range(i+1, len(L)):
p1 = L[i]
p2 = L[j]
dist = math.sqrt((p1[0]-p2[0])**2+(p1[1]-p2[1])**2)
if dist > farthest_dist:
farthest_dist = dist
return farthest_dist

16

6.100L
6.0001Lecture
LECTURE 922
 MEASURE TIME: diameter function

def diameter(L):
farthest_dist = 0
for i in range(len(L)):
for j in range(i+1, len(L)):
p1 = L[i]
p2 = L[j]
dist = math.sqrt((p1[0]-p2[0])**2+(p1[1]-p2[1])**2)
if dist > farthest_dist:
farthest_dist = dist
return farthest_dist

17

6.100L
6.0001Lecture
LECTURE 922
 MEASURE TIME: diameter function

def diameter(L):
farthest_dist = 0
for i in range(len(L)):
for j in range(i+1, len(L)):
p1 = L[i]
p2 = L[j]
dist = math.sqrt((p1[0]-p2[0])**2+(p1[1]-p2[1])**2)
if dist > farthest_dist:
farthest_dist = dist
return farthest_dist

18

6.100L
6.0001Lecture
LECTURE 922
 MEASURE TIME: diameter function

def diameter(L):
farthest_dist = 0
for i in range(len(L)):
for j in range(i+1, len(L)):
p1 = L[i]
p2 = L[j]
dist = math.sqrt((p1[0]-p2[0])**2+(p1[1]-p2[1])**2)
if dist > farthest_dist:
farthest_dist = dist
return farthest_dist

19

6.100L
6.0001Lecture
LECTURE 922
 MEASURE TIME: diameter function

def diameter(L):
farthest_dist = 0
for i in range(len(L)):
for j in range(i+1, len(L)):
p1 = L[i]
p2 = L[j]
dist = math.sqrt((p1[0]-p2[0])**2+(p1[1]-p2[1])**2)
if dist > farthest_dist:
farthest_dist = dist
return farthest_dist

20

6.100L
6.0001Lecture
LECTURE 922
 MEASURE TIME: diameter function

def diameter(L):
farthest_dist = 0
for i in range(len(L)):
for j in range(i+1, len(L)):
p1 = L[i]
p2 = L[j]
dist = math.sqrt((p1[0]-p2[0])**2+(p1[1]-p2[1])**2)
if dist > farthest_dist:
farthest_dist = dist
return farthest_dist

21

6.100L
6.0001Lecture
LECTURE 922
 MEASURE TIME: diameter function

def diameter(L):
farthest_dist = 0
for i in range(len(L)):
for j in range(i+1, len(L)):
p1 = L[i]
p2 = L[j]
dist = math.sqrt((p1[0]-p2[0])**2+(p1[1]-p2[1])**2)
if dist > farthest_dist:
farthest_dist = dist
return farthest_dist

22

6.100L
6.0001Lecture
LECTURE 922
 MEASURE TIME: diameter function

def diameter(L):
farthest_dist = 0
for i in range(len(L)):
for j in range(i+1, len(L)):
p1 = L[i]
p2 = L[j]
dist = math.sqrt((p1[0]-p2[0])**2+(p1[1]-p2[1])**2)
if dist > farthest_dist:
farthest_dist = dist
return farthest_dist

 Gets much slower as size of input grows

 Quadratic: for list of size len(L), does len(L)/2 operations
per element on average
 len(L) x len(L)/2 operations — worse than linear growth
23

6.100L
6.0001Lecture
LECTURE 922
PLOT OF INPUT SIZE vs. TIME TO RUN

is_in
binary_search

linear logarithmic

diameter

quadratic

24

6.100L Lecture 22
 TWO DIFFERENT MACHINES
My old laptop My old desktop

~2x slower for large problems

Observation 1: even for the same code, the actual machine may affect speed.
Observation 2: Looking only at the relative increase in run time from a prev run,
if input is n times as big, the run time is approx. n times as long.
25
DON’T GET ME WRONG!

 Timing is a critical tool to assess the performance of programs

 At the end of the day, it is irreplaceable for real-world
assessment

 But we will see a complementary tool (asymptotic complexity)

that has other advantages
 A priori evaluation (before writing or running code)
 Assesses algorithm independent of machine and
implementation (what is intrinsic efficiency of algorithm?)
 Provides direct insight into the design of efficient
algorithms
26

6.100L
6.0001Lecture
LECTURE 922
COUNTING

27

6.100L Lecture 22
COUNT OPERATIONS

convert_to_km  2 ops
 Assume these steps take def convert_to_km(m):
constant time: return m * 1.609
• Mathematical operations
• Comparisons
sum_of  1+len(L)*3+1 = 3*len(L)+2 ops
• Assignments def sum_of(L):
• Accessing objects in total = 0
memory for i in L:
 Count number of these total += i
operations executed as return total
function of size of input

28

6.100L
6.0001Lecture
LECTURE 822
COUNT OPERATIONS: is_in

def is_in_counter(L, x):

global count
count += 1 #return of value
for elt in L:
count += 2 # set elt, if == test
if elt==x:
return True
return False

29

6.100L
6.0001Lecture
LECTURE 922
COUNT OPERATIONS: is_in

def is_in_counter(L, x):

global count
count += 1
for elt in L:
count += 2
if elt==x:
return True
return False

30

6.100L
6.0001Lecture
LECTURE 922
COUNT OPERATIONS:
binary search

def binary_search_counter(L, x):

global count
lo = 0
hi = len(L)
count += 3
while hi-lo > 1:
count += 2
mid = (hi+lo) // 2
count += 3
if L[mid] <= x:
lo = mid
else:
hi = mid
count += 3
count += 3
return L[lo] == x
31

6.100L
6.0001Lecture
LECTURE 922
COUNT OPERATIONS

is_in testing
Observation 1: number of
for 1 element, is_in used 9 operations
operations for is_in increases by
for 10 element, is_in used 37 operations
10 as size increases by 10
for 100 element, is_in used 307 operations
for 1000 element, is_in used 3007 operations
for 10000 element, is_in used 30007 operations
for 100000 element, is_in used 300007 operations
for 1000000 element, is_in used 3000007 operations

binary_search testing
for 1 element, binary search used 15 operations Observation 2: but number
for 10 element, binary search used 85 operations of operations for binary
for 100 element, binary search used 148 operations search grows much more
for 1000 element, binary search used 211 operations slowly. Unclear at what rate.
for 10000 element, binary search used 295 operations
for 100000 element, binary search used 358 operations
for 1000000 element, binary search used
32
421 operations
10/5/20 6.100L
6.0001Lecture
LECTURE 922
PLOT OF INPUT SIZE vs. OPERATION COUNT

33

6.100L Lecture 22
PROBLEMS WITH TIMING AND COUNTING

 Timing the exact running time of the program

• Depends on machine
• Depends on implementation
• Small inputs don’t show growth
 Counting the exact number of steps
• Gets us a formula!
• Machine independent, which is good
• Depends on implementation
• Multiplicative/additive constants are irrelevant for large inputs
 Want to:
 evaluate algorithm
 evaluate scalability
 evaluate in terms of input size
34

6.100L Lecture 22
 EFFICIENCY IN TERMS OF INPUT: BIG-PICTURE
RECALL mysum (one loop) and square (nested loops)
 mysum(x)
 What happened to the program efficiency as x increased?
 10 times bigger x meant the program
 Took approx. 10 times as long to run
 Did approx. 10 times as many ops
 Express it in an “order of” way vs. the input variable: efficiency = Order of x
 square(x)
 What happened to the program efficiency as x increased?
 2 times bigger x meant the program
 Took approx. 4 times as long to run
 Did approx. 4 times as many ops
 10 times bigger x meant the program
 Took approx. 100 times as long to run
 Did approx. 100 times as many ops
 Express it in an “order of” way vs. the input variable: efficiency = Order of x2
35

6.100L Lecture 22
ORDER of GROWTH

36

6.100L Lecture 22
ORDERS OF GROWTH

 It’s a notation
 Evaluates programs when input is very big
 Expresses the growth of program’s run time
 Puts an upper bound on growth
 Do not need to be precise: “order of” not “exact” growth

 Focus on the largest factors in run time (which section of

the program will take the longest to run?)

37

6.100L Lecture 22
A BETTER WAY
A GENERALIZED WAY WITH APPROXIMATIONS

 Use the idea of counting operations in an algorithm, but not

worry about small variations in implementation
 When x is big, 3x+4 and 3x and x are pretty much the same!
 Don’t care about exact value: ops = 1+x(2+1)
 Express it in an “order of” way vs. the input: ops = Order of x

 Focus on how algorithm performs when size of problem gets

arbitrarily large
 Relate time needed to complete a computation against the
size of the input to the problem
 Need to decide what to measure. What is the input?
38

6.100L Lecture 22
WHICH INPUT TO USE TO MEASURE EFFICIENCY

 Want to express efficiency in terms of input, so need to

decide what is your input

 Could be an integer
-- convert_to_km(x)
 Could be length of list
-- list_sum(L)
 You decide when multiple parameters to a function
-- is_in(L, e)
 Might be different depending on which input you consider

39

6.100L
6.0001Lecture
LECTURE 822
DIFFERENT INPUTS CHANGE HOW
THE PROGRAM RUNS

 A function that searches for an element in a list

def is_in(L, e):
for i in L:
if i == e:
return True
return False

 Does the program take longer to run as e increases?

 No

40

6.100L
6.0001Lecture
LECTURE 822
DIFFERENT INPUTS CHANGE HOW
THE PROGRAM RUNS

 A function that searches for an element in a list

def is_in(L, e):
for i in L:
if i == e:
return True
return False

 Does the program take longer to run as L increases?

 What if L has a fixed length and its elements are big numbers?
 No
 What if L has different lengths?
 Yes!

41

6.100L
6.0001Lecture
LECTURE 822
DIFFERENT INPUTS CHANGE HOW
THE PROGRAM RUNS

 A function that searches for an element in a list

def is_in(L, e):
for i in L:
if i == e:
return True
return False
 When e is first element in the list
 BEST CASE
 When look through about half of the elements in list
 AVERAGE CASE
 When e is not in list
 WORST CASE
 Want to measure this behavior in a general way
42

6.100L
6.0001Lecture
LECTURE 822
ASYMPTOTIC GROWTH

 Goal: describe how time grows as size of input grows

 Formula relating input to number of operations
 Given an expression for the number of operations needed to
compute an algorithm, want to know asymptotic behavior as size
of problem gets large
 Want to put a bound on growth
 Do not need to be precise: “order of” not “exact” growth
 Will focus on term that grows most rapidly
 Ignore additive and multiplicative constants, since want to know how
rapidly time required increases as we increase size of input
 This is called order of growth
 Use mathematical notions of “big O” and “big Θ”
Big Oh and Big Theta
43

6.100L
6.0001Lecture
LECTURE 822
BIG O Definition
3𝑥𝑥 2 + 20𝑥𝑥 + 1 = 𝑂𝑂(𝑥𝑥 2 )

 Suppose some code runs in

𝑓𝑓(𝑥𝑥) = 3𝑥𝑥 2 + 20𝑥𝑥 + 1 steps
Not an upper bound; as 𝑥𝑥 → ∞ Think of this as the formula from
f(x) will always exceed it counting the number of ops.
Never
cross  Big OH is a way to upper bound the
𝑔𝑔(𝑥𝑥) = 𝑥𝑥 again! growth of any function

Crossover
 f(x) = O(g(x)) means that g(x) times
some constant eventually always
exceeds f(x)
Eventually means above some
threshold value of x
4𝑥𝑥 2 > 3𝑥𝑥 2 + 20𝑥𝑥 + 1∀𝑥𝑥 > 20.04

44

6.100L Lecture 22
BIG O FORMALLY
 A big Oh bound is an upper bound on the growth of some function
 𝑓𝑓(𝑥𝑥) = 𝑂𝑂(𝒈𝒈(𝒙𝒙)) means there exist
constants 𝒄𝒄𝟎𝟎 , 𝒙𝒙𝟎𝟎 for which 𝒄𝒄𝟎𝟎 𝒈𝒈(𝒙𝒙) ≥ 𝒇𝒇(𝒙𝒙) for all 𝑥𝑥 > 𝒙𝒙𝟎𝟎
Example: 𝑓𝑓(𝑥𝑥) = 3𝑥𝑥 2 + 20𝑥𝑥 + 1

𝑓𝑓(𝑥𝑥) = 𝑂𝑂(𝒙𝒙𝟐𝟐 ) ,because 𝟒𝟒 𝒙𝒙𝟐𝟐 > 𝟑𝟑𝒙𝒙𝟐𝟐 + 𝟐𝟐𝟐𝟐𝟐𝟐 + 𝟏𝟏∀𝑥𝑥 ≥ 𝟐𝟐𝟐𝟐

(𝒄𝒄𝟎𝟎 = 𝟒𝟒, 𝒙𝒙𝟎𝟎 = 𝟐𝟐𝟐𝟐. 𝟎𝟎𝟎𝟎)

Crossover at
x=20.04 orange > blue
for all x > 20.04)
These lines
will never
cross again

0 <= x <= 30 45
0 <= x <= 100
6.100L Lecture 22
 BIG Θ Definition 3𝑥𝑥 2 − 20𝑥𝑥 − 1 = 𝜃𝜃(𝑥𝑥 2 )

 A big Θ bound is a lower and upper bound on the growth of some function
Suppose 𝑓𝑓(𝑥𝑥) = 3𝑥𝑥 2 − 20𝑥𝑥 − 1
𝒇𝒇(𝒙𝒙) = Θ(𝒈𝒈(𝒙𝒙)) means:
there exist constants 𝒄𝒄𝟎𝟎 , 𝑥𝑥0 for which 𝒄𝒄𝟎𝟎 𝒈𝒈(𝒙𝒙) ≥ 𝒇𝒇(𝒙𝒙) for all 𝑥𝑥 > 𝒙𝒙𝟎𝟎
and constants 𝒄𝒄𝟏𝟏 , 𝑥𝑥1 for which 𝒄𝒄𝟏𝟏 𝒈𝒈(𝒙𝒙) ≤ 𝒇𝒇(𝒙𝒙) for all 𝑥𝑥 > 𝒙𝒙𝟏𝟏
 Example, 𝒇𝒇(𝒙𝒙) = Θ(𝒙𝒙𝟐𝟐 ) because 𝟒𝟒𝒙𝒙𝟐𝟐 > 𝟑𝟑𝒙𝒙𝟐𝟐 − 𝟐𝟐𝟐𝟐𝟐𝟐 − 𝟏𝟏 ∀𝑥𝑥 ≥ 𝟎𝟎 (𝒄𝒄𝟎𝟎 = 𝟒𝟒, 𝒙𝒙𝟎𝟎 = 𝟎𝟎)
and 𝟐𝟐𝒙𝒙𝟐𝟐 < 𝟑𝟑𝒙𝒙𝟐𝟐 − 𝟐𝟐𝟐𝟐𝟐𝟐 − 𝟏𝟏 ∀𝑥𝑥 ≥ 𝟐𝟐𝟐𝟐 (𝒄𝒄𝟏𝟏 = 𝟐𝟐, 𝒙𝒙𝟏𝟏 = 𝟐𝟐𝟐𝟐. 𝟎𝟎𝟎𝟎)

orange > blue These lines

for all x > 0 will never
cross again
blue > green
for all x > 20.04

46

6.100L Lecture 22 0 <= x <= 100

 Θ vs O

 In practice, Θ bounds are preferred, because they are “tight”

For example: 𝑓𝑓(𝑥𝑥) = 3𝑥𝑥 2 − 20𝑥𝑥 − 1

 𝑓𝑓 𝑥𝑥 = 𝑂𝑂 𝑥𝑥 2 = 𝑂𝑂 𝑥𝑥 3 = 𝑂𝑂(2𝑥𝑥 ) and anything higher order

because they all upper bound it

 𝒇𝒇 𝒙𝒙 = 𝜣𝜣 𝒙𝒙𝟐𝟐
≠ Θ 𝑥𝑥 3 ≠ Θ 2𝑥𝑥 and anything higher order because they
upper bound but not lower bound it

47

6.100L Lecture 22
SIMPLIFICATION EXAMPLES

 Drop constants and multiplicative factors

 Focus on dominant term

Θ(n2) : n2 + 2n + 2
Θ(x2) : 3x2 + 100000x + 31000
Θ(a) : log(a) + a + 4

48

6.100L
6.0001Lecture
LECTURE 822
 BIG IDEA
Express Theta in terms of
the input.
Don’t just use n all the time!

49

6.100L Lecture 22
YOU TRY IT!
Θ(x) : 1000*log(x) + x
Θ(n3) : n2log(n) + n3
Θ(y) : log(y) + 0.000001y
Θ(2b) : 2b + 1000a2 + 100*b2 + 0.0001a3
Θ(a3)
Θ(2b+a3)
All could be ok, depends on the input we care about

50

6.100L Lecture 22
USING Θ TO EVALUATE YOUR
ALGORITHM

def fact_iter(n):
"""assumes n an int >= 0"""
answer = 1
while n > 1:
answer *= n
n -= 1
return answer

 Number of steps: 5n + 2
 Worst case asymptotic complexity: Θ(n)
 Ignore additive constants
 2 doesn’t matter when n is big
 Ignore multiplicative constants
 5 doesn’t matter if just want to know how increasing n changes time
needed
51

6.100L
6.0001Lecture
LECTURE 822
COMBINING COMPLEXITY CLASSES
LOOPS IN SERIES
 Analyze statements inside functions to get order of growth
 Apply some rules, focus on dominant term

 Law of Addition for Θ():

 Used with sequential statements
 Θ(𝑓𝑓(𝑛𝑛)) + Θ(𝑔𝑔(𝑛𝑛)) = Θ(𝑓𝑓(𝑛𝑛) + 𝑔𝑔(𝑛𝑛))
 For example,
for i in range(n): Θ(n)
print('a')
for j in range(n*n): Θ(n2)
print('b')
is Θ(𝑛𝑛) + Θ(𝑛𝑛 ∗ 𝑛𝑛) = Θ(𝑛𝑛 + 𝑛𝑛2 ) = Θ(𝑛𝑛2 ) because of
dominant 𝑛𝑛2 term
52

6.100L
6.0001Lecture
LECTURE 822
COMBINING COMPLEXITY CLASSES
NESTED LOOPS
 Analyze statements inside functions to get order of growth
 Apply some rules, focus on dominant term

 Law of Multiplication for Θ():

 Used with nested statements/loops
 Θ 𝑓𝑓 𝑛𝑛 ∗ Θ(𝑔𝑔(𝑛𝑛)) = Θ(𝑓𝑓 𝑛𝑛 ∗ 𝑔𝑔(𝑛𝑛))
 For example,
Θ(n)
for i in range(n):
for j in range(n//2): Θ(n) for each outer loop iteration
print('a')
 Θ(𝑛𝑛) × Θ(𝑛𝑛) = Θ(𝑛𝑛 × 𝑛𝑛) = Θ(𝑛𝑛2 )
 Outer loop runs n times and the inner loop runs n times
for every outer loop iteration.
53

6.100L
6.0001Lecture
LECTURE 822
ANALYZE COMPLEXITY

 What is the Theta complexity of this program?

def f(x):
answer = 1
for i in range(x): Outer loop is Θ(x)
for j in range(i,x): Inner loop is Θ(x)
answer += 2 Everything else is Θ(1)
return answer

 Θ(1) + Θ(x)* Θ(x)* Θ(1) + Θ(1)

 Overall complexity is Θ(x2) by rules of addition and
multiplication

54

6.100L Lecture 22
YOU TRY IT!
 What is the Theta complexity of this program? Careful to
describe in terms of input
(hint: what matters with a list, size of elems of length?)
def f(L):
Lnew = []
for i in L:
Lnew.append(i**2)
return Lnew

ANSWER:
Loop: Θ(len(L))
f is Θ(len(L))

55

6.100L Lecture 22
YOU TRY IT!
 What is the Theta complexity of this program?
def f(L, L1, L2):
""" L, L1, L2 are the same length """
inL1 = False
for i in range(len(L)):
if L[i] == L1[i]:
inL1 = True
inL2 = False
for i in range(len(L)):
if L[i] == L2[i]:
inL2 = True
return inL1 and inL2

ANSWER:
Loop: Θ(len(L)) + Θ(len(L))
f is Θ(len(L)) or Θ(len(L1)) or Θ(len(L2))

56

6.100L Lecture 22
 COMPLEXITY CLASSES

We want to design algorithms that are as

close to top of this hierarchy as possible

 Θ(1) denotes constant running time

 Θ(log n) denotes logarithmic running time
 Θ(n) denotes linear running time
 Θ(n log n) denotes log-linear running time
 Θ(nc) denotes polynomial running time
(c is a constant)
 Θ(cn) denotes exponential running time
(c is a constant raised to a power based on input size)
57

6.100L
6.0001Lecture
LECTURE 822
COMPLEXITY GROWTH

CLASS N = 10 N = 100 N = 1000 N = 1000000

Constant 1 1 1 1
Logarithmic 1 2 3 6
Linear 10 100 1000 1000000
Log-linear 10 200 3000 6000000
Polynomial 100 10000 1000000 1000000000000
Exponential 1024 12676506 1071508607186267320948425
0490600018105614048117055
Good Luck!!
00228229 3360744375038837035105112
4936122493198378815695858
40149670 1275946729175531468251871
3205376 4528569231404359845775746
9857480393456777482423098
5421074605062371141877954
1821530464749835819412673
9876755916554394607706291
4571196477686542167660429
8316526243868372056680693
76
58

6.100L Lecture 22
SUMMARY

 Timing is machine/implementation/algorithm dependent

 Counting ops is implementation/algorithm dependent
 Order of growth is algorithm dependent

 Compare efficiency of algorithms

• Notation that describes growth
• Lower order of growth is better
• Independent of machine or specific implementation
 Using Theta
• Describe asymptotic order of growth
• Asymptotic notation
• Upper bound and a lower bound
59

6.100L Lecture 22
MITOpenCourseWare
https://ocw.mit.edu

6.100L Introduction to Computer Science and Programming Using Python

Fall 2022

For information about citing these materials or our Terms ofUse,visit: https://ocw.mit.edu/terms.

60