Computational Complexity

Lecture 3

Introduction to Complexity Theory 1

 Introduction to Big O Notation
• Big O notation is one of the most necessary
mathematical notations used in computer science to
measure an algorithm's efficiency.
• We can analyze how efficient an algorithm is from the
amount of time, storage, and other resources it takes to
run the algorithm and a change in the input size.
• Big O Notation in data structure tells us how well an
algorithm will perform in a particular situation. In other
words, it gives an algorithm's upper-bound runtime or
worst-case complexity.

Introduction to Complexity Theory 2

 Continued …
• Asymptotic Notations can describe an algorithm's run
time when the input tends toward a specific or limiting
value.
• Asymptotic analysis helps to analyze the algorithm
performance change in the order of input size.
• Big O notation in data structure is used to express
algorithmic complexity using algebraic terms.
• It describes the upper bound of an algorithm's runtime
and calculates the time and amount of memory needed
to execute the algorithm for an input value

Introduction to Complexity Theory 3

 Continued …
Mathematical Definition
• Consider the functions f(n) and g(n), where functions f
and g are defined on an unbounded set of positive real
numbers. g(n) is strictly positive for every large value of n.
• The function f is said to be O(g) (read as big- oh of g), if,
for a constant c>0 and a natural number n0, f (n) ≤ CG(n)
for all n >= n0
This can be written as:
• f(n) = O(g(n)), where n tends to infinity (n ∞) →
• We can simply write the above expression as:
• f(n) = O(g(n))

Introduction to Complexity Theory 4

 Big O Notation
Understanding Big O Notation:
• Concept: Big O notation categorizes functions based
on their dominant growth rate as their input size (n)
approaches infinity. It helps analyze algorithms and
estimate their efficiency.
• Formal Definition: We say f(n) is O(g(n)) if there exist
positive constants c and N such that |f(n)| <= c * |g(n)| for
all n >= N. This means f(n) grows at most as fast as a
constant multiple of g(n) for sufficiently large n.

Introduction to Complexity Theory 5

 Big O Notation
Common Big O Cases:
 O(1): Constant time (independent of n), e.g.,
accessing an array element
 O(log n): Logarithmic time (growth slower than
linear), e.g., binary search
 O(n): Linear time (proportional to n), e.g., iterating
through an array
 O(n log n): Log-linear time (growth between linear
and quadratic), e.g., merge sort
 O(n^2): Quadratic time (faster than linear but slower
than exponential), e.g., nested loops for comparison
 O(2^n): Exponential time (rapidly increasing), e.g.,
brute-force search

Introduction to Complexity Theory 6

 How Does Big O Notation Make a
Runtime Analysis of an Algorithm

• In order to analyze and calculate an algorithm's
performance, we must calculate and compare the worst-
case runtime complexities of the algorithm.
• The order of O(1) - known as the Constant Running Time
- is the fastest running time for an algorithm, with the
time taken by the algorithm being equal for different
input sizes.
• Runtime complexity of some common algorithmic
examples:
• Runtime Complexity for Linear Search – O(n)
• Runtime Complexity for Binary Search – O(log n)
• Runtime Complexity for Binary Search – O(log n)
• Runtime Complexity for Exponential algorithms like Tower of Hanoi - O(c^n).
• Runtime Complexity for Heap Sort, Merge Sort - O(n log n).
• Runtime Complexity for Bubble Sort, Insertion Sort, Selection Sort, Bucket Sort -
O(n^c). Introduction to Complexity Theory 7
 How Does Big O Notation Analyze Space
Complexity?

• It is also essential to determine the space complexity of an
algorithm. This is because space complexity indicates how
much memory space the algorithm occupies.
• We compare the worst-case space complexities of the
algorithm.
• Before the Big O notation analyzes the Space complexity, the
following tasks need to be implemented:
– Implementation of the program for a particular algorithm.
– The size of input n needs to be known to calculate the memory
each item will hold.
• Space Complexities of some common algorithms:
• Linear Search, Binary Search, Bubble sort, Selection sort, Heap sort,
Insertion sort - Space Complexity is O(1).
• Radix sort - Space complexity is O(n+k).
• Quick Sort - Space complexity is O(n).
• Merge sort - Space Introduction
complexity is O(logTheory
to Complexity n). 8
Big O Growth hierarchy

Introduction to Complexity Theory 9

Big O Growth hierarchy with
Example:

Introduction to Complexity Theory 10

 Example in C programming:

Implementation of Selection Sort algorithm in C to
find worst-case complexity (Big O Notation) of the
algorithm:
for(int i=0; i<n; i++) Explanation:
• The range of the first (outer) for
{ loop is i<n, meaning the order of the
int min = i; loop is O(n).
for(int j=i; j<n; j++) • The range for the second (inner) for
{ loop is j<n; so, the order of the loop
is again O(n).
if(array[j]<array[min]) • Average efficiency is calculated as
min=j; n/2 for a constant c, but we ignore
} the constant. Thus, the order comes
int temp = array[i]; to be O(n).
• We get runtime complexity by
array[i] = array[min];
multiplying the inner and outer loop
array[min] = temp; order. It is O(n^2).
}
Introduction to Complexity Theory 11
 Tower of Hanoi
• Tower of Hanoi, is a mathematical puzzle which consists of three
towers (pegs) and more than one rings is as depicted

• These rings are of different sizes and stacked upon in an ascending

order, i.e. the smaller one sits over the larger one.

Introduction to Complexity Theory 12

 Tower of Hanoi - Rules
Rules
The mission is to move all the disks to some another tower
without violating the sequence of arrangement. A few rules to
be followed for Tower of Hanoi are:
• Only one disk can be moved among the towers at any given
time.
• Only the "top" disk can be removed.
• No large disk can sit over a small disk.

Introduction to Complexity Theory 13

 A recursive algorithm for Tower of Hanoi
can be driven as follows

• Step 1 − Move n-1 disks from source to aux
• Step 2 − Move nth disk from source to dest
• Step 3 − Move n-1 disks from aux to dest

START
Procedure Hanoi(disk, source, dest, aux)

IF disk == 1, THEN
move disk from source to dest
ELSE
Hanoi(disk - 1, source, aux, dest) // Step 1
move disk from source to dest // Step 2
Hanoi(disk - 1, aux, dest, source) // Step 3
END IF

END Procedure
STOP

Introduction to Complexity Theory 14

 Project (Group)
• Implement the Tower of Hanoi using
C++ / Java

• Submission deadline February 9/2024 @Morning

• Maximum number of group is 4

https://www.mathsisfun.com/games/towerofhanoi.html

Introduction to Complexity Theory 15

 Exercises
1. Analyze the Big O complexity of the following
functions:
 f(n) = 2n + 5
 g(n) = n^2 + 3n + 1
 h(n) = log(n^3)
 i(n) = 2^(n/2)
 j(n) = (n + 1)!

2. Find the smallest function g(n) such that f(n) =

O(g(n)) for the following:
 f(n) = 100
 f(n) = 3n^2 + 2n + 1
 f(n) = log(n!)

Introduction to Complexity Theory 16

 Summary
• Big O notation is a fundamental tool in computer
science for describing the asymptotic growth rate of an
algorithm's complexity (execution time or space
requirements) as the input size (denoted by n)
approaches infinity.
• In essence, it provides an upper bound on the algorithm's
worst-case performance, offering a high-level
understanding of its scalability without getting bogged
down in implementation-specific details.

Introduction to Complexity Theory 17

 Summary
Key Properties and Interpretations:
• Asymptotic Behavior: Big O focuses on how the
algorithm's complexity grows as n becomes very large,
neglecting constant factors and lower-order terms that
have less impact in the limit.
• Upper Bound: Big O notation represents the worst-case
scenario, capturing the maximum rate of growth the
algorithm can experience. Even if the actual complexity
grows slower in some cases, the Big O expression
provides a guarantee for the upper bound.

Introduction to Complexity Theory 18

 Summary
Choosing the Right Algorithm:
• Selecting the best algorithm for a task often involves
balancing factors like time and space complexity.
• For example, if you need to search a large sorted dataset,
binary search (O(log n)) will be much faster than linear
search (O(n)).
• However, if the data is unsorted, you might have to sort
it first (O(n log n)), impacting the overall efficiency.
Benefits of Big O Notation:
• Guides algorithm design and selection based on efficiency
requirements.
• Facilitates complexity analysis and comparison of different
algorithms.
• Enables reasoning about scalability and performance
implications as input size increases.

Introduction to Complexity Theory 19

 Summary
Beyond Big O:
While Big O is a powerful tool, it only captures the upper
bound of complexity. Other notations like Ω(lower bound),
Θ(exact bound), and o(smaller-order) provide more
nuanced insights:
• Ω(lower bound): Guarantees a lower bound on the
algorithm's complexity.
• Θ(exact bound): Gives the precise asymptotic growth
rate, considering both upper and lower bounds.
• o(smaller-order): Captures the complexity when it grows
slower than a given smaller-order expression.

Introduction to Complexity Theory 20

 Exercises with Answers
1.What is Big O notation? Give some examples.
 In computer science, Big O Notation is a
fundamental tool used to find out the time complexity
of algorithms. Big O Notation allows programmers to
classify algorithms depending on how their run time or
space requirements vary as the input size varies.
Examples:
• Runtime Complexity for Linear Search – O(n)
• Runtime Complexity for Binary Search – O(log n)
• Runtime Complexity for Bubble Sort, Selection Sort,
Insertion Sort, Bucket Sort - O(n^c).
• Runtime Complexity for Exponential algorithms like Tower
of Hanoi - O(c^n).
• Runtime Complexity for Heap Sort, Merge Sort - O(n log n).

Introduction to Complexity Theory 21

 Exercises with Answers
2. Why is Big O notation used?
 Big O Notation gives the upper-bound runtime or worst-
case complexity of an algorithm. It analyzes and classifies
algorithms depending on their run time or space
requirements.

3. What are time complexity and Big O notation?

 Time complexity refers to the amount of time an algorithm
takes to run when the input tends towards a specific or limiting
value. It calculates the time taken to execute each code
statement in an algorithm. Big O Notation is a tool used to
describe the time complexity of algorithms. It calculates the
time taken to run an algorithm as the input grows. In other words,
it calculates the worst-case time complexity of an algorithm.
Big O Notation in Data Structure describes the upper bound of
an algorithm's runtime. It calculates the time and amount of
memory needed to execute the algorithm for an input value.

Introduction to Complexity Theory 22

 Exercises with Answers
4. What is the other name for Big O notation?
Big O Notation is a mathematical notation named after
the term "order of the function", meaning growth of
functions. It is also called Landau's Symbol and belongs
to the Asymptotic Notations group.

Introduction to Complexity Theory 23