Design and Analysis of Algorithms

Lecture 4:
BS Course Semester-5

Dr. Muhammad Ilyas Fakhir

Computer Science Department

GC University Lahore

February 20, 2023

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 1 / 43
 Outlines

1 Analyzing Algorithms

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 2 / 43
 Analyzing Algorithms

Analyzing Algorithms
Classifying algorithms: relative amount of time they require:
How fast growth in time required as SIZE of the problem increases?
In arrays: SIZE of the problem is the number of elements in the
array?
If size measured by variable n, then time required as a function of n;
T(n).

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 3 / 43
 Growth of T(n)

Useable and Unusable Algorithms

If T(n) grows RAPIDLY → algorithm UNUSABLE for large n.
If T(n) grows SLOWLY → algorithm USABLE even for large n.

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 4 / 43
 Θ(n)

How Time Complexity Increases

An algorithm
1 Θ(n2 ): if the time it takes quadruples ⇒ n doubles;
2 Θ(n): time it takes doubles⇒ when n doubles.
3 Θ(logn): time increases by a constant independent of n ⇒ when n
doubles.
4 Θ(1): time does not increase at all ⇒ when n increases.

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 5 / 43
 Algorithm: Problem Size

Algorithm: Problem Size

In general, an ALGORITHM is Θ(T (n)) if the time requires on problem of
size n grows proportionally to T(n) as n increases.

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 6 / 43
 Analysis of Algorithm and Recurrence Relation

Recurrence Relation
The analysis of an algorithm is often accomplished by finding and solving a
recurrence relation that describes the time required by the algorithm.
The most commonly occurring families of recurrences:
LINEAR recurrences
DIVIDE-and-CONQUER recurrences

Note
Before discussing these recurrences first we will discuss the growth of
functions.

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 7 / 43
 Asymptotic Notation

Asymptotic Notation
Asymptotic running time of an algorithm are defined in terms of
functions whose domains are set of natural numbers N = {0, 1, 2, ...}
convenient for describing the worst-case running-time function T(n).
USUALLY defined only on integer input sizes.

Basic Asymptotic Notations

Θ-notation
O-notation
Ω-notation
o-notation
ω-notation

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 8 / 43
 Θ-notation

Definition
For g(n) be a given function, we denote Θ(g(n)) the set of function

Θ(g(n)) = { f (n)|∃ positive constants c1 , c2 and n0 such that

0 ≤ c1 g(n) ≤ f (n) ≤ c2 g(n)∀n ≥ n0 }

A function f (n) ∈ Θ(g(n))

If ∃ positive constants c1 , and c2
such that f(n) is sandwiched between c1 g(n) and c2 g(n) for sufficiently
large n.

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 9 / 43
 Contd..

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 10 / 43
 Description

For n ≥ n0 the function f (n) ≥ c1 g(n), and f (n) ≤ c2 g(n).

Intuitively, for n ≥ n0 then function f(n) equals g(n) to within a
constant factor.
We say g(n) is an asymptotically tight bound for f(n).
The definition of Θ(g(n)) requires for ∀f (n) ∈ Θ(g(n)) be
asymptotically nonnegative. i.e. f (n) > 0 when n is sufficiently large.
Consequently, g(n) itself must be asymptotically nonnegative, or else
Θ(g(n)) = φ
Every function used within Θ-notation is asymptotically nonnegative.

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 11 / 43
 Example
Let f (n) = 5n2 + 3n and g(n) = n2 . We want to prove f (n) = Θ(g(n)).
That means we need three constants c1 , c2 & n0 such that
c1 n2 ≤ 5n2 + 3n ≤ c2 n2
Simplification results,
c1 ≤ 5 + 3/n ≤ c2
If we choose n0 = 1, then the expression in the middle can not be smaller
than 5, means c1 ≤ 5. Let choose c1 = 4. Similarly, the term can not be
greater than 8 so c2 ≥ 8. Let choose c2 = 9. The expression now becomes
4n2 ≤ 5n2 + 3n ≤ 9n2 for all n ≥ 1.
Hence proves 5n2 + 3n ∈ Θ(n).

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 12 / 43
 The graphs of 4n2 , 5n2 + 3n and 9n2 is shown below.

Figure: Θ notation

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 13 / 43
 Example
The same rate of increase for f (n) = n + 5n0.5 and g(n) = n because
n ≤ n + 5n0.5 ≤ 6n for n > 1.

Example
√
2n − 2 n ∈ Θ(n) , We need here c1 , c2 , and n0 , g(n) = n
√ √
Suppose c1 = 1 ⇒ n ≤ 2n − 2 n ⇒ 2 n ≤ n
√ √ √
2 ≤ n/ n = n ⇒ 2 ≤ n ⇒ n0 = 4
√
Suppose c2 = 2 ⇒ 2n − 2 n ≤ 2n
√
Hence, 2n − 2 n ∈ Θ(n)

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 14 / 43
 Example

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 15 / 43
 Example

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 16 / 43
 Example

Show that: 12 n2 − 3n = Θ(n2 )

Proof: c1 n2 ≤ 21 n2 − 3n ≤ c2 n2
for all n ≥ n0 . Dividing by n2 :
c1 ≤ 21 − n3 ≤ c2
The right-hand inequality can be made to hold for any value of n ≥ 1 by
choosing c2 ≥ 1/2. Likewise, the left-hand inequality can be made to hold for
any value of n ≥ 7 by choosing c1 ≤ 1/14. Thus, by choosing
c1 = 1/14, c2 = 1/2, and n0 = 7, we can verify that:
1 2 2
2 n − 3n = Θ(n ).

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 17 / 43
 Ω-Notation

Ω-Notation provides an asymptotic lower bound.

Definition
For a given function g(n) we denote by Ω(g(n)) the set of functions

Ω(g(n)) = {f (n)| there exit positive constants c and n0 such that

0 ≤ cg(n) ≤ f (n) for all n ≥ n0 }

Intuitively show in the diagram.

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 18 / 43
 Contd..

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 19 / 43
 Example
Let f (n) = 10n2 + 14n + 10 and g(n) = n2 . We want to prove
f (n) = Ω(g(n)). That means we need two constants c & n0 such that the
following relation holds for all n ≥ n0 :
cn2 ≤ 10n2 + 14n + 10
Simplification results,
c ≤ 10 + 14/n + 10/n2
If we choose n0 = 1, then the minimum value of the right hand side
expression can get is 10. Let choose c = 9. Therefore expression now
becomes
10n2 + 14n + 10 = Ω(n2 ).

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 20 / 43
 The graphs for functions 10n2 + 14n + 10 and 9n2 is shown
in the figure below.

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 21 / 43
 Example

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 22 / 43
 Example

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 23 / 43
 O-Notation

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 24 / 43
 Example
Let f (n) = 50n3 + 10n and g(n) = n3 . We want to prove f (n) = O(g(n)).
To prove this, we need two constants c & n0 such that the following relation
holds for all n ≥ n0 :
50n3 + 10n ≤ cn3
Simplification results,
50 + 10/n2 ≤ c
If we choose n0 = 1, then the maximum value of the left hand side expression
can get is 60. Let choose c = 61. Therefore expression now becomes
50n3 + 10n = O(n3 ).

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 25 / 43
 The graphs for functions 50n3 + 10n and 61n3 is shown in
the figure below.

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 26 / 43
 O-Notation and o-Notation
O-notation may or may not be asymptotically tight.
The bound 2n2 = O(n2 ) is asymptotically tight but 2n = O(n2 ) is not.
o-notation to denote an upper bound that is not asymptotically tight.

Definition
O(g(n)) as a set:
O(g(n)) = {f (n)| for any positive constant c > 0, there exits a constant
n0 > 0 such that 0 ≤ f (n) ≤ cg(n) for all n ≥ n0 }

Definition
o(g(n)) as a set:
o(g(n)) = {f (n)| for any positive constant c > 0, there exits a constant
n0 > 0 such that 0 ≤ f (n) < cg(n) for all n ≥ n0 }

Example
2n = o(n2 ), but 2n2 6= o(n2 )
Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 27 / 43
 Comparison

The definitions of O-notation and o-notation are similar.

The main difference is that f(n) = o(g(n)), the bound 0 ≤ f (n) ≤ cg(n)
holds for some constant c > 0, the bound 0 ≤ f (n) < cg(n) holds for all
constants c > 0.
Intuitively, in the o-notation, the function becomes insignificant relative
to g(n) as n approaches infinity;
f (n)
limn→∞ g(n) =0

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 28 / 43
 Example

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 29 / 43
 ω-Notation

By analogy, ω-notation is to Ω-notation

as o-notation is to O-notation.
ω-notation to denote a lower bound that is not asymptotically tight.
one way to define o-notation

f (n) ∈ ω(g(n)) iff g(n) ∈ o(f (n)).

Definition
ω(g(n)) = {f (n)| for any positive constant c > 0, there exists a constant
n0 > 0 such that 0 ≤ cg(n) < f (n) for all n ≥ n0 }

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 30 / 43
 Using Limits for Comparing Orders of Growth

In order to determine the relationship between f (n) and g(n), it is often

usefully to examine
f (n)
lim =L
n→∞ g(n)

The possible outcomes:

L = 0 : f (n) = o(g(n))
L = ∞ : f (n) = ω(g(n))
L 6= 0 is finite: f (n) = Θ(g(n))

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 31 / 43
 Contd...

Example
n2 /2 = ω(n), but n2 /2 6= ω(n2 )

Here’s a table that summarizes the key restrictions in these four definitions:

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 32 / 43
 Example

Example
Determine if 2n3 + n2 = Θ(n2 ). Here we need c1 , c2 and n0 , g(n) = n2
c1 n2 ≤ 2n3 + n2 ≤ c2 n2 for n ≥ n0
Dividing by n2 gives
c1 ≤ 2n + 1 ≤ c2
Lower Bound: c1 ≤ 2n + 1, here it is clear that c1 = 1, n0 = 1
Therefore: 2n3 + n2 = Ω(n2 ).
Upper Bound: Unfortunately we cannot find any constant c2 that satisfies for
all n. Therefore
2n3 + n2 6= O(n2 ).
Hence by the theorem
2n3 + n2 6= Θ(n2 ).

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 33 / 43
 Application Examples

Example
Let f (n) = 7n + 8 and g(n) = n. Is f (n) ∈ O(g(n))?
Sol: For 7n + 8 ∈ O(n), we have to find c and n0 such that
7n + 8 ≤ cn, ∀n ≥ n0 . By inspection, it’s clear that c must be larger than 7.
Let c = 8.
Now we need a suitable n0 . In this case, f (8) = 8g(8). Because the definition
of O() requires that f (n) ≤ cg(n), we can select n0 = 8, or any integer above
8 – they will all work.
We have identified values for the constants c and n0 such that 7n + 8 ≤ cn for
every n ≥ n0 , so we can say that 7n + 8 is O(n).

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 34 / 43
 Example

Example
Is 7n + 8 ∈ o(n2 )?
Sol: To prove this, we need calculus. For g(n) to be a loose upper-bound on
f(n), it must be the case that limn→∞ fg(n)
(n)
= 0. Here, limn→∞ 2n7
= 0 (by
0 2
l H ôpital). Thus, 7n + 8 ∈ o(n ).

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 35 / 43
 THEOREM

Theorem
For any two functions f(n) and g(n), we have f (n) = Θ(g(n)) iff f(n) =
O(g(n)) and f (n) = Ω(g(n)).
Example: an2 + bn + c = Θ(n2 )(a > 0)
immediately implies that an2 + bn + c = Θ(n2 ) = Ω(n2 ) and
an2 + bn + c = Θ(n2 ) = O(n2 )

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 36 / 43
 Transitivity

Transitivity
f (n) = Θ(g(n)) and g(n) = Θ(h(n)) ⇒ f (n) = Θ(h(n)),
f (n) = O(g(n)) and g(n) = O(h(n)) ⇒ f (n) = O(h(n)),
f (n) = Ω(g(n)) and g(n) = Ω(h(n)) ⇒ f (n) = Ω(h(n)),
f (n) = o(g(n)) and g(n) = o(h(n)) ⇒ f (n) = o(h(n)),
f (n) = ω(g(n)) and g(n) = ω(h(n)) ⇒ f (n) = ω(h(n)),

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 37 / 43
 Reflexivity

Reflexivity
f (n) = Θ(f (n)),
f (n) = O(f (n)),
f (n) = Ω(f (n)),

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 38 / 43
 Symmetry

Symmetry
f (n) = Θ(g(n)) iff g(n) = Θ(f (n)),

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 39 / 43
 Transpose Symmetry

Transpose Symmetry
f (n) = O(g(n)) iff g(n) = Ω(f (n)),
f (n) = o(g(n)) iff g(n) = ω(f (n)),

f(n) is asymptotically smaller than g(n) if f (n) = o(g(n)), and

f(n) is asymptotically larger than g(n) if f (n) = ω(g(n))

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 40 / 43
 Contd..

For any two real numbers a and b.

f (n) = O(g(n)) ≈ a ≤ b
f (n) = Ω(g(n)) ≈ a ≥ b
f (n) = Θ(g(n)) ≈ a = b
f (n) = o(g(n)) ≈ a < b
f (n) = ω(g(n)) ≈ a > b

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 41 / 43
 Common Algorithm Classes

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 42 / 43
 Common Algorithm Classes

Dr. Muhammad Ilyas Fakhir (CS Department) Design and Analysis of Algorithms February 20, 2023 43 / 43