CSE-221

Algorithms
Introduction to Algorithms
 Algorithm Definition
• A finite set of statements that guarantees an optimal solution in
finite interval of time

• Algorithmic thinking and problem solving skill are vital in making

efficient solutions.

• The English word "ALGORITHM" derives from the Latin word AL-
AL-KHWARIZMI’S name. He developed the concept of an
algorithm in Mathematics, and thus sometimes being called the
“Grandfather of Computer Science".

2
 Glance of Algorithm
• An algorithm is a finite set of instructions or logic, written in order,
to accomplish a certain predefined task.
• Algorithm is not the complete code or program
• Can be expressed either as an informal high level description as
pseudocode or using a flowchart.

3
 Algorithm Specifications
• Input - Every Algorithm must take zero or more number of input values
from external.
• Output - Every Algorithm must produce an output as result.
• Definiteness - Every statement/instruction in an algorithm must be clear
and unambiguous (only one interpretation)
• Finiteness - For all different cases, the algorithm must produce result
within a finite number of steps.
• Effectiveness - Every Instruction must be basic enough to be carried out
and it also must be feasible.

4
 Good Algorithms?
• Run in less time

• Consume less memory

But computational resources (time complexity) usually important

5
 Analyzing Algorithms
• Predict the amount of resources required:
• memory: how much space is needed?
• computational time: how fast the algorithm runs?

• FACT: running time grows with the size of the input

• Input size (number of elements in the input)
– Size of an array, polynomial degree, # of elements in a matrix, # of bits in the binary
representation of the input, vertices and edges in a graph

Def: Running time = the number of primitive operations (steps)

executed before termination
– Arithmetic operations (+, -, *), data movement, control, decision making (if, while),
comparison
6
 Algorithm Analysis: Example
• Alg.: MIN (a[1], …, a[n])
m ← a[1];
for i ← 2 to n
if a[i] < m
then m ← a[i];
• Running time:
– the number of primitive operations (steps) executed before termination
T(n) =1 [first step] + (n) [for loop] + (n-1) [if condition] + (n-1) [the assignment in then] =
3n - 1
• Order (rate) of growth:
– The leading term of the formula
– Expresses the asymptotic behavior of the algorithm

7
 Typical Running Time Functions
• 1 (constant running time):
– Instructions are executed once or a few times

• logN (logarithmic)
– A big problem is solved by cutting the original problem in smaller sizes, by a constant
fraction at each step
• N (linear)
– A small amount of processing is done on each input element

• N logN
– A problem is solved by dividing it into smaller problems, solving them independently and
combining the solution

8
 Typical Running Time Functions
• N2 (quadratic)
– Typical for algorithms that process all pairs of data items (double nested loops)

• N3 (cubic)
– Processing of triples of data (triple nested loops)

• NK (polynomial)
• 2N (exponential)
– Few exponential algorithms are appropriate for practical use

9
Growth of Functions

10
Complexity Graphs

√𝑛

log(n)

11
Complexity Graphs

n log(n)

√𝑛
log(n)

12
Complexity Graphs

n10 n3

n2
n log(n)

13
Complexity Graphs (log scale)

3n
nn
n20

2n

n10

1.1n

14
 Algorithm Complexity
• Worst Case Complexity:
– the function defined by the maximum number of steps taken on any
instance of size n
• Best Case Complexity:
– the function defined by the minimum number of steps taken on any
instance of size n
• Average Case Complexity:
– the function defined by the average number of steps taken on any
instance of size n

15
Best, Worst, and Average Case Complexity

Number Worst Case

of steps
Complexity

Average Case
Complexity

Best Case
Complexity

N
(input size)

16
 Doing the Analysis
• It’s hard to estimate the running time exactly
– Best case depends on the input
– Average case is difficult to compute
– So we usually focus on worst case analysis
• Easier to compute
• Usually close to the actual running time
• Strategy: find a function (an equation) that, for large n, is an upper bound to the
actual function (actual number of steps, memory usage, etc.)

Upper bound
Actual function
Lower bound

17
 Motivation for Asymptotic Analysis
• An exact computation of worst-case running time can be difficult
– Function may have many terms:
• 4n2 - 3n log n + 17.5 n - 43 n⅔ + 75
• An exact computation of worst-case running time is unnecessary

18
 Classifying functions by their
Asymptotic Growth Rates
• asymptotic growth rate, asymptotic order, or
order of functions
– Comparing and classifying functions that ignores
• constant factors and
• small inputs.
• The Sets big oh O(g), big theta (g), big omega (g)

19
 Classifying functions by their
Asymptotic Growth Rates
1. O(g(n)), Big-Oh of g of n, the Asymptotic Upper Bound
2. Q(g(n)), Theta of g of n, the Asymptotic Tight Bound
3. W(g(n)), Omega of g of n, the Asymptotic Lower Bound

20
 Big-O

𝑓 ( 𝑛)= 𝑂 ( 𝑔 ( 𝑛) ) :there exist positive constants 𝑐 and 𝑛0 such that

• What does it mean?
– If f(n) = O(n2), then:
• f(n) can be larger than n2 sometimes, but…
• We can choose some constant c and some value n0 such that for every value of n
larger than n0 : f(n) < cn2
• That is, for values larger than n0, f(n) is never more than a constant multiplier greater
than n2
• Or, in other words, f(n) does not grow more than a constant factor faster than n2.

21
Visualization of O(g(n))
cg(n)

f(n)

n0

22
 Examples

– 2n2 = O(n3): 2n2 ≤ cn3  2 ≤ cn  c = 1 and n0= 2

– n2 = O(n2): n2 ≤ cn2  c ≥ 1  c = 1 and n0= 1

– 1000n2+1000n = O(n2):

1000n2+1000n ≤ cn2 ≤ cn2+ 1000n  c=1001 and n0 = 1

– n = O(n2): n ≤ cn2  cn ≥ 1  c = 1 and n0= 1

23
Big-O

24
 More Big-O
• Prove that: 20 𝑛 + 2𝑛+ 5=𝑂 ( 𝑛 )
2 2

• Let c = 21 and n0 = 4
• 21n2 > 20n2 + 2n + 5 for all n > 4
n2 > 2n + 5 for all n > 4
TRUE

25
 Tight bounds
• We generally want the tightest bound we can find.
• While it is true that n2 + 7n is in O(n3), it is more interesting to say
that it is in O(n2)

26
 Big Omega – Notation
• () – A lower bound

𝑓 ( 𝑛)= Ω (𝑔 ( 𝑛) ) :there exist positive constants 𝑐 and 𝑛0 such that

– n2 = (n)
– Let c = 1, n0 = 2
– For all n  2, n2 > 1  n

27
Visualization of (g(n))
f(n)

cg(n)

n0

28
 -notation

• Big-O is not a tight upper bound. In other words n = O(n2)

•  provides a tight bound

𝑓 ( 𝑛)=Θ ( 𝑔 ( 𝑛) ) :there exist positive constants 𝑐1 , 𝑐2 , and 𝑛0 such that

• In other words,

𝑓 ( 𝑛 )=Θ ( 𝑔 ( 𝑛 ) ) ⇒ 𝑓 ( 𝑛 ) =𝑂 ( 𝑔 ( 𝑛 ) ) AND 𝑓 ( 𝑛 ) =Ω ( 𝑔 (𝑛 ) )

29
Visualization of (g(n))
c2g(n)

f(n)

c1g(n)

n0

30
 A Few More Examples
• n = O(n2) ≠ (n2)
• 200n2 = O(n2) = (n2)
• n2.5 ≠ O(n2) ≠ (n2)

31
 Example 2
• Prove that: 20 𝑛 + 7 𝑛+1000=Θ ( 𝑛 )
3 3

• Let c = 21 and n0 = 10
• 21n3 > 20n3 + 7n + 1000 for all n > 10
n3 > 7n + 5 for all n > 10
TRUE, but we also need…
• Let c = 20 and n0 = 1
• 20n3 < 20n3 + 7n + 1000 for all n  1
TRUE

32
 Example 3
• Show that 𝑛 2
2 + 𝑛 =O ( 2 )
𝑛

• Let c = 2 and n0 = 5

𝑛𝑛 2
2×2 >2 +𝑛 

33
 Asymptotic Notations - Examples
•  notation
– n2/2 – n/2 = (n2)
– (6n3 + 1)lgn/(n + 1)
= (n2lgn)
– n vs. n2 n ≠ (n2)
•  notation • O notation
– n3 vs. n2 n3 = (n2) – 2n2 vs. n3 2n2 = O(n3)
– n vs. logn n = (logn) – n2 vs. n2 n2 = O(n2)
– n vs. n2 n  (n2) – n3 vs. nlognn3  O(nlgn)

34
 Asymptotic Notations - Examples
• For each of the following pairs of functions, either f(n) is
O(g(n)), f(n) is Ω(g(n)), or f(n) = Θ(g(n)). Determine
which relationship is correct.
– f(n) = log n2; g(n) = log n + 5 f(n) =  (g(n))
– f(n) = n; g(n) = log n2 f(n) = (g(n))
– f(n) = log log n; g(n) = log n f(n) = O(g(n))
– f(n) = n; g(n) = log2 n f(n) = (g(n))
– f(n) = n log n + n; g(n) = log nf(n) = (g(n))
– f(n) = 10; g(n) = log 10 f(n) = (g(n))
– f(n) = 2n; g(n) = 10n2 f(n) = (g(n))
– f(n) = 2n; g(n) = 3n f(n) = O(g(n))
35
 Simplifying Assumptions
1. If f(n) = O(g(n)) and g(n) = O(h(n)), then f(n) = O(h(n))
2. If f(n) = O(kg(n)) for any k > 0, then f(n) = O(g(n))
3. If f1(n) = O(g1(n)) and f2(n) = O(g2(n)),
then f1(n) + f2(n) = O(max (g1(n), g2(n)))
4. If f1(n) = O(g1(n)) and f2(n) = O(g2(n)),
then f1(n) * f2(n) = O(g1(n) * g2(n))

36
 Some Simplified Rules
• O(1) = c , where c is a constant
• O(n) = c*n = cn , where c is constant and n is variable
• c1*O(1) = c1*c = c2 = O(1) , where c,c1,c2 are constants
– O(1) + O(1) + O(1) = 3*O(1) = O(1)
– 5*O(1) = O(1)
• n*O(1) = n*c = cn = O(n) , where c is constant and n is variable
• O(m) + O(n) ≠ O(m+n)
• O(m) * O(n) = c1mc2n = (c1*c2)(mn) = (c2)(mn) = O(mn)
• O(m)*O(n)*O(p)*O(q) = O(m(n(p(q)))) = O(mnpq)
– Example nested for loops
• O(an2 + bn + c) = O(n2) where a, b , c are constants
37
 Example #1: carry n books
from one bookshelf to another one
• How many operations?
• n pick-ups, n forward moves, n drops and n reverse moves  4 n
operations
• 4n operations = c. n = O(c. n) = O(n)
• Similarly, any program that reads n inputs from the user will have
minimum time complexity O(n).

38
 Example #2: Locating Roll-Number record in
Attendance Sheet

What is the time complexity of search?

• Binary Search algorithm at work
– O(log n)
• Sequential search?
– O(n)

39
 Example #3: Teacher of CSE 221 gives gifts
to first 10 students
• There are n students in the queue.
• Teacher brings one gift at a time.
• Time complexity = O(c. 10) = O(1)
• Teacher will take exactly same time irrespective of the line length.

40
 Loops with Break
for (j = 0; j < n; ++j) {
// 3 atomics
if (condition) break;
}

• Upper bound = O(4n) = O(n)

• Lower bound = Ω(4) = Ω(1)
• Complexity = O(n)
Ques: Why don’t we have a (…) notation here?

41
 Sequential Search
• Given an unsorted vector/list a[ ], find the location of element X.

for (i = 0; i < n; i++) {

if (a[i] == X) return true;
}
return false;

• Input size: n = array size()

• Complexity = O(n)

42
 If-then-else Statement
if(condition)
i = 0;
else
for ( j = 0; j < n; j++)
a[j] = j;

• Complexity = ??
= O(1) + max ( O(1), O(N))
= O(1) + O(N)
= O(N)

43
 Consecutive Statements

for (j = 0; j < n; ++j) {

// 3 atomics
}
for (j = 0; j < n; ++j) {
// 5 atomics
}

• Add the complexity of consecutive statements

• Complexity = O(3n + 5n) = O(n)

44
 Nested Loop Statements
• Analyze such statements inside out

for (j = 0; j < n; ++j) {

// 2 atomics
for (k = 0; k < n; ++k) {
// 3 atomics
}
}

• Complexity = O((2 + 3n)n) = O(n2)

45
 Example
• Code:
• a = b;

• Complexity:

46
 Example
• Code:
• sum = 0;
• for (i=1; i <=n; i++)
• sum += n;

• Complexity:

47
 Example
• Code:
• sum = 0;
• for (j=1; j<=n; j++)
• for (i=1; i<=j; i++)
• sum++;
• for (k=0; k<n; k++)
• A[k] = k;
• Complexity:

48
 Example
• Code:
• sum1 = 0;
• for (i=1; i<=n; i++)
• for (j=1; j<=n; j++)
• sum1++;
• Complexity:

49
 Example
• Code:
• sum2 = 0;
• for (i=1; i<=n; i++)
• for (j=1; j<=i; j++)
• sum2++;
• Complexity:

50
 Example
• Code:
• sum1 = 0;
• for (k=1; k<=n; k*=2)
• for (j=1; j<=n; j++)
• sum1++;
• Complexity:

51
 Example
• Code:
• sum2 = 0;
• for (k=1; k<=n; k*=2)
• for (j=1; j<=k; j++)
• sum2++;
• Complexity:

52
 Recursion
long factorial( int n )
{ In terms of big-Oh:
if( n <= 1 ) t(1) = 1
return 1; t(n) = 1 + t(n-1) = 1 + 1 + t(n-2)
else = ... k + t(n-k)
return n*factorial(n- 1); Choose k = n-1
}
t(n) = n-1 + t(1) = n-1 + 1 = O(n)

Consider the following time complexity:

t(0) = 1
t(n) = 1 + 2t(n-1) = 1 + 2(1 + 2t(n-2)) = 1 + 2 + 4t(n-2)
= 1 + 2 + 4(1 + 2t(n-3)) = 1 + 2 + 4 + 8t(n-3)
= 1 + 2 + ... + 2k-1 + 2kt(n-k)
Choose k = n
t(n) - 1 + 2 + ... 2n-1 + 2n = 2n+1 - 1

53
 Binary Search
• Given a sorted vector/list a[ ], find the location of element X

unsigned int binary_search(vector<int> a, int X)

{
unsigned int low = 0, high = a.size()-1;

while (low <= high) {

int mid = (low + high) / 2;
if (a[mid] < X)
low = mid + 1;
else if( a[mid] > X )
high = mid - 1;
else
return mid;
}
return NOT_FOUND;
}

• Input size: n = array size()

• Complexity = O( k iterations x (1 comparison+1 assignment) per loop)
= O(log(n))

54
 Summary
• Time complexity is a measure of algorithm efficiency
• Efficient algorithm plays the major role in determining the
running time.
Q: Is it possible to determine running time based on
algorithm’s time complexity alone?
• Minor tweaks in the code can cut down the running time by
a factor too.
• Other items like CPU speed, memory speed, device I/O
speed can help as well.
• For certain problems, it is possible to allocate additional
space & improve time complexity.
55
 Summary
• Time complexity is a measure of algorithm efficiency
• Efficient algorithm plays the major role in determining the running time.
Q: Is it possible to determine running time based on algorithm’s time
complexity alone?
• Minor tweaks in the code can cut down the running time by a factor too.
• Other items like CPU speed, memory speed, device I/O speed can help
as well.
• For certain problems, it is possible to allocate additional space &
improve time complexity.

56