Analysis of Algorithms

Chapter 1
Introduction and elementary data structures
 What is an algorithm?
 Insertion sort
 Analyzing Algorithm
 Order of growth
 Asymptotic Notation
 Elementary Data Structure
 Heaps and Heapsort

Algorithm is an abstract computational procedure which takes some value or values as input and
produces a value or values as output. Another important criterion we will assume about algorithms is that they
terminate after a finite number of operations. There is another word for an algorithm which obeys all of the
above properties except termination, and that is computational procedure. A program is the expression of an
algorithm in a programming language.
An algorithm is thus a sequence of computational steps that transform the input into output. Algorithm is
a tool for solving a well specified computational problem. The algorithm describes a specific computational
procedure for achieving input-output relationship.
For example, we might need to sort a sequence of numbers into no decreasing order. How we formally
define the solving problem:
Input: A sequence of „n‟ numbers <a1, a2, ……………, an>.
Output: A permutation <a‟1, a‟2, ……………, a‟n> of the input sequence such that a‟1 ≤ a‟2 ≤ ……… ≤ a‟n
Eg, Input sequence <15, 26, 31, 7, 4, 26, 10>
Output sequence <4, 7, 10, 15, 26, 26, 31> such input sequence is called an instance of the sorting problem.
Instance of a problem consists of the i/p needed to compute a solution to the problem.
We have large number of good sorting algorithms. Which algorithm is best for a given application
depends on some factors:
 The number of items to be sorted.
 The extent to which the items are already somewhat sorted.
 Possible restrictions on the item values.
 The architecture of the computer.
 The kind of storage devices to be used.(main memory, disks, etc).
An algorithm said to be correct if, for every input instance, it halts with the correct output. We say that a correct
algorithm solves the given computational problem. An incorrect algorithm might not halt at all or some input
instances, or it might halt with an incorrect answer.

Insertion Sort

Insertion sort is an efficient algorithm for sorting a small number of elements.

The sorting problem

Input: A sequence of „n‟ numbers <a1, a2, ……………, an>.
Output: A permutation <a‟1, a‟2, ……………, a‟n> of the input sequence such that a‟1 ≤ a‟2 ≤ ……… ≤ a‟n

1 Prepared by: Mujeeb Rahman

The sequences are typically stored in arrays. We also refer to the numbers as keys. Along with each key may be
additional information, known as satellite data.

Example:

Analyzing algorithms

We want to predict the resources that the algorithm requires. Usually, running time. In order to predict
resource requirements, we need a computational model.

Random-access machine (RAM) model

 Instructions are executed one after another. No concurrent operations.
 It is too tedious to define each of the instructions and their associated time costs.
 Instead, we recognize that we‟ll use instructions commonly found in real computers:
 Arithmetic: add, subtract, multiply, divide, remainder, floor, ceiling). Also, shift left/shift
right (good for multiplying/dividing by 2k ).
 Data movement: load, store, copy.
 Control: conditional/unconditional branch, subroutine call and return.
Each of these instructions takes a constant amount of time.

How do we analyze an algorithm’s running time?

The time taken by an algorithm depends on the input. Sorting 1000 numbers takes longer than sorting 3
numbers. A given sorting algorithm may even take differing amounts of time on two inputs of the same size. For
example, we‟ll see that insertion sort takes less time to sort n elements when they are already sorted than when
they are in reverse sorted order. In general the time taken by an algorithm grows with the size of input. So it is
traditional to describe the running time of a program as a function of size of its input.
2 Prepared by: Mujeeb Rahman
Running time: On a particular input, it is the number of primitive operations (steps) executed.
 Want to define steps to be machine-independent.
 Figure that each line of pseudo code requires a constant amount of time.
 One line may take a different amount of time than another, but each execution of line i takes the same
amount of time ci .
 This is assuming that the line consists only of primitive operations.

Analysis of insertion sort

 Assume that the ith line takes time ci , which is a constant. (Since the third line is a comment, it takes no
time.)
 For j = 2, 3, . . . , n, let tj be the number of times that the while loop test is executed for that value of j .
 Note that when a for or while loop exits in the usual way-due to the test in the loop header-the test is
executed one time more than the loop body.

The running time of the algorithm is the sum of running times for each statement executed. A statement that
takes ci steps to execute and execute n times will contribute cin to the total running time. To compute T(n), the
running time of INSERTION SORT on an input of n values, we sum the product of cost and times column.

The running time of the algorithm is

∑ ( ) ( )

Let T (n) = running time of INSERTION-SORT.

The running time depends on the values of tj . These vary according to the input.

Best case: The array is already sorted.

 Always find that A[i ] ≤ key upon the first time the while loop test is run (when i = j − 1).
 All tj are 1.
 Running time is
T (n) = c1n + c2(n − 1) + c4(n − 1) + c5(n − 1) + c8(n − 1)
= (c1 + c2 + c4 + c5 + c8)n − (c2 + c4 + c5 + c8) .
 Can express T (n) as an +b for constants a and b (that depend on the statement costs ci )⇒ T (n) is a
linear function of n.
Worst case: The array is in reverse sorted order.
 Always find that A[i ] > key in while loop test.
 Have to compare key with all elements to the left of the j th position⇒compare with j − 1 elements.
 Since the while loop exits because i reaches 0, there‟s one additional test after the j − 1 tests ⇒ tj = j .
3 Prepared by: Mujeeb Rahman
Arithmetic series is equals n(n+1)/2

Worst-case and average-case analysis

We usually concentrate on finding the worst-case running time: the longest running time for any input of size n.
Reasons:

 The worst-case running time gives a guaranteed upper bound on the running time for any input.
 For some algorithms, the worst case occurs often. For example, when searching, the worst case often
occurs when the item being searched for is not present, and searches for absent items may be frequent.
 Why not analyze the average case? Because it‟s often about as bad as the worst case.

Order of growth
For ease analysis, look only at the leading term of the formula for running time. Drop lower-order terms
and ignore the constant coefficient in the leading term.
Example: For insertion sort, we already abstracted away the actual statement costs to conclude that the worst-
case running time is an2 + bn + c.
Drop lower-order terms ⇒ an2
Ignore constant coefficient ⇒ n2
4 Prepared by: Mujeeb Rahman
But we cannot say that the worst-case running time T (n) equals n2. It grows like n2. But it doesn‟t equal n2.
We say that the running time is Θ(n2) to capture the notion that the order of growth is n2.
We usually consider one algorithm to be more efficient than another if its worst-case running time has a smaller
order of growth.
Due to constant factors and lower order terms, an algorithm whose running time has a higher order of
growth might take less time for small inputs than an algorithm whose running time has a lower order of growth.
But for large enough inputs, a Θ(n2) algorithm for example will run more quickly in the worst case than a Θ(n3)
algorithm.

Growth of Function
The order of growth of the running time of an algorithm gives a simple characterization of the
algorithm‟s efficiency and also allows us to compare the relative performance of alternative algorithms. For large
enough inputs, the multiplicative constants and lower order terms of an exact running time are dominated by the
effect of input size itself.
When we look at input sizes large enough to make only the order of growth of the running time relevant,
we are studying the asymptotic efficiency of algorithms. We are concerned with how the running time of an
algorithm increases with the size of the input in the limit, as the size of the input increases without bound.
Usually an algorithm that is asymptotically more efficient will be the best choice for all but very small
input.

Asymptotic Notation
The notation we use to describe the asymptotic running time of an algorithm are defined in terms of
functions whose domains are the set of natural numbers, N= {0,1,2,…….}. Such notations are convenient for
describing the worst-case running time function T(n), which usually is defined only on integer input sizes.
The best case step count is the minimum number of steps that can be executed for the given parameters.
The worst case step count is the maximum number of steps that can be executed for the given parameters. The
average step count is the average number of steps that can be executed for the given parameters.
If we have two algorithms with complexity c1n2+c2n and c3n respectively, then we know that the one with
complexity c3n will be faster than the one with complexity c1n2+c2n for sufficient large value of n. For small
value of „n‟, either algorithm could be faster(depending on c1, c2 and c3).
If c1 = 1, c2 = 2 and c3= 100 then
c1n2+c2n ≤ c3n for n ≤ 98
c1n2+c2n > c3n for n > 98
If c1 = 1, c2 = 2 and c3= 1000 then
c1n2+c2n ≤ c3n for n ≤ 998
No matter the values of c1, c2 and c3 , there will be an „n‟ beyond which the algorithm with complexity c3n will
be faster than the one with complexity c1n2+c2n. This value of „n‟ will be called the break-even point.
Θ – notation(Theta)
We found that the worst-case running time of insertion sort is T (n) = Θ(n2). Let us define what this
notation means. For a given function g(n), we denote by Θ(g(n)) the set of functions.
Θ(g(n)) = { f (n) : there exist positive constants c1, c2, and n0 such that 0 ≤ c1g(n) ≤ f (n) ≤ c2g(n) for all n ≥ n0}

5 Prepared by: Mujeeb Rahman

 A function f (n) belongs to the set Θ(g(n)) if there exist positive constants c1 and c2 such that it can be
“sandwiched” between c1g(n) and c2g(n), for sufficiently large n. Because Θ(g(n)) is a set, we could write
“ f (n) ∈ Θ(g(n))” to indicate that f (n) is a member of Θ(g(n)). Instead, we will usually write “f (n) = Θ(g(n))” to
express the same notion.
Figure (a) gives an intuitive picture of functions f (n) and g(n), where we have that f (n) = Θ(g(n)). For
all values of n to the right of n0 , the value of f (n) lies at or above c1g(n) and at or below c2g(n). In other words,
for all n ≥ n0, the function f(n) is equal to g(n) to within a constant factor. We say that g(n) is an asymptotically
tight bound for f (n).
The lower-order terms of an asymptotically positive function can be ignored in determining
asymptotically tight bounds because they are insignificant for large n. A tiny fraction of the highest-order term is
enough to dominate the lower-order terms. Thus, setting c1 to a value that is slightly smaller than the coefficient
of the highest-order term and setting c2 to a value that is slightly larger permits the inequalities in the definition
of Θ-notation to be satisfied. The coefficient of the highest-order term can likewise be ignored, since it only
changes c1 and c2 by a constant factor equal to the coefficient.

O – notation(Big oh)
The Θ-notation asymptotically bounds a function from above and below. When we have only an
asymptotic upper bound, we use O-notation. For a given function g(n), we denote by O(g(n)) (pronounced
“big-oh of g of n” or sometimes just “oh of g of n”) the set of functions
O(g(n)) = { f (n) : there exist positive constants c and n0 such that 0 ≤ f (n) ≤ cg(n) for all n ≥ n0}

We use O-notation to give an upper bound on a function, to within a constant factor. Figure (b) shows
the intuition behind O-notation. For all values n to the right of n0, the value of the function f (n) is on or below
g(n). We write f (n) = O(g(n)) to indicate that a function f (n) is a member of the set O(g(n)). Note that
f (n) = Θ(g(n)) implies f (n) = O(g(n)), since Θ-notation is a stronger notion than O-notation. Written set-
theoretically, we have Θ(g(n)) ⊆ O(g(n)). Thus, our proof that any quadratic function an2 + bn + c, where a > 0,
is in Θ(n2) also shows that any quadratic function is in O(n2). What may be more surprising is that any linear
function an + b is in O(n2).
Using O-notation, we can often describe the running time of an algorithm merely by inspecting the
algorithm‟s overall structure. For example, the doubly nested loop structure of the insertion sort algorithm from
immediately yields an O(n2) upper bound on the worst-case running time: the cost of each iteration of the inner
loop is bounded from above by O(1) (constant), the indices I and j are both at most n, and the inner loop is
executed at most once for each of the n2 pairs of values for i and j.
Since O-notation describes an upper bound, when we use it to bound the worst-case running time of an
algorithm, we have a bound on the running time of the algorithm on every input. Thus, the O(n2) bound on
6 Prepared by: Mujeeb Rahman
worst-case running time of insertion sort also applies to its running time on every input. The Θ(n2) bound on the
worst-case running time of insertion sort, however, does not imply a Θ(n2) bound on the running time of
insertion sort on every input. For example, we saw that when the input is already sorted, insertion sort runs in
Θ(n) time.
Ω – notation(Omega)
Just as O-notation provides an asymptotic upper bound on a function, Ω-notation provides an
asymptotic lower bound. For a given function g(n), we denote by Ω(g(n)) (pronounced “big-omega of g of n”
or sometimes just “omega of g of n”) the set of functions
Ω(g(n)) = { f (n) : there exist positive constants c and n0 such that 0 ≤ cg(n) ≤ f (n) for all n ≥ n0}

The intuition behind Ω-notation is shown in Figure (c). For all values n to the right of n0, the value of f (n) is on
or above cg(n).
Theorem: For any two functions f (n) and g(n), we have f (n) = Θ(g(n)) if and only if f (n) = O(g(n)) and f (n) =
Ω(g(n)).
As an example of the application of this theorem, our proof that an2 + bn + c = _(n2) for any constants a, b, and
c, where a > 0, immediately implies that an2 + bn + c = Ω(n2) and an2 + bn + c = O(n2).
Since Ω-notation describes a lower bound, when we use it to bound the best-case running time of an
algorithm, by implication we also bound the running time of the algorithm on arbitrary inputs as well. For
example, the best-case running time of insertion sort is Ω(n), which implies that the running time of insertion
sort is Ω(n).
The running time of insertion sort therefore falls between Ω(n) and O(n2), since it falls anywhere
between a linear function of n and a quadratic function of n. Moreover, these bounds are asymptotically as tight
as possible: for instance, the running time of insertion sort is not Ω(n2), since there exists an input for which
insertion sort runs in Θ(n) time (e.g., when the input is already sorted). It is not contradictory, however, to say
that the worst-case running time of insertion sort is Ω(n2), since there exists an input that causes the algorithm to
take Ω(n2) time.
When we say that the running time of an algorithm is Ω(g(n)), we mean that no matter what particular
input of size n is chosen for each value of n, the running time on that input is at least a constant times g(n), for
sufficiently large n.
o-notation
The asymptotic upper bound provided by O-notation may or may not be asymptotically tight. The bound
2n = O(n2) is asymptotically tight, but the bound 2n = O(n2) is not. We use o-notation to denote an upper bound
2

that is not asymptotically tight. We formally define o(g(n)) (“little-oh of g of n”) as the set

o(g(n)) = { f (n) : for any positive constant c > 0, there exists a constant n0 > 0 such that 0 ≤ f (n) < cg(n) for all n ≥ n0}

For example, 2n = o(n2), but 2n2 ≠ o(n2)

ꞷ-notation
By analogy, ω-notation is to Ω-notation as o-notation is to O-notation. We use ω-notation to denote a
lower bound that is not asymptotically tight. One way to define it is by f(n) ∈ ω(g(n)) if and only if
g(n) ∈ o(f (n)) .
Formally, however, we define ω(g(n)) (“little-omega of g of n”) as the set

ω(g(n)) = { f (n) : for any positive constant c > 0, there exists a constant n0 > 0 such that 0 ≤ cg(n) < f(n) for all n ≥ n0}

7 Prepared by: Mujeeb Rahman

Elementary Data Structure
One of the basic techniques for improving algorithms is to structure the data in such a way that the
resulting operations can be efficiently carried out.

Stacks and Queues

One of the most common forms of data organization in computer programs is the ordered or linear list,
which is often written as A = (a1,a2,…..an). The ai‟s are referred to as atoms and they are chosen from some set.
The null or empty list has n = 0 elements. A stack is an ordered list in which all insertions and deletions are made
at one end, called the top. A queue is an ordered list in which all insertions take place at one end, the rear,
while all deletions take place at the other end, the.front.

The operations of a stack imply that if the elements A,B,C,D,E are inserted into a stack, in that order,
then the first element to be removed/deleted must be E. Equivalently we say that the last element to be inserted
into the stack will be the first to be removed. For this reason stacks are sometimes referred to as Last In First Out
(LIFO) lists. The operations of a queue require that the first element which is inserted into the queue will be
the first one to be removed. Thus queues are known as First In First Out (FIFO) lists.

Trees
A tree is a finite set of one or more nodes such that (i) there is a specially designated node called the
root; (ii) the remaining nodes are partitioned into n ≥ 0 disjoint sets Tl, ... , Tn where each of these sets is a tree.
Tl, ... , Tn are called the subtrees of the root.
There are many terms which are often used when referring to trees. Consider the tree in Figure. This tree
has 13 nodes, each data item of a node being a single letter for convenience. The root contains A, (though we
will usually say node A), and we will normally draw trees with their root at the top.
The number of subtrees of a node is called its degree. The degree of A is 3, of C is 1, and of F is 0.
Nodes that have degree zero are called leaf or terminal nodes. The set {K, L, F, G, M, I, J} is the set of leaf
nodes of Figure. The other nodes are referred to as nonterminals. The roots of the subtrees of a node, X, are the
children of X. Xis the parent of its children. Thus the children of D are H, I, J; the parent of D is A.

8 Prepared by: Mujeeb Rahman

 Children of the same parent are said to be siblings. For example H, I, and J are siblings. We can extend
this terminology if we need to so that we can ask for the grandparent of M which is D, etc. The degree of a tree is
the maximum degree of the nodes in the tree. The tree in Figure has degree 3. The ancestors of a node are all the
nodes along the path from the root to that node. The ancestors of M are A, D and H.
The level of a node is defined by initially letting the root be at level one. If a node is at level p, then its
children are at level p + 1. Figure shows the levels of all nodes in that tree. The height or depth of a tree is
defined to be the maximum level of any node in the tree.
A forest is a set of n ≥ 0 disjoint trees. The notion of a forest is very close to that of a tree because if we
remove the root of a tree we get a forest. For example, in Figure 2.5 if we remove A we get a forest with three
trees.

Heaps and Heap sort

The (binary) heap data structure is an array object that can be viewed as a nearly complete binary tree as
shown in Figure. Each node of the tree corresponds to an element of the array that stores the value in the node.
The tree is completely filled on all levels except possibly the lowest, which is filled from the left up to a point.

Max Heap: A heap is known as Max Heap, if for every node „n‟(other than the root) relation between node „n‟
and its parent is the “smaller relation” (n ≤ parent (n)). In this maximum value is stored in the root.

Min Heap: A heap is known as Min Heap, if for every node „n‟(other than the root) the relation between node
„n‟ and its parent is the “greater relation” (n ≥ parent (n)). In this minimum value is stored in the root.
A heap can be stored as an array A.

Maintaining the heap property

MAX-HEAPIFY is an important subroutine for manipulating max-heaps. Its inputs are an array A and an
index i into the array.

9 Prepared by: Mujeeb Rahman

Figure: The action of MAX-HEAPIFY(A, 2), where heap-size[A] = 10. (a) The initial configuration, with A[2] at node i = 2
violating the max-heap property since it is not larger than both children. The max-heap property is restored for node 2 in (b)
by exchanging A[2] with A[4], which destroys the max-heap property for node 4. The recursive call MAX-HEAPIFY(A, 4)
now has i = 4. After swapping A[4] with A[9], as shown in (c), node 4 is fixed up, and the recursive call MAX-HEAPIFY(A,
9) yields no further change to the data structure.

Building a heap
We can use the procedure MAX-HEAPIFY in a bottom-up manner to convert an array A[1 . . n], where
n = length[A], into a max-heap.

We can compute a simple upper bound on the running time of BUILD-MAXHEAP as follows. Each call to
MAX-HEAPIFY costs O(lg n) time, and there are O(n) such calls. Thus, the running time is O(n lg n).
Eg:

10 Prepared by: Mujeeb Rahman

Figure: The operation of BUILD-MAX-HEAP, showing the data structure before the call to MAX-HEAPIFY in line 3 of
BUILD-MAX-HEAP. (a) A 10-element input array A and the binary tree it represents. The figure shows that the loop index
i refers to node 5 before the call MAX-HEAPIFY(A, i ). (b) The data structure that results. The loop index i for the next
iteration refers to node 4. (c)–(e) Subsequent iterations of the for loop in BUILD-MAX-HEAP. Observe that whenever
MAX-HEAPIFY is called on a node, the two subtrees of that node are both max-heaps. (f) The max-heap after BUILD-
MAX-HEAP finishes.

The heapsort algorithm

The heapsort algorithm starts by using BUILD-MAX-HEAP to build a max-heap on the input array
A[1 . . n], where n = length[A]. Since the maximum element of the array is stored at the root A[1], it can be put
into its correct final position by exchanging it with A[n]. If we now “discard” node n from the heap (by
decrementing heap-size[A]), we observe that A[1 . . (n − 1)] can easily be made into a max-heap. The children of
the root remain max-heaps, but the new root element may violate the max-heap property. All that is needed to
restore the max-heap property, however, is one call to MAX-HEAPIFY(A, 1), which leaves a max-heap in A[1 . .
(n − 1)]. The heapsort algorithm then repeats this process for the maxheap of size n − 1 down to a heap of size 2.

Figure shows an example of the operation of heapsort after the max-heap is initially built. Each max-heap is
shown at the beginning of an iteration of the for loop of lines 2–5. The HEAPSORT procedure takes time
O(n lg n), since the call to BUILD-MAXHEAP takes time O(n) and each of the n − 1 calls to MAX-HEAPIFY
takes time O(lg n).
 The MAX-HEAPIFY procedure, which runs in O(lg n) time, is the key to maintaining the max-heap
property.
 The BUILD-MAX-HEAP procedure, which runs in linear time, produces a maxheap from an unordered
input array.
 The HEAPSORT procedure, which runs in O(n lg n) time, sorts an array in place.
11 Prepared by: Mujeeb Rahman
Figure: The operation of HEAPSORT. (a) The max-heap data structure just after it has been built by BUILD-MAX-HEAP.
(b)–(j) The max-heap just after each call of MAX-HEAPIFY in line 5. The value of i at that time is shown. Only lightly
shaded nodes remain in the heap. (k) The resulting sorted array A.

***

12 Prepared by: Mujeeb Rahman