CS424-Design and Analysis of

Algorithms

SORTING ALGORITHMS REVISITED

Week #6
Saima Gul
Memory Usage of The Sorting
Algorithms
 An inplace sorting algorithm requires a constant amount
of additional memory.
 We have seen before insertion and merge sort
algorithms as solutions to the sorting problem.
 Insertion Sort: (n 2 )
 Merge Sort: (n lg n)
 Insertion sort is inplace (the numbers are shifted within
the same array), however merge sort needs Θ(n)
additional space.

2
Quick Sort

 It is an inplace algorithm.

2
 Worst case :  ( n )

 Average case: (n lg n)

(with small constant factors)

 It uses the divide and conquer approach.

3
Basic Idea of Quick Sort

 Given an array A, pick an element z in the array as the

pivot element:
A: z
 Move all the elements smaller than or equal to z to the
left of z.
 Move all the elements greater than z to the right of z.
≤z >z

A’: z
A1 A2
 If we sort A1 and A2, then we will be sorting the entire
array.
4
Pseudo code for Quick Sort
Quick-Sort(A,p,r) {
if (p<r) {
q=Partition(A,p,r); // index of the pivot is returned
Quick-Sort(A,p,q-1);
Quick-Sort(A,q+1,r);
}
}

5
 Pseudo code for Partition
Partition(A,p,r) {
z=A[r]; // picks the last element as the pivot (could be smthg
else)
i=p-1; // the index of the last number seen which is ≤z
for (j=p; j <r; j++) { // go over every element (except pivot itself)
if (A[j] ≤z) { // whenever we see a number ≤z
i++; // update i and
swap(A[i],A[j]); // move the number to the front
}
}
swap(A[i+1],A[r]); // move to the pivot right next to the last seen
return(i+1); // smaller number, and return the index of the
} // pivot cost : (n)
Consider application on the array: 8 9 3 7 6

6
Analysis of Quick Sort: best case

 The best case of Quick Sort is when the partitioning

produces balanced partitions at each recursion

Quick-Sort(A,p,r) { T (n) 2T (n / 2)  (n)

if (p<r) {
T (n) (n lg n)
q=Partition(A,p,r);
Quick-Sort(A,p,q-1);
Quick-Sort(A,q+1,r);
}
}

7
Analysis of Quick Sort: worst case

 The worst case of Quick Sort occurs when the

partitioning produces an empty partition at each
recursion

Quick-Sort(A,p,r) {
T (n) T (n  1)  (n)
if (p<r) {
q=Partition(A,p,r); T (n) (n 2 )
Quick-Sort(A,p,q-1);
Quick-Sort(A,q+1,r);
}
}

8
Analysis of Quick Sort: average case

 The average case for Quick Sort (when the worst case
does not occur) luckily has the same behavior as the
best case.

 Even if the partitioning does not produce equal partitions,

as long as the size of the partitions are proportional, we
will have the best case behavior.

T (n) T (n / 1000)  T (999n / 1000)  (n)

T (n) (n lg n)

9
LOWER BOUNDS FOR THE SORTING PROBLEM

10
Lower bounds for the sorting
problem
 A sorting algorithm is called a comparison sorting
algorithm if it compares the numbers to sort them.
 We have seen three sorting algorithms so far, and all of
them are comparison based sorting algorithms.
 When we consider the worst case behaviors, Merge Sort
is the most efficient one, whose worst case running time
is (n lg n)
 As of today, merge sort is the most efficient comparison
based sorting algorithm known
(invented by John von Neumann in 1945)

11
Lower bounds for the sorting
problem
 Can we hope that, someday a brilliant person will be
born, and she/he will find a more efficient comparison
based sorting algorithm?
 Or, is the merge sort an algorithm that can never be
beaten?
 Note that, not being able to find something, does not
prove that what we are looking for does not exist.
 So, may be someone wise enough will come up and
show the world how to sort faster.
 Or, may be not?
12
Lower bounds for the sorting
problem
 In order to get out of such dilemma, we need to show
lower bounds for the problems.
 Note that, we are not talking about lower bound of a
particular algorithm given as a solution to a problem.
 When we find a lower bound for a problem, it applies to
any algorithm (or a class of algorithms) for that problem.
 Hence if we can show that, n lg n is a lower bound for
the sorting problem, then we will be showing that no one
can ever find a more efficient algorithm, no matter how
smart she/he is.

13
Lower bounds for the sorting
problem
 Note that, in a comparison based sorting algorithm,
every number has to be compared to at least one other
number.
 Otherwise, the correct place for that number cannot be
found.
 This argument also applies to sorting algorithms that are
not comparison based. You need to read each number
at least once…
 Hence, any sorting algorithm is (n)

14
Lower bounds for the sorting
problem
 Is (n) a tight lower bound for comparison based
sorting algorithms?

 Or, can we find a comparison based sorting algorithm

between n and n lg n ?

15
Lower bounds for the sorting
problem
 Using decision trees to visualize sorting: (assume that we
have three numbers to sort)
a1 : a2
 
a2 : a3 a1 : a3
   
a1 a2 a3 a1 : a3 a2 a1 a3 a2 : a3
   
a1 a3 a2 a3 a1 a2 a2 a3 a1 a3 a2 a1
 The leaves are the permutations of the input (n! leaves)
 If any of the permutations are missing, then for an input whose
result corresponds to that permutation, the algorithm will not
work correctly.

16
Lower bounds for the sorting
problem
 Note that, as a comparison based sorting algorithm
performs its comparisons and derives more and more
information on the relative orders of the input numbers, it
is actually following a path in the decision tree.
 When we consider the worst case, even the most efficient
comparison based sorting algorithm has to traverse the
decision tree from the root to a deepest leaf, i.e. it needs
to perform h comparisons where h is the height of the tree.
 So if we find a lower bound on the depth of the decision
tree, we will be able to find a lower bound on the
comparison based sorting algorithms.

17
Lower bounds for the sorting
problem
 Note that, the decision tree is a binary tree.

 For any binary tree with k leaves and of height h

k 2 h
 For decision trees we know the number of leaves: k n!
Stirling’s n!2 h
approx. (n / e) n  n! 2 h
(n / e) n  n!
( n / e) n  2 h h (n lg n)
lg(n / e) n  h
n lg n  n lg e  h

18
Lower bounds for the sorting
problem
 Since the depth of a decision tree has a lower bound of
h (n lg n)
and any comparison based sorting algorithm (in its worst
case) will have to traverse the longest path in the
decision tree, n lg n is also a lower bound for the
comparison based sorting algorithms.

 Therefore, merge sort is asymptotically optimal.

 So, do not try to find a more efficient comparison based

sorting algorithm. There is none…
19
SORTING IN LINEAR TIME

20
 No need to get confused…

 “Didn’t we just prove that any sorting algorithm is (n lg n) ?”

 “So, how are we going to be able to find a linear time sorting
algorithm?”
 What we have shown is that any comparison based
sorting algorithm has this lower bound.
 A sorting algorithm which does not depend on comparing
numbers with each other is not affected by this lower bound.
 “How can we sort without comparing numbers?”
 We will see how...

21
Counting Sort
 Assume that we are given n numbers in the range 1..k
 Basic idea of counting sort is to count the number of
times each element in the given sequence appear.
 For example, consider the sequence [3,7,3,1,2] whose
elements range in 1..9 (or to be more exact in 1..7)
 In this sequence: 1,2, and 7 appear once; 3 appears
twice.
 After having this information, it is trivial to sort this
sequence by simply generating: one 1, one 2, two 3’s,
and one 7.

22
An implementation for Counting
Sort
Counting-Sort(A,k) {
for (i=1; i <= k; i++) // initialize number counters to 0
times[i] = 0;

for (j=1; j <= length[A]; j++) // find out how many times each
times[A[j]]++; // number appears in the input

// form the sorted array

m=1; // the index of the next number to be placed in the output
for ( i=1; i <= k; i++) // consider each number in the
range
for ( j=1; j <= times[ i ]; j++) { // generate that number in
A[m]=i; // the output as many times as
m++; // it occurs in the input
}
}

23
Running time of Counting Sort

 The first for-loop:

for ( i=1; i <= k; i++) // initialize number counters to 0
times[ i ] = 0;

It will surely iterate O(k) times.

 The second for-loop:

for ( j=1; j <= length[A]; j++) // decide how many times each
times[A[ j ]]++; // number appears in the input

It will surely iterate O(n) times.

24
Running time of Counting Sort
 The third for-loop:
for ( i=1; i <= k; i++) // consider each number in the range
for ( j=1; j <= times[ i ]; j++) { // generate that number in
A[m]=i; // the output as many times as
m++; // it occurs in the input
}
 The outer for-loop iterates O(k) times
 However, the inner for-loop iterates different number of times
for each i.
 A loose upper bound can be found by considering that the
inner for-loop will iterate at most n times for each i.
 In this case, the third for-loop will need O(nk) time.
 However, we can perform a better analysis.

25
Running time of Counting Sort
 Let ti denote the number of times the inner for-loop
iterates for each i.
for (i=1; i <= k; i++) // consider each number in the range
for ( j=1; j <= times[ i ]; j++) { // generate that number in
A[m]=i; // the output as many times as
m++; // it occurs in the input
}

 Note that, every time the inner for-loop iterates, we place

a new number into its correct place in the output array.

k
 Hence:
t
i 1
i n?

26
Running time of Counting Sort
 Note that, the condition of the inner for-loop will be
checked ti+1 times for each i:

 Hence, total number of times it will be checked is:

k k k

 (t
i 1
i  1)  ti   1 n  k
i 1 i 1

27
Running time of Counting Sort
 Therefore:
O(k ) for (i=1; i <= k; i++) // consider each number in the range
O(n  k ) for (j=1; j <= times[ i ]; j++) { // generate that number in
O(n) A[m]=i; // the output as many times as
O(n) m++; // it occurs in the input
}

 Hence the total cost of the third loop is:

O(k )  O(n  k )  O(n)  O(n) O(n  k )
 Therefore the total running time of Counting Sort is:
O(k )  O(n)  O(n  k ) O(n  k )
cost of the 1st loop cost of the 2nd loop cost of the 3rd loop

28
Counting Sort is a linear time
algorithm
 Note that, running time of Counting Sort is linear in the
number of elements we will sort.

 When k=O(n), Counting Sort has O(k+n)=O(n) worst

case running time, which is lower than the lower bound
of Ω(n lg n) that we have found for comparison based
sorting algorithms.

 This is of course normal, since Counting Sort does not

sort the numbers by comparing them, i.e. it is not a
comparison based sorting algorithm.

29
Can we generalize the idea of
Counting Sort?
 We have considered the Counting Sort to solve a special
case of the sorting problem, where we know the range of
the numbers to be sorted.

 Can we generalize the approach of counting sort so that

it becomes a solution to the general sorting problem?

 In other words, given a sequence of numbers whose

range is not known, can we use counting sort as an
intermediate step?

30
Using Counting Sort as a solution to
the general sorting problem
 Suppose that we use:
GetMin(A) { GetMax(A) {
min = A[1]; max = A[1];
for (i=2; i <= length[A]; i++) for (i=2; i <= length[A]; i++)
if (min > A[i]) if (max < A[i])
min=A[i]; max=A[i];
return min; return max;
} }

 Note that both algorithms run in Θ(n) time.

31
Using Counting Sort as a solution to
the general sorting problem
 Then we can:
LinearSort (A) {
min = GetMin(A); // find minimum number
max = GetMax(A); // find maximum number
k = max – min + 1; // find the length of our range
subtract “min-1” from each number in A; // transfer the numbers
// into the range 1..k
Counting-Sort(A,k); // apply the counting sort
add “min-1” to each number in A; // transfer the numbers back
}

 Note that the steps other than the counting sort take
Θ(n) time. Hence the overall time is still O(n+k).

32
Practical applicability of Counting
Sort
 The counting based solution to the sorting problem

seems to be better than the comparison based sorting

problem, if we consider the fact that the running time is
linear in the number of elements to be sorted.
 However, we should not forget that there is another term
in the running time expression of counting sort, which is
k.
 When k=O(n): O(k+n)=O(n), so using counting sort is a
wise approach.
 But, in practice k in O(k+n) can cause problems.

33
Practical applicability of Counting
Sort
 Assume that, we are trying to sort 32 bit numbers.

 How many values can we represent using 32 bits:

22x210x210x210=4 giga
 Even if we use 1 byte for each counter, we will need 4
giga bytes for the times array in counting sort.
 Note that, we will need to initialize the times array (one
of the reasons we have the term k in the running time
expression of counting sort).
 Therefore, the space and time requirement of counting
sort may not always be practical.

34