Sorting Algorithms

Slides and figures have been collected from various publicly

available Internet sources for preparing the lecture slides of
IT2001 course. I acknowledge and thank all the original
authors for their contribution to prepare the content.
The problem of sorting

Input: sequence a1, a2, …, an of numbers.

Output: permutation a'1, a'2, …, a'n such

that a'1  a'2  …  a'n .

Example:
Input: 8 2 4 9 3 6
Output: 2 3 4 6 8 9
Why Study Sorting Algorithms?

• There are a variety of situations that we can encounter

– Do we have randomly ordered keys?
– Are all keys distinct?
– How large is the set of keys to be ordered?
– Need guaranteed performance?

• Various algorithms are better suited to some of these situations

Some Definitions

• Internal Sort
– The data to be sorted is all stored in the computer’s main
memory.
• External Sort
– Some of the data to be sorted might be stored in some external,
slower, device.
• In Place Sort
– The amount of extra space required to sort the data is constant
with the input size.
Stability
• A STABLE sort preserves relative order of records with equal keys

Sorted on first key:

Sort file on second key:

Records with
key value 3 are
not in order on
first key!!
Insertion sort
INSERTION-SORT (A, n) ⊳ A[1 . . n]
for j ← 2 to n
do key ← A[ j]
i←j–1
“pseudocode” while i > 0 and A[i] > key
do A[i+1] ← A[i]
i←i–1
A[i+1] = key
1 i j n
A:
key
sorted
Example of insertion sort
8 2 4 9 3 6
Example of insertion sort
8 2 4 9 3 6
Example of insertion sort
8 2 4 9 3 6
2 8 4 9 3 6
Example of insertion sort
8 2 4 9 3 6
2 8 4 9 3 6
Example of insertion sort
8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
Example of insertion sort
8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
Example of insertion sort
8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
2 4 8 9 3 6
Example of insertion sort
8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
2 4 8 9 3 6
Example of insertion sort
8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
2 4 8 9 3 6
2 3 4 8 9 6
Example of insertion sort
8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
2 4 8 9 3 6
2 3 4 8 9 6
Example of insertion sort
8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
2 4 8 9 3 6
2 3 4 8 9 6
2 3 4 6 8 9 done
Kinds of analyses

Worst-case: (usually)
•T(n) = maximum time of algorithm on any input of size n.
Average-case: (sometimes)
•T(n) = expected time of algorithm over all inputs of size n.
•Need assumption of statistical distribution of inputs.
Best-case: (NEVER)
•Cheat with a slow algorithm that works fast on some input.
Analysis of Insertion Sort
INSERTION-SORT(A) cost times
for j ← 2 to n c1 n
do key ← A[ j ] c2 n-1
Insert A[ j ] into the sorted sequence A[1 . . j -1] 0 n-1
i←j-1 c4 n-1

n
while (i > 0 and A[i] > key) c5 j =2 j
t
c6 
n
do A[i + 1] ← A[i] j =2
(t j − 1)
c7 
n
i←i–1 j =2
(t j − 1)

A[i + 1] ← key c8 n-1

tj: # of times the while statement is executed at iteration j

T (n) = c1n + c2 (n − 1) + c4 (n − 1) + c5  t j + c6  (t j − 1) + c7  (t j − 1) + c8 (n − 1)
n n n

j =2 j =2 j =2
Insertion sort analysis
Worst case: Input reverse sorted.
n
T (n) =   ( j ) =  ( n 2) [arithmetic series]
j=2
Average case: All permutations equally likely.
n
T (n) =   ( j / 2 ) =  (n 2 )
j=2
Is insertion sort a fast sorting algorithm?
• Moderately so, for small n.
• Not at all, for large n.
 Selection Sort
• Idea:
– Find the smallest element in the array
– Exchange it with the element in the first position
– Find the second smallest element and exchange it
with the element in the second position
– Continue until the array is sorted
• Disadvantage:
– Running time depends only slightly on the amount of
order in the file
 Example
8 4 6 9 2 3 1 1 2 3 4 9 6 8

1 4 6 9 2 3 8 1 2 3 4 6 9 8

1 2 6 9 4 3 8 1 2 3 4 6 8 9

1 2 3 9 4 6 8 1 2 3 4 6 8 9
 Selection Sort
Alg.: SELECTION-SORT(A)
n ← length[A] 8 4 6 9 2 3 1

for j ← 1 to n - 1
do smallest ← j
for i ← j + 1 to n
do if A[i] < A[smallest]
then smallest ← i
exchange A[j] ↔ A[smallest]
 Analysis of Selection Sort
Alg.: SELECTION-SORT(A) cost times
n ← length[A] c1 1
for j ← 1 to n - 1 c2 n
do smallest ← j c3 n-1
n2/2 for i ← j + 1 to n c4 nj=−11 (n − j + 1)
comparisons
do if A[i] < A[smallest] c5 nj=−11 (n − j )
n
exchanges then smallest ← i c6 nj=−11 (n − j )
exchange A[j] ↔ A[smallest] c7 n-1
n −1 n −1 n −1
T (n) = c1 + c2 n + c3 (n − 1) + c4  (n − j + 1) + c5  ( n − j ) + c6  ( n − j ) + c7 (n − 1) = (n 2 )
j =1 j =1 j =2
Divide and Conquer
(Merge Sort)
 Divide and Conquer
• Recursive in structure
– Divide the problem into sub-problems that are similar
to the original but smaller in size
– Conquer the sub-problems by solving them
recursively. If they are small enough, just solve them
in a straight forward manner.
– Combine the solutions to create a solution to the
original problem
 An Example: Merge Sort

Sorting Problem: Sort a sequence of n elements into non-

decreasing order.
• Divide: Divide the n-element sequence to be sorted into two
subsequences of n/2 elements each
• Conquer: Sort the two subsequences recursively using
merge sort.
• Combine: Merge the two sorted subsequences to produce
the sorted answer.
 Merge – Example
A … 61 86 26
8 32 1 32
9 26 9 42 43 …
k k k k k k k k k

L 6 8 26 32  R 1 9 42 43 
i i i i i j j j j j
 Procedure Merge
Merge(A, p, q, r)
1 n1  q – p + 1
2 n2  r – q
Input: Array containing sorted
3 for i  1 to n1 subarrays A[p..q] and A[q+1..r].
4 do L[i]  A[p + i – 1]
5 for j  1 to n2 Output: Merged sorted subarray in
6 do R[j]  A[q + j] A[p..r].
7 L[n1+1]  
8 R[n2+1]  
9 i1
10 j  1 Sentinels, to avoid having to
11 for k p to r check if either subarray is
12 do if L[i]  R[j] fully copied at each step.
then A[k]  L[i]
13
14 ii+1
Time = (n) to merge a total
15 else A[k]  R[j] of n elements (linear time).
16 jj+1
 Merge-Sort (A, p, r)
INPUT: a sequence of n numbers stored in array A
OUTPUT: an ordered sequence of n numbers

MergeSort (A, p, r) // sort A[p..r] by divide & conquer

1 if p < r // If n = 1, done
2 then q  (p+r)/2
3 MergeSort (A, p, q)
4 MergeSort (A, q+1, r)
5 Merge (A, p, q, r) // merges A[p..q] with A[q+1..r]

Initial Call: MergeSort(A, 1, n)

 Merge Sort – Example
18 26 32 6 43 15 9 1 22 26 19 55 37 43 99 2

18 26 32 6 43 15 9 1 22 26 19 55 37 43 99 2

18 26 32 6 43 15 9 1 22 26 19 55 37 43 99 2

18 26 32 6 43 15 9 1 22 26 19 55 37 43 99 2

18 26 32 6 43 15 9 1 22 26 19 55 37 43 99 2
 Merge Sort – Example
Original Sequence Sorted Sequence
18 26 32 6 43 15 9 1 1 6 9 15 18 26 32 43

18 26 32 6 43 15 9 1 6 18 26 32 1 9 15 43
43

18 26 32 6 43 15 9 1 18 26 6 32 15 43 1 9

18 26 32 6 43 15 9 1 18 26 32 6 43 15 9 1

18 26 32 6 43 15 9 1
 Analysis of Merge Sort
• Running time T(n) of Merge Sort:
• Divide: computing the middle takes (1)
• Conquer: solving 2 subproblems takes 2T(n/2)
• Combine: merging n elements takes (n)
• Total:
T(n) = (1) if n = 1
T(n) = 2T(n/2) + (n) if n > 1
 T(n) = (n lg n) (CLRS book, Chapter 4)
 Conclusions

• (n lg n) grows more slowly than (n2).

• Therefore, merge sort asymptotically beats insertion
sort in the worst case.
• In practice, merge sort beats insertion sort for n > 30
or so.
 Comparison Sorting
Sort Worst Average Best Comments
Case Case Case

Selection Sort Θ(N2) Θ(N2) Θ(N2) Fast for

small N

Insertion Sort Θ(N2) Θ(N2) Θ(N) Fast for

small N

Merge Sort Θ(N lg N) Θ(N lg N) Θ(N lg N) Requires

memory

Heap Sort Θ(N lg N) Θ(N lg N) Θ(N lg N) Large constants

Quick Sort Θ(N2) Θ(N lg N) Θ(N lg N) Small constants

 Recurrence for merge sort

(1) if n = 1;
T(n) =
2T(n/2) + (n) if n > 1.

• We shall usually omit stating the base case when

T(n) = (1) for sufficiently small n, but only when it
has no effect on the asymptotic solution to the
recurrence.
Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
T(n)
Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
T(n/2) T(n/2)
Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
cn/2 cn/2

T(n/4) T(n/4) T(n/4) T(n/4)

Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
cn/2 cn/2

cn/4 cn/4 cn/4 cn/4

(1)
 Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
cn/2 cn/2
h = lg n cn/4 cn/4 cn/4 cn/4

(1)
 Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn cn
cn/2 cn/2
h = lg n cn/4 cn/4 cn/4 cn/4

(1)
 Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn cn
cn/2 cn/2 cn
h = lg n cn/4 cn/4 cn/4 cn/4

(1)
 Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn cn
cn/2 cn/2 cn
h = lg n cn/4 cn/4 cn/4 cn/4 cn

…
(1)
 Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn cn
cn/2 cn/2 cn
h = lg n cn/4 cn/4 cn/4 cn/4 cn

…
(1) #leaves = n (n)
 Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn cn
cn/2 cn/2 cn
h = lg n cn/4 cn/4 cn/4 cn/4 cn

…
(1) #leaves = n (n)
Total = (n lg n)
 References

❑ http://www.cse.unr.edu/~bebis/
❑ http://courses.csail.mit.edu/6.046/spring04/lectures