Datenstrukturen

und Algorithmen

Elementary Sorts
 2.1 E LEMENTARY S ORTS

h tt p : / / a l g s 4 . c s . p r i n c e t o n . e d u

2
 Sorting problem
Chen 3 A 991-878-4944 308 Blair

Rohde 2 A 232-343-5555 343 Forbes

Gazsi 4 B 766-093-9873 101 Brown

item Furia 1 A 766-093-9873 101 Brown

Kanaga 3 B 898-122-9643 22 Brown

Andrews 3 A 664-480-0023 097 Little

key Battle 4 C 874-088-1212 121 Whitman

3
 Sorting problem
Chen 3 A 991-878-4944 308 Blair

Rohde 2 A 232-343-5555 343 Forbes

Gazsi 4 B 766-093-9873 101 Brown

item Furia 1 A 766-093-9873 101 Brown

Kanaga 3 B 898-122-9643 22 Brown

Andrews 3 A 664-480-0023 097 Little

key Battle 4 C 874-088-1212 121 Whitman

Andrews 3 A 664-480-0023 097 Little

Battle 4 C 874-088-1212 121 Whitman

Chen 3 A 991-878-4944 308 Blair

Furia 1 A 766-093-9873 101 Brown

Gazsi 4 B 766-093-9873 101 Brown

Kanaga 3 B 898-122-9643 22 Brown

Rohde 2 A 232-343-5555 343 Forbes

3
Sorting applications

Library numbers

contacts

FedEx packages

playing cards Hogwarts houses

4
 G: Sort any type of data
Ex 1. Sort random real numbers in ascending order.

import sys, random % python3 Experiment.py 10

import DSA, Selection 0.08614716385210452
0.09054270895414829
N = int(sys.argv[1]) 0.10708746304898642
a = DSA.floatArray(N)
0.21166190071646818
for i in range(N):
a[i] = random.uniform(0,1) 0.363292849257276

Selection.sort(a) 0.460954145685913
for i in range(N): 0.5340026311350087
print(a[i]) 0.7216129793703496
0.9003500354411443
0.9293994908845686

5
G: Sort any type of data
Ex 2. Sort strings in alphabetical order.
import DSA, Selection

a = DSA.stdReadAllStrings()
Selection.sort(a)
for e in a:
print(e)

% cat words3.txt
bed bug dad yet zoo ... all bad yes

% python3 StringSorter.py < words3.txt

all bad bed bug dad ... yes yet zoo
[suppressing newlines]

6
 G: Sort any type of data
Ex 3. Sort the les in a given directory by lename.

% python3 FileSorter.py .
DSA.py (size: 2783 bytes)
import sys Date.py (size: 556 bytes)
import DSA, Selection
Experiment.py (size: 182 bytes)
FileSorter.py (size: 104 bytes)
a = DSA.readFiles(sys.argv[1])
Selection.sort(a) Insertion.py (size: 184 bytes)
for e in a: Selection.py (size: 187 bytes)
print(e) ShellSort.py (size: 293 bytes)
Shuffle.py (size: 231 bytes)
ShuffleTest.py (size: 604 bytes)
StringSorter.py (size: 90 bytes)

7
fi
fi
 Total order

A total order is a binary relation ≤ that satis es:

• Antisymmetry: if both v ≤ w and w ≤ v, then v = w.

• Transitivity: if both v ≤ w and w ≤ x, then v ≤ x.

• Totality: either v ≤ w or w ≤ v or both.

8
fi
 Strict Total order

A strict total order is a asymmetric binary relation <

associated to a total order that satis es:
a < b if a ≤ b and a ≠ b

9
fi
 Examples

• Standard order for natural and real numbers.

• Chronological order for dates or times.
• Alphabetical order for strings.

10
 Counterexamples
• No transitivity. Rock-paper-scissors.

• No totality. Course prerequisites.

COS 423 COS 333

COS 226 COS 217

COS 126
11
 Python Sort

• Q. How can sort() know how to compare data of type

double, string, or your own class without any information
about the type of an item's key?

• Client passes array of objects to sort() function.

• The sort() function calls object's __lt__()

(less than) method as needed.

12
 Roadmap
client
import DSA, Selection

a = DSA.stdReadAllStrings()
Selection.sort(a)
for e in a:
print(e)

13
 Roadmap
client
import DSA, Selection

a = DSA.stdReadAllStrings()
Selection.sort(a)
for e in a:
print(e)

Comparable Function (automatically called by Python)

Python)
def __lt__(self, other):
# ...

13
 Roadmap
client data-type implementation
import DSA, Selection class Date:
def __init__(self, d, m, y):
a = DSA.stdReadAllStrings() self.d = d
Selection.sort(a) self.m = m
for e in a: self.y = y
print(e)
# ...

Comparable Function (automatically called by Python)

Python)
def __lt__(self, other):
# ...

13
 Roadmap
client data-type implementation
import DSA, Selection class Date:
def __init__(self, d, m, y):
a = DSA.stdReadAllStrings() self.d = d
Selection.sort(a) self.m = m
for e in a: self.y = y
print(e)
# ...

sort implementation
def sort(a):
N = len(a)
Comparable Function (automatically called by Python) for i in range(N):
min = i
Python)
def __lt__(self, other): for j in range(i+1,N):
# ... if a[j] < a[min]:
min = j
exch(a, i, min)

key point: no dependence

on a particular data type

13
 Comparable API
Implement __lt__(self, other) so that:
• De nes a total order.
• Returns true if self is less than other.

self other
self other
other self

less than (return True) equal to (return False) greater than (return False)

14
fi
 Implementing the
"Less-Than" Function
class Date:
def __init__(self, d, m, y):
self.d = d
self.m = m
self.y = y

def __lt__(self, other):

if self.y < other.y: return True
if self.y > other.y: return False
if self.m < other.m: return True
if self.m > other.m: return False
return self.d < other.d

15
Elementary Sorts
Selection Sort
 Selection Sort

• In iteration i, nd index min of smallest remaining entry.

• Swap a[i] and a[min].

18
fi
 Selection sort demo

initial

• In iteration i, nd index min of smallest remaining entry.

• Swap a[i] and a[min].

19
fi
 Selection sort demo

remaining entries

• In iteration i, nd index min of smallest remaining entry.

• Swap a[i] and a[min].

20
fi
 Selection sort demo

i min

remaining entries

• In iteration i, nd index min of smallest remaining entry.

• Swap a[i] and a[min].

21
fi
 Selection sort demo

i min

remaining entries

• In iteration i, nd index min of smallest remaining entry.

• Swap a[i] and a[min].

22
fi
 Selection sort demo

in nal order remaining entries

• In iteration i, nd index min of smallest remaining entry.

• Swap a[i] and a[min].

23
fi
fi
 Selection sort demo

i min

in nal order remaining entries

• In iteration i, nd index min of smallest remaining entry.

• Swap a[i] and a[min].

24
fi
fi
 Selection sort demo

i min

in nal order remaining entries

• In iteration i, nd index min of smallest remaining entry.

• Swap a[i] and a[min].

25
fi
fi
 Selection sort demo

in nal order remaining entries

• In iteration i, nd index min of smallest remaining entry.

• Swap a[i] and a[min].

26
fi
fi
 Selection sort demo

i min

in nal order remaining entries

• In iteration i, nd index min of smallest remaining entry.

• Swap a[i] and a[min].

27
fi
fi
 Selection sort demo

i min

in nal order remaining entries

• In iteration i, nd index min of smallest remaining entry.

• Swap a[i] and a[min].

28
fi
fi
 Selection sort demo

in nal order remaining entries

• In iteration i, nd index min of smallest remaining entry.

• Swap a[i] and a[min].

29
fi
fi
 Selection sort demo

i min

in nal order remaining entries

• In iteration i, nd index min of smallest remaining entry.

• Swap a[i] and a[min].

30
fi
fi
 Selection sort demo

i min

in nal order remaining entries

• In iteration i, nd index min of smallest remaining entry.

• Swap a[i] and a[min].

31
fi
fi
 Selection sort demo

in nal order remaining entries

• In iteration i, nd index min of smallest remaining entry.

• Swap a[i] and a[min].

32
fi
fi
 Selection sort demo

i min

in nal order remaining entries

• In iteration i, nd index min of smallest remaining entry.

• Swap a[i] and a[min].

33
fi
fi
 Selection sort demo

i min

in nal order remaining entries

• In iteration i, nd index min of smallest remaining entry.

• Swap a[i] and a[min].

34
fi
fi
 Selection sort demo

in nal order remaining entries

• In iteration i, nd index min of smallest remaining entry.

• Swap a[i] and a[min].

35
fi
fi
 Selection sort demo

i min

in nal order remaining entries

• In iteration i, nd index min of smallest remaining entry.

• Swap a[i] and a[min].

36
fi
fi
 Selection sort demo

i min

in nal order remaining entries

• In iteration i, nd index min of smallest remaining entry.

• Swap a[i] and a[min].

37
fi
fi
 Selection sort demo

in nal order remaining entries

• In iteration i, nd index min of smallest remaining entry.

• Swap a[i] and a[min].

38
fi
fi
 Selection sort demo

i min

in nal order remaining entries

• In iteration i, nd index min of smallest remaining entry.

• Swap a[i] and a[min].

39
fi
fi
 Selection sort demo

i min

in nal order remaining entries

• In iteration i, nd index min of smallest remaining entry.

• Swap a[i] and a[min].

40
fi
fi
 Selection sort demo

in nal order remaining entries

• In iteration i, nd index min of smallest remaining entry.

• Swap a[i] and a[min].

41
fi
fi
 Selection sort demo

i min

in nal order remaining entries

• In iteration i, nd index min of smallest remaining entry.

• Swap a[i] and a[min].

42
fi
fi
 Selection sort demo

i min

in nal order remaining entries

• In iteration i, nd index min of smallest remaining entry.

• Swap a[i] and a[min].

43
fi
fi
 Selection sort demo

i min

in nal order remaining entries

• In iteration i, nd index min of smallest remaining entry.

• Swap a[i] and a[min].

43
fi
fi
 Selection sort demo

in nal order remaining entries

• In iteration i, nd index min of smallest remaining entry.

• Swap a[i] and a[min].

44
fi
fi
 Selection sort demo

i min

in nal order remaining entries

• In iteration i, nd index min of smallest remaining entry.

• Swap a[i] and a[min].

45
fi
fi
 Selection sort demo

i min

in nal order remaining entries

• In iteration i, nd index min of smallest remaining entry.

• Swap a[i] and a[min].

46
fi
fi
 Selection sort demo

i min

in nal order remaining entries

• In iteration i, nd index min of smallest remaining entry.

• Swap a[i] and a[min].

46
fi
fi
 Selection sort demo

in nal order

• In iteration i, nd index min of smallest remaining entry.

• Swap a[i] and a[min].

47
fi
fi
 Selection sort demo

sorted

• In iteration i, nd index min of smallest remaining entry.

• Swap a[i] and a[min].

48
fi
 Selection sort:
Gypsy folk dance

https://www.youtube.com/watch?v=Ns4TPTC8whw
49
 Selection sort
• Algorithm: ↑ scans from left to right.

• Invariants:

• Entries the left of ↑ (including ↑) xed and in ascending order.

• No entry to right of ↑ is smaller than any entry to the left of ↑.

↑
in nal order
51
fi
fi
 Little Helper

Exchange: Swap item in array a[] at index i with the one at index j.

def exch(a, i, j):

t = a[i]
a[i] = a[j]
a[j] = t

52
 Selection sort inner loop
To maintain algorithm invariants:

Move the pointer to the right.

i += 1

in nal order ↑

53
fi
 Selection sort inner loop
To maintain algorithm invariants:

Move the pointer to the right.

i += 1

in nal order ↑

Identify index of minimum entry on right.

min = i
for j in range(i+1,N):
if a[j] < a[min]:
min = j
in nal order ↑ ↑

53
fi
fi
 Selection sort inner loop
To maintain algorithm invariants:

Move the pointer to the right.

i += 1

in nal order ↑

Identify index of minimum entry on right.

min = i
for j in range(i+1,N):
if a[j] < a[min]:
min = j
in nal order ↑ ↑

Exchange into position.

exch(a, i, min)

in nal order ↑ ↑
53
fi
fi
fi
Python Implementation
def exch(a, i, j):
t = a[i]
a[i] = a[j]
a[j] = t

def sort(a):
N = len(a)
for i in range(N):
min = i
for j in range(i+1,N):
if a[j] < a[min]:
min = j
exch(a, i, min)

54
 20 random items

algorithm position
in nal order
not in nal order

http://www.sorting-algorithms.com/selection-sort

55
fi
fi
 20 random items

algorithm position
in nal order
not in nal order

http://www.sorting-algorithms.com/selection-sort

55
fi
fi
 20 partially-sorted items

algorithm position
in nal order
not in nal order

http://www.sorting-algorithms.com/selection-sort

56
fi
fi
 20 partially-sorted items

algorithm position
in nal order
not in nal order

http://www.sorting-algorithms.com/selection-sort

56
fi
fi
 Analysis
a[]
entries in black
i min 0 1 2 3 4 5 6 7 8 9 10 are examined to find
the minimum
S O R T E X A M P L E
0 6 S O R T E X A M P L E
entries in red
1 4 A O R T E X S M P L E are a[min]
2 10 A E R T O X S M P L E
3 9 A E E T O X S M P L R
4 7 A E E L O X S M P T R
5 7 A E E L M X S O P T R
6 8 A E E L M O S X P T R
7 10 A E E L M O P X S T R
8 8 A E E L M O P R S T X
entries in gray are
9 9 A E E L M O P R S T X in final position
10 10 A E E L M O P R S T X
A E E L M O P R S T X

Trace of selection sort (array contents just after each exchange)

57
 Analysis
• Proposition: Selection sort uses
(N – 1) + (N – 2) + ... + 1 + 0 ~ 1/2 N 2 compares and N
exchanges.
• Running time insensitive to input. Quadratic time, even
if input is sorted.
• Data movement is minimal. Linear number of
exchanges.

58
Insertion Sort
 Insertion Sort

In iteration i, swap a[i] with each larger entry to its left.

60
 Insertion sort demo

In iteration i, swap a[i] with each larger entry

to its left.
61
 Insertion sort demo

not yet seen

In iteration i, swap a[i] with each larger entry

to its left.
62
 Insertion sort demo

j i

in ascending order not yet seen

In iteration i, swap a[i] with each larger entry

to its left.
63
 Insertion sort demo

j i

not yet seen

In iteration i, swap a[i] with each larger entry

to its left.
64
 Insertion sort demo

j i

in ascending order not yet seen

In iteration i, swap a[i] with each larger entry

to its left.
65
 Insertion sort demo

j i

not yet seen

In iteration i, swap a[i] with each larger entry

to its left.
66
 Insertion sort demo

j i

not yet seen

In iteration i, swap a[i] with each larger entry

to its left.
67
 Insertion sort demo

j i

not yet seen

In iteration i, swap a[i] with each larger entry

to its left.
68
 Insertion sort demo

in ascending order not yet seen

In iteration i, swap a[i] with each larger entry

to its left.
69
 Insertion sort demo

j i

not yet seen

In iteration i, swap a[i] with each larger entry

to its left.
70
 Insertion sort demo

j i

not yet seen

In iteration i, swap a[i] with each larger entry

to its left.
71
 Insertion sort demo

j i

not yet seen

In iteration i, swap a[i] with each larger entry

to its left.
72
 Insertion sort demo

j i

not yet seen

In iteration i, swap a[i] with each larger entry

to its left.
73
 Insertion sort demo

in ascending order not yet seen

In iteration i, swap a[i] with each larger entry

to its left.
74
 Insertion sort demo

j i

not yet seen

In iteration i, swap a[i] with each larger entry

to its left.
75
 Insertion sort demo

j i

not yet seen

In iteration i, swap a[i] with each larger entry

to its left.
76
 Insertion sort demo

in ascending order not yet seen

In iteration i, swap a[i] with each larger entry

to its left.
77
 Insertion sort demo

j i

not yet seen

In iteration i, swap a[i] with each larger entry

to its left.
78
 Insertion sort demo

j i

not yet seen

In iteration i, swap a[i] with each larger entry

to its left.
79
 Insertion sort demo

j i

not yet seen

In iteration i, swap a[i] with each larger entry

to its left.
80
 Insertion sort demo

j i

not yet seen

In iteration i, swap a[i] with each larger entry

to its left.
81
 Insertion sort demo

j i

not yet seen

In iteration i, swap a[i] with each larger entry

to its left.
82
 Insertion sort demo

in ascending order not yet seen

In iteration i, swap a[i] with each larger entry

to its left.
83
 Insertion sort demo

j i

not yet seen

In iteration i, swap a[i] with each larger entry

to its left.
84
 Insertion sort demo

j i

not yet seen

In iteration i, swap a[i] with each larger entry

to its left.
85
 Insertion sort demo

j i

not yet seen

In iteration i, swap a[i] with each larger entry

to its left.
86
 Insertion sort demo

j i

not yet seen

In iteration i, swap a[i] with each larger entry

to its left.
87
 Insertion sort demo

j i

not yet seen

In iteration i, swap a[i] with each larger entry

to its left.
88
 Insertion sort demo

j i

not yet seen

In iteration i, swap a[i] with each larger entry

to its left.
89
 Insertion sort demo

j i

not yet seen

In iteration i, swap a[i] with each larger entry

to its left.
90
 Insertion sort demo

in ascending order not yet seen

In iteration i, swap a[i] with each larger entry

to its left.
91
 Insertion sort demo

j i

not yet seen

In iteration i, swap a[i] with each larger entry

to its left.
92
 Insertion sort demo

j i

not yet seen

In iteration i, swap a[i] with each larger entry

to its left.
93
 Insertion sort demo

in ascending order not yet seen

In iteration i, swap a[i] with each larger entry

to its left.
94
 Insertion sort demo

j i

In iteration i, swap a[i] with each larger entry

to its left.
95
 Insertion sort demo

j i

In iteration i, swap a[i] with each larger entry

to its left.
96
 Insertion sort demo

j i

In iteration i, swap a[i] with each larger entry

to its left.
97
 Insertion sort demo

j i

In iteration i, swap a[i] with each larger entry

to its left.
98
 Insertion sort demo

j i

In iteration i, swap a[i] with each larger entry

to its left.
99
 Insertion sort demo

sorted

In iteration i, swap a[i] with each larger entry

to its left.
100
 Insertion sort:
Romanian folk dance

https://www.youtube.com/watch?v=ROalU379l3U
101
 Invariants
• Entries to the left of ↑ (including ↑) are in
ascending order.
• Entries to the right of ↑ have not yet been seen.

in order ↑ not yet seen

103
Insertion sort inner loop
To maintain algorithm invariants:

Move the pointer to the right.

i += 1

in order not yet seen

104
Insertion sort inner loop
To maintain algorithm invariants:

Move the pointer to the right.

i += 1

in order not yet seen

Moving from right to left, exchange

a[i] with each larger entry to its left.
for j in range(i,0,-1):
if a[j] < a[j-1]:
exch(a, j, j-1)
else:
break ↑ ↑ ↑↑
in order not yet seen
104
Python Implementation
def exch(a, i, j):
t = a[i]
a[i] = a[j]
a[j] = t

def sort(a):
N = len(a)
for i in range(N):
for j in range(i,0,-1):
if a[j] < a[j-1]:
exch(a, j, j-1)
else:
break

105
40 random items

algorithm position
in order
not yet seen
http://www.sorting-algorithms.com/insertion-sort

106
40 random items

algorithm position
in order
not yet seen
http://www.sorting-algorithms.com/insertion-sort

106
40 reverse-sorted items

algorithm position
in order
not yet seen
http://www.sorting-algorithms.com/insertion-sort

107
40 reverse-sorted items

algorithm position
in order
not yet seen
http://www.sorting-algorithms.com/insertion-sort

107
40 partially-sorted items

algorithm position
in order
not yet seen
http://www.sorting-algorithms.com/insertion-sort

108
40 partially-sorted items

algorithm position
in order
not yet seen
http://www.sorting-algorithms.com/insertion-sort

108
 Analysis

• Proposition: To sort a randomly-ordered array

with distinct keys, insertion sort uses ~ ¼ N 2
compares and ~ ¼ N exchanges on average.
2

• Pf: Expect each entry to move halfway back.

109
 Analysis
a[]
i j 0 1 2 3 4 5 6 7 8 9 10
S O R T E X A M P L E entries in gray
do not move
1 0 O S R T E X A M P L E
2 1 O R S T E X A M P L E
3 3 O R S T E X A M P L E
4 0 E O R S T X A M P L E entry in red
is a[j]
5 5 E O R S T X A M P L E
6 0 A E O R S T X M P L E
7 2 A E M O R S T X P L E
entries in black
8 4 A E M O P R S T X L E moved one position
9 2 A E L M O P R S T X E right for insertion
10 2 A E E L M O P R S T X
A E E L M O P R S T X

Trace of insertion sort (array contents just after each insertion)

110
111
 Best Case
If the array is in ascending order, insertion
sort makes N – 1 compares and 0 exchanges.

A E E L M O P R S T X

112
 Worst Case
If the array is in descending order (and no duplicates),
insertion sort makes ~ ½ N 2 compares and
~ ½ N 2 exchanges.

X T S R P O M L E E A

113
Partially-sorted arrays
Def.: An inversion is a pair of keys that are
out of order.

A E E L M O T R X P S

T-R T-P T-S R-P X-P X-S

(6 inversions)

114
 Partially-sorted arrays
• Def.: An array is partially sorted if the number of
inversions is ≤ c N.

• Ex 1.: A sorted array with i inversions.

• Ex 2.: A subarray of size 10 appended to a sorted subarray
of size N.

115
 Partially-sorted arrays

• Proposition: For partially-sorted arrays, insertion sort runs in

linear time.
• Pf.: Number of exchanges equals the number of inversions.

number of compares = exchanges + (N – 1)

116
Practical improvements
Half exchanges:
• Shift items over (instead of exchanging)
• Eliminates unnecessary data movement.
• No longer uses only __lt__() and exch()
to access data.
A C H H I M N N P Q X Y K B I N A R Y

117
Practical improvements
Half exchanges:
• Shift items over (instead of exchanging)
• Eliminates unnecessary data movement.
• No longer uses only __lt__() and exch()
to access data.
A C H H I M N N P Q X Y B I N A R Y

117
Practical improvements
Half exchanges:
• Shift items over (instead of exchanging)
• Eliminates unnecessary data movement.
• No longer uses only __lt__() and exch()
to access data.
A C H H I M N N P Q X Y B I N A R Y

117
Practical improvements
Half exchanges:
• Shift items over (instead of exchanging)
• Eliminates unnecessary data movement.
• No longer uses only __lt__() and exch()
to access data.
A C H H I M N N P Q X Y B I N A R Y

117
Practical improvements
Half exchanges:
• Shift items over (instead of exchanging)
• Eliminates unnecessary data movement.
• No longer uses only __lt__() and exch()
to access data.
A C H H I K M N N P Q X Y B I N A R Y

117
 Practical improvements
Binary insertion sort:
• Use binary search to nd insertion point.
• Number of compares ~ N lg N .
• But still a quadratic number of array accesses.

A C H H I M N N P Q X Y K B I N A R Y

118
fi
 Practical improvements
Binary insertion sort:
• Use binary search to nd insertion point.
• Number of compares ~ N lg N .
• But still a quadratic number of array accesses.

A C H H I M N N P Q X Y K B I N A R Y

binary search for rst key > K

118
fi
fi
 Practical improvements
Binary insertion sort:
• Use binary search to nd insertion point.
• Number of compares ~ N lg N .
• But still a quadratic number of array accesses.

A C H H I M N N P Q X Y K B I N A R Y

binary search for rst key > K

118
fi
fi
Shellsort
 h-Sorting
Idea: Move entries more than one position at
a time by h-sorting the array.
an h-sorted array is h interleaved sorted subsequences

h=4
L E E A M H L E P S O L T S X R
L M P T
E H S S
E L O X
A E L R

h = 13
P H E L L S O R T E X A M S L E
h-sorted:
P for each i, a[i] <= a[i+h]
S
H L
E E 120
h-sorting demo
h-sorting demo

122
h-sorting demo

j i

not yet processed

123
 h-sorting demo

j i

not yet processed

124
h-sorting demo

3-sorted not yet processed

125
h-sorting demo

j i

not yet processed

126
 h-sorting demo

j i

not yet processed

127
h-sorting demo

3-sorted not yet processed

128
h-sorting demo

j i

not yet processed

129
h-sorting demo

3-sorted not yet processed

130
h-sorting demo

j i

not yet processed

131
h-sorting demo

j i

not yet processed

132
 h-sorting demo

j i

not yet processed

133
h-sorting demo

3-sorted not yet processed

134
h-sorting demo

j i

not yet processed

135
h-sorting demo

3-sorted not yet processed

136
h-sorting demo

j i

137
h-sorting demo

j i

138
h-sorting demo

3-sorted

139
h-sorting demo

3-sorted

140
h-sorting demo

3-sorted

141
h-sorting demo

3-sorted

142
 Shellsort [Shell 1959]

h-sort array for decreasing sequence of values of h.

input S H E L L S O R T E X A M P L E
13-sort P H E L L S O R T E X A M S L E
4-sort L E E A M H L E P S O L T S X R

1-sort A E E E H L L L M O P R S S T X

Shellsort trace (array contents after each pass)

143
How to h-sort an array?
Insertion sort, with stride length h.
M O L E E X A S P R T
E O L M E X A S P R T
E E L M O X A S P R T
E E L M O X A S P R T
A E L E O X M S P R T
A E L E O X M S P R T
A E L E O P M S X R T
A E L E O P M S X R T
A E L E O P M S X R T
A E L E O P M S X R T

Why insertion sort?

144
Shellsort example: increments 7, 3, 1
input
S O R T E X A M P L E

145
Shellsort example: increments 7, 3, 1
input
S O R T E X A M P L E
7-sort

S O R T E X A M P L E
M O R T E X A S P L E
M O R T E X A S P L E
M O L T E X A S P R E
M O L E E X A S P R T

145
Shellsort example: increments 7, 3, 1
input
S O R T E X A M P L E
7-sort

S O R T E X A M P L E
M O R T E X A S P L E
M O R T E X A S P L E
M O L T E X A S P R E
M O L E E X A S P R T

3-sort

M O L E E X A S P R T
E O L M E X A S P R T
E E L M O X A S P R T
E E L M O X A S P R T
A E L E O X M S P R T
A E L E O X M S P R T
A E L E O P M S X R T
A E L E O P M S X R T
A E L E O P M S X R T
145
Shellsort example: increments 7, 3, 1
input
S O R T E X A M P L E
7-sort
1-sort
S O R T E X A M P L E
M O R T E X A S P L E A E L E O P M S X R T
M O R T E X A S P L E A E L E O P M S X R T
M O L T E X A S P R E A E L E O P M S X R T
M O L E E X A S P R T A E E L O P M S X R T
A E E L O P M S X R T
3-sort
A E E L O P M S X R T
M O L E E X A S P R T A E E L M O P S X R T
E O L M E X A S P R T A E E L M O P S X R T
E E L M O X A S P R T A E E L M O P S X R T
E E L M O X A S P R T A E E L M O P R S X T
A E L E O X M S P R T A E E L M O P R S T X
A E L E O X M S P R T
A E L E O P M S X R T
A E L E O P M S X R T
A E L E O P M S X R T
145
Shellsort example: increments 7, 3, 1
input
S O R T E X A M P L E
7-sort
1-sort
S O R T E X A M P L E
M O R T E X A S P L E A E L E O P M S X R T
M O R T E X A S P L E A E L E O P M S X R T
M O L T E X A S P R E A E L E O P M S X R T
M O L E E X A S P R T A E E L O P M S X R T
A E E L O P M S X R T
3-sort
A E E L O P M S X R T
M O L E E X A S P R T A E E L M O P S X R T
E O L M E X A S P R T A E E L M O P S X R T
E E L M O X A S P R T A E E L M O P S X R T
E E L M O X A S P R T A E E L M O P R S X T
A E L E O X M S P R T A E E L M O P R S T X
A E L E O X M S P R T
A E L E O P M S X R T
A E L E O P M S X R T result

A E L E O P M S X R T A E E L M O P R S T X
145
Python Implementation

def sort(a):
3x+1 increment
N = len(a)
sequence

h = 1
while h < N//3: h = 3*h+1 # 1, 4, 13, 40, 121, 364, ...

while h >= 1: # h-sort the array.

for i in range(h,N):
j = i
insertion sort
while j >= h and a[j] < a[j-h]:
exch(a, j, j-h)
j -= h move to next

h = h//3 increment

146
input
Visual Trace
40-sorted

13-sorted

4-sorted

result

147
50 random items

algorithm position
h-sorted
current subsequence
http://www.sorting-algorithms.com/shell-sort other elements

148
50 random items

algorithm position
h-sorted
current subsequence
http://www.sorting-algorithms.com/shell-sort other elements

148
50 partially-sorted items

algorithm position
h-sorted
current subsequence
http://www.sorting-algorithms.com/shell-sort other elements

149
50 partially-sorted items

algorithm position
h-sorted
current subsequence
http://www.sorting-algorithms.com/shell-sort other elements

149
 Shellsort:
Hungarian (Sz kely) folk dance

https://www.youtube.com/watch?v=CmPA7zE8mx0 150
é
Which increment sequence to use?
• Powers of two. 1, 2, 4, 8, 16, 32, ...
No.

• Powers of two minus one. 1, 3, 7, 15, 31, 63, …

Maybe.

• 3x + 1. 1, 4, 13, 40, 121, 364, …

OK. Easy to compute.

152
Which increment sequence to use?

Sedgewick. 1, 5, 19, 41, 109, 209, 505, 929, 2161, 3905, …

Good. Tough to beat in empirical studies.

153
Which increment sequence to use?

Sedgewick. 1, 5, 19, 41, 109, 209, 505, 929, 2161, 3905, …

merging of (9 . 4i) – (9 . 2i) + 1

and 4i – (3 . 2i) + 1

Good. Tough to beat in empirical studies.

153
 Intuition
Proposition: An h-sorted array remains h-sorted after g-sorting it.

7-sort 3-sort

S O R T E X A M P L E M O L E E X A S P R T
M O R T E X A S P L E E O L M E X A S P R T
M O R T E X A S P L E E E L M O X A S P R T
M O L T E X A S P R E E E L M O X A S P R T
M O L E E X A S P R T A E L E O X M S P R T
A E L E O X M S P R T
A E L E O P M S X R T
A E L E O P M S X R T
A E L E O P M S X R T
A E L E O P M S X R T

154
 Analysis
• Proposition: The order of growth of the worst-case number of compares used by
shellsort with the 3x+1 increments is N 3/2.
• Property: The expected number of compares to shellsort a randomly-ordered array
using 3x+1 increments is….

N compares 2.5 N ln N 0.25 N ln 2 N N 1.3

5.000 93K 106K 91K 64K

10.000 209K 230K 213K 158K

20.000 467K 495K 490K 390K

40.000 1022K 1059K 1122K 960K

80.000 2266K 2258K 2549K 2366K

155
Why are we interested in shellsort?
Useful in practice.
• Fast unless array size is huge
(used for small subarrays).
• Tiny, xed footprint for code
(used in some embedded systems).
• Hardware sort prototype.

156
fi
Why are we interested in shellsort?
Useful in practice.
• Fast unless array size is huge
(used for small subarrays).
R, bzip2, /linux/kernel/groups.c

• Tiny, xed footprint for code

(used in some embedded systems).
• Hardware sort prototype. uClibc

156
fi
Why are we interested in shellsort?
Simple algorithm, nontrivial performance,
interesting questions.
• Asymptotic growth rate?
• Best sequence of increments?
• Average-case performance?

Lesson: Some good algorithms are still waiting discovery.

157
Why are we interested in shellsort?
Simple algorithm, nontrivial performance,
interesting questions.
• Asymptotic growth rate? open problem: nd a better

increment sequence

• Best sequence of increments?

• Average-case performance?

Lesson: Some good algorithms are still waiting discovery.

157
fi
 Shuf ing
fl
How to shuf e an array?
Goal. Rearrange array so that result is a
uniformly random permutation.
all permutations

equally likely

159
fl
How to shuf e an array?
Goal. Rearrange array so that result is a
uniformly random permutation.
all permutations

equally likely

160
fl
 Shuf e sort
1. Generate a random real number for each array entry.
2. Sort the array.

161
fl
 Shuf e sort
1. Generate a random real number for each array entry.
2. Sort the array.

0.8003 0.9706 0.9157 0.9649 0.1576 0.4854 0.1419 0.4218 0.9572

161
fl
 Shuf e sort
1. Generate a random real number for each array entry.
2. Sort the array.

0.1419 0.1576 0.4218 0.4854 0.8003 0.9157 0.9572 0.9649 0.9706

162
fl
 Shuf e sort
1. Generate a random real number for each array entry.
2. Sort the array.

0.1419 0.1576 0.4218 0.4854 0.8003 0.9157 0.9572 0.9649 0.9706

Proposition: Shuf e sort produces a uniformly random permutation.

assuming real numbers

uniformly at random (and no ties) 162

fl
fl
 Shuf ing done wrong
Microsoft antitrust probe by EU. Microsoft agreed to
provide a randomized ballot screen for users to select
browser in Windows 7.

http://www.browserchoice.eu (now of ine)

163
fl
fl
 Shuf ing done wrong
Microsoft antitrust probe by EU. Microsoft agreed to
provide a randomized ballot screen for users to select
browser in Windows 7.

http://www.browserchoice.eu (now of ine)

appeared last

50% of the time 163

fl
fl
 Shuf ing done wrong
Microsoft antitrust probe by EU. Microsoft agreed to
provide a randomized ballot screen for users to select
browser in Windows 7.
Solution? Implement shuf e sort by making comparator
always return a random answer.

164
fl
fl
 Shuf ing done wrong
Microsoft antitrust probe by EU. Microsoft agreed to
provide a randomized ballot screen for users to select
browser in Windows 7.
Solution? Implement shuf e sort by making comparator
always return a random answer.

Microsoft's implementation in Javascript

function RandomSort (a,b)

{
return (0.5 - Math.random()); browser comparator
} (should implement a total order)

164
fl
fl
 Knuth shuf e demo

• In iteration i, pick integer r between 0 and i uniformly at random.

• Swap a[i] and a[r].

165
fl
 Knuth shuf e
• In iteration i, pick integer r between 0 and i uniformly at random.
• Swap a[i] and a[r].

166
fl
 Knuth shuf e
• In iteration i, pick integer r between 0 and i uniformly at random.
• Swap a[i] and a[r].

i r

not yet seen

167
fl
 Knuth shuf e
• In iteration i, pick integer r between 0 and i uniformly at random.
• Swap a[i] and a[r].

i r

not yet seen

168
fl
 Knuth shuf e
• In iteration i, pick integer r between 0 and i uniformly at random.
• Swap a[i] and a[r].

i r

not yet seen

168
fl
 Knuth shuf e
• In iteration i, pick integer r between 0 and i uniformly at random.
• Swap a[i] and a[r].

shuf ed not yet seen

169
fl
fl
 Knuth shuf e
• In iteration i, pick integer r between 0 and i uniformly at random.
• Swap a[i] and a[r].

r i

shuf ed not yet seen

170
fl
fl
 Knuth shuf e
• In iteration i, pick integer r between 0 and i uniformly at random.
• Swap a[i] and a[r].

r i

shuf ed not yet seen

171
fl
fl
 Knuth shuf e
• In iteration i, pick integer r between 0 and i uniformly at random.
• Swap a[i] and a[r].

shuf ed not yet seen

172
fl
fl
 Knuth shuf e
• In iteration i, pick integer r between 0 and i uniformly at random.
• Swap a[i] and a[r].

r i

shuf ed not yet seen

173
fl
fl
 Knuth shuf e
• In iteration i, pick integer r between 0 and i uniformly at random.
• Swap a[i] and a[r].

r i

shuf ed not yet seen

174
fl
fl
 Knuth shuf e
• In iteration i, pick integer r between 0 and i uniformly at random.
• Swap a[i] and a[r].

shuf ed not yet seen

175
fl
fl
 Knuth shuf e
• In iteration i, pick integer r between 0 and i uniformly at random.
• Swap a[i] and a[r].

r i

shuf ed not yet seen

176
fl
fl
 Knuth shuf e
• In iteration i, pick integer r between 0 and i uniformly at random.
• Swap a[i] and a[r].

r i

shuf ed not yet seen

177
fl
fl
 Knuth shuf e
• In iteration i, pick integer r between 0 and i uniformly at random.
• Swap a[i] and a[r].

shuf ed not yet seen

178
fl
fl
 Knuth shuf e
• In iteration i, pick integer r between 0 and i uniformly at random.
• Swap a[i] and a[r].

r i

shuf ed not yet seen

179
fl
fl
 Knuth shuf e
• In iteration i, pick integer r between 0 and i uniformly at random.
• Swap a[i] and a[r].

r i

shuf ed not yet seen

180
fl
fl
 Knuth shuf e
• In iteration i, pick integer r between 0 and i uniformly at random.
• Swap a[i] and a[r].

shuf ed not yet seen

181
fl
fl
 Knuth shuf e
• In iteration i, pick integer r between 0 and i uniformly at random.
• Swap a[i] and a[r].

r i

shuf ed not yet seen

182
fl
fl
 Knuth shuf e
• In iteration i, pick integer r between 0 and i uniformly at random.
• Swap a[i] and a[r].

r i

shuf ed not yet seen

183
fl
fl
 Knuth shuf e
• In iteration i, pick integer r between 0 and i uniformly at random.
• Swap a[i] and a[r].

shuf ed not yet seen

184
fl
fl
 Knuth shuf e
• In iteration i, pick integer r between 0 and i uniformly at random.
• Swap a[i] and a[r].

r i

shuf ed not yet seen

185
fl
fl
 Knuth shuf e
• In iteration i, pick integer r between 0 and i uniformly at random.
• Swap a[i] and a[r].

r i

shuf ed not yet seen

186
fl
fl
 Knuth shuf e
• In iteration i, pick integer r between 0 and i uniformly at random.
• Swap a[i] and a[r].

shuf ed not yet seen

187
fl
fl
 Knuth shuf e
• In iteration i, pick integer r between 0 and i uniformly at random.
• Swap a[i] and a[r].

r i

shuf ed not yet seen

188
fl
fl
 Knuth shuf e
• In iteration i, pick integer r between 0 and i uniformly at random.
• Swap a[i] and a[r].

r i

shuf ed not yet seen

189
fl
fl
 Knuth shuf e
• In iteration i, pick integer r between 0 and i uniformly at random.
• Swap a[i] and a[r].

r i

shuf ed not yet seen

189
fl
fl
 Knuth shuf e
• In iteration i, pick integer r between 0 and i uniformly at random.
• Swap a[i] and a[r].

shuf ed not yet seen

190
fl
fl
 Knuth shuf e
• In iteration i, pick integer r between 0 and i uniformly at random.
• Swap a[i] and a[r].

r i

shuf ed

191
fl
fl
 Knuth shuf e
• In iteration i, pick integer r between 0 and i uniformly at random.
• Swap a[i] and a[r].

r i

shuf ed

192
fl
fl
 Knuth shuf e
• In iteration i, pick integer r between 0 and i uniformly at random.
• Swap a[i] and a[r].

shuf ed

193
fl
fl
 Knuth Shuf e

Proposition: [Fisher-Yates 1938] Knuth

shuf ing algorithm produces a uniformly
random permutation of the input array in
linear time. assuming integers uniformly at

random

194
fl
fl
 Knuth Shuf e
• In iteration i, pick integer r between 0
and i uniformly at random.
• Swap a[i] and a[r]. common bug: between 0 and N – 1

correct variant: between i and N – 1

def shuffle(a):
N = len(a)
for i in range(N): between 0 and i
r = random.randint(0, i)
exch(a, i, r)

195
fl
 Broken Knuth shuf e
Q. What happens if integer is chosen between 0 and N-1 ?
A. Not uniformly random!

permutation Knuth shuf e broken shuf e

ABC 1/6 4/27

ACB 1/6 5/27

BAC 1/6 5/27

BCA 1/6 5/27

CAB 1/6 4/27

CBA 1/6 4/27

196
fl
fl
fl
 Shuf ing done wrong 2
Texas hold'em poker. Software must shuf e electronic cards.

How We Learned to Cheat at Online Poker: A Study in Software Security

http://www.datamation.com/entdev/article.php/616221
197
fl
fl
 for i := 1 to 52 do begin
r := random(51) + 1; between 1 and 51
Shuf ing algorithm in FAQ at
swap := card[r];
www.planetpoker.com
card[r] := card[i];
card[i] := swap;
end;

• Bug 1: Random number r never 52 52nd card can't end up in 52nd place.

• Bug 2: Shuf e not uniform (should be between 1 and i).

• Bug 3: random() uses 32-bit seed 232 possible shuf es.

• Bug 4: Seed = milliseconds since midnight 86.4 million shuf es.

198
fl
fl
fl
fl
 for i := 1 to 52 do begin
r := random(51) + 1; between 1 and 51
Shuf ing algorithm in FAQ at
swap := card[r];
www.planetpoker.com
card[r] := card[i];
card[i] := swap;
end;

• Bug 1: Random number r never 52 52nd card can't end up in 52nd place.

• Bug 2: Shuf e not uniform (should be between 1 and i).

• Bug 3: random() uses 32-bit seed 232 possible shuf es.

• Bug 4: Seed = milliseconds since midnight 86.4 million shuf es.

“ The generation of random numbers is too important to be left to chance. ”

— Robert R. Coveyou
198
fl
fl
fl
fl
 Best practices for shuf ing
(if your business depends on it).
• Use a hardware random-number generator that has passed both
the FIPS 140-2 and the NIST statistical test suites.
• Continuously monitor statistic properties:
hardware random-number generators are fragile and fail silently.
• Use an unbiased shuf ing algorithm.

Bottom line: Shuf ing a deck of cards is hard!

199
fl
fl
fl
 Summary
algorithm best average worst

selection sort N2 N2 N2

insertion sort N N2 N2

Shellsort (3x+1) N log N ? N 3/2

order of growth of running time to sort an array of N items

Next sessions. N log N sorting algorithms (in worst case).

200
 Summary
algorithm best average worst

selection sort N2 N2 N2

insertion sort N N2 N2

Shellsort (3x+1) N log N ? N 3/2

goal N N log N N log N

order of growth of running time to sort an array of N items

Next sessions. N log N sorting algorithms (in worst case).

200
 R EADING L IST :

2.2 M ERGESORT

h tt p : / / a l g s 4 . c s . p r i n c e t o n . e d u

201