Data Structures

CS201
Time Complexity
Instructor: Atif Khattak
Algorithm
• Finite sequence of instructions.

• Each instruction having a clear meaning.

• Each instruction requiring finite amount of effort.

• Each instruction requiring finite time to complete.

FAST-NUCES Faisalabad-Chiniot Campus 12/9/2018

 Algorithm
• Finite sequence of instructions.
An input should not take the program in an infinite loop

• Each instruction having a clear meaning.

Very subjective. What is clear to me, may not be clear to you.

• Each instruction requiring finite amount of effort.

Very subjective. Finite on a super computer or a P4?

• Each instruction requiring finite time to complete.

Very subjective. 1 min, 1 hr, 1 year or a lifetime?

FAST-NUCES Faisalabad-Chiniot Campus 12/9/2018

Algorithm Definition
A finite set of statements that guarantees an
optimal solution in finite interval of time

FAST-NUCES Faisalabad-Chiniot Campus 12/9/2018

Good Algorithms?
 Run in less time
 Consume less memory

But computational resources (time

complexity) is usually more important

FAST-NUCES Faisalabad-Chiniot Campus 12/9/2018

Measuring Efficiency
 The efficiency of an algorithm is a measure of
the amount of resources consumed in solving a
problem of size n.
 The resource we are most interested in is time
 We can use the same techniques to analyze the consumption
of other resources, such as memory space.
 It would seem that the most obvious way to
measure the efficiency of an algorithm is to run it
and measure how much processor time is
needed
 Is it correct
FAST-NUCES Faisalabad-Chiniot Campus 12/9/2018
Factors
 Hardware
 Operating System
 Compiler
 Size of input
 Nature of Input
 Algorithm

Which should be improved?

FAST-NUCES Faisalabad-Chiniot Campus 12/9/2018
Running Time of an Algorithm
 Depends upon
 Input Size
 Nature of Input

 Generally time grows with size of input, so

running time of an algorithm is usually
measured as function of input size.
 Running time is measured in terms of
number of steps/primitive operations
performed
 Independent from machine, OS
FAST-NUCES Faisalabad-Chiniot Campus 12/9/2018
Finding running time of an Algorithm /
Analyzing an Algorithm
 Running time is measured by number of
steps/primitive operations performed
 Steps means elementary operation like
 ,+, *,<, =, A[i] etc
 We will measure number of steps taken in
term of size of input

FAST-NUCES Faisalabad-Chiniot Campus 12/9/2018

Simple Example (1)
// Input: int A[N], array of N integers
// Output: Sum of all numbers in array A

int Sum(int A[], int N)

{
1. int s=0;
2. for (int i=0; i< N; i++)
3. s = s + A[i];
4. return s;
}
How should we analyse this?

FAST-NUCES Faisalabad-Chiniot Campus 12/9/2018

Simple Example (2)
// Input: int A[N], array of N integers
// Output: Sum of all numbers in array A

int Sum(int A[], int N){

int s=0; 1
for (int i=0; i< N; i++)
2 3 4
s = s + A[i];
5 6 7 1,2,8: Once
return s;
} 3,4,5,6,7: Once per each iteration
8 of for loop, N iteration
Total: 5N + 3
The complexity function of the
algorithm is : f(N) = 5N +3

FAST-NUCES Faisalabad-Chiniot Campus 12/9/2018

Simple Example (3)
Growth of 5n+3
Estimated running time for different values of N:

N = 10 => 53 steps
N = 100 => 503 steps
N = 1,000 => 5003 steps
N = 1,000,000 => 5,000,003 steps

As N grows, the number of steps grow in linear proportion to N for

this function “Sum”

FAST-NUCES Faisalabad-Chiniot Campus 12/9/2018

What Dominates in Previous Example?
What about the +3 and 5 in 5N+3?
 As N gets large, the +3 becomes insignificant
 5 is inaccurate, as different operations require varying
amounts of time and also does not have any
significant importance

What is fundamental is that the time is linear in N.

Asymptotic Complexity: As N gets large,
concentrate on the highest order term:
 Drop lower order terms such as +3
 Drop the constant coefficient of the highest
order term i.e. N
FAST-NUCES Faisalabad-Chiniot Campus 12/9/2018
Be careful to differentiate
between
Performance: how much time/memory/disk/... is
actually used when a program is run. This
depends on the machine, compiler, etc. as well
as the code.

Complexity: how do the resource requirements of

a program or algorithm scale, i.e., what happens
as the size of the problem being solved gets
larger.

FAST-NUCES Faisalabad-Chiniot Campus 12/9/2018

Asymptotic Complexity
 The 5N+3 time bound is said to "grow
asymptotically" like N
 This gives us an approximation of the
complexity of the algorithm
 Ignores lots of (machine dependent)
details, concentrate on the bigger picture

FAST-NUCES Faisalabad-Chiniot Campus 12/9/2018

Comparing Functions: Asymptotic Notation
 Big Oh Notation: Upper bound
 Omega Notation: Lower bound
 Theta Notation: Tighter bound

FAST-NUCES Faisalabad-Chiniot Campus 12/9/2018

Big Oh Notation
If f(N) and g(N) are two complexity functions, we say

f(N) = O(g(N))

(read "f(N) is order g(N)", or "f(N) is big-O of g(N)")

if there are constants c and N0 such that for N > N0,
f(N) > c * g(N)
for all sufficiently large N.
 O(f(n)) =
{g(n) : there exists positive constants c and n0 such that 0 <= g(n) <=
c f(n) }
 O(f(n)) is a set of functions.
 n = O(n2) means that function n belongs to the set of functions O(n2)

FAST-NUCES Faisalabad-Chiniot Campus 12/9/2018

 Example : Comparing Functions
4000

 Which function is 3500

better?
3000
10 n2 Vs n3
2500

10 n^2
2000
n^3

1500

1000

500

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

FAST-NUCES Faisalabad-Chiniot Campus 12/9/2018

Comparing Functions
 As inputs get larger, any algorithm of a smaller order will be more
efficient than an algorithm of a larger order

0.05 N2 = O(N2)
Time (steps)

3N = O(N)

N = 60 Input (size)

FAST-NUCES Faisalabad-Chiniot Campus 12/9/2018

Big-Oh Notation
 Even though it is correct to say “7n - 3 is O(n3)”,
a better statement is “7n - 3 is O(n)”, that is, one should make the
approximation as tight as possible
 Simple Rule:

Drop lower order terms and constant

factors
7n-3 is O(n)
8n2log n + 5n2 + n is O(n2log n)

FAST-NUCES Faisalabad-Chiniot Campus 12/9/2018

Performance Classification
f(n) Classification

1 Constant: run time is fixed, and does not depend upon n. Most instructions are executed once, or only a few times,
regardless of the amount of information being processed

log n Logarithmic: when n increases, so does run time, but much slower. Common in programs which solve large problems
by transforming them into smaller problems.

n Linear: run time varies directly with n. Typically, a small amount of processing is done on each element.

n log n When n doubles, run time slightly more than doubles. Common in programs which break a problem down into smaller
sub-problems, solves them independently, then combines solutions

n2 Quadratic: when n doubles, runtime increases fourfold. Practical only for small problems; typically the program
processes all pairs of input (e.g. in a double nested loop).

n3 Cubic: when n doubles, runtime increases eightfold

2n Exponential: when n doubles, run time squares. This is often the result of a natural, “brute force” solution.
Size does matter
What happens if we double the input size N?

N log2N 5N N log2N N2 2N
8 3 40 24 64 256
16 4 80 64 256 65536
32 5 160 160 1024 ~109
64 6 320 384 4096 ~1019
128 7 640 896 16384 ~1038
256 8 1280 2048 65536 ~1076

FAST-NUCES Faisalabad-Chiniot Campus 12/9/2018

Size does matter
 Suppose a program has run time O(n!)
and the run time for
n = 10 is 1 second

For n = 12, the run time is 2 minutes

For n = 14, the run time is 6 hours
For n = 16, the run time is 2 months
For n = 18, the run time is 50 years
For n = 20, the run time is 200 centuries

FAST-NUCES Faisalabad-Chiniot Campus 12/9/2018

Standard Analysis Techniques

 Constant time statements

 Analyzing Loops
 Analyzing Nested Loops
 Analyzing Sequence of Statements
 Analyzing Conditional Statements

FAST-NUCES Faisalabad-Chiniot Campus 12/9/2018

Constant time statements
 Simplest case: O(1) time statements
 Assignment statements of simple data types
int x = y;
 Arithmetic operations:
x = 5 * y + 4 - z;
 Array referencing:
A[j] = 5;
 Array assignment:
 j, A[j] = 5;
 Most conditional tests:
if (x < 12) ...

FAST-NUCES Faisalabad-Chiniot Campus 12/9/2018

Analyzing Loops
 Any loop has two parts:
 How many iterations are performed?
 How many steps per iteration?

int sum = 0,j;

for (j=0; j < N; j++)
sum = sum +j;
 Loop executes N times (0..N-1)
 4 = O(1) steps per iteration

 Total time is N * O(1) = O(N*1) = O(N)

FAST-NUCES Faisalabad-Chiniot Campus 12/9/2018
Analyzing Loops

 What about this for loop?

int sum =0, j;
for (j=0; j < 100; j++)
sum = sum +j;
 Loop executes 100 times
 4 = O(1) steps per iteration
 Total time is 100 * O(1) = O(100 * 1) =
O(100) = O(1)
FAST-NUCES Faisalabad-Chiniot Campus 12/9/2018
Analyzing Nested Loops
 Treat just like a single loop and evaluate each
level of nesting as needed:
int j,k;
for (j=0; j<N; j++)
for (k=N; k>0; k--)
sum += k+j;
 Start with outer loop:
 How many iterations? N
 How much time per iteration? Need to evaluate inner
loop
 Inner loop uses O(N) time
 Total time is N * O(N) = O(N*N) = O(N2)
FAST-NUCES Faisalabad-Chiniot Campus 12/9/2018
Analyzing Nested Loops
 What if the number of iterations of one
loop depends on the counter of the other?
int j,k;
for (j=0; j < N; j++)
for (k=0; k < j; k++)
sum += k+j;
 Analyze inner and outer loop together:
 Number of iterations of the inner loop is:
 0 + 1 + 2 + ... + (N-1) = O(N2)
FAST-NUCES Faisalabad-Chiniot Campus 12/9/2018
Analyzing Sequence of Statements
 For a sequence of statements, compute their
complexity functions individually and add them
up
for (j=0; j < N; j++)
for (k =0; k < j; k++) O(N2)
sum = sum + j*k;
for (l=0; l < N; l++)
O(N)
sum = sum -l;
cout<<“Sum=”<<sum; O(1)

Total cost is O(N2) + O(N) +O(1) = O(N2)

SUM RULE

FAST-NUCES Faisalabad-Chiniot Campus 12/9/2018

Analyzing Conditional Statements
What about conditional statements such as

if (condition)
statement1;
else
statement2;
where statement1 runs in O(N) time and statement2 runs in O(N2)
time?

We use "worst case" complexity: among all inputs of size N, that is the
maximum running time?
The analysis for the example above is O(N2)

FAST-NUCES Faisalabad-Chiniot Campus 12/9/2018

Time complexity familiar tasks
Task Growth rate
Getting a specific element from a list (array) O(1)
Dividing a list in half, dividing one halve in half, etc O(log2N)
Binary Search O(log2N)
Scanning (brute force search) a list O(N)
O(Nk)
Nested for loops (k levels)

MergeSort
O(N log2N)
BubbleSort O(N2)
Generate all subsets of a set of data O(2N)
Generate all permutations of a set of data O(N!)

FAST-NUCES Faisalabad-Chiniot Campus 12/9/2018

 Any questions?

FAST-NUCES Faisalabad-Chiniot Campus 12/9/2018