Computer Algorithm

1
TY IT 21-22
D B Kulkarni
Walchand College of Engineering, Sangli
Schedule

➢ Online synchronous due to Covid Pandemic

➢ 2 Lect/week
➢ Tue (10:15-11:15AM) , Thu (10:15-11:15AM)
➢ 60 mins session
○ 10 mins Doubts/discussion
○ 20 mins session I
○ 5 mins break
○ 20 mins session II
○ 5 QA
Algorithm

➢ What is an algorithm? –Pseudocode, logic

➢ Phases- Devise, Validate, Analyze, Test (DVAT)
➢ Properties:
○ Written in simple English, statement, description
○ Should accept all possible input- Generic
○ Platform (s/w, H/w independent)
○ Terminate
➢ Performance evaluation
○ Priori- Performance analysis
○ Posteriori- Performance measurement
Algorithm: Analysis

➢ Analysis: Primitive operation (comparison/item manipulations/#of use of

function(module))
○ Complexity
■ Time
■ Space
➢ Asymptotic notations:
○ Big O (upper bound)
○ Omega (lower bound)
○ Theta (both bound)
➢ Strategies
○ Iterative
○ Recursive
Algorithms: Examples

1. Finding largest and smallest

2. Finding largest and second largest
3. Knapsack problem

TY IT 20-21 CA DBK 10/22/2020

Examples: Finding largest and smallest

Finding largest and smallest

➢ (N-1) comparisons for each
➢ Total (2N-2)?
➢ (┌3N/2┐-2)
Examples: Finding largest and second
largest
Finding largest and second largest
➢ (N-1) comparisons to find largest, (N-2) for second largest= Total (3N-2)
➢ N-2+(┌log N┐)
Examples: Knapsack problem

➢ Knapsack problem
○ Knapsack (Bag)
○ Weight handling capacity of Knapsack M
○ Items to be inserted
○ Item (wheat, jowar, rice…) Weight and profit factor given
➢ Optimization
○ Best the one which carries max profit!!!
○ Best the one which has least weight!!!

TY IT 20-21 CA DBK 10/22/2020

Fractional Knapsack

TY IT 20-21 CA DBK 10/22/2020

 Fractional Knapsack: Example

Item (M=100) 1 2 3 4

60 10 30 80
30 40 10 20

Optimal:
Greedy Approach 1: Less weight: Sort items w˄- 2,3,1-p80
Greedy Approach 2: More profit: Sort items p˅-2,1,3/8(4)-p77.5
Correct Approach: Less weight/profit: Sort items (w/p) ˄-2,1,3-p80

10/22/2020
0-1 Knapsack problem

➢ Fraction of item not allowed

➢ Approach?
➢ You cannot break an item, either pick the complete item or don’t pick it
(0-1 property).

➢ Brute force method: Consider all subsets of weight where sum is <M, select
the one with highest profit
➢ Dynamic programming approach

10/22/2020
Dynamic programming Approach

➢ Determine the structure of an optimal solution

○ Decompose the problem into sub-problems
➢ Recursively define the value of an optimal solution.
➢ Compute the value of an optimal solution in a bottom-up fashion.
➢ Construct an optimal solution from computed information
●Encoding messages

◆ Encode a message composed of a string of characters

◆ Codes used by computer systems
▪ ASCII
• uses 8 bits per character
• can encode 256 characters
▪ Unicode
• 16 bits per character
• can encode 65536 characters
• includes all characters encoded by ASCII
◆ ASCII and Unicode are fixed-length codes
▪ all characters represented by same number of bits
Problems

◆ Suppose that we want to encode a message constructed from the

symbols A, B, C, D, and E using a fixed-length code
▪ How many bits are required to encode each symbol?
◆ at least 3 bits are required
◆ 2 bits are not enough (can only encode four symbols)
◆ How many bits are required to encode the message DEAACAAAAABA?
◆ there are twelve symbols, each requires 3 bits
◆ 12*3 = 36 bits are required
Drawbacks of fixed-length codes

◆ Wasted space
▪ Unicode uses twice as much space as ASCII
• inefficient for plain-text messages containing only ASCII characters
◆ Same number of bits used to represent all characters
▪ ‘a’ and ‘e’ occur more frequently than ‘q’ and ‘z’

◆ Potential solution: use variable-length codes

▪ variable number of bits to represent characters when frequency of occurrence is known
▪ short codes for characters that occur frequently
Advantages of variable-length codes

◆ The advantage of variable-length codes over fixed-length is short

codes can be given to characters that occur frequently
▪ on average, the length of the encoded message is less than fixed-length encoding
◆ Potential problem: how do we know where one character ends and
another begins?
• not a problem if number of bits is fixed!

A = 00
001011011100111111111
B = 01
1
C = 10
D = 11 ACDBADDDDD
Prefix property

◆ A code has the prefix property if no character code is the prefix (start of the
code) for another character
◆ Example: Symbol Code
P 000
Q 11
0100110110001
R 01 0
S 001 RSTQPT
T 10

◆ 000 is not a prefix of 11, 01, 001, or 10

◆ 11 is not a prefix of 000, 01, 001, or 10 …
Code without prefix property

◆ The following code does not have prefix property

Symbol Code
P 0
Q 1
R 01
S 10
T 11
◆ The pattern 1110 can be decoded as QQQP, QTP, QQS, or TS
Problem

◆ Design a variable-length prefix-free code such that the message

DEAACAAAAABA can be encoded using 22 bits
◆ Possible solution:
▪ A occurs eight times while B, C, D, and E each occur once
▪ represent A with a one bit code, say 0
• remaining codes cannot start with 0
▪ represent B with the two bit code 10
• remaining codes cannot start with 0 or 10
▪ represent C with 110
▪ represent D with 1110
▪ represent E with 11110
Encoded message

DEAACAAAAAB
A
Symbol Code
A 0
B 10
C 110
D 1110
E 11110

111011110001100000010 22
0 bits
Another possible code

DEAACAAAAAB
A
Symbol Code
A 0
B 100
C 101
D 1101
E 1111

110111110010100000100 22
0 bits
Better code

DEAACAAAAAB
A
Symbol Code
A 0
B 100
C 101
D 110
E 111

1101110010100000100 20
0 bits
What code to use?

◆ Question: Is there a variable-length code that makes the most efficient use
of space?

Answer:
Yes!
Huffman coding tree

◆ Binary tree
▪ each leaf contains symbol (character)
▪ label edge from node to left child with 0
▪ label edge from node to right child with 1
◆ Code for any symbol obtained by following path from root to the leaf
containing symbol
◆ Code has prefix property
▪ leaf node cannot appear on path to another leaf
▪ note: fixed-length codes are represented by a complete Huffman tree and clearly have the
prefix property
Building a Huffman tree

◆ Find frequencies of each symbol occurring in message

◆ Begin with a forest of single node trees
▪ each contain symbol and its frequency
◆ Do recursively
▪ select two trees with smallest frequency at the root
▪ produce a new binary tree with the selected trees as children and store the sum of their
frequencies in the root
◆ Recursion ends when there is one tree
▪ this is the Huffman coding tree
Example

◆ Build the Huffman coding tree for the message

This is his message
◆ Character frequencies
A G M T E H _ I S

1 1 1 1 2 2 3 3 5

◆ Begin with forest of single trees

1 1 1 1 2 2 3 3 5
A G M T E H _ I S
Step 1

1 1 1 1 2 2 3 3 5
A G M T E H _ I S
Step 2

2 2

1 1 1 1 2 2 3 3 5
A G M T E H _ I S
Step 3

2 2 4

1 1 1 1 2 2 3 3 5
A G M T E H _ I S
Step 4

2 2 4

1 1 1 1 2 2 3 3 5
A G M T E H _ I S
Step 5

2 2 4 6

1 1 1 1 2 2 3 3 5
A G M T E H _ I S
Step 6

4 4

2 2 2 2 6

E H

1 1 1 1 3 3 5
A G M T _ I S
Step 7

8 1
1

4 4 6 5
S

2 2 2 2 3 3

E H _ I

1 1 1 1
A G M T
Step 8
1
9

8 1
1

4 4 6 5
S

2 2 2 2 3 3

E H _ I

1 1 1 1
A G M T
Label edges

1
0 9 1

8 1
1
0 1
0 1

4 4 6 5
0 1 0 1 0 1 S

2 2 2 2 3 3
0 1 0 1 E H _ I

1 1 1 1
A G M T
Huffman code & encoded message

This is his message

S 11
E 010
H 011
_ 100
I 101
A 0000
G 0001
M 0010
T 0011

001101110111100101111000111011110000100101111000000010
10
 Matrix Multiplication
• Suppose we have A1, A2 matrices to be
multiplied
– That is, we want to compute the product A1*A2
– Orders of matrix
• A1(pxq)
• A2(rxs)
• For multiplication q=r
• They must be compatible
 Matrix-chain Multiplication
• Suppose we have a sequence or chain A1, A2, …, An of
n matrices to be multiplied
– That is, we want to compute the product A1A2…An

• There are many possible ways (parenthesizations) to

compute the product

11-38
 Matrix-chain Multiplication …contd

• Example: consider the chain A1, A2, A3, A4 of 4 matrices

– Let us compute the product A1A2A3A4
• There are 5 possible ways:
1. (A1(A2(A3A4)))
2. (A1((A2A3)A4))
3. ((A1A2)(A3A4))
4. ((A1(A2A3))A4)
5. (((A1A2)A3)A4)

11-39
 Matrix-chain Multiplication …contd

• To compute the number of scalar multiplications

necessary, we must know:
– Algorithm to multiply two matrices
– Matrix dimensions

• Can you write the algorithm to multiply two matrices?

11-40
 Algorithm to Multiply 2 Matrices
Input: Matrices Ap×q and Bq×r (with dimensions p×q and q×r)
Result: Matrix Cp×r resulting from the product A·B

MATRIX-MULTIPLY(Ap×q , Bq×r)
1. for i ← 1 to p
2. for j ← 1 to r
3. C[i, j] ← 0
4. for k ← 1 to q
5. C[i, j] ← C[i, j] + A[i, k] · B[k, j]
6. return C
Scalar multiplication in line 5 dominates time to compute CNumber of scalar
multiplications = pqr

11-41
 Matrix-chain Multiplication …contd

• Example: Consider three matrices A10×100, B100×5, and

C5×50
• There are 2 ways to parenthesize
– ((AB)C) = D10×5 · C5×50
• AB ⇒ 10·100·5=5,000 scalar multiplications Total:
• DC ⇒ 10·5·50 =2,500 scalar multiplications 7,500
– (A(BC)) = A10×100 · E100×50
• BC ⇒ 100·5·50=25,000 scalar multiplications
• AE ⇒ 10·100·50 =50,000 scalar multiplications
Total:
75,000 11-42
43 Matrix chain multiplication (MCM)

● For example, if A1 is a 10 × 30 matrix, A2 is a 30 × 5 matrix, and A3 is a 5 × 60

matrix, then
(A1A2) A3 🡪 (10×30×5) + (10×5×60) = 1500 + 3000 = 4500 operations
A1 (A2A3) 🡪 (30×5×60) + (10×30×60) = 9000 + 18000 = 27000 operations
● To multiply n matrices {A1,A2,A3…. An}, all combinations can be checked and
least one can be accepted….brute force method…..complexity 2^n
● Dynamic programming approach has complexity of n^3

TY IT 20-21 CA DBK 10/22/2020

 Matrix-chain Multiplication …contd

• Matrix-chain multiplication problem

– Given a chain A1, A2, …, An of n matrices, where for i=1, 2, …,
n, matrix Ai has dimension pi-1×pi
– Parenthesize the product A1A2…An such that the total number
of scalar multiplications is minimized
• Brute force method of exhaustive search takes time
exponential in n
 Dynamic Programming Approach
• The structure of an optimal solution
– Let us use the notation Ai..j for the matrix that results from the
product Ai Ai+1 … Aj
– An optimal parenthesization of the product A1A2…An splits the
product between Ak and Ak+1 for some integer k where1 ≤ k <
n
– First compute matrices A1..k and Ak+1..n ; then multiply them to
get the final matrix A1..n
 Dynamic Programming Approach …contd

– Key observation: parenthesizations of the subchains

A1A2…Ak and Ak+1Ak+2…An must also be optimal if the
parenthesization of the chain A1A2…An is optimal (why?)

– That is, the optimal solution to the problem contains within it

the optimal solution to subproblems
 Dynamic Programming Approach …contd
• Recursive definition of the value of an optimal solution
– Let m[i, j] be the minimum number of scalar multiplications
necessary to compute Ai..j
– Minimum cost to compute A1..n is m[1, n]
– Suppose the optimal parenthesization of Ai..j splits the product
between Ak and Ak+1 for some integer k where i ≤ k < j
 Dynamic Programming Approach …contd
– Ai..j = (Ai Ai+1…Ak)·(Ak+1Ak+2…Aj)= Ai..k · Ak+1..j
– Cost of computing Ai..j = cost of computing Ai..k + cost of
computing Ak+1..j + cost of multiplying Ai..k and Ak+1..j
– Cost of multiplying Ai..k and Ak+1..j is pi-1pk pj

– m[i, j ] = m[i, k] + m[k+1, j ] + pi-1pk pj for i ≤ k < j

– m[i, i ] = 0 for i=1,2,…,n
 Dynamic Programming Approach …contd
– But… optimal parenthesization occurs at one value of k
among all possible i ≤ k < j
– Check all these and select the best one

0 if i=j
m[i, j ] =
min {m[i, k] + m[k+1, j ] + pi-1pk pj } if i<j
i ≤ k< j
 Dynamic Programming Approach …contd
• To keep track of how to construct an optimal solution,
we use a table s
• s[i, j ] = value of k at which Ai Ai+1 … Aj is split for optimal
parenthesization
• Algorithm: next slide
– First computes costs for chains of length l=1
– Then for chains of length l=2,3, … and so on
– Computes the optimal cost bottom-up
 Algorithm to Compute Optimal Cost
Input: Array p[0…n] containing matrix dimensions and n
Result: Minimum-cost table m and split table s

MATRIX-CHAIN-ORDER(p[ ], n)
for i ← 1 to n Takes O(n3) time
m[i, i] ← 0
Requires O(n 2) space
for l ← 2 to n
for i ← 1 to n-l+1
j ← i+l-1
m[i, j] ← ∞
for k ← i to j-1
q ← m[i, k] + m[k+1, j] + p[i-1] p[k] p[j]
if q < m[i, j]
m[i, j] ← q
s[i, j] ← k
return m and s
 Constructing Optimal Solution
• Our algorithm computes the minimum-cost table m and
the split table s
• The optimal solution can be constructed from the split
table s
– Each entry s[i, j ]=k shows where to split the product Ai Ai+1 …
Aj for the minimum cost
 MCM Example..contd…
● For m[i,j] let s[i,j]=k for optimal splitting (index for which m[i,j] is minimum)
● Lets split matrix chain AiAi+1…. Aj into (Ai..Ak)*( Ak+1..Aj) and compute m[i,k] and m[k+1,j]
● For m[i,k] let s[i,k]=l for optimal splitting and for m[k+1,j] let s[k+1,j]=m be the value for
optimal splitting
● split matrix chain AiAi+1…. Ak into (Ai..Al)*( Al+1..Ak) and Ak+1Ak+2…. Aj into (Ak+1..Am)*( Am+1..Aj)
● Recurse
● Example
● Minimum from which m[1,4] is calculated, will give value of corresponding k
● Record this value as s[1,4]
● m[1,s[1,4]] OR m[s[1,4]+1,4]
● Parentisize
 MCM Example
● For four matrices with vector p=
{5,4,6,2,7} such that
{A1(5x4),A2(4x6),A3(6x2),A4(2x7)} find
anesthetization
● To find k such that m[1,4] is minimum

● m[i, j ] = m[i, k] + m[k+1, j ] + pi-1pk pj

 MCM Example
● m[i, j ] = m[i, k] + m[k+1, j ] + pi-1pk pj
 MCM Example..contd…
Compute Step 1 Step 2 Step 3
Phase 1 m[1,2],m[2,3],m[3,4] m[1,3], m[2,4] m[1,4]
Record s[1,4] s[1,s[1,4]] or s[s[1,4],4] Parentisize

s[1,4]=3, M[2,4] =min2<k<4{m[2,k]+m[k+1,4]+p1pkp4}

=min{(m[2,2]+m[3,4]+p1p2p4), (m[2,3]+m[4,4]+p1p3p4)} (A1(A2A3))A4
s[1,3]=1 =Min{(254),(104)}
 MCM Example
● m[i, j ] = m[i, k] + m[k+1, j ] + pi-1pk pj

M[2,4] =min2<k<4{m[2,k]+m[k+1,4]+p1pkp4}
=min{(m[2,2]+m[3,4]+p1p2p4),
(m[2,3]+m[4,4]+p1p3p4)}
=min{(120+84+4*6*7), (48+0+4*2*7)}
=Min{(254),(104)}
 Example
• Show how to multiply this matrix Matrix Dimension
chain optimally
A1 30×35
A2 35×15
• Solution ??
– Minimum cost 15,125 A3 15×5
– Optimal parenthesization
((A1(A2A3))((A4 A5)A6))
A4 5×10
A5 10×20
A6 20×25
 Longest common sub-sequence (LCS)
● Given two sequences {X,Y}, finding the longest subsequence common to both. X={x1,x2…xn}
Y={y1,y2…ym}
● Sub-sequences are different from sub-string
● Complexity NP Hard problem with complexity 2^(lenth of sequence)
● Let Z={z1,z2…zk} be the LCS of X and Y
● If zk=xn= ym, then zk-1 is LCS of Xn -1 and Ym -1
● xn<> ym
● If zk <> xn then then Zk is LCS of Xn -1 and Ym

● If zk <> ym , then zk is LCS of Xn and Ym -1

 LCS: Formula
● Let the two sequences be X(1,2..i) and Y(1,2..j)

● Then Longest common subsequence (LCS) of X and Y is given as

● LCS(i,j) = 0, if i=0 OR j=0

= 1+LCS(i−1,j−1), if X[i]=Y[j]
= max{LCS(i,j−1),LCS(i−1,j)}, if X[i]≠Y[j]

● To find
● Length of LCS
● LCS
 LCS: Example
Xi=5={A,C,A,D,B}, Yj=4 ={C,B,D,A}

LCS(i,j) = 0, if i=0 OR j=0

= 1+LCS(i−1,j−1), if X[i]=Y[j]
= max{LCS(i,j−1),LCS(i−1,j)}, if X[i]≠Y[j]

Yj y1=C y2=B y3=D y4=A

Xi 0 0 0 0 0
x1=A 0 0 0 0 1
x2=C 0 1 1 1 1
x3=A 0 1 1 1 2
x4=D 0 1 1 2 2
x5=B 0 1 2 2 2
63
LCS: Example X={A,C,A,D,B}, Y={C,B,D,A}

{C,A}{C,D}{C,B}
LCS:Homework Example

● X={A,B,C,B,D,A,B}, Y{B,D,C,A,B,A}