CSC510-04 Analysis of Algorithms

Branch and Bound Algorithms

Jose Ortiz
jortizco@sfsu.edu

Copyright Jose Ortiz

 Overview
• 0-1 Knapsack
• Best First. Also know as Least Cost (LC)
• Depth First
• Breadth First
• Traveling Salesman

2 Copyright Jose Ortiz

What are branch and bound algorithms ?
• Branch and bound algorithms exploits the Dynamic Programing
Principle. However, it makes a passive use of this principle. Branch and
bound relies on the bounding principle from optimization.

• Conceptual example:

• Let’s say that you are exploring two sets S1 and S2 and the rules for
the upper and lower bounds of the optimal solutions are known. Then,
if S1 includes the lower and upper bounds, and if the upper bound of
the solutions from S1 is less than the lower bound for the solutions of
S2, then it is not worth exploring if S2 contains optimal solutions.

3 Copyright Jose Ortiz

0-1 Knapsack Problem
Branch and Bound

4 Copyright Jose Ortiz

 0-1 Knapsack Problem
Given a list of items item = {1,2,3,4}, their weights w = {3,4,2,5}
and their pro t p = {3,8,1,15}. Determine the number of items to
include in a knapsack so that the total weight is at most w = 11, and
the total pro t is as large as possible. (maximized pro t)

Max
Capacity
w = 11 kg

5 Copyright Jose Ortiz

fi
fi
fi
 0-1 Knapsack Problem Best-Case
upper bound = − 12
W = 11
x1 = 1
∅
Items 1 2 3 4
cost = − 18
bound = − 12
Pro t 3 8 1 15

Weigths 3 4 2 5 1

Rules :
Branch chosen : best case
Bounds :
n

∑
cost = (−1) Pi + ( fractional) X1 included :
i 15
n cost = (−1)(3 + 8 + 1 + ( (2)) = − 18
5
∑
bound = (−1) Pi
bound = (−1)(3 + 8 + 1) = − 12
i
6 Copyright Jose Ortiz
fi
 0-1 Knapsack Problem Best-Case (minimization)
upper bound = − 24
W = 11
x1 = 1
∅ x1 = 0
Items 1 2 3 4
cost = − 18 cost = − 24
bound = − 12 bound = − 24
Pro t 3 8 1 15

Weigths 3 4 2 5 1 1

Rules :
For next iteration, we consider only the minimum upper bound
Branch chosen : best case
Bounds :
n X1 not included :
∑
cost = (−1) Pi + ( fractional)
cost = (−1)(0 + 8 + 1 + 15) = − 24
i
n bound = (−1)(0 + 8 + 1 + 15) = − 24
∑
bound = (−1) Pi upper_bound = min{bound(x1 = 1), bound(x1 = 0)} = − 24
i
7 Copyright Jose Ortiz
fi
 0-1 Knapsack Problem Best-Case (minimization)
upper bound = − 24
W = 11 x1 = 1 ∅ x1 = 0
cost = − 18 cost = − 24
bound = − 24
bound = − 12
Items 1 2 3 4
1 x2 = 1 1 x2 = 0
Pro t 3 8 1 15 cost = − 24 cost = − 16
bound = − 24 bound = − 16

Weigths 3 4 2 5 2 2

Rules :
Branch chosen : best case
Bounds :
n X2 not included :
∑
cost = (−1) Pi + ( fractional)
cost = (−1)(0 + 0 + 1 + 15) = − 16
i
n bound = (−1)(0 + 0 + 1 + 15) = − 16
∑
bound = (−1) Pi upper_bound = min{bound(x1 = 1), bound(x1 = 0)} = − 24
i
8 Copyright Jose Ortiz
fi
 0-1 Knapsack Problem Best-Case (minimization)
upper bound = − 24
W = 11 x1 = 1 ∅ x1 = 0
cost = − 18 cost = − 24
bound = − 24
bound = − 12
Items 1 2 3 4
x2 = 0
1 x2 = 1 1 cost = − 16
Pro t 3 8 1 15 cost = − 24 bound = − 16
bound = − 24
Weigths 3 4 2 5 2 2
x3 = 1 x3 = 0
upper = − 24 cost = − 23
bound = − 24 bound = − 23
Rules : 3 3

Branch chosen : best case

Bounds :
n X3 not included :
∑
cost = (−1) Pi + ( fractional) cost = (−1)(0 + 8 + 0 + 15) = − 23
i
n bound = (−1)(0 + 8 + 0 + 15) = − 23
∑
bound = (−1) Pi upper_bound = min{bound(x1 = 1), bound(x1 = 0)} = − 24
i
9 Copyright Jose Ortiz
fi
 0-1 Knapsack Problem Best-Case (minimization)
upper bound = − 24
W = 11 x1 = 1 ∅ x1 = 0
cost = − 18 cost = − 24
bound = − 24
bound = − 12
Items 1 2 3 4
x2 = 0
1 x2 = 1 1 cost = − 16
Pro t 3 8 1 15 cost = − 24 bound = − 16
bound = − 24
x3 = 0 2
Weigths 3 4 2 5 x3 = 1 2
cost = − 23
cost = − 24 bound = − 23
bound = − 24
3
Rules : x4 = 1 3 x4 = 0
cost = − 24 cost = − 12
bound = − 24 bound = − 12
Branch chosen : best case
4 4
Bounds :
n X4 not included :
∑
cost = (−1) Pi + ( fractional) cost = (−1)(3 + 8 + 1 + 0) = − 12
i
n upper = (−1)(3 + 8 + +1 + 0) = − 12
∑
bound = (−1) Pi upper_bound = min{bound(x1 = 1), bound(x1 = 0)} = − 24
i
10 Copyright Jose Ortiz
fi
 0-1 Knapsack Problem Best-Case (minimization)
upper bound = − 24
W = 11 x1 = 1 ∅ x1 = 0
cost = − 18 cost = − 24
bound = − 24
bound = − 12
Items 1 2 3 4
x2 = 0
1 x2 = 1 1 cost = − 16
Pro t 3 8 1 15 cost = − 24 bound = − 16
bound = − 24
x3 = 0 2
Weigths 3 4 2 5 x3 = 1 2
cost = − 23
cost = − 24 bound = − 23
bound = − 24
3
Rules : x4 = 1 3 x4 = 0
cost = − 24 cost = − 12
bound = − 24 bound = − 12
Branch chosen : best case
4 4
Bounds :
n

∑
cost = (−1) Pi + ( fractional) n
Items included = {2,3,4} Total weight =
∑
i W2,3,4 = 11
n
n i
∑
Total profit = P2,3,4 = 24
∑
bound = (−1) Pi
i i

11 Copyright Jose Ortiz

fi
 0-1 Knapsack Problem Best-Case (minimization)
Time complexity Analysis
• This algorithm using the best-case technique has its worst time
n
complexity O(2 ) when we need to fully calculate the entire tree.
However, at best, the time complexity to nd a solution becomes linear
since it is inspecting the best-cases for every path, and rejecting the
rest of them.

• Still in the worst case, this algorithm performs better than backtracking
because some nodes are not explored based on their bounds and
infeasibility

• This algorithm is better implemented using a priority queue to keep

track of the best nodes.
12 Copyright Jose Ortiz
fi
 0-1 Knapsack Problem: Depth First
upper bound = − 3
W = 11
x1 = 1
cost = − 3 ∅
Items 1 2 3 4 bound = − 3

Pro t 3 8 1 15 1

Weigths 3 4 2 5

Rules :
Branch chosen : best case
Bounds :
n

∑
cost = (−1) Pi + ( fractional) X1 included :
i
n cost = (−1)(3) = − 3
∑
bound = (−1) Pi bound = (−1)(3) = − 3
i
13 Copyright Jose Ortiz
fi
 0-1 Knapsack Problem: Depth First
upper bound = − 11
W = 11 x1 = 1
cost = − 3 ∅
Items 1 2 3 4 bound = − 3
x2 = 1
Pro t 3 8 1 15 cost = − 11 1
bound = − 11
Weigths 3 4 2 5 2

Rules :
Branch chosen : best case
Bounds :
n

∑
cost = (−1) Pi + ( fractional) X2 included :
i
n cost = (−1)(3 + 8) = − 11
∑
bound = (−1) Pi bound = (−1)(3 + 8) = − 11
i
14 Copyright Jose Ortiz
fi
 0-1 Knapsack Problem: Depth First
upper bound = − 12
W = 11 x1 = 1
cost = − 3 ∅
Items 1 2 3 4 bound = − 3
x2 = 1
cost = − 11 1
Pro t 3 8 1 15
bound = − 11

Weigths 3 4 2 5 x3 = 1
cost = − 12 2
bound = − 12

Rules : 3
Branch chosen : best case
Bounds :
n

∑
cost = (−1) Pi + ( fractional) X3 included :
i
n cost = (−1)(3 + 8 + 1) = − 12
∑
bound = (−1) Pi bound = (−1)(3 + 8 + 1) = − 12
i
15 Copyright Jose Ortiz
fi
 0-1 Knapsack Problem: Depth First
upper bound = − 12
W = 11 x1 = 1
cost = − 3 ∅
Items 1 2 3 4 bound = − 3
x2 = 1
cost = − 11 1
Pro t 3 8 1 15
bound = − 11
x3 = 1
Weigths 3 4 2 5 cost = − 3 2
bound = − 3
x4 = 1
Rules : cost = − 18 3
bound = − 12
Branch chosen : best case
4
Bounds :
n
X4 included :
∑
cost = (−1) Pi + ( fractional)
15
i cost = (−1)(3 + 8 + 1 + ( (2)) = − 18
n 5
∑
bound = (−1) Pi bound = (−1)(3 + 8 + 1) = − 12
i
16 Copyright Jose Ortiz
fi
 0-1 Knapsack Problem: Depth First
W = 11 x1 = 1 x1 = 0
upper bound = − 24
cost = − 3 ∅ cost = − 24
Items 1 2 3 4 bound = − 3 bound = − 24
x2 = 1
Pro t 3 8 1 15
cost = − 11 1 1 ∅
bound = − 11 x2 = 0 x2 = 1
x3 = 1 cost = − 24
Weigths 3 4 2 5 cost = − 3 2 4 bound = − 24
bound = − 3 x3 = 0 2
x4 = 4 x3 = 1
Rules : 4 cost = − 24
cost = − 18 3 bound = − 24
3
bound = − 12 x4 = 0
Branch chosen : best case x4 = 1
cost = − 24
4 4 4
Bounds : bound = − 24
n
X1 (not included) and X2, X3, X4 (included) :
∑
cost = (−1) Pi + ( fractional)
i cost = (−1)(0 + 8 + 1 + 15) = − 24
n
bound = (−1)(0 + 8 + 1 + 15) = − 24
∑
bound = (−1) Pi
i upper_bound = min{bound(x1 = 1), bound(x1 = 0)} = − 24
17 Copyright Jose Ortiz
fi
 0-1 Knapsack Problem: Depth First
W = 11 x1 = 1 x1 = 0
upper bound = − 24
cost = − 3 ∅ cost = − 24
Items 1 2 3 4 bound = − 3 bound = − 24
x2 = 1 x2 = 0

Pro t 3 8 1 15
cost = − 11 1 1 ∅ cost = − 19
bound = − 19
bound = − 11 x2 = 0 x2 = 1
x3 = 1 2
Weigths 3 4 2 5 cost = − 24
cost = − 3 2 2 bound = − 24
bound = − 3 2 x3 = 0
x3 = 0 cost = − 19
x4 = 4 x3 = 1 bound = − 11
Rules : cost = − 18 3 3 cost = − 24
bound = − 24 3
3
bound = − 12 x4 = 0 x4 = 0
Branch chosen : best case x4 = 1
cost = − 12
4 cost = − 24 bound = − 12
4 4 bound = − 24 4
Bounds :
n

∑
cost = (−1) Pi + ( fractional) n
Items included = {2,3,4} Total weight =
∑
i W2,3,4 = 11
n
n i
∑
Total profit = P2,3,4 = 24
∑
bounc = (−1) Pi
i i

18 Copyright Jose Ortiz

fi
 0-1 Knapsack Problem Depth-First
Time complexity Analysis

• Since this approach needs to rst compute all the included nodes
(same as backtracking) the bound is only computed on the last
included node, and the number of computations increases in relation
to the branch and bound (best-case)

n
• Worst case scenario for this approach is still O(2 )

19 Copyright Jose Ortiz

fi
 0-1 Knapsack Problem: Breadth-First
upper bound = − 24
W = 11
x1 = 1
∅ x1 = 0
Items 1 2 3 4
cost = − 18 cost = − 24
bound = − 12 bound = − 24
Pro t 3 8 1 15

Weigths 3 4 2 5 1 2

Rules :
Branch chosen : best case
Bounds :
n X1 not included :
∑
cost = (−1) Pi + ( fractional)
cost = (−1)(0 + 8 + 1 + 15) = − 24
i
n bound = (−1)(0 + 8 + 1 + 15) = − 24
∑
bound = (−1) Pi upper_bound = min{bound(x1 = 1), bound(x1 = 0)} = − 24
i
20 Copyright Jose Ortiz
fi
 0-1 Knapsack Problem: Breadth-First
upper bound = − 24
W = 11
x1 = 1 x1 = 0
cost = − 18
∅ cost = − 24
Items 1 2 3 4 bound = − 24
bound = − 12

Pro t 3 8 1 15 x2 = 1 1 x2 = 0
cost = − 18 x2 = 0 x2 = 1 1 cost = − 16
cost = − 18 cost = − 24
bound = − 12 bound = − 16
Weigths 3 4 2 5 bound = − 12 bound = − 24
2 2 2 2

Rules :
Branch chosen : best case
Bounds :
n

∑
cost = (−1) Pi + ( fractional)
i
n

∑
bound = (−1) Pi
i
21 Copyright Jose Ortiz
fi
 0-1 Knapsack Problem: Breadth-First
upper bound = − 24
W = 11
x1 = 1 x1 = 0
cost = − 18
∅ cost = − 24
Items 1 2 3 4 upper = − 24
bound = − 12

Pro t 3 8 1 15 x2 = 1 1 x2 = 0
cost = − 18 x2 = 0 x2 = 1 1 cost = − 16
cost = − 19
bound = − 12 cost = − 24 bound = − 16
Weigths 3 4 2 5 bound = − 19 bound = − 24
2 2 2 2
x3 = 0
Rules : x3 = 1 x3 = 0 x3 = 1
cost = − 19 3 3
Branch chosen : best case bound = − 19
3 3
x3 = 1 x3 = 1
cost = − 15
Bounds : bound = − 15
cost = − 24
bound = − 24
n

∑
cost = (−1) Pi + ( fractional)
i
n

∑
bound = (−1) Pi
i
22 Copyright Jose Ortiz
fi
 0-1 Knapsack Problem: Breadth-First
upper bound = − 24
x1 = 1 x1 = 0
∅ cost = − 24
W = 11 cost = − 18
upper = − 24
bound = − 12
Items 1 2 3 4 x2 = 1 1 x2 = 0
cost = − 18 x2 = 0 x2 = 1 1 cost = − 16
cost = − 19
Pro t 3 8 1 15 bound = − 12 cost = − 24 bound = − 16
bound = − 19 bound = − 24
2 2 x3 = 0 x3 = 1 2 2
Weigths 3 4 2 5 cost = − 15 cost = − 24
bound = − 15
bound = − 24 x3 = 0
x3 = 1
cost = − 19 3 3 3
Rules : bound = − 19
3

Branch chosen : best case x4 = 1

4
cost = − 19 4 4 4 x4 = 0
Bounds : bound = − 19 x4 = 0 x4 = 1 cost = − 9
n cost = − 4 cost = − 24 bound = − 9
bound = − 4 bound = − 24
∑
cost = (−1) Pi + ( fractional) n
Items included = {2,3,4} Total weight =
∑
i W2,3,4 = 11
n n
i
∑
upper = (−1)
∑
Pi Total profit = P2,3,4 = 24
i i
23 Copyright Jose Ortiz
fi
 0-1 Knapsack Problem Breadth-First
Time complexity Analysis

• This approach needs to compute nodes at each level. This means

that until the algorithm doesn’t reach the last level, a decision
n
cannot be made. Worst case scenario for this approach is still O(2 )

24 Copyright Jose Ortiz

 0-1 Knapsack Problem Brach and Bound
Summary

From all the brach and bound approaches to solve this problem, the
best-case approach is the best suited since it only considers good
nodes with maximum pro t and rejects all the unfeasible paths. Thus,
by using the best-case approach, the number of computations made
by the algorithm decreases in relation to Depth-First and Breadth-First
ones.

25 Copyright Jose Ortiz

fi
Traveling Salesperson Problem
Branch and Bound
Least Cost

26 Copyright Jose Ortiz

 TSP Branch and Bound
• Problem Statement

• Given a a list of cities and the distances between each pair of cities, what is the shortest
possible route that visits each city exactly once and returns to the original city?

• Brute force (next slide) will compute all the possible permutations in order to get the shortest
possible route in O(n!). Not good!

• We can do slightly better using dynamic programming since it will take only into account the
2 n
minimum shortest paths computed so far in O(n 2 ). This may not seem a great improvement.
However, it will make the problem more feasible with graphs of at most 20 nodes
approximately using a typical computer.

• The branch and bound technique for solving this problem uses a breadth- rst approach to
display the nodes, and best-case to select the minimum cost path for each node

fi
 TSP Branch and Bound
Here is the adjacently matrix for the A B C D
traveling salesman problem representing
a graph of four vertices A ∞ 25 10 15

Find the shortest possible route that B 25 ∞ 10 45

visits each vertex exactly once, and
returns to the source vertex. Consider for C 10 10 ∞ 5

this problem source vertex to be V0 using

D 15 45 5 ∞
Branch and Bound (Least Cost)
TSP Branch and Bound (Least Cost)
1. For each row, compute the 2. For each cell, reduce its value using the following
minimum value formula ReducedVal = ActualVal − MinVal

A B C D Min A B C D
A ∞ 25 10 15 10 A ∞ 15 0 5
B 25 ∞ 10 45 10 B 15 ∞ 0 35
C 10 10 ∞ 5 5 C 5 5 ∞ 0
D 15 45 5 ∞ 5 D 10 40 0 ∞

3. For each column, compute the 4. For each column, reduce its value using the following
minimum value formula ReducedVal = ActualVal − MinVal
A B C D
A B C D
A ∞ 15 0 5
A ∞ 10 0 5
B 15 ∞ 0 35
C 5 5 ∞ 0 B 10 ∞ 0 35
D 10 40 0 ∞ C 0 0 ∞ 0
Min 5 5 0 0 D 5 35 0 ∞

∑ ∑
ReducedCost(A) = MinCol + MinRow = 10 + 30 = 40
TSP Branch and Bound (Least Cost)
Reduced Matrix
A -> D
upper = 55
A B C D A
A B C D
40
A ∞ 10 0 5 A ∞ ∞ ∞ ∞
B 10 ∞ 0 35 B 10 ∞ 0 ∞
C 0 0 ∞ 0 C 0 0 ∞ ∞
D 5 35 0 ∞ D ∞ 35 0 ∞ B D
50 C
45
A -> B A -> D ->B 55
A B C D A B C D
A ∞ ∞ ∞ ∞ A ∞ ∞ ∞ ∞
B ∞ ∞ 0 35 B ∞ ∞ 0 ∞
C 0 ∞ ∞ 0 C 0 ∞ ∞ ∞ B C
D 5 ∞ 0 ∞ D ∞ ∞ ∞ ∞ 80 55
A -> D -> C
A ->C
A B C D A B C D
A ∞ ∞ ∞ ∞ A ∞ ∞ ∞ ∞ Minimum Path: A − D − C − B − A = 55
B 0 ∞ ∞ 25 B 0 ∞ 0 ∞
C ∞ 0 ∞ 0 C ∞ ∞ ∞ ∞
D 0 30 ∞ ∞ D ∞ ∞ ∞ ∞
 TSP Branch and Bound Summary
• The best-case approach together with a breadth- rst search give pretty good results
in this problem because we reject unpromising vertices before exploring in deep the
tour for that vertex.

• The time complexity of this problem is still similar to the brute force one. However, in
practice, branch and bound performs better depending on the bounding function
given

• TSP still is one of the most interesting problems in algorithms and one of the most
researched.

fi