Algorithms & data structures

Branch-and-bound
Damjan Strnad

1 / 19
 Branch-and-bound
●
branch-and-bound is an improved backtracking method,
which is used mainly for solving discrete and combinatorial
optimization problems
●
we search for a solution in steps as with backtracking, but
explore the state space tree using breadth first search
●
while searching for the optimal solution we maintain the lower
(upper) bound of the optimal solution value
●
expansion of nodes is limited to those nodes, for which we
assess that they can still improve the value of optimal solution

2 / 19
 Branch-and-bound
●
general procedure: BRANCH_AND_BOUND(root,opt)
– returns only the value of 1. Q  empty queue
the best solution, can be 2. ENQUEUE(Q,root)
extended to return the 3. opt  VALUE(root)
best solution itself 4. while not EMPTY(Q) do
– for minimization 5. v  DEQUEUE(Q)
problems we change 6. for each successor u of v do
the inequality signs in 7. if FEASIBLE(u) then
rows 8 and 10 8. if VALUE(u) > opt then
– the value, bound and 9. opt  VALUE(u)
solution description is 10. if BOUND(u) > opt then
stored with each node 11. ENQUEUE(Q,u)

3 / 19
 Binary knapsack problem
●
binary or 0-1 knapsack:
– we want to fill a knapsack of capacity V with objects such
that the knapsack value is maximized
– for each of n objects we are given its volume vi and value ci
– an object can be added to the knapsack as a whole (xi=1)
or not added at all (xi=0)
– we are searching for argmax x {x⋅c } given x⋅v≤V
●
a greedy strategy does not guarantee optimal solution
irrespective of the object order (higher value per volume
first, higher value first, smaller volume first)
4 / 19
 Binary knapsack problem
●
example: v=[2, 3, 1, 6, 4], c=[30, 50, 60, 70, 40], V=8
– a greedy method with objects ordered by decreasing ratio
c/v would give solution x=[1,1,1,0,0] with value 140
– a greedy method with objects ordered by decreasing c
would give solution x=[0,0,1,1,0] with value 130
– a greedy method with objects ordered by increasing v
would give solution x=[1,1,1,0,0] with value 140
– optimal solution is x=[0,1,1,0,1] with value 150
●
a brute force method tests all combinations and selects a
feasible solution with highest value*
●
backtracking terminates branch exploration when partial
solution becomes infeasible* 5 / 19
 Binary knapsack problem
●
example: v=[2, 3, 1, 6, 4], c=[30, 50, 60, 70, 40], V=8
– a greedy method with objects ordered by decreasing ratio
c/v would give solution x=[1,1,1,0,0] with value 140
– a greedy method with objects ordered by decreasing c
would give solution x=[0,0,1,1,0] with value 130
– a greedy method with objects ordered by increasing v
would give solution x=[1,1,1,0,0] with value 140
– optimal solution is x=[0,1,1,0,1] with value 150
●
a brute force method tests all combinations and selects a
feasible solution with highest value*
●
backtracking terminates branch exploration when partial
solution becomes infeasible* 5 / 19
 Binary knapsack problem
●
example: v=[2, 3, 1, 6, 4], c=[30, 50, 60, 70, 40], V=8
– a greedy method with objects ordered by decreasing ratio
c/v would give solution x=[1,1,1,0,0] with value 140
– a greedy method with objects ordered by decreasing c
would give solution x=[0,0,1,1,0] with value 130
– a greedy method with objects ordered by increasing v
would give solution x=[1,1,1,0,0] with value 140
– optimal solution is x=[0,1,1,0,1] with value 150
●
a brute force method tests all combinations and selects a
feasible solution with highest value*
●
backtracking terminates branch exploration when partial
solution becomes infeasible* 5 / 19
 Binary knapsack problem
●
example: v=[2, 3, 1, 6, 4], c=[30, 50, 60, 70, 40], V=8
– a greedy method with objects ordered by decreasing ratio
c/v would give solution x=[1,1,1,0,0] with value 140
– a greedy method with objects ordered by decreasing c
would give solution x=[0,0,1,1,0] with value 130
– a greedy method with objects ordered by increasing v
would give solution x=[1,1,1,0,0] with value 140
– optimal solution is x=[0,1,1,0,1] with value 150
●
a brute force method tests all combinations and selects a
feasible solution with highest value*
●
backtracking terminates branch exploration when partial
solution becomes infeasible* 5 / 19
 Binary knapsack problem
●
example: v=[2, 3, 1, 6, 4], c=[30, 50, 60, 70, 40], V=8
●
with alphabetic object order we can calculate that adding
objects from:
– A on increases the solution value by at most 250
– B on increases the solution value by at most 220
– C on increases the solution value by at most 170
– D on increases the solution value by at most 110
– E on increases the solution value by at most 40
●
with branch-and-bound we terminate exploration of the
current branch when we cannot further improve the value
of currently best solution (among so far expanded 6 / 19
successors according to the breadth-first search)*
 Binary knapsack problem
●
example: v=[2, 3, 1, 6, 4], c=[30, 50, 60, 70, 40], V=8
●
with alphabetic object order we can calculate that adding
objects from:
– A on increases the solution value by at most 250
– B on increases the solution value by at most 220
– C on increases the solution value by at most 170
– D on increases the solution value by at most 110
– E on increases the solution value by at most 40
●
with branch-and-bound we terminate exploration of the
current branch when we cannot further improve the value
of currently best solution (among so far expanded 6 / 19
successors according to the breadth-first search)*
 Binary knapsack problem
●
efficiency of branch-and-bound strategy depends on
determining a good initial or current bound of optimal
solution
●
in the case of 0-1 knapsack the efficiency can be
potentially increased by using different object order:
– objects with higher value per volume unit first – generate
good solutions as soon as possible to determine tight
optimal bound
– objects with higher volume first – generate invalid
solutions as soon as possible

7 / 19
 Binary knapsack problem
●
example: v=[2, 3, 1, 6, 4], c=[30, 50, 60, 70, 40], V=8
●
let‘s order objects by decreasing volume:
– v'=[6, 4, 3, 2, 1], c'=[70, 40, 50, 30, 60], V=8

D E B A C
– adding objects from:
●
D on increases the solution value by at most 250
●
E on increases the solution value by at most 180
●
B on increases the solution value by at most 140
●
A on increases the solution value by at most 90
●
C on increases the solution value by at most 60
– in our case the search efficiency improves*
8 / 19
 Binary knapsack problem
●
example: v=[2, 3, 1, 6, 4], c=[30, 50, 60, 70, 40], V=8
●
let‘s order objects by decreasing volume:
– v'=[6, 4, 3, 2, 1], c'=[70, 40, 50, 30, 60], V=8

D E B A C
– adding objects from:
●
D on increases the solution value by at most 250
●
E on increases the solution value by at most 180
●
B on increases the solution value by at most 140
●
A on increases the solution value by at most 90
●
C on increases the solution value by at most 60
– in our case the search efficiency improves*
8 / 19
 Binary knapsack problem
●
example: v=[2, 3, 1, 6, 4], c=[30, 50, 60, 70, 40], V=8
●
let's order objects by decreasing value per volume unit:
– v'=[1, 3, 2, 6, 4], c'=[60, 50, 30, 70, 40], V=8

C B A D E
– adding objects from:
●
C on increases the solution value by at most 250
●
B on increases the solution value by at most 190
●
A on increases the solution value by at most 140
●
D on increases the solution value by at most 110
●
E on increases the solution value by at most 40
– in our case there is no efficiency improvement*
9 / 19
 Binary knapsack problem
●
example: v=[2, 3, 1, 6, 4], c=[30, 50, 60, 70, 40], V=8
●
let's order objects by decreasing value per volume unit:
– v'=[1, 3, 2, 6, 4], c'=[60, 50, 30, 70, 40], V=8

C B A D E
– adding objects from:
●
C on increases the solution value by at most 250
●
B on increases the solution value by at most 190
●
A on increases the solution value by at most 140
●
D on increases the solution value by at most 110
●
E on increases the solution value by at most 40
– in our case there is no efficiency improvement*
9 / 19
 Binary knapsack problem
●
example: v=[2, 3, 1, 6, 4], c=[30, 50, 60, 70, 40], V=8
●
let's order objects by decreasing value per volume unit:
– v'=[1, 3, 2, 6, 4], c'=[60, 50, 30, 70, 40], V=8

C B A D E
●
we obtain a tighter upper bound for each node value if we
calculate it in the following way:
– we add the values of next elements (in order) that can be
put to the knapsack as a whole
– we add the partial value of the last element as if we could
put a fraction of it in the knapsack
– a node is not expanded (i.e. we do not enqueue it), if its
upper bound is lower or equal to the best result obtained so
far* 10 / 19
 Binary knapsack problem
●
v'=[1, 3, 2, 6, 4], c'=[60, 50, 30, 70, 40], V=8
●
example – node X: C B A D E
– the object C with volume vC=1 was included, so the node
value is 60 and the remaining knapsack capacity is V=7
– the next two objects B and A with total value 50+30=80 and
volume 3+2=5 can be added to the knapsack as a whole
– the remaining knapsack capacity 7-5=2 is filled with 2/6 of
object D; the fractional value is 140/6=23,33
– the upper bound of node X is 60+80+23,33=163,33
– because the best solution so far is node X itself with value
60 and the upper bound is higher, the node is enqueued fot
later expansion
11 / 19
 Binary knapsack problem
●
v'=[1, 3, 2, 6, 4], c'=[60, 50, 30, 70, 40], V=8
●
example – node X: C B A D E
– the object C with volume vC=1 was included, so the node
value is 60 and the remaining knapsack capacity is V=7
– the next two objects B and A with total value 50+30=80 and
volume 3+2=5 can be added to the knapsack as a whole
– the remaining knapsack capacity 7-5=2 is filled with 2/6 of
object D; the fractional value is 140/6=23,33
– the upper bound of node X is 60+80+23,33=163,33
– because the best solution so far is node X itself with value
60 and the upper bound is higher, the node is enqueued fot
later expansion
11 / 19
 Binary knapsack problem
●
v'=[1, 3, 2, 6, 4], c'=[60, 50, 30, 70, 40], V=8
●
example – node Y: C B A D E
– the best solution so far has value 110
– in node Y the knapsack contains object B with volume vB=3
and value 50, the remaining knapsack capacity is V=5
– we can only add the next object A with value 30 and volume
2 as a whole to the knapsack
– the remaining knapsack capacity 5-2=3 is filled with 3/6 of
object D; the fractional value is 210/6=35
– the upper bound of node Y is 50+30+35=115
– because the upper bound ishigher than the best solution so
far, the node is enqueued fot later expansion
12 / 19
 Binary knapsack problem
●
v'=[1, 3, 2, 6, 4], c'=[60, 50, 30, 70, 40], V=8
●
example – node Y: C B A D E
– the best solution so far has value 110
– in node Y the knapsack contains object B with volume vB=3
and value 50, the remaining knapsack capacity is V=5
– we can only add the next object A with value 30 and volume
2 as a whole to the knapsack
– the remaining knapsack capacity 5-2=3 is filled with 3/6 of
object D; the fractional value is 210/6=35
– the upper bound of node Y is 50+30+35=115
– because the upper bound ishigher than the best solution so
far, the node is enqueued fot later expansion
12 / 19
 Binary knapsack problem
●
v'=[1, 3, 2, 6, 4], c'=[60, 50, 30, 70, 40], V=8
●
an improvement is the use of best-first search strategy in
place of ordinary breadth-first search:
– the nodes for expansion are enqueued into a priority queue,
where they are ordered by decreasing value per volume
unit
– the numbers next to the nodes denote the order of
expansion*
– some nodes are enqueued for expansion because their
upper bounds are higher than the best results found up to
that moment, but they are later not expanded because the
best result has improved in the meantime (coloured
green/blue)
13 / 19
 Binary knapsack problem
●
v'=[1, 3, 2, 6, 4], c'=[60, 50, 30, 70, 40], V=8
●
an improvement is the use of best-first search strategy in
place of ordinary breadth-first search:
– the nodes for expansion are enqueued into a priority queue,
where they are ordered by decreasing value per volume
unit
– the numbers next to the nodes denote the order of
expansion*
– some nodes are enqueued for expansion because their
upper bounds are higher than the best results found up to
that moment, but they are later not expanded because the
best result has improved in the meantime (coloured
green/blue)
13 / 19
 Traveling salesman problem
●
is a minimization problem, so we maintain the lower
bound of the value, i.e. the shortest cyce length, that
can be achieved by continuing from the node:
– the tree root is the starting point (and the end) of the path
– when expanding any node in the tree, we continue from
all of its unvisited successors in the graph (example of a
graph with 4 cities*; dashed lines are path continuations
that need not be expanded, because they are unique)
– when computing the lower bound, the lowest costs from
the last visited city to any of the unvisited cities and from
the unvisited cities to other unvisited cities or the starting
point is added to the path length
– if the lower bound of the node is greater than the length of
the shortest path so far, the node is not expanded further 14 / 19
 Traveling salesman problem
●
is a minimization problem, so 1 we maintain the lower
bound of the value, i.e. the shortest cyce length, that
can be achieved by continuing from the node:
– the tree root is the starting point (and the end) of the path
2 3 4
– when expanding any node in the tree, we continue from
all of its unvisited successors in the graph (example of a
graph with 4 cities*; dashed lines are path continuations
3 4 2 4 2 3
that need not be expanded, because they are unique)
– when computing the lower bound, the lowest costs from
the last visited city to any of the unvisited cities and from
4 3 4 2 3 2
the unvisited cities to other unvisited cities or the starting
point is added to the path length
– if the lower bound of the node is greater than the length of
1
1
the shortest 1
path 1
so far,1 the node is not 1expanded further
14 / 19
 Traveling salesman problem
●
is a minimization problem, so we maintain the lower
bound of the value, i.e. the shortest cyce length, that
can be achieved by continuing from the node:
– the tree root is the starting point (and the end) of the path
– when expanding any node in the tree, we continue from
all of its unvisited successors in the graph (example of a
graph with 4 cities*; dashed lines are path continuations
that need not be expanded, because they are unique)
– when computing the lower bound, the lowest costs from
the last visited city to any of the unvisited cities and from
the unvisited cities to other unvisited cities or the starting
point is added to the path length
– if the lower bound of the node is greater than the length of
the shortest path so far, the node is not expanded further 14 / 19
 Traveling salesman problem
●
example: a graph with 4 nodes – cities
●
let the starting and ending city be 3
●
let‘s compute the lower bounds of the path
length for nodes on depth 1 (i.e. nodes that
represent partial paths [3,1], [3,2] and [3,4]):
– the cost of path [3,1] is 1
– the length of the shortest edge from the last visited node 1 to any
of the unvisited nodes (i.e., 2 and 4) is min(5,4)=4
– the length of the shortest edge from the unvisited node 2 to any of
the other unvisited nodes or the origin (i.e., 3 or 4) is min(6,2)=2
– the length of the shortest edge from the unvisited node 4 to any of
the other unvisited nodes or the origin (i.e., 2 or 3) is min(4,9)=4
– the lower bound of path length through nodes [3,1] is
1+4+2+4=11 15 / 19
 Traveling salesman problem
●
example: a graph with 4 nodes – cities
●
let the starting and ending city be 3
●
let‘s compute the lower bounds of the path
length for nodes on depth 1 (i.e. nodes that
represent partial paths [3,1], [3,2] and [3,4]):
– the cost of path [3,2] is 8
– the length of the shortest edge from last visited node 2 to any of
the unvisited nodes (i.e., 1 and 4) is min(3,2)=2
– the length of the shortest edge from the unvisited node 1 to any of
the other unvisited nodes or the origin (i.e., 3 or 4) is min(11,4)=4
– the length of the shortest edge from the unvisited node 4 to any of
the other unvisited nodes or the origin (i.e., 1 or 3) is min(6,9)=6
– the lower bound of path length through nodes [3,2] is
8+2+4+6=20 16 / 19
 Traveling salesman problem
●
example: a graph with 4 nodes – cities
●
let the starting and ending city be 3
●
let‘s compute the lower bounds of the path
length for nodes on depth 1 (i.e. nodes that
represent partial paths [3,1], [3,2] and [3,4]):
– the cost of path [3,4] is 7
– the length of the shortest edge from last visited node 4 to any of
the unvisited nodes (i.e., 1 and 2) is min(6,4)=4
– the length of the shortest edge from the unvisited node 1 to any of
the other unvisited nodes or the origin (i.e., 2 or 3) is min(5,11)=5
– the length of the shortest edge from the unvisited node 2 to any of
the other unvisited nodes or the origin (i.e., 1 or 3) is min(3,6)=3
– the lower bound of path length through nodes [3,4] je
7+4+5+3=19 17 / 19
 Traveling salesman problem
●
example: a graph with 4 nodes – cities; the origin is city 3
●
when solving the problem we can use ordinary breadth-first
search or best-first search; in the example they build the
same search tree with solution 3-1-4-2-3 of length 15*

18 / 19
 Traveling salesman problem
●
example: a graph with 5 nodes – cities; the origin is city 2
●
the solution using ordinary BFD (the candidates for the
next move are ordered in numerical order)* or the best-first
search*

[ ]
0 8 13 18 20
3 0 7 8 10
C= 4 11 0 10 7
6 6 7 0 11
10 6 2 1 0

19 / 19
 Traveling salesman problem
●
example: a graph with 5 nodes – cities; the origin is city 2
●
the solution using ordinary BFD (the candidates for the
next move are ordered in numerical order)* or the best-first
search*

[ ]
0 8 13 18 20
3 0 7 8 10
C= 4 11 0 10 7
6 6 7 0 11
10 6 2 1 0

19 / 19