DAA-18CS42

Module 5 –Backtracking and Branch & Bound

Introduction:
 Some problems can be solved, by exhaustive search. The exhaustive-search technique
suggests generating all candidate solutions and then identifying the one (or the ones) with a
desired property.
 Backtracking is a more intelligent variation of this approach. The principal idea is to
construct solutions one component at a time and evaluate such partially constructed
candidates as follows. If a partially constructed solution can be developed further without
violating the problem’s constraints, it is done by taking the first remaining legitimate
option for the next component. If there is no legitimate option for the next component, no
Alternatives for any remaining component need to be considered. In this case, the
algorithm backtracks to replace the last component of the partially constructed solution
with its next option.
 It is convenient to implement this kind of processing by constructing a tree of choices being
made, called the state-space tree. Its root represents an initial state before the search for a
solution begins. The nodes of the first level in the tree represent the choices made for the
first component of a solution; the nodes of the second level represent the choices for the
second component, and so on. A node in a state-space tree is said to be promising if it
corresponds to a partially constructed solution that may still lead to a complete solution;
otherwise, it is called non-promising. Leaves represent either non-promising dead Ends or
complete solutions found by the algorithm.
 In the majority of cases, a state space tree for a backtracking algorithm is constructed in the
manner of depth-first search. If the current node is promising, its child is generated by
adding the first remaining legitimate option for the next component of a solution, and the
processing moves to this child. If the current node turns out to be non-promising, the
algorithm backtracks to the node’s parent to consider the next possible option for its last
component; if there is no such option, it backtracks one more level up the tree, and so on.
Finally, if the algorithm reaches a complete solution to the problem, it either stops (if just
one solution is required) or continues searching for other possible solutions.
 DAA-18CS42

General Algorithm for backtracking (From textbook T1)

1.1 N-Queens problem

The problem is to place n queens on an n × n chessboard so that no two queens attack each
other by being in the same row or in the same column or on the same diagonal.
So let us consider the four-queens problem and solve it by the backtracking technique.
Since each of the four queens has to be placed in its own row, all we need to do is to assign a
column for each queen on the board presented in figure.

We start with the empty board and then place queen 1 in the first possible position of its row,
which is in column 1 of row . Then we place queen 2, after trying unsuccessfully columns 1
and 2, in the first acceptable position for it, which is square (2, 3),

Figure: State-space tree of solving the four-queens problem by backtracking.

× denotes an unsuccessful attempt to place a queen in the indicated column. The
numbers above the nodes indicate the order in which the nodes are generated.
 DAA-18CS42

the square in row 2 and column 3. This proves to be a dead end because there is no acceptable
position for queen 3. So, the algorithm backtracks and puts queen 2 in the next possible
position at (2, 4). Then queen 3 is placed at (3, 2), which proves to be another dead end. The
algorithm thenbacktracks all the way to queen 1 and moves it to (1, 2). Queen 2 then goes to
(2, 4), queen 3 to (3, 1), and queen 4 to (4, 3), which is a solution to the problem. The state-
space tree of this search is shown in figure.

If other solutions need to be found, the algorithm can simply resume its operations at the leaf
at which it stopped. Alternatively, we can use the board’s symmetry for this purpose.
Finally, it should be pointed out that a single solution to the n-queens problem for any n ≥ 4
can be found in linear time.
 ADA-BCS401

1.2 Sum of subsets problem

Problem definition: Find a subset of a given set A = {a1, . . . , an } of n positive integers whose
sum is equal to a given positive integer d.
For example, for A = {1, 2, 5, 6, 8} and d = 9, there are two solutions: {1, 2, 6} and {1, 8}. Of
course, some instances of this problem may have no solutions.
It is convenient to sort the set’s elements in increasing order. So, we will assume that a1< a2 < . .
. < an.
The state-space tree can be constructed as a binary tree like that in Figure shown below for the
instance A = {3, 5, 6, 7} and d = 15.
The number inside a node is the sum of the elements already included in the subsets represented
by the node. The inequality below a leaf indicates the reason for its termination.

The root of the tree represents the starting point, with no decisions about the given elements
made as yet. Its left and right children represent, respectively, inclusion and exclusion of a1 in
a set being sought.
Similarly, going to the left from a node of the first level corresponds to inclusion of a2 while
going to the right corresponds to its exclusion, and so on. Thus, a path from the root to a node
on the ith level of the tree indicates which of the first i numbers have been included in the
subsets represented by that node.
We record the value of s, the sum of these numbers, in the node. If s is equal to d, we have a
solution to the problem. We can either report this result and stop or, if all the solutions need
to be found, continue by backtracking to the node’s parent. If s is not equal to d, we can
terminate the node as non-promising if either of the following two inequalities holds:

Page 4
 ADA-BCS401

Example 2: Apply backtracking to solve the following instance of the subset sum problem: A
= {3, 4, 5,6} and d = 13.

Page 5
 ADA-BCS401

2. Branch and Bound

Recall that the central idea of backtracking, discussed in the previous section, is to cut off a
branch of the problem’s stat -space tree as soon as we can deduce that it cannot lead to a
solution. This idea can be strengthened further if we deal with an optimization problem.
An optimization problem seeks to minimize or maximize some objective function (a tour length,
the value of items selected, the cost of an assignment, and the like), usually subject to some
constraints. An optimal solution is a feasible solution with the best value of the objective
function (e.g., the shortest Hamiltonian circuit or the most valuable subset of items that fit the
knapsack).
Compared to backtracking, branch-and-bound requires two additional items:
1. a way to provide, for every node of a state-space tree, a bound on the best value of the
objective function on any solution that can be obtained by adding further components to
the partially constructed solution represented by the node
2. the value of the best solution seen so far
In general, we terminate a search path at the current node in a state-space tree of a branch- and-
bound algorithm for any one of the following three reasons:
1. The value of the node’s bound is not better than the value of the best solution seen so far.
2. The node represents no feasible solutions because the constraints of the problem are
already violated.
3. The subset of feasible solutions represented by the node consists of a single point (and
hence no further choices can be made)—in this case, we compare the value of the
objective function for this feasible solution with that of the best solution seen so far and
update the latter with the former if the new solution is better.

0/1 Knapsack problem

Let us now discuss how we can apply the branch-and-bound technique to solving the
knapsack problem. Given n items of known weights wi and values vi , i = 1, 2, . . . , n, and a
knapsack of capacity W, find the most valuable subset of the items that fit in the knapsack.
Σ wi xi ≤ W and Σ vi xi is maximized, where xi = 0 or 1

It is convenient to order the items of a given instance in descending order by their value-to-
weight ratios.

Each node on the ith level of state space tree, 0 ≤ i ≤ n, represents all the subsets of n items
that include a particular selection made from the first i ordered items. This particular
selection is uniquely determined by the path from the root to the node: a branch going to the
left indicates the inclusion of the next item, and a branch going to the right indicates its
exclusion.

Page 6
 ADA-BCS401

We record the total weight w and the total value v of this selection in the node, along
with some upper bound ub on the value of any subset that can be obtained by adding
zero or more items to this selection. A simple way to compute the upper bound ub is to
add to v, the total value of the items already selected, the product of the remaining
capacity of the knapsack W − w and the best per unit payoff among the remaining
items, which is vi+1/wi+1:
ub = v + (W − w)(vi+1/wi+1).
Example: Consider the following problem. The items are already ordered in
descendingorder of their value-to-weight ratios.

Let us apply the branch-and-bound algorithm. At the root of the state-space tree (see
Figure 12.8), no items have been selected as yet. Hence, both the total weight of the items
already selected w and their total value v are equal to 0. The value of the upper bound is
100.
Node 1, the left child of the root, represents the subsets that include item 1. The total
weight and value of the items already included are 4 and 40, respectively; the value of the
upper bound is 40
+ (10 − 4) * 6 = 76.
Node 2 represents the subsets that do not include item 1. Accordingly, w = 0, v = 0, and
ub = 0 +(10 − 0) * 6 = 60. Since node 1 has a larger upper bound than the upper bound of
node 2, it is more promising for this maximization problem, and we branch from node 1
first. Its children— nodes 3 and 4—represent subsets with item 1 and with and without
item 2, respectively. Since the total weight w of every subset represented by node 3
exceeds the knapsack’s capacity, node 3can be terminated immediately.
Node 4 has the same values of w and v as its parent; the upper bound ub is equal to 40 + (10
− 4) * 5 = 70. Selecting node 4 over node 2 for the next branching (Due to better ub), we
get nodes 5 and 6 by respectively including and excluding item 3. The total weights and
values as well as the upper bounds for these nodes are computed in the same way as for
the preceding nodes.
Branching from node 5 yields node 7, which represents no feasible solutions, and node 8,
which represents just a single subset {1, 3} of value 65. The remaining live nodes 2 and 6
have smaller upper-bound values than the value of the solution represented by node 8.
Hence, both can be terminated making the subset {1, 3} of node 8 the optimal solution to
the problem.
Page 7
 ADA-BCS401

Solving the knapsack problem by a branch-and-bound algorithm has a rather unusual

characteristic. Typically, internal nodes of a state-space tree do not define a point of the
problem’s search space, because some of the solution’s components remain undefined. (See,
for example, the branch-and-bound tree for the assignment problem discussed in the
preceding subsection.) For the knapsack problem, however, every node of the tree represents
a subset of the items given. We can use this fact to update the information about the best
subset seen so far after generating each new node in the tree. If we had done this for the
instance investigated above, we could have terminated nodes 2 and 6 before node 8 was
generated because they both are inferior to the subset of value 65 of node 5.

Greedy Algorithms for the Knapsack Problem We can think of several greedy approaches to
this problem. One is to select the items in decreasing order of their weights; however, heavier
items may not be the most valuable in the set. Alternatively, if we pick up the items in
decreasing order of their value, there is no guarantee that the knapsack’s capacity will be used
efficiently. Can we find a greedy strategy that takes into account both the weights and values?
Yes, we can, by computing the value-to-weight ratios vi/wi, i = 1, 2, . . . , n, and selecting the
items in decreasing order of these ratios. Here is the algorithm based on this greedy heuristic.

Page 8
 ADA-BCS401

Greedy algorithm for the discrete knapsack problem

Step 1 Compute the value-to-weight ratios ri = vi/wi, i = 1, . . . , n, for the items given.
Step 2 Sort the items in nonincreasing order of the ratios computed in Step 1. (Ties can be
broken arbitrarily.)
Step 3 Repeat the following operation until no item is left in the sorted list: if the current item
on the list fits into the knapsack, place it in the knapsack and proceed to the next item;
otherwise, just proceed to the next item.
Example 1: Let us consider the instance of the knapsack problem with the
knapsack capacity 10 and the item information as follows:

Computing the value-to-weight ratios and sorting the items in nonincreasing order of these
efficiency ratios yields

The greedy algorithm will select the first item of weight 4, skip the next item of weight 7, select
the next item of weight 5, and skip the last item of weight 3. The solution obtained happens to
be optimal for this instance .
Greedy algorithm for the continuous knapsack problem
Step 1 Compute the value-to-weight ratios vi/wi, i = 1, . . . , n, for the items given.
Step 2 Sort the items in nonincreasing order of the ratios computed in Step 1.(Ties can be
broken arbitrarily.)
Step 3 Repeat the following operation until the knapsack is filled to its full capacity or no item
is left in the sorted list: if the current item on the list fits into the knapsack in its entirety, take it
and proceed to the next item; otherwise, take its largest fraction to fill the knapsack to its full
capacity and stop
The algorithm will take the first item of weight 4 and then 6/7 of the next item on the sorted list
to fill the knapsack to its full capacity.

Page 9
 ADA-BCS401

Approximation Schemes: We now return to the discrete version of the knapsack problem. For
this problem, unlike the traveling salesman problem, there exist polynomial-time approximation
schemes, which are parametric families of algorithms that allow us to get approximations sa (k)
with any predefined accuracy level:

where k is an integer parameter in the range 0 ≤ k < n. The first approximation scheme was
suggested by S. Sahni in 1975 [Sah75]. This algorithm generates all subsets of k items or less,
and for each one that fits into the knapsack it adds the remaining items as the greedy algorithm
would do (i.e., in nonincreasing order of their value-to-weight ratios). The subset of the highest
value obtained in this fashion is returned as the algorithm’s output.
Example :A small example of an approximation scheme with k = 2 is provided in below figure
The algorithm yields {1, 3, 4}, which is the optimal solution for this instance.

For each of those subsets, it needs O(n) time to determine the subset’s possible extension. Thus,
the algorithm’s efficiency is in O(knk+1).

Page 10
 ADA-BCS401

3. NP-Complete and NP-Hard problems

Basic concepts
For many of the problems we know and study, the best algorithms for their solution have
computing times can be clustered into two groups;
1. Solutions are bounded by the polynomial- Examples include Binary search O(log n),
Linear search O(n), sorting algorithms like merge sort O(n log n), Bubble sort O(n2)
& matrix multiplication O(n3) or in general O(nk) where k is a constant.
2. Solutions are bounded by a non-polynomial - Examples include travelling salesman
problem O(n22n) & knapsack problem O(2n/2). As the time increases exponentially,
even moderate size problems cannot be solved.
So far, no one has been able to device an algorithm which is bounded by the polynomial for
the problems belonging to the non-polynomial. However impossibility of such an algorithm
is not proved.

Non deterministic algorithms

We also need the idea of two models of computer (Turing machine): deterministic and non-
deterministic. A deterministic computer is the regular computer we always thinking of; a non-
deterministic computer is one that is just like we’re used to except that is has unlimited
parallelism, so that any time you come to a branch, you spawn a new “process” and examine
both sides.
When the result of every operation is uniquely defined then it is called deterministic
algorithm.
When the outcome is not uniquely defined but is limited to a specific set of possibilities, we
call it non deterministic algorithm.
We use new statements to specify such non deterministic h algorithms.
 choice(S) - arbitrarily choose one of the elements of set S
 failure - signals an unsuccessful completion
 success - signals a successful completion
The assignment X = choice(1:n) could result in X being assigned any value from the integer
range[1..n]. There is no rule specifying how this value is chosen.

“The nondeterministic algorithms terminates unsuccessfully iff there is no set of choices

which leads to the successful signal”.
Example-1: Searching an element x in a given set of elements A(1:n). We are required to
determine an index j such that A(j) = x or j = 0 if x is not present.
j := choice(1:n)
if A(j) = x then print(j); success endif

Page 11
 ADA-BCS401

print(‘0’); failure

Example-2: Checking whether n integers are sorted or not

procedure NSORT(A,n);
//sort n positive integers//
var integer A(n), B(n), n, i, j;
begin
B := 0; //B is initialized to zero//
for i := 1 to n do
begin
j := choice(1:n);
if B(j) <> 0 then failure;
B(j) := A(j);
end;

for i := 1 to n-1 do //verify order//

if B(i) > B(i+1) then failure;
print(B);
success;
end.
“A nondeterministic machine does not make any copies of an algorithm every time a choice
is to be made. Instead it has the ability to correctly choose an element from the given set”.
A deterministic interpretation of the nondeterministic algorithm can be done by making
unbounded parallelism in the computation. Each time a choice is to be made, the algorithm
makes several copies of itself, one copy is made for each of the possible choices.

Decision vs Optimization algorithms

An optimization problem tries to find an optimal solution.
A decision problem tries to answer a yes/no question. Most of the problems can be specified
in decision and optimization versions.
For example, Traveling salesman problem can be stated as two ways
 Optimization - find hamiltonian cycle of minimum weight,
 Decision - is there a hamiltonian cycle of weight  k?
For graph coloring problem,
 Optimization – find the minimum number of colors needed to color the vertices of a
graph so that no two adjacent vertices are colored the same color
 Decision - whether there exists such a coloring of the graph’s vertices with no more
than m colors?
Many optimization problems can be recast in to decision problems with the property that the
decision algorithm can be solved in polynomial time if and only if the corresponding
optimization problem.

Page 12
 ADA-BCS401

P, NP, NP-Complete and NP-Hard classes

NP stands for Non-deterministic Polynomial time.
Definition: P is a set of all decision problems solvable by a deterministic algorithm in
polynomial time.
Definition: NP is the set of all decision problems solvable by a nondeterministic algorithm in
polynomial time. This also implies P ⊆ NP
Problems known to be in P are trivially in NP — the nondeterministic machine just never
troubles itself to fork another process, and acts just like a deterministic one. One example of a
problem not in P but in NP is Integer Factorization.
But there are some problems which are known to be in NP but don’t know if they’re in P. The
traditional example is the decision-problem version of the Travelling Salesman Problem
(decision-TSP). It’s not known whether decision-TSP is in P: there’s no known poly-time
solution, but there’s no proof such a solution doesn’t exist.
There are problems that are known to be neither in P nor NP; a simple example is to enumerate
all the bit vectors of length n. No matter what, that takes 2n steps.

Now, one more concept: given decision problems P and Q, if an algorithm can transform a
solution for P into a solution for Q in polynomial time, it’s said that Q is poly-time reducible
(or just reducible) to P.
The most famous unsolved problem in computer science is “whether P=NP or P≠NP? ”

Figure: Commonly believed Figure: Commonly believed relationship between P, NP, NP-
relationship between P and NP Complete and NP-hard problems

Definition: A decision problem D is said to be NP-complete if:

• it belongs to class NP
• every problem in NP is polynomially reducible to D
The fact that closely related decision problems are polynomially reducible to each other is not
very surprising. For example, Hamiltonian circuit problem is polynomially reducible to the
decision version of the traveling salesman problem.

Page 13
 ADA-BCS401

NP-Complete problems have the property that it can be solved in polynomial time if all other
NP-Complete problems can be solved in polynomial time. i.e if anyone ever finds a poly-time
solution to one NP-complete problem, they’ve automatically got one for all the NP-complete
problems; that will also mean that P=NP.
Example for NP-complete is CNF-satisfiability problem. The CNF-satisfiability problem deals
with boolean expressions. This is given by Cook in 1971. The CNF-satisfiability problem asks
whether or not one can assign values true and false to variables of a given boolean expression in
its CNF form to make the entire expression true.
Over the years many problems in NP have been proved to be in P (like Primality Testing). Still,
there are many problems in NP not proved to be in P. i.e. the question still remains whether
P=NP? NP Complete Problems helps in solving this question. They are a subset of NP
problems with the property that all other NP problems can be reduced to any of them in
polynomial time. So, they are the hardest problems in NP, in terms of running time. If it can be
showed that any NP-Complete problem is in P, then all problems in NP will be in P (because of
NP-Complete definition), and hence P=NP=NPC.
NP Hard Problems - These problems need not have any bound on their running time. If any
NP-Complete Problem is polynomial time reducible to a problem X, that problem X belongs to
NP-Hard class. Hence, all NP-Complete problems are also NP-Hard. In other words if a NP-
Hard problem is non-deterministic polynomial time solvable, it is a NP- Complete problem.
Example of a NP problem that is not NPC is Halting Problem.
If a NP-Hard problem can be solved in polynomial time then all NP-Complete can be solved in
polynomial time.
“All NP-Complete problems are NP-Hard but not all NP-Hard problems are not NP- Complete.”
NP-Complete problems are subclass of NP-Hard
The more conventional optimization version of Traveling Salesman Problem for finding the
shortest route is NP-hard, not strictly NP-complete.

Page 14
Decision Tree:
Sorting and searching algorithms work by comparing items of their inputs. We can study the
performance of such algorithms with a device called a decision tree.

The decision tree to check to print minimum of 3 numbers.

Each internal node of a binary decision tree represents a key comparison indicated in the node.
The nodes left subtree contains the information about subsequent comparisons made if k < k,
and its right subtree does the same for the case of k >k. Each leaf represents a possible outcome
of the algorithm’s run on some input of size n. An important point is that the number of leaves
must be at least as large as the number of possible outcomes. The number of comparisons in
the worst case is equal to the height of the algorithm’s decision tree.
h ≥ _log2 l.

a. Decision tree for sorting:

Decision tree for the tree-element selection sort. A triple above a node indicates the state of the
array being sorted. Note two redundant comparisons b <a with a single possible outcome
because of the results of some previously made comparisons.

Most sorting algorithms work by comparing elements in a list. By analysing the decision trees
used in these algorithms, we can determine important lower bounds on their time efficiency. In
general, the number of possible outcomes for sorting an arbitrary n-element list is equal to n!.
The height of a binary decision tree for any comparison-based sorting algorithm and hence the
worst-case number of comparisons made by such an algorithm cannot be less than _log2 n!.
Cworst(n) ≥ [log2 n!]. Merge sort takes number of comparisons is n log2 n in its worst case
and hence is asymptotically optimal. This also implies that the asymptotic lower bound n log2
n is tight and therefore cannot be substantially improved.
 Decision tree for the three-element insertion sort.

b. Decision Trees for Searching a Sorted Array:

Consider Binary search to search a given key in a finite set of elements. The No. of comparisons
in th worst case is

Ternary decision tree for binary search in a four-element array.

We dealing here with three-way comparisons in which search key K is compared with some
element A[i] to see whether K <A[i], K = A[i], or K >A[i]. Hence it is natural to try using ternary
decision trees. The internal nodes of that tree indicate the arrays elements being compared with
the search key. The leaves indicate either a matching element in the case of a successful search
or a found interval that the search key belongs to in the case of an unsuccessful search.
For an array of n elements, all such decision trees will have 2n + 1 leaves (n for successful
searches and n + 1 for unsuccessful ones). Since the minimum height h of a ternary tree with l
leaves is [log3 l], We get the following lower bound on the number of worst-case comparisons:
Cworst(n) ≥ [log3(2n + 1)].
The lower bound is smaller than [log2(n + 1)].