MACHINE LEARNING

Unit-III:
Unit III: Ensemble Learning and Random Forests: Introduction, Voting Classifiers, Bagging and
Pasting, Random Forests, Boosting, Stacking.

Support Vector Machine: Linear SVM Classification, Nonlinear SVM ClassificationSVM Regression,
Naïve Bayes Classifiers.

Introduction:
Aggregate the predictions of a group of predictors (such as classifiers or regressors), you will often get better
predictions than with the best individual predictor. A group of predictors is called an ensemble; thus, this
technique is called Ensemble Learning, and an Ensemble Learning algorithm is called an Ensemble
method.
For example, you can train a group of Decision Tree classifiers, each on a different random subset of the
training set. To make predictions, you just obtain the predictions of all individual trees, then predict the class
that gets the most votes. Such an ensemble of Decision Trees is called a Random Forest. This is one of the
most powerful Machine Learning algorithms available today.
The most popular Ensemble methods, including
 Bagging- involves fitting many decision trees on different samples of the same dataset
andaveraging the predictions
 Boosting- involves adding ensemble members sequentially that correct the predictions
made by prior models and outputs a weighted average of the predictions.
 Stacking- involves fitting many different models types on the same data and using
anothermodel to learn how to best combine the predictions.
 And a few others.

Voting Classifiers:
Suppose you have trained a few classifiers, each one achieving about 80% accuracy. You may have a
Logistic Regression classifier, an SVM classifier, a Random Forest classifier, a K-Nearest Neighbors
classifier, and perhaps a few more

Figure . Training diverse classifiers

1
A very simple way to create an even better classifier is to aggregate the predictions of each classifier and
predict the class that gets the most votes. This majority-vote classifier is called a hard voting classifier.

Figure: Hard voting classifier predictions

Figure: Soft voting classifier predictions

This voting classifier often achieves a higher accuracy than the best classifier in the ensemble. In fact, even
if each classifier is a weak learner (meaning it does only slightly better than random guessing), the ensemble

2
can still be a strong learner (achieving high accuracy), provided there are a sufficient number of weak
learners and they are sufficiently diverse.
The following code creates and trains a voting classifier in Scikit-Learn, composed of three diverse
classifiers (the training set is the moons dataset)

Let’s look at each classifier’s accuracy on the test set:

Accuracy of
LogisticRegression 0.864
RandomForestClassifier 0.896
SVC 0.888
VotingClassifier 0.904

If all classifiers are able to estimate class probabilities (i.e., they have a predict_proba() method), then
you can tell Scikit-Learn to predict the class with the highest class probability, averaged over all the
individual classifiers. This is called soft voting. It often achieves higher performance than hard voting
because it gives more weight to highly confident votes. All you need to do is replace voting="hard" with
voting="soft" and ensure that all classifiers can estimate class probabilities. This is not the case of the SVC
class by default, so you need to set its probability hyper parameter to True (this will make the SVC class use
cross-validation to estimate class probabilities, slowing down training, and it will add a predict_proba()
method).If you modify the preceding code to use soft voting, you will find that the voting classifier achieves
over 91.2% accuracy!

3
 Bagging and Pasting

One way to get a diverse set of classifiers is to use very different training algorithms, as just discussed.
Another approach is to use the same training algorithm for every predictor, but to train them on different
random subsets of the training set. When sampling is performed with replacement, this method is called
bagging (short for bootstrap aggregating). When sampling is performed without replacement, it is called
pasting.
Both bagging and pasting allow training instances to be sampled several times across multiple predictors, but
only bagging allows training instances to be sampled several times for the same predictor. This sampling and
training process is represented in below Figure:

Figure : Pasting/bagging training set sampling and training

Once all predictors are trained, the ensemble can make a prediction for a new instance by simply
aggregating the predictions of all predictors. The aggregation function is typically the statistical mode (i.e.,
the most frequent prediction, just like a hard voting classifier) for classification, or the average for
regression. Each individual predictor has a higher bias than if it were trained on the original training set, but
aggregation reduces both bias and variance.
Generally, the net result is that the ensemble has a similar bias but a lower variance than a single predictor
trained on the original training set.

Bagging and Pasting in Scikit-Learn:

Scikit-Learn offers a simple API for both bagging and pasting with the Bagging Classifier class (or
BaggingRegressor for regression).
The following code trains an ensemble of 500 Decision Tree classifiers, each trained on 100 training
instances randomly sampled from the training set with replacement (this is an example of bagging, but if you
want to use pasting instead, just set bootstrap=False).

4
Note: The BaggingClassifier automatically performs soft voting instead of hard voting if the base classifier
can estimate class probabilities (i.e., if it has a predict_proba() method), which is the case with Decision
Trees classifiers.
Bootstrapping introduces a bit more diversity in the subsets that each predictor is trained on, so bagging
ends up with a slightly higher bias than pasting, but this also means that predictors end up being less
correlated so the ensemble’s variance is reduced. Overall, bagging often results in better models, which
explains why it is generally preferred. However, if you have spare time and CPU power you can use
crossvalidation to evaluate both bagging and pasting and select the one that works best.

In statistics, bootstrapping refers to a resample method that consists of repeatedly drawn, with replacement,
samples from data to form other smaller datasets, called bootstrapping samples.

Out-of-Bag Evaluation:
With bagging, some instances may be sampled several times for any given predictor, while others may not
be sampled at all. By default a BaggingClassifier samples m training instances with replacement
(bootstrap=True), where m is the size of the training set. This means that only about 63% of the training
instances are sampled on average for each predictor. The remaining 37% of the training instances that are
not sampled are called out-of-bag (oob) instances. Note that they are not the same 37% for all predictors.
In Scikit-Learn, you can set oob_score=True when creating a BaggingClassifier to request an automatic oob
evaluation after training. The following code demonstrates this. The resulting evaluation score is available
through the oob_score_ variable:

0.90133333333333332
According to this oob evaluation, this BaggingClassifier is likely to achieve about 90.1% accuracy on the
test set. Let’s verify this:

0.91200000000000003
We get 91.2% accuracy on the test set

5
Random Patches and Random Subspaces:
The BaggingClassifier class supports sampling the features as well. This is controlled by two
hyperparameters: max_features and bootstrap_features. They work the same way as max_samples and
bootstrap, but for feature sampling instead of instance sampling. Thus, each predictor will be trained on a
random subset of the input features.
This is particularly useful when you are dealing with high-dimensional inputs (such as images). Sampling
both training instances and features is called the Random Patches method.

Keeping all training instances (i.e., bootstrap=False and max_samples=1.0) but sampling features (i.e.,
bootstrap_features=True and/or max_features smaller than 1.0) is called the Random Subspaces method.

Sampling features results in even more predictor diversity, trading a bit more bias for a lower variance.

Random Forests
Random Forest is a popular machine learning algorithm that belongs to the supervised learning
technique. It can be used for both Classification and Regression problems in ML. It is based on the concept
of ensemble learning, which is a process of combining multiple classifiers to solve a complex problem and
to improve the performance of the model.

As the name suggests, "Random Forest is a classifier that contains a number of decision trees on various
subsets of the given dataset and takes the average to improve the predictive accuracy of that dataset." Instead
of relying on one decision tree, the random forest takes the prediction from each tree and based on the
majority votes of predictions, and it predicts the final output.

The greater number of trees in the forest leads to higher accuracy and prevents the problem of overfitting.

The below diagram explains the working of the Random Forest algorithm:

6
Random Forest Algorithm

Random Forest works in two-phase first is to create the random forest by combining N decision tree,
and second is to make predictions for each tree created in the first phase.

Step-1: Select random K data points from the training set.

Step-2: Build the decision trees associated with the selected data points (Subsets).
Step-3: Choose the number N for decision trees that you want to build.
Step-4: Repeat Step 1 & 2.
Step-5: For new data points, find the predictions of each decision tree, and assign the new data points to the
category that wins the majority votes.

The working of the algorithm can be better understood by the below example:

Example: Suppose there is a dataset that contains multiple fruit images. So, this dataset is given to the
Random forest classifier. The dataset is divided into subsets and given to each decision tree. During the
training phase, each decision tree produces a prediction result, and when a new data point occurs, then based
on the majority of results, the Random Forest classifier predicts the final decision. Consider the below
image:

Implementation Steps are given below:

1. Data Pre-processing step

2. Fitting the Random forest algorithm to the Training set
3. Predicting the test result
4. Test accuracy of the result (Creation of Confusion matrix)
5. Visualizing the test set result.
7
Applications of Random Forest
There are mainly four sectors where Random forest mostly used:

Banking: Banking sector mostly uses this algorithm for the identification of loan risk.
Medicine: With the help of this algorithm, disease trends and risks of the disease can be identified.
Land Use: We can identify the areas of similar land use by this algorithm.
Marketing: Marketing trends can be identified using this algorithm.

Advantages of Random Forest

Random Forest is capable of performing both Classification and Regression tasks.
 It is capable of handling large datasets with high dimensionality.
 It enhances the accuracy of the model and prevents the overfitting issue.
 Disadvantages of Random Forest
 Although random forest can be used for both classification and regression tasks, it is not more
suitable for Regression tasks.

Python Implementation of Random Forest Algorithm

A Random Forest is an ensemble of Decision Trees, generally trained via the bagging method (or
sometimes pasting), typically with max_samples set to the size of the training set. Instead of building a
BaggingClassifier and passing it a DecisionTreeClassifier, you can instead use the RandomForestClassifier
class, which is more convenient and optimized for Decision Trees (similarly, there is a
RandomForestRegressor class for regression tasks).

The following code trains a Random Forest classifier with 500 trees (each limited to maximum 16 nodes),
using all available CPU cores:

The Random Forest algorithm introduces extra randomness when growing trees; instead of searching for the
very best feature when splitting a node, it searches for the best feature among a random subset of features.
This results in a greater tree diversity, which (once again) trades a higher bias for a lower variance, generally
yielding an overall better model. The following BaggingClassifier is roughly equivalent to the previous
RandomForestClassifier:

Extra-Trees:
When you are growing a tree in a Random Forest, at each node only a random subset of the features is
considered for splitting (as discussed earlier). It is possible to make trees even more random by also using
random thresholds for each feature rather than searching for the best possible thresholds (like regular
8
Decision Trees do).
A forest of such extremely random trees is simply called an Extremely Randomized Trees ensemble (or
Extra-Trees for short). Once again, this trades more bias for a lower variance. It also makes Extra-Trees
much faster to train than regular Random Forests since finding the best possible threshold for each feature at
every node is one of the most time-consuming tasks of growing a tree.
You can create an Extra-Trees classifier using Scikit-Learn’s ExtraTreesClassifier class. Its API is identical
to the RandomForestClassifier class. Similarly, the Extra TreesRegressor class has the same API as the
RandomForestRegressor class.

Note: It is hard to tell in advance whether a RandomForestClassifier will perform better or worse than an
ExtraTreesClassifier. Generally, the only way to know is to try both and compare them using cross-
validation (and tuning the hyperparameters using gridsearch).

Feature Importance:

Random Forests is that they make it easy to measure the relative importance of each feature. Scikit-Learn
measures a feature’s importance by looking at how much the tree nodes that use that feature reduce impurity
on average (across all trees in the forest). More precisely, it is a weighted average, where each node’s weight
is equal to the number of training samples that are associated with it.

Scikit-Learn computes this score automatically for each feature after training, then it scales the results so that
the sum of all importances is equal to 1. You can access the result using the feature_importances_ variable.

For example, the following code trains a RandomForestClassifier on the iris dataset and outputs each
feature’s importance. It seems that the most important features are the petal length (44%) and width (42%),
while sepal length and width are rather unimportant in comparison (11% and 2%, respectively).

sepal length (cm) 0.112492250999

sepal width (cm) 0.0231192882825
petal length (cm) 0.441030464364
petal width (cm) 0.423357996355

9
 Boosting
Boosting (originally called hypothesis boosting) refers to any Ensemble method that can combine several
weak learners into a strong learner. The general idea of most boosting methods is to train predictors
sequentially, each trying to correct its predecessor. There are many boosting methods available, but by far
the most popular are:

AdaBoost (short for Adaptive Boosting) and Gradient Boosting.

AdaBoost:
One way for a new predictor to correct its predecessor is to pay a bit more attention to the training instances
that the predecessor underfitted. This results in new predictors focusing more and more on the hard cases.
This is the technique used by AdaBoost.
For example, to build an AdaBoost classifier, a first base classifier (such as a Decision Tree) is trained and
used to make predictions on the training set. The relative weight of misclassified training instances is then
increased. A second classifier is trained using the updated weights and again it makes predictions on the
training set, weights are updated, and so on.

Figure : AdaBoost sequential training with instance weight updates

Once all predictors are trained, the ensemble makes predictions very much like bagging or pasting, except
that predictors have different weights depending on their overall accuracy on the weighted training set.

Note: There is one important drawback to this sequential learning technique: it cannot be parallelized (or
only partially), since each predictor can only be trained after the previous predictor has been trained and
evaluated. As a result, it does not scale as well as bagging or pasting.

1
0
 Let’s take a closer look at the AdaBoost algorithm. Each instance weight is initially set to 1/m.
A first predictor is trained and its weighted error rate r1 is computed on the training set;
Weighted error rate of the jth predictor:

The predictor’s weight αj is then computed using below Equation, where η is the learning rate
hyperparameter (defaults to 1). The more accurate the predictor is, the higher its weight will be. If it is just
guessing randomly, then its weight will be close to zero. However, if it is most often wrong (i.e., less
accurate than random guessing), then its weight will be negative.

Predictor weight:

Next the instance weights are updated using below Equation : the misclassified instances are boosted.

Weight update rule

Then all the instance weights are normalized (i.e., divided by

Finally, a new predictor is trained using the updated weights, and the whole process is repeated (the new
predictor’s weight is computed, the instance weights are updated, then another predictor is trained, and so
on). The algorithm stops when the desired number of predictors is reached, or when a perfect predictor is
found.

AdaBoost simply computes the predictions of all the predictors and weighs them using the predictor weights
αj . The predicted class is the one that receives the majority of weighted votes

1
1
AdaBoost predictions:

Scikit-Learn actually uses a multiclass version of AdaBoost called SAMME (which stands for Stagewise
Additive Modeling using a Multiclass Exponential loss function).When there are just two classes, SAMME
is equivalent to AdaBoost.

Scikit-Learn can use a variant of SAMME called SAMME.R (the R stands for “Real”), which relies on class
probabilities rather than predictions and generally performs better.

The following code trains an AdaBoost classifier based on 200 Decision Stumps using Scikit-Learn’s
AdaBoostClassifier class (as you might expect, there is also an AdaBoostRegressor class). A Decision
Stump is a Decision Tree with max_depth=1 —in other words, a tree composed of a single decision node
plus two leaf nodes. This is the default base estimator for the AdaBoostClassifier class:

Note: If your AdaBoost ensemble is overfitting the training set, you can try reducing the number
ofestimators or more strongly regularizing the base estimator.

Gradient Boosting:

Another very popular Boosting algorithm is Gradient Boosting. Just like AdaBoost, Gradient Boosting
works by sequentially adding predictors to an ensemble, each one correcting its predecessor. However,
instead of tweaking the instance weights at every iteration like AdaBoost does, this method tries to fit the
new predictor to the residual errors made by the previous predictor.

This is called Gradient Tree Boosting, or Gradient Boosted Regression Trees (GBRT).

10
First, let’s fit a DecisionTreeRegressor to the training set (for example, a noisy quadratic train ing set):

A simpler way to train GBRT ensembles is to use Scikit-Learn’s GradientBoostingRe gressor class.

It is worth noting that an optimized implementation of Gradient Boosting is available in the popular python
library XGBoost, which stands for Extreme Gradient Boosting. This package was initially developed by
Tianqi Chen as part of the Distributed (Deep) Machine Learning Community (DMLC), and it aims at being
extremely fast, scalable and portable. In fact, XGBoost is often an important component of the winning
entries in ML competitions. XGBoost’s API is quite similar to Scikit-Learn’s:

11
XGBoost also offers several nice features, such as automatically taking care of early stopping:

Stacking
The last Ensemble method we will discuss in this chapter is called stacking (short for stacked
generalization). It is based on a simple idea: instead of using trivial functions (such as hard voting) to
aggregate the predictions of all predictors in an ensemble, why don’t we train a model to perform this
aggregation? Below Figure shows such an ensemble performing a regression task on a new instance. Each of
the bottom three predictors predicts a different value (3.1, 2.7, and 2.9), and then the final predictor (called a
blender, or a meta learner) takes these predictions as inputs and makes the final prediction (3.0).

Figure : Aggregating predictions using a blending predictor

12
To train the blender, a common approach is to use a hold-out set. Let’s see how it works. First, the training
set is split in two subsets. The first subset is used to train the predictors in the first layer

Figure: Training the first layer

Next, the first layer predictors are used to make predictions on the second (held-out) set (see Below Figure).
This ensures that the predictions are “clean,” since the predictors never saw these instances during training.
Now for each instance in the hold-out set there are three predicted values. We can create a new training set
using these predicted values as input features (which makes this new training set three-dimensional), and
keeping the target values. The blender is trained on this new training set, so it learns to predict the target
value given the first layer’s predictions.

13
 Figure: Training the blender

It is actually possible to train several different blenders this way (e.g., one using Linear Regression, another
using Random Forest Regression, and so on): we get a whole layer of blenders. The trick is to split the
training set into three subsets: the first one is used to train the first layer, the second one is used to create the
training set used to train the second layer (using predictions made by the predictors of the first layer), and the
third one is used to create the training set to train the third layer (using predictions made by the predictors of
the second layer). Once this is done, we can make a prediction for a new instance by going through each
layer sequentially, as shown in below figure:

14
 Figure : Predictions in a multilayer stacking ensemble

Unfortunately, Scikit-Learn does not support stacking directly, but it is not too hard to roll out your own
implementation (see the following exercises). Alternatively, you can use an open source implementation
such as brew (available at https://github.com/ viisar/brew).

30