Machine Learning

H Dip in DAB/ AI
CCT College Dublin

Decision Trees and Random Forest

Week 3

Lecturer: Dr. Muhammad Iqbal *

©CCT College Dublin 2022

Email: miqbal@cct.ie 1
Agenda
• Decision Tree Classifier
• Classification and Regression Trees
• Example of a Decision Tree
• General Structure based on Recursive
approach
• Decision Trees Splitting
• Categorical Attributes: Computing Gini Index
• CART Algorithm
• Random Forest
• How does the algorithm work?
2
Decision Tree Classifier
• A decision tree is a graph in the shape of a tree, a
sequential diagram that shows all of the potential
decision options and their associated results.

• Starting from the root of a tree, every internal node

represents what a decision is made based on; each
branch of a node represents how a choice may lead
to the next nodes; and finally, each terminal node,
the leaf, represents an outcome yielded.

• For example, we have made a couple of decisions

that brought us to the action of learning decision
tree to solve our advertising problem.

3
 Decision Trees
• Decision Tree is collection of decision nodes, connected by branches, extending downward from
root node to terminating leaf nodes.
• Beginning with root node, attributes tested at decision nodes, and each possible outcome results
in branch.
• Each branch leads to decision node or leaf node.

• Example Custome Saving Asset Income Credit Risk

r s s ($1000s)
• Credit Risk is the target variable. 1 Medium High 75 Good
2 Low Low 50 Bad
• Customers are classified as either “Good 3 High Mediu 25 Bad
Risk” or “Bad Risk”. m
4 Medium Mediu 50 Good
• Predictor variables are Savings (Low, Med, m

High), Assets (Low, High) and Income. 5 Low Mediu

m
100 Good

6 High High 60 Good

7 Low Low 25 Bad 4
Classification and Regression Trees

• Example
• Predict whether customer is classified “Good” or “Bad” credit risk using
three predictor fields, according to data in Table.
• All records enter root node, and CART evaluates possible binary splits.

Custome Saving Asset Income Credit Risk

r s s ($1000s) Root Node
Savings = Low, Med,
1 Medium High 75 Good High?
2 Low Low 50 Bad
Savings = Low Savings = Med Savings = High
3 High Mediu 25 Bad
m Assets = Low? Income <=
Good
4 Medium Mediu 50 Good Risk $30K?
m
Yes No Yes No
5 Low Mediu 100 Good
m
Bad Risk Good Bad Risk Good
6 High High 60 Good Risk Risk
7 Low Low 25 Bad
5
Decision Trees
• For example, all branches terminate at pure leaf nodes. This describes all subsets of records arriving at leaf
nodes with same target class value.
• Diverse leaf node has records with different target class values (“Good Risk” and “Bad Risk”). Algorithm
possibly unable to split
• For example, subset of records has Savings = “High”, Income <= $30,000, and Assets = “Low”. Leaf node
contains 2 “Good Risk”, and 3 “Bad Risk” records
• All records contain same predictor values. No way to split further leading to pure leaf node
• 3/5 records are classified “Bad Risk” with 60% confidence

Custome Saving Asset Income ($1000s) Credit Risk

r s s Root Node
1 Medium High 75 Good Savings = Low, Med,
High?
2 Low Low 50 Bad
Savings = Low Savings = Med Savings = High
3 High Mediu 25 Bad
m Income <=
Assets = Low? Good
4 Medium Mediu 50 Good Risk $30K?
m
Yes No Yes No
5 Low Mediu 100 Good
m
Bad Risk Good Bad Risk Good
6
6 High High 60 Good Risk Risk
Decision Trees
Requirements for Classification

• Decision Tree is supervised classification method.

• The target variable must be categorical.
• Pre-classified target variable must be included in training set.
• Decision trees learn by example, so training set should contain records with
varied attribute values.
• If training set systematically lacks definable subsets, classification becomes
problematic.
• There are different measures for leaf node purity.
• Classification and Regression Trees (CART) and C4.5 are two leading algorithms
used in data analytics. 7
Decision Trees
Best Splitting
• A decision tree is constructed by partitioning the training
samples into successive subsets. The partitioning process is
repeated in a recursive fashion on each subset.

• For each partitioning at a node, a condition test is conducted

based on a value of a feature of the subset.

• When the subset shares the same class label, or no further

splitting can improve the class purity of this subset, recursive
partitioning on this node is finished.

• For a partitioning on a feature (numerical or categorical) with n

different values, there are n different ways of binary splitting
(yes or no to the condition test), not to mention other ways of • CART (Classification and Regression
splitting. Tree): which we will discuss in detail

• Without considering the order of features partitioning takes

place on, there are possible trees for an m-dimensional dataset. 8
Measures of Node Impurity
Gini Index Criteria

• Gini Index (NOTE: p( i | t) is the relative frequency of class i

at node t).
Finding the Best Split
1. Compute impurity measure (P) before splitting
2. Compute impurity measure (M) after splitting
• Compute impurity measure of each child node
• M is the weighted impurity of children

3. Choose the attribute test condition that produces the highest gain

Gain = P – M
or equivalently, lowest impurity measure after splitting (M)

9
 Measures of Node Impurity
Gini Index
• Gini Index for a given node t:

(NOTE: p( i | t) is the relative frequency of class i at node t).

• Maximum (1 - 1/nc) when records are equally distributed among all classes
• Minimum (0.0) when all records belong to one class
• For 2-class or binary problem (p, 1 – p):
• GINI = 1 – p2 – (1 – p)2 = 2p (1 – p)
C1 0 C1 1 C1 2 C1 3
C2 6 C2 5 C2 4 C2 3
Gini=0.000 Gini=0.278 Gini=0.444 Gini=0.500

P(C1) = 0/6 = 0 P(C2) = 6/6 = 1 P(C1) = 1/6 P(C2) = 5/6 P(C1) = 2/6 P(C2) = 4/6 P(C1) = 3/6 P(C2) = 3/6

Gini = 1 – P(C1)2 – P(C2)2 = 1 – 0 – 1 = 0 Gini = 1 – (1/6)2 – (5/6)2 = 0.278 Gini = 1 – (2/6)2 – (4/6)2 = 0.444 Gini = 1 – (3/6)2 – (3/6)2 = 0.5
10
 CART Algorithm
Example

• Data Set
• There are 14 instances of golf playing
decisions based on outlook,
temperature, humidity and wind factors.

• Gini index
• Gini index is a metric for classification
tasks in CART. It stores sum of squared
probabilities of each class. We can
formulate it as illustrated below.
• Gini = 1 – Σ (Pi)2 for i = 1 to number of
classes https://dataaspirant.com/how-decision-tree-algorithm-works/ 11
 12
https://dataaspirant.com/how-decision-tree-algorithm-works/
 Splitting is occurred at minimum gini
index

All are Yes

https://dataaspirant.com/how-decision-tree-algorithm-works/ 13
https://dataaspirant.com/how-decision-tree-algorithm-works/ 14
https://dataaspirant.com/how-decision-tree-algorithm-works/ 15
https://dataaspirant.com/how-decision-tree-algorithm-works/ 16
 Random Forest
• Random forests is a supervised learning algorithm. It can be used both
for classification and regression. It is the most flexible and easy to use
algorithm.

• A forest is comprised of trees. We can get a robust forest if we have

more trees.

• Random forests create decision trees on randomly selected data

samples, gets prediction from each tree and selects the best solution by
means of voting.

• It provides a pretty good indicator of the feature importance.

• Random forests have a variety of applications, such as recommendation

engines, image classification and feature selection.

• It can be used to classify loyal loan applicants, identify fraudulent

17
Random Forest: Example
• Suppose you want to go on a trip and you would like to travel to a place which you will enjoy.

• So what do you do to find a place that you will like? You can search online, read reviews on travel blogs and
portals, or you can also ask your friends.

• Let’s suppose you have decided to ask your friends, and talked with them about their past travel experience
to various places. You will get some recommendations from every friend. Now you have to make a list of
those recommended places. Then, you ask them to vote (or select one best place for the trip) from the list of
recommended places you made. The place with the highest number of votes will be your final choice for the
trip.

• In the above decision process, there are two parts. First, asking your friends about their individual travel
experience and getting one recommendation out of multiple places they have visited.

• This part is like using the decision tree algorithm. In this scenario, each friend makes a selection of the places
he or she has visited so far.

• The second part, after collecting all the recommendations, is the voting procedure for selecting the best place
in the list of recommendations. This whole process of getting recommendations from friends and voting on 18
How does the algorithm work?
• It works in four steps:
1. Select random samples from a given dataset.
2. Construct a decision tree for each sample and get a prediction
result from each decision tree.
3. Perform a vote for each predicted result.
4. Select the prediction result with the most votes as the final
prediction.
Random Forests vs Decision Trees
• Random forests is a set of multiple decision trees. Working diagram of random forest

• Deep decision trees may suffer from overfitting, but the random forests prevent overfitting by creating trees
on random subsets.

• Decision trees are computationally faster.

• Random forests is difficult to interpret, while a decision tree is easily interpretable and can be converted to19
Resources/ References
• Introduction to Machine Learning with Python, Andreas C. Müller and
Sarah Guido, O'Reilly Media, Inc. October 2016.
• Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow,
2nd Edition, Aurélien Géron, O'Reilly Media, September 2019, ISBN: Copyright Notice
9781492032649. The following material has been
communicated to you by or on behalf of
• Python Machine Learning - Third Edition, Sebastian Raschka, Vahid CCT College Dublin in accordance with the
Mirjalili, Copyright © 2017 Packt Publishing. Copyright and Related Rights Act 2000 (the
Act).
• Discovering Knowledge In Data: An Introduction To Data Exploration, The material may be subject to copyright
Second Edition, By Daniel Larose And Chantal Larose, John Wiley And under the Act and any further
Sons, Inc., 2014. reproduction, communication or
• UCI Repository: http://www.ics.uci.edu/~mlearn/MLRepository.html distribution of this material must be in
accordance with the Act.
• Understanding Autoencoders. (Part I) | by Jelaleddin Sultanov | AI³ |
Do not remove this notice
Theory, Practice, Business | Medium
• Statlib: http://lib.stat.cmu.edu
• Some images are used from Google search repository
(https://www.google.ie/search) to enhance the level of learning.
20