CLASSIFICATION

CLASSIFICATION
 Task of assigning objects to one of several predefined categories.
 Examples
 Predicting tumor cells as benign or malignant
 Classifying credit card transactions as legitimate or fraudulent
 Detecting spam email messages based on message header and content
 Categorizing news stories as finance, weather, entertainment, sports, etc.
 Classifying galaxies based on their shape
CLASSIFICATION

 Input data is collection of records.

 Each record is characterized by tuple (x,y), where x is the attribute set and y is a
special attribute designated as class label.
 The attribute set could be continuous or discrete, but the class label should be
discrete.
 CLASSIFICATION
 Classification is the task of learning a target function f that maps each
attribute set x to one of the predefined class labels y.
 The target function is also informally known as classification model.
 A classification model is useful for the following purposes:
 Descriptive Modeling : A classification model can serve as an
explanatory tool to distinguish between objects of different classes.
 A descriptive model which can summarize the features that define a
vertebrate as a mammal, reptile, bird, fish or amphibian.
 Predictive Modeling: A classification model can be used to predict class
labels of unknown records.
CLASSIFICATION

 Classification is most suited for techniques for

predicting or describing datasets with binary
or nominal categories.
 Less suitable for ordinal categories as they do
not consider the implicit ordering among the
categories.
 Other forms of relationships, such as the
subclass-superclass relationships among
categories are also ignored.
GENERAL APPROACH

 A classification approach (or classifier) is a systematic approach to build classification

models from an input data set.
 Examples: Decision Tree Classifiers, Neural Networks, Support Vector Machines,
Naïve Bayes Classifier.
 Each technique employs a learning algorithm to identify a model that best fits the
relationship between the attribute set and class label of the input data.
 The model generated by the learning algorithm should both fit the input data well
and correctly predict the class label of records it has never seen before.
 The key objective of the learning algorithm is to build models with good
generalization capability.
GENERAL APPROACH

 Given a collection of records (training set )

 Each record contains a set of attributes, one of the attributes is the class.
 Find a model for class attribute as a function of the values of other attributes.
 Goal: previously unseen records should be assigned a class as accurately as possible.
 A test set is used to determine the accuracy of the model. Usually, the given data
set is divided into training and test sets, with training set used to build the model
and test set used to validate it.
 Illustration of Classification Task
ILLUSTRATION OF CLASSIFICATION TASK
Tid Attrib1 Attrib2 Attrib3 Class Learning
1 Yes Large 125K No
algorithm
2 No Medium 100K No

3 No Small 70K No

4 Yes Medium 120K No

Induction
5 No Large 95K Yes

6 No Medium 60K No

7 Yes Large 220K No Learn

8 No Small 85K Yes Model
9 No Medium 75K No

10 No Small 90K Yes

Model
10

Training Set
Apply
Tid Attrib1 Attrib2 Attrib3 Class Model
11 No Small 55K ?

12 Yes Medium 80K ?

13 Yes Large 110K ? Deduction

14 No Small 95K ?

15 No Large 67K ?
10

Test Set
CONFUSION MATRIX

 Confusion Matrix: Tabulates the counts of

test records correctly and incorrectly
predicted by the model.
 It is used to evaluate the performance of a
classification model.
 Total Number of Correct Predictions made
by the model
 TP +TN
 Total Number of Incorrect Predictions
 FP + FN
 Performance Metric: Accuracy
𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝐶𝑜𝑟𝑟𝑒𝑐𝑡 𝑃𝑟𝑒𝑑𝑖𝑐𝑡𝑖𝑜𝑛𝑠 𝑇𝑃+𝑇𝑁
 Accuracy = =
𝑇𝑜𝑡𝑎𝑙 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑝𝑟𝑒𝑑𝑖𝑐𝑡𝑖𝑜𝑛𝑠 𝑇𝑃+𝐹𝑃+𝐹𝑁+𝑇𝑁
𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝐼𝑛𝑐𝑜𝑟𝑟𝑒𝑐𝑡 𝑃𝑟𝑒𝑑𝑖𝑐𝑡𝑖𝑜𝑛𝑠 𝐹𝑃+𝐹𝑁
 Error Rate = =
𝑇𝑜𝑡𝑎𝑙 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑝𝑟𝑒𝑑𝑖𝑐𝑡𝑖𝑜𝑛𝑠 𝑇𝑃+𝐹𝑃+𝐹𝑁+𝑇𝑁

 Classification algorithms seek models that attain the highest accuracy or

equivalently the lowest error rate.
HOW A DECISION TREE WORKS
 Vertebrate Classification
 Attributes:
 Name (Whale, Frog, Cat, Leopard)
 Body Temperature (Warm Blooded/ Cold Blooded)
 Skin Cover (Hair/ Scales/ Fur/ Feather)
 Gives Birth (Yes/No)
 Aquatic Creature (Yes/No)
 Aerial Creature (Yes/No)
 Has Legs (Yes/No)
 Hibernates (Yes/No)
 Class Label: Mammal/ Non-mammal
HOW A DECISION TREE WORKS
 Suppose a new species is discovered by scientists. How can we tell whether it is a
mammal or a non-mammal?
 An approach is to pose a series of questions about the characteristics of species.
 Q1: Is the species warm blooded or cold blooded?
 If cold blooded definitely not a mammal. Otherwise it is a bird or a mammal.
 Q2: Do the females of the species give birth to their young?
 Those that give birth are definitely mammals and those that do not are likely to be
non-mammals.
 Each time we receive an answer a follow-up question is asked until we reach the
conclusion about the class label of the record.
HOW A DECISION TREE WORKS?

 The series of questions and their possible answers can be organized in the form of
decision tree.
 It is a hierarchical structure that consists of nodes and directed edges.
A tree has three types of nodes
• Root Node: One root node with no incoming
edges and zero or more outgoing edges.
• Internal Nodes: Exactly one incoming edge and
two or more outgoing edges.
• Leaf or Terminal Nodes: Exactly one incoming
edge and no outgoing edges.
HOW A DECISION TREE WORKS?

 Each leaf node is assigned a class label.

 The non-terminal nodes, which
include the root and other internal
nodes, contain attribute test
conditions to separate records that
have different characteristics.
HOW A DECISION TREE WORKS?

 In order to classify a test record, apply test condition to the

record and follow the appropriate branch based on the
outcome of the test.
 This will lead to either an internal node, for which a new
test condition is applied or to a leaf node.
 The class label associated with the leaf node is then
assigned to the record.
EXAMPLE OF A DECISION TREE
Splitting Attributes

Tid Refund Marital Taxable Refund

Status Income Cheat
Yes No
2,3,5,6,8,9,10
1 Yes Single 125K No
NO MarSt
2 No Married 100K No
Single, Divorced Married
3 No Single 70K No
3,5,8,10 2,6,9
4 Yes Married 120K No TaxInc NO
5 No Divorced 95K Yes
< 80K > 80K
6 No Married 60K No
7 Yes Divorced 220K No NO YES
8 No Single 85K Yes
9 No Married 75K No
10 No Single 90K Yes
Model: Decision Tree
10

Training Data
ANOTHER EXAMPLE OF DECISION TREE

Tid Refund Marital Taxable MarSt Single,

Status Income Cheat Married Divorced
1 Yes Single 125K No
NO Refund
2 No Married 100K No
Yes No
3 No Single 70K No
4 Yes Married 120K No NO TaxInc
5 No Divorced 95K Yes < 80K > 80K
6 No Married 60K No
NO YES
7 Yes Divorced 220K No
8 No Single 85K Yes
9 No Married 75K No
There could be more than one tree that fits
the same data!
10 No Single 90K Yes
10
DECISION TREE CLASSIFICATION TASK

Tid Attrib1 Attrib2 Attrib3 Class

Tree
1 Yes Large 125K No Induction
2 No Medium 100K No algorithm
3 No Small 70K No

4 Yes Medium 120K No

Induction
5 No Large 95K Yes

6 No Medium 60K No

7 Yes Large 220K No Learn

8 No Small 85K Yes Model
9 No Medium 75K No

10 No Small 90K Yes

Model
10

Training Set
Apply Decision
Tid Attrib1 Attrib2 Attrib3 Class
Model Tree
11 No Small 55K ?

12 Yes Medium 80K ?

13 Yes Large 110K ?

Deduction
14 No Small 95K ?

15 No Large 67K ?
10

Test Set
APPLY MODEL TO TEST DATA
Test Data
Start from the root of tree.
Refund Marital Taxable
Status Income Cheat

No Married 80K ?
Refund 10

Yes No

NO MarSt
Single, Divorced Married

TaxInc NO
< 80K > 80K

NO YES
APPLY MODEL TO TEST DATA

Test Data
Refund Marital Taxable
Status Income Cheat

No Married 80K ?
Refund 10

Yes No

NO MarSt
Single, Divorced Married

TaxInc NO
< 80K > 80K

NO YES
 Test Data
APPLY MODEL TO TEST DATA
Refund Marital Taxable
Status Income Cheat

No Married 80K ?
Refund 10

Yes No

NO MarSt
Single, Divorced Married

TaxInc NO
< 80K > 80K

NO YES
 Test Data
APPLY MODEL TO TEST DATA
Refund Marital Taxable
Status Income Cheat

No Married 80K ?
Refund 10

Yes No

NO MarSt
Single, Divorced Married

TaxInc NO
< 80K > 80K

NO YES
 Test Data
APPLY MODEL TO TEST DATA
Refund Marital Taxable
Status Income Cheat

No Married 80K ?
Refund 10

Yes No

NO MarSt
Single, Divorced Married

TaxInc NO
< 80K > 80K

NO YES
 Test Data
APPLY MODEL TO TEST DATA
Refund Marital Taxable
Status Income Cheat

No Married 80K ?
Refund 10

Yes No

NO MarSt
Single, Divorced Married Assign Cheat to “No”

TaxInc NO
< 80K > 80K

NO YES
DECISION TREE CLASSIFICATION TASK
Tid Attrib1 Attrib2 Attrib3 Class
Tree
1 Yes Large 125K No Induction Decision
2 No Medium 100K No algorithm Tree
3 No Small 70K No

4 Yes Medium 120K No

Induction
5 No Large 95K Yes

6 No Medium 60K No

7 Yes Large 220K No Learn

8 No Small 85K Yes Model
9 No Medium 75K No

10 No Small 90K Yes

Model
10

Training Set
Apply
Tid Attrib1 Attrib2 Attrib3 Class
Model
11 No Small 55K ?

12 Yes Medium 80K ?

13 Yes Large 110K ?

Deduction
14 No Small 95K ?

15 No Large 67K ?
10

Test Set
DECISION TREE INDUCTION

 Many Algorithms:
 Hunt’s Algorithm (one of the earliest)
 CART
 ID3, C4.5
 SLIQ,SPRINT
GENERAL STRUCTURE OF HUNT’S ALGORITHM
Tid Refund Marital Taxable
Cheat
 Let Dt be the set of training records that reach a
Status Income

node t 1 Yes Single 125K No

2 No Married 100K No
 General Procedure: 3 No Single 70K No
4 Yes Married 120K No
 If Dt contains records that belong the same 5 No Divorced 95K Yes
class yt, then t is a leaf node labeled as yt 6 No Married 60K No

 If Dt is an empty set, then t is a leaf node 7 Yes Divorced 220K No

labeled by the default class, yd 8

9
No
No
Single
Married
85K
75K
Yes
No

 If Dt contains records that belong to more 10 No Single 90K Yes

than one class, use an attribute test to split

10

Dt
the data into smaller subsets. Recursively apply
the procedure to each subset. ?
 Tid Refund Marital Taxable
Status Income Cheat

HUNT’S ALGORITHM
Don’t
Refund Hunt’s 1 Yes Single 125K No
Yes No
Cheat Algorithm 2 No Married 100K No
Don’t Don’t 3 No Single 70K No
Cheat Cheat 4 Yes Married 120K No
5 No Divorced 95K Yes
6 No Married 60K No

Refund Refund 7 Yes Divorced 220K No

Yes No Yes No 8 No Single 85K Yes

Don’t Don’t Marital

9 No Married 75K No
Marital
Cheat Status
Cheat Status 10 No Single 90K Yes
Single, Single, 10

Married Married
Divorced Divorced
Don’t Taxable Don’t
Cheat Cheat
Cheat Income
< 80K >= 80K
Don’t Cheat
Cheat
 Predicting whether a loan applicant will repay her loan
obligations or become delinquent, subsequently defaulting in
HUNT’S ALGORITHM
her loan

• The initial tree contains a single label with class label Defaulted = “NO”, which means most
borrowers successfully repaid their loans.
• However, the tree needs refinement.
HUNT’S ALGORITHM

 Hunt’s algorithm will work if every combination of attribute values is present in the
training data and each combination has a unique class label.
 These assumptions are too stringent for use in most practical scenarios.
 Additional conditions are needed to handle the following cases:
 It is possible for some of the child nodes created be empty; i.e. there are no
records associated with these nodes. This can happen if none of the training
records have the combination of attribute values associated with such nodes.
 In this case the node is declared a leaf node with the same class label as the
majority class of training records associated with its parent node.
HUNT’S ALGORITHM

 If all the records associated with Dt have identical attribute values (except for the
class label), then it is not possible to split the records any further.
 In this case, the node is declared a leaf node with the same class label as the
majority class of training records associated with this node.
TREE INDUCTION

 Greedy strategy.
 Split the records based on an attribute test that optimizes certain criterion.
 Issues
 Determine how to split the records
 How to specify the attribute test condition?
 How to determine the best split?
 Determine when to stop splitting
 Continue expanding a node until either all records belong to the same class or all
records have identical attribute values
METHODS FOR EXPRESSING ATTRIBUTE TEST CONDITIONS

 Depends on attribute types

 Nominal
 Ordinal
 Continuous
 Depends on number of ways to split
 2-way split
 Multi-way split
 SPLITTING BASED ON NOMINAL ATTRIBUTES

 Multi-way split: Use as many partitions as distinct values.

CarType
Family Luxury
Sports

 Binary split: Divides values into two subsets. Need to find optimal partitioning.

OR CarType
CarType {Family,
{Sports, Luxury} {Sports}
Luxury} {Family}
SPLITTING BASED ON ORDINAL ATTRIBUTES

 Can produce binary or multiway

splits.
 They can be grouped as long as
the grouping does not violate the
order property of the attribute
values.
SPLITTING BASED ON ORDINAL ATTRIBUTES
 Multi-way split: Use as many partitions as distinct values.

Size
Small Large
Medium

 Binary split: Divides values into two subsets.

Need to find optimal partitioning.

Size Size
{Small,
{Large}
OR {Medium,
{Small}
Medium} Large}

 What about this split?

Size
{Small,
Large} {Medium}
SPLITTING BASED ON CONTINUOUS ATTRIBUTES

 Different ways of handling

 Discretization to form an ordinal categorical attribute
 Static – discretize once at the beginning
 Dynamic – ranges can be found by equal interval bucketing, equal frequency bucketing
(percentiles), or clustering.

 Binary Decision: (A < v) or (A  v)

 consider all possible splits and finds the best cut
 can be more computation intensive
SPLITTING BASED ON CONTINUOUS ATTRIBUTES

Taxable Taxable
Income Income?
> 80K?
< 10K > 80K
Yes No

[10K,25K) [25K,50K) [50K,80K)

(i) Binary split (ii) Multi-way split

HOW TO DETERMINE THE BEST SPLIT?
 Various measures can be used to determine the best way to split the records.
 These measures are defined in terms of class distribution of records before and after
splitting.
 Let p(i|t) denote the fraction of records belonging to class i at a given node t. We
sometimes omit the reference to node t and express the fraction as pi.
 Greedy approach:
 Nodes with homogeneous class distribution are preferred
 Need a measure of node impurity: C0: 5 C0: 9
C1: 5 C1: 1
Non-homogeneous, Homogeneous,
High degree of impurity Low degree of impurity
HOW TO DETERMINE THE BEST SPLIT?

Before Splitting: 10 records of class 0,

10 records of class 1

Own Car Student

Car? Type? ID?

Yes No Family Luxury c1 c20

c10 c11
Sports
C0: 6 C0: 4 C0: 1 C0: 8 C0: 1 C0: 1 ... C0: 1 C0: 0 ... C0: 0
C1: 4 C1: 6 C1: 3 C1: 0 C1: 7 C1: 0 C1: 0 C1: 1 C1: 1

Which test condition is the best?

HOW TO DETERMINE THE BEST SPLIT?

 The measure developed for selecting the best split are often based on the degree of
impurity of the child nodes.
 The smaller the degree of impurity, the more skewed the class distribution.
 A node with class distribution (0,1) has zero impurity and a node with uniform class
distribution (0.5,0.5) has the highest impurity.
MEASURES OF NODE IMPURITY

 Gini Index

 Entropy

 Misclassification error
 ALGORITHM

 The best attribute is selected and used as the test at the root node of the tree.
 A descendant of the root node is then created for each possible value of this
attribute, and the training examples are sorted to the appropriate descendant
node (i.e., down the branch corresponding to the example's value for this
attribute).
 The entire process is then repeated using the training examples associated with
each descendant node to select the best attribute to test at that point in the
tree.
 This forms a greedy search for an acceptable decision tree, in which the
algorithm never backtracks to reconsider earlier choices.
ID3 ALGORITHM
Examples: Training Examples
Target-attribute: Attribute to be predicted
Attributes: List of predictors

ID3 (Examples ,Target-attribute, Attributes)

 Create a root node for the tree.
 If all examples are positive, return the single-node tree Root, with label = +
 If all examples are negative, return the single-node tree Root, with label = -
 If Attributes is empty, return the single-node tree Root with label = most common
value of Target-attribute in Examples.
ID3 Otherwise
ALGORITHM Begin ID3 ALGORITHM
 A  the attribute from Attributes that best classifies Examples
 The decision attribute for Root  A
 For each possible value vi of A
 Add a new tree branch below Root, corresponding to the test A= vi
 Let Examples vi be the subset of Examples that have vi for A
 If Examples vi is empty or below a certain threshold (Eg. 5%)
 Then below this new branch add a leaf node with label = most common value of
Target-attribute in Examples
 Else below this new branch add the subtree
ID3(Examples vi, Target-attribute, Attributes –{A})
 End
 Return Root
WHICH ATTRIBUTE IS THE BEST CLASSIFIER?

 We would like to select the attribute that is most useful for classifying examples.
 We will define a statistical property, called information gain, that
measures how well a given attribute separates the training examples
according to their target classification
ENTROPY

 A measure used from Information Theory in the ID3 algorithm and popularly used in
decision tree construction is that of Entropy.
 Entropy of a dataset measures the impurity of the dataset.
 Informally, Entropy can be considered to find out how disordered the dataset is.
ENTROPY

 It has been shown that there is a relationship between entropy and information.
 That is, higher the uncertainty or entropy of some data, implies more information is
required to completely describe that data.
 In building a decision tree, the aim is to decrease the entropy of the dataset until we
reach leaf nodes at which point the subset that we are left with is pure, or has zero
entropy and represents instances all of one class.
 ENTROPY
 Suppose C is class distribution and A is a group of attributes, the Entropy (H) of a
random variable is defined as

◼ where C has proportion of positive and the proportion

of negative examples.
ENTROPY

𝐻 𝐶𝐴 =− 𝑝(𝑎) 𝑝(𝑐|𝑎) log 2 (𝑝 𝑐 𝑎 )

𝑎∈𝐴 𝑐∈𝐶

 Conditional entropy H(C|A ) is the entropy of one random

variable conditional upon knowledge of another.
INFOGAIN
 In terms of decision trees, the information gain (Infogain) is based on the decrease
in entropy after a dataset is split on an attribute i.e. the mutual information given a
specific attribute.
 Constructing a decision tree is all about finding attribute that returns the
highest information gain (mutual information).
ALTERNATIVE SPLITTING CRITERIA BASED ON INFO

 Entropy at a given node t:

Entropy(t ) = − p( j | t ) log p( j | t )
j

(NOTE: p( j | t) is the relative frequency of class j at node t).

 Measures homogeneity of a node.
 Maximum (log nc) when records are equally distributed among all
classes implying least information
 Minimum (0.0) when all records belong to one class, implying most
interesting information
EXAMPLES FOR COMPUTING ENTROPY

Entropy(t ) = − p( j | t ) log p( j | t )
j 2

C1 0 P(C1) = 0/6 = 0 P(C2) = 6/6 = 1

C2 6 Entropy = – 0 log 0 – 1 log 1 = – 0 – 0 = 0

C1 1 P(C1) = 1/6 P(C2) = 5/6

C2 5 Entropy = – (1/6) log2 (1/6) – (5/6) log2 (1/6) = 0.65

C1 2 P(C1) = 2/6 P(C2) = 4/6

C2 4 Entropy = – (2/6) log2 (2/6) – (4/6) log2 (4/6) = 0.92
GAIN

 Decision tree induction algorithms choose a test condition that maximizes the gain
(∆).
 As, I (Parent) is same for all test conditions, maximizing the gain is equivalent to
minimizing the weighted average impurity measures of the child nodes.
 When entropy is used as an impurity measure, the difference in entropy is known as
information gain, ∆ gain
HOW TO FIND THE BEST SPLIT
C0 N00
Before Splitting: C1 N01 M0
A? B?
Yes No Yes No

Node N1 Node N2 Node N3 Node N4

C0 N10 C0 N20 C0 N30 C0 N40

C1 N11 C1 N21 C1 N31 C1 N41

M1 M2 M3 M4

M12 M34
Gain = M0 – M12 vs M0 – M34
EXAMPLE
WHICH ATTRIBUTE TO SELECT?
 EXAMPLE
 There are two classes, play tennis, “yes” and “no” in the data. Therefore the entropy can be
calculated as below:
Entropy(S) = -pyes log2(pyes) -pno log2(pno)

= -(9/14) * log2(9/14) -(5/14) * log2(5/14)

= -0.643 log2 0.643 -0.357 log2 0.357

= 0.410 + 0.530 = 0.94

EXAMPLE

 Now, the next step is to select highly significant input variable among all the four
input variables (Outlook, Temperature, Humidity, Wind) that will split the data more
purely than the other attributes.
 For this, we will calculate the information gain that would result in over the entire
dataset after splitting the attributes (Outlook, Temperature, Humidity, Wind).
EXAMPLE
Infogain (S|Outlook) = Entropy(S) -5/14 Entropy (S|Outlook =Sunny) – 4/14
Entropy(S|Outlook =Overcast) – 5/14 Entropy(S|Outlook =Rain)

= 0.94 –(5/14) (-pyes log2(pyes) -pno log2(pno) ) –(4/14) (-pyes log2(pyes) -pno log2(pno) ) –
(5/14) (-pyes log2(pyes) -pno log2(pno) )

= 0.94 -(5/14) (-2/5 log2 2/5 -3/5 log2 3/5) -(4/14) (-4/4 log2 4/4) -(5/14) (-3/5 log2 3/5 -
2/5 log2 2/5)
= 0.94 – 0.347 – 0 -0.347
= 0.246 bits
EXAMPLE
Infogain (S|Temperature) = Entropy(S) -4/14 Entropy (S| Temperature =Hot) – 6/14
Entropy(S| Temperature =Mild) – 4/14 Entropy(S|Temperature =Cold)

= 0.94 –(4/14) (-pyes log2(pyes) -pno log2(pno) ) –(6/14) (-pyes log2(pyes) -pno log2(pno) ) –
(4/14) (-pyes log2(pyes) -pno log2(pno) )

= 0.94 -(4/14) (-2/4 log2 2/4 -2/4 log2 2/4) -(6/14) (-4/6 log2 4/6 -2/6 log2 2/6) -(4/14) (-
3/4 log2 3/4 -1/4 log2 1/4)
= 0.94 – 0.286 – 0.392 -0.233
= 0.029 bits
 EXAMPLE
Infogain (S|Humidity) = Entropy(S) -7/14 Entropy (S| Humidity =High) – 7/14 Entropy(S|
Humidity =Normal)

= 0.94 –(7/14) (-pyes log2(pyes) -pno log2(pno) ) –(7/14) (-pyes log2(pyes) -pno log2(pno) )

= 0.94 -(7/14) (-3/7 log2 3/7 -4/7 log2 4/7) -(7/14) (-6/7 log2 6/7 -1/7 log2 1/7)
= 0.94 – 0.493 – 0.296 = 0.151 bits
 EXAMPLE
Infogain (S|Wind) = Entropy(S) -6/14 Entropy (S| Wind =Strong) – 8/14 Entropy(S|
Wind = Weak)

= 0.94 –(6/14) (-pyes log2(pyes) -pno log2(pno) ) –(8/14) (-pyes log2(pyes) -pno log2(pno) )

= 0.94 -(6/14) (-1/2 log2 1/2 -1/2 log2 1/2) -(8/14) (-6/8 log2 6/8 -2/8 log2 2/8)
= 0.94 – 0.428 – 0.465 = 0.047 bits
 EXAMPLE
 Now, we select the attribute for the root node which will result in the highest
reduction in entropy.
Infogain (S| Outlook) = 0.246 bits
Infogain (S| Temperature) = 0.029 bits
Infogain (S| Humidity) = 0.151 bits
Infogain (S|Wind) = 0.047 bits
 We can clearly see that the attribute Outlook results in the highest reduction in
entropy or the highest information gain.
 We would therefore select Outlook at the root node, splitting the data up into subsets
corresponding to all the different values for the Outlook attribute.
HOLDOUT METHOD

 The original data with labelled examples is partitioned into two disjoint sets, called
the training set and test set.
 A classification model is then induced from the training set and its performance is
evaluated on the test set.
 The proportion of data reserved for training and for testing is typically at the
discretion of the analysts (Eg. 50-50 or two-thirds for training and one-third for
testing).
 The accuracy of the classifier can be estimated based on the accuracy of the induced
model on the test set.
 HOLDOUT METHOD
 Limitations
 Fewer labelled examples are available for training because some of the records are
withheld for testing. The induced model may not be as good as when all the labelled
examples are used for training.
 The model may be highly dependent on the composition of the training and tests
sets.
 The smaller the training set size, the larger is the variance of the model.
 On the other hand, if the training set is too large, then the estimated accuracy computed
from the smaller test set is less reliable. Such an estimate is said to have a wide
confidence interval.
 HOLDOUT METHOD
 Limitations
 The training and test sets are
no longer independent of each
other. Because the training
and test sets are subsets of
the original data, a class that is
overrepresented in one subset
will be underrepresented in
the other, and vice versa.
RANDOM SUBSAMPLING

 The holdout method can be repeated

several times to improve the estimation
of a classifier’s performance. Known as
random subsampling.
 Let acci be the model accuracy during the
ith iteration. The overall accuracy is given
by
𝑎𝑐𝑐𝑖
accsub = σ𝑘𝑖=1
𝑘
RANDOM SUBSAMPLING

 Limitations
 It does not utilize as much as data as possible for training.
 It has no control over the number of times each record is used for testing and
training. Consequently, some records might be used for training more often than
others.
CROSS-VALIDATION
 In this approach, each record is used the same
number of times for training and exactly once for
testing.
 Example:
 Partition the data three equal-sized subsets.
 Choose two of the subsets (A, B) for training and
the remaining one for testing.
 Choose two of the other combination (A,C) of
subsets for training and the remaining one for
testing.
 Choose the remaining one combination (B,C) for
training and remaining one for testing.
 This approach is called three-fold cross validation.
CROSS-VALIDATION

 The k-fold cross validation method generalizes this approach by segmenting the data
into k-equal sized partitions.
 During each runs, one of the partition is used for testing, while the rest of them are
used for training.
 This procedure is repeated k times so that each partition is used for testing exactly
once.
 The total error is found by summing up the errors for all the k runs.
CROSS-VALIDATION

 A special case of k-fold cross-validation method sets k=N, the size of the dataset.
 Called leave-one-out approach, each test set contains only one record.
 This approach has the advantage of utilizing as much data as possible for training.
 In addition, the test sets are mutually exclusive and they effectively cover the
entire data set.
 The drawback of this approach is that it is computationally expensive to repeat the
procedure N times.
 Furthermore, since each test contains only one record, the variance of the
estimated performance metric tends to be high.
BOOTSTRAP
 The methods so far assume that the training records are sampled without
replacement. As a result, there are no duplicate records in the training and test sets.
 Bootstrap Approach: Training records sampled with replacement.
 If the original data has N records, it can be shown that, on average, a bootstrap
sample of size N contains about 63.2% of the records of the original data.
 Records that are not included in bootstrap sample, become a part of the test set.
 The module induced from the training set is then applied to the test set to obtain an
estimate of the accuracy of the bootstrap sample, εi.
 The sampling procedure is then repeated b times to generate b bootstrap samples.
BOOTSTRAP

 .632 Bootstrap
 It computes the overall accuracy by combining the accuracies of each bootstrap
sample (εi) with the accuracy computed from the training set that contains all
labelled examples in the original data (accs) i.e. no repeated labelled examples.

1
Accuracy (accboost) = σ𝑏𝑖=1 0.632 ∗ ∈ 𝑖 + 0.368 ∗ 𝑎𝑐𝑐𝑠
𝑏