Mohsen Ghodrat

Classification Evaluation
Metrics

©2022 UNIVERSITY CANADA WEST. ALL RIGHTS RESERVED. BUSI 651 – Machine Learning Tools and Techniques
 Classification Metrics

Spam Email Detection

Assume we have designed

a machine learning
classifier to identify spam
emails, and it returns the
following outcomes.

Goal: Evaluate the performance

of the designed classifier.

©2022 UNIVERSITY CANADA WEST. ALL RIGHTS RESERVED. BUSI 651 – Machine Learning Tools and Techniques
 Classification Metrics

Spam Email Detection Correct prediction

Wrong prediction

Assume we have designed

a machine learning
classifier to identify spam
emails, and it returns the
following outcomes.

Goal: Evaluate the performance

of the designed classifier.

©2022 UNIVERSITY CANADA WEST. ALL RIGHTS RESERVED. BUSI 651 – Machine Learning Tools and Techniques
 Classification Metrics

Confusion Matrix
TP
FN
Possible prediction scenarios: FN
TN
TN
TN
Actual Predicted
TN
True Positive TP TN
TN
True Negative TN
TN
False Positive FP FP
False Negative FN TN
FP
TN
TN
TN
TN
FP
TN
TN

©2022 UNIVERSITY CANADA WEST. ALL RIGHTS RESERVED. BUSI 651 – Machine Learning Tools and Techniques
 Classification Metrics

Confusion Matrix
Confusion matrix is a table layout of the
different outcomes of the prediction TP

that summarizes the performance of a FN

FN
classification model. TN
TN

Prediction TN
TN
𝟎 𝟏 TN
TN
TN FP FP
TN
FP
𝟎
TN
14 3 TN
Actual

FP
TN
FN TP TP
TN
TN
𝟏
FN
2 1 TN
FP
TN
TN

©2022 UNIVERSITY CANADA WEST. ALL RIGHTS RESERVED. BUSI 651 – Machine Learning Tools and Techniques
 Classification Metrics

Accuracy
True Negative TN + True Positives (TP)
Accuracy is the proportion of all classifications that were correct. Total Number of Predictions (TN + FP + FN + TP)

Prediction TN FP
14 3
𝟎 𝟏 FN TP
TN TP
2 1 14 1
TN FP FP = = 75%
𝟎
TN
14 3 TN
14
FP
3
TN
14
TP
1
FN
2
FP
3
Actual

FN TP

FN TP 2 1
TP

𝟏
FN
2 1

©2022 UNIVERSITY CANADA WEST. ALL RIGHTS RESERVED. BUSI 651 – Machine Learning Tools and Techniques
 Classification Metrics

Precision
Precision shows how often the classification model is correct when True Positives (TP)
predicting the target class (label 1 in the spam detection example). True Positives TP + False Pasitives (FP)

Prediction TN FP
14 3
𝟎 𝟏 FN TP TP
2 1 1
TN FP FP = = 25%
𝟎
TN
14 3 TN
14
FP
3
TP
1
FP
3
Actual

FN TP

FN TP 2 1
TP

𝟏
FN
2 1

©2022 UNIVERSITY CANADA WEST. ALL RIGHTS RESERVED. BUSI 651 – Machine Learning Tools and Techniques
 Classification Metrics

Recall
Recall shows whether an ML model can find all objects of the target True Positives (TP)
class (label 1 in the spam detection example). True Positives TP + False Negatives (FN)

Prediction TN
14
FP
3
𝟎 𝟏 FN TP TP
2 1 1
TN FP FP = = 33%
𝟎
TN
14 3 TN
14
FP
3
TP
1
FN
2
Actual

FN TP

FN TP 2 1
TP

𝟏
FN
2 1

©2022 UNIVERSITY CANADA WEST. ALL RIGHTS RESERVED. BUSI 651 – Machine Learning Tools and Techniques
 Classification Metrics

Precision vs Recall
Scenario A Scenario B

Prediction Prediction
𝟎 𝟏 𝟎 𝟏
TN FP TN FP
𝟎 𝟎
14 1 14 4

Actual

Actual
FN TP FN TP
𝟏 𝟏
4 1 1 1

Precision: 20% More non-spam emails

Precision: 50%
More spam emails flagged as spam
went undetected Recall: 20% Recall: 50%

©2022 UNIVERSITY CANADA WEST. ALL RIGHTS RESERVED. BUSI 651 – Machine Learning Tools and Techniques
 Classification Metrics

F1-Score
When both Precision and Recall are important, and you need the 2 ∗ Precision ∗ Recall 2 ∗ 25% ∗ 33%
= = 28%
classifier to do well in both, judge the model performance based Precision + Recall 25% + 33%

on F1-Score. It provides a balance between Precision and Recall.

F𝟏

High Precision Low Precision

Low Recall High Recall

Precision 0 100%
Recall 100% 0

©2022 UNIVERSITY CANADA WEST. ALL RIGHTS RESERVED. BUSI 651 – Machine Learning Tools and Techniques
 Thank You!

©2022 UNIVERSITY CANADA WEST. ALL RIGHTS RESERVED. BUSI 651 – Machine Learning Tools and Techniques