Department of

CSE
MACHINE LEARNING
21CS2226F

Topic:

Performance metrics

Session - 02
 AIM OF THE
SESSION
To build an accurate and efficient machine learning model that can handle both classification and
regression tasks.

INSTRUCTIONAL
OBJECTIVES

This session is designed to:

1. Understand the metrics to monitor and measure the performance of a machine learning
model.
2. Apply metrics to solve classification and regression problems.

LEARNING OUTCOMES

At the end of this session, you should be able to:

1. Describe the different metrics used to monitor and measure the performance of a machine learning model, and
2. Apply metrics to validate the performance of output generated by a machine learning model.

2
 Performance metrics

• How to validate the performance of output generated by a machine learning model?

• Metrics are needed to monitor and measure the performance of a model.
• In general, machine learning problems have been divided into regression and
classification problems.
• Hence, metrics are divided into:
• Regression metrics
• Classification metrics.

3
 Regression metrics

• Regression models generate continuous output.

• Hence, a distance-based calculation between the predicted output and ground truth data
is essential.
• The most popular metrics to evaluate the regression models are:
• Mean Absolute Error (MAE),
• Mean Squared Error (MSE),
• Root Mean Squared Error (RMSE),
• R² (R-Squared),
• Adjusted R².

4
 Regression metrics
• Mean Absolute Error (MAE)

• Mean Absolute Error is the average of the difference between the ground truth and the
predicted values.
• Mathematically, it is represented as :

• It’s more robust towards outliers.

• Error interpretation needs no second thoughts.
• It gives us a measure of how far the predictions were from the actual output.

5
 Regression metrics
• Mean Squared Error (MSE)

• Mean Squared Error is the average of the squared difference between the ground truth
and the value predicted by the regression model.
• Mathematically, it is represented as :

• It’s more prone to outliers than other metrics.

• It is differentiable; hence it can be optimized better.

6
 Regression metrics
• Root Mean Squared Error (RMSE)

• Root Mean Squared Error is the square root of the average of the squared difference
between the ground truth and the value predicted by the regression model.
• Mathematically, it is represented as :

• Error interpretation can be done smoothly.

• It’s less prone to outliers.
• It is differentiable; hence it can be optimized better.

7
 Regression metrics
• R-Squared (R²)

• The R-squared metric enables to comparison of the model with a constant baseline to
determine the performance of the regression model.
• Mathematically, it is represented as :

• If the sum of the Squared Error of the regression line is small, R² will be close to 1 (Ideal),
meaning the regression was able to capture 100% of the variance in the target variable.

8
 Regression metrics
• Adjusted R²

• When the model overfits the data, the variance will be 100% but the model learning
hasn’t happened. To overcome this problem, R² is adjusted with the number of
independent variables.
• Mathematically, it is represented as :
n = number of observations,

k = number of independent variables

• The adjusted R² always lower than R².
• It only shows improvement if there is a real improvement.

9
 classification metrics

• Classification models generate discrete output.

• Hence, a metric is required that compares discrete classes.
• The most popular metrics to evaluate the classification models are:
• Confusion Matrix,
• Precision and Recall,
• F1-score,
• AU-ROC,
• Accuracy.

10
 classification metrics
• Confusion Matrix

• Confusion Matrix is the easiest way to measure the performance of the classification

• model.
TP signifies how many positive class
True value
samples your model predicted correctly.
1 0
• TN signifies how many negative class
True False
samples your model predicted correctly. 1 Positive Positive
Predicted (TP) (FP)
• FP signifies how many negative class
value False True
samples your model predicted incorrectly. 0 Negative Negative
• FN signifies how many positive class (FN) (TN)

samples your model predicted incorrectly.

11
 classification metrics
• Precision and Recall

• Precision is defined as the ratio of TP to the total number of predictions as positives.

• Mathematically, it is represented as :

• Recall is defined as the ratio of TP to the total number of actual positives.

• Mathematically, it is represented as :

12
 classification metrics
• F1-score

• F1-score is the harmonic mean of precision and recall.

• Mathematically, it is represented as :

• It gives equal importance to precision and recall.

• It presents a good balance between precision and recall and gives good results on
imbalanced classification problems.

13
 classification metrics
• AU-ROC (Area Under Receiver Operating Characteristics
Curve)
• AU-ROC makes use of True Positive Rates (TPR) and False Positive Rates (FPR) to visualize
the performance of the classification model.
• Mathematically, it is represented as :

• High ROC means that the probability of a randomly chosen positive example is indeed
positive.
• ROC curves aren’t a good choice when your problem has a huge class imbalance.

14
 classification metrics
• Accuracy

• Accuracy tells the overall effectiveness of the classifier.

• Mathematically, it is represented as :

N is the total sample size.

• It is the simplest metric to use and implement.

15
 Self-Assessment Questions

1. Which among the following evaluation metrics would you NOT use to measure the
performance of a classification model?

(a) Precision
(b) Recall
(c) Mean Squared Error
(d) F1-score

2. The true-positive rate is also referred to as

(a) Recall
(b) Accuracy
(c) Precision
(d) F1-score

16
 Self-Assessment Questions

3. A single metric which combines both precision and recall is the

(a) Precision
(b) Recall
(c) Mean Squared Error
(d) F1-score

4. What is called the average squared difference between classifier predicted output and actual
output?

(a) Mean Squared Error

(b) Mean Absolute Error
(c) Root Mean Squared Error
(d) Mean Relative Error

17
 REFERENCES FOR FURTHER LEARNING OF THE
SESSION

Text Books:
1. Mitchell, Tom. Machine Learning. New York, NY: McGraw-Hill, 1997. ISBN: 9780070428072.
2. MacKay, David. Information Theory, Inference, and Learning Algorithms. Cambridge, UK: Cambridge University Press,
2003. ISBN: 9780521642989.

Reference Books:
3. EthemAlpaydin “Introduction to Machine Learning “, The MIT Press (2010).
4. Stephen Marsland, “Machine Learning an Algorithmic Perspective” CRC Press, (2009).

Sites and Web links:

5. Data Science and Machine Learning: https://www.edx.org/course/data-science-machinelearning.
2. Machine Learning: https://www.ocw.mit.edu/courses/6-867-machine-learning-fall-2006/.

18
 THANK YOU

Team – MACHINE LEARNING

19