Introduction

Formalism and terminology

Tuning and evaluation

Data Science Lecture Notes 07

Donghui Yan

University of Massachusetts Dartmouth

Donghui Yan UMassD Lecture Notes

 Introduction
Formalism and terminology
Tuning and evaluation

Outline

• Introduction
• Formalism and terminology
• Evaluation methodology

Donghui Yan UMassD Lecture Notes

 Introduction
Formalism and terminology
Tuning and evaluation

Machine learning in real life

• Search engine design

I To max chance one gets what he searches in top K entries
• Computational advertising
I Placement of ads to maximize profit
• Design of e-commerce web site
I Selection of selling items to max click thru rate (or profit)
• Selection of headline news
I e.g., which news as headline in news portal at Yahoo, CNN etc
• Object recognition
I OCT hand digits recognition by USPS
• House (pool) cleaning robot.

Donghui Yan UMassD Lecture Notes

 Introduction
Formalism and terminology
Tuning and evaluation

State of the art in AI

• AlphaGo
I Beat Ke Jie (ranked #1 in world) 3:0 in 2017
I Major milestone in AI research
• Self-driving
• Conversation robot.

Donghui Yan UMassD Lecture Notes

 Introduction
Formalism and terminology
Tuning and evaluation

What is machine learning (ML)?

Definition. Machine learning refers to application, methodology,

and theory relevant to the automatic learning of patterns or
regularities from data.

Donghui Yan UMassD Lecture Notes

 Introduction
Formalism and terminology
Tuning and evaluation

Two important assumptions in Machine learning

• The future is related to the past
I The phenomenon is stationary, or the past and future drawn from
the same probability distribution
• Knowledge about the problem under study
I Generalization only possible when knowledge is encoded
I Features being the most elementary form.

Donghui Yan UMassD Lecture Notes

 Introduction
Formalism and terminology
Tuning and evaluation

Problems in machine learning

• Classification
I Y ∈ C = {c1 , c2 , ..., ck }, called labels
• Clustering
I Y not given (often called unsupervised learning)
• Regression
I Y ∈ R, called response
• Ranking
• And a lot more new topics emerging in recent years
I Topic model (e.g., what is the topic of a blogger article)
I Manifold (topological) learning
I Salient sentence extraction
I Graph learning etc.

Donghui Yan UMassD Lecture Notes

 Introduction
Formalism and terminology
Tuning and evaluation

Problems in machine learning

Donghui Yan UMassD Lecture Notes

 Introduction
Formalism and terminology
Tuning and evaluation

A little history about the evolving of machine learning

• Early days
I The AI approach
– 1956 Dartmouth conference marks the start of AI
– Perceptron (Rosenblatt, 1957)
– Dying of the research on Neural network late 1960’s
– Various induction machine, expert system, fuzzy system etc
– PAC learning (Valiant 1984)
I The statistical approach
– Statistical learning theory (Vapnik and Chervonenkis, 1964-1974)
– Fisher’s LDA, logistic regression, k-means, mixture analysis etc
– Early nonparametric statistics (e.g., kNN)
• The revitalization of Neural network in mid 1980’s
• SVM, boosting, Random Forests from early 1990’s and on
I The statistical approach is gaining popularity
• Neural network back again under guise of deep learning (2008-).
Donghui Yan UMassD Lecture Notes
 Introduction
Formalism and terminology
Tuning and evaluation

Connections to other subjects

• Aspects of machine learning

I Machine
– Computer science (algorithms to realize machine learning)
I Learning
– Mathematics, probability and statistics (analysis and theory)
I Applications
– Provides motivation and ultimate testbed for learning algorithms.

• What’s the connection of (nonparametric) statistics and ML?

I Both learn from the data
I Nonparametric statistics ⊆ ML (by my definition)
I But, as a matter of fact, ML focus more on discrete problems
(e.g., classification) while (nonparametric) statistics more on the
continuous world (e.g., regression).

Donghui Yan UMassD Lecture Notes

 Introduction
Formalism and terminology
Tuning and evaluation

Predictive learning or classification

• Given data (X1 , Y1 ), ..., (Xn , Yn ), we wish to learn the

relationship f : X 7→ Y s.t.

The future prediction is the best

e.g., smallest error rate (or precision/recall, AUC etc)

I Xi called features, Yi ∈ {1, 2, ..., J} called labels or classes
I (X1 , Y1 ), ..., (Xn , Yn ) is called a training sample
I f is called the trained or fitted model

♠ The best possible decision rule (Bayes rule)

I As if one knows the distribution (X, Y ) (often unknown).

Donghui Yan UMassD Lecture Notes

 Introduction
Formalism and terminology
Tuning and evaluation

The classification problem

• What does the solution to a classification problem really do?
– Identifying the decision (or class) boundary.

Donghui Yan UMassD Lecture Notes

 Introduction
Formalism and terminology
Tuning and evaluation

Loss function
Depends on the application, typical loss functions
• The 0-1 loss

1, if f (X) 6= Y
cost =
0, otherwise.
I A loss function of special interest and most commonly used
• Cost-sensitive loss functions, i.e., a cost matrix, for a 6= b,
 
0 b
a 0
I Suitable when errors in diff classes have diff consequence
– e.g., fraud detection, cost a when mistaking fraud as normal and b
when mistaking normal as fraud.

Donghui Yan UMassD Lecture Notes

 Introduction
Formalism and terminology
Tuning and evaluation

Function class

Function class F = {f } determines the type of classifiers

• Linear classifiers
I Logistic regression: logit(P (Y |X)) = Xβ
I SVM: f (X) = ni=1 wi K(Xi , X) + w0
P

• Boosting
PT (n)
I f (X) = i=1 ai h(X1 , ..., Xn , X)
with h from some data dependent basis library
• Tree-based classifiers
I C4.5, CART
I Random Forests and its variants.

Donghui Yan UMassD Lecture Notes

 Introduction
Evaluation
Formalism and terminology
Performance metrics
Tuning and evaluation

Evaluation of a machine learning algorithm

• Train on training data and evaluate on test data

I Most common
• Cross-validation
I Split data into J partitions
I Use any one of the J partitions as test and rest for training
I Average result on all J tests
• Bootstrap and use out of bag estimate
I Train on a sample with replacement of all observations in the data
I Test on the rest
I Repeat many times and average results.

Donghui Yan UMassD Lecture Notes

 Introduction
Evaluation
Formalism and terminology
Performance metrics
Tuning and evaluation

Evaluation methodology

• Have separate training/test set

• Fit a model (e.g., logistic model) on the training set
• Evaluate the trained model on the test set
I Correct classification when the label matches (0/1 loss)
I More advanced metric like AUC.

Donghui Yan UMassD Lecture Notes

 Introduction
Evaluation
Formalism and terminology
Performance metrics
Tuning and evaluation

Evaluation illustrated

Donghui Yan UMassD Lecture Notes

 Introduction
Evaluation
Formalism and terminology
Performance metrics
Tuning and evaluation

Selection of tuning parameters

• Treat the training set as the entire data

• Split training data = training set + tuning set
• Treat tuning set → test set and proceed as usual evaluation
I Calculate a performance metric, e.g., accuracy
I Select parameters that lead to the best performance
I Use selected parameters for final performance evaluation.

Donghui Yan UMassD Lecture Notes

 Introduction
Evaluation
Formalism and terminology
Performance metrics
Tuning and evaluation

Performance metrics

Some popular performance metrics

• Error rate
– Most commonly used in statistics and machine learning
• Kappa statistics
– Commonly used in remote sensing, medical assessment
• Area under curve (AUC)
– When detection and false alarm rate matter, e.g., biomarker
discovery, anomaly/fraud detection.

Donghui Yan UMassD Lecture Notes

 Introduction
Evaluation
Formalism and terminology
Performance metrics
Tuning and evaluation

Kappa statistic

• The idea is to measure the amount of departure from that arises

purely by chance
I κ is calculated from the confusion matrix
I κ takes into account sample sizes for different classes
I Controversial (many modifications going on).

Donghui Yan UMassD Lecture Notes

 Introduction
Evaluation
Formalism and terminology
Performance metrics
Tuning and evaluation

The confusion matrix

Label 1 ... j ... C Total

1 n11 ... n1j ... n1C n1.
... ... ... ... ... ... ...
i ni1 ... nij ... niC ni.
... ... ... ... ... ... ...
C nC1 ... nCj ... nCC nC.
Total n.1 ... n.j ... n.C n

I C = # classes
I nij = # points from class i but classified as j
I n = size of the sample.

Donghui Yan UMassD Lecture Notes

 Introduction
Evaluation
Formalism and terminology
Performance metrics
Tuning and evaluation

Calculation of κ

Definition. The κ coefficient is calculated as

Pobserved − Pexpected
κ=
1 − Pexpected

where
C C
X ni. n.i 1X
Pexpected = . , Pobserved = nii .
n n n
i=1 i=1

I Pexpected measures chance that observed and true labels agree

I Pobserved measures proportion of observations labeled correctly.

Donghui Yan UMassD Lecture Notes

 Introduction
Evaluation
Formalism and terminology
Performance metrics
Tuning and evaluation

Example κ statistics
Confusion matrix of Logit on the South Africa Heart data

True/Predicted 1 2 Total
1 130 41 171
2 22 38 60
Total 152 79 231

I n1. = 171, n2. = 60, n.1 = 151, n.2 = 79, n = 231

I Pexpected = n1. n.1 /n + n2. n.2 /n =
171 · 152/2312 + 60 · 79/2312 = 0.5759
I Pobserved = (n11 + n22 )/n = (130 + 38)/231 = 0.7273
I κ = (Pobserved − Pexpected )/(1 − Pexpected ) =
(0.7273 − 0.5759)/(1 − 0.5759) = 0.36.

Donghui Yan UMassD Lecture Notes

 Introduction
Evaluation
Formalism and terminology
Performance metrics
Tuning and evaluation

Area under curve

• ROC curve is a graphical plot of true positive rate (TPR) vs.

false positive rate (FPR) as discrimination threshold varies
I TPR = % true positives out of the positives
I FPR = % false positives out of the negatives
• Example (assume class 1 = positive, 2 = negative)
True/Predicted 1 2 Total
1 130 41 171
(true pos) (false neg) (pos)
2 22 38 60
(false pos) (true neg) (neg)
Total 152 79 231

Donghui Yan UMassD Lecture Notes

 Introduction
Evaluation
Formalism and terminology
Performance metrics
Tuning and evaluation

Area under curve (AUC)

• Another measure to assess a machine learning algorithm
• Often used when cost for mis-classification is asymmetric
I e.g., intrusion as normal Vs normal as intrusion in cyber security
• R package “AUC”.

Donghui Yan UMassD Lecture Notes