EE2211 Introduction to Machine

Learning
Lecture 1
Semester 1
2020/2021

Li Haizhou (haizhou.li@nus.edu.sg)

Electrical and Computer Engineering Department

National University of Singapore

Acknowledgement:
EE2211 development team
(Thomas, Kar-Ann, Chen Khong, Helen, Robby and Haizhou)

© Copyright EE, NUS. All Rights Reserved.

 Course Overview
• EE2211 is organized into 12 lectures, 12 tutorials, 3 assignments (40%
marks), 1 Quiz (30% marks) at mid-term, and 1 final Exam (30% marks).

• Tutorials 1+1 hours

• Lectures, tutorials, quiz (mid-term) and final exam are conducted online

• Important dates
– A1 released in Week 4 on on 31 Aug (Monday) and submitted in 3 weeks
– A2 released in Week 6 on 14 Sept (Monday) and submitted in 4 weeks
– A3 released in Week 10 on 19 Oct (Monday) and submitted in 4 weeks
– Quiz (mid-term) will be on 25 Sept (Friday)

EE2211 Introduction to Machine Learning L1.2

© Copyright EE, NUS. All Rights Reserved.
 Reading List
Main textbooks: Book1 (text) and Book2 (python)
Supplementary reading: Book3

References
• [Book1] Andriy Burkov, “The Hundred-Page Machine Learning Book”,
2019. (read first, buy later: http://themlbook.com/wiki/doku.php)
• [Book2] Andreas C. Muller and Sarah Guido, “Introduction to Machine
Learning with Python: A Guide for Data Scientists”, O’Reilly Media, Inc.,
2017.
• [Book3] Jeff Leek, “The Elements of Data Analytic Style: A guide for
people who want to analyze data”, Lean Publishing, 2015.

EE2211 Introduction to Machine Learning 3

© Copyright EE, NUS. All Rights Reserved.
 Course Contents
• Introduction and Preliminaries (Haizhou)
– Introduction
– Data engineering
– Introduction to probability and statistics
• Fundamental Machine Learning Algorithms I (Kar-Ann / Helen)
– Systems of linear equations
– Least squares, linear regression
– Ridge regression, polynomial regression
• Fundamental Machine Learning Algorithms II (Thomas)
– Over-fitting, bias/variance Trade-off
– Optimization, gradient descent
– Decision trees, random forest
• Performance and More Algorithms (Haizhou)
– Performance issues
– K-means clustering
– Neural networks

EE2211 Introduction to Machine Learning 4

© Copyright EE, NUS. All Rights Reserved.
 Introduction
Module I Content
• What is machine learning and types of learning
• How supervised learning works
• Regression and classification tasks
• Induction versus deduction reasoning
• Types of data
• Data wrangling and cleaning
• Data integrity and visualization
• Causality and Simpson’s paradox
• Random variable, Bayes’ rule
• Parameter estimation
• Parameters vs. Hyperparameters

5
© Copyright EE, NUS. All Rights Reserved.
 2001: A Space Odyssey

HAL listens, talks, sings, reads lips, plays chess, and solves problems !

© Copyright EE, NUS. All Rights Reserved.

 AlphaGo

https://www.bbc.com/news/technology-35785875

https://www.businessinsider.com/googles-alphago-made-
artifical-intelligence-history-2016-3

EE2211 Introduction to Machine Learning 7

© Copyright EE, NUS. All Rights Reserved.
 Shannon Game

the frog => the frog jumped

frog jumped => frog jumped into
jumped into => jumped into the
into the => into the pond

The Bit Player (2018) tells the story of

an overlooked genius, Claude
Shannon (the "Father of Information
Theory")
C. Shannon, A mathematical theory of communication, Bell System Technical Journal, vol. 27, pp. 379-423 and 623-656, 1948.

8
© Copyright EE, NUS. All Rights Reserved.
 What is Machine Learning?
❑ Machine learning
➢ is a subfield of computer science that is concerned with
building algorithms which, to be useful, rely on a
collection of examples of some phenomenon. - Andriy
Burkov
✓These examples can come from nature, be handcrafted by
humans or generated by another algorithm.

Ref: Book1, chp1 , p3

EE2211 Introduction to Machine Learning 9

© Copyright EE, NUS. All Rights Reserved.
 Early Definitions
• Arthur Samuel (1959): A field of study that gives
computers the ability to learn without being explicitly
programmed.

• Tom Mitchell (1998): A computer program is said to learn

from experience E with respect to some task T and some
performance measure P, if its performance on T, as
measured by P, improves with experience E.

How do we create computer programs that improve with experience?"

Tom Mitchell
http://videolectures.net/mlas06_mitchell_itm/

EE2211 Introduction to Machine Learning 10

© Copyright EE, NUS. All Rights Reserved.
 Example: 2 or 7?

Image credit: Geoffrey Hinton and CIS 419/519, Eric Eaton

Task: Digit recognition

Performance: Classification accuracy
Experience: Labelled images
EE2211 Introduction to Machine Learning 11
© Copyright EE, NUS. All Rights Reserved.
 Application Examples
• Speech recognition
• Face recognition
• Handwriting recognition
• Object recognition
• Housing price prediction
• Etc, etc.

EE2211 Introduction to Machine Learning 12

© Copyright EE, NUS. All Rights Reserved.
 Types of Learning

• Supervised Learning Supervised Unsupervised

Learning Learning
• Unsupervised Learning
• Semi-Supervised Learning

Discrete
• Reinforcement Learning Classification Clustering

discrete

Continuous
x Classification y Dimensionality
Regression
continuous Reduction
x Regression y

Ref: Book1 and https://towardsdatascience.com/supervised-vs-unsupervised-learning-14f68e32ea8d

EE2211 Introduction to Machine Learning 13

© Copyright EE, NUS. All Rights Reserved.
 Supervised Learning

Training
Apple

Orange

Model Testing

This is an orange

EE2211 Introduction to Machine Learning 14

© Copyright EE, NUS. All Rights Reserved.
 Supervised Learning
• In supervised learning, the dataset is the collection of labeled
examples ,

𝑥1
𝑇
𝐱 𝑖 = ⁞ or 𝐱 𝑖 = 𝑥1 , … , 𝑥𝑗 , … , 𝑥𝐷 , i = 1, . . . , N
𝑖
𝑥𝐷 𝑖
– Each element 𝐱 𝑖 among N is called a feature vector.
• A feature vector is a vector in which each dimension j = 1, . .
. , D contains a value that describes the example somehow.

– The label 𝑦𝑖 can be either an element belonging to a finite set

of classes {1, 2, . . . ,C}, or a real number.

EE2211 Introduction to Machine Learning 15

© Copyright EE, NUS. All Rights Reserved.
 • For instance, if your examples are email messages and your
problem is spam detection, then you have two classes {spam,
not-spam}.
• Classification: predict discrete valued output (e.g., 0 or 1)

y 1-Dimensional Case
“0” “1”

0 x
(e.g., repeated word count)

(1D view)
Decision line
(threshold)

EE2211 Introduction to Machine Learning 16

© Copyright EE, NUS. All Rights Reserved.
 Classification: Breast Cancer (malignant, benign)

(age)
x 2 2-Dimensional Case
Malignant (harmful)
Benign (not harmful)

x 1 (tumor size)

EE2211 Introduction to Machine Learning 17

© Copyright EE, NUS. All Rights Reserved.
 Regression: predict continuous valued output
(e.g., house price prediction)

(price)
y

x
(size in meter square)

EE2211 Introduction to Machine Learning 18

© Copyright EE, NUS. All Rights Reserved.
 How Supervised Learning Works

Training Test
𝑀 𝑀 𝑁 𝑁
𝐱𝑖 𝑖=1 𝑦𝑖 𝑖=1 𝐱𝑘 𝑘=1 𝑦𝑘 𝑘=1

Model Model
Data Known Data Predicted
Parameters to label Learned label
learn parameters
Goal: to learn the model’s Goal: to predict the label of
parameters from the given novel data 𝐱 𝑘 𝑁
𝑘=1 using the
data and labels 𝐱 𝑖 , 𝑦𝑖 𝑀
𝑖=1 learned parameters

Slides courtesy: Professor Robby Tan

EE2211 Introduction to Machine Learning 19

© Copyright EE, NUS. All Rights Reserved.
 Example

source: SUTD

Task: Number of new cases prediction

Performance: Prediction accuracy
Experience: Historical data

Supervised or unsupervised learning?

EE2211 Introduction to Machine Learning L1.20
© Copyright EE, NUS. All Rights Reserved.
 Example

* Ref: Book1, sec1.3, pp.5-7 EE2211 doesn’t discuss constraint

optimization in detail.
EE2211 Introduction to Machine Learning 21
© Copyright EE, NUS. All Rights Reserved.
 Unsupervised Learning
• In unsupervised learning, the dataset is a collection of
unlabeled examples

• Again, x is a feature vector, and the goal of an

unsupervised learning algorithm is to create a model
that takes a feature vector x as input and either
transforms it into another vector or into a value that can
be used to solve a practical problem.

Ref: Book1, sec1.2.2

EE2211 Introduction to Machine Learning 22

© Copyright EE, NUS. All Rights Reserved.
 Unsupervised Learning

Training

I found two
types of fruits!

EE2211 Introduction to Machine Learning 23

© Copyright EE, NUS. All Rights Reserved.
 Pictorial summary
• Supervised Learning

(age)
x2 2-Dimensional Case
Malignant (harmful)
Benign (not harmful)

x1 (tumor size)

EE2211 Introduction to Machine Learning 24

© Copyright EE, NUS. All Rights Reserved.
 Pictorial summary
• Unsupervised Learning

(age)
x2 2-Dimensional Case

x1 (tumor size)

EE2211 Introduction to Machine Learning 25

© Copyright EE, NUS. All Rights Reserved.
 Pictorial summary
• Unsupervised Learning

(age) Find the distribution of data

x2 2-Dimensional Case

x1 (tumor size)

EE2211 Introduction to Machine Learning 26

© Copyright EE, NUS. All Rights Reserved.
 Pictorial summary
• Unsupervised Learning: Clustering

(age) Discover the underlying structures

x2 of data
2-Dimensional Case

x1 (tumor size)

EE2211 Introduction to Machine Learning 27

© Copyright EE, NUS. All Rights Reserved.
 Example: Social Network

https://en.wikipedia.org/wiki/Social_network_analysis#/media/File:Kencf0618FacebookNetwork.jpg

EE2211 Introduction to Machine Learning 28

© Copyright EE, NUS. All Rights Reserved.
 Semi-Supervised Learning

Supervised Semi-supervised Unsupervised

Learning Learning Learning

Labelled +
Unlabelled data
Labelled data Unlabelled data

+
Typically plenty of
unlabelled data

Learning Model

EE2211 Introduction to Machine Learning 29

© Copyright EE, NUS. All Rights Reserved.
 Reinforcement Learning

action

Environment
Agent
S1 S2

reward

EE2211 Introduction to Machine Learning 30

© Copyright EE, NUS. All Rights Reserved.
 Reinforcement Learning

• A policy is a function (similar to the model in supervised

learning) that takes the feature vector of a state as input
and outputs an optimal action to execute in that state.

• The action is optimal if it maximizes the expected

average reward.

* EE2211 doesn’t discuss reinforcement learning in detail.

EE2211 Introduction to Machine Learning 31

© Copyright EE, NUS. All Rights Reserved.
 Inductive vs. Deductive Reasoning
Main task of Machine Learning: to make inferences

Type of inferences
Example
Inductive Deductive
• To reach probable conclusions. • To reach logical conclusions
• All needed information is unavailable or deterministically: all information that can
unknown, causing uncertainty in the conclusions lead to the correct conclusion is available

Statistical Machine Learning (as

Probability and Statistics are required
opposed to logic machine learning)

Basic properties Basic rules

Product rule Sum rule

• 0 ≤ p(x) ≤ 1
• Independent variables: • Dependent
• ʃ p(x)dx = 1
p(a,b) = p(a)p(b) variables:
• ∑ p(x) = 1
x • Dependent variables: p(a) = ∑ p(a,b)
• p(a,b) = p(a|b) p(b)
b
(Marginalization)
= p(b|a) p(a)
Slides courtesy: Professor Robby Tan

EE2211 Introduction to Machine Learning 32

© Copyright EE, NUS. All Rights Reserved.
 Inductive Reasoning
Note: humans use inductive reasoning all the time and not
in a formal way like using probability/statistics.

Ref: Gardener, Martin (March 1979). "MATHEMATICAL GAMES: On the

fabric of inductive logic, and some probability paradoxes" (PDF). Scientific
American. 234

EE2211 Introduction to Machine Learning 33

© Copyright EE, NUS. All Rights Reserved.
 The End

EE2211 Introduction to Machine Learning 34

© Copyright EE, NUS. All Rights Reserved.