Scribd
Upload
49 views26 pages

Perceptron

This document discusses classification algorithms. It begins by defining classification tasks using training data with discrete labels. Linear classification models are described as using a linear function to learn a classification boundary. Examples of classification problems include fruit, email, and medical classifications. The perceptron algorithm is then introduced as the simplest form of neural network for linearly separable data. It works by iteratively adjusting the weights of the linear boundary based on misclassified examples. An example demonstrates how the perceptron converges on a solution. In summary, this document provides an overview of classification problems and introduces the perceptron algorithm as the simplest form of neural network classifier.

Uploaded by

Simmon Shaji
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
49 views26 pages

Perceptron

This document discusses classification algorithms. It begins by defining classification tasks using training data with discrete labels. Linear classification models are described as using a linear function to learn a classification boundary. Examples of classification problems include fruit, email, and medical classifications. The perceptron algorithm is then introduced as the simplest form of neural network for linearly separable data. It works by iteratively adjusting the weights of the linear boundary based on misclassified examples. An example demonstrates how the perceptron converges on a solution. In summary, this document provides an overview of classification problems and introduces the perceptron algorithm as the simplest form of neural network classifier.

Uploaded by

Simmon Shaji
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 26

Classification

Given: Training data: (x1, y1), . . . , (xn, yn)/xi 2 Rd and yi is


discrete (categorical/qualitative), yi 2 Y.

Example Y = { 1, +1}, Y = {0, 1}.

Task: Learn a classification function:

f : Rd ! Y

Linear Classification: A classification model is said to be linear


if it is represented by a linear function f (linear hyperplane)
Classification: examples
1. Fruit classification ! Banana/Orange?

2. Email Spam/Ham ! Which email is junk?

3. Tumor benign/malignant ! Which patient has cancer?

4. Credit default/not default ! Which customers will default on


their credit card debt?

Balance Income Default


300 $20,000.00 no
2000 $60,000.00 no
5000 $45,000.00 yes
. . .
. . .
. . .
Classification: example

60000
Income
40000
20000
0

0 500 1000 1500 2000 2500

Balance

Credit: Introduction to Statistical Learning.


Perceptron
• Belongs to Neural Networks class of algorithms (algorithms
that try to mimic how the brain functions).

• The first algorithm used was the Perceptron (Resenblatt


1959).

• Worked extremely well to recognize:


1. handwritten characters (LeCun et a. 1989),
2. spoken words (Lang et al. 1990),
3. faces (Cottrel 1990)

• NN were popular in the 90’s but then lost some of its popularity.

• Now NN back with deep learning.


Perfectly separable data

!"#$%&"'('

!"#$%&"')'
Perceptron

!"#$%&"'('

!"#$%&"')'

• Linear classification method.


• Simplest classification method.
• Simplest neural network.
• For perfectly separated data.
Perceptron

Given n examples and d features.

d
X
f (xi) = sign( wj xij )
j=0
Perceptron
• Works perfectly if data is linearly separable. If not, it will not
converge.

• Idea: Start with a random hyperplane and adjust it using your


training data.

• Iterative method.
Perceptron
Perceptron Algorithm

Input: A set of examples, (x1, y1), · · · , (xn, yn)


Output: A perceptron defined by (w0, w1, · · · , wd)

Begin
2. Initialize the weights wj to 0 8j 2 {0, · · · , d}
3. Repeat until convergence
4. For each example xi 8i 2 {1, · · · , n}
5. if yif (xi)  0 #an error?
6. update all wj with wj := wj + yixi #adjust the weights
End
Perceptron
Some observations:

• The weights w1, . . . , wd determine the slope of the decision


boundary.

• w0 determines the o↵set of the decision boundary (sometimes


noted b).

• Line 6 corresponds to:


Mistake on positive: add x to weight vector.
Mistake on negative: substract x from weight vector.
Some other variants of the algorithm add or subtract 1.

• Convergence happen when the weights do not change anymore


(di↵erence between the last two weight vectors is 0).
Perceptron: Example
Perceptron: Example
Perceptron: Example
Perceptron: Example
Perceptron: Example
Perceptron: Example
Perceptron: Example
Perceptron: Example
Perceptron: Example
Perceptron: Example
Perceptron: Example
Finally converged!
Perceptron: Example
With some test data:
Perceptron
• The wi determine the contribution of xi to the label.
Pn
• w0 is a quantity that i=1 wi x1 needs to exceed for the per-
ceptron to output 1.

• Can be used to represent many Boolean functions: AND, OR,


NAND, NOR, NOT but not all of them (e.g., XOR).
From perceptron to NN
• Neural networks use the ability of the perceptrons to represent
elementary functions and combine them in a network of layers
of elementary questions.

• However, a cascade of linear functions is still linear!

• And we want networks that represent highly non-linear func-


tions.
Choice of the hyperplane

+ +
-
+ +
-
+
- +
- - +
- +
-

Lots of possible solutions!


Digression: Idea of SVM is to find the optimal solution.
Credit
• The elements of statistical learning. Data mining, inference,
and prediction. 10th Edition 2009. T. Hastie, R. Tibshirani,
J. Friedman.
• Machine Learning 1997. Tom Mitchell.

You might also like

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy