Scribd
Upload
13 views46 pages

DL 1

The document discusses the concept of artificial neurons, focusing on the mathematical models of the MP neuron and the Perceptron. It outlines the structure and function of neurons, the importance of weights and biases, and the Perceptron learning algorithm, which updates weights based on training data. The document highlights the convergence of the algorithm for linearly separable datasets and the role of activation functions in neural networks.

Uploaded by

poojab230080ec
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
13 views46 pages

DL 1

The document discusses the concept of artificial neurons, focusing on the mathematical models of the MP neuron and the Perceptron. It outlines the structure and function of neurons, the importance of weights and biases, and the Perceptron learning algorithm, which updates weights based on training data. The document highlights the convergence of the algorithm for linearly separable datasets and the role of activation functions in neural networks.

Uploaded by

poojab230080ec
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 46

Deep Learning

1 Artificial Neuron (MP Neuron and Perceptron)

Dr. Konda Reddy Mopuri


kmopuri@ai.iith.ac.in
Dept. of AI, IIT Hyderabad
Jan-May 2023

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 1


The Neuron

About 100 billion neurons in human brain

Figure credits: Wikipedia


Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 2
Neuron in action

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 3


Neuron in action

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 4


Neuron in action

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 5


Neuron in action

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 6


Neuron in action

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 7


Neurons in the brain have a hierarchy

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 8


Threshold Logic Unit
1 First Mathematical Model for a neuron

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 9


Threshold Logic Unit
1 First Mathematical Model for a neuron
2 McCulloch and Pitts, 1943 → MP neuron

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 9


Threshold Logic Unit
1 First Mathematical Model for a neuron
2 McCulloch and Pitts, 1943 → MP neuron
3 Boolean inputs and output

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 9


Threshold Logic Unit
1 First Mathematical Model for a neuron
2 McCulloch and Pitts, 1943 → MP neuron
3 Boolean inputs and output

f (x) = 1(
X
xi ≥ θ)
i

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 9


Threshold Logic Unit

1 Inputs can be of excitatory or inhibitory nature

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 10


Threshold Logic Unit

1 Inputs can be of excitatory or inhibitory nature


2 When an inhibitory input is set (=1) output → 0

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 10


Threshold Logic Unit

1 Inputs can be of excitatory or inhibitory nature


2 When an inhibitory input is set (=1) output → 0
3 Counts the number of ‘ON’ signals on the excitatory inputs versus the
inhibitory

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 10


Threshold Logic Unit

Example Boolean functions

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 11


Threshold Logic Unit
1 let’s implement simple functions

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 12


Threshold Logic Unit
1 let’s implement simple functions
2 xy’

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 12


Threshold Logic Unit
1 let’s implement simple functions
2 xy’

3 NOR

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 12


Threshold Logic Unit

1 What one unit does? - Learn linear separation

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 13


Threshold Logic Unit

1 What one unit does? - Learn linear separation


line in 2D, plane in 3D, hyperplane in higher dimensions

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 13


Threshold Logic Unit

1 What one unit does? - Learn linear separation


line in 2D, plane in 3D, hyperplane in higher dimensions
2 No learning; heuristic approach

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 13


Perceptron

1 Frank Rosenblatt 1957 (American Psychologist)

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 14


Perceptron

1 Frank Rosenblatt 1957 (American Psychologist)


2 Very crude biological model

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 14


Perceptron

1 Frank Rosenblatt 1957 (American Psychologist)


2 Very crude biological model
3 Similar to MP neuron - Performs linear classification

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 14


Perceptron

1 Frank Rosenblatt 1957 (American Psychologist)


2 Very crude biological model
3 Similar to MP neuron - Performs linear classification
4 Inputs can be real, weights can be different for different i/p
components

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 14


Perceptron

1 Frank Rosenblatt 1957 (American Psychologist)


2 Very crude biological model
3 Similar to MP neuron - Performs linear classification
4 Inputs can be real, weights can be different for different i/p
components
5
(
+b≥0
P
1 when i wi x i
f (x) =
0 else

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 14


Perceptron
1 For simplicity we consider +1 and -1 responses
(
1 when x ≥ 0
σ(x) =
−1 else

f (x) = σ(wT · x + b)

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 15


Perceptron
1 For simplicity we consider +1 and -1 responses
(
1 when x ≥ 0
σ(x) =
−1 else

f (x) = σ(wT · x + b)

2 In general, σ(·) that follows a linear operation is called an activation


function

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 15


Perceptron
1 For simplicity we consider +1 and -1 responses
(
1 when x ≥ 0
σ(x) =
−1 else

f (x) = σ(wT · x + b)

2 In general, σ(·) that follows a linear operation is called an activation


function
3 w are referred to as weights and b as the bias

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 15


Perceptron vs. MP neuron

1 Perceptron is more general computational model

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 16


Perceptron vs. MP neuron

1 Perceptron is more general computational model


2 Inputs can be real

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 16


Perceptron vs. MP neuron

1 Perceptron is more general computational model


2 Inputs can be real
3 Weights are different on the input components

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 16


Perceptron vs. MP neuron

1 Perceptron is more general computational model


2 Inputs can be real
3 Weights are different on the input components
4 Mechanism for learning weights

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 16


Weights and Bias

1 Why are the weights important?

Figure credits: DeepAI


Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 17
Weights and Bias

1 Why are the weights important?


2 Why is it called ‘bias’? What does it capture?

Figure credits: DeepAI


Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 17
Perceptron

Figure credits: François Fleuret


Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 18
Perceptron

Figure credits: François Fleuret


Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 19
Perceptron Learning algorithm

1 Training data (xi , y i ) ∈ RD × {−1, 1}, i = 1, . . . , N

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 20


Perceptron Learning algorithm

1 Training data (xi , y i ) ∈ RD × {−1, 1}, i = 1, . . . , N


2 Start with k ← 1 and wk = 0

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 20


Perceptron Learning algorithm

1 Training data (xi , y i ) ∈ RD × {−1, 1}, i = 1, . . . , N


2 Start with k ← 1 and wk = 0
3 While ∃ i ∈ {1, 2 . . . N } such that y i (wkT · xi ) ≤ 0, update
wk+1 = wk + yi · xi
k ←k+1

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 20


Perceptron Learning algorithm

1 Training data (xi , y i ) ∈ RD × {−1, 1}, i = 1, . . . , N


2 Start with k ← 1 and wk = 0
3 While ∃ i ∈ {1, 2 . . . N } such that y i (wkT · xi ) ≤ 0, update
wk+1 = wk + yi · xi
k ←k+1
4 Note that the bias b is absorbed as a component of w and x is
appended with 1 suitably

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 20


Perceptron Learning Algorithm

Colab Notebook: Perceptron-learning

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 21


Perceptron Learning Algorithm

1 Convergence result: Can be shown that for linearly separable dataset,


algorithm converges after finite iterations

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 22


Perceptron Learning Algorithm

1 Convergence result: Can be shown that for linearly separable dataset,


algorithm converges after finite iterations
2 Stops as soon as it finds a separating boundary

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 22


Perceptron Learning Algorithm

1 Convergence result: Can be shown that for linearly separable dataset,


algorithm converges after finite iterations
2 Stops as soon as it finds a separating boundary
3 Other algorithms maximize the margin from boundary to the samples

Dr. Konda Reddy Mopuri dlc-1/Artificial Neuron 22

You might also like

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy