MODULE 1

January 7, 2025
Watermark
 Contents

1 Perceptron
Basics of perceptron

2 Perceptron Models
Single Layer Perceptron Model
Watermark

Multilayer Perceptron Model

 This is my Title
Perceptron
Basic Problem

Contents

1 Perceptron
Basics of perceptron

2 Perceptron Models
Watermark

Single Layer Perceptron Model

Multilayer Perceptron Model
 This is my Title
Perceptron
Basic Problem

What is Perceptron?

Perceptron is a Machine Learning algorithm for supervised learning

of various binary classification tasks.
Further, Perceptron is also understood as an Artificial Neuron or
neural network unit that helps to detect certain input data
computations in business intelligence.
The Perceptron model is one of the best and simplest types of
Artificial Neural Networks.
It is a supervised learning algorithm for binary classifiers.
Watermark

A Perceptron can be considered a single-layer neural network

with four main parameters:
1 Input values
2 Weights and Bias
3 Net sum
4 Activation function
 This is my Title
Perceptron
Basic Problem

Basic Components of Perceptron

Inventor: Mr. Frank Rosenblatt developed the perceptron model
as a binary classifier.
Main Components:
1 Inputs (x1 , x2 , . . . , xm ): These are real numerical values fed into the
perceptron for further processing.
2 Weights (w1 , w2 , . . . , wm ): Each input is multiplied by its associated
weight. A bias term (w0 ) is also included.
3 Net Input Function: The perceptron computes the weighted sum of
inputs:
m
Watermark

X
Net Input = wi xi + w0
i =1

4 Activation Function: Determines the output based on the net input

(e.g., a step function for binary classification).
Input Nodes or Input Layer:
The primary component that accepts initial data for processing.
Each input node contains a real numerical value.
 This is my Title
Perceptron
Basic Problem
Watermark

Figure: Perceptron Model Diagram

 This is my Title
Perceptron
Basic Problem

Perceptron Components

Input Nodes or Input Layer: This is the primary component of the

Perceptron that accepts the initial data into the system for further
processing. Each input node contains a real numerical value.
Weight and Bias:
Weight: Represents the strength of the connection between units.
Weight is directly proportional to the strength of the associated
Watermark

input neuron in deciding the output.

Bias: Can be considered as the line of intercept in a linear
equation.
 This is my Title
Perceptron
Basic Problem

Activation Function

Activation Function: These are the final and important

components that help to determine whether the neuron will fire or
not. The Activation Function can be considered primarily as a step
function.
Types of Activation Functions:
Watermark

Sign function
Step function
Sigmoid function
 This is my Title
Perceptron
Basic Problem

Figure: Activation function

Watermark
 This is my Title
Perceptron
Basic Problem

Activation Function

The data scientist uses the activation function to take a subjective

decision based on various problem statements and forms the desired
outputs. Activation function may differ (e.g., Sign, Step, and Sigmoid) in
perceptron models by checking whether the learning process is slow or
has vanishing or exploding gradients.
Watermark
 This is my Title
Perceptron
Basic Problem

How does Perceptron work?

In Machine Learning, Perceptron is considered as a single-layer neural

network that consists of four main parameters:
1 Input values (Input nodes)
2 Weights and Bias
3 Net sum
4 Activation function
Watermark

The perceptron model begins with the multiplication of all input values
and their weights, then adds these values together to create the
weighted sum. This weighted sum is then applied to the activation
function f to obtain the desired output. This activation function is also
known as the step function and is represented by f .
 This is my Title
Perceptron
Basic Problem
Watermark
 This is my Title
Perceptron
Basic Problem

Activation Function and Its Role

1 The step function or Activation function plays a vital role in

ensuring that the output is mapped between the required values
(0, 1) or (−1, 1).
2 The weight of the input is indicative of the strength of a node.
3 An input’s bias value gives the ability to shift the activation
Watermark

function curve up or down.

 This is my Title
Perceptron
Basic Problem

Perceptron Model: Two Important Steps

Step-1: In the first step, multiply all input values with corresponding
weight values and then add them to determine the weighted sum.
Mathematically, we can calculate the weighted sum as follows:
X
wi · xi = x1 · w1 + x2 · w2 + · · · + xn · wn

Add a special term called bias b to this weighted sum to improve the
model’s performance: X
wi · xi + b
Watermark

Step-2: In the second step, an activation function is applied to the

weighted sum, which gives us an output either in binary form or a
continuous value: X 
Y =f wi · xi + b
 This is my Title
Perceptron
Basic Problem

Types of Perceptron Models

Based on the layers, Perceptron models are divided into two types.
These are as follows:
1 Single-layer Perceptron Model
2 Multi-layer Perceptron Model
Watermark
 This is my Title
Perceptron Models
Single Layer Perceptron Model

Contents

1 Perceptron
Basics of perceptron

2 Perceptron Models
Watermark

Single Layer Perceptron Model

Multilayer Perceptron Model
 This is my Title
Perceptron Models
Single Layer Perceptron Model

Single Layer Perceptron Model

This is one of the easiest Artificial neural networks (ANN) types. A

single-layered perceptron model consists of a feed-forward network
and also includes a threshold transfer function inside the model. The
main objective of the single-layer perceptron model is to analyze the
linearly separable objects with binary outcomes.
In a single-layer perceptron model, its algorithms do not contain
recorded data, so it begins with inconsistently allocated input for
Watermark

weight parameters.
The model sums up all inputs (weight), and after adding them, if
the total sum of all inputs is more than a pre-determined value, the
model gets activated and shows the output value as +1.
 This is my Title
Perceptron Models
Single Layer Perceptron Model

Single Layer Perceptron Model (Continued)

If the outcome is the same as the pre-determined or threshold

value, the performance of this model is stated as satisfied, and
weight demand does not change.
However, this model consists of a few discrepancies triggered
when multiple weight input values are fed into the model. Hence,
to find the desired output and minimize errors, some changes
Watermark

should be necessary for the weights input.

"Single-layer perceptron can learn only linearly separable patterns."
 This is my Title
Perceptron Models
Multilayer Perceptron Model

Contents

1 Perceptron
Basics of perceptron

2 Perceptron Models
Watermark

Single Layer Perceptron Model

Multilayer Perceptron Model
 This is my Title
Perceptron Models
Multilayer Perceptron Model

Multilayer Perceptron Model: Overview

Like a single-layer perceptron model, a multilayer perceptron model

also has the same model structure but with a greater number of hidden
layers. The multilayer perceptron model is also known as the
backpropagation algorithm, which executes in two stages as follows:
1 Forward Stage: The activation functions start from the input layer in
the forward stage and terminate on the output layer.
2 Backward Stage: In the backward stage, weight and bias values
Watermark

are modified as per the model’s requirement. In this stage, the

error between the actual output and the demanded output
propagates backward from the output layer and ends on the
input layer.
 This is my Title
Perceptron Models
Multilayer Perceptron Model

Multilayer Perceptron Model: Key Features

A multilayer perceptron model can be considered as multiple artificial

neural networks having various layers. Unlike a single-layer perceptron
model, the activation function in this model does not remain linear.
Instead, it can be executed as:
Sigmoid
Watermark

TanH
ReLU (Rectified Linear Unit)
 This is my Title
Perceptron Models
Multilayer Perceptron Model

Multilayer Perceptron Model: Key Features (Continued)

This flexibility allows for better deployment of the model.

A multilayer perceptron model has greater processing power and can
process both linear and nonlinear patterns. Further, it can also
implement logic gates such as:
AND
OR
XOR
Watermark

NAND
NOT
XNOR
NOR
 This is my Title
Perceptron Models
Multilayer Perceptron Model

Advantages of Multi-Layer Perceptron

A multi-layered perceptron model can be used to solve complex

non-linear problems.
It works well with both small and large input data.
It helps us to obtain quick predictions after the training.
Watermark

It helps to obtain the same accuracy ratio with large as well as

small data.
 This is my Title
Perceptron Models
Multilayer Perceptron Model

Disadvantages of Multi-Layer Perceptron

In Multi-layer perceptron, computations are difficult and

time-consuming.
In multi-layer Perceptron, it is difficult to predict how much the
dependent variable affects each independent variable.
Watermark

The model functioning depends on the quality of the training.

 This is my Title
Perceptron Models
Multilayer Perceptron Model

Perceptron Function

Perceptron function f (x ) can be achieved as output by multiplying the

input x with the learned weight coefficient w. Mathematically, we can
express it as: 
1 if w · x + b > 0


f (x ) = 
0 otherwise

Watermark

w represents the real-valued weights vector

b represents the bias
x represents a vector of input x values.
 This is my Title
Perceptron Models
Multilayer Perceptron Model

How Does Backpropagation Work in a Multilayer Perceptron?

(1/2)

Backpropagation is a supervised learning algorithm used to train the

network by adjusting the weights of the connections between neurons.
Here’s how it works:
1 Forward Pass: During the forward pass, input data is fed through
the network, and the output is calculated based on the current
Watermark

weights and biases.

2 Error Calculation: The difference between the predicted output
and the actual output is calculated using a loss function, such as
mean squared error or cross-entropy loss.
 This is my Title
Perceptron Models
Multilayer Perceptron Model

How Does Backpropagation Work in a Multilayer Perceptron?

(2/2)

1 Backward Pass: In the backward pass, the algorithm works by

propagating the error backward through the network, starting
from the output layer and moving towards the input layer. This is
where the name \backpropagation" comes from.
2 Weight Update: As the error is propagated backward, the
algorithm adjusts the weights of the connections between neurons
Watermark

to minimize the error. This is done using the gradient of the loss
function with respect to the weights, calculated via the chain rule
of calculus.
3 Repeat Until Convergence: The forward and backward passes are
repeated for multiple iterations (epochs) until the network’s
performance converges to a satisfactory level.
 This is my Title
Perceptron Models
Multilayer Perceptron Model

Characteristics of Perceptron

The perceptron model has the following characteristics:

1 Perceptron is a machine learning algorithm for supervised learning
of binary classifiers.
2 In Perceptron, the weight coefficient is automatically learned.
3 Initially, weights are multiplied with input features, and the decision
is made whether the neuron is fired or not.
4 The activation function applies a step rule to check whether the
Watermark

weight function is greater than zero.

5 The linear decision boundary is drawn, enabling the distinction
between the two linearly separable classes +1 and -1.
6 If the added sum of all input values is more than the threshold
value, it must have an output signal; otherwise, no output will be
shown.
 This is my Title
Perceptron Models
Multilayer Perceptron Model

Limitations of Perceptron Model

A perceptron model has limitations as follows:

The output of a perceptron can only be a binary number (0 or 1)
due to the hard limit transfer function.
Perceptron can only be used to classify the linearly separable sets
of input vectors. If input vectors are non-linear, it is not easy to
Watermark

classify them properly.

 This is my Title
Perceptron Models
Multilayer Perceptron Model

Stochastic Gradient Descent (SGD)

Stochastic Gradient Descent (SGD) is an optimization algorithm used to
minimize the loss function by iteratively adjusting the model’s
parameters. It works by using small, random subsets (mini-batches) of
the training data, providing computational efficiency and helping to
escape local minima.
Initialization: Start with an initial set of weights and biases.
Epoch Process:
Randomly shuffle the training data.
Watermark

Divide the data into mini-batches.

For Each Mini-Batch:
Perform a forward pass to compute predictions and loss.
Calculate gradients of the loss with respect to the model’s
parameters.
Update the model’s parameters using the gradients and learning
rate.
 This is my Title
Perceptron Models
Multilayer Perceptron Model

Comparison: Perceptron vs Multi-Layer Perceptron (MLP) -

Part 1

Feature Perceptron Multi-Layer Perceptron (MLP)

Structure Single layer (input and output Multiple layers: input, one or more
neurons only). hidden layers, and output layer.
Hidden Layers No hidden layers. One or more hidden layers
present.
Watermark

Capability Solves only linearly separable Solves both linearly and non-
problems. linearly separable problems.
Activation Function Step function (outputs binary val- Non-linear functions like ReLU, sig-
ues like 0 or 1). moid, or tanh.
 This is my Title
Perceptron Models
Multilayer Perceptron Model

Comparison: Perceptron vs Multi-Layer Perceptron (MLP) -

Part 2

Feature Perceptron Multi-Layer Perceptron (MLP)

Learning Algorithm Simple rule adjusting weights Backpropagation with optimiza-
based on misclassified samples. tion techniques like gradient de-
scent.
Complexity Simple and computationally inex- More complex and computation-
pensive. ally intensive.
Watermark

Use Cases Binary classification of linearly Complex tasks like image recog-
separable data. nition, NLP, regression, and more.
Training Data Limited to simple datasets with Suitable for large and complex
linear patterns. datasets.