Lecture

UNIT-I
Soft Computing
A8711
Introduction
Computing Methods, Fundamentals of Artificial Neural
Syllabus Details
Network: Model of Biological Neuron, Mathematical Model of
Neuron, ANN Architec- ture, Learning Rules, Learning
Paradigms, Perceptron Network, Adaline and Madaline
Networks, Applications of Neural Network, Associative
Memory.

Classical Sets and Fuzzy Sets: Crisp Sets, Fuzzy Sets:

History and Origin, Fuzzy Sets: Basic Concepts, Paradigm
Shift, Representations of Fuzzy Sets, Alpha-cuts, Basic
Operations on Fuzzy Sets, Fuzzy Complements,
Intersections, and Unions, Extension Principle for Fuzzy
Sets, Intuitionistic Fuzzy Sets, Operations on Intuitionistic
Fuzzy Sets, Alpha–Beta Cuts. Crisp Relations, Fuzzy
Relations, Binary Fuzzy Relations, Intuitionistic Fuzzy
Relations Intuitionistic Fuzzy Relations, Intuitionistic Fuzzy
Relations, Classical Logic and Fuzzy Logic: Logic, Interval
Analysis, Fuzzy Numbers, Fuzzy Logic, Fuzzy Rule-Based
Systems: Linguistic Variables and Linguistic Hedges, Rule-
Based Systems, Conventional Programs Versus Rule-Based
Systems, Fuzzy Propositions, Fuzzification and
Defuzzification.
 Syllabus Details
Genetic Algorithms: History of Evolutionary Computing, Crossover
and Mutation Prop- erties, Genetic Algorithm Cycle, Fitness Function,
Applications of Genetic Algorithm. His- tory of Genetic Algorithms (GA),
Working Principle, Various Encoding methods, GA Operators-
Reproduction, Crossover, Mutation, Convergence of GA, Bit wise operation
in GA, Multi - level Optimization.

Particle Swarm Optimization: Particle Swarm Optimization:

I particles moves, particle swarm optimization, variable length PSO,
applications of PSO, case studies. Artificial Bee Colony algorithms - ABC
basics, ABC in optimization, multi-dimensional bee colony algorithms,
applications of bee algorithms, case studies.

Rough Sets: Fundamentals of Rough Set Theory, Rough

Approximations, Properties of Approximations, Measures of
Accuracy, Topological Characterization of Imprecision, Rough
Membership Function Attribute Reduction, Approximation of
Classification, De- pendency of Knowledge.
 Course Outcomes (COs)

A8711.1 Develop intelligent systems leveraging the

paradigm of soft computing tech- niques.
A8711.2 Implement, evaluate and compare solutions by
various soft computing approaches for finding the optimal
solutions.
A8711.3 Recongnize the feasibility of applying a soft
computing methodology for a particluar problem.
A8711.4 Design the methodolgy to solve the optimization
problmes using Fuzzy logic, genetic algorithms and nueral
networks.
A8711.5 Design hybrid systems to revise the principles of
soft computing in various applications.
 Books and Materials

Text Books:
B K Tripathy | J Anuradha, Soft Computing Advances and
Applications:, CENGAGE Learning, 2015.
Reference Books:
J.S.R. Jang, C.T. Sun and E. Mizutani., Neuro-Fuzzy and Soft
Computing, Pearson Education, 2004
N.P.Padhy., Artificial Intelligence and Intelligent Systems,
Oxford University Press, 2005.
, Himalaya Publications, 2006

Hours Per Week Hours Per Semester Credits Assessment Marks

L T P L T P C CIE SEE Total

3 0 0 45 0 0 3 40 60 100
Computing
 Hard Computing
 Hard computing uses traditional mathematical
methods to solve problems, such as algorithms and
mathematical models.
 It is based on deterministic and precise calculations
and is ideal for solving problems that have well-
defined mathematical solutions.
 The term, hard computing, was coined by Dr Lotfi Zadeh.
Here ‘hard’ has nothing to do with hardware.
 The principles of hard computing are precision, certainty
and rigor.
 Hard computing is achieved using sequential programs that
use binary logic. It is deterministic in nature. The input data
should be exact and the output will be precise and
verifiable.
 Hard Computing

Advantages
Accurate solutions can be obtained
Faster
Disadvantages
Not suitable for real world problems
Cannot handle imprecision and partial truth
 Soft Computing
 Soft Computing refers to a consortium of
computational methodologies like fuzzy logic,
neural networks, genetic algorithms etc
 All having their roots in the Artificial Intelligence
 Artificial Intelligence is an area of computer
science concerned with designing intelligent
computer systems.
 Systems that exhibit the characteristics we
associate with intelligence in human behavior.
 Soft Computing
 Soft Computing was introduced by Lotfi A zadeh of
the university of California, Berkley, U.S.A
 The soft computing differs from hard computing in
its tolerance to imprecision, uncertainty and partial
truth.
 Soft Computing is the fusion of methodologies
designed to model and enable solutions to real world
problems, which are not modeled or too difficult to
model mathematically.
 The aim of Soft Computing is to exploit the tolerance
for imprecision, uncertainty, approximate reasoning,
and partial truth in order to achieve close
resemblance with human like decision making.
Soft Computing
 Soft Computing
 The Soft Computing consists of several computing
paradigms mainly :
 Fuzzy Systems, Neural Networks, and Genetic
Algorithms.
 Fuzzy set : for knowledge representation via fuzzy
If – Then rules.
 Neural Networks : for learning and adaptation
 Genetic Algorithms : for evolutionary computation
 Soft Computing
 Its aim is to exploitthetolerance for
Approximation,Uncertainty, Imprecision, and Partial
Truth in order to achieve close resemblance with
human like decision making.
 Approximation : here the model features are similar to
the real ones, but not the same.
 Uncertainty : here we are not sure that the features of
the model are the same as that of the entity (belief).
 Imprecision : here the model features (quantities) are
not the same as that of the real ones, but close to them.
Soft Computing
Soft Computing
Soft Computing
Soft Computing
Soft Computing

Rough Sets:
 Introduction to Neural Network

As we have noted, a glimpse into the natural world reveals that
even a small child is able to do numerous tasks at once.
The example of a child walking, probably the first time that child
sees an obstacle, he/she may not know what to do.
But afterward, whenever he/she meets obstacles, she simply takes
another route.
It is natural for people to both appreciate the observations and
learn from them.
 An intensive study by people coming from multidisciplinary fields
marked the evolution of what we call the artificial neural network
(ANN).
 ANNs- How They Operate

ANNs represent a highly connected network of neurons - the basic

processing unit
They operate in a highly parallel manner.
Each neuron does some amount of information processing.
It derives inputs from some other neuron and in return gives its
output to other neuron for further processing.
This layer-by-layer processing of the information results in great
computational capability.
As a result of this parallel processing, ANNs are able to achieve
great results when applied to real-life problems.
 ANN Architecture

Connection Neurons
Strengths
Connections

w
w
Input 1
Output 1

Input 2
Output 2

Input 3
w w

Input Layer Hidden Layer Output Layer

Historical Note
 Historical Note

Warren McCulloch and mathematical prodigy Walter Pitts gave

McCulloch-Pitts theory of formal neural networks.
 In 1949, Donald Hebb further extended the work in this field, when
he described how neural pathways are strengthened each time they
are used
In 1954, Marvin Minsky presented his thesis “Theory of Neural-
Analog Reinforcement Systems and Its Application to the Brain-
Model Problem” and also wrote a paper titled “Steps Toward
Artificial Intelligence.”
 Historical Note
Later John von Neumann invented the von Neumann machine.
1958, Frank Rosenblatt, a neurobiologist, proposed the perceptron,
which is believed to be the first physical ANN.
Between 1959 and 1960, Bernard Wildrow and Marcian Hoff
developed the Adaptive Linear Elements (ADALINE) and the Multiple
Adaptive Linear Elements (MADELINE) models.
In 1986, Rumelhart, Hinton and Williams proposed the back
propagation algorithm.
 The Biological Neuron
The entire human brain consists of small interconnected processing
units called neurons and are connected to each other by nerve fibers
The interneuron information communication makes it possible for
multilevel hierarchical information processing, which gives the brain
all its problem-solving power.
Each neuron is provided with many inputs, each of which comes
from other neurons.
Each neuron takes a weighted average of the various inputs
presented to it.
The weighted average is then made to pass over a nonlinear
inhibiting function that limits the neuron’s output.
The nonlinearity in biological neurons is provided by the presence of
potassium ions within each neuron cell and sodium ions outside the
cell membrane.
 Structural Components of a Neuron

The difference in concentrations of these two elements causes an

electrical potential difference, which in turn causes a current to flow
from outside the cell to inside the cell. This is how the neuron takes
its inputs
A neuron has four main structural components - the dendrites, the
cell body, the axon, and the synapse.
 Structural Components of a Neuron

Dendrites are responsible for receiving the signals from other

neurons.
These signals are passed through a small, thick fiber called a
dendron.
The received signals collected at different dendrites are processed
within the cell body, and the resulting signal is transferred through a
long fiber called the axon.
At the other end of the axon exists an inhibiting unit called a
synapse, which controls the flow of neuronal current from the
originating neuron to the receiving dendrites of neighboring
neurons.
 The Artificial Neuron
The neural network, by its simulating a biological neural network,
is a novel computer architecture and a novel algorithmization
architecture relative to conventional computers.
It allows using very simple computational operations (additions,
multiplication, and fundamental logic elements) to solve complex,
mathematically ill-defined problems, nonlinear problems, or
stochastic problems.
The artificial neuron is the most basic computational unit of
information processing in ANNs.
Each neuron takes information from a set of neurons, processes it,
and gives the result to another neuron for further processing.
These neurons are a direct result of the study of the human brain
and attempts to imitate the biological neuron.
The Artificial Neuron
 Structure of an Artificial Neuron

The neuron consists of a number of inputs. The information is

given as inputs via input connections, each of which has some
weight associated with it.
An additional input, known as bias, is given to the artificial
neuron.
Biological neural network vs artificial neural network
Relationship between Biological neural network and artificial
neural network:

Biological Neural Network Artificial Neural Network

Dendrites Inputs

Cell nucleus Nodes

Synapse Weights

Axon Output
 McCulloch-Pitts neuron model

• A set of input connections brings in activations from

other neurons.
• A processing unit sums the inputs, and then applies a
non-linear activation function (i.e. squashing /
transfer / threshold function).
• An output line transmits the result to other neurons.
 McCulloch-Pitts neuron model

The processing in a single artificial neuron.

the inputs are marked X1, X2, X3, … , Xn; the weights
associated with each Connection are given by W1, W2, W3, … ,
Wn; b denotes the bias; and the output is denoted by O.
Because there is one weight for every input, the number of
inputs is equal to the number of weights in a neuron.
 The Processing of the Neuron

Functionality of neuron that is performed can be broken down into

two steps. The first is the weighted summation, and the second is
the activation function.
The two steps are applied one after the other, as shown in the
previous slide.
The function of the summation block is given by Equation

The summation forms the input to the next block. This is the block
of the activation function, where the input is made to pass through
a function called the activation function.
 The Activation Function

• The activation function performs several important tasks

• One of the more important of which is to introduce nonlinearity to the

network.

• Another important feature of the activation function is its ability to limit

the neuron’s output.

• The complete mathematical function of the artificial neuron can be

given as:

where f is any activation function

 The Activation Function

• The activation function performs several important tasks

• One of the more important of which is to introduce nonlinearity to the

network.

• Another important feature of the activation function is its ability to limit

the neuron’s output.

• The complete mathematical function of the artificial neuron can be

given as:

where f is any activation function

 Common Activation Functions

Note: f(x) = tanh(x) = 2/(1 + e-2x) – 1 = 2 * sigmoid(2x) - 1

Common Activation Functions
Common Activation Functions
Common Activation Functions
Common Activation Functions
Common Activation Functions
Common Activation Functions
Common Activation Functions
Common Activation Functions
Common Activation Functions
Common Activation Functions
Common Activation Functions
 Example

• Calculate the output for a neuron. The inputs are (0.10, 0.90, 0.05), and the
corresponding weights are (2, 5, 2). Bias is given to be 1. The activation
function is linear. Also draw the neuron architecture.

Solution:
Using Equation, we have

where I1 = 0.1, W1 = 2, I2 = 0.9, W2 = 5, I3 = 0.05, W3 = 2, and b = 1.

f(x)=x
Hence the neuron’s output is
f(W1 * I1 + W2 * I2 + W3 * I3+b)
= f(2 * 0.1 + 5 * 0.9 + 2 * 0.05 +1)
= f(0.2 + 4.5 + 0.1+1)
= f(5.8)
=5.8
 Example
• Calculate the output for a neuron. The inputs are (0.10, 0.90, 0.05), and the
corresponding weights are (2, 5, 2). Bias is given to be 0. The activation
function is logistic. Also draw the neuron architecture.

Solution:
Using Equation, we have

where I1 = 0.1, W1 = 2, I2 = 0.9, W2 = 5, I3 = 0.05, W3 = 2, and b = 0.

Hence the neuron’s output is

f(W1 * I1 + W2 * I2 + W3 * I3)
= f(2 * 0.1 + 5 * 0.9 + 2 * 0.05)
= f(0.2 + 4.5 + 0.1)
= f(4.8)
= 1.008
 Example

• Calculate the output for a neuron. The inputs are (0.10, 0.90, 0.05), and the
corresponding weights are (2, 5, 2). Bias is given to be 0. The activation
function is Binary. Also draw the neuron architecture.

Solution:
Using Equation, we have

where I1 = 0.1, W1 = 2, I2 = 0.9, W2 = 5, I3 = 0.05, W3 = 2, and b = 0.

Hence the neuron’s output is

f(W1 * I1 + W2 * I2 + W3 * I3)
= f(2 * 0.1 + 5 * 0.9 + 2 * 0.05)
= f(0.2 + 4.5 + 0.1)
= f(4.8)
=1
 Example

• Calculate the output for a neuron. The inputs are (0.10, 0.90, 0.05), and the
corresponding weights are (2, 5, 2). Bias is given to be 0. The activation
function is bipolar. Also draw the neuron architecture.

Solution:
Using Equation, we have

where I1 = 0.1, W1 = 2, I2 = 0.9, W2 = 5, I3 = 0.05, W3 = 2, and b = 0.

Hence the neuron’s output is

f(W1 * I1 + W2 * I2 + W3 * I3)
= f(2 * 0.1 + 5 * 0.9 + 2 * 0.05)
= f(0.2 + 4.5 + 0.1)
= f(4.8)
=
 Example

• Calculate the output for a neuron. The inputs are (0.10, 0.90, 0.05), and the
corresponding weights are (2, 5, 2). Bias is given to be 0. The activation
function is signum. Also draw the neuron architecture.

Solution:
Using Equation, we have

where I1 = 0.1, W1 = 2, I2 = 0.9, W2 = 5, I3 = 0.05, W3 = 2, and b = 0.

Hence the neuron’s output is

f(W1 * I1 + W2 * I2 + W3 * I3)
= f(2 * 0.1 + 5 * 0.9 + 2 * 0.05)
= f(0.2 + 4.5 + 0.1)
= f(4.8)
=1
 Example

• Calculate the output for a neuron. The inputs are (0.10, 0.90, 0.05), and the
corresponding weights are (2, 5, 2). Bias is given to be 0. The activation
function is Relu. Also draw the neuron architecture.

Solution:
Using Equation, we have

where I1 = 0.1, W1 = 2, I2 = 0.9, W2 = 5, I3 = 0.05, W3 = 2, and b = 0.

f(x)=max(0,x)
Hence the neuron’s output is

f(W1 * I1 + W2 * I2 + W3 * I3)
= f(2 * 0.1 + 5 * 0.9 + 2 * 0.05)
= f(0.2 + 4.5 + 0.1)
= f(4.8)
=4.8
 Example

• Calculate the output for a neuron. The inputs are (0.10, 0.10, 0.2), and the
corresponding weights are (2, -6, 2). Bias is given to be 0. The activation
function is sigmoid. Also draw the neuron architecture.

Solution:
Using Equation, we have

where I1 = 0.1, W1 = 2, I2 = 0.1, W2 = -6, I3 = 0.2, W3 = 2, and b = 0.

f(x)=sigmoid
Hence the neuron’s output is

f(W1 * I1 + W2 * I2 + W3 * I3)
=f(0)=1/(1+e-0)=1/(1+1)=0.5
 Example

• Calculate the output for a neuron. The inputs are (0.10, 0.10, 0.2), and the
corresponding weights are (2, -6, 2). Bias is given to be 0. The activation
function is hyperbolic. Also draw the neuron architecture.

Solution:
Using Equation, we have

where I1 = 0.1, W1 = 2, I2 = 0.1, W2 = -6, I3 = 0.2, W3 = 2, and b = 0.

f(x)=Hyperbolic
Hence the neuron’s output is

f(W1 * I1 + W2 * I2 + W3 * I3)
=f(0)=(e0-e-0)/ (e0+e-0)= (1-1)/(1+1)=0/2=0
Classification of ANNs
Classification of ANNs
 Classification of ANNs
A network topology is the arrangement of a network along with
its nodes and connecting lines. According to the topology, ANN
can be classified as the following kinds −

Feedforward Network:
It is a non-recurrent network having processing units/nodes in
layers and all the nodes in a layer are connected with the
nodes of the previous layers.
The connection has different weights upon them.
There is no feedback loop means the signal can only flow in
one direction, from input to output.
It may be divided into the following two types −
1. Single layer feedforward network
2. Multilayer feedforward network
 Classification of ANNs
Single layer feedforward network:
The concept is of feedforward ANN having only one
weighted layer.
 In other words, we can say the input layer is fully
connected to the output layer.
 Classification of ANNs
Multy layer feedforward network:
The concept is of feedforward ANN having more than one
weighted layer.
 As this network has one or more layers between the input
and the output layer, it is called hidden layers.
 Classification of ANNs
Feedback Network:
As the name suggests, a feedback network has feedback
paths, which means the signal can flow in both directions
using loops.
This makes it a non-linear dynamic system, which
changes continuously until it reaches a state of equilibrium.

Recurrent networks: They are feedback networks with

closed loops. Following are the two types of recurrent
networks.
 Classification of ANNs
1. Fully recurrent network: It is the simplest neural
network architecture because all nodes are connected to all
other nodes and each node works as both input and output.
 Classification of ANNs
2. Jordan network − It is a closed loop network in which
the output will go to the input again as feedback as shown
in the following diagram.
McCulloch-Pitts(MCP) neuron model
McCulloch-Pitts neuron model
McCulloch-Pitts neuron model

x1*w1+x2*w2=sum
0*1+0*1=0
0*1+1*1=1
1*1+0*1=1
1*1+1*1=2
McCulloch-Pitts neuron model

x1*w1+x2*w2=sum
0*2+0*2=0
0*2+1*2=2
1*2+0*2=2
1*2+1*2=4
 Q&A

1. Whatwill be threshold value for or function using

McCulloch Pitts neural network:
A.0
B.1
C.2
D.Not Solvable
 Q&A

1. Whatwill be threshold value for or function using

McCulloch Pitts neural network:
A.0
B.1
C.2
D.Not Solvable
 Q&A
2. Whatwill be threshold value for and function using
McCulloch Pitts neural network:
A.0
B.1
C.2
D.Not Solvable
 Q&A
2. Whatwill be threshold value for and function using
McCulloch Pitts neural network:
A.0
B.1
C.2
D.Not Solvable
 Q&A
3. What was the name of the first model which can
perform wieghted sum of inputs?
a) McCulloch-pitts neuron model
b) Marvin Minsky neuron model
c) Hopfield model of neuron
d) none of the mentioned
 Q&A
3. What was the name of the first model which can
perform wieghted sum of inputs?
a) McCulloch-pitts neuron model
b) Marvin Minsky neuron model
c) Hopfield model of neuron
d) none of the mentioned
x1 x2 Not x1 Not x2 X1 and not x2 X2 and not x1 X1 and not x2
Or X2 and not x1
0 0 1 1 0 0 0
0 1 1 0 0 1 1
1 0 0 1 1 0 1
1 1 0 0 0 0 0
Classification of Learning Algorithms
Classification of Learning Algorithms
Classification of Learning Algorithms
Classification of Learning Algorithms
Classification of Learning Algorithms
Classification of Learning Algorithms
Classification of Learning Algorithms
 The Perceptron
The perceptron is the most basic model of the ANN. It consists of a
single neuron. The perceptron may be seen as a binary classifier
that maps every input to an output that is either 0 or 1.

 The perceptron is given by the function represented by Equation

where w is the vector of real-world weights, x is the input, “.” is the

dot product of the two vectors, and b is the bias.

The perceptron has learning capabilities in that it can learn from

the inputs to adapt itself. As a result, the perceptron is able to learn
historical data.
 Multilayer Perceptron

• Multilayer perceptrons are networks used to overcome the linear

reparability limitation of the perceptrons.
 Multilayer Perceptron
An MLP consists of
 1. An input layer,
 2. At least one intermediate or “hidden” layer,
 3. And one output layer
The neurons from each layer being fully connected (in some particular
applications, partially connected) to the neurons from the next layer.
The number of neurons in the input layer is equal to the number of
inputs of the system. Each neuron corresponds to one input.
In output layer, the number of neurons is equal to the number of outputs
that the system is supposed to provide.
The number of layers decides the computational complexity of the ANN.
 Types of Data Involved

The data is generally divided into three sets

 Training data : These data are used by the training algorithm to
set the ANN’s parameters, weights, and biases. Training data make
up the largest set of data, comprising almost 80 percent of the data.
Testing data: This data set is used when the final ANN is ready.
Testing data, which are completely unknown, are used to test the
system’s actual performance. The testing data set consists of about
10 percent of the data.
Validation data: These data are used for validation purposes
during the training period. At any point during training, we measure
the error on both the training data and the validation data. The size
of this data set is about 10 percent of the entire data set.
 Training

ANNs are able to learn the data presented in a process known as

training
The training algorithm tries to adjust the various weights (and
biases) and set them to a value such that the ANN performs better
at the applied input.
Note that each ANN might not be able to train itself well for all
architectures.
This performance and training depends on the number of hidden
layers as well as on the neurons in these hidden layers.
Changing the architecture cannot be performed during training.
Because this entire activity is manual, it may take a long time and a
great deal of experience before a designer is able to selects a final
design.
 Stopping Condition
The algorithm stops according to the stopping condition. Normally
one or more of the following criteria are used as stopping conditions:
 Time: The algorithm may be stopped when the time taken to
execute exceeds more than a threshold.
 Epoch: The algorithm has a specified maximum number of
epochs. Upon exceeding this number, the algorithm may be stopped
 Goal: The algorithm may be stopped if the error measured by the
system reduces to more than a specific value. It may not be useful to
continue training after this point.
 Validating data: If the error on validation data starts increasing,
even if there is a decrease in the testing data, it would be better to stop
further training.
 Gradient: Gradient refers to the improvement of the performance or
the lowering of the error in between epochs.
 Q&A

1. Artificial Intelligence is the XOR problem exceptionally

exciting to neural network researchers because it can be?
(A). expressed in a manner that permits you to use a neural
network
(B). the simplest linearly inseparable problem that exists.
(C). solved by a single layer perceptron
(D). it is a complex binary operation and it is unsolvable
using neural networks
 Q&A

1. Artificial Intelligence is the XOR problem exceptionally

exciting to neural network researchers because it can be?
(A). expressed in a manner that permits you to use a neural
network
(B). the simplest linearly inseparable problem that exists.
(C). solved by a single layer perceptron
(D). it is a complex binary operation and it is unsolvable
using neural networks
 Q&A
2. A perceptron receives the weighted inputs and totals up,
and if it increases a certain value, the value of its output
will be 1, otherwise, it just outputs the value of 0.
(A). True
(B). False
(C). Occasionally, it can also output intermediary values as
well
(D). None of these
 Q&A
2. A perceptron receives the weighted inputs and totals up,
and if it increases a certain value, the value of its output
will be 1, otherwise, it just outputs the value of 0.
(A). True
(B). False
(C). Occasionally, it can also output intermediary values as
well
(D). None of these
 Q&A
3. What is hebb’s rule of learning
a) the system learns from its past mistakes
b) the system recalls previous reference inputs & respective
ideal outputs
c) the strength of neural connection get modified
accordingly
d) none of the mentioned
 Q&A
3. What is hebb’s rule of learning
a) the system learns from its past mistakes
b) the system recalls previous reference inputs & respective
ideal outputs
c) the strength of neural connection get modified
accordingly
d) none of the mentioned
 Q&A

4. Who proposed the first perceptron model in 1957?

a) McCulloch-pitts
b) Marvin Minsky
c) Hopfield
d) Rosenblatt
 Q&A
4. Who proposed the first perceptron model in 1957?
a) McCulloch-pitts
b) Marvin Minsky
c) Hopfield
d) Rosenblatt
• Suppose w1=0.6 and w2=0.8
• Threshold =1, α=0.5
• Learn the NN using Delta learning rule and minimise the error
• Round- 1
• Case 1: A=x1=0, B=x2=0
So, sum=0*0.6+0*0.8=0
Since 0<1 , predicted output=0
Thus no error because predicted value= Target value
• Case-2: A=x1=0, B=x2=1
• So, sum=0*0.6+1*0.8=0.8
• Since 0.8<1 , predicted output=0
• But here target output=1
• Find e=T-Y=1-0=1
∆wi= α*e*xi
Here ∆wi = weight change for ith ⁡pattern;
α = the positive and constant learning rate;
xi= the input value from pre-synaptic neuron;
e = (t−yin) the difference between the desired/target output and the
actual output ⁡yin
when t ≠ y, then
w(new)=w(old)+Δw
∆w1= α*e*x1 = 0.5*1*0=0
∆w2= α*e*x2 = 0.5*1*1=0.5
W1(new)=w1+ ∆w1=0.6+0=0.6
W2(new)=w2+ ∆w2=0.8+0.5=1.3
• Case 3:
• A=x1=1, B=x2=0
• So, sum=1*0.6+0*1.3=0.6
• Since 0.6<1 , predicted output=0
• But here target output=1
• Find e=T-Y=1-0=1
∆w1= α*e*x1 = 0.5*1*1=0.5
∆w2= α*e*x2 = 0.5*1*0=0
W1(new)=w1+ ∆w1=0.6+0.5=1.1
W2(new)=w2+ ∆w2=1.3+0=1.3
• Case 4:
A=x1=1, B=x2=1
So, sum=1*1.1+1*1.3=2.4
Since 2.4>1 , predicted output=1
But here target output=1
Find e=T-Y=1-1=0
• Round-2
• Case 1:
• A=x1=0, B=x2=0
• So, sum=0*1.1+0*1.3=0
• Since 0<1 , predicted output=0
• But here target output=0
• Find e=T-Y=0
• Case 2:
A=x1=0, B=x2=1
So, sum=0*1.1+1*1.3=1.3
Since 1.3>1 , predicted output=1
But here target output=1
Find e=T-Y=1-1=0
• Case 3:
A=x1=1, B=x2=0
So, sum=1*1.1+0*1.3=1.1
Since 1.1>1 , predicted output=1
But here target output=1
Find e=T-Y=1-1=0
• Case 4:
A=x1=1, B=x2=1
So, sum=1*1.1+1*1.3=2.4
Since 2.4>1 , predicted output=1
But here target output=1
Find e=T-Y=1-1=0
Back propagation neural networks
Back propagation neural networks
Back propagation neural networks
Back propagation neural networks
 Back propagation neural networks
• Back propagation algorithm calculates the gradient of the error function.
• Back propagation can be written as a function of the neural network.
• Back propagation algorithms are a set of methods used to efficiently train artificial neural networks following a gradient descent approach which exploits the chain rule.
 Back propagation neural networks
• The main features of Backpropagation are the iterative, recursive and efficient method through which it calculates the updated weight to
improve the network until it is not able to perform the task for which it is being trained.
• Derivatives of the activation function to be known at network design time is required to Back propagation.
Back propagation neural networks
Back propagation neural networks
Input values
X1=0.05 X2=0.10

Initial weight
W1=0.15 w5=0.40
W2=0.20 w6=0.45
W3=0.25 w7=0.50
W4=0.30 w8=0.55

Bias Values
b1=0.35 b2=0.60

Target Values
T1=0.01 T2=0.99
Back propagation Algorithm
Back propagation Algorithm
Back propagation Algorithm
Back propagation Algorithm
Back propagation Algorithm
Back propagation Algorithm
Back propagation Algorithm
Example
Input values
X1=0.05 X2=0.10

Initial weight
W1=0.15 w5=0.40
W2=0.20 w6=0.45
W3=0.25 w7=0.50
W4=0.30 w8=0.55

Bias Values
b1=0.35 b2=0.60

Target Values
T1=0.01 T2=0.99
Assignment
 ADALINE MODEL OF ANN
ADALINE (Adaptive Linear Neuron or later Adaptive Linear Element) is
an early single-layer artificial neural network and the name of the
physical device that implemented this network.
The network uses memistors.
It was developed by Professor Bernard Widrow and his graduate
student Ted Hoff at Stanford University in 1960.
It is based on the McCulloch–Pitts neuron. It consists of a weight, a
bias and a summation function.
The difference between Adaline and the standard (McCulloch–Pitts)
perceptron is that in the learning phase, the weights are adjusted
according to the weighted sum of the inputs (the net).
In the standard perceptron, the net is passed to the activation
(transfer) function and the function's output is used for adjusting the
weights.
 ADALINE MODEL OF ANN
The basic structure of Adaline is similar to perceptron having an extra
feedback loop with the help of which the actual output is compared
with the desired/target output. After comparison on the basis of
training algorithm, the weights and bias will be updated.
ADALINE MODEL OF ANN
ADALINE MODEL OF ANN
Architecture of MADALINE
Architecture of MADALINE
Architecture of MADALINE
Architecture of MADALINE
Architecture of MADALINE
 Applications of ANN
1.Social Media:Artificial Neural Networks are used heavily in Social Media. For
example, let’s take the‘People you may know’feature on Facebook that suggests
people that you might know in real life so that you can send them friend requests.
Well, this magical effect is achieved by using Artificial Neural Networks that analyze
your profile, your interests, your current friends, and also their friends and various
other factors to calculate the people you might potentially know. Another common
application ofMachine Learningin social media isfacial recognition. This is done by
finding around 100 reference points on the person’s face and then matching them
with those already available in the database using convolutional neural networks.
2.Marketing and Sales:When you log onto E-commerce sites like Amazon and
Flipkart, they will recommend your products to buy based on your previous browsing
history. Similarly, suppose you love Pasta, then Zomato, Swiggy, etc. will show you
restaurant recommendations based on your tastes and previous order history. This is
true across all new-age marketing segments like Book sites, Movie services,
Hospitality sites, etc. and it is done by implementingpersonalized marketing. This
uses Artificial Neural Networks to identify the customer likes, dislikes, previous
shopping history, etc., and then tailor the marketing campaigns accordingly.
 Applications of ANN
3.Healthcare: Artificial Neural Networks are used in Oncology to train algorithms
that can identify cancerous tissue at the microscopic level at the same accuracy as
trained physicians. Various rare diseases may manifest in physical characteristics
and can be identified in their premature stages by usingFacial Analysison the
patient photos. So the full-scale implementation of Artificial Neural Networks in the
healthcare environment can only enhance the diagnostic abilities of medical experts
and ultimately lead to the overall improvement in the quality of medical care all over
the world.
4.Personal Assistants:I am sure you all have heard of Siri, Alexa, Cortana, etc.,
and also heard them based on the phones you have!!! These are personal assistants
and an example of speech recognition that usesNatural Language Processingto
interact with the users and formulate a response accordingly. Natural Language
Processing uses artificial neural networks that are made to handle many tasks of
these personal assistants such as managing the language syntax, semantics,
correct speech, the conversation that is going on, etc.
Applications of ANN
Applications of ANN
 Associative Memory
 Associative memory is also known as content addressable
memory (CAM) or associative storage or associative array. It is
a special type of memory that is optimized for performing
searches through data, as opposed to providing a simple direct
access to the data based on the address.
 it can store the set of patterns as memories when the
associative memory is being presented with a key pattern, it
responds by producing one of the stored pattern which closely
resembles or relates to the key pattern.
 It can be viewed as data correlation here. input data is
correlated with that of stored data in the CAM.
 It forms of two type:
 1. auto associative memory network
2. hetero associative memory network :
 Auto Associative Memory
auto associative memory network :
An auto-associative memory network, also known as a
recurrent neural network, is a type of associative memory that is
used to recall a pattern from partial or degraded inputs.
 In an auto-associative network, the output of the network is
fed back into the input, allowing the network to learn and
remember the patterns it has been trained on.
 This type of memory network is commonly used in
applications such as speech and image recognition, where the
input data may be incomplete or noisy.
 Auto Associative Memory
 AANN contains five-layer perceptron feed-forward network,
that can be divided into two neural networks of 3 layers
each connected in series (similar to autoencoder
architecture).
 The network consists of an input layer followed by a hidden
layer and bottleneck layer.
 This bottleneck layer is common between both the network
and a key component of the network.
 It provides data compression to the input and topology with
powerful feature extraction capabilities.
 The bottleneck layer is followed by a second non-linear
hidden layer and the output layer of the second network.
 The first network compresses the information of the n-dimensional
vector to smaller dimension vectors that contain a smaller number of
characteristic variables and represent the whole process.
 The second network works opposite to the first and uses
compressed information to regenerate the original n redundant
Auto Associative Memory
Auto Associative Memory
Hetero Associative Memory
 A hetero-associative memory network is a type of
associative memory that is used to associate one set
of patterns with another.
 In a hetero-associative network, the input pattern is
associated with a different output pattern, allowing the
network to learn and remember the associations
between the two sets of patterns.
 This type of memory network is commonly used in
applications such as data compression and data
retrieval.
Hetero Associative Memory
Hetero Associative Memory
 Bidirectional Associative Network
 Bidirectional Associative Memory (BAM) is a
supervised learning model in Artificial Neural Network.
 This is hetero-associative memory, for an input
pattern, it returns another pattern which is potentially
of a different size. This phenomenon is very similar to
the human brain. Human memory is necessarily
associative.
 It uses a chain of mental associations to recover a
lost memory like associations of faces with names, in
exam questions with answers, etc.
 In such memory associations for one type of object
with another, a Recurrent Neural Network (RNN) is
needed to receive a pattern of one set of neurons as
an input and generate a related, but different, output
pattern of another set of neurons.
 Bidirectional Associative Network
 Why BAM is required? The main objective to
introduce such a network model is to store hetero-
associative pattern pairs. This is used to retrieve a
pattern given a noisy or incomplete pattern.
 BAM Architecture: When BAM accepts an input of n-
dimensional vector X from set A then the model
recalls m-dimensional vector Y from set B. Similarly
when Y is treated as input, the BAM recalls X.
Bidirectional Associative Network
Bidirectional Associative Network
Organization of associative memory
Organization of associative memory
 Argument Register: It contains words to be searched.
It contains ‘n’ number of bits.
 Match Register: It has m-bits, One bit corresponding
to each word in the memory array. After the making
process, the bits corresponding to matching words in
match register are set to ‘1’.
 Key Register: It provides a mask of choosing a
particular field/key in argument register. It specifies
which part of the argument word need to be
compared with words in memory.
 Associative Memory Array: It combines word in that
are to be compared with the arguments word in
parallel. It contains ‘m’ words with ‘n’ bit per word.
 Applications of Associative Memory
 It can be only used in memory allocation format.
 It is widely used in the database management systems, etc.
 Networking: Associative memory is used in network routing
tables to quickly find the path to a destination network based
on its address.
 Image processing: Associative memory is used in image
processing applications to search for specific features or
patterns within an image.
 Artificial intelligence: Associative memory is used in artificial
intelligence applications such as expert systems and pattern
recognition.
 Database management: Associative memory can be used in
database management systems to quickly retrieve data based
on its content.
12/12/14 CSE510, Lovely Professional University, Phagwara