Artificial Neural

Networks
History of the Artificial Neural Networks
 history of the ANNs stems from the 1940s, the decade of the first electronic
computer.
 However, the first important step took place in 1957 when Rosenblatt
introduced the first concrete neural model, the perceptron. Rosenblatt also
took part in constructing the first successful neurocomputer, the Mark I
Perceptron. After this, the development of ANNs has proceeded as
described in Figure.
History of the Artificial Neural Networks
 Rosenblatt's original perceptron model contained only one layer. From this,
a multi-layered model was derived in 1960. At first, the use of the multi-
layer perceptron (MLP) was complicated by the lack of a appropriate
learning algorithm.
 In 1974, Werbos came to introduce a so-called backpropagation algorithm
for the three-layered perceptron network.
History of the Artificial Neural Networks
 in 1986, The application area of the MLP networks remained rather limited
until the breakthrough when a general back propagation algorithm for a
multi-layered perceptron was introduced by Rummelhart and Mclelland.
 in 1982, Hopfield brought out his idea of a neural network. Unlike the
neurons in MLP, the Hopfield network consists of only one layer whose
neurons are fully connected with each other.
History of the Artificial Neural Networks
 Since then, new versions of the Hopfield network have been developed.
The Boltzmann machine has been influenced by both the Hopfield network
and the MLP.
History of the Artificial Neural Networks
 in 1988, Radial Basis Function (RBF) networks were first introduced by
Broomhead & Lowe. Although the basic idea of RBF was developed 30
years ago under the name method of potential function, the work by
Broomhead & Lowe opened a new frontier in the neural network
community.
History of the Artificial Neural Networks
 in 1982, A totally unique kind of network model is the Self-Organizing
Map (SOM) introduced by Kohonen. SOM is a certain kind of topological
map which organizes itself based on the input patterns that it is trained with.
The SOM originated from the LVQ (Learning Vector Quantization)
network the underlying idea of which was also Kohonen's in 1972.
 History of Artificial Neural
Networks
Since then, research on artificial neural networks has
remained active, leading to many new network types, as
well as hybrid algorithms and hardware for neural
information processing.
 Artificial Neural Network
An artificial neural network consists of a pool of simple
processing units which communicate by sending signals to
each other over a large number of weighted connections.
Artificial Neural Network
 A set of major aspects of a parallel distributed model include:
 a set of processing units (cells).
 a state of activation for every unit, which equivalent to the output of the
unit.
 connections between the units. Generally each connection is defined by a
weight.
 a propagation rule, which determines the effective input of a unit from its
external inputs.
 an activation function, which determines the new level of activation based
on the effective input and the current activation.
 an external input for each unit.
 a method for information gathering (the learning rule).
 an environment within which the system must operate, providing input
signals and _ if necessary _ error signals.
 Computers vs. Neural Networks
“Standard” Computers Neural Networks

 one CPU highly parallel processing

fast processing units slow processing units

reliable units unreliable units

static infrastructure dynamic infrastructure

Why Artificial Neural Networks?
There are two basic reasons why we are interested in
building artificial neural networks (ANNs):

• Technical viewpoint: Some problems such as

character recognition or the prediction of future
states of a system require massively parallel and
adaptive processing.

• Biological viewpoint: ANNs can be used to

replicate and simulate components of the human
(or animal) brain, thereby giving us insight into
natural information processing.
 Artificial Neural Networks
• The “building blocks” of neural networks are the
neurons.
• In technical systems, we also refer to them as units or nodes.
• Basically, each neuron
 receives input from many other neurons.
 changes its internal state (activation) based on the current
input.
 sends one output signal to many other neurons, possibly
including its input neurons (recurrent network).
 Artificial Neural Networks
• Information is transmitted as a series of electric
impulses, so-called spikes.

• The frequency and phase of these spikes encodes the

information.

• In biological systems, one neuron can be connected to as

many as 10,000 other neurons.

• Usually, a neuron receives its information from other

neurons in a confined area, its so-called receptive field.
 How do ANNs work?
 An artificial neural network (ANN) is either a hardware
implementation or a computer program which strives to
simulate the information processing capabilities of its biological
exemplar. ANNs are typically composed of a great number of
interconnected artificial neurons. The artificial neurons are
simplified models of their biological counterparts.
 ANN is a technique for solving problems by constructing software
that works like our brains.
How do our brains work?
 The Brain is A massively parallel information processing
system.
 Our brains are a huge network of processing elements. A
typical brain contains a network of 10 billion neurons.
How do our brains work?
 A processing element

Dendrites: Input
Cell body: Processor
Synaptic: Link
Axon: Output
How do our brains work?
 A processing element

A neuron is connected to other neurons through about 10,000

synapses
 How do our brains work?
 A processing element

A neuron receives input from other neurons. Inputs are combined.

 How do our brains work?
 A processing element

Once input exceeds a critical level, the neuron discharges a spike ‐

an electrical pulse that travels from the body, down the axon, to
the next neuron(s)
 How do our brains work?
 A processing element

The axon endings almost touch the dendrites or cell body of the
next neuron.
 How do our brains work?
 A processing element

Transmission of an electrical signal from one neuron to the next is

effected by neurotransmitters.
 How do our brains work?
 A processing element

Neurotransmitters are chemicals which are released

from the first neuron and which bind to the
Second.
 How do our brains work?
 A processing element

This link is called a synapse. The strength of the signal that

reaches the next neuron depends on factors such as the amount of
neurotransmitter available.
How do ANNs work?

An artificial neuron is an imitation of a human neuron

 How do ANNs work?
• Now, let us have a look at the model of an artificial neuron.
 How do ANNs work?
.........
Input xm
...
x2 x1

Processing ∑
∑= X1+X2 + ….+Xm =

Output y
 How do ANNs work?
Not all inputs are equal
xm ......... x2 x1
...
Input
w ...
weights m
..
w2 w1

Processing ∑ ∑= X1w1+X2w2 + ….
+Xmwm =y

Output y
 How do ANNs work?
The signal is not passed down to the
next neuron verbatim
xm ......... x2 x1
...
Input
w
w w
...
weights m
..
2
1

Processing ∑
Transfer Function
f(vk)
(Activation Function)

Output y
The output is a function of the input, that is
affected by the weights, and the transfer
functions
 Neural Network
• Humans have an ability to identify patterns within
the accessible information with an astonishingly hi
gh degree of accuracy.
• Whenever you see a car or a bicycle you can imm
ediately recognize what they are. This is because
we have learned over a period of time how a car a
nd bicycle looks like and what their distinguishing
features are.
• Artificial neural networks are computation systems
that intend to imitate human learning capabilities
via a complex architecture that resembles the hu
man nervous system.
Human Nervous System
Human Nervous System
• Human nervous system consists of billions of neuron
s. These neurons collectively process input received
from sensory organs, process the information, and d
ecides what to do in reaction to the input.
• A typical neuron in the human nervous system has t
hree main parts: dendrites, nucleus, and axons.
– The information passed to a neuron is received by
dendrites.
– The nucleus is responsible for processing this infor
mation.
– The output of a neuron is passed to other neurons
via the axon, which is connected to the dendrites
of other neurons further down the network.
 Perceptron
• A perceptron is a simple binary classification
algorithm, proposed by Cornell scientist Frank
Rosenblatt.
• It helps to divide a set of input signals into tw
o parts—“yes” and “no”.
• But unlike many other classification algorithm
s, the perceptron was modeled after the esse
ntial unit of the human brain—the neuron and
has an uncanny ability to learn and solve com
plex problems.
Perceptron
 Perceptron
• A perceptron is a very simple learning ma
chine. It can take in a few inputs, each of
which has a weight to signify how importa
nt it is, and generate an output decision of
“0” or “1”.
• However, when combined with many othe
r perceptrons, it forms an artificial neural
network.
• A neural network can, theoretically, answe
r any question, given enough training dat
a and computing power.
 Multilayer Perceptron
• A multilayer perceptron (MLP) is a percept
ron that teams up with additional perceptr
ons, stacked in several layers, to solve co
mplex problems.
• Each perceptron in the first layer on the le
ft (the input layer), sends outputs to all th
e perceptrons in the second layer (the hid
den layer), and all perceptrons in the seco
nd layer send outputs to the final layer on
the right (the output layer).
Multilayer Perceptron
Multilayer Perceptron
• Each layer can have a large number of percep
trons, and there can be multiple layers, so the
multilayer perceptron can quickly become a v
ery complex system.
• The multilayer perceptron has another, more
common name—a neural network.
• A three-layer MLP, like the diagram in previou
s slide, is called a Non-Deep or Shallow Neural
Network.
• An MLP with four or more layers is called a De
ep Neural Network.
Multilayer Perceptron
• One difference between an MLP and a ne
ural network is that in the classic percept
ron, the decision function is a step functi
on and the output is binary.
• In neural networks that evolved from ML
Ps, other activation functions can be use
d which result in outputs of real values, u
sually between 0 and 1 or between -1 an
d 1.
• This allows for probability-based predicti
ons or classification of items into multipl
e labels.
Structure of a Perceptron
 The Percpetron Learning
Process
1 Takes the inputs, multiplies them by their
weights, and computes their sum
2 Adds a bias factor, the number 1 multiplie
d by a weight
3 Feeds the sum through the activation func
tion
4 The result is the perceptron output
Step-1 Backpropogation
• Takes the inputs, multiplies them by their
weights, and computes their sum
• Why It’s Important ?
– The weights allow the perceptron to evaluate
the relative importance of each of the outputs.
– Neural network algorithms learn by discoverin
g better and better weights that result in a more
accurate prediction.
– There are several algorithms used to fine tune
the weights, the most common is called backpro
pagation.
Step-2 Neural Network
Bias

• Adds a bias factor, the number 1 multiplie

d by a weight
• This is a technical step that makes it possi
ble to move the activation function curve
up and down, or left and right on the num
ber graph.
• It makes it possible to fine-tune the numer
ic output of the perceptron.
 Step-3 Activation Function
• Feeds the sum through the activation functi
on
• The activation function maps the input value
s to the required output values.
• For example, input values could be between
1 and 100, and outputs can be 0 or 1. The a
ctivation function also helps the perceptron
to learn, when it is part of a multilayer perce
ptron (MLP).
• Certain properties of the activation function,
especially its non-linear nature, make it poss
ible to train complex neural networks.
 Step-4 Output
• The perceptron output is a classification
decision.
• In a multilayer perceptron, the output of
one layer’s perceptrons is the input of t
he next layer.
• The output of the final perceptrons, in t
he “output layer”, is the final prediction
of the perceptron learning model.
 Artificial Neural Networks
An ANN can:
1. compute any computable function, by the appropriate
selection of the network topology and weights values.
2. learn from experience!
 Specifically, by trial‐and‐error
 Learning by trial‐and‐error
Continuous process of:
Trial:
Processing an input to produce an output (In
terms of ANN: Compute the output function of a
given input)
Evaluate:
Evaluating this output by comparing the
actual output with the expected output.
Adjust:
Adjust the weights.
 How it works?
 Set initial values of the weights randomly.
 Input: truth table of the XOR
 Do
 Read input (e.g. 0, and 0)
 Compute an output (e.g. 0.60543)
 Compare it to the expected output. (Diff= 0.60543)
 Modify the weights accordingly.
 Loop until a condition is met
 Condition: certain number of iterations
 Condition: error threshold
Design Issues
Initial weights (small random values ∈[‐1,1])
Transfer function (How the inputs and the weights are
combined to produce output?)
Error estimation
Weights adjusting
Number of neurons
Data representation
Size of training set
 Transfer Functions
Linear: The output is proportional to the total
weighted input.
Threshold: The output is set at one of two values,
depending on whether the total weighted input is
greater than or less than some threshold value.
Non‐linear: The output varies continuously but not
linearly as the input changes.
 Error Estimation
The root mean square error (RMSE) is a
frequently-used measure of the differences between
values predicted by a model or an estimator and the
values actually observed from the thing being
modeled or estimated
Weights Adjusting
After each iteration, weights should be adjusted to
minimize the error.
– All possible weights
– Back propagation
Back Propagation
Back-propagation is an example of
supervised learning is used at each layer
to minimize the error between the
layer’s response and the actual data
The error at each hidden layer is an
average of the evaluated error
Hidden layer networks are trained this
way
Back Propagation
N is a neuron.
Nw is one of N’s inputs weights
Nout is N’s output.
Nw = Nw +Δ Nw
Δ Nw = Nout * (1‐ Nout)* NErrorFactor
NErrorFactor = NExpectedOutput – NActualOutput
This works only for the last layer, as we can know
the actual output, and the expected output.
Number

of neurons
Many neurons:
Higher accuracy
Slower
Risk of over‐fitting
 Memorizing, rather than understanding
 The network will be useless with new problems.

Few neurons:
 Lower accuracy
Inability to learn at all
Optimal number.
 Data representation
Usually input/output data needs pre‐processing
 Pictures
 Pixel intensity
Text:
A pattern
 Size of training set
No one‐fits‐all formula
Over fitting can occur if a “good” training set is not
chosen
What constitutes a “good” training set?
 Samples must represent the general population.
Samples must contain members of each class.
Samples in each class must contain a wide range of
variations or noise effect.
The size of the training set is related to the number of
hidden neurons
Learning Paradigms
Supervised learning
Unsupervised learning
Reinforcement learning
 Supervised learning
This is what we have seen so far!
A network is fed with a set of training
samples (inputs and corresponding
output), and it uses these samples to learn
the general relationship between the
inputs and the outputs.
This relationship is represented by the
values of the weights of the trained
network.
Unsupervised learning
No desired output is associated with the
training data!
Faster than supervised learning
Used to find out structures within data:
 Clustering
 Compression
Reinforcement learning
Like supervised learning, but:
Weights adjusting is not directly related to the error
value.
The error value is used to randomly, shuffle weights!
Relatively slow learning due to ‘randomness’.
 Applications Areas
Function approximation
including time series prediction and modeling.
Classification
including patterns and sequences recognition, novelty
detection and sequential decision making.
 (radar systems, face identification, handwritten text recognition)

Data processing
including filtering, clustering blinds source separation and
compression.
 (data mining, e-mail Spam filtering)
Advantages / Disadvantages
Advantages
Adapt to unknown situations
Powerful, it can model complex functions.
Ease of use, learns by example, and very little user
domain‐specific expertise needed
Disadvantages
Forgets
Not exact
Large complexity of the network structure
 Conclusion
 Artificial Neural Networks are an imitation of the biological
neural networks, but much simpler ones.
 The computing would have a lot to gain from neural networks.
Their ability to learn by example makes them very flexible and
powerful furthermore there is need to device an algorithm in
order to perform a specific task.
 Conclusion
 Neural networks also contributes to area of research such a
neurology and psychology. They are regularly used to model
parts of living organizations and to investigate the internal
mechanisms of the brain.
 Many factors affect the performance of ANNs, such as the
transfer functions, size of training sample, network topology,
weights adjusting algorithm, …
 References
 Craig Heller, and David Sadava, Life: The Science of Biology, fifth edition,
Sinauer Associates, INC, USA, 1998.
 Introduction to Artificial Neural Networks, Nicolas Galoppo von Borries
 Tom M. Mitchell, Machine Learning, WCB McGraw-Hill, Boston, 1997.
Thank You
Q. How does each neuron work in ANNS?
What is back propagation?
A neuron: receives input from many other neurons;
changes its internal state (activation) based on the
current input;
sends one output signal to many other neurons, possibly
including its input neurons (ANN is recurrent network).

Back-propagation is a type of supervised

learning, used at each layer to minimize the
error between the layer’s response and the
actual data.