INTRODUCTION

What is a neural network?

Work on artificial neural networks, commonly referred to as '''neural networks," has been
motivated right from its inception by the recognition that the human brain computes in an entirely
different way from the conventional digital computer. The brain is a highly complex, nonlinear,
and parallel computer (information-processing system). It has the capability to organize its
structural constituents known as neurons. so as to perform certain computations many times faster
than the fastest digital computer in existence today.
Consider, for example, human vision, which is an information-processing task. It is the
function of the visual system to provide a representation of the environment around us and supply
the information we need to interact with the environment.
For another example, consider the sonar of a bat. Sonar is an active echo-location system.
In addition to providing information about how far away a target (e.g., a flying insect) is, a bat
sonar conveys information about the relative velocity of the target, the size of the target, the size
of various features of the target.
At birth, a brain has great structure and the ability to build up its own rules through what
we usually refer to as "experience."
Indeed, experience is built up over time, with the most dramatic development of the human brain
taking place during the first two years from birth; but the development continues well beyond that
stage.
A "developing" neuron is synonymous with a plastic brain: Plasticity permits the
developing nervous system to adapt to its surrounding environment. Just as plasticity appears to
be essential to the functioning of neurons as information-processing units in the human brain, so it
is with neural networks made up of artificial neurons.
A neural network is a machine that is designed to model the way in which the brain
performs a particular task & network is usually implemented by using electronic
components or is simulated in software on a digital computer.
To achieve good performance, neural networks employ a massive interconnection of
simple computing cells referred to as "neurons" or "processing units."
A neural network is a massively parallel distributed processor made up of simple
processing units, which has a natural propensity for storing experiential knowledge and making it
available for use. It resembles the brain in two respects:
1. Knowledge is acquired by the network from its environment through a learning process.
2. Interneuron connection strengths, known as synaptic weights, are used to store the acquired
knowledge.
The procedure used to perform the learning process is called a learning algorithm, the
function of which is to modify the synaptic weights of the network in an orderly fashion to attain a
desired design objective. Neural networks are also referred to in literature as neurocomputers,
connectionist networks, parallel distributed processors, etc.

Benefits of Neural Networks

It is apparent that a neural network derives its computing power through,
1. Its massively parallel distributed structure.
2. Its ability to learn and therefore generalize.
Generalization refers to the neural network producing reasonable outputs for inputs not
encountered during training (learning). These two information-processing capabilities make it
possible for neural networks to solve complex (large-scale) problems by decomposing into a
number of relatively simple tasks, and neural networks are assigned a subset of the tasks that
match their inherent capabilities.
The use of neural networks offers the following useful properties and capabilities:
1. Nonlinearity An artificial neuron can be linear or nonlinear. A neural network, made up of an
interconnection of nonlinear neurons, is itself nonlinear. Nonlinearity is a highly important
 property if the underlying physical mechanism responsible for generation of the input signal is
inherently nonlinear.
2. Input-Output Mapping A popular paradigm of learning called learning with a teacher or
supervised learning involves modification of the synaptic weights of a neural network by applying
a set of labeled training samples or task examples. Each example consists of a unique input signal
and a corresponding desired response. The network is presented with an example picked at
random from the set, and the synaptic weights of the network are modified to minimize the
difference between the desired response and the actual response of the network produced by the
input signal in accordance with an appropriate statistical criterion. The training of the network is
repeated for many examples in the set until the network reaches a steady state where there are no
further significant changes in the synaptic weights.
3. Adaptivity Neural networks have a built-in capability to adapt their synaptic weights to
changes in the surrounding environment. Moreover, when it is operating in a nonstationary
environment, a neural network can be designed to change its synaptic weights in real time. The
natural architecture of a neural network for pattern classification, signal processing, and control
applications, coupled with the adaptive capability of the network, make it a useful tool in adaptive
pattern classification, adaptive signal processing, and adaptive control.
For example, an adaptive system with short time constants may change rapidly and therefore tend
to respond to spurious disturbances, causing a drastic degradation in system performance. To
realize the full benefits of adaptivity, the principal time constants of the system should be long
enough for the system to ignore spurious disturbances and yet short enough to respond to
meaningful changes in the environment.
4. Evidential Response In the context of pattern classification, a neural network can be designed
to provide information not only about which particular pattern to select, but also about the
confidence in the decision made.
5. Contextual Information Knowledge is represented by the very structure and activation state of
a neural network. Every neuron in the network is potentially affected by the global activity of all
other neurons in the network. Consequently, contextual information is dealt with naturally by a
neural network.
6. Fault Tolerance. A neural network, implemented in hardware form, has the potential to be
inherently fault tolerant, or capable of robust computation, in the sense that its performance
degrades gracefully under adverse operating conditions.
For example, if a neuron or its connecting links are damaged, due to the distributed nature of
information stored in the network, the damage has to be extensive before the overall response of
the network is degraded seriously. Thus, in principle, a neural network exhibits a graceful
degradation in performance rather than catastrophic failure.
7. VLSI Implementability The massively parallel nature of a neural network makes it potentially
fast for the computation of certain tasks. This makes a neural network well suited for
implementation using very-large-scale-integrated (VLSI) technology. One particular beneficial
virtue of VLSI is that it provides a means of capturing truly complex behavior in a highly
hierarchical fashion.
8. Uniformity of Analysis and Design Basically, neural networks enjoy universality as
information processors. We say this in the sense that the same notation is used in all domains
involving the application of neural networks. This feature manifests itself in different ways:
• Neurons, in one form or another, represent an ingredient common to all neural networks.
• This commonality makes it possible to share theories and learning algorithms in different
applications of neural networks.
• Modular networks can be built through a seamless integration of modules.
9. Neurobiological Analogy The design of a neural network is motivated by analogy with the brain,
which is a living proof that fault tolerant parallel processing is not only physically possible but also
fast and powerful. Neurobiologists look to (arlificial) neural networks as a research tool for the
interpretation of neurobiological phenomena. On the other hand, engineers look to neurobiology for
 new ideas to solve problems more complex than those based on conventional hard-wired design
techniques.
Biological neurons and Artificial neurons
A biological neuron consists of a cell nucleus that receives input from other neurons
through a web of input terminals or branches called dendrites. The combination of dendrites is
often referred to as a dendritic tree, which receives excitatory or inhibitory signals from other
neurons via an electrochemical exchange of neurotransmitters. Depending on this signal
aggregated from all synapses from the dendritic tree, the neuron is either activated or inhibited, or
in other words, switched on or switched off respectively, after a process called neural summation.
The neuron has an electrochemical threshold, analogous to an activation function in
artificial neural networks, which governs whether the accumulated information is enough to
activate the neuron. The final result is then fed into other neurons and the process begins again.
When humans meet new people or see new things every day, they learn what they look like and
how they evolve with time. This is achieved by making minor alterations to the neural networks
residing in their brains as they evolve. The same phenomenon applies to other tasks as well.

Artificial Neural Network (ANN) tries to approximate the structure of human brain and are
fed with massive amount of data to learn, all at once. Neural network architecture is arranged into
layers, where each layer consists of many simple processing units called nodes, further connected
to several nodes in the layers above and below it. The data is fed into the lowest layer which is
then relayed to the next layer. However, ANNs can also learn based on a pre-existing
representation.

This process is called fine-tuning and consists of adjusting the weights from a pre-trained network
topology at a relatively slow learning rate to perform well on newly supplied input training data.
While artificial neural nets were initially designed to function like biological neural networks, the
neural activity and capability in our brains is far more complex than suggested by artificial
neurons. Unlike Human Neural Networks, ANN cannot yet be trained to work well for many
heterogeneous tasks simultaneously.

Major Differences between these two are -

 Field of application: ANNs are specialized. They can perform one task. They might be
 perfect at playing chess, but they fail at playing go (or vice versa). Biological neural networks
can learn completely new tasks.
 Signal transport and processing: The human brain works asynchronously, ANNs work
synchronously.
 Parameter count: Humans have many billions of adjustable parameters. Even the most
complicated ANNs only have several million learnable parameters.
 Training algorithm: ANNs use Gradient Descent for learning. Human brains use something
different (but we don't know what)
 Processing speed: Single biological neurons are slow, while standard neurons in ANNs are
fast.
 Topology: Biological neural networks have complicated topologies, while ANNs are often in
a tree structure
 Power consumption: Biological neural networks use very little power compared to artificial
networks.
 Training time: Biological networks usually don't stop / start learning. ANNs have different
fitting (train) and prediction (evaluate) phases.

The Functionalities that describe Biological and Artificial neurons are as follows.
1. Size: our brain contains about 10 billion neurons and more than 60 trillion synapses
(connections). The number of “neurons” in artificial networks is much less than that but
comparing their numbers this way is misleading. Deep Neural Networks usually consist of
input neurons, output neurons, and neurons in the hidden layers in between. All the layers are
usually fully connected to the next layer, meaning that artificial neurons usually have as many
connections as there are artificial neurons in the preceding and following layers combined.
The limitation in size isn’t just computational: simply increasing the number of layers and
artificial neurons does not always yield better results in machine learning tasks.
Topology: all artificial layers compute one by one, instead of being part of a network that has
nodes computing asynchronously. Feedforward networks compute the state of one layer of
artificial neurons and their weights, then use the results to compute the following layer the
same way. During backpropagation, the algorithm computes some change in the weights in the
opposing way to reduce the difference of the feedforward computational results in the output
layer from the expected values of the output layer. Layers aren’t connected to non-
neighboring layers, but it’s possible to somewhat mimic loops with recurrent and LSTM
networks. In biological networks, neurons can fire asynchronously in parallel, have small
world nature with a small portion of highly connected neurons (hubs) and a large amount of
lesser connected ones. Since artificial neuron layers are usually fully connected, this small-
world nature of biological neurons can only be simulated by introducing weights that are 0 to
mimic the lack of connections between two neurons.
2. Speed: certain biological neurons can fire around 200 times a second on average. Signals
travel at different speeds depending on the type of the nerve impulse, ranging from 0.61 m/s
up to 119 m/s. Signal travel speeds also vary from person to person depending on their gender,
age, height, temperature, medical condition, lack of sleep, etc. Information in artificial neurons
is instead carried over by the continuous, floating-point number values of synaptic weights.
How quickly feedforward or backpropagation algorithms are calculated carries no
information, other than making the execution and training of the model faster.
3. Fault-tolerance: biological neuron networks, due to their topology, are also fault-tolerant.
Information is stored redundantly so minor failures will not result in memory loss. They don’t
have one “central” part. The brain can also recover and heal to an extent. Artificial neural
networks are not modeled for fault tolerance or self-regeneration, though recovery is possible
by saving the current state (weight values) of the model and continuing the training from that
saved state. Training artificial neural networks for longer periods of time will not affect the
efficiency of the artificial neurons.
4. Power consumption: the brain consumes about 20% of all the human body’s energy —
 despite it’s large cut, an adult brain operates on about 20 watts (barely enough to dimly light a
bulb) being extremely efficient. Our machines are way less efficient than biological systems.
Computers also generate a lot of heat when used, with consumer GPUs operating safely
between 50–80 degrees Celsius instead of 36.5–37.5 °C.
5. Signals: an action potential is either triggered or not — biological synapses either carry a
signal or they don’t. Artificial neurons accept continuous values as inputs and apply a simple
non-linear, easily differentiable function (an activation function) on the sum of its weighted
inputs to restrict the outputs’ range of values.
6. Learning: we still do not understand how brains learn, or how redundant connections store
and recall information. Brain fibers grow and reach out to connect to other neurons,
neuroplasticity allows new connections to be created or areas to move and change function,
and synapses may strengthen or weaken based on their importance. Artificial neural networks
in the other hand have a predefined model, where no further neurons or connections can be
added or removed. Only the weights of the connections can change during training. Learning
can be understood as the process of finding optimal weights to minimize the differences
between the network’s expected and generated output.

Model of an ANN
A neuron is an information-processing unit that is fundamental to the operation of a neural
network. The following diagram shows the model of a neuron, which forms the basis for
designing (artificial) neural networks.

Here we identify three basic elements of the neuronal model:

1. A set of synapses or connecting links, each of which is characterized by a weight or strength of
its own. Specifically, a signal xj at the input of synapse j connected to neuron k is multiplied by
the synaptic weight wkj"
It is important to make a note of the manner in which the subscripts of the synaptic weight wkj are
written. The first subscript refers to the neuron in question and the second subscript refers to the
input end of the synapse to which the weight refers. Unlike a synapse in the brain, the synaptic
weight of an artificial neuron may lie in a range that includes negative as well as positive values.
2. An adder for summing the input signals, weighted by the respective synapses of the neuron; the
operations described here constitute a linear combiner.
3. An activation function for limiting the amplitude of the output of a neuron. The activation
function is also referred to as a squashing function in that it squashes (limits) the permissible
amplitude range of the output signal to some finite value. Typically, the normalized amplitude
range of the output of a neuron is written as the closed unit interval [0,1] or alternatively [- 1,1].
The neuronal model also includes an externally applied bias, denoted by bk. The bias bk has the
effect of increasing or lowering the net input of the activation function, depending on whether it is
positive or negative, respectively.
In mathematical terms, we may describe a neuron k by writing the following pair of equations:

(1)
and

(2)
 Where x1,x2,….xm are the input signals; Wk1, Wk2, ….. Wkm are the synaptic weights of
neuron k; Uk is the linear combiner output due to the input signals; bk is the bias;
is the activation function; and Yk is the output signal of the neuron.
The use of bias bk has the effect of applying an affine transformation to the output uk of
the linear combiner in the model of Figure, as shown by
(3)
The bias bk is an external parameter of artificial neuron k.

and

We may therefore reformulate the model of neruon k as in Following Figure.

In this figure the effect of the bias is accounted for by doing two things: (1) adding a new
input signal fixed at + 1. and (2) adding a new synaptic weight equal to the bias bk.
Although the models of Fig. 1.5 and 1.7 are different in appearance, they are
mathematically equivalent.
Activation functions used in ANNs
The activation function denoted by defines the output of a neuron in
terms of the induced local field v. Here we identify three basic types of activation
functions:
1. Threshold Function
2. Piecewise-Linear Function
3. Sigmoid Function

Threshold Function For this type of activation function described in following figure , we
have
 The output of neuron k employing such a threshold function is expressed as

where vk is the induced local field of the neuron;

Such a neuron is referred to in the literature as the McCulloch-Pitts model, in

recognition of the pioneering work done by McCulloch and Pitt.
In this model, the output of a neuron takes on the value of 1 if the induced local field of
that neuron is nonnegative and 0 otherwise.
This statement describes the all-or-none property of the McCulloch-Pitts model.

Piecewise - Linear Function For the piecewise-linear function described in following

figure, we have

The following two situations may be viewed as special forms of the piecewise-linear
function:
• A linear combiner arises if the linear region of operation is maintained without
running into saturation.
• The piecewise-linear function reduces to a threshold function if the amplification
factor of the linear region is made infinitely large.

Sigmoid Function The sigmoid function, whose graph is s-shaped, is the most common
form of activation function used in the construction of artificial neural networks. It is
defined as a strictly increasing function that exhibits a graceful balance between linear
and nonlinear behavior.
An example of the sigmoid function is the logistic function defined by

where a is the slope parameter of the sigmoid function. By varying the parameter,
‘ a’, we obtain sigmoid functions of different slopes, as illustrated in the following
figure.
In fact, the slope at the origin equals a/4. In the limit, as the slope parameter approaches
infinity, the sigmoid function becomes simply a threshold function. Whereas a threshold
function assumes the value of 0 or 1, a sigmoid function assumes a continuous range of
values from 0 to 1 .

The activation functions defined in Eqs. range from 0 to + 1. It is sometimes desirable to

have the activation function range from -1 to + 1, in which case the activation function
assumes an antisymmetric form with respect to the origin; that is, the activation function
is an odd function of the induced local field. Specifically, the threshold function is now

which is commonly referred to as the signum function.

For the corresponding form of a sigmoid function we may use the hyperbolic tangent function
defined by

References: JNTU Fastupdates.com;

Neural Networks : A comprehensive Foundation- Simon Haykin.

Typical classes of network architectures

The manner in which the neurons of a neural network are structured is intimately
linked with the learning algorithm used to train the network. In this section we focus our
attention on network architectures (structures). In general, we may identify three
fundamentally different classes of network architectures:
1. Single-Layer Feedforward Networks
2. Multilayer Feedforward Networks
3. Recurrent Networks

Single-Layer Feedforward Networks

In a layered neural network the neurons are organized in the form of layers. In the
simplest form of a layered network, we have an input layer of source nodes that projects
onto an output layer of neurons (computation nodes), but not vice versa, In other words,
this network is strictly a feedforward or acyclic type, It is illustrated in following figure
for the case of four nodes in both the input and output layers. Such a network is called a
single- layer network, with the designation "single-layer" referring to the output layer of
computation nodes (neurons).
Multilayer Feedforward Networks
The second class of a feedforward neural network distinguishes itself by the presence of
one or more hidden layers, whose computation nodes are correspondingly called hidden
neurons or hidden units, The function of hidden neurons is to intervene between the
external input and the network output in some useful manner, By adding one or more
hidden layers, the network is enabled to extract higher-order statistics. The ability of
hidden neurons to extract higher-order statistics is particularly valuable when the size of
the input layer is large.
The source nodes in the input layer of the network supply respective elements of the activation
pattern (input vector), which constitute the input signals applied to the neurons (computation
nodes) in the second layer (i.e., the first hidden layer). The output signals of the second layer are
used as inputs to the third layer, and so on for the rest of the network. Typically, the neurons in
each layer of the network have as their inputs the output signals of the preceding layer only.
The set of output signals of the neurons in the output (final) layer of the network
constitutes the overall response of the network to the activation pattern supplied by the source
nodes in the input (first) layer. The architectural graph in the following Figure illustrates the
layout of a multilayer feedforward neural network for the case of a single hidden layer.
the network in Figure is referred to as a 10-4-2 network because it has 10 source nodes, 4
hidden neurons, and 2 output neurons. As another example, a feedforward network with m
source nodes, h1 neurons in the first hidden layer, h 2 neurons in the second hidden layer, and q
neurons in the output layer is referred to as an m-h1-h2-q network.

The neural network in the belowFigure is said to be fully connected in the sense that
every node in each layer of the network is connected to every other node in the adjacent
forward layer. If some of the communication links (synaptic connections) are missing
from the network, we say that the network is partially connected.
Recurrent Networks
A recurrent neural network distinguishes itself from a feedforward neural network in
that it has at least one feedback loop. For example, a recurrent network may consist of a
single layer of neurons with each neuron feeding its output signal back to the inputs of
all the other neurons, as illustrated in the architectural graph in following Figure. In the
structure depicted in this figure there are no self-feedback loops in the network; self
feedback refers to a situation where the output of a neuron is fed back into its own input.
The recurrent network illustrated in following Figure also has no hidden neurons.