A Review of Artificial Neural Network [ANN]

In machine
learning and cognitive
science, artificial
neural
networks (ANNs) are a family of statistical learning models inspired by biological
neural networks (the central nervous systems of animals, in particular the brain)
and are used to estimate or approximate functions that can depend on a large
number of inputs and are generally unknown. Artificial neural networks are
generally presented as systems of interconnected "neurons" which send messages
to each other. The connections have numeric weights that can be tuned based on
experience, making neural nets adaptive to inputs and capable of learning.
For example, a neural network for handwriting recognition is defined by a
set of input neurons which may be activated by the pixels of an input image. After
being weighted and transformed by a function (determined by the network's
designer), the activations of these neurons are then passed on to other neurons.
This process is repeated until finally, an output neuron is activated. This
determines which character was read.
Like other machine learning methods - systems that learn from data - neural
networks have been used to solve a wide variety of tasks that are hard to solve
using ordinary rule-based programming, including computer vision and speech
recognition.
The first artificial neural network was invented in 1958 by psychologist
Frank Rosenblatt. Called Perceptron, it was intended to model how the human
brain processed visual data and learned to recognize objects. Other researchers
have since used similar ANNs to study human cognition.
Artificial neural networks are a computational tool, based on the properties
of biological neural systems. Neural networks excel in a number of problem areas
where conventional von Neumann computer systems have traditionally been slow
and inefficient. This book is going to discuss the creation and use of artificial
neural networks.
Neural networks, with their remarkable ability to derive meaning from
complicated or imprecise data, can be used to extract patterns and detect trends that

are too complex to be noticed by either humans or other computer techniques. A

trained neural network can be thought of as an "expert" in the category of
information it has been given to analyse. This expert can then be used to provide
projections given new situations of interest and answer "what if" questions.
Other advantages include:
Adaptive learning: An ability to learn how to do tasks based on the data given for
training or initial experience.
Self-Organisation: An ANN can create its own organisation or representation of
the information it receives during learning time.
Real Time Operation: ANN computations may be carried out in parallel, and
special hardware devices are being designed and manufactured which take
advantage of this capability.
Fault Tolerance via Redundant Information Coding: Partial destruction of a
network leads to the corresponding degradation of performance. However, some
network capabilities may be retained even with major network damage.
The back propagation algorithm (Rumelhart and McClelland, 1986) is used
in layered feed-forward ANNs. This means that the artificial neurons are organized
in layers, and send their signals forward, and then the errors are propagated
backwards. The network receives inputs by neurons in the input layer, and the
output of the network is given by the neurons on an output layer. There may be one
or more intermediate hidden layers. The back propagation algorithm uses
supervised learning, which means that we provide the algorithm with examples of
the inputs and outputs we want the network to compute, and then the error
(difference between actual and expected results) is calculated. The idea of the back
propagation algorithm is to reduce this error, until the ANN learns the training
data. The training begins with random weights, and the goal is to adjust them so
that the error will be minimal. The activation function of the artificial neurons in
ANNs implementing the back propagation algorithm is a weighted sum (the sum of
the inputs x multiplied by their ji respective weights w ):
(1) We can see that the activation depends only on the inputs and the
weights. If the output function would be the identity (output=activation), then the

neuron would be called linear. But these have severe limitations. The most
common output function is the sigmoidal function:
(2) The sigmoidal function is very close to one for large positive numbers,
0.5 at zero, and very close to zero for large negative numbers. This allows a
smooth transition between the low and high output of the neuron (close to zero or
close to one). We can see that the output depends only in the activation, which in
turn depends on the values of the inputs and their respective weights. Now, the
goal of the training process is to obtain a desired output when certain inputs are
given. Since the error is the difference between the actual and the desired output,
the error depends on the weights, and we need to adjust the weights in order to
minimize the error. We can define the error function for the output of each neuron:
(3) We take the square of the difference between the output and the desired
target because it will be always positive, and because it will be greater if the
difference is big, and lesser if the difference is small. The error of the network will
simply be the sum of the errors of all the neurons in the output layer:
(4) The back propagation algorithm now calculates how the error depends on
the output, inputs, and weights. After we find this, we can adjust the weights using
the method of gradient descendent:
(5) This formula can be interpreted in the following way: the adjustment of
each weight ji ()w ) will be the negative of a constant eta (0) multiplied by the
dependance of the previous weight on the error of the network, which is the
derivative of E in respect to w . The size of the adjustment will depend on 0, and
on the contribution of the weight to the error of the function. This is, if the weight
contributes a lot to the error, the adjustment will be greater than if it contributes in
a smaller amount. (5) Is used until we find appropriate weights (the error is
minimal). If you do not know derivatives, dont worry, you can see them now as
functions that we will replace right away with algebraic expressions. If you
understand derivatives, derive the expressions yourself and compare your results
with the ones presented here. If you are searching for a mathematical proof of the
back propagation algorithm, you are advised to check it in the suggested reading,
since this is out of the scope of this material. So ji , we only need to find the
derivative of E in respect to w . This is the goal of the back propagation algorithm,

since we need to achieve this backwards. First, we need to calculate how much the
error depends on the output, which isthe derivative of E in respect j to O (from
(3)).
(6) And then, how much the output depends on the activation, which in turn
depends on the weights (from (1) and (2)):
(7) And we can see that (from (6) and (7)):
(8) And so, the adjustment to each weight will be (from (5) and (8)):
(9) We can use (9) as it is for training an ANN with two layers. Now, for
training the network with one more layer we need to make some considerations. If
we want to adjust ik the weights (lets call them v ) of a previous layer, we need
first to calculate how the error depends not on the weight, but in the input from the
previous layer. This is easy, we would i ji just need to change x with w in (7), (8),
and (9). But we also need to see how the error of ik the network depends on the
adjustment of v . So:
(10) Where:
(11) And, assuming that there k ik are inputs u into the neuron with v (from
(7)):
(12) If we want to add yet another layer, we can do the same, calculating
how the error depends on the inputs and weights of the first layer. We should just
be careful with the indexes, since each layer can have a different number of
neurons, and we should not confuse them. For practical reasons, ANNs
implementing the back propagation algorithm do not have too many layers, since
the time for training the networks grows exponentially. Also, there are refinements
to the back propagation algorithm which allow a faster learning.
One way to visualize what goes on is to turn the network upside down and
ask it to enhance an input image in such a way as to elicit a particular
interpretation. Say you want to know what sort of image would result in Banana.
Start with an image full of random noise, then gradually tweak the image towards
what the neural net considers a banana. By itself, that doesnt work very well, but

it does if we impose a prior constraint that the image should have similar statistics
to natural images, such as neighboring pixels needing to be correlated.
The individual neurons are complicated. They have a myriad of parts, subsystems, and control mechanisms. They convey information via a host of
electrochemical pathways. There are over one hundred different classes of neurons,
depending on the classification method used. Together these neurons and their
connections form a process which is not binary, not stable, and not synchronous. In
short, it is nothing like the currently available electronic computers, or even
artificial neural networks.
With the delta rule, as with other types of back propagation, 'learning' is a
supervised process that occurs with each cycle or 'epoch' (i.e. each time the
network is presented with a new input pattern) through a forward activation flow of
outputs, and the backwards error propagation of weight adjustments. More simply,
when a neural network is initially presented with a pattern it makes a random
'guess' as to what it might be. It then sees how far its answer was from the actual
one and makes an appropriate adjustment to its connection weights. More
graphically, the process looks something like this:
The most common application of neural networks in computing today is
to perform one of these easy-for-a-human, difficult-for-a-machine tasks,
often referred to as pattern recognition. Applications range from optical
character recognition (turning printed or handwritten scans into digital text)
to facial recognition. We dont have the time or need to use some of these
more elaborate artificial intelligence algorithms here, but if you are interested
in researching neural networks, Id recommend the books Artificial
Intelligence: A Modern Approach by Stuart J. Russell and Peter Norvig
and AI for Game Developers by David M. Bourg and Glenn Seemann.