Deep Learning

Artificial Neural Network (ANN)

Agenda
● Limitations of simple Neural Network
● Artificial Neural Network (ANN)

2
1. Artificial Neural Network (ANN)
The following section gives a brief introduction of perceptron, illustrates different components
and layers of multilayer neural network along with backpropagation algorithm, and discusses
different use cases of ANNs as well.

1. Introduction – A Perceptron
An Artificial Neural Network is a computational network, based on biological neural networks
that construct the structure of the human brain. Similar to how a human brain has neurons
interconnected to each other, artificial neural networks also have neurons that are linked to each
other in various layers of the networks. A typical single neuron (also known as perceptron) is
represented below.

3
 1. Multi-layer ANN Layers
A typical Neural Network contains a large number of artificial neurons
called units arranged in a series of layers. ANNs have an input layer
and output layer. Between these two layers there are other hidden layers
that perform the mathematical computations that help determine the
decision or action the machine should take. Ultimately, these hidden
layers are in place to transform the input data into something the output
unit can use. A typical multi-layer ANN is represented in the image
below.

4
 1. Back Propagation Algorithm
It is the training or learning algorithm. It learns by example. If you submit to the algorithm the example of what you want the
network to do, it changes the network’s weights so that it can produce desired output for a particular input on finishing the
training.
The backprop algorithm cycles through two distinct passes, a forward pass followed by a backward pass through the layers of the
network. The algorithm alternates between these passes several times as it scans the training data. Typically, the training data has
to be scanned several times before the networks “learns” to make good classifications.
Forward Pass: Computation of outputs of all the neurons in the network.
The input is fed to the input layer, the neurons perform a linear transformation on this input using the weights and biases. Post
that, an activation function is applied on this linear transformation to add non-linearity in the model in order to learn complex
pattern from the data. Finally, the output from the activation function moves to the next hidden layer and the same process is
repeated. This forward movement of information is known as the forward pass or propagation.

5
Backward pass: Propagation of error and adjustment of weights
The task is to make the output to the neural network as close to the actual (desired) output. Each of these
neurons is contributing some error to the final output. How do you reduce the error? We try to minimize
the value/weight of neurons that are contributing more to the error and this happens while traveling back
to the neurons of the neural network and finding where the error lies. In order to minimize the error, the
neural networks use a common algorithm known as “Gradient Descent”, which helps to optimize the task
quickly and efficiently.
The one round of forwarding and backpropagation iteration is known as one training iteration aka
“Epoch“. Most applications of feedforward networks and backprop require several epochs before errors
are reasonably small.

6
 1. Activation Functions
Activation function is one of the building blocks of Neural Network. Activation function decides,
whether a neuron should be activated or not by calculating weighted sum and further adding bias with
it. The purpose of the activation function is to introduce non-linearity into the output of a neuron. A
neural network without an activation function is essentially just a linear regression model.
Popular types of activation functions are:
∙ Binary Step Function
∙ Linear Function
∙ Sigmoid
∙ Tanh
∙ Rectified Linear Unit (ReLU)
∙ Leaky ReLU
∙ Parameterised ReLU
∙ ELU
Softmax

7
Refer to the following links and the lab session to understand activation functions in
details:
∙ https://www.analyticsvidhya.com/blog/2020/01/fundamentals-deep-learning-act
ivation-functions-when-to-use-them/
∙ https://medium.com/the-theory-of-everything/understanding-activation-function
s-in-neural-networks-9491262884e0
https://towardsdatascience.com/activation-functions-neural-networks-1cbd9f8d91d6

8
 1. ANN Use Cases
ANNs are designed to spot patterns in the data. This makes ANNs an optimal solution for:
∙ classification (sorting data into predetermined categories)
∙ clustering (finding similar characteristics among data and pulling that data together into
categories)
∙ making predictions from data (such as helping determine infection rates for COVID, the next
catastrophic weather event or box-office smash).

1. Classification Evaluation Metrics

Evaluating a model is a major part of building an effective machine
learning model. The most frequent classification evaluation metric that
we use should be ‘Accuracy’. When building and optimizing your
classification model, measuring how accurately it predicts your
expected outcome is crucial. However, this metric alone is never the
entire story, as it can still offer misleading results. You might believe
that the model is good when the accuracy rate is 99%! However, it is
not always true. This will be discussed later. For this lab, you will
calculate the accuracy.