ML Module 5: 30 Marks

Q.1 Explain neural networks architecture.

1. Single-layer feed-forward network
2. Multilayer feed-forward network
3. Single node with its own feedback
4. Single-layer recurrent network
5. Multilayer recurrent network
1. Single-layer feed-forward network

In this type of network, we have only two layers input layer and the output layer but the input
layer does not count because no computation is performed in this layer. The output layer is
formed when different weights are applied to input nodes and the cumulative effect per node is
taken. After this, the neurons collectively give the output layer to compute the output signals.
2. Multilayer feed-forward network

This layer also has a hidden layer that is internal to the network and has no direct contact with
the external layer. The existence of one or more hidden layers enables the network to be
computationally stronger, a feed-forward network because of information flow through the input
function, and the intermediate computations used to determine the output Z. There are no
feedback connections in which outputs of the model are fed back into itself.
3. Single node with its own feedback

When outputs can be directed back as inputs to the same layer or preceding layer nodes, then it
results in feedback networks. Recurrent networks are feedback networks with closed loops. The
above figure shows a single recurrent network having a single neuron with feedback to itself.
4. Single-layer recurrent network
The above network is a single-layer network with a feedback connection in which the processing
element’s output can be directed back to itself or to another processing element or both. A
recurrent neural network is a class of artificial neural networks where connections between nodes
form a directed graph along a sequence. This allows it to exhibit dynamic temporal behavior for
a time sequence. Unlike feedforward neural networks, RNNs can use their internal state
(memory) to process sequences of inputs.
5. Multilayer recurrent network

In this type of network, processing element output can be directed to the processing element in
the same layer and in the preceding layer forming a multilayer recurrent network. They perform
the same task for every element of a sequence, with the output being dependent on the previous
computations. Inputs are not needed at each time step. The main feature of a Recurrent Neural
Network is its hidden state, which captures some information about a sequence.
Q.2 Justify Perceptron works only for linear separable problems.
The Perceptron algorithm is like drawing a straight line to separate two different groups of
things. Imagine you have red and blue balls scattered on a table, and you want to draw a line that
separates them. But here's the catch: the line can only be straight, no curves allowed.
1. Straight Line Rule: The Perceptron algorithm follows this rule strictly—it can only draw
straight lines to divide things. So, if the balls are arranged in a way that you can't draw a straight
line to separate all the red ones from the blue ones, the Perceptron algorithm can't do its job
properly.
2. Fixing Mistakes: The algorithm tries to draw the line in the best way possible, but it's not
perfect. It might make mistakes, like drawing the line in a way that some red balls end up on the
blue side, and vice versa. It keeps adjusting the line to fix these mistakes, but there's a limit to
how much it can adjust.
3. Never-ending Mistakes: If the balls are arranged in a way that no matter where you draw a
straight line, there are always some red balls on the blue side or vice versa, then the Perceptron
algorithm keeps trying to fix the mistakes forever. It never reaches a point where it's satisfied
with how the line separates the balls.
So, in simple terms, the Perceptron algorithm only works well when you can draw a straight line
to neatly separate different groups of things. If things are mixed up in a way that a straight line
can't do the job, the Perceptron algorithm struggles and might never be happy with the result.
Q.3 Explain Backpropagation algorithm
 Backpropagation is an effective algorithm used to train artificial neural networks,
especially in feed-forward neural networks.
 It is an iterative algorithm that helps to minimize the cost function by determining
which weights and biases should be adjusted.
 During every epoch, the model learns by adapting the weights and biases to minimize
the loss by moving down toward the gradient of the error¹.
 It involves the two most popular optimization algorithms, such as gradient descent or
stochastic gradient descent.
 Computing the gradient in the backpropagation algorithm helps to minimize the cost
function and it can be implemented by using the mathematical rule called chain rule
from calculus to navigate through complex layers of the neural network.

Working of Backpropagation Algorithm:

 The backpropagation algorithm works by computing the gradient of the loss function
with respect to each weight via the chain rule.
 It computes the gradient one layer at a time, iterating backward from the last layer to
avoid redundant computation of intermediate terms in the chain rule.
  The algorithm starts with the forward pass, where the input is passed through the
network to generate the output.
 Then, the error is calculated by comparing the generated output with the actual
output.
 In the backward pass, the error is propagated back into the network which provides a
measure of how much each neuron in the last layer contributed to the error.
 This process is repeated for each layer in the network.
 The weights are then updated in a manner that minimizes the error.

Advantages:
 Ease of Implementation: Backpropagation does not require prior knowledge of neural
networks, making it accessible to beginners.
 Simplicity and Flexibility: The algorithm’s simplicity allows it to be applied to a wide
range of problems and network architectures.
 Efficiency: Backpropagation accelerates the learning process by directly updating
weights based on the calculated error derivatives.