ASSIGNMENT 1

Name- Md. Faisal Nadeem

Reg No-2024GE10
Geotechnical Engg Mtech 1st Yr
Soft Computing Methods in Engineering Problem Solving

Ques 1: Define an Artificial Neural Network and explain its basic components.
Ans: An Artificial Neural Network (ANN) is a computational model inspired by the structure and functioning of
biological neural networks, such as those in the human brain. ANNs are used to recognize patterns, classify data, and
make predictions. They consist of interconnected nodes (or neurons) organized into layers, which work together to
process input data and produce an output.

Basic Components of an Artificial Neural Network:

1. Neurons (Nodes):
o Neurons are the basic units of an ANN. Each neuron receives input, processes it, and passes
the result to the next layer.
o A neuron typically performs two functions:
 It calculates a weighted sum of its inputs.
 It applies an activation function to this sum to introduce non-linearity.
2. Layers: ANNs consist of multiple layers of neurons. These layers determine how the network
processes data.
o Input Layer:
 The first layer that receives raw input data (features) and passes it on to the next layer.
Each neuron in this layer corresponds to a feature in the input data.
o Hidden Layers:
 These are intermediate layers between the input and output layers. They transform the
input into something that the output layer can use. The complexity of the neural network
increases with more hidden layers and neurons.
o Output Layer:
 The final layer produces the network’s output, which could be a prediction (like a
classification label or a regression value). Each neuron in this layer corresponds to a
possible output.
3. Weights:
o Weights determine the strength of the connection between two neurons. When data passes
through the network, it gets multiplied by the weights at each connection. The ANN adjusts
these weights during training to improve its predictions.
4. Bias:
 o Bias is an additional parameter added to the weighted sum in each neuron, allowing the model
to fit the data better by shifting the activation function. It helps the network model relationships
in the data more effectively.
5. Activation Function:
o After calculating the weighted sum, an activation function is applied to introduce non-linearity
to the network. Common activation functions include:
 Sigmoid: Maps values to a range between 0 and 1.
 ReLU (Rectified Linear Unit): Replaces negative values with zero.
 Tanh: Maps values to a range between -1 and 1.
 Softmax: Often used in classification tasks, providing a probability distribution over
multiple output classes.

Ques 2: How does an artificial neural network model the brain?

Ans: An Artificial Neural Network (ANN) models the brain by mimicking the way biological neurons communicate
and process information. In the brain, neurons are connected in a vast network where each neuron receives signals from
others, processes them, and passes on the information. Similarly, an ANN consists of artificial neurons (or nodes)
arranged in layers, with each neuron receiving input from the previous layer, performing calculations, and transmitting
the result to the next layer.

In both systems, the strength of connections between neurons is crucial. In the brain, this strength is controlled
by synapses, which determine how strongly one neuron influences another. In ANNs, this is modeled by
weights assigned to connections between nodes. Just as the brain strengthens or weakens synapses through
learning, ANNs adjust these weights during training, using algorithms like backpropagation to minimize
error and improve performance.

Ques 3: What are the limitations of using a perceptron as a model of biological neurons?
Why is the perceptron only capable of learning linearly separable functions?
Ans: The perceptron is a simple model of a biological neuron, but it has several limitations when compared to the
complexity of actual neural activity. Biological neurons can process a vast range of signals and exhibit complex
behaviors, such as handling non-linear relationships, memory retention, and plasticity, whereas a perceptron only outputs
a binary result based on a linear combination of its inputs. Biological neurons also interact in highly dynamic and
adaptive ways, influenced by the entire brain's network, whereas the perceptron’s structure and learning process are
rigid and simplistic, lacking the capacity to handle nuanced patterns or continuously adapt beyond a basic set of input-
output transformations.

The perceptron is only capable of learning linearly separable functions because of its linear decision
boundary. The perceptron computes a weighted sum of its inputs and applies a step function (or threshold
activation), which can only divide the input space with a straight line (or hyperplane in higher dimensions).
This makes it effective for problems like AND or OR, but it cannot solve non-linearly separable problems like
XOR, where the decision boundary is not a simple straight line. To solve more complex tasks, neural networks
with multiple layers (multilayer perceptrons) and non-linear activation functions are required, as they can
create non-linear decision boundaries.

Ques 4: What is the difference between autoassociative and heteroassociative memory?

Ans: The difference between autoassociative and heteroassociative memory lies in the type of associations they
form:
  Autoassociative memory retrieves an entire pattern from a partial or noisy version of the same pattern.
It stores and recalls patterns by associating them with themselves. An example is reconstructing a
complete image from a distorted version.
 Heteroassociative memory associates two different patterns, meaning it stores a relationship between
an input and a distinct output. For example, it might recall a word ("cat") when given an image of a
cat.
In short, autoassociative memory links a pattern to itself, while heteroassociative memory links different
patterns.

Ques 5: What are the key differences between Hebbian learning and competitive
learning?
Ans: Hebbian learning and competitive learning are both unsupervised learning mechanisms, but they operate
based on different principles. Hebbian learning follows the idea that "neurons that fire together, wire
together." This means that when two neurons are activated simultaneously, the synaptic connection between
them is strengthened. Hebbian learning focuses on reinforcing correlations between neuron activations,
making it effective for tasks like pattern recognition and associative memory, where similar inputs need to
strengthen their connections over time.
On the other hand, competitive learning introduces competition among neurons, where only the most
responsive neuron (or a small set of neurons) "wins" and gets its weights updated. This process, often called
a winner-take-all strategy, encourages neurons to specialize and respond to distinct patterns or inputs. Instead
of reinforcing co-activation like Hebbian learning, competitive learning reduces redundancy by ensuring that
different neurons are trained to handle different input patterns. This makes it useful for tasks like clustering,
where the goal is to allocate different neurons to represent different categories or regions of the input space.

Ques 6: Explain the two major classes of learning paradigms: supervised learning and
unsupervised (selforganized) learning. What are the key differences that distinguish
these two learning paradigms?
Ans: The two major classes of learning paradigms are supervised learning and unsupervised (self-organized)
learning, and they differ primarily in the nature of the data and the learning process.

1. Supervised Learning:
 In supervised learning, the model is trained on a dataset that includes both inputs and corresponding
labeled outputs (target values). The algorithm learns by comparing its predictions to the actual output
and adjusting its parameters to minimize the error.
 Examples: Classification (e.g., identifying if an email is spam) and regression (e.g., predicting house
prices).
 Key Process: The algorithm uses feedback (labeled data) to learn the mapping from input to output.
 Goal: Minimize the difference between predicted and actual outputs (often using a loss function like
mean squared error or cross-entropy).
2. Unsupervised (Self-Organized) Learning:
 In unsupervised learning, the model is provided with input data without labeled outputs. The goal
is to discover hidden patterns or structures in the data. The algorithm organizes the data by grouping
similar inputs together or identifying relationships, but there’s no explicit feedback from correct
outputs.
  Examples: Clustering (e.g., grouping similar customers in marketing), dimensionality reduction,
anomaly detection.
 Key Process: The algorithm explores patterns in the input data, relying on intrinsic data structures
rather than feedback from labeled examples.
 Goal: Organize or categorize the data in a meaningful way without prior knowledge of the outputs.
Key Differences:
 Labeled vs. Unlabeled Data: Supervised learning relies on labeled data (input-output pairs), while
unsupervised learning works with only input data without predefined labels.
 Learning Objective: Supervised learning aims to make accurate predictions by learning a mapping
from inputs to outputs, while unsupervised learning focuses on uncovering the underlying structure or
relationships within the data.
 Feedback: Supervised learning uses explicit feedback to correct errors, whereas unsupervised learning
does not have this feedback, relying on the inherent structure of the input data for learning.

Ques 7: Explain the structure of a single-layer perceptron and how it makes decisions.
Include a discussion on the activation function.
Ans: A single-layer perceptron is the simplest form of an artificial neural network, consisting of an input layer and
an output layer, with no hidden layers. The input layer receives data, and each input is assigned a weight, representing
the strength of its contribution to the output. The perceptron computes a weighted sum of the inputs and adds a bias
term to shift the output. This sum is then passed through an activation function, typically a step function or a threshold
function, which determines the perceptron’s output. The step function outputs a binary result, either 0 or 1 (or -1 and 1
in some variations), depending on whether the weighted sum exceeds a certain threshold.

In decision-making, the perceptron uses this process to classify inputs into two categories. For instance, in a
binary classification task, the perceptron assigns inputs to one of two classes based on whether the weighted
sum meets the activation threshold. However, because the perceptron uses a linear activation function, it can
only solve problems where the data is linearly separable—i.e., where a straight line (or hyperplane) can
clearly divide the input space into two distinct classes. This limits its ability to handle more complex, non-
linear problems, which require deeper or more complex network structures

Ques 8: Describe the backpropagation algorithm used for training ANNs. How does it
minimize the error in predictions?
Ans: The backpropagation algorithm is a key method used to train artificial neural networks (ANNs) by minimizing
the error between the network’s predictions and the actual target values. It works by iteratively adjusting the weights of
the network to reduce the error, following a process of gradient descent. The algorithm begins with a forward pass,
where input data is fed through the network to generate predictions. The difference between these predictions and the
actual target values is calculated using a loss function (e.g., mean squared error for regression tasks or cross-entropy
for classification).

In the backward pass, the algorithm computes the gradient of the loss function with respect to each weight in
the network using the chain rule of calculus. This involves calculating how much each weight contributed to
the error by propagating the error backward, layer by layer, starting from the output layer and moving towards
the input. The weights are then updated in the direction opposite to the gradient (hence, "gradient descent") to
reduce the error. The learning rate controls how large each weight adjustment is, ensuring that the changes are
gradual and stable.
By repeating this process over many iterations (or epochs), the backpropagation algorithm allows the network
to "learn" by continuously adjusting the weights to minimize the prediction error, improving its accuracy over
time.

Ques 9: Discuss the significance of a loss function in ANN training. Provide examples of
commonly used loss functions.
Ans: The loss function plays a crucial role in training artificial neural networks (ANNs) as it quantifies the difference
between the predicted outputs of the network and the actual target values. Its primary significance lies in guiding the
optimization process during training; the loss function provides a measurable objective that the training algorithm seeks
to minimize. By evaluating how well the network performs, the loss function enables adjustments to the model's weights
through techniques like backpropagation, effectively steering the learning process toward better predictions. A well-
defined loss function not only helps in achieving accurate models but also influences the stability and convergence of
the training process.

Commonly used loss functions vary based on the type of problem being addressed. For regression tasks, the
Mean Squared Error (MSE) is frequently used, calculating the average of the squares of the errors between
predicted and actual values. This function penalizes larger errors more significantly, encouraging the model
to make more precise predictions. In classification tasks, particularly for binary classification, the Binary
Cross-Entropy Loss is often employed, measuring the dissimilarity between the predicted probabilities and
the actual binary outcomes. For multi-class classification, the Categorical Cross-Entropy Loss is utilized,
extending the concept to multiple classes. Each of these loss functions serves to reflect the specific nature of
the task at hand, thus enabling effective training of ANNs to perform well across various applications.

Ques 10: Explain the role of activation functions in ANNs. Provide examples of common
activation functions and their characteristics.
Ans: Ac va on func ons play a vital role in ar ficial neural networks (ANNs) by introducing non-linearity into the
model, allowing the network to learn complex pa erns and rela onships in the data. Without ac va on func ons, the
output of a neuron would simply be a linear combina on of its inputs, severely limi ng the network's ability to solve
non-linear problems. By applying non-linear ac va on func ons, ANNs can approximate a wide range of func ons and
learn intricate data distribu ons.

Common activation functions include:

1. Sigmoid: Maps input values to a range between 0 and 1. It is particularly useful for binary classification
tasks but can suffer from the vanishing gradient problem, where gradients become very small for
extreme input values, hindering effective learning.
2. Hyperbolic Tangent (tanh): Similar to the sigmoid but outputs values in the range of -1 to 1. This can
lead to better convergence during training compared to the sigmoid function, but it still faces vanishing
gradient issues.
3. ReLU (Rectified Linear Unit): Outputs the input directly if it is positive; otherwise, it returns zero.
ReLU is widely used in hidden layers due to its simplicity and ability to mitigate the vanishing gradient
problem, but it can suffer from the dying ReLU problem, where neurons can become inactive and
stop learning.
4. Leaky ReLU: A variation of ReLU that allows a small, non-zero gradient when the input is negative,
helping to address the dying ReLU problem.
5. Softmax: Typically used in the output layer for multi-class classification, it converts the raw output
scores into probabilities, ensuring that the sum of the outputs equals one.
Each activation function has its characteristics, advantages, and drawbacks, influencing the training dynamics
and performance of ANNs across different tasks. Selecting the appropriate activation function is crucial for
achieving optimal learning and convergence in neural networks.

Ques 11: Discuss strategies to prevent overfitting in ANNs, including regularization

techniques.
Ans: Overfitting occurs when a model learns the training data too well, resulting in poor generalization to
unseen data. Strategies to prevent overfitting include:
Regularization Techniques:
 L1 and L2 Regularization: Add a penalty to the loss function based on the size of the weights,
discouraging overly complex models.
 Dropout: Randomly drops a fraction of neurons during training, preventing co-adaptation of neurons
and promoting redundancy.
 Early Stopping: Monitor validation loss during training and stop when it starts to increase, indicating
overfitting.
 Data Augmentation: Increase the diversity of training data through transformations (e.g., rotations,
translations) to help the model generalize better.

Ques 12: Discuss how ANNs can be applied to image recognition tasks, including a brief
overview of the steps for image processing.
Ans: Artificial Neural Networks (ANNs), particularly Convolutional Neural Networks (CNNs), have
become the backbone of modern image recognition tasks due to their ability to learn spatial hierarchies of
features from images. The application of ANNs to image recognition typically involves several key steps in
the image processing pipeline. First, images are preprocessed, which may include resizing to a consistent
size, normalizing pixel values to a specific range (often between 0 and 1), and augmenting the dataset
through techniques like rotation, flipping, or cropping to enhance model robustness.
Once preprocessed, images are fed into the CNN, which consists of multiple layers that automatically learn
to extract features from the images. Convolutional layers apply filters to detect features such as edges,
textures, and patterns. These layers are often followed by activation functions (like ReLU) and pooling
layers, which reduce the dimensionality of the feature maps while retaining essential information, thus
allowing the network to focus on the most relevant features. After several convolutional and pooling layers,
the high-level features are flattened and passed through fully connected layers, where final classifications
are made.
During training, the network learns to minimize the error in its predictions using a loss function and
backpropagation. After training, the model can effectively classify new, unseen images by identifying the
learned patterns. This stepwise approach enables ANNs to achieve high accuracy in various image
recognition tasks, such as facial recognition, object detection, and medical imaging analysis.

Ques 13: Describe the architecture of CNN and discuss their applications, particularly
in image processing and computer vision.?
Ans: Convolutional Neural Networks (CNNs) are a specialized type of artificial neural network designed
primarily for processing grid-like data, such as images. The architecture of a CNN typically consists of
several layers, including convolutional layers, activation functions, pooling layers, and fully connected
layers. In the convolutional layers, small filters (or kernels) slide over the input image to detect local
features like edges and textures by performing convolution operations. Each filter extracts different features,
which are then passed through non-linear activation functions, such as ReLU, to introduce non-linearity into
the model.
Following the convolutional layers, pooling layers are employed to downsample the feature maps, reducing
dimensionality and computational complexity while preserving important information. This process helps
the network become more invariant to translations and distortions in the input images. After several
convolutional and pooling layers, the high-level feature maps are flattened and connected to one or more
fully connected layers, which serve to make final classifications or predictions based on the learned features.
CNNs have found extensive applications in image processing and computer vision, including tasks such as
image classification (e.g., recognizing objects in photos), object detection (locating objects within images),
semantic segmentation (classifying each pixel in an image), and facial recognition. Their hierarchical feature
extraction capabilities make them particularly well-suited for handling the complexity and variability
inherent in visual data, leading to significant advancements in fields such as autonomous driving, medical
imaging analysis, and augmented reality. The ability of CNNs to learn directly from raw pixel data has
revolutionized image-related tasks, achieving state-of-the-art performance in many applications.

Ques 14: Explain the structure and unique properties of RNNs. How do they differ
from traditional feedforward neural networks?
Ans: Recurrent Neural Networks (RNNs) are a class of neural networks specifically designed for
processing sequential data, such as time series, text, or speech. The unique structure of RNNs includes
cycles or loops in the network, allowing them to maintain a hidden state that captures information from
previous time steps in the sequence. This ability to retain context enables RNNs to model dependencies
across different points in the sequence, making them well-suited for tasks where the order of input matters,
such as language modeling, machine translation, and speech recognition.
One of the key properties of RNNs is their temporal dynamics, which means they can take into account the
sequential nature of data and update their internal states based on both current inputs and previous hidden
states. This contrasts sharply with traditional feedforward neural networks, where information flows in one
direction—from input to output—without any feedback loops. Consequently, feedforward networks are
limited to fixed-size inputs and outputs and cannot effectively handle sequential or temporal data. While
RNNs are powerful in learning patterns over time, they can face challenges like the vanishing and exploding
gradient problems, which can hinder training over long sequences. To address these issues, more advanced
architectures like Long Short-Term Memory (LSTM) networks and Gated Recurrent Units (GRUs) have
been developed, incorporating mechanisms to better manage memory and context in sequential learning
tasks.
Ques 15: Provide examples of popular neural network simulators and discuss their key features and
applications.
Ans: Several popular neural network simulators are widely used for research, education, and practical
applications, each offering unique features tailored to different needs. TensorFlow, developed by Google, is
one of the most popular frameworks for building and training neural networks. It provides a comprehensive
ecosystem that includes high-level APIs (like Keras) for rapid prototyping, as well as lower-level APIs for
more complex model customization. TensorFlow is highly scalable and supports distributed computing,
making it suitable for large-scale machine learning tasks, including image recognition, natural language
processing, and reinforcement learning.
Another widely used simulator is PyTorch, favored for its dynamic computation graph, which allows for
more intuitive model building and debugging. PyTorch’s flexibility makes it particularly appealing for
researchers who need to experiment with new architectures and algorithms. It has robust support for GPU
acceleration and offers a rich library of pre-trained models, facilitating rapid development in areas such as
computer vision and deep learning.
Keras, initially an independent high-level API, is now integrated with TensorFlow, providing a user-friendly
interface for building neural networks. Keras is particularly suited for beginners due to its simplicity and
ease of use while still supporting complex neural network architectures.
Caffe, developed by the Berkeley Vision and Learning Center (BVLC), is known for its efficiency and
speed, especially in image processing tasks. It is optimized for image classification, segmentation, and
convolutional neural networks, making it popular in computer vision research.
Lastly, MXNet, with its support for both symbolic and imperative programming, is designed for efficiency
and flexibility, making it well-suited for training large-scale deep learning models, particularly in the context
of cloud computing.
Each of these simulators is tailored to specific use cases, from academic research and experimentation to
industrial applications in computer vision, natural language processing, and beyond, providing developers
and researchers with the tools they need to create sophisticated neural network models.
.

Ques 16: Consider a neural network with a single hidden layer, as described below:
Input Layer: 2 neurons (x1, x2)
Hidden Layer: 2 neurons (h1, h2) with a sigmoid activation function
Output Layer: 1 neuron (o1) with a sigmoid activation function
Loss Function: Mean Squared Error
The initial weights and biases are given as follows: w1 (from x1 to h1): 0.15, w2 (from
x2 to h1): 0.20,
w3 (from x1 to h2): 0.25, w4 (from x2 to h2): 0.30, w5 (from h1 to o1): 0.40, w6 (from h2
to o1): 0.50,
b1 (bias to h1): 0.35, b2 (bias to h2): 0.35, b3 (bias to o1): 0.60. Given an input (x1, x2) =
(0.05, 0.10)
and a target output of 0.01.
Perform two iterations of backpropagation to update the weights. Use a learning rate
(η) of 0.5.
Ans: Problem Description
The problem involves performing two iterations of backpropagation on a neural network with the following
characteristics:

Input Layer: 2 neurons (x1, x2)

Hidden Layer: 2 neurons (h1, h2) with sigmoid activation functions
Output Layer: 1 neuron (o1) with a sigmoid activation function
Loss Function: Mean Squared Error (MSE)
Learning Rate (η): 0.5
Initial Weights and Biases:
- w1 (from x1 to h1): 0.15
- w2 (from x2 to h1): 0.20
 - w3 (from x1 to h2): 0.25
- w4 (from x2 to h2): 0.30
- w5 (from h1 to o1): 0.40
- w6 (from h2 to o1): 0.50
- Biases:
- b1 (to h1): 0.35
- b2 (to h2): 0.35
- b3 (to o1): 0.60

Input Values: (x1, x2) = (0.05, 0.10)

Target Output: 0.01

1. Step-by-Step Solution
Step 1: Forward Pass
1. Calculate the weighted sum for each neuron in the hidden layer:
For h1:
z_h1 = (x1 * w1) + (x2 * w2) + b1
= (0.05 * 0.15) + (0.10 * 0.20) + 0.35
= 0.0075 + 0.02 + 0.35
= 0.3775
For h2:
z_h2 = (x1 * w3) + (x2 * w4) + b2
= (0.05 * 0.25) + (0.10 * 0.30) + 0.35
= 0.0125 + 0.03 + 0.35
= 0.3925

2. Apply the sigmoid activation function to the weighted sums:

Sigmoid function: σ(z) = 1 / (1 + e^(-z))
For h1:
a_h1 = σ(0.3775) = 1 / (1 + e^(-0.3775)) ≈ 0.593
For h2:
a_h2 = σ(0.3925) = 1 / (1 + e^(-0.3925)) ≈ 0.596

Step 2: Output Neuron Calculation

1. Calculate the weighted sum for the output neuron:
z_o1 = (a_h1 * w5) + (a_h2 * w6) + b3
= (0.593 * 0.40) + (0.596 * 0.50) + 0.60
= 0.2372 + 0.298 + 0.60
= 1.1352
2. Apply the sigmoid activation function:
a_o1 = σ(1.1352) = 1 / (1 + e^(-1.1352)) ≈ 0.757
3. Calculate the error (using Mean Squared Error):
Error = 0.5 * (target - a_o1)^2
= 0.5 * (0.01 - 0.757)^2 ≈ 0.278

Step 3: Backpropagation (Output Layer)

1. Calculate the output error (delta) for the output neuron o1:
δ_o1 = (a_o1 - target) * a_o1 * (1 - a_o1)
 = (0.757 - 0.01) * 0.757 * (1 - 0.757)
= 0.747 * 0.757 * 0.243
≈ 0.136

2. Update the weights from the hidden layer to the output layer (w5, w6):
Δw5 = -η * δ_o1 * a_h1
= -0.5 * 0.136 * 0.593
≈ -0.0404
Updated w5 = 0.40 + Δw5 ≈ 0.3596

Δw6 = -η * δ_o1 * a_h2

= -0.5 * 0.136 * 0.596
≈ -0.0405
Updated w6 = 0.50 + Δw6 ≈ 0.4595

3. Update the bias for the output layer (b3):

Δb3 = -η * δ_o1
= -0.5 * 0.136
≈ -0.068
Updated b3 = 0.60 + Δb3 ≈ 0.532

Step 4: Backpropagation (Hidden Layer)

1. Calculate the error (delta) for each hidden neuron:
For h1:
δ_h1 = δ_o1 * w5 * a_h1 * (1 - a_h1)
= 0.136 * 0.40 * 0.593 * (1 - 0.593)
≈ 0.0138

For h2:
δ_h2 = δ_o1 * w6 * a_h2 * (1 - a_h2)
= 0.136 * 0.50 * 0.596 * (1 - 0.596)
≈ 0.0164

2. Update the weights from the input layer to the hidden layer (w1, w2, w3, w4):
Δw1 = -η * δ_h1 * x1
= -0.5 * 0.0138 * 0.05
≈ -0.000345
Updated w1 = 0.15 + Δw1 ≈ 0.1497

Δw2 = -η * δ_h1 * x2
= -0.5 * 0.0138 * 0.10
≈ -0.00069
Updated w2 = 0.20 + Δw2 ≈ 0.1993

Δw3 = -η * δ_h2 * x1
= -0.5 * 0.0164 * 0.05
≈ -0.00041
Updated w3 = 0.25 + Δw3 ≈ 0.2496

Δw4 = -η * δ_h2 * x2
 = -0.5 * 0.0164 * 0.10
≈ -0.00082
Updated w4 = 0.30 + Δw4 ≈ 0.2992

Step 5: Update Bias for Hidden Layer

1. Update the biases for the hidden neurons:
Δb1 = -η * δ_h1
= -0.5 * 0.0138
≈ -0.0069
Updated b1 = 0.35 + Δb1 ≈ 0.3431

Δb2 = -η * δ_h2
= -0.5 * 0.0164
≈ -0.0082
Updated b2 = 0.35 + Δb2 ≈ 0.3418

Step 6: Repeat for Second Iteration

After completing the first iteration, repeating the same process for the second iteration, using the updated
weights and biases from the first iteration.

2. Second Iteration
Step 1: Forward Pass (Second Iteration)
1. Calculate the weighted sum for each neuron in the hidden layer using updated weights and biases from the
first iteration:
For h1:
z_h1 = (x1 * w1) + (x2 * w2) + b1
= (0.05 * 0.1497) + (0.10 * 0.1993) + 0.3431
= 0.007485 + 0.01993 + 0.3431
= 0.370515
For h2:
z_h2 = (x1 * w3) + (x2 * w4) + b2
= (0.05 * 0.2496) + (0.10 * 0.2992) + 0.3418
= 0.01248 + 0.02992 + 0.3418
= 0.3842

2. Apply the sigmoid activation function to the weighted sums:

For h1:
a_h1 = σ(0.370515) = 1 / (1 + e^(-0.370515)) ≈ 0.5915
For h2:
a_h2 = σ(0.3842) = 1 / (1 + e^(-0.3842)) ≈ 0.5948

Step 2: Output Neuron Calculation (Second Iteration)

1. Calculate the weighted sum for the output neuron using updated weights:
z_o1 = (a_h1 * w5) + (a_h2 * w6) + b3
= (0.5915 * 0.3596) + (0.5948 * 0.4595) + 0.532
 = 0.2126 + 0.2733 + 0.532
= 1.0179

2. Apply the sigmoid activation function:

a_o1 = σ(1.0179) = 1 / (1 + e^(-1.0179)) ≈ 0.7346

3. Calculate the error (using Mean Squared Error):

Error = 0.5 * (target - a_o1)^2
= 0.5 * (0.01 - 0.7346)^2 ≈ 0.261

Step 3: Backpropagation (Output Layer - Second Iteration)

1. Calculate the output error (delta) for the output neuron o1:
δ_o1 = (a_o1 - target) * a_o1 * (1 - a_o1)
= (0.7346 - 0.01) * 0.7346 * (1 - 0.7346)
= 0.7246 * 0.7346 * 0.2654
≈ 0.1412

2. Update the weights from the hidden layer to the output layer (w5, w6):
Δw5 = -η * δ_o1 * a_h1
= -0.5 * 0.1412 * 0.5915
≈ -0.0417
Updated w5 = 0.3596 + Δw5 ≈ 0.3179

Δw6 = -η * δ_o1 * a_h2

= -0.5 * 0.1412 * 0.5948
≈ -0.0420
Updated w6 = 0.4595 + Δw6 ≈ 0.4175

3. Update the bias for the output layer (b3):

Δb3 = -η * δ_o1
= -0.5 * 0.1412
≈ -0.0706
Updated b3 = 0.532 + Δb3 ≈ 0.4614

Step 4: Backpropagation (Hidden Layer - Second Iteration)

1. Calculate the error (delta) for each hidden neuron:
For h1:
δ_h1 = δ_o1 * w5 * a_h1 * (1 - a_h1)
= 0.1412 * 0.3179 * 0.5915 * (1 - 0.5915)
≈ 0.0153

For h2:
δ_h2 = δ_o1 * w6 * a_h2 * (1 - a_h2)
= 0.1412 * 0.4175 * 0.5948 * (1 - 0.5948)
≈ 0.0170

2. Update the weights from the input layer to the hidden layer (w1, w2, w3, w4):
Δw1 = -η * δ_h1 * x1
= -0.5 * 0.0153 * 0.05
 ≈ -0.000383
Updated w1 = 0.1497 + Δw1 ≈ 0.1493

Δw2 = -η * δ_h1 * x2
= -0.5 * 0.0153 * 0.10
≈ -0.000765
Updated w2 = 0.1993 + Δw2 ≈ 0.1985

Δw3 = -η * δ_h2 * x1
= -0.5 * 0.0170 * 0.05
≈ -0.000425
Updated w3 = 0.2496 + Δw3 ≈ 0.2492

Δw4 = -η * δ_h2 * x2
= -0.5 * 0.0170 * 0.10
≈ -0.000850
Updated w4 = 0.2992 + Δw4 ≈ 0.2984

Step 5: Update Bias for Hidden Layer (Second Iteration)

1. Update the biases for the hidden neurons:
Δb1 = -η * δ_h1
= -0.5 * 0.0153
≈ -0.00765
Updated b1 = 0.3431 + Δb1 ≈ 0.3354

Δb2 = -η * δ_h2
= -0.5 * 0.0170
≈ -0.0085
Updated b2 = 0.3418 + Δb2 ≈ 0.3333