PATTERN RECOGNITION

ECECE-26

Submitted by:- Uddhav Mittal

Submitted to:- Mr. Amit Singhal
Roll No:- 2022UEC2646
 INDEX

Exp No. Experiment Name Remarks

1 Gaussian function evaluation. Write a
MATLAB/Python function that computes the
value of the Gaussian distribution N (m, S), at a
given vector x. Hence, plot the effect of
varying mean and variance to the normal
distributions.
2 Write a MATLAB/Python function that will take as
inputs: (a) the mean vectors, (b) the covariance
matrices of the class distributions of a c-class problem,
(c) the a priori probabilities of the c classes, and (d) a
matrix X containing column vectors that stem from the
above classes. It will give as output an Ndimensional
vector whose ith component contains the class where
the corresponding vector is assigned, according to the
Bayesian classification rule.
3 Write a MATLAB/Python function that will take as
inputs: (a) the mean vectors, and (b) a matrix X
containing column vectors that stem from the above
classes. It will give as output an N-dimensional vector
whose ith component contains the class where the
corresponding vector is assigned, according to the
minimum Euclidean distance classifier.

4 Write a MATLAB/Python function that will take as

inputs: (a) the mean vectors,(b) the covariance matrix
of the class distributions of a c-class problem, and (c) a
matrix X containing column vectors that stem from the
above classes. It will give as output an N-dimensional
vector whose ith component contains the class where
the corresponding vector is assigned according to the
minimum Mahalanobis distance classifier
5 Write a MATLAB/Python function that takes as
inputs: (a) a set of N1 vectors packed as columns of a
matrix Z,(b) an N1-dimensional vector containing the
classes where each vector in Z belongs, (c) the value
for the parameter k of the classifier,(d) a set of N
vectors packed as columns in the matrix X. It returns
an dimensional vector whose ith component contains
the class where the correspondin vector of X is
assigned, according to the k-nearest neighbor
classifier.
6 Write a MATLAB/Python function that will take as
inputs: (a) an N-dimensional vector, each component
of which contains the class where the corresponding
data vector belongs and (b) a similar N- dimensional
vector each component of which contains
the class where the corresponding data
vector is assigned from a certain classifier. Its output
will be the percentage of the places where the two
vectors differ (i.e., the classification error of the
classifier).
7 Write a MATLAB/Python function for the perceptron
algorithm. This will take as inputs: (a) a matrix X
containing N ldimensional column vectors, ( b) an N -
dimensional row vector y, whose ith
component contains the class (-1 or +1)
where the corresponding vector belongs, and (c) an
initial value vector wini for the parameter vector. It
returns the estimated parameter vector.
8 Design a three-layer FFN, using gradient descent to
perform the {x1, x2} → {y1, y2} mapping. The
activation function for all the nodes is the hyperbolic
tangent one. For training, one may select one of the
following algorithms: a) the standard gradient descent
backpropagation algorithm, ( b) the back propagation
algorithm with momentum , and (c) the
backpropagation algorithm with adaptive learning rate
9 Write MATLAB/Python function to compute the
principal components of the covariance matrix of an l
× N dimensional data matrix X as well as the
corresponding variances. Hence, write a
MATLAB/Python function that
evaluates the performance of the PCA
method when applied on a data matrix
 Experiment – 1
AIM : Gaussian function evaluation. Write a MATLAB/Python function that computes the
value of the Gaussian distribution N (m, S), at a given vector x. Hence, plot the effect of varying
mean and variance to the normal distributions.

Software Required : Jupyter Notebook

Theory

The Gaussian distribution is characterized by two parameters: the mean (μ) and the standard
deviation (σ). The mean is the central tendency of the distribution, while the standard
deviation measures the spread of the distribution. The probability density function (PDF) of
the Gaussian distribution is given by:

f(x) = (1 / (σ√(2π))) * e^(-((x-μ)^2 / (2σ^2)))

where x is a random variable, μ is the mean, σ is the standard deviation, and e is the
mathematical constant e (approximately equal to 2.71828).

The Gaussian distribution has several important properties. It is symmetric about the mean,
meaning that the probability of a value being above the mean is the same as the probability
of it being below the mean. The total area under the curve of the Gaussian distribution is
equal to 1, which means that the sum of the probabilities of all possible values of x is equal
to 1. The Gaussian distribution is also asymptotic, meaning that the tails of the curve
approach but never touch the x-axis.

The Gaussian distribution is widely used in statistics and probability theory because of its
simplicity and versatility. It is used to model a wide variety of natural phenomena, including
the heights of individuals in a population, the errors in scientific measurements, and the
noise in electronic signals. The central limit theorem states that the sum of a large number of
independent random variables, regardless of their original distribution, will tend to be
normally distributed, which further highlights the importance of the Gaussian distribution in
probability theory.
Python Code

Plots
 Experiment -2

Aim : Write a MATLAB/Python function that will take as inputs: (a) the mean vectors, (b) the
covariance matrices of the class distributions of a c-class problem, (c) the a priori probabilities of
the c classes, and (d) a matrix X containing column vectors that stem from the above classes. It
will give as output an N-dimensional vector whose ith component contains the class where the
corresponding vector is assigned, according to the Bayesian classification rule.

Software Required : Jupyter Notebook

Theory

The Bayes' theorem states that the probability of a hypothesis (H) given some observed evidence
(E) is proportional to the probability of the evidence given the hypothesis, multiplied by the prior
probability of the hypothesis. In the context of classification, the hypothesis corresponds to the
category or class to which an object or event belongs, and the evidence corresponds to the
observed features or attributes.

The Bayesian classification rule works by first calculating the prior probability of each class,
based on the frequency or proportion of objects or events belonging to each class in the training
data. Then, for each observed feature or attribute, the conditional probability of that feature given
each class is calculated using the training data. Finally, the posterior probability of each class
given the observed features is calculated using Bayes' theorem, and the object or event is
classified as belonging to the class with the highest posterior probability.

In mathematical terms, the Bayesian classification rule can be expressed as:

P(C|X) = P(X|C) * P(C) / P(X)

where P(C|X) is the posterior probability of class C given observed features X

where P(X|C) is the conditional probability of observed features X given class C

where P(C) is the prior probability of class C, and P(X) is the probability of observed features X.

1. Convert the given dataset into frequency tables.

2 .Generate Likelihood table by finding the probabilities of given features.

3. Now, use Bayes theorem to calculate the posterior probability.

Simulation Code
import numpy as np
import matplotlib.pyplot as plt

def bayesian_classifier(mean_vectors, covariance_matrices, a_priori_probs, X):

n, d = X.shape
c = len(mean_vectors)
class_assignments = np.zeros(n)
for i in range(n):
x = X[i,:]
max_posterior = -np.inf
max_class = -1
for j in range(c):
mean = mean_vectors[j]
covariance = covariance_matrices[j]
diff = x - mean
exponent = -0.5 * np.dot(np.dot(diff.T, np.linalg.inv(covariance)), diff)
determinant = np.linalg.det(covariance)
posterior = np.log(a_priori_probs[j]) - 0.5 * np.log(determinant) + exponent
if posterior > max_posterior:
max_posterior = posterior
max_class = j
class_assignments[i] = max_class
return class_assignments

# Example input
np.random.seed(1)
mean_vectors = [np.array([0, 0]), np.array([5, 5])]
covariance_matrices = [np.array([[1, 0], [0, 1]]), np.array([[2, 0], [0, 2]])]
a_priori_probs = [0.5, 0.5]
X = np.concatenate((np.random.multivariate_normal(mean_vectors[0], covariance_matrices[0],
100),
np.random.multivariate_normal(mean_vectors[1], covariance_matrices[1], 100)))

# Call the bayesian_classifier function

class_assignments = bayesian_classifier(mean_vectors, covariance_matrices, a_priori_probs, X)

# Plot the results

colors = ['b', 'y']
for i in range(2):
class_samples = X[class_assignments == i, :]
plt.scatter(class_samples[:, 0], class_samples[:, 1], c=colors[i], alpha=0.5)
plt.show()
Result

Conclusion

The N - dimensional vector whose ith component contains the class where the corresponding vector
is assigned, according to the Bayesian classification rule is been identified
3
 Experiment – 4

AIM : Write a MATLAB/Python function that will take as inputs: (a) the mean
vectors,(b) the covariance matrix of the class distributions of a c-class problem, and (c) a
matrix X containing column vectors that stem from the above classes. It will give as output
an N-dimensional vector whose ith component contains the class where the corresponding
vector is assigned according to the minimum Mahalanobis distance classifier.

Software Required : Python

Theory

The Mahalanobis distance classifier is a classification algorithm that measures the distance
between a test point and the mean of the training data in a way that takes into account the
correlation structure of the input variables. It is based on the Mahalanobis distance, which is a
measure of the distance between two points in a multivariate space.
The Mahalanobis distance is defined as the distance between a test point and the mean of a set of
points, divided by the standard deviation of the set of points in the direction of the test point. This
distance metric takes into account the correlations between the input variables, which can be used
to improve classification accuracy in situations where the variables are correlated.

To use the Mahalanobis distance as a classifier, we first calculate the Mahalanobis distance
between the test point and the mean of each class in the training data. The test point is then
assigned to the class with the smallest Mahalanobis distance.

The Mahalanobis distance classifier has several advantages over other classification algorithms,
such as the k-nearest neighbors classifier. One advantage is that it can handle correlated input
variables, which can improve classification accuracy. Another advantage is that it can handle
missing data, as long as the covariance matrix is estimated correctly.
However, the Mahalanobis distance classifier also has some limitations. One limitation is that it
can be sensitive to outliers in the training data, as these can have a large impact on the covariance
matrix. Another limitation is that it assumes that the data are normally distributed, which may not
be the case in some situations.

Overall, the Mahalanobis distance classifier is a useful tool for classification in situations where
the input variables are correlated, and where the assumption of normality holds.
Code

import numpy as np
import matplotlib.pyplot as plt

def mahalanobis_classifier(mean_vectors, covariance, X):

class_assignments = np.zeros((X.shape[1],))
for i in range(X.shape[1]):
x = X[:,i]
min_distance = float('inf')
for j in range(len(mean_vectors)):
mean = mean_vectors[j]
diff = x - mean
mahal_distance = np.dot(np.dot(diff.T, np.linalg.inv(covariance)), diff)
if mahal_distance < min_distance:
min_distance = mahal_distance
min_class = j
class_assignments[i] = min_class
return class_assignments

# Generate three clusters of data

np.random.seed(42)
N = 500
X1 = np.random.multivariate_normal([2,2], [[1,0],[0,1]], N).T
X2 = np.random.multivariate_normal([5,5], [[1,0.5],[0.5,1]], N).T
X3 = np.random.multivariate_normal([8,2], [[1,0.3],[0.3,1]], N).T

# Plot the data

plt.scatter(X1[0], X1[1], color='yellow')
plt.scatter(X2[0], X2[1], color='lime')
plt.scatter(X3[0], X3[1], color='cyan')

# Stack the data into a single array

X = np.hstack((X1,X2,X3))

# Define the mean vectors

mean_vectors = [np.mean(X1,axis=1), np.mean(X2,axis=1), np.mean(X3,axis=1)]

# Define the covariance matrix

covariance = np.cov(X)

# Call the mahalanobis_classifier function

class_assignments = mahalanobis_classifier(mean_vectors, covariance, X)

# Plot the results

for i in range(X.shape[1]):
if class_assignments[i] == 0:
 plt.scatter(X[0,i], X[1,i], color='red', alpha=0.2)
elif class_assignments[i] == 1:
plt.scatter(X[0,i], X[1,i], color='green', alpha=0.2)
elif class_assignments[i] == 2:
plt.scatter(X[0,i], X[1,i], color='blue', alpha=0.2)
for mean in mean_vectors:
plt.scatter(mean[0], mean[1], marker='x', s=100, linewidth=3)
plt.show()

Result
 Experiment - 5

Aim : Write a MATLAB/Python function that takes as inputs: (a) a set of N1 vectors packed as
columns of a matrix Z,(b) an N1-dimensional vector containing the classes where each vector in Z
belongs, (c) the value for the parameter k of the classifier,(d) a set of N vectors packed as columns
in the matrix X. It returns an N-dimensional vector whose ith component contains the class where
the corresponding vector of X is assigned, according to the k-nearest neighbour classifier.

Software Required : Python

Theory

The k-nearest neighbor (k-NN) classifier is a classification algorithm that works by finding
the k nearest training data points to a given test data point and classifying the test point
based on the most common class among its k-nearest neighbors. The choice of k is an
important parameter in the algorithm, and it can be chosen through cross-validation or other
methods.

The k-NN classifier is a non-parametric algorithm, meaning it does not make any
assumptions about the distribution of the data. It can handle both continuous and
categorical data, and can be used for both binary and multi-class classification problems.

One advantage of the k-NN classifier is its simplicity and ease of implementation. It also
has the potential to be very accurate, particularly if the training data set is large and the
value of k is chosen carefully. However, the algorithm can be computationally intensive,
especially for large data sets.

One limitation of the k-NN classifier is that it can be sensitive to noisy data and outliers. It
also requires the choice of an appropriate distance metric, which can be challenging in
some situations.

Overall, the k-NN classifier is a useful algorithm for classification problems where the data
is not easily modeled by a parametric distribution and the training data set is sufficiently
large.
Code

import numpy as np
import matplotlib.pyplot as plt

def euclidean_distance(x, y):

return np.sqrt(np.sum((x - y)**2))

def knn_classifier(Z, classes, k, X):

class_assignments = np.zeros((X.shape[1],))
for i in range(X.shape[1]):
x = X[:,i]
distances = np.zeros((Z.shape[1],))
for j in range(Z.shape[1]):
z = Z[:,j]
distances[j] = euclidean_distance(x, z)
closest_classes = classes[np.argsort(distances)[:k]]
(values, counts) = np.unique(closest_classes, return_counts=True)
class_assignments[i] = values[np.argmax(counts)]
return class_assignments

# Define the matrix Z with 1000 vectors

Z = np.random.rand(2, 1000)

# Define the classes for Z

classes = np.random.randint(0, 2, 1000)

# Define the value for k

k=5

# Define the matrix X with 200 vectors

X = np.random.rand(2, 200)

# Call the knn_classifier function

class_assignments = knn_classifier(Z, classes, k, X)

# Plot the results

colors = ['b', 'y']
for i in range(X.shape[1]):
plt.scatter(X[0,i], X[1,i], color=colors[int(class_assignments[i])])
plt.show()
Result
 Experiment - 6

Aim : Write a MATLAB/Python function that will take as inputs: (a) an N-dimensional vector,
each component of which contains the class where the corresponding data vector belongs and (b)
a similar N- dimensional vector each component of which contains the class where the
corresponding data vector is assigned from a certain classifier. Its output will be the percentage of
the places where the two vectors differ (i.e., the classification error of the classifier).

Software Required : Python

Theory
Error analysis is the process to isolate, observe and diagnose erroneous ML predictions thereby
helping understand pockets of high and low performance of the model. When it is said that “the
model accuracy is 90%” it might not be uniform across subgroups of data and there might be
some input conditions which the model fails more. So, it is the next step from aggregate metrics to
a more in-depth review of model errors for improvement. The classification error for a supervised
learning classifier is given by the difference of total samples and the erroneous predictions
divided by the total samples predicted by the classifier. Since, we are calculating this for a
supervised model, we will already have the values of the actual classes of the inputs.

The classifier used in this case is the k-nearest neighbour classifier. The k- nearest neighbors
algorithm, also known as KNN or k-NN, is a non- parametric, supervised learning classifier,
which uses proximity to make

classifications or predictions about the grouping of an individual data point.

While it can be used for either regression or classification problems.
Code
import numpy as np

def classification_error(true_classes, predicted_classes):

return np.sum(true_classes != predicted_classes) / true_classes.size

# Example usage
true_classes = np.array([0, 1, 0, 1, 0])
predicted_classes = np.array([0, 1, 1, 1, 0])
error = classification_error(true_classes, predicted_classes)
print(f"Classification error: {error:.2%}")

Result

Conclusion

The classification of the data points using the KNN classifier is done and the
error for classification has been calculated as above
 Experiment - 7

Aim : Write a MATLAB/Python function for the perceptron algorithm. This will take as inputs:
(a) a matrix X containing N l-dimensional column vectors, ( b) an N -dimensional row vector y,
whose I th component contains the class (-1 or +1) where the corresponding vector belongs, and
(c) an initial value vector wini for the parameter vector. It returns the estimated parameter vector.

Software Required : Python

Theory

The perceptron algorithm is a supervised learning algorithm used for binary classification tasks. It
is based on the concept of a simple artificial neuron called a perceptron, which takes in a set of
input features and produces a binary output, either 0 or 1.

The algorithm starts by initializing the weights for each input feature to a random value. It then
takes in a set of training data, where each data point consists of a set of input features and a binary
output. The algorithm computes the weighted sum of the input features, adds a bias term, and
applies an activation function to the result to obtain the predicted output.

The activation function used in the perceptron algorithm is usually the step function, which
outputs 1 if the weighted sum is greater than or equal to a certain threshold, and 0 otherwise. The
threshold value is a hyperparameter that can be set by the user.

During training, the algorithm compares the predicted output with the actual output and adjusts
the weights based on the error between them. The weights are updated according to the following
formula:

new_weight = old_weight + learning_rate * (desired_output - predicted_output) * input_feature

where learning_rate is a hyperparameter that controls the rate at which the algorithm learns, and
input_feature is the value of the input feature for the current data point.

The algorithm continues to update the weights for each data point in the training set until it
converges to a set of weights that can accurately classify the training data. Once the algorithm has
converged, it can be used to make predictions on new data points by computing the weighted sum
of the input features, adding the bias term, and applying the activation function.
Code

import numpy as np

class Perceptron:
def _init_(self, learning_rate=0.1, epochs=100):
self.learning_rate = learning_rate
self.epochs = epochs

def fit(self, X, y):

# add a column of ones to X for the bias term
X = np.hstack((np.ones((X.shape[0], 1)), X))

# initialize the weights to zeros

self.weights = np.zeros(X.shape[1])

for epoch in range(self.epochs):

for i in range(X.shape[0]):
# calculate the predicted label
y_pred = np.sign(np.dot(self.weights, X[i]))

# update the weights if the prediction is incorrect

if y_pred != y[i]:
self.weights += self.learning_rate * y[i] * X[i]

def predict(self, X):

# add a column of ones to X for the bias term
X = np.hstack((np.ones((X.shape[0], 1)), X))

# calculate the predicted labels

y_pred = np.sign(np.dot(self.weights, X.T))

return y_pred

def plot_decision_boundary(self, X, y):

# plot the data points
plt.scatter(X[:, 0], X[:, 1], c=y)

# plot the decision boundary

x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5
y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5
xx, yy = np.meshgrid(np.arange(x_min, x_max, 0.02), np.arange(y_min, y_max, 0.02))
Z = self.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)
plt.contour(xx, yy, Z, colors='k', levels=[-1, 0, 1], alpha=0.5)
plt.show()
# generate some random data for binary classification
np.random.seed(1453)
 X = np.random.normal(size=(100, 2))
y = np.ones(100)
y[X[:, 0] < 0] = -1

# create a Perceptron instance with default hyperparameters

perceptron = Perceptron()

# train the Perceptron on the data

perceptron.fit(X, y)

# plot the decision boundary

perceptron.plot_decision_boundary(X, y)

Result
 Experiment - 8

Aim : Design a three-layer FFN, using gradient descent to perform the{x1, x2} → {y1, y2}
mapping. The activation function for all the nodes is the hyperbolic tangent one. For training, one
may select one of the following algorithms: a) the standard gradient descent backpropagation
algorithm, ( b) the backpropagation algorithm with momentum , and (c) the backpropagation
algorithm with adaptive learning rate.

Software Required : Python

Theory

A three-layer feedforward neural network (FFN) is a type of neural network architecture that
consists of three layers of nodes: an input layer, a hidden layer, and an output layer. The input
layer contains nodes that receive input data, the hidden layer processes the input data through a set
of nonlinear transformations, and the output layer produces a set of predictions or outputs.

Here is a high-level description of the three-layer FFN architecture:

Input layer: The input layer consists of nodes that receive input data, which can be a vector of
features or raw input data such as images or audio signals. Each node in the input layer
corresponds to a single feature or input dimension.

Hidden layer: The hidden layer consists of nodes that process the input data through a set of
nonlinear transformations. Each node in the hidden layer receives input from all nodes in the
previous layer and produces an output that is fed forward to the next layer. The activations of the
hidden layer nodes are computed using a nonlinear activation function, such as the sigmoid
function, ReLU function, or hyperbolic tangent function.

Output layer: The output layer consists of nodes that produce the final set of predictions or
outputs. The number of nodes in the output layer depends on the task at hand, such as regression
or classification. For example, in a binary classification task, there would be a single output node
that produces a value between 0 and 1, which can be interpreted as the probability of belonging to
the positive class.

During training, the weights and biases of the network are learned through backpropagation,
which is a gradient-based optimization algorithm. Backpropagation computes the gradients of the
loss function with respect to the weights and biases of the network, and updates them using a
learning rate and a momentum parameter. The process of computing the gradients and updating
the weights and biases is repeated iteratively until the network converges to a set of weights and
biases that minimize the loss function on the training data.
Code
import numpy as np
from sklearn.datasets import make_regression

# Generate a sample dataset

X, y = make_regression(n_samples=10000, n_features=2, n_targets=2, random_state=42)

# Normalize the dataset

X = (X - np.mean(X, axis=0)) / np.std(X, axis=0)
y = (y - np.mean(y, axis=0)) / np.std(y, axis=0)

# Define the hyperbolic tangent activation function

def tanh(x):
return np.tanh(x)

# Define the derivative of the hyperbolic tangent activation function

def tanh_derivative(x):
return 1 - np.tanh(x) ** 2

# Define the number of neurons in each layer

input_size = 2
hidden_size = 4
output_size = 2

# Initialize the weights for the first and second layers

W1 = np.random.randn(input_size, hidden_size)
W2 = np.random.randn(hidden_size, output_size)

# Define the learning rate

learning_rate = 0.01

# Define the number of epochs

num_epochs = 1000

# Choose one of the following optimization algorithms:

# a) Standard gradient descent backpropagation algorithm
# b) Backpropagation algorithm with momentum
# c) Backpropagation algorithm with adaptive learning rate

# a) Standard gradient descent backpropagation algorithm

for epoch in range(num_epochs):
# Forward pass
hidden_layer_input = np.dot(X, W1)
hidden_layer_output = tanh(hidden_layer_input)
output_layer_input = np.dot(hidden_layer_output, W2)
y_pred = tanh(output_layer_input)
 # Backward pass
error = y_pred - y
d_y_pred = error * tanh_derivative(y_pred)
error_hidden = np.dot(d_y_pred, W2.T)
d_hidden = error_hidden * tanh_derivative(hidden_layer_output)

# Update weights
W2 -= learning_rate * np.dot(hidden_layer_output.T, d_y_pred)
W1 -= learning_rate * np.dot(X.T, d_hidden)

# b) Backpropagation algorithm with momentum

# Initialize the velocity for the first and second layers
v1 = np.zeros_like(W1)
v2 = np.zeros_like(W2)

# Define the momentum coefficient

momentum = 0.9

for epoch in range(num_epochs):

# Forward pass
hidden_layer_input = np.dot(X, W1)
hidden_layer_output = tanh(hidden_layer_input)
output_layer_input = np.dot(hidden_layer_output, W2)
y_pred = tanh(output_layer_input)

# Backward pass
error = y_pred - y
d_y_pred = error * tanh_derivative(y_pred)
error_hidden = np.dot(d_y_pred, W2.T)
d_hidden = error_hidden * tanh_derivative(hidden_layer_output)

# Update velocity
v2 = momentum * v2 - learning_rate * np.dot(hidden_layer_output.T, d_y_pred)
v1 = momentum * v1 - learning_rate * np.dot(X.T, d_hidden)

# Update weights
W2 += v2
W1 += v1

# c) Backpropagation algorithm with adaptive learning rate

# Initialize the learning rate
initial_learning_rate = 0.01

for epoch in range(num_epochs):

# Forward pass
hidden_layer_input = np.dot(X, W1)
hidden_layer_output = tanh(hidden_layer_input)
output_layer_input = np.dot(hidden_layer_output, W2)
 y_pred = tanh(output_layer_input)
# Backward pass
error = y_pred - y
d_y_pred = error * tanh_derivative(y_pred)
error_hidden = np.dot(d_y_pred, W2.T)
d_hidden = error_hidden * tanh_derivative(hidden_layer_output)

# Compute the learning rate

learning_rate = initial_learning_rate / (1 + epoch * 0.001)

# Update weights
W2 -= learning_rate * np.dot(hidden_layer_output.T, d_y_pred)
W1 -= learning_rate * np.dot(X.T, d_hidden)

Result

The 3 layer feed-forward network using gradient descent has been created
and the output for a given set of inputs is as shown.
 Experiment - 9

Aim :
Write MATLAB/Python function to compute the principal components of the covariance matrix
of an l × N dimensional data matrix X as well as the corresponding variances. Hence, write a
MATLAB/Python function that evaluates the performance of the PCA method when applied on a
data matrix X

Software Required : Python

Theory:-

Principal Component Analysis (PCA) is a technique used for dimensionality reduction and data
compression. It works by identifying the directions, or principal components, in which the data
varies the most, and then projecting the data onto these directions.
Here are the steps involved in PCA:

Standardize the data: Before applying PCA, it is important to standardize the data by subtracting
the mean and dividing by the standard deviation. This ensures that all features are on the same
scale and have equal importance in the analysis.

Compute the covariance matrix: The next step is to compute the covariance matrix, which
measures the linear relationship between the features. The covariance matrix is a square matrix
where the (i,j)th element is the covariance between the i-th and j-th features.

Compute the eigenvectors and eigenvalues: The eigenvectors and eigenvalues of the covariance
matrix represent the directions and magnitudes of the principal components, respectively. The
eigenvectors are the directions in which the data varies the most, and the eigenvalues represent the
amount of variance explained by each principal component.

Choose the number of principal components: The next step is to choose the number of principal
components to retain. This can be done by examining the eigenvalues and selecting the top k
components that explain the most variance in the data. Typically, a scree plot is used to visualize
the eigenvalues and identify the "elbow point" where the curve levels off.

Project the data onto the principal components: The final step is to project the data onto the
selected principal components. This can be done by multiplying the standardized data matrix by
the matrix of eigenvectors corresponding to the selected principal components.

PCA is widely used in a variety of fields, including image processing, signal processing, and
finance. It can be used for tasks such as dimensionality reduction, feature extraction, and data
compression. PCA can also be extended to nonlinear and kernelized versions, such as Kernel
PCA, which can handle nonlinear and non-Gaussian data distributions.
Code a

import numpy as np

# generate sample data

l = 100 # number of samples
N = 25 # number of features
X = np.random.normal(size=(l,N))

# compute covariance matrix and its eigenvalues and eigenvectors

cov = np.cov(X.T)
eigenvalues, eigenvectors = np.linalg.eig(cov)

# sort eigenvalues in descending order

idx = eigenvalues.argsort()[::-1]
eigenvalues = eigenvalues[idx]
eigenvectors = eigenvectors[:,idx]

# compute principal components

pcs = np.dot(X, eigenvectors)

# compute variances
variances = np.var(pcs, axis=0)

# evaluate performance of PCA method

total_variance = np.sum(np.var(X, axis=0))
explained_variance = np.sum(variances)
explained_variance_ratio = explained_variance / total_variance
print('Explained variance ratio:', explained_variance_ratio)
Code b
import numpy as np
def pca(X):
# Subtract the mean of each column from X

# Compute the mean of each column of X

mean_X = np.mean(X, axis=0)

# Compute the centered data matrix

X_centered = X - mean_X

# Compute the covariance matrix

cov_X = np.cov(X_centered.T)

# Compute the eigenvectors and eigenvalues of the covariance matr

ix
eig_vals, eig_vecs = np.linalg.eig(cov_X)

# Sort the eigenvectors in descending order of their correspo

nding eigenvalues
idxs = np.argsort(eig_vals)[::-1]
eig_vals = eig_vals[idxs]
eig_vecs = eig_vecs[:,idxs]

# Compute the principal components

V = eig_vecs

# Compute the variances corresponding to each principal compo

nent
variances = eig_vals / (X.shape[0] - 1)

return V, variances
# Generate random data matrix
X = np.random.normal(size=(100, 5))
principal_components, variances = pca(X)
plt.scatter(principal_components[:, 0], principal_components[:
, 1])
plt.xlabel('PC1 (Variance = {:.2f})'.format(variances[0]))
plt.ylabel('PC2 (Variance = {:.2f})'.format(variances[1]))
plt.show()

Result

Explained variance ratio : 1.0 expected

Conclusion

The principal components of the given covariance matrix have been

calculated and plotted as shown