Unit I:Introduction History of Deep Learning, McCulloch Pitts Neuron, Multilayer

Perceptions (MLPs), Representation Power of MLPs, Sigmoid Neurons, Feed Forward

Neural Networks, Back propagation, weight initialization methods, Batch Normalization,
Representation Learning, GPU implementation, Decomposition – PCA and SVD

Introduction History of Deep Learning

Deep learning is a subfield of machine learning, which itself is a branch of artificial
intelligence (AI). Deep learning focuses on algorithms inspired by the structure and function
of the human brain, called artificial neural networks (ANNs). These algorithms are capable
of learning from vast amounts of data and improving their performance over time. Over the
past several decades, deep learning has evolved significantly, and its impact is seen across
various domains, including natural language processing (NLP), computer vision, speech
recognition, and robotics.

McCulloch Pitts Neuron

The McCulloch-Pitts Neuron is one of the earliest models of a neural network and a
precursor to modern artificial neural networks. It was introduced in 1943 by Warren
McCulloch and Walter Pitts in a paper titled "A Logical Calculus of Ideas Immanent in
Nervous Activity." The model was inspired by the workings of biological neurons but in a
simplified form. Despite its simplicity, the McCulloch-Pitts Neuron laid the foundation for
future developments in neural networks and artificial intelligence.

Key Features of the McCulloch-Pitts Neuron

1. Binary Inputs and Outputs:

o The McCulloch-Pitts Neuron works with binary inputs (either 0 or 1) and

produces a binary output (either 0 or 1). This mirrors the firing mechanism of
a biological neuron, where a neuron either fires (outputs 1) or doesn't fire
(outputs 0) based on its input.
2. Summation of Inputs:

o The neuron receives a set of binary inputs. Each input is associated with a
weight, which determines the strength of that input. The neuron computes a
weighted sum of these inputs.

o The weighted sum of the inputs is denoted as: S=∑i=1nwi⋅xiS =

\sum_{i=1}^{n} w_i \cdot x_i Where:
 xix_i are the inputs,
 wiw_i are the weights,
 nn is the number of inputs.
3. Threshold Function:
 o The neuron applies a threshold function (or activation function) to the
weighted sum. If the weighted sum exceeds a certain threshold, the neuron
"fires" and outputs 1. If the weighted sum is less than the threshold, the neuron
doesn't fire and outputs 0.

o Mathematically, this is expressed as: Output={1if S≥θ0if S<θ\text{Output} =

\begin{cases} 1 & \text{if } S \geq \theta \\ 0 & \text{if } S < \theta
\end{cases} Where θ\theta is the threshold value.
4. Simplified Neuron Model:

o The McCulloch-Pitts neuron is a very basic model and does not incorporate
complex features like learning, dynamic weight adjustments, or multiple
layers (as in modern neural networks). It only considers linear combinations
of inputs and applies a step function to determine the output.

Activation Function of McCulloch-Pitts Neuron

The McCulloch-Pitts neuron uses a threshold function (also known as a step function) as its
activation function. This function is defined as:

f(S)={1if S≥θ0if S<θf(S) = \begin{cases} 1 & \text{if } S \geq \theta \\ 0 & \text{if } S <
\theta \end{cases}

Where θ\theta is the threshold. If the sum of the weighted inputs reaches or exceeds the
threshold, the neuron fires (output = 1); otherwise, it remains inactive (output = 0).

Working of the McCulloch-Pitts Neuron

1. Inputs: The neuron receives a set of binary inputs, say x1,x2,x3,…,xnx_1, x_2, x_3,
\dots, x_n, each representing some signal.
2. Weighted Sum: The neuron calculates the weighted sum SS of these inputs, where
each input xix_i has a corresponding weight wiw_i.

3. Thresholding: The weighted sum SS is compared to a threshold value θ\theta. If the

sum is greater than or equal to θ\theta, the output is 1 (neuron fires); otherwise, the
output is 0 (neuron does not fire).

Significance and Impact

 Binary Logic: The McCulloch-Pitts Neuron can perform simple logical operations
such as AND, OR, and NOT by adjusting the weights and threshold. This ability to
perform logic gates was one of the key reasons the McCulloch-Pitts model was
influential in early neural network research.

 Foundation for Neural Networks: Despite its simplicity, the McCulloch-Pitts

Neuron laid the groundwork for the development of more complex neural networks.
The concepts of weights, thresholds, and the idea of neurons firing based on inputs are
still foundational in modern neural networks.
  Limitations:

o No Learning Capability: The McCulloch-Pitts model does not have a

learning mechanism. In modern neural networks, learning occurs through
processes like backpropagation.

o Limited to Linear Problems: The McCulloch-Pitts Neuron can only model

problems that are linearly separable. This limitation was one of the key
insights that led to the development of more advanced neural network
architectures (e.g., multilayer perceptrons).

o Binary Outputs: The binary output also limits the applicability of the
McCulloch-Pitts neuron to problems where binary classification is sufficient.

Logical Operations with McCulloch-Pitts Neuron

The McCulloch-Pitts neuron can perform several fundamental logic operations based on the
weights and threshold:
1. AND Operation:

o For an AND gate, the weights are set so that the sum of the inputs must exceed
a certain threshold for the neuron to output 1.

o Example: If the inputs are x1=x2=1x_1 = x_2 = 1, then the output is 1;

otherwise, the output is 0.
2. OR Operation:

o The OR gate requires the sum of the inputs to exceed the threshold if at least
one of the inputs is 1.
o Example: If x1=1x_1 = 1 or x2=1x_2 = 1, the output is 1.
3. NOT Operation:

o A NOT gate can be modeled by setting the threshold such that the output is the
opposite of the input.
Multilayer Perceptions (MLPs)

A Multilayer Perceptron (MLP) is a type of artificial neural network consisting of multiple

layers of neurons, with each layer fully connected to the next one. MLPs are the fundamental
architecture for many deep learning models and are particularly powerful for supervised
learning tasks such as classification, regression, and function approximation.

Basic Structure of an MLP

An MLP consists of the following layers:

1. Input Layer: This layer takes in the raw data as input. The input layer is typically a
vector, where each element of the vector represents a feature of the data.

2. Hidden Layers: These are the intermediate layers between the input and output
layers. There can be one or more hidden layers in an MLP, and they are responsible
for learning representations of the input data. Each hidden layer contains multiple
neurons that apply a non-linear activation function to the weighted sum of their
inputs.

3. Output Layer: The output layer produces the final prediction or classification result.
For a classification task, this might be the probabilities of belonging to each class,
while for a regression task, it might be a continuous value.

Each layer of an MLP is made up of neurons (also called units), and each neuron is connected
to neurons in the subsequent layer by weighted connections.

Training an MLP
The training of an MLP involves the following steps:
1. Forward Pass: Compute the output of the network for a given input.

2. Loss Calculation: Compute the difference between the predicted output and the
actual output (using a loss function).
3. Backpropagation: Calculate the gradients of the loss with respect to each weight by
applying the chain rule.
 4. Weight Update: Use an optimization algorithm (such as gradient descent) to update
the weights in the direction that minimizes the loss.

5. Iteration: Repeat the process for many iterations (epochs) to progressively reduce the
loss and improve the model's predictions.

Activation Functions in MLPs

The activation function plays a critical role in enabling MLPs to model complex non-linear
relationships. Here are some commonly used activation functions in MLPs:

 Sigmoid: Often used in binary classification tasks. It squashes the input to a range
between 0 and 1.
 Tanh: Like sigmoid but with a range of [-1, 1], often used in hidden layers.
 ReLU: A very popular activation function due to its simplicity and efficiency. It
outputs the input if it’s positive, and zero otherwise.

 Leaky ReLU: A variant of ReLU that allows small negative values for inputs less
than zero, helping to avoid dead neurons.
 Softmax: Typically used in the output layer for multi-class classification tasks. It
converts the raw output into probabilities that sum to 1.

Advantages of MLPs
1. Non-linear Mapping:
o MLPs can model highly non-linear relationships between inputs and outputs,
which makes them more powerful than simple linear models.
2. Flexibility:
o MLPs can be applied to a wide range of tasks, including classification,
regression, function approximation, and more.
3. Learning Complex Patterns:

o Through multiple hidden layers, MLPs can learn hierarchical representations

of the data, capturing increasingly complex patterns.
4. Universal Approximation:

o Thanks to the Universal Approximation Theorem, a sufficiently large MLP

can approximate any continuous function, making it theoretically capable of
solving any learning problem, given enough data and computational resources.
Limitations of MLPs
1. Overfitting:

o MLPs have a large number of parameters, which can lead to overfitting if the
training data is not sufficiently large or diverse. Regularization techniques like
dropout, L2 regularization, and data augmentation are used to prevent
overfitting.
2. Training Complexity:

o Training deep MLPs can be computationally expensive and time-consuming,

especially for large datasets. Proper weight initialization, gradient descent
optimization methods (e.g., Adam), and the use of GPUs can mitigate some of
these issues.
3. Vanishing/Exploding Gradient Problem:

o In deep networks, gradients can either become too small (vanishing gradient)
or too large (exploding gradient), which can slow down training or lead to
instability. Techniques like batch normalization and careful weight
initialization can help mitigate these problems.
4. Need for Large Datasets:

o MLPs generally require large amounts of data to effectively train the network
and avoid overfitting. Insufficient data can lead to poor generalization.

Applications of MLPs
1. Classification:

o MLPs are widely used for classification tasks, such as image recognition,
sentiment analysis, spam detection, and more.
2. Regression:

o MLPs can predict continuous values, such as house prices, stock prices, and
other real-valued outputs.
3. Function Approximation:
o MLPs can approximate any continuous function, making them suitable for
solving problems in control systems, robotic motion, and more.
4. Time Series Prediction:
o With appropriate modifications (like recurrent layers), MLPs can be applied to
time series forecasting and sequential data.
Representation Power of MLPs

Sigmoid Neurons

Sigmoid Neurons

A Sigmoid Neuron is a type of artificial neuron in a neural network that uses the sigmoid
activation function to introduce non-linearity into the network. Sigmoid neurons are
commonly used in traditional neural networks, especially in the earlier stages of deep
learning research.

Sigmoid Activation Function

The sigmoid function is a mathematical function that maps input values to an output range
between 0 and 1. It is an S-shaped curve, which makes it suitable for binary classification
tasks where outputs are constrained to represent probabilities.

Properties of Sigmoid Neurons

1. Non-linearity:
o The sigmoid function introduces non-linearity, enabling the network to model
complex relationships in data. This is important because, without non-
linearity, a neural network would only be able to learn linear patterns,
regardless of the depth of the network.
2. Output Range:

o The output of a sigmoid neuron is always between 0 and 1, making it ideal for
tasks where the output needs to be interpreted as a probability (e.g., binary
classification).
3. Smooth Gradient:

o The sigmoid function has a smooth gradient, which helps with the
optimization of weights during training (e.g., gradient descent).
 4. Differentiability:

o The sigmoid function is differentiable, which is essential for backpropagation,

as it allows the gradient of the loss function to be calculated with respect to the
weights.

Advantages of Sigmoid Neurons

1. Probabilistic Interpretation:

o Since the output of a sigmoid neuron is between 0 and 1, it can be interpreted

as a probability. This is particularly useful in binary classification problems,
where the goal is to predict one of two classes.
2. Differentiable:
o The sigmoid function is continuous and differentiable, which makes it suitable
for gradient-based optimization methods like backpropagation.
3. Simple and Easy to Compute:

o The sigmoid function is mathematically simple, and its derivative can be

computed efficiently, making it easy to use in training neural networks.

Limitations of Sigmoid Neurons

1. Vanishing Gradient Problem:

o The sigmoid function saturates at both ends of the range, meaning that for very
large or very small values of xx, the gradient (derivative) approaches zero.
This leads to a problem known as vanishing gradients, which can hinder the
learning process in deep networks, especially when training on large datasets.
In the regions where the gradient is very small, weight updates become very
small, slowing down or even halting learning.
2. Non-zero Centered Output:

o The output of the sigmoid function is not zero-centered. This means that for
large positive inputs, the output is close to 1, and for large negative inputs, the
output is close to 0. This can lead to inefficient gradient updates during
training, as the gradients are not balanced around zero.
3. Slow Convergence:

o Due to the squashing nature of the sigmoid function, learning can be slower
compared to other activation functions like ReLU (Rectified Linear Unit),
especially when deep networks are used.
4. Limited Range:
 o The sigmoid function’s output range is between 0 and 1, which can sometimes
be too restrictive for some tasks, especially when a larger output range is
needed.

Applications of Sigmoid Neurons

1. Binary Classification:

o Sigmoid neurons are ideal for binary classification tasks, such as

distinguishing between two classes (e.g., spam vs. not spam, positive vs.
negative sentiment).
2. Probabilistic Output:
o In models where the output needs to be interpreted as a probability (e.g.,
logistic regression), the sigmoid function is commonly used.
3. Neural Network Output Layer:

o Sigmoid neurons are often used in the output layer of neural networks for
binary classification tasks, where the network outputs a value between 0 and 1,
representing the probability of belonging to one of the two classes.

Feed Forward Neural Networks

A Feed Forward Neural Network (FFNN) is one of the simplest types of artificial neural
networks where the connections between the nodes (neurons) do not form any cycles. In this
type of network, information moves in one direction: from the input layer, through hidden
layers (if any), to the output layer. FFNNs are used for a variety of tasks, including
regression, classification, and function approximation.

Basic Structure of FFNN

1. Input Layer:
 o This is the first layer, where the input features are provided to the network.
Each node in the input layer represents a feature of the input data.
2. Hidden Layers:

o These layers consist of neurons that process the inputs received from the
previous layer. A network can have one or more hidden layers, each
containing multiple neurons. The more hidden layers, the deeper the network,
making it capable of learning more complex representations of the data.
3. Output Layer:

o This is the final layer that produces the network’s output. The output layer’s
size depends on the task (e.g., one neuron for binary classification, multiple
neurons for multi-class classification or regression).
4. Activation Function:

o Each neuron in the network (except the input neurons) has an activation
function, which introduces non-linearity into the network, enabling it to learn
complex patterns.
o Common activation functions include:
 ReLU (Rectified Linear Unit): f(x)=max⁡(0,x)f(x) = \max(0, x)
 Sigmoid: f(x)=11+e−xf(x) = \frac{1}{1 + e^{-x}}
 Tanh: f(x)=21+e−2x−1f(x) = \frac{2}{1 + e^{-2x}} - 1

 Softmax: Typically used in the output layer for multi-class

classification problems.

Applications of Feed Forward Neural Networks

FFNNs are widely used in several domains, including:
1. Classification:

o Identifying the category of an object or classifying an input into one of several

categories.
o Examples: Image recognition, speech recognition, spam detection.
2. Regression:
o Predicting continuous output values.
o Examples: Stock price prediction, weather forecasting, medical diagnosis.
3. Function Approximation:
o Learning complex mappings from input to output (e.g., approximating
mathematical functions).
 4. Time Series Prediction:

o Using past data to predict future values, although more complex models like
RNNs or LSTMs are typically used for sequential data.

Advantages of Feed Forward Neural Networks

 Simple to implement: FFNNs are conceptually simple and easy to implement.

 Universal approximator: MLPs can approximate any continuous function, as per the
universal approximation theorem.
 Flexible: Can handle a variety of tasks, from regression to classification.

Limitations of Feed Forward Neural Networks

 Limited to Fixed Structure: FFNNs are not good for tasks involving sequential data
(e.g., time series, speech), for which specialized models like RNNs are more suited.

 Requires Large Datasets: They often require large datasets to achieve high
performance.

 Training Time: Training FFNNs can be time-consuming, especially with large

networks and datasets.

Summary of Feed Forward Neural Networks

 Structure: Consists of an input layer, one or more hidden layers, and an output layer.
 Forward Pass: Input data is passed through the network to produce output.

 Activation Function: Introduces non-linearity, enabling the network to learn complex

patterns.
 Training: Uses backpropagation and gradient descent to minimize the loss and update
weights.

 Applications: Used in various tasks like classification, regression, and function

approximation.

Feed forward neural networks are the foundation for many deep learning models and serve as
the basis for more complex neural network architectures, such as Convolutional Neural
Networks (CNNs) and Recurrent Neural Networks (RNNs).

Back propagation
Backpropagation (short for backward propagation of errors) is the cornerstone of learning
in neural networks. It is the process by which a neural network adjusts its weights to
minimize the error (or loss) in its predictions. Backpropagation uses the chain rule of calculus
to compute the gradient of the loss function with respect to each weight by propagating the
error backward through the network.
Key Steps in Backpropagation:
1. Forward Pass:
o Input is processed through the network to generate output.
2. Loss Function:
o Compute the error between the predicted and actual output.
3. Backward Pass:

o Compute the gradient of the loss function with respect to each weight in the
network using the chain rule.
4. Gradient Descent:

o Update the weights using the computed gradients to minimize the loss
function.

Optimizers in Backpropagation
Backpropagation is often combined with gradient-based optimization algorithms to
improve training. Some of the commonly used optimizers are:

1. Stochastic Gradient Descent (SGD): Updates the weights after every batch or
sample, making the updates more noisy but potentially faster.
2. Momentum: A variation of SGD that uses a moving average of past gradients to
smooth out updates and help escape local minima.
 3. Adam (Adaptive Moment Estimation): Combines momentum and adaptive learning
rates, adjusting the learning rate for each parameter individually.

Challenges in Backpropagation
1. Vanishing Gradients:

o When gradients become very small during the backward pass (especially in
deep networks), the weights stop updating properly, and learning slows down
or stagnates. This is particularly a problem with activation functions like
sigmoid or tanh.
2. Exploding Gradients:
o When gradients become excessively large, the weights can grow very large,
leading to instability in training.
3. Overfitting:

o Backpropagation might overfit the model to the training data if the model is
too complex or the training process is not regularized.

Weight initialization methods

Batch Normalization

Batch Normalization
Batch Normalization (BatchNorm) is a technique used to improve the training of deep
neural networks by normalizing the inputs to each layer. It helps to stabilize and accelerate
training, reduce the sensitivity to hyperparameters like the learning rate, and mitigate issues
like vanishing and exploding gradients.

What is Batch Normalization?

Batch Normalization normalizes the output of a layer in a neural network by adjusting and
scaling activations. It does this by computing the mean and variance of each feature in the
mini-batch and then standardizing the activations.

Why Use Batch Normalization?

1. Improves Convergence Speed:

o Normalizing the input to each layer stabilizes the training process, allowing
the network to train faster.
2. Reduces Internal Covariate Shift:

o Internal covariate shift refers to the change in the distribution of activations

during training as the parameters of the previous layers are updated.
BatchNorm reduces this shift, making the optimization process easier.
3. Acts as a Regularizer:

o By adding noise due to mini-batch statistics, it has a slight regularizing effect,

helping to prevent overfitting. This can sometimes eliminate the need for other
regularization techniques like dropout.
4. Mitigates Vanishing/Exploding Gradients:

o BatchNorm helps prevent the vanishing and exploding gradient problems by

ensuring that the activations maintain a reasonable scale, improving gradient
flow through deep networks.

How Does Batch Normalization Work?

1. During Training:
o The mean and variance are computed for each mini-batch.
o The activations are normalized using the batch statistics.
o The network learns the scaling (γ\gamma) and shifting (β\beta) parameters.
2. During Inference:
o The mean and variance are computed across the entire training dataset and
used as fixed values for normalization.
 o This prevents the network from depending on mini-batch statistics during
inference, which could cause inconsistencies.

When to Apply Batch Normalization?

 After the Fully Connected (Dense) Layer:

o In most cases, BatchNorm is applied after fully connected layers (before the
activation function).
 After Convolutional Layers:

o BatchNorm is also commonly applied after convolutional layers in

convolutional neural networks (CNNs).
 Before Activation Functions:

o BatchNorm is typically applied before the activation function (ReLU,

Sigmoid, etc.), as it normalizes the activations that are passed to the activation
function.

Benefits of Batch Normalization

1. Faster Training:

o By mitigating internal covariate shift, BatchNorm allows the use of higher

learning rates and speeds up training convergence.
2. Stability:

o Reduces the risk of exploding and vanishing gradients, ensuring stable

backpropagation.
3. Regularization:

o Adds noise to the training process (due to mini-batch statistics), which helps to
prevent overfitting, reducing the need for other regularization methods.
4. Better Performance:

o BatchNorm can often lead to better model performance by allowing the use of
deeper networks and faster convergence.

Limitations of Batch Normalization

1. Dependency on Mini-batch Size:

o The effectiveness of BatchNorm can be influenced by the mini-batch size. For

very small mini-batches, the statistics might be noisy, which can hurt
performance.
 2. Training Time:

o Although it speeds up convergence, the normalization process itself introduces

some additional computational cost during training.
3. Not Always Useful for Small Networks:

o For small networks or tasks with relatively simple data, BatchNorm might not
have a significant impact.
4. Not Suitable for Recurrent Networks:

o BatchNorm is less commonly used in Recurrent Neural Networks (RNNs)

because of the sequential nature of RNNs, where the dependencies between
time steps complicate batch-wise normalization.

Alternatives to Batch Normalization

1. Layer Normalization:

o Instead of normalizing over the mini-batch, layer normalization normalizes

across the features in a single sample. It is more suitable for RNNs and
transformer models.
2. Group Normalization:

o A compromise between BatchNorm and LayerNorm, GroupNorm divides the

channels into groups and normalizes within each group. It works well in
situations with smaller batch sizes.
3. Instance Normalization:

o Typically used in image generation tasks, instance normalization normalizes

each individual sample's features instead of using the entire batch.

Batch Normalization Summary

 Purpose: Stabilizes and speeds up the training process by normalizing the activations
in deep networks.

 Key Operation: Normalizes activations based on the mean and variance of each
mini-batch, then scales and shifts them using learnable parameters.
 Benefits: Faster training, improved stability, and slight regularization.

 Limitations: Sensitive to mini-batch size, may not be suitable for very small
networks or sequential models.
Representation Learning
Representation Learning is a key concept in machine learning where a model learns
meaningful features or representations of data that can be used for various tasks. Unlike
traditional methods where features are hand-engineered, representation learning enables the
model to automatically extract and organize features directly from raw data.

Why Representation Learning?

1. Eliminates Manual Feature Engineering:
o Automatically discovers relevant features without human intervention.
2. Captures Complex Patterns:

o Learns hierarchical or abstract representations, especially for tasks like image

recognition, speech, and text.

3. Generalizes Across Tasks:

o Features learned for one task can often be reused for related tasks (transfer
learning).

Key Techniques in Representation Learning

1. Unsupervised Representation Learning
 Learns features from unlabeled data.
 Examples:
o Autoencoders
o Principal Component Analysis (PCA)
o Clustering methods (e.g., K-means)

2. Supervised Representation Learning

 Learns representations tailored to a specific task using labeled data.
 Examples:
o Convolutional Neural Networks (CNNs) for image classification.
o Recurrent Neural Networks (RNNs) for sequence data.

3. Semi-Supervised Representation Learning

 Combines a small amount of labeled data with a large amount of unlabeled data.
 Example:
 o Self-supervised learning (e.g., contrastive learning).

4. Self-Supervised Representation Learning

 Creates pretext tasks (pseudo-labels) to generate supervisory signals from the data
itself.
 Example:
o Predicting missing parts of an image or word embeddings.

Applications of Representation Learning

1. Image Processing:
o Face recognition, object detection, image segmentation.
2. Natural Language Processing:
o Sentiment analysis, machine translation, text summarization.
3. Speech Recognition:
o Speaker identification, voice-to-text systems.
4. Recommendation Systems:
o Learning user and item embeddings for collaborative filtering.
5. Healthcare:
o Predicting patient outcomes, medical imaging.

Benefits of Representation Learning

1. Improved Performance:
o Learns task-specific features, leading to better accuracy.
2. Scalability:
o Automatically adapts to new data or tasks.
3. Reusability:
o Learned representations can be transferred to other tasks.
4. Reduced Human Effort:
o No need for domain experts to design features.
Challenges in Representation Learning
1. Data Requirements:
o Deep models often require large datasets to learn effective representations.
2. Interpretability:
o Representations can be abstract and difficult to interpret.
3. Computational Costs:
o Training deep models for representation learning can be expensive.
4. Overfitting:
o Risk of overfitting, especially with limited data.

GPU implementation
Modern deep learning frameworks heavily rely on Graphics Processing Units (GPUs) to
accelerate computations. GPUs are designed for parallel processing, making them highly
efficient for the matrix and tensor operations central to deep learning. Here's a detailed
breakdown of GPU implementation:

Why GPUs for Deep Learning?

1. Parallelism:

o GPUs have thousands of cores that execute many operations simultaneously,

ideal for tasks like matrix multiplication.
2. High Throughput:
o GPUs are optimized for high data throughput, making them efficient for
training models on large datasets.
3. Optimized for Matrix/Tensor Operations:
o Most deep learning workloads involve tensor computations (e.g., dot products,
convolutions), where GPUs outperform CPUs.

Decomposition – PCA and SVD

Decomposition methods are powerful mathematical tools for analyzing and simplifying data.
Principal Component Analysis (PCA) and Singular Value Decomposition (SVD) are two
commonly used techniques in machine learning and data science for dimensionality
reduction, data compression, and noise reduction.
1) PCA
Principal Component Analysis (PCA)
Principal Component Analysis (PCA) is a technique used to reduce the dimensionality of a
dataset while retaining as much variance as possible. It transforms the original data into a
new coordinate system where the axes (principal components) correspond to the directions of
maximum variance.
 Dimensionality Reduction: PCA reduces the number of features in the data by
projecting it onto a smaller subspace.

 Principal Components: Linear combinations of the original features, ranked by the

amount of variance they capture.

 Variance Explained: Each principal component explains a portion of the total

variance in the data.

Applications of PCA
1. Dimensionality Reduction:

o Reducing the number of features simplifies models, reduces overfitting, and

speeds up computation.
o Example: From 100 features to 10 principal components.
2. Data Visualization:
o Helps visualize high-dimensional data by projecting it into 2D or 3D space.
3. Noise Filtering:
o PCA removes noise by discarding components with low variance.
4. Feature Extraction:
o Derives new, uncorrelated features from the original data.
5. Preprocessing:
o Used in tasks like face recognition, image compression, and NLP.

Advantages of PCA
1. Reduces computational complexity by lowering dimensions.
2. Helps eliminate multicollinearity by creating uncorrelated features.
3. Enhances model performance by removing redundant or noisy features.
2) SVD
Singular Value Decomposition (SVD) is a matrix factorization technique used in linear
algebra. It decomposes a matrix into three distinct matrices that capture its intrinsic
properties. SVD has a wide range of applications, from dimensionality reduction to solving
linear equations, and is a foundational method in machine learning and data science.

Applications of SVD
1. Dimensionality Reduction:
2. Image Compression:

o SVD can compress images by storing only the largest singular values and
vectors.
3. Latent Semantic Analysis (LSA):

o In Natural Language Processing (NLP), SVD is used to discover latent

relationships between terms and documents.
4. Recommender Systems:

o SVD is used to decompose user-item matrices for collaborative filtering in

recommendation systems.
5. Solving Linear Systems:

o SVD can solve systems of linear equations, especially when the system is ill-
conditioned or has no unique solution.
6. Noise Filtering:
o SVD isolates noise in the lower singular values, allowing the data to be
reconstructed without it.

Advantages of SVD
1. Handles any m×nm \times n matrix.
2. Effective for noise reduction and dimensionality reduction.
3. Robust for ill-conditioned matrices.

Limitations of SVD
1. Computationally expensive for large matrices.
2. Not scalable for very high-dimensional data.