Scribd
Upload
2 views9 pages

Incomplete 1

Linear Discriminant Analysis (LDA) is a supervised dimensionality reduction technique that aims to maximize class separability while minimizing within-class variability. It operates under the assumptions of Gaussian distribution, equal covariance matrices for classes, and linear separability in the projected space. The process involves projecting data onto a lower-dimensional space using eigenvalue problems to determine optimal feature combinations for classification.

Uploaded by

PRAKHAR JHA
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
2 views9 pages

Incomplete 1

Linear Discriminant Analysis (LDA) is a supervised dimensionality reduction technique that aims to maximize class separability while minimizing within-class variability. It operates under the assumptions of Gaussian distribution, equal covariance matrices for classes, and linear separability in the projected space. The process involves projecting data onto a lower-dimensional space using eigenvalue problems to determine optimal feature combinations for classification.

Uploaded by

PRAKHAR JHA
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 9

What Is LDA?

LDA (Linear Discriminant Analysis) is a supervised dimensionality reduction technique


based on Fisher’s Linear Discriminant, a statistical method introduced by Sir Ronald Fisher
in the 1930s and later extended by C. R. Rao for multi-class problems. LDA focuses on
projecting data in a way that best separates multiple known classes.

This method finds a linear combination of features that maximizes between-class


separability while minimizing within-class variability, enabling better class discrimination
in reduced dimensions.

Assumptions of LDA
1. The input dataset has a Gaussian distribution.

Ideal Gaussian Distribution Gaussian data fit for LDA

For checking whether the given data is Gaussian, perform the Shapiro-Wilk Test.

2. Each class has the same covariance matrix. For checking covariance, either evaluate
the covariance matrix of all features or check using Box’s M Test of equality of
covariance matrices.
3. The Classes are linearly separable in the projected space — not necessarily in the
original space

.
Intuition Behind LDA
The core idea behind Linear Discriminant Analysis (LDA) is to find the best projection that
maximizes the separation between different classes while minimizing the spread within each
class.

Maths and Formulation in LDA


• Let X ∈ Rn×d : the data matrix with n samples and d features.
• Let y ∈ {1, 2, … ,K}: the class labels
• There are K distinct classes (we’ll start with 2-class case for intuition, then
generalize)

We want to project the data onto a lower-dimensional space such that:

• Between-class distance is maximized


• Within-class scatter is minimized

Step 1: Transform the axises

• Consider a unit vector v ∈ Rd . Project each sample onto v:

ai = vTxi
Step 2: Calculate mean and variance of projected class
• Let μi be the mean of class i after its projection.

• Let si be the standard deviation about its mean of class i after its projection.
Variance is:
Step 3: Calculate the inter class distance and within class variance
• Inter class distance: maxv (µ1 - µ2)2
• Within class variance: minv (s12 + s22)

Therefore, our aim is to calculate an optimal value of v such that:

•
• The total within-class scatter of the two classes in the projection space is

vT Sb v
• Equation 1 becomes: maxv
vT Sw v

This result is valid for more than 2 classes also. Here’s the proof:
Step 4: Rayleigh Quotient Corollary 0.5 gives a direct simplification of
such forms to the eigenvalue equation:
Sbv = λSwv
To solve this, we transform it into a regular eigenvalue problem:

Sw-1Sbv = λv
Now this is a simple eigen value problem which can be used to easily find all the
eigen values.
For f features and c label classes, we will get f eigen values. However, we choose
only min[f,c-1] eigen values chosen in descending order, all other values are
discarded.
In essence, it is generally seen that the first two eigen values carry the most
importance seen by its
explained ratio:

However, a good practise is to check this always.

For 5 features and 3 classes


Scikit-learn Python code for LDA

You might also like

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy