G E N E R AT I V E

LEARNING
ALGORITHMS
PRESENTERS

o ANNAY SAXENA
o ANISH KUMAR
o ANANT KUMAR
 W H AT A R E G E N E R AT I V E L E A R N I N G
ALGORITHMS?

Definition:

Generative learning algorithm models are designed to model how data is

generated from an underlying distribution.
They learn the joint probability distribution of the input data and labels.
Then generate new data points that resemble the original data.
Characteristics:

1. Learn the joint probability distribution of the data.

2. Can generate new data samples.
3. Often use probabilistic models.
4. Can handle missing data.
G E N E R AT I V E V S .
D I S C R I M I N AT I V E
MODELS
 D I S C R I M I N AT I V E M O D E L S

Modelling the Decision Boundary

Tries to create decision boundary based on different features of

input.

Learn the Conditional Probability of label y given input x: P(y | x)

Examples: Logistic Regression, Support Vector Machines

G E N E R AT I V E M O D E L S

Modelling the data Distribution.

The model will try to find the distribution of both classes in a n-

dimensional plane.

Learn joint probability P(x, y) and can generate new samples from the
data distribution

Examples: Naive Bayes, Gaussian Mixture Models , Hidden Markov

Models
 T Y P E S O F G E N E R AT I V E
MODELS

Non- Deep
Parametric
Parametric Generative
Models
Models Models
 Parametric generative
PA R A M E T R I C models are statistical
G E N E R AT I V E models that assume the
underlying data
MODELS distribution can be
described using a fixed set
of parameters.
These models learn the
parameters from the data
and use them to generate
new samples.
 T Y P E S O F PA R A M E T R I C
G E N E R AT I V E M O D E L S

Gaussian Hidden
Bayesian
Mixture Markov
Networks
Models Models
G AU SS I A N
MIXTURE
MODELS
What is GMM?
A Gaussian Mixture Model assumes that the data is generated from a mixture of several
Gaussian distributions, where each Gaussian distribution represents a cluster or a
component of the data.
Concept of GMM
• Data is divided into multiple Gaussian distributions.
• Each Gaussian distribution has its own mean, variance, and weight.
• The sum of weights equals 1.
 G AU SS I A N
MIXTURE
MODELS

Mathematical Formulation
• K is the number of Gaussian components.
• πk is the mixing coefficient for the kth Gaussian.
• N is the Gaussian distribution with mean μk
and covariance Σk.
GAUSSIAN MIXTURE MODELS

Applications of Gaussian Mixture Models (GMM)

1. Clustering: Groups data into clusters with different shapes (soft

clustering).
2. Density Estimation: Estimates the underlying data distribution.

3. Anomaly Detection: Identifies outliers based on low-probability regions.

4. Image Segmentation: Classifies pixels into different regions in images.

5. Speech Recognition: Models phonemes or sound patterns for feature

extraction.
 What is HMM
• A Hidden Markov Model assumes that a
system is modelled as a Markov process with
hidden states
Key Components
HIDDEN • States : A finite set of hidden states that the
model transitions between
MARKOV • Observations: A sequence of observations
MODELS that are probabilistically linked to the hidden
states
• Transition Probabilities: Probability of
transitioning from one hidden state to another
• Emission Probabilities: Probability of an
observation being emitted from a hidden state
• Initial State Distribution: Probabilities of
starting in each hidden state
 H I DDE N M A R KOV M O DE LS
Mathematical Representation

• Transition Matrix A: A [i, j] is the probability of moving from state i

to state j.
• Emission Matrix B: B[i, j] is the probability of observing k from
hidden state i.
Learning Algorithms

• Forward-Backward Algorithm: Used to compute the probabilities of

a sequence of states given a sequence of observations
• Viterbi Algorithm: Used to find the most likely sequence of hidden
states given the observed sequence
• Baum-Welch Algorithm: A special case of the EM algorithm for
HMMs, used to estimate model parameters
A P P L I C AT I O N S
OF HIDDEN
M A R KOV M O D E L

• Speech Recognition: Each phoneme

corresponds to a hidden state, and the
acoustic signal corresponds to observations
• Bioinformatics: Modelling sequences of
genes or proteins
• Stock Market Prediction: States can
represent market conditions, and the
observations are price movements
B AY E S I A N
NETWORKS
• What is a Bayesian Network
• A Bayesian Network is a directed acyclic graph that represents a set of random
variables and their conditional dependencies.

• Key Concepts
• Conditional Independence: The graph structure encodes which variables
are independent of others given their parents in the graph.
• Joint Probability Distribution: The joint distribution of all variables can be
factored as the product of the conditional probabilities of each variable given
its parents in the graph
B AY E S I A N Learning in Bayesian
NETWORK Networks
Structure Learning: Involves finding
S the best graph structure that
represents the relationships among
variables
Parameter Learning: Once the
structure is fixed, parameters of the
conditional distributions are learned
using methods like maximum likelihood
estimation or Bayesian inference
B AY E S I A N
NETWORK Advantages Applications
Interpretability: The graphical Medical Diagnosis: Can
S structure provides insights into model the relationships
the relationships between between symptoms and
variables diseases
Efficient algorithms: exist for Decision Support Systems:
querying the network, making it Used in expert systems for
useful for decision support reasoning under uncertainty
systems
N O N - PA R A M E T R I C
G E N E R AT I V E
MODELS
• Definition: Non-parametric generative models do
not assume a fixed number of parameters or a fixed
form of the data distribution
• Key Idea: Unlike parametric models, the number of
parameters in non-parametric models increases as
more data is available, providing greater flexibility
• Popular Non- Parametric Models
• Kernel Density Estimation
• k-Nearest Neighbors
KERNEL DENSITY
E S T I M AT I O N
• Definition: KDE is a method used to estimate the probability density function of a random
variable non-parametrically
• Key Concepts
• The data is smoothed by summing up contributions from "kernels" placed at each data point
• The width of the kernels is controlled by a bandwidth parameter, which defines the
smoothness of the density estimate
• Mathematical Representation:

Where,
• n is the number of data points
• K is the kernel function, typically a Gaussian
• h is the bandwidth, which controls the smoothness of the estimate
 K E R N E L D E N S I T Y E S T I M AT I O N

Advantages Limitations Applications

• No need to assume • Choosing the right • Density estimation
a specific form of bandwidth is for continuous
distribution crucial; too small data
• Can capture can lead to • Data smoothing
complex and multi- overfitting, while and visualization
modal distributions too large can
underfit
• Computational
cost increases with
more data
K- N E A R E S T
NEIGHBORS
• Definition: k-NN is a non-parametric method used for
classification and regression, based on the proximity of
data points to each other
• Key Concepts:
• For classification, an unlabeled point is assigned
the label that is most common among its k nearest
neighbors
• For regression, the value of the point is predicted
as the average of the values of its nearest
neighbors
 K- N E A R E S T
NEIGHBORS
• Mathematical Representation
• Classification:

• Regression:

• Advantages
• Simple to implement and understand
• Non-parametric and flexible, adapting to data
distribution
K- N E A R E S T
NEIGHBORS
• Limitations
• Sensitive to the choice of k, and performance
can vary based on the distance metric used
• Computationally expensive for large
datasets, as distances need to be calculated
for each point
• Applications
• Pattern recognition and image classification
• Recommendation systems
 Definition: Deep Generative Models use
neural networks to generate new data
samples that resemble the training data

DEEP
G E N E R AT I V Key Features
Can generate new instances that
Learn the underlying distribution of
look like they come from the
E MODELS data
original dataset

Popular Deep Generative Models

Variational Autoencoders Generative Adversarial Networks
V A R I AT I O N A L
A U T O E N C O D E R S ( VA E
S)
• Definition: VAE is a generative model that combines
probabilistic graphical models with deep learning
• Concept:
• Learn a latent space representation
• Learns the distribution of the latent space to generate
new data
• Encoder and Decoder architecture

• Key Idea: Unlike standard autoencoders, VAEs model the

latent space as a probability distribution, allowing them to
generate new data by sampling from the latent space
• Applications:
• Image generation, anomaly detection, data compression
 How GANs Work:
• Two competing networks (generator and
discriminator).
• Generator tries to create data similar to
G E N E R AT I V E the training data, while the discriminator
ADVERSARIA attempts to distinguish real from fake data.
L NETWORKS • Training process involves a min-max
(GAN) game.
Applications of GANs:
• Synthetic data generation.
• Art and design, drug discovery, video
prediction.
A P P L I C AT I O N S O F G E N E R AT I V E
LEARNING ALGORITHMS

Data Augmentation: Generative models can create additional data for training

Anomaly Detection: GMM and other generative models can model "normal" data
and detect deviations

Natural Language Processing : Used for generating text or language models

Image Synthesis and Super-Resolution: Generating high-resolution images

from low-resolution data

Drug Discovery: Generative models are used to generate new molecular

structures
 CHALLENGES OF
G E N E R AT I V E M O D E L S

High Computational Complexity: Training generative

models can be computationally expensive

Mode Collapse : The generator may produce limited

variations of outputs, ignoring parts of the data
distribution

Overfitting: Especially in VAEs and other generative

models with limited data
 F U T U R E O F G E N E R AT I V E
MODELS

Advances in AI: More powerful generative models

using transformers and diffusion models

Ethical Considerations: Synthetic data generation and

its impact on society

AI-driven Creativity: Generative models pushing

boundaries in art, design, and media
CONCLUSION

• Generative learning algorithms play a crucial

role in various ML applications
• Powerful tools for data generation, anomaly
detection, and creative processes
• Challenges remain, but the field is rapidly
advancing
THANK YOU