Module 3: Advanced ML Algorithms and Hardware

Design Optimization
Syllabus
Ensemble Methods: Random Forest, Gradient Boosting. Dimensionality
Reduction Techniques: PCA, t-SNE. Model Evaluation and
Hyperparameter Tuning.

VLSI Application: Optimization of hardware design parameters using

ML algorithms.
Overview
• Ensemble methods like Random Forest (RF) and Gradient Boosting (GB) improve model
accuracy by combining multiple learners, with RF reducing overfitting through averaging
multiple decision trees and GB sequentially refining predictions to minimize errors.
• Dimensionality reduction techniques such as Principal Component Analysis (PCA),
which transforms correlated features into orthogonal components to retain maximum
variance, and t-SNE(t-Distributed Stochastic Neighbor Embedding), a nonlinear method
for visualizing high-dimensional data, help simplify complex datasets.
• Model evaluation relies on metrics like accuracy, precision, recall, F1-score, and cross-
validation, while hyperparameter tuning techniques such as Grid Search, Random
Search, and Bayesian Optimization optimize model performance.
• In VLSI hardware design, ML algorithms optimize power efficiency, performance, and
layout by predicting power consumption, detecting faults, and automating circuit
optimization through neural networks and reinforcement learning, making chip design
more efficient and cost-effective.
 Ensemble method
• An ensemble method is a machine learning technique that combines multiple models to
improve overall performance, accuracy, and robustness compared to a single model.
• The idea is that by aggregating the predictions of multiple models, the ensemble reduces
errors, avoids overfitting, and enhances generalization. Ensemble methods can be broadly
categorized into bagging, boosting, and stacking.

➢ Bagging (Bootstrap Aggregating): Trains multiple independent models on random subsets of the dataset
and averages (for regression) or takes a majority vote (for classification). Example: Random Forest, which
builds multiple decision trees and averages their outputs.
➢ Boosting: Sequentially trains weak models, where each new model corrects the errors of the previous
ones, gradually improving performance. Examples: Gradient Boosting, AdaBoost, XGBoost.
➢ Stacking: Combines predictions from multiple models using a meta-model that learns how to best
combine their outputs.

• Ensemble methods are widely used in real-world applications like fraud detection, medical
diagnosis, and recommendation systems due to their ability to enhance predictive power
and reduce variability.
 Random Forest Algorithm
• Random Forest is a well-known machine learning algorithm that uses the supervised
learning method.
• In machine learning, it can be used for both classification and regression problems.
• It is based on ensemble learning, which is a method of combining multiple classifiers
to solve a complex problem and improve the model's performance.
• Random Forest is a classifier that combines a number of decision trees on different
subsets of a dataset and averages the results to improve the dataset's predictive
accuracy.
• Instead of relying on a single decision tree, the random forest takes the predictions
from each tree and predicts the final output based on the majority votes of
predictions.
Basic Structure of RF
Working of RF algorithm
Decision Tree vs Random Forest
Decision tree Random forest

An algorithm that generates a tree-like set of rules for An algorithm that combines many decision trees to
classification or regression. produce a more accurate outcome.

When a dataset with certain features is ingested into a Builds decision trees on random samples of data and
decision tree, it generates a set of rules for prediction. averages the results.

High dependency on the initial data set; low accuracy of

High precision and reduced bias of results.
prediction in the real world as a result.

Prone to overfitting because of the possibility to adapt The use of many trees allows the algorithm to avoid
to the initial data set too much. and/or prevent overfitting.
Gradient Boosting
• Gradient Boosting is a popular boosting algorithm in machine learning used
for classification and regression tasks.
• Boosting is one kind of ensemble Learning method which trains the model
sequentially and each new model tries to correct the previous model.
• Gradient Boosting is a powerful boosting algorithm that combines several
weak learners into strong learners, in which each new model is trained to
minimize the loss function such as mean squared error or cross-entropy of
the previous model using gradient descent.
• In each iteration, the algorithm computes the gradient of the loss function
with respect to the predictions of the current ensemble and then trains a
new weak model to minimize this gradient.
• The predictions of the new model are then added to the ensemble, and the
process is repeated until a stopping criterion is met.
 Working of gradient boosting
• The ensemble consists of M trees. Tree1 is trained using the
feature matrix X and the labels y. The predictions
labeled y1(hat) are used to determine the training set residual
errors r1.
• Tree2 is then trained using the feature matrix X and the
residual errors r1 of Tree1 as labels.
• The predicted results r1(hat) are then used to determine the
residual r2.
• The process is repeated until all the M trees forming the
ensemble are trained. There is an important parameter used in
this technique known as Shrinkage.
• Shrinkage refers to the fact that the prediction of each tree in
the ensemble is shrunk after it is multiplied by the learning rate
(eta) which ranges between 0 to 1.
• There is a trade-off between eta and the number of estimators,
decreasing learning rate needs to be compensated with 𝒀
𝒑𝒓𝒆𝒅 =𝒚𝟏 + 𝒆𝒕𝒂 ∗ 𝒓𝟏 + 𝒆𝒕𝒂 ∗ 𝒓𝟐 + ⋯ … . . + 𝒆𝒕𝒂 ∗ 𝒓𝑵
increasing estimators in order to reach certain model
performance. Since all trees are trained now, predictions can be
made.
 Gradient Boosting
Key Features
• Boosting: Unlike bagging,
boosting focuses on learning
from previous mistakes by giving Advantages
more weight to misclassified • Handles missing values well
instances. • Works well with structured/tabular
• Weak Learners: Uses shallow data
decision trees (stumps) to • Provides feature importance for
gradually refine predictions. interpretation
• Learning Rate: Controls the • Can handle both regression and
contribution of each weak classification tasks
learner, preventing overfitting.
Disadvantages
• Computationally expensive
• Sensitive to hyper parameter tuning
• Leads to overfitting if not properly
regularized
Dimensionality Reduction Techniques
• While working with machine learning models, we often encounter
datasets with a large number of features. These datasets can lead to
problems such as increased computation time and overfitting.
• To address these issues, we use dimensionality reduction techniques.

Dimensionality reduction is the process of reducing the

number of features (or dimensions) in a dataset while
retaining as much information as possible.
 Working Dimensionality Reduction Techniques

• On the left, data points exist in a 3D space (X, Y,

Z), but the Z-dimension appears unnecessary
since the data primarily varies along the X and Y
axes. The goal of dimensionality reduction is to
remove less important dimensions without losing
valuable information.
• On the right, after reducing the dimensionality,
the data is represented in lower-dimensional
spaces. The top plot (X-Y) maintains the
meaningful structure, while the bottom plot (Z-Y)
shows that the Z-dimension contributed little
useful information.
• This process makes data analysis more efficient,
improving computation speed and visualization
while minimizing redundancy
 What is Feature selection and Feature Extraction?
Feature Selection Feature Extraction
Feature selection chooses the most relevant features Feature extraction involves creating new features by combining
from the dataset without altering them. It helps or transforming the original features. There are several
remove redundant or irrelevant features, improving methods for feature extraction stated above in the introductory
model efficiency. There are several methods for feature part which is responsible for creating and transforming the
selection including features.
Filter methods PCA is a popular technique that projects the original features
Wrapper methods onto a lower-dimensional space while preserving as much of
Embedded methods. the variance as possible.
• Filter methods rank the features based on their
relevance to the target variable.
• Wrapper methods use the model performance as
the criteria for selecting features.
• Embedded methods combine feature selection with
the model training process.
Advantages of Dimensionality Reduction

• Faster Computation: With fewer features, machine learning

algorithms can process data more quickly. This results in faster model
training and testing, which is particularly useful when working with
large datasets.
• Better Visualization: As we saw in the earlier figure, reducing
dimensions makes it easier to visualize data, revealing hidden
patterns.
• Prevent Overfitting: With fewer features, models are less likely to
memorize the training data and overfit. This helps the model
generalize better to new, unseen data, improving its ability to make
accurate predictions.
Principal Component Analysis (PCA)-Linear Method
• Principal Component Analysis (PCA) is a linear dimensionality
reduction technique that transforms a high-dimensional dataset into
a lower-dimensional space while preserving as much variance as
possible.
• It achieves this by finding a new set of orthogonal axes (principal
components) along which the data varies the most.
How PCA Works for Dimensionality Reduction?
It works by transforming high-dimensional data into a lower-
dimensional space while maximizing the variance (or spread) of the
data in the new space.
PCA is an unsupervised learning algorithm, meaning it doesn’t require
prior knowledge of target variables. It’s commonly used in
exploratory data analysis and machine learning to simplify datasets
without losing critical information.

Note: It prioritizes the directions where the data varies

the most because more variation = more useful information.
Steps in PCA
1. Standardization of Data
• Since PCA is sensitive to different feature scales, we first standardize the dataset.
• Convert all features to have zero mean and unit variance using:
𝑋−𝜇
𝑋𝑠𝑐𝑙𝑎𝑒𝑑 =
𝜎
where μ is the mean and σ(sigma) is the standard deviation.

2. Compute the Covariance Matrix

• The covariance matrix captures relationships between different features.
• Given a dataset with n features, the covariance matrix is:
σ𝑛𝑖=1 𝑥1 − 𝑥1 𝑥2 − 𝑥2
𝑐𝑜𝑣 𝑥1 , 𝑥2 =
𝑛−1
The value of covariance can be positive, negative, or zeros.
• Positive: As the x1 increases x2 also increases.
• Negative: As the x1 increases x2 also decreases.
• Zeros: No direct relation.
Steps in PCA
3: Find the “Magic Directions” (Principal Components)
• PCA identifies new axes (like rotating a camera) where the data spreads out the most:
• 1st Principal Component (PC1): The direction of maximum variance (most spread).
• 2nd Principal Component (PC2): The next best direction, perpendicular to PC1, and so on.
• These directions are calculated using Eigenvalues and Eigenvectors.
• For a square matrix A, an eigenvector X (a non-zero vector) and its corresponding eigenvalue λ (a scalar) satisfy:
𝐀𝐗 = 𝛌𝐗
This means:
• When A acts on X, it only stretches or shrinks X by the scalar λ.
• The direction of X remains unchanged (hence, eigenvectors define “stable directions” of A).
The Characteristics equation is :
𝐀 − 𝛌 𝐗=0
𝐀 − 𝛌𝑰 𝑿 = 𝟎
where I is the identity matrix of the same shape as matrix A. And the above conditions will be true only if (A–λI) will
be non-invertible (i.e. singular matrix). That means,
𝐀 − 𝛌𝑰 =0
• In PCA, the covariance matrix C (from Step 2) acts as matrix A.
• Eigenvectors of C are the principal components (PCs).
• Eigenvalues represent the variance captured by each PC.
 Steps in PCA
4: Pick the Top Directions & Transform Data
• Keep only the top 2–3 directions (or enough to capture ~95% of the variance).
• Project the data onto these directions to get a simplified, lower-dimensional version.

• PC₁ (First Principal Component): The direction along which the data has the maximum variance. It captures the most
important information.
• PC₂ (Second Principal Component): The direction orthogonal (perpendicular) to PC₁. It captures the remaining
variance but is less significant.
• Now, The red dashed lines indicate the spread (variance) of data along different directions . The variance along PC₁ is
greater than PC₂, which means that PC₁ carries more useful information about the dataset.
• The data points (blue dots) are projected onto PC₁, effectively reducing the dataset from two dimensions (Radius,
Area) to one dimension (PC₁).
• This transformation simplifies the dataset while retaining most of the original variability.
t-Distributed Stochastic Neighbor Embedding (t-SNE) -
Nonlinear Method
In t-Distributed Stochastic Neighbor Embedding (t-SNE), the "t" stands for the Student’s t-
distribution, which is used to model the pairwise similarities in the lower-dimensional
space.
Why Use the t-Distribution?
• In the high-dimensional space, t-SNE models pairwise similarities using a Gaussian
(normal) distribution.
• However, when mapping to a lower-dimensional space, using a Gaussian would cause
crowding because high-dimensional distances don’t directly translate well to lower
dimensions.
• The t-distribution allows distant points to stay apart, preventing crowding in the low-
dimensional space.
• Ensures that well-separated clusters remain visually distinct. This choice of distribution is
what makes t-SNE more effective than PCA for visualizing high-dimensional data.
t-Distributed Stochastic Neighbor Embedding
(t-SNE) - Nonlinear Method
• T-distributed Stochastic Neighbor Embedding (t-SNE) is
a nonlinear dimensionality reduction technique and it is suited
for visualizing high-dimensional data in a lower-dimensional
space typically in 2D or 3D. It is a widely used dimensionality
reduction technique and in this article we will learn about it.
• Dimensionality reduction is a process that simplifies complex
dataset by combining similar or correlated features. It helps in
improving analysis and computational efficiency.
• t-SNE is a dimensionality reduction technique that uses a
randomized, non-linear approach to reduce the dimensionality of
data. Unlike linear methods such as Principal Component
Analysis (PCA), t-SNE focuses on preserving the local structure
and pattern of the data.
• It is especially effective for visualizing high-dimensional datasets
as it keeps similar data points close to each other in the lower-
dimensional space making it easier to see patterns and clusters.
• This ability to retain the local structure of the dataset helps in
exploring and understanding complex, high-dimensional data.
Visualizing the data in 2D or 3D can provide us valuable insights
into the relationships between different data points.
t-SNE Working
• t-SNE works by looking at the similarity between data points in the high-dimensional space. The
similarity is computed as a conditional probability. It calculates how likely it is that one data point
would be near another.
• Once the similarities are calculated, t-SNE tries to keep similar points close when it reduces the data to
lower dimensions (like 2D or 3D). The goal is to make sure that points that are close in the original
space stay close in the lower-dimensional space, preserving the structure of the data.
Step 1: Compute Pairwise Similarities in High-Dimensional Space
For each data point 𝑥𝑖 ​, compute the probability that another point 𝑥𝑗 is its neighbor using a Gaussian
(normal) distribution: 𝟐
− 𝒙𝒊 −𝒙𝒋 ൘
𝒆 𝟐𝝈𝟐
𝑷𝒊𝒋 = − 𝒙𝒌 −𝒙𝒍 𝟐൘
σ𝒌≠𝒍 𝒆 𝟐𝝈𝟐

where:
• 𝑷𝒊𝒋 ​ represents how similar 𝒙𝒊 and 𝒙𝒋 ​ are in high-dimensional space.
• σ(sigma) perplexity parameter controls the neighbourhood size.
• The sum ensures that probabilities are normalized.
• The probability 𝑷𝒊𝒋 ​ is high for similar points and low for dissimilar points.
t-SNE Working
Step 2: Compute Pairwise Similarities in Low-Dimensional Space
• In the low-dimensional space (2D or 3D), we compute a similar probability ​, but this time using a
Student’s t-distribution with 1 degree of freedom (also called a Cauchy distribution):
2 −1
1 + 𝑦𝑖 − 𝑦𝑗
𝑄𝑖𝑗 = 2 −1
σ𝑘≠𝑙 1 + 𝑦𝑘 − 𝑦𝑙
where:
• 𝑄𝑖𝑗 ​ represents how similar points are in the lower-dimensional space.
• The t-distribution has heavier tails than a Gaussian, preventing points from crowding together.
t-SNE Working
Step 3: Minimize the Kullback-Leibler (KL) Divergence
• The goal is to match the similarity distributions 𝑃𝑖𝑗 ​ and 𝑄𝑖𝑗 ​.
• This is done by minimizing the KL divergence, which measures how different the two distributions are:
𝑃𝑖𝑗
𝐾𝐿 𝑃 ∥ 𝑄 = ෍ 𝑃𝑖𝑗 𝑙𝑜𝑔
𝑄𝑖𝑗
𝑖≠𝑗
• The optimization is performed using gradient descent, where points are adjusted iteratively to make 𝑄𝑖𝑗 ​ as
close to 𝑃𝑖𝑗 ​ as possible.

Step 4: Update the Points and Repeat

• t-SNE adjusts the positions of points in the low-dimensional space to reduce KL divergence.
• Points that were similar in high-dimensional space move closer together, while dissimilar points move apart.
• This process continues until convergence.
Key Differences Between PCA and t-SNE
t-SNE (t-Distributed Stochastic Neighbor
Feature PCA (Principal Component Analysis)
Embedding)
Type Linear Nonlinear

Purpose Feature extraction & dimensionality reduction Data visualization

Works By Finding orthogonal axes that maximize variance Preserving local neighborhood similarities

Mathematical Probabilistic similarity mapping (KL divergence

Eigen decomposition of covariance matrix
Basis minimization)
Components can be interpreted in relation to original Components are abstract and not directly
Interpretability
features interpretable
Computationally efficient, works well on large Computationally expensive, not suitable for
Scalability
datasets large datasets

Data clustering, exploratory visualization of

Use Case Feature selection, noise reduction, data compression
high-dimensional data
Evaluation and Hyperparameter Tuning.
Model evaluation and hyperparameter tuning are crucial steps in machine learning to ensure the
model performs optimally and generalizes well to unseen data.
• For Classification Problems, the parameters of confusion matrix like: Accuracy, Precision, Recall,
F1 Score, Selectivity, Sensitivity.
• For regression problems, the parameters like:
➢ Mean Absolute Error (MAE), Measures absolute differences between actual and predicted values.
1
𝑀𝐴𝐸 = ෍ 𝑦𝑖 − 𝑦ഥ𝑖
𝑛
➢ Mean Squared Error (MSE) / Root Mean Squared Error (RMSE), gives more weight to larger errors.
2
1
𝑀𝑆𝐸 = ෍ 𝑦𝑖 − 𝑦ഥ𝑖
𝑛
➢ R² Score (Coefficient of Determination), Measures how well the model explains variance in the data.
σ 𝑦𝑖 − 𝑦ഥ𝑖 2
2
𝑅 =1− 2
σ 𝑦𝑖 − 𝑦ෝ𝑖
Hyperparameter tuning
• Hyperparameter tuning is the process of selecting the optimal values for
a machine learning model’s hyperparameters.
• Hyperparameters are settings that control the learning process of the model,
such as the learning rate, the number of neurons in a neural network, or the
kernel size in a support vector machine.
• The goal of hyperparameter tuning is to find the values that lead to the best
performance on a given task.
• In the context of machine learning, hyperparameters are configuration variables
that are set before the training process of a model begins.
• They control the learning process itself, rather than being learned from the data.
• Hyperparameters are often used to tune the performance of a model, and they
can have a significant impact on the model’s accuracy, generalization, and other
metrics.
Different Ways of Hyperparameters Tuning
Neural networks have several essential hyperparameters that need to be adjusted, including:
• Learning rate: This hyperparameter controls the step size taken by the optimizer during each iteration of
training. Too small a learning rate can result in slow convergence, while too large a learning rate can lead to
instability and divergence.
• Epochs: This hyperparameter represents the number of times the entire training dataset is passed through
the model during training. Increasing the number of epochs can improve the model’s performance but may
lead to overfitting if not done carefully.
• Number of layers: This hyperparameter determines the depth of the model, which can have a significant
impact on its complexity and learning ability.
• Number of nodes per layer: This hyperparameter determines the width of the model, influencing its
capacity to represent complex relationships in the data.
• Architecture: This hyperparameter determines the overall structure of the neural network, including the
number of layers, the number of neurons per layer, and the connections between layers. The optimal
architecture depends on the complexity of the task and the size of the dataset
• Activation function: This hyperparameter introduces non-linearity into the model, allowing it to learn
complex decision boundaries. Common activation functions include sigmoid, tanh, and Rectified Linear Unit
(ReLU).
Hyperparameter Tuning techniques
Models can have many hyperparameters and finding the best combination of parameters can be
treated as a search problem. The two best strategies for Hyperparameter tuning are:
➢ Grid Search CV
➢ Randomized Search CV
➢ Bayesian Optimization

1. GridSearchCV
• Grid search can be considered as a “brute force” approach to hyperparameter optimization. We
fit the model using all possible combinations after creating a grid of potential discrete
hyperparameter values. We log each set’s model performance and then choose the combination
that produces the best results. This approach is called Grid Search CV, because it searches for the
best set of hyperparameters from a grid of hyperparameters values.
• An exhaustive approach that can identify the ideal hyperparameter combination is grid search.
But the slowness is a disadvantage. It often takes a lot of processing power and time to fit the
model with every potential combination, which might not be available.
• For example: if we want to set two hyperparameters C and Alpha of the Logistic Regression
Classifier model, with different sets of values. The grid search technique will construct many
versions of the model with all possible combinations of hyperparameters and will return the best
one.
• As in the image, for C = [0.1, 0.2, 0.3, 0.4, 0.5] and Alpha = [0.1, 0.2, 0.3, 0.4]. For a combination
of C=0.3 and Alpha=0.2, the performance score comes out to be 0.726(Highest), therefore it is
selected.
2. Randomized Search CV
• As the name suggests, the random search method selects values at random as
opposed to the grid search method’s use of a predetermined set of numbers.
• Every iteration, random search attempts a different set of hyperparameters and
logs the model’s performance.
• It returns the combination that provided the best outcome after
several iterations. This approach reduces unnecessary computation.
• Randomized Search CV solves the drawbacks of Grid Search CV, as it goes through
only a fixed number of hyperparameter settings.
• It moves within the grid in a random fashion to find the best set of
hyperparameters.
• The advantage is that, in most cases, a random search will produce a comparable
result faster than a grid search.
3. Bayesian Optimization
• Grid search and random search are often inefficient because they evaluate
many unsuitable hyperparameter combinations without considering the
previous iterations’ results.
• Bayesian optimization, on the other hand, treats the search for optimal
hyperparameters as an optimization problem.
• It considers the previous evaluation results when selecting the next
hyperparameter combination and applies a probabilistic function to choose
the combination that will likely yield the best results.
• This method discovers a good hyperparameter combination in relatively
few iterations.
• Data scientists use a probabilistic model when the objective function is
unknown. The probabilistic model estimates the probability of a
hyperparameter combination’s objective function result based on past
evaluation results.
Key Differences Between GridSearchCV, RandomizedSearchCV, and
Bayesian Optimization
Feature GridSearchCV RandomizedSearchCV Bayesian Optimization

Exhaustive search over all possible Randomly selects parameter Uses probabilistic models to find the
Search Method
combinations combinations best parameters

Efficiency Computationally expensive More efficient than GridSearchCV Most efficient

Intelligent trade-off between

Exploration vs. Exploitation Exhaustive exploration Random exploration
exploration and exploitation

Medium (faster, but may miss Low (fast, finds optimal values
Time Complexity High (slow for large parameter grids)
optimal values) efficiently)
Large search spaces, approximate
Works Well For Small parameter grids, exact tuning Complex, high-dimensional spaces
tuning

Scalability Poor for large search spaces Better scalability Excellent scalability

Some domain knowledge helps for

Requires Prior Knowledge? No, but needs a predefined grid No, but requires a range of values
setting priors

Parallel Processing? Yes Yes Yes (in some implementations)

VLSI Application: Optimization of hardware
design parameters using ML algorithms.
Optimization of hardware design parameters using
ML algorithms.
• Machine Learning (ML) is increasingly used in VLSI (Very Large-Scale
Integration) design, FPGA optimization, and circuit design to improve
performance, reduce power consumption, and optimize other
hardware parameters.
• Why Use ML for Hardware Optimization?
Traditionally, hardware design optimization involves manual tuning or
heuristic-based methods. However, ML can:
➢Automate parameter tuning for circuits and chips.
➢Speed up simulations and reduce computation cost.
➢Find optimal trade-offs between power, performance, and area (PPA).
➢Predict performance for unseen hardware configurations.
 ML-Based Optimization Techniques
A. Regression-Based Parameter Prediction
It Predict optimal hardware parameters (e.g., transistor sizing, clock speed, power consumption)
using regression models.
The Algorithms are:
• Linear Regression
• Decision Trees
• Random Forest
• Neural Networks
B. Reinforcement Learning (RL) for Hardware Tuning
It use an RL agent to iteratively optimize circuit parameters.
• How It Works:
• The RL agent chooses a hardware design configuration.
• It simulates or tests the design.
• It receives rewards based on performance metrics (e.g., power efficiency).
• Over time, the model learns the best design choices.
ML-Based Optimization Techniques
C. Evolutionary Algorithms (GA, PSO) for Circuit Design
It Use genetic algorithms (GA) or particle swarm optimization (PSO) to evolve an optimal hardware
configuration.
• How It Works:
• Starts with random solutions (initial hardware parameters).
• Applies mutation and crossover to generate better designs.
• Evaluates the fitness function (e.g., power, area, speed).
• Continues evolving until an optimal design is found.

D. Deep Learning for Hardware Synthesis

It Use Neural Networks to model and optimize complex circuit behaviours.
• Application Areas:
• Power Estimation: Predicting power usage based on design inputs.
• RTL-to-GDSII Optimization: Automating layout generation.
• Thermal Modeling: Predicting heat distribution in chips.