CSIT332 - Principles of Machine Learning

Assignment 1
Prithvi Dhyani
November 5, 2024

1 Introduction and Data Exploration

Our goal in this project is to explore four supervised machine learning algorithms(Decision Tree,
Random Forest, KNN, Neural Network) on three different datasets. Before we begin any kind of mod-
eling, lets start by exploring the three datasets. Please note that each dataset contains exactly 500
points(350 for training, 150 for testing).

1.1 Linearly Separable Dataset

The above dataset is generated by two Gaussian clusters(σ “ 1), and each cluster has a randomly
selected mean. By visual inspection, it is clear that the two clusters are linearly separable. This
information is crucial as we will later create the hyper-parameter search grids with this in mind.

1
1.2 Two Moons(Non-Linearly Separable) Dataset

The above dataset is generated by some finite samples of the sets of points on two semi circles(upper
semi-circle centered at (0,0), lower semi-circle centered at (1, 0.5). These points are then shifted by some
noise from a standard normal distribution, and the noise itself is scaled by a factor of 0.1. Clearly, this
dataset isn’t linearly separable, and would hence require a non-linear decision boundary for classification.

2
1.3 Concentric Circles(Non-Linearly Separable) Dataset

The above dataset is generated by some finite samples of the sets of points on two concentric circles
with radii 1 and 2 respectively, both centered at (0, 0). Each point is shifted by some noise sampled
from a standard normal distribution, and the noise itself is scaled by a factor of 0.5. Once again, this
dataset is clearly not linearly separable.

2 Hyper-parameter Considerations
Recall that our goal is to train, tune and test 4 supervised learning algorithms, namely Decision
Tree, Random Forest, KNN and Neural Network. In order to compare the performance of models across
different datasets, we need to make sure that the search space(grid) for hyper-parameters is standardized
for each model across all 3 datasets. Hence, we need only define 4 unique hyper-parameter grids, for
each of the 4 models.

2.1 Decision Tree

In the case of the decision tree learning algorithm, we consider the following 4 hyper-parameters:

• criterion: This specifies whether the measure we are using to determine the best split is en-
tropy(information gain) or Gini impurity.

3
• max depth: This specifies the maximum depth that the decision tree can reach, in terms of
the number of splits. In this case, we consider the values tN one, 1, 2, 5, 10u. The reason for
considering such small values is that by visual examination of the datasets, and the fact that our
data is 2-dimensional, clearly 10 linear boundaries is more than enough to separate the two classes.

• min samples split: This specifies the minimum number of data-points that need to be present in
a node in order to split it. We consider the values t2, 4, 6u. Higher values are included to prevent
over-fitting by splitting nodes with few points.

• min samples leaf: This specifies the minimum number of data-points required in a node for it
to be a leaf node. We consider the values t1, 2, 4u.

2.2 Random Forest

In the case of the Random Forest learning algorithm, we consider the following 6 hyper-parameters:

– n estimators: This specifies the number of trees in the forest. We consider the values
t10, 25, 50u.
– max depth: This specifies the maximum depth of each tree. We consider the values
tN one, 1, 2, 5, 10u.
– min samples split: This specifies the minimum number of data-points that need to be
present in a node in order to split it. We consider the values t2, 4, 6u.
– min samples leaf: This specifies the minimum number of data-points required in a node
for it to be a leaf node. We consider the values t1, 2, 4u.
– criterion: This specifies whether the measure we are using to determine the best split is
Gini impurity or entropy (information gain). We consider the values t1 gini1 ,1 entropy 1 u.
– max features: This controls the number of features to consider at each split. We consider
the values t1, N oneu.
– bootstrap: This controls whether bootstrapping is used in the forest. We consider the value
tT rueu.

2.3 K Nearest Neighbors

In the case of the K Nearest Neighbors (KNN) learning algorithm, we consider the following 3
hyper-parameters:

– n neighbors: This specifies the number of neighbors to consider. We consider the values
t3, 5, 7, 9u.
– weights: This specifies the weighting scheme used. We consider the options t1 unif orm1 ,1 distance1 u.
– metric: This specifies the distance metrics for KNN. We consider the options t1 euclidean1 ,1 manhattan

4
2.4 Neural Network
In the case of the Neural Network learning algorithm, we consider the following 4 hyper-parameters:

– hidden sizes: This specifies the number of units in each hidden layer. We consider the
values t2, 5, 10u.
– learning rates: This specifies the learning rates for training. We consider the values
t0.01, 0.001u.
– num hidden layers: This specifies the number of hidden layers in the network. We consider
the values t1, 2, 3u.
– activation functions: This specifies the activation functions to use in the hidden layers.
We consider the options tReLU, Sigmoid, Tanhu.

3 Results Analysis
In this section, we analyze the results obtained from the different supervised machine learning
models on the datasets. To evaluate the performance of each model, we utilize several key metrics:
accuracy, precision, recall, and F1-score. These metrics provide insights into how well the models
perform, especially in the context of classification tasks.

3.1 Evaluation Metrics

– Accuracy measures the proportion of correctly classified instances among the total instances.
It is calculated using the formula:
TP ` TN
Accuracy “
TP ` TN ` FP ` FN
where TP is true positives, TN is true negatives, FP is false positives, and FN is false
negatives.
– Precision indicates the accuracy of the positive predictions. It is given by:
TP
Precision “
TP ` FP

– Recall (also known as sensitivity or true positive rate) measures the ability of a model to
identify all relevant instances. It is defined as:
TP
Recall “
TP ` FN

– F1-score is the harmonic mean of precision and recall, providing a single metric that balances
both concerns. It is computed as:
Precision ¨ Recall
F1 “ 2 ¨
Precision ` Recall

5
These metrics serve different purposes in our evaluation. Accuracy gives an overall measure of
performance, but it can be misleading in imbalanced datasets where one class is more prevalent.
Precision and recall provide a more nuanced view of performance, especially in scenarios where
the cost of false positives and false negatives differs significantly. The F1-score combines both
precision and recall into a single metric, making it useful when we need a balance between the
two.
We will use these metrics to evaluate and compare the performance of the Decision Tree, Random
Forest, KNN, and Neural Network models across the various datasets. By analyzing these metrics,
we can determine which model best suits the characteristics of each dataset and the specific
classification task at hand. The goal is to select a model that not only performs well on accuracy
but also maintains a high precision and recall, ensuring that our classification outcomes are both
reliable and relevant.

3.2 Linearly Separable Dataset

The results for the various models on the linearly separable dataset are summarized in Table 1.
As observed, all models achieved perfect accuracy, successfully classifying all instances correctly.
This outcome can be attributed to the simple nature of the dataset, which is characterized by
distinct Gaussian clusters that are easily separable.

Table 1: Results for the Linearly Separable Dataset

Model Best Parameters Accuracy Classification Report

precision: 1.00
Decision Tree criterion: gini, 1.0 recall: 1.00
max depth: None, f1-score: 1.00
min samples leaf: 1,
min samples split: 2
precision: 1.00
Random Forest bootstrap: True, 1.0 recall: 1.00
criterion: gini, f1-score: 1.00
max depth: None,
max features: 1,
min samples leaf: 1,
min samples split: 2,
n estimators: 10
precision: 1.00
KNN metric: euclidean, 1.0 recall: 1.00
n neighbors: 3, f1-score: 1.00
weights: uniform
precision: 1.00
Neural Network hidden size: 2, 1.0 recall: 1.00
learning rate: 0.01, f1-score: 1.00
num hidden layers: 1,
activation function:
ReLU

6
7
All four models managed to classify all instances accurately, demonstrating their effectiveness in
handling linearly separable data. The optimal sets of hyperparameters indicate that the models
leverage their respective strengths, such as the flexibility of the Decision Tree and Random Forest

8
algorithms and the adaptability of the KNN and Neural Network approaches. The simplicity of
the dataset allowed for straightforward parameter tuning, resulting in optimal performances across
the board.

3.3 Two Moons (Non-Linearly Separable) Dataset

The results for the various models on the Two Moons dataset are summarized in Table 2. Re-
markably, all models achieved perfect accuracy, successfully classifying all instances correctly. This
outcome highlights the models’ capabilities to learn the complex boundaries presented by the two
moons shape, which, despite being a non-linearly separable dataset, was effectively navigated by
each algorithm.

Table 2: Results for the Two Moons Dataset

Model Best Parameters Accuracy Classification Report

precision: 1.00
Decision Tree criterion: entropy, 1.0 recall: 1.00
max depth: None, f1-score: 1.00
min samples leaf: 1,
min samples split: 2
precision: 1.00
Random Forest bootstrap: True, 1.0 recall: 1.00
criterion: entropy, f1-score: 1.00
max depth: None,
max features: None,
min samples leaf: 1,
min samples split: 2,
n estimators: 50
precision: 1.00
KNN metric: euclidean, 1.0 recall: 1.00
n neighbors: 3, f1-score: 1.00
weights: uniform
precision: 1.00
Neural Network hidden size: 2, 1.0 recall: 1.00
learning rate: 0.01, f1-score: 1.00
num hidden layers: 2,
activation function:
Tanh

9
10
All four models effectively classified all instances, showcasing their proficiency in handling the
intricate structure of the Two Moons dataset. The optimal hyperparameters for each model suggest
an effective learning strategy; for instance, the Decision Tree and Random Forest utilized entropy
for splitting, which is well-suited for such complex distributions. The KNN model maintained a

11
consistent performance with uniform weights, while the Neural Network adapted effectively with
two hidden layers and a Tanh activation function. This demonstrates that even in non-linearly
separable cases, well-tuned models can achieve remarkable accuracy by capturing the underlying
patterns in the data.

3.4 Concentric Circles (Non-Linearly Separable) Dataset

The results for the various models on the Concentric Circles dataset are summarized in Table 3.
The Decision Tree and Random Forest models achieved high accuracy of 0.98, indicating their
effectiveness in classifying the instances despite the complex structure of the dataset. The KNN
model performed even better, with an accuracy of approximately 0.99. In contrast, the Neural
Network excelled with a perfect accuracy of 1.0, showcasing its capacity to learn the intricate
patterns present in this non-linearly separable data.

Table 3: Results for the Concentric Circles Dataset

Model Best Parameters Accuracy Classification Report

precision: 0.95
Decision Tree criterion: gini, 0.98 recall: 1.00
max depth: None, f1-score: 0.98
min samples leaf: 1,
min samples split: 2
precision: 0.95
Random Forest bootstrap: True, 0.98 recall: 1.00
criterion: gini, f1-score: 0.98
max depth: None,
max features: None,
min samples leaf: 1,
min samples split: 2,
n estimators: 25
precision: 0.98
KNN metric: euclidean, 0.99 recall: 1.00
n neighbors: 9, f1-score: 0.99
weights: uniform
precision: 1.00
Neural Network hidden size: 10, 1.0 recall: 1.00
learning rate: 0.01, f1-score: 1.00
num hidden layers: 3,
activation function:
Tanh

12
13
The performance of the models demonstrates their ability to adapt to the unique challenges posed
by the Concentric Circles dataset. While the Decision Tree and Random Forest models maintained
a high accuracy, their precision for class 0 was slightly lower, reflecting a trade-off between pre-
cision and recall. The KNN model also showed robust performance, benefitting from an optimal

14
number of neighbors that enabled it to balance bias and variance effectively. The Neural Network’s
configuration, with three hidden layers and a Tanh activation function, allowed it to learn and gen-
eralize well, resulting in perfect accuracy. Overall, these results underline the importance of model
selection and hyperparameter tuning in achieving effective classification for complex datasets.

15