Assignment 3

Q: What is KNN algorithm?

A: KNN (K-Nearest Neighbors) algorithm is a non-parametric and lazy
machine learning algorithm used for classification and regression
tasks. It works by finding the K nearest data points (neighbors) to the
query point and predicts the output based on the labels of these
neighbors.

Q: What is the distance metric used in KNN algorithm?

A: Euclidean distance is the most commonly used distance metric in
KNN algorithm. However, other distance metrics such as Manhattan
distance, Minkowski distance, and Hamming distance can also be used
depending on the problem.

Q: What is the curse of dimensionality in KNN algorithm?

A: The curse of dimensionality refers to the problem that arises when
the number of dimensions in the feature space increases. As the
number of dimensions increases, the distance between any two points
in the space becomes increasingly large, making it difficult to find
meaningful nearest neighbors. This problem can be addressed by
reducing the dimensionality of the feature space or by using
dimensionality reduction techniques such as PCA.

Q: What are the advantages of using KNN algorithm?

A: The advantages of using KNN algorithm are:
 • Simple to implement

• Non-parametric and does not make assumptions about the

underlying distribution of the data

• Can be used for both classification and regression tasks

• Can handle multi-class classification problems

• Can handle both numerical and categorical data

Q: What are the disadvantages of using KNN algorithm?

A: The disadvantages of using KNN algorithm are:

• Computationally expensive, especially for large datasets

• Sensitive to the choice of K and distance metric

• Requires a large amount of memory to store the entire dataset

• Can be affected by the presence of noisy or irrelevant features

• Cannot handle missing data

Q: How do you choose the value of K in KNN algorithm?

A: The choice of K in KNN algorithm depends on the problem and the
dataset. A small value of K (e.g., K=1) will result in a more flexible
model but may be prone to overfitting. A large value of K (e.g., K=n,
where n is the size of the dataset) will result in a more stable model but
may not capture the local variations in the data. The choice of K can be
determined using techniques such as cross-validation or grid search.

Q: What is the difference between classification and

regression in KNN algorithm?
A: In classification, the output of the KNN algorithm is a categorical
variable (e.g., class label), whereas in regression, the output is a
continuous variable (e.g., real number). The distance metric and the
choice of K are the same for both classification and regression, but the
prediction function is different.

Q: How do you handle imbalanced data in KNN algorithm?

A: One approach to handling imbalanced data in KNN algorithm is to
use weighted voting, where the vote of each neighbor is weighted by its
inverse distance to the query point. This gives more weight to the
closer neighbors and less weight to the farther neighbors, which can
help to reduce the effect of the majority class. Another approach is to
oversample the minority class or undersample the majority class to
balance the dataset.

Q: Can KNN algorithm be used for text classification?

A: Yes, KNN algorithm can be used for text classification by
representing the text data as a bag-of-words or TF-IDF vector and
using a distance metric such as cosine similarity. However, KNN
algorithm may not be the most efficient algorithm for text
classification, especially for large datasets. Other algorithms such as
Naive Bayes, SVM, and neural networks may be more suitable.
Q: What are the parameters of KNN in scikit-learn?
A: In scikit-learn library, the main parameters of KNN algorithm are:

• n_neighbors: The number of neighbors to consider for

classification or regression. This is the K parameter in the
KNN algorithm.

• weights: The weight function used in prediction. Possible

values are "uniform", where all neighbors have equal weight,
or "distance", where the weight of each neighbor is
proportional to its inverse distance from the query point.

• algorithm: The algorithm used to compute nearest neighbors.

Possible values are "brute", which performs a brute-force
search over all possible neighbors, "kd_tree", which uses a k-d
tree to find the nearest neighbors, and "ball_tree", which uses
a ball tree to find the nearest neighbors.

• leaf_size: The number of points at which the k-d tree or ball

tree algorithm switches to brute-force search. Larger values
lead to faster queries but higher memory consumption.

• metric: The distance metric used to compute the distance

between two points. Possible values are "euclidean" (default),
"manhattan", "chebyshev", "minkowski", "wminkowski",
"seuclidean", "mahalanobis", and others.
 • p: The power parameter for the Minkowski distance metric.
When p=1, this is equivalent to the Manhattan distance, and
when p=2, this is equivalent to the Euclidean distance.

There are also additional parameters that can be used for specific
purposes, such as n_jobs to control the number of CPU cores used for
computation, and metric_params to pass additional parameters to the
distance metric function.

Q: What are the default values of parameters for KNN in

scikit-learn?
A: In scikit-learn library, the default values of the main parameters for
KNN algorithm are:

• n_neighbors: 5

• weights: "uniform"

• algorithm: "auto"

• leaf_size: 30

• metric: "minkowski"

• p: 2

These default values are used when no values are specified for these
parameters during the initialization of the KNeighborsClassifier or
KNeighborsRegressor classes. However, it is recommended to tune
these parameters for the specific task and dataset to achieve the best
performance of the model.

Q: What are the evaluation metrics for KNN algorithm in

classification tasks?
A: The common evaluation metrics for KNN algorithm in classification
tasks are:

• Accuracy: The proportion of correctly classified instances over

the total number of instances.

• Precision: The proportion of true positives over the total

number of predicted positives.

• Recall: The proportion of true positives over the total number

of actual positives.

• F1 score: The harmonic mean of precision and recall.

• ROC curve and AUC: The ROC (Receiver Operating

Characteristic) curve shows the trade-off between the true
positive rate and false positive rate for different threshold
values, while the AUC (Area Under the Curve) measures the
overall performance of the classifier.

Q: What are the evaluation metrics for KNN algorithm in

regression tasks? A: The common evaluation metrics for KNN
algorithm in regression tasks are:
 • Mean Absolute Error (MAE): The average absolute difference
between the predicted values and the actual values.

• Mean Squared Error (MSE): The average squared difference

between the predicted values and the actual values.

• Root Mean Squared Error (RMSE): The square root of the

MSE.

• R-squared: The proportion of the variance in the dependent

variable that is explained by the independent variable.

Q: How do you perform cross-validation for KNN algorithm?

A: Cross-validation is a technique used to evaluate the performance of
a machine learning model. The common approach for performing
cross-validation for KNN algorithm is k-fold cross-validation, where
the dataset is divided into k equally sized folds. The KNN model is
trained on k-1 folds and tested on the remaining fold, and this process
is repeated k times with a different fold used for testing each time. The
average performance over the k iterations is then used as the estimate
of the model performance.

Q: Can KNN algorithm handle imbalanced classes?

A: Yes, KNN algorithm can handle imbalanced classes by using
weighted voting or adjusting the decision threshold. In weighted
voting, each neighbor’s vote is weighted by its inverse distance to the
query point, giving more weight to the closer neighbors and less weight
to the farther neighbors. Adjusting the decision threshold involves
changing the threshold used to classify an instance as positive or
negative. By increasing the threshold, the algorithm becomes more
conservative and tends to classify more instances as negative, which
can help to balance the classes.

Q: How do you tune the hyperparameters of KNN algorithm?

A: The two main hyperparameters of KNN algorithm are the number of
neighbors (K) and the distance metric. The optimal values of these
hyperparameters can be determined using techniques such as grid
search or randomized search. Grid search involves testing a range of
values for each hyperparameter and selecting the combination of
hyperparameters that gives the best performance. Randomized search
is similar to grid search but samples hyperparameters randomly from a
distribution rather than testing all possible combination