Pattern Recognition

PECCS702B
WorkBook
Semester - 7
Prof. Bavrabi Ghosh

What is Pattern Recognition?

Pattern recognition is the scientific discipline whose goal is the classification of
objects into a number of categories or classes. Depending on the application, these
objects can be images or signal waveforms or any type of measurements that need
to be classified. Pattern recognition is an integral part in most machine intelligence systems
built for decision making.

What is machine vision?

A machine vision system captures images via a camera and analyzes them to
produce descriptions of what is imaged. A typical application of a machine vision
system is in the manufacturing industry, either for automated visual inspection or
for automation in the assembly line. For example, in inspection, manufactured
objects on a moving conveyor may pass the inspection station, where the camera
stands, and it has to be ascertained whether there is a defect. Thus, images have
to be analyzed on line, and a pattern recognition system has to classify the objects
into the “defect” or “non-defect”c1ass. After that, an action has to be taken, such as
to reject the offending parts. In an assembly line, different objects must be located
and “recognized,” that is, classified in one of a number of classes known a priori.
Examples are the “screwdriver class,” the “German key class,” and so forth in a
tools’ manufacturing unit. Then a robot arm can place the objects in the right place.

What is a feature?
The input variables that we give to our machine learning models are called features. Each
column in our dataset constitutes a feature.

What is feature extraction?

Feature extraction is the process of determining the features to be used for learning. The
description and properties of the patterns are known. However, for the classification task at
hand, it is necessary to extract the features to be used. It may involve carrying out some
arithmetic operations on the features like linear combinations of the features or finding the
value of a function.
What is feature selection?

To train an optimal model, we need to make sure that we use only the essential features. If
we have too many features, the model can capture the unimportant patterns and learn
from noise. The method of choosing the important parameters of our data is called Feature
Selection.

Why is feature selection needed?

Machine learning models follow a simple rule: whatever goes in, comes out. If we put
garbage into our model, we can expect the output to be garbage too. In this case, garbage
refers to noise in our data.

To train a model, we collect enormous quantities of data to help the machine learn better.
Usually, a good portion of the data collected is noise, while some of the columns of our
dataset might not contribute significantly to the performance of our model. Further, having
a lot of data can slow down the training process and cause the model to be slower. The
model may also learn from this irrelevant data and be inaccurate.

A case study on feature selection

Consider a table which contains information on old cars. The model decides which cars must
be crushed for spare parts.

In the above table, we can see that the model of the car, the year of manufacture, and the
miles it has traveled are pretty important to find out if the car is old enough to be crushed
or not. However, the name of the previous owner of the car does not decide if the car
should be crushed or not. Further, it can confuse the algorithm into finding patterns
between names and the other features. Hence we can drop the column.
Elaborate the concept of feature selection.

Feature Selection is the method of reducing the input variable to your model by using only
relevant data and getting rid of noise in data.

It is the process of automatically choosing relevant features for your machine learning
model based on the type of problem you are trying to solve. We do this by including or
excluding important features without changing them. It helps in cutting down the noise in
our data and reducing the size of our input data.

What are the different feature selection models?

Feature selection models are of two types:

Supervised Models: Supervised feature selection refers to the method which uses the
output label class for feature selection. They use the target variables to identify the
variables which can increase the efficiency of the model
Unsupervised Models: Unsupervised feature selection refers to the method which does not
need the output label class for feature selection. We use them for unlabelled data.
What is the filter method of feature selection?

Filter Method: In this method, features are dropped based on their relation to the output,
or how they are correlating to the output. We use correlation to check if the features are
positively or negatively correlated to the output labels and drop features accordingly. Eg:
Information Gain, Chi-Square Test, Fisher’s Score, etc.

What is the wrapper method of feature selection?

We split our data into subsets and train a model using this. Based on the output of the
model, we add and subtract features and train the model again. It forms the subsets using a
greedy approach and evaluates the accuracy of all the possible combinations of features. Eg:
Forward Selection, Backwards Elimination, etc.
What is intrinsic method of feature selection?

This method combines the qualities of both the Filter and Wrapper method to create the
best subset. This method takes care of the machine training iterative process while
maintaining the computation cost to be minimum. Eg: Lasso and Ridge Regression.

How to choose a feature selection model?

The process is relatively simple, with the model depending on the types of input and output
variables.

Variables are of two main types:

Numerical Variables: Which include integers, float, and numbers.

Categorical Variables: Which include labels, strings, boolean variables, etc.
Based on whether we have numerical or categorical variables as inputs and outputs, we can
choose our feature selection model as follows:

Input
Output Variable Feature Selection Model
Variable

• Pearson’s correlation coefficient

Numerical Numerical
• Spearman’s rank coefficient

• ANOVA correlation coefficient (linear).

Numerical Categorical
• Kendall’s rank coefficient (nonlinear).

• Kendall’s rank coefficient (linear).

Categorical Numerical • ANOVA correlation coefficient

(nonlinear).

• Chi-Squared test (contingency tables).

Categorical Categorical
• Mutual Information.

Short note on KNN Algorithm.

1. K-Nearest Neighbour is one of the simplest Machine Learning algorithms based on

Supervised Learning technique.
2. K-NN algorithm assumes the similarity between the new case/data and available cases
and put the new case into the category that is most similar to the available categories.
3. K-NN algorithm stores all the available data and classifies a new data point based on the
similarity. This means when new data appears then it can be easily classified into a well suite
category by using K- NN algorithm.
4. K-NN algorithm can be used for Regression as well as for Classification but mostly it is
used for the Classification problems.
5. K-NN is a non-parametric algorithm, which means it does not make any assumption on
underlying data.
6. It is also called a lazy learner algorithm because it does not learn from the training set
immediately instead it stores the dataset and at the time of classification, it performs an
action on the dataset.

KNN algorithm at the training phase just stores the dataset and when it gets new data, then
it classifies that data into a category that is much similar to the new data.
Example: Suppose, we have an image of a creature that looks similar to cat and dog, but we
want to know either it is a cat or dog. So for this identification, we can use the KNN
algorithm, as it works on a similarity measure. Our KNN model will find the similar features
of the new data set to the cats and dogs images and based on the most similar features it
will put it in either cat or dog category.

Why do we need a KNN algorithm ?

Suppose there are two categories, i.e., Category A and Category B, and we have a new data
point x1, so this data point will lie in which of these categories. To solve this type of
problem, we need a K-NN algorithm. With the help of K-NN, we can easily identify the
category or class of a particular dataset. Consider the below diagram:
How does KNN algorithm work?

The K-NN working can be explained on the basis of the below algorithm:

o Step-1: Select the number K of the neighbors

o Step-2: Calculate the Euclidean distance of K number of neighbors
o Step-3: Take the K nearest neighbors as per the calculated Euclidean distance.
o Step-4: Among these k neighbors, count the number of the data points in each
category.
o Step-5: Assign the new data points to that category for which the number of the
neighbor is maximum.
o Step-6: Our model is ready.

Suppose we have a new data point and we need to put it in the required category. Consider
the below image:

o Firstly, we will choose the number of neighbors, so we will choose the k=5.
o Next, we will calculate the Euclidean distance between the data points. The
Euclidean distance is the distance between two points, which we have already
studied in geometry. It can be calculated as:
 o By calculating the Euclidean distance we got the nearest neighbors, as three nearest
neighbors in category A and two nearest neighbors in category B. Consider the below
image:

o As we can see the 3 nearest neighbors are from category A, hence this new data
point must belong to category A.

How to select the value of K in KNN algorithm?

There is no particular way to determine the best value for "K", so we need to try some
values to find the best out of them. The most preferred value for K is 5.
A very low value for K such as K=1 or K=2, can be noisy and lead to the effects of outliers in
the model.

Large values for K are good, but it may find some difficulties.

What are the advantages of KNN algorithm?

o It is simple to implement.
o It is robust to the noisy training data
o It can be more effective if the training data is large.

What are the disadvantages of KNN algorithm?

o Always needs to determine the value of K which may be complex some time.
o The computation cost is high because of calculating the distance between the data
points for all the training samples.

Write short notes on different variants of KNN algorithm.

Locally Adaptive KNN - In standard KNN algorithm global value of input parameter k is used.
But this proposed algorithm suggested using different values of the parameter k for
different portions of input space. Each time for classification of a query, the value of k is
determined via applying cross-validation in its local neighbourhood.

Weighted adjusted KNN - In standard KNN algorithm all the attributes have equal
importance. All the attributes or features give equal contribution for classification of novel
tuples. But not all the attributes in the data set are equally important. A weight-adjusted
KNN algorithm which first learns weights for different attributes and according to the
weights assigned, each attribute would affect the process of classification that much only.

Improved KNN for Text Categorization - As we know how much the value of input parameter
k influences the performance of KNN algorithm. So, it is very crucial to choose appropriate
value of the parameter k. In general the classes are not evenly distributed in the data set.
Therefore, using a fixed value of k for all the classes would result in bias towards the class
which has larger number of tuples. Therefore one can use different values of k for different
classes according to their class distribution. More number of tuples is used to classify a new
tuple in a class which has large number of tuples.

Adaptive KNN – Rather than using a fixed value of k, to use non-fixed number of nearest
neighbours i.e. k. Large value of parameter k would also increase the computational cost
and time in case of large data sets. To solve this problem it has applied three heuristics so
that early break of the algorithm can be possible. These heuristics on fulfilment of a fixed
condition would break out from the algorithm. This would save computational time of the
algorithm.

KNN with Shared Nearest Neighbours - another variant of KNN algorithm which uses shared
nearest neighbours to classify documents. To find neighbours of a novel tuple, it uses BM25
similarity measure. A threshold is set, only that much number of nearest neighbours can
vote for classification of an unknown tuple.

KNN with K-Means - One of the shortcomings of KNN algorithm is its high computation
complexity. To alleviate this drawback an effort has been made by combining KNN algorithm
with the clustering algorithm K-Means. In the proposed algorithm first the clusters of the
different categories in training data set are formed. The centres of these newly formed
clusters will now act as new training samples. To classify an unknown tuple the distance of it
is computed with these new training tuples and with which the tuple has least distance it
will be assigned to that class. The benefit of this variant of KNN is that there is no need of
passing the input parameter k as we have to do in standard KNN.

KNN with Mahalanobis Metric - The performance of KNN algorithm largely depends on the
distance metric which is used to find the distance between any two tuples. A new distance
metric called Mahalanobis distance metric was introduced. It transforms the whole input
space using linear transformation. In this transformed input space the Euclidean distance is
same as Mahalanobis distance between any two data points. Euclidean distance is the
distance between any two points whereas Mahalanobis distance is a distance between a
point and a distribution. If the point represents the mean of the distribution then the
Mahalanobis distance would be zero. The main benefit of taking Mahalanobis distance
metric instead of Euclidean distance metric is that it also reckoned the correlation between
data tuples.