SUPPORT VECTOR MACHINE

KIT-601
 SUPPORT VECTOR MACHINE

Support Vector Machine or SVM is one of the most popular Supervised Learning
algorithms, which is used for Classification as well as Regression problems.

• “Support Vector Machine” (SVM) is a supervised learning machine learning algorithm that can

be used for both classification or regression challenges.

• In the SVM algorithm, we plot each data item as a point in n-dimensional space (where n is the

number of features you have), with the value of each feature being the value of a particular

coordinate. Then, we perform classification by finding the optimal hyper-plane that differentiates the

two classes very well

LINEAR SVM
When the data is perfectly linearly separable only then we can use Linear SVM. Perfectly
linearly separable means that the data points can be classified into 2 classes by using a single
straight line(if 2D).

NON-LINEAR SVM
When the data is not linearly separable then we can use Non-Linear SVM, which means when
the data points cannot be separated into 2 classes by using a straight line (if 2D) then we use
some advanced techniques like kernel tricks to classify them. In most real-world applications we
do not find linearly separable datapoints hence we use kernel trick to solve them.
 BASIC TERMS
• SUPPORT VECTORS: These are the points that are closest to the hyperplane. A separating line will be defined with the
help of these data points.

• MARGIN: it is the distance between the hyperplane and the observations closest to the hyperplane (support vectors).
In SVM large margin is considered a good margin. There are two types of margins hard margin and soft margin. I will
talk more about these two in the later section.

• HYPERPLANE: There can be multiple lines/decision boundaries to segregate the classes in n-dimensional space, but
we need to find out the best decision boundary that helps to classify the data points. This best boundary is known as
the hyperplane of SVM.

• DECISION BOUNDARY: A decision boundary can be thought of as a demarcation line (for simplification) on one side of
which lie positive examples and on the other side lie the negative examples. On this very line, the examples may be
classified as either positive or negative. This same concept of SVM will be applied in Support Vector Regression as well
 IDENTIFY THE RIGHT HYPER-PLANE (SCENARIO-1)

we have three hyper-planes (A, B, and C). Now, identify the right hyper-plane
to classify stars and circles.

You need to remember a thumb rule to identify the right hyper-plane: “Select the
hyper-plane which segregates the two classes better.” In this scenario, hyper-plane
“B” has excellently performed this job.
 IDENTIFY THE RIGHT HYPER-PLANE (SCENARIO-2)
We have three hyper-planes (A, B, and C), and all segregate the classes well. Now, How can we
identify the right hyper-plane?

Here, maximizing the distances between the nearest data point (either class) and the
hyper-plane will help us to decide the right hyper-plane. This distance is called a Margin.
Above, you can see that the margin for hyper-plane C is high as compared to both A and B.
Hence, we name the right hyper-plane as C. Another lightning reason for selecting the
hyper-plane with a higher margin is robustness. If we select a hyper-plane having a low
margin, then there is a high chance of misclassification.
 IDENTIFY THE RIGHT HYPER-PLANE (SCENARIO-3)
Use the rules as discussed in the previous section to identify the right hyper-plane.

Some of you may have selected hyper-plane B as it has a higher margin compared to
A. But, here is the catch, SVM selects the hyper-plane which classifies the classes
accurately prior to maximizing the margin. Here, hyper-plane B has a classification
error, and A has classified all correctly. Therefore, the right hyper-plane is A.
 IDENTIFY THE RIGHT HYPER-PLANE (SCENARIO-4)
Below, I am unable to segregate the two classes using a straight line, as one of the stars lies in the territory of the
other (circle) class as an outlier.

As I have already mentioned, one star at the other end is like an outlier for the star class. The SVM algorithm has
a feature to ignore outliers and find the hyper-plane that has the maximum margin. Hence, we can say SVM
classification is robust to outliers.
 IDENTIFY THE RIGHT HYPER-PLANE (SCENARIO-5)
among these different hyperplanes, which is the best hyperplane?

If you observed in the above figure, we can clearly see like among three-line, red line (H3) which has the
maximum margin with the data points and also classified between the data properly, if you see blue color line
(H2) the margin is small with one data and large with another data, whereas, green line (H1), it has not
classified between the data itself.
 SUPPORT VECTOR, HYPERPLANE, AND
MARGIN
1. The data/vector points closest to the hyperplane (black line) are known as the support vector
(SV) data points because only these two points are contributing to the result of the algorithm
(SVM), other points are not.

2. If a data point is not an SV, removing it has no effect on the model.

3. Deleting the SV will then change the position of the hyperplane.

4. The dimension of the hyperplane depends upon the number of features. If the number of
input features is 2, then the hyperplane is just a line. If the number of input features is 3, then
the hyperplane becomes a two-dimensional plane. It becomes difficult to imagine when the
number of features exceeds 3.
 SUPPORT VECTOR CLASSIFIER (SVC)

MARGIN
The distance of the vectors from the hyperplane is called the margin which is a separation of a line
to the closest class points. We would like to choose a hyperplane that maximizes the margin
between classes.
 TYPES OF MARGIN

1. SOFT MARGIN – As most of the real-world data are not fully linearly separable,
we will allow some margin violation to occur which is called soft margin
classification. It is better to have a large margin, even though some constraints
are violated. Margin violation means choosing a hyperplane, which can allow
some data points to stay on either the incorrect side of the hyperplane and
between the margin and correct side of the hyperplane.

2. HARD MARGIN – If the training data is linearly separable, we can select two
parallel hyperplanes that separate the two classes of data, so that the distance
between them is as large as possible.
 NON- LINEAR SVM
If data is linearly arranged, then we can separate it by using a straight line, but for non-linear data, we
cannot draw a single straight line.

Non-linear SVM When the data is not linearly separable, we use the non-linear SVM classifier to
separate the data points.

We cannot separate the above data points with a single line. Also if we gave more than two classes, it is
impossible to separate them with a single straight line.
 KERNEL
• A kernel is a function used in SVM for helping to solve problems. They provide shortcuts to
avoid complex calculations. The amazing thing about kernel is that we can go to higher
dimensions and perform smooth calculations with the help of it.

• We can go up to an infinite number of dimensions using kernels.

• A kernel helps to form the hyperplane in the higher dimension without raising the complexity.

• Kernels are a way to solve non-linear problems with the help of linear classifiers. This is known
as the kernel trick method.

• The kernel functions are used as parameters in the SVM codes.

• They help to determine the shape of the hyperplane and decision boundary.
 KERNEL FUNCTION WORK

• The function of kernel is to take data as input and transform it into the
required form. Different SVM algorithms use different types of kernel
functions.

• These functions can be different types. For example linear, nonlinear,

• The kernel functions return the inner product between two points in a
suitable feature space. Thus by defining a notion of similarity, with little
computational cost even in very high-dimensional spaces.
TYPES OF KERNEL FUNCTIONS