Scribd
Upload
14 views12 pages

AIML Lec-14

Uploaded by

shubhodippal01
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
14 views12 pages

AIML Lec-14

Uploaded by

shubhodippal01
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 12

Artificial Intelligence & Machine Learning

Dr. Sankhadeep Chatterjee


Department of Computer Science and Technology
&
Department of Computer Science and Information Technology
University of Engineering and Management, Kolkata
Topics
• Concept of Maximum Margin
• Support Vectors
• Support Vector Machine for linear decision boundary
• Kernels for non linear decision boundary

Reference
1. “Introduction to Statistical Learning” Hastie, Tibshirani
2. “Machine Learning” Pater Flach
3. Course on “Artificial Intelligence” by Patrick Winston, MIT (Lecture 16)
Maximum Margin
• Binary classification problem
• Need to draw decision boundary
• Draw decision boundary such
that
• The separation between ‘+’ and ‘-’
samples is maximum
• Training SVM is again an
optimization problem
SVM for linear decision boundary
• Assume vector 𝑤 which is
perpendicular with decision
boundary (Red dashed line)
• Consider unknown data sample
represented by vector 𝑢
• Project vector 𝑢 on 𝑤 using dot
product (𝑤. 𝑢) 𝑢

• If 𝑤. 𝑢 ≥ 𝑐 for some 𝑐 conclude that 𝑤


𝑢 is in ′ + ′ category else ′ − ′
SVM for linear decision boundary
𝑤. 𝑢 ≥ 𝑐
𝑤. 𝑢 + 𝑏 ≥ 0 (consider 𝑏 = −𝑐) …(1)
Now for support vectors we write;
𝑤. 𝑥+ + 𝑏 ≥ 1 (if 𝑢 ∈ ′ + ′)
𝑤. 𝑥− + 𝑏 ≤ −1 (if 𝑢 ∈′ −′)
Assume that; 𝑢

𝑦𝑖 = +1 if 𝑢 ∈ ′ + ′ 𝑤
𝑦𝑖 = −1 if 𝑢 ∈′ −′
SVM for linear decision boundary
Combining the following:
𝑤. 𝑥+ + 𝑏 ≥ 1 and 𝑦𝑖 = +1 if 𝑢 ∈ ′ + ′

We get: 𝑦𝑖 (𝑤. 𝑥+ + 𝑏) ≥ 1

And similarly for 𝑥− we get: 𝑦𝑖 (𝑤. 𝑥− + 𝑏) ≥ 1

Hence, for both class we have a single decision 𝑢


function
𝑤
On decision boundary we define:
𝑦𝑖 𝑤. 𝑥𝑖 + 𝑏 − 1 = 0
SVM for linear decision boundary
• Width of the lines separating ‘+’ and ‘-’
samples can be calculated by taking the
dot product of (𝑥+ − 𝑥− ) with a unit
vector perpendicular to decision
boundary.
𝑤
= 𝑥+ − 𝑥− . 𝑥+ − 𝑥−
𝑤
As, 𝑤 is already perpendicular with 𝑥− 𝑥+
decision boundary and dividing it by 𝑤
makes it a unit vector.
SVM for linear decision boundary
• From equation of the decision boundary 𝑦𝑖 𝑤. 𝑥𝑖 + 𝑏 − 1 = 0 we get:
For ‘+’ samples: For ‘-’ samples:
+1. 𝑤. 𝑥+ + 𝑏 = 1 −1. 𝑤. 𝑥− + 𝑏 = 1
𝑤. 𝑥+ + 𝑏 = 1 −𝑤. 𝑥− − 𝑏 = 1
𝑤. 𝑥+ = 1 − 𝑏 −𝑤. 𝑥− = 1 + 𝑏

Now, the distance between the lines (width of margin) can be calculated as
𝑤 𝑥+ . 𝑤 − 𝑥− . 𝑤 1−𝑏+1+𝑏 2
= 𝑥+ − 𝑥− . = = =
𝑤 𝑤 𝑤 𝑤
SVM for linear decision boundary
• Objective: Maximize the width of the margin between separating
lines of ‘+’ and ‘-’ samples.
2 1 2
• Maximize ≡ minimize 𝑤 ≡ minimize 𝑤
𝑤 2
• Constrain: 𝑦𝑖 𝑤. 𝑥Ԧ + 𝑏 − 1 = 0
• In order to solve constrained optimization problem we use Lagrangian
method. Hence we define objective function as:
1 2
𝐿= 𝑤 − ∑𝛼𝑖 [𝑦𝑖 𝑤. 𝑥𝑖 + 𝑏 − 1]
2
where 𝛼𝑖 are Lagrangian multipliers
SVM for linear decision boundary
1 2
Minimize 𝐿 = 𝑤 − ∑𝛼𝑖 [𝑦𝑖 𝑤. 𝑥𝑖 + 𝑏 − 1]
2

𝜕𝐿
= 𝑤 − ∑𝛼𝑖 𝑦𝑖 𝑥𝑖 = 0 …(2)
𝜕𝑤
𝜕𝐿
= −∑𝛼𝑖 𝑦𝑖 = 0 …(3)
𝜕𝑏

Substituting (2) in objective function we get:

1
𝐿 = ∑𝛼𝑖 𝑦𝑖 𝑥𝑖 . ∑𝛼𝑗 𝑦𝑗 𝑥𝑗 − ∑𝛼𝑖 𝑦𝑖 𝑥𝑖 . ∑𝛼𝑗 𝑦𝑗 𝑥𝑗 − ∑𝛼𝑖 𝑦𝑖 𝑏 + ∑𝛼𝑖
2

Substituting (3) and simplifying further we get:

1
= ∑𝛼𝑖 − ∑𝑖 ∑𝑗 𝛼𝑖 𝛼𝑗 𝑦𝑖 𝑦𝑗 𝑥𝑖 . 𝑥𝑗
2
SVM for non-linear decision boundary
• Decision Boundary: ∑𝛼𝑖 𝑦𝑖 𝑥𝑖 . 𝑥Ԧ + 𝑏 ≥ 1
Kernel
• Apply Kernel function (𝜙(𝑥𝑖 , 𝑥))
Ԧ to project data points to higher
dimensions.
1
• Objective function: L = ∑𝛼𝑖 − ∑𝑖 ∑𝑗 𝛼𝑖 𝛼𝑗 𝑦𝑖 𝑦𝑗 𝜙 𝑥𝑖 , 𝑥𝑗
2
• Some kernel functions:
• Dot product: 𝜙 𝑥𝑖 , 𝑥Ԧ = 𝑥𝑖 . 𝑥Ԧ
2
𝑥𝑖 −𝑥
−
• Radial Basis Function (RBF): 𝜙 𝑥𝑖 , 𝑥Ԧ = 𝑒 𝜎2

• Polynomial: 𝜙 𝑥𝑖 , 𝑥Ԧ = (𝛾𝑥𝑖 . 𝑥Ԧ + 𝑐)𝑝

You might also like

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy