Scribd
Upload
12 views14 pages

Lecture14 Final

Uploaded by

Shraddha
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
12 views14 pages

Lecture14 Final

Uploaded by

Shraddha
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 14

CSE 151 Machine Learning

Instructor: Kamalika Chaudhuri


Announcements

Final: June 13, 7-9pm (2 hour final)

Material Covered: Everything! (Linear algebra,


Probability review to Gaussian Mixture Models)
Review session will be announced
Homework 4 is up, due Mon Jun 4 in class
Projecting Data

How to find the subspace S to project to?


One option: PCA subspace
Eigenvectors and Eigenvalues

Given a n x n matrix A,
c: eigenvalue of A
x: the corresponding eigenvector if:

Ax = cx x �= 0

x cx
A
O O
Symmetric A

1. All ci are real


2. All xi are orthonormal

AX=XC
A = X C XT (for symmetric A)
since XT = X-1 for orthonormal matrices
X = an orthonormal basis of Rn
Properties of Symmetric A

maxn x� Ax
x∈R

Such that: �x� = 1

Solution: x = x1 (eigenvector for the largest eigenvalue)


maximum value of xTAx is the largest eigenvalue

Question: If x is an eigenvector with eigenvalue c, what is xTAx?


Properties of Symmetric A

max trace(X � AX)


X∈Rn×k
Such that columns of X are orthonormal

Solution: X = [x1 x2 ... xk ]


(eigenvectors for the k largest eigenvalues)
maximum value of tr(XTAX) is sum of top k eigenvalues
In Class Problem

Let A = a symmetric d x d matrix


X = [x1, .., xk], where xi = i-th eigenvector of A
Each xi has norm 1

Show that:
(a) Entry (i, i) of XTAX is ci, the i-th eigenvalue of A
(b) Entry (i, j) of XTAX is 0, when i is not equal to j
Covariance Matrix

Suppose x1,..,xn are data points in Rd

Covariance matrix of x1,..,xn:


�n
1
Σ= (xi − µ)(xi − µ)�
n i=1

where: 1 �n
µ= xi
n i=1
Covariance Matrix

Covariance matrix of x1,..,xn:


�n
1
Σ= (xi − µ)(xi − µ)�
n i=1

where: 1 �n
µ= xi
n i=1

Properties:
1. Entry at row i, column j is covariance of coord. i and coord. j
2. Symmetric
3. Positive semi-definite (0 or positive eigenvalues)
How to find a good subspace?

Direction 1 projections
Direction 1

Direction 2
Direction 2 projections

Data lies on a line


How to find a good subspace?

Direction 1 projections
Direction 1

Direction 2
Direction 2 projections

Data lies almost on a line


Find the direction of maximum variance
Max Variance Direction

Suppose data in Rd is centered so that the mean is 0

Variance along a direction u is:


n n n
� �
1 � 1 � 1 � 1 � �
� �
2
�xi , u� = 2
(u xi ) = u xi x�
i u = u xi x�
i u
n i=1
n i=1
n i=1
n i

�
Recall that: A = xi x�
i is symmetric
i

Finding the max. variance direction is equivalent to


finding the top eigenvector of A. Similar fact can be
shown for the k-dimensional subspace of max variance.
Dimension Reduction Algorithm

Given data matrix Xn x d


1. Compute mean m of the data vectors

2. Compute covariance matrix:


n
1�
Σ= (xi − m)(xi − m)�
n i=1

3. Compute Ukxd, the top k eigenvectors of Σ


4. Project:
P(X) = XUT

You might also like

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy