Optimal Feature Generation

● In general, feature generation is a problem-dependent task.

However, there are a few general directions common in a
number of applications. We focus on three such alternatives.
➢ Optimized features based on Scatter matrices (Fisher’s linear
discrimination).
• The goal: Given an original set of m measurements

, compute , by the linear transformation

x ∈ ℜm y ∈ ℜ!
T
y = A x criterion involving Sw, Sb is
so that the J3 scattering matrix
maximized. AT is an matrix.

xm
!

0
• The basic steps in the proof:
– J3 = trace{Sw-1 Sm}
– Syw = ATSxwA, Syb = ATSxbA,
– J3(A)=trace{(ATSxwA)-1 (ATSxbA)}
– Compute A so that J3(A) is maximum.

• The solution:

– Let B be the matrix that diagonalizes simultaneously

matrices Syw, Syb , i.e:
BTSywB = I , BTSybB = D
where B is a ℓxℓ matrix and D a ℓxℓ diagonal matrix.

0
– Let C=AB an mxℓ matrix. If A maximizes J3(A) then
(S −1
xw )
S xb C = CD
The above is an eigenvalue-eigenvector problem. For an
M-class problem, is of −rank
1
M-1.
● If ℓ =M-1, choose C to S xw S xb of the M-1 eigenvectors,
consist
corresponding to the non-zero eigenvalues.

The above guarantees maximum T

J3 value. In this case:
J3,x = J3,y. y=C x
● For a two-class problem, this results to the well known
Fisher’s linear discriminant

For Gaussian classes, this is the optimal Bayesian

(
classifier, with aydifference
= µ − of
1
)
S −1 x value .
µ a threshold
2 xw

0
● If ℓ<M-1, choose the ℓ eigenvectors corresponding to the
ℓ largest eigenvectors.
● In this case, J3,y<J3,x, that is there is loss of information.

– Geometric interpretation. The vector is the projection of

onto the subspace spanned by the eigenvectors
y of
.
x
−1
S xw S xb

0
● Principal Components Analysis
(The Karhunen – Loève transform):
➢ The goal: Given an original set of m measurements x ∈ ℜm
compute
y ∈ ℜ!
y = AT x
for an orthogonal A, so that the elements of are optimally
mutually uncorrelated. y
That is

➢ Sketch of the proof:

E [y (i ) y ( j )]= 0, i ≠ j

[ ] [T T
R y = E y y = E AT x x A = AT Rx A ]
0
• If A is chosen so that its columns are thea i orthogonal
eigenvectors of Rx, then
R y = AT Rx A = Λ
where Λ is diagonal with elements the respective
eigenvalues λi.
• Observe that this is a sufficient condition but not
necessary. It imposes a specific orthogonal structure on
A.

➢ Properties of the solution

• Mean Square Error approximation.
Due to the orthogonality of A:

m
T
x = ∑ y (i )a i , y (i ) = a i x
i =0

0
- Define −1
!
xˆ = ∑ y (i )a i
i =0
- The Karhunen – Loève transform minimizes the
square error:
2
⎡ m ⎤
- [
The error is:E x − xˆ
2
]
= E ⎢ ∑ y (i )a i ⎥
⎢⎣ i =! ⎥⎦
m
[
E x − xˆ
2

It can be also shown that this isi =the

!
i]= ∑ λ
minimum mean
square error compared to any other representation of x
by an ℓ-dimensional vector.

0
- In other words, x̂ is the projection of x the
into
subspace spanned by the principal ℓ eigenvectors.
However, for Pattern Recognition this is not the
always the best solution.

0
• Total variance: It is easily seen that

[ ]
σ 2y ( i ) = E y 2 (i ) = λi
Thus Karhunen – Loève transform makes the total
variance maximum.

• Assuming to be a zero mean multivariate Gaussian,

then the K-L transform
y maximizes the entropy:

of the resulting process.

[
H y = − E ln Py ( y ) ]
y

0
➢ Subspace Classification. Following the idea of projecting in a
subspace, the subspace classification classifies an unknown to
the class whose
x subspace is closer to . x
The following steps are in order:

• For each class, estimate the autocorrelation matrix Ri, and

compute the m largest eigenvalues. Form Ai, by using
respective eigenvectors as columns.

• Classify to the class ωi, for which the norm of the

subspace projection
x is maximum

T T
A x > A j x ∀i this
According to Pythagoras theorem,
i ≠ j corresponds to the
subspace to which is closer.
x

0
● Independent Component Analysis (ICA)
In contrast to PCA, where the goal was to produce uncorrelated
features, the goal in ICA is to produce statistically
independent features. This is a much stronger requirement,
involving higher to second order statistics. In this way, one
may overcome the problems of PCA, as exposed before.
➢ The goal: Given , compute
x y ∈ ℜ!
so that the components of
y =are
W xstatistically independent. In
order the problem to have a solution, y the following
assumptions must be valid:
• Assume that is indeed generated by a linear combination
of independent components
x

x =Φy
0
 Φ is known as the mixing matrix and W as the demixing
matrix.
• Φ must be invertible or of full column rank.
• Identifiability condition: All independent components, y(i),
must be non-Gaussian. Thus, in contrast to PCA that can
always be performed, ICA is meaningful for non-Gaussian
variables.
• Under the above assumptions, y(i)’s can be uniquely
estimated, within a scalar factor.

0
➢ Common’s method: Given , and x under the previously
stated assumptions, the following steps are adopted:
• Step 1: Perform PCA on :
x
• Step 2: Compute a unitary AT x , so that the fourth order
y =matrix,
cross-cummulants of the transform vector
Â
are zero. This is equivalent to searching for an that makes
y = Aˆ T yˆ maximum,
the squares of the auto-cummulants

Â
where, is the 4th order auto-cumulant.
max ( ˆ ) = k (y (i ) )
A
2
ˆ ˆT
AA
Ψ ∑ 4
k 4 ()
⋅
0
• Step 3: ( )
W = AAˆ
T

➢ A hierarchy of components: which ℓ to use? In PCA one

chooses the principal ones. In ICA one can choose the ones
with the least resemblance to the Gaussian pdf.

0
➢ Example:

The principal component is α 1 , thus according to PCA one

chooses as y the projection of x into α 1 . According to ICA,
one chooses as y the projection on α 2 . This is the least
Gaussian. Indeed:
K4(y1) = -1.7
K4(y2) = 0.1
Observe that across α , the statistics is bimodal. That is, no
2
resemblance to Gaussian. 0