Elementary Linear Algebra (2024-25 @IITH)

Amit Tripathi

Indian Institute of Technology, Hyderabad

(IIT) Elementary Linear Algebra (2024-25 @IITH) 1 / 36

Gaussian elimination

Basic algorithm:
(a) Given a system of linear equations, apply elementary row operations
to convert it into a triangular system.
(b) Use backward substition to solve the system, one unknown at a time.

(IIT) Elementary Linear Algebra (2024-25 @IITH) 2 / 36

Gaussian elimination

Example
Consider the linear system:

2x + y + z =5
4x − 6y = −2
−2x + 7y + 2z = 9

We apply elementary row-operations to the augmented matrix

   
2 1 1 5 2 1 1 5
R →R +R1 and R2 →R2 −2R1
4 −6 0 −2 −−3−−−3−−− −−−−−−−−−−→ 0 −8 −2 −12
−2 7 2 9 0 8 3 14

(IIT) Elementary Linear Algebra (2024-25 @IITH) 3 / 36

Gaussian elimination

Example
   
2 1 1 5 2 1 1 5
R3 →R3 +R2
0 −8 −2 −12 −
−−−−−−→ 0 −8 −2 −12
0 8 3 14 0 0 1 2

Now, we apply back substitution to get z = 2, y = 1 and then x = 1 (in

reverse order).

It is possible that we end up with an inconsistent system or a system with

infinitely many solutions.

(IIT) Elementary Linear Algebra (2024-25 @IITH) 4 / 36

Linear functionals

Let V be a vector space over F .

Definition
A linear functional is a linear transformation

T : V → F.

Equivalently, an element of Hom(V , F ).

We denote Hom(V , F ) by V ∗ .

(IIT) Elementary Linear Algebra (2024-25 @IITH) 5 / 36

Linear functionals

Example
Let V = F n . For any scalars c1 , · · · , cn ∈ F , define

T (x1 , · · · , xn ) = c1 x1 + · · · + cn xn .

Clearly, T is a linear functional.

Example
Let [a, b] be a closed interval on the real line. Let V = C ([a, b]) be the
vector space of continuous real valued functions on [a, b]. Then
Z b
L(f ) = f (t) dt
a

is a linear functional on C ([a, b]).

(IIT) Elementary Linear Algebra (2024-25 @IITH) 6 / 36

Dual basis

Example
Let V = R2 and let B = {e1 , e2 } be the standard ordered basis. For any
vector α ∈ R2 , let  
x
[α]B = 1
x2
be the coordinate matrix.
Question: Can you define two linear functionals fi : R2 → R for i = 1, 2 ?

(IIT) Elementary Linear Algebra (2024-25 @IITH) 7 / 36

Dual basis

More generally, let V be an n-dimensional vector space and let

B = {α1 , · · · , αn } be a basis. Giving a linear functional

fi : V → F

is equivalent to giving n vectors in F . We set

fi (αj ) = δij , for i, j ∈ {1, · · · , n}.

In other words, for each i,

fi (αi ) = 1, and fi (αj ) = 0 for j ̸= i.

(IIT) Elementary Linear Algebra (2024-25 @IITH) 8 / 36

Dual basis

Theorem
The linear functionals B ∗ = {f1 , · · · , fn } as defined above form a basis of
V ∗.

Proof.
Spanning set: Let f ∈ V ∗ . Any linear transformation is uniquely
determined by its action on a basis set. Thus, the scalars

f (α1 ), · · · , f (αn )

uniquely determine f . We write

f = f (α1 )f1 + f (α2 )f2 + · · · + f (αn )fn .

This shows that B ∗ spans V ∗ .

(IIT) Elementary Linear Algebra (2024-25 @IITH) 9 / 36

Dual basis

Proof.
Independent set: Let c1 , · · · , cn be scalars such that

c1 f1 + · · · + cn fn = 0.

Applying α1 on both sides, we get c1 = 0. Continuing similarly, we

conclude that c1 = · · · = cn = 0.

Corollary
If V is a finite dimensional vector space, then dim(V ∗ ) = dim(V ).

Definition
The basis B ∗ obtained by the construction fi (αj ) = δij is called the dual
basis of B.

(IIT) Elementary Linear Algebra (2024-25 @IITH) 10 / 36

The trace map

Example
Let V be the space of all polynomials over F . Fix a scalar a ∈ F . Define
the evaluation map at a as

Ta (g ) = g (a).

This is a linear functional.

(IIT) Elementary Linear Algebra (2024-25 @IITH) 11 / 36

The trace map

Let F n×n be the vector space of n × n matrices over F . The trace map
defined as
Tr (A) = a11 + · · · + ann
is a linear functional.

(IIT) Elementary Linear Algebra (2024-25 @IITH) 12 / 36

The trace map

Some properties of the trace functional:

(a) Tr (A + B) = Tr (A) + Tr (B).
(b) Tr (A) = Tr (At ).
(c) Tr (AB) = Tr (BA).
(d) Tr (ABC ) = Tr (BCA) = Tr (CAB).
(e) In general, Tr (ABC ) ̸= Tr (BAC ).

Question: Can we define the trace of a linear operator T ?

(IIT) Elementary Linear Algebra (2024-25 @IITH) 13 / 36

Trace of a linear operator

Definition
The trace of a linear operator T : V → V is defined to be the sum of its
eigenvalues.

We are taking of sum of complex roots here.

For example, if the characteristic polynomial of T is h(x) = x 2 + 1, then
Tr (T ) = 0.
Question: Will this definition agree with the trace of a matrix
representing T ?

(IIT) Elementary Linear Algebra (2024-25 @IITH) 14 / 36

The usual dot product

Let V be a finite dimensional real vector space.

Given vectors α = (x1 , · · · , xn ) and β = (y1 , · · · , yn ) in V , we define the
dot product
α · β = x1 y1 + · · · + xn yn .
It satisfies:
(a) (α + α′ ) · β = α · β + α′ · β.
(b) (cα) · β = c(α · β).
(c) (Symmetry) α · β = β · α.
(d) (Positivity) α · α ≥ 0 and equality holds if and only if α = 0.

(IIT) Elementary Linear Algebra (2024-25 @IITH) 15 / 36

Inner product

Definition
An inner product on a real vector space V is a map

<, >: V × V → R

satisfying the following 3 axioms for all u, v , w ∈ V and all scalars

c, d ∈ R.
(i) (Bilinearity)

< cu + dv , w > = c < u, w > +d < v , w >

< w , cu + dv > = c < w , u > +d < w , v >

(ii) (Symmetry) < u, v >=< v , u > .

(iii) (Positivity) < u, u > ≥ 0 and equality holds if and only if α = 0.

(IIT) Elementary Linear Algebra (2024-25 @IITH) 16 / 36

Inner product

Definition
√
We refer to < u, u > as the norm of the vector u. It is denoted by the
symbol ||u||.

Inner product ⇐⇒ norm.

1
||u + v ||2 − ||u − v ||2 .


< u, v >=
4

(IIT) Elementary Linear Algebra (2024-25 @IITH) 17 / 36

Inner product

Example
We define the following inner product on R2

< v , w >= 2v1 w1 + 7v2 w2 .

Verify that it satisfies bilinearity, symmetry and positivity.

Recall that ||v ||usual norm = 1 is the unit circle.
Question: What is the locus of the curve ||v ||this norm = 1 ?

(IIT) Elementary Linear Algebra (2024-25 @IITH) 18 / 36

Inner product

Question: Which of the following define an inner product on R2

(a) < v , w >= v1 + w1 .
(b) < v , w >= v1 w1 + v1 w2 + v2 w1 + v2 w2 .
(c) < v , w >= (v1 + w1 )2 − (v1 − w1 )2 .
(d) < v , w >= 2026v1 w1 + 2025(v1 w2 + w1 v2 ) + 2026v2 w2 .

(IIT) Elementary Linear Algebra (2024-25 @IITH) 19 / 36

Inner product

Example
Let V = Rn×n be the space of n × n matrices. For any A and B in V ,
define
< A, B >= Tr (At B).
Then <, > is an inner product on V . Here
n
X
t
(A B)ik = aji bjk .
j=1

Bilinear, symmetric and positivity. For any matrix A, the norm

v
u n X n
p uX
t
||A|| = Tr (A A) = t aij2 .
i=1 j=1

(IIT) Elementary Linear Algebra (2024-25 @IITH) 20 / 36

Inner product

Example
Let W be a vector space with an inner product <, > . Let T : V → W be
any injective linear transformation. Then <, > induces an inner product on
V as follows:
< u, v >induced =< T (u), T (v ) >

Bilinear, symmetric and positivity.

(IIT) Elementary Linear Algebra (2024-25 @IITH) 21 / 36

The L2 norm

Example
Let [a, b] ⊂ R be a closed interval. Let V be the vector space C [a, b]
consisting of all continuous real valued functions on [a, b]. We define an
inner product <, > on this space as following
Z b
< f , g >= f · g dt.
a

The associated norm is Z b

||f || = f 2 dt.
a

It is called the L2 norm of the function f over the interval [a, b].

(IIT) Elementary Linear Algebra (2024-25 @IITH) 22 / 36

The Cauchy-Schwartz inequality

Theorem
Let (V , <, >) be an inner product space. For every α, β ∈ V , we have

< α, β >≤ ||α|| · ||β||

Equality holds if and only if α and β are parallel vectors.

(IIT) Elementary Linear Algebra (2024-25 @IITH) 23 / 36

Applications of Cauchy-Schwartz inequality

Let V = Rn and consider the usual dot product

< u, v >= u1 v1 + · · · + un vn .

Cauchy-Schwartz implies

(u1 v1 + · · · + un vn )2 ≤ (u12 + · · · + un2 )(v12 + · · · + vn2 ).

(IIT) Elementary Linear Algebra (2024-25 @IITH) 24 / 36

Applications of Cauchy-Schwartz inequality

For matrices, we get

Tr (At B) ≤ ||A|| · ||B||.
For the L2 -norm defined earlier on the space C[a, b], we get
s
Z b Z b Z b
f · g dt ≤ 2
f dt · g 2 dt.
a a a

Since expectation of continuous random variables is same as an integral,

we deduce q
|E [XY ]| ≤ E [X 2 ]E [Y 2 ].

(IIT) Elementary Linear Algebra (2024-25 @IITH) 25 / 36

The triangle inequality

Corollary (Triangle inequality)

For every α, β ∈ V ,
||α + β|| ≤ ||α|| + ||β||

Proof.
We have

||α + β||2 =< α + β, α + β >

= ||α||2 + ||β||2 + 2 < α, β >
≤ ||α||2 + ||β||2 + 2||α|| · ||β||
= (||α|| + ||β||)2

(IIT) Elementary Linear Algebra (2024-25 @IITH) 26 / 36

Proof of Cauchy-Schwartz
Proof.
Let t be any real number. We know

0 ≤ ||α + tβ||2 =< α + tβ, α + tβ >

= ||α||2 + t 2 ||β||2 + 2t < α, β >

Completing the square, we get

< α, β > 2 < α, β > 2

   
2
t||β|| + + ||α|| − ≥ 0.
||β|| ||β||

Since t was arbitrary, we get

< α, β >2 ≤ ||α||2 · ||β||2 .

This completes the proof.

(IIT) Elementary Linear Algebra (2024-25 @IITH) 27 / 36

A matrix associated to an inner product

Let (V , <, >) be an n-dimensional inner product space i.e. there is a map

<, >: V × V → R

satisfying the following 3 axioms for all u, v , w ∈ V and all scalars

c, d ∈ R.
(i) (Bilinearity)

< cu + dv , w > = c < u, w > +d < v , w >

< w , cu + dv > = c < w , u > +d < w , v >

(ii) (Symmetry) < u, v >=< v , u > .

(iii) (Positivity) < u, u > ≥ 0 and equality holds if and only if α = 0.

(IIT) Elementary Linear Algebra (2024-25 @IITH) 28 / 36

A matrix associated to an inner product

Let B = {α1 , · · · , αn } be a basis of V . Consider the scalars:

a11 =< α1 , α1 >

a12 =< α1 , α2 >
··· = ······
aij =< αi , αj >
··· = ······
ann =< αn , αn >

Let A be the n × n matrix whose (i, j)th entry is aij .

Question: What is the size of A ?
Question: What kind of matrix is A ?

(IIT) Elementary Linear Algebra (2024-25 @IITH) 29 / 36

Recovering the inner product from the matrix

Let α = x1 α1 + · · · + xn αn and α′ = x1′ α1 + · · · + xn′ αn . We recover the

inner product < α, α′ > as following

< α, α′ > =< x1 α1 + · · · + xn αn , x1′ α1 + · · · + xn′ αn >

= x1 x1′ < α1 , α1 > +x2 x1′ < α2 , α1 > + · · · + xn xn′ < αn , αn >
X
= xi xj′ aij
i,j

We can rewrite this more compactly as

< α, α′ >:= [α]tB A[α′ ]B .

(IIT) Elementary Linear Algebra (2024-25 @IITH) 30 / 36

Recovering the inner product from the matrix

Example
Let V = Rn and B = {e1 , · · · , en } be the standard ordered basis. Let
<, > be the usual dot product

< x1 α1 + · · · + xn αn , x1′ α1 + · · · + xn′ αn >:= x1 x1′ + x2 x2′ + · · · + xn xn′ .

To find the matrix associated to this inner product, we define the scalars

aij :=< ei , ej >= δij .

Thus, the matrix of the dot product (with respect to the standard ordered
basis) is In .

(IIT) Elementary Linear Algebra (2024-25 @IITH) 31 / 36

Symmetric positive definite matrix

Question: Does any symmetric matrix A determines an inner product?

Example
Let V = R2 and let B = {e1 , e2 }. Set
 
−1 0
A= .
0 1

Does < α, β >:= [α]tB A[α′ ]B defines an inner product on R2 ?

Symmetric ? Bilinear ? Positivity?

(IIT) Elementary Linear Algebra (2024-25 @IITH) 32 / 36

Symmetric positive definite matrices

Definition
We say that a real n × n matrix A is (symmetric) positive definite if it is
symmetric and X t KX > 0 for all nonzero X ∈ Rn .

Theorem
Any inner product on Rn is given by

< α, β >= αt Aβ, for α, β ∈ Rn ,

where A is a symmetric positive definite n × n matrix.

(IIT) Elementary Linear Algebra (2024-25 @IITH) 33 / 36

Symmetric positive definite matrices

Corollary
Upto a choice of basis B, any inner product on an n-dimensional vector
space V is given by

< α, β >= [α]tB A[β]B , for α, β ∈ V ,

where A is a symmetric positive definite n × n matrix.

Exercise: A symmetric positive definite matrix is non-singular.

(IIT) Elementary Linear Algebra (2024-25 @IITH) 34 / 36

Symmetric positive definite matrices

Question: Identifying a 1 × 1 matrix symmetric positive definite?

Question: Identifying a 2 × 2 matrix symmetric positive definite?

(IIT) Elementary Linear Algebra (2024-25 @IITH) 35 / 36

Sylvester’s criterion

Theorem
A real symmetric n × n matrix is symmetric and positive definite if and
only if all its principal minors have positive determinants.

(IIT) Elementary Linear Algebra (2024-25 @IITH) 36 / 36