Linear Algebra

Jenish P.

July 19, 2024

ii
Contents

1 Vector Spaces 1
1.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Bases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Dimension of a Vector Space . . . . . . . . . . . . . . . . . . 3
1.4 Sums and Direct Sums . . . . . . . . . . . . . . . . . . . . . . 4

2 Matrices 7
2.1 Matrices and Arithmetic Operations on Them . . . . . . . . . 7
2.2 The Inverse of a Matrix . . . . . . . . . . . . . . . . . . . . . 9

3 Linear Mappings 13

4 Solving Linear Equations 15

4.1 Systems of Linear Equations . . . . . . . . . . . . . . . . . . . 15
4.2 Row Reduction and Echelon Forms . . . . . . . . . . . . . . . 16
4.3 Vector Equations . . . . . . . . . . . . . . . . . . . . . . . . . 16
4.4 The Matrix Equation Ax = b . . . . . . . . . . . . . . . . . . 17
4.5 Solution Sets of Linear Systems . . . . . . . . . . . . . . . . . 18
4.6 Linear Independence . . . . . . . . . . . . . . . . . . . . . . . 18
4.7 Linear Transformations . . . . . . . . . . . . . . . . . . . . . 19

5 Matrices 21
5.1 Subspaces of Rn . . . . . . . . . . . . . . . . . . . . . . . . . 21

6 Determinants 23
6.1 Determinants and some other Concepts . . . . . . . . . . . . 23
6.2 Properties of Determinants . . . . . . . . . . . . . . . . . . . 24

7 Vector Spaces 27
7.1 Coordinate Systems . . . . . . . . . . . . . . . . . . . . . . . 28
7.2 Rank . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
7.3 Change of Basis . . . . . . . . . . . . . . . . . . . . . . . . . . 29

iii
iv CONTENTS

8 Eigenvalues and Eigenvectors 31

8.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
8.2 Diagonalization . . . . . . . . . . . . . . . . . . . . . . . . . . 32
8.3 Complex Eigenvalues . . . . . . . . . . . . . . . . . . . . . . . 32

9 Orthogonality 33
9.1 Orthogonal Sets . . . . . . . . . . . . . . . . . . . . . . . . . . 33
9.2 Orthogonal Projections . . . . . . . . . . . . . . . . . . . . . . 34

10 Symmetric Matrices and Quadratic Forms 37

1

Vector Spaces

1.1 Definitions
Definition. A vector space over a field F is a set V with two closed op-
erations, vector addition or simply addition and scalar multiplication, that
satisfy the following axioms:

1. Associativity of addition;

2. Commutativity of addition;

3. Identity element of addition;

4. Inverse elements of addition;

5. Compatibility of scalar multiplication with field multiplication;

a(bv) = (ab)v

6. Identity element of scalar multiplication;

7. Distributivity of scalar multiplication with respect to vector addition;

8. Distributivity of scalar multiplication with respect to field addition.

Definition. A subspace of a vector space V is a subset H of V that has

three properties:

• The zero vector of V is in H.

• H is closed under vector addition.

• H is closed under multiplication by scalars.

If a subspace only contains the zero vector 0, it is called a zero subspace and
written as {0}.

1
2 1. VECTOR SPACES

Definition. Given vectors v1 , v2 , · · · , vn in an arbitrary vector space V and

given scalars c1 , c2 , · · · , cn , the vector y defined by

y = c1 v1 + · · · + cn vn

is called a linear combination of v1 , v2 , · · · , vn using weights c1 , c2 , · · · , cn .

It is easy to verify that the set W of all linear combinations of v1 , v2 , · · · , vn
is a subspace of V .

The subspace W as above is called the subspace generated by v1 , v2 , · · · , vn .

If W = V , then we say that v1 , v2 , · · · , vn generate V .
Definition. The dot product or scalar product of two vectors is defined
as the sum of the product of their corresponding components. It has the
following properties:
1. The commutativity of dot product.
2. The distributivity of dot product over vector addition and vice versa.
3. The associativity of scalar multiplication and dot product.
Two vectors are perpendicular or orthogonal if their dot product is zero.

1.2 Bases
Definition. An indexed set of vectors {v1 , · · · , vp } ∈ V is said to be linearly
independent if the vector equation

c1 v1 + c2 v2 + · · · + cp vp = 0

has only the trivial solution. Otherwise it’s said to be linearly dependent.
Definition. A collection of linearly independent vectors that generated the
vector space V is called a basis of V .
Definition. Let V be a vector space and B be a basis for it. The coordinate
with respect to basis B of an element is a n-tuple of numbers such that the
desired element can be expressed with the linear combination of the n-tuple
and basis elements. By the definition of basis, the coordinate of any element
is unique.
Definition. The maximal subset of linearly independent elements is a subset
of any collection of vectors of a vector space that adding another element
from the original collection that is not in this maximal subset will result in
the loss of linearly independence.
Theorem 1.2.1. The maximal subset of linearly independent elements of
any collection of vectors that generate the vector space is a basis for it.
1.3. DIMENSION OF A VECTOR SPACE 3

1.3 Dimension of a Vector Space

Theorem 1.3.1. Let V be a vector space over the field K. Let B be a basis
that has m elements. Then any collection of more than m vectors in V are
linearly dependent.

Proof. Let the basis B be {v1 , · · · , vm } and the collection of n vectors(n >
m) in V be w1 , · · · , wn . Assume this collection of vectors are linearly inde-
pendent. Since B is a basis, the equation

w1 = a1 v1 + · · · + am vm

holds for some elements in the field K that are not all zero. Without loss
of generality, we can assume that a1 ̸= 0(otherwise we could just rearrange
the equation to make it so). Solve for v1

v1 = a−1 −1 −1
1 w1 − a1 a2 v2 − · · · a1 am vm

Then the subspace generated by w1 , v2 , · · · , vm is all of V since it contains

v1 . Now assume by induction that there is an integer r, 1 ⩽ r < m, such that
after a suitable renumbering of v1 , · · · , vm , the elements w1 , · · · , wr , vr+1 , · · · , vm
generate V . Then the equation

wr+1 = b1 w1 + · · · + br wr + cr+1 vr+1 + · · · + cm vm

holds for some element in K. cr+1 , · · · , cm can’t be all zero, since then
we have expressed wr+1 as a linear combination of w1 , · · · , wr , contradict
out assumption. Without loss of generality, assume that cr+1 is not zero.
Repeat the same process when we solve for v1 to solve for vr+1 , we then find
that vr+1 is in the subspace generated by w1 , · · · , wr+1 , vr+2 , · · · , vm . By
the induction assumption, these vectors also generate V . Therefore vectors
w1 , · · · , wm generate V , and for n > m we can expressed wn as a linear
combination of w1 , · · · , wm . The theorem is then proved.

Corollary. Any two basis of the same vector space have the same number
of elements.

Definition. The number of vectors in a basis is called the dimension of this

vector space.

Definition. The maximal set of linearly independent elements of a vector

space V is a collection of linearly independent vectors such that adding extra
"outside" vectors would result in the loss of its linearly independence.

Theorem 1.3.2. The maximal set of linearly independent elements of a

vector space V is a basis for it.
4 1. VECTOR SPACES

Proof. Let this maximal set be {v1 , · · · , vn } and w be a element of the

vector space V . By hypotheses the union of the maximal set and w is
linearly dependent, then

x0 w + x1 v1 + · · · + xn vn = 0

has a nontrivial solution. By the linearly independence of {v1 , · · · , vn }, x0

is not zero. Then we can solve for w
x1 xn
w=− v1 − · · · − vn
x0 x0

Corollary. For a vector space of dimension n, any collection of n linearly

independent vectors is a basis for it.

Corollary. Let V be a vector space and let W be a subspace for it. If

dim W = dim V , then V = W .

Corollary. Any linearly independent collections of a vector space can be

made to a basis for it by adding some (or 0) vectors that are/is linearly
independent to its elements.

Theorem 1.3.3. The dimension of the subspace W that does not consist of
the zero vector alone of a vector space V is no greater than dim V .

1.4 Sums and Direct Sums

Definition. If U and W are subspaces of a vector space V , then U + W ,
the set of all elements u + w with u ∈ U and w ∈ W , is a subspace of V ,
said to be generated by U and W , and called the sum of U and W . V is a
direct sum of U and W if for any v ∈ V there exist unique elements u ∈ U
an w ∈ W such that v = u + w. When V is the direct sum of subspaces U ,
W , we write
V =U ⊕W

Theorem 1.4.1. Let V be a vector space over the field K, and let U , W be
subspaces. If U + W = V and U ∩ W = {0}, then V = U ⊕ W .

Theorem 1.4.2. Let V be a finite dimensional vector space over the field K.
Let W be a subspace. Then there exists a subspace U such that V = W ⊕ U .

Theorem 1.4.3. If V is a finite dimensional vector space and is the direct

sum of subspaces U and W , then

dim V = dim W + dim U

1.4. SUMS AND DIRECT SUMS 5

Definition. The direct product of two vector spaces U and W over the field
K is the cartesian product of elements in U and W , i.e (u1 , w1 ) ∈ U ×W . The
vector addition and scalar multiplication are defined to be componentwise.

Immediately, the direct product of two vector spaces over the same field
is a vector space.

Theorem 1.4.4.

dim(U × W ) = dim U + dim W

Proof. For two basis in each space {u1 , · · · , un } and {w1 , · · · , wm }, the col-
lection of vectors in U ×W consists of {(u1 , 0), · · · , (un , 0), (0, w1 ), · · · , (0, wm )}
is a basis for U × W of n + m elements.

The notion of direct sum and direct product can be extended to several
factors with and . In this circumstance vector addition and scalar
P Q

multiplication are also defined to be componentwise.

6 1. VECTOR SPACES
2

Matrices

2.1 Matrices and Arithmetic Operations on Them

Definition. A diagonal matrix is a square matrix whose nondiagonal entries
are zero.

Definition. Two matrices are equal if they have the same size and each
entries are equal.

Definition. The sum of two matrices is the sum of each corresponding

entries in these two matrices. Thus the sum is defined only when they have
the same size.

Definition. The scalar multiple of a matrix has entries of the product of

the scalar and each corresponding original entries.

Theorem 2.1.1. The set of matrices of the same size with respect to matrix
addition and scalar multiplication over the field of real numbers is a vector
space.

Definition. A square matrix is called lower triangular if all the entries

above the main diagonal are zero. Similarly, a square matrix is called upper
triangular if all the entries below the main diagonal are zero. A triangular
matrix is one that is either lower triangular or upper triangular. A matrix
that is both upper and lower triangular is called a diagonal matrix.

Definition. If A is an m×n matrix, and if B is an n×p matrix with columns

b1 , · · · , bp , then the product AB is the m × p matrix whose columns are
Ab1 , · · · , Abp . Multiplication of matrices corresponds to composition of
linear transformations.

Theorem 2.1.2. The multiplication has the following properties:

• Associativity of multiplication;

7
8 2. MATRICES

• Left distribution;

• Right distribution;

• Associativity over scalar multiplication;

• Identity for matrix multiplication; i.e. If A is a matrix of size m × n,

then
Im A = A = AIn

where In is the n × n identity matrix.

Definition. In general, matrix multiplication is not commutative and the

cancellation law do not hold. When two matrices’ multiplication is commu-
tative, they are said to be commute with one another. Also, if a product
AB is the zero matrix, in general it does not mean that either A = 0 or
B = 0.

Definition. If A is an m × n matrix and k is a positive integer, then Ak

denoted the product of k copies of A, i.e. the kth power of A. The 0th
power of a matrix is the identity matrix.

Definition. If A is an m×n matrix, the transpose of A is the n×m matrix,

denoted AT , whose columns are formed from the corresponding rows of A.

Definition. If A is an m×n matrix, the transpose of A is the n×m matrix,

denoted AT , whose columns are formed from the corresponding rows of A.

Definition. A matrix is said to be symmetric if it is equal to its transpose.

Definition. The unit n × n matrix, denoted by In , is the square matrix

having all its components equal to 0 except the diagonal components, equal
to 1.

Theorem 2.1.3. The transpose operation has the following properties:

• (AT )T = A;

• (A + B)T = AT + BT ;

• Associativity with scalar multiplication;

• (AB)T = BT AT , that is, the transpose of a product of arbitrary num-

ber of matrices equals the product of their transpose in the reverse
order.
2.2. THE INVERSE OF A MATRIX 9

2.2 The Inverse of a Matrix

Definition. If A is an n × n matrix, then if

AA−1 = In

we say that A is invertible and A−1 an inverse of A. The inverse of a matrix

is unique. If a matrix is not invertible, it is called a singular matrix.
Theorem 2.2.1. A matrix A is invertible only if det(A) ̸= 0, and in this
case
1
A−1 = Adj(A)
det(A)
Theorem 2.2.2. If A is an invertible n × n matrix, then for each b ∈ Rn ,
the equation Ax = b has the unique solution x = A−1 b.
Theorem 2.2.3. • The inverse of the inverse of a invertible matrix is
the matrix itself.

• The inverse of the product of arbitrary number of invertible square

matrices is the product of the inverse of themselves multiplied in the
reverse order.

• The transpose of a invertible matrix is also invertible. Moreover, the

inverse of a matrix’s transpose is the transpose of the matrix’s inverse.
Definition. An elementary matrix is a matrix obtained by performing a
single elementary row operation on a identity matrix.
Theorem 2.2.4. If an elementary row operations is performed on an m × n
matrix A, the resulting matrix can be written as EA, where the m × m
matrix E is created by performing the same row operation on Im .
Theorem 2.2.5. Each elementary matrix E is invertible. The inverse of E
is the elementary matrix of the same type that transforms E back into I.
Theorem 2.2.6. An n×n matrix A is invertible iff A is a row equivalent to
In , and in this case, any sequence of elementary row operations that reduces
A to In also transforms In into A−1 .
Theorem 2.2.7 (The Invertible Matrix Theorem). Let A be a square n × n
matrix. Then the following statements are equivalent.
• A is an invertible matrix.

• A is row equivalent to the n × n identity matrix.

• A has n pivot positions.

10 2. MATRICES

• The equation Ax = 0 has only the trivial solution.

• The columns of A form a linearly independent set.

• The linear transformation x 7→ Ax is injective.

• The equation Ax = b has at least one solution for each b in Rn .

• The columns of A span Rn .

• The linear transformation x 7→ Ax maps Rn onto Rn .

• There is an n × n matrix C such that CA = I.

• There is an n × n matrix D such that AD = I.

• AT is an invertible matrix.

• The columns of A form a basis of Rn .

• Col A = Rn

• dim Col A = n

• rank A = n

• Nul A = {0}

• dim Nul A = 0

Proposition. Let A and B be square matrices. If AB = I, then A and B

are both invertible, with B = A−1 and A = B−1

Definition. A linear transformation T : Rn → Rn is invertible if there exists

a function S : Rn → Rn such that

(∀x ∈ Rn ) S(T (x)) = x

(∀x ∈ Rn ) T (S(x)) = x

and S is called the inverse of T and denoted T −1 .

Theorem 2.2.8. A linear transformation is invertible iff its standard matrix

is invertible. In this case its inverse is unique.
2.2. THE INVERSE OF A MATRIX 11

Theorem 2.2.9. If A is m × n and B is n × p, then

Row1 (B)
 

 Row2 (B) 
i
h 
AB = Col1 (A) Col2 (A) · · · Coln (A)  .. 
.
 
 
Rown (B)
= Col1 (A) Row1 (B) + · · · Coln (A) Rown (B)

Definition. A block matrix is a partitioned matrix with zero blocks off the
main diagonal. Such matrix is invertible iff each block on the diagonal is
invertible.

Definition. A factorization of a matrix is an equation that expresses it as

a product of two or more matrices.

Definition. An square matrix is said to be strictly diagonally dominant if

the absolute of each diagonal entry exceeds the sum of the absolute values
of the other entries in the same row.
12 2. MATRICES
3

Linear Mappings

Definition. Let V, W be vector spaces over K, and let F : V → W be a

linear map. The kennel of F , denoted Ker F , is the set of elements v ∈ V
such that F (v = 0).
The kernel of a linear map on vector space is a subspace of the vector
space.
Proposition. The following two conditions are equivalent
1. The kernel of F is {0}.

2. F is injective.
Proof.

Theorem 3.0.1. Let F : V → W be a linear map whose kernel is {0}. If

v1 , · · · , vn are linearly independent elements of V , then F (v1 ), · · · , F (vn )
are linearly independent elements of W .
Proof. content...

Theorem 3.0.2. Let V be a vector space. Let L : V → W be a linear map

of V into another space W . Then

dim V = dim Ker L + dim Im L

Proof. content...

Theorem 3.0.3. Let L : V → W be a linear map. Assume that

dim V = dim W

If Ker L = {0}, or if Im L = W , then L is bijective.

Proof.

13
14 3. LINEAR MAPPINGS
4

Solving Linear Equations

4.1 Systems of Linear Equations

Definition. A linear equation in the variables x1 , · · · , xn is an equation
that can be written in the form
a1 x1 + a2 x2 + · · · + an xn = b
where b and coefficients ai are real or complex numbers. A linear system is
a collection of one or more linear equations involving the same variables. A
solution of the system is a list of numbers that makes each equation a true
statement when their values are substituted for x1 , · · · , xn respectively. The
set of all possible solutions is called the solution set of the linear system.
Two linear systems are called equivalent if they have the same solution set.
Definition. A linear system is consistent if it has either one solution or
infinitely many solutions. If it has no solution, it is called inconsistent.
Definition. The coefficient matrix is the matrix where the coefficients of
each variable in a system aligned in columns. If additionally the coefficient
of the right-hand side of equations are added to the coefficient matrix, a new
matrix called augmented matrix is generated.
Definition. Elementary row operations on a matrix include:
• (Replacement) Replace one row by the sum of itself and a multiple of
another row.
• (Interchange) Interchange two rows.
• (Scaling) Multiply all entries in a row by a nonzero constant.
Two matrices are row equivalent if there is a sequence of elementary opera-
tions that transforms one matrix into the other.
Theorem 4.1.1. If the augmented matrices of two linear systems are row
equivalent, then the two systems have the same solution set.

15
16 4. SOLVING LINEAR EQUATIONS

4.2 Row Reduction and Echelon Forms

Definition. A rectangular matrix is in echelon form if it has the following
properties:

• All nonzero rows are above any rows of all zeros;

• Each leading entry of a row is in a column to the right of the leading

entry of the row above it;

• All entries in a column below a leading entry are zero.

If a matrix in echelon form satisfies the following additional conditions, then

it is in reduced echelon form:

• The leading entry in each nonzero row is 1;

• Each leading 1 is the only nonzero entry in its column.

Theorem 4.2.1. Each matrix is row equivalent to an unique reduced echelon

matrix.

If a matrix A is row equivalent to an (reduced)echelon matrix U , U is

called an (reduced) echelon form of A. The abbreviation RREF and REF
are used for reduced (row) echelon form and (row) echelon form respectively.

Definition. A pivot position in a matrix A is a location in A that corre-

sponds to a leading entry in an echelon form of A. A pivot column is a
column of A that contains a pivot position.

Theorem 4.2.2. A linear system is consistent iff the rightmost column of

the augmented matrix is not a pivot column, that is, iff an echelon form of
the augmented matrix has no row of the form
h i
0 ··· 0 b with b nonzero

If a linear system is consistent, then the solution set contains either

• a unique solution, when there are no free variables.

• infinitely many solutions, when there is at least one free variable.

4.3 Vector Equations

Definition. A matrix with only one column is called a column vector, or
simply a vector.
4.4. THE MATRIX EQUATION Ax = b 17

Definition. If v1 , v2 , · · · , vp are in Rn , then the set of all linear combi-

nations of them is denoted by Span{v1 , · · · , vp } and is called the subset of
Rn spanned (or generated) by v1 , · · · , vp . That is, Span{v1 , · · · , vp } is the
collection of all vectors that can be written in the form

c1 v1 + · · · + cp vp

with c1 , c2 , · · · , cp scalars.

4.4 The Matrix Equation Ax = b

Definition. If A is an m × n matrix, with columns a1 , · · · , an , and if x is in
Rn , then the product of A and x, denoted by Ax, is the linear combination
of the columns of A using the corresponding entries in x as weights, that is,
 
h i x1
Ax = a1 a2 · · · an · · · = x1 a1 + x2 a2 + · · · + xn an
 
xn

Ax is defined only if the number of columns of A equals the number of

entries in x.

Definition. Equations having the form Ax = b are called matrix equations.

Theorem 4.4.1. If A is an m × n matrix, with columns a1 , · · · , an , and b

is in Rm , the matrix equation

Ax = b

has the same solution set as the vector equation

x1 a1 + x2 a1 + · · · + xn an = b

which has the same solution set as the system of linear equations whose
augmented matrix is h i
a1 a2 · · · an b

Definition. A set of vectors {v1 , · · · , vp } in Rm spans (or generates) Rm if

Span{v1 , · · · , vp } = Rm .

Theorem 4.4.2. Let A be an m × n coefficient matrix. Then the following

statements are logically equivalent, that is, for a particular A, either they
are all true statements or they are all false.

• For each b in Rm , the equation Ax = b has a solution.

• The columns of A spans Rm .

18 4. SOLVING LINEAR EQUATIONS

• A has a pivot position in every row.

Theorem 4.4.3. If A is an m × n matrix, u and v are vectors in Rn , and

c is a scalar, then

• A(u + v) = Au + Av.

• A(cu) = c(Au).

4.5 Solution Sets of Linear Systems

Definition. A system of Linear equations is said to be homogeneous if it
can be written in the form Ax = 0. Such a system always has at least
one solution, namely, x = 0, and this solution is usually called the trivial
solution. A homogeneous equation has a nontrivial solution iff the equation
has at least one free variable.

Definition. Vector addition can be considered as a translation. e.g. the

vector v is translated by p to v + p.

Definition. A parametric vector equation can be written as

x = su + tv (s, t ∈ R)

which describes explicitly the spanned plane by u and v. Whenever a so-

lution set is described explicitly with vectors, we say that the solution is in
parametric vector form.

Theorem 4.5.1. Suppose the equation Ax = b is consistent for some given

b, and let p be a nonzero solution. Then the solution set of it is the set of all
vectors of the form w = p+vh , where vh is any solution of the homogeneous
equation Ax = 0.

4.6 Linear Independence

Definition. An indexed set of vectors {v1 , · · · , vp } in Rn is said to be
linearly independent if the vector equation

x1 v1 + x2 v2 + · · · + xp vp = 0

has only the trivial solution. The set {v1 , · · · , vp } is said to be linearly
dependent if there exist weights c1 , · · · , cp , not all zero, such that

c1 v1 + c2 v2 + · · · + cp vp = 0

and this equation is called a linear dependence relation among v1 , · · · , vp .

4.7. LINEAR TRANSFORMATIONS 19

Theorem 4.6.1. The columns of a matrix A are linearly independent iff

the equation Ax = 0 has only the trivial solution.
Theorem 4.6.2. A set of two vectors {v1 , v2 } is linearly dependent iff one
of the vectors is a multiple of the other.
Theorem 4.6.3. An indexed set S = {v1 , · · · , vp } of two or more vectors is
linearly dependent iff at least one of the vectors in S is a linear combination
of the others.
Theorem 4.6.4. Any set {v1 , · · · , vp } in Rn is linearly dependent if p >
n(Same as the criterion for the existence of solutions in a system of equa-
tions).
Proof. Since p > n, there are more variables than equations, and therefore
nontrivial solutions exist.

Theorem 4.6.5. If a set S = {v1 , · · · , vp } in Rn contains the zero vector,

then the set is linearly dependent.

4.7 Linear Transformations

Definition. A transformation(or function or mapping) from Rn to Rm is a
rule that assigns to each vector x ∈ Rn a vector T (x) ∈ Rm . Rn is called the
domain of T , and Rm is called the codomain of T . For x ∈ Rn , the vector
T (x) ∈ Rm is called the image of x under T . The set of all images T (x) is
called the range of T .
Example 4.1. Given a scalar r, define T : R2 → R2 by T (x) = rx. T is
called a contraction when 0 ⩽ r ⩽ 1 and a dilation when r > 1.
Theorem 4.7.1. Let T : Rn → Rm be a linear transformation. Then there
exists a unique matrix A such that

T (x) = Ax ∀x ∈ Rn

In fact, A is the m × n matrix whose jth column is the vector T (ej ), where
ej is the jth column of the identity matrix in Rn .
h i
A = T (e1 ) · · · T (en )

The matrix A is called the standard matrix for the linear transformation T .
Theorem 4.7.2. Let T : Rn → Rm be a linear transformation. Then T is
injective iff the equation T (x) = 0 has only the trivial solution.
Theorem 4.7.3. Let T : Rn → Rm be a linear transformation and let A be
the standard matrix for T . Then
20 4. SOLVING LINEAR EQUATIONS

• T is surjective iff the columns of A span Rm ;

• T is injective iff the columns of A are linearly independent.

Definition. If there is a matrix A such that

xk+1 = Axk for k = 0, 1, 2, · · ·

then the equation above is called a linear difference equation (or recurrence
relation).
5

Matrices

5.1 Subspaces of Rn
Definition. A subspace of Rn is any set H ∈ Rn that has three properties:

• The zero vector is in H;

• For each vector u and v in H, their sum is in H (addition is closed on

H);

• For each u in H and each scalar c, the vector cu is in H (scalar

multiplication is closed on H).

Definition. The column space of a matrix A is the set Col A of all linear
combinations of the columns of A.

Definition. The null space of a matrix A is the set Nul A of all solutions
to the homogeneous equation Ax = 0.

Theorem 5.1.1. The null space of a m × n matrix is a subspace of Rn .

Definition. A basis for a subspace H of Rn is a linearly independent set in

H that spans H.

Example 5.1. The standard basis for Rn are vectors e1 , · · · , en , where

1 0
   

0
 
0
 
e1 =   , · · · , en =  . 
 
.
. .
. .
0 1

Theorem 5.1.2. The pivot columns of a matrix A form a basis for the
column space of A.

21
22 5. MATRICES

Definition. Suppose the set B = {b1 , · · · , bp } is the basis for a subspace

H. For each x ∈ H, the coordinates of x relative to the basis B are the
weights c1 , · · · , cp such that x = c1 b1 + · · · + cp bp , and the vector in Rp
 
c1
[x]B = · · ·
 
cp

is called the coordinate vector of x relative to B.

Definition. The dimension of a nonzero subspace H, denoted by dim H,

is the number of vectors in any basis for H. The dimension of the zero
subspace {0} is defined to be zero.

Definition. The rank of a matrix A, denoted by rank A, is the dimension

of the column space of A.

Theorem 5.1.3 (The Rank Theorem). If a matrix A has n columns, then

rank A + dim Nul A = n.

Proof. The nonpivot columns correspond to the free variables in Ax = 0,

and since the number of pivot columns plus the number of nonpivot columns
are the number of columns in the matrix, the proof completes.

Theorem 5.1.4 (The Basis Theorem). Let H be a p-dimensional subspace

of Rn . Any linearly independent set of exactly p elements in H is automati-
cally a basis for H. Also, any set of p elements of H that spans H is a basis
for H.
6

Determinants

6.1 Determinants and some other Concepts

Definition. The determinant of the matrix A
" #
a b
c d

denoted det A and equals ad − bc. Determinant is only defined for a square
matrix, but the procedure above can be repeated on higher dimension ma-
trices, for example
 
a b c d        
e f g h e g h e f h e f g
f g h
det   = a det  j k l −b det  i k l +c det  i j l −d det  i j k 
        
i j k l
n o p m o p m n p m n o
m n o p

By the Leibniz formula for the determinant of an n × n matrix A is

n
det(A) = (sgn(σ) ai,σi )
X Y

σ∈Sn i=1

Here the sum is computed over all permutations σ of the set {1, 2, . . . , n}.
A permutation is a function that reorders this set of integers. The value in
the ith position after the reordering σ is denoted by σi . For example, for
n = 3, the original sequence 1, 2, 3 might be reordered to σ = [2, 3, 1], with
σ1 = 2, σ2 = 3, and σ3 = 1. The set of all such permutations (also known
as the symmetric group on n elements) is denoted by Sn .
For each permutation σ, sgn(σ) denotes the signature of σ, a value that
is +1 whenever the reordering given by σ can be achieved by successively
interchanging two entries an even number of times, and −1 whenever it can
be achieved by an odd number of such interchanges.

23
24 6. DETERMINANTS

Definition. If A is a square matrix, then the minor of the entry in the

i-th row and j-th column (also called the (i, j) minor, or a first minor) is
the determinant of the submatrix formed by deleting the i-th row and j-th
column. This number is often denoted Mi,j . The (i, j) cofactor is obtained
by multiplying the minor by (−1)i+j and is denoted Ci,j .

In general, let A be an m × n matrix and k an integer with 0 < k ⩽ m,

and k ⩽ n. A k × k minor of A, also called minor determinant of order k of
A or, if m = n, (n − k)th minor determinant of A, is the determinant of a
k × k matrix obtained from A by deleting m − k rows and n − k columns.
Definition. The matrix formed by all of the cofactors of a square matrix
A is called the cofactor matrix.
Definition. The adjugate is the transpose of the cofactor matrix of it, that
is, if A is a matrix and C is its cofactor matrix, then

Adj(A) = CT

Theorem 6.1.1. For a invertible matrix n × n A

A Adj(A) = det(A)I

or equivalently
1
A−1 = Adj A
det A
Theorem 6.1.2. The determinant of an square matrix can be computed by
a cofactor expansion across any row or down any column. The expansion
across the ith row is

det A = ai1 Ci1 + · · · + ain Cin

Theorem 6.1.3. If A is a triangular matrix, then det A is the product of

the entries on the main diagonal of A.

6.2 Properties of Determinants

Definition. An elementary matrix is called an row replacement if it is ob-
tained from the identity matrix by adding a multiple of one row to another;
it’s called an interchange if it is obtained by interchanging two rows of iden-
tity; and it’s called a scale by r if it is obtained by multiplying a row of
identity by a nonzero scalar r.
Theorem 6.2.1. Let A be a square matrix.
• If a multiple of one row of A is added to another row to produce a
matrix B, then det A = det B.
6.2. PROPERTIES OF DETERMINANTS 25

• If two rows of A are interchanged to produce B, then det B = − det A.

• If one row of A is multiplied by k to produce B, then det B = k ·det A.

or, equivalently, if A is an n × n matrix and E is an n × n elementary

matrix, then
det EA = (det E)(det A)
where det E assumes 1, −1, r respectively for E is a row replacement, an
interchange, and a scale by r.

Theorem 6.2.2. If A is an n × n matrix, then det AT = det A.

Theorem 6.2.3. If A and B are n×n matrices, then det AB = (det A)(det B).

Example 6.1. If all columns except one are held fixed in a square matrix,
then its determinant is a linear function of that one(vector) variable.

Let Ai (b) denote the matrix obtained from A by replacing column i by

the vector b.

Theorem 6.2.4. If A is an invertible n × n matrix. For any b ∈ Rn , then

unique solution x of Ax = b has entries given by

det Ai (b)
xi = , i = 1, 2, · · · , n
det A
Theorem 6.2.5. If A is a 2 × 2 matrix, the area of the parallelogram de-
termined by the columns of A is | det A|. If A is a 3 × 3 matrix, the volume
of the parallelepiped determined by the columns of A is | det A|.

Theorem 6.2.6. Let T : R2 → R2 be the linear transformation determined

by a 2 × 2 matrix A. If S is a parallelogram in R2 , then

{area of T (S)} = | det A| · {area of S}

and similar, if in R3 S is a parallelepiped, then

{volume of T (S)} = | det A| · {volumn of S}

These conclusions hold whenever S has finite area or finite volume.

26 6. DETERMINANTS
7

Vector Spaces

Theorem 7.0.1. If v1 , · · · , vp are in a vector space V , then Span{v1 , · · · , vp }

is a subspace of V and is called the subspace spanned (or generated) by
{v1 , · · · , vp }. Given any subspace H of V , a spanning (or generating) set
for H is a set {v1 , · · · , vp } in H such that H = Span{v1 , · · · , vp }.
Definition. The null space of an m × n matrix A, written as Nul A, is the
set of all solutions to the homogeneous equation Ax = 0.
Theorem 7.0.2. The null space of an m × n matrix A is a subspace of Rn .
Definition. The column space of an m × n matrix, written as Col A, is the
set of all linear combinations of the columns of A.
Theorem 7.0.3. The column space of an m × n matrix is a subspace of Rn .
Theorem 7.0.4. The column space of an m × n matrix A is all of Rm iff
the equation Ax = b has a solution for each b ∈ Rm .
Definition. For a linear transformation T from a vector space V into a
vector space W , the kernel(or null space) of T is the set of all u ∈ V such
that T (u) = 0. The range of T is the set of all vectors in W of the form
T (x) for some x ∈ V . If T can be written as a matrix transformation, then
the kernel and the range of T are just the null space and the column space
of that matrix. Kernel is a subspace of V , and range is a subspace of W .
Theorem 7.0.5. An indexed set {v1 , · · · , vp } of two or more vectors, with
v1 = 0, is linearly dependent iff some vj with j > 1 is a linear combination
of the preceding vectors, v1 , · · · , vj−1 .
Definition. Let H be a subspace of a vector space V . An indexed set of
vectors B = {b1 , · · · , bp } in V is a basis for H if
1. B is a linearly independent set;

2. the subspace spanned by B coincides with H.

27
28 7. VECTOR SPACES

Theorem 7.0.6. Let S = {v1 , · · · , vp } be a set in V and let H = Span{v1 , · · · , vp }.

1. If one of the vectors in S is a linear combination of the remaining

vectors in S, then the set formed from S by removing this vector still
spans H.

2. If H ̸= {0}, some subset of S is a basis for H.

Theorem 7.0.7. The pivot columns of a matrix A form a basis for Col A.

7.1 Coordinate Systems

Theorem 7.1.1. Let B = {b1 , · · · , bp } be a basis for a vector space V .
Then for each x ∈ V , there exists a unique set of scalars c1 , · · · , cn such
that
x = c1 b1 + · · · + cn bn

Definition. Suppose the set B = {b1 , · · · , bp } is a basis for V and x ∈ V .

The coordinate of x relative to the basis B (or the B-coordinates of x) are
the weights c1 , · · · , cn such that x = c1 b1 + · · · + cn bn .

The vector
c1
 
 .. 
[x]B =  . 
cn
is the coordinate vector of x(relative to B), or the B-coordinate vector of x.
The mapping x 7→ [x]B is the coordinate mapping(determined by B).

Definition. The matrix

PB = [b1 , · · · , bp ]

is called the change-of-coordinates matrix from B to the standard basis in

Rn , since for a vector x = c1 b1 + · · · + cn bn we obtain the relationship

x = PB [x]B

Theorem 7.1.2. Let B be a basis for a vector space V . Then the coordinate
mapping x 7→ [x]B is an injective linear transformation from V into Rn .

In general, an injective linear transformation from a vector space V onto

another vector space W is called an isomorphism from V onto W .

Theorem 7.1.3. If a vector space V has a basis B = b1 , · · · , bn , then any

set in V containing more than n vectors must be linearly dependent.
7.2. RANK 29

Theorem 7.1.4. If a vector space V has a basis of n vectors, then every

basis of V must consist of exactly n vectors.

Definition. If V is spanned by a finite set, then V is said to be finite-

dimensional, and the dimension of V , written as dim V , is the number of
vectors in a basis for V . If V is not spanned by a finite set, then V is said
to be infinite-dimensional.

Theorem 7.1.5. Let H be a subspace of a finite-dimensional vector space

V . Any linearly independent set in H can be expanded, if necessary, to a
basis for H. Also, H is finite-dimensional and

dim H ⩽ dim V

Theorem 7.1.6 (The Basis Theorem). Let V be a p-dimensional vector

space, p ⩾ 1. Any linearly independent set of exactly p elements in V is
automatically a basis for V . Any set of exactly p elements that spans V is
a basis for V .

The dimension of Nul A is the number of free variables in Ax = 0, and

the dimension of Col A is the number of pivot columns in A.

7.2 Rank
Definition. The set of all linear combinations of the row vectors in A is
called the row space of A and denoted Row A.

Theorem 7.2.1. If two matrices A and B are row equivalent, then their
row spaces are the same. If B is in echelon form, the nonzero rows of B
form a basis for the row space of A as well as B.

Definition. The rank of A is the dimension of the column space of A.

Theorem 7.2.2 (The Rank Theorem). The dimensions of the column space
and the row space of an m×n matrix A are equal. This common dimension,
the rank of A, also equals the number of pivot positions in A and satisfies
the equation
rank A + dim Nul A = n

7.3 Change of Basis

Theorem 7.3.1. Let B = {b1 , · · · , bn } and C = {c1 , · · · , cn } be bases of a
vector space V . Then there is an n × n matrix P , called the change-of-
C←B
coordinates matrix from B to C, such that

[x]C = P [x]B
C←B
30 7. VECTOR SPACES

The columns of P are the C-coordinate vectors of the vectors in the basis
C←B
B, that is h i
P = [b1 ]C [b2 ]C · · · [bn ]C
C←B

Because the columns of this matrix are linearly independent, since they
are the coordinate vectors of the linearly independent set B, it follows that
P is invertible, and we have
C←B

( P )−1 = P
C←B B←C
8

Eigenvalues and Eigenvectors

8.1 Definition
Definition. An eigenvector of an n × n matrix A is a nonzero vector x
such that Ax = λx for some scalar λ. A scalar λ is called an eigenvalue
of A if there is a nontrivial solution x of Ax = λx; such an x is called an
eigenvector corresponding to λ.
Definition. The set of all solutions of
(A − λI)x = 0
is a subspace of Rn and is called the eigenspace of A corresponding to λ.
Theorem 8.1.1. The eigenvalues of a triangular matrix are the entries on
its main diagonal.
Theorem 8.1.2 (The Invertible Matrix Theorem). Let A be an n × n ma-
trix. Then A is invertible iff the number 0 is not an eigenvalue of A.
Theorem 8.1.3. If v1 , · · · , vr are eigenvectors that corresponding to dis-
tinct eigenvalues λ1 , · · · , λr of an n × n matrix A, then the set {v1 , · · · , vr }
is linearly independent.
Definition. The scalar equation det(A−λI) = 0 is called the characteristic
equation of A. If A is an n × n matrix, then det(A − λI) is a polynomial of
degree n called the characteristic polynomial of A.
Theorem 8.1.4. A scalar λ is an eigenvalue of an n × n matrix A iff λ
satisfies the characteristic equation.
Definition. If A and B are n × n matrices, then A and B are similar if
there is an invertible matrix P such that P−1 AP = B. Changing A into
P−1 AP is called a similarity transformation.
Theorem 8.1.5. If n × n matrices A and B are similar, then they have
the same characteristic polynomial and hence the same eigenvalues(with the
same multiplicities).

31
32 8. EIGENVALUES AND EIGENVECTORS

8.2 Diagonalization
Definition. A square matrix A is said to be diagonalizable if A is similar
to a diagonal matrix.
Theorem 8.2.1. An n × n matrix A is diagonalizable iff A has n linearly
independent eigenvectors. In other words, A is diagonalizable iff there are
enough eigenvectors to form a basis of Rn , and such basis is called an eigen-
vector basis.
Theorem 8.2.2. An n × n matrix with n distinct eigenvalues is diagonal-
izable.
Theorem 8.2.3. Let A be an n × n matrix whose distinct eigenvalues are
λ1 , · · · , λp .
1. For 1 ⩽ k ⩽ p, the dimension of the eigenspaces for λk is less than or
equal to the multiplicity of the eigenvalue λk .
2. The matrix A is diagonalizable iff the sum of the dimensions of the
distinct eigenspaces equals n, and this happens iff the dimension of the
eigenspace for each λk equals the multiplicity of λk .
3. If A is diagonalizable and Bk is a basis for the eigenspace correspond-
ing to λk for each k, then the total collection of vectors in the sets
B1 , · · · , Bp forms an eigenvector basis for Rn .
Definition. The matrix
h i
M = [T (b1 )]C [T (b2 )]C · · · [T (bn )]C
where B = {b1 , · · · , bn } is a basis for the vector space V , C is a basis in
W , and T is a linear transformation from V to W , is called the matrix for
T relative to the bases B and C. If W is the same as V and the basis C is
the same as B, the matrix M is called the matrix for T relative to B or the
B-matrix for T , and denoted [T ]B .
Theorem 8.2.4 (Diagonal Matrix Representation). Suppose A = PDP−1 ,
where D is a diagonal n × n matrix. If B is the basis for Rn formed from
the columns of P, then D is the B-matrix of the transformation x 7→ Ax.

8.3 Complex Eigenvalues

The theory of eigenvalues and eigenvectors developed for Rn applies equally
well on Cn .
Theorem 8.3.1. Let A be a real 2 × 2 matrix with a complex eigenvalue
λ = a − bi(b ̸= 0) and associated eigenvector v ∈ C2 . Then
" #
−1
h i a −b
A = PCP , where P = Re v Im v and C=
b a
9

Orthogonality

Definition. If a vector z is orthogonal to every vector in a subspace W of

Rn , then z is said to be orthogonal to W . The set of all vectors z that are
orthogonal to W is called the orthogonal complement of W and denoted by
W ⊥.

Theorem 9.0.1. 1. A vector x is in W ⊥ iff x is orthogonal to every

vector in a set that spans W .

2. W ⊥ is a subspace of Rn .

Proof. Left for Exercise

Theorem 9.0.2. Let A be an m × n matrix. Then the orthogonal com-

plement of the row space of A is the nullspace of A, and the orthogonal
complement of the column space of A is the nullspace of AT :

(Row A)⊥ = Nul A, (Col A)⊥ = Nul AT

9.1 Orthogonal Sets

Definition. A set of vectors in Rn is said to be an orthogonal set if each
pair of distinct vectors from the set is orthogonal.

Theorem 9.1.1. If S = {u1 , · · · , up } is an orthogonal set of nonzero vectors

in Rn , then S is linearly independent and hence is a basis for the subspace
spanned by S.

Definition. An orthogonal basis for a subspace W of Rn is a basis for W

that is also an orthogonal set.

Theorem 9.1.2. Each vector in a subspace of Rn has a unique representa-

tion as a linear combination of its orthogonal basis.

33
34 9. ORTHOGONALITY

Definition. The orthogonal projection of v on an arbitrary non-zero vector

b can be written as:
v·b
projb v = b (9.1)
∥b∥2
Moreover, we can see that v−projb v is the vector component of v orthogonal
to b. The projection projb v is determined by the subspace Span{b}, and
we may call it the orthogonal projection onto Span{b}.

Definition. A set is an orthonormal set if it is an orthogonal set of unit

vectors. It is also an orthonormal basis for a subspace spanned by it.

Theorem 9.1.3. An m×n matrix U has orthonormal columns iff UT U = I.

Theorem 9.1.4. Let U be an m × n matrix with orthonormal columns, and

let x, y ∈ Rn . Then

1. ∥Ux∥ = ∥x∥.

2. ∥Ux∥ · ∥Uy∥ = x · y, and it equals zero iff x · y = 0.

Equivalently, they say that the linear mapping x 7→ Ux preserves lengths

and orthogonality.

Definition. An orthogonal matrix is a square invertible matrix U such that

U−1 = UT .

9.2 Orthogonal Projections

Theorem 9.2.1. Let W be a subspace of Rn . Then each y ∈ Rn can be
written uniquely in the form

y = ŷ + z

where ŷ is in W and z is in W ⊥ . In fact, if {u1 , · · · , up } is any orthogonal

basis of W , then
ŷ = proju1 y + · · · + projup y
and z = y − ŷ.

The vector ŷ is called the orthogonal projection of y onto W and written

as projW y.

Theorem 9.2.2. Let W be a subspace of Rn , y ∈ Rn , and ŷ the orthogonal

projection of y onto W . Then ŷ is the closest point in W to y, in the sense
that
∥y − ŷ∥ < ∥y − v∥
for all v ∈ W distinct from ŷ.
9.2. ORTHOGONAL PROJECTIONS 35

Theorem 9.2.3. If {u1 , · · · , up } is an orthonormal basis for a subspace

W ∈ Rn , then

projW y = (y · u1 )u1 + · · · + (y · up )up

h i
If U = u1 · · · up , then

projW y = UUT y
36 9. ORTHOGONALITY
10

Symmetric Matrices and

Quadratic Forms

Definition. A symmetric matrix is a matrix such that it equals to the

transpose of itself.
Definition. A quadratic form on Rn is a function Q defined on Rn whose
value at a vector x ∈ Rn can be computed by an expression of the form
Q(x) = xAT x, where A is an n × n symmetric matrix. The matrix A is
called the matrix of the quadratic form. The simplest example of a nonzero
quadratic form is where the matrix of the quadratic form is the n×n identity
matrix.
Definition. A quadratic form Q is:
1. positive definite if Q(x) > 0 for all x ̸= 0,

2. negative definite if Q(x) < 0 for all x ̸= 0,

3. indefinite if Q(x) assumes both positive and negative values.

Theorem 10.0.1. Let A be an n × n symmetric matrix. Then a quadratic
form xAT x is:
1. positive definite iff the eigenvalues of A are all positive,

2. negative definite iff the eigenvalues of A are all negative,

3. indefinite iff the eigenvalues of A has both positive and negative eigen-
values.
Definition. A positive definite matrix A is a symmetric matrix for which
the quadratic form is positive definite. The matrix is a positive semidefi-
nite matrix if its quadratic form is nonnegative. Other terms are defined
analogously.

37