Linear Algebra

Math 308
S. Paul Smith
Department of Mathematics, Box 354350, University of Washington, Seattle, WA 98195
E-mail address: smith@math.washington.edu

Contents
Chapter 0. Introduction
1. What its all about
2. Some practical applications
3. The importance of asking questions

1
1
2
2

Chapter 1. Linear Geometry

1. Linearity
2. Lines in R2
3. Points, lines, and planes in R3
4. Higher dimensions
5. Parametric descriptions of lines, planes, etc.

3
3
10
13
15
15

Chapter 2. Matrices
1. What is a matrix?
2. A warning about our use of the word vector
3. Row and column vectors = points with coordinates
4. Matrix arithmetic: addition and subtraction
5. The zero matrix and the negative of a matrix
6. Matrix arithmetic: multiplication
7. Pitfalls and warnings
8. Transpose
9. Some special matrices
10. Solving an equation involving an upper triangular matrix
11. Some special products

17
17
18
18
20
20
21
25
25
26
27
28

Chapter 3. Matrices and motions in R2 and R3

1. Linear transformations
2. Rotations in the plane
3. Projections in the plane
4. Contraction and dilation
5. Reflections in the plane
6. Reflections in R3
7. Projections from R3 to a plane

31
31
31
31
31
31
31
31

Chapter 4. Systems of Linear Equations

1. Systems of equations
2. A single linear equation

33
33
36

iii

iv

CONTENTS

3.
4.
5.
6.
7.
8.

Systems of linear equations

A system of linear equations is a single matrix equation
Specific examples
The number of solutions
A geometric view on the number of solutions
Homogeneous systems

37
38
38
41
41
42

Chapter 5. Row operations and row equivalence

1. Equivalent systems of equations
2. Echelon Form
3. An example
4. You already did this in high school
5. The rank of a matrix
6. Inconsistent systems
7. Consistent systems
8. Parametric equations for lines and planes
9. The importance of rank
10. The word solution
11. Elementary matrices

45
45
45
46
47
48
49
49
51
52
53
54

Chapter 6. The Vector space Rn

1. Arithmetic in Rn
2. The standard basis for Rn
3. Linear combinations and linear span
4. Some infinite dimensional vector spaces

57
57
58
58
60

Chapter 7. Subspaces
1. The definition and examples
2. The row and column spaces of a matrix
3. Lines, planes, and translations of subspaces
4. Linguistic difficulties: algebra vs. geometry

61
61
64
65
67

Chapter 8. Linear dependence and independence

1. The definition
2. Criteria for linear (in)dependence
3. Linear (in)dependence and systems of equations

69
69
69
73

Chapter 9. Non-singular matrices, invertible matrices, and inverses

1. Singular and non-singular matrices
2. Inverses
3. Elementary matrices are invertible
4. Making use of inverses
5. The inverse of a 2 2 matrix
6. If A is non-singular how do we find A1 ?

75
75
76
78
78
79
81

Chapter 10. Bases, coordinates, and dimension

1. Definitions

83
83

CONTENTS

2. A subspace of dimension d is just like Rd

3. All bases for W have the same number of elements
4. Every subspace has a basis
5. Properties of bases and spanning sets
6. How to find a basis for a subspace
7. How to find a basis for the range of a matrix
8. Rank + Nullity
9. How to compute the null space and range of a matrix
Chapter 11. Linear transformations
1. A reminder on functions
2. First observations
3. Linear transformations and matrices
4. How to find the matrix representing a linear transformation
5. Invertible matrices and invertible linear transformations
6. How to find the formula for a linear transformation
7. Rotations in the plane
8. Reflections in R2
9. Invariant subspaces
10. The one-to-one and onto properties
11. Two-to-two
12. Gazing into the distance: differential operators as linear
transformations

83
84
84
85
86
86
86
90
91
91
92
94
95
96
96
96
97
98
98
98
99

Chapter 12. Determinants

1. The definition
2. Elementary row operations and determinants
3. The determinant and invertibility
4. Properties
5. Elementary column operations and determinants

103
103
106
108
108
109

Chapter 13. Eigenvalues

1. Definitions and first steps
2. Reflections in R2 , revisited
3. The 2 2 case
4. The equation A2 + I
5. The characteristic polynomial
6. How to find eigenvalues and eigenvectors
7. The utility of eigenvectors

113
113
114
115
116
117
118
122

Chapter 14. Complex vector spaces and complex eigenvalues

1. The complex numbers
2. The complex numbers
3. Linear algebra over C
4. The complex norm
5. An extended exercise

125
125
125
129
129
131

vi

CONTENTS

Chapter 15. Orthogonality

1. Geometry
2. The dot product
3. Orthogonal vectors
4. Classical orthogonal polynomials
5. Orthogonal and orthonormal bases
6. The Gram-Schmidt process
7. Orthogonal matrices
8. Orthogonal projections
9. Least-squares solutions to inconsistent systems
10. Approximating data by polynomial curves
11. Gazing into the distance: Fourier series
12. Infinite sequences

133
133
133
134
135
135
136
138
140
143
146
147
147

Chapter 16. Similar matrices

1. Definition
2. Definition and first properties
3. Example
4.
5. Diagonalizable matrices

149
149
149
150
150
152

Chapter 17. Words and terminology

1. The need for clarity and precision
2. Some, all, each, every, and only
3. Singular and plural
4. The word it
5. A and The
6. If and only if

157
157
157
159
159
160
160

Chapter 18. Applications of linear algebra and vector spaces

1. Simple electrical circuits
2. Magic squares

161
161
162

Chapter 19. Last rites

1. A summary of notation
2. A sermon

165
165
166

CHAPTER 0

Introduction
1. What its all about
The practical problem, solving systems of linear equations, that motivates the subject of Linear Algebra, is introduced in chapter 4. Although
the problem is concrete and easily understood, the methods and theoretical
framework required for a deeper understanding of it are abstract. Linear algebra can be approached in a completely abstract fashion, untethered from
the problems that give rise to the subject.
Such an approach is daunting for the average student so we will strike a
balance between the practical, specific, abstract, and general.
Linear algebra plays a central role in almost all parts of modern technology. Systems of linear equations involving hundreds, thousands, even
billions of unknowns are solved every second of every day in all corners of
the globe. One of the more fantastic uses is the way in which Google prioritizes pages on the web. All web pages are assigned a page rank that measures
its importance. The page ranks are the unknowns in an enormous system
of linear equations. To find the page rank one must solve the system of
linear equations. To handle such large systems of linear equations one uses
sophisticated techniques that are developed first as abstract results about
linear algebra.
Systems of linear equations are rephrased in terms of matrix equations,
i.e., equations involving matrices. The translation is straightforward but
after mastering the basics of matrix arithmetic one must interpret those
basics in geometric terms. That leads to linear geometry and the language
of vectors and vector spaces.
Chapter 1 provides a brief account of linear geometry. As the name
suggests, linear geometry concerns lines. The material about lines in the
plane is covered in high school. Unless you know that material backwards
and forwards linear algebra will be impossible for you. Linear geometry
also involves higher dimensional analogues of lines, for examples, lines and
planes in 3-space, or R3 as we will denote it. I am assuming you met that
material in a multivariable calculus course. Unless you know that material
backwards and forwards linear algebra will be impossible for you.1
1I expect you to know the material about linear geometry in R2 and R3 . By that I
dont mean that you have simply passed a course where that material is covered. I expect
you to have understood and mastered that material and that you retain that mastery today. If
1

0. INTRODUCTION

After reviewing linear geometry we will review basic facts about matrices
in chapter 2. That will be familiar to some students. Very little from chapter
2 is required to understand the initial material on systems of linear equations
in chapter 4.
2. Some practical applications
(1) solving systems of linear equations
(2) birth-death processes, Leslie matrices (see Ch. 6 in Difference equations. From rabbits to chaos), Lotka-Volterra
(3) Google page rank
(4) X-ray tomography, MRI, density of tissue, a slice through the
body is pixellated and each pixel has a different density to be
inferred/computed from the input/output of sending various rays
through the body. Generally many more measurements than unknowns so almost always obtain an inconsistent system but want
the least-squares solution.
3. The importance of asking questions
You will quickly find that this is a difficult course. Everyone finds it
difficult. The best advice I can give you is to ask questions. When you read
something you dont understand dont just skim over it thinking you will
return to it later and then fail to return to it. Ask, ask, ask. The main
reason people fail this course is because they are afraid to ask questions.
They think asking a question will make them look stupid. Smart people
arent afraid to ask questions. No one has all the answers.
The second most common reason people fail this course is that they wait
till the week before the exam to start thinking seriously about the material.
If you are smart you will try to understand every topic in this course at the
time it first appears.

you are not in command of the material in chapter 1 master it as soon as possible. That
is essential if you want to pass this course.

CHAPTER 1

Linear Geometry
1. Linearity
The word linear comes from the word line.
The geometric aspect of linear algebra involves lines, planes, and their
higher dimensional analogues: e.g., lines in the plane, lines in 3-space, lines
in 4-space, planes in 3-space, planes in 4-space, 3-planes in 4-space, 5-planes
in 8-space, and so on, ad infinitum. Such things form the subject matter of
linear geometry.
Curvy things play no role in linear algebra or linear geometry. We ignore
circles, spheres, ellipses, parabolas, etc. All is linear.
1.1. What is a line? You already know what a line is. More accurately, you know something about lines in the plane, R2 , or in 3-space, R3 .
In this course, you need to know something about lines in n-space, Rn .
1.2. What is Rn ? Rn is our notation for the set of all n-tuples of
real numbers. We run out of words after pair, triple, quadruple, quintuple,
sextuple, ... so invent the word n-tuple to refer to an ordered sequence of n
numbers where n can be any positive integer.
For example, (8,7,6,5,4,3,2,1) is an 8-tuple. It is not the same as the 8tuple (1,2,3,4,5,6,7,8). We think of these 8-tuples as labels for two different
points in R8 . We call the individual entries in (8,7,6,5,4,3,2,1) the coordinates
of the point (8,7,6,5,4,3,2,1); thus 8 is the first coordinate, 7 the second
coordinate, etc.
Often we need to speak about a point in Rn when we dont know its
coordinates. In that case we will say something like this: let (a1 , a2 , . . . , an )
be a point in Rn . Here a1 , a2 , . . . , an are some arbitrary real numbers.
1.3. The origin. The origin is a special point in Rn : it is the point
having all its coordinates equal to 0. For example, (0,0) is the origin in R2 ;
(0,0,0) is the origin in R3 ; (0,0,0,0) is the origin in R4 ; and so on. We often
write 0 for the origin.
There is a special notation for the set of all points in Rn except the origin,
namely Rn {0}. We use the minus symbol because Rn {0} is obtained by
taking 0 away from Rn . Here taking away is synonymous with removing.
This notation permits a useful brevity of expression: suppose p Rn {0}
means the same thing as suppose p is a point in Rn that is not 0.
3

1. LINEAR GEOMETRY

1.4. Adding points in Rn . We can add points in Rn . For example, in

(1, 2, 3, 4) + (6, 4, 2, 0) = (7, 6, 5, 4). The origin is the unique point with
the property that 0 + p = p for all points p in Rn . For that reason we also
call 0 zero.
We can also subtract: for example, (6, 4, 2, 0)(1, 2, 3, 4) = (5, 2, 1, 4).
We can not add a point in R4 to a point in R5 . More generally, if m 6= n,
we cant add a point in Rm to a point in Rn .
R4 ,

1.5. Multiplying a point in Rn by a number. First, some examples. Consider the point p = (1, 1, 2, 3) in R4 . Then 2p = (2, 2, 4, 6),
5p = (5, 5, 10, 15), 3p = (3, 3, 6, 9); more generally, if t is any
real number tp = (t, t, 2t, 3t). In full generality, if is a real number and
p = (a1 , . . . , an ) is a point in Rn , p denotes the point (a1 , . . . , an ); thus,
the coordinates of p are obtained by multiplying each coordinate of p by
. We call p a multiple of p.
1.6. Lines. We will now define what we mean by the word line. If p is
a point in Rn that is not the origin, the line through the origin in the direction
p is the set of all multiples of p. Formally, if we write Rp to denote that
line,1
Rp = {p | R}.
n
If q is another point in R , the line through q in the direction p is
{q + p | R};
we often denote this line by q + Rp.
When we speak of a line in Rn we mean a subset of Rn of the form
q + Rp. Thus, if I say L is a line in Rn I mean there is a point p Rn {0}
and a point q Rn such that L = q + Rp. The p and q are not uniquely
determined by L.
Proposition 1.1. The lines L = q + Rp and L0 = q 0 + Rp0 are the same
if and only if p and p0 are multiples of each other and q q 0 lies on the line
Rp; i.e., if and only if Rp = Rp0 and q q 0 Rp.
Proof.2 () Suppose the lines are the same, i.e., L = L0 . Since q 0 is on the
line L0 it is equal to q + p for some R. Since q is on L, q = q 0 + p0 for
1I like the notation Rp for the set of all multiples of p because it is similar to the
notation 2p: when x and y are numbers we denote their product by xy, the juxtaposition
of x and y. So Rp consists of all products p where ranges over the set of all real
numbers.
2To show two sets X and X 0 are the same one must show that every element of X is
in X 0 , which is written as X X 0 , and that every element of X 0 is in X.
To prove an if and only if statement one must prove each statement implies the
other; i.e., the statement A if and only if B is true if the truth of A implies the truth of
B and the truth of B implies the truth of A. Most of the time, when I prove a result of
the form A if and only if B I will break the proof into two parts: I will begin the proof
that A implies B by writing the symbol () and will begin the proof that B implies A
by writing the symbol ()

1. LINEARITY

some R. Hence q q 0 = p = p0 . In particular, q q 0 Rp. There

are now two cases.
Suppose q 6= q 0 . Then and are non-zero so p and p0 are multiples of
each other whence Rp0 = Rp.
Suppose q = q 0 . Since q + p q + Rp = q 0 + Rp0 = q + Rp0 , q + p = q + p0
for some R. Thus p = p0 . Likewise, q + p0 q + Rp0 = q + Rp so
q + p0 = q + p for some R. Thus p0 = p. We have shown that p and
p0 are multiples of each other so Rp = Rp0 .
() Suppose q q 0 Rp and Rp = Rp0 . Then q q 0 = p for some
R and p = p0 for some non-zero R.
If R, then
q + p = q 0 + p + p = q 0 + ( + )p0
which is in q 0 + Rp0 ; thus q + Rp q 0 + Rp0 .
Similarly, if R, then
q 0 + p0 = q p + 1 p = = q + ( 1 )p
which is in q + Rp; thus q 0 + Rp0 q + Rp.
This completes the proof that q 0 + Rp0 = q + Rp if q q 0 Rp and
Rp = Rp0 .

1.7. Fear and loathing. I know most of you are pretty worried about
this mysterious thing called Rn . What does it look like? What properties
does it have? How can I work with it if I cant picture it? What the heck
does he mean when he talks about points and lines in Rn ?
Although we cant picture Rn we can ask questions about it.
This is an important point. Prior to 1800 or so, mathematicians and
scientists only asked questions about things they could see or touch. For
example, R4 wasnt really considered in a serious way until general relativity
tied together space and time. Going further back, there was a time when
negative numbers were considered absurd. Indeed, there still exist primitive
cultures that have no concept of negative numbers. If you tried to introduce
them to negative numbers they would think you were some kind of nut.
Imaginary numbers were called that because they werent really numbers.
Even when they held that status there were a few brave souls who calculated
with them by following the rules of arithmetic that applied to real numbers.
Because we have a definition of Rn we can grapple with it, explore it,
and ask and answer questions about it. That is what we have been doing in
the last few sections. In olden days, mathematicians rarely defined things.
Because the things they studied were real it was obvious what the words
meant. The great counterexample to this statement is Euclids books on
geometry. Euclid took care to define everything carefully and built geometry
on that rock. As mathematics became increasingly sophisticated the need
for precise definitions became more apparent. We now live in a sophisticated
mathematical world where everything is defined.

1. LINEAR GEOMETRY

So, take a deep breath, gather your courage, and plunge into the bracing
waters. I will be there to help you if you start sinking. But I cant be
everywhere at once, and I wont always recognize whether you are waving
or drowning. Shout help if you are sinking. Im not telepathic.
1.8. Basic properties of Rn . Here are some things we will prove:
(1) If p and q are different points in Rn there is one, and only one, line
through p and q. We denote it by pq.
(2) If L is a line through the origin in Rn , and q and q 0 are points in
Rn , then either q + L = q 0 + L or (q + L) (q 0 + L) = .
(3) If L is a line through the origin in Rn , and q and q 0 are points in
Rn , then q + L = q 0 + L if and only if q q 0 L.
(4) If p and q are different points in Rn , then pq is the line through p
in the direction q p.
(5) If L and L0 are lines in Rn such that L L0 , then L = L0 . This is
a consequence of (2).
If L is a line through the origin in Rn , and q and q 0 are points in Rn , we say
that the lines q + L and q 0 + L are parallel.
Dont just accept these as facts to be memorized. Learning is more than
knowinglearning involves understanding. To understand why the above
facts are true you will need to look at how they are proved.
Proposition 1.2. If p and q are different points in Rn there is one, and
only one, line through p and q, namely the line through p in the direction
q p.
Proof. The line through p in the direction q p is p + R(q p). The line
p + R(q p) contains p because p = p + 0 (q p). It also contains q because
q = p + 1 (q p). We have shown that the line through p in the direction
q p passes through p and q.
It remains to show that this is the only line that passes through both p
and q. To that end, let L be any line in Rn that passes though p and q. We
will use Proposition 1.1 to show that L is equal to p + R(q p).
By our definition of the word line, L = q 0 + Rp0 for some points q 0 R
and p0 R {0}. By Proposition 1.1, q 0 + Rp0 is equal to p + R(q p) if
Rp0 = R(q p) and q 0 p Rp0 .
We will now show that Rp0 = R(q p) and q 0 p Rp0 .
Since p L and q L, there are numbers and such that p = q 0 + p0 ,
which implies that q 0 p Rp0 , and q = q 0 + p0 . Thus p q = q 0 + p
q 0 q 0 = ( )q 0 . Because p 6= q, p q 6= 0; i.e., ( )p0 6= 0. Hence
6= 0. Therefore p0 = ( )1 (p q). It follows that every multiple of
p0 is a multiple of p q and every multiple of p q is a multiple of p0 . Thus
Rp0 = R(q p). We already observed that q 0 p Rp0 so Proposition 1.1
tells us that q 0 + Rp0 = p + R(q p), i.e., L = p + R(q p).


1. LINEARITY

1.8.1. Notation. We will write pq for the unique line in Rn that passes
through the points p and q (when p 6= q). Proposition 1.2 tells us that
pq = p + R(q p).
Proposition 1.1 tells us that pq has infinitely many other similar descriptions.
For example, pq = q + R(p q) and pq = q + R(q p).
1.8.2. Although we cant picture R8 we can ask questions about it. For
example, does the point (1,1,1,1,1,1,1,1) lie on the line through the points
(8,7,6,5,4,3,2,1) and (1,2,3,4,5,6,7,8)? Why? If you can answer this you
understand what is going on. If you cant you dont, and should ask a
question.
1.9. Parametric description of lines. You already know that lines
in R3 can be described by a pair of equations or parametrically. For example,
the line given by the equations
(
x+y+z
= 4
(1-1)
x + 3y + 2z = 9
is the set of points of the form (t, 1 + t, 3 2t) as t ranges over all real
numbers; in set-theoretic notation, the line is
(1-2)

{(t, 1 + t, 3 2t) | t R};

we call t a parameter; we might say that t parametrizes the line or that the
line is parametrized by t.
The next result shows that every line in Rn can be described parametrically.
Proposition 1.3. If p and q are distinct points in Rn , then
(1-3)

pq = {tp + (1 t)q | t R}.

Proof. We must show that the sets pq and {tp + (1 t)q | t R} are the
same. We do this by showing each set contains the other one.
Let p0 be a point on pq. Since pq = p + R(q p), p0 = p + (q p) for
some R. Thus, if t = 1 , then
p0 = (1 )p + q = tp + (1 t)q.
Therefore pq {tp + (1 t)q | t R}. If t is any number, then
tp + (1 t)q = q + t(p q) q + R(p q) = pq
so {tp + (1 t)q | t R} pq. Thus, pq = {tp + (1 t)q | t R}.

If you think of t as denoting time, then the parametrization in (1-3) can

be thought of as giving the position of a moving point at time t; for example,
at t = 0 the moving point is at q and at time t = 1, the moving point is at
p; at time t = 21 the moving point is exactly half-way between p and q. This
perspective should help you answer the question in 1.8.2.

1. LINEAR GEOMETRY

1.9.1. Dont forget this remark. One of the key ideas in understanding
systems of linear equations is moving back and forth between the parametric
description of linear subsets of Rn and the description of those linear subsets
as solution sets to systems of linear equations. For example, (1-2) is a
parametric description of the set of solutions to the system (1-1) of two
linear equations in the unknowns x, y, and z. In other words, every solution
to (1-1) is obtained by choosing a number t and then taking x = t, y = 1 + t,
and z = 3 2t.
1.9.2. Each line L in Rn has infinitely many parametric descriptions.
1.9.3. Parametric descriptions of higher dimensional linear subsets of
Rn .
1.10. Linear subsets of Rn . Let L be a subset of Rn . We say that L
is linear if
(1) it is a point or
(2) whenever p and q are different points on L every point on the line
pq lies on L.
1.11. What is a plane? Actually, it would be better if you thought
about this question. How would you define a plane in Rn ? Look at the
definition of a line for inspiration. We defined a line parametrically, not by
a collection of equations. Did we really need to define a line in two steps,
i.e., first defining a line through the origin, then defining a general line?
1.12. Hyperplanes. The dot product of two points u = (u1 , . . . , un )
and v = (v1 , . . . , vn ) in Rn is the number
u v = u1 v1 + + un vn .
Notice that u v = v u. If u and v are non-zero and u v = 0 we say u and
v are orthogonal. If n is 2 or 3, the condition that u v = 0 is equivalent to
the condition that the line 0u is perpendicular to the line 0v.
Let u be a non-zero point in Rn and c any number. The set
H := {v Rn | u v = c}
is called a hyperplane in Rn .
Proposition 1.4. If p and q are different points on a hyperplane H,
then all the points on the line pq lie on H.
Proof. Since H is a hyperplane there is a point u Rn {0} and a number
c such that H = {v Rn | u v = c}.


1. LINEARITY

1.13. Solutions to systems of equations: an example. A system

of equations is just a collection of equations. For example, taken together,
(1-4)

x2 + y 2 = z 2

and

x+y+z =1

form a system of two equations. We call x, y, and z, unknowns. A solution

to the system (1-4) consists of three numbers a, b, c such that a2 + b2 = c2
and a+b+c = 1. Such a solution
corresponds to the point (a, b, c) in 3-space,


R3 . For example, 0, 21 , 12 is a solution to this system of equations because
 2  2
1
1 1
1
=
and
0 + + = 1.
(0)2 +
2
2
2 2


5
is a solution to the system of equations because
Similarly, 14 , 13 , 12
 2  2  2
1
5
1 1
5
1
+
=
and
+ +
= 1.
4
3
12
4 3 12
Thus, we think of solutions to the system (1-4) as points in R3 . These
points form a geometric object. We use our visual sense to perceive them,
and our visual sense organize those individual points into a single organized
geometric object.
If you are really interested in math you might try finding all solutions
to the system (1-4). If not all, several. How did I find the second solution?
Can you find another solution?
1.14. Solutions to systems of equations. A solution to a system of
equations in n unknowns x1 , . . . , xn is a set of numbers s1 , . . . , sn such that
every equation is a true equality when each xi is replaced by the corresponding number si . This is the famous plugging in procedure. We will think of
s1 , . . . , sn as the coordinates of a point in n-space, Rn . Thus, each solution
is a point p = (s1 , . . . , sn ) in Rn .
1.15. Systems of linear equations. The distinguishing feature of a
system of linear equations is that
if p and q are different solutions to a system of linear equations, then every point on the line through p and q is a
solution to the system.
It is this linearity feature that gives rise to the terminology linear algebra.
It is convenient to write pq for the line through p and q.
Suppose p, q, and r, are solutions to a system of linear equations and r
is not on the line through p and q. Then every point on the three lines pq,
pr, and qr, is a solution to the system. Now take any point p0 on pq and any
point q 0 on pr; since p0 and q 0 are solutions to the linear system so is every
point on the line p0 q 0 . Repeating this operation we can draw more and more
lines. You should be able to convince yourself that all the lines we obtain in
this way eventually fill up a plane, the plane in Rn that contains the points
p, q, and r. We denote that plane by pqr.

10

1. LINEAR GEOMETRY

1.16. Grumbling and preaching. It has become popular over the

last two or three decades to speak of linear thinking and non-linear
thinking. This is a fine example of the way in which scientific terms are
hijacked by people who have no idea what they mean. To those people, linear thinking is not as good as non-linear thinking. Linear thinking
is considered rigid, not flexible, something creative/imaginative people
dont engage in. This is utter nonsense promulgated by ignorant, pompous,
people. (Yes, Im pompous too but perhaps not so ignorant.)
The recognition of linearity in mathematics and the subsequent development of linear algebra has produced methods of tremendous power and
extraordinarily wide applicability. Indeed, the modern world could not function as it does if it did not make use of linear algebra and linear geometry
millions of times every second. I mean that literally. No exaggeration. Linear algebra is used millions of times every second, in every corner of the
globe, to ensure that (some of) our society functions smoothly.
The methods that have been developed to address problems in linear
algebra are so effective that it is often beneficial to approximate a nonlinear problem (and here I use the word non-linear in a precise scientific
sense) by a linear problem.
2. Lines in R2
2.1. An ordered pair of real numbers consists of two real numbers where
their order matters. For example, (0, 0), (0, 1), (1, 1), (2, 7), and (3, 5),
are ordered pairs of real numbers, and they differ from the ordered pairs
(1, 0), (7, 2), and (5, 3).
R2 denotes the set of ordered pairs of real numbers. For example, (0, 0),
(0, 1), (1, 1), (2, 7), and (3, 5), are points in R2 . We use the word ordered
because (1, 2) and (2, 1) are different points of R2 . A formal definition is
R2 := {(a, b) | a, b R}.
At high school you learned about x- and y-axes and labelled points of
in the plane by ordered pairs of real numbers. The point labelled (3, 7) is
the point obtained by starting at the origin, going 3 units of distance in the
x-direction, then 7 units of distance in the y-direction. Thus, we have a
geometric view of the algebraically defined object R2 .
Linear algebra involves a constant interplay of algebra and geometry.
To master linear algebra one must keep in mind both the algebraic and
geometric features. A simple example of this interplay is illustrated by the
fact that the unique solution to the pair of equations
2x + 3y =4
2x 3y = 8
is the point where the lines 2x + 3y = 4 and 2x 3y = 8 intersect. This
pair of equations is a system of two linear equations.

2. LINES IN R2

11

2.2. Notation. If p and q are two different points in the plane R2 or

in 3-space, R3 , we write pq for the line through them.
2.3. Set notation. We will use standard set notation throughout this
course. You will find it helpful to read something about set notation. Use
google.
2.4. Prerequisites for every linear algebra course. You must be
able to do the following things:
Find the slope of the line through two given points in R2 .
Find the equation of the line through two points in R2 .
Give a parametric form for a line that is given in the form ax+by =
c. There are infinitely many different parametric forms for a given
line. For example, the line x = y is the line consisting of all points
(t, t) as t ranges over R, the set of real numbers. The same line is
given by the parametrization (1 + 2t, 1 + 2t), t R. And so on.
Give a parametric form for the line through two given points. If
p = (a, b) and q = (c, d) are two different points in the plane the set
of points t(a, b) + (1 t)(c, d), or, equivalently, tp + (1 t)q, gives all
points on pq as t ranges over all real numbers. For example, t = 0
gives the point q, t = 1 gives the point p, t = 12 gives the point
halfway between p and q. One gets all points between p and q by
letting t range over the closed interval [0, 1]. More formally, the set
of points between p and q is {tp + (1 t)q | t [0, 1]}. Similarly,
pq = {tp + (1 t)q | t R}.
Notice that t(a, b) + (1 t)(c, d) = (ta + (1 t)c, tb + (1 t)d).
Decide whether two lines are parallel.
Decide whether two lines are perpendicular.
Given a line L and a point not on L, find the line parallel to L that
passes through the given point.
Given a line L and a point not on L, find the line perpendicular to
L that passes through the given point.
Find the point at which two lines intersect if such a point exists.
Proposition 2.1. Let p = (a, b) and q = (c, d) be two different points
in the plane, R2 . The set of points
{tp + (1 t)q | t R}
is exactly the set of points on the line through p and q.
Proof. The line given by the equation
(2-1)

(c a)(y b) = (d b)(x a)

passes through p because when we plug in a for x and b for y (2-1) is an

equality, namely (c a)(b b) = (d b)(a a). The line given by equation
(2-1) also passes through q because when we plug in c for x and d for y we

12

1. LINEAR GEOMETRY

get a true equality, namely (c a)(d b) = (d b)(c a). Thus, the line
given by equation (2-1) is the line through p and q which we denote by pq.
Lets write L for the set of points {tp + (1 t)q | t R}. Less elegantly,
but more explicitly,
n
o


L = ta + (1 t)c, tb + (1 t)d  t R .
The proposition claims that L = pq. To prove this we must show that every
point in L belongs to pq and every point on pq belongs to L.
We first show that L pq, i.e., every point in L
belongs to pq. A point
in L has coordinates ta + (1 t)c, tb + (1 t)d . When we plug these
coordinates into (2-1) we obtain
(c a)(tb + (1 t)d b) = (d b)(ta + (1 t)c a),
i.e., (c a)(1 t)(d b) = (d b)(1 t)(c a).
Since this really is an equality
we have shown that ta + (1 t)c, tb + (1 t)d lies on pq. Therefore L pq.
To prove the opposite inclusion, take a point (x, y) that lies on pq; we
will show that (x, y) is in L. Do do that we must show there is a number
t such that (x, y) = (ta + (1 t)c, tb + (1 t)d). Because (x, y) lies on pq,
(c a)(y b) = (d b)(x a). This implies that
yd
xc
=
.
ac
bd
Lets call this number t. If a = c we define t to be the number (y d)/(bd);
if b = d, we define t to be the number (x c)/(a c); we cant have a = c
and b = d because then (a, b) = (c, d) which violates the hypothesis that
(a, b) and (c, d) are different points.
In any case, we now have x c = t(a c) and y d = t(b d). We
can rewrite these as x = ta + (1 t)c and y = tb + (1 t)d. Thus (x, y) =
(ta + (1 t)c, tb + (1 t)d), i.e., (x, y) belongs to the set L. This completes

the proof that L pq.
2.5. Remarks on the previous proof. You might ask How did you
know that (2-1) is the line through (a, b) and (c, d)?
Draw three points (a, b), (c, d), and (x, y), on a piece of paper. Draw the
line segment from (a, b) to (c, d), and the line segment from (c, d) to (x, y).
Can you see, or convince yourself, that (x, y) lies on the line through (a, b)
and (c, d) if and only if those two line segments have the same slope? I hope
so because that is the key idea behind the equation (2-1). Let me explain.
The slopes of the two lines are
db
yb
and
.
ca
xa
Thus, (x, y) lies on the line through (a, b) and (c, d) if and only if
db
yb
=
.
ca
xa

3. POINTS, LINES, AND PLANES IN R3

13

It is possible that a might equal c in which case the expression on the left
of the previous equation makes no sense. To avoid that I cross multiply and
rewrite the required equality as (d b)(x a) = (c a)(y b).
Of course, there are other reasonable ways to write the equation for the
line pq but I like mine because it seems elegant and fits in nicely with the
geometry of the situation, i.e., the statements I made about slopes.
2.6. About proofs. A proof is a narrative argument intended to convince other people of the truth of a statement.
By narrative I mean that the argument consists of sentences and paragraphs. The sentences should be grammatically correct and easy to understand. Those sentences will contain words and mathematical symbols.
Not only must the words obey the rules of grammar, so too must the symbols. Thus, there are two kinds of grammatical rules: those that govern the
English language and those that govern mathematical phrases.
By saying other people I want to emphasize that you are writing for
someone else, the reader. Make it easy for the reader. It is not enough that
you understand what your sentences say. Others must understand them too.
Indeed, the quality of your proof is measured by the effect it has on other
people. If your proof does not convince other people you have done a poor
job.
If your proof does not convince others it might be that your proof is
incorrect. You might have fooled yourself. If your proof does not convince
others it might be because your argument is too long and convoluted. In
that case, you should re-examine your proof and try to make it simpler.
Are parts of your narrative argument unnecessary? Have you said the same
thing more than once? Have you used words or phrases that add nothing to
your argument? Are your sentences too long? Is it possible to use shorter
sentences. Short sentences are easier to understand. Your argument might
fail to convince others because it is too short. Is some essential part of the
argument missing? Is it clear how each statement follows from the previous
ones? Have you used an undefined symbol, word, or phrase?
When I figure out how to prove something I usually rewrite the proof
many times trying to make it as simple and clear as possible. Once I have
convinced myself that my argument works I start a new job, that of polishing
and refining the argument so it will convince others.
Even if you follow the rules of grammar you might write nonsense. A
famous example is Red dreams sleep furiously. For some mathematical
nonsense, consider the sentence The derivative of a triangle is perpendicular
to its area.
3. Points, lines, and planes in R3
3.1. R3 denotes the set of ordered triples of real numbers. Formally,
R3 := {(a, b, c) | a, b, c R}.

14

1. LINEAR GEOMETRY

The corresponding geometric picture is 3-space. Elements of R3 , i.e., ordered

triples (a, b, c) label points in 3-space. We call a the x-coordinate of the point
p = (a, b, c), b the y-coordinate of p, and c the z-coordinate of p.
3.2. Notation. If p and q are two different points in R3 we write pq
for the line through them. If p, q, and r, are three points in R3 that do not
lie on a line, there is a unique plane that contains them. That plane will be
labelled pqr.
3.3. The first difference from R2 is that a single linear equation, 2x+y
3z = 4 for example, gives a plane in R3 . By gives I mean the following.
Every solution to the equation is an ordered triple of numbers, e.g., (2, 6, 2)
is a solution to the equation 2x+y3z = 4 and it represents (is the label for)
a point in R3 . We say that the point (2, 6, 2) is a solution to the equation.
The collection (formally, the set) of all solutions to 2x + y 3z = 4 forms a
plane in R3 .
Two planes in R3 are said to be parallel if they do not intersect (meet).
3.4. Prerequisites for every linear algebra course. You must be
able to do the following things:
Fiind the equation of the plane pqr when p, q, and r, are three
points in R3 that do not lie on a line. A relatively easy question
of this type is to find the equation for the plane that contains
(0, 0, 0), (2, 3, 7), and (3, 2, 3). That equation will be of the form
ax + by + cz = d for some real numbers a, b, c, d. Since (0, 0, 0) lies
on the plane, d = 0. Because (2, 3, 7) and (3, 2, 3) lie on the plane
ax + by + cz = 0 the numbers a, b, c must have the property that
2a + 3b + 7c = 0
3a + 2b + 3c = 0.

and

This is a system of 2 linear equations in 3 unknowns. There are

many ways you might solve this system but all are essentially the
same. The idea is to multiply each equation by some number and
add or subtract the two equations to get a simpler equation that
a, b, and c, must satisfy.
Find the equation of the line through two given points.
Give a parametric form for a line that is given in the form ax+by =
c. There are infinitely many different parametric forms for a given
line. For example, the line x = y is the line consisting of all points
(t, t) as t ranges over R, the set of real numbers. The same line is
given by the parametrization (1 + 2t, 1 + 2t), t R. And so on.
Give a parametric form for the line through two given points. If
p = (a, b) and q = (c, d) are two different points in the plane the set
of points t(a, b) + (1 t)(c, d), or, equivalently, tp + (1 t)q, gives all
points on pq as t ranges over all real numbers. For example, t = 0
gives the point q, t = 1 gives the point p, t = 12 gives the point

5. PARAMETRIC DESCRIPTIONS OF LINES, PLANES, ETC.

15

halfway between p and q. One gets all points between p and q by

letting t range over the closed interval [0, 1]. More formally, the set
of points between p and q is {tp + (1 t)q | t [0, 1]}. Similarly,
pq = {tp + (1 t)q | t R}.
Notice that t(a, b) + (1 t)(c, d) = (ta + (1 t)c, tb + (1 t)d).
Decide whether two lines are parallel.
Decide whether two lines are perpendicular.
Given a line L and a point not on L, find the line parallel to L that
passes through the given point.
Given a line L and a point not on L, find the line perpendicular to
L that passes through the given point.
Find the point at which two lines intersect if such a point exists.
As t ranges over all real numbers the points (1, 2, 3) + t(1, 1, 1) form a
line in R3 . The line passes through the point (1, 2, 3) (just take t = 0). We
sometimes describe this line by saying it is the line through (1, 2, 3) in the
direction (1, 1, 1).
4. Higher dimensions
5. Parametric descriptions of lines, planes, etc.
The solutions to the system of equations x+y 2z = 0 and 2xy z = 3
is a line in R3 . Lets call it L. Although these two equations give a complete
description of L that description does not make it easy to write down any
points that lie on L. In a sense it is a useless description of L.
In contrast, it is easy to write down points that lie on the line (1, 2, 3) +
t(1, 1, 1), t R; just choose any real number t and compute. For example,
asking t = 1, t = 0, t = 1, and t = 2, gives us the points (0, 1, 2), (1, 2, 3),
(2, 3, 4), and (3, 4, 5). In fact, the line in this paragraph is the line in the
previous paragraph.
Solving a system of linear equations means, roughly, taking a line (usually a higher dimensional linear space) given as in the first paragraph and
figuring out how to describe that line by giving a description of it like that
in the second paragraph.
5.1. Exercise. Show that the line (1, 2, 3)+t(1, 1, 1), t R, is the same
as the line given by the equations x + y 2z = 0 and 2x y z = 3. (Hint:
to show two lines are the same it suffices to show they have two points in
common.)

CHAPTER 2

Matrices
You can skip this chapter if you want, start reading at chapter 4, and
return to this chapter whenever you need to.
Matrices are an essential part of the language of linear algebra and linear
equations. This chapter isolates this part of the language so you can easily
refer back to it when you need to.

1. What is a matrix?
1.1. An m n matrix (we read this as an m-by-n matrix) is a rectangular array of mn numbers arranged into m rows and n columns. For
example,


1 3 0
(1-1)
A :=
4 5 2
is a 2 3 matrix. The prefix m n is called the size of the matrix.
1.2. We use upper case letters to denote matrices. The numbers in the
matrix are called its entries. The entry in row i and column j is called the
ij th entry, and if the matrix is denoted by A we often write Aij for its ij th
entry. The entries in the matrix A in (1-1) are
A11 = 1, A12 = 3, A13 = 0
A21 = 4, A22 = 5, A23 = 2.
In this example, we call 5 the 22-entry; read this as two-two entry. Likewise,
the 21-entry (two-one entry) is 4.
We sometimes write A = (aij ) for the matrix whose ij th entry is aij . For
example, we might write


a11 a12 a13
A=
.
a21 a22 a23
Notice that aij is the entry in row i and column j
1.3. Equality of matrices. Two matrices A and B are equal if and
only if they have the same size and Aij = Bij for all i and j.
17

18

2. MATRICES

1.4. Square matrices. An n n matrix is called a square matrix. For

example,


1 2
(1-2)
A=
1 0
is a 2 2 square matrix.
2. A warning about our use of the word vector
In the past you have probably used the word vector to mean something
with magnitude and direction. That is not the way the word vector
will be used in this course.
Later we will define things called vector spaces. The familiar spaces
2
R and R3 are vector spaces. But there are many other vector spaces.
For example, the set of all polynomials in one variable is a vector space.
The set of all continuous functions f : R R is a vector space. The set
of all continuous functions f : [0, 1] R is a vector space.The set of all
polynomials of degree 56 fis a vector space; there is nothing special about
56. All vector spaces have various common properties so rather than proving
such a property for this vector space, then that one, then another one, we
develop a general theory of vector spaces proving results about all vector
spaces at once.
That is what abstraction is about: by throwing away the inessential
data and keeping the essential data we are able to prove results that apply
to many different things at once.
When talking about vector spaces we need a name for the elements in
them. The word used for an element in a vector space is vector. However,
when we look at a particular vector space the word vector might be confusing.
For example, the set of all polynomials is a vector space but rather than
calling the elements in it vectors we call them polynomials.
One might argue that we should not have hijacked the word vectorI
tend to agree with you, but the naming was done long ago and is now part
of the language of linear algebra. We are stuck with it. Sometimes I will use
the word point to refer to an element in a vector space. For example, it is
common to call elements of the plane R2 points. You probably are used to
calling elements of R3 points.
Bottom line: in this course, when you here the word vector think of a
point or whatever is appropriate to the situation. For example, the row and
column vectors we are about to define are just matrices of a particular kind.
It is not helpful to think of row and column vectors as having magnitude
and direction.
3. Row and column vectors = points with coordinates
3.1. Matrices having just one row or column are of particular importance and are often called vectors.
A matrix having a single row is called a row matrix or a row vector.

3. ROW AND COLUMN VECTORS = POINTS WITH COORDINATES

19

A matrix having a single column is called a column matrix or a column vector.

For example, the 1 4 matrix (1 2 3 4) is a row vector, and the 4 1
matrix

1
2

3
4
is a column vector.
In parts of this course it is better to call row and column vectors points
and think of the entries in them as the coordinates of the points. For example, think of (1,2,3,4) and

1
2

3
4
as points having 4 coordinates. Because it has 4 coordinates such a point
belongs to 4-space which we will denote by R4 .
At high school you met points with 2 coordinates, points in the plane,
R2 . Either there, or at college, you met points in 3-space,R3 , often labelled
(x, y, z).
Points with n coordinates, (a1 , . . . , an ) for example, belong to n-space
which we denote by Rn . We call the number n in the notation Rn the
dimension of the space. Later we will give an abstract and formal definition
of dimension but for now your intuition should suffice.
3.2. Underlining row and column vectors. Matrices are usually
denoted by upper case letters but this convention is usually violated for
row and column vectors. We usually denote a row or column vector by
a lower case letter that is underlined. The underlining is done to avoid
confusion between a lower case letters denoting numbers and lower case
letters denoting row or column vectors. For example, I will write

1
2

and
v=
u = (1 2 3 4)
3 .
4

Many students ask me whether they can write v to denote a vector.

My answer is NO. It is best to avoid that notation because that arrow
suggests direction and as I said in 2 we are not using the word vector to
mean things with magnitude and direction. Also it takes more work to write

v than v.
Warning. .

20

2. MATRICES

4. Matrix arithmetic: addition and subtraction

4.1. We add two matrices of the same size by adding the entries of one
to those of the other, e.g.,

 
 

1 2 3
1 2 0
0 4 3
+
=
.
4 5 6
3 2 1
7 3 7
The result of adding two matrices is called their sum. Subtraction is defined
in a similar way, e.g.,

 
 

1 2 3
1 2 0
2 0 3

=
.
4 5 6
3 2 1
1 7 5
Abstractly, if (aij ) and (bij ) are matrices of the same size, then
(aij ) + (bij ) = (aij + bij )

and

(aij ) (bij ) = (aij bij ).

4.2. Matrices of different sizes can not be added. The sum of

two matrices of different sizes is not defined. Whenever we write A + B we
tacitly assume that A and B have the same size.
4.3. Addition of matrices is commutative:
A + B = B + A.
The commutativity is a simple consequence of the fact that addition of
numbers is commutative.
4.4. Addition of matrices, like addition of numbers, is associative:
(A + B) + C = A + (B + C).
This allows us to dispense with the parentheses: the expression A + B + C
is unambiguousyou get the same answer whether you first add A and B,
then C, or first add B and C, then A. Even more, because + is commutative
you also get the same answer if you first add A and C, then B.
5. The zero matrix and the negative of a matrix
The zero matrix is the matrix consisting entirely of zeroes. Of course, we
shouldnt really say the zero matrix because there is one zero matrix of size
m n for each choice of m and n. We denote the zero matrix of any size by
0. Although the symbol 0 now has many meanings it should be clear from
the context which zero matrix is meant.
The zero matrix behaves like the number 0:
(1) A + 0 = A = 0 + A for all matrices A, and
(2) given a matrix A, there is a unique matrix A0 such that A + A0 = 0.
The matrix A0 in (2) is denoted by A and its entries are the negatives of
the entries in A.
After we define multiplication of matrices we will see that the zero matrix
shares another important property with the number zero: the product of a
zero matrix with any other matrix is zero.

6. MATRIX ARITHMETIC: MULTIPLICATION

21

6. Matrix arithmetic: multiplication

6.1. Let A be an m n matrix and B a p q matrix. The product AB
is only defined when n = p and in that case the product is an m q matrix.
For example, if A is a 2 3 matrix and B a 3 4 matrix we can form the
product AB but there is no product BA! In other words,
the product AB can only be formed if the number of
columns in A is equal to the number of rows in B.
The rule for multiplication will strike you as complicated and arbitrary
at first though you will later understand why it is natural. Let A be an
m n matrix and B an n q matrix. Before being precise, let me say that
the
the ij th entry of AB is obtained by multiplying
the ith row of A by the j th column of B.
What we mean by this can be seen by reading the next example carefully.
The product AB of the matrices


1 2
A=
0 1

and



5 4 3
B=
2 1 2

is the 2 3 matrix whose

11-entry is (row 1)(column
12-entry is (row 1)(column
13-entry is (row 1)(column
21-entry is (row 2)(column
22-entry is (row 2)(column
23-entry is (row 2)(column

 
5
1) = (1 2)
=15+22=9
2
 
4
=14+21=6
2) = (1 2)
1
 
3
3) = (1 2)
=13+22=7
2
 
5
1) = (0 1)
= 0 5 1 2 = 2
2
 
4
2) = (0 1)
= 0 4 1 1 = 1
1
 
3
3) = (0 1)
= 0 3 1 2 = 2
2

In short,

AB =


9
6
7
.
2 1 2

The product BA does not make sense in this case.

22

2. MATRICES

My own mental picture for remembering how to multiply matrices is

encapsulated in the diagram
/
q q q q q q
o

q
q
q
/q
q
q

/

q
q
q
q
q
q


q
q
q
q
q
q


Then one computes the dot product.

6.2. Powers of a matrix. A square matrix, i.e., an n n matrix, has
the same number of rows as columns so we can multiply it by itself. We
write A2 rather that AA for the product. For example, if








1 1
2 1
3 2
5 3
2
3
4
A=
, then A =
, A =
, A =
.
1 0
1 1
2 1
3 2
Please check those calculations to test your understanding.
6.3. The formal definition of multiplication. Let A be an m n
matrix and B an n q matrix. Then AB is the m q matrix whose ij th
entry is
(6-1)

(AB)ij =

n
X

Ait Btj .

t=1

Do you understand this formula? Is it compatible with what you have

understood above? If A is the 3 4 matrix with Aij = |4 i j| and
B is the 4 3 matrix with Bij = ij 4, what is (AB)23 ? If you cant answer
this question there is something about the definition of multiplication or
something about the notation I am using that you do not understand. Do
yourself and others a favor by asking a question in class.
6.4. The zero matrix. For every m and n there is an mn matrix we
call zero. It is the m n matrix with all its entries equal to 0. The product
of any matrix with the zero matrix is equal to zero. Why?
6.5. The identity matrix. For every n there is an n n matrix we
call the n n identity matrix. We often denote it by In , or just I if the n is
clear from the context. It is the matrix with entries
(
1 if i = j
Iij =
0 if i 6= j.

6. MATRIX ARITHMETIC: MULTIPLICATION

23

In other words every entry on the NW-SE diagonal is 1 and all other entries
are 0. For example, the 4 4 identity matrix is

1 0 0 0
0 1 0 0

0 0 1 0 .
0 0 0 1
The key property of the n n identity matrix I is that if A is any m n
matrix, then AI = A and, if B is any n p matrix IB = B. You should
check this with a couple of examples to convince yourself it is true. Even
better give a proof of it for all n by using the definition of the product. For
example, if A is an m n matrix, then
(AI)ij =

n
X

Aik Ikj = Aij

k=1

where we used the fact that Ikj is zero except when kji, i.e., the only nonzero term in the sum is Aij Ijj = Aij .
6.6. Multiplication is associative. Another way to see whether you
understand the definition of the product in (6-1) is to try using it to prove
that matrix multiplication is associative, i.e., that
(6-2)

(AB)C = A(BC)

for any three matrices for which this product makes sense. This is an important property, just as it is for numbers: if a, b, and c, are real numbers
then (ab)c = a(bc); this allows us to simply write abc, or when there are four
numbers abcd, because we know we get the same answer no matter how we
group the numbers in forming the product. For example
(2 3) (4 5) = 6 20 = 120
and


(2 3) 4 5 = (6 4) 5 = 24 5 = 120.
Formula (6-1) only defines the product of two matrices so to form the
product of three matrices A, B, and C, of sizes k `, ` m, and m n,
respectively, we must use (6-1) twice. But there are two ways to do that:
first compute AB, then multiply on the right by C; or first compute BC,
then multiply on the left by A. The formula (AB)C = A(BC) says that
those two alternatives produce the same matrix. Therefore we can write
ABC for that product (no parentheses!) without ambiguity. Before we can
do that we must prove (6-2) by using the definition (6-1) of the product of
two matrices.

24

2. MATRICES

6.7. Try using (6-1) to prove (6-2). To show two matrices are equal you
must show their entries are the same. Thus, you must prove the ij th entry of
(AB)C is equal to the ij th entry of A(BC). To begin, use (6-1) to compute
((AB)C)ij . I leave you to continue.
This is a test of your desire to pass the course. I know you want to pass
the course, but do you want that enough to do this computation?1
Can you use the associative law to prove that if J is an n n matrix
with the property that AJ = A and JB = B for all m n matrices A and
all n p matrices B, then J = In ?
6.8. The distributive law. If A, B, and C, are matrices of sizes such
that the following expressions make sense, then
(A + B)C = AC + BC.
6.9. The columns of the product AB. Suppose the product AB
exists. It is sometimes useful to write B j for the j th column of B and write
B = [B 1 , . . . , B n ].
The columns of AB are then given by the formula
AB = [AB 1 , . . . , AB n ].
You should check this asssertion.
6.10. The product Ax. Suppose A = [A1 , A2 , . . . , An ] is a matrix
with n columns and

x1
x2

x = .. .
.
xn
Then
Ax = x1 A1 + x2 A2 + + xn An .
This last equation is one of the most important equations in the course.
Tattoo it on your body. If you dont know this equation you will probably
fail the course.
Try to prove it yourself. First do two simple examples, A a 2 3 matrix,
and A a 3 2 matrix. Then try the general case, A an m n matrix.
1Perhaps I will ask you to prove that (AB)C = A(BC) on the midtermwould you

prefer to discover whether you can do that now or then?

8. TRANSPOSE

25

6.11. Multiplication by a scalar. There is another kind of multiplication we can do. Let A be an m n matrix and let c be any real number.
We define cA to be the m n matrix obtained by multiplying every entry
of A by c. Formally, (cA)ij = cAij for all i and j. For example,

 

1 3 0
3
9 0
3
=
.
4 5 2
12 15 6
We also define the product Ac be declaring that it is equal to cA, i.e.,
Ac = cA.
7. Pitfalls and warnings
7.1. Warning: multiplication is not commutative. If a and b are
numbers, then ab = ba. However, if A and B are matrices AB need not
equal BA even if both products exist. For example, if A is a 2 3 matrix
and B is a 3 2 matrix, then AB is a 2 2 matrix whereas BA is a 3 3
matrix.
Even if A and B are square matrices of the same size, which ensures
that AB and BA have the same size, AB need not equal BA. For example,


 

0 0
1 0
0 0
(7-1)
=
1 0
0 0
1 0
but
(7-2)



 

1 0
0 0
0 0
=
.
0 0
1 0
0 0

7.2. Warning: a product of non-zero matrices can be zero. The

calculation (7-2) shows AB can be zero when both A and B are non-zero.
7.3. Warning: you cant always cancel. If A is not zero and AB =
AC it need not be true that B = C. For example, in (7-2) we see that
AB = 0 = A0 but the A cannot be cancelled to deduce that B = 0.
8. Transpose
The transpose of an m n matrix A is the n m matrix AT defined by
(AT )ij := Aji .
Thus the transpose of A is the reflection of A in the diagonal, and the
rows of A become the columns of AT and the columns of A become the rows
of AT . An example makes it clear:


T
1 4
1 2 3
= 2 5 .
4 5 6
3 6
Notice that (AT )T = A.

26

2. MATRICES

The transpose of a column vector is a row vector and vice versa. We

often use the transpose notation when writing a column vectorit saves
space2 and looks better to write v T = (1 2 3 4) rather than

1
2

v=
3 .
4
Check that
(AB)T = B T AT .
A lot of students get this wrong: they think that (AB)T is equal to AT B T .
(If the product AB makes sense B T AT need not make sense.)
9. Some special matrices
We have already met two special matrices, the identity and the zero
matrix. They behave like the numbers 1 and 0 and their great importance
derives from that simple fact.
9.1. Symmetric matrices. We call A a symmetric matrix if AT =
A. A symmetric matrix must be a square matrix. A symmetric matrix is
symmetric about its main diagonal. For example

0 1 2 3
1 4 5 6

2 5 7 8
3 6 8 9
is symmetric.
A square matrix A is symmetric if Aij
the matrix

1 2
2 0
5
4

= Aji for all i and j. For example,

5
4
9

is symmetric. The name comes from the fact that the entries below the
diagonal are the same as the corresponding entries above the diagonal where
corresponding means, roughly, the entry obtained by reflecting in the
diagonal. By the diagonal of the matrix above I mean the line from the top
left corner to the bottom right corner that passes through the numbers 1,
0, and 9.
If A is any square matrix show that A + AT is symmetric.
Is AAT symmetric?
Use the definition of multiplication to show that (AB)T = B T AT .
2Printers dislike blank space because it requires more paper. They also dislike black

space, like N, F, , , because it requires more ink.

10. SOLVING AN EQUATION INVOLVING AN UPPER TRIANGULAR MATRIX

27

9.2. Skew-symmetric matrices. A square matrix A is skew-symmetric

if AT = A.
If B is a square matrix show that B B T is skew symmetric.
What can you say about the entries on the diagonal of a skew-symmetric
matrix.
9.3. Upper triangular matrices. A square matrix is upper triangular
if all its entries below the diagonal are zero. For example,

0 1 2 3
0 4 5 6

0 0 7 8
0 0 0 9
is upper triangular.
9.4. Lower triangular matrices. A square matrix is lower triangular
if all its entries above the diagonal are zero. For example,

1 0 0
4 5 0
7 8 9
is lower triangular. The transpose of a lower triangular matrix is upper
triangular and vice-versa.
10. Solving an equation involving an upper triangular matrix
Here is an easy problem: find numbers x1 , x2 , x3 , x4 such that

1 2 3 4
x1
1
0 2 1 0 x2 2

Ax =
0 0 1 1 x3 = 3 = b.
0 0 0 2
x4
4
Multiplying the bottom row of the 4 4 matrix by the column x gives 2x4
and we want that to equal b4 which is 4, so x4 = 2. Multiplying the third
row of the 4 4 matrix by x gives x3 + x4 and we want that to equal b3
which is 3. We already know x4 = 2 so we must have x3 = 1. Repeating
this with the second row of A gives 2x2 x3 = 2, so x2 = 21 . Finally the
first row of A gives x1 + 2x2 + 3x3 + 4x4 = 1; plugging in the values we have
found for x4 , x3 , and x2 , we get x1 + 1 3 + 8 = 1 so x1 = 5. We find that
a solution is given by

5
1
2
x=
1 .
4
Is there any other solution to this equation? No. Our hand was forced
at each stage.
Here is a hard problem:

28

2. MATRICES

11. Some special products

Here we consider an m n matrix B and the effect of multiplying B on
the left by some special m m matrices. In what follows, let

x1
x2

X = .. .
.
xm
11.1. Let D denote

1 0
0 2

0 0

(11-1)
..
.

0 0
0 0

the m m matrix on the left in the product.

x1
1 x1
0
0
0
x x
0
0
0
.2 2. 2

3
0
0
.. ..

.. =
..

.
.
. .
0 m1 0 .. ..
0
0
m
xm
m xm

Since DB = [DB 1 , . . . , DB n ] it follows from (11-1) that the ith row of DB

is i times the ith row of B.
A special case of this appears when we discuss elementary row operations
in chapter 5. There we take i = c and all other j s equal to 1. In that case
DB is the same as B except that the ith row of DB is c times the ith row
of DB.
11.2. Fix two different integers i and j and let E be the m m matrix
obtained by interchanging the ith and j th rows of the identity matrix. If X
is an m 1 matrix, then EX is the same as X except that the entries in the
ith and j th positions of X are switched.
Thus, if B is an m n matrix, then EB is the same as B except that the
ith and j th rows are interchanged. (Check that.) This operation, switching
two rows, will also appear in chapter 5 when we discuss elementary row
operations.
11.3. Fix two different integers i and j and let F be the m m matrix
obtained from the identity matrix by changing the ij th entry from zero to
c. If X is an m 1 matrix, then F X is the same as X except that the entry
xi has been replaced by xi + cxj . Thus, if B is an m n matrix, then F B
is the same as B except that the ith row is now the sum of the ith row and
c times the j th row. (Check that.) This operation too appears when we
discuss elementary row operations.
11.4. The matrices D, E, and F all have inverses, i.e., there are matrices
D0 , E 0 , and F 0 , such that I = DD0 = D0 D = EE 0 = E 0 E = F F 0 = F 0 F ,
where I denotes the m m identity matrix. (Oh, I need to be careful here
I mean D is the matrix in (11-1) with i = c and all other j s equal to 1.)

11. SOME SPECIAL PRODUCTS

29

11.5. Easy question. Is there a positive integer n such that


n 

0 1
1 0
=
?
1 1
0 1
If so what is the smallest such n? Can you describe all other n for which
this equality holds?

CHAPTER 3

Matrices and motions in R2 and R3

1. Linear transformations
Functions play a central role in all parts of mathematics. The functions
relevant to linear algebra are called linear transformations.
We write Rn for the set of n 1 column vectors. This has a linear
structure:
(1) if u and v are in Rn so is u + v;
(2) if c R and u Rn , then cu Rn .
A non-empty subset W of Rn is called a subspace if it has these two
properties: i.e.,
(1) if u and v are in W so is u + v;
(2) if c R and u W , then cu W .
A linear transformation from Rn to Rm is a function f : Rn Rm such
that
(1) f (u + v) = f (u) + f (v) for all u, v Rn , and
(2) f (cu) = cf (u) for all c R and all u Rn .
In colloquial terms, these two requirements say that linear transformations
preserve the linear structure.
Left multiplication by an m n matrix A is a linear function from Rn
to Rm . If x Rn , then Ax Rm .
2. Rotations in the plane
3. Projections in the plane
4. Contraction and dilation
5. Reflections in the plane
6. Reflections in R3
7. Projections from R3 to a plane

31

CHAPTER 4

Systems of Linear Equations

The meat of this chapter begins in 2 below. However, some of the
terminology and framework for systems of linear equations applies to other
systems of equations so 1 starts with a discussion of the general framework.
Nevertheless, it is probably best to go directly to 2, read through to the
end of the chapter then return to 1.

1. Systems of equations
You have been solving equations for years. Sometimes you have considered the solutions to a single equation. For example, you are familiar with
the fact that the solutions to the equation y = x2 4x + 1 form a parabola.
Likewise, the solutions to the equation (x 2)2 + (y 1)2 + z 2 = 1 form a
sphere of radius 1 centered at the point (2, 1, 0).

1.1. Lets consider these statements carefully. By a solution to the

equation y = x2 x 4 we mean a pair of numbers (a, b) that give a true
statement when they are plugged into the equation, i.e., (a, b) is called a
solution if b does equal a2 4a + 1. If a2 4a + 1 6= b, then (a, b) is not a
solution to y = x2 x 4.

1.2. Pairs of numbers (a, b) also label points in the plane. More precisely, we can draw a pair of axes perpendicular to each other, usually called
the x- and y-axes, and (a, b) denotes the point obtained by going a units
along the x-axis and b-units along the y-axis. The picture below illustrates
33

34

4. SYSTEMS OF LINEAR EQUATIONS

the situation:
yO

(a, b)

/x

When we label a point by a pair of numbers the order of the numbers

matters. For example, if a 6= b, the points (a, b) and (b, a) are different. In
the above example, we have
yO
a

(b, a)

(a, b)

/x

The order also matters when we say (3, 2) is a solution to the equation
y = x2 x 4. Although (3, 2) is a solution to y = x2 x 4, (2, 3) is not.
1.3. R2 and R3 . We write R2 for the set of all ordered pairs of numbers
(a, b). Here R denotes the set of real numbers and the superscript 2 in R2
indicates pairs of real numbers. The general principle is that a solution to
an equation in 2 unknowns is an ordered pair of real numbers, i.e., a point
in R2 .
A solution to an equation in 3 unknowns, (x 2)2 + (y 1)2 + z 2 = 1
for example, is an ordered triple of numbers, i.e., a triple (a, b, c) such that
(a 2)2 + (b 1)2 + c2 does equal 1. The set of all ordered triples of numbers
is denoted by R3 .

1. SYSTEMS OF EQUATIONS

35

1.4. A system of equations. We can consider more than one equation at a time. When we consider several equations at once we speak of a
system of equations. For example, we can ask for solutions to the system of
equations
(
y = x2 x 4
(1-1)
y = 3x 7.
A solution to this system of equations is an ordered pair of numbers (a, b)
with the property that when we plug in a for x and b for y both equations
are true. That is, (a, b) is a solution to the system (1-1) if b = a2 a 4
and b = 3a 7. For example, (3, 2) is a solution to the system (1-1). So is
(1, 4). Although (4, 8) is a solution to y = x2 x 4 it is not a solution
to the system (1-1) because 8 6= 3 4 7.
On the other hand, (4, 8) is not a solution to the system (1-1): it is
a solution to y = x2 x 4 but not a solution to y = 3x 7 because
8 6= 3 4 7.
1.5. More unknowns and Rn . The demands of the modern world are
such that we often encounter equations with many unknowns, sometimes
billions of unknowns. Lets think about a modest situation, a system of 3
equations in 5 unknowns. For example, a solution to the system of equations

2
3
4
5

x1 + x2 + x3 + x4 + x5 = 100
(1-2)
x1 + 2x2 + 3x3 + 4x4 + 5x5 = 0

x1 x2 + x3 x4 x25 = 20
is an ordered 5-tuple1 of numbers, (s1 , s2 , s3 , s4 , s5 ), having the property
that when we plug in s1 for x1 , s2 for x2 , s3 for x3 , s4 for x4 , and s5 for x5 ,
all three of the equations in (1-2) become true. We use the notation R5 to
denote the set of all ordered 5-tuples of real numbers. Thus, the solutions
to the system (1-2) are elements of R5 .
We often call elements of R5 points just as we call points in the plane
points. A point in R5 has 5 coordinates, whereas a point in the plane has 2
coordinates.
More generally, solutions to a system of equations with n unknowns are,
or can be thought of as, points in Rn . We sometimes call Rn n-space. For
example, we refer to the physical world around us as 3-space. If we fix
an origin and x-, y-, and z-axes, each point in our physical world can be
1We run out of words after pair, triple, quadruple, quintuple, sextuple, ... so invent the

word n-tuple to refer to an ordered sequence of n numbers where n can be any positive
integer. For example, (8,7,6,5,4,3,2,1) is an 8-tuple. It is not the same as the 8-tuple
(1,2,3,4,5,6,7,8). We think these 8-tuples as labels for two different points in R8 . Although
we cant picture R8 we can ask questions about it. For example, does (1,1,1,1,1,1,1,1) lie
on the line through the points (8,7,6,5,4,3,2,1) and (1,2,3,4,5,6,7,8)? The answer is yes.
Why? If you can answer this you understand what is going on. If you cant you dont,
and should ask a question.

36

4. SYSTEMS OF LINEAR EQUATIONS

labelled in a unique way by an ordered triple of numbers. If we take time as

well as position into account we need 4-coordinates and then speak of R4 .
Physicists like to call R4 space-time. It has 3 space-like coordinates and 1
time coordinate.
2)2

1.6. Non-linear equations. The equations y = x2 x 4 and (x

+ (y 1)2 + z 2 = 1 are not linear equations.
2. A single linear equation
The general equation of a line in R2 is of the form
ax + by = c.

The general equation of a plane in R3 is of the form

ax + by + cz = d.
These are examples of linear equations.
A linear equation is an equation of the form
(2-1)

a1 x1 + + an xn = b

in which the ai s and b belong to R and x1 , . . . , xn are unknowns. We call

the ai s the coefficients of the unknowns.
A solution to (2-1) is an ordered n-tuple (s1 , . . . , sn ) of real numbers that
when substituted for the xi s makes equation (2-1) true, i.e.,
a1 s1 + + an sn = b.
When n = 2 a solution is a pair of numbers (s1 , s2 ) which we can think of
as the coordinates of a point in the plane. When n = 3 a solution is a triple
of numbers (s1 , s2 , s3 ) which we can think of as the coordinates of a point
in 3-space. We will use the symbol R2 to denote the plane and R3 to denote
3-space. The idea for the notation is that the R in R3 denotes the set of real
numbers and the 3 means a triple of real numbers. The notation continues:
R4 denotes quadruples (a, b, c, d) of real numbers and R4 is referred to as
4-space.2
A geometric view. If n 2 and at least one ai is non-zero, equation
(2-1) has infinitely many solutions: for example, when n = 2 the solutions
are the points on a line in the plane R2 ; when n = 3 the solutions are the
points on a plane in 3-space R3 ; when n = 4 the solutions are the points on
a 3-plane in R4 ; and so on.
It will be important in this course to have a geometric picture of the set
of solutions. We begin to do this in earnest in chapter 6 but it is useful to
keep this in mind from the outset. Solutions to equation (2-1) are ordered
n-tuples of real numbers (s1 , . . . , sn ) and we think of the numbers si as the
coordinates of a point in n-space, i.e., in Rn .
2Physicists think of R4 as space-time with coordinates (x, y, z, t), 3 spatial coordinates,

and one time coordinate.

3. SYSTEMS OF LINEAR EQUATIONS

37

The collection of all solutions to a1 x1 + + an xn = b is a special kind

of subset in n-space: it has a linear structure and that is why we call (2-1)
a linear equation. By a linear structure we mean this:
Proposition 2.1. If the points p = (s1 , . . . , sn ) and q = (t1 , . . . , tn ) are
different solutions to (2-1), then all points on the line through p and q are
also solutions to (2-1).
This result is easy to see once we figure out how to describe the points
that lie on the line through p and q.
Lets write pq for the line through p and q. The prototypical line is the
real number line R so we want to associate to each real number a point
on pq. We do that by presenting pq in parametric form: pq consists of all
points of the form
p + (1 )q = (s1 + (1 )t1 , . . . , sn + (1 )tn )
as ranges over all real numbers. Notice that p is obtained when = 1 and
q is obtained when = 0.
Proof of Proposition 2.1. The result follows from the calculation




a1 s1 + (1 )t1 + + an sn + (1 )tn =
a1 s1 + + an sn ) + (1 )(a1 t1 + + an tn )
which equals b + (1 )b, i.e., equals b, if (s1 , . . . , sn ) and (t1 , . . . , tn ) are
solutions to (2-1).

3. Systems of linear equations

An m n system of linear equations is a collection of m linear equations
in n unknowns. We usually write out the general m n system like this:
a11 x1 + a12 x2 + + a1n xn = b1
a21 x1 + a22 x2 + + a2n xn = b2
..
..
.
.
am1 x1 + am2 x2 + + amn xn = bm .
The unknowns are x1 , . . . , xn , and the aim of the game is to find them when
the aij s and bi s are specific real numbers. The aij s are called the coefficients
of the system. We often arrange the coefficients into an m n array

a11 a12 a1n

a21 a22 a2n

A := ..
..
..
.
.
.
am1 am2
called the coefficient matrix.

amn

38

4. SYSTEMS OF LINEAR EQUATIONS

4. A system of linear equations is a single matrix equation

We can assemble the bi s and the unknowns xj into column vectors


b1
x1
b2
x2


b := ..
and
x := .. .
.
.
bm

xn

The system of linear equations at the beginning of 3 can now be written as

a single matrix equation
Ax = b.
You should multiply out this equation and satisfy yourself that it is the same
system of equations as at the beginning of chapter 3. This matrix interpretation of the system of linear equations will be center stage throughout this
course.
We often arrange all the data into a single m (n + 1) matrix

a11 a12 a1n | b1

a21 a22 a2n | b2

(A| b) = ..
..
..
.
.
.
.
.
.
am1 am2

amn | bm

that we call the augmented matrix of the system.

5. Specific examples
5.1. A unique solution. The only solution to the 2 2 system
x1 + x2 = 2
2x1 x2 = 1
is (x1 , x2 ) = (1, 1). You can think of this in geometric terms. Each equation
determines a line in R2 , the points (x1 , x2 ) on each line corresponding to the
solutions of the corresponding equation. The points that lie on both lines
therefore correspond to simultaneous solutions to the pair of eqations, i.e.,
solutions to the 2 2 system of equations.
5.2. No solutions. The 2 2 system
x1 + x2 = 2
2x1 + 2x2 = 6
has no solution at all: if the numbers x1 and x2 are such that x1 + x2 = 2
then 2(x1 + x2 ) = 4, not 6. It is enlightening to think about this system
geometrically. The points in R2 that are solutions to the equation x1 +x2 = 2
lie on the line of slope 1 passing through (0, 2) and (2, 0). The points in R2
that are solutions to the equation 2(x1 + x2 ) = 6 lie on the line of slope 1
passing through (0, 3) and (3, 0). Thus, the equations give two parallel lines:

5. SPECIFIC EXAMPLES

39

there is no point lying on both lines and therefore no common solution to the
pair of equations, i.e., no solution to the given system of linear equations.
5.3. No solutions. The 3 2 system
x1 + x2 = 2
2x1 x2 = 1
x1 x2 = 3
has no solutions because the only solution to the 2 2 system consisting
of the first two equations is (1, 1) and that is not a solution to the third
equation in the 3 2 system. Geometrically, the solutions to each equation
lie on a line in R2 and the three lines do not pass through a common point.
5.4. A unique solution. The 3 2 system
x1 + x2 = 2
2x1 x2 = 1
x1 x2 = 0
has a unique solution, namely (1, 1). The three lines corresponding to the
three equations all pass through the point (1, 1).
5.5. Infinitely many solutions. It is obvious that the system consisting of the single equation
x1 + x2 = 2
has infinitely many solutions, namely all the points lying on the line of slope
1 passing through (0, 2) and (2, 0).
5.6. Infinitely many solutions. The 2 2 system
x1 + x2 = 2
2x1 + 2x2 = 4
has infinitely many solutions, namely all the points lying on the line of slope
1 passing through (0, 2) and (2, 0) because the two equations actually give
the same line in R2 . A solution to the first equation is also a solution to the
second equation in the system.
5.7. Infinitely many solutions. The 2 3 system
x1 + x2 + x3 = 3
2x1 + x2 x3 = 4
also has infinitely many solutions. The solutions to the first equation are
the points on the plane x1 + x2 + x3 = 3 in R3 . The solutions to the second
equation are the points on the plane 2x1 +x2 x3 = 4 in R3 . The two planes
meet one another in a line. That line can be described parametrically as the
points
(1 + 2t, 2 3t, t)

40

4. SYSTEMS OF LINEAR EQUATIONS

as t ranges over all real numbers. You should check that when x1 = 1 + 2t,
x2 = 2 3t, and x3 = t, both equations in the 2 3 system are satisfied.
5.8. A unique solution. The 3 3 system
x1 + x2 + x3 = 3
2x1 + x2 x3 = 4
4x1 + 3x2 2x3 = 13
has a unique solution, namely (x1 , x2 , x3 ) = (1, 5, 1). The solutions to
each equation are the points in R3 that lie on the plane given by the equation.
There is a unique point lying on all three planes , namely (1, 5, 1). This
is typical behavior: two planes in R3 meet in a line (unless they are parallel),
and that line will (usually!) meet a third plane in a point. The next two
example show this doesnt always happen: in Example 5.9 the third plane is
parallel to the line that is the intersection of the first two planes; in Example
5.10 the third plane is contains the line that is the intersection of the first
two planes.
5.9. No solution. We will now show that the 3 3 system
x1 + x2 + x3 = 3
2x1 + x2 x3 = 4
3x1 + x2 3x3 = 0
has a no solution. In Example 5.7 we showed that all solutions to the
system consisting of the first two equations in the 3 3 system we are now
considering are of the form (1 + 2t, 2 3t, t) for some t in R. However, if
(x1 , x2 , x3 ) = (1 + 2t, 2 3t, t), then
(5-1)

3x1 + x2 3x3 = 3(1 + 2t) + (2 3t) 3t = 5,

not 0, so no solution to the first two equations is a solution to the third

equation. Thus the 3 3 system has no solution.
5.10. Infinitely many solutions. The 3 3 system
x1 + x2 + x3 = 3
2x1 + x2 x3 = 4
3x1 + x2 3x3 = 5
has infinitely many solutions because, as equation (5-1) showed, every solution to the system consisting of the first two equations in the 3 3 system
we are currently considering is a solution to the equation third equation
3x1 + x2 3x3 = 5.

7. A GEOMETRIC VIEW ON THE NUMBER OF SOLUTIONS

41

6. The number of solutions

The previous examples show that a system of linear equations can have
no solution, or a unique solution, or infinitely many solutions. The next
proposition shows these are the only possibilities.
Proposition 6.1. If a system of linear equations has 2 solutions, it
has infinitely many.
Proof. Suppose (s1 , . . . , sn ) and (t1 , . . . , tn ) are different solutions to the
m n system
ai1 x1 + ai2 x2 + + ain xn = bi ,

(1 i m).

Then
(6-1)

(s1 + (1 )t1 , . . . , sn + (1 )tn )

is also a solution for every R because

ai1 (s1 + (1 )t1 ) + + ain (sn + (1 )tn ) =
(ai1 s1 + + ain sn ) + (1 )(ai1 t1 + + ain tn )
= bi + (1 )bi
= bi .
Since there are infinitely many s in R there are infinitely many solutions
to the system.

If we think of the solutions to an m n system as points in Rn , the
points in (6-1) are the points on the line through the points (s1 , . . . , sn ) and
(t1 , . . . , tn ). Thus, the proof of Proposition 6.1 says that if two points in Rn
are solutions to Ax = b, so are all points on the line through them. It is
this linear nature of the set of solutions that gives rise to the name linear
equation. It is also why we call this subject linear algebra. I think Linear
algebra and linear geometry would be a better name.
The following result is an immediate consequence of Proposition 6.1.
Corollary 6.2. A system of linear equations has either
(1) a unique solution or
(2) no solutions or
(3) infinitely many solutions.
A system of linear equations is consistent if it has a solution, and inconsistent
if it does not.
7. A geometric view on the number of solutions
Two
(1)
(2)
(3)

lines in the plane intersect at either

a unique point or
no point (they are parallel) or
infinitely many points (they are the same line).

42

4. SYSTEMS OF LINEAR EQUATIONS

Lines in the plane are the same things as equations a1 x + a01 y = b1 . The
points on the line are the solutions to the equation. The solutions to a pair
(system) of equations a1 x + a01 y = b1 and a2 x + a02 y = b2 are the points that
lie on both lines, i.e., their intersection.
In 3-space, which we will denote by R3 , the solutions to a single equation
form a plane. Two planes in R3 intersect at either
(1) no point (they are parallel) or
(2) infinitely many points (they are not parallel).
In the second case the intersection is a line if the planes are different, and a
plane if they are the same.
Three planes in R3 intersect at either
(1) a unique point or
(2) no point (all three are parallel) or
(3) infinitely many points (all three planes are the same or the intersection of two of them is a line that lies on the third plane).
I leave you to consider the possibilities for three lines in the plane and
four planes in R3 . Discuss with friends if necessary.
8. Homogeneous systems
A system of linear equations of the form Ax = 0 is called a homogeneous
system. A homogeneous system always has a solution, namely x = 0, i.e.,
x1 = = xn = 0. This is called the trivial solution because little brainpower
is needed to find it. Other solutions to Ax = 0 are said to be non-trivial.
For a homogeneous system the issue is to describe the set of non-trivial
solutions.
8.1. A trivial but important remark. If (s1 , . . . , sn ) is a solution to
a homogeneous system so is (s1 , . . . , sn ) for all R, i.e., all points on
the line through the origin 0 = (0, . . . , 0) and (s1 , . . . , sn ) are solutions.
8.2. A geometric view. The next result, Proposition 8.1, shows that
solutions to the system of equations Ax = b are closely related to solutions to
the homogeneous system of equations Ax = 0.3 We can see this relationship
in the setting of a single equation in 2 unknowns.
The solutions to the equation 2x + 3y = 0 are points on a line in the
plane R2 . Draw that line: it is the line through the origin with slope 23 .
The origin is the trivial solution. Other points on the line such as (3, 2)
are non-trivial solutions. The solutions to the non-homogeneous equation
2x + 3y = 1 are also points on a line in R2 . Draw it. This line has slope 32
too. It is parallel to the line 2x + 3y = 0 but doesnt go through the origin.
Pick any point on the second line, i.e., any solution to 2x + 3y = 1. Lets
pick (1, 1). Now take a solution to the first equation, say (3, 2) and add
3Students seem to have a hard time understanding Proposition 8.1. Section 8.3 dis-

cusses an analogy to Proposition 8.1 that will be familiar to you from your calculus courses.

8. HOMOGENEOUS SYSTEMS

43

it to (1, 1) to obtain (2, 1). The point (2, 1) is on the line 2x + 3y = 1.

In fact, if we take any solution to 2x + 3y = 0, say (a, b), and add it to
(1, 1) we obtain a solution to 2x + 3y = 1 because
(a, b) + (1, 1) = (a 1, b + 1)
and
2(a 1) + 3(b + 1) = 2a + 3b 2 + 3 = 0 + 1 = 1.
Every solution to 2x + 3y = 1 can be obtained this way. That is what
Proposition 8.1 says, though in much greater generailty.
There was nothing special about our choice of (1, 1). Everything would
work in the same way if we had chosen (5, 3) for example. Check it out!
The line 2x + 3y = 1 is obtained by translating (moving) the line 2x +
3y = 0 to the parallel line through (1, 1). Alternatively, the line 2x+3y = 1
is obtained by translating the line 2x + 3y = 0 to the parallel line through
(5, 3). The picture in chapter 3 might increase your understanding of this
important point (though a different equation is used there). The discussion
in chapter 3 might also help.
If I had taken 2x + 3y = 3 instead of 2x + 3y = 1 the same sort of thing
happens. The solutions to 2x + 3y = 3 lie on a line parallel to 2x + 3y = 0.
In fact, every line parallel to 2x+3y = 0 is the set of solutions to 2x+3y = c
for a suitable c.
Proposition 8.1. Suppose u is a solution to the equation Ax = b. Then
the set of all solutions to Ax = b consists of all vectors u + v in which v
ranges over all solutions to the homogeneous equation Ax = 0.
Proof. Fix a solution u to the equation Ax = b; i.e., Au = b. If u0 is a
solution to Ax = b, then u0 u is a solution to the homogeneous equation
Ax = 0, and u0 = u + (u0 u).
Conversely, if Av = 0, i.e., if v is a solution to the homogeneous equation
Ax = 0, then A(u + v) = Au + Av = b + 0 = b, so u + v is a solution to
Ax = b.

As we shall later see, the set of solutions to a system of homogeneous
linear equations is a subspace. Subspaces are fundamental objects in linear
algebra. You should compare Proposition 8.1 with the statement of Proposition 3.1. The two results say the same thing in slightly different words.
Corollary 8.2. Suppose the equation Ax = b has a solution. Then
Ax = b has a unique solution if and only if Ax = 0 has a unique solution.
Proof. Let u be a solution to the equation Ax = b; i.e., Au = b. Let v Rn .
Then A(u + v) = b + Av so u + v is a solution to the equation Ax = b if and
only if v is a solution to the equation Ax = 0.

You might find it easier to understand what Proposition 8.1 is telling
you by looking at it in conjunction with chapter 7.1.

44

4. SYSTEMS OF LINEAR EQUATIONS

8.3. An analogy for understanding Proposition 8.1. You probady

bly know that every solution to the differential equation dx
= 2x is of the
2
form x + c where c is an arbitrary constant. One particular solution to the
dy
dy
= 2x is x2 . Now consider the equation dx
= 0; the solutions to
equation dx
dy
this are the constant functions c, c R. Thus every solution to dx
= 2x is
dy
2
obtained by adding the particular solution x to a solution to dx = 0.
Compare the previous paragraph with the statement of Proposition 8.1:
2x is playing the role of b, x2 is playing the role of u, and c is playing the
role of v.
In Proposition 8.1 it doesnt matter which u we choose; all solutions to
Ax = b are given by u + v where v runs over all solutions to Ax = 0.
dy
Similarly, we can take any solution to dx
= 2x, x2 +17 for example, and
dy
= 2x are obtained by adding x2 + 17 to a solution
still all solutions to dx
dy
dy
to dx
= 0. That is, every solution to dx
= 2x is of the form x2 + 17 + c for
some c R.
8.4. A natural bijection between solutions to Ax = 0 and solutions to Ax = b. A bijection between two sets X and Y is a rule that
sets up a one-to-one correspondence between the elements on X and the elements of Y . We sometimes write x y to denote the fact that an element
x in X corresponds to the element y in Y . For example, there is a bijection
between the set of all integers and the set of even integers given by n 2n.
There is also a bijection between the even numbers and the odd numbers
given by 2m 2m + 1.
Suppose the equation Ax = b has a solution, say u. Then Proposition
8.1 says there is a bijection between the set of solutions to Ax = b and the
set of solutions to Ax = 0 given by
v u + v.
In other words, v is a solution to Ax = 0 if and only if u + v is a solution to
Ax = b. In plainer language, if Av = 0, then A(u + v) = b and conversely.
This is an arithmetic explanation, and generalization, of what was said
in the discussion prior to Proposition 8.1. There
(a, b) (a, b) + (1, 1).
As I said before there is nothing special about our choice of (1, 1). We could
take any point on 2x + 3y = 1. For example, if we pick (5, 3) we obtain
another bijection between the solutions to 2x + 3y = 0 and the solutions to
2x + 3y = 1, namely
(a, b) (a, b) + (5, 3).

CHAPTER 5

Row operations and row equivalence

1. Equivalent systems of equations
Two systems of linear equations are equivalent if they have the same
solutions. An important strategy for solving a system of linear equations is
to repeatedly perform operations on the system to get simpler but equivalent
systems, eventually arriving at an equivalent system that is very easy to
solve. The simplest systems to solve are those in row-reduced echelon form.
We discuss those in the next chapter. In this chapter we discuss the three
row operations you can perform on a system to obtain a simpler but
equivalent system.
We will write Ri to denote the ith row in a matrix.
The three elementary row operations on a matrix are the following:
(1) switch the positions of Ri and Rj ;
(2) replace Ri by a non-zero multiple of Ri ;
(3) replace Ri by Ri + cRj for any number c.
If A0 can be obtained from A by a single row operation, then A can be
obtained from A0 by a single row operation. We say that two matrices
are row equivalent if one can be obtained from the other by a sequence of
elementary row operations.
Proposition 1.1. Two m n systems of linear equations Ax = b and
A0 x = b0 are equivalent if and only if the augmented matrices (A|b) and
(A0 |b0 ) are row equivalent.
You might take a look back at chapter 11 to better understand the
elementary row operations.
1.1. Exercise. Find conditions on a, b, c, d such that the matrices




a b
1 3
and
c d
0 0
are row equivalent. (Use the fact that two matrices are row equivalent if
they have the same sets of solutions.)
2. Echelon Form
A matrix E is in row echelon form (REF) if
(1) all rows consisting entirely of zeroes are at the bottom of E; and
45

46

5. ROW OPERATIONS AND ROW EQUIVALENCE

(2) in each non-zero row the left-most non-zero entry, called the leading
entry of that row, is to the right of the leading entry in the row above
it.
A matrix E is in row-reduced echelon form (RREF) if
(1) it is in row echelon form, and
(2) every leading entry is a 1, and
(3) every column that contains a leading entry has all its other entries
equal to zero.
For example,

0 1 0 2 1 0 0 3
0 0 1 3 1 0 0 4

0 0 0 0 0 1 0 5

0 0 0 0 0 0 1 0
0 0 0 0 0 0 0 0
is in RREF.
Theorem 2.1. Given a matrix A, there is a unique row-reduced echelon
matrix that is row equivalent to A.
Proof.

The uniqueness result in Theorem 2.1 allows us to speak of the rowreduced echelon matrix that is equivalent to A. We will denote it by
rref(A).
2.1. Reminder. It is important to know there are infinitely many sequences of elementry row operations that can get you from A to rref(A).
In an exam, if you are given a specific matrix A and asked to put A in
row reduced echelon form there is a unique matrix, rref(A), that you must
produce, but the row operations that one student uses to get from A to
rref(A) will probably not be the same as the row operations used by another
student to get from A to rref(A). There is no best way to proceed in doing
this and therefore no guiding rule I can give you to do this. Experience and
practice will teach you how to minimize the number and complexity of the
operations and how to keep the intermediate matrices reasonably nice. I
often dont swap the order of the rows until the very end. I begin by trying
to get zeroes in the columns, i.e., all zeroes in some columns, and only one
non-zero entry in other columns, and I dont care which order of columns I
do this for.
3. An example
Doing elementary row operations by hand to put a matrix in row-reduced
echelon form can be a pretty tedious task. It is an ideal task for a computer!
But, lets take a deep breath and plunge in. Suppose the system is
MORE TO DO

4. YOU ALREADY DID THIS IN HIGH SCHOOL

47

4. You already did this in high school

You probably had to solve systems of equations like
x1 + x2 + x3 = 3
2x1 + x2 x3 = 4
4x1 + 3x2 2x3 = 13
in high school. (This is the system in chapter 5.8.) I want to persuade you
that the approach you used then is pretty much identical to the method we
are using here, i.e., putting the matrix (A | b) in row-reduced echelon form.
A typical high school approach to the problem is to add the first two
equations (to eliminate the x3 variable) and to add twice the first equation
to the third equation to get another equation without an x3 term. The two
new equations are
3x1 + 2x2 = 7
6x1 + 5x2 = 19
Now subtract twice the first of these two equations from the second to get
x2 = 5. Substitute this into 3x1 + 2x2 = 7 and get x1 = 1. Finally, from
the first of the original equations, x1 + x2 + x3 = 3, we obtain x3 = 1.
Thus, the system has a unique solution, (x1 , x2 , x3 ) = (1, 5, 1).
Lets rewrite what we did in terms of row operations. The first two
operations we performed are, in matrix notation,

1 1 1 | 3
1 1 1 | 3
3 2 0 | 7 .
(A | b) = 2 1 1 | 4
4 1 1 | 13
6 5 0 | 19
Then we subtracted twice the second row

1 1 1 |
3 2 0 |
0 1 0 |

from the third to get

3
7 .
5

The last row tells us that x2 = 5. Above, we substituted x2 = 5 into

3x1 + 2x2 = 7 and solved for x1 . That is the same as subtracting twice the
third row from the second, then dividing by three, i.e.,

1 1 1 | 3
1 1 1 | 3
1 1 1 | 3
3 2 0 | 7
3 0 0 | 3
1 0 0 | 1
0 1 0 | 5
0 1 0 | 5
0 1 0 | 5
which gives x1 = 1. Now subtract the second and third rows from the top
row to get

1 1 1 | 3
0 0 1 | 1
1 0 0 | 1
1 0 0 | 1
0 1 0 | 5
0 1 0 | 5
which gives x3 = 1. Admittedly the last matrix is not in row-reduced
echelon form because the left-most ones do not proceed downwards in the

48

5. ROW OPERATIONS AND ROW EQUIVALENCE

southeast direction, but that is fixed by one

the top row to the bottom, i.e.,

0 0 1 | 1
1 0
1 0 0 | 1
0 1
0 1 0 | 5
0 0

more row operation, moving

0 | 1
0 | 5 .
1 | 1

5. The rank of a matrix

The rank of an m n matrix A is the number
rank(A) := the number of non-zero rows in rref(A).
5.1. A very important remark about rank. Later on we will see
that the rank of A is equal to the dimension of the range of A. After
proving that equality it will be important to keep in mind these two different
interpretations of rank(A), i.e., as the number of non-zero rows in rref(A)
and as the dimension of the range of A.
5.2. Remark. If rref(A) = rref(B), then rank(A) = rank(B).
Proposition 5.1. If A is an m n matrix, then
rank(A) m
and
rank(A) n.
Proof. The rank of A is at most the number of rows in rref(A). But rref(A)
and A have the same size so rank(A) m.
Each non-zero row of rref(A) has a left-most 1 in it so the rank of A is
the number of left-most 1s. But a column of rref(A) contains at most one
left-most 1, so the rank of A is at most the number of columns in A, i.e.,
rank(A) n.

Proposition 5.2. Let A be an n n matrix. Then
(1) rank(A) = n if and only if rref(A) = I, the identity matrix;
(2) if rank(A) < n, then rref(A) has a row of zeroes.
Proof. (1) If rank(A) = n, then rref(A) has n non-zero rows and each of
those rows has a left-most 1, i.e., every row of rref(A) contains a left-most 1.
Suppose the left-most 1 in row i appear in column ci . The 1s move at least
one column to the right each time we go from each row to the one below it
so
1 c1 < c2 < < cn n.
It follows that c1 = 1, c2 = 2, . . . , cn = n. In other words, the left-most 1
in row j appears in column j; i.e., all the left-most 1s lie on the diagonal.
Therefore rref(A) = I.
(2) This is clear.


7. CONSISTENT SYSTEMS

49

6. Inconsistent systems
A system of linear equations is inconsistent if it has no solution (section
6). If a row of the form

(0 0 0 | b)

(6-1)

with b 6= 0 appears after performing some row operations on the augmented

matrix (A | b), then the system is inconsistent because row (6-1) says the
equation 0x1 + + 0xn = b appears in a system that is equivalent to the
original system of equations, and it is utterly clear that there is no solution
to the equation 0x1 + + 0xn = b when b 6= 0, and therefore no solution
to the original system of equations.

7. Consistent systems
Suppose Ax = b is a consistent m n system of linear equations .
Form the augmented matrix (A | b).
Perform elementary row operations on (A | b) until you obtain (rref(A) | d).
If column j contains the left-most 1 that appears in some row of
rref(A) we call xj a dependent variable.
The other xj s are called independent or free variables.
The number of dependent variables equals rank(A) and, because
A has n columns, the number of independent/free variables is n
rank(A).
All solutions are now obtained by allowing each independent variable to take any value and the dependent variables are now completely determined by the equations corresponding to (rref(A) | d)
and the values of the independent/free variables.
An example will clarify my meaning. If

(7-1)

1
0

0
(rref(A) | d) =
0

0
0

0
1
0
0
0
0

2
3
0
0
0
0

0
0
1
0
0
0

0 3
4 | 1
0 4
0 | 3

0 2 1 | 0

1 1 2 | 5

0 0
0 | 0
0 0
0 | 0

50

5. ROW OPERATIONS AND ROW EQUIVALENCE

the variables are x1 , x2 , x4 , and x5 . The independent/free variables x3 , x6 ,

and x7 , may take any values and then the dependent variables are determined by the equations
x1
x2
x3
x4
x5
x6
x7

= 1 2x3 3x6 4x7

= 3 3x3 4x6
=
x3
=
2x6 + x7
=5
x6 + 2x7
=
x6
=
x7

In other words, the set of all solutions to this system is given by

1 2x3 3x6 4x7

3 3x3 4x6

x3

2x6 + x7
x=

x
+
2x
6
7

x6
x7
or, equivalently,

(7-2)

1
2
3
4
3
3
4
0




0
1
0
0

x = 0 + x3 0 + x6 2 + x7
1
5
0
1
2




0
0
1
0
0
0
0
1

as x3 , x6 , x7 range over all real numbers. We call (7-2) the general solution
to the system.1
7.1. Particular vs. general solutions. If we make a particular choice
of x3 , x6 , and x7 , we call the resulting x a particular solution to the system.
1You will later see that the vectors obtained from the three right-most terms of (7-2)

as x3 , x6 , and x7 , vary over all of R form a subspace, namely the null space of A. Thus (72) is telling us that the solutions to Ax = b belong to a translate of the null space of A by
a particular solution of the equation, the particular solution being the term immediately
to the right of the equal sign in (7-2). See chapter 3 and Proposition 3.1 below for more
about this.

8. PARAMETRIC EQUATIONS FOR LINES AND PLANES

51

For example, when (x3 , x6 , x7 ) = (0, 0, 0) we obtain the particular solution

1
3

0
.
5

0
0
Notice that

2
3

1

0 ,

0

0
0

3
4

0

2 ,

1

1
0

4
0

0

1

2

0
1

are solutions to the homogeneous system Ax = 0. The general solution is

obtained by taking one particular solution, say u, perhaps that given by
setting all the independent variables to zero, and adding to it solutions to
the homogeneous equation Ax = 0. That is exactly what Proposition 8.1
below says.
The number of independent variables, in this case 3, which is nrank(A),
gives a measure of how many solutions there are to the equation Ax = b.
The fact that x3 , x6 , and x7 , can be any real numbers says there are 3
degrees of freedom for the solution space. Later we will introduce more
formal language which will allow us to say that the space of solutions is a
translate of a 3-dimensional subspace of R7 . That 3-dimensional subspace
is called the null space of A. You will understand this paragraph later. I
hope you read it again after you have been introduced to the null space,
The discussion in chapter 3 is also relevant to what we have discussed
in this chapter. In the language of chapter 3, the solutions to Ax = b are
obtained by translating the solutions to Ax, i.e., the null space of A, by a
particular solution to Ax = b.
Sections 8.2 and 8.4 are also related to the issue of particular versus
general solutions. Section 8.2 discusses the particular solution (1, 1) to
the equation 2x + 3y = 1 and the general solution (1, 1) + (3, 2) where
runs over all real numbers. As runs over all real numbers the points
(3, 2) give all the points on the line 2x + 3y = 0.
8. Parametric equations for lines and planes
You should review the material about parametric equations for curves
and surfaces that you encountered in a course prior to this. The only curves
and surfaces relevant to this course are lines and planes. These have particularly simple parametric descriptions.

52

5. ROW OPERATIONS AND ROW EQUIVALENCE

For example, the line 2x 3y = 1 in the plane consists of all points of

the form (3t + 2, 2t + 1), t R. Another way of saying this is the line is
given by the equations
x = 3t + 2
y = 2t + 1.
In other words, t is allowed to be any real number, each choice of t gives
exactly one point in the plane, and the totality of all points obtained is
exactly all points that lie on the line 2x 3y = 1. Another way of looking
at this is that the points in the plane that are solutions to the equation
2x 3y = 1 are the points on the line. For example, when t = 0 we get
the solution x = 2 and y = 1 or, equivalently, the point (2, 1); when t = 5
we get the solution (x, y) = (13, 9); when t = 4 we get the solution
(x, y) = (14, 9); and so on.
In a similar fashion, the points on the plane 2x y z = 3 are given by
the parametric equations
1
x = (3 + s + t)
2
y=s
z=t
or, if you prefer,
3
+s+t
2
y = 2s
z = 2t.

x=

As (s, t) ranges over all points of R2 , i.e., as s and t range over R, the points
( 23 + s + t, 2s, 2t) are all points on the plane 2x y z = 3.
In a similar fashion, the points on the line that is the intersection of the
planes 2xy z = 3 and x+y +z = 4 are given by the parametric equations
x=
y=
z=
9. The importance of rank
We continue to discuss a consistent m n system Ax = b.
We have seen that rank(A) m and rank(A) n (Proposition
5.1) and that the number of independent variables is n rank(A).
If rank(A) = n there are no independent variables, so the system
has a unique solution.
If rank(A) < n there is at least one independent variable, so the
system has infinitely many solutions.

10. THE WORD SOLUTION

53

Theorem 9.1. Let A be an nn matrix and consider the equation Ax =

b. If rank(A) = n, and the elementary row operations on the augmented
matrix (A | b) produce (rref(A) | d) = (I | d) where I is the identity matrix,
then d is the unique matrix such that Ad = b.
Proof. By Proposition 5.2, rref(A) = I. Hence the elementary row operations change (A | b) to (I | d) where I is the identity matrix. Therefore the
equations Ax = b and Ix = d have the same set of solutions. But Ix = x so
the only solution to Ix = d is x = d.

Proposition 9.2. Let Ax = 0 be an mn system of homogeneous linear
equations. If m < n, the system has a non-trivial solution.
Proof. By Proposition 5.1, rank(A) m. The number of independent
variables for the equation Ax = 0 is n rank(A) n m 1. Hence there
are infinitely many solutions by the remarks in chapter 9.

A shorter, simpler version of Proposition 9.2 is the following.
Corollary 9.3. A homogeneous system of equations with more unknowns than equations always has a non-trivial solution.
9.1. A simple proof of Corollary 9.3. Let Ax = 0 be a homogeneous
system of m equations in n unknowns and suppose m < n. Since rref(A) has
more columns than rows there will be a column that does not have a leading
1. Hence there will be at least one independent variable. That variable can
be anything; in particular, it can be given a non-zero value which leads to a
non-trivial solution.

10. The word solution

A midterm contained the following question: Suppose u is a solution to
the equation Ax = b and that v is a solution to the equation Ax = 0. Write
out a single equation, involving several steps, that shows u + v is a solution
to the equation Ax = b.
The correct answer is
A(u + v) = Au + Av = b + 0 = b.
Done! I talked to some students who could not answer the question. They
did not understand the meaning of the phrase u is a solution to the equation
Ax = b. The meaning is that Au = b. In principle, it is no more complicated than the statement 2 is a solution to the equation x2 + 3x = 10
which means that when you plug in 2 for x you obtain the true statement
4 + 6 = 10.

54

5. ROW OPERATIONS AND ROW EQUIVALENCE

11. Elementary matrices

The n n identity matrix, which is denoted by In , or just I when n is
clear from the context, is the n n matrix having each entry on its diagonal
equal to 1 and 0 elsewhere. For example, the 4 4 identity matrix is

1 0 0 0
0 1 0 0

0 0 1 0
0 0 0 1
An elementary matrix is a matrix that is obtained by performing a single
elementary row operation on an identity matrix. For example, if we switch
rows 2 and 4 of I4 we obtain the elementary matrix

1 0 0 0
0 0 0 1

0 0 1 0 .
0 1 0 0
If we multiply the third row of I4 by a
mentary matrix

1 0 0
0 1 0

0 0 c
0 0 0

non-zero number c we get the ele

0
0
.
0
1

If we replace the first row of I4 by itself + c times the second row we get
the elementary matrix

1 c 0 0
0 1 0 0

0 0 1 0 .
0 0 0 1
Theorem 11.1. Let E be the elementary matrix obtained by performing
an elementary row operation on the mm identity matrix. If we perform the
same elementary row operation on an m n matrix A the resulting matrix
is equal to EA.
We wont prove this. The proof is easy but looks a little technical because
of the notation. Lets convince ourselves that the theorem is plausible by
looking at the three elementary matrices above and the matrix

1 2 3
3 2 1

A=
1 1 1 .
1 0 2

11. ELEMENTARY MATRICES

Now carry out the

1
0

0
0
and

and

1
0

0
0

1
0

0
0

c
1
0
0

0
0
1
0

55

following three multiplications and check your answers:

0 0 0
1 2 3
1 2 3

0 0 1
3 2 1 = 1 0 2 ;

0 1 0
1 1 1
1 1 1
1 0 0
1 0 2
3 2 1
0
1
0
0

0
0
c
0

0
1
3
0

0 1
1
1

0
1
3
0

0 1
1
1

2
2
1
0

2
2
1
0

3
1
3
1
=
1 c
2
1

2
2
c
0

3
1
;
c
2

3
1 + 3c 2 + 2c 3 + c

1
2
1
= 3
.
1 1
1
1
2
1
0
2

Theorem 11.2. The following conditions on an n n matrix A are

equivalent:
(1) rref(A) = In ;
(2) A is a product of elementary matrices;
(3) A is invertible.
Proof. (1) = (2) Suppose rref(A) = In . Then In can be obtained from
A by a sequence of elementary row operations. Therefore A can be obtained from In by a sequence of elementary row operations. By Theorem
11.1, the result of each of those elementary row operations is the same
as the result obtained by multiplying on the left by an elementary matrix. Therefore a sequence of elementary row operations on In produces
a matrix of the form Ek Ek1 E1 where each Ej is an elementary matrix, i.e., A = Ek Ek1 E1 for a suitable collection of elementary matrices
E1 , . . . , E k .
(2) = (3)
(3) = (1)


CHAPTER 6

The Vector space Rn

We write Rn for the set of all n 1 column vectors. Elements of Rn are
called vectors or points in Rn . We call them points to encourage you to think
of Rn as a geometric object, a space, in the same sort of way as you think
of the number line R, the case n = 1, and the plane R2 , in which the points
of R2 are labelled by row vectors (a, b) after choosing a coordinate system,
and 3-space R3 , the space in which we live or, when you include time, R4 .
Solutions to an m n system Ax = b of linear equations are therefore
points in Rn . The totality of all solutions is a certain subset of Rn . As we
said just after Proposition 6.1, the set of solutions has a linear nature: if p
and q are solutions to Ax = b so are all points on the line pq.
1. Arithmetic in Rn
Because the elements of Rn are matrices, we can add them to one another, and we can multiply them by scalars (numbers). These operations
have the following properties which we have been using for some time now:
(1) if u, v Rn so is u + v;
(2) if u Rn and R, then u Rn ;
(3) u + v = v + u for all u, v Rn ;
(4) (u + v) + w = u + (v + w) for all u, v, w Rn ;
(5) there is an element 0 Rn having the property that 0 + u = u for
all u Rn ;1
(6) if u Rn there is an element u0 Rn such that u + u0 = 0;2
(7) ()u = (u) for all , R and all u Rn ;
(8) (u + v) = u + v for all R and all u, v Rn ;
(9) ( + )u = u + u for all , R and all u Rn ;
(10) 1.u = u for all u Rn .
1It is important to observe that there is only one element with this propertyif
0

0 Rn has the property that 00 + u = u for all u Rn , then 0 = 00 + 0 = 0 + 00 = 00

where the first equality holds because 00 +u = u for all u, the second equality holds because
of the commutativity property in (3), and the third equality holds because 0 + u = u for
all u.
2We denote u0 by u after showing that it is uniqueit is unique because if u + u0 =
u + u00 = 0, then
u0 = u0 + 0 = u0 + (u + u00 ) = (u0 + u) + u00 = (u + u0 ) + u00 = 0 + u00 = u00 .

57

6. THE VECTOR SPACE Rn

58

2. The standard basis for Rn

The vectors
e1 := (1, 0, . . . , 0, 0),
e2 := (0, 1, 0, . . . , 0),
..
..
.
.
en := (0, 0, . . . , 0, 1)
in Rn will appear often from now on. We call {e1 , . . . , en } the standard basis
for Rn . The notion of a basis is defined and discussed in chapter 10 below.
From now on the symbols
e1 , . . . , e n
will be reserved for the special vectors just defined. There is some potential
for ambiguity since we will use e1 to denote the vector (1, 0) in R2 , the vector
(1, 0, 0) in R3 , the vector (1, 0, 0, 0) in R4 , etc. Likewise for the other ei s. It
will usually be clear from the context which one we mean.
One reason for the importance of the ej s is that
(a1 , . . . , n ) = a1 e1 + a2 e2 + + an en .
In other words, every vector in Rn can be written as a sum of multiples of
the ej s in a unique way.
3. Linear combinations and linear span
A linear combination of vectors v 1 , . . . , v n in Rm is a vector that can be
written as
a1 v 1 + + an v n
for some numbers a1 , . . . , an R. We say the linear combination is non-trivial
if at least one of the ai s is non-zero.
The set of all linear combinations of v 1 , . . . , v n is called the linear span
of v 1 , . . . , v n and denoted
Sp(v 1 , . . . , v n )

or

hv 1 , . . . , v n i.

For example, the linear span of e1 , . . . , en is Rn itself: if (a1 , a2 , . . . , an ) is

any vector in Rn , then
(a1 , a2 , . . . , an ) = a1 e1 + a2 e2 + + an en .
3.1. Note that the linear span of v 1 , . . . , v n has the following properties:
(1) 0 Sp(v 1 , . . . , v n );
(2) if u and v are in Sp(v 1 , . . . , v n ) so are u + v;
(3) if a R and v Sp(v 1 , . . . , v n ) so is av.
You should check these three properties hold. We will return to them later
on they are the defining properties of a subspace of Rn (see chapter 1).

3. LINEAR COMBINATIONS AND LINEAR SPAN

59

Proposition 3.1. Let A be any m n matrix and x Rn . Then Ax is

a linear combination of the columns of A. Explicitly,
(3-1)

Ax = x1 A1 + + xn An

where Aj denotes the j th column of A.

Proof. Lets write A = (aij ) and x = (x1 , . . . , xn )T . Then

a11
a12
a1n
a21
a22
a2n

x1 A1 + + xn An = x1 .. + x2 .. + + xn ..
.
.
.
am1

am2

amn

a11 x1
a12 x2
a1n xn
a21 x1 a22 x2
a2n xn

= .. + .. + + ..
. .
.
am1 x1

am2 x2

amn xn

a11 x1 + a12 x2 + + a1n xn

a21 x1 + a22 x2 + + a2n xn
..
..
.
.

am1 x1 + am2 x2 + + amn xn

a11
a21

= ..
.

a12
a22

am1 am2

a1n
x1
x2
a2n

.. ..
. .
amn

xn

=Ax.
The above calculation made use of scalar multiplication (6.11): if r R and
B is a matrix, then rB is the matrix whose ij th entry is rBij ; since r and
Bij are numbers rBij = Bij r.

A useful way of stating Proposition 3.1 is that for a fixed A the set
of values Ax takes as x ranges over all of Rn is Sp(A1 , . . . , An ), the linear
span of the columns of A. Therefore, if b Rm , the equation Ax = b has
a solution if and only if b Sp(A1 , . . . , An ). This is important enough to
state as a separate result.
Corollary 3.2. Let A be an m n matrix and b Rm . The equation
Ax = b has a solution if and only if b is a linear combination of the columns
of A.

60

6. THE VECTOR SPACE Rn

4. Some infinite dimensional vector spaces

You already know some infinite dimensional vector spaces. Perhaps the
most familiar one is the set of all polynomials with real coefficients. We
denote it by R[x].
4.1. Polynomials. At high school one considers polynomial functions
of arbitrary degree. If f (x) and g(x) are polynomials so is their sum f (x) +
g(x); if R the function f (x) is a polynomial whenever f is; the zero
polynomial has the wonderful property that f (x) + 0 = 0 + f (x) = f (x).
Of course, other properties like associativity and commutativity of addition
hold, and the distributive rules (f (x) + g(x) = f (x) + g(x) and +
)f (x) = f (x)+f (x) hold. Every polynomial is a finite linear combination
of the powers 1, x, x2 , . . . of x. The powers of x are linearly independent
because the only way a0 + a1 x + + an xn can be the zero polynomial
is if all the ai s are zero. The powers of x form a basis for the space of
polynomials. Thus R[x] is an infinite dimensional vector space.
4.2. Other spaces of functions. One need not restrict attention to
polynomials. Let C 1 (R) denote the set of all differentiable functions f :
R R. This is also a vector space: you can add em, multiply em by
scalars, etc, and the usual associative, commutative and distributive rules
apply. Likewise, C(R), the set of all continuous functions f : R R is a
vector space.
MORE to say...
4.3. Infinite sequences. The set of all infinite sequences s = (s0 , s1 , . . .)
of real numbers is a vector space with respect to the following addition and
scalar multiplication:
(s0 , s1 , . . .)+(t0 , t1 , . . .) = (s0 +t0 , s1 +t1 , . . .),

(t0 , t1 , . . .) = (t0 , t1 , . . .).

CHAPTER 7

Subspaces
Let W denote the set of solutions to a system of homogeneous equations
Ax = 0. It is easy to see that W has the following properties: (a) 0 W ;
(b) if u and v are in W so is u + v; (c) if u W and R, then u W .
Subsets of Rn having this property are called subspaces and are of particular
importance in all aspects of linear algebra. It is also important to know
that every subspace is the set of solutions to some system of homogeneous
equations.
Another important way in which subspaces turn up is that the linear
span of a set of vectors is a subspace.
Now we turn to the official definition.
1. The definition and examples
A subset W of Rn is called a subspace if
(1) 0 W , and
(2) u + v W for all u, v W , and
(3) u W for all R and all u W .
The book gives a different definition but then proves that a subset W of Rn
is a subspace if and only if it satisfies these three conditions. We will use
our definition not the books.
We now give numerous examples.
1.1. The zero subspace. The set {0} consisting of the zero vector in
Rn is a subspace of Rn . We call it the zero subspace. It is also common to
call it the trivial subspace.
1.2. Rn itself is a subspace. This is clear.
1.3. The null space and range of a matrix. Let A be an m n
matrix. The null space of A is
N (A) := {x | Ax = 0}.
The range of A is
R(A) := {Ax | x Rn }.
Proposition 1.1. Let A be an m n matrix. Then N (A) is a subspace
of Rn and R(A) is a subspace of Rm .
61

62

7. SUBSPACES

Proof. First, we consider N (A). Certainly 0 N (A) because A0 = 0. now

suppose that u and v belong to N (A) and that a R. Then
A(u + v) = Au + Av = 0 + 0 = 0
and
A(au) = aAu = a0 = 0
so both u + v and au belong to N (A). Hence N (A) is a subspace of Rn .
Now consider R(A). Certainly 0 R(A) because A0 = 0. now suppose
that u and v belong to R(A) and that a R. Then there exist x and y in
Rn such that u = Ax and v = Ay. Therefore
u + v = Ax + Ay = A(x + y)
and
au = aAx = Aax
so both u + v and au belong to R(A). Hence R(A) is a subspace of Rm . 
It is sometimes helpful to think of R(A) as all multiples of A where a
multiple of A means a vector of the form Ax. It is clear that a multiple
of AB is a multiple of A; explicitly, ABx = A(Bx). It follows that
R(AB) R(A).
The following analogy might be helpful: every multiple of 6 is a multiple of
2 because 6 = 2 3. In set-theoretic notation
{6x | x Z} {2x | x Z}.
Similarly,
{ABx | x Rn }. {Ax | x Rn }
which is the statement that R(AB) R(A).
There is a similar inclusion for null spaces. If Bx = 0, then ABx = 0,
so
N (B) N (AB).
1.4. Lines through the origin are subspaces. Let u Rn . We
introduce the notation
Ru := {u | R}
for the set of all real multiples of u. We call Ru a line if u 6= 0. If u = 0,
then Ru is the zero subspace.
Lemma 1.2. Let u Rn . Then Ru is a subspace of Rn .
Proof. The zero vector is a multiple of u so is in Ru. A sum of two multiples
of u is again a multiple of u, and a multiple of a multiple of u is again a
multiple of u.


1. THE DEFINITION AND EXAMPLES

63

1.5. Planes through the origin are subspaces. Suppose that {u, v}
is linearly independent subset of Rn . We introduce the notation
Ru + Rv := {au + bv | a, b R}.
We call Ru + Rv the plane through u and v.
Lemma 1.3. Ru + Rv is a subspace of Rn .
Proof. Exercise. It is also a special case of Proposition 1.4 below.

You already know something about the geometry of R3 . For example,

if you are given three points in R3 that do not lie on a single line, then
there is a unique plane containing them. In the previous paragraph the
hypothesis that {u, v} is linearly independent implies that 0, u, and v, do
not lie on a single line. (In fact, the linear independence is equivalent to
that statementwhy?) The unique plane containing 0, u, and v is Ru + Rv.
You also know, in R3 , that given two lines meeting at a point, there is a
unique plane that contains those lines. Here, the lines Ru and Rv meet at
0, and Ru + Rv is the unique plane containing them.
1.6. The linear span of any set of vectors is a subspace.
Proposition 1.4. If v 1 , . . . , v d are vectors in Rn , then their linear span
Sp(v 1 , . . . , v d ) is a subspace of Rn . It is the smallest subspace of Rn containing v 1 , . . . , v d .
Proof. We leave the reader to show that Sp(v 1 , . . . , v d ) is a subspace of Rn .
That is a straightforward exercise that you should do in order to check that
you not only understand the meanings of linear combination, linear span,
and subspace, but can put those ideas together in a simple proof.
Suppose we have shown that Sp(v 1 , . . . , v d ) is a subspace of Rn . It
certainly contains v 1 , . . . , v d . However, if V is a subspace of Rn , it contains
all linear combinations of any set of elements in it. Thus if V is a subspace
containing v 1 , . . . , v d it contains all linear combinations of those vectors, i.e.,
V contains Sp(v 1 , . . . , v d ). Hence Sp(v 1 , . . . , v d ) is the smallest subspace of
Rn containing v 1 , . . . , v d .

Proposition 1.5. The range of a matrix is equal to the linear span of
its columns.1
1This result can be viewed as a restatement of Corollary 3.2 which says that Ax = b

has a solution if and only if b is a linear combination of the columns of A because the
range of an m n matrix can be described as
R(A) = {b Rm | b = Ax for some x Rn }
= {b Rm | the equation b = Ax has a solution}.

64

7. SUBSPACES

Proof. Let A be an m n matrix with columns A1 , . . . , An . If x Rn , then

Ax = x1 A1 + + xn An
so
R(A) = {x1 A1 + + xn An | x1 , . . . , xn R}.
But the right hand side is the set of all linear combinations of the columns
of A, i.e., the linear span of those columns.

2. The row and column spaces of a matrix
The row space of a matrix A is the linear span of its rows. The column space
of a matrix A is the linear span of its columns.
Lemma 2.1. If A and B are equivalent matrices they have the same row
space.
Proof. Since equivalent matrices are obtained from one another by a sequence of elementary row operations it suffices to show that the row space
does not change when we perform a single elementary row operation on A.
Let R1 , . . . , Rm denote the rows of A. If Ri and Rj are switched the row
space certainly stays the same. If Ri is replaced by Ri for any non-zero
number the row space stays the same. Since aRi + bRj = a(Ri + Rj ) +
(b a)Rj the row space does not change when Ri is replaced by Ri + Rj .

2.1. Sums of subspaces are subspaces. If S and T are subsets of
Rn , their sum is the set
S + T := {u + v | u S and v T }.
Lemma 2.2. If U and V are subspaces of Rn so is U + V .
Proof. Since U and V are subspaces, each contains 0. Hence 0 + 0 U + V ;
i.e., U + V contains 0.
If x, x0 U + V , then x = u + v and x0 = u0 + v 0 for some u, u0 U and
v, v 0 V . Using the fact that U and V are subspaces, it follows that
x + x0 = (u + v) + (u0 + v 0 ) = (u + u0 ) + (v + v 0 ) U + V
and, if a R, then
ax = a(u + v) = au + av U + V.
Hence U + V is a subspace.

The result can be extended to sums of more than subspaces using an

induction argument. Youll see the idea by looking at the case of three
subspaces U , V , and W , say. First, we define
U + V + W = {u + v + w | u U, v V, and w W }.
It is clear that U + V + W = (U + V ) + W . By the lemma, U + V is a
subspace. Now we may apply the lemma to the subspaces U + V and W ;

3. LINES, PLANES, AND TRANSLATIONS OF SUBSPACES

65

the lemma then tells us that (U + V ) + W is a subspace. Hence U + V + W

is a subspace.
As an example of the sum of many subspaces, notice that Sp(v 1 , . . . , v d ) =
Rv 1 + + Rv d so repeated application of the lemma tells us that the linear
span of any set of vectors is a subspace. This gives an alternative proof of
Proposition 1.4.
2.2. Intersections of subspaces are subspaces.
Lemma 2.3. If U and V are subspaces of Rn so is U V .
Proof. Because U and V are subspaces each of them contains the zero
vector. Hence 0 U V . Now suppose u and v belong to U V , and that
a R. Because u and v belong to the subspace U so does u + v; similarly, u
and v belong to V so u + v V . Hence u + v U V . Finally if w U V
and a R, then aw U because U is a subspace and aw V because V is
a subspace so aw U V .

2.3. Planes through the origin are subspaces. If the set {u, v} is
linearly independent we call the linear span of u and v a plane.
3. Lines, planes, and translations of subspaces
Let m and c be non-zero real numbers. Let ` be the line y = mx + c
and L the line y = mx. Both lines lie in R2 . Now L is a subspace but `
is not because it does not contain (0, 0). However, ` is parallel L, and can
be obtained by sliding L sideways, not changing the slope, so the it passes
through (0, c). We call the line y = mx + c a translation of the line y = mx;
i.e., ` is a translation of the subspace L. The picture below is an example
in which
` is y = 12 (x 3) and L is y = 12 x
O

`
L
?
?
?

66

7. SUBSPACES

Returning to the general case, every point on ` is the sum of (0, c) and a
point on the line y = mx so, if we write L for the subspace y = mx, then
` = (0, c) + L
= {(0, c) + v | v L}
= {(0, c) + (x, mx) | x R}
= (0, c) + R(1, m).
Let V be a subspace of Rn and z Rn . We call the subset
z + V := {z + v | v V }
the translation of V by z. We also call z + V a translate of V .
Proposition 3.1. Let S be the set of solutions to the equation Ax = b
and let V be the set of solutions to the homogeneous equation Ax = 0. Then
S is a translate of V : if z is any solution to Ax = b, then S = z + V .
Proof. Let x Rn . Then A(z + x) = b + Ax, so z + x S if and only if
x V . The result follows.

Now would be a good time to re-read chapter 7: that chapter talks
about obtaining solutions to an equation Ax = b by translating N (A) by
a particular solution to the equation Ax = b. In fact, Proposition 3.1 and
Proposition 8.1 really say the same thing in slightly different languages, and
the example in chapter 7 illustrates what these results mean in a concrete
situation.
3.1. The linear geometry of R3 . This topic should already be familiar.
Skew lines, parallel planes, intersection of line and plane, parallel lines.
For example, find an equation for the subspace of R3 spanned by (1, 1, 1)
and (1, 2, 3). Since the vectors are linearly independent that subspace will be
a plane with basis {(1, 1, 1), (1, 2, 3)}. A plane in R3 is given by an equation
of the form
ax + by + cz = d
and is a subspace if and only if d = 0. Thus we want to find a, b, c R such
that ax + by + cz = 0 is satisfied by (1, 1, 1) and (1, 2, 3). More concretely,
we want to find a, b, c R such that
x+y+z =0
x + 2y + 3z = 0.
This is done in the usual way:




1 1 1 | 0
1 1 1 | 0
1 2 3 | 0
0 1 2 | 0



1 0 1 | 0
.
0 1 2 | 0

4. LINGUISTIC DIFFICULTIES: ALGEBRA VS. GEOMETRY

67

Thus c is an independent variable and a = c and b = 2c. We just want a

particular solution so take c = 1, a = 1, and b = 2. The subspace we seek
is given by the equation
x 2y + z = 0.
The fact that c can be anything is just saying that this plane can also be
given by non-zero multiples of the equation x 2y + z = 0. For example,
2x 4y + 2z = 0 is another equation for the same plane. And so on.
4. Linguistic difficulties: algebra vs. geometry
A midterm question asked students to find all points on the plane
x1 2x2 + 3x3 4x4 = 0
x1 x2 x3 + x4 = 0
in R4 . I was surprised how many students found this a hard problem. But
I suspect they could have answered the question find all solutions to the
system of equations
x1 2x2 + 3x3 4x4 = 0
x1 x2 x3 + x4 = 0.
The two problems are identical! The students problem was linguistic not
mathematical. The first problem sounds geometric, the second algebraic. A
first course on linear algebra contains many new words and ideas. To understand linear algebra you must understand the connections and relationship
between the new words and ideas. Part of this involves the interaction between algebra and geometry. This interaction is a fundamental aspect of
mathematics. You were introduced to it in high school. There you learned
that the solutions to an equation like y = 3x 2 are the points on a line
in R2 ; that the solutions to the equation 2x2 + 3y 2 = 4 are the points on
an ellipse ;and so on. Translating between the two questions at the start of
this chapter is the same kind of thing.
The reason students find the question of finding all solutions to the system of equations easier is that they have a recipe for finding the solutions:
find rref(A), find the independent variables, etc. Recipes are a very important part of linear algebraafter all we do want to solve problems and we
want to find methods to solve particular kinds of problems. But linear algebra involves ideas, and connections and interactions between those ideas. As
you understand these you will learn how to come up with new recipes, and
learn to recognize when a particular recipe can be used even if the problem
doesnt have the code words that suggest a particular recipe can be used.

CHAPTER 8

Linear dependence and independence

The notion of linear independence is central to the course from here on
out.
1. The definition
A set of vectors {v 1 , . . . , v n } is linearly independent if the only solution
to the equation
a1 v 1 + + an v n = 0
is a1 = = an = 0. We say {v 1 , . . . , v n } is linearly dependent if it is not
linearly independent.
In other words, {v 1 , . . . , v n } is linearly dependent if there are numbers
a1 , . . . , an R, not all zero, such that
a1 v 1 + + an v n = 0.
1.1. The standard basis for Rn is linearly independent. Because
a1 e1 + a2 e2 + + an en = (a1 , a2 , . . . , an )
this linear combination is zero if and only if a1 = a2 = = am = 0. The
ei s form the prototypical example of a linearly independent set.
1.2. Example. Any set containing the zero vector is linearly dependent
because
1.0 + 0v 1 + + 0.v n = 0.
2. Criteria for linear (in)dependence
I wont ask you to prove the next theorem on the exam but you will
learn a lot if you read and re-read its proof until you understand it.
Theorem 2.1. The non-zero rows of a matrix in row echelon form are
linearly independent.
Proof. The key point is that the left-most non-zero entries in each non-zero
row of a matrix in echelon form move to the right as we move downwards
69

70

8. LINEAR DEPENDENCE AND INDEPENDENCE

from row 1 to row 2 to row 3 to ... etc. We

looking at a single example. The matrix

1 2 3 4
0 1 2 3
E=
0 0 0 1
0 0 0 0

can get an idea of the proof by

5
4
2
0

6
5

3
0

is in echelon form. Its non-zero rows are

v 1 = (1, 2, 3, 4, 5, 6),
v 2 = (0, 1, 2, 3, 4, 5),
v 3 = (0, 0, 0, 1, 2, 3).
To show these are linearly independent we must show that the only solution
to the equation
a1 v 1 + a2 v 2 + a3 v 3 = 0
is a1 = a2 = a3 = 0. Suppose a1 v 1 + a2 v 2 + a3 v 3 = 0. The sum a1 v 1 +
a2 v 2 + a3 v 3 is equal to
(a1 , 2a1 , 3a1 , 4a1 , 5a1 , 6a1 ) + (0, a2 , 2a2 , 3a2 , 4a2 , 5a2 ) + (0, 0, 0, a3 , 2a3 , 3a3 ).
The left most entry of this sum is a1 so the fact that the sum is equal to
0 = (0, 0, 0, 0, 0, 0) implies that a1 = 0. It follows that a2 v 2 + a3 v 3 = 0.
But a2 v 2 + a3 v 3 = (0, a2 , . . .) (we dont care about what is in the positions . . .) so we must
have a2 = 0. Therefore a3 v3 = 0; but a3 v3 =
0 0 0 a3 2a3 3a3 so we conclude that a3 = 0 too. Hence {v 1 , v 2 , v 3 }
is linearly independent.

Theorem 2.2. Vectors v 1 , . . . , v n are linearly dependent if and only if
some v i is a linear combination of the other v j s.
Proof. () Suppose v 1 , . . . , v n are linearly dependent. Then there are
numbers a1 , . . . , an , not all zero, such that a1 v 1 + + an v n = 0. Suppose
ai 6= 0. Then
X
vi =
(a1
i aj )v j .
j6=i

() Suppose v i is a linear combination of the other v j s. Then there are

numbers bj such that
X
vi =
bj v j .
j6=i

Therefore
1.v i

bj v j = 0,

j6=i

thus showing that v 1 , . . . , v n are linearly dependent.

2. CRITERIA FOR LINEAR (IN)DEPENDENCE

71

2.1. Using Theorem 2.2. Theorem 2.2 is an excellent way to show

that a set of vectors is linearly dependent. For example, it tells us that the
vectors



0
1
1
1 , 3 , 4 ,
2
4
6
are linearly dependent because

1
0
1
4 = 1 + 3 .
6
2
4
Please understand why this shows these three vectors are linearly independent by re-writing it as



0
1
1
0
1 1 + 1 3 + (1) 4 = 0
2
4
6
0
and re-reading the definition of linear dependence.
2.2. Remark. The important word in the next theorem is the word
unique. It is this uniqueness that leads to the idea of coordinates which will
be taken up in chapter 10.
Theorem 2.3. Vectors v 1 , . . . , v n are linearly independent if and only
if each vector in their linear span can be written as a linear combination of
v 1 , . . . , v n in a unique way.
Proof. Since Sp(v 1 , . . . , v n ) is, by definition, the set of linear combinations of v 1 , . . . , v n we must prove that the linear independence condition is
equivalent to the uniqueness condition.
() Suppose v 1 , . . . , v n are linearly independent. Let w Sp(v 1 , . . . , v n ).
If w = a1 v 1 + + an v n and w = b1 v 1 + + bn v n , then
0=ww




= a1 v 1 + + an v n b1 v 1 + + bn v n
= (a1 b1 )v 1 + + (an bn )v n .
Because the set {v 1 , . . . , v n } is linearly independent,
a1 b1 = = an bn = 0.
Hence the two representations of w as a linear combination of v 1 , . . . , v n are
the same, i.e., there is a unique way of writing w as a linear combination of
those vectors.
() Suppose each element in Sp(v 1 , . . . , v n ) can be written as a linear
combination of v 1 , . . . , v n in a unique way. In particular, there is only one
way to write the zero vector as a linear combination. But 0 = 0v 1 + + 0v n
so if a1 v 1 + + an v n = 0, then a1 = 0, a2 = 0, . . . , an = 0. In other words,
the set {v 1 , . . . , v n } is linearly independent.


72

8. LINEAR DEPENDENCE AND INDEPENDENCE

Theorem 2.4. Every subset of Sp(v 1 , . . . , v d ) having d + 1 vectors is

linearly dependent.
Proof. Let r d + 1 and let w1 , . . . , wr be vectors in Sp(v 1 , . . . , v d ). For
each i = 1, . . . , r there are numbers ai1 , . . . , aid R such that
(2-1)

wi = ai1 v 1 + ai2 v 2 + aid v d .

The homogeneous system

a11 c1 + a21 c2 + + ar1 cr = 0
a12 c1 + a22 c2 + + ar2 cr = 0
..
..
.
.
a1d c1 + a2d c2 + + ard cr = 0
of d equations in the unknowns c1 , . . . , cr has more unknowns than equations
so has a non-trivial solution; i.e., there are numbers c1 , . . . , cr , not all equal
to zero, that satisfy this system of equations. It follows from this that
c1 w1 + + cr wr = (c1 a11 + c2 a21 + + cr ar1 )v 1 +
(c1 a12 + c2 a22 + + cr ar2 )v 2 +
..
.
(c1 a1d + c2 a2d + + cr ard )v d
= 0.
This shows that {w1 , . . . , wr } is linearly dependent.

Corollary 2.5. All bases for a subspace have the same number of elements.
Proof. Suppose that {v 1 , . . . , v d } and {w1 , . . . , wr } are bases for W . Then
W = Sp(v 1 , . . . , v d ) = Sp(w1 , . . . , wr )
and the sets {v 1 , . . . , v d } and {w1 , . . . , wr } are linearly independent.
By Theorem 2.4, a linearly independent subset of Sp(v 1 , . . . , v d ) has d
elements so r d. Likewise, by Theorem 2.4, a linearly independent subset
of Sp(w1 , . . . , wr ) has r elements so d r. Hence d = r.

The next result is a special case of Theorem 2.4 because Rn = Sp{e1 , . . . , en }.
Nevertheless, we give a different proof which again emphasizes the fact that
a homogeneous system of equations in which there are more unknowns than
equations has a non-trivial solution.
Theorem 2.6. If n > m, then every set of n vectors in Rm is linearly
dependent.
Proof. Lets label the vectors A1 , . . . , An and make them into the columns
of an m n matrix that we call A. Then Ax = x1 A1 + + xn An so
the equation Ax = 0 has a non-trivial solution (i.e., a solution with some

3. LINEAR (IN)DEPENDENCE AND SYSTEMS OF EQUATIONS

73

xi 6= 0) if and only if A1 , . . . , An are linearly dependent. But Ax = 0 has a

non-trivial solution because it consists of m equations in n unknowns and
n > m (see Proposition 9.2 and Corollary 9.3).

3. Linear (in)dependence and systems of equations
Theorem 3.1. The homogeneous system Ax = 0 has a non-trivial solution if and only if the columns A1 , . . . , An of A are linearly dependent.
Proof. The proof of Theorem 2.6 used the fact that Ax = x1 A1 + +xn An
to conclude that Ax = 0 has a non-trivial solution (i.e., a solution with some
xi 6= 0) if and only if A1 , . . . , An are linearly dependent.

The next result is obtained directly from the previous one it is the
contrapositive. By that we mean the following general principle of logic: if
a statement P is true if and only if a statement Q is true, then P is false if
and only if Q is false.
We apply this principle to Theorem 3.1 by noting that not having a
non-trivial solution is the same as having only the trivial solution (which
is the same as having a unique solution for a homogeneous system), and
not linearly dependent is the same as linearly independent.
Theorem 3.2. The homogeneous system Ax = 0 has a unique solution
if and only if the columns A1 , . . . , An of A are linearly independent.
The reason for restating Theorem 3.1 as Theorem 3.2 is to emphasize
the parallel between Theorem 3.2 and Theorem 3.3.
Theorem 3.3. The system Ax = b has a unique solution if and only if b
is a linear combination of the columns A1 , . . . , An of A and A1 , . . . , An are
linearly independent.
Proof. We have already proved that Ax = b has a solution if and only
if b is a linear combination of the columns of A; see Corollary 3.2 and the
discussion just before it.
() Suppose Ax = b has a unique solution. Because Ax = b has a
solution, Corollary 3.2 and the discussion just before it tell us that b is a
linear combination of the columns of A.
We will now use the fact that Ax = b has a unique solution to show that
the columns of A are linearly independent. Suppose c1 A1 + + cn An = 0;
in other words Ac = 0 where

c1
..
c = . .
cn
If u is a solution to Ax = b, so is u + c because
A(u + c) = Au + Ac = b + 0 = b.

74

8. LINEAR DEPENDENCE AND INDEPENDENCE

But Ax = b has a unique solution so u = u + c. It follows that c = 0. In

other words, the only solution to the equation c1 A1 + + cn An = 0 is c1 =
= cn = 0 which shows that the columns of A are linearly independent.
() Suppose that b is a linear combination of the columns A1 , . . . , An of
A and the columns are linearly independent. The first of these assumptions
tells us there are numbers c1 , . . . , cn R such that b = c1 A1 + + cn An ;
i.e., Ac = b where c is the n 1 matrix displayed above. If

d1
..
d= .
dn
were another solution to Ax = b, then b = Ad = d1 A1 + + dn An so
(c1 d1 )A1 + + (cn dn )An = Ac Ad = b b = 0.
However, A1 , . . . , An are linearly independent so c1 = d1 , . . . , cn = dn , i.e.,

c = d thereby showing that Ax = b has a unique solution.
Lemma 2.1 tells us that equivalent matrices have the same row space.
Proposition 3.4. Let A be any matrix. Then A and rref(A) have the
same row space, and the non-zero rows of rref(A) are linearly independent.
Proof. Let E = rref(A). The fact that A and E have the same row space
is a special case of Lemma 2.1. Suppose r1 , . . . , rt are the non-zero rows of
E. Suppose the left-most 1 in ri appears in the jith column. Then j1 < j2 <
. . . < jt .
Now suppose that a1 r1 + + at rt = 0.
The entry in column j1 of r1 is 1 and the entry in column j1 of all other
ri s is 0, so the entry in column j1 of a1 r1 + + at rt is a1 . Hence a1 = 0.
The entry in column j2 of r2 is 1 and the entry in column j2 of all other
ri s is 0, so the entry in column j2 of a1 r1 + + at rt is a2 . Hence a2 = 0.
Carry on in this way to deduce that each ai is zero. We will then
have shown that the only solution to the equation a1 r1 + + at rt = 0 is
a1 = a2 = = at = 0. Hence {r1 , . . . , rt } is linearly independent.

After we have defined the word basis you will see that Proposition 3.4
says that the non-zero rows of rref(A) are a basis for the row space of A.
If one is interested in the column space of a matrix A rather than its
row space, one can replace A by AT and perform elementary row operations
on AT . The transposes of the rows of rref(AT ) span the column space of A.
We will see later that they form a basis for the column space of A.

CHAPTER 9

Non-singular matrices, invertible matrices, and

inverses
The adjectives singular, non-singular, and invertible, apply to square
matrices and square matrices only.
1. Singular and non-singular matrices
An n n matrix A is called non-singular if the only solution to the
equation Ax = 0 is x = 0. We say A is singular if Ax = 0 has a non-trivial
solution.
When we say the following are equivalent in the next theorem what we
mean is that if one of the three statements is true so are the others. It
also means that if one of the three statements is false so are the others.
Therefore to prove the theorem we must do the following: assume one of
the statements is true and then show that assumption forces the other two
to be true. In the proof below we show that (1) is true if and only if (2) is
true. Then we show that (3) is true if (2) is. Finally we show (1) is true if
(3) is. Putting all those implications together proves that the truth of any
one of the statements implies the truth of the other two.
Theorem 1.1. Let A be an n n matrix. The following are equivalent:
(1) A is non-singular;
(2) {A1 , . . . , An } is linearly independent.
(3) Ax = b has a unique solution for all b Rn .
Proof. (1) (2) This is exactly what Theorem 3.2 says.
(2) (3) Suppose that (2) is true.
Let b Rn . By Theorem 2.6, the n + 1 vectors b, A1 , . . . , An in Rn are
linearly dependent so there are numbers a1 , . . . , an , an+1 , not all zero, such
that
a0 b + a1 A1 + + an An = 0.
I claim that a0 can not be zero.
Suppose the claim is false. Then a1 A1 + + an An = 0, But A1 , . . . , An
are linearly independent so a1 = = an = 0. Thus if a0 = 0, then all the
ai s are zero. That contradicts the fact (see the previous pragraph) that not
all the ai s are zero. Since the assumption that a0 = 0 leads to a contradiction
we conclude that a0 must be non-zero.
75

76

9. NON-SINGULAR MATRICES, INVERTIBLE MATRICES, AND INVERSES

Hence
1
b = (a1
0 a1 )A1 + + (a0 an )An .

Thus b is a linear combination of A1 , . . . , An . By hypothesis, A1 , . . . , An

linearly independent. Hence Theorem 3.3 tells us there is a unique solution
to Ax = b.
(3) (1) Assume (3) is true. Then, in particular, the equation Ax = 0
has a unique solution; i.e., A is non-singular, so (1) is true.

2. Inverses
Mankind used whole numbers, integers, before using fractions. Fractions
were created because they are useful in the solution of equations. For example, the equation 3x = 7 has no solution if we require x to be an integer but
it does have a solution if we allow x to be a fraction, that solution being 37 .
One could take the sophisticated (too sophisticated?) view that this
solution was found by multiplying both sides of the equation by the inverse
of 3. Explicitly, 3x is equal to 7 if and only if 13 3x equals 13 7. Now use
the associative law for multiplication and perform the complicated equation
1

1
3x =
3 x=1x=x
3
3
so the previous sentence now reads 3x is equal to 7 if and only if x equals
1
3 7. Presto! we have solved the equation.
Sometimes one encounters a matrix equation in which one can perform
the same trick. Suppose the equation is Ax = b and suppose you are in the
fortunate position of knowing there is a matrix E such that EA = I, the
identity matrix. Then Ax = b implies that E Ax = E b. Thus, using
the associative law for multiplication Ax = b implies
Eb = E b = E Ax = (E A) x = I x = x.
Presto! We have found a solution, namely x = Eb. We found this by the
same process as that in the previous paragraph.
Lets look at an explicit example. Suppose we want to solve

   
1 2
x1
1
(2-1)
=
.
3 4
x2
5
I know that the matrix
E=

1
2


4 2
3 1

has the property that EA = I. You should check that. The analysis in the
previous paragraph tells me that a solution to the equation (2-1) is
 
 

   
1 4 2
x1
1
1
7
=E
=
=
.
x2
5
5
4
2 3 1

2. INVERSES

77

Lemma 2.1. Let A be an n n matrix and I the n n identity matrix.

If there are matrices E and E 0 such that
EA = AE 0 = I
then E = E 0 .
Proof. This is a single calculation:
E 0 = IE 0 = (EA)E 0 = E(AE 0 ) = EI = E.
All we used was the associative law.
The inverse of an n n matrix A is the matrix


A1

with the property

AA1 = I = A1 A,
provided it exists. If A has an inverse we say that A is invertible.
2.1. Uniqueness of A1 . Our definition says that the inverse is the
matrix . . .. The significance of Lemma 2.1 is that it tells us there is at
most one matrix, E say, with the property that AE = EA = I. Check you
understand that!
2.2. Not every square matrix has an inverse. The phrase provided
it exists suggests that not every n n matrix has an inversethat is correct.
For example, the zero matrix does not have an inverse. We will also see
examples of non-zero square matrices that do not have an inverse.
2.3. The derivative of a function f (x) at a R is defined to be
f (a + h) f (a)
h0
h
provided this limit exists. The use of the phrase provided it exists in
the definition of the inverse of a matrix plays the same role as the phrase
provided this limit exists plays in the definition of the derivative.
lim

2.4. The inverse or an inverse. You should compare our definition

of inverse with that in the book. The book says an n n matrix is invertible
if there is a matrix A1 such that AA1 = A1 A = I and calls such A1 an
inverse of A (if it exists). Lemma 2.1 tells us there is at most one inverse
so we can immediately speak of the inverse (provided it exists). You might
want to look at the definition of the inverse in several books to gain a finer
appreciation of this fine point.
Here is the definiton from the second edition of Linear Algebra by M.
ONan:
If A is an m m matrix and there is an m m matrix B
such that AB = BA = I, A is said to be invertible and
B is said to be the inverse of A.
Having made this definition, the first natural question
to ask is whether a matrix A can have two different inverses. Theorem 2 shows this is impossible.

78

9. NON-SINGULAR MATRICES, INVERTIBLE MATRICES, AND INVERSES

Theorem 2 Let A be an m m matrix with inverse

B. If C is another matrix such that AC = CA = I, then
C = B.
3. Elementary matrices are invertible
As we will now explain, the fact that elementary matrices are invertible
is essentially the same as the fact that if a matrix B can be obtained from
a matrix A by performing a single elementary row operation, then A can be
obtained from B by performing a single elementary row operation
First recall that elementary matrices are, by definition, those matrices
obtained by performing a single elementary row operation on an identity
matrix.
Let A be an m n matrix. By Theorem 11.1, the result of performing
an elementary row operation on A produces a matrix that is equal to EA
for a suitable m m elementary matrix E. Since E is obtained from Im
by an elementary row operation, Im can be obtained from E by performing
an elementary row operation. Therefore Im = E 0 E for some elementary
matrix E 0 . If you check each of the three elementary row operations you
will see that the transpose of an elementary matrix is an elementary matrix.
Hence E T is an elementary matrix and by the argument just given, there is
an elementary matrix F such that F E T = Im . Hence EF T = Im = E 0 E.
By Lemma 2.1, the fact that EF T = I = E 0 E implies F T = E 0 . Hence
E 0 E = EE 0 = I and E 0 is the inverse of E.
4. Making use of inverses
4.1. If A has an inverse and AB = C, then B = A1 C. To see
this just multiply both sides of the equality AB = C on the left by A1 to
obtain A1 C = A1 (AB) = (A1 A)B = IB = B.
4.2. Solving Ax = b when A is invertible. The only solution to this
equation is x = A1 b. This is a special case of the previous remark. In
chapter 2 we put this remark into practice for an explicit example.
4.3. Wicked notation. Never, never, never write B/A to denote BA1 .
4.4. If AB = 0 and B 6= 0, then A is not invertible. If A is invertible and B is a non-zero matrix such that AB = 0, then B = IB =
(A1 A)B = A1 (AB) = 0 so contradicting the fact that B is non-zero. We
are therefore forced to conclude that if AB = 0 for some non-zero matrix
B, then A is not invertible. Likewise, if BA = 0 for some non-zero matrix
B, then A is not invertible.
This is a useful observation because it immediately gives a large source
of matrices that are not invertible. For example, the matrix


a b
0 0

5. THE INVERSE OF A 2 2 MATRIX

79

does not have an inverse because



 

a b
w x

=
;
0 0
y z
0 0
in particular, the product can never be the identity matrix because the
22-entry is zero regardless of what w, x, y, z are.
4.5. We can cancel invertible matrices. Let A be an n n matrix.
If AB = AC, then B = C because AB = AC implies A1 (AB) = A1 (AC)
and the associative law then allows us to replace A(A1 by I to obtain
IB = IC, i.e., B = C. I prefer the argument
B = IB = (A1 A)B = A1 (AB) = A1 (AC) = (A1 A)C = IC = C.
A similar argument shows that if A is invertible and EA = F A, then
E = F.
Be warned though that you cannot cancel in an expression like AB = CA
because matrices do not necessarily commute with each other.
4.6. The inverse of a product. If A and B are n n matrices have
inverses, then AB is invertible and its inverse is B 1 A1 because
(AB)(B 1 A1 ) = A(BB 1 )A1 = AIA1 = AA1 = I
and similarly (B 1 A1 )(AB) = I.
A popular error is to claim that the inverse of AB is A1 B 1 . An
analogy might be helpful: the inverse to the operation put on socks then
shoes is take off shoes then socks. The inverse to a composition of operations
is the composition of the inverses of the operations in the reverse order.
5. The inverse of a 2 2 matrix
There is a simple arithmetic criterion for deciding if a given 2 2 matrix
has an inverse.
Theorem 5.1. The matrix


a b
A=
c d
has an inverse if and only if ad bc 6= 0. If ad bc 6= 0, then


1
d b
1
A =
ad bc c a
Proof. (=) Suppose A has an inverse. Just prior to this theorem, in
chapter 4.4, we observed that


a b
0 0
does not have an inverse, so either c or d must non-zero. Therefore
   
 

 


0
d
d
a b
d
ad bc
6=
= A1 A
= A1
= A1
0
c
c
c d
c
0

80

9. NON-SINGULAR MATRICES, INVERTIBLE MATRICES, AND INVERSES

so we conclude that ad bc 6= 0.
(=) If ad bc 6= 0 a simple calculation shows that the claimed inverse
is indeed the inverse.

The number ad bc is called the determinant of


a b
.
c d
We usually write det(A) to denote the determinant of a matrix. Theorem 5.1
therefore says that a 2 2 matrix A has an inverse if and only if det(A) 6= 0.
In chapter 12 we define the determinant for n n matrices and a prove
that an n n matrix A has an inverse if and only if det(A) 6= 0. That
is a satisfying result. Although the formula for the determinant appears
complicated at first it is straightforward to compute, even for large n, if one
has a computer handy. It is striking that the invertibility of a matrix can be
determined by a single computation even if doing the computation by hand
can be long and prone to error.
Theorem 5.2. Let A be an n n matrix. Then A has an inverse if and
only if it is non-singular.
Proof. () Suppose A has an inverse. If Ax = 0, then
x = Ix = (A1 A)x = A1 (Ax) = A1 .0 = 0
so A is non-singular.
() Suppose A is non-singular. Then Ax = b has a unique solution for
all b Rn by Theorem 1.1. Let
e1 = (1, 0, . . . , 0)T ,

e2 = (0, 1, 0, . . . , 0), . . . , en = (0, . . . , 0, 1)

Rn

and define uj
to be the unique column vector such that Auj = ej . Let
U = [u1 , . . . , un ] be the n n matrix whose columns are u1 , . . . , un . Then
AU = [Au1 , . . . , Aun ] = [e1 , . . . , en ] = I.
We will show that U is the inverse of A but we do not compute U A to do
this!
First we note that U is non-singular because if U x = 0, then
x = Ix = (AU )x = A(U x) = A.0 = 0.
Because U is non-singular we may apply the argument in the previous paragraph to deduce the existence of a matrix V such that U V = I. (The
previous paragraph showed that if a matrix is non-singular one can multiply
it on the right by some matrix to obtain the identity matrix.) Now
V = IV = (AU )V = A(U V ) = AI = A
so AU = U A = I, proving that U = A1 .

We isolate an important part of the proof. After finding U and showing

AU = I we were not allowed to say U is the inverse of A and stop the proof
because the definition of the inverse (read it carefully, now) says that we

6. IF A IS NON-SINGULAR HOW DO WE FIND A1 ?

81

need to show that U A = I too before we are allowed to say U is the inverse
of A. It is not obvious that AU = I implies U A = I because there is no
obvious reason for A and U to commute. That is why we need the last
paragraph of the proof.
Corollary 5.3. Let A be an n n matrix. If AU = I, then U A = I,
i.e., if AU = I, then U is the inverse of A.
Corollary 5.4. Let A be an n n matrix. If V A = I, then AV = I,
i.e., if V A = I, then V is the inverse of A.
Proof. Apply Corollary 5.3 with V playing the role that A played in Corollary 5.3 and A playing the role that U played in Corollary 5.3.

Theorem 5.5. The following conditions on an nn matrix A are equivalent:
(1) A is non-singular;
(2) A is invertible;
(3) rank(A) = n;
(4) rref(A) = I, the identity matrix;
(5) Ax = b has a unique solution for all b Rn ;
(6) the columns of A are linearly independent;
6. If A is non-singular how do we find A1 ?
We use the idea in the proof of Theorem 5.2:
the j th column of A1 is the solution to the equation
Ax = ej so we may form the augmented matrix (A | ej )
and perform row operations to get A in row-reduced echelon form.
We can carry out this procedure in one great big military operation. If we
want to find the inverse of the matrix

1 2 1
A = 3 0 1
1 1 1
we perform elementary row operations

1 2 1 |
3 0 1 |
1 1 1 |

on the augmented matrix

1 0 0
0 1 0
0 0 1

so we get the identity matrix on the left hand side. The matrix on the
right-hand side is then A1 . Check it out!

CHAPTER 10

Bases, coordinates, and dimension

1. Definitions
A basis for a subspace W is a linearly independent subset of W that
spans it.
The dimension of a subspace is the number of elements in a basis for it.
To make this idea effective we must show every subspace has a basis and
that all bases for a subspace have the same number of elements.
Before doing that, let us take note of the fact that {e1 , . . . , en } is a
basis for Rn , the standard basis. We made the trivial observation that these
vectors span Rn in section 3, and the equally trivial observation that these
vectors are linearly independent in chapter 8. Combining these two facts we
see that {e1 , . . . , en } is a basis for Rn . Therefore
dim Rn = n.
Strictly speaking, we cant say this until we have proved that every basis for
Rn has exactly n elements but we will prove that in Corollary 2.5 below.
Theorem 1.1. Suppose v 1 , . . . , v d is a basis for W . If w W , then
there are unique scalars a1 , . . . , ad such that w = a1 v 1 + + ad v d .
Proof. This theorem is a reformulation of Theorem 2.3.

If v 1 , . . . , v d is a basis for W and w = a1 v 1 + +ad v d we call (a1 , . . . , ad )

the coordinates of w with respect to the basis v 1 , . . . , v d . In this situation we
can think of (a1 , . . . , ad ) as a point in Rd .
2. A subspace of dimension d is just like Rd
The function
T : Rd W,

T (a1 , . . . , ad ) := a1 v 1 + + ad v d

is bijective (meaning injective and surjective, i.e., one-to-one and onto). It

also satisfies the following properties:
(1) T (0) = 0;
(2) if x, y Rd , then T (x + y) = T (x) + T (y);
(3) if x Rd and R, then T (x) = T (x).
These properties are all special cases of the following property:
(2-1)

T (1 x1 + + s xs ) = 1 T (x1 ) + + s T (xs )
83

84

10. BASES, COORDINATES, AND DIMENSION

for all x1 , . . . , xs Rd and all 1 , . . . , s R.

The function T is an example of a linear transformation.
It is the existence of the bijective function T having these properties that
justifies my repeated remark that a d-dimensional subspace looks exactly like
Rd . That is the way you should think of a d-dimensional subspace of Rn . It
is a copy of Rd sitting inside Rn . For example, all planes in R3 are copies of
R2 sitting inside R3 . Likewise, all lines in R3 are copies of R1 = R sitting
inside R3 .
3. All bases for W have the same number of elements
4. Every subspace has a basis
Our next goal, Theorem 4.2, is to show that every subspace has a basis.
First we need a lemma.
By Theorem 2.6, every subset of Rn having n + 1 elements is linearly
dependent. The lemma isolates the idea of building up a basis for a subspace
one element at a time, i.e., by getting larger and larger linearly independent
subsets of the subspace, a process that must eventually stop.
Lemma 4.1. Let {w1 , . . . , wr } be a linearly independent subset of Rn .
Let v be an element in Rn that does not belong to Sp(w1 , . . . , wr ). Then
{v} {w1 , . . . , wr } is linearly independent.
Proof. Suppose
v + a1 w1 + + an wn = 0
for some , a1 , . . . , an R. If is non-zero, then
v = (1 a1 )w1 + + (1 an )wn
which contradicts the hypothesis that v is not in Sp(w1 , . . . , wr ). Hence =
0. Now a1 w1 + +an wn = 0. Because {w1 , . . . , wr } is linearly independent
a1 = = an = 0. Hence {v, w1 , . . . , wr } is linearly independent.

Theorem 4.2. Every subspace of Rn has a basis.
Proof. Let W be a subspace of Rn . We adopt the standard convention
that the empty set is a basis for the subspace {0} so we may assume that
W 6= 0. This ensures there is an element w W such that {w} is linearly
independent.
Pick d as large as possible such that W contains a linearly independent
subset having d elements. There is a largest such d because every subset of
Rn having n + 1 elements is linearly dependent (Theorem 2.6). Suppose
{w1 , . . . , wd } is a linearly independent subset of W . It is a basis for W if
it spans W . However, if Sp(w1 , . . . , wd ) 6= W there is a v in W that is not
in Sp(w1 , . . . , wd ). By Lemma 4.1, {w1 , . . . , wd , v} is linearly independent,
thus contradicting the maximality of d. We conclude that no such v can
exist; i.e., Sp(w1 , . . . , wd ) = W . Hence {w1 , . . . , wd } is a basis for W .


5. PROPERTIES OF BASES AND SPANNING SETS

85

5. Properties of bases and spanning sets

The following is a useful way to think of d = dim W ; W has a linearly independent set having d elements and every subset of W having > d elements
is linearly dependent.
Corollary 5.1. Let W be a subspace of Rn . Then dim W is the largest
number d such that W contains d linearly independent vectors.
Proof. This is a consequence of the proof of Theorem 4.2.

Corollary 5.2. Let V and W be subspaces of Rn and suppose V W .

Then
(1) dim V < dim W if and only if V 6= W ;
(2) dim V = dim W if and only if V = W .
Proof. Let {v 1 , . . . , v c } be a basis for V and {w1 , . . . , wd } a basis for W .
Since {v 1 , . . . , v c } is a linearly independent subset of V , and hence of W ,
c d.
(1) () Suppose c < d. Since all bases for W have d elements, {v 1 , . . . , v c }
cant be a basis for W . Hence V 6= W .
() Suppose V 6= W . Then there is an element w W that is not in
V , i.e., not in span{v 1 , . . . , v c }. By Lemma 4.1, {v 1 , . . . , v c , w} is linearly
independent. It is also a subset of W and contains c+1 elements so c+1 d.
In particular, c < d.
(2) This is the contrapositive of (1).

Corollary 5.3. Let W = span{v 1 , . . . , v t }. Some subset of {v 1 , . . . , v t }
is a basis for W .
Proof. If {v 1 , . . . , v t } is linearly independent it is a basis for W and there
is nothing more to prove.
If {v 1 , . . . , v t } is linearly dependent, some v i is a linear combination of
the other v j s so we can remove that v i without changing the linear span
and so get a set of t 1 vectors that spans W . By repeating this process
we eventually get a linearly independent subset of {v 1 , . . . , v t } that spans
W and is therefore a basis for W .

Theorem 5.4. Let W be a d-dimensional subspace of Rn and let S =
{w1 , . . . , wp } be a subset of W .
(1) If p d + 1, then S is linearly dependent;
(2) If p < d, then S does not span W .
(3) If p = d, the following are equivalent:
(a) S is linearly independent;
(b) S spans W ;
(c) S is a basis for W .

86

10. BASES, COORDINATES, AND DIMENSION

Proof. Let V be the subspace spanned by S. Because S W , V W . By

Corollary 5.3, some subset of S is a basis for V . Hence dim V p.
(1) is a restatement of Corollary 5.1.
(2) If p < d, then dim V < dim W so V 6= W .
(3) Certainly, (c) implies both (a) and (b).
(a) (b) If S is linearly independent it is a basis for V so dim V = p =
d = dim W . Hence V = W .
(b) (c) If S spans W , then V = W so dim V = dim W

6. How to find a basis for a subspace
When a subspace is described as the linear span of an explicit set of
vectors, say W = Sp(w1 , . . . , wr ), the given vectors are typically linearly
dependent so are not a basis for W . One can find a basis for W in the
following way:
if the wj s are row vectors form the matrix A with rows w1 , . . . , wr ;
Sp(w1 , . . . , wr ) is equal to the row space of A;
if the wj s are column vectors form the matrix A with the rows
wT1 , . . . , wTr ;
Sp(wT1 , . . . , wTr ) is equal to the row space of A;
perform elementary row operations on A to obtain a matrix E in
echelon form (in row-reduced echelon form if you want, though it
isnt necessary for this);
by Theorem 2.1, the non-zero rows of E are linearly independent;
the non-zero rows of E are therefore a basis for the row space of E;
by Lemma 2.1, A and E have the same row space;
the non-zero rows of E are therefore a basis for the row space of A;
if the wj s are row vectors the non-zero rows of E are a basis
for Sp(w1 , . . . , wr );
if the wj s are column vectors the transposes of the non-zero
rows of E are a basis for Sp(w1 , . . . , wr ).
7. How to find a basis for the range of a matrix
Proposition 1.5 proved that the range of a matrix is the linear span of
its columns. A basis for R(A) is therefore a basis for the column space of
A. Take the transpose of A and compute rref(AT ). The non-zero rows of
rref(AT ) are a basis for the row space of AT . The transposes of the non-zero
rows in rref(AT ) are therefore a basis for the column space, and hence the
range, of A.
Try an example!
8. Rank + Nullity
The nullity of a matrix A is the dimension of its null space.

8. RANK + NULLITY

87

As a consequence of the following important result we shall see that the

dimension of the range of A is equal to the rank of A (Corollary 8.2). In
many books the rank of a matrix is defined to be the dimension of its range.
Accordingly, the formula in the next result is often stated as
nullityA + rank(A) = n.
Theorem 8.1. Let A be an m n matrix. Then
dim N (A) + dim R(A) = n.
Proof. Pick bases {u1 , . . . , up } for N (A) and {w1 , . . . , wq } for R(A). Remember that N (A) Rn and R(A) Rm .
There are vectors v 1 , . . . , v q in Rn such that wi = Av i for all i. We will
show that the set
B := {u1 , . . . , up , v 1 , . . . , v q }
is a basis for Rn .
Let x Rn . Then Ax = a1 w1 + + aq wq for some a1 , . . . , aq R. Now


A x a1 v 1 aq v q = Ax a1 Av 1 aq Av q = 0
so x a1 v 1 aq v q is in the null space of A and therefore
x a1 v 1 aq v q = b1 u1 + . . . + bp up
for some b1 , . . . , bp R. In particular,
x = a1 v 1 + + aq v q + b1 u1 + . . . + bp up
Rn .

so B spans
We will now show that B is linearly independent. If
c1 v 1 + + cq v q + d1 u1 + . . . + dp up = 0,
then


0 = A c1 v 1 + + cq v q + d1 u1 + . . . + dp up
= c1 Av 1 + + cq Av q + d1 Au1 + . . . + dp Aup
= c1 w1 + + cq wq + 0 + . . . + 0.
But {w1 , . . . , wq } is linearly independent so
c1 = = cq = 0.
Therefore
d1 u1 + . . . + dp up = 0.
But {u1 , . . . , up } is linearly independent so
d1 = = dp = 0.
Hence B is linearly independent, and therefore a basis for Rn . But B has
p + q elements so p + q = n, as required.

We now give the long-promised new interpretation of rank(A).

88

10. BASES, COORDINATES, AND DIMENSION

Corollary 8.2. Let A be an m n matrix. Then

rank(A) = dim R(A).
Proof. In chapter 7, we defined rank(A) to be the number of non-zero rows
in rref(A), the row-reduced echelon form of A. From the discussion in that
chapter we obtain
n rank(A) = the number of independent variables
for the equation Ax = 0
= dim{x Rn | Ax = 0}
= dim N (A).
But dim N (A) = n dim R(A) by Theorem 8.1, so rank(A) = dim R(A). 
Corollary 8.3. Let A be an m n matrix. Then
nullityA + rank(A) = n.
Corollary 8.4. The row space and column spaces of a matrix have the
same dimension.
Proof. Let A be an m n matrix. By Proposition 1.5, the column space of
A is equal to R(A). The row space of A is equal to the row space of rref(A) so
the dimension of the row space of A is rank(A). But rank(A) = dim R(A).
The result follows.

Corollary 8.5. Let A be an m n matrix. Then
rank(A) = rank(AT ).
Proof. The rank of AT is equal to the dimension of its row space which
is equal to the dimension of the column space of A which is equal to the
dimension of the row space of A which is equal to the rank of A.

We will now give some examples that show how to use Theorem 8.1.
8.1. 3 3 examples. There are four possibilities for the rank and nullity of a 3 3 matrix.

0 1 1
0 0 1
0 0 0

0 0 1
0 0 0
0 0 0

1 1 1
0 0 0
0 0 0

8. RANK + NULLITY

89

8.2. Planes in R4 . Let V and V 0 be two dimensional subspaces of R4 .

The intersection of V and V 0 is a subspace of R4 of dimension 2 so their
are three possibilities: the intersection is 2-dimensional if and only if V = V 0 ;
if the intersection is 1-dimensional, then V and V 0 have a line in common;
if V V 0 = {0}, then V + V 0 = R4 .
Let P and P 0 be planes in R4 , i.e., translations of 2-dimensional subspaces. We say that P and P 0 are parallel if they are translates of the same
2-dimensional subspace. In that case, P P 0 = but that is not the only
reason two planes can fail to meet.
8.3. 3-planes in R5 . We call a 3-dimensional subspace of Rn a 3-plane.
Let V and V 0 be 3-planes
8.4. Three formulas. There are many examples in mathematics where
the same idea pops up in very different settings. Here is an example:
If U and V are subspaces of Rn , then
dim(U + V ) = dim U + dim V dim(U V )
If A and B are finite sets, then
|A B| = |A| + |B| |A B|.
Let A and B denote events; let A B denote the event either A or
B occurs, and let A B denote the event both A and B occur. If
P [event] denotes the probability of an event, then
P [A B] = P [A] + P [B] P [A B].
Lemma 8.6. Let W and W 0 be subspaces of Rn and suppose that W
If w1 , . . . , wm is a basis for W , then there is a basis for W 0 that contains
w1 , . . . , wm .
W 0.

Proof. We argue by induction on dim W 0 dim W . Suppose dim W 0 =

1+dim W . Let w0 W 0 W . Then {w0 , w1 , . . . , wm } is linearly independent,
and hence a basis for W 0 . etc.

Proposition 8.7. If U and V are subspaces of Rn , then
dim(U + V ) = dim U + dim V dim(U V )
Proof. Let B be a basis for U V . Then there is a basis B t BU for U and
a basis B t BV for V . We will now show that B t BU t BV is a basis for
U +V.
Hence
dim(U + V ) = |B| + |BU | + |BV |.
But |B| = dim U V , |B| + |BU | = dim U , and |B| + |BV | = dim V , so
|B| + |BU | + |BV | = dim U + dim V dim U V
thus proving the result.

90

10. BASES, COORDINATES, AND DIMENSION

9. How to compute the null space and range of a matrix

Suppose you are given a matrix A and asked to compute its null space
and range. What would constitute an answer to that question? One answer
would be to provide a basis for its null space and a basis for its range.
By definition, the null space of A is the set of solutions to the equation
Ax = 0 so we may use the methods in chapters 7 and 8 to find the solutions.
Do elementary row operations to find rref(A). Write E for rref(A). Since
the equations Ax = 0 and Ex = 0 have the same solutions,
N (A) = N (E).
It is easy to write down the null space for E. For example, suppose If

1 0 2 0 0 3
4
0 1 3 0 0 4
0

0 0 0 1 0 2 1

(9-1)
E =
0 0 0 0 1 1 2

0 0 0 0 0 0
0
0 0 0 0 0 0
0
the dependent variables are x1 , x2 , x4 , and x5 . The independent variables
x3 , x6 , and x7 , may take any values and then the dependent variables are
determined by the equations
x1
x2
x4
x5

= 2x3 3x6 4x7

= 3x3 4x6
=
2x6 + x7
=
x6 + 2x7 .

Now set any one of the independent variables to 1 and set the others to zero.
This gives solutions



2
3
4
3
4
0



1
0
0



0 , 2 , 1 .



0
1
2



0
1
0
0
0
1
These are elements of the null space and are obviously linearly independent.
They span the null space.
MORE TO SAY/DO...basis for null space

CHAPTER 11

Linear transformations
Linear transformations are special kinds of functions between vector
spaces. Subspaces of a vector space are of crucial importance and linear
transformations are precisely those functions between vector spaces that
send subspaces to subspaces. More precisely, if T : V W is a linear
transformation and U is a subspace of V , then
{T (u) | u U }
is a subspace of W . That subspace is often called the image of U under T .
1. A reminder on functions
Let X and Y be any sets. A function f from X to Y , often denoted
f : X Y , is a rule that associates to each element x X a single element
f (x) that belongs to Y . It might be helpful to think of f as a machine: we
put an element of X into f and f spits out an element of Y . We call f (x)
the image of x under f . We call X the domain of f and Y the codomain.
The range of f is the subset R(f ) := {f (x) | x X} Y . The range of
f might or might not equal Y . For example, the functions f : R R and
g : R R0 given by the formulas f (x) = x2 and g(x) = x2 are different
functions even though they are defined by the same formula. The range g
is equal to the codomain of g. The range of f is not equal to the codomain
of f .
Part of you might rebel at this level of precision and view it as pedantry
for the sake of pedantry. However, as mathematics has developed it has
become clear that such precision is necessary and useful. A great deal of
mathematics involves formulas but those formulas are used to define functions and we add a new ingredient, the domain and codomain.
You have already met some of this. For example, the function f (x) = x1
is not defined at 0 so is not a function from R to R. It is a function from
R {0} to R. The range of f is not R but R {0}. Thus the function
f : R {0} R {0} defined by the formula f (x) = x1 has an inverse,
namely itself.
At first we learn that the number 4 has two square roots, 2 and 2,
but when we advance a little we define the square root function R0
R0 . This is because we want the function f (x) = x2 to have an inverse.
The desire to have inverses often makes it necessary to restrict the domain
or codomain. A good example is the inverse sine function. Look at the
91

92

11. LINEAR TRANSFORMATIONS

definition of sin1 () in your calculus book, or on the web, and notice how
much care is taken with the definition of the codomain.
2. First observations
Let V be a subspace of Rn and W a subspace of Rm . A linear transformation
from V to W is a function F : V W such that
(1) F (u + v) = F (u) + F (v) for all u, v V , and
(2) F (av) = aF (v) for all a R and all v V .
It is common to combine (1) and (2) into a single condition: F is a linear
transformation if and only if
F (au + bv) = aF (u) + bF (v)
for all a, b R and all u, v V .
2.1. Equality of linear transformations. Two linear transformations from V to W are equal if they take the same value at all points of V .
This is the same way we define equality of functions: f and g are equal if
they take the same values everywhere.
2.2. Linear transformations send 0 to 0. The only distinguished
point in a vector space is the zero vector, 0. It is nice that linear transformations send 0 to 0.
Lemma 2.1. If T : V W is a linear transformation, then T (0) = 0.1
Proof. The following calculation proves the lemma:
0 = T (0) T (0)
= T (0 + 0) T (0)
= T (0) + T (0) T (0)

by property (1)

= T (0) + 0
= T (0).

This proof is among the simplest in this course. Those averse to reading
proofs should look at each step in the proof, i.e., each = sign, and ask what
is the justification for that step. Ask me if you dont understand it.
2.3. The linear transformations R R.
1When we write T (0) = 0 we are using the same symbol 0 for two different things: the

0 in T (0) is the zero in V and the other 0 is the zero in W . We could write T (0V ) = 0W
but that looks a little cluttered. It is also unnecessary because T is a function from V
to W so the only elements we can put into T () are elements of V , so for T (0) to make
sense the 0 in T (0) must be the zero in V ; likewise, T spits out elements of W so T (0)
must be an element of W .

2. FIRST OBSERVATIONS

93

2.4. T respects linear combinations. Let T : V W be a linear

transformation. If v 1 , v 2 , v 3 V and a1 , a2 , a3 R, then
T (a1 v 1 + a2 v 2 + a3 v 3 ) = T ((a1 v 1 + a2 v 2 ) + a3 v 3 )
= T (a1 v 1 + a2 v 2 ) + a3 T (v 3 )
= a1 T (v 1 ) + a2 T (v 2 ) + a3 T (v 3 )
An induction argument shows that if v 1 , . . . , v p V and a1 , . . . , ap R,
then
(2-1)

T (a1 v 1 + + ap v p ) = a1 T (v 1 ) + + ap T (v p ).

2.5. T is determined by its action on a basis. There are two ways

to formalize the statement that a linear transformation is completely determined by its values on a basis.
Let T : V W be a linear transformation and suppose {v 1 , . . . , v p } is
a basis for V .
(1) If you know the values of T (v 1 ), . . . , T (v p ) you can compute its
value at every point of V because if v V there are unique numbers
a1 , . . . , ap R such that v = a1 v 1 + + ap v p and it now follows
from (2-1) that T (v) = a1 T (v 1 ) + + ap T (v p ).
(2) If S : V W is another linear transformation such that S(v i ) =
T (v i ) for i = 1, . . . , p, then S = T . To see this use (2-1) twice: once
for S and once for T .
2.6. How do we tell if a function T : V W is a linear transformation? As with all such questions, the answer is to check whether T
satisfies the definition or not.
PAUL - examples,
The function f : R2 R2 given by
(
(x + y, x y) if x 0 and y 0
f (x, y) =
(x y, x + y) otherwise
is not a linear transformation because
2.7. The zero transformation. The function V W that sends
every x V to the zero vector in W is a linear transformation (check!). We
call this the zero linear transformation and denote it by 0 if necessary.
2.8. The identity transformation. The function V V that sends
every x V to itself is a linear transformation (check!). We call it the
identity transformation of V and denote it by idV .

94

11. LINEAR TRANSFORMATIONS

2.9. A composition of linear transformations is a linear transformation. If S : V W and T : U V are linear transformations, their
composition, denoted by ST or S T , is defined in the same way as the
composition of any two functions, namely
(ST )(u) = S(T (u))

(S T )(u) = S(T (u)).

or

This makes sense because T (u) V and we can apply S to elements of V .

Thus
ST : U W.
To see that ST is really a linear combination suppose that u and u0 belong
to U and that a, a0 R. Then
(ST )(au + a0 u0 ) = S(T (au + a0 u0 ))

by definition of ST

= S(aT (u) + a0 T (u0 ))

0

because T is a linear transformation

0

= aS(T (u)) + a S(T (u ))

0

= a(ST )(u) + a (ST )(u )

because S is a linear transformation

by definition of ST

so ST is a linear transformation.
2.10. A linear combination of linear transformations is a linear
transformation. If S and T are linear transformations from U to V and
a, b R, we define the function aS + bT by
(aS + bT )(x) := aS(x) + bT (x).
It is easy to check that aS + bT is a linear transformation from U to V too.
We call it a linear combination of S and T . Of course the idea extends to
linear combinations of larger collections of linear transformations.
3. Linear transformations and matrices
Let A be an m n matrix and define T : Rn Rm by T (x) := Ax.
Then T is a linear transformation from Rn to Rm . The next theorem says
that every linear transformation from Rn to Rm is of this form.
Theorem 3.1. Let T : Rn Rm be a linear transformation. Then there
is a unique m n matrix A such that T (x) = Ax for all x Rn .
Proof. Let A be the m n matrix whose j th column is Aj := T (ej ). If
x = (x1 , . . . , xn )T is an arbitrary element of Rn , then
T (x) = T (x1 e1 + + xn en )
= x1 T (e1 ) + + xn T (en )
= x 1 A1 + + x n An
= Ax
as required.

In the context of the previous theorem, we sometimes say that the matrix
A represents the linear transformation T .

4. HOW TO FIND THE MATRIX REPRESENTING A LINEAR TRANSFORMATION 95

Suppose T : Rk Rm and S : Rm Rn are linear transformations and

A and B matrices such that T (x) = Ax for all x Rk and S(y) = By for
all y Rm , then (S T )(x) = BAx for all x Rk . Thus
multiplication of matrices corresponds to
composition of linear transformations
or
multiplication of matrices is defined so that the matrix
representing the composition of two linear transformations
is the product of the matrices representing each linear transformation.
This is of great importance. It explains why the multiplication of matrices
is defined as it is. That last sentence bears repeating. The fact that matrix
multiplication has real meaning, as opposed to something that was defined
by a committee that just wanted to be able to multiply matrices, is due
to the fact that matrix multiplication corresponds to that most natural of
operations, composition of functions, first do one, then the other.
Mathematics is not about inventing definitions just for the heck of it
and seeing what one can deduce from them. Every definition is motivated
by some problem, question, or naturally occurring object or feature. Linear
transformations and compositions of them are more fundamental than matrices and their product. Matrices are tools for doing calculations with linear
transformations. They sometimes convey information that is concealed in
other presentations of linear transformations.
We introduce you to matrices before linear transformations because they
are more concrete and down to earth, not because they are more important
than linear transformations.
3.1. Isomorphisms. Suppose that S : V U and T : U V are
linear transformations. Then ST : U U and T S : V V . If ST = idU
and T S = idV we say that S and T are inverses to one another. We also
say that S and T are isomorphisms and that U and V are isomorphic vector
spaces.
Theorem 3.2. Let V be a subspace of Rn . If dim V = d, then V is
isomorphic to Rd .
Proof. Let v 1 , . . . , v d be a basis for V . Define T : Rd V by
T (a1 , . . . , ad ) := a1 v 1 + + ad v d .
MORE TO DO
S : V Rd by

4. How to find the matrix representing a linear transformation

The solution to this problem can be seen from the proof of Theorem 3.1.
There, the matrix A such that T (x) = Ax for all x Rn is constructed one
column at a time: the ith column of A is T (ei ).

96

11. LINEAR TRANSFORMATIONS

PAULexplicit example.
5. Invertible matrices and invertible linear transformations
6. How to find the formula for a linear transformation
First, what do we mean by a formula for a linear transformation? We
mean the same thing as when we speak of a formula for one of the functions
you met in calculusthe only difference is the appearance of the formula.
For example, in calculus f (x) = x sin x is a formula for the function f ;
similarly, g(x) = |x| is a formula for the function g; we can also give the
formula for g by
(
x
if x 0
g(x) =
x if x 0.
The expression

2a b
a

T b =
a + b + c
c
3c 2b
is a formula for a linear transformation T : R3 R4 .
Although a linear transformation may not be given by an explicit formula, we might want one in order to do calculations. For example, rotation
about the origin in the clockwise direction by an angle of 45 is a linear
transformation, but that description is not a formula for it. A formula for
it is given by



 
 
a+b
1 1
a
a
1
1

= 2
.
T
= 2
ba
1 1
b
b
Let T : Rn Rm be a linear transformation. Suppose {v 1 , . . . , v n } is
a basis for Rn and you are told the values of T (v 1 ), . . . , T (v n ). There is a
simple procedure to find the m n matrix A such that T (x) = Ax for all
x Rn . The idea is buried in the proof of Theorem 3.1.
MORE TO DO
7. Rotations in the plane
Among the more important linear transformations R2 R2 are the
rotations. Given an angle we write T for the linear transformation that
rotates a point by an angle of radians in the counterclockwise direction.
Before we obtain a formula for T some things should be clear. First,
T T = T+ . Second, T0 = 0. Third, T2n = idR2 and T(2n+1) = idR2
for all n Z. Fourth, rotation in the counterclockwise direction by an angle
of is the same as rotation in the clockwise direction by an angle of so
T has an inverse, namely T .
Lets write A for the unique 2 2 matrix such that T x = A x for all
x R2 . The first column of A is given by T (e1 ) and the second column

8. REFLECTIONS IN R2

97

of A is given by T (e2 ). Elementary trigonometry (draw the diagrams and

check!) then gives


cos sin
(7-1)
A =
.
sin cos
There are many interesting aspects to this:
if you have forgotten your formulas for sin( + ) and cos( + )
you can recover them by using the fact that A A = A+ and
computing the product on the left;
the determinant of A is 1 because cos2 + sin2 = 1;
A1
= A and using the formula for the inverse of a 22 matrix in
Theorem 5.1 you can recover the formulae for sin() and cos()
if you have forgotten them.
8. Reflections in R2
 
a
Let L be the line in
through the origin and a non-zero point
.
b
 
a
to itself, i.e.,
The reflection in L is the linear transformation that sends
b
 
b
it fixes every point on L, and sends the point
, which is orthogonal to
a
 
b 2
L, to
. Let A be the matrix implementing this linear transformation.
a
Since
 
 !
1
a
b
a
b
e1 = 2
b
a
a + b2
R2

and

 
 !
1
a
b
e2 = 2
b
+a
2
b
a
a +b

we get
 
 !
 2

1
1
a
b
a b2
a
= 2
Ae1 = 2
b
b
a
2ab
a + b2
a + b2
and

 
 !


1
1
2ab
a
b
Ae2 = 2
b
+a
= 2
b
a
a + b2
a + b2 b2 a2

so

 2

1
a b2
2ab
2ab
b2 a2
a2 + b2
In summary, if x R2 , then Ax is the reflection of x in the line ay = bx.
A=

2Draw a picture to convince yourself that this does what you expect the reflection in

L to do.

98

11. LINEAR TRANSFORMATIONS

You might exercise your muscles by showing that all reflections are similar to one another. Why does it suffics to show each is similar to the
reflection


1 0
?
0 1
8.1. Reflections in higher dimensions. Let T : Rn Rn . We call
T a reflection if T 2 = I and the 1-eigenspace of T has dimension n 1.
9. Invariant subspaces
10. The one-to-one and onto properties
For a few moments forget linear algebra. The adjectives one-to-one and
onto can be used with any kinds of functions, not just those that arise in
linear algebra.
Let f : X Y be a function between any two sets X and Y .
We say that f is one-to-one if f (x) = f (x0 ) only when x = x0 .
Equivalently, f is one-to-one if it sends different elements of X to
different elements of Y , i.e., f (x) 6= f (x0 ) if x 6= x0 .
The range of f , which we denote by R(f ), is the set of all values it
takes, i.e.,
R(f ) := {f (x) | x X}.
I like this definition because it is short but some prefer the equivalent definition
R(f ) := {y Y | y = f (x) for some x X}.
We say that f is onto if R(f ) = Y . Equivalently, f is onto if every
element of Y is f (x) for some x X.
10.1. The sine function sin : R R is not one-to-one because sin =
sin 2. It is not onto because its range is [1, 1] = {y | 1 y 1}.
However, if we consider sin as a function from R to [1, 1] it is becomes
onto, though it is still not one-to-one.
The function f : R R given by f (x) = x2 is not one-to-one because
f (2) = f (2) for example, and is not onto because its range is R0 := {y
R | ly 0}. However, the function
f : R0 R0 ,

f (x) = x2 ,

is both one-to-one and onto.

11. Two-to-two
One-to-one is a lousy choice of name. Two-to-two would be much better
because a one-to-one function f : X Y is a function having the property
that it always sends two different elements x and x0 in X to two two different
elements f (x) and f (x0 ) in Y . Of course, once one sees this, it follows that a
one-to-one function is not only two-to-two but three-to-three, four-to-four,

12. GAZING INTO THE DISTANCE: DIFFERENTIAL OPERATORS AS LINEAR TRANSFORMATIONS

99

and so on; i.e., if f : X Y is one-to-one and x1 , x2 , x3 , x4 are four different

elements of X, then f (x1 ), f (x2 ), f (x3 ), f (x4 ) are four different elements of
X
11.1. We can and do apply these ideas in the context of linear algebra.
If T is a linear transformation we can ask whether T is one-to-one or onto.
For example, the linear transformation T : R2 R2 given by
   
x1
x2
T
=
x2
x1
is both one-to-one and onto.
The linear transformation T : R2 R3 given by

 
x2
x1
T
= x1
x2
x1 + x2
is one-to-one but not onto because (0, 0, 1)T is not in the range. Of course,
since the range of T is a plane in R3 there are lots of other elements of R3
not in the range of T .
Proposition 11.1. A linear transformation is one-to-one if and only if
its null space is zero.
Proof. Let T be a linear transformation. By definition, N (T ) = {x | T (x) =
0}. If T is one-to-one the only x with the property that T (x) = T (0) is x = 0,
so N (T ) = {0}.
Conversely, suppose that N (T ) = {0}. If T (x) = T (x0 ), then T (xx0 ) =
T (x) T (x0 ) = 0 so x x0 N (T ), whence x = x0 thus showing that T is
one-to-one.

Another way of saying this is that a linear transformation is one-to-one
if and only if its nullity is zero.
12. Gazing into the distance: differential operators as linear
transformations
You do not need to read this chapter. It is only for those who are curious
enough to raise their eyes from the road we have been traveling to pause,
refresh themselves, and gaze about in order to see more of the land we are
entering. Linear algebra is a big subject that permeates and provides a
framework for almost all areas of mathematics. So far we have seen only a
small piece of this vast land.
We mentioned earlier that the set R[x] of all polynomials in x is a vector
space with basis 1, x, x2 , . . .. Differentiation is a linear transformation
d
: R[x] R[x]
dx
because
d
df
dg
(af (x) + bg(x)) = a
+b .
dx
dx
dx

100

11. LINEAR TRANSFORMATIONS

If we write d/dx as a matrix with respect

0 1 0 0

d
0 0 2 0
= 0 0 0 3
dx
..
.

to the basis 1, x, x2 , . . ., then

0 0
0 0

0 0

..
.

d
(xn ) = nxn1 .
because dx
d
The null space of dx
is the set of constant functions, the subspace R.1.
d
The range of dx is all of R[x]: every polynomial is a derivative of another
polynomial.
Another linear transformation from R[x] to itself is multiplication by
x, i.e., T (f ) = xf . The composition of two linear transformations is a linear
d
to obtain another
transformation. In particular, we can compose x and dx
linear transformation. There are two compositions, one in each order. It is
interesting to compute the difference of these two compositions. Lets do
that
Write T : R[x] R[x] for the linear transformation T (f ) = xf and
D : R[x] R[x] for the linear transformation D(f ) = f 0 , the derivative
of f with respect to x. Then, applying the product rule to compute the
derivative of the product xf , we have

(DT T D)(f ) = D(xf ) xf 0 = f + xf 0 xf 0 = f.

Thus DT T D = I, the identity transformation.
MORE to say, Heisenbergs Uncertainty Principle.
12.1. Higher-order matrix equations. This course is all about solving equations in matrices. At high school you learned something about solving equations in which the solution is a number, i.e., a 1 1 matrix. In
this course you are learning about solving equations in which the solution
is a column vector, an n 1 matrix. An obvious next step is to consider
equations in which the solutions are matrices.
For example, at high school solving the two equations x2 + y 2 = 1 and
x y = 12 amounts to finding the two points where the circle meets the
line. The solutions are 1 2 matrices. However, you could also consider the
problem of finding all 2 2 matrices x and y such that x2 + y 2 = 1 and
x y = 12 .
MORE to say
Find all n n matrices E, F , H, such that
HE EH = 2E
EF F E = H
HF F H = 2F.
Because of the equivalence between matrices and linear transformations,
which we discussed in chapter 3, this problem is equivalent to finding all

12. GAZING INTO THE DISTANCE: DIFFERENTIAL OPERATORS AS LINEAR TRANSFORMATIONS

101

linear transformations E, F , and H, from Rn to itself that simultaneously

satisfy the three equations.
Let R[x, y]n denote the (n + 1)-dimensional vector space consisting of all
homogeneous polynomials in two variables having degree n. A basis for this
vector space is the set
xn , xn1 y, xn2 y 2 , . . . , xy n1 , y n .
Thus the elements of R[x, y]n consists of all linear combinations
an xn + an1 xn1 y + + a1 xy n1 + a0 y n .
Let x and y denote the partial derivatives with respect to x and y, respectively. These are linear transformations R[x, y]n R[x, y]n1 . We will
write fx for x (f ) and fy for y (f ). Multiplication by x and multiplication
by y are linear transformations R[x, y]n R[x, y]n+1 . Therefore
E := xy ,

H := xx yy ,

F := yx

are linear transformations R[x, y]n R[x, y]n for every n. The action of
EF F E on a polynomial f is


(EF F E)(f ) = xy yx yx xy (f )
= xy (yfx ) yx (xfy )
= x(fx + yfxy ) y(fy + xfyx )
= xfx + xyfxy yfy yxfyx
= xfx yfy


= xx yy (f )
= H(f ).
Since the transformations EF F E and H act in the same way on every
function f , they are equal; i.e., EF F E = H. We also have









(HE EH)(f ) = xx yy xy xy xx yy (f )


= xx yy (xfy ) xy )(xfx yfy )


= x(fy + xfyx ) yxfyy x2 fxy + x(fy + yfyy )
= 2xfy


= 2 xy (f )
= 2E(f ).
Therefore HE EH = 2E. I leave you to check that HF F H = 2F .
This example illuminates several topics we have discussed. The null
space of E is Rxn . The null space of F is Ry n . Each xni y i is an eigenvector
for H having eigenvalue n 2i. Thus H has n + 1 distinct eigenvalues
n, n2, n4, . . . , 2n, n and, as we prove in greater generality in Theorem
1.2, these eigenvectors are linearly independent.
MORE to say spectral lines....

CHAPTER 12

Determinants
The determinant of an n n matrix A, which is denoted det(A), is a
number computed from the entries of A having the wonderful property that:
A is invertible if and only if det(A) 6= 0.
We will prove this in Theorem 3.1 below.
Sometimes, for brevity, we denote the determinant of A by |A|.

1. The definition
We met the determinant of a 2 2 matrix in Theorem 5.1:


a b
det
= ad bc.
c d
The formula for the determinant of an n n matrix is more complicated. It is defined inductively by which I mean that the formula for the
determinant of an n n matrix involves the formula for the determinant of
an (n 1) (n 1) matrix.
1.1. The 3 3 case. The determinant det(A) = det(aij ) of a 3 3
matrix is the number
a11 a12 a13
a
a
a
a
a
a
a21 a22 a23 = a11 22 23 a12 21 23 + a13 21 22
a32 a33
a31 a33
a31 a32
a31 a32 a33
= a11 (a22 a33 a23 a32 ) a12 (a21 a33 a23 a31 ) + a13 (a21 a32 a22 a31 )
The coefficients a11 , a12 , and a13 , on the right-hand side of the formula are
the entries in the top row of A and they appear with alternating signs +1
and 1; each a1j is then multiplied by the determinant of the 2 2 matrix
obtained by deleting the row and column that contain a1j , i.e., deleting the
top row and the j th column of A.
In the second line of the expression for det(A) has 3 2 = 6 = 3!
terms. Each term is a product of three entries and those three entries are
distributed so that each row of A contains exactly one of them and each
column of A contains exactly one of them. In other words, each term is a
product a1p a2q a3r where {p, q, r} = {1, 2, 3}.
103

104

12. DETERMINANTS

1.2. The 4 4 case. The determinant det(A)

matrix is the number
a11 a12 a13 a14
a22 a23 a24
a21 a22 a23 a24
= a11 a32 a33 a34 a12
a31 a32 a33 a34
a42 a43 a44
a41 a42 a43 a44
+ a13

= det(aij ) of a 4 4
a21 a23 a24
a31 a33 a34
a41 a43 a44

a21 a22 a24

a21 a22 a23
a31 a32 a34 a14 a31 a32 a33 .
a41 a42 a44
a41 a42 a43

Look at the similarities with the 3 3 case. Each entry in the top row of A
appears as a coefficient with alternating signs +1 and 1 in the formula on
the right. Each a1j is then multiplied by the determinant of the 3 3 matrix
obtained by deleting the row and column that contain a1j , i.e., deleting the
top row and the j th column of A. Multiplying out the formula on the right
yields a total of 4 6 = 24 = 4! terms. Each term is a product of four
entries and those four entries are distributed so that each row of A contains
exactly one of them and each column of A contains one of them. In other
words, each term is a product a1p a2q a3r a4s where {p, q, r, s} = {1, 2, 3, 4}.
1.3. The n n case. Let A be an n n matrix. For each i and j
between 1 and n we define the (n 1) (n 1) matrix Aij to be that
obtained by deleting row i and column j from A. We then define
(1-1)

det(A) := a11 |A11 | a12 |A12 | + a13 |A13 | + (1)n1 a1n |A1n |.

Each |Aij | is called the ij th minor of A.

The expression (1-1) is said to be obtained by expanding A along its
first column. There are similar expressions for the determinant of A that
are obtained by expanding A along any row or column.
There are a number of observations to make about the formula (1-1).
When the right-hand side is multiplied out one gets a total of n! terms each
of which looks like
(1-2)

a1j1 a2j2 anjn

where {j1 , . . . , jn } = {1, . . . , n}. In particular, each row and each column
contains exactly one of the factors a1j1 , a2j2 , . . . , anjn .
Proposition 1.1. If A has a column of zeros, then det(A) = 0.
Proof. Suppose column j consists entirely if zeroes, i.e., a1j = a2j = =
anj = 0. Consider one of the n! terms in det(A), say a1j1 , a2j2 , . . . , anjn .
Because {j1 , . . . , jn } = {1, . . . , n}, one of the factors apjp in the product
a1j1 , a2j2 , . . . , anjn belongs to column j, so is zero; hence a1j1 , a2j2 , . . . , anjn =
0. Thus, every one of the n! terms in det(A) is zero. Hence det(A) = 0. 
A similar argument shows that det(A) = 0 if one of the rows of A consists
entirely of zeroes.

1. THE DEFINITION

105

By Theorem ??, A is not invertible if its columns are linearly dependent.

Certainly, if one of the columns of A consists entirely of zeroes its columns
are linearly dependent. As just explained, this implies that det(A) = 0.
This is consistent with the statement (not yet proved) at the beginning of
this chapter that A is invertible if and only if det(A) 6= 0.
1.4. The signs. The signs 1 are important. First lets just note that
the coefficient of a11 a22 . . . ann is +1. This can be proved by induction: we
have
a11 a12 . . . a1n
a22 . . . a2n
a21 a22 . . . a2n
.
.. + other terms
..
.. = a11 ..
.
.
.
an2 . . . ann
an1 an2 . . . ann
but a11 does not appear in any of the other terms. The coefficient of
a11 a22 . . . ann in det(A) is therefore the same as the coefficient of a22 . . . ann
in
a22 . . . a2n
..
..
.
.
an2 . . . ann
However, by an induction argument we can assume that the coefficient
of a22 . . . ann in this smaller determinant is +1. Hence the coefficient of
a11 a22 . . . ann in det(A) is +1.
In general, the sign in front of a1j1 a2j2 anjn is determined according
to the following recipe: it is +1 if one can change the sequence j1 , j2 , . . . , jn
to 1, 2, . . . , n by an even number of switches of adjacent numbers, and is -1
otherwise. For example, the coefficient of a12 a23 a34 a45 a51 in the expression
for the determinant of a 5 5 matrix is +1 because we need 4 switches
23451 23415 23145 21345 12345.
On the other hand, the coefficient of a13 a24 a35 a42 a51 is 1 because we need
7 switches
34521 34251 34215 32415 32145 23145 21345 12345.
1.5. The upper triangular case. An nn matrix A is upper triangular
if every entry below the main diagonal is zero; more precisely, if aij = 0 for
all i > j. The next result shows that it is easy to compute the determinant
of an upper triangular matrix: it is just the product of the diagonal entries.
Theorem 1.2.

a11 a12 a13

0 a22 a23

0 a33
det 0
..
..
.
.
0
0
0

..
.

a1n
a2n

a3n
= a11 a22 . . . ann .
..
.
a1n

106

12. DETERMINANTS

Proof. The determinant is a sum of terms of the form

a1j1 a2j2 anjn
where {j1 , . . . , jn } = {1, . . . , n}. For an upper triangular matrix this term
can only be non-zero if j1 1, and j2 2, and . . ., and jn n. But this
can only happen if j1 = 1, j2 = 2, . . ., and jn = n. Hence the only non-zero
term is a11 a22 . . . ann which, as we noted earlier, has coefficient +1.

Thus, for example,
det(I) = 1.
There is an analogue of Theorem 1.2 for lower triangular matrices.
2. Elementary row operations and determinants
Computing the determinant of a matrix directly from the definition is
tedious and prone to arithmetic error. However, given A we can perform
elementary row operations to get a far simpler matrix, say A0 , perhaps upper
triangular, compute det(A0 ) and, through repeated us of the next result, then
determine det(A).
The row reduced echelon form of a square matrix is always upper triangular.
Proposition 2.1. If B is obtained from A by the following elementary
row operation the determinants are related in the following way:
(2-1)
Elementary row operation
switch two rows
det(B) = det(A)
multiply a row by c R {0}
det(B) = c det(A)
replace row i by (row i + row k) with i 6= k
det(B) = det(A)
In particular, det(A) = 0 if and only if det(B) = 0.
Proof. The second of these statements is obvious. Suppose we multiply row
p by the number c. Each term a1j1 a2j2 anjn in det(A) contains exactly
one factor from row p so each term in det(B) is c times the corresponding
term in det(A). Hence det(B) is c times det(A).
We now prove the other two statements in the 2 2 case.
Switching two rows changes the sign because




c d
a b
det
= cb ad = (ad bc) = det
.
a b
c d
Adding one row to another doesnt change the sign because




a+c b+d
a b
det
= (a + c)b (b + d)c = ad bc = det
.
c
d
c d
and

(2-2) det




a
b
a b
= a(d + b) b(c + a) = ad bc = det
.
c+a d+b
c d

2. ELEMENTARY ROW OPERATIONS AND DETERMINANTS

107

Hence all three statements are true for a 2 2 matrix.

We now turn to the n n case and argue by induction on n, i.e., we
assume the result is true for (n 1) (n 1) matrices and prove the result
for the an n n matrix.
Actually, the induction argument is best understood by looking at how
the truth of the result for 2 2 matrices implies its truth for 3 3 matrices.
Consider switching rows 2 and 3 in a 3 3 matrix. Since
(2-3)

a11 a12 a13

a
a
a
a
a
a
a21 a22 a23 = a11 22 23 a12 21 23 + a13 21 22
a32 a33
a31 a33
a31 a32
a31 a32 a33

it follows that
(2-4)

a11 a12 a13

a
a
a
a
a
a
a31 a32 a33 = a11 32 33 a12 31 33 + a13 31 32 .
a22 a23
a21 a23
a21 a22
a21 a22 a23

But the determinants of the three 22 matrices in this sum are the negatives
of the three 2 2 determinants in (2-3). Hence the determinant of the 3 3
matrix in (2-4) is the negative of the determinant of the 3 3 matrix in
(2-3). This proves that det(B) = det(A) if B is obtained from A by
switching two rows.
Now consider replacing the third row of the 3 3 matrix A in (2-3) by
the sum of rows 2 and 3 to produce B. Then det(B) is
a11
a12
a13
a21
a22
a23
a21 + a31 a22 + a32 a23 + a33

= a11

a22
a23
a22 + a32 a23 + a33

a12

a21
a23
a21 + a31 a23 + a33

+ a13

a21
a22
.
a21 + a31 a22 + a32

However, by the calculation in (2-2), this is equal to

a11

a32 a33
a
a
a
a
a12 31 33 + a13 31 32
a22 a23
a21 a23
a21 a22

i.e., det(B) = det(A).

PAUL ... didnt cover the case of an ERO that involves row 1.

Proposition 2.2. If A is an n n matrix and E is an elementary

matrix, then det(EA) = det(E) det(A).
Proof.

Proposition 2.3. Let A and A0 be row equivalent n n matrices. Then

det(A) = 0 if and only if det(A0 ) = 0.

108

12. DETERMINANTS

Proof. By hypothesis, A0 is obtained from A by a sequence of elementary

row operations. By Proposition 2.1, a single elementary row operation does
not change whether the determinant of a matrix is zero or not. Therefore a sequence of elementary row operations does not change whether the
determinant of a matrix is zero or not.


3. The determinant and invertibility

Theorem 3.1. A square matrix is invertible if and only if its determinant is non-zero.
Proof. Suppose A is an n n matrix and let E = rref(A). Then A and E
have the same rank.
By Proposition 5.2, E = I if rank E = n, and E has a row of zeroes
if rank E < n. Since A is invertible if and only if rank(A) = n, it follows
that det(E) = 1 if A is invertible and det(E) = 0 if A is not invertible. The
result now follows from Proposition 2.3.


4. Properties
Theorem 4.1. det(AB) = (det A)(det B).
Proof. We break the proof into two separate cases.
Suppose A and B are invertible. Then A = E1 . . . Em and B = Em+1 . . . En
where each Ej is an elementary matrix. By repeated application of Proposition 2.2,
det(A) = det(E1 . . . Em )
= det(E1 ) det(E2 . . . Em )
=
= det(E1 ) det(E2 ) det(Em ).
Applying the same argument to det(B) and det(AB) we see that det(AB) =
(det A)(det B).
Suppose that either A or B is not invertible. Then AB is not invertible,
so det(AB) = 0 by Theorem 3.1. By the same theorem, either det(A) or
det(B) is zero. Hence det(AB) = (det A)(det B) in this case too.

Corollary 4.2. If A is an invertible matrix, then det(A1 ) = (det A)1 .
Proof. It is clear that the determinant of the identity matrix is 1, so the
result follows from Theorem 4.1 and the fact that AA1 = I.

det(A) = 0 if two rows (or columns) of A are the same.
det(AT ) = det(A).

5. ELEMENTARY COLUMN OPERATIONS AND DETERMINANTS

109

5. Elementary column operations and determinants

As you might imagine, one can define elementary column operations in
a way that is analogous to our definition of elementary row operations and
the effect on the determinant of those operations on the determinant of a
matrix parallels that on the effect of the elementary row operations.
5.1. Amusement. Here is an amusing use of the observation about the
effect of the elementary column operations on the determinant.
The numbers 132, 144, and 156, are all divisible by 12. Then the determinant of

1 3 2
1 4 4
1 5 6
is divisible by 12 because it is not changed by adding 100 times the 1st
column and 10 times the second column to the third column whence

1 3 2
1 3 132
1 3 11
det 1 4 4 = det 1 4 144 = 12 det 1 4 12 .
1 5 6
1 5 156
1 5 13
Similarly, the

2 8
det 3 9
8 6

numbers 289, 391,

9
2 8
1 = det 3 9
7
8 6

and 867, are all divisible

289
2 8
391 = 17 det 3 9
867
8 6

is divisible by 17.
Amuse your friends at parties.
5.2. A nice example. If the matrix

b c
a d
A=
d a
c b

d
c

b
a

has rank 2, then

(1) rref(A) has 2 non-zero rows so
(2) the row space of A has dimension 2 so
(3) any three rows of A are linearly dependent
(4) so det(any 3 rows of A) = 0.
For example,

b c
d
0 = det a d c
d a b
=b(db ac) c(ab cd) + d(a2 + d2 )
=d(a2 + b2 + c2 + d2 )

by 17 so

17
23 .
51

110

12. DETERMINANTS

Similarly, taking the determinant of the other three 3 3 submatrices of A

gives
a(a2 + b2 + c2 + d2 ) = b(a2 + b2 + c2 + d2 ) = c(a2 + b2 + c2 + d2 ) = 0.
If a, b, c, d are real numbers it follows that a = b = c = d = 0, i.e., A
must be the zero matrix. However, if a, b, c, d are complex numbers A need
not be zero because there are non-zero complex numbers a, b, c, d such that
a2 + b2 + c2 + d2 = 0.
5.3. Permutations. You have probably met the word permutation
meaning arrangement or rearrangement as in the question How many permutations are there of the numbers 1, 2, . . . , n? The answer, as you know,
is n!.
We now define a permutation of the set {1, 2, . . . , n} as a function
: {1, 2, . . . , n} {1, 2, . . . , n}
that is one-to-one (or, equivalently, onto). Thus each term in the expression
for the determinant of an n n matrix (aij ) is of the form
(5-1)

a1(1) a2(2) an(n) .

The condition that is one-to-one ensures that each row and each column
contains exactly one of the terms in (5-1).
There area total of n! permutations and the determinant of (aij ) is the
sum of n! terms (5-1), one for each permutation. However, there remains
the issue of the signs.
Let be a permutation of {1, 2, . . . , n}. Write (1), (2), . . . , (n) one
after the other. A transposition of this string of numbers is the string of
numbers obtained by switching the positions of any two adjacent numbers.
For example, if we start with 4321, then each of 3421, 4231, and 4312, is a
transposition of 4321. It is clear that after some number of transpositions
we can get the numbers back in their correct order. For example,
4321 4312 4132 1432 1423 1243 1234
or
4321 4231 2431 2341 2314 2134 1234.
If it takes an even number of transpositions to get the numbers back into
their correct order we call the permutation even. If it takes an odd number
we call the permutation odd. We define the sign of a permutation to be the
number
(
+1 if is even
sgn() :=
1 if is odd
It is not immediately clear that the sign of a permutation is well-defined. It
is, but we wont give a proof.

5. ELEMENTARY COLUMN OPERATIONS AND DETERMINANTS

111

The significance for us is that the coefficient of the term a1(1) a2(2) an(n)
in the expression for the determinant is sgn(). Hence
X
sgn() a1(1) a2(2) an(n)
det(aij ) =

where the sum is taken over all permutations of {1, 2, . . . , n}.

Thus, the term a14 a23 a32 a41 appears in the expression for the determinant of a 4 4 matrix with coefficient +1 because, as we saw above, it took
an even number of transpositions (six) to get 4321 back to the correct order.

CHAPTER 13

Eigenvalues
Consider a fixed n n matrix A. Viewed as a linear transformation
from Rn to Rn , A will send a line through the origin to either another line
through the origin or zero. Some lines might be sent back to themselves.
That is a nice situation: the effect of A on the points on that line is simply
to scale them by some factor, say. The line Rx is sent by A to either zero
or the line R(Ax), so A sends this line to itself if and only if Ax is a multiple
of x. This motivates the next definition.
1. Definitions and first steps
Let A be an n n matrix. A number R is an eigenvalue for A if
there is a non-zero vector x such that Ax = x. If is an eigenvalue for A
we define
E := {x | Ax = x}.
We call E the -eigenspace for A and the elements in E are called eigenvectors
or -eigenvectors for A.
We can define E for any R but it is non-zero if and only if is an
eigenvalue.1
Since
Ax = x 0 = Ax x = Ax Ix = (A I)x,
it follows that
E = N (A I).
We have therefore proved that E is a subspace of Rn . The following theorem
also follows from the observation that E = N (A I).
Theorem 1.1. Let A be an n n matrix. The following are equivalent:
(1) is an eigenvalue for A;
(2) A I is singular;
(3) det(A I) = 0.
Proof. Let R. Then is an eigenvalue for A if and only if there is a
non-zero vector v such that (A I)v = 0; but such a v exists if and only
1Eigen is a German word that has a range of meanings but the connotation here is
probably closest to peculiar to, or particular to, or characteristic of. The eigenvalues
of a matrix are important characteristics of it.
113

114

13. EIGENVALUES

if A I is singular. This proves the equivalence of (1) and (2). Theorem

3.1 tells us that (2) and (3) are equivalent statements.

Theorem 1.2. Let A be an n n matrix and 1 , . . . , r distinct eigenvalues for A. If v 1 , . . . , v r are non-zero vectors such that Av i = i v i for
each i, then {v 1 , . . . , v r } is linearly independent.
Proof. Suppose the theorem is false. Let m 1 be the smallest number
such that {v 1 , . . . , v m } is linearly independent but {v 1 , . . . , v m , v m+1 } is linearly dependent. Such an m exists because {v 1 } is a linearly independent
set.
There are numbers a1 , . . . , am such that v m+1 = a1 v 1 + + ar v m . It
follows that
m+1 a1 v 1 + + m+1 ar v m = m+1 v m+1
= Av m+1
= a1 Av 1 + + ar Av m
= a1 1 v 1 + + ar m v m .
Therefore
a1 (m+1 1 )v 1 + + ar (m+1 m )v m = 0.
But {v 1 , . . . , v m } is linearly independent so
a1 (m+1 1 ) = = ar (m+1 m ) = 0.
By hypothesis, m+1 6= i for any i = 1, . . . , m so
a1 = a2 = = am = 0.
But this implies v m+1 = 0 contradicting the hypothesis that v m+1 is nonzero. We therefore conclude that the theorem is true.

2. Reflections in R2 , revisited
A simple example to consider is reflection in a line lying in R2 , the
situation we considered
 in
 chapter 8. There we examined
  a pair of orthogonal
a
b
lines, L spanned by
, and L0 spanned by
. We determined the
b
a
matrix A that fixed the points
 onL and sent other points to their reflections
b
in L. In particular, A sent
to its negative. We found that
a
 2

1
a b2
2ab
.
A= 2
2ab
b2 a2
a + b2
 
 
a
b
The matrix A has eigenvectors
and
with eigenvalues, +1 and
b
a
1, and the corresponding eigenspaces are L and L0 .

3. THE 2 2 CASE

115

The point of this example is that the matrix A does not initially appear
to have any special geometric meaning but by finding its eigenvalues and
eigenspaces the geometric meaning of A becomes clear.
3. The 2 2 case


a b
Theorem 3.1. The eigenvalues of A =
are the solutions of the
c d
quadratic polynomial equation
(3-1)

2 (a + d) + ad bc = 0.

Proof. We have already noted that is an eigenvalue for A if and only if

A I is singular, i.e., if and only if det(A I) = 0. But


a
b
det(A I) = det
c
d
= (a )(d ) bc
= 2 (a + d) + ad bc,
so the result follows.

A quadratic polynomial with real coefficients can have either zero, one,
or two roots. It follows that a 2 2 matrix with real entries can have either
zero, one, or two eigenvalues. (Later we will see that an n n matrix with
real entries can have can have anywhere from zero to n eigenvalues.)
3.1. The polynomial x2 + 1 has no real zeroes, so the matrix


0 1
1 0
has no eigenvalues. You might be happy to observe that this is the matrix
that rotates the plane through an angle of 90 in the counterclockwise direction. It is clear that such a linear transformation will not send any line
to itself: rotating by 90 moves every line.
Of course, it is apparent that rotating by any angle that is not an integer
multiple of will have the same property: it will move every line, so will have
no eigenvalues. You should check this by using the formula for A in(7-1)
and then applying the quadratic formula to the equation (2x2.eigenvals).
3.2. The polynomial x2 4x + 4 has one zero, namely 2, because it is a
square, so the matrix


3 1
A=
1 1
has exactly one eigenvalue, namely 2. The 2-eigenspace is


 
1 1
1
E2 = N (A 2I) = N
=R
.
1 1
1

116

13. EIGENVALUES

 
 
1
1
is 2
. We know that dim E2 = 1
1
1
because the rank of A 2I is 1 and hence its nullity is 1.
You should check that A times

3.3. The polynomial x2 x 2 has two zeroes, namely 1 and 2, so the

matrix


2 2
B=
2
3
has two eigenvalues, 1 and 2. The 2-eigenspace is


 
4 2
1
E2 = N (B 2I) = N
=R
.
2
1
2
The (-1)-eigenspace is
E1



 
1 2
2
= N (B + I) = N
=R
.
2
4
1

3.4. The examples we have just done should not mislead

you into think1 2
ing that eigenvalues must be integers! For example, 1 1 has two eigenvalues, neither of which is an integer. It would be sensible to try computing
those and the associated eigenspaces.
4. The equation A2 + I
You already know there is no real number x such that x2 + 1 = 0. Since
a number is a 1 1 matrix it is not unreasonable to ask whether there is a
2 2 matrix A such that A2 + I = 0. I will leave you to tackle that question.
You can attack the question in a naive way by taking a 2 2 matrix


a b
c d
squaring it, setting the square equal to


1 0
0 1
and asking whether the system of four quadratic equations in a, b, c, d so
obtained have a solution. Not pretty, but why not try it and find out what
happens.
As a cute illustration of the use of eigenvalues I will now show there is
no 3 3 matrix A such that A2 + I = 0. I will prove this by contradiction;
i.e., I will assume A is a 3 3 matrix such that A2 + I = 0 and deduce
something false from that, thereby showing there can be no such A.
So, lets get started by assuming A is a 33 matrix such that A2 +I = 0.
As a preliminary step I will show that A has no eigenvalues. If R
were an eigenvalue and v a non-zero vector such that Av = v, then
v = Iv = A2 v = A(Av) = A(v) = Av = 2 v.
But v 6= 0 so the equality v = 2 v implies 1 = 2 . But this can not be
true: there is no real number whose square is 1 so I conclude that A can

5. THE CHARACTERISTIC POLYNOMIAL

117

not have an eigenvector. (Notice this argument did not use anything about
the size of the matrix A. If A is any square matrix with real entries such
that A2 + I = 0, then A does not have an eigenvalue. We should really say
A does not have a real eigenvalue.
With that preliminary step out of the way lets proceed to prove the
result we want. Let v be any non-zero vector in R3 and let V = Sp(v, Av).
Then V is an A-invariant subspace by which I mean if w V , then Aw
V . More colloquially, A sends elements of V to elements of V . Moreover,
dim V = 2 because the fact that A has no eigenvectors implies that Av is
not a multiple of v; i.e., {v, Av} is linearly independent.
Since V is a 2-dimensional subspace of R3 there is a non-zero vector w
R3 that is not in V . By the same argument, the subspace W := Sp(w, Aw)
/ V , V + W = R3 . From the
is A-invariant and has dimension 2. Since w
formula
dim(V + W ) = dim V + dim W dim(V W )
we deduce that dim(V W ) = 1. Since V and W are A-invariant, so is
V W . However, V W = Ru for some u and the fact that Ru is Ainvariant implies that Au is a scalar multiple of u; i.e., u is an eigenvector
for A. But that is false: we have already proved A has no eigenvectors.
Thus we are forced to conclude that there is no 3 3 matrix A such that
A2 + I = 0.
4.1. Why did I just prove that? A recent midterm contained the
following question: if A is a 33 matrix such that A3 +A = 0, is A invertible.
The answer is no, but the question is a very bad question because the
proof I had in mind is flawed. Let me explain my false reasoning: if
A3 + A = 0, then A(A2 + I) = 0 so A is not invertible because, as we
observed in chapter 4.4, if AB = 0 for some non-zero matrix B, then A
is not invertible. However, I can not apply the observation in chapter 4.4
unless I know that A2 + I is non-zero, and it is not easy to show that A2 + I
is not zero. The argument above is the simplest argument I know, and it
isnt so simple.
There is a shorter argument using determinants but at the time of the
midterm I had not introduced either determinants or eigenvalues.
5. The characteristic polynomial
The characteristic polynomial of an n n matrix A is the degree n polynomial
det(A tI).
You should convince yourself that det(A tI) really is a polynomial in t of
degree n. Lets denote that polynomial by p(t). If we substitute a number
for t, then
p() = det(A I).
The following result is an immediate consequence of this.

118

13. EIGENVALUES

Proposition 5.1. Let A be an n n matrix, and let R. Then is

an eigenvalue of A if and only if it is a zero of the characteristic polynomial
of A.
Proof. Theorem 1.1 says that is an eigenvalue for A if and only det(A
I) = 0.

A polynomial of degree n has at most n roots. The next result follows
from this fact.
Corollary 5.2. An n n matrix has at most n eigenvalues.


a b
The characteristic polynomial of
is
c d
t2 (a + d)t + ad bc.
6. How to find eigenvalues and eigenvectors
Finding eigenvalues and eigenvectors is often important. Here we will
discuss the head-on, blunt, full frontal method. Like breaking into a bank,
that is not always the most effective way, but it is certainly one thing to try.
But keep in mind that a given matrix might have some special features that
can be exploited to allow an easier method. Similarly, when breaking into a
bank, there might be some special features that might suggest an alternative
to the guns-ablazing method.
Let A be an n n matrix. By Proposition 5.1, is an eigenvalue if and
only if det(AI) = 0, i.e., if and only if is a root/zero of the characteristic
polynomial. So, the first step is to compute the characteristic polynomial.
Dont forget to use elementary row operations before computing det(A tI)
if it looks like that will helpremember, the more zeroes in the matrix the
easier it is to compute its determinant . The second step is to find the roots
of the characteristic polynomial.
Once you have found an eigenvalue, say, remember that the -eigenspace
is the null space of A I. The third step is to compute that null space,
i.e., find the solutions to the equation (A I)x = 0, by using the methods
developed earlier in these notes. For example, put A I in row-reduced
echelon form, find the independent variables and proceed from there.
6.1. An example. In order to

1
1
B :=
1
1

compute the eigenvalues of the matrix

1 1 1
1 1 1

1 1 1
1 1 1

we perform some elementary row operations on B tI before computing its

determinant: subtracting the bottom row of B tI from each of the other

6. HOW TO FIND EIGENVALUES AND EIGENVECTORS

rows does not

1t
1
det
1
1

119

change the determinant, so det(B tI) is equal to

1
1
1
2t
0
0
t2

1 t 1
1
2t
0
t 2
= det 0
.
0
1 1 t 1
0
2 t t 2
1
1 1 t
1
1
1 1 t

If we replace a row by c times that row we change the determinant by a

factor of c, so

1
0
0
1
0
1
0
1
.
det(B tI) = (2 t)3 det
0
0
1
1
1 1 1 1 t
The determinant does not change if
the last row so

0
det(B tI) = (2 t)3 det
0
0

we add each of the first three rows to

0
1
0
0

0
1
0
1
= (t 2)3 (t + 2).
1
1
0 2 t

The eigenvalues are 2. Now

1
1
E2 = N (B 2I) = N
1
1

1
1
1
1

1
1
1
1

1
1
.
1
1

The rank of B 2I, i.e., the dimension of the linear span of its columns, is 1,
and its nullity is therefore 3. Therefore dim E2 = 3. It is easy to find three
linearly independent vectors x such that (B 2I)x = 0 and so compute



1
1
1
1
0
0

E2 = R
0 + R 1 + R 0 .
0
0
1
Finally,

1
3 1 1 1
1 3 1 1
1

= N (B + 2I) = N
1 1 3 1 = R 1 .
1 1 1 3
1

E2

6.2. Another example. Let

1
1

A :=
1
1
1

1
1
1
1
1

1
1
1
0
1

1
1
1
1
0

1
1

1
.
1
1

120

13. EIGENVALUES

The characteristic polynomial of A is

x2 (x3 5x2 + 2x 1).
The x2 factor implies that the 0-eigenspace, i.e., the null space of A, has
dimension two. It is easy to see that


1
1
1
0

N (A) = R
0 + R 0 .
0
0
0
1
We dont need to know the zeroes of the cubic factor f (x) = x3 5x2 +2x1
just yet. However, a computation gives


f (1) = 3,
f (10) > 0,
f (0) = 1,
f 32 > 0,
so the graph y = f (x) crosses the x-axis at points in the open intervals




2
and
(1, 10).
0, 32 ,
3, 1 ,
Hence f (x) has three real zeroes, and A has three distinct non-zero eigenvalues. Because A has a 2-dimensional 0-eigenspace it follows that R5 has a
basis consisting of eigenvectors for A.
Claim: If is a non-zero eigenvalue for A, then

1 1
1 1

4 1 2

3
1
2 + 1
is a -eigenvector for A.
Proof of Claim: We must show that Ax = x. Since is an eigenvalue
it is a zero of the characteristic polynomial; since 6= 0, f () = 0, i.e.,
3 52 + 2 1 = 0. The calculation

1 2
1 1
1 1 1 1 1
1 1 1 1 1 1 1 1 2

2
2

Ax =
1 1 1 1 1 4 1 = 1 = x
3
1 1 0 1 1
3
1
1 1 1 0 1
2 + 2
2 + 1
shows that the Claim is true, however please, please, please, notice we used
the fact that is a solution to the equation x3 5x2 + 2x 1 = 0 in the
calculation in the following way: because 3 52 + 2 1 = 0 it follows
that 1 2 = (4 1 2 ), so the third entry in Ax is times the
third entry in x, thus justifying the final equality in the calculation.

Notice we did not need to compute the eigenvalues in order to find the
eigenspaces. If one wants more explicit information one needs to determine
the roots of the cubic equation x3 5x2 + 2x 1 = 0.

6. HOW TO FIND EIGENVALUES AND EIGENVECTORS

121

6.3. How did I find the eigenvector in the previous example?

That is the real problem! If someone hands you a matrix A and a vector v
and claims that v is an eigenvector one can test the truth of the claim by
computing Av and checking whether Av is a multiple of v.
To find the eigenvector I followed the proceedure described in the first
paragraph of section 6. By definition, the -eigenspace is the space of solutions to the homogeneous equation (A I)x = 0. To find those solutions
we must put

1
1
1
1
1
1
1
1
1
1

1
1
1
1
1

1
1
0
1
1
1
1
1
0
1
in row-reduced echelon form. To do this first subtract the top row from each
of the other rows to obtain

1 1
1
1
1

0
0
0

0 0
0

.

0 1 0

0
0 1
Now add the bottom row to the top row and subtract the bottom row from
each of the middle three rows, then move the bottom row to the top to get

0
0
1

1 1
1
0
1

0 0
.
1

0 0
1

0 0 1 1

Write = 1 (this is allowed

and fourth rows by to get

1 0
1 1

0 1

0 0
0 0

since 6= 0) and multiply the first, third,

1
1
0
1

1
.
1

1
1 1

Replace the second row by the second row minus the first row plus the third
row to get

1 0
0

1
0 0
1
2
3

0 1 0

.
0 0 1

1
0 0 1 1

122

13. EIGENVALUES

Now multiply the third row by 1, add the bottom row to the second row,
and subtract the bottom row from the fourth row, then multiply the bottom
row by 1, to get

1 0 0

1
0 0 0 2 + 1
3

1
B := 0 1 0
.
0 0 0 1 + 1
0 0 1
1 +

There is no need to do more calculation to get A in row-reduced echelon

form because it is quite easy to write down the solutions to the system
Bx = 0 which is equivalent to the system (A I)x = 0. Write x =
(x1 , x2 , x3 , x4 , x5 )T . From the second row of B we see that we should take
x4 = 3

and

x5 = 2 + 1 .

From the other rows of B we see that we must have

x1 = x4 + x5
= 3 + (2 + 1 )
x2 = x4 + x5
= 3 + (2 + 1 )
x3 = (1 )x4 + x5 = 3( 1) + (2 + 1 ).
Thus

1
1

2
x=
4 1 .

3
2 + 1
This is the matrix in the statement of the claim we made on page 120.
6.4. It aint easy. Before finding the eigenvector in the claim I made
many miscalculations, a sign here, a sign there, and pretty soon I was
doomed and had to start over. So, take heart: all you need is perseverance,
care, and courage, when dealing with a 5 5 matrix by hand. Fortunately,
there are computer packages to do this for you. The main thing is to understand the principle for finding the eigenvectors. The computer programs
simply implement the principle.
7. The utility of eigenvectors
Let A be an n n matrix, and suppose we are in the following good
situation:
Rn has a basis consisting of eigenvectors for A;
given v Rn we have an effective means of writing v as a linear
combination of the eigenvectors for A.
Then, as the next result says, it is easy to compute Av.

7. THE UTILITY OF EIGENVECTORS

123

Proposition 7.1. Suppose that A is an n n matrix such that Rn has a

basis consisting of eigenvectors for A. Explicitly, suppose that {v 1 , . . . , v n }
is a basis for Rn and that each v i is an eigenvector for A with eigenvalue
i . If v = a1 v 1 + + an v n , then
Av = a1 1 v 1 + + an n v n .

CHAPTER 14

Complex vector spaces and complex eigenvalues

1. The complex numbers
The set of complex numbers is denoted by C. I assume you have already
met complex numbers but will give a short refresher course in chapter 2
below.
The complex numbers were introduced to provide roots for polynomials
that might have no real roots. The simplest example is the polynomial
x2 + 1. It has no real roots because the square of a real number is never
equal to -1.
Complex numbers play a similar role in linear algebra: a matrix with
real entries might have no real eigenvalues but it will always have at least
one complex eigenvalue (unless it is a 1 1 matrix, i.e., a number.
For example, the matrix


0 1
1 0
representing counterclockwise rotation by 90 moves every 1-dimensional
subspace of R2 so has no eigenvectors and therefore no eigenvalues, at least
no real eigenvalues. I like that geometric argument but one can also see it
has no real eigenvalues because its characteristic polynomial x2 + 1 has no
real roots. However, x2 + 1 has two complex roots, i, and the matrix has
these as complex eigenvalues and has two corresponding eigenvectors in C2 .
Explicitly,

   
 
0 1
1
i
1
=
= i
1 0
i
1
i
and

   
 
0 1
1
i
1
=
=i
.
1 0
i
1
i
As this calculation suggests, all the matrix algebra and linear algebra
we have done so far extends with essentially no change from R to C. We
simply allow matrices to have entries that are complex numbers and instead
of working with Rn we work with Cn , n 1 column matrices, or vectors,
whose entries are complex numbers.
2. The complex numbers
This interlude is to remind you of some basic facts about complex numbers.
125

126

14. COMPLEX VECTOR SPACES AND COMPLEX EIGENVALUES

The set of complex numbers is an enlargement of the set of real numbers

created by adjoining a square root of -1, denoted by i, and then making the
minimal enlargement required to permit addition, subtraction and multiplication of all the numbers created by this process. Thus C contains R and
because we want to add and multiply complex numbers we define
C := {a + ib | a, b R}.
Notice that C contains R. The real numbers are just those complex numbers
a + ib for which b = 0. Of course we dont bother to write a + i0; we just
write a.
We often think of the elements of R as lying on a line. It is common to
think of C as a plane, a 2-dimensional vector space over R, the linear span
of 1 and i.
Every complex number can be written in a unique way as a + ib for
suitable real numbers a and b, We add by
(a + ib) + (c + id) = (a + c) + i(b + d).
and multiply by
(a + ib)(c + id) = ac + iad + ibc + i2 bd = (ac bd) + i(ad + bc).
If b = d = 0 we get the usual addition and multiplication of real numbers.
An important property enjoyed by both real and complex numbers is
that non-zero numbers have inverses. If a + ib 6= 0, then
1
a ib
=
.
a2 + b2
a + ib
You should check this by multiplying the left-hand term by a+ib and noting
that the answer is 1.
2.1. Reassurance. If you havent met the complex numbers before
dont be afraid. You have done something very similar before and survived
that so you will survive this. Once upon a time your knowledge of numbers
was limited to fractions, i.e., all you knew was Q. Imagine yourself in that
situation again. I now come to you saying I have a new number that I will
denote by the symbol 1 and I wish to enlarge the number system Q to a
new number system that I will denote by Q(1). I propose that in my new
number system the product of 1 with itself is 2, i.e., 12 = 2, and I further
propose that Q(1) consists of all expressions
a+b1
with the addition and multiplication in Q(1) defined by
(a + b 1) + (c + d 1) := (a + c) + (b + d) 1
and
(a + b 1)(c + d 1) := (ac + 2bd) + (ad + bc) 1 .
It might be a little tedious to check that the multiplication is associative,
that the distributive law holds, and that every non-zero number in Q(1) has

2. THE COMPLEX NUMBERS

127

an inverse that also belongs to Q(1). The number system Q(1) might also
seem a little mysterious until I tell you that a copyof Q(1) can already be
found inside R. In
fact, 1 is just a new symbol for 2. The number a + b 1
is really just a + b 2.
Now re-read the previous paragraph with this additional information.
2.2. The complex plane. As mentioned above, it is useful to picture
the set of real numbers as the points on a line, the real line, and equally
useful to picture the set of complex numbers as the points on a plane. The
horizontal axis is just the real line, sometimes called the real axis, and the
vertical axis, sometimes called the imaginary axis, is all real multiples of i.
We denote it by iR.
iR
O
a+ib

The
complex
plane C

1+2i

i1

1+i

2+i

/R

2.3. Complex conjugation. Let z = a + ib C. The conjugate of z,

denoted z, is defined to be
z := a ib.
You can picture complex conjugation as reflection in the real axis. The
important points are these
z z = a2 + b2 is a real number;
z z 0;
zz = 0 if and only if z = 0;
z z is the distance from zero to z;
wz = w
z;
w+z =w
+ z;
z = z if and only if z R.

It is common to call z z the absolute value of z and denote it by |z|. It is

also common to call z z the norm of z and denote it by ||z||.
Performing the conjugate twice gets us back where we started: z = z.
The geometric interpretation of this is that the conjugate of a complex

128

14. COMPLEX VECTOR SPACES AND COMPLEX EIGENVALUES

number is its relection in the real axis and z = z because reflecting twice in
the real axis is the same as doing nothing.
2.4. Roots of polynomials. Complex numbers were invented to provide solutions to quadratic polynomials that had no real solution. The
ur-example is x2 + 1 = 0 which has no solutions in R but solutions i in
C. This is the context in which you meet complex numbers at high school.
You make the observation there that the solutions of ax2 + bx + c = 0 are
given by

b b2 4ac
2a
provided a 6= 0 and the equation has no real solutions when b2 4ac < 0
but has two complex zeroes,

b
4ac b2
b
4ac b2

= +i
and
= i
.
2a
2a
2a
2a
are complex conjugates of each other.
Notice that the zeroes and
It is a remarkable fact that although the complex numbers were invented
with no larger purpose than providing solutions to quadratic polynomials,
they provide all solutions to polynomials of all degrees.
Theorem 2.1. Let f (x) = xn + a1 xn1 + a2 xn2 + + an1 x + an
be a polynomial with coefficients a1 , . . . , an belonging to R. Then there are
complex numbers r1 , . . . , rn such that
f (x) = (x r1 )(x r2 ) (x rn ).
Just as for quadratic polynomials, the zeroes of f come in conjugate pairs
because if f () = 0, then, using the properties of conjugation mentioned at
the end of chapter 2.3,
=
n + a1
n1 + a2
n2 + + an1
+ an
f ()
= n + a1 n1 + a2 n2 + + an1 + an
= n + a1 n1 + a2 n2 + + an1 + an
= f ()
= 0.
2.5. Application to the characteristic polynomial of a real matrix. Let A be an n n matrix whose entries are real numbers. Then
det(A tI) is a polynomial with real coefficients. If C is a zero of

det(A tI) so is .
Recall that the eigenvalues of A are exactly (the real numbers that are)
the zeroes of det(A tI). We wish to view the complex zeroes of det(A tI)
as eigenvalues of A but to do that we have to extend our framework for
linear algebra from Rn to Cn .

4. THE COMPLEX NORM

129

3. Linear algebra over C

We now allow matrices to have entries that are complex numbers. Because R is contained in C linear algebra over C includes linear algebra over
R, i.e., every matrix with real entries is a matrix with complex entries by
virtue of the fact that every real number is a complex number.
Hence, left multiplication by an m n matrix with real entries can be
viewed as a linear transformation Cn Cm in addition to being viewed as
a linear transformation Rn Rm .
We extend the operation of complex conjugation to matrices. If A =
(aij ) is a matrix with entries aij C, we define the conjugate of A to be
A := (aij ),
i.e., the matrix obtained by replacing every entry of A by its complex conjugate.
and A + B = A + B.

It is straightforward to verify that AB = AB

Proposition 3.1. Let A be an n n matrix with real entries. If C
Furthermore, if x is a -eigenvector for A,
is an eigenvalue for A so is .

then x
is a eigenvector for A.
Proof. This is a simple calculation. If Ax = x, then
x
x = Ax = x =
A
x = A
as claimed.

The first half of Proposition 3.1 is a consequence of the fact that if is

a zero of a polynomial with real coefficients so is .

3.1. The conjugate transpose. For complex matrices, i.e., entries
having complex entries, one often combines the operations of taking the
transpose and taking the conjugate. Notice that these operations commute,
i.e., it doesnt matter which order one performs them: in symbols

|
A = A| .
We often write

A := A
and call this the conjugate transpose of A.

4. The complex norm

Let z C. The norm of z is the number

||z|| = z z.
If z 6= 0,
z 1 = z/||z||.
Check this.
We can extend the notion of norm, or size, from complex numbers to
vectors in Cn .

130

14. COMPLEX VECTOR SPACES AND COMPLEX EIGENVALUES

One important difference between complex and real linear algebra involves the definition of the norm of a vector. Actually, this change already
begins when dealing with complex numbers. The distance of a real number x from zero is given by its absolute
value |x|. The distance of a point
p
x = (x1 , x2 ) R2 from zero is x21 + x22 . Picturing the complex plane as
R2 , the distance of z = a + ib from 0 is
p

a2 + b2 = z z.
If Rn , the distance of a point x = (x1 , . . . , xn )| from the origin is given
by taking the square root of its dot product with itself:
q

||x|| = x21 + + x2n = x x = x| x.

Part of the reason this works is that a sum of squares of real numbers is
always 0 and is equal to zero only when each of those numbers is zero.
This is no longer the case for complex numbers.
The norm of x Cn is
p
||x|| = x x.
Since
n
X
X

x x=
x
j xj =
|xj |2
j=1

j=1

is a real number 0 the definition of ||x|| makes senseit is the non-negative

square root of the real number x x.
4.1. Real symmetric matrices have real eigenvalues. The next
result is proved by a nice application of the complex norm.
Theorem 4.1. All the eigenvalues of a symmetric matrix with real entries are real.
Proof. Let A be real symmetric n n matrix and an eigenvalue for A.
Because is an
We will show is a real number by proving that = .
n
eigenvalue, there is a non-zero vector x C such that Ax = x. Then
)| x = (A
x)| x
||x||2 = x x = x x = x Ax = (A| x
where the last equality follows from the fact that A is symmetric. Continuing
this string of equalities, we obtain
x)| x = x
x = ||x||
2.
(A
x)| x = (
We have shown that
2
||x||2 = ||x||
because ||x||2 6= 0. Hence R.
from which we deduce that =

This is a clever proof. Lets see how to prove the 2 2 case in a less
clever way. Consider the symmetric matrix


a b
A=
b d

5. AN EXTENDED EXERCISE

131

where a, b, d are real numbers. The characteristic polynomial of A is

t2 (a + d)t + ad b2 = 0
and your high school quadratic formula tells you that the roots of this polynomial are

p
12 a + d (a + d)2 4(ad b2 )
However,
(a + d)2 4(ad b2 ) = (a d)2 + 4b2
p
which is 0 because it is a sum of squares. Hence (a + d)2 4(ad b2 )
is a real number and we conclude that the roots of the characteristic polynomial, i.e., the eigenvalues of A, are real.
It might be fun for you to try to prove the 3 3 case by a similar
elementary argument.
5. An extended exercise
Let

2 1 1 1
1 1 1 0

A=
1 0 1 1 .
1 1 0 1
Compute the characteristic polynomial of A and factor it as much as you can.
Remember to use elementary row operations before computing det(A xI).
Can you write the characteristic polynomial of A as a product of 4 linear
terms, i.e., as (x )(x )(x )(x ) with , , , C?
What are the eigenvalues and corresponding eigenspaces for A in C4 ?
Show that A6 = I and that no lower power of A is the identity.
Suppose that is a complex eigenvalue for A, i.e., there is a non-zero
vector x C4 such that Ax = x. Show that 6 = I.
Let = e2i/3 . Thus 3 = 1. Show that (t 1)(t )(t 2 ) = 0.
Show that
{1, , 2 } = { C | 6 = 1}.
Compute Av 1 , Av 2 , Av 3 , and Av 4 , where




0
0
1
3
1
1
1
1

v1 =
1 , v 2 = 1 , v 3 = , v 4 = 2 .
1
1
2

CHAPTER 15

Orthogonality
The word orthogonal is synonymous with perpendicular.
1. Geometry
In earlier chapters I often used the word geometry and spoke of the
importance of have a geometric view of Rn and the sets of solutions to
systems of linear equations. However, geometry really involves lengths and
angles which have been absent from our discussion so far. The word geometry
is derived from the Greek word geometrein which means to measure the land:
cf. or meaning earth and metron or o meaning an instrument
to measuremetronome, metre, geography, etc.
In order to introduce angles and length in the context of vector spaces
we use the dot product.
2. The dot product
The dot product of vectors u = (u1 , . . . , un ) and v = (v1 , . . . , vn ) in Rn
is the number
u v := u1 v1 + + un vn .
The norm or length of u is

||u|| := u u.
The justification for calling this the length is Pythagorass Theorem.
Notice that
u u = 0 if and only if u = 0this makes sense: the only vector
having length zero is the zero vector.
If u and v are column vectors of the same size, then u v = uT v =
v T u.
2.1. Angles. We define the angle between two non-zero vectors u and
v in Rn to be the unique angle [0, ] such that
uv
cos() :=
.
||u|| ||v||
Since cos( 2 ) = 0, the angle between u and v is 2 = 90 if and only if uv = 0.
Before accepting this definition as reasonable we should check that it
gives the correct answer for the plane.
Suppose u, v R2 . If u or v is a multiple of the other, then the angle
between them is zero and this agrees with the formula because cos(0) = 1.
133

134

15. ORTHOGONALITY

Now assume u 6= v. The cosine rule applied to the triangle with sides u, v,
and u v, gives
||u v||2 = ||u||2 + ||v||2 2||u|| ||v|| cos().
The left-hand side of this expression is ||u||2 2(u v) + ||v||2 . The result now
follows.
General case done by reducing to the R2 case using the fact that every
pair of vectors in Rn lies in a plane.
3. Orthogonal vectors
Two non-zero vectors u and v are said to be orthogonal if u v = 0.
The following result is a cute observation.
Proposition 3.1. Let A be a symmetric nn matrix and suppose and
are different eigenvalues of A. If u E and v E , then u is orthogonal
to v.
Proof. The calculation
uT v = (u)T v = (Au)T v = uT AT v = uT Av = uT v
shows that ( )uT v = 0. But 6= 0 so uT v = 0.

Let S be a non-empty subset of Rn . We define

S := {w Rn | w v = 0 for all v S},
i.e., S consists of those vectors that are perpendicular to every vector in S.
We call S the orthogonal to S and usually say S perp or S orthogonal
when we read S .
Rn .

Lemma 3.2. If S is a non-empty subset of Rn , then S is a subspace of

Proof. Certainly 0 S . If w1 and w2 are in S so is w1 + w2 because if

u S, then
(w1 + w2 ) u = w1 u + w2 u = 0 + 0 = 0.
If w S and R, then w S because if u S, then (w) u =
(w u) = 0 = 0.

Lemma 3.3. If V is a subspace of Rn , then V V = {0}.
Proof. If v is in V V , then v v = 0, whence v = 0.

5. ORTHOGONAL AND ORTHONORMAL BASES

135

4. Classical orthogonal polynomials

In chapter ?? we introduced the infinite dimensional vector space R[x]
consisting of all polynomials with real coefficients. You are used to writing
polynomials with respect to the basis 1, x, x2 , . . .. However, there are many
other bases some of which are extremely useful.
Suppose we define a dot product on R[x] by declaring that
Z 1
f (x)g(x)

f g :=
dx.
1 x2
1
It is easy to see that f g = g f , that f (g + h) = f g + f g, and
(f ) g = (f g) for all R.
It is common practice to call this dot product the inner product and write
it as (f, g) rather than f g. We will do this.
The Chebyshev polynomials are defined to be
T0 (x) := 1
T1 (x) := x
Tn (x) = 2xTn (x) Tn1 (x)

and

for all n 2.

Theorem 4.1. The set of Chebyshev polynomials are orthogonal to one

another with respect to the inner product

if m 6= n
Z 1
0
Tm (x)Tn (x)

dx =
if m = n = 0

1 x2
1

/2 if m = n 1
Proof. Use a change of variables x = cos() and use an induction argument
to prove that Tn (cos()) = cos(n).

5. Orthogonal and orthonormal bases
A set of non-zero vectors {v 1 , . . . , v d } is said to be orthogonal if v i is
orthogonal to v j whenever i 6= j. A set consisting of a single non-zero vector
is declared to be an orthogonal set.
Lemma 5.1. An orthogonal set of vectors is linearly independent.
Proof. Let {v 1 , . . . , v d } be an orthogonal set of vectors and suppose that
a1 v 1 + + ad v d = 0. Then
0 = v i (a1 v 1 + + ad v d ).
However, when we multiply this out all but one term will vanish because
v i v j = 0 when j 6= i. The non-vanishing term is ai v i , so ai v i = 0. But v i 6= 0,
so ai = 0. Thus all the ai s are zero and we conclude that {v 1 , . . . , v d } is
linearly independent.


136

15. ORTHOGONALITY

An orthogonal basis is just that, a basis consisting of vectors that are

orthogonal to one another. If in addition each vector has length one we call
it an orthonormal basis.
The standard basis {e1 , . . . , en } for Rn is an orthonormal basis.
It is usually a tedious chore to express an element w in a subspace W as
an explicit linear combination of a given basis {v 1 , . . . , v d } for W . To find the
numbers a1 , . . . , ad such that w = a1 v 1 + +ad v d usually involves solving a
system of linear equations. However, if {v 1 , . . . , v d } is an orthonormal basis
for W it is trivial to find the ai s. Thats one reason we like orthogonal bases.
Proposition 5.2. Suppose {v 1 , . . . , v d } is an orthonormal basis for W .
If w W , then
w = (w v 1 )v 1 + + (w v d )v d .
Proof. Notice first that each (w v i ) is a number!
We know there are numbers a1 , . . . , ad such that w = a1 v 1 + + ad v d .
It follows that
w v i = (a1 v 1 + + ad v d ) v i
= a1 (v 1 v i ) + + ad (v d v i ).
Because {v 1 , . . . , v d } is an orthonormal set of vectors v j v i = 0 when j 6= i
and v i v i = 1. Hence
wT v i = a1 (v 1 v i ) + + ad (v d v i ) = ai .
The result now follows.

That is the great virtue of orthonormal bases. Unfortunately, one is

not always fortunate enough to be presented with an orthonormal basis.
However, there is a standard mechanism, the Gram-Schmidt process, that
will take any basis for W and produce from it an orthonormal basis for W .
That is the subject of the next chapter.
6. The Gram-Schmidt process
Suppose B = {v 1 , . . . , v s } is a basis for a subspace W . The GramSchmidt process is a method for constructing from B an orthogonal basis
{u1 , . . . , us } for W .
Theorem 6.1. Every subspace of Rn has an orthogonal basis.
Proof. Let B = {v 1 , . . . , v s } be a basis for W . Define u1 , u2 , u3 , . . . by
u1 := v 1
u1 v 2
u
||u1 ||2 1
u v
u v
u3 := v 3 1 23 u1 2 23 u2
||u1 ||
||u2 ||
etc.
u2 := v 2

6. THE GRAM-SCHMIDT PROCESS

137

We will prove the theorem by showing that ui uj = 0 when i 6= j.

The first case is i = 2 and j = 1. We get
u v
u2 u1 = v 2 u1 1 22 u1 u1 .
||u1 ||
But u1 u1 = ||u1 ||2 so those factors cancel in the second term to leave
u2 u1 = v 2 u1 u1 v 2
which is zero because u v = v u. Hence u2 is orthogonal to u1 .
The next case is i = 3 and j = 1. We get
u v
u v
(6-1)
u3 u1 = v 3 u1 1 23 u1 u1 2 23 u2 u1 .
||u1 ||
||u2 ||
The right-most term in (6-1) is zero because u2 u1 = 0. Arguing as in the
previous paragraph shows that the two terms immediately to the right of
the = sign in (6-2) cancel so we get u3 u1 = 0.
Carry on like this to get ui u1 for all i 2.
The next case is to take i = 3 and j = 2. We get
u v
u v
(6-2)
u3 u2 = v 3 u2 1 23 u1 u2 2 23 u2 u2 .
||u1 ||
||u2 ||
The second term on the right-hand-side is zero because u1 u2 = 0; in the
third term, u2 u2 = ||u2 ||2 so those factors cancel to leave
u3 u2 = v 3 u2 u2 v 3
which is zero. Hence u3 is orthogonal to u2 .
Now do the case i = 4 and j = 2, and carry on like this to get ui u2 = 0
for all i 3. Then start again with i = 4 and j = 3, et cetera, et cetera!
Eventually, we prove that ui uj = 0 for all i 6= j.
We still need to prove that {u1 , . . . , us } is a basis for W . Because the
ui s are an orthogonal set they are linearly independent. Their linear span
is therefore a vector space of dimension s. By definition, each ui is in
Sp(v 1 , . . . , v s ) which equals W . Thus Sp(u1 , . . . , us ) is an s-dimensional
subspace of the s-dimensional vector space W , and therefore equal to it by
Corollary 5.2.1

Corollary 6.2. Every subspace of Rn has an orthonormal basis.
1Alternative proof that the u s span W . Rewrite the definition of u by putting v
i
i
i

on one side and so obtain

v i = ui + a linear combination of u1 , . . . , ui1 .
Hence v i Sp(u1 , . . . , us ). Hence Sp(v 1 , . . . , v s ) Sp(u1 , . . . , us ); but Sp(u1 , . . . , us )
Sp(v 1 , . . . , v s ) too so the two linear spans are equal. Since Sp(v 1 , . . . , v s ) = W we conclude
that Sp(u1 , . . . , us ) = W .

138

15. ORTHOGONALITY

Proof. Let W be a subspace of Rn . By Theorem 6.1, W has an orthogonal

basis. Let {u1 , . . . , us } be an orthogonal basis for W . If we divide each ui
by its length we obtain a set of orthogonal vectors
u1
u2
us
,
, ...,
||u1 || ||u2 ||
||us ||
each of which has length one; i.e., we obtain an orthonormal basis for W . 
Theorem 6.3. Let A be an n n symmetric matrix. Then Rn has an
orthonormal basis consisting of eigenvectors for A.
Proof. By Theorem 4.1, all As eigenvalues are real.

7. Orthogonal matrices
A matrix A is orthogonal if AT = A1 . By definition, an orthogonal
matrix has an inverse so must be a square matrix.
An orthogonal linear transformation is a linear transformation implemented
by an orthogonal matrix, i.e., if T (x) = Ax and A is an orthogonal matrix
we call T an orthogonal linear transformation.
Theorem 7.1. The following conditions on an nn matrix Q are equivalent:
(1) Q is an orthogonal matrix;
(2) Qx Qy = x y for all x, y Rn ;
(3) ||Qx|| = ||x|| for all x Rn ;
(4) the columns of Q, {Q1 , . . . , Qn }, form an orthonormal basis for Rn .
Proof. We start by pulling a rabbit out of a hat. Let x, y Rn . Then

1




2
2
1
= 4 x x + 2x y + y y 14 x x 2x y + y y
4 ||x + y|| ||x y||
= x y.


We will use the fact that 14 ||x + y||2 ||x y||2 later in the proof.
(1) (2) If Q is orthogonal, then QT Q = I so
(Qx) (Qy) = (Qx)T (Qy) = xT QT Qy = xT y = x y.
(2) (3) because (3) is a special case of (2), the case x = y.
(3) (4) By using the hypothesis in (3) and the rabbit we obtain


x y = 14 ||x + y||2 ||x y||2


= 14 ||Q(x + y)||2 ||Q(x y)||2


= 14 ||Qx + Qy||2 ||Qx Qy||2
= (Qx) (Qy).
(We have just shown that (3) implies (2).) In particular, if {e1 , . . . , en } is
the standard basis for Rn , then
ei ej = (Qei ) (Qej ) = Qi Qj .

7. ORTHOGONAL MATRICES

139

But ei ej = 0 if i 6= j and = 1 if i = j so the columns of Q form an

orthonormal set and therefore an orthonormal basis for Rn .
(4) (1) The ij th entry in QT Q is equal to the ith row of QT times the
ith column of Q, i.e., to QTi Qj = Qi Qj which is 1 when i = j and 0 when
i 6= j so QT Q = I.

7.1. Orthogonal linear transformations are of great practical importance

because they do not change the lengths of vectors or, as a consequence of
that, the angles between vectors. For this reason we often call the linear
transformations given by orthogonal matrices rigid motions. Moving a
rigid object, by which we mean some some kind of physical structure, a
submarine, a rocket, a spacecraft, or piece of machinery, in space does not
change the lengths of its pieces or the angles between them.
7.2. The orthogonal groups. You should check the following facts:
the identity matrix is orthogonal; a product of orthogonal matrices is an
orthogonal matrix; the inverse of an orthogonal matrix is orthogonal. The
set of all n n orthogonal matrices is called the n n orthogonal group
and is denoted by O(n). The orthogonal groups are fundamental objects in
mathematics.
7.3. Rigid motions in R2 . We have met two kinds of rigid motions
in R2 , rotations and reflections. Lets check that both are orthogonal transformations. If A denotes the matrix representing the rotation by an angle
in the counterclockwise direction, then


 

cos sin
cos sin
1 0
A (A )T =
=
.
sin cos
sin cos
0 1
Hence A is an orthogonal matrix.
On the other
hand, reflection in the line L through the origin and a
 
a
non-zero point
is given by the matrix
b
 2

1
a b2
2ab
A= 2
.
2ab
b2 a2
a + b2
This is an orthogonal linear transformation because
 4

1
a + 2a2 b2 + b4
0
T
2
A A=A = 2
= I.
0
a4 + 2a2 b2 + b4
(a + b2 )2
Notice an important difference between rotations and reflections: the
determinant of a rotation is +1 but the determinant of a reflection is 1.
Proposition 7.2. Every rigid motion in R2 is a composition of a rotation and a reflection.
Proof. Let A be an orthogonal 2 2 matrix.

140

15. ORTHOGONALITY

7.4. The standard basis vectors e1 , . . . , en are orthogonal to one another.

We said that an orthogonal matrix preserves the angles between vectors so,
if that is correct, an orthogonal matrix A will have the property that the
vectors Aej are orthogonal to one another. But Aej is the j th column of A.
Thus, the next result confirms what we have been saying.
Theorem 7.3. Let A be a symmetric matrix with real entries. Then
there is an orthogonal matrix Q such that Q1 AQ is diagonal.
Proof.

Theorem 7.4. Let A be a diagonal matrix with real entries. For every
orthogonal matrix Q, Q1 AQ is symmetric.
Proof. A matrix is symmetric if it is equal to its transpose. We have

T

T

T
Q1 AQ = QT AT Q1 = Q1 AT QT = Q1 AQ
so Q1 AQ is symmetric.

8. Orthogonal projections
Let V be a subspace of Rn . Let {u1 , . . . , uk } be an orthonormal basis
for V .
The function PV : Rn Rn defined by the formula
PV (v) := (v u1 )u1 + + (v uk )uk

(8-1)

is called the orthogonal projection of Rn onto V .

We are not yet justified in calling PV the orthogonal projection of Rn
onto V because the definition of PV depends on the basis and every subspace
of Rn of dimension 2 has infinitely many orthogonal bases. For example,
if a2 + b2 = 1, then {(a, b), (b, a)} is an orthonormal basis for R2 . In ??
we will show that PV does not depend on the choice of orthonormal basis;
that will justify our use of the word the.2
Proposition 8.1. Let {u1 , . . . , uk } be an orthonormal basis for V . The
orthogonal projection PV : Rn Rn has the following properties:
(1)
(2)
(3)
(4)
(5)

it is a linear transformation;
PV (v) = v for all v V ;
R(PV ) = V ;
ker(PV ) = V ;
PV PV = PV .3

2Thus, although the formula on the right-hand side of (8-1) depends on the choice of

orthonormal basis the value PV (v) is independent of how it is computed.

3We usually abbreviate this by writing P 2 = P .
V
V

8. ORTHOGONAL PROJECTIONS

141

Proof. We will write P for PV in this proof.

(1) To show that P is a linear transformation we must show that P (av +
bw) = aP (v)+bP (w) for all a, b R and all v, w Rn . To do this we simply
calculate:




P (av + bw) = (av + bw) u1 u1 + (av + bw) u2 u2 +




= a(v u1 ) + b(w u1 ) u1 + a(v u2 ) + b(w u2 ) u2 +
= a(v u1 )u1 + b(w u1 )u1 + a(v u2 )u2 + b(w u2 )u2 +
= a(v u1 )u1 + a(v u2 )u2 + + b(w u1 )u1 + b(w u2 )u2 +
= aP (v) + bP (w).
(2) Because the basis is orthonormal, ui ui = 1 and ui uj = 0 when
i 6= j. Therefore P (ui ) = ui for each i = 1, . . . , k.
Let v V . Then v = a1 u1 + + ak uk because V = span{u1 , . . . , uk }.
Therefore
P (v) = P (a1 u1 + + ak uk )
= a1 P (u1 ) + + ak P (uk ) because P is a linear transformation
= a1 u1 + + ak uk
= v.
(3) Because (2) is true V R(P ). On the other hand, if v is any
element in Rn , then P (v) is a linear combination of the ui s so belongs to V ;
i.e., R(P ) V . Hence R(P ) = V .
(4) If v V , then v ui = 0 for all ui so P (v) = 0. Therefore V
ker(P ). Conversely, if w Rn and P (w) = 0, then
(w u1 )u1 + + (w uk )uk = 0;
but {u1 , . . . , uk } is linearly independent so w ui = 0 for all i. Therefore
w Sp{u1 , . . . , uk } = V . We have shown that ker(P ) V . Hence
ker(P ) = V .
(5) Let v Rn . By (3), P (v) V . However, P (w) = w for all w V by
(2). Applying this fact to w = P (v) we see that (P P )(v) = P (P (v)) = v.
Since this equality holds for all v in Rn we conclude that P P = P .

The next result shows that V + V = Rn , i.e., every element in Rn is
the sum of an element in V and an element in V . Actually, the result
shows more: there is only one way to write a vector v Rn as the sum of
an element in V and an element in V .
Proposition 8.2. Let V be a subspace of Rn . If v Rn there are unique
elements w V and w0 V such that v = w + w0 , namely w := PV (v) and
w0 := v PV (v).
Proof. We write P for PV in this proof.
We start with the trivial observation that
(8-2)

v = P (v) + v P (v) .

142

15. ORTHOGONALITY

By Proposition 8.1, P (v) belongs to V . On the other hand,

P (v P (v)) = P (v) P (P (v)) = P (v) (P P )(v) = P (v) P (v) = 0
so v P (v) ker(P ). But ker(P ) = V by Proposition 8.1. Therefore the
expression (8-2) shows that v is in V + V .
To show the uniqueness, suppose that w1 , w2 V , that x1 , x2 V , and
that w1 + x1 = w2 + x2 . Therefore w1 w2 = x2 x1 ; since V is a subspace
w1 w2 V ; since V is a subspace x2 x1 V ; but V V = {0}
so w1 w2 = x2 x1 = 0. Thus w1 = w2 and x1 = x2 . This proves that
an element of Rn is the sum of an element in V and an element in V in a
unique way.

The next result is just another way of stating Proposition 8.2.
Corollary 8.3. Let V is a subspace of Rn and v Rn . The only point
w V such that v w V is w = P (v).
Proof. Suppose that w V and v w V . The equation v = w + (v w)
expresses v as the sum of an element in V and an element in V . But the
only way to express v as such a sum is v = PV (v) + (v PV (v). Therefore
w = PV (v).

Proposition 8.4. If V is a subspace of Rn , then
dim V + dim V = n.
Proof. Since PV is a linear transformation from Rn to itself there is a
unique n n matrix B such that PV (v) = Bv for all v Rn . It is clear that
R(PV ) = R(B) and ker(PV ) = N (B). But R(PV ) = V and ker(PV ) = V
so
n = rank(B) + nullity(B) = dim(V ) + dim(V )
as claimed.

8.1. PV is well-defined. The projection PV was defined in terms of
an orthonormal basis {u1 , . . . , uk } for V . However, if dim(V ) 2, then V
has infinitely many orthonormal bases so why are we justified in calling PV
the orthogonal projection onto V ? It is conceivable that if {w1 , . . . , wk } is a
different orthonormal basis for V then the linear transformation T : Rn
Rn defined by the formula
T (v) := (v w1 )w1 + + (v wk )wk
is not the same as PV , i.e., perhaps there is a v such that PV (v) 6= T (v).
This does not happen because we can use Proposition 8.2 to define
(
the unique w V such that
PV (v) :=
v = w + w0 for some w0 V .
Alternatively, using the point of view expressed by Corollary 8.3, we could
define PV (v) to be the unique element of V such that v PV (v) belongs to
V .

9. LEAST-SQUARES SOLUTIONS TO INCONSISTENT SYSTEMS

143

9. Least-squares solutions to inconsistent systems

A system of linear equations is inconsistent if it doesnt have a solution.
Until now we have ignored inconsistent systems for what might seem sensible
reasonsif Ax = b has no solutions what more is there to say? Actually, a
lot!
Although there may be no solution to Ax = b we might still want to
find an x such that Ax is as close as possible to b. For example, if b was
obtained as a result of making error-prone measurements it might be that
the true value is not b but some other point b . If so, we should be seeking
solutions to the equation Ax = b not Ax = b.
The equation Ax = b has a solution if and only if b is in the range of A
or, equivalently, if and only if b is a linear combination of the columns of A.
Thus, if Ax = b has no solution it is sensible to replace b by the point b in
R(A) closest to b and find a solution x to the equation Ax = b ; there is
such an x because b is in the range of A.
We will now implement this idea. The first step is the next result. It
says that the point in R(A) closest to b is P (v) where P : Rn Rn is the
orthogonal projection onto R(A).4
Theorem 9.1 (The best approximation theorem,5 BAT). Let V be a
subspace of Rn and b a point in Rn . Then
||PV (b) b|| < ||w b||
for all other w V . More precisely, the unique point in V closest to b is
PV (b).6
Proof. To keep the notation simple lets write P for PV .
Suppose b V . Then b is the point in V closest to b, duh! But P (b) = b
the theorem is true.
From now on we assume b
/ V . Let w V and assume w 6= P (b).
The sides of the triangle in Rn with vertices 0, b P (b), and w P (b)
have lengths ||b w||, ||b P (b)||, and ||w P (b)||. Notice that b P (b) V ,
and w P (b) V because it is a difference of elements in the subspace V .
It follows that the line through 0 and b P (b) is perpendicular to the line
4When you read the statement of the theorem you will see that it does not exactly

say that. The theorem states a more general result. Try to understand why the theorem
implies P (v) is the point in R(A) closest to b. If you cant figure this out, ask.
5
The adjective best applies to the word approximation, not to the word
theorem.
6We use the word unique to emphasize the fact that there is only one point in V that
is closest to b. This fact is not obvious even though your geometric intuition might suggest
otherwise. If we drop the requirement that V is a subspace there can be more than one
point in V that is closest to some given point b. For example, if V is the circle x2 + y 2 = 1
in R2 and b = (0, 0), then all points in V have the same distance from b. Likewise, if
2
2
b = (0, 2) andq
V is the
 parabola
 qy = x in R , there are two points on V that are closest
to b, namely

3 3
,
2 2

and

3 3
,
2 2

144

15. ORTHOGONALITY

through 0 and w P (b). Thus, the triangle has a right angle at the origin.
The longest side of a right-angled triangle is its hypotenuse which, in this
case, is the line through b P (b), and w P (b). That side has length
||(w P (b)) (b P (b))|| = ||w b||.
Therefore ||w b|| > ||b P (b)||.

9.1. The point in V closest to v. Let V be a subspace of Rn and v

a point in Rn . How do we find/compute the point in V that is closest to v?
We will show that PV (v) is the point in V that is closest to v.
We begin with a high school question. Let L be a line through the origin
in R2 and v a point in R2 ; which point on L is closest to v? Heres a picture
to help you think about the question:
L

L0

It is pretty obvious that one should draw a line L0 through v that is perpendicular to L and the intersection point L L0 will be the point on L closest
to v.
That is the idea behind the next result because the point LL0 is PL (v).
9.2. A problem: find the line that best fits the data. Given
points
(1 , 1 ), . . . , (m , m )
in R2 find the line y = dx + c that best approximates those points. In
general there will not be a line passing through all the points but we still
want the best line. This sort of problem is typical when gathering data
to understand the relationship between two variables x and y. The data
obtained might suggest a linear relationship between x and y but, especially
in the social sciences, it is rare that the data gathered spells out an exact
linear relationship.
We restate the question of finding the line y = dx + c as a linear algebra
problem. The problem of finding the line that fits the data (i , i ), 1 i

9. LEAST-SQUARES SOLUTIONS TO INCONSISTENT SYSTEMS

145

m is equivalent to finding d and c such that

1 =
2 =
3 =
..
.
m =

d1 + c
d2 + c
d3 + c
..
.
dm + cm

which is, in turn, Equivalent to finding d and c such that

1 1
1
2 1   2
 

d
3 1 d
3
=
or
A
=b

c
c
..
..
..
.

.
.
m 1
m
where A is an m 2 matrix and b Rm .
As we said, there is usually no solution ( dc ) to this system of linear
equations but we still want to find a line that is as close as possible to
the given points. The next definition makes the idea as close as possible
precise.
Definition 9.2. Let A be an m n matrix and b a point in Rm . We
call x a least-squares solution to the equation Ax = b if Ax is as close
as possible to b; i.e.,
||Ax b|| ||Ax b||

for all x Rn .

We will now explain how to find x

Because Ax is in the range of A we want Ax to be the (unique!) point
in R(A) that is closest to b. Lets write b for the point in R(A) that is
closest to b. A picture helps:
b

R(A)
b

R(A)

Lemma 9.3. Let A be an m n matrix and v Rm . Consider v as a

column vector. Then v R(A) if and only if AT v = 0.

146

15. ORTHOGONALITY

Proof. Since the range of A is the linear span of the columns of A, v

R(A) if and only if A1 v = = An v = 0. However, Aj v = ATj v so
v R(A) if and only if AT1 v = = ATn v = 0. But ATj is the j th row of

AT so v R(A) if and only if AT v = 0.
With reference to the picture above, b b R(A) if and only if
b ) = 0.

AT (b

Lemma 9.4. Let A be an m n matrix and let P : Rn Rn be the

orthogonal projection onto R(A). Define
b := P (b).
Then b
(1)
(2)
(3)

is the unique point in R(A)

that is closest to b;
such that b b R(A) ;
such that AT (b b ) = 0.

Proof. (1) This is what Theorem 9.1 says.

(2) This is what Corollary 8.3 says.
(3) This follows from (2) and Lemma 9.3.

The point in R(A) closest to b is the unique

point b in R(A) such that AT b = AT b .
Since b is in R(A), there is a vector x such that Ax = b . It follows that
AT Ax = AT b = AT b.
The preceding argument has proved the following result.
Theorem 9.5. Let A be an m n matrix and b a point in Rm . There
is a point x in Rn having the following equivalent properties:
(1) AT Ax = AT b;
(2) Ax is the point in R(A) that is closest to b;
(3) x is a least-squares solution to Ax = b.
Moreover, x is unique if and only if N (A) = 0.
Theorem 9.6. Let Ax = b be an m n system of linear equations.
(1) The system AT Ax = AT b is consistent.
(2) The solutions to the system AT Ax = AT b are least-squares solutions to Ax = b.
(3) There is a unique least squares solution if and only if rank(A) = n.
10. Approximating data by polynomial curves
When engineers lay out the path for a new a freeway they often do so
under the constraint that the curve/path is a piecewise cubic curve. Let me
explain. Suppose one wants a freeway to begin at a point (a, b) in R2 and end
at the point (a0 , b0 ). Of course, it is unreasonable to have the freeway be a

12. INFINITE SEQUENCES

147

straight line: there might be geographical obstacles, or important buildings

and other things to be avoided, or one might want the freeway to pass close
to some particular points, towns, other roads, etc. At the other extreme one
doesnt want the freeway to be too curvy because a very curvy road is
dangerous and more expensive to build. What one tends to do is break the
interval [a, a0 ] into subintervals, say
[a, a1 ], [a1 , a2 ], . . . , [an , a0 ]
where a < a1 < . . . < an < a0 and then decide on some points (a1 , b1 ), . . . , (an , bn )
that one would like the freeway to pass through.
Calculus tells us that a cubic curve y = x3 + x2 + x + has at most
two maxima and minima or, less formally, two major bends.
Lets look at
Given points
(1 , 1 ), . . . , (m , m )
2
in R we might want to find a degree 2 polynomial f (x) such that the curve
y = f (x) best approximate those points. In general there will not be a
degree 2 curve that passes through all the points but we still want the
best curve. We could also ask for the best cubic curve, etc. All these
problems are solved by using the general method in section 9. For example,
if we are looking for a degree two curve we are looking for a, b, c such that
y = ax2 + bx + c best fits the data. That problem reduces to solving the
system
2

1 1 1
1

2 2 1 2
a
2
a

2 3 1 3

3
b
b = b
=
or
A

..
..
..
.. c
c
.
.
.
.
2
m
m m 1
where A is an m 3 matrix and b Rm .
So, using the ideas in chapter 9 we must solve the equation
AT Ax = AT b
to get an a , b , and c that gives a parabola y = a x2 + b x + c that best
fits the data.
10.1. Why call it least-squares? We are finding an x that minimizes
||Ax b|| and hence ||Ax b||2 But
||Ax b||2 = (Ax b) (Ax b) = a sum of squares.
11. Gazing into the distance: Fourier series
12. Infinite sequences

CHAPTER 16

Similar matrices
1. Definition
Similar matrices are just that, similar, very similar, in a sense so alike
as to be indistinguishable in their essential properties. Although a precise
definition is required, and will be given shortly, it should be apparent that
the matrices




1 0
0 0
A=
and
B=
0 0
0 1
are, in the everyday sense of the word, similar. Multiplication by A and
multiplication by B are linear transformations from R2 to R2 ; A sends e1 to
itself and kills e2 ; B sends e2 to itself and kills e1 . Pretty similar behavior,
huh!?

2. Definition and first properties

.
Let A and B be n n matrices. We say that A is similar to B and write
A B if there is an invertible matrix S such that
A = SBS 1 .
Similarity of matrices is an equivalence relation meaning that
(1) A A because A = IAI 1 ;
(2) if A B, then B A because A = SBS 1 implies S 1 AS =
S 1 (SBS 1 )S = (S 1 S)B(S 1 S) = IBI = B;
(3) if A B and B C, then A C because A = SBS 1 and B =
T CT 1 implies A = SBS 1 = S(T CT 1 )S 1 = (ST )C(T 1 S 1 ) =
(ST )C(ST )1 .
Thus, the word similar behaves as it does in its everyday use:
(1) a thing is similar to itself;
(2) if one thing is similar to another thing, then the other thing is
similar to the first thing;
(3) if a thing is similar to a second thing and the second thing is similar
to a third thing, then the first thing is similar to the third thing.
149

150

16. SIMILAR MATRICES

3. Example
Lets return to the example in chapter 1. The only difference between
A and B is the choice of labeling: which order do we choose to write
 
 
1
0
and
0
1
or, if you prefer, which is labelled e1 and which is labelled e2 .
There is a linear transformation from R2 to R2 that interchanges e1 and
e2 , namely


0 1
S=
;
Se1 = e2 and Se2 = e1 .
1 0
Since S 2 = I, S 1 = S and



 

 

0 1
0 0
0 1
0 1
0 1
1 0
1
SBS =
=
=
= A.
1 0
0 1
1 0
0 0
1 0
0 0
The calculation shows that A = SBS 1 ; i.e., A is similar to B in the
technical sense of the definition in chapter 2.
4
We now show that similar matrices have some of the same properties.
Theorem 4.1. Similar matrices have the same
(1) determinant;
(2) characteristic polynomial;
(3) eigenvalues.
Proof. Suppose A and B are similar. Then A = SBS 1 for some invertible
matrix S.
(1) Since det(S 1 ) = (det S)1 , we have
det(A) = (det S)(det B)(det S 1 ) = (det S)(det B)(det S)1 = det(B).
We used the fact that the determinant of a matrix is a number.
(2) Since S(tI)S 1 = tI, A tI = S(B tI)S 1 ; i.e., A tI and B tI
are similar. (2) now follows from (1) because the characteristic polynomial
of A is det(A tI).
(3) The eigenvalues of a matrix are the roots of its characteristic polynomial. Since A and B have the same characteristic polynomial they have
the same eigenvalues.

4.1. Warning: Although they have the same eigenvalues similar matrices do not usually have the same eigenvectors or eigenspaces. Nevertheless,
there is a precise relationship between the eigenspaces of similar matrices.
We prove that in Proposition 4.2

151

4.2. Notation: If A is an n n matrix and X a subset of Rn , we use

the notation
AX := {Ax | x Rn }.
This shorthand is similar to the notation 2Z = {2n | n Z} for the even
numbers.
To check whether you understand this notation try the following problems.
(1) Show that AX is a subspace if X is.
(2) Show that A(BX) = (AB)X if A and B are n n matrices.
(3) Show that IX = X.
(4) If S is an invertible matrix and Y = SX, show that X = S 1 Y .
(5) If S is an invertible matrix and X is a subspace of Rn show that
dim SX = dim X by proving the following claim: if {v 1 , . . . , v d } is
a basis for X, then {Sv 1 , . . . , Sv d } is a basis for SX.
Proposition 4.2. Suppose A = SBS 1 . Let E (A) be the -eigenspace
for A and E (B) the -eigenspace for B. Then E (B) = S 1 E (A), i.e.,
E (B) = {S 1 x | x E (A)}.
In particular, the dimensions of the -eigenspaces for A and B are the same.
Proof. If x E (A), then x = Ax = SBS 1 x so
B(S 1 x) = S 1 x = (S 1 x);
i.e., S 1 x is a -eigenvector for B or, equivalently,
S 1 E (A) E (B).
Starting from the fact that B = S 1 AS, the same sort of argument shows
that SE (B) E (A).
Therefore
E (B) = I.E (B) = S 1 S.E (B) S 1 .E (A) E (B).
In particular, E (B) S 1 .E (A) E (B) so these three sets are equal,
i.e., E (B) = S 1 .E (A) = {S 1 x | x E (A)}.
That dim E (A) = dim E (B) is proved by method suggested in exercise
5 just prior to the statement of this proposition.

If A and B are similar and Ar = 0, then B r = 0. To see this one uses
the marvelous cancellation trick, S 1 S = I.
Corollary 4.3. Similar matrices have there same rank and the same
nullity.
Proof. Suppose A and B are similar matrices. By definition, the nullity of
A is the dimension of its null space. But N (A) = E0 (A), the 0-eigenspace
of A. By the last sentence in the statement of Proposition 4.2, E0 (A) and
E0 (B) have the same dimension. Hence A and B have the same nullity.
Since rank +nullity = n, A and B also have the same rank.


152

16. SIMILAR MATRICES

4.3. Intrinsic and extrinsic properties of matrices. It is not unreasonable to ask whether a matrix that is similar to a symmetric matrix is symmetric. Suppose A is symmetric and S is invertible. Then
(SAS 1 )T = (S 1 )T AT S T = (S T )1 AS T and there is no obvious reason
which this should be the same as SAS 1 . The explicit example



1 

 

1 2
2 3
1 2
8 11
1 2
8 5
=
=
0 1
3 4
0 1
3 4
0 1
3 2
shows that a matrix similar to a symmetric matrix need not be symmetric.
PAUL - More to say.

5. Diagonalizable matrices
5.1. Diagonal matrices. A square matrix D = (dij ) is diagonal if
dij = 0 for all i 6= j. In other words,

1 0 0
0
0
0 2 0
0
0

0 0 3

0
0

D = ..
.
..
..
.

0 0 0 n1 0
0 0 0
0
n
for some 1 , . . . , n R. We sometimes use the abbreviation

1 0 0
0
0
0 2 0
0
0

0 0 3
0
0

diag(1 , . . . , n ) = ..
.. .
.
.
.
.
.

0 0 0 n1 0
0 0 0
0
n
The identity matrix and the zero matrix are diagonal matrices.
Let D = diag(1 , . . . , n ). The following facts are obvious:
(1)
(2)
(3)
(4)

the i s are the eigenvalues of D;

det(D) = 1 2 n ;
the characteristic polynomial of D is (1 t)(2 t) (n t);
Dej = j ej , i.e., the standard basis vectors ej are eigenvectors for
D.

Warning: Although the axis Rej is contained in j -eigenspace for D

its j -eigenspace might be bigger. For example, if i = j , then the Ei
contains the plane Rei + Rej .

5. DIAGONALIZABLE MATRICES

153

5.2. Definition. An n n matrix A is diagonalizable if it is similar to

a diagonal matrix, i.e., if S 1 AS is diagonal for some S.
A diagonal matrix is diagonalizable: if D is diagonal, then I 1 DI is
diagonal!
Since similar matrices have the same characteristic polynomials, the
characteristic polynomial of a diagonalizable matrix is a product of n linear
terms. See obvious fact (3) above.
5.3. Example. If


5 6
A=
3 4

and



2 1
S=
,
1 1

then
S


AS =

1 1
1 2



 

5 6
2 1
2 0
=
3 4
1 1
0 1

so A is diagonalizable.
5.4. The obvious questions are how do we determine whether A is diagonalizable or not and if A is diagonalizable how do we find an S such
that S 1 AS is diagonal. The next theorem and its corollary answer these
questions.
Theorem 5.1. An n n matrix A is diagonalizable if and only if it has
n linearly independent eigenvectors.
Proof. We will use the following fact. If B is a p q matrix and C a q r
matrix, then the columns of BC are obtained by multiplying each column
of C by B; explicitly,
BC = [BC 1 , . . . , BC r ]
where C j is the j th column of C.
() Let {u1 , . . . , un } be linearly independent eigenvectors for A with
Auj = j uj for each j. Define
S := [u1 , . . . , un ],
i.e., the j th column of S is uj . The columns of S are linearly independent
so S is invertible. Since
I = S 1 S = S 1 [u1 , . . . , un ] = [S 1 u1 , . . . , S 1 un ],
S 1 uj = ej , the j th standard basis vector.

154

16. SIMILAR MATRICES

Now
S 1 AS = S 1 A[u1 , . . . , un ]
= S 1 [Au1 , . . . , Aun ]
= S 1 [1 u1 , . . . , n un ]
= [1 S 1 u1 , . . . , n S 1 un ]
= [1 e1 , . . . , n en ]

1 0 0
0 2 0

0 0 3

= ..
..
.
.

0 0 0
0 0 0

0
0
0
n1
0

0
0
0
..
.

0
n

Thus A is diagonalizable.
() Now suppose A is diagonalizable, i.e.,
S 1 AS = D = diag(1 , . . . , n )
for some invertible S. Now
[AS 1 , . . . , AS n ] = AS
= SD
= S[1 e1 , . . . , n en ]
= [1 Se1 , . . . , n Sen ]
= [1 S 1 , . . . , n S n ]
so AS j = j S j for all j. But the columns of S are linearly independent so
A has n linearly independent eigenvectors for A.

The proof of Theorem 5.1 established the truth of the following corollary.
Corollary 5.2. If A is a diagonalizable nn matrix and u1 , . . . , un are
linearly independent eigenvectors for A and S = [u1 , . . . , un ], then S 1 AS
is diagonal.
Corollary 5.3. If an n n matrix has n distinct eigenvalues it is
diagonalizable.
Proof. Suppose 1 , . . . , n are the distinct eigenvalues for A and for each
i let v i be a non-zero vector such that Av i = i v i . By Theorem 1.2,
{v 1 , . . . , v n } is a linearly independent set. It now follows from Theorem
5.1 that Ais diagonalizable.


5. DIAGONALIZABLE MATRICES

155

5.5. Previous example revisited. We showed above that



5 6
A=
3 4
is diagonalizable. Lets suppose we did not know that and had to find out
whether is was diagonalizable and, if so to find a matrix S such that S 1 AS
is diagonal. Theorem 5.1 says we must determine whether A has two linearly
independent eigenvectors.
The characteristic polynomial of A is t2 t(a + d) + ad bc, i.e.,
t2 t 2 = (t 2)(t + 1).
The eigenvalues for A are 2 and 1. By Corollary 5.3, A is diagonalizable.
To find an S we need to find eigenvectors for A. The 2-eigenspace is


 
3 6
2
E2 = N (A 2I) = N
=R
.
3 6
1
The (1)-eigenspace is
E1



 
6 6
1
= N (A + I) = N
=R
.
3 3
1

Thus S can be any matrix with one column a non-zero multiple of (2 1)T
and the other a non-zero multiple of (1 1)T . For example, the matrix we
used before, namely


2 1
,
1 1
works.
It is important to realize that S is not the only matrix that works.
For example, if


3 2
R=
.
3 1
Now



 

1 1 2
5 6
3 2
1 0
1
=
.
R AR =
3 4
3 1
0 2
3 3 3
Notice that R1 AR 6= S 1 AS. In particular, A is similar to two different
diagonal matrices.

CHAPTER 17

Words and terminology

1. The need for clarity and precision
Definitions constitute the rock on which mathematics rests. Definitions
must be precise and unambiguous. The should be expressed in simple language that is easy to understand. The mathematical edifice built upon this
rock consists of results, usually called Lemmas, Propositions, Theorems, and
Corollaries. I wont discuss the question of which types of results should receive which of those labels. Results, like definitions, must be stated clearly,
accurately, and precisely. They should be unambiguous and easy to understand.
Students taking linear algebra find it a challenge to state definitions and
results with the required clarity and precision. But clarity of thought and
clarity of statement go hand-in-hand. You cant have one without the other
and as the clarity and precision of one increases so does that of the other.
Do not confuse the requirement for clarity and precision with a pedantic
adherence to the rules of grammar. The demand for clarity and precision is
a demand that you truly understand what you think, write, and say.
Be clear in what you write. Put yourself in the shoes of the reader.
Particularly when stating a definition or result put yourself in the shoes of
a reader who has never seen that definition or result before. When I see
a students inaccurate statement of a definition or result I can guess what
they want to say. But I am not grading you on what you want to say. I am
grading you on the basis of what you do say. When I read your statement
I ask myself is this statement good enough to appear in a textbook? and
does this statement tell someone who has never before seen the statement,
or a variation on it, exactly what the author wants to, or should, convey?
2. Some, all, each, every, and only
In your everyday life you know that a statement beginning Some dogs
... is very different from one that begins All dogs .... The three words
some, all, and every, play a crucial role in mathematics. Try to use
them whenever it is appropriate to do sothere is rarely a situation in which
a variation on one of those three words is better than using the word itself.
Lets consider the definition of a linear transformation.
2.1. The following definition is perfect:
157

158

17. WORDS AND TERMINOLOGY

A function T : V W between two vector spaces is a

linear transformation if T (au + bu) = aT (u) + bT (v) for all
u, v V and all a, b R.
Here is a less than perfect version of that definition.
A function T : V W between two vector spaces is a
linear transformation if T (au + bu) = aT (u) + bT (v) where
u, v V and a, b R.
The words for all have been replaced by where. The word where
has many meanings (check the dictionary if you need convincing) and in
this situation it does not have the clarity that for all does. When I read
where I have to pause and ask myself which of its meanings is being used
but when I read for all I do not have to pause and think.
Here is an incorrect version of that definition.
A function T : V W between two vector spaces is a
linear transformation if T (au + bu) = aT (u) + bT (v) for
some u, v V and a, b R.
This is wrong because some has a different meaning than all. Consider
the sentences Some dogs have rabies and All dogs have rabies.
There are mathematical statements in which the word where is appropriate. For example:
If V is a subspace of Rn , then every point w Rn can be
written in a unique way as w = v + v 0 where v V and
v0 V .
The sentence is intended to state the fact that every point in Rn is the sum
of a point in V and a point in V , and there is only one way in which that
happens, i.e., if u and v are in V and u0 and v 0 are in V and v + v 0 = u + u0 ,
then u = v and u0 = v 0 . I think the displayed sentence does convey that but
perhaps there is a better way to convey that fact. What do you think? One
alternative is this:
If V is a subspace of Rn and w Rn , then there is a unique
v V and a unique v 0 V such that w = v + v 0 .
What do you think of the alternatives?
2.2. The following is the definition of what it means for a set of vectors
to be orthogonal.
We say that {v 1 , . . . , v k } is orthogonal if v i v j = 0 for all
i 6= j.
I think it is implicit in the statement that this must be true for all pairs of
different numbers between 1 and n but it is not unreasonable to argue that
one should make it clearer that v i v j = 0 for all i, j {1, . . . , n} such that
i 6= j. The words for all are suggesting that one is making a statement
about all pairs of distinct numbers i and j in some collection of pairs. In any

4. THE WORD IT

159

case, it seems to me that the word all should be used in this definition.
The following version of the definition seems less clear:
We say that {v 1 , . . . , v k } is orthogonal if v i v j = 0 where
i 6= j.
Where seems to need some interpretation and may be mis-interpreted.
Better to use the word all because there is little possibility it can be
misunderstood.
2.3. I used the word where in my definition of linear span.
A linear combination of vectors v 1 , . . . , v n in Rm is a vector
that can be written as
a1 v 1 + + an v n
for some numbers a1 , . . . , an R.
Let A be any m n matrix and x Rn . Then Ax is a
linear combination of the columns of A. Explicitly,
(2-1)

Ax = x1 A1 + + xn An
where Aj denotes the j th column of A.
A function T : V W between two vector spaces is a
linear transformation if T (au + bu) = aT (u) + bT (v) where
u, v V and a, b R.

2.4. Try to avoid where. I think it is a good general rule to avoid

using the word where whenever possible. I dont think it is good to never
use the word where in mathematics, but certainly it should be avoided if
there is a clear alternative that conveys your meaning without making the
sentence clunky.
3. Singular and plural
A system of linear equations is consistent if it has a solution
and inconsistent if it doesnt.
The subject of this sentence, A system is singular. The system may consist
of many equations but the system is a single entity.
The subject of a sentence beginning The people of America ... is
plural so it would be correct to proceed with the word are. The subject
of a sentence beginning The population of America ... is singular so it
would be correct to proceed with the word is.
4. The word it
The word it is versatile. But danger lurks in in its versatility. A writer
must ensure that it refers to what it is supposed to refer to.
Examples...
In the following definition it refers to the system of linear equations,
the subject of the sentence.

160

17. WORDS AND TERMINOLOGY

A system of linear equations is consistent if it has a solution

and inconsistent if it doesnt.
5. A and The
If V is a non-zero vector space it has infinitely many bases so one cannot
speak of the basis for a vector space unless it is clear from the context
which basis is being referred to. That is why we say a basis in the definition
of dimension.
The dimension of a vector space is the number of elements
in a basis for it.
The definition
The dimension of a vector space is the number of elements
in the basis for it.
is wrong because it suggests, incorrectly, that there is only one basis for a
vector space.
6. If and only if

CHAPTER 18

Applications of linear algebra and vector spaces

There are so many applications of linear algebra and vector spaces that
many volumes are needed to do justice to the title of this chapter. Linear algebra and vector spaces are embedded deep in the heart of modern
technology and the modern world.

1. Simple electrical circuits

The flow of current in a simple electrical circuit consisting of batteries
and resistors is governed by three laws:
Kirchoff s First Law The sum of the currents flowing into a node
is equal to the sum of the currents flowing out of it.
Kirchoff s Second Law The sum of the voltage drops around a
closed loop is equal to the total voltage in the loop.
Ohms Law The voltage drop across a resistor is the product of
the current and the resistor.
To make apply these we must first draw a picture of the circuit. To
simplify I will begin by leaving out the resistors and batteries. A typical
circuit might look like this:
A

There are three loops and three nodes in this circuit. We must put arrows on
each loop indicating the direction the current is flowing. It doesnt matter
how we label these currents, or the direction we choose for the current flowing
161

162

18. APPLICATIONS OF LINEAR ALGEBRA AND VECTOR SPACES

around each loop.

A

I1


O

I3

I2
D

I4

O
I5
G

Applying Kirchoffs First Law at the nodes B, D, and E, we deduce that

I1 + I3 = I2
I1 + I5 = I4
I2 + I5 = I3 + I4 .
I am going to refer you to the book for more on this because I cant draw
the diagrams!
2. Magic squares
You have probably seen the following example of a magic square:
4 3 8
9 5 1
2 7 6
The numbers in each row, column, and diagonal add up to 15. Lets look
at this from the point of view of linear algebra and learn a little about this
magic square and then try to construct some larger ones.
Lets call any n n matrix with the property that the numbers in
each row, column, and diagonal add up to the same number, say, a
-valued magic square. We will call the magic value. We will write
M Sn ()
for the set of -valued magic squares. If the numbers in the matrix are
1, 2, . . . , n2 we will call it a classical magic square.
The sum of all integers from 1 to n2 is 21 n2 (n2 + 1), so in a classical n n
magic square the sum of the numbers in a row is 12 n(n2 + 1). For example
when n = 3, the magic value is 15. When n = 4 it is 34.
The nn matrix with every entry equal to 1 is an n-valued magic square.
We will write En for this matrix or just E if the n is clear. Although this will
not strike you as a particularly interesting n-valued magic square it will play
a useful role in showing that it an appropriate system of linear equations

2. MAGIC SQUARES

163

is consistent it will be a solution to themand in reducing the problem

from to a system of homogenous linear equations.
The set of all n n matrices forms a vector space: we can add them and
multiply by scalars and the properties in Proposition ?? hold. We usually
write Mn (R) for the set of all n n matrices. The subset of 0-valued magic
squares M Sn (0) is a subspace of Mn (R). The next lemma says that
M Sn () = n E + M Sn (0);
i.e., the set of -valued magic squares is a translation of the subspace
M Sn (0).
Lemma 2.1. Let A be an n n matrix. Then A is a -valued magic
square if and only if A n E is a 0-valued magic square.
For example, when

4
9
2

n = 3,

3 8
1 2 3
5 1 5E = 4
0 4
7 6
3 2
1

is a 0-valued magic square.

We now consider the problem of finding all 0-valued 3 3 magic squares.
We will write xij for the unknown value of the ij th entry. For example, the
fact that sum of the top row of the matrix must be zero gives us the equation
x11 + x12 + x13 = 0. And so on. Doing the obvious thing we get a total
of 8 homogeneous equations in 9 unknowns3 equations from the rows, 3
from the columns, and 2 from the diagonals. By Proposition ?? a system of
homogeneous equations always has a non-trivial solution when the number
of rows is strictly smaller than the number of unknowns. Hence there is a
non-trivial 3 3 0-valued magic square.
Before we can write down the coefficient matrix for the system of equations we must agree on an ordering for the unknowns. We will just use the
naive ordering and write them in the order
x11 , x12 , x13 , x21 , x22 , x23 , x31 , x32 , x33 .
The coefficient

1
0

0
0

matrix is
0
1
0
0
0
1
0
0

0
0
1
0
0
1
1
0

0
0
0
1
1
0
0
0

1
1
0
0
1
0
1
0

0
0
1
0
1
0
0
0

0
0
0
1
0
0
1
1

0
1
0
0
0
0
0
1

1
0

0
1

main diagonal
column 2
column 3
column 1
row 2
row 1
anti-diagonal
row 3

164

18. APPLICATIONS OF LINEAR ALGEBRA AND VECTOR SPACES

I dont want
progressively

1 0 0
0 1 0

0 0 1

0 0 0

0 0 0

0 0 0

0 0 0
0 0 0

to present all the row operations in gory detail, but we can

obtain the equivalent matrices

0 1
0 0 0
1
MD
0 1
0 0 1
0
C2

0 0
1 0 0
1
C3

1 1 0 1 0 1
C1-MD

0 1 1 1 0 1
AD-C3

0 0
0 0 0
0
R2+R1+R3-C1-C2-C3

0 2 1 0 1 2
R1-MD-C2-C3
0 0
0 1 1
1
row 3

then

1
0

0
0

0 1
0 0 0
1
0 1
0 0 1
0

0 0
1 0 0
1

1 1 0 1 0 1

0 1 1 1 0 1

0 0
0 0 0
0

0 0 3 2 1 4
0 0
0 1 1
1

0
1
0
0
0
0
0
0

0
0
1
0
0
0
0
0

MD
C2
C3
C1-MD
AD-C3
R2+R1+R3-C1-C2-C3
R1-MD-C2-3C3+2AD
row 3

We can already see from this that x32 and x33 will be the independent
variables, so lets take
x33 = 1, x32 = 2.
Evaluating the dependent variables gives
x31 = 3, x23 = 4, x22 = 0, x21 = 4, x13 = 3, x12 = 2, x11 = 1,
which is the 0-valued magic square we obtained before.
The rank of the matrix is 8-1=7, so its nullity is 2. That is equivalent
to the existence of two independent variables, so lets find another solution
that together with the first will give us a basis. It is obvious though that a
3 3 matrix has various symmetries that we can exploit.
For the n n case we get a homogeneous system of 2n + 2 equations in
n2 variables. If n > 2 this system has a non-trivial solution.

CHAPTER 19

Last rites
1. A summary of notation
Sets are usually denoted by upper case letters.
f : X Y denotes a function f from a set X to a set Y . If f : X Y
and g : Y Z are functions, then g f or gf denotes the composition first
do f then do g. Thus gf : X Z and is defined by
(gf )(x) = g(f (x)).
This makes sense because f (x) is an element of Y so g can be applied to it.
The identity function idX : X X is defined by
idX (x) = x

for all x X.

If f : X Y and g : Y X are such that gf = idX and f g = idY

we call g the inverse of f and denote it by f 1 . Of course, if g = f 1 , then
f = g 1 .
We denote the range of a function f : X Y by R(f ). Thus, R(f ) =
{f (x) | x X}. It is a subset of Y .
If A is an m n matrix we often write A1 , . . . , An for the columns of A
and
A = [A1 , . . . , An ]
to indicate this. This notation usually appears when we want to make use
of the fact that
BA = B[A1 , . . . , An ] = [BA1 , . . . , BAn ],
i.e., the j th column of BA is B times the j th column of A.
We write ej for the element in Rn having zeroes in all positions except
the j th which is 1. Thus, the identity matrix is I = [e1 , . . . , en ]. More
generally, if A is any m n matrix, then Aej = Aj , the j th column of A.
You can see this either by computation or using the fact that AI = A.
If x1 , . . . , xn are vectors, then Rx1 + + Rxn is synonymous with
Sp(x1 , . . . , xn ). It denotes the set of all linear combinations of the vectors
x1 , . . . , x n .
The -eigenspace is denoted by E . If it is necessary to name the matrix
for which it is the eigenspace we write E (A).
165

166

19. LAST RITES

2. A sermon
Learning a new part of mathematics is like learning other things: photography, cooking, riding a bike, playing chess, listening to music, playing
music, tennis, tasting wine, archery, seeing different kinds of art, etc. At
some point one does, or does not, make the transition to a deeper involvement where one internalizes and makes automatic the steps that are first
learned by rote. At some point one does, or does not, integrate the separate
discrete aspects of the activity into a whole.
You will know whether you are undergoing this transition or not. It may
be happening slowly, more slowly than you want, but it is either happening
or not. Be sensitive to whether it is happening. Most importantly, ask
yourself whether you want it to happen and, if you do, how much work
you want to do to make it happen. It will not happen without your active
involvement.
Can you make fluent sentences about linear algebra? Can you formulate
the questions you need to ask in order to increase your understanding?
Without this internalization and integration you will feel more and more like
a juggler with too many balls in the air. Each new definition, concept, idea,
is one more ball to keep in the air. It soon becomes impossible unless one
sees the landscape of the subject as a single entity into which the separate
pieces fit together neatly and sensibly.
Linear algebra will remain a great struggle if this transition is not happening. A sense of mastery will elude you. Its opposite, a sense that the
subject is beyond you, will take rootparalysis and fear will set in. That
is the dark side of all areas of life that involve some degree of competence,
public performance, evaluation by others, and consequences that you care
about.
Very few people, perhaps none, get an A in 300- and higher-level math
courses unless they can integrate the separate parts into a whole in which
the individual pieces not only make sense but seem inevitable and natural.
It is you, and only you, who will determine whether you understand
linear algebra. It is not easy.
Good luck!