Topic 0 -

A Brief Review of Linear Algebra with Python

AMA3724 Further Mathematical Methods(2024/25 Semester 2 )

Lecturer: Dr. Guofeng Zhang

I Basic matrix operations I Partitioned matrices

I Determinant I System of linear equations
I Nonsingular matrices I Elementary row operations

Basic matrix operations P. 2 / 44

Matrix
An m × n matrix A is a rectangular array of numbers with m rows and n
columns
a11 a12 ··· a1n
 
 a21 a22 ··· a2n 
A = [aij ] = 
 ... .. .. .
.. 
. . .
am1 am2 ··· amn
The element aij is the entry at the ith row and the jth column of A and is
called the (i, j)th entry of A.

" #
2    
  1 2 1 −2 4
A= 1 1 −1 B= 7 C= D=
3 4 5 0 3
| {z } 8
1×3 | {z }
|{z} | {z }
2×2 2×3
3×1
Basic matrix operations P. 3 / 44

Download Anaconda:
https://www.anaconda.com/products/individual

Basic matrix operations P. 4 / 44

Basic matrix operations P. 5 / 44

Basic matrix operations P. 6 / 44

https://docs.sympy.org/latest/tutorials/intro-tutorial/matrices.html
Basic matrix operations P. 7 / 44

[1]: # Import SymPy package [7]: # Import NumPy package

import sympy as sp import numpy as np

[8]: # Define matrices (NumPy package)

[2]: # Define matrices (SymPy package) A = np.array([[1,1,-1]]); A
A = sp.Matrix([[1,1,-1]]); A

[8]: array([[ 1, 1, -1]])

[2]:
 
1 1 −1

[9]: B = np.array([[2],[7],[8]]); B
[3]: B = sp.Matrix([[2],[7],[8]]); B

[3]: [9]: array([[2],


2
7 [7],
8 [8]])

[4]: C = sp.Matrix([[1,2],[3,4]]); C [10]: C = np.array([[1,2],[3,4]]); C

[4]:
h i
1 2 [10]: array([[1, 2],
3 4 [3, 4]])

[5]: D = sp.Matrix([[1,-2,4],[5,0,3]]); D [11]: D = np.array([[1,-2,4],[5,0,3]]); D

[5]:
h i
1 −2 4 [11]: array([[ 1, -2, 4],
5 0 3 [ 5, 0, 3]])

[6]: # Size of matrices

A.shape,B.shape,C.shape,D.shape [12]: # Size of matrices
A.shape,B.shape,C.shape,D.shape

[6]: ((1, 3), (3, 1), (2, 2), (2, 3)) [12]: ((1, 3), (3, 1), (2, 2), (2, 3))

Basic matrix operations P. 8 / 44

Addition, Subtraction and Scalar Multiplication

For any two m × n matrices A = [aij ] and B = [bij ] and any scalar c,
1. A + B is the m × n matrix with aij + bij as its (i, j)th entry.
2. A − B is the m × n matrix with aij − bij as its (i, j)th entry.

3. cA is the m × n matrix with caij as its (i, j)th entry.

Properties for Addition and Scalar Multiplication

Let A, B and C be m × n matrices and let c and d be scalars.
1. A + B = B + A,
2. (A + B) + C = A + (B + C),

3. c(A + B) = cA + cB,

4. (c + d)A = cA + dA,

5. c(dA) = (cd)A,  
0 ··· 0
 .. .. .. 
6. A + 0 = A. Here, 0 =  . . .  is the m × n zero matrix.
0 ··· 0
Basic matrix operations P. 9 / 44

   
4 0 5 1 1 1
Example 1 For A = and B = ,
−1 3 2 3 5 7

[1]: import sympy as sp [4]: A+B

[4]:
h i
5 1 6
2 8 9
[2]: A = sp.Matrix([[4,0,5],[-1,3,2]]); A
[5]: A-B
[2]:
h i
4 0 5
[5]:
h i
−1 3 2 3 −1 4
−4 −2 −5

[3]: B = sp.Matrix([[1,1,1],[3,5,7]]); B
[6]: 2*B

[3]: [6]:
h i h i
1 1 1 2 2 2
3 5 7 6 10 14

h i h i h i
4 0 5 1 1 1 5 1 6
A+B = −1 3 2 + 3 5 7 = 2 8 9
h i h i h i
4 0 5 1 1 1 3 −1 4
A−B = −1 3 2 − 3 5 7 = −4 −2 −5
h i h i
1 1 1 2 2 2
2B = 2· 3 5 7 = 6 10 14

Basic matrix operations P. 10 / 44

Matrix Multiplication
For any m × n matrix A = [aij ] and n × p matrix B = [bij ], the product of A
n
X
and B, denoted by AB, is the m × p matrix with aik bkj as the (i, j)th
k=1
entry. Equivalently, if C = [cij ] = AB, then
n
X
cij = aik bkj = ai1 b1j + ai2 b2j + · · · + ain bnj .
k=1
"
4
  #
  4  
Example 2 For A = 2 3 ,B= ,C= 2 3 4 , and D = −1 ,
−1
2×1 1
1×2 z }| { 1×1
z }| {  
4
z }| {  
AB = 2 3 = 2 · 4 + 3 · (−1) = 5
−1
"
4
#
     
CD = 2 3 4 −1 = 2 · 4 + 3 · (−1) + 4 · 1 = 9
| {z } 1 | {z }
1×3 | {z } 1×1
3×1
Basic matrix operations P. 11 / 44

     
2 3 4 3 6 4 3
Example 3 For A = ,B= , and C = .
1 −5 1 −2 3 1 −2
AB [1]: import sympy as sp
h ih i A = sp.Matrix([[2,3],[1,-5]])
2 3 4 3 6 B = sp.Matrix([[4,3,6],[1,-2,3]])
= 1 −5 1 −2 3 C = sp.Matrix([[4,-3],[1,2]])
| {z } | {z }
2×2 2×3
h i [2]: A
2·4+3·1 2 · 3 + 3 · (−2) 2·6+3·3
= 1 · 4 + (−5) · 1 1 · 3 + (−5) · (−2) 1 · 6 + (−5) · 3 [2]:
h i
2 3
| {z } 1 −5
  2×3
11 0 21 [3]: B
= .
−1 13 −9 [3]:
h i
4 3 6
   1 −2 3
4 3 6 2 3
BA = is NOT defined. [4]: C
1 −2 3 1 −5
[4]:
| {z } | {z } h i
2×3 2×2 4 −3
1 2
    
2 3 4 −3 11 0 [5]:
AC = = # Matrix Multiplication
1 −5 1 2 −1 −13 A@B
    
[5]:
h i
4 −3 2 3 5 27 11 0 21
CA = = 6= AC −1 13 −9
1 2 1 −5 4 −7

Basic matrix operations P. 12 / 44

[6]: B@A

---------------------------------------------------------------------------
ShapeError Traceback (most recent call last)
/var/folders/wh/1v8jgf9d5gbfykybbcypgbt40000gn/T/ipykernel_25582/2247667524.py in <module>
----> 1 B@A

/opt/anaconda3/lib/python3.9/site-packages/sympy/core/decorators.py in binary_op_wrapper(self,␣
,→other)
134 if f is not None:
135 return f(self)
--> 136 return func(self, other)
137 return binary_op_wrapper
138 return priority_decorator

/opt/anaconda3/lib/python3.9/site-packages/sympy/matrices/common.py in __matmul__(self, other)

2727 return NotImplemented
2728
-> 2729 return self.__mul__(other)
2730
2731 def __mod__(self, other):

/opt/anaconda3/lib/python3.9/site-packages/sympy/core/decorators.py in binary_op_wrapper(self,␣
,→other)
134 if f is not None:
135 return f(self)
--> 136 return func(self, other)
137 return binary_op_wrapper
138 return priority_decorator
ShapeError: Matrix size mismatch: (2, 3) * (2, 2).
Basic matrix operations P. 13 / 44

[7]:
Power of A # Power
A**3
If A is an n × n matrix and k is a positive integer,
[7]:
h i
then 5 66
22 −149
Ak = A| · A ·{zA · · · A} .
k copies [8]: A**10

[8]:
h i
1110172 −8172999
Transpose of A −2724333 20180503

Given an m × n matrix A = [aij ], the transpose of [9]: # Transpose

A is the n × m matrix, denoted by AT , whose A.T
columns are formed from the corresponding rows
[9]:
h i
2 1
of A. That is, the (i, j)th entry of AT is equal to 3 −5
aji .
[10]: B.T

Identity Matrix [10]:


4 1

The n × n identity matrix In has the (i, j)th entry 3 −2
6 3
1 0 ··· 0
 
0 1 · · · 0 [11]: # Identity matrix

1 i=j sp.eye(4)
δij = . That is, In = 
 ... ... . . . ..  .
0 i 6= j .

[11]: 1
" #
0 0 0
0 0 ··· 1 0 1 0 0
| {z } 0 0 1 0
n×n 0 0 0 1

Basic matrix operations P. 14 / 44

Properties for Transpose

Let A and B denote matrices whose sizes are appropriate for the following
sums and products and c be any scalar.

1. (AT )T = A 3. (cA)T = cAT

2. (A + B)T = AT + B T 4. (AB)T = B T AT
   
2 3 4 3 [12]: A@C
Example 4 For A = and C = ,
1 −5 1 −2
[12]:
h i
11 0
−1 −13
    
2 3 4 −3 11 0
AC = =
1 −5 1 2 −1 −13 [13]: A.T@C.T

[13]:
     h i
T T 2 1 4 1 5 4 5 4
A C = = 27 −7
3 −5 −3 2 27 −7
     [14]: C.T@A.T
4 1 2 1 11 −1
C T AT = = [14]:
h
11 −1
i
−3 2 3 −5 0 −13 0 −13
Determinant P. 15 / 44

Determinant of 2 × 2 / 3 × 3 matrices
The determinant of a 2 × 2 matrix is defined and denoted by
 
a11 a12 a11 a12
det A = |A| = det = = a11 a22 − a12 a21 .
a21 a22 a21 a22

The determinant of a 3 × 3 matrix is defined and denoted by

a11 a12 a13

det(A) = a21 a22 a23 = a11 a22 a33 + a31 a12 a23 + a21 a32 a13
a31 a32 a33
−a31 a22 a13 − a21 a12 a33 − a11 a23 a32 .

Cofactor
Let A be an n × n matrix.
1. Denote by Aij the (n − 1) × (n − 1) matrix obtained from A by deleting
its ith row and jth column.
2. The (i, j)-th cofactor Cij is defined by

Cij = (−1)i+j det Aij .

Determinant P. 16 / 44

a11 a12 a13

" #  
a22 a23 a22 a23
A = a21 a22 a23 A11 = C11 = (−1)1+1
a32 a33 a32 a33
a31 a32 a33
a11 a12 a13
" #  
a21 a23 a21 a23
A = a21 a22 a23 A12 = C12 = (−1)1+2
a31 a33 a31 a33
a31 a32 a33
a11 a12 a13
" #  
a21 a22 a21 a22
A = a21 a22 a23 A13 = C13 = (−1)1+3
a31 a32 a31 a32
a31 a32 a33
" #
a11 a12 a13 a14  
a22 a23 a24 a22 a23 a24
a21 a22 a23 a24
A= a31 a32 a33 a34 A11 = a32 a33 a34 C11 = (−1)1+1 a32 a33 a34
a42 a43 a44 a42 a43 a44
a41 a42 a43 a44
" #
a11 a12 a13 a14  
a11 a12 a14 a11 a12 a14
a21 a22 a23 a24
A= a31 a32 a33 a34 A23 = a31 a32 a34 C23 = (−1)2+3 a31 a32 a34
a41 a42 a44 a41 a42 a44
a41 a42 a43 a44
" #
a11 a12 a13 a14  
a11 a12 a13 a11 a12 a13
a21 a22 a23 a24
A= a31 a32 a33 a34 A34 = a21 a22 a23 C34 = (−1)3+4 a21 a22 a23
a41 a42 a43 a41 a42 a43
a41 a42 a43 a44
Determinant P. 17 / 44

Determinant
For any n ≥ 2, the determinant of an n × n matrix A = [aij ] is defined by
n
X
det A = a11 C11 + a12 C12 + a13 C13 + a14 C14 + · · · + a1n C1n = a1j C1j .
j=1

It can also be computed by a cofactor expansion across any row or down any
column.
I The expansion across the ith row is
n
X
det A = ai1 Ci1 + ai2 Ci2 + ai3 Ci3 + ai4 Ci4 + · · · + ain Cin = aij Cij .
j=1

I The expansion down the jth column is

n
X
det A = a1j C1j + a2j C2j + a3j C3j + a4j C4j + · · · + anj Cnj = aij Cij .
i=1

Properties of determinant
For any n × n matrices A and B and scalar c,

det AT = det A, det(cA) = cn det A and det(AB) = det A · det B.

Determinant P. 18 / 44

−2
 
1 5 0
2 0 4 −1
Example 5 Suppose A =  .
3 1 0 7
0 4 −2 0

det A

= a11 C11 + a12 C12 + a13 C13 + a14 C14

0 4 −1 2 4 −1 2 0 −1 2 0 4
= 1· 1 0 7 − (−2) · 3 0 7 +5· 3 1 7 −0· 3 1 0
4 −2 0 0 −2 0 0 4 0 0 4 −2

= 1 · (114) + 2 · (34) + 5 · (−68) − 0 = −158

= a41 C41 + a42 C42 + a43 C43 + a44 C44

−2 5 0 1 5 0 1 −2 0 1 −2 5
= −0 · 0 4 −1 + 4 · 2 4 −1 − (−2) · 2 0 −1 + 0 · 2 0 4
1 0 7 3 0 7 3 1 7 3 1 0

= 0 + 4 · (−57) + 2 · (35) + 0 = −158.

Determinant P. 19 / 44

Adjoint
For any n × n matrix A, the matrix of cofactors

C11 C21 ··· Cn1

 
 C12 C22 ··· Cn2 
.
 . .. .. .. 
 .. . . .
C1n C2n ··· Cnn

is called the adjoint (adjugate) of A, denoted by adj A.

 
C11 C21 C31 C41
C11 C21 C31
  " #
C11 C21 C12 C22 C32 C42 
adj A = C12 C22 C32
C12 C22 C
13 C23 C33 C43 
C13 C23 C33
C14 C24 C34 C44
(2 × 2 case) (3 × 3 case) (4 × 4 case)

An important property for adjoint

For any n × n matrix A,

A · adj A = (det A) In .

Determinant P. 20 / 44

[1]: import sympy as sp;import numpy as np

[2]: A = sp.Matrix([[1,-2,5,0],[2,0,4,-1],[3,1,0,7],[0,4,-2,0]]); A

[2]:
" #
1 −2 5 0
2 0 4 −1
3 1 0 7
0 4 −2 0

[3]: # Determinant @ SymPy

A.det()

[3]: −158
[4]: # Adjugate @ SymPy
A.adjugate()

[4]:
" #
114 −112 −16 61
−34 14 2 −57
−68 28 4 −35
−44 46 −16 −18

[5]: A = np.array([[1,-2,5,0],[2,0,4,-1],[3,1,0,7],[0,4,-2,0]]); A

[5]: array([[ 1, -2, 5, 0],

[ 2, 0, 4, -1],
[ 3, 1, 0, 7],
[ 0, 4, -2, 0]])

[6]: # Determinant @ NumPy

np.linalg.det(A)

[6]: -158.00000000000003
Determinant P. 21 / 44

[1]: import sympy as sp; import numpy as np; import time

[2]: # generate a 5x5 random matrix between 0 and 9 @ Sympy

A = sp.randMatrix(5,5,0,9); A

[2]:
 
2 2 6 2 2
7 9 6 3 2
 7 2 1 2 1 
9 4 4 5 0
5 2 2 6 7

[3]: start = time.time(); a = A.det(); end = time.time(); a

[3]: −4422
[4]: print("The computation time is :", end-start)

The computation time is : 0.0014178752899169922

[5]: # generate a 200x200 random matrix between 0 and 9 @ SymPy

A = sp.randMatrix(200,200,0,9); A.shape

[5]: (200, 200)

[6]: start = time.time(); a = A.det(); end = time.time(); a

[6]: −146836563203882402681100099842173195419442530298048364881649588442071600424473837277364531
[7]: print("The computation time is :", end-start)

The computation time is : 58.75287485122681

Determinant P. 22 / 44

[8]: # generate a 5x5 random matrix between 0 and 9 @ NumPy

A = np.random.randint(10,size=(5,5)); A

[8]: array([[3, 1, 7, 5, 7],

[8, 8, 5, 4, 3],
[1, 1, 0, 6, 2],
[6, 3, 0, 5, 9],
[0, 7, 1, 7, 7]])

[9]: start = time.time(); a = np.linalg.det(A); end = time.time(); a

[9]: 20678.999999999978

[10]: print("The computation time is :", end-start)

The computation time is : 0.0001728534698486328

[11]: # generate a 200x200 random matrix between 0 and 9 @ NumPy

A = np.random.randint(10,size=(200,200)); A.shape

[11]: (200, 200)

[12]: start = time.time(); a = np.linalg.det(A); end = time.time(); a

[12]: 6.309923345255805e+279

[13]: print("The computation time is :", end-start)

The computation time is : 0.0021011829376220703

Nonsingular matrices P. 23 / 44

Nonsingular matrices
I An n × n matrix A is said to be nonsingular (or invertible) if there is
the unique inverse, denoted by A−1 , such that

A−1 A = In and AA−1 = In .

I A matrix that is NOT nonsingular is called a singular matrix.

Properties for nonsingular matrices

Suppose A and B are n × n nonsingular matrices. Then A−1 , AT and AB are
nonsingular and
1. (A−1 )−1 = A, 3. (AB)−1 = B −1 A−1 .
2. (AT )−1 = (A−1 )T ,

Inverse Formula
A matrix A is nonsingular if and only if det A 6= 0 and the inverse, if exists, is
equal to
1
A−1 = adj A.
det A

Nonsingular matrices P. 24 / 44

Inverse formula for 2 × 2 and 3 × 3 nonsingular matrices

Given a 2 × 2 or 3 × 3 matrix A = [aij ] with det A 6= 0. Then

(2 × 2 case) :
   
−1 1 C11 C21 1 a22 −a12
A = =
det A C12 C22 det A −a21 a11

(3 × 3 case) :
C11 C21 C31
" #
1
A−1 = C12 C22 C32
det A
C13 C23 C33
 
a22 a23 a12 a13 a12 a13
−
 a32 a33 a32 a33 a22 a23 
 
 
1  a21 a23 a11 a13 a11 a13
 
=  − a −

det A  a33 a31 a33 a21 a23

31 
 
 
 a 21 a22 a11 a12 a11 a12 
−
a31 a32 a31 a32 a21 a22
Nonsingular matrices P. 25 / 44

[1]: import sympy as sp; import numpy as np

[2]: A = sp.Matrix([[1,-2,5,0],[2,0,4,-1],[3,1,0,7],[0,4,-2,0]]); A

[2]:
" #
1 −2 5 0
2 0 4 −1
3 1 0 7
0 4 −2 0

[3]: # Inverse @ SymPy

A.inv()

[3]:
"− 57 56 8 61
− 158
#
79 79 79
17 7
− 79 1
− 79 57
79 158
34 14
− 79 2
− 79 35
79 158
22 23
− 79 8 9
79 79 79

[4]: A = np.array([[1,-2,5,0],[2,0,4,-1],[3,1,0,7],[0,4,-2,0]]); A

[4]: array([[ 1, -2, 5, 0],

[ 2, 0, 4, -1],
[ 3, 1, 0, 7],
[ 0, 4, -2, 0]])

[5]: # Inverse @ NumPy

np.linalg.inv(A)

[5]: array([[-0.72151899, 0.70886076, 0.10126582, -0.38607595],

[ 0.21518987, -0.08860759, -0.01265823, 0.36075949],
[ 0.43037975, -0.17721519, -0.02531646, 0.22151899],
[ 0.27848101, -0.29113924, 0.10126582, 0.11392405]])

Partitioned matrices P. 26 / 44

In general, given a matrix

[1]: A = sp.Matrix([[2,-3,1,0,-4,1],␣
−3 −4
"
2 1 0 1
#
,→[1,5,-2,3,-1,2],␣
A= 1 5 −2 3 −1 2 . ,→[0,-4,-2,7,-1,3]]); A
0 −4 −2 7 −1 3
[1]:
 
2 −3 1 0 −4 1
1 5 −2 3 −1 2
We can partition A as 0 −4 −2 7 −1 3
 
2 −3 1 0 −4 1 [2]: A11 = A[0:2,0:3]; A11

A=  1 5 −2 3 −1 2 . [2]:
h
2 −3 1
i
0 −4 −2 7 −1 3 1 5 −2

[3]: A12 = A[0:2,3:5]; A12

The matrix can be written as the 2×3 partitioned
(block) matrix [3]:
h
0 −4
i
  3 −1
A11 A12 A13
[4]: A13 = A[0:2,5]; A13
A21 A22 A23
[4]:
hi
whose entries are the blocks (submatrices) 1
2
     
2 −3 1 0 −4 1 [5]: A21 = A[2,0:3]; A21
A11 = A12 = A13 =
1 5 −2 3 −1 2
[5]:
 
0 −4 −2
     
A21 = 0 −4 −2 A22 = 7 −1 A23 = 3
Partitioned matrices P. 27 / 44

Addition and Scalar Multiplication

Suppose
   
A11 ··· A1s B11 ··· B1s
 .. .. ..   .. .. .. 
A = [Aij ] =  . . .  and B = [Bij ] =  . . . 
Ar1 ··· Ars Br1 ··· Brs

are r × s block matrices c is a scalar. Then

 
cA11 ··· cA1s
 . .. .. 
1. cA = [cAij ] =  .. . . ,
cAr1 ··· cArs
 
A11 + B11 ··· A1s + B1s
.. .. ..
2. A + B = [Aij + Bij ] =  .
 
. . .
Ar1 + Br1 ··· Ars + Brs
Here, Aij and Bij have sizes for which the indicated sums and products are
well defined.

Partitioned matrices P. 28 / 44

Block Multiplication
Suppose    
A11 ··· A1s B11 ··· B1t
 . .. ..   . .. .. 
A = [Aij ] =  .. . .  and B = [Bij ] =  .. . . 
Ar1 ··· Ars Bs1 ··· Bst

are r × s and s × t block matrices respectively. Then

Ps Ps 
" s
# A B
k=1 1k k1
··· A B
k=1 1k kt
.. .. ..
X
AB = Aik Bkj =
 
.
Ps . . 
k=1
Ps
A B
k=1 rk k1
··· A B
k=1 1k kt

 
In particular, if b1
 

 b2 

A= a1 a2 ··· as and B = 
 ..
.

 . 
bs
Then  
AB = a1 b1 + a2 b2 + · · · + as bs .
Partitioned matrices P. 29 / 44

Example
  6 Assume that A, B, X, and Y are square
 matrices. If the matrix
A C X Z
is invertible and its inverse is . Find X, Y , Z in terms of A,
0 B 0 Y
B, and C.
   
X Z A C
Solution. Since is the inverse of ,
0 Y 0 B
      
I 0 A C X Z AX AZ + CY
=I= = .
0 I 0 B 0 Y 0 BY

Then AX = I, BY = I, and AZ + CY = 0. The first equation AX = I

implies that A is invertible and X = A−1 . The second equation BY = I
implies that B is invertible and Y = B −1 . Finally,

AZ + CY = 0 =⇒ AZ = −CY = −CB −1 =⇒ Z = −A−1 CB −1

Therefore, the inverse

   
X Z A−1 −A−1 CB −1
= .
0 Y 0 B −1

System of linear equations P. 30 / 44

System of linear equations

I A system of linear equations (or a linear system) is a collection of one
or more linear equations

 a11 x1 + a12 x2 + · · · + a1n xn = b1


 a21 x1 + a22 x2 + · · · + a2n xn = b2


.. ..
. .






am1 x1 + am2 x2 + · · · + amn xn = bm

I The system of linear equations can be written in matrix form Ax = b,

where
a11 a12 ··· a1n x1 b1
     
 a11 a22 ··· a2n   x2   b2 
A=
 ... .. .. ..  , x=
 ... 
 and  ...  .
b= 
. . .

am1 am2 ··· amn xm bm

The matrix A is called the coefficient matrix (or matrix of coefficients)

of the system.
System of linear equations P. 31 / 44

System of linear equations

I The matrix

a11 a12 ··· a1n b1

 
   a21 a22 ··· a2n b2 
A b =
 ... .. .. .. .. 
. . . .

am1 am2 ··· amn bm

is called the augmented matrix of the system.

I A system of linear equations Ax = b has three possibilities.

I No solution
I Exactly one solution
I Infinitely many solutions
It is said to be consistent if it has either one or infinitely many
solution(s); a system is inconsistent if it has no solution.

System of linear equations P. 32 / 44

Example 7 Consider the system of linear equations

x1 − x3
(
2x2 + = 0
2x2 − 8x3 = 8
−4x1 + 5x2 + 9x3 = −9
[1]: import sympy as sp The solution is
x1
(
[2]: A = sp. = 29
,→Matrix([[1,-2,1],[0,2,-8],[-4,5,9]]); A
x2 = 16
[2]: x3 = 3
 
1 −2 1
0 2 −8
−4 5 9

[3]: b = sp.Matrix([[0],[8],[-9]]); b

[3]:
 
0
8
−9

[4]: # define x1, x2, x3

x1, x2, x3 = sp.symbols('x1 x2 x3')

[5]: # Solve Ax = b @ SymPy

sp.linsolve((A,b),x1,x2,x3)

[5]: {(29, 16, 3)}

System of linear equations P. 33 / 44

Example 7 Consider the system of linear equations

x1 − x3
(
2x2 + = 0
2x2 − 8x3 = 8
−4x1 + 5x2 + 9x3 = −9
[1]: import numpy as np The solution is
x1
(
[2]: A = np. = 29
,→array([[1,-2,1],[0,2,-8],[-4,5,9]]); A
x2 = 16
[2]: array([[ 1, -2, 1],
x3 = 3
[ 0, 2, -8],
[-4, 5, 9]])

[3]: b = np.array([[0],[8],[-9]]); b

[3]: array([[ 0],

[ 8],
[-9]])

[4]: # Solve Ax = b @ NumPy

np.linalg.solve(A,b)

[4]: array([[29.],
[16.],
[ 3.]])

System of linear equations P. 34 / 44


x2 − 4x3 = 8
Example 8 Solve the system 2x1 − 3x2 + 2x3 = 1
5x1 − 8x2 + 7x3 = 1

[1]: import sympy as sp [1]: import numpy as np

[2]: A = sp. [2]: A = np.

,→Matrix([[0,1,-4],[2,-3,2],[5,-8,7]]); A ,→array([[0,1,-4],[2,-3,2],[5,-8,7]]); A

[2]:
 
0 1 −4
2 −3 2 [2]: array([[ 0, 1, -4],
5 −8 7 [ 2, -3, 2],
[ 5, -8, 7]])
[3]: b = sp.Matrix([[8],[1],[1]]); b
[3]: b = np.array([[8],[1],[1]]); b
[3]:

8
1
1 [3]: array([[8],
[1],
[4]: # Solve Ax = b @ SymPy [1]])
x1, x2, x3 = sp.symbols('x1 x2 x3')
sp.linsolve((A,b),x1,x2,x3)
[4]: # Solve Ax = b @ NumPy
np.linalg.solve(A,b)
[4]: ∅

The system is inconsistent (has no [4]: array([[-9.00719925e+15],

[-7.20575940e+15],
solution). [-1.80143985e+15]])
System of linear equations P. 35 / 44

Example 9 Solve the system of linear equations


3x2 − 6x3 + 6x4 + 4x5 = −5
3x1 − 7x2 + 8x3 − 5x4 + 8x5 = 9
3x1 − 9x2 + 12x3 − 9x4 + 6x5 = 15

[1]: import sympy as sp So the general solution of the

system is
[2]: A = sp.Matrix([[0,3,-6,6,4],␣
,→[3,-7,8,-5,8], [3,-9,12,-9,6]]);A

[2]:
  x = −24 + 2x3 − 3x4
0 3 −6 6 4  1

3 −7 8 −5 8 x2 = −7 + 2x3 − 2x4
3 −9 12 −9 6
 x5 =
 4
[3]: b = sp.Matrix([[-5],[9],[15]]); b x3 , x4 are free.
[3]:
 
−5
9
the solution can also be rewritten
15 as
[5]: x1, x2, x3, x4, x5 = sp.symbols('x1 x2 x3␣
   
x1 −24 + 2x3 − 3x4
,→x4 x5')
x2   −7 + 2x3 − 2x4 
x = x3  =  x3 .
   
[6]: sp.linsolve((A,b),x1,x2,x3,x4,x5)
x4   x4 
[6]: {(2x3 − 3x4 − 24, 2x3 − 2x4 − 7, x3 , x4 , 4)} x5 4

System of linear equations P. 36 / 44

[7]: import numpy as np

A = np.array([[0,3,-6,6,4], [3,-7,8,-5,8], [3,-9,12,-9,6]]); A

[7]: array([[ 0, 3, -6, 6, 4],

[ 3, -7, 8, -5, 8],
[ 3, -9, 12, -9, 6]])

[8]: b = np.array([[-5],[9],[15]]); b

[8]: array([[-5],
[ 9],
[15]])

[9]: # np.linalg.solve cannot solve infinity many solution case

np.linalg.solve(A,b)

---------------------------------------------------------------------------
LinAlgError Traceback (most recent call last)
/var/folders/wh/1v8jgf9d5gbfykybbcypgbt40000gn/T/ipykernel_26654/351369431.py in <module>
1 # np.linalg.solve cannot solve infinity many solution case
----> 2 np.linalg.solve(A,b)

<__array_function__ internals> in solve(*args, **kwargs)

/opt/anaconda3/lib/python3.9/site-packages/numpy/linalg/linalg.py in solve(a, b)
378 a, _ = _makearray(a)
379 _assert_stacked_2d(a)
--> 380 _assert_stacked_square(a)
381 b, wrap = _makearray(b)
382 t, result_t = _commonType(a, b)

LinAlgError: Last 2 dimensions of the array must be square

Elementary row operations P. 37 / 44

Row Echelon Form

The leading entry of a row refers to the leftmost nonzero entry.
     
0 3 −6 6 4 −5 3 −9 12 −9 6 15 3 −9 12 −9 6 15
3 −7 8 −5 8 9 3 −7 8 −5 8 9 0 2 −4 4 2 −6
3 −9 12 −9 6 15 0 3 −6 6 4 −5 0 0 0 0 1 4

A rectangular matrix is in row echelon form (REF) if it has the following

two properties:
1. If both the k-th and (k + 1)-th rows have nonzero entries, the number
of zeros before the leading entry in the k-th row is strictly smaller than
the number of zeros before the leading entry in the (k + 1)-th row.
" # " # " #
∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗
0 3 ∗ ∗ ∗ ∗ 0 0 0 4 ∗ ∗ 0 3 ∗ ∗ ∗ ∗
0 0 0 4 ∗ ∗ 0 3 ∗ ∗ ∗ ∗ 0 4 ∗ ∗ ∗ ∗
0 0 0 0 ∗ ∗ 0 0 0 0 ∗ ∗ 0 0 0 ∗ ∗ ∗

(Satisfied) (Not satisfied) (Not satisfied)

2. If there are rows whose entries are all zero, they are below the rows
having nonzero entries.
" # " # " #
∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗
0 3 ∗ ∗ ∗ ∗ 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 4 ∗ ∗ 0 3 ∗ ∗ ∗ ∗ 0 4 ∗ ∗ ∗ ∗
0 0 0 0 0 0 0 0 0 4 ∗ ∗ 0 0 0 0 0 0

(Satisfied) (Not satisfied) (Not satisfied)

Elementary row operations P. 38 / 44

Reduced Row Echelon Form

If a matrix in row echelon form satisfies the following additional conditions,
then it is in reduced row echelon form (RREF) :
3. The leading entry in each nonzero row is 1.
4. Each leading 1 is the only nonzero entry in its column.
" # " # " #
1 0 ∗ 0 ∗ ∗ 1 0 ∗ ∗ ∗ ∗ 2 0 ∗ 0 ∗ ∗
0 1 ∗ 0 ∗ ∗ 0 1 ∗ ∗ ∗ ∗ 0 3 ∗ 0 ∗ ∗
0 0 0 1 ∗ ∗ 0 0 0 1 ∗ ∗ 0 0 0 4 ∗ ∗
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

(RREF) (Not RREF) (Not RREF)

Uniqueness of the Reduced Row Echelon Form

Each matrix is row equivalent to one and only one reduced row echelon form.
Remark A matrix can have many different row echelon forms.

Pivot position / column

I A pivot position in a matrix A is a location in A that corresponds to a
leading 1 in the reduced row echelon form of the matrix A.
I A pivot column is a column of A that contains a pivot position.
Elementary row operations P. 39 / 44

Elementary row operations

1. (Replacement) Replace one row by the sum of itself and a multiple of
another row.
" # " #
a1 a2 a3 a4 b a1 a2 a3 a4
R2 − a1 R1 → R2
1 b b b
b1 b2 b3 b4 ∼ 0 b2 − a1 a2
1
b3 − a1 a3
1
b4 − a1 a4
1
c1 c2 c3 c4 c1 c2 c3 c4

2. (Scaling) Multiply all entries in a row by a nonzero constant.

" # " #
a1 a2 a3 a4 a1 a2 a3 a4
k×R2 → R2
b1 b2 b3 b4 ∼ k b1 k b2 k b3 k b4
c1 c2 c3 c4 c1 c2 c3 c4

3. (Interchange) Interchange two rows.

" # " #
a1 a2 a3 a4 b1 b2 b3 b4
R1 ↔R2
b1 b2 b3 b4 ∼ a1 a2 a3 a4
c1 c2 c3 c4 c1 c2 c3 c4

Elementary row operations P. 40 / 44

Example 10 [Re-visit] Consider the system of linear equations

x1 − x3
(
2x2 + = 0
2x2 − 8x3 = 8
−4x1 + 5x2 + 9x3 = −9

[1]: A = sp.Matrix([[1,-2,1,0], [0,2,-8,8],␣ The RREF of the augmented matrix

,→[-4,5,9,-9]]); A
is
[1]:
 
1 −2 1 0
0 2 −8 8    
1 −2 1 0 1 0 0 29
−4 5 9 −9
0 2 −8 8 ∼ ··· ∼ 0 1 0 16 .
−4 5 9 −9 0 0 1 3
[2]: # RREF
A.rref()
The solution is
[2]: (Matrix([
[1, 0, 0, 29],
x1
(
[0, 1, 0, 16], = 29
[0, 0, 1, 3]]),
(0, 1, 2))
x2 = 16
x3 = 3
(The first is the reduced row echelon form, and
the second is a tuple of indices of the pivot columns.)
Elementary row operations P. 41 / 44

Example 11 [Re-visit] Solve the system of linear

equations
 [1]: A = sp.
3x2 − 6x3 + 6x4 + 4x5 = −5 ,→Matrix([[0,3,-6,6,4,-5],␣
3x1 − 7x2 + 8x3 − 5x4 + 8x5 = 9 ,→[3,-7,8,-5,8,9],␣
3x1 − 9x2 + 12x3 − 9x4 + 6x5 = 15 ,→[3,-9,12,-9,6,15]]);A

[1]:
 
Solution. Consider the augmented matrix 0 3 −6 6 4 −5
  3 −7 8 −5 8 9
A b . 3 −9 12 −9 6 15

    [2]: # RREF
0 3 −6 6 4 −5 1 0 −2 3 0 −24
A.rref()
3 −7 8 −5 8 9 ∼ 0 1 −2 2 0 −7 .
3 −9 12 −9 6 15 0 0 0 0 1 4
[2]: (Matrix([
The associated system now is [1, 0, -2, 3, 0, -24],
[0, 1, -2, 2, 0, -7],
[0, 0, 0, 0, 1, 4]]),
x1 − −24
(
2x3 + = 3x4 (0, 1, 4))
x2 − 2x3 + = 2x4 −7
x5 = 4
So the general solution of the system is    
 x1 −24 + 2x3 − 3x4
 x1
 = −24 + 2x3 − 3x4 x2   −7 + 2x3 − 2x4 
x2 = −7 + 2x3 − 2x4 ⇐⇒ x = x3  =  x3 .
   
 x5 =
 4 x4   x4 
x3 , x4 are free. x5 4

Elementary row operations P. 42 / 44

Example 12 [Re-visit] Solve the system of linear equations


3x2 − 6x3 + 6x4 + 4x5 = 0
3x1 − 7x2 + 8x3 − 5x4 + 8x5 = 0
3x1 − 9x2 + 12x3 − 9x4 + 6x5 = 0
 
Solution. Consider the augmented matrix A b .
   
0 3 −6 6 4 0 1 0 −2 3 0 0
3 −7 8 −5 8 0 ∼ ··· ∼ 0 1 −2 2 0 0 .
3 −9 12 −9 6 0 0 0 0 0 1 0

So the general solution of the system is

−
 x1 = 2x3 3x4
x2 = 2x3 − 2x4
x5 = 0
x3 , x4 are free.

The above solution can be rewritten in the parametric vector form

    2 −3
x1 2x3 − 3x4
x2 2x3 − 2x4 2 −2
x= = x3 x3  = x3 1 +x4  0  = x3 v1 + x4 v2 .
x4 x4 0 1
x5 0 0 0
|{z} | {z }
v1 v2
This shows that every solution is a linear combination of v1 and v2
(Explain later).
Elementary row operations P. 43 / 44

[1]: A = sp.Matrix([[0,3,-6,6,4],␣ Method 2

,→[3,-7,8,-5,8], [3,-9,12,-9,6]]); A
[4]: A.rref()

[1]:
 
0 3 −6 6 4
3 −7 8 −5 8 [4]: (Matrix([
3 −9 12 −9 6 [1, 0, -2, 3, 0],
[0, 1, -2, 2, 0],
[0, 0, 0, 0, 1]]),
(0, 1, 4))
[2]: # define 3x1 zero vector
b = sp.zeros(3,1); b
Method 3

[2]: [5]: A.nullspace()


0
0
0
[5]: [Matrix([
[2],
[2],
[1],
Method 1
[0],
[0]]),
[3]: x1,x2,x3,x4,x5 = sp.symbols('x1 x2 x3 x4␣ Matrix([
,→x5') [-3],
[-2],
sp.linsolve((A,b),x1,x2,x3,x4,x5)
[ 0],
[ 1],
[ 0]])]
[3]: {(2x3 − 3x4 , 2x3 − 2x4 , x3 , x4 , 0)}

Homogeneous system P. 44 / 44

Homogeneous System
I A system of linear equation is said to be homogeneous if it can be
written in the form
Ax = 0,
where A is an m × n matrix and 0 is the zero vector in Rm .
I A homogeneous system Ax = 0 always has at least one solution,
namely, x = 0. This zero solution is usually called the trivial solution.
I A nonzero vector x that satisfies Ax = 0 is called a nontrivial solution.

Properties of homogeneous system

I A homogeneous system Ax = 0 is always consistent.
I A homogeneous equation Ax = 0 has a nontrivial solution if and only if
the equation has at least one free variable.
I Suppose A is a square matrix. Then Ax = 0 has a nontrivial solution if
and only if A is singular.
I If A is an m × n matrix with m < n, then Ax = 0 has a nontrivial
solution.