CHAPTER III SYSTEM OF EQUATIONS

3.1 Matrix Notation

 We often have the need to solve a linear system of equations
 a11x1 + a12 x2 + ... + a1n xn = b1
a x + a x + ... + a x = b
 21 1 22 2 2n n 2

 ...
an1x1 + an2 x2 + ... + ann xn = bn

aij = constant coefficients,

bj = constant vector ( known);
xj = solution vector ( unknown till solved).
 For convenience, we write the above as
 a11 a12 ... a1n   x1   b1 
a ... a 2n   x 2  b2 
 21 a 22 =
 ... ... ... ...   ...   ... 
    
 a n1 a n2 ... a nn   x n  bn 

or Ax=b
where A =[ aij] is a nxn matrix,
x and b are nx1 matrices or n-dimension vectors;
they are written in column form.

Notes organized by Prof. Renwei Mei, University of Florida 1

 Definitions of numerous matrices
◊ An identity matrix or unit matrix I is

 1 0 ... 0 
 0 1 ... 0 
I = ... ... ... ...
 
 0 0 ... 1 
where the diagonal elements = 1,
and the off-diagonal elements = 0.
Note: A I = A.

 d1 0 ... 0 
◊ Diagonal matrix:
 0 d2 ... 0 
 ... ... ... ... 
 0 0 ... dn 
 u11 u12 ... u1n 
◊ Upper-triangular matrix U: U=
 0 u22 ... u2n 
 ... ... ... ... 
 0 0 ... unn 

 ll1121 l 022 ... 0

... 0

◊ Lower-triangular matrix L: L=  ... ... ... ... 
 l n1 l n2 ... l nn 
◊ Tri-diagonal matrix is in the form of:

 db12 ad12 a02 00 … 0 

... 0
 0 b3 d3 a3 ... 0 
 ... ... ... ... ... ... 
 0 ... 0 bn-1 dn-1 an-1 
 0 0 ... 0 bn dn 

Notes organized by Prof. Renwei Mei, University of Florida 2

 * Tri-diagonal system equations are in the form of:
d1x1 + a1x2 = c1
b2x1 + d2x2 + a2x3 = c2
b3x2 + d3x3 + a3x4 = c3
b4x3 + d4x4+ a4x5 = c4
... ...
bn-2xn-3+dn-2xn-2 + an-2xn-1 = cn-2
bn-1xn-2 +dn-1xn-1 + an-1xn = cn-1
bnxn-1 + dn xn = cn
(This is widely encountered in solving a 2nd order ODE)
 a11 a21 ... an1 
T   T
◊ Transpose of A: A = a12 a22 ... an2  ; i.e., A = [aji].
 ... ... ... ... 
 
a1n a2n ... ann 

e.g. Using transpose, we can write a column vector x or b as

T
x = (x1, x2, x3,…, xn)
◊ Symmetric matrix: aji = aij for all i and j.
n
◊ Trace of A: tr[A] =  aii = sum of diagonal elements.
i =1

◊ Determinant of A:
n=2: det[A] = |A| = a11 a22 - a12 a21
n
n≥3: det[A] = |A| =  (−1)
1+ j a1j |M1j|
j =1

Minor Mij is an (n-1)x(n-1) matrix obtained from A

by striking out its ith row and jth column.
(−1) i + j |Mij| is called the cofactor of elemen aij.
Notes organized by Prof. Renwei Mei, University of Florida 3
  5 7 3 4 
 
3 
9 2 3 8 2 3
 8 9 2
Example: det   =55 3 8 -7 1 3 8
1 5 3 8 
  6 9 0 3 9 0
 3 0 
 6 9

8 9 3 8 9 2
+ 3 1 5 8 - 4 1 5 3 = -36
3 6 0 3 6 9
-1 -1
◊ The inverse, A , of A is defined as A A = I.
Example:
 1 − 1 3  1 −1 −1 

= − 6 / 5 7 / 5 4 / 5
-1
A =  2 1 2  A
   
− 2 − 2 1 − 2 / 5 4 / 5 3 / 5
-1 -1
 A A = I (you can verify it by multiplying A with A)
Inversion of 2x2 matrices can be done easily as follows:
−1
−1 a b  1  d − b
=  =
ad − bc − c a 
A
c d 
-1
For n>2, it is easier to compute A numerically.
-1
◊ The matrix A is non-singular if A exists.
-1
◊ If A is not singular, Ax = b x = A b
◊ If A is singular, Ax=b  x may have multitude or no solution.
◊ Eigenvalues of an nxn matrix are the roots, , of the nth order
polynomial
det (A - I) = 0.
* If A is symmetric, all 's are real.

Notes organized by Prof. Renwei Mei, University of Florida 4

 * In some applications, 's may be repeated or complex valued.

* Example: Consider the matrix

 4 − 1
A =   .
 − 2 3 
Soln: The equation: det (A - I) = 0 translates into
4 −  −1
− 2 3−
= (4 −  )(3 −  ) − 2 = 0

which is equivalent to 2 − 7 + 10 = 0 .
The solution is:  = 2 &  = 5

◊ In general, for a square matrix A of order n, the equation

det(A − I n ) = 0
will give all eigenvalues of A.
This equation is called the characteristic equation or
characteristic polynomial of A. It is a polynomial
function in  of degree n.
The square matrix A of order n will not have more than n
eigenvalues.
◊ Scalar product of vectors b1 and b2:
n
b1  b2 =  b1j b2j or simply b1  b2 = b1j b2j.
j =1

It is also called inner product of vectors b1 and b2 .

◊ Unit vector b: b  b = 1, i.e. a unit vector b has a length of 1.

Notes organized by Prof. Renwei Mei, University of Florida 5

3.1.2 Matrix Operations
◊ Addition: C=A+B  cij = aij + bij

* Examples:

a b   e f  a  e b  f 
c d    g =
h  c  g d  h 
  

1 2 5 6  6 8 
3 4 + 7 8 = 10 12 ,
     
1 2 5 6 − 4 − 4
3 4 − 8 7 =  − 5 − 3
     
* A + B = B + A (commutative property)
n
◊ Multiplication: P = AB  pij =  aik bkj
k =1

 pp1121 pp1222 ...... pp1m 

P=  ... ... ... ...  2m

 pm1 pm2 ... pmm 

 aa1121 aa1222 ...... aa1n2n   bb1121 bb1222 ...... bb2m
1n

=  ... ... ... ...   ... ... ... ... 
 am1 am2 ... amn   bn1 bn2 ... bnm 
That is, (m x m) = (m x n) (n x m)
In general, (m x k) = (m x n) (n x k)
* AB ≠ BA but AI = IA since I is an identity matrix.
Notes organized by Prof. Renwei Mei, University of Florida 6
 * Examples:

a b  ag + bj ah + bk ai + bl 
c d   g h i   
   j k l  = cg + dj ch + dk ci + dl 
 e f     eg + fj eh + fk ei + fl 
 

1 2 27 30 33 
3 4  7 8 9  
= 
e.g.   
 10 11 12  61 68 75 
5 6   95 106 117

3.1.3 Cramer's rule for system of equations

det(Ak)
◊ Given Ax=b  xk = det(A) , k =1, 2, ...n.

Ak = A with kth column replaced by b.

◊ Number of operations (multiplication/divisions) for each
determinant is O(n3) using Gauss elimination.
 Number of operations using Cramer's rule = O(n4)
 inefficient for solving x.
◊ Need to develop more efficient practical methods
(direct and iterative)
◊ In order to assess errors involved in solving system of eqns.,
we need to first define some measures to quantify the
magnitude of a matrix.

Notes organized by Prof. Renwei Mei, University of Florida 7

3.2 Matrix Norm
◊ If a= -10 and b =0.1 are two numbers, we can compare their
magnitudes to conclude that |a| > |b|.
◊ If u = 0.3 ex + 0.4 ey & v = 5 ex + 12 ey are two vectors,
 |u| = 0.5 & |v| = 13
so that we conclude |u| < |v|.
◊ How do we compare the "magnitude" of two matrices A & B?
Answer: need to use NORM || A ||.
◊ Norm || A || will be used to estimate the accuracy of the solution
x for system Ax = b.
3.2.1 Properties of a norm
A properly defined norm must possess the following properties:
i. || A || ≥ 0; || A || = 0 if and only if A=0
ii. || kA || = |k| || A ||
iii. || A + B || ≤ || A || + || B || -- triangle inequality.
iv. || AB || ≤ || A || || B ||
3.2.2 Lp-norm of a vector x
◊ For a vector x, we define the Lp-norm of x as
1/p n 1/p
|| x ||p = (|x1| p + |x2| p + ... + |xn| p) = (  |xi| p)
i =1

Notes organized by Prof. Renwei Mei, University of Florida 8

 * Commonly used norms:
p = 1: || x ||1 = sum of magnitude of each component.
n
2 2 2 1/2
p = 2: || x ||2 = || x ||e = x1 + x2 + ... + xn = (  xi2 )
i =1

is also called Euclidean norm of vector x.

p → ∞: || x || = maximum of {|x1|, |x2|, ... |xn|}
= maximum norm.
* e.g. For u = 5 ex + 12 ey, || u ||1 = 17, || u ||2 = 13, || u || = 12.
* In general, || u ||1 ≥ || u ||2 ≥ || u ||.
◊ Inner product: <u, v> = u1v1 + u2v2 + ... + unvn
◊ Cauchy-Schwartz inequality:
For all real or complex valued vectors u and v ,
2 2
|<u, v>| ≤ <u, u>  <v, v> or |<u, v>| ≤ || u ||2  || v ||2
3.2.3 Lp-norm of a matrix A
n
◊ Define || A ||1 = Max  | aij | = max. column sum
1 j  n i =1

n
|| A || = Max  | aij | = max. row sum
1 i  n j =1

n n 1/2
|| A ||e = (   aij2 ) = Euclidean or Frobenius norm
i =1 j =1

Note: || A ||2 = spectral norm = max{|i|1/2}

= smallest among all norms.
where i’s are the eigenvalues of (A*A).
& A* = conjugate transpose of A.
Notes organized by Prof. Renwei Mei, University of Florida 9
  5 − 5 − 7
− 4 2 − 4
◊ Example: consider A=  
− 7 − 4 5 
n
column sums:  | aij | = (16, 11, 16)
i =1

 maximum is 16
 || A ||1 = 16 (column norm)
n T
row sums:  | aij | = (17, 10, 16)
j =1
 maximum element is 17
 || A || = 17 (row norm)
Euclidean norm || A ||e
1/2
= (25+25+49+16+4+16+49+16+25)
1/2
= 225 = 15.
Sp ectral norm || A ||2: (let A* be conjugate transpose of A)
 90 − 5 − 54
A*A =  − 5 45 7 
 
− 54 7 90 

Eigenvalues of A*A: 35.7666, 44.5113, 144.7221

1/2
|| A ||2=144.7221 = 12.0301
You can also obtain this result using Matlab:
>> r= norm(A)
r=
12.0301
Notes organized by Prof. Renwei Mei, University of Florida 10
3.3 Gauss Elimination Method and Gauss-Jordan Method
◊ It is a direct method (in contrast to iterative method)
◊ It is good for dense matrix (i.e. aij have very few zeroes).
 Gauss elimination method
◊ Example:
3 1 − 1  x1   2  − − −1
1 4 1   x  = 12 − − − 2
Solve x in :    2  
2 1 2   x3  10 − − − 3
Soln.
3 1 − 1 2  --- 1
i. Augmented matrix: [A | b] = 1 4 1 12 --- 2
2 1 2 10 --- 3

ii. Reduction (to reduce A to an upper triangular matrix)

row1  row 1' 3 1 −1 2  ....1'
row 2 − row1* (1/ 3)  row 2'
0 3.6667 1.3333 11.3333 ....2'
*  
row 3 - row1'*(2/3) row 3' 0 0.3333 2.6667 8.6668  ....3'

R 1'  R 1" 3 1 −1 2  ....1"

R 2'  R 2" 0 3.6667 1.3333 11.3333 ....2"
* 0.3333  
R 3'-R 2'*( )  R 3" 0 0 2.5455 7.6365  ....3"
3.6667
* Now A has become an upper-triangular matrix.
iii. Back substitution
* Solve for x3: x3= 7.6365 / 2.5454 = 3.0 (<= 3")
* Solve for x2: x2= (11.3333-1.3333*3.0)/3.6667=2.0 (<= 2")
* Solve for x1: x1= (2+ x3 -x2)/3=1.0 (<= 1")
iv. Determinant: det(A) = 3*3.6667*2.5455 = 28.0

Notes organized by Prof. Renwei Mei, University of Florida 11

 Gauss -Jordan Method
* Consider the same example.
* Start from rows 1" to 3", NO BACK SUBSTITUTION!
* (row 1")/3.0, (row 2")/3.6667, (row 3")/2.5455 =>
1 0.3333 − 0.3333 0.6667 ....i
0 1 0.3636 3.0909 ....ii
 
0 0 1 3.0  ....iii

* (row i) - (row ii)*0.3333=>

1 0 − 0.4545 0.3636 ....i'
0 1 0.3636 3.0909 ....ii'
 
0 0 1 3.0  ....iii'

(row iii ' )  row 3 *

* (row ii' ) − 0.3636 (row 3*)  row 2 * :
(row i' ) + 0.4545 *(row 3*)  row1*

 1 0 0 1.0000  --1*
 0 1 0 2.0000  --2*
 0 0 1 3.0000  --3*
* The solution vector is
T T
x = (x1, x2, x3) = (1.0000, 2.0000, 3.0000) .
 Comments:
* Gauss elimination involves n3/3 multiplication/division.
* Gauss-Jordan method involves n3/2 multiplication/division.
=> Gauss-Jordan method involves 50% more work.
* Gauss-Jordan method is used mainly to find inverse matrix.

Notes organized by Prof. Renwei Mei, University of Florida 12

 Pivoting

 -0.002 4.0 4.0   x1   7.998 

Consider  -2 2.906 -5.387   x2  =  -4.481 
 3 -4.031 -3.112   x3   -4.143 
T T
◊ Exact solution is (x1, x2, x3) = (1, 1, 1) .
◊ Augmented matrix:

 -0.002 4.0 4.0 7.998  ---row 1

 -2 2.906 -5.387 -4.481  ---row 2
 3 -4.031 -3.112 -4.143  ---row 3
◊ Reduction using standard Gaussian elimination
& keep 4 significant digits:
-2 3
row 2 - -0.002 * row 1, row 3 - -0.002 * row 1:

 -0.002 4.0 4.0 7.998  ---row 1'

  -0 -3997 -4005 -8002  ---row 2'
 0 5996 5997 11990  ---row 3'
5996
row 3' - -3997 * row 2'  row 3"

 -0.002 4.0 4.0 7.998  ---row 1"

 -0 -3997 -4005 -8002  ---row 2"
 0 0 -10.5 -13.  ---row 3"
◊ Back substitution:
x3 = 13/10.5 = 1.238
x2 = (-8002 + 4005 x3 )/(-3997) = 0.762
x1 = (7.998 - 4.0 x3 -4.0 x2) / (-0.002) = 1.0
◊ Results are inaccurate due to round off error.

Notes organized by Prof. Renwei Mei, University of Florida 13

 ◊ Now applying partial PIVOTING:
rearrange A by exchanging row 3 with row 1 so that the
element with the largest magnitude in the column is now in
the pivot position (as diagaonl element)

 3 -4.031 -3.112 -4.143  ---R1

 -2 2.906 -5.387 -4.481  ---R2
 -0.002 4.0 4.0 7.998  ---R3
( R 1) / 3  ( R1' )  1 -1.344 -1.037 -1.381  ---R1'
( R 2) + 2  ( R1' )  R 2' :  0 0.219 -7.461 -7.243  ---R2'
( R3) + 0.002 ( R1' )  R3'  0 3.997 3.998 7.995  ---R3'
◊ Exchange R2' with R3', apply pivoting  diagonal dominance

 1 -1.344 -1.037 -1.381  ---R1"

  0 3.997 3.998 7.995  ---R2"
 0 0.219 -7.461 -7.243  ---R3"

 1 -1.344 -1.037 -1.381  --R1"'

◊ ( R2" ) / 3.997  R2" '  0 1.000 1.000 2.000  --R2"'
( R3") − 0.219* ( R2"' )  R3" '  0 0 -7.679 -7.679  --R3"'
◊ Back substitution
* Solve for x3: x3 = -7.679 /(-7.679)=1.000 from (R3"')
* Solve for x2: x2 = 1.000 from (R2"')
* Solve for x1: x1 = 1.000 from (R1"')
which are exact within 4 significant digits.
◊ It is important to apply pivoting to reduce round off error due to
dividing small numbers.
◊ You should study a Gauss elimination code to appreciate how the
pivoting, elimination, and reduction are implemented efficiently.
Notes organized by Prof. Renwei Mei, University of Florida 14
3.3.4 Operation count in Gaussian elimination method
◊ There are usually approximately the same number of
addition or subtraction (A/S) steps as multiplication or
division (M/D) steps in an algorithm.
◊ Therefore, since operation counts are only an approximation,
we will only consider the M/D steps.
◊ Now consider following 3x3 system as an example.
3 1 − 1  x1   2  − − − R1
1 4 1   x  = 12 − − − R2
* System of equations:    2  
2 1 2   x3  10 − − − R3

* Reduction (first column):

R1  R1' 3 1 −1 2  ....R1'
R 2 − R1* (1 / 3)  R 2'
0 3.6667 1.3333 11.3333 ....R2'
 
R3 - R1'*(2/3)  R3' 0 0.3333 2.6667 8.6668  ....R3'

◊ Counting:
• First, we need to find the coefficients such as 1/3 and 2/3…
o For each coefficient, it involves 1 division
o The 1st column involves (n-1) divisions
o The 2nd column involves (n-2) divisions.
o …
o Totally, it requires (n-1)+(n-2)+…+3+2+1 = n(n-1)/2
divisions in order to obtain all the coefficients.

Notes organized by Prof. Renwei Mei, University of Florida 15

 • Next, we need to reduce all the elements below diagonal to 0.
o To reduce (n-1) elements in the 1st column, it involves
n(n-1) multiplications for the augmented matrix.
(For LU decomposition, b is not involved. Hence it
involves (n-1) rows and (n-1) columns.
o To reduce (n-2) elements in the 2nd column, it involves
(n-1)(n-2) multiplications.
o …
n
o Totally, it needs  i(i-1) = (n3 –n )/3 multiplications.
i=1
n
(It needs  (i-1)(i-1) = n(n-1)(2n-1)/6 multiplications
i=1

without augmentation or involving b)

• Now, count the operations in back substitution step:
3 1 −1 2  ....1"
0 3.6667 1.3333 11.3333 ....2"
 
0 0 2.5455 7.6365  ....3"

x3 = 7.6365 / 2.5454 = 3.0

x2 = (11.3333-1.3333*3.0)/3.6667 …
o Solution for xn needs 1 division.
o Solution for xn-1 needs 1 multiplication & 1 division.
o Solution for x1 needs (n-1) multiplications & 1 division.
o Totally, it needs n(n+1)/2 operations.
• Add: n(n-1)/2+ (n3–n)/3+ n(n+1)/2 = n3/3+n2–n/3 operations

Notes organized by Prof. Renwei Mei, University of Florida 16

3.3.5 Operation count in Gaussian-Jordan method
◊ For the purpose of counting the number of operations, we
consider a slight variation of the procedure in the Gauss-Jordan
method.
* Start from the following nx(n+1) augmented matrix,
 a11 a12 a13 ... a1n b1 
a b2 
 21 a22 a23 ... a 2n
a31 a32 a33 ... a3n b3 
 
 ... ... ... ... ... ... 
 an1 an2 a n3 ... ann bn 
Step 1: reduce (n-1) elements below a11 to 0 =>
a11 a12 a13 ... a1n b1 
 0 a 
a23 ... a2 n b2 
 22
 0 a32  ... a3 n
a33 b3 
 
 ... ... ... ... ... ... 
 0 an 2 an 3 ... ann
 bn 

Each row requires 1 division (D) & n multiplications (M).

Number of operation = (n-1)(n+1) for (n-1) rows.
Step 2: reduce elements on 2nd column (except a′22) =>
a11 0 
a13 ... a1n b1 
 0 a 
a23 ... a2 n b2 
 22
 0 0 
a33 ... a3n b3 
 
 ... ... ... ... ... ... 
 0 0 an 3 ... 
ann bn 

There are still (n-1) rows; with n M/D operation.

Number of operation = (n-1)(n)

Notes organized by Prof. Renwei Mei, University of Florida 17

  ) 
Step n: reduce elements on nth column (except ann

a11 0 0 ... 0 b1

 0 a 0 ... 0 b 
 22 2 
0 0 a33  ... 0 b3
 
 ... ... ... ... ... ... 
 0 0 0 ... ann bn
There are still (n-1) rows; only two columns left: ajn & b.
need 1 dividion & 1 multiplication per row.
Number of operation = (n-1)(2)
At last, we need n divisions to get xj = ( bj / ajj )

Total number of operations:

(n-1){(n+1)+n+…+2} + n = n3/2+ n2–n/2 operations.
which is 50% more work than Gauss elimination for large n.

◊ If we use Cramer’s rule to compute xj by using Gauss

elimination to compute (n+1) determinants, it requires
(n+1)(n3/3+…) = O(n4/3) operations

– much more costly for large n!!!

◊ This is why Cramer's rule is a useful theoretical tool,

not a practical computational tool.

Notes organized by Prof. Renwei Mei, University of Florida 18

 Tri-diagonal algorithm (Thomas algorithm)
Consider the following tri-diagonal system of equations:

 db12 da12 a02 00 ...... 00   xx12   cc12 

 0 b3 d3 a3 ... 0   x3   c3 
 ... ... ... ... ... ...   ...  =  ... 
 0 ... 0 bn-1 dn-1 an-1   xn-1   cn-1 
 0 0 ... 0 bn dn   xn   cn 
It can be solved using the attached Fortran program.
The algorithm is described below.
* REDUCTION STEP
row 1 stays the same p 1 = d 1; f1 =c1
row 2 -b2/p1 x row 1: p2 = d2 -a1*b2/p1; f2 =c2 -f1b2/p1 ...
row j - bj/p1 x row 1: pj = dj- aj-1*bj/pj-1; fj =cj -fj-1bj/pj-1
(j=2, ... n)
so that the resulting system becomes
p1 x1 + a1x2 = f1
p2 x2 + a2 x3 = f2
...
pn-1xn-1 + anxn = fn-1
pnxn = fn
* BACK SUBSTITUTION
xn = fn / pn
xn-1 = (fn-1 -an-1 xn-1)/pn-1
...
xn-j = (fn-j -a n-j xn-j +1)/ pn-j, j=1,2, ... n-1

Notes organized by Prof. Renwei Mei, University of Florida 19

* This algorithm is called Thomas algorithm. It is widely used.
c ********************************************************
c The Triadiagonal system of equations is arranged in the following form
c bb(i)* x(i-1) + dd(i)*x(i) + aa(i)*x(i+1) = cc(i)
c for i = IL to IU
c (for the system considered here, IL=1, IU=n)
c The solution x(i) is returned in the array cc(i).
c
SUBROUTINE TDA(IL, IU, BB, DD, AA, CC)
PARAMETER (NYD=129)
IMPLICIT DOUBLE PRECISION (A-H,O-Z)
DIMENSION BB(NYD), DD(NYD), CC(NYD), AA(NYD)
C
C REDUCTION STEP
LP = IL+1
DO 10 I = LP,IU
R = BB(I)/DD(I-1)
DD(I)=DD(I) - R*AA(I-1)
CC(I)=CC(I) - R*CC(I-1)
10 CONTINUE
C
C BACK SUBSTITUTION
CC(IU) = CC(IU)/DD(IU)
DO 20 I = LP,IU
J = IU-I+IL
CC(J) = (CC(J)-AA(J)*CC(J+1))/DD(J)
20 CONTINUE
RETURN
END
c ********************************************************
Notes organized by Prof. Renwei Mei, University of Florida 20
3.4 L-U Decomposition and Matrix Inversion
3.4.1 L-U Decomposition based on Gaussian elimination

 3 1 -1   x1  2
◊ Back to 1 4 1   x2  =  12  using Gaussian
2 1 2   x3   10 
elimination.

3 1 -1  --- 1'
† (row 2) - (row 1)  1/3  row 2'  0 3.6667 1.3333  --- 2'
(row 3) - (row 1)  2/3 row 3'  0 0.3333 2.6667  --- 3'

0.3333 3 1 -1  --- 1"

† (row 3')-(row 2')x   0 3.6667 1.3333  --- 2"
3.6667
 0 0 2.5455  --- 3"
† Now put the "coefficients" back 

 1 0 0 3 1 -1 
A = 1/3 1 0   0 3.6667 1.3333  = L U
 2/3 0.0909 1   0 0 2.5455 
U = upper triangular matrix (with diagonal elements ≠ 1)
The non-zero coefficients are from Gaussian elimination.
L = lower triangular matrix (with diagonal elements = 1)
The rest of non-zero coefficients are from the coefficients
used in the Gaussian elimination to get U.

Notes organized by Prof. Renwei Mei, University of Florida 21

 ◊ Note: L & U are NOT unique.
We can express A as

3 0 0   1 1/3 -1/3 
A =  1 11/3 0   0 1 4/11  = L1 U1
 2 1/3 28/11   0 0 1 
3 0 0 
where L1 = L  0 11/3 0 
 0 0 28/11 
 1/3 0 0 
U1 = U  0 3/11 0 
 0 0 11/28 
◊ If we interchanged rows during the reduction step, the resulting
LU=A' is a permutation of A (i.e. by exchanging rows, A' can be
made back into A).
◊ Once LU decomposition is obtained, we can find,
det(A);
& the solution for Ax = b as follows:
Ax = b
 A'x = b' where A' = LU or LU x = b'
(due to pivoting=> row exchange so that
the elements of b must be exchanged if A≠A').
Let Ux = y.
Then LUx = b'
 Ly = b',

Notes organized by Prof. Renwei Mei, University of Florida 22

  3 1 -1   x1   2 
* Example: 1 4 1   x2  =  12 
2 1 2   x3   10 
Solution:

 3 1 -1   1 0 0 3 1 -1 
Note  1 4 1  =  1/3 1 0   0 3.6667 1.3333 
 2 1 2   2/3 0.0909 1   0 0 2.5455 
 1 0 0  y1   2 
L y =  1/3 1 0   y2  =  12 
 2/3 0.0909 1   y3   10 
 y1   2 
  y2  =  11.333 
 y3   7.6364 
Since Ux=y

3 1 -1   x1   2 
  0 3.6667 1.3333   x2  =  11.333 
 0 0 2.5455   x3   7.6364 
 x1  1
  x2  =  2 
 x3  3

Notes organized by Prof. Renwei Mei, University of Florida 23

3.4.2 Crout reduction
 Note: for this method, U has 1’s in the diagonal.
 Idea:
Consider a 4x4 matrix A. Decompose it into LU as

 aa1121 aa1222 aa1323 aa1424 

A=  a31 a32 a33 a34  =L U
 a41 a42 a43 a44 
 ll1121 l022 00 00   10 u112 uu1323 uu2414 
 
 l31 l32 l33 0   00 00 10 u134 
= P
 l41 l42 l43 l44 
with uii =1 for i=1 to 4.
◊ Procedure: (you need to multiply L & U out to get pij,
then compare each pij with aij in the original matrix A)
o First column in P gives the 1st column in L; it is easy to find:
l11 = a11, l21 = a21, l31 = a31, l41 = a41  li1 = ai1
o Now, the 1st row in P gives the 1st row in U as follows:
l11 u12 = a12, l11 u13 = a13, l11 u14 = a14  u1j = a1j/ l11
o Now, 2nd column in P gives the 2nd column in L:
l21 u12 + l22 = a22
l31 u12 + l32 = a32
l41 u12 + l42 = a42
or li1 u12 + li2 = a i2, i=2, 3, 4
so that li2 = ai2 - li1 u12, i=2, 3, 4
Notes organized by Prof. Renwei Mei, University of Florida 24
 o The 2nd row in P gives the 2nd row in U as:
l21u13 + l22 u23 = a23
l21u14 + l22 u24 = a24
so that u2j = (a2j - l21u1j)/l22, j=3, 4
o The 3rd column in P gives the 3rd column in L …
...
o Generalize to:
j −1
lij = aij -  lik ukj, j≤i, i=1, 2, ..., n
k =1

1 j −1
uij = (aij −  likukj ) , i≤j, j=2, 3, ..., n
lii k =1

◊ Note:

 ll1121 ul2212 uu1323 uu1424 

LU can be stored in a compact form as
 
 l31 l32 l33 u34 
 l41 l42 l43 l44 
to save memory—important for large systems.
◊ Pivoting in Crout reduction
It is needed to avoid small lii or lii = 0 in computing uij.
 Need to record how rows are interchanged.

Notes organized by Prof. Renwei Mei, University of Florida 25

 ◊ Example (of pivoting in Crout reduction)

0 2 1 --- 1
* Consider A = 1 0 0 --- 2
3 0 1 --- 3
* Interchange row 1 with row 3 to avoid dividing by 0.

3 0 1
A′ =  1 0 0 
0 2 1
* We can then proceed to compute L & U of A′.

 3 0 1/3 
Finally, LU =  0 2 1/2  ,
 1 0 -1/3 
* Suppose b = [5, -1, -2]T is the RHS of Ax = b.
Then since the original row 1 and row 3 are switched,
 b → b' = [-2, 5, -1]T.
Thus, we need to solve LU x = b' as

3 0 0   1 0 1/3   x1   -2 
0 2 0   0 1 1/2   x2  = b' =  5 
 1 0 -1/3  0 0 1   x3   -1 
The solution can be easily found as x = [-1, 2, 1]T.

Notes organized by Prof. Renwei Mei, University of Florida 26

3.4.3 Evaluation of determinant
◊ After LU decomposition using Gaussian elimination, say,
with lii =1, the determinant of A' is
det (A') = det (L) det (U) = det (U) = u11u22...unn
◊ However, A  A' through m time of row interchanges. Hence,
det (A) = (-1)m u11u22...unn .
◊ Thus, Gauss elimination is an effective method for det(A).
◊ The amount of work: O(n3/3) M/D

Notes organized by Prof. Renwei Mei, University of Florida 27

3.4.4 Matrix Inversion
◊ Inverse matrix can be obtained using Gaussian-Jordan method
1 − 1 2
* Consider: A= 3 0 1 .
1 0 2

* Need to find: A-1.

1 − 1 2 1 0 0

* Add unit vectors to A: 3 0 1 0 1 0
1 0 2 0 0 1

3 0 1 0 1 0

* Pivoting: R2 R 1  1 − 1 2 1 0 0
1 0 2 0 0 1

3 0 1 0 1 0

* Gaussian elimination  0 − 1 1.667 1 − 0.333 0
0 0 1.667 0 − 0.333 1

* Continue with Gauss-Jordan elimination to reduce the upper

triangular matrix to obtain identity matrix on the left of the
augmented matrix
1 0 0 0 0.4 − 0.2

 0 1 0 −1 0 1 
0 0 1 0 − 0.2 0.6 

the right part gives the inverse matrix of A.

 0 0.4 − 0.2
 1 
 A-1 = − 1 0
 0 − 0.2 0.6 

◊ No need to worry about row interchanges, no matter how often.

Notes organized by Prof. Renwei Mei, University of Florida 28
3.5 Condition Number and Error Analysis
3.5.1 Ill-conditioned system
 Consider the following 4x4 system:
10 7 8 7 32
7 23
5 6 5 
Ax= 
8
x = 33 =b
6 10 9 
   
7 5 9 10 31

* Exact solution: x = (1, 1, 1, 1)T

* Let x* = (-7.2, 14.6, -2.5, 3.1)T
 A x* = (31.9, 23.1, 32.9, 31.1)T
which differes from b by (0.1, -0.1,0.1, -0.1)T.
* Let x** = (0.18, 2.36, 0.65, 1.21)T
 A x** = (31.99, 23.01, 32.99, 31.01)T
which differes from b by (0.01, -0.01, 0.01, -0.01)T.
* Clearly, a poor approximation to the true solution can almost
satisfy the relevant equations! Scary, isn’t it?
* For completeness, here is some relevant information about A:
Determinant: det(A) = 1
25 -41 10 -6
 -41 68 -17 10 
Inverse: A =  10 -17 5 -3 
-1

 -6 10 -3 2 
1 0 0 0 10 7.0 8.0 7.00 10 7 8 7
 0.8 1 0 0   0 0.4 3.6 3.40   8 6 10 9 
LU:  0.7 0.25 1 0   0 0 2.5 4.25  = 7 5 9 10 =A′
 0.7 0.25 -0.2 1   0 0 0 0.10   7 5 6 5 
Eigenvalues:  = 0.01015, 0.84311, 3.85806, 30.28869.
Notes organized by Prof. Renwei Mei, University of Florida 29
  3.02 -1.05 2.53   -1.61 
◊ Consider A =  4.33 0.56 -1.78  , b =  7.23 
 -0.83 -0.54 1.47   -3.38 
 Exact solution: x = (1, 2, -1)T
◊ In general, for A x = b  x = A-1 b
Expect: small change in some elements of A or b
 small change in x.
◊ Now, change a11 from 3.02 to 3.00  ≤ 1% change in a11.

 Exact solution: x* = (0.73326, 6.52208, 1.1276)T

Difference: x - x* = (0.26674, -4.52208, -2.1276)T

or || x - x*||2 = 5.06583 while || x ||2 = 61/2 = 2.449.

|| x - x* ||2
Hence || x ||2 = 2.068 or 207% ! WHY?

◊ Look at Gaussian elimination of Ax = b, with PIVOTING,

keep 3 significant digits

 4.33 0.56 -1.74 7.23 

...   0 -1.44 3.77 -6.65 
 0 0 -0.00362 0.00962 
 x = (0.875, -2.33, -2.66)T ≠ xExact = (1, 2, -1)T
PROBLEM: small diagonal elements after Gaussian elimination:
-0.00362  loss of accuracy in the process  need more
SD.
◊ If we use 6 significant digits, accuracy is improved.
x = (0.9998, 1.9995, -1.0020) T.

Notes organized by Prof. Renwei Mei, University of Florida 30

  1.01 0.99   x1   2.02 
◊ Consider     =   =b
 0.99 1.01   x2   1.98 

Exact solution: x = (2, 0)T.

Now change b to b* = (1.98, 2.02) T  b - b* = (0.04, -
0.04)T

But, the new solution is x* = (0, 2) T  x - x* = (2, -2)T

4 4
y 1.01x + 0.99y = 2.02
y
3 3
0.99x + 1.01y = 1.98

2 2

1 1

0 0
1.01x + 0.99y = 1.98
-1 -1 0.99x + 1.01y = 2.02

-2 -2
-2 -1 0 x 1 2 3 4 -2 -1 0 x 1 2 3 4

◊ This kind of variation (a few %) in b or A is typical in

experimental measurements, but the resulting variation in x
of this kind is considered large.
◊ Such system is called "ill-conditioned".
◊ The extent of "ill-conditionedness" can be measured using
condition number of the matrix A – Cond(A).

Notes organized by Prof. Renwei Mei, University of Florida 31

3.5.2 Condition number
-1
◊ Definition: Cond (A) = || A || || A ||
 1.01 0.99  -1  25.25 -24.75 
◊ For A =  0.99 1.01  , A =  -24.75 25.25 
   
-1
 || A ||∞ = 2, || A ||∞ = 50.
Hence Cond(A) = 100. (This is not a small number!!)
◊ If Cond(A) is large, the solution will be sensitive to the change in
A or b or to ROUND-OFF ERROR (which is more
worrisome)!!!

 107 75 86 75  25 -41 10 -6
 -41 68 -17 10 
◊ For A =  8 6 10 9  , A-1 =  10 -17 5 -3 
 7 5 9 10   -6 10 -3 2 
use maximum column sum  ||A||1 = 33, ||A-1||1 =136
 cond(A) = 33*136 = 4216 (quite big!)
Using spectral norm, cond(A) =2984.1  similar magnitude.

 3.02 -1.05 2.53   5.6611 -7.2732 -18.550 

◊ For A = 4.33 0.56 -1.78  , A = 200.50 -268.26 -669.91 
-1

 -0.83 -0.54 1.47   76.851 -102.65 -255.88 

 cond(A) = 6.67*1138.676= 7595 based on max row sum.
 cond(A) =4293.5 based on spectral norm.
-1
◊ Note cond(A) 1 since || A A || = || I || =1
-1 -1
& || A A ||  ||A || || A ||
◊ How does cond(A) affect the solution accuracy of the system?

Notes organized by Prof. Renwei Mei, University of Florida 32

3.5.3 Error analysis and condition number
* Eqn. A x = b; (x denotes the exact solution)

* Numerical solution is x- which is an approximate solution

* Error in x: e = x - x-

* Residual of eqn. r = b - A x- = b - A (x - e) = A e
due to round-off error.
-1 -1 -1
 e=A r  || e || = || A r || ≤ || A || || r ||
* Also note: r = A e  || r || = || A e || ≤ || A || || e ||
|| r ||
 || e || ≥ || A ||

|| r || -1
Hence || A || ≤ || e || ≤ || A || || r || (1)

* We are interested in how large the relative error of the solution is:
|| e ||
|| x || = ?
* Ax=b  || b || ≤ || A || || x ||
-1 -1
* x=A b  || x || ≤ || A || || b ||
|| b || -1
 || A || ≤ || x || ≤ || A || || b || (2)

* Combining (1-2) 
1 || r || || e || -1 || r ||
-1 ≤ ≤ ||A || || A ||
|| A || || A || || b || || x || || b ||
1 || r || || e || || r ||
or Cond(A) || b || ≤ || x || ≤ Cond(A) || b ||

Notes organized by Prof. Renwei Mei, University of Florida 33

 * Typically, || r || is of the order of machine epsilon
-7
(e.g. ~10 for single precision)
and || b || ~ O(1) or larger for a non-homogeneous system.
* If || b || is close to zero, the system is nearly homogeneous which
means that the solution x should be close to 0.
* If Cond(A) = O(1), small round-off error
 small relative error in solution || e || / || x ||.
5
* If Cond(A) is large, say O(10 ), the relative error in solution,
-2
|| e || / || x ||, can be of O(10 ).
Further computation using x suffers significant loss of accuracy.
5 6
* If Cond(A) is of O(10 ) or O(10 ), one remedy is to use double
precision to obtain x. However, even double precision is not
14
enough if cond(A) = O(10 ).
* Cond(A) is a very important measure of the matrix system.
max(|i|)
* Practically, we can use the ratio to compute Cond(A)
min(|i|)
where 's are all eigenvalues of matrix A if we have the
knowledge of 's or their ratios. This ratio is equivalent to
Cond(A) based on spectral norm.

Notes organized by Prof. Renwei Mei, University of Florida 34

3.6 Iterative Method
3.6.1 When do we use the iterative method?
* Consider a sparse matrix with a lot of zero elements. If we use
direct method such as Gauss elimination method, since there
are a lot of zero elements already, a lot of effort will be
wasted using the general procedure of Gauss elimination.
* On the other hand, it is difficult to devise an algorithm similar to
Thomas algorithm (for tri-diagonal system) for a system such
as shown below where the matrix involves several non-zero
elements outside the three diagonals

 db1 da1 a0 0 ..... 0   xx12   cc12 

 2 b2 d2 a ... 0   x3   c3 
 ... ...3 ...3 ...3 ... ...   ...  =  ... 
 0 ... 0 bn-1 dn-1 an-1   xn-1   cn-1 
  0 ... 0 bn dn   xn   cn 
* Iterative methods take advantages of many zeroes in the matrix.
* Thus, iterative methods are often used for sparse matrix.
 It is important to have diagonal dominance
n
| aii | >  | aij |
j=1‚j≠i
* Pivoting before starting:

 6 -2 1   x1   11 
Consider  1 2 -5   x2  =  -1 
 -2 7 2   x3  5
It is important to have the diagonal elements as large as possible.

Notes organized by Prof. Renwei Mei, University of Florida 35

  switch row 2 with row 3

 6 -2 1   x1   11 
  -2 7 2   x2  =  5 
 1 2 -5   x3   -1 
3.6.2 Jacobi method
 Divide the ith row by aii, express xi as a linear function of other
x's for i=1, 2, 3.
11 2 1
x1 = 6 + 6 x2 - 6 x3 = 1.8333 + 0.3333 x2 - 0.1667 x3

5 2 2
x2 = 7 + 7 x1 - 7 x3 = 0.7143 + 0.2857 x1 - 0.2857 x3

1 1 2
x3 = 5 + 5 x1 + 5 x2 = 0.2 + 0.2 x1 + 0.4 x2

 Starting iteration using the following procedure:

x1n +1 = 1.8333 + 0.3333 x2n - 0.1667 x3n

x2n +1 = 0.7143 n
+ 0.2857 x1 - 0.2857 x3
n

x3n +1 = 0.2 n n
+ 0.2 x1 + 0.4 x2
Initial guess: x(0) = (0, 0, 0)T  x(1) = (1.833, 0.714, 0.2)T
x\ 0 1 2 ... 7
x1 0 1.833 2.038 2.085 2.00 ... 2.000
n x2n 0
3
0.714 1.181 1.053
4
1.00 ... 1.000
3 4
x3 0 0.2 0.852 1.080 1.03 ... 1.000
3 1
8

Notes organized by Prof. Renwei Mei, University of Florida 36

  Some important considerations
a11x1 + a12 x2 + a13 x3 + a14 x4 = b1
a21x1 + a22 x2 + a23 x3 + a24 x4 = b2
Ax=b  a31x1 + a32 x2 + a33 x3 + a34 x4 = b3
a41x1 + a42 x2 + a43 x3 + a44 x4 = b4

Rewrite the matrix equation

a a a b
x1 = − 12 x2 − 13 x3 − 14 x4 + 1
a11 a11 a11 a11
a a a b
x2 = − 21 x1 − 23 x3 − 24 x4 + 2
a22 a22 a22 a22
a a a b
x3 = − 31 x1 − 32 x2 − 34 x4 + 3
a33 a33 a33 a33
a a a b
x4 = − 41 x1 − 42 x2 − 43 x3 + 4
a44 a44 a44 a44
n +1
 x = Cxn + d ; Cii = 0 for general system

 a a a   b1 
 0 − 12 − 13 − 14  a 
a11 a11 a11
   11 
− a21 0
a
− 23 −
a24   b2 
 a a22 a22  a 
with C =  22  ; d =  22 
a a a b
 − 31 − 32 0 − 34   3 
 a33 a33 a33   a 33 
 a41 a a   b4 
− − 42 − 43 0   
 a44 a44 a44   a 44 

 For system of equations, convergence requirement on C (not A)

i  1,  = eig(C)
 This is very similar to fix-point iteration method.

Notes organized by Prof. Renwei Mei, University of Florida 37

  Note, when we use iterative method, a significant number of aij's
are zero => a small number of operations in aij*xj are needed.
3.6.3 Gauss-Seidel point iterative method
 Improve over Jacobi method by using the most recently updated
results on x = Cx + d ; Cii = 0 ,
n+1 n n
x1 = 1.8333 + 0.3333 x2 - 0.1667 x3
n+1 n+1 n
x2 = 0.7143 + 0.2857x1 - 0.2857 x3
n+1
(note: x1 is just recently obtained)
n+1 n+1 n+1
x3 = 0.2 + 0.2x1 + 0.4 x2
n+1 n+1
(both x1 & x2 are recently updated)

 Initial guess: x(0) = (0, 0, 0) T 

 x1(1) = 1.8333 + 0.3333 * 0 - 0.1667 * 0 = 1.8333

x2(1) = 0.7143 + 0.2857 * 1.8333 - 0.2857 * 0 = 1.238071

x3(1) = 0.2 + 0.2*1.8333 + 0.4*1.23807 = 1.06189

n x1 x2 x3
0 0 0 0
1 1.8333 1.23807 1.06189
2 2.06896 1.00202 1.014602
3 1.99817 0.99531 0.997756
4 1.99874 1.00028 0.99986
5 2.00005 1.00005 1.000031
6 1.99994 0.99997 0.999978

Notes organized by Prof. Renwei Mei, University of Florida 38

 7 1.99993 0.99998 0.999979
8 1.99993 0.99999 0.99998
9 1.99993 0.99999 0.99998
 Comparison between Gauss-Seidel (G-S) and Jacobi method
i. G-S converges faster than Jacobi.
ii. On a parallel machine, Jacobi's method is more appropriate
since only the nth iterative value are used on RHS.
n +1 n +1 n +1
iii. In parallel computing, x1 , x2 , x3 are all updated
simultaneously. A scheme like

x2n +1 = 0.7143 + 0.2857 x1n +1 - 0.2857 x3n

n +1 n +1
requires x1 to be obtained first before computing x2 .
This seriously compromises the main advantage of parallel
computing.
 Necessary and sufficient condition -- the magnitude of the largest
eigenvalue of C (not A) is less than 1.
 Fortunately, many engineering problems of practical importance
satisfy this requirement.
 Use partial pivoting to rearrange equations!

Notes organized by Prof. Renwei Mei, University of Florida 39

3.6.4 Relaxation method
n
 General algorithm description for Ax = b:  aij xj = bi
j=1
i-1 n
 aii xi = bi -  aij xj -  aij xj
j=1 j=i+1

n+1 1 i-1 n n n
Jacobi: xi = a ( bi -  aij xj -  aij xj )
ii j=1 j=i+1

n+1 1 i-1 n+1 n n

G-S: xi = a ( bi -  aij xj -  aij xj )
ii j=1 j=i+1
n
 Add & subtract xi to G-S scheme:

n+1 n 1 i-1 n+1 n n

 xi = xi + aii (b i -  aij xj -  aij xj )
j=1 j=i

 Introduce relaxation factor :

n+1 n 1 i-1 n+1 n n
xi = xi +  a (bi -  aij xj -  aij xj )
ii j=1 j=i

⎯ 1 i-1 n+1 n n
or let xn+1
i
= aii (b i -  aij xj -  aij xj )
j=1 j=i+1

n+1 n ⎯ n
 xi = xi +  ( xn+1
i
- xi )

Notes organized by Prof. Renwei Mei, University of Florida 40

  Optimal values for  is system dependent.
Some systems converge very fast with =1.8 while other
systems converges very slowly with this value of .
For systems of equations resulting from nonlinear ODE's or
PDE's, <1 is often used to establish convergence of
diverging iterative process.

 -41 -41 11 11   xx12   11 

* Example:  1 1 -4 1   x3  = 1
 1 1 1 -4   x4  1
Use relaxation method, set convergence at | xi - xi | < 10-5.
n+1 n

How does the number of iterations vary with ?

 Number of iteration
1.0 24
1.1 18
1.2 13
1.3 11
1.4 14
1.5 18
1.6 24
1.7 35
1.8 55
1.9 >100

Notes organized by Prof. Renwei Mei, University of Florida 41

◊ Application of relaxation method to solving Poisson (or Laplace)
Equations
* Consider
2 2
 
 u = f, ( = ). (1)
2 2
+
x 2 y 2

Here, we assume that f(x, y) is a known function for simplicity.

In practice, f(x, y) may be another unkown that needs to be coupled
with u in a very nonlinear manner ---such as in stream-function
vorticity formulation in fluid dynamics problems.
* Discretization of equations using central difference scheme:

y x

j+
1j y

j-1

i- i i+ x
1 1
 2u ui +1, j − 2ui, j + ui −1, j
|
2 (x=xi, y=yj)
= 2
+ O((x)2) (2)
x (x)

 2u ui, j +1 − 2ui, j + ui, j −1

|
2 (x=xi, y=yj)
= 2
+ O((y)2) (3)
y (y)

Notes organized by Prof. Renwei Mei, University of Florida 42

  u = f =>
2

ui +1, j − 2ui, j + ui −1, j ui, j +1 − 2ui, j + ui, j −1

2
+ 2
= fi,j (4)
(x) (y)

* If we use Gaussian elimination method to solve it directly, we

will end up with a sparse matrix with 5 bands. It will be
inefficient.
* Gauss-Seidel iteration:
n +1 n +1 n +1 n +1
u i +1, j − 2ui, j + ui −1, j
n n
ui, j +1 − 2ui, j + ui, j −1
+ = fi,j (5)
(x) 2 (y) 2

for the internal points: i=2 to nx-1 and j=2 to ny-1.

* Note: at i=1 and i=nx, we need to apply boundary conditions;
the same is true at j=1 and j=ny boundaries.
Hence, we only need to solve for the internal points.
* Suppose x=y=h. Then Eqn. (5) 

uin, +j 1 = (h 2 f i, j + uin−+11, j + uin, +j 1−1 + uin+1, j + uin, j +1 )

1
(6)
4
* You can apply relaxation to Eqn. (6).

Notes organized by Prof. Renwei Mei, University of Florida 43

3.7 System of Nonlinear Equations
◊ Consider a 2-equation system
Circle x2 + y2 = r2
to start with:
f1(x1, x2) = 0
f2(x1, x2) = 0 f1(x,y)=0
f2(x,y)=0
An example is the intersection of a
circle and an ellipse shown on the
right.
Ellipse (x/a)2 + (y/b)2 = 1

◊ For m variables, it can be cast in the vector form as f (x) = 0

with x = (x1, x2,..., xm)T,
& f = (f1, f2,..., fm)T,
roots: r = (r1, r2,..., rm)T.

An example of 3
surface intersection is
shown on the right.

Notes organized by Prof. Renwei Mei, University of Florida 44

3.7.1 Fixed-point iteration method
* Rearanging: f (x) = 0  x = h(x) and r = h(r),
* Iteration scheme: xn+1 = h(xn)
* Initial guess: x0 = (x1,0, x2,0)
(n+1)th iteration x1,n+1 = h1(x1,n, x2,n)
x2,n+1 = h2(x1,n, x2,n)
* Convergence behavior:
xn+1 - r = h(xn) - r = h(xn) - h(r)  h n  ( xn - r) + ...
 h1 h1 h1 
 x ...
x2 xm 
 1 
 h2 h2
...
h2 
where h =  x1 x2 xm  = mxm matrix
 ... ... ... ... 
 h hm hm 
 m ... 
 x1 x2 xm 

That is en+1 = xn+1 - r  hn  ( xn - r) = hn  en

so that || en+1 || ≤ || hn || || en ||
Let  = max || hn || in the domain of x, we have
|| en+1 || ≤  || en ||  || en+1 || ≤ n+1 || e0 ||.
It is clear that we need  <1 for convergence.
* Advantage of the method: easy to implement.
Disadvantage of the method: too slow to converge.
 Example:
Use fixed-point method to solve the following nonlinear system:

Notes organized by Prof. Renwei Mei, University of Florida 45

f1( x1, x2 , x3 ) = x12 + 50x1 + x22 + x32 − 200 = 0
f 2 ( x1, x2 , x3 ) = x12 + 20x2 + x32 − 50 = 0
f3 ( x1, x2 , x3 ) = − x12 − x22 + 40x3 + 75 = 0 1.E+01
sum |e(i)|
Solution: 1.E+00
1.E-01
* Rearrange (initial guess:
1.E-02
x0=y0=z0=2) 1.E-03
x1 = (200 − x12 − x22 − x32 ) /50 1.E-04
1.E-05
x2 = (50 − x12 − x32 ) /20
1.E-06
x3 = ( x12 + x22 − 75) /40
1.E-07
3 n
n +1
Sum |e(i)| =  | xi − xi |
n 0 5 10 15
i =1

n x1 x2 x3 Sum |e(i)|
0 2 2 2
1 3.76 2.1 -1.675 5.535
2 3.5729355 1.6528388 -1.41131 0.897916
3 3.6502092 1.7621168 -1.4875564 0.262798
4 3.6271619 1.7231574 -1.4642729 0.08529
5 3.6346066 1.7349801 -1.4718606 0.026855
6 3.6322621 1.7311631 -1.469487 0.008535
7 3.6330071 1.732364 -1.4702437 0.002703
8 3.6327712 1.7319822 -1.4700044 0.000857
9 3.632846 1.732103 -1.4700803 0.000272
10 3.6328223 1.7320647 -1.4700562 8.61E-05
11 3.6328298 1.7320769 -1.4700639 2.73E-05
12 3.6328274 1.732073 -1.4700614 8.65E-06
13 3.6328282 1.7320742 -1.4700622 2.74E-06
14 3.6328279 1.7320738 -1.4700620 8.69E-07
15 3.6328280 1.7320740 -1.4700620 2.76E-07

Notes organized by Prof. Renwei Mei, University of Florida 46

3.7.2 Newton's method for nonlinear equations
 Fixed-point iteration is too slow.
 Taylor series expansion for f(x): f(x) = f(xn) + fn(x - xn) + ...
Set f(x) = 0  fn  (xn+1 - xn) = - f(xn) (*)
 f1 f1 f1 
 x ...
x2 xm 
 1 
 f 2 f 2
...
f 2  Jacobian
Denote G( x n ) = fn =  x1 x2 xm  =
matrix
 ... ... ... ... 
 f m f m f m 
 ... 
 xm xm xm 

Eqn. (*)  G( x n )  xn+1 = G( x n )  xn - f(xn)

This becomes a linear system for xn+1.
 xn+1 = xn - G −n1  f(xn) (**)

The idea is very similar to Newton’s method for one variable.

 Convergence:
2
|| en+1 || ≤ B || en || ; quadratic.
 Problem – cost is high!
# of function evaluation in each iteration: m
2
# of derivative evaluation in each iteration: m
# of algebraic operation (M/D+A/S) for solving a linear
2
system in each iteration: 3 m3
--- the most expensive part of the algorithm.

Notes organized by Prof. Renwei Mei, University of Florida 47

  Practical remedy:
Suppose it takes a total of n=15 iterations using this method.
2
If G −n1 is updated every iteration would involve
3
3 m algebraic

operations (M/D + A/S) in inverting G( x n ) in each iteration.

But if we only update G( x k ) after every 5 iterations for xk,
carry out the solution using LU decomposition,
store the LU matrix coefficient,
2
3
 only 3x 3 m algebraic operations in inverting G( x k )

2 3
instead of 15x 3 m .

The convergence will be slower since G( x k ) is not updated;

but the overall computational time is much less since the
most time consuming part of the whole process (matrix
inversion) is reduced.
This often results in significant saving.
 Reality: Newton’s method is the most widely used method for
solving system of nonlinear equations.

Notes organized by Prof. Renwei Mei, University of Florida 48