Numerical Methods

Simultaneous Linear Equations

Topic: Gaussian Elimination
Gaussian Elimination
Basic Principles
• The general description of a set of linear equations in the matrix
form: [A][X] = [B]
• Where, [A] is ( m x n ) matrix, the [X] is a ( n x 1 ) vector, and
the [B] is a (m x 1 ) vector.
• How we solve such a system:

• Write the equations in natural form

• Identify unknowns and order them
• Isolate unknowns
• Write equations in matrix form
Types of Matrix Formulation
Suppose we have an ( m x n ) Array

a11 x1  a12 x2  a13 x3    a1n xn  b1

a21 x1  a22 x2  a23 x3    a2 n xn  b2
a31 x1  a32 x2  a33 x3    a3n xn  b3
   
am1 x1  am 2 x2  am 3 x3    amn xn  bm

If m = n The solution of [A]{x} ={b} with n unknowns and m(n) equations

If m > n The system is overdetermined system (Least Square Problems)
If m < n The system is underdetermined system (Optimization Problems)
Matrix Representation
The set of n linear equations
a11 x1  a12 x2  a13 x3    a1n xn  b1
with n unknowns:
a21 x1  a22 x2  a23 x3    a2 n xn  b2
a31 x1  a32 x2  a33 x3    a3n xn  b3
   
am1 x1  am 2 x2  am3 x3    amn xn  bm

a11 a12 a13  a1n   x1  b1 

The matix form: a  a2 n   x2  b2 
 21 a22 a23
a31 a32 a33  a3n   x3   b3 
    
          
an1 an 2 an 3  ann   xn  bn 

Consistency
a11 a12 a13  a1n   x1  b1 
a a22 a23  a2 n   x2  b2 
 21
AX   b  a31 a32 a33  a3n   x3   b3 
    
          
an1  ann   xn  bn 
 an 2 an 3

• The problem is consistent, if a solution exists for the problem.

• The problem is inconsistent, if there is no solution for the problem.
Rank of Matrix
Rank of a matrix is the number of linearly
independent column vectors in the matrix.

For n x n matrix, A:
• If rank(A) = n and is consistent, A has an unique
solution exists
• If rank(A) = n and is inconsistent, A has no solution
exists
• If rank(A) < n and system is consistent, A has an
infinite number of solutions
Matrix
For an n x n system, rank(A) = n automatically
guarantees
that the system is consistent.

• The columns of A are linearly independent

• The rows of A are linearly independent
• rank(A) = n
• det(A) ~= 0
• A-1 exists;
• The solution to [A]{x} ={b} exist and is unique.
Matrix Definition
Consider Consider
y = -2.0 x + 6 y = -2 x + 6
y = 0.5 x + 1 y = -2 x + 5
There are 2 unknowns x , y and
There are 2 unknowns x , y rank is 1 and is inconsistent
and rank is 2

 2 1  x  6 2 1  x  6
 0.5 1  y   1 2 1  y   5
         
Gaussian Elimination
a11 x1  a12 x2  a13 x3    a1n xn  b1
a21 x1  a22 x2  a23 x3    a2 n xn  b2
a31 x1  a32 x2  a33 x3    a3n xn  b3
   
am1 x1  am 2 x2  am3 x3    amn xn  bm

Gaussian Elimination is one of the most popular techniques for

solving simultaneous linear equations of the form: [A][X] =[b]

The method consists of 2 steps

1. Forward Elimination of Unknowns.
2. Back Substitution

Let us learn the method first by examples:

Example 1:
Consider following three linear equations with three unknowns:
2.37 x1  3.06 x2  4.28 x3  1.76 (1)
1.46 x1  0.78 x2  3.75 x3  4.69 (2)
 3.69 x1  5.13x2  1.06 x3  5.74 (3)

First step is to divide by 2.37 so that the coefficient of x1 is one. Thus

x1  1.2911x2  1.8059 x3  0.7426 (4)
Now multiply this by -1.46 and adding it with Eq2.
 1.46 x1  1.885 x2  2.6366 x3  1.0842
 1.46 x1  0.780 x2  3.75 x3  4.69
0  2.6650 x2  6.366 x3  3.6058 (5)

Now multiply Eq.4 with 3.69 and adding it to Eq3.

0  9.8942 x2  5.6038x3  8.4802 (6)
Example 1:
Re-writing Equations 4, 5 and 6:
x1  1.2911x2  1.8059 x3  0.7426 (4)
(0) x1  2.6650 x2  6.3866 x3  3.6058 (5)
(0) x1  9.8942 x2  5.6038 x3  8.4802 (6)

The second step is to divide by -2.6650 so that the coefficient of x2 is one. Thus
0  x2  2.3965 x3  1.3530 (7)

Now multiply this by -9.8942 and adding it to Eq 6.

0  (0) x2  18.1077 x3  21.8671

dividing it by 18.1077, we get

0  (0) x2  x3  1.2076 (8)
Example 1:
Final form of Eq. 4, 7 and 8 is
x1  1.2911x2  1.8059 x3  0.7426 (4)
0 x2  2.3965 x3  1.3530 (7 )
0 0 x3  1.2076 (8)
Now x3 is known and we can back-substitute it into Eq. 7 to find x2.
x2  1.3530  2.3965 x3
 1.3530  2.3965 1.2076  1.5410

Similarly, we can find x1 using values of x2 and x3 from Eq. 4

x1  0.7426  1.2911x2  2.3965 x3
 0.7426  1.29111.5410  2.3965 1.2076
 0.9338
Therefore, the solution is given as
x1 = 0.9338 ; x2 = 1.5410 ; x3 = 1.2076
Forward Elimination
The goal of Forward Elimination is to transform the coefficient
matrix into an Upper Triangular Matrix

 25 5 1 25 5 1 
 64   0  4.8  1.56
 8 1   
144 12 1  0 0 0.7 
Forward Elimination
Linear Equations:
A set of n equations and n unknowns

a11x1  a12 x2  a13 x3  ...  a1n xn  b1 ( Eq.1 )

a21x1  a22 x2  a23 x3  ...  a2 n xn  b2 ( Eq.2 )

. .
. .
. .

an1x1  an 2 x2  an3 x3  ...  ann xn  bn

Forward Elimination
Transform to an Upper Triangular Matrix
(Eq.1)
a11x1  a12 x2  a13 x3  ...  a1n xn  b1
Step 1: Eliminate x1 in 2nd equation using equation 1 as the pivot equation

 Eqn1
   (a21 )
 a11 
Which will change the Eq. 1 as following:

a21 a21 a21

a21 x1  a12 x2  ...  a1n xn  b1
a11 a11 a11
Forward Elimination
Zeroing out the coefficient of x1 in the 2nd equation.
a21 a a ( Eq.1 )
a21 x1  a12 x2  ...  21 a1n xn  21 b1
a11 a11 a11
a21x1  a22 x2  a23 x3  ...  a2 n xn  b2 ( Eq.2 )

Subtract Equation 1 from Eq.2

 a21   a21  a
 a22  a12  x2  ...   a2 n  a1n  xn  b2  21 b1
 a11   a11  a11

a 21
Or Where
'
a 22  a 22  a12
a11

'
a22 x2  ...  a2' n xn  b2'
a 21
a 2' n  a 2 n  a1n
a11
Forward Elimination
Repeat this procedure for the remaining equations to reduce the set of
equations as

a11x1  a12 x2  a13 x3  ...  a1n xn  b1

'
a22 x2  a23
'
x3  ...  a2' n xn  b2'
'
a32 x2  a33
'
x3  ...  a3' n xn  b3'

. . .
a x  a x  ...  ann
'.
n.2 2
.'
. n3 3
'
xn  bn'
.
.
Forward Elimination
Step 2: Eliminate x2 in the 3rd equation.
Equivalent to eliminating x1 in the 2nd equation using equation 2 as the pivot
equation.

 Eqn2 
Eqn3     )
 (a32

 a22 
Forward Elimination
This procedure is repeated for the remaining equations to reduce the
set of equations as

a11x1  a12 x2  a13 x3  ...  a1n xn  b1

'
a22 x2  a23
'
x3  ...  a2' n xn  b2'
"
a33 x3  ...  a3" n xn  b3"
. .
. .
. .

an" 3 x3  ...  ann

"
xn  bn"
Forward Elimination
Continue this procedure by using the third equation as the pivot equation
and so on.
At the end of (n-1) Forward Elimination steps, the system of equations will
look like:

a11 x1  a12 x2  a13 x3  ...  a1n xn  b1

'
a22 x2  a23
'
x3  ...  a2' n xn  b2'

a x  ...  a x  b
"
33 3
"
n n
"
3

 n 1 n 1 
ann. xn.  bn
. .
. .
Forward Elimination
At the end of the Forward Elimination steps:

a11 a12 a13  a1n   x1   b1 

 ' ' '    
a22 a23  a2 n   x2   b2  ' 

 "
a33  a3" n   x3    b3" 
    
        
 ( n 1)  
ann x  b (n-1 ) 
 n   n 
Back Substitution
The goal of Back Substitution is to solve each of the equations using
the upper triangular matrix.

a11 a12 a13   x1   b1 

0 a    
a23  x 2   b2  
 22

 0 0 a33   x 3  b3 
Example of a system of 3 equations
Back Substitution
Start with the last equation because it has only
one unknown
bn( n 1)
xn  ( n1)
a nn
Solve the second from last equation (n-1)th using xn solved for
previously.
This solves for xn-1.
Back Substitution
Representing Back Substitution for all equations by formula

i 1
  aiji 1 x j
n
bi
j i 1
xi  i 1 For i=n-1, n-2,….,1
aii

bn( n 1)
and xn  ( n1)
a nn
Computer Program
function x = gaussE(A,b,ptol)
% GEdemo Show steps in Gauss elimination and back substitution
% No pivoting is used.
%
% Synopsis: x = GEdemo(A,b)
% x = GEdemo(A,b,ptol)
%
% Input: A,b = coefficient matrix and right hand side vector
% ptol = (optional) tolerance for detection of zero pivot
% Default: ptol = 50 * eps
%
% Output: x = solution vector, if solution exists
A=[25 5 1; 64 8 1; 144 12 1]
b=[106.8; 177.2; 279.2]
if nargin<3, ptol = 50*eps; end
[m,n] = size(A);
if m~=n, error('A matrix needs to be square'); end
nb = n+1; Ab = [A b]; % Augmented system
fprintf('\n Begin forward elimination with Augmented system;\n'); disp(Ab);

% --- Elimination
Computer Program (continued)
% program continued
for i =1:n-1
pivot = Ab(i,i);
if abs(pivot)<ptol, error('zero pivot encountered'); end
for k=i+1:n
factor = - Ab(k,i)/pivot;
Ab(k,i:nb) = Ab(k,i:nb) - (Ab(k,i)/pivot)*Ab(i,i:nb);
fprintf('Multiplication factor is %g\n',factor);
disp(Ab);
pause;
end
fprintf('\n After elimination in column %d with pivot = %f
\n\n',i,pivot);
disp(Ab);
pause;
end
% --- Back substitution
x = zeros(n,1); % Initializing the x vector to zero
x(n) = Ab(n,nb) /Ab(n,n);
for i= n-1:-1:1
x(i) = (Ab(i,nb) - Ab(i,i+1:n)*x(i+1:n))/Ab(i,i);
end