Numerical Methods for

Engineers
EE0903301
Lecture 16: Iterative Methods (Part 2)
Dr. Yanal Faouri
Nonlinear Systems

• The following is a set of two simultaneous nonlinear equations with two unknowns:
x12 + x1x2 = 10
x2 + 3x1 x22 = 57
• In contrast to linear systems which plot as straight lines, these equations plot as curves on an x2 versus x1 graph,
the solution is the intersection of the curves.
Successive Substitution

• Nonlinear systems of equations can be expressed generally as

f1(x1, x2, . . . , xn) = 0
f2(x1, x2, . . . , xn) = 0
...
fn(x1, x2, . . . , xn) = 0
• Therefore, the solution are the values of the x’s that make the equations equal to zero.
• A simple approach for solving this system is to use the same strategy that was employed for fixed-point iteration
and the Gauss-Seidel method.
• That is, each one of the nonlinear equations can be solved for one of the unknowns.
• These equations can then be implemented iteratively to compute new values which (hopefully) will converge on
the solutions.
• This approach, which is called successive substitution.
Example: Successive Substitution for a
Nonlinear System
• Problem Statement.
• Use successive substitution to determine the roots of
x12 + x1x2 = 10
x2 + 3x1 x22 = 57
• Note that a correct pair of roots is x1 = 2 and x2 = 3. Initiate the computation with guesses of x1 = 1.5 and x2 = 3.5.

• Solution. The first equation can be solved for ➔

• Then x2 = 57 − 3x1 x22
• Based on the initial guesses, the new value of x1 is:

• This result and the initial value of x2 = 3.5 can be substituted into x2 equation to determine a new value of x2:

x2 = 57 − 3(2.21429)(3.5)2 = −24.37516
Example: Successive Substitution for a
Nonlinear System
• Thus, the approach seems to be diverging. This behavior is even more pronounced on the second iteration:

x2 = 57 − 3(−0.20910)(−24.37516)2 = 429.709

• Obviously, the approach is deteriorating. Now we will repeat the computation but with the original equations set
up in a different format. For example, an alternative solution of x1 and x2 are:
Example: Successive Substitution for a
Nonlinear System
• Now the results are more satisfactory:

• Thus, the approach is converging on the true values of x1 = 2 and x2 = 3.

• The previous example illustrates the most serious shortcoming of successive substitution—that is, convergence
often depends on the way the equations are formulated.
• Additionally, even in those instances where convergence is possible, divergence can occur if the initial guesses
are insufficiently close to the true solution.
• These criteria are so restrictive that fixed-point iteration has limited utility for solving nonlinear systems.
Newton-Raphson

• Just as fixed-point iteration can be used to solve systems of nonlinear equations, other open root location
methods such as the Newton-Raphson method can be used for the same purpose.
• Recall that the Newton-Raphson method was predicated on employing the derivative (i.e., the slope) of a
function to estimate the root. a graphical derivation was used to compute this estimate.
• An alternative is to derive it from a first-order Taylor series expansion:
f(xi+1) = f(xi) + (xi+1 − xi) f ʹ(xi)
• where xi is the initial guess at the root and xi+1 is the point at which the slope intercepts the x axis.
• At this intercept, f(xi+1) equals zero and it can be rearranged to yield

• Which is the single-equation form of the Newton-Raphson method.

• The multi-equation form is derived in an identical fashion. However, a multivariable Taylor series must be used to
account for the fact that more than one independent variable contributes to the determination of the root.
Newton-Raphson

• For the two-variable case, a first-order Taylor series can be written for each nonlinear equation as:

• Just as for the single-equation version, the root estimate corresponds to the values of x1 and x2, where f1,i+1 and
f2,i+1 equal zero. For this situation, the above equations can be rearranged to give:
Newton-Raphson

• Because all values subscripted with i’s are known (they correspond to the latest guess or approximation), the
only unknowns are x1,i+1 and x2,i+1. Thus, the last two equations in the previous slide are a set of two linear
equations with two unknowns. Consequently, algebraic manipulations (e.g., Cramer’s rule) can be employed to
solve for x1,i+1 and x2,i+1

• The denominator of each of these

equations is formally referred to as
the determinant of the Jacobian of
the system.

• These equation are the two-equation

version of the Newton-Raphson
method. It can be employed
iteratively to home in on the roots of
two simultaneous equations.
Example: Newton-Raphson for a Nonlinear
System
• Problem Statement.
• Use the multiple-equation Newton-Raphson method to determine roots of
x12 + x1x2 = 10
x2 + 3x1 x22 = 57
• Initiate the computation with guesses of x1 = 1.5 and x2 = 3.5.
• Solution. First compute the partial derivatives and evaluate them at the initial guesses of x and y:
Example: Newton-Raphson for a Nonlinear
System
• Thus, the determinant of the Jacobian for the first iteration is ➔ 6.5(32.5) − 1.5(36.75) = 156.125
• The values of the functions can be evaluated at the initial guesses as
f1,0 = (1.5)2 + 1.5(3.5) − 10 = −2.5
f2,0 = 3.5 + 3(1.5)(3.5)2 − 57 = 1.625
• These values can be substituted into the Newton-Raphson equation to give:

• Thus, the results are converging to the true values of x1 = 2 and x2 = 3. The computation can be repeated until an
acceptable accuracy is obtained.
Newton-Raphson

• When the multi-equation Newton-Raphson works, it exhibits the same speedy quadratic convergence as the
single-equation version. However, just as with successive substitution, it can diverge if the initial guesses are not
sufficiently close to the true roots. Whereas graphical methods could be employed to derive good guesses for
the single-equation case, no such simple procedure is available for the multi-equation version.

• Although there are some advanced approaches for obtaining acceptable first estimates, often the initial guesses
must be obtained based on trial and error and knowledge of the physical system being modeled.

• The two-equation Newton-Raphson approach can be generalized to solve n simultaneous equations. To do this, a
first-order Taylor series can be written for the kth equation and rearranged as:
Jacobian Matrix

• Where the first subscript k represents the equation or unknown and the second subscript denotes whether the
value or function in question is at the present value (i) or at the next value (i + 1).
• Notice that the only unknowns in the previous equation are the xk,i+1 terms on the left-hand side. All other
quantities are located at the present value (i) and, thus, are known at any iteration.
• Consequently, the set of equations generally represented by the previous equation (i.e., with k = 1, 2, . . . , n)
constitutes a set of linear simultaneous equations that can be solved numerically by the elimination methods
discussed before.
• Matrix notation can be employed to express the previous equation as ➔ [J]{xi+1} = −{f} + [J]{xi}
• where the partial derivatives evaluated at i are written as the Jacobian matrix consisting of the partial
derivatives:
Jacobian Matrix

• The initial and final values are expressed in vector form as ➔ {xi}T = ⌊x1,i x2,i · · · xn,i⌋
• And {xi+1}T = ⌊x1,i+1 x2,i+1 · · · xn,i+1⌋
• Finally, the function values at i can be expressed as ➔ {f}T = ⌊f1,i f2,i · · · fn,i⌋
• [J]{xi+1} = −{f} + [J]{xi} can be solved using a technique such as Gauss elimination. This process can be
repeated iteratively to obtain refined estimates of the solution.
• Recall that the single-equation version of the Newton-Raphson method is:

• If [J]{xi+1} = −{f} + [J]{xi} is solved by multiplying it by the inverse of the Jacobian, the result is:
{xi+1} = {xi} − [J]−1{f}
Solution Using MATLAB

• The solution of the last example using MATLAB can be done as follow.
• After defining the initial guesses, we can compute the Jacobian and the function values as:
>> x = [1.5;3.5];
>> J = [2*x(1)+x(2) x(1) ; 3*x(2)^2 1+6*x(1)*x(2)]
➔ J = 6.5000 1.5000
36.7500 32.5000
>> f = [x(1)^2 + x(1)*x(2) - 10 ; x(2)+3*x(1)*x(2)^2 - 57]
➔ f = −2.5000
1.6250
• Then, we can implement {xi+1} = {xi} − [J]−1{f} to yield the improved estimates
>> x = x-J\f
➔ x = 2.0360
2.8439
MATLAB M-file to Implement Newton-Raphson Method
for Nonlinear Systems of Equations: newtmult

function [x,f,ea,iter]=newtmult(func,x0,es,maxit,varargin) if nargin<2,error('at least 2 input arguments

% newtmult: Newton-Raphson root zeroes nonlinear required'),end
systems if nargin<3||isempty(es),es = 0.0001;end
% [x,f,ea,iter] = newtmult(func,x0,es,maxit,p1,p2,...): if nargin<4||isempty(maxit),maxit = 50;end
% uses the Newton-Raphson method to find the roots of iter = 0;
% a system of nonlinear equations x = x0;
% input: while (1)
% func = name of function that returns f and J [J,f] = func(x,varargin{:});
% x0 = initial guess dx = J\f;
% es = desired percent relative error (default = 0.0001%) x = x - dx;
% maxit = maximum allowable iterations (default = 50) iter = iter + 1;
% p1,p2,... = additional parameters used by function ea=100*max(abs(dx./x));
% output: if iter>= maxit||ea<= es, break, end
% x = vector of roots end
% f = vector of functions evaluated at roots
% ea = approximate percent relative error (%)
% iter = number of iterations
MATLAB Function: fsolve

• The fsolve function solves systems of nonlinear equations with several variables. A general representation of its
syntax is:
>> [x, fx] = fsolve(function, x0, options)
• where [x, fx] = a vector containing the roots x and a vector containing the values of the functions evaluated at
the roots, function = the name of the function containing a vector holding the equations being solved, x0 is a
vector holding the initial guesses for the unknowns, and options is a data structure created by the optimset
function. Note that if you desire to pass function parameters but not use the options, pass an empty vector [] in
its place.
• The optimset function has the syntax
>> options = optimset('par1',val1,'par2',val2,...)
• where the parameter pari has the value vali. A complete listing of all the possible parameters can be obtained by
merely entering optimset at the command prompt. The parameters commonly used with the fsolve function are
display: When set to 'iter' displays a detailed record of all the iterations.
tolx: A positive scalar that sets a termination tolerance on x.
tolfun: A positive scalar that sets a termination tolerance on fx.
MATLAB Function: fsolve
• As an example, we can solve the last system using MATLAB built in function fsolve as follows:
f(x1, x2) = 2x1 + x1x2 − 10
f(x1, x2) = x2 + 3x1x22 − 57
• First, set up a function to hold the equations
function f = fun(x)
f = [x(1)^2+x(1)*x(2) - 10;x(2)+3*x(1)*x(2)^2 - 57];
• A script can then be used to generate the solution,
clc, format compact
[x,fx] = fsolve(@fun,[1.5;3.5])
➔ x = 2.0000
3.0000
➔ fx = 1.0e−13 *
0
0.1421