IJISET - International Journal of Innovative Science, Engineering & Technology, Vol.

4 Issue 1, January 2017

ISSN (Online) 2348 – 7968 | Impact Factor (2015) - 4.332
www.ijiset.com

Solving Nonlinear Least Squares Problem Using Gauss-Newton

Method
Wen Huey Lai1, Sie Long Kek2 and Kim Gaik Tay3
1
Department of Mathematics and Statistics, Universiti Tun Hussein Onn Malaysia,
Batu Pahat, Johor, Malaysia
2
Center for Research on Computational Mathematics, Universiti Tun Hussein Onn Malaysia,
Batu Pahat, Johor, Malaysia
3
Department of Communication Engineering, Universiti Tun Hussein Onn Malaysia,
Batu Pahat, Johor, Malaysia

Abstract fit in the least-squares sense minimizes the SSE which is

For a nonlinear function, an observation model is proposed to the difference between the observed value and the fitted
approximate the solution of the nonlinear function as closely as value provided by the model used [4]. The SSE is used
possible. Since the parameters in the model are unknown, a instead of the offset absolute values because the value of
successive approximation scheme is required. In this study, the
SSE allows the residuals to be treated as a continuous
Gauss-Newton algorithm is derived briefly. In doing so, a
residual between the nonlinear function and the model proposed
differentiable quantity. Minimizing SSE as the objective
is constructed, and the sum squares of error (SSE) is then function can be justified by consistency of the optimization
minimized. During the computational procedure, the gradient method. The approaches for solving nonlinear least
equations are calculated. By taking the gradient equation equals squares problems include Gauss-Newton method,
to zero and doing some algebraic manipulations, the normal Newton’s method, Quasi-Newton method, and Levenberg-
equations are resulted. These normal equations are the basis for Marquardt method are well-defined [5].
constructing the Gauss-Newton algorithm. For illustration,
nonlinear least squares problems with nonlinear model proposed
are solved by using the Gauss-Newton algorithm. In conclusion, 2. Problem Statement
it is highly recommended that the iterative procedure of the
Gauss-Newton algorithm gives the best fit solution and its Mathematical criterion is the basis of selecting a model in
efficiency is proven.
order to obtain an accurate estimated model. The SSE can
Keywords: Gauss-Newton Method, Sum of Squares Error,
Parameter Estimation, Jacobian matrix, Iteration Calculation.
be used to measure the variation between the real data and
the estimated value. In general, the least value of the SSE
presents the better estimate in the model. Each model may
1. Introduction give the different value of SSE. The best model has the
smallest SSE and it is a better model if compared to others
In a statistical model, the unknown parameters can be estimated models. If SSE is exactly zero, the appropriate
estimated by using maximum likelihood estimation and method can be known as the most efficient method.
least squares approach [1]. For these methods, the method However, this perfect situation is impossible to be
of least squares is a standard method to approximate the occurred in the reality. In other words, the exact model
solution of over determined systems. Least squares is which presents the real situation is hard to formulate. In
defined that the overall solution minimizes the sum squares this point of view, an approximation model, which has the
of error (SSE) made in the results of every single equation. least value of SSE, could be constructed. On the other
The first clearly and brief presentation of the method of hand, a scatter plot is used in deciding what kind of
least square was published by Legendre in 1805 [2]. mathematical model is more appropriate. This could
Nonetheless, Gauss declared that this method had been happen when any data set is expressed by mathematical
used before that of proposing by Legendre [3]. Basically, equation. By examining the character of the data plotted, a
least squares problems are divided into linear and suitable function can be chosen to represent the data of
nonlinear least squares problems, depending on the observation. Then, the values of the estimated parameters
linearity of the model used and the corresponding unknown could make a function of the model match the data of
parameters. This method of least squares is most observation as closely as possible.
commonly used in curve fitting. The reason is that the best

258
 IJISET - International Journal of Innovative Science, Engineering & Technology, Vol. 4 Issue 1, January 2017
ISSN (Online) 2348 – 7968 | Impact Factor (2015) - 4.332
www.ijiset.com

3. Gauss-Newton Method
J ( xk )T r ( xk ) + ( S ( xk ) + J ( xk )T J ( xk ))( x − xk ) =
0 . (7)
An optimization problem occurs when an objective
function is, either minimized or maximized, over a set of Rearrange (7), the following normal equation is obtained:
constraints. In our study, the nonlinear least squares
problem is formulated as an optimization problem without
( S ( xk ) + J ( xk )T J ( xk ))( x − xk ) =
− J ( xk )T r ( xk ) . (8)
constraints, where the SSE is defined as an objective
function. During the computation procedure, the SSE is
minimized, whereas the unknown parameters in the After some algebra manipulations, let x = xk +1 , then (8)
proposed model are determined in the optimal sense [5]. becomes
Here, the nonlinear least squares problem, which is an
unconstraint optimization problem, is defined by xk ( S ( xk ) + J ( xk )T J ( xk )) −1 J ( xk )T r ( xk ) .
xk +1 =− (9)

∑ (r ( x))
1 1 By neglecting the term S ( x) , the Gauss-Newton recursion
=
minn f ( x) = r ( x)T r ( x) i
2
(1)
x∈ℜ 2 2 equation is resulted as follows:
i =1

xk += xk + sk (10)
where f ( x) : ℜn → ℜ is the objective function, and ri ( x)
1
with
is the residual function, which is defined by
sk = −( J ( xk )T J ( xk )) −1 J ( xk )T r ( xk ) . (11)
ri ( x)= φ (ti , x) − yi , i= 1, 2,..., m, (2)
From the discussion above, the computation procedure of
the Gauss-Newton algorithm is summarized as follows.
where φ (ti , x) is the proposed function and yi is the
observation data. The gradient of f ( x) is given by Algorithm 1: Gauss-Newton Algorithm
Step 0 Set starting values of x0 , tolerance ε > 0,
g ( x) = J ( x)T r ( x) (3) and k = 0.
Step 1 Compute the residual function ri ( xk ), i = 0, , m,
where J ( x) is the Jacobian matrix of the residual function from (2).
r ( x) . The Hessian matrix is given by Step 2 Calculate the Jacobian matrices J ( xk ),
J ( xk )T J ( xk ) and (J ( xk )T J ( xk )) −1 .
m
Calculate the gradient g k = g ( xk ) from (3). If
∑ ( S ( x ) + ∇r ( x )
Step 3
=
G ( x) T
∇ri ( x)) (4)
|| g k || ≤ ε , stop.
i i
i =1
Step 4 Calculate sk from (11).
where Step 5 Update xk += xk + sk from (10). Set k = k + 1,
1
and go to Step 1.
S ( x) = ∇ 2 r ( x)r ( x) . (5)
Remarks:
Now, write the objective function f ( x) as the second (a) For the accuracy purpose, the value of the
order Taylor series expansion, which is the quadratic tolerance is chosen as small as possible.
model, as follows: (b) The initial value x0 is chosen arbitrarily.
(c) The inverse of the matrix (J ( xk )T J ( xk )) −1 exist
1
q ( x) = r ( xk )T r ( xk ) + J ( xk )T r ( xk )( x − xk ) during the calculation procedure.
2 (6)
1
+ ( S ( xk ) + J ( xk )T J ( xk ))( x − xk ) 2 .
2 4. Results and Discussion
Notice that differentiate (6) with respect to x and let it In this section, three examples are discussed, where the
equal to zero, we have corresponding models with the unknown parameters are
proposed. By using the Gauss-Newton approach, these

259
 IJISET - International Journal of Innovative Science, Engineering & Technology, Vol. 4 Issue 1, January 2017
ISSN (Online) 2348 – 7968 | Impact Factor (2015) - 4.332
www.ijiset.com

parameters are determined in which to give the least value

12
of the SSE. Then, the fitted line of each model and the
original data are presented. 10

4.1 Example 1: Exponential Data 8

Given a set of data points (ti , yi ) as below [6]: 6

y
4
Table 1: Observations
t y
2
1 10
2 5.49
0
3 0.89
4 -0.14 -2
5 -1.07 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6
t
6 0.84

Fig. 1 Curve fitting for observation.

The function

=y x1 exp(−t ⋅ x2 )
4.2 Example 2: Sinusoidal Data
is proposed to fit the data in Table 1. By using the initial A list of average monthly high temperatures for the city of
guesses Monroe Louisiana [7] is given below:

x1(0) = 10 and x2(0) = 0.001 , Table 2: Average High Temperature per Month
Month Temperature (in Fahrenheit)
January 56
the following model is obtained: February 60
March 69
=y 25.487 exp(−0.904 ⋅ t ) April 77
May 84
where the final values of x1 and x2 are 25.487041 and June 90
July 94
0.904330, respectively. From the calculation result, the August 94
value of SSE is 5.490956 with 22 iteration steps. The September 88
Jacobian is October 79
November 68
(−tx1 exp(−tx2 ), exp(−tx2 ))T . December 58

The graph of the data points and the model proposed is The relation between the temperature ( yi ) and the month
shown in Fig. 1. ( ti ) would be determined. For convenience, we label
January as 1, February as 2, and so forth. The function

=
y (t ) x1 sin( x2 t + x3 ) + x4

is proposed. With the initial guesses

( x1(0) , x2(0) , x3(0) , x4(0) ) = (15, 0.4, 10, 1),

the Gauss-Newton algorithm has drastically decreased the

value of SSE from 70648.991746 to 21.690917. The final
values of the parameters are

260
 IJISET - International Journal of Innovative Science, Engineering & Technology, Vol. 4 Issue 1, January 2017
ISSN (Online) 2348 – 7968 | Impact Factor (2015) - 4.332
www.ijiset.com

= =
x1 19.898335, x2 0.475806, (a) y (=
t ) x1t + x2 , (b) y (t ) = x1 exp( x2 t ) , and
= =
x3 10.771250, x4 74.500317. (c) y=
(t ) x1t + x2 . 2

The algorithm takes 16 number of iterations to converge. Taking the initial guesses for x1 and x2 as 6 and 0.3,
The Jacobian is
respectively, for these models, and after running the
algorithm, the optimal parameter values for the respective
( sin( x2 t + x3 ) , tx1 cos( x2 t + x3 ) , x1 cos( x2 + x3 ) , 1)T. model are given by

The model obtained is (a) x1 = 6.770238 and x2 = −3.478571,

(b) x1 = 7.000152 and x2 = 0.262077,
=
y (t ) 19.898 ⋅ sin(0.476 ⋅ t + 10.771) + 74.5 .
(c) x1 = 0.751331 and x2 = 7.828571 .
The fitted line of the model proposed and the observed
data are shown in Fig. 2. Hence, the models proposed are

95 (a) y= (t ) 6.770 ⋅ t − 3.479 ,

90 (b) y (t ) =7.000 ⋅ exp(0.262 ⋅ t )

85 (c) y (t=
) 0.751 ⋅ t 2 + 7.829 .
80
The respective Jacobians are
75
y

(a) J 1 = (t, 1)T,

J 2 = (exp(x 2 ⋅t), tx 1 exp(x 2 ⋅t))T,
70
(b)
65 (c) J 3 = (t2, 1)T.
60
The values of SSE for the respective model used are
55
0 2 4 6 8 10 12
t (a) 90.451548, (b) 6.013081, and (c) 0.312430.

Fig. 2 Curve fitting for temperature per month. As a result, Model (c) is the best model to be selected to fit
with the real data given in Table 3. The plotting of the data
points and these three models are shown in Fig. 3.
4.3 Example 3: Model Validity
60
A set of data for the population (in millions) in the United Actual Data
States and the corresponding year [8] is used to discuss the (a)
(b)
50
model selection with curve fitting. (c)

40
Table 3: Population of United States (in millions)
Year ti Population, yi
30
y

1815 8.3
1825 11.0
20
1835 14.7
1845 19.7
10
1855 26.7
1865 35.2
1875 44.4 0
1 2 3 4 5 6 7 8
1885 55.9 t

For convenience, 1815 is labeled as 1, 1825 as 2, and so Fig. 3 Comparison of Plotting between Three Models and the Actual
forth. Next, three models are proposed, which are: Data.

261
 IJISET - International Journal of Innovative Science, Engineering & Technology, Vol. 4 Issue 1, January 2017
ISSN (Online) 2348 – 7968 | Impact Factor (2015) - 4.332
www.ijiset.com

5. Conclusions
Wen Huey Lai received the Bachelor degree in industrial statistics from
The iterative algorithm of the Gauss-Newton method that the Universiti Tun Hussein Onn Malaysia, Johor, Malaysia, in 2016.
is used for solving the nonlinear least squares problem was
discussed in this paper. In general, it is typically difficult to Sie Long Kek received the M.Sc. degree and the Ph.D. in mathematics
decide an appropriate nonlinear function for a set of data. from the Universiti Teknologi Malaysia, Johor, Malaysia, in 2002 and
2011, respectively. He is currently a Senior Lecturer in the Department of
However, the model could be determined by observing the
Mathematics and Statistics, Universiti Tun Hussein Onn Malaysia. His
trend of the plotting of the data points. Then, the model research interests includes optimization and control, operation research
proposed could be used to perform the curve fitting. In this and management science, and modelling and simulation.
study, the unknown parameters were determined by using
Kim Gaik Tay received the M.Sc. degree and the Ph.D. degree in
the Gauss-Newton method, where the value of SSE was mathematics from the Universiti Teknologi Malaysia, Johor, Malaysia, in
minimized. The calculation was started from an initial 2000 and 2007, respectively. She is currently an Associate Professor in
point, and at each iteration step, a step-size was computed. the Department of Communication Engineering, Universiti Tun Hussein
During the iteration procedure, the value of the unknown Onn Malaysia. Her research interests includes nonlinear waves, optical
solitons and numerical methods.
parameters was then updated. The iteration stopped when
the convergence was achieved within a given tolerance.
The parameter value with the minimum SSE is known as
the optimum solution. For illustration, three examples were
discussed. The results showed that the model proposed
approximates closely to the actual data. In conclusion, the
efficiency of the Gauss-Newton algorithm has been
proven.

Acknowledgments

The authors would like to thank the Universiti Tun Hussein

Onn Malaysia (UTHM) for financial supporting to this
study under Incentive Grant Scheme for Publication (IGSP)
VOT. U417.

References
[1] J. P. Hoffmann, Linear Regression Analysis: Applications
and Assumptions. 2nd Ed. United States of America:
Department of Sociology Brigham Young University, 2010.
[2] M. Merriman, “On the History of the Method of Least
Squares”, The Analyst, Vol. 4, No. 2, 1877, pp. 33-36.
[3] S. M. Stigler, “Gauss and the Invention of Least Squares”,
the Annals of Statistics, Vol. 9, No. 3, 1981, pp. 465-474.
[4] T. Kariya and H. Kurata, Generalized Least Squares. Great
Britain: John Wiley & Sons, Ltd, 2004.
[5] J. Eriksson, (1996). “Optimization and Regularization of
Nonlinear Least Squares Problems”, Ph.D. thesis, Umea
University.
[6] G. Recktenwald, “Least Squares Fitting of Data to a Curve”,
Department of Mechanical Engineering, Portland State
University, 2001.
[7] T. Wherman, Applications of the Gauss-Newton Method.
Retrieved Oct 22, 2015 from https://ccrma.stanford.edu
[8] A. Croeze, L. Pittman and W. Reynolds, Nonlinear Least-
Squares Problems with the Gauss-Newton and Levenberg-
Marquardt Methods. Retrieved May 21, 2015 from
https://www.math.lsu.edu_system_files_MunozGroup1–
Presentation.

262