THE UNIVERSITY OF JORDAN
School of Engineering
Department of Civil Engineering
(0951301): Numerical Methods
Dr. Ramia Al-Ajarmeh
Curve Fitting
Least-Squares Regression
Part 3
Least-Squares Regression
Refer to the Textbook, Chapter 17
• Linear Regression
• Polynomial Regression
• Multiple linear Regression
• Non-linear Regression
Non-Linear Regression
Models are defined as those that have a nonlinear dependence on their
parameters.
e.g.:
𝑓 𝑥 = 𝑎0 1 − 𝑒 −𝑎1𝑥 + 𝑒
Non-Linear Regression
General linear least square model:
𝑦 = 𝑎0 𝑧0 + 𝑎1 𝑧1 + 𝑎2 𝑧2 + ⋯ +𝑎𝑚 𝑧𝑚 + 𝑒 ….(1)
where 𝑧0 , 𝑧1 , ⋯ , 𝑧𝑚 are m+1 basis functions.
It can be easily seen how simple & multiple linear regression fall
within this model- that is,
𝑧0 = 1 , 𝑧1 = 𝑥1 , 𝑧2 = 𝑥2 , ⋯ , 𝑧𝑚 = 𝑥𝑚
Further, polynomial regression can be also included if the basis
functions are simple monomials as in :
𝑧0 = 𝑥 0 = 1 , 𝑧1 = 𝑥 , 𝑧2 = 𝑥 2 , ⋯ , 𝑧𝑚 = 𝑥 𝑚
Note that the term in linear refers only to the model's dependence
on its parameters (i.e. a's).
Non-Linear Regression
General linear least square model:
Equation (1) can be expressed in matrix notation as:
𝑦 = 𝑍 𝐴 + 𝐸
𝑍 is a matrix of the calculated values of the basis functions at the
measured values of the independent variables.
𝑧01 𝑧11 ⋯⋯ 𝑧𝑚1
𝑧02 𝑧12 ⋯⋯ 𝑧𝑚2
𝑍 = ⋮ ⋮ ⋯⋯ ⋮
𝑧0𝑛 𝑧1𝑛 ⋯⋯ 𝑧𝑚𝑛
𝑚 = # 𝑜𝑓 𝑣𝑎𝑟𝑖𝑎𝑏𝑙𝑒𝑠 𝑖𝑛 𝑡ℎ𝑒 𝑚𝑜𝑑𝑒𝑙
𝑛 = # 𝑜𝑓 𝑑𝑎𝑡𝑎 𝑝𝑜𝑖𝑛𝑡𝑠
because n>m+1 ➔ 𝑍 is not a square matrix.
Non-Linear Regression
General linear least square model:
❖ The column vector 𝑦 contains the observed values of the
dependent variable.
𝑦 𝑇 = 𝑦1 𝑦2 ⋯ 𝑦𝑛
❖ The col. vector 𝐴 contains the unknown coefficients
𝐴 𝑇 = 𝑎1 𝑎2 ⋯ 𝑎𝑚
❖ The col. vector 𝐸 contains the residuals
𝐸 𝑇 = 𝑒1 𝑒2 ⋯ 𝑒𝑛
❖ Sum of squares of the residuals for this model:
𝑛 𝑚
𝑆𝑟 = (𝑦𝑖 − 𝑎𝑗 𝑧𝑗𝑖 )2
𝑖=1 𝑗=0
Non-Linear Regression
General linear least square model:
Sr quantity can be minimized by taking its partial derivative w.r.t
each of the coefficients & setting to zero.
The outcome of this process is the normal equations that can be
expressed concisely in matrix form as:
𝑍 𝑇 𝑍 𝐴 = 𝑍 𝑇 𝑦
employ matrix inverse:
𝑇 −1 𝑇
𝐴 = 𝑍 𝑍 𝑍 𝑦
Non-Linear Regression
Non-linear regression is based on determining the values of the
parameters that minimize the sum of squares of the residuals.
However, for the nonlinear case, the solution must proceed in an
iterative fashion.
----------------------------------------------------------------------------------------------
Gauss-Newton method is one algorithm for minimizing the sum of the
squares of the residuals between data & non linear equations.
Key concept: Taylor series expansion is used to express the original
non-linear equations in an approximate, linear form:
(𝑥1 − 𝑥0 ) 𝑥1 − 𝑥0 2 𝑥1 − 𝑥0 𝑛
𝑓 𝑥1 = 𝑓 𝑥0 + 𝑓′ 𝑥0 + 𝑓 ′′ 𝑥0 + ⋯ + 𝑓 (𝑛) 𝑥0
1! 2! 𝑛!
Non-Linear Regression
Gauss-Newton method:
How to apply?
𝑦𝑖 = 𝑓 𝑥𝑖 ; 𝑎0 , 𝑎1 , ⋯ , 𝑎𝑚 + 𝑒𝑖
𝑦𝑖 = a measured value of the dependent variable.
𝑓 𝑥𝑖 ; 𝑎0 , 𝑎1 , ⋯ , 𝑎𝑚 : the equation that is a function of the independent
variable xi & a non linear function of the parameters 𝑎0 , 𝑎1 , ⋯ , 𝑎𝑚
𝑒𝑖 : a random error
abbreviated form:
𝑦𝑖 = 𝑓 𝑥𝑖 + 𝑒𝑖 … (2)
Non-Linear Regression
Gauss-Newton method:
The non-linear model can be expanded in a Taylor Series around the
parameter values & curtailed after the 1st derivative.
Case of two parameters:
𝜕𝑓(𝑥𝑖 )𝑗 𝜕𝑓(𝑥𝑖 )𝑗
𝑓 𝑥𝑖 𝑗 + 1 = 𝑓(𝑥𝑖 )𝑗 + ∆𝑎0 + ∆𝑎1 … (3)
𝜕𝑎0 𝜕𝑎1
𝑗 = 𝑖𝑛𝑖𝑡𝑖𝑎𝑙 𝑔𝑢𝑒𝑠𝑠𝑒𝑠 , 𝑗 + 1 = 𝑝𝑟𝑒𝑑𝑖𝑐𝑡𝑖𝑜𝑛
∆𝑎0 = 𝑎0 𝑗+1 − 𝑎0 𝑗 & ∆𝑎1 = 𝑎1 𝑗+1 − 𝑎1 𝑗
Non-Linear Regression
We have linearized the original model w.r.t the parameters.
eq.(3) can be substituted into eq.(2):
𝜕𝑓(𝑥𝑖 )𝑗 𝜕𝑓(𝑥𝑖 )𝑗
𝑦𝑖 − 𝑓 𝑥𝑖 𝑗 = ∆𝑎0 + ∆𝑎1 + 𝑒𝑖
𝜕𝑎0 𝜕𝑎1
or in a matrix form:
𝐷 = 𝑍𝑗 ∆𝐴 + 𝐸
𝑍𝑗 is the matrix of partial derivatives of the function evaluated @ the
initial guesses j,
𝜕𝑓1 𝜕𝑓1
ൗ𝜕𝑎 ൗ𝜕𝑎
0 1
𝜕𝑓2 𝜕𝑓2
𝑍𝑗 = ൗ𝜕𝑎 ൗ𝜕𝑎
0 1
⋮ ⋮
𝜕𝑓𝑛 𝜕𝑓𝑛
ൗ𝜕𝑎 ൗ𝜕𝑎
0 1
Non-Linear Regression
𝑦1 − 𝑓 𝑥1 ∆𝑎0
𝑦 − 𝑓 𝑥2 ∆𝑎1
𝐷 = 2 ∆𝐴 =
⋮
⋮
𝑦𝑛 − 𝑓 𝑥𝑛 ∆𝑎𝑚
Apply linear least squares theory to the following equation:
𝐷 = 𝑍𝑗 ∆𝐴 + 𝐸
𝑇 𝑇
𝑍𝑗 𝑍𝑗 ∆𝐴 = 𝑍𝑗 𝐷
used to solve for parameters:
𝑎0, 𝑗+1 = 𝑎0 𝑗 + ∆𝑎0
𝑎1, 𝑗+1 = 𝑎1 𝑗 + ∆𝑎1
𝑎𝑘, 𝑗+1 − 𝑎𝑘, 𝑗
𝜀𝑎 𝐾 = 100% < 𝜀𝑠
𝑎𝑘, 𝑗+1
Non-Linear Regression
Example: Gauss-Newton method:
Fit the function 𝑓 𝑥; 𝑎0 , 𝑎1 = 𝑎0 1 − 𝑒 −𝑎1𝑥 to the data:
𝑥 0.25 0.75 1.25 1.75 2.25
𝑦 0.28 0.57 0.68 0.74 0.79
use initial guesses of 𝑎0 = 1.0 & 𝑎1 = 1.0 𝑓𝑜𝑟 𝑡ℎ𝑒 𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟𝑠.
Note that for these guesses, the initial sum of squares of residuals is
0.0248
Non-Linear Regression
Example Solution:
The partial derivatives of the function with respect to the parameters
are:
Non-Linear Regression
Example Solution:
