Econometrics [EM2008/EM2Q05]
Lecture 4
Maximum likelihood and generalized least squares
estimators
Irene Mammi
irene.mammi@unive.it
Academic Year 2018/2019
1 / 19
outline
I maximum likelihood and generalized least squares estimators
I maximum likelihood estimators (MLEs)
I ML estimation of the linear model
I ML-based tests: LR, Wald and LM
I generalized least squares
I References:
I Johnston, J. and J. DiNardo (1997), Econometrics Methods, 4th
Edition, McGraw-Hill, New York, Chapter 5.
2 / 19
MLEs in a nutshell
I let y 0 = y1 y2 · · · yn be a n-vector of sample values,
dependent on some k-vector of unknown parameters,
θ0 = θ1 θ2 · · · θk
I let the joint density be written f (y; θ): this density may either
indicate, for given θ, the probability of a set of sample outcomes, or it
may be interpreted as a function of θ, conditional on a set of sample
outcomes
I in the latter interpretation it is referred as likelihood function:
Likelihood function = L(θ; y ) = f (y; θ)
I maximizing the likelihood function wrt θ implies finding a specific
value, say θ̂, that maximizes the probability of obtaining the sample
values actually observed
I θ̂ is the MLE of the unknown parameter vector θ
3 / 19
MLEs in a nutshell (cont.)
I in most cases, it is simpler to maximize the log of the likelihood
fuction:
` = ln L
I then
∂` 1 ∂L
=
∂θ L ∂θ
and the θ̂ that maximizes ` will also maximize L
I the derivative of ` wrt θ is the score, s (θ; y )
I the MLE θ̂ is obtained by setting the score to zero, i.e. by finding θ
that solves
∂`
s (θ; y ) = =0
∂θ
4 / 19
ML estimation of the linear model
I consider the linear model
y = X β + u with u ∼ N (0, ff 2 I )
I the multivariate normal density for u is
1 2 0
f (u ) = e −(1/2σ )(u u )
(2πσ2 )n/2
I so that the multivariate density for y conditional on X is
∂u
f (y |X ) = f (u )
∂y
where |(∂u/∂y )| is the absolute value of the determinant from the
n × n matrix of partial derivatives of u wrt y, which is here simply
the identity matrix
5 / 19
ML estimation of the linear model (cont.)
I the log-likelihood function is
n n 1
` = ln f (y |X ) = ln f (u ) = − ln 2π − ln σ2 − 2 u 0 u
2 2 2σ
n n 1
= − ln 2π − ln σ − 2 (y − X β)0 (y − X β)
2
2 2 2σ
I The vector of unknown parameters, θ, has k + 1 elements, namely,
θ0 = β0 , σ2
I taking partial derivatives gives
∂` 1
= − 2 (−X 0 y + X 0 X β)
∂β σ
∂` n 1
2
= − 2 + 4 (y − X β ) 0 (y − X β )
∂σ 2σ 2σ
6 / 19
ML estimation of the linear model (cont.)
I setting these partial derivatives to zero gives the MLEs as
β̂ = (X 0 X )−1 X 0 y
and
σ̂2 = (y − X β̂)0 (y − X β̂)/n
I the MLE, β̂ is the OLS estimator, b, and σ̂2 is e 0 e/n with e being
OLS residuals
I the max of the likelihood function is L( β̂, σ̂2 ) = constant · (e 0 e )−n/2
7 / 19
ML-based tests
consider the general framework of linear hypotheses about β
H0 : R β = r
where R is a q × k (q < k ) matrix of known constants and r a q × 1
known vector
Likelihood ratio (LR) test
I L( β̂, σ̂2 ) is the unrestricted maximum likelihood and can be
expressed as a function of the unrestricted sum of squares, e 0 e
I the model may also be estimated in restricted form by maximizing
the likelihood subject to the restrictions R β = r . Denote the resulting
estimators as β̃ and σ̃2 : the maximum of the likelihood is L( β̃, σ̃2 )
I if the restrictions are valid, we expect the restricted maximum to be
close to the unrestricted maximum
8 / 19
ML-based tests (cont.)
I the likelihood ratio is defined as
L( β̃, σ̃2 )
λ=
L( β̂, σ̂2 )
I a generally applicable large-sample test is
a
LR = −2 ln λ = 2[ln L( β̂, σ̂2 ) − ln L( β̃, σ̃2 )] ∼ χ2 (q )
which can be alternatively expressed as
LR = n(ln e 0∗ e ∗ − ln e 0 e )
I the calculation of the LR statistic requires the fitting of both the
restricted and unrestricted model
9 / 19
ML-based tests (cont.)
Wald (W) test
I the Wald test only requires to calculate the unrestricted β̂
I the vector (R β̂ − r ) indicates the extent to which the unrestricted
ML estimates fit the null hypothesis: a vector close to zero would
support H0
I under the null, R β̂ − r is asymptotically distributed as multivariate
normal with zero mean and variance-covariance matrix RI −1 ( β)R 0
where I −1 ( β) = σ2 (X 0 X )−1
I we have
a
(R β̂ − r )0 [RI −1 ( β)R 0 ]−1 (R β̂ − r ) ∼ χ2 (q )
I the asymptotic distribution still holds when σ2 is replaced by
σ̂2 = e 0 e
10 / 19
ML-based tests (cont.)
I the resulting Wald statistic is
(R β̂ − r )0 [R (X 0 X )−1 R 0 ]−1 (R β̂ − r ) a 2
W = ∼ χ (q )
σ̂2
which can also be expressed as
n(e 0∗ e ∗ − e 0 e ) a 2
W = ∼ χ (q )
e0e
11 / 19
ML-based tests (cont.)
Lagrange multiplier (LM) test
I the LM test, also known as the score test, is based on the score vector
∂ ln L ∂`
s (θ) = =
∂θ ∂θ
I the unrestricted estimator, θ̂, is found by solving s (θ̂) = 0; the score
vector will in general not be zero when evaluated at θ̃, the restricted
estimator
I if the restrictions are valid, the restricted maximum, `(θ̃), should be
close to the unrestricted maximum, `(θ̂), and so the gradient of the
former should be close to zero
I under the null hypothesis,
a
LM = s 0 (θ̃)I −1 (θ̃)s (θ̃) ∼ χ2 (q )
I notice that there is no need to compute the unrestricted estimator
12 / 19
ML-based tests (cont.)
I it can be shown that the LM statistic is
LM = nR 2
where R 2 is the squared multiple correlation coefficient from the
regression of e ∗ on X
I the LM test can be implemented in two steps: first compute the
restricted estimator θ̃ and obtain the residual vector e ∗ ; then regress
e ∗ on X and refer nR 2 from this regression to χ2 (q )
I it can be shown that
n(e 0∗ e ∗ − e 0 e )
LM =
e 0∗ e ∗
I it can also be proved that W ≥ LR ≥ LM.
13 / 19
ML-based tests (cont.)
Figure 1: ML-based tests
14 / 19
ML estimation with nonspherical disturbances
I consider the model
y = X β + u with u ∼ N (0, σ2 Ω)
where Ω is a positive definite matrix of order n, whose elements are
assumed to be known
I e.g. assume
var(ui ) = σi2 = σ2 X2i2 i = 1, 2, . . . , n
so that the error variance-covariance matrix is
2
··· 0
X21 0
0 X222 ··· 0
var(u ) = σ2 . 2
2 2 2
.. = σ diag X21 X22 · · · X2n
. .
. . .
. . . .
0 0 2
· · · X2n
15 / 19
ML estimation with nonspherical disturbances (cont.)
I the multivariate normal density for u is
1
f (u ) = (2π )−n/2 |σ2 Ω|−1/2 exp − u 0 (σ2 Ω)−1 u
2
which, noting that |σ2 Ω| = σ2n |Ω|, can be rewritten as
f (u ) = (2π )−n/2 (σ2 )−n/2 |Ω|−1/2 exp[(−1/2σ2 )u 0 Ω−1 u ]
I the log-likelihood is then
n n 1 1
` = − ln(2π ) − ln σ2 − ln |Ω| − 2 (y − X β)0 Ω−1 (y − X β)
2 2 2 2σ
I differentiating with respect to β and σ2 and setting the partial
derivatives to zero gives the ML estimators
β̂ = (X 0 Ω−1 X )−1 X 0 Ω−1 y
and
1
σ̂2 = (y − X β̂)0 Ω−1 (y − X β̂)
n
16 / 19
generalized least squares
I since Ω is positive definite, its inverse is positive definite. Thus it is
possible to find a nonsingular matrix P such that
Ω −1 = P 0 P
I substitution into the MLE formula gives
β̂ = (X 0 P 0 PX )−1 X 0 P 0 Py = [(PX )0 (PX )]−1 (PX )0 (Py )
which is exactly the vector of estimated coefficients that would be
obtained from the OLS regression of the vector Py on the matrix PX
I to see this, premultiply the linear model by P and obtain
y ∗ = X ∗ β + u∗
where y ∗ = Py, X ∗ = PX, and u ∗ = Pu
17 / 19
generalized least squares (cont.)
I since Ω = P −1 (P 0 )−1 , we have
var(u ∗ ) = E(Puu 0 P 0 )
= σ2 PΩP 0
= σ2 PP −1 (P 0 )−1 P 0
= σ2 I
I the coefficient vector from the OLS regression of y ∗ on X ∗ is the
generalized least squares (GLS) estimator:
b GLS = (X 0∗ X ∗ )−1 X 0∗ y ∗
= (X 0 Ω −1 X ) −1 X 0 Ω −1 y
I it follows directly that
var(b GLS ) = σ2 (X 0∗ X ∗ )−1
= σ 2 (X 0 Ω −1 X ) −1
18 / 19
generalized least squares (cont.)
I an unbiased estimate of σ2 is obtained as
s 2 = (y ∗ − X ∗ b GLS )0 (y ∗ − X ∗ b GLS )/(n − k )
= [P (y − Xb GLS )]0 [P (y − Xb GLS )]/(n − k )
= (y ∗ − Xb GLS )0 Ω−1 (y − Xb GLS )/(n − k )
I an exact finite sample test of the linear restrictions
H0 : R β = r
can be based on
(r − Rb GLS )0 [R 0 (X 0 Ω−1 X )−1 R 0 ]−1 (r − Rb GLS )/q
F =
s2
19 / 19
