| Statistical Inferences
| & Gauss Markov Model
yethods of Estimation
timate’ means ‘to judge’.
peimating parameter has a particular value (quantitative in the form of real number),
repr it method of estimation is called point estimation
Fesimating parameter has a value (quantity in the form of real number) lying in an
va, then for it method of estimation is called interval estimation.
Characteristics of Estimators
# Consistency
‘= Unbiasedness
« Efficiency
+ Sufficiency
Ley be a random variable with pdf f(x, 6) 6 being a parameter.
Problem is to find the value of @.
The collection of all possible values that @ can tal
parameter space
inset theory notation above information can be expressed as
ve consider a random sample Xj, %)y--«/%9 of size n from a population with probability
density function.
F(X; Oy, Op, ==)
Where @,0,,...,8_ are unknown population parameters.
te one or more parameters is called a statistic.
Aninfnite number of statistics can be created to estimate population parameters.
Among these statistics, a statistic which has value 1
Study) is called best estimate of the parameter.
Symbolically if x,, x9, ..-)X, is a sample of size n.
Ty 8X 0X), Ty =9p(y Korver Klee Th = OC %r
salues i.e, statistics), which have values near to population parameters,
tofind the statistic which is best estimate of the population parameter (s)
function of sample values used to estimat
T
hese statistics are called estimators,
Consistency
{0%} be a fandom sample of size
aie
ke is denoted by @ and is called
s {fx 0)|8€ OP,
yearest to the parameter value (under
s.=)%q) be functions of sample
then problem is
_ be a sequence of estimators of 710),
syllabus
"Methods of Estimation
1 Properties of Estimators
* Confidence intervals
1 Tests of Hypothesis
Most powerful and Uniformly most
Powerful Tests
= Ukelnood Ratio Tests
Analysis of Discrete Data and chi-square
‘est of goodness of Fit
= Large Sample Tests
' Simple Non-parametric Tests for One
and'Two Sample Problems
"Rank Correlation and Test for
Independence
‘= Elementary Bayesian Inference
* Gauss-Markov Models
= Estimabilty of Parameters
Best Linear Unbiased Estimators
‘= Tests for Linear Hypothesis and.
Confidence Intervals
‘= Analysis of Variance and Covariance
‘= Fixed, Random and Mixed Eects
Models
‘Simple and Multiple Linear Regression
Elementary Regression Diagnostics
Logistic Regression
‘Multivariate Normal Distribution
\Wishart cistrbution and their Properties
Distribution of Quadratic Forms
Inference for Parameters
Partial and Multiple Correlation
Coefficients and Related Tests
‘= Data Reduction Techniques
* Principle Component Analysis.
‘Discriminant Analysis
= Cluster Analysis.
Canonical Correlation,
Simple RandomSampling
+ Stratified Sampling and Systematic Sampling
1 Probability Proportional to Size Sampling
= Ratio and Regression Methods
= Completely Randomized
‘Randomized Blocks and Latin-square
Designs
Connectedness and Orthogonalty of
Block Designs
‘IBD. 2k Factorial Experiments
Confounding and Construction
Series and Parallel Systems
‘Hazard Function and Failure Rates
= Censoring and Life Testing 1002
This T, is said to be consistent if
lim T, 2 70)
This can also be expressed as, is said to be consistent to 110),
if for any arbitrarily chosen €>0 and n>0 there exists a
Positive integer m, (depending upon ¢ and 1) such that
FLIT, ~ 10)|>€]— 1as n> 0
ie, FIlT, ~ @)|
1-9, Wn2m
Here, T,=T0xy,%).-.:%) iS an estimator of 10). Its value
depends upon the sample size n. AS n increases, Ty may
cchange on making n very large, we study how does T, behave.
Theorem Invariance Property of Consistent Estimators.
If T, is a consistent estimator of r(@) and w(y(@)) is a
continuous function of y(@), then y(T,) is a consistent
estimator of y(i(9))
Theorem Sufficient Condition for Consistency. Let {T,}
be a sequence of estimators such that for all @<@
(i) E,(T,) > ¥(0),n 0
and (ii) Varg(T,) > 0 as n>
Then, 7, is a consistent estimator of (0).
Unbiasedness
Unbiasedness is associated with sample size n
A Statistic Ty =Thx%y-%qh is said to be an unbiased
estimator of 710) if
E,)= 7), VOEe@
If E(,)> ¥@), then T, is said to be positively biased and if
ET,)< ¥@), it is said to be negatively biased.
The amount of bias b@)is given by
b@)= ET,)- ¥@),6€O
Unbiased Estimate for Population Mean (11)
and Variance (co)
Let a large population be {X,X,,....Xy} (of size N) with mean
Hand variance 0?
Let {4 %4y+:11%,} be a random sample of size m taken from it.
Then, sample mean ¥ is given by
Joey + apathy)
n
ake
=3o4
and sample variance,
i
a ze Ea
ate
Now, a=1(23)
In the sample, each x, is from the population {X;,X,-.. Xu}
and so x, can take anyone of the values X,, Xs, ..,X, each with
equal probability 1/ N.
x
1
£ a)
2,
a
UGC-CSIR NET Tutor Mathematical Sciences
Thus, probability distribution for x
aX | %
| uN
- Ay
UN | ee TW
1 ae 1
BEX th tench
Therefore, x)= 581+ 5% ne
1
= LOG th to AXy)=
wet u
Now,
=> Sample mean 5 is unbiased estimate ofthe pop
mean ().
Te
Now, are [23 =|
Vox) = Bx?) ~ (£0)? 4
For x = xj this gives
Vox) = Ex) [ba
Fd) = Vox) + 1x )P =o? +H? -*
Eq. (i) gives
VQ) = BR?) — (ERP ea
Abo, v@=vi1¥x)=1 ¥
so, e ( 3s) es 2 Vox)
- SMV tat Vo
or Statistical
tatistical Inferences and Gauss Markov Model
- «iy and Gv), we get
o ated
2) -[6@)F =
at) - [EOF = —
2
o
EG?) = — +p, ri
; eer HSER =n) 0
13 pe) — Ag?
03) =
3 nee) BG)
13492 ay2y_ (2
« (Soe
ane bg fe +)
segs. (i) and (0), we get
a o
= Lin’ +e y-(S ae)
ate
oo" 4) ee
-(cr alla
(n
ea?
i as)=
sample variance is not an unbiased estimate of
npulation variance.
Fc... n-1a
3 LAs) =0'
a 185 gle?
are -xf}=
fo oe “| g
* AS')=o?
s unbiased estimate of the population variance,
o.
Me know
Gaal
2 ftge sample, n> so and so s? = S*
°ecomes unbiased estimate of population variance.
1003
Test of Signifi
ean gnificance for Sample
For
{oc large samples, the standard normal variate corresponding
oi vn
Null Hypothesis, H, = The samy
ypothesis, iple has been di
Population with mean and variance 6, aa
significant difference between the sample mean (z) and
Population mean (u), the test statistic (for large samples) is
o/vn
If @ is unknown, its estimate is 6”
sample.
Confidence limits for u 95% confidence interval for u is given
by
[Z|s196, ie, Ets 196
In)
= £196 © are known as 95% confidence lit
vn
or 6=5 for large
ts forp as
oe cee
196 S<, the likelihood equatot
{jp 100 =0, has a solution which converges in probstit
to the true value 0." In other words MLES **
consistent.
MLE's are always consistent but need not he unbiss@!
Theorem 2 (Hazoor Bazar's Theorem) Any MSS
solution of the likelihood equation is 25}
U< Moo. Statistical Inferences and Gauss Markov Model
jy distributed about the true value @, Thus,
f
om 1 Vesn->—
110)
nce
» ne s
1°70 A-—2;log 1]
rs
43 If MLE exists, it is the most efficient in the
em
euch estimators.
of MLE is given by
rem 4 if @ sufficient estimator exists, it is a
fore maximum likelihood estimator
‘is theorem 1S useful in finding, if a sufficient
s sor exists OF not)
seorem 5 lf for a given population with pdf f(x,6),
xe (Minimum variance bound) estimator T exists for 6,
Wye likelihood equation will have a solution equal to
ppestimator T.
theorem 6 (Invariance property of MLE) If T is the
Tate and y(@) is one-to-one function of @, then (7) is
be MLE of v (0)
theorem 7 Cramer-Rao Inequality. If t is an unbiased
‘emator for ¥(@),@ function of parameter @, then
4 ee
0)
[°°] vor
1)
mt
itere, @) is the information on @ supplied by the
sample
var(t)>
Regularity Conditions for Cramer-Rao
‘inequality
() The parameter space © is a non-degenerate open
interval on the real line R’=]— =, =
(i) For almost all x = (xy, %9,-+%qh and for all 8 @,
a ,
5;1%0) exits the exceptional set if any is
independent of 8.
Gi) The range of integration is independent of the
parameter @, so that f{x,6) is differentiable under
integral sign.
(v) The conditions of uniform convergence of integrals,
are satisfied so that differentiation under the integral
sign is valid,
2
Wi@= { a oat ol |
yf and is postive forall 8 © corny.
‘is an unbiased estimator of @ ie., Et)=8
2 =
1005
50 (OP cary
real)
(s dl
1@)
(
a
where, ra-d( 1) | is called the amount of
information on @ supplied by the sample {1% ++}
and its reciprocal 755 as the information limit to the
variance of estimator t = tty X20 %q)
(itis by RA Fisher)
MVBE.
‘An unbiased estimator t of (@) for which Cramer-Rao lower
bound in
iste
)
varit) => Yor
a 10)
ke
ts asi]
is attained is called a minimum variance bound estimator
(MVBE)
Condition for the Equality Sign in
Crammer-Rao (C-R) inequality
CR inequality is
tier
{e=)|
The necessary and sufficient condition for an unbiased
estimator t to attain the lower bound ofits variance is given by
Svartt)= Bt - yO)P
a 1),
sk =It- 7040)
30 °8 xe) It - OnAe
1
where, REA)
Theorem 1 If the likelihood functior Lis expressible as
2 jog =f = tt -YOA00),
Fog = FG alt OAc
then
(i) t is an unbiased estimator of 110).
(i) Minimum Variance Bound (MVB) estimator ¢ for
100) exists and :
i =U! - tyr)
tai) var) = gy = ©0011
Theorem 2.A MVB estimator for (0) exists ifand ni7 if
there exists a sufficient estimator for 1) 1006
Complete Family of
Distributions
Let a statistic T= Tb %,%,.14%9) be based on a random
sample of size n from the population f\x,0),0¢ ©,
The distribution of the statistic T will, in general depend on 0.
So, corresponding to T, we again have a family of
distributions, say {g(t,0),0€ @)
Definition The statistic T = tx, x),...,x,) or more precisely,
the family of distributions {g(t,6)|@€@)} is said to be
complete for 0 if
E\(MT)1=0 for allo
= RIT) =01=1
ie, J h)-g¢,Oxdt=0, vee
or Et) gtt,0)=0, Voc @
= A{T) =0 Ve N almost surely (a.s)
MVUE (Minimum Variance
Unbiased Estimator) and
Blackmellisation
Theorem (Rao-Blackwell Theorem) Let X and Y be
random variables such that
E(Y) = and var(Y) =o’y >0
Let EX Ox)
Then, (i) E90)
(ii) varfo(o) < var(y)
The Principle of Maximum Likelihood
Estimation
It consists of finding an estimator 6 for the unknown parameter
6 =(@,,0,,..., 6 )say, which maximises the likelihood function
(@) for variations in parameter, ie.,
To find 6=(6,6,,....6,)
such that 16)> 16), ¥0¢O
or 16) = sup L(0),¥8e
6 is called Maximum t ikelihood Estimator (MLE)
Gis the solution of,
2
* ond X @)= a, ag
and PU, <0)=0 a
4, and a, are constants independent of 8 In Eqs. () and i
and 7 may be taken as C, and G B
(On combining Eqs. (i) and (i), we get
Pq, <0 Hy or Hy
Hi; H> Ho
(i) Hi 2
The value of Z given by Eq, (i) is known as
fortwo tailed test, at ‘a’ level of significance,
2, Thus, P|Z|> Zq)= a
The level of significance is the size of the type | error (or the
"aximum producers risk). It is generally 5% or 1%. Its fixed
' advance before collecting the sample information.
test statistict,
Z is denoted by
Etrors in Sampling
- ain objective of sampling theory is
‘out the population parameters on th
ts,
{fi Beerally in the form : To accept oF re
prexamining a sample from it. n this wor
ibity of error. It is of two types
to draw conclusion
1e basis of sample
ject the population
king there may be
he Error: Reject Hy, when it is true.
We Error: Accept Hy, when itis wrong, ie, accept Ho
\e rear
Reject Hy when it is true}
1007
= Pf Reject Hy [b} =
and P{ Accept Hy when it is wrong)
= Pf Accept Hp | Hi 1}=B
Then, o.is called size of type | error, B is
called size of type I! error.
@ is also called Producer's risk and B is
called consumer risk.
In two dimensional space
test statistics = tlxy %y---)%9), each
clement is in the form of ordered pairs.
Thus, ne YN)
The values can be plotted in 2-dimensional plane.
Critical Region
A region (corresponding
which amounts to rejecti
or region of rejection.
IFW is the critical region and iftt = ty %,-
the statistic based on a random sample of size n,
PteW/ Hy
Pte W/ Hy
W (complement of W) is called acceptance region,
Here, wuW=S,WoW=0
Probability ‘«’ that a random value of the stati
the critical region is known as the level of si
Power of the test
‘We know
a= Probability of type | error
= P (Reject Hy| Hp is true)
= Probability of type Il error
= PiAccept Hy| Hp is false)
Power of the test is defined as 1—B. It is alo called power
function of the test hypothesis H, against the alternative
hypothesis H,.
The value of the power function at a parameter point is called
the power of the test at that point.
to a statistic t)in the sample space S
jon of H, is termed as critical region
,,)is the value of
then
ict belongs to
ificance.
Steps in Solving Testing of Hypothesis
Problem
() From the problem, to know the form of the population
distribution, and ‘the parameter(s) about which the
hypothesis are set up.
(ii) Setting up of the null hypothesis Hy and the alternative
hypothesis Hy, in terms of the range of the parameter
values.
Gli) Choosing of statistic
= they %y,-0/%) Called test statistic which control
and B to minimum level. oe
(iv) Partition the set of all possible values of stat
disjoint sets W (rejection region or critical region) and W_
(acceptance region) such that 1008
(a) Reject Hy (ie., accept H,),
if the value of t falls in W.
(b) Accept Hy, if the value of t fall in W.
(W) Take observation (Experimentally sample observations)
calculate test statistic and further follow the steps and
take decision
Optimum Test under Different
Situations
In any testing problem the first two steps, viz.,
( the form of population distribution
Gi) the parameter(s) of interest are considered from the
problem. The most important step is to choose the best
test, ie., the best statistic ‘t’ and the critical region W
where by the best test we mean one which in addition to
controlling a at any desired low level has the minimum
type Il error B or maximum power 1, compared to B
of all other tests having this o:
Most Powerful Test (MP test)
Let test hypothesis be
Hy: 0-8
Hy: 828
The critical region W is the most powerful (MP) critical region
of size a (and corresponding test a most powerful test of level
@) for testing Hy: 8 = 8, against Hy: @=6,, if
Pix e W| Hy) = fu bo dx =
and Pixe W /H,)2 Poxe W| Hy)
for every other critical region W, satisfying Eq. (.
Uniformly Most Powerful Text (UMP
Test)
Let Hy:8=8)
Hy: 0 #8),
Then, for a predetermined ar, the best test for H, is called the
uniformly most powerful test of level a.
Definition The region W is called uniformly most powerful
(UMP) critical region of size a (and corresponding test as
uniformly most powerful (UMP) test of level a) for testing
Hy: 8 = 6, against H,: 0 #9 i.e.,
Hy: 8 =0, #8) if
PaceW] Hy) = flock =a wi)
and Pie W|H,)2 Pic e W,|H,)
for all #6,
Whatever the region W, satisfying Eq. (i) may be.
Neyman-Pearson Lemma
Let 4,%/-04,%)} be a random sample of size n from the
population whose density function is f(x, 8).
UGC-CSIR NET Tutor Mathematical Sciences
Let null hypothesis be Hy: © = % and alternative Pete
Hy 0=6, P
Let K>0, be a constant and W be a critical region of
4 p
such that
: spo}
= were a
wafresttsa]
and So
nd L; are the likelihood functions of the
weet ae a by Mayon %q) UNDE Hy an respect
the most powerful critical region of the
Then, W is U
hypothesis Hy: = @ se the alternative H,:0=0, test
Let We={re Se >K
be the most powerful H; :@=0 1 critical region of size a
testing Hy: @=0) against and let it be independent of
8) € 0, = 8 @,, where O) isthe parameter space under
‘Then, 'we say that critical region W is the uniformly mag
powerful critical region (UMP CR) of size a for tesing
Hy: 8 =@, against H,: 0 € @;,
Unbiased Test and Unbiased Critical
Region
Let testing be of
Hy: @= 6, against
Hy: =)
The critical region W and consequently the test based on ts
said to be unbiased, if the power of the test exceeds the sie
of the critical region, ie.,if
power of the test size of the CR.
- 1-B2a
= RW)2 RW)
= Pix|x€ W| Hy12 Plx|xe WH)
In other words, the critical region W is said to be unbiased. i
PAW) 2 Py, (W), VO Be ©
Theorem Every most powerful (MP) or uniformly 108
powerful (UMP) (CR) is necessarily unbiased.
(IEW be MPCR of size « for testing Hy: 0=6 agit
Hy: @=0,, then it is necessarily unbiased.
(id) Similarly, if W be UMPCR of size a for testis
Hy:8= 0) against Hy: 8 ©, then itis ako wnbiaes
um Regions and
Sufficient Statistics
Let X,X,
Population Statistical Inferences and Gauss Markov Model
wy, Let T be a sufficient statisti
e
vecration theorem.
iC for @ Then, by
4 orzation °
Lx,0)= TT, 8) = galtexyih(x) o
ve, ggt®)) 15 the marginal distribut
wie’ By NP-Lemma, the MPCR for recy the, Satie
st Hy: 0=0, is given by Bh
W = {3 Lx 8;)2 KL, 04}, VK >.0
son fap (0. and (i, we get
W = {818 000)- HO K. By (60). Ha}, WK>0
= 43] 8, €00)2 K- Bo, GX), WK>0
sence, fT =) is sufficient statistic for 8 then Ml
ves gven in terms of marginal distribution of 7 =e) cats
1A int distibution of XX, Xp
Likelihood Function
let XpMrer%y be a random sample of size n from
population with density function f(x,@). Then, the TikelIhood
theo of the sample values xy xysu/%, 1s denoted by
{8} and is defined as their joint density function given by
L= fbx, 8), 0) nfl 8) = Tf, 0)
{ gies the relative likelihood that the random variables
assume a particular set of values XX... Xp:
fora given sample X,%/..0% & becomes a function of the
variable 8, the parameter.
Likelihood Ratio Test
8 Xy%yyy%y be a random sample of size n>1 from a
opulation with pf f(x, ;,8;,---/ 0) where, @ the parameter
sguce is the totality Of all points that (6,/@;, ...8,) can
sume.
want to test the null hypothesis Ho: (8027-8) € Np
int the alternative hypothesis of the type Hi (810, ®x)
<6-,. The likelihood function of the sample observations
given by
[1,010.0 0)
8) the principle of maximum likelihood,
The likelihood equation for estimating any parameter @, is
Bien by
L
aL
Hp iat
aL
8,
Me can obtain the maximum likelihood estimates for the
Diameters (8, 8,,-~,8,) these parameters vary over the
Siameter space @ and subspace Q)- ;
“8 the maximum likelihood estimates sO obtained, in
LP] 6,8, 8,0)
=
Using
0, (= 42-0) KD
@
1009
We obtain maximum values of the likelihood function for
variation of the parameters in © and @,, respectively
The likelihood ratio test is defined as the quotient of these two
‘maximas, given by
sup L0x,@)
De Aay Mo eo __
0 Xero) = TG) = Sup Li 8)
0
Where, (5) and L(@) are the maxima of the likelihood
function (i) with respect to the parameters in the regions ©,
and , respectively.
Theorem 1 If i is the likelihood ratio test for testing a
null hypothesis H, and if U=9() is a monotonic
increasing (decreasing) function of 2, then the test based
‘on U is equivalent to the likelihood ratio test. The critical
region of the test based on U Is
4(0)< <6)
[o(Ag)< UV < 00)
Theorem 2 Let %,,%;,...%, be a random sample from @
population with Pdf /(x;0,,6,,....0,) where the parameter
space N is kdimensional. Suppose, we want to test the
composite hypothesis.
Fh: = 0%, 8, = 05...
Where 64,6;,...6 are specified numbers. When, Hy is
true, -2log, 4 is asymptotically distributed as chi-square
with r degrees of freedom i.e., under Hy, ~2log ~ z;, ifn
is large.
Test for the mean of a normal population N(u,0*)..
Let (%,%p,-.-.%,) be a random sample of size n from it
Let Hy: =H (specified) 00,
= {(H, 64, 3) |- o0, i= 12}
In case of population variances are unequal, 03 # 3
In case of population variances are equal, 03 1010
Test for the Equality of Means of Several
Normal Populations having Unknown
Common Variance o”
Let Hy =n
of=
Wy == We =H
=o] =0°
1, are not all equal
=oi =o?
Test for the Variance of a Normal
Population
Let (specified)
Test for Equality of Variance of Two Normal
Populations
Niw,03) and Niu,,03)
Ho} (unspecified)
Hy and p, are unspecified
Heo} #03, Wy tty (unspecified)
Test for the Equality of Variances of Several
Normal Populations
Nii,o7) 1542 wk.
Hy:03 =03 =...=0} =o? (unspecified)
With by Bhar by (unspecified)
Hy lim 12,41, K) are not all equal;
eel ee (unspecified)
IIDR Variables
A collection of random variables is independent and
identically distributed (IID) if each random variable has the
same probability distribution as the others and all are mutually
independent.
Central Limit Theorem The probability distribution of the
sum (or average) of IID variables with finite variance
approaches a normal distribution,
Example of IID
(i) A sequence of fair due rolls is ID
(ii)_A sequence of fair coin flips is IID
Chi-square Test of Goodness of Fit
It tests ifthe deviation of the experiment from theory is just by
cchance or is it really due to the inadequacy of theory to fit the
observed data.
1 O, and E, (i= 42, ..,k), be a set of observed and expected
frequencies, then
£(0,-
e Se EF
a
UGC-CSIR NET Tutor Mathematical Sciences
{follows chi-square distribution with (k ~ 1) df
ditions :
os Sample observations should be independent
) Conaains on the cll Fequences any shag
linear,
84 2nj= ZA, oF Fo) = BE
N; the total frequency should be reasonably. leg,
greater than 50. a
iv) No theoritical cell frequency shou
(9 Mt square is a continuous distribution, bur aut
inant character of continuity i call equa
fess than 5. It is a non-parametric test (or distgug
free test)
Degrees of Freedom (DF)
The number of independent variates which make
statistic (e.g., 12 )is known as the degrees of freedom i anj
is usually denoted by v.
The number of degrees of freedom is the total number of
observations less the number of independent constnins
imposed on the observations.
If k is the number of independent constraints in the set of daa
of n observations, then v
If a random variable X has a chi-square distribution with
n-degrees of freedom,then we write it as X ~ 32...
Its probability density function is given by
1
fox
Pima te Osten
Properties of Degree of Freedom
(@ If X,i=12,...,n are n independent normal varies
with mean; and standard deviation 6, then
S54) is a y?-variate with n degrees of
AL 6,
freedom,
Normal distrib is a particular case
%C--distribution when n=1
We know pdf for x? distribution is
= a ete OS Kem
Tin/ 2)
Putting x = 4, probability differential is given by
PO?) = fg dy?
is 1 HIP 2 y2 falda? gg yc*
Pippy eke Nano ¢
for n=1
POP) = fog ay? = yy Mt gg 21
2 T7/2)
1
Teer iy ecycn
= 1 isa standard normal variate.
