Statistics
CVEN2002/2702
Week 12
This lecture
11. Analysis of Variance (ANOVA)
11.1 Introduction
11.2 One-way Analysis of Variance
11.3 Multiple pairwise comparisons
11.4 Adequacy of the ANOVA model
11.5 Blocking factor
Additional reading: Sections 9.1, 9.2 and 9.4 in the textbook
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
2 / 48
11. Analysis of Variance (ANOVA)
11. ANOVA
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
3 / 48
11. Analysis of Variance (ANOVA)
11.1 Introduction
Introduction
In Chapter 10, we introduced testing procedures for comparing the
means of two different populations, having observed two random
samples drawn from those populations (two-sample z- and t-tests)
However, in applications, it is common that we want to detect a
difference in a set of more than two populations
Imagine the following context: four groups of students were subjected
to different teaching techniques and tested at the end of a specified
period of time. Do the data shown in the table below present sufficient
evidence to indicate a difference in mean achievement for the four
teaching techniques?
Tech. 1
65
87
73
79
81
69
CVEN2002/2702 (Statistics)
Tech. 2
75
69
83
81
72
79
90
Dr Justin Wishart
Tech. 3
59
78
67
62
83
76
Tech. 4
94
89
80
88
Session 2, 2012 - Week 12
4 / 48
11. Analysis of Variance (ANOVA)
11.1 Introduction
Introduction: randomisation
To answer this question, we should first note that the method of division of the
students into 4 groups is of vital importance
For instance, some basic visual inspection of the data suggests that the
members of group 4 scored higher than those in the other groups. Can we
conclude from this that teaching technique 4 is superior? Perhaps, students
in group 4 are just better learners
; it is essential that we divide the students into 4 groups in such a
way to make it very unlikely that one of the group is inherently
superior to others (regardless of the teaching technique it will be
subjected to)
; the only reliable method for doing this is to divide the students in a
completely random fashion, to balance out the effect of any
nuisance variable that may influence the variable of interest
This kind of consideration is part of a very important area of statistical
modelling called experimental design, which is not addressed in this course
(Chapter 10 in the textbook). In this course, we will always assume that the
division of the individuals into the groups was indeed done at random
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
5 / 48
11. Analysis of Variance (ANOVA)
11.1 Introduction
Introduction
75
69
83
81
72
79
90
78.43
7.11
59
78
67
62
83
76
94
89
80
88
70.83
9.58
87.75
5.80
65
75.67
8.17
85
65
87
73
79
81
69
80
Tech. 4
75
Tech. 3
70
Tech. 2
60
x
s
Tech. 1
90
95
Now, numerical summaries and a graphical display of the data are
always useful:
; the boxplots show the variability of the observations within a group
and the variability between the groups
; comparing the between-group with the within-group variability
is the key in detecting any significant difference between the groups
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
6 / 48
11. Analysis of Variance (ANOVA)
11.2 One-way Analysis of Variance
Between-group and within-group variability
Group 1
5.90
5.92
5.91
5.89
5.88
Group 2
5.51
5.50
5.50
5.49
5.50
Group 3
5.01
5.00
4.99
4.98
5.02
Between-group variance = 1.017, within-group variance = 0.00018
(ratio = 5545)
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
7 / 48
11. Analysis of Variance (ANOVA)
11.2 One-way Analysis of Variance
Between-group and within-group variability
Group 1
5.90
4.42
7.51
7.89
3.78
Group 2
6.31
3.54
4.73
7.20
5.72
Group 3
4.52
6.93
4.48
5.55
3.52
Between-group variance = 1.017, within-group variance = 2.332
(ratio = 0.436)
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
8 / 48
11. Analysis of Variance (ANOVA)
11.2 One-way Analysis of Variance
Analysis of Variance
Comparing (intelligently) the between-group variability and the
within-group variability is the purpose of the Analysis of Variance
; often shortened to the acronym ANOVA
Suppose that we have k different groups (k populations, or k
sub-populations of a population) that we wish to compare
Often, each group is called a treatment or treatment level (general
terms that can be traced back to the early applications of this
methodology in the agricultural sciences)
The response for each of the k treatments is the random variable of
interest, say X
Denote Xij the jth observation (j = 1, . . . , ni ) taken under treatment i
; we have k independent samples (one sample from each of the
treatments)
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
9 / 48
11. Analysis of Variance (ANOVA)
11.2 One-way Analysis of Variance
ANOVA samples
The k random samples are often presented as:
Treatment
Mean
St. Dev.
1
X11
X21
..
.
X1n1
2
X21
X22
..
.
X1n2
...
k
Xk 1
Xk 2
..
.
Xknk
1
X
S1
2
X
S2
...
...
k
X
Sk
...
...
i and Si are the sample mean and standard deviation of the ith
where X
sample. The total number of observations is
n = n1 + n2 + . . . + nk
, is
and the grand mean of all the observations, usually denoted X
k
i
XX
1 + n2 X
2 + . . . + nk X
k
n1 X
=1
X
Xij =
n
n
i=1 j=1
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
10 / 48
11. Analysis of Variance (ANOVA)
11.2 One-way Analysis of Variance
ANOVA model
The ANOVA model is the following:
Xij = i + ij
where
i is the mean response for the ith treatment (i = 1, 2, . . . , k )
ij is an individual random error component (j = 1, 2, . . . , ni )
As usual for errors, we will assume that the random variables ij are
normally and independently distributed with mean 0 and variance 2 :
i.i.d.
ij N (0, )
for all i, j
Therefore, each treatment can be thought of as a normal population
with mean i and variance 2 :
ind.
Xij N (i , )
Important: the variance 2 is common for all treatments
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
11 / 48
11. Analysis of Variance (ANOVA)
11.2 One-way Analysis of Variance
ANOVA hypotheses
We are interested in detecting differences between the different
treatment means i , which are population parameters
; hypothesis test!
The null hypothesis to be tested is
H0 : 1 = 2 = . . . = k
versus the general alternative
Ha : not all the means are equal
Careful! The alternative hypothesis should be that at least two of the
means differ, not that they are all different !
As pointed out previously, the primary tool when testing for equality of
the means is based on a comparison of the variances within the
groups and between the groups
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
12 / 48
11. Analysis of Variance (ANOVA)
11.2 One-way Analysis of Variance
Variability decomposition
The ANOVA partitions the total variability in the sample data,
described by the total sum of squares
ni
k X
X
SSTot =
)2
(Xij X
(df = n 1)
i=1 j=1
into the treatment sum of squares (= variability between groups)
SSTr =
k
X
)2
i X
ni (X
(df = k 1)
i=1
and the error sum of squares (= variability within groups)
SSEr =
ni
k X
X
i )2
(Xij X
(df = n k )
i=1 j=1
The sum of squares identity is
SSTot = SSTr + SSEr
CVEN2002/2702 (Statistics)
Dr Justin Wishart
(Proof: exercise)
Session 2, 2012 - Week 12
13 / 48
11. Analysis of Variance (ANOVA)
11.2 One-way Analysis of Variance
Variability decomposition
The total sum of squares SSTot quantifies the total amount of variation
contained in the global sample
The Treatment sum of squares SSTr quantifies the variation between
the groups, that is the variation between the means of the groups
(giving more weight to groups with more observations)
The Error sum of squares SSEr quantifies the variation within the
groups
treatment sample
error samples
95
global sample
95
95
samples
65
65
CVEN2002/2702 (Statistics)
10
5
X
Dr Justin Wishart
80
X
75
60
10
70
65
80
X
75
60
70
70
75
85
85
80
60
90
85
90
90
Session 2, 2012 - Week 12
14 / 48
11. Analysis of Variance (ANOVA)
11.2 One-way Analysis of Variance
Mean Squared Error
In sample i, the sample variance is given by Si2 =
1
ni 1
Pni
j=1 (Xij
i )2
X
which is an unbiased estimator for 2 : E(Si2 ) = 2 (Slide 43 Week 10)
Since,
SSEr =
ni
k X
X
i )2 =
(Xij X
i=1 j=1
hence
E(SSEr ) =
k
X
(ni 1)Si2
i=1
k
k
X
X
(ni 1)E(Si2 ) = 2
(ni 1) = (n k ) 2
i=1
i=1
; another unbiased estimator for 2 is the Mean Squared Error MSEr
MSEr =
SSEr
nk
(generalisation of the pooled sample variance, Slide 21 Week 10)
; the number of degrees of freedom for this error estimator of 2 is
nk
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
15 / 48
11. Analysis of Variance (ANOVA)
11.2 One-way Analysis of Variance
Treatment mean square
Now if H0 is true, that is if 1 = 2 = . . . = k = , we have
i ) N (0, ), for all i = 1, . . . , k
i N (, ), that is ni (X
X
ni
1 ), n2 (X
2 ), . . . , nk (X
k ), is a random sample
; n1 (X
whose sample variance
k
1 X
)2 = SSTr
i X
ni (X
k 1
k 1
i=1
is an unbiased estimator for 2
; the Treatment Mean Square MSTr , defined by
MSTr =
SSTr
k 1
is also an unbiased estimator for 2
; the number of degrees of freedom for this treatment estimator
of 2 is k 1
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
16 / 48
11. Analysis of Variance (ANOVA)
11.2 One-way Analysis of Variance
ANOVA test
Thus we have two potential estimators of 2 :
1
MSEr , which always estimates 2
2
MSTr , which estimates 2 only when H0 is true
Actually, if H0 is not true, MSTr tends to exceed 2 , as we have
E(MSTr ) = 2 + true variance between the groups
; the idea of the ANOVA test now takes shape
Suppose we have observed k samples xi1 , xi2 , . . . , xini , for
i = 1, 2, . . . , k , from which we can find through calculations the
observed values msTr and msEr . Then:
if msTr ' msEr , then H0 is probably reasonable
if msTr msEr , then H0 should be rejected
; this will thus be a one-sided hypothesis test
We need to determine what msTr msEr means so as to obtain a
hypothesis test at given significance level
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
17 / 48
11. Analysis of Variance (ANOVA)
11.2 One-way Analysis of Variance
Sampling distribution
It can be shown that, if H0 is true, the ratio
MSTr
F =
=
MSEr
SSTr
k 1
SSEr
nk
follows a particular distribution known as the Fishers F -distribution
with k 1 and n k degrees of freedom, which is usually denoted by
F Fk 1,nk
Note: Ronald A. Fisher (1890-1962) was an English statistician and
biologist. Some say that he almost single-handedly created the
foundation for modern statistical science. As a biologist, he is also
regarded as the greatest biologist since Charles Darwin
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
18 / 48
11. Analysis of Variance (ANOVA)
11.2 One-way Analysis of Variance
The Fishers F -distribution
A random variable, say X , is said to follow Fishers F -distribution with
d1 and d2 degrees of freedom, i.e.
X Fd1 ,d2
if its probability density function is given by
f (x) =
((d1 + d2 )/2)(d1 /d2 )d1 /2 x d1 /21
(d1 /2)(d2 /2)((d1 /d2 )x + 1)(d1 +d2 )/2
for x > 0
; SX = [0, +)
for some integers d1 and d2
Note: the Gamma function is given by
Z +
(y ) =
x y 1 ex dx,
for y > 0
It can be shown that (y ) = (y 1) (y 1), so that, if y is a positive
integer n,
(n) = (n 1)!
There is usually no simple expression for the F -cdf
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
19 / 48
11. Analysis of Variance (ANOVA)
11.2 One-way Analysis of Variance
The Fishers F -distribution
1.0
Some F -distributions
d1=4,d2=4
d1=100,d2=6
0.6
d1=4,d2=100
0.2
0.4
0.2
0.4
f(x)
F(x)
0.6
0.8
d1=3,d2=10
d1=4,d2=4
d1=100,d2=6
d1=3,d2=10
0.0
0.0
d1=4,d2=100
pdf f (x) = F 0 (x)
cdf F (x)
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
20 / 48
11. Analysis of Variance (ANOVA)
11.2 One-way Analysis of Variance
The Fishers F -distribution
It can be shown that the mean and the variance of the F -distribution
with d1 and d2 degrees of freedom are
E(X ) =
and
Var(X ) =
d2
d2 2
for d2 > 2
2d22 (d1 + d2 2)
d1 (d2 2)2 (d2 4)
for d2 > 4
Note that a F -distributed random variable is nonnegative, as expected
(ratio of two positive random quantities) and the distribution is highly
skewed to the right
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
21 / 48
11. Analysis of Variance (ANOVA)
11.2 One-way Analysis of Variance
The Fishers F -distribution: quantiles
Similarly to what we did for other distributions, we can define the
quantiles of any F -distribution:
Fdistribution
Let fd1 ,d2 ; be the value such that
P(X > fd1 ,d2 ; ) = 1
The F -distribution is not symmetric,
however it can be shown that
fd1 ,d2 ; =
f(x)
for X Fd1 ,d2
1
fd2 ,d1 ;1
1
fd1, d2,
x
For any d1 and d2 , the main quantiles of interest may be found in the
F -distribution critical values tables
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
22 / 48
11. Analysis of Variance (ANOVA)
11.2 One-way Analysis of Variance
ANOVA test
The null hypothesis to test is H0 : 1 = 2 = . . . = k
versus the general alternative Ha : not all the means are equal
Evidence against H0 is shown if MSTr MSEr , so we will reject H0
whenever MSTr is much larger than MSEr , i.e.
MSTr
MSEr
much larger than 1
; for testing H0 at significance level , we need a constant c such that
MSTr
=P
> c if H0 is true
MSEr
We know that, if H0 is true, F =
MSTr
MSEr
Fk 1,nk
; we have directly that c = fk 1,nk ;1
From observed values msTr and msEr , the decision rule is:
reject H0 if
CVEN2002/2702 (Statistics)
msTr
> fk 1,nk ;1
msEr
Dr Justin Wishart
Session 2, 2012 - Week 12
23 / 48
11. Analysis of Variance (ANOVA)
11.2 One-way Analysis of Variance
ANOVA test: p-value
F(k1), (nk)distribution
The observed value of the test
statistic is
msTr
msEr
and thus the p-value is given by
f(x)
f0 =
p = P(X > f0 ),
p
where X Fk 1,nk
f0
(the probability that the test statistic will take on a value that is at least
as extreme as the observed value when H0 is true, definition on Slide
21 Week 9) ; from the F -distribution table, only bounds can be found
for this p-value (use software to get an exact value)
x
This test is also often called the F -test or ANOVA F -test
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
24 / 48
11. Analysis of Variance (ANOVA)
11.2 One-way Analysis of Variance
ANOVA table
The computations for this test are usually summarised in tabular form
Source
degrees
of freedom
sum of
squares
mean
square
Treatment
dfTr = k 1
ssTr
msTr =
ssTr
k 1
Error
dfEr = n k
ssEr
msEr =
ssEr
nk
Total
dfTot = n 1
ssTot
F -statistic
f0 =
msTr
msEr
Note 1: dfTot = dfTr + dfEr and ssTot = ssTr + ssEr
Note 2: this table is the usual computer output when an ANOVA
procedure is run
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
25 / 48
11. Analysis of Variance (ANOVA)
11.2 One-way Analysis of Variance
ANOVA: example
Example
Consider the data shown on Slide 5. Test at significance level = 0.05 the
null hypothesis that there is no difference in mean achievement for the four
teaching techniques
We have k = 4, n1 = 6, n2 = 7, n3 = 6 and n4 = 4, with x1 = 75.67,
x2 = 78.43, x3 = 70.83, x4 = 87.75 and s1 = 8.17, s2 = 7.11, s3 = 9.58,
s4 = 5.80. Besides,
4
n = 6 + 6 + 7 + 4 = 23
and
x =
1X
ni xi = 77.35
n
i=1
Thus, from the expressions on Slides 15 and 16,
ssEr = 5 8.172 + 6 7.112 + 5 9.582 + 3 5.802 = 1196.63
and
ssTr = 6 (75.67 77.35)2 + 7 (78.43 77.35)2
+ 6 (70.83 77.35)2 + 4 (87.75 77.35)2 = 712.59
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
26 / 48
11. Analysis of Variance (ANOVA)
11.2 One-way Analysis of Variance
ANOVA: example
From there, the ANOVA table can be easily completed:
Source
degrees
of freedom
sum of
squares
mean
square
F -statistic
Treatments
dfTr = 3
ssTr = 712.59
msTr = 237.53
f0 = 3.77
Error
dfEr = 19
ssEr = 1196.63
msEr = 62.98
Total
dfTot = 22
ssTot =1909.22
Is f0 = 3.77 much larger than 1?
; compare to the appropriate F -distribution critical value
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
27 / 48
11. Analysis of Variance (ANOVA)
11.2 One-way Analysis of Variance
ANOVA: example
According to M ATLAB, f3,19;0.95 = 3.1274 (in the table: f3,20;0.95 = 3.10)
; the decision rule is:
reject H0 if
msTr
> 3.1274
msEr
Here, we have observed
f0 =
msTr
= 3.77
msEr
; reject H0
We can claim that the teaching technique does have an influence on
the mean achievement of the students (with less than 5% chance of
being wrong)
The associated p-value is
p = P(X > 3.77) = 0.0281
(M ATLAB again)
CVEN2002/2702 (Statistics)
for X F3,19
; indeed, p < = 0.05 (reject H0 )
Dr Justin Wishart
Session 2, 2012 - Week 12
28 / 48
11. Analysis of Variance (ANOVA)
11.3 Multiple pairwise comparisons
ANOVA: confidence intervals on treatment means
The ANOVA F -test will tell you whether the means are all equal or not,
but nothing more
When the null hypothesis of equal means is rejected, we will usually
want to know which of the i s are different from one another
A first step in that direction is to build confidence intervals for the
different means i
From our assumptions (normal populations, random samples, equal
variance 2 in each group), we have
Xi N i ,
ni
The value of 2 is unknown, however we have (numerous!) estimators
for it
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
29 / 48
11. Analysis of Variance (ANOVA)
11.3 Multiple pairwise comparisons
ANOVA: confidence intervals on treatment means
For instance, the MSEr is an unbiased estimator for 2 with n k
degrees of freedom
This one is based on all the n observations from the global sample
; it has smaller variance (i.e. it is more accurate) than any other
(like e.g. Si ), and should always be used in the ANOVA framework!
Acting as usual, we can conclude that
i i
X
ni p
tnk
MSEr
and directly write a 100 (1 )% two-sided confidence interval for
i , from the observed values xi and msEr :
r
r
msEr
msEr
xi tnk ,1/2
, xi + tnk ,1/2
ni
ni
; these confidence intervals for each group will tell which values i s
are much different from one another and which ones are close
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
30 / 48
11. Analysis of Variance (ANOVA)
11.3 Multiple pairwise comparisons
ANOVA: confidence intervals on treatment means
For instance, in the previous example (mean achievement for teaching
techniques), we would find, with t19;0.975 = 2.093 (table) and msEr = 62.98:
q
95% CI for 1 = [75.67 2.093 62.98
] = [68.89, 84.45]
q 6
95% CI for 2 = [78.43 2.093 62.98
] = [72.15, 84.71]
q 7
] = [64.05, 77.61]
95% CI for 3 = [70.83 2.093 62.98
q 6
95% CI for 4 = [87.75 2.093 62.98
4 ] = [79.45, 96.06]
3
2
2
1
teaching technique
4
3
60
70
80
90
100
; it seems clear that 3 6= 4 is the main reason for rejecting H0
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
31 / 48
11. Analysis of Variance (ANOVA)
11.3 Multiple pairwise comparisons
ANOVA: pairwise comparisons
It is also possible to build confidence intervals for the differences
between two means i and j . From observed values xi , xj and msEr , a
100 (1 ) % confidence interval for i j is
s
"
1
1
(xi xj ) tnk ;1/2 msEr
+
,
ni
nj
s
#
1
1
(xi xj ) + tnk ;1/2 msEr
+
ni
nj
for any pair of groups (i, j) (compare Slide 26 Week 10)
Finding the value 0 in such an interval is an indication that i and j
are not significantly different. On the other hand, if the interval does
not contain 0, that is evidence that i 6= j
However, these confidence intervals are sometimes misleading and
must be carefully analysed, in particular when related to the global null
hypothesis H0 : 1 = 2 = . . . = k
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
32 / 48
11. Analysis of Variance (ANOVA)
11.3 Multiple pairwise comparisons
ANOVA: pairwise comparisons
Suppose that for a pair (i, j), the 100 (1 )% confidence interval for
i j does not contain 0
(i,j)
; at significance level %, you would reject H0
: i = j (Sl. 44 W9)
If i 6= j then automatically H0 : 1 = 2 = . . . = k is contradicted
; should you also reject H0 at significance level %? No
(i,j)
When you reject H0 : i = j at significance level %, you essentially
keep a % chance of being wrong
(1,2)
Successively testing H0
(1,3)
: 1 = 2 , and then H0
: 1 = 3 , and
(k 1,k )
H0
then . . ., and then finally
: k 1 = k , that is
k
k!
K =
=
pairwise comparisons,
2
2!(k 2)!
greatly increases the chance of making a wrong decision
(look back at Example Slide 32 Week 3)
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
33 / 48
11. Analysis of Variance (ANOVA)
11.3 Multiple pairwise comparisons
ANOVA: pairwise comparisons
Suppose that H0 : 1 = 2 = . . . = k is true
(i,j)
If the decisions made for each of the K pairwise tests H0 : i = j
were independent (which they are not! why?), we would wrongly reject
at least one null hypothesis with probability 1 (1 )K (why?)
If the decisions were perfectly dependent (which they are not either!),
we would wrongly reject at least one null hypothesis with probability
(why?)
; if we based our decision about H0 : 1 = 2 = . . . = k on the
pairwise comparison tests, we would wrongly reject H0 with a
probability strictly between and 1 (1 )K , larger than !
To fix ideas, suppose k = 4 groups, which would give K = 42 = 6
pairwise comparisons, and = 0.05
; the test based on pairwise comparisons would be of effective
significance level between 0.05 and 1 (1 0.05)6 = 0.265
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
34 / 48
11. Analysis of Variance (ANOVA)
11.3 Multiple pairwise comparisons
Pairwise comparisons: Bonferonni adjustments
It is usually not possible to determine exactly the significance level of
such a test: it all depends on the exact level of dependence between
the decisions about the different pairwise comparisons
Several procedures have been proposed to overcome this difficulty, the
simplest being the Bonferonni adjustment method
It is based on the Bonferonni inequality (see Exercise 1 Tut. Week 5):
P(A1 A2 . . . AK ) P(A1 ) + P(A2 ) + . . . + P(AK )
Suppose that Aq is the event we wrongly reject H0 for the qth pairwise
comparison. Then, the event B = (A1 A2 . . . AK ) is the event we
wrongly reject H0 : 1 = . . . = k
; if we want P(B) , it is enough to take P(Aq ) =
for all q
Hence, to guarantee an overall significance level of at most %, the
pairwise comparison tests must be carried
out at significance level
/K % (instead of %), where K = k2
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
35 / 48
11. Analysis of Variance (ANOVA)
11.3 Multiple pairwise comparisons
Pairwise comparisons: example
In our running example, we have k = 4 groups, and we can run K = 6
pairwise two-sample t-tests
We can find:
t-test for H0 : 1 = 2
; p-value = 0.5276
t-test for H0 : 1 = 3
; p-value = 0.3691
t-test for H0 : 1 = 4
; p-value = 0.0346
t-test for H0 : 2 = 3
; p-value = 0.1293
t-test for H0 : 2 = 4
; p-value = 0.0537
t-test for H0 : 3 = 4
; p-value = 0.0139
At level 5%, we reject H0 : 1 = 4 and H0 : 3 = 4
From this, can we reject H0 : 1 = 2 = 3 = 4 at level 5%? No
; we must compare the above p-values to /K = 0.05/6 = 0.0083
None are smaller than 0.0083 ; do not reject H0 : 1 = 2 = 3 = 4 !
The ANOVA test did reject H0 . Is that a contradiction?
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
36 / 48
11. Analysis of Variance (ANOVA)
11.4 Adequacy of the ANOVA model
Adequacy of the ANOVA model
The ANOVA model is based on several assumptions that should be
carefully checked
The central assumption here is that the random variables ij = Xij i ,
i = 1, . . . , k and j = 1, . . . , ni , are (1) independent and (2) normally
distributed:
i.i.d.
ij N (0, ),
with (3) the same variance in each group
We do not have access to values for ij (i s are unknown!), however
we can approximate these values by the observed residuals
ij = xij xi
e
Note that these residuals are the quantities arising in ssEr
; as for a regression model (see Slides 41-42 Week 11), the
adequacy of the ANOVA model is established by examining the
residuals
; residual analysis
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
37 / 48
11. Analysis of Variance (ANOVA)
11.4 Adequacy of the ANOVA model
Residuals analysis
The normality assumption can be checked by constructing a normal
quantile plot for the residuals
The assumption of equal variances in each group can be checked by
plotting the residuals against the treatment level (that is, xi )
; the spread in the residuals should not depend on any way on xi
A rule-of-thumb is that, if the ratio of the largest sample standard
deviation to the smallest one is smaller than 2, the assumption of equal
population variances is reasonable
The assumption of independence can be checked by plotting the
residuals against time, if this information is available
; no pattern, such sequences of positive and negative residuals,
should be observed
As for the regression, the residuals are everything the model will not
consider ; no information should be observed in the residuals, they
should look like random noise
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
38 / 48
11. Analysis of Variance (ANOVA)
11.4 Adequacy of the ANOVA model
Residual analysis: example
For our running example, a normal quantile plot and a plot against the
fitted values xi for the residuals are shown below:
Normal QQ Plot
residuals
10
residuals
Theoretical Quantiles
10
10
10
65
Sample Quantiles
70
75
80
85
90
95
; nothing (obvious) to report
; the assumptions we made look valid
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
39 / 48
11. Analysis of Variance (ANOVA)
11.5 Blocking factor
Residual analysis: example
Example
To assess the reliability of timber structures, researchers have studied
strength factors of structural lumber. Three species of Canadian softwood
were analysed for bending strength (Douglas Fir, Hem-Fir and
Spruce-Pine-Fir). Wood samples were selected from randomly selected
sawmills. The results of the experiment are given below. Is there any
significant difference in the mean bending parameters among the three types
of wood?
Douglas (1)
370
150
372
145
374
365
Hem (2)
381
401
175
185
374
390
Spruce (3)
440
210
230
400
386
410
; an ANOVA was run to test the null hypothesis H0 : 1 = 2 = 3 , against
the alternative Ha : not all the means are equal
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
40 / 48
11. Analysis of Variance (ANOVA)
11.5 Blocking factor
Residual analysis: example
We computed values for the ANOVA table:
Source
degrees
of freedom
sum of
squares
mean
square
F -statistic
Treatment
7544
3772
0.33
Error
15
172929
11529
Total
17
180474
In the F -distribution table, we can find that f2,15;0.95 = 3.68
; here we have observed f0 = 0.33 ; do not reject H0 !
Associated p-value: p = P(X > 0.33) = 0.726 for X F2,15
; we confidently claim that there is no significant difference in the
mean bending parameters for the different wood types
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
41 / 48
11. Analysis of Variance (ANOVA)
11.5 Blocking factor
Residual analysis: example
Residual analysis:
residuals
100
Normal QQ Plot
50
residuals
50
Theoretical Quantiles
100
150
150
100
50
50
100
280
290
300
Sample Quantiles
310
320
330
340
350
; the assumptions are clearly not fulfilled!
; the above conclusion is certainly not reliable!
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
42 / 48
11. Analysis of Variance (ANOVA)
11.5 Blocking factor
Blocking factor
450
If we had plotted the data first, we would have seen:
400
300
250
bending parameter
350
200
150
Mill 1
Mill 2
Mill 3
Mill 4
Mill 5
Mill 6
tree type
(Bottom line: always plotting the data before analysing them!)
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
43 / 48
11. Analysis of Variance (ANOVA)
11.5 Blocking factor
Blocking factor
It is clear that over and above the wood type, the mills where the
lumber was selected is another source of variability, in this example
even more important than the main treatment of interest (wood type)
This kind of extra source of variability is known as a blocking factor,
as it essentially groups some observations in blocks across the initial
groups ; the samples are not independent! (assumption violation)
; a potential blocking factor must be taken into account!
When a blocking factor is present, the initial Error Sum of Squares,
, that is the whole amount of variability not due to the
say SSEr
treatment, can in turn be partitioned into:
1
the variability due to the blocking factor, quantified by SSBlock
2
the true natural variability in the observations SSEr
= SS
We can write thatSSEr
Block + SSEr , and thus
SSTot = SSTr + SSBlock + SSEr
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
44 / 48
11. Analysis of Variance (ANOVA)
11.5 Blocking factor
Blocking factor
The ANOVA table becomes:
Source
degrees
of freedom
sum of
squares
mean
square
Treatment
k 1
ssTr
msTr =
Block
b1
ssBlock
msBlock =
Error
nk b+1
ssEr
Total
n1
ssTot
msEr =
ssTr
k 1
F -statistic
f0 =
msTr
msEr
ssBlock
b1
ssEr
nk b+1
where b is the number of blocks
msTr
Note: the test statistic is again the ratio ms
(we have just removed the
Er
variability due to the blocking factor first), to be compared with the
quantile of the Fk 1,nk b+1 distribution
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
45 / 48
11. Analysis of Variance (ANOVA)
11.5 Blocking factor
Blocking factor
In the previous example, we would have found:
Source
degrees
of freedom
sum of
squares
mean
square
F -statistic
Treatment
7544
3772
15.87
Block
170552
34110
Error
10
2378
238
Total
17
180474
In the F -distribution table, we can find that f2,10;0.95 = 4.10
Here, we have observed f0 = 15.87 ; clearly reject H0 !
Associated p-value: p = P(X > 15.87) = 0.0008 for X F2,10
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
46 / 48
11. Analysis of Variance (ANOVA)
11.5 Blocking factor
Blocking factor: comments
The SSEr in the first ANOVA (without block) was 172,929 which contains an
amount of variability 170,552 due to mills
; about 99% of the initial SSEr
was due to mill to mill variability, and so was
no natural variability!
The second ANOVA (with blocking factor) adjusts for this effect
The net effect is a substantial reduction in the genuine MSEr , leading to a
larger F -statistic (increased from 0.33 to 15.87!)
; with very little risk of being wrong (p ' 0), we can now conclude that
there is a significant difference in the mean bending parameters for
the three different wood types
An analysis of the residuals in this second ANOVA would not show anything
peculiar ; valid conclusion
Generally speaking, ignoring a blocking factor leads to a misleading
conclusion, and it should always be carefully assessed whether a
blocking factor may exist or not (plot the data!)
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
47 / 48
11. Analysis of Variance (ANOVA)
Objectives
Objectives
Now you should be able to:
conduct engineering experiments involving a treatment with a
certain number of levels
understand how the ANOVA is used to analyse the data from
these experiments
assess the ANOVA model adequacy with residual plots
understand the blocking principle and how it is used to isolate the
effect of nuisance factors
Recommended exercises: Q3, Q6 p.406, Q9 p.407, Q10, Q11 p.412,
Q13, Q15, Q17 p.413, Q19 p.414, Q22, Q23 p.415, Q35 p.428
CVEN2002/2702 (Statistics)
Dr Justin Wishart
Session 2, 2012 - Week 12
48 / 48
