AIAA JOURNAL

Vol. 33, No. 10, October 1995

Engineering Application of Experimental Uncertainty Analysis

Hugh W. Coleman*
University of Alabama in Huntsville, Huntsville, Alabama 35899
and
W. Glenn Steele1
Mississippi State University, Mississippi State, Mississippi 39762

Publication in late 1993 by the International Organization for Standardization (ISO) of the Guide to the Expres-
sion of Uncertainty in Measurement in the name of ISO and six other international organizations has, in everything
but name only, established a new international experimental uncertainty standard. In this article, an analysis of
the assumptions and approximations used in the development of the methods in the ISO guide is presented, and
a comparison of the resulting equation with previously published uncertainty analysis approaches is made. Also
Downloaded by BIBLIOTHEK DER TU MUENCHEN on August 12, 2013 | http://arc.aiaa.org | DOI: 10.2514/3.12742

discussed are the additional assumptions necessary to achieve the less complex "large sample" methodology that
is recommended in AIAA Standard S-071-1995, Assessment of Wind Tunnel Data Uncertainty, issued in 1995. It
is shown that these assumptions are actually less restrictive than those associated with some previously accepted
methodologies. The article concludes with a discussion of some practical aspects of implementing experimental
uncertainty analysis in engineering testing.

I. Introduction with a summary and a discussion of some practical aspects of imple-

menting experimental uncertainty analysis in engineering testing.
D URING the past decade, and particularly in the past two years,
there have been major developments in the field of experi-
mental uncertainty analysis (or "measurement uncertainty," as it is II. Uncertainty Analysis Overview
often called). Publication in late 1993 by the International Organi- The word accuracy is generally used to indicate the relative close-
zation for Standardization (ISO) of the Guide to the Expression of ness of agreement between an experimentally determined value of
Uncertainty in Measurement1 in the name of ISO and six other inter- a quantity and its true value. Error (8) is the difference between the
national organizations has, in everything but name only, established experimentally determined value and the truth, and thus as error
a new international experimental uncertainty standard. The method- decreases, accuracy is said to increase. Only in rare instances is the
ology (but not the complete terminology) of the ISO guide is being true value of a quantity known. Thus, one is forced to estimate error,
adopted by groups working on formulating or revising guidelines, and that estimate is called an uncertainty U. Uncertainty estimates
procedures, and standards in the experimental uncertainty area—a are made at some confidence level—a 95% confidence estimate, for
NATO AGARD Working Group on Quality Assessment for Wind example, means that the true value of the quantity is expected to be
Tunnel Testing,2 of which the first author was a member, adopted the within the ±C7 interval about the experimentally determined value
methodology, as has the committee (of which the second author is 95 times out of 100.
a member), currently revising the American National Standards In- As shown in Fig. la, total error 8 can be considered to be com-
stitute (ANSI)/American Society of Mechanical Engineers (ASME) posed of two components: a precision (random) component £ and a
Standard on Measurement Uncertainty.3 bias (systematic) component f>. An error is classified as precision if
The authors' objective in this article is to present an analysis of it contributes to the scatter of the data; otherwise, it is a bias error. It
the assumptions and approximations used in the development of is assumed that corrections have been made for all systematic errors
the methods in the ISO guide and a discussion of the assumptions whose values are known. The remaining bias errors are thus equally
necessary to achieve a less complex "large sample" methodology as likely to be positive as negative.
that is applicable to the vast majority of engineering testing. As will Suppose that we are making a number of measurements of the
be seen, these assumptions are actually less restrictive than those value of a variable X that is steady. The k and k -+- 1 measurements
associated with some previously accepted methodologies.3"5 The are shown in Fig. la. Since the bias is a fixed error, it is the same
large sample methodology is that recommended in the AGARD for each measurement. However, the precision error will have a
report,2 the draft of the revision to Ref. 3, and the newly issued different value for each measurement. It then follows that the total
AIAA standard.6 error in each measurement will be different, since the total error is
In Sec. II of this article a brief overview of uncertainty analysis the sum of the bias error and precision error in a measurement.
is presented; in Sec. Ill a derivation of the uncertainty propagation If we continued to take measurements as previously described un-
equation is presented with particular discussion of the assumptions til we had a sample of N readings, more than likely as TV approached
and approximations made; in Sec. IV a comparison of the resulting infinity the data would behave as shown in Fig. Ib. The bias error
equation with previously published uncertainty analysis approaches would be given by the difference between the mean (average) value
is presented; in Sec. V additional assumptions are discussed that lead IJL of the N readings and the true value of X, whereas the precision
to a less complex uncertainty analysis approach that is suitable for errors would cause the frequency of occurrence of the readings to
most engineering applications; and the article concludes in Sec. VI be distributed about the mean value.
As an estimator of ft, a bias limit B is defined.7 A 95% confidence
Received June 3, 1994; revision received March 13, 1995; accepted for estimate is interpreted as the experimenter being 95% confident that
publication March 15, 1995. Copy right © 1995 by Hugh W. Coleman and the true value of the bias error, if known, would fall within ±B. A
W. Glenn Steele. Published by the American Institute of Aeronautics and
Astronautics, Inc., with permission.
useful approach to estimating the magnitude of a bias error is to
*Eminent Scholar in Propulsion and Professor, Propulsion Research Cen- assume that the bias error for a given case is a single realization
ter, Department of Mechanical and Aerospace Engineering. Senior Member drawn from some statistical parent distribution of possible bias er-
AIAA. rors. For example, suppose a thermistor manufacturer specifies that
^Professor and Head, Department of Mechanical Engineering. Senior 95% of samples of a given model are within ±1.0 C of a reference
Member AIAA. resistance-temperature (R-T) calibration curve supplied with the
1888
 COLEMAN AND STEELE 1889

- 0(bias)-

a) Xtn
Downloaded by BIBLIOTHEK DER TU MUENCHEN on August 12, 2013 | http://arc.aiaa.org | DOI: 10.2514/3.12742

Fig. 2 Propagation of bias errors and precision errors into an experi-

b) Magnitude of X mental result.

Fig. 1 Errors in the measurement of a variable X: a) two readings and where the function is continuous and has continuous derivatives in
b) infinite number of readings. the domain of interest. The situation is shown schematically in Fig. 2
for the kih set of measurements (x*, yk) that is used to determine rk.
thermistors. One might assume that the bias errors (the differences Here, pxk and €Xk are the bias and precision errors, respectively, in
between the actual, but unknown, R-T curves of the various thermis- the fcth measurement of x, with a similar convention for the errors
tors and the reference curve) belong to a Gaussian parent distribution in y and in r. Assume that the test instrumentation and/or apparatus
with a standard deviation b — 0.5 C. Then the interval defined by is changed for each measurement so that different values of f>Xk and
±B = ±2b = ±1.0 C would include about 95% of the possible Pyk will occur for each measurement. Therefore, the bias errors and
bias errors that could be realized from the parent distribution. (The precision errors will be random variables so that
bias limit is sometimes called the "systematic uncertainty.") xk = jctrue + .fix + ex (4)
As an estimator of the magnitude of the precision errors (the
width of the distribution of readings in Fig. Ib) a precision limit P (5)
= )>true + Pyk + ^
is defined.7 A 95% confidence estimate of P is interpreted to mean
that the ±P interval about a single reading of X, should cover [L Now approximate the function r in the DRE using a Taylor series
95 times out of 100. (The precision limit is sometimes called the expansion. Expanding to the general point rk from rtrue gives
"precision uncertainty.")
In nearly all experiments, the measured values of different vari- = ''true + —— (Xk ~ *true) + —— (yk ~ Jtrue) + (6)
ables are combined using a data reduction equation (DRE) to form
some desired result. A good example is the experimental determi- where R2 is the remainder term and where the partial derivatives
nation of drag coefficient of a particular model configuration in a are evaluated at (jctrue, ytrue)- Since the true values of x and y are un-
wind-tunnel test. Defining drag coefficient as known, an approximation is always introduced when the derivatives
are evaluated at some measured values (xk, yk).
(1) The remainder term has the form8
pV2A
~ "T~r ~ *true)
one can envision that errors in the values of the variables on the right- 2\\_dx2 dxdy
hand side of Eq. (1) will cause errors in the experimental result CD.
A more general representation of a data reduction equation is
— (yk- (7)
= r(XlfX2,...,Xj) (2)
where the partial derivatives are evaluated at (£, x) which is some-
where between (xk, yk) and (jctrue, ytrue). This term is usually assumed
where r is the experimental result determined from / measured
to be negligible, and so it is useful to consider the conditions under
variables X/. Each of the measured variables contains bias errors and
which this assumption might be reasonable. The factors (xk — jctrue)
precision errors. These errors in the measured values then propagate
and (yk — ytrue) are the total errors in x and y. If the derivatives are of
through the data reduction equation, thereby generating the bias
reasonable magnitude and the total errors in x and y are small, then
and precision errors in the experimental result r. In the following
R2, containing the squares of the errors, will approach zero more
section, a derivation of the equation describing the error propagation
quickly than will the first-order terms. Also, if r(jc, y) is a linear
is presented, with particular attention paid to the assumptions and
function, then the partial derivatives in Eq. (7) are identically zero
approximations that are made to obtain the final equation.
(as is /?2).
Neglecting R2, the expansion gives [taking rtrue to the left-hand
III. Derivation of the Uncertainty side (LHS)]
Propagation Equation
Rather than present the derivation for the case in which the result 3r 9r
——(Xk- *true) + —— (j* ~ (8)
is a function of many variables, the simpler case in which the result dx ay
is a function of only two variables will be considered first. The
This expression relates the total error 8 in the Ath determination of
expressions for the more general case will then be presented as the result r to the total errors in the measured variables and using
extensions of the two-variable case.
the notation
Suppose that the data reduction equation is

r = r(x, y) (3)
', = (9)
 1890 COLEMAN AND STEELE

can be written as Table 1 Two-tailed t distribution giving t values

for a confidence level C and number of degrees of
(10) freedom i/
V C = 95.0% C = 99.0%
We are interested in obtaining a measure of the distribution of the
various 8r for (some large number) N determinations of the result r. 1 12.706 63.657
The variance of the parent distribution is defined by 2 4.303 9.925
3 3.182 5.841
4 2.776 4.604
5 2.571 4.032
(11) 6 2.447 3.707
7 2.365 3.499
8 2.306 3.355
Substituting Eq. (10) into Eq. (11) but deferring taking the limit 9 2.262 3.250
gives 10 2.228 3.169
11 2.201 3.106
12 2.179 3.055
13 2.160 3.012
14 2.145 2.977
15 2.131 2.947
16 2.120 2.921
Downloaded by BIBLIOTHEK DER TU MUENCHEN on August 12, 2013 | http://arc.aiaa.org | DOI: 10.2514/3.12742

17 2.110 2.898
k=\ k=\ 18 2.101 2.878
19 2.093 2.861
-, N N . N
20 2.086 2.845
* tf E y 25 2.060 2.787
N E 30 2.042 2.750
60 2.000 2.660
00 1.960 2.576

(termed the expanded uncertainty in the ISO guide) at some speci-

(12) fied confidence level (95%, 99%, etc.), the combined standard un-
certainty uc must be multiplied by a coverage factor K,
Taking the limit as N approaches infinity and using definitions of
variances similar to that in Eq. (1 1), we obtain Ur = Kuc (17)

It is in choosing K that assumptions about the type(s) of the error

(13) distributions must be made.
An argument is presented in the ISO guide1 that the error distri-
assuming that there are no bias error/precision error correlations so bution of the result r may often be considered Gaussian because of
that in Eq. (12) the final four terms containing p€ products are zero. the Central Limit Theorem, even if the error distributions of the X/
Since in reality we never know the various a exactly, we must are not normal. In fact, the same argument can be made for approx-
use estimates of them. Defining u2 as an estimate of the variance imate normality of the error distributions of the X,- since the errors
of the distribution of total errors in the result, b2 as an estimate of typically are composed of a combination of errors from a number
the variance of a bias error distribution, and S2 as an estimate of the of elemental sources.
variance of a precision error distribution, we can write If it is assumed that the error distribution of the result r is nor-
mal, then the value of K for C% coverage corresponds to the C%
+ 02b] 20x6ybxy + 02S2 + 02S2 + 20xOySxy (14) confidence level t value from the t distribution (Table 1) so that
In Eq. (14) bxy is an estimate of the covariance of the bias errors in
x and the bias errors in 3; that is defined exactly by E ftM'2M
= \im(N -> oo) J-l
(15)
(18)

Likewise, Sxy is an estimate of the covariance of the precision errors

in x and y. In keeping with the nomenclature of the ISO guide,1 uc The effective number of degrees of freedom vr for deter-
is called the combined standard uncertainty. For the more general mining the t value is given (approximately) by the so-called
case in which the experimental result is determined from Eq. (2), uc Welch-Satterthwaite formula as1
is given by

e
«? = E +2 E +E > vr =
E/.,
(19)

/-i
where the v.y/ are the number of degrees of freedom associated with
(16) the Si and the v/,/ are the number of degrees of freedom to associate
with the hi.
where (&/) 2 is the estimate of the variance of the bias error dis- If an Si has been determined from Ni readings of X, taken
tribution of variable X/, etc. The derivation to this point has been over an appropriate interval, the number of degrees of freedom is
presented by the authors in Ref. 9 and in Appendix B of Ref. 7. given by
No assumptions about type(s) of error distributions are made to
obtain the preceding equation for uc. To obtain an uncertainty Ur v,t =N,-l (20)
 COLEMAN AND STEELE 1891

For the number of degrees of freedom v/,/ to associate with a non- Approach of Coleman and Steele7'9
statistical estimate of £/, it is suggested in the ISO guide1 that one The authors, expanding on the ideas advanced by Kline and
might use the approximation McClintock10 and assuming Gaussian error distributions, proposed
viewing Eq. (14) as a propagation equation for 68% confidence in-
1 /Afr, tervals. A 95% coverage estimate of the uncertainty in the result was
(21) then proposed as that given by an equation similar to Eq. (23)
2\ bi
(28)
where the quantity in parentheses is the relative uncertainty of £/.
For example, if one thought that the estimate of &/ was reliable to with the bias limit of the result defined by
within ±25%, then j-i
B
=
(22)

Consideration of the 95% confidence t values in Table 1 reveals and the precision limit of the result given by
that the value of t approaches 2.0 (approximately) as the number of
degrees of freedom increases. We thus face the somewhat paradox- (30)
ical situation that as we have more information (vr increases), we
can take t equal to 2.0 and do not have to deal with Eq. (19), but for
Downloaded by BIBLIOTHEK DER TU MUENCHEN on August 12, 2013 | http://arc.aiaa.org | DOI: 10.2514/3.12742

the cases in which we have little information (vr small) we need to where pbik is the correlation coefficient appropriate for the bias errors
make the more detailed estimates required by Eq. (19). In Sec. V of in X/ and Xk, and psik is the correlation coefficient appropriate for
this article, we examine the assumptions required to discard Eq. (19) the precision errors in X, and X*. The precision limit of the variable
and to simply use t equal to 2 (for 95% confidence). X/ is given by
Pi=ttSi (31)
IV. Comparison with Published Approaches
Approach of Abernethy et al.5 where ^ is determined with v, = TV/ — 1 degrees of freedom.
An approach that was widely used in the 1970s and 1980s was Equations (29) and (30) were viewed as propagation equations
the URss, ^ADD technique formulated by Abernethy et al.5 and used for 95% confidence bias limits and precision limits, and thus this
in Refs. 3 and 4 and other Society of Automotive Engineers (SAE), approach avoided use of the Welch-Satterthwaite formula. Compar-
Instrument Society of America (ISA), JANNAF, National Research ison of uncertainty coverages for a range of sample sizes using this
Council (NRC), U.S. Air Force (USAF), NATO, and ISO standards approach and the Abernethy et al. approach have been previously
documents.5 According to Abernethy et al., presented.11 As was stated earlier in reference to the Abernethy et
al. approach, consideration of these expressions in the context of the
derivation presented in the previous section shows that they cannot
URSS = [B2r + (tSr)2] i (23) be justified on a rigorous basis.
Both the approach of Coleman and Steele7 and the Abernethy et al.
for a 95% confidence estimate and URSS approach5 (properly modified to account for correlated bias
effects) agree with the 95% confidence form of Eq. (18) for "large
= Br + (tSr) (24) sample sizes"—that is, Nt (and vr) large enough so that t can be taken
as 2.0. As will be discussed in Sec. V, an argument can be made that
for a 99% confidence estimate, where Br is given by this corresponds roughly to Nt > 10.

Approach of ISO Guide1

The ISO guide1 was published in late 1993 in the name of seven
(25) international organizations: the Bureau International des Poids et
Mesures (BIPM), the International Electrotechnical Commission
(IEC), the International Federation of Clinical Chemistry (IFCC),
and Br and the various Bt are 95% confidence bias limit esti- the International Organization for Standardization (ISO), the Inter-
mates, and national Union of Pure and Applied Chemistry (IUPAC), the In-
ternational Union of Pure and Applied Physics (IUPAP), and the
International Organization of Legal Metrology (OIML). It is now,
(26) in the authors' opinion, the de facto international standard.
One fundamental difference between the approach of the guide
and that of Eqs. (16), (18), and (19) in this article is that the guide
where t is the 95% confidence t value from the t distribution for vr uses u (xt), a "standard uncertainty," to represent the quantities b[ and
degrees of freedom given by Si used in this article. Instead of categorizing uncertainties as either
bias (systematic) or precision (random), the various u are divided
(ft-*)4
into type A standard uncertainties and type B standard uncertainties.
yr = (27) TVpe A uncertainties are those evaluated "by the statistical analysis
of series of observations," whereas type B uncertainties are those
evaluated "by means other than the statistical analysis of series of ob-
Consideration of these expressions in the context of the derivation servations." These do not correspond to the categories described by
presented in the previous section shows that they cannot be justified the traditional engineering usage: precision (or random or repeata-
on a rigorous basis. The t/ADD approach has always been advanced bility) uncertainty and bias (or systematic or fixed) uncertainty.
on the basis of ad hoc arguments and with results from a few Monte Arguments can, of course, be made for both sets of nomencla-
Carlo simulations, but (as argued in the ISO guide1) for a 99% ture. Types A and B unambiguously define how an uncertainty es-
confidence level, the t value appropriate for 99% confidence should timate was made, whereas systematic and precision uncertainties
be used as the value of K in Eq. (17) to obtain a 99% confidence can change from one category to the other in a given experimental
estimate for an assumed Gaussian distribution. The Abernethy et al. program depending on the experimental process used—a systematic
approaches also ignore the possibility of correlated bias error effects calibration uncertainty can become a source of scatter (and thus a
[taken into account in the bik covariance terms in Eq. (18)], although precision uncertainty) if a new calibration is done before each read-
Ref. 3 does consider this effect in one example. ing in the sample is taken, for example. On the other hand, if one
 1892 COLEMAN AND STEELE

wants an estimate of the expected dispersion of results for a particu- little consideration in the engineering "measurement uncertainty"
lar experimental approach or process, then the systematic/precision literature. As stated in the ISO guide,1
categorization is useful—particularly when used in the "debugging"
phase of an experiment.7 This ... shows that the standard deviation of a statistically esti-
Considering the tradition in engineering of the bias/precision mated standard deviation is not negligible for practical values
(systematic/random) uncertainty categorization and its usefulness of n. One may therefore conclude that Type A evaluations
in engineering experimentation as mentioned earlier, this article's of standard uncertainty are not necessarily more reliable than
Type B evaluations, and that in many practical measurement
authors have chosen to retain that categorization while adopting the situations where the number of observations is limited, the
mathematical procedures of the ISO guide. This categorization is components obtained from Type B evaluations may be better
also used in the AGARD2 document, by the committee currently known than the components obtained from Type A evaluations.
revising Ref. 3, and in the AIAA standard.6
Considering the 95% confidence t table (Table 1), one can ob-
12
serve that for vr > 9 the values of t are within about 13% of the
Approach of NIST large sample t value of 2. This difference is relatively insignificant
Taylor and Kuyatt12 reported guidelines for the implementation compared with the uncertainties inherent in estimating the S/ and
of a National Institute of Standards and Technology (NIST) policy bj as discussed earlier. Therefore, for most engineering applications
that states the following: it is proposed that Gaussian error distributions and vr > 9 be as-
sumed so that t = 2 always. (This could be called the "large sample
assumption.") This assumption eliminates the need for evaluation
Downloaded by BIBLIOTHEK DER TU MUENCHEN on August 12, 2013 | http://arc.aiaa.org | DOI: 10.2514/3.12742

Use expanded uncertainty U to report the results of all NIST

measurements other than those for which uc has traditionally of vr using the Welch-Satterthwaite formula and thus the need to
been employed. To be consistent with current international estimate all of the vvl- and the v/,/.
practice, the value of k to be used at NIST for calculating U Consideration of the Welch-Satterthwaite formula [Eq. (19)]
is, by convention, k = 2. Values of k other than 2 are only to shows that, because of the exponent of 4 in each term, vr is most
be used for specific applications dictated by established and influenced by the number of degrees of freedom of the largest
documented requirements. of the OiSi or 0/fc/ terms. If, for example, O^S^ is dominant, then
vr ™ Vs3 > 9 for 7V3 > 10 [recalling Eq. (20)]. If, on the other hand,
(The coverage factor k corresponds to the K used in this article.) 03&3 is dominant, then vr ~ VM > 9 when the relative uncertainty
The NIST approach is thus that in the ISO guide,1 and no confidence in bi is about 24% or less [recalling Eq. (21)]. Therefore, invoking
level is associated with U when reported by NIST even though the the large sample assumption essentially means that if a 0/5; is dom-
coverage factor is specified as 2.0. inant, then its /V; > 10, or if a 0/fr/ is dominant, then the relative
uncertainty in that &/ is about 24% or less. If there is no single dom-
inant term, but there are M different 0/5/ and 0/fc/ that all have the
Approaches of AGARD2 and AIAA Standard6 same magnitude and same number of degrees of freedom va, then
The first author was a participant in AGARD Fluid Dynamics
Panel Working Group 15 on Quality Assessment for Wind Tunnel vr = Mva (32)
Testing and was the principal author of the methodology chapter
in the resulting report.2 This AGARD report, with minor revisions, If M = 3, for example, va would only have to be 3 or greater for vr
has been issued in 1995 as an AIAA standard.6 The recommended to be equal to or greater than 9. Therefore, t can often legitimately
methodology is that discussed in this article, including the additional be taken as 2 for estimating the uncertainty in a result determined
assumptions discussed later in Sec. V so that t is taken as 2 "unless from several measured variables even when the numbers of degrees
there are other overriding considerations."2 of freedom associated with the measured variables are very small.
If the large sample assumption is made so that t = 2, then the
95% confidence expression for Ur becomes
V. Additional Assumptions for
Engineering Applications
In much engineering testing (such as in most wind-tunnel tests,
for example) it seems that the use of the preceding complex but still
approximate equations [Eqs. (18) and (19)] for Ur and vr derived
(33)
in Sec. Ill of this article would be excessively and unnecessarily
complicated and would tend to give a false sense of the degree of
significance of the numbers computed using them.2-6 The following Recalling the definitions of bias limit and precision limit from
discussion examines what additional simplifying approximations Sec. II, the (2fc/) factors are equal to the 95% confidence bias limits
can reasonably be made for application of uncertainty analysis in Bi. The 2St factors, however, do not necessarily correspond to the
most engineering testing. 95% confidence precision limits, as discussed later. If no further
assumptions are made, we can rewrite Eq. (33) as
Approximating the Coverage Factor
Consider the process of estimating the uncertainty components fBf + 2 etekBik +
(tbi) and (f S,) and obtaining Ur. The propagation equation and the
Welch-Satterthwaite formula are approximate, not exact, and the 7-1
Welch-Satterthwaite formula does not include the influence of cor- (34)
related uncertainties. In addition, unavoidable uncertainties are al-
ways present in estimating the bias uncertainties bf and in estimating
their associated degrees of freedom, vbi. where Bik is the 95% confidence estimator of the covariance of the
In fact, the uncertainty associated with an S, calculated from bias errors in X/ and Xk.
N readings of X/ can be surprisingly large.1 For samples from a
Gaussian parent population with standard deviation or, 95 out of 100 Estimating Precision Limits
determinations of S/ will scatter within an interval of approximately Using a Monte Carlo type simulation, the behavior of the preci-
±0.45a if the Si are determined from N = 10 readings and within sion limit determined from N readings of a variable X was investi-
an interval of approximately ±0.25a if the S/ are determined from gated as a function o f N . A Gaussian parent population with mean ^
N = 30 readings. (A sample with N > 30 has traditionally been and standard deviation a was specified for a variable X. For 7V = 8,
considered a "large" sample.3) This effect seems to have received for example, the results were calculated as follows. Eight readings
 COLEMAN AND STEELE 1893

were randomly drawn from the specified parent population, and a where % is determined with TV/ — 1 degrees of freedom.
sample standard deviation was calculated using In cases in which all TV/ > 10, then the various (25/) in Eq. (34)
correspond to the precision limits P/ so that
j j-i j J
(35)
N-:
i = \ k = i+i 1= 1
where the mean value is defined as
(38)
(36)
where Pik equals 4(Sik).
If we define the bias limit (systematic uncertainty) of the result as
A check was then made to see if the ±(tS) and ±(25) intervals cen-
tered about the next (ninth) randomly chosen reading covered /x,
and the appropriate counter was incremented when the check was (39)
positive. This procedure was followed for 100,000 trials, and the
percent coverage was the percentage of the 100,000 trials in which
JJL was covered. An average, 5avg, of the 100,000 S was calculated, and the precision limit (precision uncertainty) of the result as
Downloaded by BIBLIOTHEK DER TU MUENCHEN on August 12, 2013 | http://arc.aiaa.org | DOI: 10.2514/3.12742

and the precision limit ratios (25avg/2o-) and (f Savg/2cr) were deter- j j-\ j
mined. This entire procedure was followed for N from 3 to 30, and
•P?+2YY
1
' ^ ' -1
e&P* (40)
the results are shown in Figs. 3 and 4.
i = l jfe=i+l
For N = 10, the percent coverage by 2S is 92.5% and the average
25" interval is about 0.98 of the 2a interval, whereas the percent then Eq. (38) can be written as
coverage by t S is 95% and the average t S interval is about 1.10 of the
2<r interval. As N increases, the percent coverage by 2S approaches (41)
95% and the average 25 interval approaches the 2a interval value.
Considering the previous discussion of the dispersion associated and Eqs. (39) and (40) can be viewed as propagation equations for
with determinations of S using N readings, it is proposed that the the bias limits and precision limits, respectively.
95% confidence precision limit for a variable X, be estimated as
Pi = *iSi N -< 10 Single Test
(37) When the result is determined from a single test, that is, at a given
Pi = 2St N > 10 test condition the result is determined once using Eq. (2)
100—• _ r/y v v \ fO\

and when the various X/ are considered single measurements, then

Eq. (40) is used to find the precision limit of the result. This situation
is often encountered in large-scale engineering tests in which mea-
surements of the variables are made at a given set point over a period
that is small compared with the periods of the factors causing vari-
ability in the experiment. A proper precision limit (one indicative of
the dispersion of the variable over several cycles of the factors caus-
ing its variation) cannot be calculated from readings taken over such
a small time interval. For such data, the measurement(s) of a variable
Xj should be considered a single reading—whether the value of X/
is the average of 10,103, or 106 readings taken during the short mea-
10 12 14 16 18 20 22 24 26 28 30
surement time. In such a test, the value for the precision limit to be
0 2
associated with a single reading would have to be based on previous
Number of Readings, N
information about that measurement obtained over the appropriate
Fig. 3 Percent coverage of population mean p, for N readings of a time interval.11 If previous readings of a variable over an appropri-
Gaussian-distributed variable. ate interval are not available, then the experimenter must estimate a
value for P/ using the best information available at that time.2-7
For single tests in which some of the variables (X2 and X3, for
instance) can be determined as averages from multiple readings over
an appropriate time period but the other variables cannot be, then
= r(Xl,X29X3,...,Xj) (42)

and Eq. (40) is used to find the precision limit of the result as follows.
For the variables that are single readings, the various P, are the
precision limits determined from previous information or estimated
from the best available information. For the averaged variables when
N2 and N3 are equal to or greater than 10, P2 and P3 should be taken
as precision limits of means, (2S2)/(N2)1/2 and (2S3)/(W3)1/2, with
the various S calculated using Eq. (35). When N2 and N3 are less
than 10, it is the authors' recommendation that the precision limits
used in Eq. (40) for the averaged variables be taken as (P2)/(A^2)1/2
and (P 3 )/(N 3 ) 1/2 , where P2 and P3 are determined from previous
10 12 14 16 18 20 22 24 26 28 30 information, as is done for the single reading variables.
Number of Readings, N For tests in which multiple readings of all of the variables can
Fig. 4 Ratio of calculated to true precision limit for N readings of a be obtained over an appropriate period, the following method is
Gaussian-distributed variable. recommended.
 1894 COLEMAN AND STEELE

Multiple Tests conducted, these estimates can be updated using the additional infor-
If a test is performed so that M multiple sets of measurements mation gained about the accuracy of the calibration standards, errors
(Xi, X2,..., Xj)k at the same test condition are obtained, then M associated with the calibration process and curvefit procedures, and
results can be determined using Eq. (2) and an average result r can perhaps analytical estimates of installation errors.
be determined using As Moffat14 suggests, there can be additional conceptual bias er-
rors resulting from not measuring the variable whose symbol appears
in the data reduction equation. An example would be a freestream
(43) velocity value measured at a particular axial position on a wind-
tunnel centerline and used as the freestream velocity at that cross
If the M sets of measurements were obtained over an appropriate section in determining CD, but there may be a cross-sectional gra-
time period, the precision limit that should be associated with a dient of velocity at that location causing the average value to be
single result would be different. Hence, the inclusion of an elemental bias term for the
conceptual error would be appropriate.
Pr = tSr (44) Examples of estimating elemental bias limits in engineering ex-
periments are presented in Refs. 2, 6,7, and 15, and the procedures
where t is determined with M — I degrees of freedom [and as in for estimating bias uncertainties assumed from a non-Gaussian dis-
Eq. (37) is taken as 2 for M > 10] and Sr is the standard deviation tribution are presented in Ref. 1.
of the sample of M results
Correlated Bias Limits
Downloaded by BIBLIOTHEK DER TU MUENCHEN on August 12, 2013 | http://arc.aiaa.org | DOI: 10.2514/3.12742

Correlated bias limits are those that are not independent of each
(45) other, typically a result of different measured variables sharing some
identical elemental error sources. It is not unusual for the uncertain-
ties in the results of experimental programs to be influenced by the
The precision limit that should be associated with the average result effects of correlated bias errors in the measurements of several of
is given by the variables. A typical example occurs when different variables
are measured using the same transducer, such as multiple pressures
Pr
(46) sequentially ported to and measured with the same transducer or
temperatures at different positions in a flow measured with a single
with Pr given by Eq. (44). The uncertainty that should be associated probe that is traversed across the flowfield. Obviously, the bias errors
with r is (using the large sample assumption) in the variables measured with the same transducer are not indepen-
dent of one another. Another common example occurs when differ-
(47)
ent variables are measured using different transducers all of which
have been calibrated against the same standard, a situation typical
with Br given by Eq. (39). of the electronically scanned pressure (ESP) measurement systems
in wide use in aerospace test facilities. In such a case, at least a part
Correlated Precision Uncertainties
of the bias error arising from the calibration procedure will be the
The Stk terms in Eq. (34) and the Pik terms in Eq. (38) take into same for each transducer, and thus some of the elemental bias error
account the possibility of precision errors in different variables be- contributions in the measurements of the variables will be correlated.
ing correlated. These terms have traditionally been neglected,3"5'7'10 The Bjk terms in Eqs. (34) or (39) must be approximated—there is
although precision errors in different variables caused by the same in general no way to obtain the data with which to make a statistical
uncontrolled factor(s) are certainly possible and can have a sub- estimate of the covariance of the bias errors in Xi and the bias errors
stantial impact on the value of the precision limit.13 In such cases, in Xj. The approximation of such terms was considered in detail
one would need to acquire sufficient data to allow a valid statisti- in Ref. 16, where it was shown that the approach that consistently
cal estimate of the precision covariance terms to be made if using gives the most satisfactory approximation for the correlated bias
Eqs. (34) or (38). Note, however, that the multiple tests approach us-
limits was
ing Eq. (47) implicitly includes the correlated error effect—a definite
advantage when multiple sets of measurements over an appropriate (49)
time period are available.

Estimating Bias Limits where L is the number of elemental systematic error sources that
Bias Limits of Individual Variables are common for measurements of variables Xt and Xk.
When attempting to estimate the bias limits Bt of the individual If, for example,
variables in Eqs. (34) or (39), one might separate the bias errors that
influence the measurement of a variable into different categories: (50)
calibration errors, data acquisition errors, data reduction errors, test and it is possible for portions of the bias limits and B2 to arise
technique errors, etc. Within each category, there may be several from the same source(s), then Eq. (39) gives
elemental sources of bias. For instance, if for the Jth variable, Xj,
there are M elemental bias errors identified as significant and whose f = Q(E{ -f 02B2 + 20102B12 (51)
bias limits are estimated as (#7)1, ( B j ) 2 , . . . , (Bj)M> then the bias
limit for the measurement of Xj is calculated as the root-sum-square For a case in which the measurements of X\ and X2 are each influ-
(rss) combination of the elemental limits enced by four elemental error sources and sources 2 and 3 are the
same for both X\ and X2, Eq. (48) gives
B,= (48) (52)
*=i
and
The elemental bias limits, (£/)*, must be estimated for each vari-
able Xi using the best information one has available at the time. In the = (B2)\ + (£2)2 + (B2)l + (#2)4 (53)
design phase of an experimental program, manufacturer's specifi-
cations, analytical estimates, and previous experience will typically whereas Eq. (49) gives
provide the basis for most of the estimates. As the experimental
program progresses, equipment is assembled, and calibrations are (54)
 COLEMAN AND STEELE 1895

Elemental
this study has been accomplished, resources can be concentrated,
error sources if necessary, on obtaining detailed estimates of the uncertainties in
the most important variables.
3) A detailed uncertainty analysis7 should then be performed con-
sidering both systematic uncertainties and precision uncertainties.
In doing this, it is sometimes useful to visualize the experiment and
the propagation of uncertainties as shown schematically in Fig. 5.
For each X/, estimates of the bias limit and the precision limit are
usually made.
It is generally easiest to obtain an estimate of the bias limit for
Xi by estimating the bias limits of the significant elemental sources
and combining them using Eq. (48). To achieve this, the sources of
systematic uncertainty for each X/ should be identified by consider-
ing the potential elemental error sources affecting the determination
of that variable. Once the sources of uncertainty have been identi-
fied, their relative significance should be established. This is often
done using order of magnitude estimates of the sources. As a rule
of thumb for a given X,-, those uncertainty sources that are smaller
than one-fourth or one-fifth of the largest sources can usually be
Downloaded by BIBLIOTHEK DER TU MUENCHEN on August 12, 2013 | http://arc.aiaa.org | DOI: 10.2514/3.12742

considered negligible. Resources can then be concentrated on ob-

taining good estimates of those uncertainties of most importance.
In the authors' experience, for most engineering testing it is gener-
ally not cost effective or necessary to try to estimate precision limits
Fig. 5 Propagation of uncertainties into an experimental result. at the elemental error source level. (The authors have observed cases
in which significant resources were squandered through attempts to
obtain good, large sample estimates of elemental precision uncer-
VI. Summary tainties that were of absolutely no consequence in the test program.)
The 95% confidence large sample uncertainty analysis approach It is generally far more effective to estimate the precision limits of
recommended for use in the vast majority of engineering testing can the measured variables (at the Pf level in Fig. 5) and obtain Pr using
be summarized as follows. Eq. (57) or, if possible, computing the direct estimate of Sr as given
In single tests with precision limits estimated using previous in- by Eq. (45) if multiple results at the same set point are available
formation and all NI > 10, (perhaps from previous tests of a similar type).
Tj2 _ r>2 , p2 4) The overall uncertainty, bias limit, and precision limit for the
u r — DT -r rr (55) experimental result r are then found using the preceding equations.
The bias limit of the result is 5) Each experimental result that is reported should be accompa-
nied by its overall uncertainty, reference to an explanation of the
y y-i y estimation of its precision and bias limits, and a discussion of any
(56) implications of the uncertainty on the use that should be made of
i = l k=i+[ the result.
where Bik is given by Eq. (49). The precision limit of the result is Acknowledgments
given by The authors gratefully acknowledge the discussions with and
comments from their faculty colleagues and their students, the mem-
(57) bers of NATO/AGARD Working Group 15, the members of the
ASME Performance Test Codes committee revising Ref. 3, and the
many people who reviewed drafts of Refs. 2 and 6. Kendall K. Brown
assuming no correlated precision uncertainties. performed the Monte Carlo analysis that produced the results shown
In multiple tests, for which there are M results at the same exper- in Figs. 3 and 4.
imental set point, the uncertainty that should be associated with r is
References
(58) 1
Anon., Guide to the Expression of Uncertainty in Measurement, Interna-
tional Organization for Standardization, Geneva, Switzerland, 1993, ISBN
with Br given by Eq. (56) and Sr by Eq. (45). 92-67-10188-9.
2
Experimental uncertainty analysis using the large sample ap- Anon., Quality Assessment for Wind Tunnel Testing, AGARD-AR-304,
proach can be implemented in an engineering test as described next. July 1994.
3
(The authors emphasize that the partial derivatives in the equations Anon., Measurement Uncertainty, American National Standards In-
do not have to be determined analytically; they can be numerically stitute/American Society of Mechanical Engineers, PTC 19.1-1985 Pt. 1,
approximated using a finite difference scheme, for example, that per- American Society of Mechanical Engineers, New York, 1986.
4
Anon., Measurement Uncertainty for Fluid Flow in Closed Conduits,
turbs the data reduction equation and computes a numerical value
American National Standards Institute/American Society of Mechanical
for each 0/.) Engineers, MFC-2M-1983, American Society of Mechanical Engineers,
1) For each experimental result, the data reduction equa- New York, 1984.
tion [Eq. (2)] should be identified; once this has been done, the 5
Abernethy, R. B., Benedict, R. P., and Dowdell, R. B., "ASME
various Xi that must be considered are known. How well each re- Measurement Uncertainty," Journal of Fluids Engineering, Vol. 107,
sult needs to be known (what Ur is acceptable?) should be decided June 1985, pp. 161-164.
6
in the initial planning phase of the experimental program, and this Anon., Assessment of Wind Tunnel Data Uncertainty, AIAA Standard
should be used to guide decisions made in the phases that follow. S-071-1995, Washington, DC, June 1995.
7
2) Those variables that are the most important from an uncer- Coleman, H. W., and Steele, W. G., Experimentation and Uncertainty
tainty viewpoint should be identified. The authors suggest7 initially Analysis for Engineers, Wiley, New York, 1989.
8
Hildebrand, KB., Advanced Calculus for Applications, Prentice-Hall,
assuming all uncertainties are random and using Eq. (57) in a general Englewood Cliffs, NJ, 1962.
uncertainty analysis to investigate the sensitivity of the uncertainty 9
Coleman, H. W., and Steele, W. G., "Some Considerations in the Prop-
in the result to the uncertainties in the variables. This analysis is agation of Bias and Precision Errors into an Experimental Result," Experi-
sometimes done in a parametric study using an assumed range of mental Uncertainty in Fluid Measurements, FED Vol. 58, American Society
uncertainties over the range of anticipated test conditions. Once of Mechanical Engineers, New York, 1987, pp. 57-62.
HEK DER TU MUENCHEN on August 12, 2013 | http://arc.aiaa.or

1896 COLEMAN AND STEELE

1()
Kline, S. J., and McClintock, F. A., "Describing Uncertainties in Precision Errors on the Uncertainty of a Subsonic Venturi Calibration,"
Single-Sample Experiments," Mechanical Engineering, Vol. 75, Jan. 1953, AIAA Paper 95-0797, Jan. 1995.
I4
pp 3-8. Moffat, R. J., "Describing the Uncertainties in Experimental Results,"
n
Steele, W. G., Taylor, R. P., Burrell, R. E., and Coleman, H. W., "The Experimental Thermal and Fluid Science, Vol. 1, No. 1, 1988, pp. 3-17.
15
Use of Data from Previous Experience to Estimate the Precision Uncertainty Coleman, H. W., Hosni, M. H., Taylor, R. P., and Brown, G. B., "Using
of Small Sample Experiments," AIAA Journal, Vol. 31, No. 10, 1993, pp. Uncertainty Analysis in the Debugging and Qualification of a Turbulent Heat
1891-1896. Transfer Test Facility," Experimental Thermal and Fluid Science, Vol. 4,
12
Taylor, B. N., and Kuyatt, C. E., "Guidelines for Evaluating and Ex- No. 6,1991, pp. 673-683.
16
pressing the Uncertainty of NIST Measurement Results," National Inst. of Brown, K. K., Coleman, H. W., Steele, W. G., and Taylor, R. P., "Evalu-
Standards and Technology, NIST TN 1297, Gaithersburg, MD, Sept. 1994. ation of Correlated Bias Approximations in Experimental Uncertainty Anal-
13
Hudson, S. T., Bordelon, W., and Coleman, H. W., "Effect of Correlated ysis," AIAA Paper 94-0772, Jan. 1994.