Data Analytics
for Civil & Environmental Engineers
Lecture 8
Univariate Statistics
1
Univariate Statistics
Empirical distributions
Histograms
Mean, Median, Quartiles, Variance, Skewness,
Kurtosis
Boxplot
Statistical distributions
Discrete distributions
Continuous distributions
• Gaussian distribution and the central limit theorem
• Chi-squared, F, and Student’s t-distributions
2
Univariate Statistics
Statistical testing
Chi-squared test
F-test
Student’s t-test
Extreme value distributions
Generalized extreme value distribution
• Return period
Extreme threshold distributions
• Weibull distribution
3
Empirical distribution: Histogram
Histogram shows the number of data points in a
given data bin
[n,xout]=hist(data)
Syntax %n: row vector if the number of data in each bin
%xout: bin locations
hist (data)
hist(data, # of bins)
hist(data, vector of data bins)
Updated functions:
hist histogram
[n, edges]=histcounts(data)
center=edges(1:end-1)+diff(edges)/2
4
Empirical distribution: Histogram
x=randn (1000,1);
histogram (x)
hist(x, 22) %gives similar results
histogram (x, 50) %50 bins
y=-2:0.1:2;
hist(x,y) %not pretty
histogram(x,y) %much better
5
Empirical distributions
How do we describe a dataset?
Discrete parameters
min, max, mean
Median, quartile
Standard deviation
Variance
Skewness
kurtosis
6
Mean: Why different definitions?
Arithmetic mean
Geometric mean
Harmonic mean
7
Median: write a median function
function m=mymedian(x)
a=sort(x);
b=length(x);
b2=floor(b/2);
if (b/2 > b2) %if mod(b,2)
m=a(b2+1);
else
m=0.5*(a(b2)+a(b2+1));
end
end 8
Quantiles
Divide ordered data into (approximately) equal-
sized subsets of data.
4-quantiles: quartiles
100-quantiles: percentiles
1st quartile: 25th percentile
2nd quartile: median: 50th percentile
9
Quantiles
x=1:15, what is the 3rd quartile?
1. Use the median to divide the data to 2 subsets (do not
include the median value)
2. The upper quartile is the median of the upper half.
The 3rd quartile is 12.
Matlab uses linear interpolation:
prctile(x,[25 50 75])
10
Dispersion of the data: Central moments
11
Moment statistics
12
Moment statistics
Which variable is most sensitive to the difference between the mean
with median values?
13
Moment statistics
14
Moment statistics
15
Dealing with NaN
x=[1:120, NaN];
mean(x), var(x)
nanmean(x), nanvar(x)
skewness(x)
kurtosis(x)
How do we remove the NaN values?
x(isnan(x))=[]
x=x(~isnan(x))
NaN==NaN always return 0; must use isnan
16
Organic matter data
org=load('organicmatter_one.txt');
%checkout the data
plot(org,'o-'), ylabel('wt %')
%histogram
%sqrt of the number of data is often a good
first guess of intervals to use
hist(org,8)
17
Statistics:
18
Historgram
org=load('organicmatter_one.txt');
[n,xout]=hist(org,8);
%n: raw with the number of data of each bin
%xout: bin locations
bar(xout, n, 'r') %red bar
%3d bar
bar3(xout, n, 'b')
19
Sensitivity to outliers
sodium = load('sodiumcontent.txt');
whos sodium
hist(sodium,11)
%add an outliner
sodium2=sodium;
sodium2(121,1) = 0.1;
%sodim2=[sodium;0.1];
Which variable is most sensitive?
20
Sensitivity to outliers
21
Boxplot
boxplot(org)
Box shows the lower quartile, median, and upper quartile
values.
Whiskers show the most extreme data within 1.5 times
interquatile range (25th-75th percentile) from the ends of
the box (25th, 75th percentile)
Red + signs: outliners
load carsmall
boxplot(MPG,Origin)
%MPG is a vector of numbers, Origin a vector of strings that
define “group”
22
Boxplot
23
Box plot: group assignment {}
sodium = load('sodiumcontent.txt');
sodium2=[sodium;0.1];
data=[sodium; sodium2];
name(1:length(sodium))={'original'};
ed= length(sodium);
name(ed+1:ed+length(sodium2))={'outlier'};
boxplot(data, name)
24
Statistical distribution
25
Discrete distribution: Poisson
26
Continuous PDF: Boltzman
Maxwell–Boltzmann distribution
27
https://en.wikipedia.org/wiki/Maxwell%E2%80%93Boltzmann_distribution#/media/File:MaxwellBoltzmann-en.svg
Gaussian (normal) distributions
Syntax
Y=pdf(name, p1,..)
Y=cdf(name, p1,…)
name: distribution name
pi: parameters for the distribution
Guassian
Y=pdf(‘norm’,data vector, mean,std)
Y=cdf(‘norm’,data vector, mean,std)
Or
Y=normpdf(data vector, mean,std)
Y=normcdf(data vector, mean,std)
28
Gaussian distribution
mu=[0, 2, -2, 0]; sig=[0.2,1,0.5,3];
x=linspace(-5,5,100);
for i=1:4
xpdf(:,i)=pdf('norm',x,mu(i),sig(i));
xcdf(:,i)=cdf('norm',x,mu(i),sig(i));
end
subplot(2,1,1), plot(x,xpdf)
subplot(2,1,2), plot(x,xcdf)
29
Gaussian distribution
30
Central limit theorem
The sum of a large number of independent and
identically distributed random variables, each
with finite mean and variance, is approximately
normally distributed.
-the 2nd fundamental theorem of probability
31
Central limit theorem
To simulate flipping a coin, we use a
random number generator. If the
random number is less than 0.5, it is for i=1:2000
considered as heads (represented x=rand(1000,1)<0.5;
by 1); if the random number is equal heads=sum(x);
to or greater than 0.5, it is tails=1000-heads;
considered as tails (represented by y(i)=heads-tails;
-1). The sum of 1000 random flips in end
2000 game plays can be calculated hist(y)
by generating random numbers
1000 times in 2000 trips and
summing the results (1 or -1).
32
Central limit theorem
33
Flipping a rigged coin
What are the probabilities of losing more than
$50 and $100 if the winning odds is 50%?
ymean=mean(y)
ystd=std(y)
cdf('norm',-50,ymean,ystd)
cdf('norm',-100,ymean,ystd)
34
Central limit theorem
Let X1, X2, X3, ... be a set of n independent and identically
distributed (not necessarily normal) random variables having
finite values of mean μ and variance σ2. As the sample size n
increases, the distribution of the sample average approaches
the normal distribution with a mean μ and variance σ2/n
irrespective of the shape of the original distribution.
The PDF of the sum of two or more independent variables is
the convolution of their densities (if these densities exist). The
convolution of a number of density functions tends to the
normal density as the number of density functions increases
without bound, under the conditions stated above.
35
Gaussian distribution
36
Estimate of the errors
37
Propagation of error (normal distribution)
38
Central limit theorem
The log of a product of random variables that
take only positive values tends to have a normal
distribution, which makes the product itself have
a log normal distribution.
39
Log-normal distribution
If Y is a random variable with a normal
distribution, then X = exp(Y) has a log-normal
distribution.
40
Log-normal distribution
41
Log-normal distribution
mu=[0, 0, 1, 1]; sig=[1/4,1/2,1,2];
x=linspace(0,3,100);
for i=1:4
xpdf(:,i)=pdf('logn',x,mu(i),sig(i));
xcdf(:,i)=cdf('logn',x,mu(i),sig(i));
end
subplot(2,1,1), plot(x,xpdf)
subplot(2,1,2), plot(x,xcdf)
42
Chi-squared distribution
One of the most widely used in statistical significance
tests.
43
Chi-squared distribution
https://en.wikipedia.org/wiki/Chi-squared_distribution
44
F-distribution
U1 and U2 have chi-squared distributions with d1 and
d2 degrees of freedom respectively, and
U1 and U2 are independent
F-distribution is
https://en.wikipedia.org/wiki/F-distribution
The F-distribution arises frequently as the null
distribution in the analysis of variance
45
Student’s t-distribution
The t-distribution is often
used as an alternative to
the normal distribution as
a model for data. It is
frequently the case that
real data have heavier
tails than the normal
distribution allows for. The
classical approach was to
identify outliers and
exclude or downweight
them in some way.
46
https://en.wikipedia.org/wiki/Student%27s_t-distribution
Student’s t-distribution
Arises in the problem of estimating the mean of a
normally distributed population when the sample size is
small (hence the standard deviation is not known well).
Suppose X1, ..., Xn are independent random variables
that are normally distributed with expected value μ and
variance σ2. DOF=n-1 Let
47
Using t-distribution
Scaled distribution with a mean of 0 and
a standard deviation of 1.
If T were a Gaussian distribution,
PDF
CDF
48
t-distribution
A random sampling of screws gives weights 30.02,
29.99, 30.11, 29.97, 30.01, 29.99.
Calculate a 95% confidence interval for the
population's mean weight.
49
t-distribution
The probability that x
is greater than the
confidence interval
zα/2 is α.
50
https://www.studypug.com/statistics-help/confidence-intervals-with-t-distribution
t-distribution
This shows the confidence interval on μ at the 1- α
confidence level. This is a commonly used statistic for
estimation of the population mean. Often the level used
is 95%, or 2σ.
51
t-distribution
Find the data value corresponding to probability
α/2= cdf(‘t’, zα/2, DOF) (forward)
zα/2 = tinv(α/2, DOF) (invert)
For Gaussian distribution
zα/2 = norminv(α/2, mean, std/sqrt(n))
52
t-distribution
x=[30.02, 29.99, 30.11, 29.97, 30.01, 29.99];
xmean=mean(x)
xstd=std(x)
n=length(x)
%t-value at 2.5% (5%/2), DOF=n-1
%t-distribution cdf is symmetrical
tvalue=abs(tinv(0.025,n-1))
%low/high bound
low=xmean-tvalue*xstd/sqrt(n)
high=xmean+tvalue*xstd/sqrt(n)
53
Comparison to normal distribution
%if the sample size is large
xmean=mean(x);
t-distribution
xsig=xstd/sqrt(n); [29.963, 30.067]
tvalue=2.57
tvalue=abs(norminv(0.025,0,1))
Normal dist
xmean+tvalue*xsig [29.975, 30.055]
tvalue=1.96
xmean-tvalue*xsig
…….…or ………….
norminv(0.975,xmean,xsig)
norminv(0.025,xmean,xsig)
54
Student’s t and Guassian distributions
As the number of
sample increases,
the t-distribution
value for 95%
confidence interval
approaches that of
Gaussian.
55
Statistical testing
Null hypothesis is used to test differences in
treatment and control groups, and the assumption
at the outset of the experiment is that no difference
exists between the two groups for the variable being
compared.
"A statistically significant difference" simply means
there is statistical evidence that there is a difference;
it does not mean the difference is necessarily large,
important or significant in the common meaning of
the word.
Confidence level:
56
Statistical testing
The null hypothesis must be stated in mathematical/statistical terms
that make it possible to calculate the probability of possible samples
assuming the hypothesis is correct.
A test must be chosen that will summarize the information in the
sample that is relevant to the hypothesis. In the example given
above, it might be the numerical difference between the two sample
means, m1 − m2.
The distribution of the test statistics is used to calculate the
probability sets of possible values (usually an interval or union of
intervals).
Among all the sets of possible values, we must choose one that we
think represents the most extreme evidence against the hypothesis.
That is called the critical region of the test statistic. The probability
of the test statistic falling in the critical region when the null
hypothesis is correct, is called when the null hypothesis is correct,
is called the p value (“surprise” value) of the test. 57
Probability
Frequency probability (frequentists) is the interpretation of
probability that defines an event's probability as the limit of its
relative frequency in a large number of trials. The problems
and paradoxes of the classical interpretation motivated the
development of the relative frequency concept of probability.
Bayesian probability is an interpretation of the probability
calculus which holds that the concept of probability can be
defined as the degree to which a person (or community)
believes that a proposition is true. A posteriori is a function of
a priori and observations.
The groups have agreed that Bayesian and Frequentist
analyses answer genuinely different questions, but disagreed
about which class of question it is more important to answer
in scientific and engineering contexts.
58
Pearson’s Chi-squared test
The null hypothesis: the relative frequencies of
occurrence of observed events follow a specified
frequency distribution.
Pearson's chi-squared is the original and most
widely-used chi-squared test.
The test compares the difference between each
observed and theoretical frequency for each
possible outcome.
59
Pearson’s Chi-squared test
DOF = n-1
60
Pearson’s Chi-squared test
A chi-squared probability of 0.05 or less (alpha
value, α =0.05) is commonly used for rejecting the
null hypothesis.
Critics of α values point out that the criterion used to
decide "statistical significance" is based on the
somewhat arbitrary choice of level (often set at
0.05).
61
Pearson’s Chi-squared test
A random sample of 100 people has been drawn
from a population in which men and women are
equal in frequency. There were 45 men and 55
women in the sample, what is chi-squared value?
Probability in Chi-squared distribution: cdf(‘chi2’, value, DOF)
Is this Chi-squared value within 95% of the distribution?
cdf('chi2',1,1)
62
Pearson’s Chi-squared test
What should we use?
x=linspace(0,8,100);
v (probability)
v=cdf('chi2',x,ones(1,100));
plot(x,v)
%cdf('chi2',1,1) = 0.68
x (chi-squared value)
63
chi2inv
%chi2inv(probability, DOF)
%the value exceeds 95% of the data with DOF =1
chi2inv(0.95,1)
if the chi2 values < chi2inv(0.95,1), the hypothesis
cannot be rejected.
64
Pearson’s Chi-squared test
65
Pearson’s Chi-squared test
corg = load('organicmatter_one.txt');
% 60 data points, define 8 bins
[n_exp,v] = hist(corg,8);
% generate a synthetic dataset
n_syn = pdf('norm',v,mean(corg),std(corg));
%redistribute;
n_syn = n_syn.* sum(n_exp)/sum(n_syn);
subplot(1,2,1), bar(v,n_syn,'r')
subplot(1,2,2), bar(v,n_exp,'b')
66
Pearson’s Chi-squared test
%test
chi2 = sum((n_exp - n_syn).^2./n_syn)
%dof (# of bins - # of parameters -1)
%For Gaussian, # of parameters is 2, mean & std
dof=8-2-1;
%0.05 value
chi2inv(0.95,dof)
67
F-test
The null hypothesis: the standard deviations of
two normally distributed populations are
equal.
F_crit=finv(0.95, DOF1,DOF2)
68
F-test
load('organicmatter_four.mat');
%compare std
s1 = std(corg1)
s2 = std(corg2)
%DOF
df1 = length(corg1) -1;
df2 = length(corg2) -1;
69
F-test
if s1>s2
Freal=(s1/s2)^2
else
Freal=(s2/s1)^2
end
%find the table value for 5% extreme, inv-f cdf
Ftable = finv(0.95,df1,df2)
70
Student’s t-Test
The null hypothesis: the means of two
distributions are equal.
Assumptions
Normal distribution of data (What test to use?)
Equality of variances (what test to use?)
71
Student’s t-Test
Matlab syntax
[h,s,ci] = ttest2(x,y,p-value)
h: 1 rejects the null hypothesis;
0 cannot reject
s: significance for the difference
of the means between x and y
ci: confidence interval
p-value=0.05
72
Student’s t-Test
load('organicmatter_two.mat');
[n1,x1] = hist(corg1);
[n2,x2] = hist(corg2);
h1 = bar(x1,n1);
hold on
h2 = bar(x2,n2);
set(h1,'FaceColor','none','EdgeColor','r')
set(h2,'FaceColor','none','EdgeColor','b')
hold off
73
Student’s t-Test
%difference of mean
mean(corg1)-mean(corg2)
%h=1 rejects the null hypothesis
%significance: p-value associated with the t-statistic
%ci: 95% confidence interval on the difference of the
means (if there is no statistically significant difference)
[h, significance, ci] = ttest2(corg1,corg2,0.05)
74
There are more!
(e.g., Extremal type distribution)
75
Tutorial
We compare the observed and simulated ozone mixing
ratios (ppbv, parts per billion by volume) in July and August.
Negative values represent no measurements.
Data: Atlanta_O3.txt
1. Apply a statistical test to show if the two datasets are
normal distributions.
2. Assuming that both are normal distributions, apply
statistical tests to show if the two datasets have equivalent
variances and if the two datasets have equivalent means.
76
