University of Padua
Statistics for Management
Simple Linear Regression (1)
Omar Paccagnella
Department of Statistical Sciences
University of Padua
omar.paccagnella@unipd.it
http://www.stat.unipd.it/~paccagnella
Statistics for Management, a.y. 2018/19 - Simple linear regression (1) 1/ 23
University of Padua
Introduction
What happens if:
• Data are time-oriented
• There is more than 1 variable
Unit Shop surface Weekly sales (1000 e)
1 95 43.2
2 144 132.0
3 210 155.0
4 156 76.0
5 188 100.9
6 321 187.4
7 250 185.0
8 115 60.7
9 178 82.9
10 105 61.3
Statistics for Management, a.y. 2018/19 - Simple linear regression (1) 2/ 23
University of Padua
Statistics for Management, a.y. 2018/19 - Simple linear regression (1) 3/ 23
University of Padua
Statistics for Management, a.y. 2018/19 - Simple linear regression (1) 4/ 23
University of Padua
Introduction
• A scatter diagram or scatter plot (that is a two-dimensional graph)
may help to show a relationship between two variables
• Is this relationship linear? Is this positive or negative? If linear, could
we summarise such relationship by fitting a straight line through the
data points (like a trend for a time series)?
Statistics for Management, a.y. 2018/19 - Simple linear regression (1) 5/ 23
University of Padua
Statistics for Management, a.y. 2018/19 - Simple linear regression (1) 6/ 23
University of Padua
Statistics for Management, a.y. 2018/19 - Simple linear regression (1) 7/ 23
University of Padua
Statistics for Management, a.y. 2018/19 - Simple linear regression (1) 8/ 23
University of Padua
The Correlation Coefficient
It measures the extent to which 2 variables (usually called X and Y ) are
linearly related to each other
(in other words, the strength of such linear relationship)
• In the population (that contains all possible values of the pair X − Y
of interest): ρ
• In the (random) sample drawn from this population: r
Often the 2 variables are measured in different units (in the example,
square metres & e); nevertheless it is important to measure the extent to
which X and Y are related
Statistics for Management, a.y. 2018/19 - Simple linear regression (1) 9/ 23
University of Padua
The Correlation Coefficient
• Standardize the variables (constructing the Z -scores)
(X − X) ( Y − Y)
ZX = ZY =
SX SY
I X: average value (mean) of X
I SX : standard deviation of X
I n: number of units
• Calculate the mean cross product of Z -scores:
P
1 X (X − X)(Y − Y)
r = ZX ZY = qP q
n−1
(X − X)2 (Y − Y)2
I Correlation and not causation is measured
I −1 ≤ r ≤ 1 (check the value of r = 0.8927 in the previous example)
Statistics for Management, a.y. 2018/19 - Simple linear regression (1) 10/ 23
University of Padua
Fitting a Straight Line
• Could we find a straight line that is able to summarise the pattern of
all X − Y data points?
• Could we fit the best straight line?
• Could we exploit this best straight line to forecast unknown (future?)
values of the variable of interest (Y )?
Statistics for Management, a.y. 2018/19 - Simple linear regression (1) 11/ 23
University of Padua
Fitting a Straight Line
• We may introduce a mathematical procedure to calculate both the
Y -intercept and the slope of the best-fitting straight line.
• Since many straight lines can be calculated, the most common
approach to determine the best fit is the method of least squares
(OLS - Ordinary Least Squares)
The best fitting line is the one that minimises the sum of
the squared distances between the data points and the line itself,
as measured in the vertical (Y ) direction
Statistics for Management, a.y. 2018/19 - Simple linear regression (1) 12/ 23
University of Padua
Fitting a Straight Line
Statistics for Management, a.y. 2018/19 - Simple linear regression (1) 13/ 23
University of Padua
Fitting a Straight Line
• In the population the straight line may be mathematically defined as:
Y = β 0 + β1 X
• In the sample the straight line may be mathematically defined as:
Y = b0 + b1 X
where b0 and b1 are estimates of the true (but unknown) population
intercept and slope.
According to the values of the sample, we can predict the Y values in the
fitted line
Ŷ = b̂0 + b̂1 X
Ŷ is the value that we would observe if X ’s were on the line
Statistics for Management, a.y. 2018/19 - Simple linear regression (1) 14/ 23
University of Padua
Least Squares
The idea behind this method is that the line will be appropriate to
describe the relationship under investigation if the observed values are
closed to the straight line.
The distance between observed and fitted values is the residual:
ei = Yi − Ŷi = Yi − b0 − b1 Xi
According to the OLS criterium, the values of b0 and b1 are chosen in
order to minimise the sum of squared errors (residuals):
n
X n
X
SSE = f (b0 , b1 ) = ei2 = (Yi − b0 − b1 Xi )2
i =1 i =1
Statistics for Management, a.y. 2018/19 - Simple linear regression (1) 15/ 23
University of Padua
Least Squares
First order conditions (that is the derivates of f (b0 , b1 ) with respect to b0
and b1 ) are applied to minimise SSE. Using little calculi:
Pn
i =1 (X − X)(Y − Y)
b̂1 = Pn 2
i =1 (X − X)
b̂0 = Y − b̂1 X
Least Squares slope is related to sample correlation coefficient, so that
qP
n
i = 1 (Y − Y)2
b̂1 = qP r
n
i =1 (X − X)2
Hence, b̂1 and r are proportional to one another and have the same sign.
Statistics for Management, a.y. 2018/19 - Simple linear regression (1) 16/ 23
University of Padua
The linear regression model
According to the least squares criterion, we have the identity
Observation = Fit + Residual
formally
Y = Ŷ + (Y − Ŷ )
• The fit represents the overall pattern in the data
• The residuals represent deviations from the pattern
Statistics for Management, a.y. 2018/19 - Simple linear regression (1) 17/ 23
University of Padua
The linear regression model
Observed data is a sample of observations on an underlying relation that
holds in the population.
For all values of X , the observed values for Y are identically distributed
around a mean µ that depends linearly on X :
µy = β0 + β1 X
As X changes, the means of the distributions of the possible values of Y
lie along a straight line. This is the so-called
population regression line
• Observed values for Y vary because of the presence of unknown
(and unmeasured) factors.
• This variation is the same for all X ’s values and is measured by the
standard deviation σ .
• The distance between a Y value and its mean is called error ().
Statistics for Management, a.y. 2018/19 - Simple linear regression (1) 18/ 23
University of Padua
The linear regression model
In the simple linear regression model:
• Y is the response or dependent variable.
• X is the controlled or explanatory (independent) variable.
• The dependent variable is the sum of its mean and a random
deviation () from this mean.
• Deviations represent variation in Y due to unobserved factors that
prevent the pair (X , Y ) values from lying exactly on the straight line.
The population regression line may be defined as:
Y = β0 + β1 X +
Statistics for Management, a.y. 2018/19 - Simple linear regression (1) 19/ 23
University of Padua
Statistics for Management, a.y. 2018/19 - Simple linear regression (1) 20/ 23
University of Padua
The linear regression model
The sample regression line may be regarded as an estimate of the
population regression line,
µY = β0 + β1 X
and the residuals e = Y − Ŷ may be regarded as estimates of the error
components
Therefore:
Y = b0 + b1 X + e
Statistics for Management, a.y. 2018/19 - Simple linear regression (1) 21/ 23
University of Padua
Some notes
• We may also write
Cov (X , Y )
b1 =
Var (X )
if Var (X ) 6= 0
• b1 = 0 if and only if Cov (X , Y ) = 0, that is the two variables are
linearly independent
• Cov (X , Y ) provides the sign of b1 estimate
• The regression line always passes through the means of X and Y
Statistics for Management, a.y. 2018/19 - Simple linear regression (1) 22/ 23
University of Padua
Steps of a Linear Regression Analysis
• Hypothesis on the linear functional relationship between the variable
of interest and the other variable(s)
• Estimation of the parameters of this functional relationship,
based on the available sample data
• Statistical testing of model estimates and goodness of fit
• Robustness checks on the main assumptions of the linear
regression model
Statistics for Management, a.y. 2018/19 - Simple linear regression (1) 23/ 23
