Multiple Regression
Analysis
Multiple Regression
Cautions About Linear Regression
Correlation and regression describe only linear
relations.
Correlation and least-squares regression line
are not resistant to outliers.
Predictions outside the range of observed data
are often inaccurate.
Relationship between two variables often
influenced by lurking variables not included in
our model.
General Principle of Data Analysis
Plot your data To understand the data, always start with a
series of graphs
Interpret what Look for all pattern and deviations on that
you see pattern
Numerical Choose an appropriate measure to
summary ? describe the pattern and deviation
Mathematical If the pattern is regular, summarize the data
model ? in a compact mathematical model
Analysis of Two Quantitative Variables
Plot your data For two quantitative variables, use a
scatterplot
Interpret what Describe the direction, form and strength of
you see the relationship
Numerical If pattern is roughly linear, summarize with
summary ? correlation, means and standard deviations
Mathematical Regression gives a compact model of
model ? overall pattern if relationship is roughly
linear
Analysis of Three or More Quantitative
Variables
Plot your data To examine relationships among all
possible pairs, use a scatterplot matrix
Interpret what Describe the direction, form and strength of
you see the relationships
Numerical If pattern is roughly linear, summarize with
summary ? correlations, means and standard
deviations
Mathematical Multiple Regression gives a compact model
model ? of relationship between response variable
and a set of predictors
Multiple Regression
• Can we predict job performance (Y) from overall school
achievement (X1) and IQ scores (X2)?
- How much variance in Y is explained by X1 and X2 in
combination?
- how important is each predictor of job performance?
• Two kinds of research questions in Multiple Regression:
- Is the model significant and important??
- Are the individual predictors significant and important?
The Structural Model
Y c b1 X 1 b2 X 2 ... b p X p e
Y - any dependent variable score is predicted according to :
c - an intercept on the Y axis, plus
b1 X 1 - a weighted effect of predictor X1
b2 X 2 - a weighted effect of predictor X2
bp X p - a weighted effect of predictor Xp
e - error
The Structural Model
Y c b1 X 1 b2 X 2 ... b p X p e
DATA = MODEL + RESIDUAL
The Regression Plane – Two Predictors
(3D space)
Unstandardized Partial Regression
Coefficients - b
• Y is calculated according to Least Square Criterion (LSC)
• solved for by finding a set of weights (b) minimising errors
of prediction (around the plane)
- b1 indicates change in Y given unit change in X1 when X2 …
Xp = 0
- when standardised, indicates SD change in Y given SD
change in X, and is denoted by
• c is the Y intercept
• Y is therefore a weighted combination of the predictors
(and intercept) called a linear composite (LC)
Bivariate regression
Multiple Regression
Multiple Regression
Variance Explained – R2
R 2 is simply the r 2 representing the proportion of variance in Y
which is explained by Yˆ – the linear composite
r2
SS regression
SSYˆ
Yˆ Y
2
Y Y
2
SS total SSY
a ratio reflecting the proportion of variance captured by our
model relative to the overall variance in our data
R 2 =.50 means 50% of the variance in Y is explained by the
combination of X1, X2… Xp
2
R vs r 2
Significance of the Model
• R 2 tells us how important the model is
• the model can also be tested for statistical
significance
• test is conducted on R the multiple correlation
coefficient, against df = p, N - p - 1
( N p 1) R 2
MS regression
F
p (1 R ) 2
MS residual
Importance of Individual Predictors
r – simple correlation coefficient
b – partial regression coefficient
– standardized partial regression coefficient
pr – partial correlation coefficient
sr – semi-partial correlation coefficient
r – simple correlation coefficient
• indicates importance of predictor in terms of its
direct relationship with the criterion
• not very useful in Multiple Regression as it does
not take into account inter-correlations with other
predictors.
b – Partial Regression Coefficient
• indication of the importance of a predictor in
terms of the model (not the data).
• scale-bound so can’t compare magnitude.
• can however compare significance – each b is
tested by dividing it by its standard error to give
a t-value:
– standardized partial regression coefficient
• indication of the importance of a predictor in
terms of the model (not the data).
• standardized (scale free) so you can compare
magnitude
• test of significance is same as for b
pr – Partial Correlation Coefficient
sr – semi-partial correlation coefficient
Unique, Shared and Total Variance
Assumptions of Multiple Regression
• Scale (predictor and criterion scores)
• measured using a continuous scale (interval
or ratio)
• normality (variables are normally distributed)
• linearity (there is a straight line relationship
between predictors and criterion)
• predictors are not multicollinear or singular
(extremely highly correlated)
Assumptions of Multiple Regression
• Residuals
• normality: array of Y values are normally
distributed around Yˆ (assumption of normality in
arrays)
• homoscedasticity: variance of Y values are
constant across full range Y values (assumption
of homogeneity of variance in arrays)
• linearity: straight-line relationship between Y
and residuals (with mean = 0 and slope = 0)
• independence (residuals uncorrelated)
Multicollinearity and Singularity
• occurs when predictors are highly correlated (>.90)
• causes unstable calculation of regression weights (b)
• diagnosed with inter-correlations, tolerance and VIF
Tolerance = (1 Rx2 )
2
• where Rx is the overlap between a particular
predictor and all the other predictors
• values below .10 considered problematic
Variance Inflation Factor (VIF) = 1/tolerance
- values above 4 considered problematic
• best solution is to remove or combine collinear predictors
Outliers – Extreme Cases
• distort solution and inflate standard error
• univariate outliers
• cases beyond 3 SD on any variable
• multivariate outliers
• described in terms of:
• leverage (h) – distance of case from group
centroid along line/plane of best fit
• discrepancy – extent to which case
deviates from line/plane of best fit
• influence – combined effect of leverage
and discrepancy: effect of the outlier on
the solution
Multivariate Outliers – high influence
high discrepancy
Multivariate Outliers – low influence
high discrepancy
Multivariate Outliers – Testing
Leverage
• Leverage statistic (h): varies from 0 to 1, values > .50 are
problematic
• Mahalanobis Distance h x (n-1), distributed as chi-square and
tested as such (df = p, <.001)
discrepancy – not directly tested
influence –
assesses change in solution when case is removed
Cook’s Distance, values > 1 are problematic
Working example
A marketing manager of a large supermarket chain wanted to
determine the effect of shelf space and price on the sales of pet food.
A random sample of 15 equal-sized shops was selected, and the
sales, shelf space in square metres and price per kilogram were
recorded
1. What contribution do both shelf space and price make to the
prediction of sales of pet food?
2 . Which is a better predictor of sales of pet food?
3. Do a residual analysis
The data file can be found in Work17.sav
Using SPSS
Graphs
Scatter/Dot
Matrix Scatter
Using SPSS
Graph
[DataSet1] C:\Users\demo\Desktop\Corr&RegressPresentation\SPSS DataFile\Work17.sav
Using SPSS
Multiple Linear Regression:
Starting the Procedure
• In the menu, click on
Analyze
• Point to
Regression
• Point to
Linear…
… and click.
Using SPSS
Multiple Linear Regression:
Selecting Variables
Choose the variables
for analysis from the
list in the variable box.
To select multiple
variables, hold down
the Ctrl Key and chose
the variables that you
want.
Using SPSS
Multiple Linear Regression:
Selecting Variables
Move shelf space
(space) & price per kg
(price), which are
already highlighted, to
the box labeled
Independent(s) then
click the arrow.
Move sales of pet food
(sales) to the box
labeled Dependent by
clicking the arrow.
Using SPSS
Multiple Linear Regression:
Requesting Statistics
Request
descriptive
statistics by
clicking the
button
labeled
Statistics…
Using SPSS
Multiple Linear Regression:
Requesting Statistics
Statistics for the Model
fit and Estimates for
Regression
Coefficients will be
produced by default.
Click the checkbox for
Descriptives. Also,
click the checkbox for
Durbin-Watson for
Residuals.
Click the Continue
button.
Using SPSS
Multiple Linear Regression:
Standardized Residual Plots
You can also request
several different plots.
Click the Plots… button.
In the box labeled
Standardized Residual
Plots, first click the
checkbox for
Histogram,
then click the box
for Normal
probability plot.
Click the Continue
button.
Using SPSS
Multiple Linear Regression:
Enter Method
The independent
variables can be
entered into the
analysis using
five different
methods.
Enter Method, a procedure for variable selection in which all variables in a
block are entered in a single step.
Using SPSS
Multiple Linear Regression:
Enter Method
Enter is the
default method
of variable entry.
Click the OK
button to run the
Multiple Linear
Regression
procedure.
Using SPSS
Multiple Linear Regression Output:
Descriptive Statistics
Regression
[DataSet1] C:\Users\demo\Desktop\Corr&RegressPresentation\SPSS DataFile\Work17.sav
Using SPSS
Multiple Linear Regression Output:
Correlations
Using SPSS
Multiple Linear Regression Enter Method Output:
Variables Entered
Using SPSS
Multiple Linear Regression Enter Method Output:
Model Summary
Correlation Standard Deviation
Coefficient of around the
Determination regression line
Durbin-Watson
Statistic
Using SPSS
Multiple Linear Regression Enter Method Output:
Model Summary
Independence
Durbin-Watson Statistic.
The D-W statistic is
defined as:
Another way to look at the Durbin-Watson Statistic is:
D = 2(1-ρ)
where ρ = the correlation between consecutive errors.
Using SPSS
Multiple Linear Regression Enter Method Output:
ANOVA
Measures of
Variation
Using SPSS
Multiple Linear Regression Enter Method Output:
Coefficients
Regression Equation:
ŷi = 10.50x1 + 0.057x2 + 2.029
Using SPSS
Multiple Linear Regression Enter Method Output:
Residuals Statistics
Using SPSS
Multiple Linear Regression Enter Method Output:
Residuals Histogram
Normality
Normality of residuals
is only required for
valid hypothesis
testing, that is, the
normality assumption
assures that the p-
values for the t-tests
and F-test will be
valid. Normality is not
required in order to
obtain unbiased
estimates of the
regression
coefficients
Using SPSS
Multiple Linear Regression Enter Method Output:
Plot of Standardized Residuals
Normality
A standardized
normal
probability (P-P)
plot is sensitive
to non-normality
in the middle
range of data
tails.
Using SPSS
Multiple Linear Regression Enter Method Output:
Interpretation of Output
1. What contribution do both shelf space and price make to the
prediction of sales of pet food?
Both independent variables (shelf space and price) together explain 85 per
cent of the variance (R Square) in sales of pet food, which is highly
significant as indicated by the F-value of 34.08
Using SPSS
Multiple Linear Regression Enter Method Output:
Interpretation of Output
2. Which of the two variable is a better predictor of sales of pet food?
An examination of the t-values and Beta values indicate that price contributes
better to the prediction of sales. Therefore, you can say that price
significantly predicts sales of pet food with t = 3.22, P < .05. However, the
shelf space allocated is not a significant predictor.
