ENGINEERING
DATA ANALYSIS
Prepared by: Engr. Marlo Dexie Dale G. Malaluan
Module 8
Multiple Linear Regression
8 Multiple Linear Regression
MODULE OUTLINE
12.1 Multiple Linear Regression Model 12.2 Hypothesis Tests in Multiple Linear
12.1.1 Introduction Regression
12.1.2 Least squares estimation of the 12.2.1 Test for significance of regression
parameters 12.2.2 Tests on individual regression
12.1.3 Matrix approach to multiple linear coefficients & subsets of coefficients
regression
12.1.4 Properties of the least squares
estimators
2
Learning Objectives for Module 8
After careful study of this module, you should be able to do the following:
1. Use multiple regression techniques to build empirical models to engineering and
scientific data
2. Understand how the method of least squares extends to fitting multiple regression
models.
3. Assess regression model adequacy
4. Test hypotheses and construct confidence intervals on the regression coefficients
3
Multiple Linear Regression Model
• Many applications of regression analysis involve situations in which there are more than
one regressor variable.
• A regression model that contains more than one regressor variable is called a multiple
regression model.
4
Multiple Linear Regression Model
• For example, suppose that the gasoline mileage performance of a vehicle depends on the
vehicle weight and the engine displacement. A multiple regression model that might
describe this relationship is
𝑌𝑌 = 𝛽𝛽0 + 𝛽𝛽1 𝑥𝑥1 + 𝛽𝛽2 𝑥𝑥2 + 𝜖𝜖
where 𝑌𝑌 = 𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚, 𝑥𝑥1 = 𝑤𝑤𝑤𝑤𝑤𝑤𝑤𝑤𝑤𝑤𝑤, 𝑥𝑥2 = 𝑒𝑒𝑒𝑒𝑒𝑒𝑒𝑒𝑒𝑒𝑒𝑒 𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑
5
Least Squares Estimation of Parameters
The method of least squares may be used to estimate the regression coefficients in the multiple
regression model, 𝑌𝑌 = 𝛽𝛽0 + 𝛽𝛽1𝑥𝑥1 + 𝛽𝛽2 𝑥𝑥 2 + ⋯ + 𝛽𝛽𝐾𝐾 𝑥𝑥 𝐾𝐾 + 𝜖𝜖.
Suppose that 𝑛𝑛 > 𝑘𝑘 observations are available, and let 𝑥𝑥𝑖𝑖j denote the 𝑖𝑖𝑖𝑖𝑖 observation or level of variable
𝑥𝑥𝑗𝑗. The observations are 𝑥𝑥𝑥𝑥1, 𝑥𝑥𝑖𝑖𝑖 , … , 𝑥𝑥𝑖𝑖𝑖𝑖 , 𝑦𝑦𝑖𝑖 , 𝑖𝑖 = 1, 2, …, 𝑛𝑛 𝑎𝑎𝑎𝑎𝑎𝑎 𝑛𝑛 > 𝑘𝑘.
It is customary to present the data for multiple regression in a table such as Table 12.1
6
Least Squares Estimation of Parameters
7
Least Squares Estimation of Parameters
The solution to the normal equations are the least
squares estimators of the regression coefficients.
8
Example 12.1 | Wire Bond Strength
• In Chapter 1, we used data on pull strength of a wire bond in a semiconductor
manufacturing process, wire length, and die height to illustrate building an empirical
model.
• We will use the same data, repeated for convenience in Table 12.2, and show the
details of estimating the model parameters.
• A three-dimensional scatter plot of the data is presented in Fig. 1.15. Figure 12.4
shows a matrix of two-dimensional scatter plots of the data.
• These displays can be helpful in visualizing the relationships among variables in a
multivariable data set.
• For example, the plot indicates that there is a strong linear relationship between
strength and wire length.
9
Example 12.1 | Wire Bond Strength (ctd)
10
Example 12.1 | Wire Bond Strength (ctd)
11
Example 12.1 | Wire Bond Strength (ctd)
12
Example 12.1 | Wire Bond Strength (ctd)
Practical Interpretation: This
equation can be used to predict
pull strength for pairs of values of
the regressor variables wire
length (𝑥𝑥1) and die height
(𝑥𝑥2). This is essentially the same
regression model given in Section
1.3. Figure 1.16 shows a three-
dimensional plot of the plane of
predicted values generated from
this equation.
13
Matrix Approach to Multiple Linear Regression
14
Matrix Approach to Multiple Linear
Regression
The resulting least squares estimate is
15
Matrix Approach to Multiple Linear
Regression
16
Example 12.2 | Wire Bond Strength
1 2 50 9.95
1 8 110 24.45
1 11 120 31.75
1 10 550 35.00
1 8 295 25.02
1 4 200 16.86
1 2 375 14.38
1 2 52 9.60
The model matrix 𝐗𝐗 1
1
9
8
100
300
24.35
27.50
and 𝐲𝐲 vector for this
1 4
412
17.08
1 11 400 37.00
model are
X = 1 12
500 y = 41.95
1 2 360 11.66
1 4 205 21.65
1 4 400 17.89
1 20 600 69.00
1 1 585 10.30
1 10 540 34.93
1 15 250 46.59
1 15 290 44.88
1 16 510 54.12
1 17 590 56.63
1 6 100 22.13
1 400
5 21.15
17
Example 12.2 | Wire Bond Strength (ctd)
18
Example 12.2 | Wire Bond Strength (ctd)
19
Example 12.2 | Wire Bond Strength (ctd)
20
Example 12.2 | Wire Bond Strength (ctd)
21
Estimator of Variance
This is an unbiased estimator of 𝜎𝜎 2
22
Properties of the Least Squares Estimators
Unbiased estimators Individual variances and covariances
Covariance Matrix In general,
23
Test for Significance of Regression
24
Test for Significance of Regression
25
Example 12.3 | Wire Bond Strength ANOVA
We will test for significance of regression (with α = 0.05) using the wire
bond pull strength data from2
Example 12.1. The total sum of squares is
n
∑ yi
(725.82) 2
i =1
SST = y ′y − = 27,178.5316 −
n 25
= 6105.9447
The regression or model sum of squares is computed as follows:
2
n
∑ yi
(725.82) 2
ˆ i =1
SS R = β′ X′ y − = 27,063.3581 −
n 25
and by subtraction = 5990.7712
26
Example 12.3b | Wire Bond Strength ANOVA
• The analysis of variance is shown in Table 12.6. To test H0: β1 = β2 = 0, we
calculate the statistic
MS R 2995.3856
f0 = = = 572.17
MS E 5.2352
• Since f0 > f0.05,2,22= 3.44 (or since the P-value is considerably smaller than
α = 0.05), we reject the null hypothesis and conclude that pull strength is
linearly related to either wire length or die height, or both.
• Practical Interpretation: Rejection of H0 does not necessarily imply that the relationship found is an
appropriate model for predicting pull strength as a function of wire length and die height. Further
tests of model adequacy are required before we can be comfortable using this model in practice.
27
Example 12.3c | Wire Bond Strength ANOVA
28
Test for Significance of Regression
29
Tests on Individual Regression Coefficients
and Subsets of Coefficients
• Reject H0 if |t0| > tα/2,n-p.
• This is called a partial or marginal test
30
Example 12.4 | Wire Bond Strength
Coefficient Test
The wire bond pull strength data, and suppose that we want to test the
hypothesis that the regression coefficient for x2 (die height) is zero. The
hypotheses are
H0: β2 = 0
H1: β2 ≠ 0
The main diagonal element of the (X′X)−1 matrix corresponding to is C22
= 0.0000015, so the t-statistic in Equation 12.25 is
βˆ 2 0.01253
t0 = = = 4.477
σˆ 2C22 (5.2352)(0.0000015)
31
Example 12.4b | Wire Bond Strength
Coefficient Test
Note that we have used the estimate of σ2 reported to four decimal places in Table 12.6.
Since t0.025,22 = 2.074, we reject H0: β2 = 0 and conclude that the variable x2 (die height)
contributes significantly to the model. We could also have used a P-value to draw
conclusions. The P-value for t0 = 4.477 is P = 0.0002, so with α = 0.05 we would reject the
null hypothesis.
Practical Interpretation: Note that this test measures the marginal or partial contribution of
x2 given that x1 is in the model. That is, the t-test measures the contribution of adding the
variable x2 = die height to a model that already contains x1 = wire length. Table 12.4 shows
the computer-generated value of the t-test computed. The t-test statistic is reported to two
decimal places. Note that the computer produces a t-test for each regression coefficient in
the model. These t-tests indicate that both regressors contribute to the model.
32
Tests on Individual Regression Coefficients
and Subsets of Coefficients
The general regression significance test or the extra sum of
squares method:
The model may be written as
33
Tests on Individual Regression Coefficients
and Subsets of Coefficients
For the full model:
SS R (β) = βˆ ′ X′ y ( p = k + 1 degrees of freedom)
y ′ y − β ′ X′ y
MS E =
n− p
If H0 is true, the reduced model is
SS R (β 2 ) βˆ ′2 X′2 y
= ( p − r degrees of freedom )
34
Tests on Individual Regression Coefficients
and Subsets of Coefficients
Reject H0 if f0 > fα,r,n-p
The test in Equation (12.33) is often referred to as a partial F-test
35
Example 12.6 | Wire Bond Strength General
Regression Test
Consider the wire bond pull-strength data in Example 12.1. We investigate the
contribution of two new variables, x3 and x4, to the model using the partial F-test
approach. The new variables are explained at the end of this example. That is,
we wish to test
H0: β3 = β4 = 0 H1: β3 ≠ 0 or β4 ≠ 0
To test this hypothesis, we need the extra sum of squares due to β3 and β4 or
SSR(β4, β3| β2, β1, β0) = SSR (β4, β3, β2, β1, β0) − SSR(β2, β1, β0)
= SSR(β4, β3, β2, β1|β0) − SSR (β2, β1|β0)
36
Example 12.6b | Wire Bond Strength General
Regression Test
• In Example 12.3 we calculated
2
n
∑ yi
′ ′
SS R (β2 , β1 |β0 ) = β X y − i =1 = 5990.7712 ( two degrees of freedom)
n
• Also, Table 12.4 shows the Minitab output for the model with only x1 and x2 as predictors. In
the analysis of variance table, we can see that SSR = 5990.8 and this agrees with our
calculation. In practice, the computer output would be used to obtain this sum of squares.
• If we fit the model Y = β0 + β1x1 + β2x2 + β3x3 + β4x4, we can use the same matrix formula.
Alternatively, we can look at SSR from computer output for this model. The analysis of
variance table for this model is shown in Table 12.7 and we see that
SSR(β4, β3, β2, β1|β0) = 6024.0 (four degrees of freedom)
Therefore,
SSR(β4, β3|β2, β1, β0) = 6024.0 − 5990.8 = 33.2 (two degrees of freedom)
37
Example 12.6c | Wire Bond Strength General
Regression Test
• This is the increase in the regression sum of squares due to adding x3 and x4 to a model
already containing x1 and x2. To test H0, calculate the test statistic
• Note that MSE from the full model using x1, x2, x3 and x4 is used in the denominator of the
test statistic. Because f0.05, 2, 20 = 3.49, we reject H0 and conclude that at least one of the new
variables contributes significantly to the model. Further analysis and tests will be needed to
refine the model and determine if one or both of x3 and x4 are important.
• The mystery of the new variables can now be explained. These are quadratic powers of the
original predictors wire length and wire height. That is, 𝑥𝑥3 =𝑥𝑥12 and 𝑥𝑥4 =𝑥𝑥22 . A test for
quadratic terms is a common use of partial F-tests. With this information and the original
data for x1 and x2, you can use computer software to reproduce these calculations. Multiple
regression allows models to be extended in such a simple manner that the real meaning of
x3 and x4 did not even enter into the test procedure.
38
Important Terms and Conditions
• All possible regressions • Hat matrix • Prediction interval on a future
observation
• Analysis of variance test in multiple • Hidden extrapolation
regression • PRESS statistic
• Indicator variables
• Backward elimination • 𝑅𝑅 2
• Inference (tests and intervals) on
• 𝐶𝐶𝑃𝑃 statistic individual model parameters • Residual analysis and model
adequacy checking
• Categorical variables • Influential observations
• Significance of regression
• Confidence interval on the mean • Model parameters and their
response interpretation in multiple regression • Standardized residuals
• Confidence interval on the • Multicollinearity • Stepwise regression
regression coefficient
• Multiple regression model • Studentized residuals
• Cook’s distance measure
• Outliers • Variable selection
• Extra sum of squares method
• Partial or marginal test • Variance inflation factor (VIF)
• Forward selection
• Polynomial regression model
• Full model
39
END OF
PRESENTATION
