Topic 12.

Multiple Linear Regression

Any questions?

During the
Lecture
E6-2004
Skype: L2ahmadi
Office Ext. 37160
L2ahmadi @uw...
Hour
After the
Lecture

In Person Message Call

1
Course Topics
1. Role of Statistics in Engineering
2. Probability
3. Treatment of Data
4. Discrete Probability Distributions
5. Continuous Probability Distributions
6. Point Estimation of Parameters
7. Statistical Intervals for a Single Sample
8. Tests of Hypothesis for a Single Sample
9. Statistical Intervals for Two Samples
10. Topic 6 to 9 in one Picture
11. Simple Linear Regression
12. Multiple Linear Regression (Sections 11.4 - 11.7)
2
13. Process Quality Improvement (If time permits)
Topic Outline
12-1. Multiple Linear Regression Models
12-2. Hypothesis Tests
12-3. Confidence Intervals
12-4. Prediction of New Observations (Not in the exam)
12-5. Model Adequacy (by R2)
12-6. Polynomial Regression Models

3
Learning Objectives

4
Previous Topic
STEP I: BUILD THE MODEL STEP II: GOOD ENOUGH?
SST

SSE SSR

Tests:
- ANOVA Table - Confidence Intervals - R2

5
12-1 Multiple Linear Regression Models

12-1.1 Introduction
• Many applications of regression analysis involve
situations in which there are more than one
regressor variable.
• A regression model that contains more than one
regressor variable is called a multiple regression
model.

6
12-1 Multiple Linear Regression Models

12-1.1 Introduction
• For example, suppose that the effective life of a cutting
tool depends on the cutting speed and the tool angle. A
possible multiple regression model could be

where
Y – tool life
x1 – cutting speed
x2 – tool angle
7
12-1 Multiple Linear Regression Models
12-1.1 Introduction

Figure 12-1 (a) The regression plane for the model

E(Y) = 50 + 10x1 + 7x2. (b) The contour plot
8
12-1 Multiple Linear Regression Models

12-1.1 Introduction

9
12-1 Multiple Linear Regression Models

12-1.1 Introduction

Figure 12-2 (a)

Three-dimensional
plot of the
regression model
E(Y) = 50 + 10x1 +
7x2 + 5x1x2. (b) The
contour plot
10
12-1 Multiple Linear Regression Models

12-1.1 Introduction

Figure 12-3 (a)

Three-dimensional
plot of the
regression model
E(Y) = 800 + 10x1 +
7x2 – 8.5x12 – 5x22 +
4x1x2. (b) The
contour plot 11
12-1 Multiple Linear Regression Models

12-1.2 Least Squares Estimation of the Parameters

12
12-1 Multiple Linear Regression Models

12-1.2 Least Squares Estimation of the Parameters

• The least squares function is given by

• The least squares estimates must satisfy

13
12-1 Multiple Linear Regression Models

12-1.2 Least Squares Estimation of the Parameters

• The least squares normal Equations are

• The solution to the normal Equations are the least

squares estimators of the regression coefficients.
14
12-1 Multiple Linear Regression Models

Example 12-1

15
12-1 Multiple Linear Regression Models
Example 12-1

16
 12-1 Multiple Linear Regression Models
12-1.3 Matrix Approach to Multiple Linear Regression

Suppose the model relating the regressors to the

response is

In matrix notation this model can be written as

17
 12-1 Multiple Linear Regression Models
12-1.3 Matrix Approach to Multiple Linear Regression

where

18
 12-1 Multiple Linear Regression Models
12-1.3 Matrix Approach to Multiple Linear Regression

We wish to find the vector of least squares

estimators that minimizes:

The resulting least squares estimate is

19
 12-1 Multiple Linear Regression Models
12-1.3 Matrix Approach to Multiple Linear Regression

20
12-1 Multiple Linear Regression Models

Example 12-2

21
Example 12-2

22
12-1 Multiple Linear Regression Models

Example 12-2

23
12-1 Multiple Linear Regression Models

Example 12-2

24
12-1 Multiple Linear Regression Models
Example 12-2

25
12-1 Multiple Linear Regression Models
Example 12-2

26
27
12-1 Multiple Linear Regression Models
Estimating 2

An unbiased estimator of 2 is

28
12-2 Hypothesis Tests in Multiple Linear
Regression
12-2.1 Test for Significance of Regression
The appropriate hypotheses are

The test statistic is

29
12-2 Hypothesis Tests in Multiple Linear
Regression
12-2.1 Test for Significance of Regression
ANOVA Table

30
12-2 Hypothesis Tests in Multiple Linear
Regression
An Example

31
12-2 Hypothesis Tests in Multiple Linear
Regression
R2 and Adjusted R2
The coefficient of multiple determination

• For the wire bond pull strength data, we find that R2 =

SSR/SST = 5990.7712/6105.9447 = 0.9811.
• Thus, the model accounts for about 98% of the
variability in the pull strength response.
32
12-2 Hypothesis Tests in Multiple Linear
Regression
R2 and Adjusted R2
The adjusted R2 is

• The adjusted R2 statistic penalizes the analyst for

adding terms to the model.
• It can help guard against overfitting (including
regressors that are not really useful)
33
12-3 Confidence Intervals in Multiple
Linear Regression
12-3.1 Confidence Intervals on Individual
Regression Coefficients

Definition

34
12-3 Confidence Intervals in Multiple
Linear Regression
12-3.2 Confidence Interval on the Mean Response

The mean response at a point x0 is estimated by

The variance of the estimated mean response is

35
12-3 Confidence Intervals in Multiple
Linear Regression
12-3.2 Confidence Interval on the Mean Response

Definition

36
 12-5 Model Adequacy
12-5.1 Residual Analysis
Example 12-9

Figure 12-6 Normal

probability plot of residuals
37
 12-5 Model Adequacy
12-5.1 Residual Analysis
Example 12-9

38
 12-5 Model Adequacy
12-5.1 Residual Analysis
Example 12-9

Figure 12-7 Plot of

residuals against ŷi.
39
 12-5 Model Adequacy
12-5.1 Residual Analysis
Example 12-9

Figure 12-8 Plot of

residuals against x1.
40
 12-5 Model Adequacy Checking
12-5.1 Residual Analysis

Example 12-9

Figure 12-9 Plot of

residuals against x2.
41
12-6 Polynomial Regression Models

42
 12-6 Polynomial Regression Models

Example 12-11

43
 12-6 Polynomial Regression Models

Example 12-11

Figure 12-11 Data for

Example 12-11.
44
Example 12-11

45
 12-6 Polynomial Regression Models

Example 12-11

46
 Concept Map

Ho
Multiple H1
Linear CI Polynomial Next ?
Regression R2 Models
Model
ANOVA

47
Practice Problems

48
12-1 Multiple Linear Regression Models

Example 12-1

49
12-1 Multiple Linear Regression Models

Example 12-1

50
12-1 Multiple Linear Regression Models

Figure 12-4 Matrix of scatter plots (from Minitab) for the

wire bond pull strength data in Table 12-2. 51
12-1 Multiple Linear Regression Models
Example 12-1

52
12-1 Multiple Linear Regression Models
Example 12-1

53
12-1 Multiple Linear Regression Models
Example 12-1

54
12-2 Hypothesis Tests in Multiple Linear
Regression
Example 12-3

55
12-2 Hypothesis Tests in Multiple Linear
Regression
Example 12-3

56
12-2 Hypothesis Tests in Multiple Linear
Regression
Example 12-3

57
12-2 Hypothesis Tests in Multiple Linear
Regression
Example 12-3

58
12-3 Confidence Intervals in Multiple
Linear Regression

Example 12-6

59
12-3 Confidence Intervals in Multiple
Linear Regression
Example 12-7

60
12-3 Confidence Intervals in Multiple
Linear Regression
Example 12-7

61
 MORE?
(Not in the exam)

62
12-2 Hypothesis Tests in Multiple Linear
Regression
12-2.2 Tests on Individual Regression Coefficients and
Subsets of Coefficients

The hypotheses for testing the significance of any

individual regression coefficient:

63
12-2 Hypothesis Tests in Multiple Linear
Regression
12-2.2 Tests on Individual Regression Coefficients and
Subsets of Coefficients
The test statistic is

• Reject H0 if |t0| > t/2,n-p.

• This is called a partial or marginal test

64
12-2 Hypothesis Tests in Multiple Linear
Regression
Example 12-4

65
12-2 Hypothesis Tests in Multiple Linear
Regression

Example 12-4

66
12-2 Hypothesis Tests in Multiple Linear
Regression
The general regression significance test or the extra
sum of squares method:

We wish to test the hypotheses:

67
12-2 Hypothesis Tests in Multiple Linear
Regression
A general form of the model can be written:

where X1 represents the columns of X associated with

1 and X2 represents the columns of X associated with
2

68
12-2 Hypothesis Tests in Multiple Linear
Regression

For the full model:

If H0 is true, the reduced model is

69
12-2 Hypothesis Tests in Multiple Linear
Regression

The test statistic is:

Reject H0 if f0 > f,r,n-p

The test in Equation (12-32) is often referred to as a
partial F-test

70
12-2 Hypothesis Tests in Multiple Linear
Regression
Example 12-5

71
12-2 Hypothesis Tests in Multiple Linear
Regression
Example 12-5

72
12-2 Hypothesis Tests in Multiple Linear
Regression
Example 12-5

73
12-4 Prediction of New Observations

A point estimate of the future observation Y0 is

A 100(1-)% prediction interval for this future

observation is

74
12-4 Prediction of New Observations

Figure 12-5 An example

of extrapolation in
multiple regression
75
12-4 Prediction of New Observations

Example 12-8

76
 12-5 Model Adequacy Checking
12-5.1 Residual Analysis

77
 12-5 Model Adequacy Checking
12-5.1 Residual Analysis

The variance of the ith residual is

78
 12-5 Model Adequacy Checking
12-5.1 Residual Analysis

79
 12-6 Aspects of Multiple Regression
Modeling
12-6.2 Categorical Regressors and Indicator Variables

• Many problems may involve qualitative or

categorical variables.
• The usual method for the different levels of a
qualitative variable is to use indicator variables.
• For example, to introduce the effect of two different
operators into a regression model, we could define an
indicator variable as follows:

80
12-6 Aspects of Multiple Regression
Modeling
Example 12-12

81
12-6 Aspects of Multiple Regression
Modeling
Example 12-12

82
12-6 Aspects of Multiple Regression
Modeling
Example 12-12

83
Example 12-12

84
12-6 Aspects of Multiple Regression
Modeling
Example 12-12

85
12-6 Aspects of Multiple Regression
Modeling

Example 12-12

86
 12-6 Aspects of Multiple Regression
Modeling
12-6.3 Selection of Variables and Model Building

87
 12-6 Aspects of Multiple Regression
Modeling
12-6.3 Selection of Variables and Model Building
All Possible Regressions – Example 12-13

88
 12-6 Aspects of Multiple Regression
Modeling
12-6.3 Selection of Variables and Model Building
All Possible Regressions – Example 12-13

89
 12-6 Aspects of Multiple Regression
Modeling
12-6.3 Selection of Variables and Model Building
All Possible Regressions – Example 12-13

Figure 12-12 A
matrix of Scatter
plots from Minitab
for the Wine
Quality Data.

90
91
12-6.3 Selection of Variables and Model Building
Stepwise Regression
Example 12-13

92
12-6.3 Selection of Variables and Model Building
Backward Regression
Example 12-13

93
 12-6 Aspects of Multiple Regression
Modeling
12-6.4 Multicollinearity

Variance Inflation Factor (VIF)

94
 12-6 Aspects of Multiple Regression
Modeling
12-6.4 Multicollinearity
The presence of multicollinearity can be detected in several
ways. Two of the more easily understood of these are:

95
Matrix Inverse

96
 To find the inverse matrix:
we have to find the adjugate matrix and
divide by the determinant

97
98