This Book is Available on YakiBooki.

com

Methods of Multivariate Analysis

Second Edition

https://www.yakibooki.com/download/methods-of-multivariate-analysis-2nd-edition/
 This Book is Available on YakiBooki.com

Methods of Multivariate Analysis

Second Edition

ALVIN C. RENCHER
Brigham Young University

A JOHN WILEY & SONS, INC. PUBLICATION

https://www.yakibooki.com/download/methods-of-multivariate-analysis-2nd-edition/
 This Book is Available on YakiBooki.com

This book is printed on acid-free paper. ∞

Copyright

c 2002 by John Wiley & Sons, Inc. All rights reserved.

Published simultaneously in Canada.

No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form
or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as
permitted under Sections 107 or 108 of the 1976 United States Copyright Act, without either the prior
written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to
the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978)
750-4744. Requests to the Publisher for permission should be addressed to the Permissions Department,
John Wiley & Sons, Inc., 605 Third Avenue, New York, NY 10158-0012, (212) 850-6011, fax (212)
850-6008. E-Mail: PERMREQ@WILEY.COM.
For ordering and customer service, call 1-800-CALL-WILEY.

Library of Congress Cataloging-in-Publication Data

Rencher, Alvin C., 1934–
Methods of multivariate analysis / Alvin C. Rencher.—2nd ed.
p. cm. — (Wiley series in probability and mathematical statistics)
“A Wiley-Interscience publication.”
Includes bibliographical references and index.
ISBN 0-471-41889-7 (cloth)
1. Multivariate analysis. I. Title. II. Series.
QA278 .R45 2001
519.5 35—dc21 2001046735
Printed in the United States of America
10 9 8 7 6 5 4 3 2 1

https://www.yakibooki.com/download/methods-of-multivariate-analysis-2nd-edition/
 This Book is Available on YakiBooki.com

Contents

1. Introduction 1
1.1 Why Multivariate Analysis?, 1
1.2 Prerequisites, 3
1.3 Objectives, 3
1.4 Basic Types of Data and Analysis, 3

2. Matrix Algebra 5
2.1 Introduction, 5
2.2 Notation and Basic Definitions, 5
2.2.1 Matrices, Vectors, and Scalars, 5
2.2.2 Equality of Vectors and Matrices, 7
2.2.3 Transpose and Symmetric Matrices, 7
2.2.4 Special Matrices, 8
2.3 Operations, 9
2.3.1 Summation and Product Notation, 9
2.3.2 Addition of Matrices and Vectors, 10
2.3.3 Multiplication of Matrices and Vectors, 11
2.4 Partitioned Matrices, 20
2.5 Rank, 22
2.6 Inverse, 23
2.7 Positive Definite Matrices, 25
2.8 Determinants, 26
2.9 Trace, 30
2.10 Orthogonal Vectors and Matrices, 31
2.11 Eigenvalues and Eigenvectors, 32
2.11.1 Definition, 32
2.11.2 I + A and I − A, 33
2.11.3 tr(A) and |A|, 34
2.11.4 Positive Definite and Semidefinite Matrices, 34
2.11.5 The Product AB, 35
2.11.6 Symmetric Matrix, 35

v
https://www.yakibooki.com/download/methods-of-multivariate-analysis-2nd-edition/
 This Book is Available on YakiBooki.com
vi CONTENTS

2.11.7 Spectral Decomposition, 35

2.11.8 Square Root Matrix, 36
2.11.9 Square Matrices and Inverse Matrices, 36
2.11.10 Singular Value Decomposition, 36

3. Characterizing and Displaying Multivariate Data 43

3.1 Mean and Variance of a Univariate Random Variable, 43
3.2 Covariance and Correlation of Bivariate Random Variables, 45
3.2.1 Covariance, 45
3.2.2 Correlation, 49
3.3 Scatter Plots of Bivariate Samples, 50
3.4 Graphical Displays for Multivariate Samples, 52
3.5 Mean Vectors, 53
3.6 Covariance Matrices, 57
3.7 Correlation Matrices, 60
3.8 Mean Vectors and Covariance Matrices for Subsets of
Variables, 62
3.8.1 Two Subsets, 62
3.8.2 Three or More Subsets, 64
3.9 Linear Combinations of Variables, 66
3.9.1 Sample Properties, 66
3.9.2 Population Properties, 72
3.10 Measures of Overall Variability, 73
3.11 Estimation of Missing Values, 74
3.12 Distance between Vectors, 76

4. The Multivariate Normal Distribution 82

4.1 Multivariate Normal Density Function, 82
4.1.1 Univariate Normal Density, 82
4.1.2 Multivariate Normal Density, 83
4.1.3 Generalized Population Variance, 83
4.1.4 Diversity of Applications of the Multivariate Normal, 85
4.2 Properties of Multivariate Normal Random Variables, 85
4.3 Estimation in the Multivariate Normal, 90
4.3.1 Maximum Likelihood Estimation, 90
4.3.2 Distribution of y and S, 91
4.4 Assessing Multivariate Normality, 92
4.4.1 Investigating Univariate Normality, 92
4.4.2 Investigating Multivariate Normality, 96

https://www.yakibooki.com/download/methods-of-multivariate-analysis-2nd-edition/
 This Book is Available on YakiBooki.com
CONTENTS vii

4.5 Outliers, 99
4.5.1 Outliers in Univariate Samples, 100
4.5.2 Outliers in Multivariate Samples, 101

5. Tests on One or Two Mean Vectors 112

5.1 Multivariate versus Univariate Tests, 112
5.2 Tests on ␮ with ⌺ Known, 113
5.2.1 Review of Univariate Test for H0 : µ = µ0
with σ Known, 113
5.2.2 Multivariate Test for H0 : ␮ = ␮0 with ⌺ Known, 114
5.3 Tests on ␮ When ⌺ Is Unknown, 117
5.3.1 Review of Univariate t-Test for H0 : µ = µ0 with σ
Unknown, 117
5.3.2 Hotelling’s T 2 -Test for H0 : ␮ = ␮0 with ⌺ Unknown, 117
5.4 Comparing Two Mean Vectors, 121
5.4.1 Review of Univariate Two-Sample t-Test, 121
5.4.2 Multivariate Two-Sample T 2 -Test, 122
5.4.3 Likelihood Ratio Tests, 126
5.5 Tests on Individual Variables Conditional on Rejection of H0 by
the T 2 -Test, 126
5.6 Computation of T 2 , 130
5.6.1 Obtaining T 2 from a MANOVA Program, 130
5.6.2 Obtaining T 2 from Multiple Regression, 130
5.7 Paired Observations Test, 132
5.7.1 Univariate Case, 132
5.7.2 Multivariate Case, 134
5.8 Test for Additional Information, 136
5.9 Profile Analysis, 139
5.9.1 One-Sample Profile Analysis, 139
5.9.2 Two-Sample Profile Analysis, 141

6. Multivariate Analysis of Variance 156

6.1 One-Way Models, 156
6.1.1 Univariate One-Way Analysis of Variance (ANOVA), 156
6.1.2 Multivariate One-Way Analysis of Variance Model
(MANOVA), 158
6.1.3 Wilks’ Test Statistic, 161
6.1.4 Roy’s Test, 164
6.1.5 Pillai and Lawley–Hotelling Tests, 166

https://www.yakibooki.com/download/methods-of-multivariate-analysis-2nd-edition/
 This Book is Available on YakiBooki.com
viii CONTENTS

6.1.6 Unbalanced One-Way MANOVA, 168

6.1.7 Summary of the Four Tests and Relationship to T 2 , 168
6.1.8 Measures of Multivariate Association, 173
6.2 Comparison of the Four Manova Test Statistics, 176
6.3 Contrasts, 178
6.3.1 Univariate Contrasts, 178
6.3.2 Multivariate Contrasts, 180
6.4 Tests on Individual Variables Following Rejection of H0 by the
Overall MANOVA Test, 183
6.5 Two-Way Classification, 186
6.5.1 Review of Univariate Two-Way ANOVA, 186
6.5.2 Multivariate Two-Way MANOVA, 188
6.6 Other Models, 195
6.6.1 Higher Order Fixed Effects, 195
6.6.2 Mixed Models, 196
6.7 Checking on the Assumptions, 198
6.8 Profile Analysis, 199
6.9 Repeated Measures Designs, 204
6.9.1 Multivariate vs. Univariate Approach, 204
6.9.2 One-Sample Repeated Measures Model, 208
6.9.3 k-Sample Repeated Measures Model, 211
6.9.4 Computation of Repeated Measures Tests, 212
6.9.5 Repeated Measures with Two Within-Subjects
Factors and One Between-Subjects Factor, 213
6.9.6 Repeated Measures with Two Within-Subjects
Factors and Two Between-Subjects Factors, 219
6.9.7 Additional Topics, 221
6.10 Growth Curves, 221
6.10.1 Growth Curve for One Sample, 221
6.10.2 Growth Curves for Several Samples, 229
6.10.3 Additional Topics, 230
6.11 Tests on a Subvector, 231
6.11.1 Test for Additional Information, 231
6.11.2 Stepwise Selection of Variables, 233

7. Tests on Covariance Matrices 248

7.1 Introduction, 248
7.2 Testing a Specified Pattern for ⌺, 248
7.2.1 Testing H0 : ⌺ = ⌺0 , 248

https://www.yakibooki.com/download/methods-of-multivariate-analysis-2nd-edition/
 This Book is Available on YakiBooki.com
CONTENTS ix

7.2.2 Testing Sphericity, 250

7.2.3 Testing H0 : ⌺ = σ 2 [(1 − ρ)I + ρJ], 252
7.3 Tests Comparing Covariance Matrices, 254
7.3.1 Univariate Tests of Equality of Variances, 254
7.3.2 Multivariate Tests of Equality of Covariance Matrices, 255
7.4 Tests of Independence, 259
7.4.1 Independence of Two Subvectors, 259
7.4.2 Independence of Several Subvectors, 261
7.4.3 Test for Independence of All Variables, 265

8. Discriminant Analysis: Description of Group Separation 270

8.1 Introduction, 270
8.2 The Discriminant Function for Two Groups, 271
8.3 Relationship between Two-Group Discriminant Analysis and
Multiple Regression, 275
8.4 Discriminant Analysis for Several Groups, 277
8.4.1 Discriminant Functions, 277
8.4.2 A Measure of Association for Discriminant Functions, 282
8.5 Standardized Discriminant Functions, 282
8.6 Tests of Significance, 284
8.6.1 Tests for the Two-Group Case, 284
8.6.2 Tests for the Several-Group Case, 285
8.7 Interpretation of Discriminant Functions, 288
8.7.1 Standardized Coefficients, 289
8.7.2 Partial F-Values, 290
8.7.3 Correlations between Variables and Discriminant
Functions, 291
8.7.4 Rotation, 291
8.8 Scatter Plots, 291
8.9 Stepwise Selection of Variables, 293

9. Classification Analysis: Allocation of Observations to Groups 299

9.1 Introduction, 299
9.2 Classification into Two Groups, 300
9.3 Classification into Several Groups, 304
9.3.1 Equal Population Covariance Matrices: Linear
Classification Functions, 304
9.3.2 Unequal Population Covariance Matrices: Quadratic
Classification Functions, 306

https://www.yakibooki.com/download/methods-of-multivariate-analysis-2nd-edition/
 This Book is Available on YakiBooki.com
x CONTENTS

9.4 Estimating Misclassification Rates, 307

9.5 Improved Estimates of Error Rates, 309
9.5.1 Partitioning the Sample, 310
9.5.2 Holdout Method, 310
9.6 Subset Selection, 311
9.7 Nonparametric Procedures, 314
9.7.1 Multinomial Data, 314
9.7.2 Classification Based on Density Estimators, 315
9.7.3 Nearest Neighbor Classification Rule, 318

10. Multivariate Regression 322

10.1 Introduction, 322
10.2 Multiple Regression: Fixed x’s, 323
10.2.1 Model for Fixed x’s, 323
10.2.2 Least Squares Estimation in the Fixed-x Model, 324
10.2.3 An Estimator for σ 2 , 326
10.2.4 The Model Corrected for Means, 327
10.2.5 Hypothesis Tests, 329
10.2.6 R 2 in Fixed-x Regression, 332
10.2.7 Subset Selection, 333
10.3 Multiple Regression: Random x’s, 337
10.4 Multivariate Multiple Regression: Estimation, 337
10.4.1 The Multivariate Linear Model, 337
10.4.2 Least Squares Estimation in the Multivariate Model, 339
10.4.3 Properties of Least Squares Estimators B̂, 341
10.4.4 An Estimator for ⌺, 342
10.4.5 Model Corrected for Means, 342
10.5 Multivariate Multiple Regression: Hypothesis Tests, 343
10.5.1 Test of Overall Regression, 343
10.5.2 Test on a Subset of the x’s, 347
10.6 Measures of Association between the y’s and the x’s, 349
10.7 Subset Selection, 351
10.7.1 Stepwise Procedures, 351
10.7.2 All Possible Subsets, 355
10.8 Multivariate Regression: Random x’s, 358

11. Canonical Correlation 361

11.1 Introduction, 361
11.2 Canonical Correlations and Canonical Variates, 361

https://www.yakibooki.com/download/methods-of-multivariate-analysis-2nd-edition/
 This Book is Available on YakiBooki.com
CONTENTS xi

11.3 Properties of Canonical Correlations, 366

11.4 Tests of Significance, 367
11.4.1 Tests of No Relationship between the y’s and the x’s, 367
11.4.2 Test of Significance of Succeeding Canonical
Correlations after the First, 369
11.5 Interpretation, 371
11.5.1 Standardized Coefficients, 371
11.5.2 Correlations between Variables and Canonical Variates, 373
11.5.3 Rotation, 373
11.5.4 Redundancy Analysis, 373
11.6 Relationships of Canonical Correlation Analysis to Other
Multivariate Techniques, 374
11.6.1 Regression, 374
11.6.2 MANOVA and Discriminant Analysis, 376

12. Principal Component Analysis 380

12.1 Introduction, 380
12.2 Geometric and Algebraic Bases of Principal Components, 381
12.2.1 Geometric Approach, 381
12.2.2 Algebraic Approach, 385
12.3 Principal Components and Perpendicular Regression, 387
12.4 Plotting of Principal Components, 389
12.5 Principal Components from the Correlation Matrix, 393
12.6 Deciding How Many Components to Retain, 397
12.7 Information in the Last Few Principal Components, 401
12.8 Interpretation of Principal Components, 401
12.8.1 Special Patterns in S or R, 402
12.8.2 Rotation, 403
12.8.3 Correlations between Variables and Principal
Components, 403
12.9 Selection of Variables, 404

13. Factor Analysis 408

13.1 Introduction, 408
13.2 Orthogonal Factor Model, 409
13.2.1 Model Definition and Assumptions, 409
13.2.2 Nonuniqueness of Factor Loadings, 414
13.3 Estimation of Loadings and Communalities, 415
13.3.1 Principal Component Method, 415
13.3.2 Principal Factor Method, 421

https://www.yakibooki.com/download/methods-of-multivariate-analysis-2nd-edition/