De Gruyter Graduate Lectures

Börm / Mehl · Numerical Methods for Eigenvalue Problems

Steffen Börm
Christian Mehl

Numerical Methods for

Eigenvalue Problems

De Gruyter
0DWKHPDWLFV6XEMHFW&ODVVL¿FDWLRQ3ULPDU\ 15A18, 15A22, 15A23, 15A42, 65F15,
65F25; 6HFRQGDU\ 65N25, 15B57.

ISBN 978-3-11-025033-6
e-ISBN 978-3-11-025037-4

/LEUDU\RI&RQJUHVV&DWDORJLQJLQ3XEOLFDWLRQ'DWD

A CIP catalog record for this book has been applied for at the Library of Congress.

%LEOLRJUDSKLFLQIRUPDWLRQSXEOLVKHGE\WKH'HXWVFKH1DWLRQDOELEOLRWKHN
7KH'HXWVFKH1DWLRQDOELEOLRWKHNOLVWVWKLVSXEOLFDWLRQLQWKH'HXWVFKH1DWLRQDOELEOLRJUD¿H
detailed bibliographic data are available in the internet at http://dnb.d-nb.de.

© 2012 Walter de Gruyter GmbH & Co. KG, Berlin/Boston

Typesetting: Da-TeX Gerd Blumenstein, Leipzig, www.da-tex.de

Printing and binding: Hubert & Co. GmbH & Co. KG, Göttingen
’3ULQWHGRQDFLGIUHHSDSHU

Printed in Germany

www.degruyter.com
Preface

Eigenvalues and eigenvectors of matrices and linear operators play an important role
when solving problems from structural mechanics, and electrodynamics, e.g., by de-
scribing the resonance frequencies of systems, when investigating the long-term be-
haviour of stochastic processes, e.g., by describing invariant probability measures,
and as a tool for solving more general mathematical problems, e.g., by diagonalizing
ordinary differential equations or systems from control theory.
This book presents a number of the most important numerical methods for finding
eigenvalues and eigenvectors of matrices. It is based on lecture notes of a short course
for third-year students in mathematics, but it should also be accessible to students of
physics or engineering sciences.
We discuss the central ideas underlying the different algorithms and introduce the
theoretical concepts required to analyze their behaviour. Our goal is to present an
easily accessible introduction to the field, including rigorous proofs of all important
results, but not a complete overview of the vast body of research.
For an in-depth coverage of the theory of eigenvalue problems, we can recommend
the following monographs:
 J. H. Wilkinson, “The Algebraic Eigenvalue Problem” [52]
 B. N. Parlett, “The Symmetric Eigenvalue Problem” [33]
 G. H. Golub and C. F. Van Loan, “Matrix Computations” [18]
 G. W. Stewart, “Matrix Algorithms” [43, 44]
 D. S. Watkins, “The matrix eigenvalue problem” [49]
We owe a great debt of gratitude to their authors, since this book would not exist
without their work.
The book is intended as the basis for a short course (one semester or trimester) for
third- or fourth-year undergraduate students. We have organized the material mostly
in short sections that should each fit one session of a course. Some chapters and sec-
tions are marked by an asterisk  . These contain additional results that we consider
optional, e.g., rather technical proofs of general results or algorithms for special prob-
lems. With one exception, the results of these optional sections are not required for
the remainder of the book. The one exception is the optional Section 2.7 on non-
unitary transformations, which lays the groundwork for the optional Section 4.8 on
the convergence of the power iteration for general matrices.
vi Preface

In order to keep the presentation self-contained, a number of important results are

proven only for special cases, e.g., for self-adjoint matrices or a spectrum consisting
only of simple eigenvalues. For the general case, we would like to refer the reader to
the monographs mentioned above.
Deviating from the practice of collecting fundamental results in a separate chapter,
we introduce some of these results when they are required. An example is the Bauer–
Fike theorem given as Proposition 3.11 in Section 3.4 on error estimates for the Jacobi
iteration instead of in a separate chapter on perturbation theory. While this approach is
certainly not adequate for a reference work, we hope that it improves the accessibility
of lecture notes like this book that are intended to be taught in sequence.
We would like to thank Daniel Kressner for his valuable contributions to this book.
Kiel, December 2011 Berlin, December 2011
Steffen Börm Christian Mehl
Contents

Preface v
1 Introduction 1
1.1 Example: Structural mechanics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Example: Stochastic processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3 Example: Systems of linear differential equations . . . . . . . . . . . . . . . . 5
2 Existence and properties of eigenvalues and eigenvectors 8
2.1 Eigenvalues and eigenvectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.2 Characteristic polynomials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.3 Similarity transformations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.4 Some properties of Hilbert spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.5 Invariant subspaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.6 Schur decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.7 Non-unitary transformations  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3 Jacobi iteration 39
3.1 Iterated similarity transformations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.2 Two-dimensional Schur decomposition . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.3 One step of the iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.4 Error estimates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
3.5 Quadratic convergence  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
4 Power methods 61
4.1 Power iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
4.2 Rayleigh quotient . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
4.3 Residual-based error control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
4.4 Inverse iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
4.5 Rayleigh iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
4.6 Convergence to invariant subspace . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
viii Contents

4.7 Simultaneous iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

4.8 Convergence for general matrices  . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
5 QR iteration 100
5.1 Basic QR step . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
5.2 Hessenberg form . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
5.3 Shifting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
5.4 Deflation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
5.5 Implicit iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
5.6 Multiple-shift strategies  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
6 Bisection methods  132
6.1 Sturm chains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
6.2 Gershgorin discs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
7 Krylov subspace methods for large sparse eigenvalue problems 145
7.1 Sparse matrices and projection methods . . . . . . . . . . . . . . . . . . . . . . . . 145
7.2 Krylov subspaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
7.3 Gram–Schmidt process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
7.4 Arnoldi iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
7.5 Symmetric Lanczos algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
7.6 Chebyshev polynomials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
7.7 Convergence of Krylov subspace methods . . . . . . . . . . . . . . . . . . . . . . 172
8 Generalized and polynomial eigenvalue problems  182
8.1 Polynomial eigenvalue problems and linearization . . . . . . . . . . . . . . . . 182
8.2 Matrix pencils . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
8.3 Deflating subspaces and the generalized Schur decomposition . . . . . . . 189
8.4 Hessenberg-triangular form . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
8.5 Deflation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
8.6 The QZ step . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
Bibliography 203
Index 206
Chapter 1

Introduction

Eigenvalue problems play an important role in a number of fields of numerical math-

ematics: in structural mechanics and electrodynamics, eigenvalues correspond to res-
onance frequencies of systems, i.e., to frequencies to which these systems respond
particularly well (or badly, depending on the context). When studying stochastic pro-
cesses, invariant probability measures correspond to eigenvectors for the eigenvalue 1,
and finding them yields a description of the long-term behaviour of the corresponding
process.
This book gives an introduction to the basic theory of eigenvalue problems and
focuses on important algorithms for finding eigenvalues and eigenvectors.

1.1 Example: Structural mechanics

Before we consider abstract eigenvalue problems, we turn our attention to some ap-
plications that lead naturally to eigenvalue problems.
The first application is the investigation of resonance frequencies. As an example,
we consider the oscillations of a string of unit length. We represent the string as a
function

u W R  Œ0; 1 ! R; .t; x/ 7! u.t; x/;

where u.t; x/ denotes the deflection of the string at time t and position x (cf. Fig-
ure 1.1).
The oscillations are then described by the wave equation

@2 u @2 u
.t; x/ D c .t; x/ for all t 2 R; x 2 .0; 1/;
@t 2 @x 2
where c > 0 is a parameter describing the string’s properties (e.g., its thickness). We
assume that the string is fixed at both ends, i.e., that

u.t; 0/ D u.t; 1/ D 0 holds for all t 2 R:

Since the differential equation is linear, we can separate the variables: we write u in
the form

u.t; x/ D u0 .x/ cos.!t / for all t 2 R; x 2 Œ0; 1

2 Chapter 1 Introduction

∂ 2u ∂2u
u(t,x) ∂t 2 (t,x) = c ∂x 2 (t,x)

Figure 1.1. Mathematical model of a string.

with a frequency parameter ! 2 R0 and a function

u0 W Œ0; 1 ! R; x 7! u0 .x/;

depending only on the location, but not on the time. The differential equation takes
the form
@2 u @2 u
! 2 u0 .x/ cos.!t / D .t; x/ D c .t; x/
@t 2 @x 2
D cu000 .x/ cos.!t / for all t 2 R; x 2 .0; 1/;

and eliminating the time-dependent factor yields

cu000 .x/ D ! 2 u0 .x/ for all x 2 .0; 1/: (1.1)

We introduce  WD ! 2 2 R0 and define the differential operator L by

LŒu0 .x/ WD cu000 .x/ for all u0 2 C 2 .0; 1/; x 2 .0; 1/

in order to obtain
LŒu0  D u0 :
This is an eigenvalue problem in the infinite-dimensional space C 2 .0; 1/.
In order to be able to treat it by a numerical method, we have to discretize the
problem. A simple approach is the finite difference method: Taylor expansion yields

h2 00 h3 .3/ h4 .4/
u0 .x C h/ Du0 .x/ C hu00 .x/ C u0 .x/ C u0 .x/ C u0 .C /;
2 6 24
2 3 4
h h h
u0 .x  h/ Du0 .x/  hu00 .x/ C u000 .x/  u0 .x/ C u0 . /
.3/ .4/
2 6 24
Section 1.1 Example: Structural mechanics 3

for h 2 R>0 with 0  x  h  x C h  1, where C 2 Œx; x C h and  2 Œx  h; x.

Adding both equations and using the intermediate value theorem yields
h4 .4/
u0 .x  h/  2u0 .x/ C u0 .x C h/ D h2 u000 .x/ C u ./
12 0
with  2 Œx  h; x C h. Dividing by h2 gives us an equation for the second derivative:
u0 .x  h/  2u0 .x/ C u0 .x C h/ 00 h2 .4/
D u0 .x/ C u ./:
h2 12 0
We obtain a useful approximation by dropping the right-most term: fixing n 2 N and
setting
1
xk WD hk; h WD for all k 2 ¹0; : : : ; n C 1º;
nC1
we find
u0 .xk1 /  2u0 .xk / C u0 .xkC1 /
 u000 .xk / for all k 2 ¹1; : : : ; nº;
h2
and the term on the left-hand side requires only values of u0 in the discrete points
x0 ; : : : ; xnC1 . We collect these values in a vector
0 1
u0 .x1 /
B C
e WD @ ::: A ; e0 D enC1 D 0;
u0 .xn /

and replace u000 .x/ in (1.1) by the approximation to get

2ek  ek1  ekC1
c  ek for all k 2 ¹1; : : : ; nº:
h2
This system can be written as the algebraic eigenvalue problem
0 10 1 0 1
2 1 e1 e1
B CB : C B :C
B :: :: C
c B1 : : CB :C B:C
B : C  B : C;
B C B :: C B :: C (1.2)
h2 B : 1C
:: ::
@ : A@ : A @:A
en en
1 2
and solving the system yields approximations u0 .xk /  ek of the values of u0 in the
points x1 ; : : : ; xn .
In order to reach a high accuracy, we have to ensure that h is small, so we have to
be able to handle large values of n. We are typically only interested in computing a
small number of the smallest eigenvalues, and this problem can be solved efficiently
by specialized algorithms (e.g., the inverse iteration discussed in Chapter 4).
4 Chapter 1 Introduction

1.2 Example: Stochastic processes

The next example is not motivated by physics, but by computer science: we are inter-
ested in determining the “most important” pages of the world wide web. Let n 2 N be
the number of web pages, and let L 2 Rnn represent the hyperlinks between these
pages in the following way:
´
1 if page j contains a link to page i;
`ij D for all i; j 2 ¹1; : : : ; nº:
0 otherwise
We follow the PageRank [31] approach: we consider a “random web user” that moves
.m/
from page to page and compute the probability pj 2 Œ0; 1 of him visiting a certain
page j in his m-th step. We denote the number of links on page j by
X
n
`j WD `ij for all j 2 ¹1; : : : ; nº
i D1

and assume that the random user chooses each of the links with equal probability
1=`j . In order to make this approach feasible, we have to assume `j ¤ 0 for all
j 2 ¹1; : : : ; nº, i.e., we have to assume that each page contains at least one link.
This means that the probability of switching from page j to page i is given by
´
`ij 1=`j if `ij D 1;
sij WD D for all i; j 2 ¹1; : : : ; nº:
`j 0 otherwise
The probability of visiting page i in step m C 1 is given by

.mC1/
X
n
.m/
pi D sij pj for all i 2 ¹1; : : : ; nº; m 2 N0 :
j D1

Since this equation corresponds to a matrix-vector multiplication by S D .sij /ni;j D1 ,

we can write it in the compact form
p .mC1/ D Sp .m/ for all m 2 N0 : (1.3)
In order to ensure that the result does not depend on the arbitrarily chosen starting
vector p .0/ , the PageRank algorithm uses the limit
p  WD lim p .m/ (1.4)
m!1

to determine the “importance” of a web page: if pj is large, the probability of a user
visiting the j -th web page is high, therefore it is assumed to be important. Due to
p  D lim p .m/ D lim p .mC1/ D lim Sp .m/ D S lim p .m/ D Sp  ;
m!1 m!1 m!1 m!1

the vector p 2 Rn is an eigenvector of the matrix S for the eigenvalue one.

Section 1.3 Example: Systems of linear differential equations 5

In this example, we not only reduce a problem related to stochastic processes (the
“random walk” of the web user) to an algebraic eigenvalue problem, but we also find
a simple algorithm for computing at least the eigenvector: due to (1.4), we can hope
to approximate p  by computing p .m/ for a sufficiently large value of m. Due to
(1.3), this requires only m matrix-vector multiplications, and since we can assume
that each web page contains only a small number of links, these multiplications can
be performed very efficiently.
In order to ensure convergence, the PageRank algorithm replaces the matrix S by
the matrix bS D .Osij /ni;j D1 given by

sOij D .1  ˛/sij C ˛ui for all i; j 2 ¹1; : : : ; nº;

where ˛ 2 .0; 1 (a typical value is 0.15) is a parameter controlling how close b S

is to the original matrix S, while u 2 R n is a vector (the “teleportation vector”)

describing the event that a user switches to a different page without following a link:
when visiting page j , the user either follows one of the links with a total probability
of 1  ˛ or switches to another page with a total probability of ˛. In the latter case,
ui is the relative probability of switching to page i . For a suitable choice of u (e.g.,
a vector with strictly positive entries and u1 C : : : C un D 1), the Perron–Frobenius
theorem [34] ensures convergence of the sequence .p .m/ /1 mD0 to p .


1.3 Example: Systems of linear differential equations

Another, more general example for eigenvalue problems are systems of linear dif-
ferential equations that appear frequently in natural and engineering sciences. If
F 2 ¹R; Cº and A 2 F nn then
y 0 D Ay (1.5)
is a (homogeneous) system of linear differential equations and a solution is defined to
be a continuously differentiable function y W R ! F n satisfying y 0 .t / D Ay.t / for
all t 2 R. For some vector y0 2 F n n ¹0º the ansatz y.t / D e t y0 yields the identity

e t y0 D y 0 .t / D Ay.t / D Ae t y0

which, after division on both sides by e t , reduces to the characteristic equation

y0 D Ay0 :

Thus, if  2 F is an eigenvalue of A and y0 2 F n n ¹0º is an associated eigenvector,

then y.t / D e t y0 is a solution of the corresponding system of differential equations.
It is well known from the theory of differential equations that if A is diagonalizable
and if v1 ; : : : ; vn is a basis of F n consisting of eigenvectors of A associated with
6 Chapter 1 Introduction

the eigenvalues 1 ; : : : ; n , then any solution y of the system of differential equa-

tions (1.5) has the form
Xn
y.t / D ci e i t vi
i D1

for some coefficients c1 ; : : : ; cn 2 F. In the non-diagonalizable case, the general

solution can be constructed from the so called Jordan normal form.
Instead of systems of linear differential equations of first order as in the form (1.5),
one may also consider systems of linear differential equations of higher order having
the general form

X̀
Ak y .k/ D A` y .`/ C A`1 y .`1/ C    C A2 y 00 C A1 y 0 C A0 y D 0;
kD0

where A0 ; : : : ; A` 2 F nn . In this case the ansatz y.t / D e t y0 for some nonzero
vector y0 2 F n yields the identity

X̀
Ak k e t y0 D 0;
kD0

or equivalently, after division by e t ,

0 1
X̀
@ k Ak A y0 D 0: (1.6)
kD0

The problem of solving (1.6) is called a polynomial eigenvalue problem.

A particular example in applications can be found in the theory of mechanical vi-
bration. The equations of motion for a viscously damped linear system with n degrees
of freedom are given by

My 00 .t / C Cy 0 .t / C Ky.t / D 0;

where M , C , K are n  n matrices called mass matrix, damping matrix, and stiffness
matrix, respectively. The corresponding quadratic eigenvalue problem has the form

.2 M C C C K/x D 0:

A simple example for the case n D 1 is the spring-mass system with damping by
friction, see Figure 1.2.
By Hooke’s law, the equation of motion for this system without friction is

my 00 .t / C ky.t / D 0;
Section 1.3 Example: Systems of linear differential equations 7

m y(t)

Figure 1.2. Mass-spring system.

where m is the mass attached to the spring and k is the spring constant. If friction
is considered, it is usually modeled in such a way that the friction is assumed to be
proportional to the velocity y 0 .t / thus yielding the equation of motion

my 00 .t / C cy 0 .t / C ky.t / D 0;

for some constant c.

Chapter 2

Existence and properties of eigenvalues and

eigenvectors

Summary
This chapter investigates existence and uniqueness of eigenvalues and eigenvectors
for a given matrix. The key result is the Schur decomposition introduced in Theo-
rem 2.46, a very useful tool for the investigation of eigenvalue problems. One of its
most important consequences is the fact that a matrix can be diagonalized unitarily
if and only if it is normal. The optional Section 2.7 presents a block-diagonalization
result for general square matrices

Learning targets
 Recall the definition and some of the most important properties of eigenvalues,
eigenvectors, similarity transformations and the characteristic polynomial corre-
sponding to a matrix.
 Introduce a number of basic concepts of Hilbert space theory, e.g., the Cauchy–
Schwarz inequality, self-adjoint, normal, isometric and unitary matrices.
 Prove the existence of the Schur decomposition.
 Use it to establish the existence of eigenvector bases for normal and self-adjoint
matrices and of invariant subspaces in the general case.

2.1 Eigenvalues and eigenvectors

Let n; m 2 N, and let F 2 ¹R; Cº be the field of real or complex numbers.
We denote the space of matrices with n rows and m columns by F nm . The coeffi-
cients of a matrix A 2 F nm are given by
0 1
a11 : : : a1m
B
A D @ ::: :: :: C :
: : A
an1 : : : anm
Section 2.1 Eigenvalues and eigenvectors 9

Zero coefficients in a matrix are frequently omitted in our notation, e.g., the n-di-
mensional identity matrix is usually represented by
0 1
1
B C
In D @ : : : A :
1

If the dimension is clear from the context, we denote the identity matrix by I .
The product of a matrix by a vector x 2 F m is given by
0 10 1 0 1
a11 : : : a1m x1 a11 x1 C    C a1m xm
B
Ax D @ ::: :: :: C B :: C D B :: C
: : A@ : A @ : A
an1 : : : anm xm an1 x1 C    C anm xm

or, more precisely, by

X
m
.Ax/i D aij xj for all i 2 ¹1; : : : ; nº: (2.1)
j D1

The mapping

F m ! F n; x 7! Ax

is a linear operator mapping F m to F n , and we use the corresponding notations:

Definition 2.1 (Null space and range). Let A 2 F nm . The space

N .A/ WD ¹x 2 F m W Ax D 0º

is called the null space of A, and its dimension is called the nullity of A. The space

R.A/ WD ¹y 2 F n W there exists a vector x 2 F m with Ax D yº

is called the range of A, and its dimension is called the rank of A.

The matrix A is called injective if N .A/ D ¹0º holds, and it is called surjective if
R.A/ D F n holds.

We recall the rank-nullity theorem: let .yi /kiD1 be a basis of R.A/. By definition,
x i /kiD1 in F m such that
we can find .b

x i D yi
Ab for all i 2 ¹1; : : : ; kº:

Since the family .yi /kiD1 is linearly independent, the same holds for .bx i /kiD1 .
This means that we can expand the family to a basis .b x i /i D1 of F . For each
m m
10 Chapter 2 Existence and properties of eigenvalues and eigenvectors

j 2 ¹k C 1; : : : ; mº, we obviously have Ab xj 2 R.A/ and can therefore find zj 2

x k º such that Ab
x 1 ; : : : ;b
span¹b xj D Azj holds, i.e., b
xj  zj 2 N .A/. We define
´
bxi if i  k;
xi WD for all i 2 ¹1; : : : ; mº
bx i  zi otherwise

i D1 is a basis of F such that span¹xkC1 ; : : : ; xm º  N .A/ holds.

and see that .xi /m m

This yields dim N .A/ D m  k, i.e.,

dim R.A/ C dim N .A/ D m for all A 2 F nm : (2.2)

Definition 2.2 (Eigenvalue and Eigenvector). Let A 2 F nn , and let  2 F.  is

called an eigenvalue of A, if there is a vector x 2 F n n ¹0º such that

Ax D x (2.3)

holds. Any such vector is called an eigenvector of A for the eigenvalue . A pair
.; x/ consisting of an eigenvalue and a corresponding eigenvector is called an eigen-
pair.
The set
.A/ WD ¹ 2 F W  is an eigenvalue of Aº
is called the spectrum of A.

Let k 2 N. The product AB of two matrices A 2 F nk and B 2 F km is given by

X
k
.AB/ij D ai` b`j for all i 2 ¹1; : : : ; nº; j 2 ¹1; : : : ; mº: (2.4)
`D1
The definition ensures
ABx D A.Bx/ for all x 2 F m ;
i.e., it is compatible with the matrix-vector multiplication.
Exercise 2.3 (Polynomials). Let A 2 F nn . We define the `-th power of the
matrix by
´
` I if ` D 0;
A WD `1
for all ` 2 N0 :
AA otherwise
This definition allows us to apply polynomials to matrices: for each polynomial
p.t / D a0 C a1 t C    C am t m ;
we define
p.A/ WD a0 A0 C a1 A1 C    C am Am :
Prove p..A//  .p.A//. Can you find A and p with .p.A// ¤ p..A//?
Hint: consider Exercise 2.6.
Section 2.1 Eigenvalues and eigenvectors 11

Exercise 2.4 (Projection). Let P 2 F nn be a projection, i.e., let it satisfy P 2 D

P . Prove .P /  ¹0; 1º.
Is any matrix A 2 F nn satisfying .A/  ¹0; 1º a projection?
Hint: consider A 2 F 22 with a21 D 0.

Exercise 2.5 (Nil-potent matrix). Let N 2 F nn be a nil-potent matrix, i.e., let
there be a k 2 N with N k D 0. Prove .N /  ¹0º.

Exercise 2.6 (Empty spectrum). Let F D R. Consider the matrix

 
0 1
A WD 2 R22 :
1 0

Prove .A/ D ;, i.e., show that A has no eigenvalues. Does the situation change
if we let F D C?

If x is an eigenvector of a matrix A 2 F nn , multiplying x by any non-zero number

will again yield an eigenvector. Instead of dealing with non-unique eigenvectors, it is
preferable to use an alternative characterization of eigenvalues:

Proposition 2.7 (Null space). Let A 2 F nn , and let  2 F.  is an eigenvalue of A

if and only if
N .I  A/ ¤ ¹0º
holds, i.e., if I  A is not injective.

Proof. We first observe that for all x 2 F n and all  2 F, the following statements
are equivalent:

x D Ax;
x  Ax D 0;
.I  A/x D 0;
x 2 N .I  A/:

If  2 F is an eigenvalue, we can find a corresponding eigenvector x 2 F n n ¹0º,

i.e., we have Ax D x, and therefore x 2 N .I A/ and N .I A/ ¹0; xº ¤ ¹0º.
If, on the other hand, N .I  A/ ¤ ¹0º holds, we can find x 2 N .I  A/ n ¹0º,
and this vector x is an eigenvector.

Since the null space of I  A is uniquely determined by A and , working with it

instead of individual eigenvectors offers significant advantages both for practical and
theoretical investigations.
12 Chapter 2 Existence and properties of eigenvalues and eigenvectors

Definition 2.8 (Eigenspace). Let A 2 F nn , and let  2 .A/. Then

E.A; / WD N .I  A/

is called the eigenspace of A for the eigenvalue .

Definition 2.9 (Geometric multiplicity). Let A 2 F nn , and let  2 .A/. The
dimension of the eigenspace E.A; / is called the geometric multiplicity of  and
denoted by g .A; /.

Instead of looking for individual eigenvectors, we look for a basis of an eigenspace.

This offers the advantage that we can change the basis during the course of our algo-
rithms in order to preserve desirable properties like isometry or non-degeneracy.

Exercise 2.10 (Eigenspaces). Let A 2 F nn , and let ;  2 .A/ with  ¤ .

Prove
E.A; / \ E.A; / D ¹0º:

Exercise 2.11 (Geometric multiplicity). Let A 2 F nn . Prove

X
g .A; /  n:
2.A/

2.2 Characteristic polynomials

By the rank-nullity theorem (2.2),  is an eigenvalue of a matrix A if and only if
I  A is not invertible. Using this property, we can characterize the eigenvalues
without explicitly constructing eigenvectors.
Let n; m 2 N, and let j 2 ¹1; : : : ; mº. The j -th canonical unit vector ıj 2 F m is
given by
´
1 if i D j;
.ıj /i WD for all i 2 ¹1; : : : ; mº; (2.5)
0 otherwise

and for a matrix A 2 F nm , we denote the j -th column vector by

0 1
a1j
B C
aj WD Aıj D @ ::: A :
anj

The matrix A is injective if and only if its columns a1 ; : : : ; am are linearly indepen-
dent.
Section 2.2 Characteristic polynomials 13

We can use the determinant to characterize tuples of linearly independent vectors.

The determinant is a mapping
det W .F n /n ! F
of n-tuples of n-dimensional vectors to scalar values that is multilinear, i.e., we have
det.x1 ; : : : ; xj 1 ; xj C ˛z; xj C1 ; : : : ; xn /
D det.x1 ; : : : ; xj 1 ; xj ; xj C1 ; : : : ; xn /
C ˛ det.x1 ; : : : ; xj 1 ; z; xj C1 ; : : : ; xn /
for all x1 ; : : : ; xn ; z 2 F n ; ˛ 2 F; j 2 ¹1; : : : ; nº:

The determinant is also alternating, i.e., we have

det.x1 ; : : : ; xi 1 ; xi ; xi C1 ; : : : ; xj 1 ; xj ; xj C1 ; : : : ; xn /
D  det.x1 ; : : : ; xi 1 ; xj ; xi C1 ; : : : ; xj 1 ; xi ; xj C1 ; : : : ; xn /
for all x1 ; : : : ; xn 2 F n ; i; j 2 ¹1; : : : ; nº with i < j
and satisfies det.ı1 ; : : : ; ın / D 1.
Let x1 ; : : : ; xn 2 F n . If there are i; j 2 ¹1; : : : ; nº with i < j and xi D xj , the
fact that the determinant is alternating implies
det.x1 ; : : : ; xi 1 ; xi ; xi C1 ; : : : ; xj 1 ; xj ; xj C1 ; : : : ; xn /
D  det.x1 ; : : : ; xi 1 ; xj ; xi C1 ; : : : ; xj 1 ; xi ; xj C1 ; : : : ; xn /
D  det.x1 ; : : : ; xi 1 ; xi ; xi C1 ; : : : ; xj 1 ; xj ; xj C1 ; : : : ; xn /;
and therefore det.x1 ; : : : ; xn / D 0. Since the determinant is also multilinear, we can
add any multiple of any argument to any other argument without changing the value
of the determinant. In particular, it is possible to prove that the determinant is equal
to zero if and only if its arguments are linearly dependent.
This means that a matrix A 2 F nn is invertible if and only if det.a1 ; : : : ; an / ¤ 0,
i.e., if the determinant of its column vectors vanishes. We extend the definition of the
determinant to quadratic matrices by setting
det W F nn ! F; A 7! det.a1 ; : : : ; an /;
and have det.A/ ¤ 0 if and only if A is invertible. Combining this property with
Proposition 2.7, we obtain a characterization of eigenvalues that does not require
eigenvectors:

Definition 2.12 (Characteristic polynomial). Let A 2 F nn .

pA W F ! F; t 7! det.tI  A/;

is a polynomial of degree n. We call it the characteristic polynomial of A.

14 Chapter 2 Existence and properties of eigenvalues and eigenvectors

Proposition 2.13 (Zeros of pA ). Let A 2 F nn .  2 F is an eigenvalue of A if and

only if pA ./ D 0 holds.
Proof. Let  2 F. If  is an eigenvalue of A, Proposition 2.7 implies that I  A
is not injective, therefore this matrix has to be non-invertible, and we have pA ./ D
det.I  A/ D 0.
If, on the other hand, we have 0 D pA ./ D det.I A/, the matrix I A is non-
invertible. Since it is a quadratic matrix, the rank-nullity theorem (2.2) implies that it
cannot be injective, and Proposition 2.7 yields that  has to be an eigenvalue.
This result allows us to characterize the spectrum of a matrix A 2 F nn as the set
of zeros of its characteristic polynomial:
.A/ D ¹ 2 F W pA ./ D 0º:
Given a polynomial p and a  with p./ D 0, we can find a polynomial q such that
p.t / D .  t /q.t / for all t 2 F:
If  is a zero of q, we can apply this construction repeatedly to find the maximal
power k 2 N such that .  t /k is a divisor of p: the multiplicity of  is the number
k 2 N uniquely defined by the property that there is a polynomial q satisfying
p.t / D .  t /k q.t / for all t 2 F;
q./ ¤ 0:

Definition 2.14 (Algebraic multiplicity). Let A 2 F nn and  2 .A/. By Proposi-

tion 2.13,  is a zero of pA . We call its multiplicity the algebraic multiplicity of the
eigenvalue  and denote it by a .A; /.
If a .A; / D 1,  is called a simple eigenvalue.

Exercise 2.15 (Algebraic multiplicity). Let n 2 N and A 2 C nn . Prove

X
a .A; / D n: (2.6)
2.A/

Hint: the fundamental theorem of algebra states that every non-constant complex-
valued polynomial has at least one root.

Exercise 2.16 (Companion matrix). Let n 2 N2 and c0 ; c1 ; : : : ; cn1 2 F. Let

0 1
0 1
B :: C
B 0 0 : C
B C
B
C WD B :: : : : C:
: : : : : C
B : : C
@ 0 0 ::: 0 1 A
c0 c1 : : : cn2 cn1
Section 2.3 Similarity transformations 15

Prove

pC .t / D c0 C c1 t C    C cn1 t n1 C t n for all t 2 F:

This means that, given a monic polynomial p of order n, we can find a matrix
C 2 F nn such that pC D p holds, i.e., the task of finding the zeros of a
polynomial and the task of finding the eigenvalues of a matrix are equivalent.
The matrix C is called the companion matrix of the polynomial p.
(Hint: use Laplace’s theorem and induction).

The Abel–Ruffini theorem states that there is no general closed-form algebraic

solution to polynomial equations of degree five or higher. Since finding the roots
of a polynomial is equivalent to finding the eigenvalues of the companion matrix,
we cannot hope to find a general algorithm for computing the exact eigenvalues
of matrices of dimension five or higher.
In fact, all practical algorithms compute arbitrarily accurate approximations
of eigenvalues and eigenvectors.

2.3 Similarity transformations

Many popular algorithms for solving systems of linear equations are based on trans-
forming the given matrix to a simple form (e.g., upper triangular) that can be handled
efficiently. We are interested in following a similar approach.
Let A 2 F nn , let  2 .A/, and let x 2 F n n ¹0º be an eigenvector of A for the
eigenvalue . By definition (2.3), we have

Ax D x:

Let now B 2 F nn be an invertible matrix. Multiplying our equation on both sides
with B 1 yields
B 1 Ax D B 1 x;
but since B 1 appears on the right-hand side, this equation is no longer related to
eigenvalue problems. We fix this problem by introducing bx WD B 1 x and find

B 1 ABb
x D b
x:

This is again an eigenvalue problem. Instead of looking for eigenvalues and eigenvec-
tors of A, we can also look for eigenvalues and eigenvectors of the transformed matrix
A WD B 1 AB. Any eigenvalue of b
b A is also an eigenvalue of A, and any eigenvector
of b
A can be transformed to an eigenvector of A by multiplying by B.
16 Chapter 2 Existence and properties of eigenvalues and eigenvectors

Definition 2.17 (Similar matrices). Let A; b A 2 F nn . We call A and b

A similar, if
there is a invertible matrix B 2 F nn such that

b
A D B 1 AB:

The mapping A 7! B 1 AB is called a similarity transformation. It can be inter-

preted as a change of basis: the linear mapping defined by A is represented in the basis
given by the columns of B. As explained above, a change of basis cannot change the
eigenvalues, and we can see that a number of other important properties are also left
unchanged:

Proposition 2.18 (Similar matrices). Let A; b

A 2 F nn be similar matrices, and let
B 2 F nn be invertible with b
A D B 1 AB. Then we have
 .A/ D .b A/,
 E.A; / D BE.b
A; / for all  2 .A/ D .b
A/,
 pA D pb
A
and

 g .A; / D g .b
A; / and a .A; / D a .b
A; / for all  2 .A/ D .b
A/.

Proof. Let  2 .A/, and let x 2 F n n ¹0º be a corresponding eigenvector. We let

x WD B 1 x and observe b
b x ¤ 0, Bb
x D x. We find

bx D B 1 ABB 1 x D B 1 Ax D B 1 x D b
Ab x;

x is an eigenvector of b
so b A, and therefore  2 .b
A/. This implies

.A/  .b
A/; E.A; /  BE.b
A; / for all  2 .A/:

AB 1 , so we can exchange the roles of A and b

We also have A D B b A in order to
obtain

.A/ D .b
A/; E.A; / D BE.b
A; /; g .A; / D g .b
A; / for all  2 .A/:

Since the determinant of the product of two matrices is the product of their deter-
minants, we have

pb
A
.t / D det.tI  bA/ D det.tB 1 B  B 1 AB/ D det.B 1 .tI  A/B/
D det.B 1 / det.tI  A/ det.B/ D det.B 1 / det.B/ det.tI  A/
D det.B 1 B/pA .t / D det.I /pA .t / D pA .t / for all t 2 F;

and this completes the proof.

Section 2.3 Similarity transformations 17

Similarity transformations are an important tool for the investigation of eigenval-

ues and eigenvectors. As an example of its many uses we consider the relationship
between the geometric and the algebraic multiplicity.

Proposition 2.19 (Geometric and algebraic multiplicity). Let A 2 F nn , and let  2
.A/. Then we have
g .A; /  a .A; /:

Proof. Let k WD g .A; / D dim N .I  A/ 2 N. By definition, we can find k

linearly independent vectors e1 ; : : : ; ek 2 F n spanning N .I  A/, i.e., satisfying

Aej D ej for all j 2 ¹1; : : : ; kº:

We can extend .ej /jkD1 to a basis .ej /jnD1 of F n . Then the matrix E 2 F nn
defined by using the basis vectors as columns, i.e., by

eij WD .ej /i for all i; j 2 ¹1; : : : ; nº; (2.7)

is invertible and satisfies Eıj D ej . Let

b
A WD E 1 AE:

For j 2 ¹1; : : : ; kº, we obtain

b
Aıj D E 1 AEıj D E 1 Aej D E 1 ej D E 1 Eıj D ıj

and therefore  
b 1 Ik B
A WD E AE D
C
with B 2 F k.nk/ and C 2 F .nk/.nk/ . Since A and b A are similar, we can apply
Proposition 2.18 to get
 
b tIk  Ik B
pA .t / D pb .t / D det.tIn  A/ D det
A tInk  C
D det..t  /Ik / det.tIkn  C /
D .t  /k det.tIkn  t / for all t 2 F;

i.e.,  is a zero of pA of multiplicity at least k D g .A; /.

Exercise 2.20 (Multiplicities). Consider the matrix

 
1 1
A WD 2 R22 :
1

Compute its eigenvalues and their algebraic and geometric multiplicities.

18 Chapter 2 Existence and properties of eigenvalues and eigenvectors

Of particular interest are similarity transformations turning a matrix into a diagonal

matrix, since these are very useful both for theoretical investigations and practical
applications.
Definition 2.21 (Diagonalizable matrix). Let A 2 F nn . The matrix A is called
diagonalizable if it is similar to a diagonal matrix, i.e., if there are an invertible matrix
B 2 F nn and a diagonal matrix D 2 F nn satisfying

A D BDB 1 :

Proposition 2.22 (Diagonalizable matrix). Let A 2 F nn . The matrix A is diagonal-

izable if and only if there is a basis of F n consisting only of its eigenvectors. This is
equivalent to X
g .A; / D n:
2.A/

Proof. Assume that a basis .ej /jnD1 of eigenvectors exists, and let .j /jnD1 be the
corresponding eigenvalues. Then the matrix E 2 F nn defined by (2.7) is invertible
and satisfies
E 1 AEıj D E 1 Aej D E 1 j ej D j E 1 Eıj D j ıj for all j 2 ¹1; : : : ; nº;
so D WD E 1 AE is a diagonal matrix, and therefore A is diagonalizable.
Assume now that A is diagonalizable, and let E 2 F nn be a invertible matrix and
D 2 F nn a diagonal matrix satisfying A D EDE 1 . We define vectors .ej /jnD1 by
ej WD Eıj for all j 2 ¹1; : : : ; nº
and obtain
Aej D EDE 1 ej D EDıj D Edjj ıj D djj ej ;
i.e., ej is an eigenvector corresponding to the eigenvalue djj . Since E is invertible,
the vectors .ej /jnD1 are linearly independent and therefore a basis of F n .

Exercise 2.23 (Complex diagonalizability). Let n 2 N and A 2 C nn . Prove

that A is diagonalizable if and only if

g .A; / D a .A; / holds for all  2 .A/:

Hint: Proposition 2.19 and Exercise 2.15 could help.

Exercise 2.24 (Matrix exponential). Let A 2 C nn be diagonalizable. Prove
that the matrix exponential
1
X 1 m
exp.A/ WD A
mŠ
mD0
is well-defined and diagonalizable.
Section 2.4 Some properties of Hilbert spaces 19

The matrix exponential is useful for solving linear ordinary differential equa-
tions: prove

@
exp.tA/ D A exp.tA/ for all t 2 R:
@t

Exercise 2.25 (Harmonic oscillator). Consider the ordinary differential equation

u0 .t / D v.t /; v 0 .t / D u.t / for all t 2 R:

Use Exercise 2.24 to describe the space of solutions t 7! .u.t /; v.t // explicitly.

2.4 Some properties of Hilbert spaces

Our next goal is to find criteria for checking whether a given matrix is diagonalizable.
F n is a Hilbert space with the inner product given by
X
n
hx; yi WD xj yNj for all x; y 2 F n
j D1

and the norm given by

v
uX
p u n
kxk D hx; xi D t jxj j2 for all x 2 F n : (2.8)
j D1

One of the most important consequences of the close relationship between the inner
product and the norm is the Cauchy–Schwarz inequality. Its proof serves to illustrate
typical Hilbert space arguments:

Proposition 2.26 (Cauchy–Schwarz inequality). Let x; y 2 F n . We have

jhx; yij  kxk kyk: (2.9)

Both sides are equal if and only if x and y are linearly dependent.

Proof. We assume y ¤ 0 without loss of generality. Let  2 F. We have

0  kx  yk2 D hx  y; x  yi D hx; xi  hx; yi  hy; xi C hy; yi

D kxk2  hx;
N yi  hx; yi C jj2 kyk2 :

In order to minimize the right-hand side, we choose

hx; yi
 WD
kyk2
20 Chapter 2 Existence and properties of eigenvalues and eigenvectors

and obtain
jhx; yij2 jhx; yij2 jhx; yij2
0  kx  yk2 D kxk2  2 C kyk 2
D kxk 2
 ; (2.10)
kyk2 kyk4 kyk2
and multiplying both sides by kyk2 yields the estimate (2.9).
If we have equality, i.e., if jhx; yij D kxk kyk holds, (2.10) implies
kx  yk D 0.

In the following we consider a number of properties of Hilbert spaces that prove

useful for the investigation of eigenvalue problems.

Definition 2.27 (Adjoint matrix). Let n; m 2 N and A 2 F nm . The matrix B 2

F mn given by

bij WD aNj i for all i 2 ¹1; : : : ; mº; j 2 ¹1; : : : ; nº

is called the adjoint of A and denoted by A .

Lemma 2.28 (Adjoint matrix). Let n; m 2 N and A 2 F nm . We have

hAx; yi D hx; A yi for all x 2 F m ; y 2 F n : (2.11)

Proof. Let B D A . We have

X
n X
n X
m X
m X
n
hAx; yi D .Ax/i yNi D aij xj yNi D xj bNj i yNi
iD1 i D1 j D1 j D1 i D1
Xm
D xj .By/j D hx; Byi D hx; A yi for all x 2 F m ; y 2 F n :
j D1

Applying (2.11) to x D ıi and y D ıj for i 2 ¹1; : : : ; mº and y 2 ¹1; : : : ; nº

shows that this equation actually defines A . It can be used to generalize the concept
of an adjoint to operators in general Hilbert spaces.

Lemma 2.29 (Null space and range). Let n; m 2 N and A 2 F nm . We have

hx; yi D 0 for all x 2 R.A/; y 2 N .A /: (2.12)

In particular, A x D 0 implies x D 0 for all x 2 R.A/.

Proof. Let x 2 R.A/ and y 2 N .A /. By definition, we can find z 2 F m such that
x D Az and obtain

hx; yi D hAz; yi D hz; A yi D hz; 0i D 0;

since y 2 N .A /.
Section 2.4 Some properties of Hilbert spaces 21

If now A x D 0 holds, we have x 2 N .A / and find

kxk2 D hx; xi D 0

by applying (2.12) to y D x.

Two vectors x; y 2 F n are called perpendicular, denoted by x ? y, if hx; yi D 0

holds. A vector x 2 F n is called perpendicular on a subspace W  F n , denoted
x ? W , if x and y are perpendicular for all y 2 W . Two subspaces V ; W  F n are
called perpendicular if all pairs .x; y/ 2 V  W are perpendicular. Lemma 2.29 states
that the range of A and the null space of A are perpendicular.

Proposition 2.30 (Spectrum of adjoint matrix). Let n 2 N and A 2 F nn . We have

.A / D ¹N W  2 .A/º:

Proof. Let  2 .A/, and let x 2 F n n ¹0º be a corresponding eigenvector. By

definition, we have .I  A/x D 0 and therefore
N /yi
0 D h.I  A/x; yi D hx; .I  A/ yi D hx; .A  I for all y 2 F n :

Since x is perpendicular on R.A  I N /, the rank-nullity theorem (2.2) implies

N .A  IN / ¤ ¹0º, and any non-zero element of N .A  I N / is an eigenvector
 N N 
of A for , so we have proven  2 .A /.
Due to A D A and NN D , the proof is complete.

Definition 2.31 (Self-adjoint matrix). Let n 2 N and A 2 F nn . If A D A holds,

A is called self-adjoint.

Lemma 2.32 (Identity of self-adjoint matrices). Let n 2 N, and let A 2 F nn be

self-adjoint. If

hAx; xi D 0 holds for all x 2 F n ;

we have A D 0.

Proof. Let x; y 2 F n . We have

0 D hA.x C y/; x C yi D hAx; xi C hAx; yi C hAy; xi C hAy; yi

D hAx; yi C hAy; xi D hAx; yi C hy; A xi D hAx; yi C hy; Axi:

Using y WD Ax, we find

0 D 2hAx; Axi D 2kAxk2 for all x 2 F n ;

and this implies A D 0.

22 Chapter 2 Existence and properties of eigenvalues and eigenvectors

Definition 2.33 (Normal matrix). Let n 2 N and A 2 F nn . If AA D A A holds,

A is called normal.

Exercise 2.34. Prove that the following matrices are normal:

0 1
    1 2 3
0 1 2 3
A1 WD ; A2 WD ; A3 WD @3 1 2A :
1 0 3 2
2 3 1

Lemma 2.35 (Metric equivalence). Let n 2 N and A 2 F nn . A is normal if and

only if

kAxk D kA xk holds for all x 2 F n : (2.13)

Proof. Assume that A is normal. For x 2 F n , we find

kAxk2 D hAx; Axi D hA Ax; xi D hAA x; xi D hA x; A xi D kA xk2 ;

and this implies (2.13).

Assume now (2.13). We find

0 D kAxk2  kA xk2 D hAx; Axi  hA x; A xi D hA Ax; xi  hAA x; xi

D h.A A  AA /x; xi for all x 2 F n :

Since A A  AA is obviously self-adjoint, we can apply Lemma 2.32 to obtain

A A D AA and can conclude that A is normal.

Proposition 2.36 (Eigenvectors). Let n 2 N, and let A 2 F nn be a normal matrix.

Let  2 .A/ be an eigenvalue, and let e 2 F n n ¹0º be a corresponding eigenvector.
Then we have
N
A e D e;
N
i.e., e is an eigenvector of the adjoint A for the eigenvalue .

In particular, A is diagonalizable if and only if A is, and both matrices can be
diagonalized by the same similarity transformation.

Proof. Since e is an eigenvector, we have Ae D e, and therefore

0 D kAe  ek D k.A  I /ek: (2.14)

Since A is normal, we have

N / D AA  A  A
.A  I /.A  I / D .A  I /.A  I N C jj2 I
N  A C jj2 I D .A  I
D A A  A N /.A  I /
D .A  I / .A  I /; (2.15)
Section 2.4 Some properties of Hilbert spaces 23

so I  A is also a normal matrix. Applying Lemma 2.35 to (2.14), we obtain

N /ek D kA e  ek
0 D k.A  I /ek D k.A  I / ek D k.A  I N

N
i.e., A e D e.

Applying general similarity transformations can cause numerical problems if the

transformation is ill-conditioned, e.g., if rounding errors lead to mixed eigenspaces.
We can avoid these problems by using unitary transformations, since these leave
lengths and angles of vectors unchanged and therefore lead to very stable algorithms.

Definition 2.37 (Isometric matrix). Let n; m 2 N, and let Q 2 F nm . If Q Q D I

holds, Q is called isometric. A square isometric matrix is called unitary.

A real unitary matrix is usually also called orthogonal. We always use the term
“unitary” in the interest of consistency.

Lemma 2.38 (Isometry). Let n; m 2 N, and let Q 2 F nm . Q is isometric if and

only if

kQxk D kxk holds for all x 2 F n : (2.16)

Proof. Assume that Q is isometric. For x 2 F n , we find

kQxk2 D hQx; Qxi D hQ Qx; xi D hx; xi D kxk2 ;

and this implies (2.16).

Assume now (2.16). We find

0 D kQxk2  kxk2 D hQx; Qxi  hx; xi D hQ Qx; xi  hx; xi

D h.Q Q  I /x; xi for all x 2 F n :

Since Q  Q  I is self-adjoint, we can apply Lemma 2.32 to obtain Q Q D I and

conclude that Q is isometric.

Proposition 2.39 (Unitary inverse). Let n 2 N and Q 2 F nm be isometric. Then

we have

QQ x D x for all x 2 R.Q/: (2.17)

If Q is square, i.e., unitary, Q is also unitary and we have Q D Q1 .

24 Chapter 2 Existence and properties of eigenvalues and eigenvectors

Proof. Let x 2 R.Q/. By definition, we can find y 2 F m such that x D Qy and

obtain
QQ x D QQ Qy D Qy D x:
Let now Q be square. Due to (2.16), Q is injective. Since Q is square, the rank-
nullity theorem (2.2) implies that Q is also surjective, i.e., R.Q/ D F n . (2.17) yields
QQ D I D Q Q.

We frequently use unitary similarity transformations, and Proposition 2.39 allows

us to express them using the readily available adjoint instead of the inverse.

2.5 Invariant subspaces

In general, we cannot diagonalize a matrix by unitary similarity transformations, but
we can at least transform it to triangular form. This is the Schur decomposition, a
useful tool that allows us to draw conclusions regarding the unitary diagonalizability
of matrices.

Definition 2.40 (Triangular matrix). Let n 2 N and L; R 2 F nn . The matrix R is

called upper triangular if

rij D 0 holds for all i; j 2 ¹1; : : : ; nº with j < i

and the matrix L is called lower triangular if

lij D 0 holds for all i; j 2 ¹1; : : : ; nº with i < j:

Consider an upper triangular matrix R 2 F nn . For j 2 ¹1; : : : ; nº, multiplying R

by the canonical unit vector ıj yields
´
rij if i  j;
.Rıj /i D rij D for all i 2 ¹1; : : : ; nº;
0 otherwise

i.e., we have
X
j
Rıj D rij ıi 2 span¹ı1 ; : : : ; ıj º:
i D1

Introducing the subspace

X WD span¹ı1 ; : : : ; ıj º;
we can write this property in the form

Rx 2 X for all x 2 X: (2.18)

Section 2.5 Invariant subspaces 25

Eigenspaces share the same structure: if A 2 F nn is a matrix and  2 .A/ is one
of its eigenvalues, we have

Ax D x 2 E.A; / for all x 2 E.A; /:

In this sense, subspaces with the property (2.18) are a generalization of eigenspaces.

Definition 2.41 (Invariant subspace). Let n 2 N and A 2 F nn . A subspace X  F n

is called invariant with respect to A if

Ax 2 X holds for all x 2 X: (2.19)

Exercise 2.42 (Invariant subspaces). Find at least five different invariant sub-
spaces for the matrix
0 1
2
B4 3 C
A WD B
@
C 2 C 44 :
7 8A
6

In practical algorithms, we can only work with a basis of a subspace X instead of

the subspace itself. Combining the basis vectors as in (2.7), we can represent the basis
by a matrix X such that X D R.X /.

Proposition 2.43 (Invariant subspace). Let n; m 2 N, let A 2 F nn and X 2 F nm .

The range X WD R.X / of X is an invariant subspace with respect to A if and only if

AX D Xƒ (2.20)

holds for a matrix ƒ 2 F mm .

Proof. Assume that X is invariant with respect to A. Let j 2 ¹1; : : : ; mº. Due to
X ıj 2 R.X / D X, we have

AXıj 2 X D R.X /;

so we can find yj 2 F m with AXıj D Xyj . We define

ƒij WD .yj /i for all i; j 2 ¹1; : : : ; mº

and conclude ƒıj D yj and therefore

AXıj D Xyj D Xƒıj for all j 2 ¹1; : : : ; mº:

This implies (2.20).

26 Chapter 2 Existence and properties of eigenvalues and eigenvectors

Assume now that (2.20) holds. For x 2 X D R.X /, we can find y 2 F m with
x D Xy and obtain

Ax D AXy D Xƒy 2 R.X / D X;

i.e., X is invariant with respect to A.

The equation (2.20) bears a close resemblance to the equation (2.3) defining eigen-
values and eigenvectors: the eigenvector x is replaced by the matrix X, and the eigen-
value  is replaced by the matrix ƒ.

Exercise 2.44 (Invariant subspaces and eigenvalues). Show that any eigenvalue
of the matrix ƒ in (2.20) is an eigenvalue of A if X is injective.
If  is an eigenvalue of A for an eigenvector x 2 F n with x 2 R.X /, is it also
an eigenvalue of ƒ?

2.6 Schur decomposition

We can use equation (2.20) to find a similarity transformation that moves A “closer”
to an upper triangular matrix: if X 2 F nm is injective and satisfies (2.20) with
ƒ 2 F mm , we can use the basis extension theorem (e.g., as in the proof of Proposi-
tion 2.19) to find an invertible matrix T 2 F nn with
 
T D X Y

for a matrix Y 2 F n.nm/ . (2.20) implies

     
I ƒ ƒ B
T 1 AX D T 1 Xƒ D m ƒ D ; T 1 AT D (2.21)
0 0 C

for some B 2 F m.nm/ and C 2 F .nm/.nm/ . If we can find suitable invariant

subspaces for ƒ and C , we can repeat the procedure until we reach upper triangular
matrices.
We can even ensure that the similarity transformations are unitary. To this end, we
introduce the sign function
´
z=jzj if z ¤ 0;
sgn W F ! F; z 7! (2.22)
1 otherwise:

and define a unitary transformation mapping an arbitrary vector to a multiple of the

first canonical unit vector:
Section 2.6 Schur decomposition 27

Lemma 2.45 (Householder reflection). Let n 2 N and q 2 F n n ¹0º. The matrix

P 2 F nn defined by
ww 
w WD q C sgn.q1 /kqkı1 ; P WD I  2
kwk2

(where we identify F with F 11 and F n with F n1 in the obvious way) is called an
elementary Householder reflection [21]. It is unitary and self-adjoint and satisfies

P q D  sgn.q1 /kqkı1 : (2.23)

Proof. P is obviously self-adjoint. Due to w  w D kwk2 and

  
 ww  ww  ww  ww  ww 
P P D I 2 I  2 D I  4 C 4 D I;
kwk2 kwk2 kwk2 kwk4
it is also unitary. With  WD sgn.q1 /, we have

hq; wi D hq; qi C hq;  kqkı1 i D kqk2 C kqk jq1 j;

kwk2 D hq C kqkı1 ; q C kqkı1 i
D hq; qi C kqkhq;
N ı1 i C kqkhı1 ; qi C kqk2 hı1 ; ı1 i
D 2.kqk2 C kqk jq1 j/;
hq; wi kqk2 C kqk jq1 j
P q D q  2w D q  2.q C  kqkı1 /
kwk2 2.kqk2 C kqk jq1 j/
D q  q  kqkı1 D kqkı1 ;

and this completes the proof.

By the fundamental theorem of algebra, any complex polynomial has at least one
zero. Applying this result to the characteristic polynomial yields that any complex ma-
trix has at least one eigenvalue, and we can use a Householder reflection to transform
the matrix to the “almost upper triangular” form (2.21). Repeating the process allows
us to reach an upper triangular matrix using only unitary similarity transformations.

Theorem 2.46 (Schur decomposition). Let n 2 N and A 2 C nn . There are a

unitary matrix Q 2 C nn and an upper triangular matrix R 2 C nn satisfying

Q AQ D R: (2.24)

Proof. By induction on n 2 N. The case n D 1 is trivial.

Let now n 2 N and assume that any A 2 C nn is unitarily similar to an upper
triangular matrix. Let A 2 C .nC1/.nC1/ . Due to the fundamental theorem of algebra,
we can find  2 C with pA ./ D 0. Since therefore I  A is not injective, we can
find a vector q 2 N .I  A/ n ¹0º with kqk D 1. Obviously, q is an eigenvector for
28 Chapter 2 Existence and properties of eigenvalues and eigenvectors

the eigenvalue . Let P 2 C .nC1/.nC1/ be the elementary Householder reflection

(cf. Lemma 2.45) satisfying

P q D  sgn.q1 /ı1 :

Due to P D P  and 1= sgn.q1 / D sgn.qN 1 /, we also have P ı1 D  sgn.qN 1 /q and

obtain

P  AP ı1 D  sgn.qN1 /P  Aq D  sgn.qN 1 /P  q D  sgn.qN 1 /P q D ı1 :

This means that we can find b

A 2 C nn and B 2 C 1n satisfying
 
  B
P AP D b :
A

Since b
A is only an n  n matrix, we can apply the induction assumption and find a
b 2 C nn with
b 2 C nn and an upper triangular matrix R
unitary matrix Q

b b
Q AQb D R:
b

Now we let
  !
1  BQb
Q WD P b ; R WD
Q b
R

and observe
!   !  
 1  1 1  B 1
Q AQ D b b D
P AP
b 
b b
Q Q Q A Q
! !
 BQb  BQb
D D D R:
b b
Q AQb b
R

Since R is upper triangular, this completes the induction.

Theorem 2.46 relies on the fundamental theorem of algebra that holds in C, but
not in R. Since most of the following results are corollaries, they also hold only
in the field of complex numbers.

Remark 2.47 (Order of eigenvalues). In the proof of Theorem 2.46, we can put any
eigenvalue  2 .A/ into the upper left entry of R, and by extension we can choose
any order for the eigenvalues on the diagonal of R.
Section 2.6 Schur decomposition 29

Exercise 2.48 (Trace). The trace of a matrix A 2 F nn is defined by

X
n
tr.A/ WD ai i ;
i D1

i.e., by the sum of diagonal entries. Let Q 2 F nn be unitary. Prove

tr.Q AQ/ D tr.A/

and conclude X
tr.A/ D a .A; /;
2.A/

i.e., that the trace is invariant under similarity

P transformations.
Hint: Start by proving tr.Q AQ/ D ni;j;kD1 qNj i qki aj k and considering the
first two factors.

While the upper triangular form is already useful, e.g., for finding all eigenvalues
and the corresponding algebraic multiplicities, we are mainly interested in diagonal-
izing a matrix. The following Lemma suggests a possible approach:

Lemma 2.49 (Normal triangular matrix). Let R 2 F nn be upper triangular. If R is

normal, it is a diagonal matrix.

Proof. By induction on n 2 N. The case n D 1 is trivial.

Let now n 2 N and assume that any normal upper triangular matrix R 2 F nn is
diagonal. Let R 2 F .nC1/.nC1/ be normal and upper triangular. Due to Lemma 2.35,
we have
jr11 j2 D kRı1 k2 D kR ı1 k2 D jr11 j2 C    C jr1;nC1 j2
and conclude jr12 j D    D jr1;nC1 j D 0. This means that we can find an upper
b 2 F nn with
triangular matrix R
 
r11
RD b :
R

Due to ! !
jr11 j2 jr11 j2
bRb D RR D R R D b ;
b R
R R
b is normal, and we can use the induction assumption to complete the
the matrix R
proof.
30 Chapter 2 Existence and properties of eigenvalues and eigenvectors

Corollary 2.50 (Normal matrix). Let n 2 N, and let A 2 C nn . A is normal if and
only if it is unitarily diagonalizable, i.e., if there are a unitary matrix Q 2 C nn and
a diagonal matrix D 2 C nn with

Q AQ D D: (2.25)

Proof. Assume that A is normal. Due to Theorem 2.46, we can find a unitary matrix
Q 2 C nn and an upper triangular matrix R 2 C nn with

A D QRQ :

Since A is a normal matrix, we have

RR D Q AQQ A Q D Q AA Q D Q A AQ D Q A QQ AQ D R R;

so R is also normal, and Lemma 2.49 yields that R is, in fact, diagonal.
Assume now that we can find a unitary matrix Q 2 C nn and a diagonal matrix
D 2 C nn satisfying (2.25). Then we have

AA D QDQ QD  Q D QDD  Q D QD  DQ D QD  Q QDQ D A A

and conclude that A is normal.

Corollary 2.51 (Self-adjoint matrix). Let n 2 N, and let A 2 C nn . A is self-

adjoint if and only if there are a unitary matrix Q 2 C nn and a real diagonal matrix
D 2 Rnn with
Q AQ D D: (2.26)

Proof. Assume that A is self-adjoint. Then it is also normal, and Corollary 2.50 gives
us a unitary matrix Q 2 C nn and a diagonal matrix D 2 C nn with A D QDQ .
Since A is self-adjoint, we obtain

D D Q  AQ D Q A Q D .Q AQ/ D D  ;

so all diagonal entries of D have to be real.

Assume now that we can find a unitary matrix Q 2 C nn and a real diagonal matrix
D 2 C nn satisfying (2.26). Then we have

A D QDQ D QD  Q D A

and conclude that A is self-adjoint.

Our proofs so far depend on the fundamental theorem of algebra and therefore apply
only to complex matrices. Even if A is self-adjoint and real, Corollary 2.51 still may
yield a complex matrix Q. By slightly modifying the proof of Theorem 2.46, we can
improve the result:
Section 2.6 Schur decomposition 31

Corollary 2.52 (Real self-adjoint matrix). Let n 2 N and A 2 Rnn . A is self-

adjoint if and only if there are a unitary matrix Q 2 Rnn and a diagonal matrix
D 2 Rnn with
Q AQ D D: (2.27)

Proof. Assume that we can find a unitary matrix Q 2 Rnn and a diagonal matrix
D 2 Rnn satisfying (2.27). Then we have

A D QDQ D QD  Q D A ;

so A has to be self-adjoint.
Assume now that A is self-adjoint. According to Corollary 2.51, we can find an
eigenvalue  2 R of A, and due to N .I  A/ ¤ ¹0º also a real eigenvector q 2 Rn
with kqk D 1. Using this vector, we can proceed by induction as in the proof of
Theorem 2.46.

According to Theorem 2.46, any complex-valued matrix can be transformed to tri-

angular form by a unitary transformation. We now investigate whether it is possible
to get close to diagonal form by using further unitary transformations.

Definition 2.53 (Frobenius norm). Let n; m 2 N and A 2 F nm . The Frobenius

norm of A is given by
0 11=2
Xn X
m
kAkF WD @ jaij j2 A :
i D1 j D1

Proposition 2.54 (Frobenius norm). Let n; m 2 N and A 2 F nm . We have

 kAkF D kA kF ,
 kAkF D kQAkF for all unitary matrices Q 2 F nn ,
 kAkF D kAQkF for all unitary matrices Q 2 F mm .

Proof. Let B WD A . Definition 2.53 implies

X
n X
m X
m X
n
kAk2F D jaij j2 D jbj i j2 D kBk2F D kA k2F :
i D1 j D1 j D1 i D1
32 Chapter 2 Existence and properties of eigenvalues and eigenvectors

Let Q 2 F nn be a unitary matrix. Using the canonical unit vectors .ıj /jmD1 in F m
defined by (2.5), we obtain
X
m X
n X
m X
n X
m
kAk2F D jaij j2 D j.Aıj /i j2 D kAıj k2
j D1 i D1 j D1 i D1 j D1
X m
D kQAıj k2 D kQAk2F
j D1

by Lemma 2.38. Applying this result to A completes the proof.

We measure the “distance” to diagonal form by taking the Frobenius norm of all
entries except for the diagonal.

Definition 2.55 (Off-diagonal part). Let n 2 N and A 2 F nn . We introduce the

quantity
0 11=2
BXn X
n
C
off.A/ WD B
@ jaij j2 C
A
i D1 j D1
j ¤i

describing the Frobenius norm of the off-diagonal part of A.

We have off.A/ D 0 if and only if A is a diagonal matrix.

We compare two unitary transformations of a matrix A to triangular form.

Proposition 2.56 (Invariant). Let n 2 N and A 2 C nn . Let Q1 ; Q2 2 C nn be

unitary matrices and let R1 ; R2 2 C nn be upper triangular matrices satisfying

Q1 R1 Q1 D A D Q2 R2 Q2 :

Then we have off.R1 / D off.R2 /.

Proof. (cf. [18, Section 7.1]) Since A, R1 and R2 are similar matrices, Proposition
2.18 yields
Y
n Y
n
.r1;i i  / D pR1 ./ D pA ./ D pR2 ./ D .r2;jj  / for all  2 C;
iD1 j D1

where r1;i i and r2;jj denote the i -th and j -th diagonal elements of R1 and R2 , re-
spectively. Two factorizations of pA into linear factors can only differ by the ordering
of the factors, and we obtain
X
n X
n
jr1;i i j2 D jr2;jj j2 :
i D1 j D1
Section 2.6 Non-unitary transformations 33

Proposition 2.54 yields

X
n X
n X
n X
n
2
off.R1 / D 2
jr1;ij j D kR1 k2F  2
jr1;i i j D kQ1 AQ1 k2F  jr1;i i j2
iD1 j D1 i D1 i D1
j ¤i
X
n X
n
D kAk2F  jr1;i i j2 D kQ2 AQ2 k2F  jr2;jj j2
i D1 j D1
Xn
D kR2 k2F  jr2;jj j2 D off.R2 /2 ;
j D1

and the proof is complete.

Let A 2 C nn , and let Q; R 2 C nn be as in Theorem 2.46. Corollary 2.50

implies that A is normal if and only if R is diagonal, and this is the case if and only if

.A/ WD off.R/

equals zero. In this sense, .A/ measures the departure from normality of A, and due
to Proposition 2.56, it does not depend on the particular Schur decomposition.

2.7 Non-unitary transformations 

Let A 2 C nn . Theorem 2.46 implies that we can find a unitary transformation
Q 2 C nn such that
R D Q AQ
is upper triangular, and Proposition 2.56 implies that we cannot get “closer” to diag-
onal form using unitary transformations.
Therefore we now consider non-unitary transformations. We assume that an upper
triangular matrix R 2 C nn is given and split it into
 
R11 R12
RD (2.28)
R22

with R11 2 C kk , R12 2 C k` and R22 2 C `` with n D k C `. We consider the
similarity transformation
   
I X 1 I X
T WD ; T D (2.29)
I I

for a matrix X 2 C k` and obtain

     
1 I X R11 R12 I X R11 R12 C R11 X  XR22
T RT D D :
I R22 I R22
34 Chapter 2 Existence and properties of eigenvalues and eigenvectors

In order to eliminate the upper-right block, we have to find X 2 C k` solving

Sylvester’s equation
R11 X  XR22 D R12 :
This is a linear matrix equation.

Proposition 2.57 (Sylvester’s equation). Let n; m 2 N, A 2 C nn and B 2 C mm .

Sylvester’s equation
AX  XB D C (2.30)
has unique solutions X 2 C nm for all C 2 C nm if and only if .A/ \ .B/ D ;,
i.e., if A and B have no eigenvalue in common.

Proof. We consider the linear mapping

S W C nm ! C nm ; X 7! AX  XB:

If it is injective, the rank-nullity theorem implies that it is also bijective, and there-
fore equation (2.30) is uniquely solvable for all C 2 C nm .
We will prove that .A/ \ .B/ ¤ ; is equivalent to S being not injective.
Let first .A/\.B/ ¤ ;. We pick  2 .A/\.B/ and eigenvectors x 2 C n n¹0º
and y 2 C m n ¹0º satisfying

Ax D x; N
B  y D y:

The latter is possible due to Proposition 2.30. Now we let X WD xy  and find

S ŒX  D AX  XB D Axy   xy  B D .Ax/y   x.B  y/

N  D xy   xy  D 0;
D .x/y   x.y/

i.e., the nullspace of S contains the non-zero matrix X , therefore S cannot be injec-
tive.
Let now S be not injective. Then we can find X 2 C nm n ¹0º with S ŒX  D 0, i.e.,

AX D XB: (2.31)

We first consider a special case: assume that B is upper triangular. Let j 2 ¹1; : : : ; mº
be the smallest index satisfying x WD X ıj ¤ 0, i.e., the number of the first non-zero
column of X. Then we have
X
j X
j
Ax D AXıj D XBıj D X bij ıi D bij X ıi :
i D1 i D1

Since X ıi D 0 for all i < j , this yields

Ax D bjj X ıj D bjj x;
Section 2.7 Non-unitary transformations 35

i.e., x is an eigenvector of A for the eigenvalue bjj . Since B is upper triangular, bjj is
also an eigenvalue of B, i.e., bjj 2 .A/ \ .B/.
Now we return to the general case. According to Theorem 2.46, we can find a Schur
decomposition of B, i.e., a unitary matrix Q 2 C mm and an upper triangular matrix
b such that
B
B D QBQb 
holds. (2.31) yields
b ;
AX D XB D XQBQ b
AXQ D XQB;

b WD XQ, we find
due to Q Q D I , and introducing X
bDX
AX b B:
b

b ¤ ;, and since B and

We can apply the first step of the proof to show .A/ \ .B/
b
B are similar, Proposition 2.18 yields .A/ \ .B/ ¤ ;.

If we can ensure .R11 / \ .R22 / D ; in (2.28), Proposition 2.57 implies that the
transformation T introduced in (2.29) can eliminate R12 . Repeating this procedure
allows us to transform any square complex matrix to block-diagonal form:

Theorem 2.58 (Block-diagonalization). Let A 2 C nn . Let .A/ D ¹1 ; : : : ; k º

with k WD #.A/. For each ` 2 ¹1; : : : ; kº there are an integer n` 2 N and an upper
triangular matrix R` 2 Rn` n` such that
0 1
R1
B C
B 1 AB D @ ::
: A (2.32)
Rk

holds for an invertible matrix B 2 C nn . For each ` 2 ¹1; : : : ; kº, we have .R` / D
¹` º and n` D a .A; ` /.

Proof. (cf. [18, Theorem 7.1.6]) By induction on the cardinality #.A/ of the spec-
trum. If #.A/ D 1, the Schur decomposition of Theorem 2.46 yields the desired
factorization with B WD Q and R1 WD R.
Assume now we have a k 2 N such that the desired factorization exists for all
A 2 C nn satisfying #.A/ D k.
Let A 2 C nn be given with #.A/ D k C 1 and .A/ D ¹1 ; : : : ; kC1 º. Due to
Theorem 2.46, we can find a unitary matrix Q 2 C nn and an upper triangular matrix
R 2 C nn satisfying
Q AQ D R:
We define

n1 WD #¹i 2 ¹1; : : : ; nº W ri i D 1 º; nO WD n  n1 :
36 Chapter 2 Existence and properties of eigenvalues and eigenvectors

According to Remark 2.47, we can ensure

ri i D 1 for all i  n1 ;
ri i ¤ 1 for all i > n1 :

b 2 Rn
We split R into R1 2 Rn1 n1 , R O nO and Y 2 Rn1 nO such that

 
R1 Y
RD b :
R

By construction, we have .R1 / D ¹1 º and 1 62 .R/. b Due to Proposition 2.57,

we can find X 2 R n1 .nn 1 / satisfying Sylvester’s equation

b D Y:
R1 X  X R

We define T 2 Rnn as in (2.29) and obtain

 
1 R1
T RT D b :
R

Due to 1 62 .R/ b  .A/, we have #.R/

b D k and can apply the induction
b 2 C n
assumption to find an invertible matrix B O nO and upper triangular matrices

R2 ; : : : ; RkC1 such that

0 1
R2
B bB
b1 R bDB
@ ::
:
C
A:
RkC1

We let  
I
B WD QT b
B
and obtain
       
1 I 1  I I R1 I
B AB D b1 T Q AQT b D b1 b b
B B B R B
0 1
R1
  B C
R1 B R2 C
D b1 R
bBb D B :: C:
B @ : A
RkC1

This completes the induction.

Section 2.7 Non-unitary transformations 37

Applying Proposition 2.18 to the representation (2.32) and taking advantage of the
fact that the determinant of a block diagonal matrix is the product of the determinants
of the diagonal blocks yields
Y
k Y
k
pA ./ D pR` ./ D .  ` /n` for all  2 C;
`D1 `D1

and therefore a .A; ` / D n` .

Even if a matrix is not diagonalizable, this result allows us to transform it to block-
diagonal structure with one block corresponding to each eigenvalue. If we split the
transformation B into
 
B D B1 : : : Bk ; B` 2 C nn` for all ` 2 ¹1; : : : ; kº;
the representation (2.32) takes the form
AB` D B` R` for all ` 2 ¹1; : : : ; kº; (2.33)
i.e., the columns of each B` are a basis of an invariant subspace.
Let ` 2 ¹1; : : : ; kº. Since ` is the only eigenvalue of the upper triangular matrix
R` , all diagonal elements are equal to ` , and therefore all diagonal elements of ` I 
R` are equal to zero, i.e., the matrix is strictly upper triangular. We let bj WD B` ıj
for all j 2 ¹1; : : : ; n` º. Since ` I  R` is strictly upper triangular, we have
.` I  A/bj D .` I  A/B` ıj D B` .` I  R` /ıj
1
jX 1
jX
D B` r`;i;j ıi D r`;i;j bi for all j 2 ¹1; : : : ; n` º:
i D1 i D1
In particular, this implies
.` I  A/b1 D 0;
.` I  A/bj 2 span¹b1 ; : : : ; bj 1 º for all j 2 ¹2; : : : ; n` º:
A simple induction yields
.` I  A/j bj D 0 for all j 2 ¹1; : : : ; n` º;
so b1 is an eigenvector, while bj for j
2 can be considered as a generalized eigen-
vector.
Definition 2.59 (Generalized eigenvector). Let A 2 C nn ,  2 .A/ and x 2 C n n
¹0º. If there is a number k 2 N with

.I  A/k x D 0; (2.34)

the vector x is called a generalized eigenvector of A for the eigenvalue . The smallest
integer k 2 N satisfying (2.34) is called the order of the generalized eigenvector.
38 Chapter 2 Existence and properties of eigenvalues and eigenvectors

By definition, if x is a generalized eigenvector of order k > 1 for a matrix A and

an eigenvalue , y WD .I  A/x has to be a generalized eigenvector of order k  1,
since
.I  A/k1 y D .I  A/k x D 0
and y ¤ 0 due to the minimality of k.
Theorem 2.58 states that for any complex matrix A 2 C nn , we can find a basis of
C consisting of generalized eigenvectors. This basis can be used to derive the even
n

more specialized Jordan normal form of the matrix A. For our purposes, however, the
block-diagonal decomposition of Theorem 2.58 is sufficient.
Chapter 3

Jacobi iteration

Summary
In this chapter, we consider the simple, but reliable, Jacobi iteration. Given a self-
adjoint matrix A 2 F nn , it performs a sequence of unitary similarity transformations
that eliminate off-diagonal entries and thus moves the matrix closer to diagonal form.
The Jacobi iteration may not be the fastest method, but it is guaranteed to converge
for any self-adjoint matrix. The optional Section 3.5 proves that the Jacobi iteration
will converge quadratically if a matrix is already close to diagonal form.

Learning targets
 Compute the two-dimensional Schur decomposition in a numerically stable way.
 Construct the elementary Jacobi step that eliminates a sub-diagonal entry of a
matrix (and, due to symmetry, a corresponding super-diagonal entry as well).
 Use these elementary steps to construct a globally convergent iteration.
 Derive a posteriori bounds for the iteration error that allow us to judge the accu-
racy of each iteration step.

3.1 Iterated similarity transformations

The basic idea of the Jacobi iteration, and a number of other important iterative tech-
niques for finding Schur decompositions, is to construct a sequence

A0 D A; AmC1 D QmC1 Am QmC1 for all m 2 N0

of matrices .Am /1mD0 by applying unitary similarity transformations .Qm /mD1 . If

1

these transformations are chosen appropriately, the sequence A0 ; A1 ; A2 ; : : : will con-

verge to a diagonal matrix.
Accumulating the transformations in a sequence
b 0 D I;
Q b mC1 D Q
Q b m QmC1 for all m 2 N0

yields
b m AQ
Am D Q bm for all m 2 N0 ;
40 Chapter 3 Jacobi iteration

and if Am approximates a diagonal matrix D, we obtain

b mDQ
Q b m  Q b m D A;
b m Am Q

i.e., an approximation of the Schur decomposition.

This approach has the significant advantage that the convergence behaviour of the
sequence A0 ; A1 ; A2 ; : : : can be controlled explicitly. Sophisticated iterations could,
e.g., take advantage of submatrices that have already converged.

3.2 Two-dimensional Schur decomposition

Like many other numerical algorithms, the Jacobi iteration [22] replaces a compli-
cated problem, in this case finding the Schur decomposition, by a sequence of simple
problems. In the case of the Jacobi iteration, the simple problems consist of finding
the Schur decomposition of 2  2 matrices. This decomposition can then be used to
define a transformation of the original matrix.
In this section, we focus on the self-adjoint matrix
 
a b
AD N 2 F 22 ; a; d 2 R; b 2 F:
b d

Due to Corollary 2.51, we know that there is a unitary transformation Q 2 F 22

diagonalizing the matrix A.
Following [18, Section 8.4.2], we consider unitary matrices of the form
 
c sN
QD ; c; s 2 F : (3.1)
s cN
A matrix of this form is unitary if and only if
    2 
cN Ns c sN jcj C jsj2 cN sN  sN cN
I D Q Q D D
s c s cN sc  cs jsj2 C jcj2

holds, i.e., we have to use c; s 2 F with jcj2 C jsj2 D 1. In order to diagonalize A,

we have to find Q such that
   
 cN Ns a b c sN
B WD Q AQ D
s c bN d s cN
  
cN Ns ac  bs aNs C b cN
D N  ds bN sN C d cN
s c bc
 
N
c.ac N  ds/ c.aN
 bs/  sN .bc N s C b c/ N  sN .bN sN C d c/N
D N  ds/ s.aNs C b c/
s.ac  bs/ C c.bc N C c.bN sN C d c/ N
 
ajcj2 C d jsj2  .b csN C bcN sN / .a  d /cN sN C b cN 2  bN sN 2
D N 2 N sN /
.a  d /cs  bs 2 C bc ajsj2 C d jcj2 C .b cs N C bc
Section 3.2 Two-dimensional Schur decomposition 41

is a diagonal matrix, i.e., that

N 2
0 D b21 D .a  d /cs  bs 2 C bc (3.2)

holds. If we have b D 0, we can let c D 1 and s D 0 and are done. Otherwise, we

notice that, at least for c; s 2 R, the matrix Q defined in (3.1) can be considered as a
clockwise rotation with the angle ˛ given by c D cos ˛ and s D sin ˛. The tangent is
given by
sin ˛
t WD s=c D D tan ˛
cos ˛
and can be used to simplify (3.2): with s D t c, we obtain
N 2 D ..a  d /t  bt 2 C b/c
0 D .a  d /t c 2  bt 2 c 2 C bc N 2

and can eliminate c 2 to get the quadratic equation

N
0 D bt 2  .a  d /t  b: (3.3)

Multiplying by b, we obtain
 
2 ad 2 a  d 2 .a  d /2
0 D .bt /  2 .bt /  jbj D bt    jbj2 :
2 2 4
Taking the square root yields
s sˇ ˇ
ad .a  d /2 ad jbj ˇˇ a  d ˇˇ2
bt D ˙ C jbj2 ; tD ˙ C 1:
2 4 2b b ˇ 2b ˇ

We introduce
ad
 WD ;
2b
recall (cf. 2.22) that the sign of a complex number is denoted by
´
z=jzj if z ¤ 0;
sgn.z/ WD for all z 2 F;
1 otherwise

and arrive at q
1
t D˙ j j2 C 1: (3.4)
sgn.b/
Using t , we can reconstruct c and s by
1
1 D jcj2 C jsj2 D jcj2 C jt cj2 D .1 C jt j2 /jcj2 ; cDp ; s D t c:
1 C jt j2
(3.5)
42 Chapter 3 Jacobi iteration

We still have to choose the sign of the square root in (3.4). In order to obtain proper
convergence of the iterative method, we have to ensure that the next iterate B D
Q AQ is close to A. For the rotation Q given in (3.1), this means that the sine s
should be as small as possible, i.e., jt j should be as small as possible: we have to
choose the zero that is smaller in modulus. It is given by
q q
sgn.a  d /
t D j j2 C 1 D   sgn. / j j2 C 1;
sgn.b/
but computing it this way can lead to problems when using floating-point arithmetic.
Computing the zero q
tO WD  C sgn. / j j2 C 1

that is larger in modulus, on the other hand, is stable. By construction, t and tO are the
zeros of the monic polynomial
N
p.z/ WD z 2  2 z  b=b for all z 2 F;

i.e., we have
N D p.z/ D .z  t /.z  tO/ D z 2  .t C tO/z C t tO
z 2  2 z  b=b for all z 2 F;

and comparing terms yields

N
b=b
N
t tO D b=b; tD :
tO
N D sgn. / sgn.b/= sgn.b/
Using sgn.N / D sgn.a  d /= sgn.b/ N allows us to find a
more convenient expression:
N sgn.b/
sgn.b/= 1
tD D p
Ot sgn.b/. C sgn. / j j2 C 1/= sgn.b/ N
1  
D p D p D p
N C sgn.N / j j2 C 1  N C  sgn.N / j j2 C 1 j j2 C j j j j2 C 1
sgn. /
D p : (3.6)
j j C j j2 C 1
The last term offers a numerically stable way of computing t .

Exercise 3.1 (Singular value decomposition). Let A 2 F 22 be an arbitrary

matrix. Find unitary matrices U; V 2 F 22 such that † WD U  AV is diagonal
with real and non-negative diagonal entries.
If the diagonal entries of † are in descending order by magnitude, we call
A D U †V  the singular value decomposition of A.
Section 3.3 One step of the iteration 43

3.3 One step of the iteration

Let now A 2 F nn be a self-adjoint matrix. In order to apply the result of the previous
section, we choose p; q 2 ¹1; : : : ; nº with p < q and consider the 2  2 submatrix
 
b app apq
A WD :
aqp aqq

We have seen that we can find c; s 2 F such that

 
b WD c N
s
Q
s cN

b b
is unitary and Q AQb is diagonal. Applying the transformation Q
b to the p-th and q-th
row of an n-dimensional vector is a unitary transformation in F nn corresponding to
the matrix 0 1
Ip1
B c sN C
B C
Q WD B B Iqp1 C; (3.7)
C
@ s cN A
Inq
where Ik 2 F kk again denotes the k-dimensional identity matrix. We compute
the transformation B WD Q AQ in two steps: multiplication of A by Q yields the
intermediate result
0 1
: : : a1p c  a1q s : : : a1p sN C a1q cN : : :
B :: :: C
M WD AQ D @ : : A; (3.8)
: : : anp c  anq s : : : anp sN C anq cN : : :

where only the p-th and q-th column differ from the ones of the original matrix A.
In the second step, we compute B D Q M . Multiplication by Q changes only the
p-th and q-th row:
0 1
:: ::
B : : C
B cm
N  N
s m : : : N
cm  sN mq n C
B p1 q1 pn C
B :: :: C
B D Q M D B : : C: (3.9)
B C
Bsmp1 C cmq1 : : : smpn C cmq n C
@ A
:: ::
: :

We can easily verify

   
c WD mpp mpq D b
M b
AQ; b WD bpp bpq D Q
B cDQ
b M b b b
AQ; (3.10)
mqp mqq bqp bqq
44 Chapter 3 Jacobi iteration

procedure jacobi_step(p, q, var A);

begin
if apq D 0 then
t D0
else begin
 .app  aqq /=.2a p pq /;
t sgn. /=.j j C j j2 C 1/;
end; p
c 1= jt j2 C 1; s t c;
for i 2 ¹1; : : : ; nº do begin
h aip ; aip hc  aiq s; aiq hNs C aiq cN
end;
for j 2 ¹1; : : : ; nº do begin
h apj ; apj N  sN aqj ; aqj
ch sh C caqj
end
end
Figure 3.1. One step of the Jacobi iteration eliminates the entries apq and aqp .

b b
and since Q AQb is diagonal by construction, we have bqp D bpq D 0. The resulting
algorithm is summarized in Figure 3.1, A is overwritten with B D Q AQ.
Of course, it is not clear that eliminating one entry of A will get us closer to a
diagonal matrix: it could just move entries around without actually reducing the off-
diagonal part. Fortunately, the elimination step works perfectly:

Theorem 3.2 (Jacobi step). Let n 2 N, and let A 2 F nn be self-adjoint. Let p; q 2
¹1; : : : ; nº with p < q. For the Jacobi matrix Q 2 F nn defined by (3.7), we have

off.B/2 D off.A/2  2japq j2 ; B D Q AQ:

Proof. (cf. [18, eq. (8.4.2)]) The transformation Q can change only the p-th and q-th
row and column. In particular, this implies

ai i D bi i for all i 2 ¹1; : : : ; nº; i ¤ p; i ¤ q:

bDQ
Due to B b b
AQb and Proposition 2.54, we have

japp j2 C 2japq j2 C jaqq j2 D kb b 2 D jbpp j2 C jbqq j2 ;

Ak2F D kBkF
Section 3.3 One step of the iteration 45

since bqp D bpq D 0 and aqp D aNpq . Using Proposition 2.54 again, we obtain

X
n X
n
off.B/2 D kBk2F  jbi i j2 D kBk2F  jbpp j2  jbqq j2  jbi i j2
i D1 i D1
i 62¹p;qº
X
n
D kAk2F  japp j2  2japq j2  jaqq j2  jai i j2
i D1
i 62¹p;qº
X
n
D kAk2F  jai i j2  2japq j2 D off.A/2  2japq j2 :
i D1

Exercise 3.3 (Singular value decomposition). Let A 2 F nn , and let p; q 2

¹1; : : : ; nº be given with p < q. Find unitary matrices U; V 2 F nn of similar
form as (3.7) such that

off.U  AV /2 D off.A/2  japq j2  jaqp j2 :

As long as we choose non-zero entries apq , each Jacobi step reduces the norm of
the off-diagonal part of the matrix. If we choose the maximal entries, we can obtain a
simple linear convergence estimate:

Corollary 3.4 (Convergence). Let n 2 N and let A 2 F nn be self-adjoint. Let

p; q 2 ¹1; : : : ; nº with p < q and

jaij j  japq j for all i; j 2 ¹1; : : : ; nº:

For the Jacobi matrix Q 2 F nn defined by (3.7), we have

 
2
2
off.B/  1  off.A/2 ; B D Q AQ:
n.n  1/
Proof. (cf. [18, Section 8.4.3]) We have

X
n X
n
off.A/2 D jaij j2  japq j2 D japq j2 n.n  1/;
i;j D1 i;j D1
i ¤j i ¤j

so Theorem 3.2 yields

2
off.B/2  off.A/2  2japq j2  off.A/2  off.A/2
n.n  1/
 
2
D 1 off.A/2 :
n.n  1/
46 Chapter 3 Jacobi iteration

procedure jacobi_classical( , var A);

begin
Choose 1  p < q  n with japq j D max¹jaij j W i; j 2 ¹1; : : : ; nºº;
while n.n  1/japq j2 > 2 do begin
jacobi_step(p, q, A);
Choose 1  p < q  n with japq j D max¹jaij j W i; j 2 ¹1; : : : ; nºº
end
end
Figure 3.2. Classical Jacobi iteration.

These results give rise to the classical Jacobi iteration: in each step, we choose
the off-diagonal entry of maximal modulus and eliminate it, and we repeat until the
off-diagonal entries are sufficiently small. Using off.A/2  n.n  1/japq j2 , we can
compute an upper bound for off.A/ and use it as a stopping criterion. The resulting
algorithm is summarized in Figure 3.2.
An important advantage of the classical Jacobi iteration, as compared to other
eigenvalue algorithms, is its guaranteed rate of convergence. Unfortunately, this ad-
vantage comes at a high cost: finding the maximal japq j directly requires checking
all n.n  1/=2 superdiagonal entries, and even for moderately large values of n, this
task dominates the computational cost. It is possible to reduce the complexity by
using more sophisticated algorithms, but in practice it is more convenient to simply
replace the search for the maximum by a strategy that cycles through all off-diagonal
entries. An example is the cyclic-by-row Jacobi iteration given in Figure 3.3 that cy-
cles through all rows and within each row through all columns. For a 5  5 matrix,
the elements are eliminated in the following pattern:
0 1
 1 2 3 4
B10  5 6
B 0 0 7C C
B2 5  8
B 9C C:
@30 60 80  10A
40 70 90 100 

The entry m 2 ¹1; : : : ; 10º is eliminated in the m-th elementary step together with the
entry m0 . Diagonal elements marked with  are not eliminated.
The elementary Jacobi step given in Figure 3.1 requires 13 arithmetic operations to
compute c and s and 12n operations to update A. The complexity of the overall Jacobi
algorithm, both in the classical version given in Figure 3.2 and the cyclic version
given in Figure 3.3 is usually measured in terms of one “sweep”, i.e., a sequence of
n.n  1/=2 elementary Jacobi steps, since exactly this number is required in the cyclic
algorithm to eliminate all off-diagonal elements once. One sweep of the classical
Section 3.4 Error estimates 47

procedure jacobi_cyclic( , var A);

begin

0;
for i; j 2 ¹1; : : : ; nº; i < j do

C jaij j2 ;
2
while 2
> do begin
for p D 1 to n  1 do
for q D p C 1 to n do
jacobi_step(p, q, A);

0;
for i; j 2 ¹1; : : : ; nº; i < j do

C jaij j2
end
end
Figure 3.3. Cyclic-by-row Jacobi iteration.

Jacobi algorithm requires

n.n  1/
.12n C 13/  6n3 C 13n2 =2
2
operations for the elementary steps and
n.n  1/ n.n  1/
 n4 =4
2 2
operations for finding the maximal elements in each step. One sweep of the cyclic
Jacobi algorithm, including the computation of
required for the stopping criterion,
requires
n.n  1/ n.n  1/
.12n C 13/ C 2  6n3 C 15n2 =2
2 2
operations, i.e., it can be significantly more efficient than the classical algorithm.

3.4 Error estimates

We have seen that the matrices constructed by the Jacobi iteration converge to di-
agonal form, so we can expect the diagonal entries of these matrices to converge to
eigenvalues. In this section, we provide a simple bound for the speed of convergence.
The Frobenius norm introduced in Definition 2.53 is not particularly well-suited for
our purposes, so we introduce a second norm in the space of matrices:

Definition 3.5 (Spectral norm). Let A 2 F nm .

kAk WD max¹kAxk W x 2 F m ; kxk D 1º

is called the spectral norm of A.

48 Chapter 3 Jacobi iteration

Although we use the same notation for the norm of a vector and the spectral norm
of a matrix, the meaning is always clear, since the first only applies to vectors and the
second only to matrices.

The spectral norm is compatible with the norm k  k in F m : for x 2 F m n ¹0º, the
vector x=kxk has unit norm, and we find
 
 x 
kAxk D  
A kxk  kxk  kAk kxk:

This estimate obviously also holds for x D 0, so we have proven

kAxk  kAk kxk for all x 2 F m : (3.11)

Exercise 3.6 (Spectral norm). Let A 2 F nm . Prove

² ³
jhAx; yij
kAk D max W x 2 F m n ¹0º; y 2 F n n ¹0º :
kxk kyk

Show that this implies kA k D kAk and kA Ak D kAk2 D kAA k.

The Frobenius norm and the spectral norm are related:

Proposition 3.7 (Norm estimates). Let A 2 F nm . We have

p
kAk  kAkF  mkAk:

Proof. As in the proof of Proposition 2.54, we write the Frobenius norm as a sum of
norms of the columns of A and apply (3.11):

X
n X
m X
m X
m
kAk2F D jaij j2 D kAıj k2  kAk2 kıj k2 D mkAk2 :
i D1 j D1 j D1 j D1

For the first estimate, we let x 2 F m with kxk D 1 and use the Cauchy–Schwarz
inequality to obtain

X
n n ˇX
X m ˇ2
2 2 ˇ ˇ
kAxk D j.Ax/i j D ˇ aij xj ˇ
i D1 i D1 j D1
X n Xm Xm
 jaij j2 jxk j2 D kAk2F kxk2 D kAk2F :
i D1 j D1 kD1

Definition 3.5 yields kAk  kAkF .

Section 3.4 Error estimates 49

If the off-diagonal part of A has been sufficiently reduced, A is “almost” a diagonal

matrix. Since it is very easy to find the eigenvalues and eigenvectors of a diagonal
matrix, we would like to know how “close” they are to the eigenvalues and eigenvec-
tors of A. For our proof of the corresponding perturbation estimate, we rely on the
following simple version of the Courant–Fischer–Weyl theorem (cf. [6, §4.1]).

Lemma 3.8 (Minimization problem). Let A 2 F nn be a self-adjoint matrix, and let
 2 .A/ be its minimal eigenvalue. We have
 D min¹hAx; xi W x 2 F n ; kxk D 1º:
Proof. Due to Corollary 2.51, we find a unitary matrix Q 2 F nn and a real diagonal
matrix D 2 Rnn with
A D QDQ :
Since  is the minimal eigenvalue, Proposition 2.18 yields
 D min¹di i W i 2 ¹1; : : : ; nºº;
and we can find j 2 ¹1; : : : ; nº with  D djj .
For all x 2 F n with kxk D 1, we have
X
n
hAx; xi D hQDQ x; xi D hDQ x; Q xi D hDb xi D
x ;b di i jxO i j2
i D1

for b
x WD Q x.
Lemma 2.38 yields kb
x k D kxk, and we obtain
´ n μ
X
n 2 n
min¹hAx; xi W x 2 F ; kxk D 1º D min di i jxO i j W b
x 2 F ; kb
xk D 1 :
i D1

Due to
X
n X
n
x k2 D
 D kb djj jxO i j2  di i jxO i j2 x 2 F n ; kb
for all b x k D 1;
i D1 i D1

 is a lower bound, and using b

x D ıj demonstrates that it is the minimum.

Exercise 3.9 (Maximization problem). Let A 2 F nn be a self-adjoint matrix,

and let  2 .A/ be its maximal eigenvalue. Prove

 D max¹hAx; xi W x 2 F n ; kxk D 1º:

Combine the result with Exercise 3.6 to show

kAk D max¹jj W  2 .A/º:

The spectral norm owes its name to this property.

50 Chapter 3 Jacobi iteration

Exercise 3.10 (Eigenvectors). Let A 2 F nn be a self-adjoint matrix, and let

 2 .A/ be its minimal eigenvalue. Let x 2 F n with kxk D 1. Prove that x is
an eigenvector of A for the eigenvalue  if and only if hAx; xi D  holds.

Based on Lemma 3.8, we can investigate the sensitivity of eigenvalues with respect
to perturbations of the matrix. The result is the following simple version of the Bauer–
Fike theorem [3]:

Proposition 3.11 (Perturbed eigenvalues). Let A; B 2 F nn be self-adjoint matrices.

For each eigenvalue  of B, we can find an eigenvalue  of A such that

j  j  kA  Bk:

Proof. Let  2 .B/. We choose  2 .A/ with

j  j D min¹j0  j W 0 2 .A/º:

This choice implies that    is the smallest eigenvalue in modulus of A  I . Since

A and B are self-adjoint, we have    2 R, so .  /2 has to be the smallest
eigenvalue of .A  I /2 . Applying Lemma 3.8 to the latter matrix yields

.  /2 D min¹h.A  I /2 x; xi W x 2 F n ; kxk D 1º:

Let y 2 F n n ¹0º be an eigenvector of B for the eigenvalue . Since y ¤ 0, we can

assume kyk D 1 without loss of generality. Using (2.11) and (3.11), we find

.  /2 D min¹h.A  I /2 x; xi W x 2 F n ; kxk D 1º

 h.A  I /2 y; yi D h.B  I C A  B/2 y; yi
D h.B  I /2 y; yi C h.A  B/.B  I /y; yi
C h.B  I /.A  B/y; yi C h.A  B/2 y; yi
D h.B  I /y; .A  B/yi C h.A  B/y; .B  I /yi C h.A  B/2 y; yi
D h.A  B/y; .A  B/yi D k.A  B/yk2  kA  Bk2 kyk2
D kA  Bk2

due to .B  I /y D 0, and since    is real, this implies j  j  kA  Bk.

Finding an estimate for the eigenvectors is slightly more challenging: if the “gap”
between eigenvalues is small, perturbing the matrix may lead to large changes in the
eigenvectors. As an example, we consider
       
1 C 2 1 1 1 1 1C 1 1
A WD ; B WD D
1 1 2 1 1 1 1 1
Section 3.4 Error estimates 51

for a small value > 0. Using Exercise 3.9 and Proposition 2.13, we can compute the
spectral norm
 
 2  
kA  Bk D    D max¹jj W .  2 /  2 D 0º
 0 
p p p
D max¹.1  2/ ; .1 C 2/ º D .1 C 2/ :

Although the norm of the difference of A and B can be very small if is, the angles
between each of the eigenvectors of both matrices always equal =2. The reason
is that the “gap” between the two eigenvalues of A equals only 2 , and so a small
perturbation of the eigenvalue can lead to a completely different eigenvector. In order
to obtain a useful estimate for the approximation of eigenvectors, we can introduce a
lower bound for this gap.

Definition 3.12 (Spectral gap). Let A 2 F nn be a self-adjoint matrix. For all  2
.A/, we define the spectral gap of  and A by

A ./ WD inf¹j  j W  2 .A/ n ¹ºº: (3.12)

Using this quantity, we can formulate the following estimate for the eigenvectors
of a perturbed matrix:

Proposition 3.13 (Perturbed eigenvectors). Let A; B 2 F nn be self-adjoint matri-

ces. For each eigenvector y of B for an eigenvalue  we can find an eigenvalue 
of A and a vector x 2 E.A; / in the corresponding eigenspace satisfying

kA  Bk
kx  yk  2 kyk:

A ./

Proof. Let y 2 F n n ¹0º be an eigenvector for an eigenvalue  of B. Due to Proposi-

tion 3.11, we can find an eigenvalue  of A such that j  j  kA  Bk holds.
According to Corollary 2.51, we can find a unitary matrix Q 2 F nn and a real
diagonal matrix D 2 Rnn satisfying

A D QDQ :

y WD Q y and define b
We let b x 2 F n by
´
yOj if djj D ;
xOj WD for all j 2 ¹1; : : : ; nº:
0 otherwise

For x WD Qb
x , we obtain

Ax D QDQ Qb
x D QDb
x D Qb
x D x
52 Chapter 3 Jacobi iteration

since D is a diagonal matrix and xOj is only non-zero if djj D , and conclude x 2
E.A; /. Now we compute
k.A  I /.x  y/k D k.A  I /x  .A  I /yk D k.A  I /yk
D k.B  I /y C .A  B/y C .  /yk
 k.B  I /yk C k.A  B/yk C j  j kyk
 kA  Bk kyk C kA  Bk kyk D 2kA  Bk kyk
using the compatibility property (3.11) and the fact that y is an eigenvector of B. Now
we only have to eliminate the matrix A  I in this estimate:
k.A  I /.x  y/k2 D kQ.D  I /Q .x  y/k2 D k.D  I /.b y /k2
x b
X
n X
n
D jdjj  j2 jxOj  yOj j2 D jdjj  j2 jxOj  yOj j2
j D1 j D1
djj ¤

X
n X
n


A ./2 jxOj  yOj j2 D
A ./2 jxOj  yOj j2
j D1 j D1
djj ¤

D
A ./2 kb y k2 D
A ./2 kQ.b
x b y /k2
x b
D
A ./2 kx  yk2 ;
where we have used our special choice of b
x to avoid inconvenient terms in the sum
and Lemma 2.38.

The arguments used to eliminate A  I in this proof are closely related to the
concept of a pseudo-inverse: due to  2 .A/, the matrix A  I is not injective
and therefore not invertible. Still, we can define an inverse in the range of AI ,
and this is called a pseudo-inverse. In our case, the range of AI is the invariant
subspace spanned by all eigenspaces except E.A; /, and the vector x is chosen
to ensure that x  y is an element of this space.

Using this estimate, we can estimate the accuracy of the eigenvalue approximations:

Corollary 3.14 (Eigenvalue accuracy). Let A 2 F nn be a self-adjoint matrix. For

each i 2 ¹1; : : : ; nº, we can find  2 .A/ such that
jai i  j  off.A/;
and we can find a vector x 2 E.A; / such that
off.A/
kıi  xk  2 :

A ./
Section 3.4 Quadratic convergence 53

Proof. We choose B 2 F nn to be the diagonal part of A, i.e.,

´
ai i if i D j;
bij D for all i; j 2 ¹1; : : : ; nº:
0 otherwise

Due to Proposition 3.7, this implies

kA  Bk2  kA  Bk2F D off.A/2 :

Let i 2 ¹1; : : : ; nº. Since ai i is an eigenvalue of the diagonal matrix B, Proposi-

tion 3.11 states that we can find an eigenvalue  2 .A/ satisfying

jai i  j  kB  Ak  off.A/:

Since ıi is a corresponding eigenvector, we can apply Proposition 3.13 to find the

required vector x 2 E.A; /.

The estimate given here is suboptimal: combining Proposition 3.13 with the esti-
mate provided by Theorem 4.6 for the Rayleigh quotient introduced in Section 4.2
yields that jai i  j converges like off.A/2 .

Under certain additional conditions, Jacobi’s method offers the advantage that it
can be used to compute eigenvalues to a very high relative accuracy, while most
other algorithms typically reach only a high absolute accuracy [8].

3.5 Quadratic convergence 

Under certain conditions, the Jacobi iteration converges quadratically, i.e., it can be
significantly faster if the matrix A is sufficiently close to diagonal form [19]. This
section follows the outline of the proofs presented in [51] and [39], slightly adapted
to fit our definition of the Jacobi iteration.
Obviously, the Jacobi iteration can only converge if the matrices do not change too
much when applying the unitary transformation. This means that jsj in (3.1), the sine
of the rotation angle, has to be small. Due to (3.6), we have

1 1 japq j
jsj  jt j jcj  jt j  p  D (3.13)
j j C j j C 1
2 2j j japp  aqq j

for the elementary Jacobi step that eliminates apq with 1  p < q  n. This bound is
only useful if we can ensure that the diagonal elements of the matrix are sufficiently
far from each other.
54 Chapter 3 Jacobi iteration

Due to Corollary 3.4, we know that the diagonal elements converge to eigenval-
ues, so we can only expect well-separated diagonal elements if the eigenvalues are
well-separated. For the remainder of this section, we assume that A has only simple
eigenvalues, i.e., that
#.A/ D n (3.14)
holds. The minimal distance of two eigenvalues is denoted by

 WD min¹j  j W ;  2 .A/;  ¤ º; (3.15)

and we can use it in combination with Proposition 3.11 to obtain the following esti-
mate for the diagonal entries:

Lemma 3.15 (Diagonal entries). Let 2 off.A/ < , i.e., ı WD   2 off.A/ > 0. Then
we have

japp  aqq j
ı for all p; q 2 ¹1; : : : ; nº; p ¤ q:

Proof. (cf. [51, eq. (1.6)]) Let D 2 F nn be the diagonal part of A, i.e.,
0 1
a11
B :: C
DD@ : A;
ann

We have kA  DkF D off.A/, and combining Proposition 3.11 with Proposition 3.7
yields that for each  2 .A/ we can find i 2 ¹1; : : : ; nº with

j  ai i j  off.A/: (3.16)

Let now  2 .A/ with

j  ai i j  off.A/:
By the triangle inequality, this implies

j  j D j  ai i C ai i  j  j  ai i j C j  ai i j  2 off.A/ < ;

i.e.,  D  by our assumption (3.15). This means that the mapping  7! i defined
by (3.16) is injective, and (3.14) yields that it also has to be bijective.
Let now p; q 2 ¹1; : : : ; nº be fixed with p ¤ q. We have seen that there are
;  2 .A/ with  ¤  such that p D i and q D i , and conclude

japp  aqq j D jai i  ai i j D j    .  ai i /  .ai i  /j

j  j  j  ai i j  j  ai i j
  2 off.A/ D ı

by the triangle inequality.

Section 3.5 Quadratic convergence 55

We consider one sweep of the algorithm, i.e., N WD n.n  1/=2 elementary Jacobi
steps. In the r-th step, we choose 1  pr < qr  n and eliminate the entry in the
pr -th row and qr -th column of the matrix. We denote the sine of the rotation angle
(cf. (3.5)) by sr and the cosine by cr . The original matrix is written as A.0/ , and the
matrix resulting from applying the r-th transformation to A.r1/ as A.r/ . Due to our
.r1/
construction the r-th transformation is chosen to eliminate apr qr . The key to the
convergence analysis is the following bound for the Jacobi angles:

Lemma 3.16 (Jacobi angles). Let ı WD   2 off.A/ > 0. We have

X
N
off.A.0/ /2
jsr j2  : (3.17)
rD1
2ı 2

Proof. (cf. [51, eq. (2.2)] and [39, eq. (9)]) Theorem 3.2 yields
off.A.r/ /2 D off.A.r1/ /2  2jap.r1/
r qr
j2 for all r 2 ¹1; : : : ; N º;
and a simple induction gives us
X
N
0  off.A.N / /2 D off.A.0/ /2  2 jap.r1/
r qr
j2 ;
rD1

which implies
X
N
off.A.0/ /2
jap.r1/
r qr
j2  : (3.18)
2
rD1

Due to off.A.r1/ /  off.A.0/ /, we can apply Lemma 3.15 to A.r1/ , and (3.13)
yields
X
N X
N
jap.r1/ j2 X
N
jap.r1/ j2 off.A.0/ /2
r qr r qr
jsr j2   2
 :
.r1/ .r1/
japr pr  aqr qr j2 ı 2ı 2
rD1 rD1 rD1

In the r-th step of the sweep, the entry ap.r1/

r qr
is eliminated, we have ap.r/r qr
D 0.
Unfortunately, this entry will usually not stay eliminated: if there is an ` 2 ¹1; : : : ; N º
.`/
with pr D p` and qr ¤ q` , the pr -th row will be changed and apr qr may no longer
be equal to zero. In fact, (3.9) yields
jap.`/
r qr
j D jcNr ap.`1/
r qr
 sNr aq.`1/
` qr
j  jcNr j jap.`1/
r qr
j C jNsr j jaq.`1/
` qr
j
 jap.`1/
r qr
j C jsr j jaq.`1/
` qr
j: (3.19a)
If we have pr D q` and qr ¤ p` , the pr -th row will also be changed:
jap.`/
r qr
j D jsr ap.`1/
` qr
C cr ap.`1/
r qr
j  jsr j jap.`1/
` qr
j C jcr j jap.`1/
r qr
j
 jap.`1/
r qr
j C jsr j jap.`1/
` qr
j: (3.19b)
56 Chapter 3 Jacobi iteration

The same reasoning applies to the columns: if qr D p` and pr ¤ q` , (3.8) implies

jap.`/
r qr
j D jap.`1/
r qr
cr  ap.`1/
r q`
sr j  jap.`1/
r qr
j jcr j C jap.`1/
r q`
j jsr j
 jap.`1/
r qr
j C jsr j jap.`1/
r q`
j; (3.19c)

while qr D q` and pr ¤ p` leads to

jap.`/
r qr
j D jap.`1/ sN C ap.`1/
r p` r r qr
cNr j  jap.`1/
r p`
j jsr j C jap.`1/
r qr
j jcr j
 jap.`1/
r qr
j C jsr j jap.`1/
r p`
j: (3.19d)

For pr D p` and qr D q` , ap.`1/

r qr
is eliminated again, and pr D q` and qr D p` is
impossible due to pr < qr and p` < q` . If ¹pr ; qr º \ ¹p` ; q` º D ;, ap.`1/
r qr
is not
changed. We can summarize (3.19) as follows:

Lemma 3.17 (Bound for one Jacobi step). Let ` 2 ¹1; : : : ; N º. Define the matrix
E .`/ 2 F nn by
8 .`1/
ˆ
ˆ aq` j if i D p` ; j ¤ q` ;
ˆ
ˆ
ˆ
ˆ
.`1/
if i D q` ; j ¤ p` ;
<ap` j
.`/ .`1/
eij D aiq if j D p` ; i ¤ q` ; for all i; j 2 ¹1; : : : ; nº: (3.20)
ˆ
ˆ `
ˆ
ˆaip .`1/
if j D q` ; i ¤ p` ;
ˆ
:̂ `
0 otherwise

Then we have
.`/ .`1/ .`/
jaij j  jaij j C js` j jeij j for all i; j 2 ¹1; : : : ; nº:

Proof. Combine the estimates (3.19) and take advantage of the fact that A D A
implies .A.`1/ / D A.`1/ , which in turn implies .E .`/ / D E .`/ .

If we use a cyclic Jacobi method, i.e., if each off-diagonal entry is eliminated once
during the sweep, we obtain the following convergence estimate:

Theorem 3.18 (Cyclic Jacobi method). Let ı WD   2 off.A.0/ / > 0. If for each
1  p < q  n there is an r 2 ¹1; : : : ; N º with pr D p and qr D q, i.e., if each
off-diagonal entry is eliminated once during the sweep, we have
r
.N / N
off.A /  off.A.0/ /2 ; (3.21)
2ı 2
i.e., the Jacobi method converges quadratically.
Section 3.5 Quadratic convergence 57

Proof. (cf. [51, Section 2]) Let 1  p < q  n. By our assumption, we can find
.r1/
r 2 ¹1; : : : ; N º with pr D p and qr D q. Since the Jacobi rotation eliminates apr qr ,
.r/
we have apq D 0. Lemma 3.17 and a simple induction yield

X
N X
N
.N / .`/ .`/
japq j js` j jepq j js` j jepq j:
`DrC1 `D1

By the Cauchy–Schwarz inequality, we find

!2 ! !
X
N X
N X
N
.N / 2 .`/
japq j  js` j jepq j  js` j2 .`/ 2
jepq j :
`D1 `D1 `D1

Using Lemma 3.16, we obtain

! N
XN X off.A.0/ /2 X
N
.N / 2 2
off.A /  js` j off.E .`/ /2  off.E .`/ /2 : (3.22)
2ı 2
`D1 `D1 `D1

A look at (3.20) in combination with Theorem 3.2 reveals

off.E .`/ /2  off.A.`/ /2  off.A.0/ /2 ;

so we have found

off.A.0/ /2 X
N
N
off.A.N / /2  2
off.E .`/ /2  2 off.A.0/ /4 ;
2ı 2ı
`D1

and this implies (3.21).

Remark 3.19 (Improvements). This is obviously a very rough estimate, particularly

the inequality off.E .`/ /2  off.A.0/ /2 is very pessimistic, since most of the entries
of E .`/ are equal to zero.
For the cyclic-by-row Jacobi method, the convergence estimate can be improved
significantly to obtain r
.N / 1
off.A /  off.A.0/ /2
2ı 2
by analyzing the entries of the intermediate matrices more carefully [51, Section 3].

Exercise 3.20 (Improved estimate). Take the structure of the matrices .E .`/ /N
`D1
into account to improve the result of Theorem 3.18 to
r
.N / n
off.A /  off.A.0/ /2 :
ı2
58 Chapter 3 Jacobi iteration

Hint: Theorem 3.2 implies off.A.`/ /  off.A.0/ / for all ` 2 ¹0; : : : ; N º. How
often is each of the entries in the matrix changed during a sweep? Can you
improve the estimate even further, maybe using additional assumptions regarding
the sequence of elementary steps?

Theorem 3.21 (Classical Jacobi method). Let ı WD   2 off.A.0/ / > 0, and let
the indices .pr /N N
rD1 and .qr /rD1 be chosen as in the classical Jacobi method, i.e.,
satisfying
.r1/
jap.r1/
r qr
j
jaij j for all 1  i < j  n; r 2 ¹1; : : : ; N º: (3.23)

Then we have r
n2
.N /
off.A off.A.0/ /2 ;
/ (3.24)
ı2
i.e., the classical Jacobi method also converges quadratically.

Proof. (cf. [39, Satz 2]) Let IN WD ¹.i; j / W 1  i < j  nº. We prove by induction
that for each r 2 ¹1; : : : ; N º we can find Ir  IN with
!2
X .r/ 2
Xr
#Ir D r; jaij j  2.n  2/ js` j jap.`1/
` q`
j : (3.25)
.i;j /2Ir `D1

We can see that for r D N , this estimate implies an upper bound for off.A.N / /2 =2.
For r D 1, we choose I1 D ¹.p1 ; q1 /º. The Jacobi step guarantees ap.1/ 1 q1
D 0, so
(3.25) holds trivially.
Assume now that r 2 ¹1; : : : ; N  1º is given such that a subset Ir  IN satisfying
(3.25) exists. Let J  IN be a subset of cardinality #J D r such that
X .r/ 2
jaij j
.i;j /2J

is minimal among all those subsets. This implies

X .r/ 2
X .r/ 2
jaij j  jaij j :
.i;j /2J .i;j /2Ir

Since r < N and (3.23) holds, we can assume .pr ; qr / 62 J , since otherwise we could
replace it by a different index pair without increasing the sum. Therefore the set

IrC1 WD J [ ¹.pr ; qr /º

has cardinality #IrC1 D r C 1. Due to Lemma 3.17, we have

.rC1/ .r/ .rC1/
jaij j  jaij j C jsrC1 j jeij j for all .i; j / 2 J:
Section 3.5 Quadratic convergence 59

A look at (3.20) reveals that only 4.n  2/ entries of E .`/ , namely the p` -th and q` -
th rows and columns with the exception of their intersections, are non-zero. To take
advantage of this property, let

JC WD ¹.i; j / 2 J W .i 2 ¹prC1 ; qrC1 º ^ j 2

6 ¹prC1 ; qrC1 º/_
.j 2 ¹prC1 ; qrC1 º ^ i 26 ¹prC1 ; qrC1 º/º

denote the set of all coefficients in J changed during the update from A.r/ to A.rC1/ .
Its cardinality equals #JC D 2.n  2/, since J contains only index pairs above the
.rC1/
diagonal. We use (3.23) to bound jeij j by jap.r/
rC1 qrC1
j for all .i; j / 2 JC and
apply the Cauchy–Schwarz inequality to obtain
X .rC1/ 2
X .r/ .rC1/ 2
jaij j  .jaij j C jsrC1 j jeij j/
.i;j /2J .i;j /2J
X .r/ 2 .r/ .rC1/ .rC1/ 2
D .jaij j C 2jaij j jsrC1 j jeij j C jsrC1 j2 jeij j /
.i;j /2J
X .r/ 2
X
 jaij j C jsrC1 j2 jap.r/
rC1 qrC1
j2
.i;j /2J .i;j /2JC
X .r/
C 2jap.r/
rC1 qrC1
j jsrC1 j jaij j
.i;j /2JC
X .r/ 2
 jaij j C 2.n  2/jsrC1 j2 jap.r/
rC1 qrC1
j2
.i;j /2J
p s X
.r/
C 2jap.r/
rC1 qrC1
j jsrC1 j 2.n  2/ jaij j2
.i;j /2JC
0 12
s X p
D@ .r/ 2
jaij j C 2.n  2/jsrC1 j jap.r/
rC1 qrC1
jA :
.i;j /2J

Applying the induction assumption (3.25) to the first term yields

!2
X .rC1/ 2
p X
r p
jaij j  2.n  2/ js` j jap.`1/
` q`
jC 2.n  2/jsrC1 j jap.r/
rC1 qrC1
j
.i;j /2JC `D1
!2
X
rC1
D 2.n  2/ js` j jap.`1/
` q`
j :
`D1

This completes the induction.

60 Chapter 3 Jacobi iteration

We apply (3.25) to r D N and use the Cauchy–Schwarz inequality, Lemma 3.16

and (3.18) to obtain
!2
X .rC1/ 2
X
N
.N / 2
off.A / D2 jaij j  4.n  2/ js` j jap.`1/
` q`
j
.i;j /2IN `D1
! !
X
N X
N
2
 4.n  2/ js` j jap.`1/
` q`
j2
`D1 `D1
off.A.0/ /2 off.A.0/ /2 n2
 4.n  2/ D off.A.0/ /4 :
2ı 2 2 ı2
Remark 3.22 (Multiple eigenvalues). In order to prove quadratic convergence, we
rely on the assumption that all eigenvalues of the matrix A are simple. This assump-
tion can be avoided if we instead require all diagonal elements converging to the same
multiple eigenvalue to be grouped together [46].
Applying generalized rotations to entire blocks of the matrix A, a variant of the
cyclic Jacobi method can be derived that converges globally [13].
Chapter 4

Power methods

Summary
This chapter introduces the power iteration, a very simple method for computing
eigenvectors that paves the way for a number of very important and efficient meth-
ods like the inverse iteration, the Rayleigh iteration, and the simultaneous iteration.
We investigate the convergence of these iterations and provide simple estimates for
the accuracy of the approximations of eigenvalues and eigenvectors that can be used
to construct stopping criteria for the iteration.

Learning targets
 Introduce the power iteration and the (shifted) inverse iteration.
 Analyze the convergence of these methods.
 Introduce the Rayleigh iteration and establish its cubic convergence.
 Investigate the convergence to an invariant subspace instead of an eigenspace.
 Introduce the simultaneous iteration to compute a basis of this invariant sub-
space.

4.1 Power iteration

The power method (also known as the Von Mises iteration [47]) is a very simple and
yet very flexible method for computing an eigenvector. The algorithm can be traced
back to a publication [35] that appeared in 1921 and uses it (or more precisely a
variant of the related inverse iteration) to solve a generalized eigenproblem arising
in the context of structural mechanics. Our discussion of the method and its proof
follows the theory already outlined in [35, 47].
Let n 2 N, and let A 2 F nn be a matrix. In order to keep the theoretical investiga-
tion simple, we assume that A is a normal matrix if F D C or self-adjoint if F D R.
Note that the theory can be extended to more general matrices without changing the
algorithm (cf. Section 4.8).
The idea of the power iteration is the following: assume that a vector x 2 F n is
given and that it can be represented as a linear combination
x D ˛1 e1 C    C ˛k ek
62 Chapter 4 Power methods

of eigenvectors e1 ; : : : ; ek 2 F n corresponding to eigenvalues 1 ; : : : ; k . Then we

have
Am x D ˛1 m m
1 e1 C    C ˛k k ek for all m 2 N0 :
If one of the eigenvalues is larger in modulus than the others, the corresponding term
grows faster than all others for m ! 1, and we can expect to obtain an approximation
of an eigenvector.
In order to analyze the behaviour of the sequence .x .m/ /1
mD0 given by

x .m/ WD Am x .0/ D Ax .m1/ for all m 2 N (4.1)

more closely, we represent the vectors in a suitable basis consisting of eigenvectors.
Since A is normal or self-adjoint, Corollaries 2.50 or 2.51, respectively, imply that we
can find a unitary matrix Q 2 F nn and a diagonal matrix D 2 F nn such that the
Schur decomposition takes the form
A D QDQ ; D D Q AQ: (4.2)
The diagonal elements of D are its eigenvalues, and therefore also the eigenvalues
of A. We denote these eigenvalues by 1 ; : : : ; n given by
0 1
1
B :: C
DD@ : A: (4.3)
n
We assume that A has a dominant eigenvalue, i.e., that one eigenvalue is simple and
larger in modulus than all others. Since changing the order of the columns of Q and D
will leave the Schur decomposition intact, we can assume
j1 j > j2 j
  
jn j: (4.4)
Let a starting vector x .0/ 2 F n n ¹0º be given, and let the iteration vectors x .1/ ;
x .2/ ; : : : be defined as in (4.1). We transform the vectors into the eigenvector basis,
i.e., define
x .m/ WD Q x .m/
b for all m 2 N0 ;
and (4.1) takes the form
x .mC1/ D Q x .mC1/ D Q Ax .m/ D Q AQb
b x .m/ D Db
x .m/ ;
x .m/ D D mb
b x .0/ for all m 2 N0 :
Since D is a diagonal matrix, this equation allows us to compute the components of
the iteration vectors directly:
.mC1/ .m/
b
xj D j b
xj holds for all m 2 N0 ; j 2 ¹1; : : : ; nº: (4.5)
Section 4.1 Power iteration 63

Due to (4.4), the first component of b x .m/ will grow more rapidly than any other com-
ponent as m ! 1, so we can expect the transformed iteration vectors to “converge”
to multiples of the eigenvector ı1 and the original vectors to multiples of the eigen-
vector e1 WD Qı1 .
Unfortunately, if 1 ¤ 1, there can be no convergence in the usual sense, since the
length of the iteration vectors is changed by applying the matrix. Instead, we look for
convergence of the angle between the iteration vectors and the eigenvector.

Definition 4.1 (Angle between vectors). Let x; y 2 F n n ¹0º. We define

jhx; yij
cos †.x; y/ WD ;
kxk kyk
q
sin †.x; y/ WD 1  cos2 †.x; y/;
sin †.x; y/
tan †.x; y/ WD :
cos †.x; y/

If cos †.x; y/ D 0 holds, we let tan †.x; y/ D 1.

Since Q is unitary, Q is isometric, and Lemma 2.38 yields

jhQ x; Q yij jhQQ x; yij
cos †.Q x; Q y/ D D
kQ xk kQ yk kxk kyk
jhx; yij
D D cos †.x; y/ for all x; y 2 F n n ¹0º: (4.6)
kxk kyk
Due to our definition, the sine and the tangent also remain unchanged by unitary
transformations. Now we can formulate a first convergence result: we let e1 WD Qı1
and observe
Ae1 D AQı1 D QQ AQı1 D QDı1 D 1 Qı1 D 1 e1 ;
i.e., e1 is a convenient eigenvector for the dominant eigenvalue 1 . In the case of a
dominant eigenvalue, i.e., if (4.4) holds, we can now establish the following conver-
gence estimate:

Theorem 4.2 (Convergence). Let (4.2) and (4.3) hold with

j1 j > j2 j
  
jn j:
Let e1 WD Qı1 and cos †.x .0/ ; e1 / ¤ 0. Let .x .m/ /1mD0 be given by (4.1). Then we
have
j2 j
tan †.x .mC1/ ; e1 /  tan †.x .m/ ; e1 / for all m 2 N0 ;
j1 j
and due to j1 j > j2 j, this means that the angle between the iteration vectors and
the eigenvector e1 converges to zero.
64 Chapter 4 Power methods

Proof. Due to (4.6), we have

.m/
x .m/ ; ı1 ij2
jhb jb
x 1 j2
cos2 †.x .m/ ; e1 / D cos2 †.b
x .m/ ; ı1 / D D ;
kb
x .m/ k2 kı1 k2 kb
x .m/ k2
.m/
x 1 j2
jb kb x .m/
x .m/ k2  jb 1 jr
sin2 †.x .m/ ; e1 / D 1  cos2 †.x .m/ ; e1 / D 1  D
kbx k.m/ 2 kb
x k
.m/ 2
Pn .m/ 2 .m/ 2 Pn .m/ 2
j D1 jb
xj j  jb x1 j j D2 jb
xj j
D D ;
kbx .m/ k2 kb
x .m/ k2
Pn
2 .m/ sin2 †.x .m/ ; e1 / xj.m/ j2
j D2 jb
tan †.x ; e1 / D D for all m 2 N0 :
cos2 †.x .m/ ; e1 / x .m/
jb j 2
1

Using (4.5) and (4.4) yields

Pn Pn
2 .mC1/ j D2 jbxj.mC1/ j2 2 x .m/ j2
j D2 jj j jb j
tan †.x ; e1 / D .mC1/ 2
D .m/
jb
x1 j j1 j2 jb
x 1 j2
P .m/  2
j2 j2 jnD2 jb xj j2 j2 j
 D tan2 †.x .m/ ; e1 /
x .m/
j1 j2 jb j1 j
1 j
2

for all m 2 N0 ;

and taking the square root of both sides of the inequality yields the result.

The condition cos †.x .0/ ; e1 / ¤ 0 is required to ensure that the denominators in the
.0/
proof are well-defined: it implies b x 1 ¤ 0, and since we also have j1 j > j2 j
0
due to (4.4), this implies
.m/ .0/
b
x1 D m
1bx 1 ¤ 0; x .m/ k ¤ 0
kb for all m 2 N0 :

.0/
This requirement is also quite natural: if cos †.x .0/ ; e1 / D 0 holds, we have b
x 1 D 0,
so the starting vector contains no component in the direction of the desired eigenvec-
tor, and this will not change no matter how often we apply the iteration step.
Section 4.1 Power iteration 65

The assumption that A is normal or self-adjoint is not necessary, it only serves

to make the theoretical investigation of the convergence more elegant. If A is
merely diagonalizable, i.e., if there is a regular matrix T 2 F nn such that
0 1
1
B C
T 1 AT D D D @ ::
: A
n

holds, we can introduce transformed iteration vectors

x .m/ WD T 1 x .m/
b for all m 2 N0

and obtain

x .mC1/ D T 1 x .mC1/ D T 1 Ax .m/ D T 1 AT b

b x .m/ D Db x .m/
for all m 2 N0 ;

i.e., the transformed vectors result from a power iteration using the matrix D and
the starting vector b
x .0/ . Applying Theorem 4.2 to the transformed vectors yields
convergence estimates.

In a practical implementation, we have to replace the field F by machine numbers,

and these numbers cannot exceed a given maximum. This can lead to problems if j1 j
is larger that one: the component b x .m/
1 D m1bx .0/
1 can grow beyond the maximum,
and this usually leads to an overflow. A similar effect occurs if j1 j is smaller than
.m/
one: bx 1 tends to zero, and truncation will eliminate this component after a certain
number of iterations. In order to avoid these undesirable effects, we normalize the
iteration vectors, i.e., we ensure that they always have unit norm.

Using machine numbers also means that rounding effects will influence the con-
vergence behaviour, cf. [52, Chapter 9].
66 Chapter 4 Power methods

procedure power_iteration(A, var x);

begin
while „not sufficiently accurate“ do begin
a Ax;
x a=kak
end
end

Figure 4.1. Power iteration, still missing a stopping criterion.

Since the angle between vectors does not depend on their scaling, this leaves the
convergence estimate of Theorem 4.2 intact and leads to the numerically stable algo-
rithm summarized in Figure 4.1.
An important advantage of the power iteration is its flexibility: we only have to
be able to evaluate matrix-vector products, we do not require the matrix A itself. In
particular, we do not have to apply transformations to the matrix like in the case of
the Jacobi iteration. This allows us to use the power iteration in situations where A
is only given implicity: e.g., if A is the inverse of a matrix B, we can replace the
matrix-vector multiplication a D Ax D B 1 x by solving the system Ba D x, and
this is usually easier and more stable.

Exercise 4.3 (Indefinite matrix). Consider the matrix

 
2
A WD :
2

Investigate the convergence of the power iteration for the starting vectors
     
1 0 1
x .0/ D
b ; b x .0/ D ; bx .0/ D :
0 1 1

Does the angle between b x .m/ and an eigenspace always converge? If not, explain
why this is not a counterexample for Theorem 4.2.

4.2 Rayleigh quotient

Before we introduce refined variants of the power iteration, we investigate the question
of a proper stopping criterion. The power iteration gives us a sequence .x .m/ /1
mD0 of
vectors that we expect to converge to an eigenvector e1 in the sense that the angles
between the iterates and this eigenvector converge to zero, and we would like to know
when the angle between x .m/ and e1 has become sufficiently small.
Section 4.2 Rayleigh quotient 67

If we know the dominant eigenvalue 1 , we have

Ae1 D 1 e1 ; 0 D 1 e1  Ae1 ;

so the norm
k1 x .m/  Ax .m/ k
should provide an estimate of how close we are to the eigenvector. In most applica-
tions, the eigenvalue is not known, so we have to approximate it as well. An elegant
solution to this problem is provided by the Rayleigh quotient: assume that e ¤ 0 is
an eigenvector of A, but the corresponding eigenvalue  is unknown. Our goal is to
reconstruct  using e. By our assumption, we have

Ae D e:

Taking the inner product with e on both sides yields

hAe; ei D he; ei;

and using
he; ei D kek2 > 0;
we can obtain
hAe; ei
D
he; ei
and have found . This result gives rise to the following definition:

Definition 4.4 (Rayleigh quotient). Let A 2 F nn . The mapping

hAx; xi
ƒA W F n n ¹0º ! F; x 7! ;
hx; xi
is called the Rayleigh quotient corresponding to the matrix A.

We have already encountered another important property of the Rayleigh quo-

tient in Lemma 3.8: for a self-adjoint matrix A, its minimum and maximum are
the smallest and largest eigenvalue, respectively.

For an approximation of an eigenvector, the Rayleigh quotient yields an approxi-

mation of an appropriate eigenvalue. In order to prove the corresponding estimate, we
require the following alternate characterization of the sine of the angle between two
vectors:
68 Chapter 4 Power methods

Lemma 4.5 (Sine). Let x; y 2 F n n ¹0º. Then we have

² ³
kx  ˛yk
sin †.x; y/ D min W ˛2F : (4.7)
kxk
Proof. We let ˛0 WD hx; yi=kyk2 and observe
hx; yi
hx  ˛0 y; yi D hx; yi  hy; yi D hx; yi  hx; yi D 0;
kyk2
i.e., x ˛0 y and y are perpendicular vectors. For ˛ 2 F and ˇ WD ˛ ˛0 , this implies
kx  ˛yk2 D kx  ˛0 y  ˇyk2 D h.x  ˛0 y/  ˇy; .x  ˛0 y/  ˇyi
D kx  ˛0 yk2  hˇy; .x  ˛0 y/i  h.x  ˛0 y/; ˇyi C jˇj2 kyk2
D kx  ˛0 yk2 C jˇj2 kyk2 ;
i.e., the right-hand side attains its minimum for ˛ D ˛0 . Due to
kx  ˛0 yk2 D kxk2  ˛N 0 hx; yi  ˛0 hy; xi C j˛0 j2 kyk2
 
2 jhx; yij2 jhx; yij2 jhx; yij2 2 jhx; yij2
D kxk   C D kxk 1 
kyk2 kyk2 kyk2 kxk2 kyk2
2 2 2 2
D kxk .1  cos †.x; y// D kxk sin †.x; y/;
this minimum has to be sin †.x; y/.
Using this result, we can give the following elegant bound for the approximation
error of the Rayleigh quotient:

Theorem 4.6 (Rayleigh quotient). Let A 2 F nn , and let x 2 F n n ¹0º. Let e 2
F n n ¹0º be an eigenvector of A for the eigenvalue  2 F. Then we have
jƒA .x/  j  kA  I k sin †.x; e/  kA  I k tan †.x; e/:
If A is a normal matrix, we even have
jƒA .x/  j  kA  I k sin2 †.x; e/  kA  I k tan2 †.x; e/:
Proof. Since e is an eigenvector, we have .AI /e D 0, and we can use the Cauchy–
Schwarz inequality (2.9) and the compatibility inequality (3.11) of the spectral norm
to find
ˇ ˇ
ˇ hAx; xi hx; xi ˇ
jƒA .x/  j D ˇˇ  ˇ D jh.A  I /x; xij
hx; xi hx; xi ˇ hx; xi
jh.A  I /.x  ˛e/; xij k.A  I /.x  ˛e/k kxk
D 
hx; xi kxk2
kA  I k kx  ˛ek kxk kx  ˛ek
 D kA  I k for all ˛ 2 F:
kxk2 kxk
Section 4.2 Rayleigh quotient 69

procedure power_iteration(A, var x);

begin
x x=kxk;
a Ax;
 ha; xi;
while ka  xk > jj do begin
x a=kak;
a Ax;
 ha; xi
end
end
Figure 4.2. Practical power iteration.

Due to Lemma 4.5, this yields

jƒA .x/  j  kA  I k sin †.x; e/;

i.e., our first estimate. Let now A be a normal matrix. Due to Proposition 2.36, we
now not only have .A  I /e D 0, but also .A  I / e D 0, and this allows us to
improve the estimate as follows:

jh.A  I /.x  ˛e/; xij jhx  ˛e; .A  I / xij

jƒA .x/  j D D
kxk2 kxk2
 .x  ˛e/ij
jhx  ˛e; .A  I / jh.A  I /.x  ˛e/; x  ˛eij
D D
kxk 2 kxk2
kA  I k kx  ˛ek2
 for all ˛ 2 F;
kxk2
and using the optimal ˛ of Lemma 4.5 yields

jƒA .x/  j  kA  I k sin2 †.x; e/:

Due to the Cauchy–Schwarz inequality (2.9), we have cos †.x; e/  1 and therefore
sin †.x; e/  tan †.x; e/.

Using the Rayleigh quotient, we can construct a stopping criterion for the power
iteration: for the m-th iteration vector x .m/ , we compute an approximation .m/ D
ƒA .x .m/ / of the eigenvalue and check whether kAx .m/  .m/ x .m/ k is sufficiently
small. In our algorithm, x .m/ is a unit vector, so the Rayleigh quotient takes the
simple form ƒA .x .m/ / D hAx .m/ ; x .m/ i and we obtain the practical algorithm given
in Figure 4.2.
70 Chapter 4 Power methods

4.3 Residual-based error control

Since the error analysis in this section relies on Propositions 3.11 and 3.13, we require
the matrix A to be self-adjoint. Since this is a particularly useful vector, it deserves to
be given a name:

Definition 4.7 (Residual). Let A 2 F nn , and let x 2 F n n ¹0º. The vector

r WD ƒA .x/x  Ax

is called the residual of x with respect to A.

The practical power iteration given in Figure 4.2 stops as soon as the norm of the
residual drops below j.m/ j, and we can use backward error analysis to derive error
bounds for the eigenvalue and the corresponding eigenvector: given x .m/ and .m/ D
ƒA .x .m/ /, we construct a matrix B such that

Bx .m/ D .m/ x .m/

holds, i.e., such that x .m/ is an eigenvector of B for the eigenvalue .m/ . If we can
bound kA  Bk, the perturbation theory of Propositions 3.11 and 3.13 can be applied.
In order to find a convenient bound for kA  Bk, we require the following lemma:

Lemma 4.8 (Rank-2 matrix). Let a; b 2 F n with ha; bi D 0, and let E WD ab  Cba .
Then we have kEk D kak kbk.

Proof. We first consider the special case kak D 1, kbk D 1.

Let x 2 F n n¹0º. We split x into components in the kernel of E and in the subspaces
spanned by a and b:

x D y C ˛a C ˇb; ˛ WD hx; ai; ˇ WD hx; bi; y WD x  ˛a  ˇb:

By assumption, we have ha; bi D 0, and this implies

hy; ai D hx; ai  ˛ha; ai  ˇhb; ai D hx; ai  hx; ai D 0;

hy; bi D hx; bi  ˛ha; bi  ˇhb; bi D hx; bi  hx; bi D 0;
Ey D ab  y C ba y D ahy; bi C bhy; ai D 0;

i.e., the vectors y, a and b are pairwise perpendicular and y 2 N .E/. We find

Ex D E.y C ˛a C ˇb/ D Ey C ˛Ea C ˇEb

D ˛ab  a C ˛ba a C ˇab  b C ˇba b D ˛b C ˇa;
Section 4.3 Residual-based error control 71

and Pythagoras’ theorem yields

kExk2 D j˛j2 kbk2 C jˇj2 kak2 D j˛j2 C jˇj2
 kyk2 C j˛j2 C jˇj2 D kyk2 C k˛ak2 C kˇbk2 D kxk2 ;
i.e., kEk  1. Due to
Ea D ab  a C ba a D b; kEak D kbk D 1;
we also have kEk
1, and therefore kEk D 1.
Now we consider the general case. If a D 0 or b D 0, we have E D 0 and
kEk D 0 D kak kbk. Otherwise we let b a WD a=kak and b
b WD b=kbk and apply the
b D 1 for E
special case to prove kEk b WD b b C b
ab a . Observing E D kak kbk E
bb b
completes the proof.
Using this result, the backward error analysis for eigenvectors is straightforward:

Theorem 4.9 (Accuracy). Let y 2 F n n ¹0º. Let  WD ƒA .y/, and let r WD y  Ay

denote the residual of y. Then there are an eigenvalue  2 .A/ and a vector x 2
E.A; / in the corresponding eigenspace satisfying
krk krk
j  j  ; kx  yk  2 :
kyk
A ./
Here
A ./ is the spectral gap introduced in (3.12).
Proof. We define the self-adjoint matrix
ry  yr 
B WD A C C ;
kyk2 kyk2
where we again interpret y and r as elements of F n1 in the obvious way. Due to
r  y D hy; ri D hy; y  Ayi D hy;
N yi  hy; Ayi D hAy; yi  hAy; yi D 0;
our definition yields
y y r y kyk2
By D Ay C r C y D Ay C r D Ay C y  Ay D y;
kyk2 kyk2 kyk2
i.e., y is an eigenvector of B for the eigenvalue . In order to be able to apply
Propositions 3.11 and 3.13, we have to find a bound for the spectral norm of
ry  yr  1
B AD C D .ry  C yr  /:
kyk 2 kyk 2 kyk2
Due to hr; yi D 0, we can apply Lemma 4.8 and obtain
krk kyk krk
kA  Bk  D :
kyk 2 kyk
Applying Propositions 3.11 and 3.13 completes the proof.
72 Chapter 4 Power methods

The practical power iteration in Figure 4.2 stops as soon as the norm of the residual
r .m/ defined by

r .m/ WD .m/ x .m/  Ax .m/ ; .m/ WD hAx .m/ ; x .m/ i for all m 2 N0 (4.8)

drops below j.m/ j. In this case, we can obtain the following estimates for the accu-
racy of the approximations of eigenvalues and eigenvectors:

Corollary 4.10 (Accuracy). Let 2 Œ0; 1/ and kr .m/ k  j.m/ j. Then we can find
 2 .A/ and x 2 E.A; / such that

j  .m/ j  j.m/ j  jj; (4.9a)
1
j.m/ j   jj
kx  x .m/ k  2  2 1 C : (4.9b)

A ./ 1 
A ./

Proof. By Theorem 4.9, kr .m/ k  j.m/ j implies that we can find an eigenvalue
 2 .A/ and a vector x 2 E.A; / such that the bounds

j.m/ j
j  .m/ j  j.m/ j; kx  x .m/ k  2

A ./
hold. In order to obtain an estimate for the relative error of the eigenvalue, we apply
the triangle inequality to the first bound to get

j  .m/ j  j.m/ j  jj C j.m/  j;

.1  /j  .m/ j  jj;

j  .m/ j  jj;
1
i.e., the relative error is bounded by =.1  /. Using this result, we can also derive a
bound for the eigenvector:

j.m/ j jj C j.m/  j jj C . =.1  //jj

kx  x .m/ k  2  2  2

A ./
A ./
A ./
  jj
D 2 1 C :
1 
A ./
We conclude that the stopping criterion guarantees a high relative accuracy of the
eigenvalue. If the relative spectral gap
A ./=jj is not too small, we can also expect
an accurate approximation of an eigenvector.
Section 4.4 Inverse iteration 73

4.4 Inverse iteration

The power iteration given in Figures 4.1 and 4.2 can only be used to compute the
largest eigenvalues (in modulus). We now introduce a modification that allows us to
find arbitrary eigenvalues: assume for the moment that A is invertible. Then  is an
eigenvalue of A if and only if 1= is an eigenvalue of A1 , since
1
Ax D x ” x D A1 x ” x D A1 x for all  2 F n ¹0º; x 2 F n :

(4.10)
This means that applying the power iteration to A1 instead of A should yield a se-
quence of vectors converging to the eigenspace corresponding to the smallest eigen-
value in modulus if
j1 j < j2 j      jn j;
and Theorem 4.2 yields a rate of j1 j=j2 j for the convergence. The resulting al-
gorithm is called the inverse iteration. It appears to have been developed first for
applications in the field of structural mechanics [35].
We can refine the idea to obtain a method that can find eigenvectors for any eigen-
value, not only for the largest and smallest ones: we introduce a shift parameter  2 F
and consider the matrix A  I . Its eigenvalues are “shifted”: we have
Ax D x ” .A  I /x D .  /x for all  2 F; x 2 F n ;
and (4.10) yields
1
Ax D x ” x D .A  I /1 x for all  2 F n ¹º; x 2 F d

(4.11)
if A  I is invertible. In this case, we can apply the power iteration to the matrix
.A  I /1 and arrive at the shifted inverse iteration:
x .m/ WD .A  I /1 x .m1/ for all m 2 N: (4.12)
The convergence results of Theorem 4.2 carry over directly:

Corollary 4.11 (Convergence). Let  2 F, and let A  I be invertible. Let (4.2)

and (4.3) hold with
j1  j < j2  j      jn  j: (4.13)
Let e1 WD Qı1 and cos †.x .0/ ; e1 / ¤ 0, and let .x .m/ /1
mD0 be given by (4.12). Then
we have
j1  j
tan †.x .mC1/ ; e1 /  tan †.x .m/ ; e1 / for all m 2 N0 ;
j2  j
and due to j1 j < j2 j, this means that the angle between the iteration vectors
and the eigenvector e1 converges to zero.
74 Chapter 4 Power methods

Proof. Let b
A WD .A  I /1 and
1
O j WD for all j 2 ¹1; : : : ; nº:
j  

The equations (4.2) and (4.3) yield

0 1
1  
B C
Q .A  I /Q D Q AQ  Q Q D D  I D @ ::
: A;
n  
0 1
O 1
B C
Qb
AQ D Q .A  I /1 Q D .D  I /1 D@ ::
: A:
O n

Using (4.13), we obtain

jO 1 j > jO 2 j
  
jO n j;
so we can apply Theorem 4.2 to b
A in order to find

jO 2 j
tan †.x .mC1/ ; e1 /  tan †.x .m/ ; e1 /
O
j1 j
j1  j
D tan †.x .m/ ; e1 / for all m 2 N0 :
j2  j
If we can find a suitable shift parameter , the shifted inverse iteration converges
rapidly and reliably: as a first example, we consider the matrix
 
1
AD :
1

Its eigenvalues are 1 and 1, and due to j1j D j  1j, we can expect no convergence
for the power iteration. If we use a shifted inverse iteration with, e.g.,  D 1=2,
Corollary 4.11 predicts a convergence rate of

j1  1=2j j1=2j
D D 1=3:
j  1  1=2j j  3=2j
By choosing  closer to 1, we can get even better rates.
The shift parameter can also help if the gap between eigenvectors is small: let > 0
and consider  
1C
AD :
1
Section 4.4 Inverse iteration 75

procedure inverse_iteration(A, , var x);

begin
Prepare a solver for A  I ;
x x=kxk;
Solve .A  I /a D x;
O ha; xi;  1=O C ;
O
while ka  xk O do begin
> jj
x a=kak;
Solve .A  I /a D x;
O ha; xi;  1=O C 
end
end
Figure 4.3. Inverse iteration.

The matrix A now has a dominant eigenvalue 1 C and a second eigenvalue 1, and
Theorem 4.2 predicts a convergence rate of 1=.1C /. Using a shifted inverse iteration
with  D 1 C 2 =3, we get a rate of

j1 C  .1 C 2 =3/j j =3j
D D 1=2;
j1  .1 C 2 =3/j j  2 =3j
and again the rate can be improved by moving  close to 1 C . In both examples, we
can also compute the second eigenvalue by choosing  appropriately.
In a practical implementation of the inverse iteration, it is frequently more efficient
to handle the inverse .A  I /1 implicitly: instead of computing a D .A  I /1 x,
which would require the inverse, we solve the linear system .A  I /a D x for the
unknown vector a. Since each step of the iteration requires us to solve one of these
systems for another right-hand side vector, it is usually a good idea to prepare the
matrix A  I in advance, e.g., by computing a suitable factorization. The resulting
algorithm is summarized in Figure 4.3.
The stopping criterion relies on the residual given by

O .m/ WD h.A  I /1 x .m/ ; x .m/ i;

br .m/ WD O .m/ x .m/  .A  I /1 x .m/ for all m 2 N0

instead of (4.8), since this vector can be computed without additional matrix-vector
multiplications.
76 Chapter 4 Power methods

r .m/ k  jO .m/ j. Then we can find

Corollary 4.12 (Accuracy). Let 2 Œ0; 1/ and kb
an eigenvalue  2 .A/ and x 2 E.A; / in the corresponding eigenspace such that
j  .m/ j  j  j;
  
.m/  j  j
kx  x k  2 1 C 1C :
1
A ./

A WD .A  I /1 in order to find O 2 .b

Proof. We apply Corollary 4.10 to b A/ and
x 2 E.b O
A; / such that
  jj
O
jO  O .m/ j  jO .m/ j; kx  x .m/ k  2 1 C : (4.14)
O
1 
./
b
A

Due to (4.11), we can find  2 .A/ satisfying O D 1=.  / and obtain

ˇ ˇ
ˇ 1 ˇ
ˇ D j.
1 .m/  /  .  /j
O O
j   j D ˇˇ
.m/
 .m/
   ˇ j.m/  j j  j
j  .m/ j
D ;
j.m/  j j  j
ˇ ˇ
ˇ 1 ˇ j  j
jO .m/ j D ˇˇ .m/ ˇD
ˇ :
  j.m/  j j  j
This already implies the first estimate:
j  .m/ j D jO  O .m/ j j.m/  j j  j  jO .m/ j j.m/  j j  j D j  j:
An estimate for the eigenvector can be obtained by investigating the spectral gap for
A: by definition, we find an eigenvalue O 0 2 .b
b O such that
A/ n ¹º

b O D jO  O 0 j:
./
A

Using (4.11) again, we find an eigenvalue 0 2 .A/ n ¹º such that O 0 D 1=.0  /.
This means
ˇ ˇ
ˇ 1 1 ˇˇ j.0  /  .  /j j0  j

b . O
/ D j O
  O
 0
j D ˇ  D D ;
A ˇ    0   ˇ j  j j0  j j  j j0  j
and (4.14) yields
  jjO   j  j j0  j
kx  x .m/ k  2 1 C D 2 1 C
1 
./O 1  j  j j0  j
b
A
  j0  j   j0  j C j  j
D 2 1 C  2 1 C
1  j0  j 1 j0  j
  
 j  j
 2 1 C 1C :
1
A ./
Section 4.5 Rayleigh iteration 77

4.5 Rayleigh iteration

According to Corollary 4.11, the shifted inverse iteration converges rapidly if the shift
parameter  is close to an eigenvalue. Since Theorem 4.6 states that the Rayleigh
quotient yields an approximation of an eigenvalue, using the Rayleigh quotient as
shift should lead to a fast method. The resulting algorithm is called the Rayleigh
iteration, and its iteration vectors are given by
hAx; xi
.m/ WD ; x .mC1/ WD .A  .m/ I /1 x .m/ for all m 2 N0 : (4.15)
hx; xi
In practical implementations, we have to ensure that the entries of the iteration vectors
do not become too large or too small by dividing x .mC1/ by its norm. As in previous
algorithms, this also allows us to avoid dividing by the norm when computing the
Rayleigh quotient. Using the residual-based stopping criterion already employed for
the power iteration, we obtain the algorithm given in Figure 4.4.
Compared to the shifted inverse iteration, the Rayleigh iteration typically requires
more operations per step: the shift changes in each step, therefore we have to solve
linear systems with different matrices that cannot be prepared in advance. On the
other hand, the Rayleigh iteration can be significantly faster:

Proposition 4.13 (Convergence). Let A  .0/ I be invertible, let (4.2) and (4.3) hold
with
j1  .0/ j < j2  .0/ j      jn  .0/ j: (4.16)
Let e1 WD Qı1 and let x .1/ be given by (4.15). If there is a constant c 2 R>0 such
that s
j1  2 j
tan †.x .0/ ; e1 /  c 
2kA  1 I k

holds, we have tan †.x .1/ ; e1 /  c and

kA  1 I k
tan †.x .1/ ; e1 /  2 tan3 †.x .0/ ; e1 /;
j1  2 j
i.e., for a sufficiently good initial vector, the Rayleigh iteration is cubically convergent.

Proof. Since .0/ D ƒA .x .0/ / by definition, Theorem 4.6 yields

j.0/  1 j  kA  1 I k tan2 †.x .0/ ; e1 /;

since we have assumed A to be normal (or even self-adjoint), and we also find

j.0/  2 j
j1  2 j  j.0/  1 j
j1  2 j  kA  1 I k tan2 †.x .0/ ; e1 /

j1  2 j  kA  1 I kc 2
j1  2 j  j1  2 j=2 D j1  2 j=2:
78 Chapter 4 Power methods

procedure rayleigh_iteration(A, var , x);

begin
Solve .A  I /y D x;
x y=kyk;
a Ax;
 ha; xi;
while ka  xk > jj do begin
Solve .A  I /y D x;
x y=kyk;
a Ax;
 ha; xi
end
end
Figure 4.4. Rayleigh iteration.

Combining these estimates with Corollary 4.11 for  D .0/ yields

j1  .0/ j
tan †.x .1/ ; e1 /  tan †.x .0/ ; e1 /
j2  .0/ j
kA  1 I k tan2 †.x .0/ ; e1 /
 tan †.x .0/ ; e1 /
j1  2 j=2
kA  1 I k
D2 tan3 †.x .0/ ; e1 /
j1  2 j
kA  1 I k j1  2 j
2 tan †.x .0/ ; e1 / D tan †.x .0/ ; e1 /  c:
j1  2 j 2kA  1 I k
These are the required estimates.

Example 4.14 (One-dimensional model problem). The advantages of the Rayleigh

iteration can be illustrated using the model problem (1.2). We let c D 1 and n D 1000,
.0/
use the starting vector given by xj D j=n, and compute the second eigenvalue by
a shifted inverse iteration using the shift parameter  D 30 and a Rayleigh iteration
starting with this initial shift. We have

1 D 4h2 sin2 .2h=2/  39:478; 2 D 4h2 sin2 .h=2/  9:8696

Section 4.6 Convergence to invariant subspace 79

and obtain the following results:

Inverse it. Rayleigh

m kr .m/ k Ratio kr .m/ k Ratio
1 1:72  10C1 1:72  10 C1

2 1:10  10C1 0:64 1:44  10C1 0:84

3 5:93  10C0 0:54 1:13  10C1 0:78
4 2:89  10C0 0:49 2:91  10C0 0:26
5 1:37  10C0 0:47 2:89  10 2 9:9  103
6 6:45  101 0:47 2:73  10 8 9:5  107
7 3:04  101 0:47 conv.
8 1:43  101 0:47

As we can see, the inverse iteration converges at a rate of approximately 0:47, which
coincides with the expected value of j1  j=j2  j. The Rayleigh iteration shows
inferior convergence rates during the first few steps, but once cubic convergence takes
hold in the fourth step, machine accuracy is attained rapidly.

As with most non-linear iterations, the performance of the Rayleigh iteration

depends crucially on the choice of the starting vector. If an unsuitable vector
is chosen, the method may converge to an eigenspace corresponding to another
eigenvalue, or it may fail to converge at all.

4.6 Convergence to invariant subspace

The convergence analysis presented so far relies on the dominance of one simple
eigenvalue, the case of multiple eigenvalues with identical modulus has not been ad-
dressed. Since the investigation of the inverse iteration and the Rayleigh iteration
relies on the convergence result provided by Theorem 4.2 for the power iteration, we
focus on this simple case.
Instead of assuming that one eigenvalue is dominant, we assume that there is a
k 2 ¹1; : : : ; n  1º such that

j1 j
  
jk j > jkC1 j
  
jn j (4.17)

holds, i.e., we allow k eigenvalues to have the same norm. In this case, we cannot
expect convergence to the eigenspace corresponding to 1 , but we can still prove
convergence to an invariant subspace.
80 Chapter 4 Power methods

Definition 4.15 (Angle between a vector and a subspace). Let x 2 F n n ¹0º, and let
Y  F n be a non-trivial subspace. We define

cos †.x; Y/ WD max¹cos †.x; y/ W y 2 Y n ¹0ºº;

sin †.x; Y/ WD min¹sin †.x; y/ W y 2 Y n ¹0ºº;
sin †.x; Y/
tan †.x; Y/ WD :
cos †.x; Y/

If cos †.x; Y/ D 0 holds, we let tan †.x; Y/ D 1.

In order to compute the angle between a vector and a subspace, the minimization
property of Lemma 4.5 is useful: we have

sin †.x; Y/ D min¹sin †.x; y/ W y 2 Yº

² ² ³ ³
kx  ˛yk
D min min W ˛2F W y2Y
kxk
² ³
kx  yk
D min W y2Y for all x 2 F n n ¹0º;
kxk
since Y is a subspace and therefore invariant under scaling. If we can find the best
approximation of x in Y, we can compute the angle. Such best approximations can
be conveniently described by appropriate projection operators:

Definition 4.16 (Orthogonal projection). Let P 2 F nn . If P 2 D P holds, P is

called a projection. If P is a selfadjoint projection, i.e., if P 2 D P D P  holds, we
call it an orthogonal projection. If Y  F n is a subspace such that Y D R.P /, we
call P a projection onto Y.

Exercise 4.17 (Factorized projection). Let m 2 N, and let V 2 F nm be an

isometric matrix (cf. Definition 2.37). Prove that P WD V V  is an orthogonal
projection onto R.V /.

Proposition 4.18 (Orthogonal projection). Let Y  F n be a subspace, and let P 2

F nn . P is an orthogonal projection onto Y if and only if R.P /  Y and

hx; yi D hP x; yi for all x 2 F n ; y 2 Y: (4.18)

In this case, we have

kx  P xk  kx  yk for all y 2 Y; (4.19a)

2 2 2
kx  P xk D kxk  kP xk (4.19b)

i.e., P x 2 Y is the best approximation of x in Y.

Section 4.6 Convergence to invariant subspace 81

Proof. Let P be an orthogonal projection onto Y. Let x 2 F n and y 2 Y. By

definition, we have y 2 R.P /, i.e., we can find z 2 F n such that y D P z. This
implies

hx; yi D hx; P zi D hx; P 2 zi D hP  x; P zi D hP x; P zi D hP x; yi:

Let now P 2 F nn be a matrix satisfying (4.18). We first consider y 2 Y and find

ky  P yk2 D hy  P y; y  P yi D hy; yi  hP y; yi  hy; P yi C hP y; P yi

D .hy; yi  hP y; yi/  .hy; P yi  hP y; P yi/ D 0

due to P y 2 R.P /  Y. This implies P y D y for all Y and therefore R.P / D Y.

Let x 2 F n . Due to the previous equation, we have y WD P x 2 R.P / D Y and
2 x D P y D y D P x, and therefore P 2 D P .
P
Let x; y 2 F n . Due to P x; P y 2 R.P / D Y, we find

hx; P yi D hP x; P yi D hP y; P xi D hy; P xi D hP x; yi;

and this implies P D P  by (2.11). We have proven that P is an orthogonal projec-

tion.
Let now x 2 F n and y 2 Y. We have z WD P x  y 2 Y, and (4.18) yields

kx  yk2 D kx  P x C P x  yk2 D h.x  P x/ C z; .x  P x/ C zi

D hx  P x; x  P xi C hx  P x; zi C hz; x  P xi C hz; zi
D kx  P xk2 C hx  P x; zi C hx  P x; zi C kzk2
D kx  P xk2 C kzk2 ;

and this implies (4.19a) due to kzk2

0. Applying the equation to the special choice
y WD 0 yields (4.19b).

If we have an orthogonal projection at our disposal, we can find explicit equations

for the angle between a vector and the range of the projection.

Proposition 4.19 (Angle characterized by projection). Let x 2 F n n ¹0º, let Y  F n

be a non-trivial subspace, and let P 2 F nn be an orthogonal projection onto this
subspace. Then we have (assuming P x ¤ 0 for the third equation)

kP xk kx  P xk kx  P xk
cos †.x; Y/ D ; sin †.x; Y/ D ; tan †.x; Y/ D :
kxk kxk kP xk
Proof. According to Definitions 4.1 and 4.15 and Lemma 4.5, we have
² ³
kx  yk
sin †.x; Y/ D min¹sin †.x; y/ W y 2 Y n ¹0ºº D min W y 2 Y n ¹0º :
kxk
82 Chapter 4 Power methods

Since P is an orthogonal projection onto Y, Proposition 4.18 can be applied, and

(4.19a) yields
² ³
kx  yk kx  P xk
sin †.x; Y/ D min W y 2 Y n ¹0º D :
kxk kxk

Using sin2 †.x; y/ C cos2 †.x; y/ D 1 and (4.19b), we obtain

kx  P xk2 kxk2  kx  P xk2

cos2 †.x; Y/ D 1  sin2 †.x; Y/ D 1  D
kxk 2 kxk2
kxk2  .kxk2  kP xk2 / kP xk2
D D ;
kxk2 kxk2
and taking the square root gives us the desired equation for the cosine. Applying the
definition of the tangent completes the proof.

Theorem 4.20 (Convergence). Let (4.2) and (4.3) hold with

j1 j
  
jk j > jkC1 j
  
jn j: (4.20)

Let Ek WD span¹Qıj W j 2 ¹1; : : : ; kºº denote the invariant subspace spanned by

the first k columns of Q, and let cos †.x .0/ ; Ek / ¤ 0. Let .x .m/ /1
mD0 be given by
(4.1). Then we have
jkC1 j
tan †.x .mC1/ ; Ek /  tan †.x .m/ ; Ek / for all m 2 N0 ;
jk j
i.e., the angle between the iteration vectors and the subspace Ek converges to zero.
b 2 F nn by
Proof. We define the matrix P
 
b I
P D k ;
0

where Ik 2 F kk denotes the k-dimensional identity matrix, and let

b Q :
P WD QP (4.21)

Due to

P 2 D QPb Q Q Pb Q D Q P bP
b Q D Q P
b Q D P;
P  D .QPb Q  / D Q P
b  Q D QPb Q D P;

the matrix P is an orthogonal projection, and R.P b/ D b E k WD span¹ıj W j 2

¹1; : : : ; kºº implies R.P / D Ek . We conclude that P is an orthogonal projection into
Ek , so we can use Proposition 4.19 to handle the angle.
Section 4.7 Simultaneous iteration 83

We can proceed as in the proof of Theorem 4.2: we let

x .m/ WD Q x .m/
b for all m 2 N0 :

b x .0/ ¤ 0 and
Due to cos †.x .0/ ; Ek / ¤ 0, we have P x .0/ ¤ 0 and therefore Pb
observe
kx .m/  P x .m/ k kQbx .m/  QP b Q Qb x .m/ k
tan †.x .m/ ; Ek / D D
kP x .m/ k kQP b Q Qb x .m/ k
b x .m/ k
x .m/  Pb
kb
D D tan †.bx .m/ ; b
E k / for all m 2 N0 (4.22)
b
kPbx k
.m/

due to (2.16). The tangent of the transformed vector and subspace can be computed
directly: Proposition 4.19 combined with the definition of the norm yields
Pn
kb b y k2
y  Pb j DkC1 jbyj j2
2
tan †.b b
y; E k/ D D Pk for all b b y ¤ 0;
y 2 F n with Pb
b y k2
kPb jb
y j2
j D1 j

and this results in

Pn 2 x .m/ j2
2 .mC1/ b 2 .m/ b j DkC1 jj j jb j
tan †.b
x ; E k / D tan †.Db x ; Ek/ D P
k .m/ 2
j D1 jj j jb
xj j
2
P .m/   Pn
n
jkC1 j2 j DkC1 jb xj j2 jkC1 j 2 j DkC1 jb xj.m/ j2
 P D Pk
jk j2 jkD1 jb
.m/
xj j2 jk j .m/
j D1 jb
xj j2
 
jkC1 j 2
D x .m/ ; b
tan2 †.b E k / for all m 2 N0 :
jk j
Using (4.22) gives us the final result.

If 1 is a multiple eigenvalue, i.e., if 1 D 2 D    D k holds, we have Ek D

E.A; 1 /, so the power iteration will compute an eigenvector for 1 .

4.7 Simultaneous iteration

The result of Theorem 4.20 is not immediately useful, since it yields only convergence
of the iteration vectors to the k-dimensional invariant subspace. In order to describe
the entire subspace, we need a basis, i.e., k linearly independent vectors. A simple
.0/ .0/
approach would be to pick k linearly independent starting vectors x1 ; : : : ; xk 2 F n
and compute the corresponding iteration vectors
.mC1/ .m/
xj WD Axj for all m 2 N0 ; j 2 ¹1; : : : ; kº
84 Chapter 4 Power methods

in parallel. We avoid the additional index by introducing matrices .X .m/ /1

mD0 in
.m/
F nk .m/
such that xj is the j -th column of X , i.e., such that

xj.m/ D X .m/ ıj for all m 2 N0 ; j 2 ¹1; : : : ; kº;

similar to the approach used in the proof of Proposition 2.19. The resulting iteration
takes the form

X .mC1/ D AX .m/ for all m 2 N0 : (4.23)

According to Theorem 4.20, the columns of the matrices X .m/ will converge to the
invariant subspace Ek , but they may stop being linearly independent, e.g., if one of
the columns lies in the nullspace of A.
Since we are interested in finding a basis, we have to avoid this problem: we impose
suitable starting conditions.

Proposition 4.21 (Linear independence). Let A, Q, D, 1 ; : : : ; n be as in Theo-

rem 4.20, and let P 2 F nn be the projection onto Ek defined by (4.21).
If PX .0/ is injective, i.e., if its columns are linearly independent, the same holds
for all PX .m/ with m 2 N0 . In particular, the columns of X .m/ are also linearly
independent, i.e., they span a k-dimensional subspace.

Proof. Let PX .0/ be injective. Following the approach of Theorem 4.2, we define
transformed matrices
b .m/ WD Q X .m/
X for all m 2 N0 :

b D Q PQ projects into the subspace F k  ¹0º of F n , it makes sense to

Since P
consider the restriction of our matrices to the first k rows and columns: we define
0 1
1
b WD Djkk D B
D @ ::
:
C
A;
k
 .m/ 
b
b.m/ WD X
Y b .m/ jkk ; P
bXb .m/ D Y for all m 2 N0 :
0

b is invertible. Since
Due to (4.17), we have j1 j
  
jk j > jkC1 j
0, so D
.0/
PX is injective, the matrix
 .0/ 
b
Y
DP bXb .0/ D Q PQQ X .0/ D Q PX .0/
0

b.0/ has to be injective.

is also injective, so Y
Section 4.7 Simultaneous iteration 85

b and D are diagonal matrices, we find

Let m 2 N0 . Since P
b Q X .m/ D QP
PX .m/ D QP bXb .m/ D QP
bDmXb .0/
 .0/   m .0/ 
b b Y b
bX
D QD m P b .0/ D QD m Y DQ
D
:
0 0

We have already seen that D b is invertible, so the same holds for D

b m . Since Y
b.0/ is
b Y
injective, the matrix D m b also has to be injective, and thus the matrix PX .
.0/ .m/

Although the matrices .X .m/ /1

mD0 will remain injective if PX
.0/ is, their columns

may become “numerically linearly dependent”, e.g., they could all converge to
E.A; 1 / if 1 is dominant. In order to avoid numerical instability, we guarantee
linear independence by orthogonalization.

Definition 4.22 (QR factorization). Let A 2 F nm and k WD min¹n; mº. If an iso-
metric matrix Q 2 F nk and an upper triangular matrix R 2 F km satisfy

A D QR;

we call the representation of A as the product QR a (skinny) QR factorization.

Proposition 4.23 (QR factorization). For every matrix A 2 F nm , there is a QR

factorization A D QR.

Proof. (cf. [43, Section 4.1]) By induction on n 2 N. The case n D 1 is trivial,

since A is already upper triangular.
Let now n 2 N and assume that for every matrix A 2 F nm with any m 2 N0 ,
there is a QR factorization. Let m 2 N0 , and let A 2 F .nC1/m . If m D 0, we are
done.
Otherwise we can use Lemma 2.45 to find a unitary matrix Q1 2 F .nC1/.nC1/
with Q1 D Q1 mapping the first column of A to a multiple of the first canonical unit
vector. If m D 1, this yields
   
r 1
Q1 A D 11 ; A D Q1 r D QR
0 0 11

with the matrices

 
1  
Q WD Q1 ; R WD r11 :
0

For m > 1, on the other hand, we have

 
r11 R1
Q1 A D b
A
86 Chapter 4 Power methods

with r11 2 F , R1 2 F 1.m1/ and bA 2 F n.m1/ . By the induction assumption,

O O
b 2 F k.m1/
b 2 F nk and an upper triangular matrix R
we find an isometric matrix Q
with kO D min¹n; m  1º and b
ADQ b R.
b We obtain
      
r11 R1 r11 R1 1 r11 R1
Q1 A D b D bRb D b b
A Q Q R

and conclude   
1 r11 R1
A D Q1 b
Q b D QR
R
with the matrices
   
1 O
.nC1/.kC1/ r11 R1 O
Q WD Q1 b 2F ; R WD b 2 F .kC1/m :
Q R

Observing k D min¹nC1; mº D min¹n; m1ºC1 D kO C1 completes the proof.

This implies that we can find isometric matrices .Q.m/ /1

mD0 in F
nk and upper

triangular matrices .R.m/ /1

mD0 in F
kk such that

X .m/ D Q.m/ R.m/ for all m 2 N0 : (4.24)

In order to avoid numerical instabilities, we do not work directly with X .m/ : in a first
step, we compute Q.0/ and R.0/ satisfying (4.24) directly. If Q.m/ and R.m/ have
been computed, we have

X .mC1/ D AX .m/ D AQ.m/ R.m/ :

We compute a skinny QR factorization of

Y .mC1/ WD AQ.m/ ;
b.mC1/ 2 F kk
i.e., we find an isometric Q.mC1/ 2 F nk and an upper triangular R
such that
Y .mC1/ D Q.mC1/ Rb.mC1/ :
This implies
b.mC1/ R.m/ ;
X .mC1/ D AQ.m/ R.m/ D Y .mC1/ R.m/ D Q.mC1/ R

and by choosing
b.mC1/ R.m/ ;
R.mC1/ WD R
we can obtain the required factorization without computing X .mC1/ explicitly. The
resulting algorithm is called the simultaneous iteration (sometimes also subspace it-
eration or orthogonal iteration) and summarized in Figure 4.5.
Section 4.7 Simultaneous iteration 87

procedure simultaneous_iteration(A, var Q);

begin
while „not sufficiently accurate“ do begin
Y AQ;
Compute skinny QR factorization Y D QR b
end
end
Figure 4.5. Simultaneous iteration, still missing a stopping criterion.

For k D 1, the skinny QR decomposition coincides with dividing a one-column

matrix by the norm of this column, so the simultaneous iteration is a generalization of
the simple power iteration to the case of multiple vectors.
We can generalize the stopping criterion used for the power iteration: instead of
computing the Rayleigh quotient

.m/ WD hAx .m/ ; x .m/ i for all m 2 N0 ;

(recall that we have kx .m/ k D 1 in the practical power iteration), we compute the
k  k matrix

ƒ.m/ WD .Q.m/ / AQ.m/ for all m 2 N0 ; (4.25)

and instead of considering the norm of the residual vector r .m/ D Ax .m/  .m/ x .m/ ,
we use the norm of the residual matrix

R.m/ WD Q.m/ ƒ.m/  AQ.m/ for all m 2 N0 :

Note that this is not the same matrix as the one used in (4.24), we only use the same
letter, for obvious reasons. The resulting practical simultaneous iteration is given in
Figure 4.6. The error analysis of Section 4.3 can be extended to this case:

Exercise 4.24 (Rank-2k matrix). Let C; D 2 F nk with D  C D 0, and let

X WD CD  C DC  . Prove kX k2F  2kC k2F kDk2F and kXk  kC k kDk.
(Hint: Consider the Cauchy–Schwarz inequality for the first estimate and the
proof of Lemma 4.8 for the second).

Exercise 4.25 (Eigenvalues). Let m 2 N0 and  2 .ƒ.m/ /. Prove that there is

a  2 .A/ such that
j  j  kR.m/ k:
(Hint: Consider the proof of Theorem 4.9).
88 Chapter 4 Power methods

procedure simultaneous_iteration(A, var Q);

begin
Y AQ;
ƒ Q Y ;
while kY  Qƒk > do begin
Compute skinny QR factorization Y D QR; b
Y AQ;
ƒ Q Y
end
end
Figure 4.6. Practical simultaneous iteration.

Due to Theorem 4.20, we know that the angles between the columns of the matrices
X .m/ and the invariant subspace Ek spanned by the eigenvectors corresponding to the
first k eigenvalues will converge to zero. Proposition 4.21 ensures that the columns of
X .m/ span the same subspace as those of Q.m/ , so the convergence result holds for
the latter as well.
In view of Proposition 2.43, we can characterize “convergence to an invariant sub-
space” also in terms of equation (2.20) and obtain the following result.

Theorem 4.26 (Convergence to invariant subspace). Let (4.2) and (4.3) hold with
j1 j
  
jk j > jkC1 j
  
jn j:
Let P 2 F nn be the projection onto Ek defined by (4.21).
Let X .0/ 2 F nk , and assume that PX .0/ is injective. We can find a matrix ƒ 2
F kk such that
 
k.AX .m/  X .m/ ƒ/yk jkC1 j m k.AX .0/  X .0/ ƒ/yk

kPX .m/ yk jk j kPX .0/ yk
for all y 2 F k n ¹0º and all m 2 N0 :
Proof. As in the previous proofs, we introduce transformed matrices
b .m/ WD Q X .m/
X for all m 2 N0
and observe
b .mC1/ D Q X .mC1/ D Q AX .m/ D Q AQX
X b .m/ D D X
b .m/ for all m 2 N0 :
b
We define X
.m/ b .m/ 2 F .nk/k by
2 F kk and X
k ?
!
b
X
.m/
b .m/ D
X k for all m 2 N0 :
b .m/
X ?
Section 4.7 Simultaneous iteration 89

Due to (4.21) we have

    ! !
I I b .0/
X b .0/
X

Q PX .0/  b
D Q PQX D k
.0/ b
X D k
.0/ k D k ;
0 0 b .0/
X 0
?

b .0/ has to be invertible.

and since PX .0/ is injective, the matrix X k
We split the diagonal matrix D into
0 1 0 1
1 kC1
B : C B :: C
Dk WD @ :: A ; D? WD @ : A
k n
and obtain
AX .m/  X .m/ ƒ D QDQ X .m/  QQ X .m/ ƒ D Q.D X b .m/  Xb .m/ ƒ/
b .0/  D m X
D Q.D mC1 X b .0/ ƒ/
!
b .0/  D m X
DkmC1 X b .0/ ƒ
DQ k
mC1 b .0/
k k
m b .0/
for all m 2 N0 ; ƒ 2 F kk :
D? X ?  D? X? ƒ

b is invertible, we can eliminate the first row by using

Since X
.0/
k

b .0/
b .0/ /1 Dk X
ƒ WD .X k k

and use (2.16) and (3.11) to conclude

 !
 
 0 
k.AX .m/  X .m/ ƒ/yk D  m .D b .0/ b .0/ 
 D? ?X ?  X ? ƒ/y 
b .0/  X
 kD? km k.D? X b .0/ ƒ/yk
? ?
b .0/  X
D jkC1 jm k.D? X b .0/ ƒ/yk
? ?
for all m 2 N0 ; y 2 F k :
This yields the numerator of the required estimate. For the denominator, we use
kPX .m/ yk D kQPb Q X .m/ yk D kQPbX
b .m/ yk D kQP b .0/ yk
bDmX
  m  ! 
 b .0/ 
 Ik Dk X 
D k y 
 0 m
D? b .0/
X 
?

D b .0/ yk
kDkm X b .0/ yk

jk j kX m
for all y 2 F k ; m 2 N0 ;
k k

where we have used

0 11=2 0 11=2
Xk Xk
kDk zk D @ ji j2 jzi j2 A
@ jk j2 jzi j2 A D jk j kzk for all z 2 F k
iD1 i D1
90 Chapter 4 Power methods

b .0/ , we have
in the last step. Using the definitions of P and X

kX bX
b .0/ yk D kP b .0/ yk D kPX .0/ yk for all m 2 N0 ; y 2 F k ;
k

and this yields the required estimate for the denominator.

During the simultaneous iteration, we do not compute X .m/ but the isometric ma-
trices Q.m/ . For these matrices, the convergence result takes a particularly simple
form.

Corollary 4.27 (Invariant subspace). Let the conditions of Theorem 4.26 hold, let
.Q.m/ /1
mD0 and .R
.m/ /1
mD0 be as in (4.24), and let .ƒ
.m/ /1
mD0 be as in (4.25). Then
we have
 
.m/ .m/ .m/ jkC1 j m
kAQ  Q ƒ k  C0 for all m 2 N0 ;
jk j
where the constant is given by
´ μ
k.AX .0/  X .0/ ƒ/yk k
C0 WD max W y 2 F n ¹0º :
kPX .0/ yk

Proof. Let m 2 N0 . By definition, we have

AX .m/  X .m/ ƒ D AQ.m/ R.m/  Q.m/ R.m/ ƒ;

and Proposition 4.21 guarantees that R .m/ is invertible, so we obtain

AX .m/  X .m/ ƒ D AQ.m/ R.m/  Q.m/ R.m/ ƒ.R.m/ /1 R.m/

e.m/ /R.m/
D .AQ.m/  Q.m/ ƒ

with the matrix

e.m/ WD R.m/ ƒ.R.m/ /1 :
ƒ
Let z 2 F k n ¹0º, and let y WD .R.m/ /1 z. Since (4.19b) and (2.16) imply kPQ.m/ zk
 kQ .m/ zk D kzk, we find
e.m/ /zk
k.AQ.m/  Q.m/ ƒ k.AQ.m/  Q.m/ ƒe .m/ /zk

kzk kPQ.m/ zk
.m/  Q .m/ e .m/ /R.m/ yk
k.AQ ƒ
D
kPQ R.m/ yk
.m/

k.AQ.m/ R.m/  Q.m/ R.m/ ƒ.R.m/ /1 R.m/ /yk

D
kPQ.m/ R.m/ yk
k.AX .m/  X .m/ ƒ/yk
D :
kPX .m/ yk
Section 4.7 Convergence for general matrices 91

Applying Theorem 4.26 and the Definition 3.5 yields

 m
kAQ.m/  Q.m/ ƒ e.m/ k  C0 jkC1 j : (4.26)
jk j

Now we only have to replace ƒ e .m/ by ƒ.m/ . To this end let … WD Q.m/ .Q.m/ / .
Since Q .m/ is isometric, we have …2 D Q.m/ .Q.m/ / Q.m/ .Q.m/ / D Q.m/ .Q.m/ /
D … and … D …, so … is an orthogonal projection onto the range R.Q.m/ /
of Q.m/ . Let z 2 F k . Due to Q.m/ ƒ e.m/ z 2 R.Q.m/ /, we can use the best-
approximation property (4.19a) to obtain

k.AQ .m/  Q.m/ ƒ.m/ /zk D kAQ.m/ z  Q.m/ .Q.m/ / AQ.m/ zk

e.m/ zk
D kAQ .m/ z  …AQ.m/ zk  kAQ.m/ z  Q.m/ ƒ
e.m/ /zk
D k.AQ.m/  Q.m/ ƒ
e.m/ k kzk
 kAQ.m/  Q.m/ ƒ

and using the intermediate result (4.26) in combination with the Definition 3.5 of the
spectral norm completes the proof.

4.8 Convergence for general matrices 

The convergence analysis presented so far relies on the diagonalizability of the ma-
trix A. In this section, we consider the convergence of the power iteration for non-
diagonalizable matrices. In practice, a non-diagonalizable matrix can be represented
as a limit of diagonalizable matrices, since we can always add a small perturbation
that ensures that all eigenvalues are single. Since this approach would require a sub-
stantial generalization of the perturbation theory presented so far, we prefer to handle
the non-diagonalizable case directly using the results of Section 2.7 to introduce a
block-diagonal representation of A.
Let A 2 C nn . According to Theorem 2.58, we can find an invertible matrix
B 2 C nn and upper triangular matrices R1 2 C n1 n1 ; : : : ; Rk 2 C nk nk such that
0 1
R1
B :: C 1
ADB@ : AB (4.27)
Rk

and .R` / D ¹` º for all ` 2 ¹1; : : : ; kº. This decomposition can take the place
of the diagonalization (4.2) we relied on in the analysis of the power iteration in the
diagonalizable case: the matrix is block-diagonalizable.
We have to investigate the matrices R` more closely. Let ` 2 ¹1; : : : ; kº. Since
R` is upper triangular, Proposition 2.13 implies that the diagonal elements of R` are
its eigenvalues. Due to .R` / D ¹` º, all diagonal elements have to equal ` . This
92 Chapter 4 Power methods

suggests a decomposition of R` into the diagonal part and the remainder, and the
remainder will be a special kind of triangular matrix.

Definition 4.28 (˛-triangular matrix). Let N 2 F nn and ˛ 2 N0 . We call N ˛-

upper triangular, if

nij D 0 holds for all i; j 2 ¹1; : : : ; nº with j < i C ˛: (4.28)

A 1-upper triangular matrix is also called strictly upper triangular.

We split the matrix R` into the diagonal part ` I and a strictly upper triangular
matrix N` 2 C n` n` :
R` D ` I C N` :
The vectors computed during the power iteration result from applying powers of
the matrix to the starting vector, therefore we have to investigate the powers of R` .
Since I and N` commute, these powers are given by
!
Xm
m mj j
m m
R` D .` I C N` / D ` N` for all m 2 N0 :
j
j D0

We can simplify this equations significantly by taking advantage of the properties of

triangular matrices:

Lemma 4.29 (Products of triangular matrices). Let N; M 2 F nn be ˛- and ˇ-upper

triangular, respectively. Then the product NM is .˛ C ˇ/-upper triangular.

Proof. We use contraposition: let i; j 2 ¹1; : : : ; nº be given such that .NM /ij ¤ 0.
Due to
X n
0 ¤ .NM /ij D nik mkj ;
kD1
there has to be at least one index k 2 ¹1; : : : ; nº with nik ¤ 0 and mkj ¤ 0. Since
N is ˛-upper triangular and M is ˇ-upper triangular, this implies k
i C ˛ and
j
k C ˇ due to (4.28). Combining both estimates gives us j
k C ˇ
i C ˛ C ˇ.
We conclude that .NM /ij ¤ 0 implies j
i C ˛ C ˇ, and contraposition yields that
j < i C ˛ C ˇ implies .NM /ij D 0, so NM has to be .˛ C ˇ/-upper triangular.

Since N` is strictly upper triangular, Lemma 4.29 and a simple induction yield
that N`n` is n` -upper triangular, and since the matrix is of size n`  n` , this implies
N`n` D 0. Therefore the m-th power of R` takes the form
!
` 1
nX
m mj j
R`m D ` N` for all m 2 Nn` 1 :
j
j D0
Section 4.8 Convergence for general matrices 93

In order to obtain a generalized version of Theorem 4.2, we have to find lower and
upper bounds for matrices of this type.

Lemma 4.30 (Bounds for iteration vectors). Let  2 F , let N 2 F nn be a strictly
upper triangular matrix, and let R D I C N .
Let x 2 F n . There are a constant c 2 R>0 and a polynomial p of degree not higher
than n  1 such that
cjjm kxk  kRm xk  p.m/jjm.n1/ kxk for all m 2 Nn1 :
Proof. Since the result is trivial for x D 0, we consider only the case x ¤ 0. Let
m 2 Nn , P WD Rm and y WD P x D Rm x. Due to Lemma 4.29, the product
of upper triangular matrices is upper triangular, and a simple induction yields that
P WD Rm is also upper triangular.
We first prove the lower bound. Since x ¤ 0, we can find an index k such that
xk ¤ 0. The largest such index is given by
` WD max¹k 2 ¹1; : : : ; nº W xk ¤ 0º:
Since P is upper triangular and xk D 0 for all k > `, we have
X
n X
n
y` D .P x/` D p`j xj D p`j xj D p`` x` :
j D1 j D`

Since I and N commute, the binomial theorem yields

!
X
m
m
P D Rm D .I C N /m D mj N j : (4.29)
j
j D0

N is strictly upper triangular, and the same holds for N j with j > 0 by Lemma 4.29.
Therefore the only term in (4.29) contributing to the diagonal of P is the term for
j D 0, i.e., m I . We conclude that all diagonal elements of P equal m , and in
particular
y` D p`` x` D m x` :
Since x` ¤ 0, the constant
jx` j
c WD >0
kxk
is well-defined, and we obtain
kRm xk D kyk
jy` j D jjm jx` j D cjjm kxk:
Now we consider the upper bound. Applying the triangle inequality to (4.29) yields
 !  !
X 
 m m mj j  X m
m
kyk D kP xk D    N x  jjmj kN j xk:
j D0 j  j D0 j
94 Chapter 4 Power methods

Since N is strictly upper triangular, Lemma 4.29 yields that N n is n-upper triangular,
i.e., N n D 0, and therefore
!
X m
n1
kyk  jjmj kN j xk:
j
j D0

Using the compatibility (3.11) of the norms, we conclude

! !
X m
n1 X m
n1
mj j m.n1/
kyk  jj kN k kxk D jj jjn1j kN kj kxk:
j j
j D0 j D0
(4.30)
Now we only have to prove that the sum can be expressed by a polynomial in m. Due
to
! Qm
m mŠ kDmj C1 k
Yj
kCmj
D D Qj D for all j 2 ¹0; : : : ; n  1º;
j .m  j /Šj Š k k
kD1 kD1

the polynomials

Y
j
kCzj
pj W F ! F; z 7! ;
k
kD1
 
satisfy pj .m/ D m j for all m 2 Nn and j 2 ¹0; : : : ; n  1º.
Since each pj is the product of j linear factors, its order is bounded by j , and

X
n1
p W F ! F; z 7! pj .z/jjn1j kN kj ;
j D0

is a polynomial of degree not larger than n  1 satisfying

!
X m
n1
p.m/ D jjn1j kN kj
j
j D0

for all m. Combining this equation with (4.30) yields

!
X m
n1
m.n1/
kyk  jj jjn1j kN kj kxk D jjm.n1/ p.m/kxk;
j
j D0

and this is the required upper bound.

Section 4.8 Convergence for general matrices 95

Note that the constant c of Lemma 4.30 depends on the vector x, particularly on
the ratio between its last non-zero entry and its Euclidean norm.

We can apply this Lemma to the decomposition (4.27) if we assume that one eigen-
value is dominant.

Theorem 4.31 (Convergence). Let A 2 C nn , and let B; R1 ; : : : ; Rk ; n1 ; : : : ; nk and

1 ; : : : ; k be as in (4.27). Let

j1 j > j2 j

  
jk j; (4.31)

and let
b
G 1 WD C n1  ¹0º  C n ; G1 WD B b
G 1:

G1 is the invariant subspace of generalized eigenvectors (cf. Definition 2.59) for the
dominant eigenvalue 1 . Let x .0/ 2 C n n ¹0º satisfy cos †.B 1 x .0/ ; b
G 1 / ¤ 0. Let
.x .m/ /1
mD0 be the iteration vectors of the usual power iteration given by (4.1). Then
there is a polynomial p of degree not higher than nO WD max¹n2  1; : : : ; nk  1º such
that
 
.m/ j2 j mnO
sin †.x ; G1 /  p.m/ for all m 2 NnO ;
j1 j

i.e., the angle between the iteration vectors and the subspace G1 converges to zero.

Proof. We follow the approach already used in the proof of Theorem 4.20: we define

x .m/ WD B 1 x .m/
b for all m 2 N0

and observe
0 1
R1
B :: C .m/
x .mC1/ D @
b : Ab
x for all m 2 N0 : (4.32)
Rk

.m/
We define b
x` 2 C n` for all m 2 N0 and ` 2 ¹1; : : : ; kº by
0 .m/ 1
b
x1
B :: C
x .m/
b D@ : A for all m 2 N0 ;
.m/
b
xk
96 Chapter 4 Power methods

and (4.32) yields

.mC1/
b
x` x .m/
D R`b `
for all m 2 N0 ; ` 2 ¹1; : : : ; kº:

Straightforward inductions gives us

.m/ .0/
b
x` D R`mb
x` for all m 2 N0 ; ` 2 ¹1; : : : ; kº; (4.33)

so we can apply Lemma 4.30 to each of the subvectors.

We are interested in an upper bound of the angle between the transformed iteration
x .m/ and the subspace b
vectors b G 1 corresponding to the first eigenvalue 1 . In order
to be able to apply Proposition 4.19, we introduce the orthogonal projection
0 1
In1
B 0 C
b WD B C
P B : C 2 C nn
@ : : A
0

into the relevant subspace C n1  ¹0º  C n .

We first apply Lemma 4.30 to the block R1 . The Lemma yields a constant c1 2
R>0 , independent of m, such that

x .0/
c1 j1 jm kb m .0/
1 k  kR1 b
x1 k for all m 2 Nn1 :

Due to the definition (2.8) of the Euclidean norm and (4.33), this implies

b x .m/ k2 D kb
kPb
.m/
x 1 k2 D kR1mb
.0/
x 1 k2
c12 j1 j2m kb
x 1 k2
.0/

b x .0/ k2 for all m 2 NnO :

D c12 j1 j2m kPb (4.34)

By assumption and Proposition 4.19, we have kPb b x .0/ k D cos †.bx .0/ ; b
G 1 / ¤ 0, so
both sides of (4.34) are strictly positive. Now we apply Lemma 4.30 to the remaining
blocks R2 ; : : : ; Rk . We find polynomials p2 ; : : : ; pk , independent of m, such that

x .0/
kR`mb `
k  p` .m/j` jm.n` 1/ kb
x .0/
`
k for all m 2 Nn` 1 ; ` 2 ¹2; : : : ; kº:

Using again (2.8) and (4.33), we obtain

X
k X
k
b /b
k.I  P x .m/ k2 D
.m/
x ` k2 D
kb kR`mb
.0/
x ` k2
`D2 `D2
X
k
O .0/
 p` .m/2 j` j2.mn/ x ` k2
kb
`D2
Section 4.8 Convergence for general matrices 97

X
k
O
 p` .m/2 j2 j2.mn/ x .0/
kb `
k2
`D2
X
k
O .0/
 max¹p` .m/2 W ` 2 ¹2; : : : ; kººj2 j2.mn/ x ` k2
kb
`D2
for all m 2 NnO :

By taking the maximum of the absolute values of the coefficients of the polynomials
p2 ; : : : ; pk , we can construct a polynomial pmax of degree not larger than nO such that

jp` .m/j  pmax .m/ for all m 2 NnO ; ` 2 ¹2; : : : ; kº;

and conclude

X
k
b /b
k.I  P O
x .m/ k2  pmax .m/2 j2 j2.mn/ x .0/
kb k2
`
`D2
O
D pmax .m/2 j2 j2.mn/ b /b
k.I  P x .0/ k2 for all m 2 NnO :
(4.35)

Now we have to translate the estimates back to the original iteration vectors x .m/ . We
e WD B P
let P b B 1 and observe that it is a projection onto G1 , although unfortunately
not orthogonal in general. We denote the orthogonal projection onto G1 by P 2 C nn .
The best-approximation property (4.19a) and (3.11) yield

e /x .m/ k D kB.I  P
k.I  P /x .m/ k  k.I  P b /B 1 x .m/ k
D kB.I  Pb /bx .m/ k  kBk k.I  Pb /b
x .m/ k for all m 2 N0 ;

so we can apply (4.35) to obtain

b /b
k.I  P /x .m/ k  kBkpmax .m/j2 jmnO k.I  P x .0/ k for all m 2 NnO : (4.36)

Using (3.11) and (4.19b) yields

kB 1 k kx .m/ k
kB 1 x .m/ k D kb b x .m/ k
x .m/ k
kPb for all m 2 N0 ;

so we can apply (4.34) to obtain

b x .m/ k
kPb b x .0/ k
c1 j1 jm kPb
kx .m/ k

for all m 2 N0 : (4.37)
kB 1 k kB 1 k
98 Chapter 4 Power methods

Combining (4.36) and (4.37) with Proposition 4.19, we find

k.I  P /x .m/ k
sin †.x .m/ ; G1 / D
kx .m/ k
  b /b
kBk kB 1 k j2 j mnO k.I  P x .0/ k
 pmax .m/
c1 j1 jnO j1 j b x .0/ k
kPb
  O
kBk kB 1 k j2 j mn
D pmax .m/ x .0/ ; b
tan †.b G 1/
c1 j1 jnO j1 j
for all m 2 NnO ;
so setting
kBk kB 1 k
p.m/ WD x .0/ ; b
pmax .m/ tan †.b G 1/ for all m 2 N
c1 j1 jnO
gives us the required estimate.

Exercise 4.32 (Transformed vectors). Given the assumptions of Theorem 4.31,

prove that there is a polynomial pO of degree not above n such that
 
.m/ b j2 j mn
tan †.bx ; G 1 /  p.m/
O tan †.bx .0/ ; b
G 1 / for all m 2 Nn :
j1 j

Remark 4.33 (Linear convergence). Theorem 4.31 yields “almost” the same conver-
gence behaviour as Theorem 4.2, at least in the long run: let % WD j2 j=j1 j < 1
denote the rate of convergence for the diagonalizable case, and let %O 2 .%; 1/. Then
we have q WD %=%O < 1 and find
sin †.x .m/ ; G1 /  %mn p.m/ D %O mn q mn p.m/ for all m 2 Nn :
O WD m  n and p.
To simplify the expression, we introduce m O m/
O WD p.m
O C n/ and
obtain
O
sin †.x .mCn/ ; G1 /  %O mO q mO p.
O m/
O O 2 N0 :
for all m
Due to q < 1, we can find
2 R>0 such that exp.
/ D 1=q, and therefore
exp.
/ D q and q mO D exp.
m/.
O Since m
O is non-negative, we have
X
nC1
O `
.
m/
nC1
O

exp.
m/
O nC1
m O 2 N0
for all m
`Š .n C 1/Š
`D0

and conclude
O O m/
p. O .n C 1/Šp.
O m/
O
sin †.x .mCn/ ; G1 /  %O mO  %O mO nC1 nC1 O 2 N0 :
for all m
O
exp.
m/
mO
Section 4.8 Convergence for general matrices 99

Since the degree of pO is bounded by n, we can find a constant

² ³
.n C 1/Šp.
O m/
O
Cq WD max W mO 2 N 0
O nC1

nC1 m
and obtain
O
sin †.x .m/ ; G1 / D sin †.x .mCn/ ; G1 /  Cq %O mO D Cq %O mn for all m 2 Nn ;

i.e., the rate of convergence will get arbitrarily close to %.

Chapter 5

QR iteration

Summary
The most successful algorithm for finding all eigenvalues and eigenvectors of a given
basis is the QR iteration. The basic formulation of the method can be derived from the
simultaneous iteration (cf. Section 4.7). In order to obtain an efficient algorithm, sev-
eral improvements are required: the matrix has to be reduced to Hessenberg form, an
appropriate shift strategy has to be used, and partially converged intermediate results
have to be handled. The resulting practical QR algorithm can be expected to show
quadratic or even cubic convergence and compute all eigenvalues and eigenvectors of
an n  n matrix in O.n3 / operations.

Learning targets
 Introduce the basic QR iteration.
 Derive an algorithm for reducing a matrix to Hessenberg or tridiagonal form.
 Rewrite the iteration as an implicit algorithm requiring only a small amount of
auxiliary storage.
 Get to know shift strategies for improving the speed of convergence.

5.1 Basic QR step

Historically, the QR iteration can be traced back to the quotient-difference algorithm
[36] that characterizes eigenvalues as singularities of a meromorphic function that can
be approximated by a series of fractions. It turned out [37] that the algorithm can be
interpreted as a sequence of simple LR factorizations, and this interpretation led to an
efficient method [38] for computing eigenvalues.
Since LR factorizations may be numerically unstable or even fail to exist for certain
matrices, they were later replaced by the far more stable QR factorizations [14, 15,
26], resulting in the QR iteration. In this introduction, we follow an elegant derivation
[48] of the method based on the simultaneous iteration (cf. Section 4.7).
Let A 2 F nn . We consider the simultaneous iteration applied to a unitary first
matrix Q.0/ 2 F nn , i.e., to a matrix with n columns. By construction, the iteration
Section 5.1 Basic QR step 101

matrices satisfy

Q.mC1/ R.mC1/ D AQ.m/ for all m 2 N0 : (5.1)

Let k 2 ¹1; : : : ; n  1º, and split the matrices into submatrices

 
.m/
Q.m/ D Qk.m/ Q? .m/
; Qk 2 F nk ; (5.2)
.m/ .m/
!
.m/ Rkk Rk? .m/
R D .m/
; Rkk 2 F kk for all m 2 N0 :
R??

Using these submatrices, (5.1) can be rewritten as

!
  R.mC1/ R.mC1/  
.mC1/ .mC1/ .m/ .m/
Qk Q? kk k?
.mC1/
D A Qk
Q?
for all m 2 N0 ;
R??

and due to the triangular structure of the second factor R.mC1/ , taking the first block
column of this equation yields
.mC1/ .mC1/
Qk Rkk D AQk.m/ for all m 2 N0 :

Obviously this equation shares the structure of (5.1): the matrices .Qk /1
.m/
mD0 are the
.0/
result of a simultaneous iteration starting with the isometric matrix Qk .
We assume for the moment that A is self-adjoint (for F D R) or normal (for
F D C), respectively, and that its eigenvalues satisfy the condition (4.20). Let
P 2 F nn denote the orthogonal projection into the invariant subspace spanned by
the eigenvectors corresponding to the first k eigenvalues. We also assume PQk.0/ to
be injective.
Corollary 4.27 states
 
.m/ .m/ .m/ jkC1 j m
kAQk  Qk ƒkk k  C0 for all m 2 N0 (5.3)
jk j

for the matrices .ƒkk /1

.m/
mD0 given by

ƒkk WD .Qk / AQk

.m/ .m/ .m/
for all m 2 N0 :

Since the matrices .Q.m/ /1

mD0 are unitary, the matrices .A
.m/ /1
mD0 defined by

A.m/ WD .Q.m/ / AQ.m/ for all m 2 N0

are unitarily similar to A.

102 Chapter 5 QR iteration

In order to investigate these matrices, we fix m 2 N0 . Using the splitting (5.2), we

obtain
!
.Q
.m/ 
/  
A.m/ D .Q.m/ / AQ.m/ D k
.m/ 
A Q
.m/
k
Q?
.m/
.Q? /
.m/ .m/
!
.Qk / AQk .Qk.m/ / AQ?.m/
D .m/  .m/ 
: (5.4)
.Q? / AQk.m/ .Q? / AQ? .m/

We are interested in the lower left block. Due to the estimate (5.3), we know that
.m/ .m/ .m/
kAQk  Qk ƒkk k converges to zero, and since Q.m/ is unitary, we have
! !
.Qk.m/ /  
.m/ .m/
.Qk / Qk .Qk.m/ / Q?
.m/
.m/ .m/
.m/  .m/ .m/  .m/
D .m/  Qk Q?
.Q? / Qk .Q? / Q? .Q? /
 
.m/  .m/ I
D .Q / Q DI D
I

.m/ .m/
and therefore .Q? / Qk D 0 in particular. For the lower left block of A.m/ , this
means

k.Q? / AQk k D k.Q? / Qk ƒkk C .Q? / .AQk  Qk ƒkk /k

.m/ .m/ .m/ .m/ .m/ .m/ .m/ .m/ .m/

D k.Q? / .AQk  Qk ƒkk /k

.m/ .m/ .m/ .m/
 ! 
 .Q.m/ / 
 .m/ .m/ .m/ 
 k
.AQ  Q ƒ kk 
/
 .Q.m/ / k k 
?
D k.Q.m/ / .AQk
.m/ .m/ .m/
 Qk ƒkk /k
 
.m/ .m/ .m/ jkC1 j m
D kAQk  Qk ƒkk k  C0 ; (5.5)
jk j

where we have used the definition of the norm (2.8) in the third step and Proposi-
tion 2.39 and (2.16) in the fifth step.
This estimate means that the lower left block of A.m/ in (5.4) converges to zero,
therefore A.m/ converges to block upper triangular form. We can use this property to
develop an algorithm that takes the matrices arbitrarily close to triangular form, i.e.,
we can approximate the Schur decomposition introduced in Theorem 2.46.
We can judge the convergence of the iteration by checking the entries of the matri-
ces A.m/ , therefore we are interested in finding an efficient way for computing them.
Fortunately, we can rewrite the simultaneous iteration in a way that allows us to com-
pute A.mC1/ directly from A.m/ : let m 2 N0 . By definition, we have

Q.mC1/ R.mC1/ D AQ.m/ ; (5.6)

Section 5.1 Basic QR step 103

procedure qr_iteration(var A, Q);

begin
while „not sufficiently triangular“ do begin
Compute QR factorization A D QR; b
A b
RQ;
Q QQ b
end
end
Figure 5.1. Simple QR iteration.

and multiplying both sides of the equation by .Q.m/ / yields

.Q.m/ / Q.mC1/ R.mC1/ D .Q.m/ / AQ.m/ D A.m/ :

We introduce the unitary matrix

b .mC1/ WD .Q.m/ / Q.mC1/
Q (5.7)

and obtain
b .mC1/ R.mC1/ D A.m/ :
Q (5.8)
We can reverse this construction: we compute a QR factorization b .mC1/ R.mC1/
Q D
A.m/ , let Q.mC1/ WD Q.m/ Q b .mC1/ , and observe that the defining equation (5.6)
holds. We are interested in finding

A.mC1/ D .Q.mC1/ / AQ.mC1/ :

Due to (5.7), we have

b .mC1/
Q.mC1/ D Q.m/ Q (5.9)
and find

A.mC1/ D .Q.mC1/ / AQ.mC1/ D .Q b .mC1/ / .Q.m/ / AQ.m/ Q

b .mC1/
b .mC1/ :
b .mC1/ / A.m/ Q
D .Q

Using (5.8), we conclude

b .mC1/ D .Q
b .mC1/ / A.m/ Q
A.mC1/ D .Q b .mC1/ R.mC1/ Q
b .mC1/ / Q b .mC1/
b .mC1/ ;
D R.mC1/ Q

i.e., the matrix A.mC1/ can be obtained by multiplying the factors of the QR decom-
position of A.m/ in reversed order. The resulting simple QR iteration is summarized
in Figure 5.1. If we are not interested in the eigenvectors, we can neglect to update
the matrix Q in each step.
104 Chapter 5 QR iteration

5.2 Hessenberg form

The simple QR iteration given in Figure 5.1 can require a large number of operations
per step: finding the QR factorization Q b .mC1/ R.mC1/ D A.m/ of an n  n matrix
A.m/ takes approximately 4n3 =3 arithmetic operations, and multiplying R.mC1/ and
b .mC1/ to compute A.mC1/ can be expected to require a similar amount of opera-
Q
tions.
We can improve the efficiency of the algorithm dramatically if the matrices A.m/
are of a special form that allows us to compute the factorization more efficiently. In
this context the Hessenberg form [20] is very useful, a close relative of the upper
triangular form:

Definition 5.1 (Hessenberg matrix). Let n 2 N and H 2 F nn . The matrix H is

called Hessenberg matrix or in Hessenberg form if

hij D 0 for all i; j 2 ¹1; : : : ; nº with j C 1 < i:

If H  is also in Hessenberg form, H is called tridiagonal.

For a triangular matrix, all entries below the diagonal have to be equal to zero. For a
Hessenberg matrix, the entries immediately below the diagonal (the first subdiagonal)
are allowed to be non-zero. For a tridiagonal matrix, only the entries on the diagonal
and immediately below and immediately above the diagonal are allowed to be non-
zero.
The resulting patterns of non-zero entries (denoted by “”) for Hessenberg (left)
and tridiagonal matrices (right) look as follows:
0 1 0 1
      
B    C B   C
B C B C
B    C C B    C:
B B C
@    A @   A
   

The QR factorization of a Hessenberg matrix H can be computed efficiently using

Givens rotations [18, Section 5.1.8]: for x; y 2 F with .x; y/ ¤ 0, we have
    p 
cN Ns x jxj2 C jyj2 x y
D ; c WD p ; s WD p ;
s c y 0 jxj2 C jyj2 jxj2 C jyj2

so we can apply the unitary transformation

 
cN Ns
G WD
s c
Section 5.2 Hessenberg form 105

procedure givens_setup(var x, y, c, s);

begin
if y D 0 then begin
c 1; s 0
end elsepbegin
% jxj2 C jyj2 ;
c x=%; s y=%;
x %; y 0
end
end;
procedure givens_apply(c, s, var x, y);
begin
 x; x N  sN y; y
c s C cy;
end;
procedure givens_conjapply(c, s, var x, y);
begin
 x; x c  sy; y sN  C cy;
N
end
Figure 5.2. Preparation and application of Givens rotations.

to eliminate entries in a two-dimensional vector. Algorithms for finding c and s and

applying G and G  to vectors are given in Figure 5.2. In order to handle
n-dimensional vectors, we use the same approach as in (3.7): we choose p; q 2
¹1; : : : ; nº with p ¤ q and embed G in a matrix Gpq 2 F nn such that
8
ˆ
<cxN p  sN xq if i D p;
.Gpq x/i D sxp C cxq if i D q; for all x 2 F n ; i 2 ¹1; : : : ; nº:
:̂
xi otherwise

By multiplying with this matrix, we can eliminate the q-th entry of a given vector
changing only the p-th and the q-th component.
We consider a matrix of the form
0 1
    
B    C
B C
H DB B    C
C
@   A
 
106 Chapter 5 QR iteration

as an example. We apply a Givens rotation G .1/ to the first two rows in order to
eliminate the first sub-diagonal element h21 :
0 1
˝ ˝ ˝ ˝ ˝
B0 ˝ ˝ ˝ ˝C
B C
H .1/ WD G .1/ H D B
B    CC;
@   A
 

where “˝” denotes matrix entries changed by the transformation. Next, we eliminate
the second sub-diagonal element h32 by applying a Givens rotation G .2/ to the second
and third row: 0 1
    
B ˝ ˝ ˝ ˝C
B C
H WD G H D B
.2/ .2/ .1/
B 0 ˝ ˝ ˝C C:
@   A
 
Givens rotations G .3/ and G .4/ applied to the third and fourth and fourth and fifth
rows, respectively, lead to the required upper triangular form:
0 1
    
B    C
B C
.3/
H WD G H D B .3/ .2/ B ˝ ˝ ˝C C;
@ 0 ˝ ˝A
 
0 1
    
B    C
B C
.4/ .3/
R WD G H D B B   C C:
@ ˝ ˝A
0 ˝

In general, we require n  1 Givens rotations, and each rotation works on only two
rows of the matrix, so the QR factorization H D QR b of a Hessenberg matrix can
be constructed in O.n2 / operations. In the case of a tridiagonal matrix, we can avoid
computing zeros in the upper right triangle and require only O.n/ operations.
For the next step of the QR iteration, we have to compute Hb WD RQ, b i.e., we have
to apply the adjoint Givens rotations to the columns of R. In our example, we have
Section 5.2 Hessenberg form 107

Q D .G .1/ / .G .2/ / .G .3/ / .G .4/ / and compute

0 1
˝ ˝   
B˝ ˝   C
B C
H WD R.G / D B
b .1/ .1/ 
B   CC;
@  A

0 1
 ˝ ˝  
B ˝ ˝  C
B C
H WD H .G / D B
b .2/ b .1/ .2/ 
B ˝ ˝   C;
C
@  A

0 1
  ˝ ˝ 
B  ˝ ˝ C
B C
H WD H .G / D B
b .3/ b .2/ .3/ 
B  ˝ ˝  C;
C
@ ˝ ˝ A

0 1
   ˝ ˝
B   ˝ ˝C
B C
H WD H .G / D B
b b .3/ .4/ 
B   ˝ ˝CC:
@  ˝ ˝A
˝ ˝

Since each step changes only two columns of the matrix, we require only O.n2 /
operations to perform one step of the QR iteration. This is a significant improvement
compared to the cubic complexity required by a general QR factorization.
We can see that one step of the QR iteration preserves the Hessenberg form: if A.0/
is a Hessenberg matrix, the same holds for all matrices A.m/ . This property allows us
to perform the entire iteration more efficiently. The resulting efficient algorithm for
performing one step of the QR iteration is summarized in Figure 5.3.

Proposition 5.2 (Complexity). The algorithm given in Figure 5.3 requires not more
than
12n2 operations
to perform one step of the QR iteration.

Proof. For a given k 2 ¹1; : : : ; n  1º, computing %, ck and sk requires not more than
8 operations. The application of the corresponding Givens rotation to all columns
takes not more than 6.n  k/ operations, the application to all rows 6k C 2 operations,
108 Chapter 5 QR iteration

procedure qr_hessenberg(var A, Q);

begin
for k D 1 to n  1 do ¹ Find factorization A D QR b º
givens_setup(akk , akC1;k , ck , sk );
for j 2 ¹k C 1; : : : ; nº do givens_apply(ck , sk , akj , akC1;j )
end;
for k D 1 to n  1 do begin ¹ Compute RQ b and QQ bº
for i 2 ¹1; : : : ; kº do givens_conjapply(ck , sk , aik , ai;kC1 );
akC1;k sk akC1;kC1 ; akC1;kC1 cNk akC1;kC1 ;
for i 2 ¹1; : : : ; nº do givens_conjapply(ck , sk , qik , qi;kC1 )
end
end
Figure 5.3. One step of the Hessenberg QR iteration.

and the application to all rows of Q 6n operations. The total number of operations is
bounded by

X
n1
.8 C 6.n  k/ C .6k C 2/ C 6n/
kD1
X
n1
D .10 C 12n/
kD1
D 10.n  1/ C 12n.n  1/  12n2 :

The Hessenberg form has the additional advantage that it is easy to monitor conver-
gence to upper triangular form: we only have to check the n  1 sub-diagonal entries
for convergence to zero.
If A is self-adjoint, the same holds for all matrices A.m/ , since they are unitarily
similar. If A.0/ is also a Hessenberg matrix, the same holds for all A.m/ , so all of these
matrices are self-adjoint Hessenberg matrices and therefore tridiagonal. This means
that for self-adjoint matrices, we can take advantage of the tridiagonal structure to
reduce the number of operations.

Example 5.3 (QR iteration). We consider the QR iteration applied to the matrix
0 1
4
B1 3 C
ADB @ 1 2 A:
C

1 1
Section 5.2 Hessenberg form 109

Since it is lower triangular, we easily find .A/ D ¹1; 2; 3; 4º. Given the results of
Corollary 4.27, we expect convergence rates of 3=4, 2=3 and 1=2 for the first, second
and third subdiagonal entries, respectively. The program gives the following results:
.m/ .m/ .m/
m ja21 j Ratio ja32 j Ratio ja43 j Ratio
1 7:46  101 6:95  10 1 4:13  101
2 5:48  101 0:73 4:62  101 0:66 1:75  101 0:42
3 3:98  101 0:73 2:95  101 0:64 8:24  102 0:47
4 2:86  101 0:72 1:87  101 0:63 4:09  102 0:50
5 2:06  101 0:72 1:20  101 0:64 2:07  102 0:51
6 1:49  101 0:72 7:81  102 0:65 1:05  102 0:51
7 1:08  101 0:72 5:13  102 0:66 5:33  103 0:51
8 7:88  102 0:73 3:39  102 0:66 2:70  103 0:51
The subdiagonal entries appear to converge at the rate predicted by our theory.
Exercise 5.4 (Self-adjoint QR iteration). Derive a version of the Hessenberg QR
iteration given in Figure 5.3 for self-adjoint matrices that requires only O.n/
operations for each iteration.

Apparently the Hessenberg form is useful, therefore we are looking for an algorithm
that transforms an arbitrary matrix into this form. We consider a matrix of the form
0 1
    
B    C
B C
ADB B    C :
C
@    A
    
We first look for a unitary similarity transformation that eliminates the entries a31 ,
a41 and a51 . This can be done by Givens rotations, but we choose to use Householder
reflections for the sake of efficiency: according to Lemma 2.45, we can find a unitary
self-adjoint matrix P1 such that
0 1 0 1
a21 1
Ba31 C B 0 C
P1 B C B C
@a41 A D @ 0 A
a51 0
holds for a 1 2 F, and applying this transformation to the second to fifth row of
A yields 0 1
    
  B˝ ˝ ˝ ˝ ˝C
1 B C
ADB B0 ˝ ˝ ˝ ˝CC:
P1 @0 ˝ ˝ ˝ ˝A
0 ˝ ˝ ˝ ˝
110 Chapter 5 QR iteration

Since we are looking for a similarity transformation, we have to apply the inverse of
P1 to the second to fifth column. Due to P1 D P1 D P11 , computing the inverse is
not really a challenge, and we obtain
0 1
 ˝ ˝ ˝ ˝
    B ˝ ˝ ˝ ˝C
1 1 B C
b .1/
A WD A DBB ˝ ˝ ˝ ˝C C:
P1 P1 @ ˝ ˝ ˝ ˝A
˝ ˝ ˝ ˝

.1/ .1/
Now we find a Householder reflection P2 that eliminates aO 42 and aO 52 , i.e., such that
0 .1/
1 0 1
aO 32 2
B .1/ C @ A
P2 @aO 42 A D 0
.1/ 0
aO 52

holds for a 2 2 F. Applying it to the third to fifth rows and columns yields
0 1
    
  B     C
I2 B C
A DB
b .1/
B ˝ ˝ ˝ ˝C C;
P2 @ 0 ˝ ˝ ˝A
0 ˝ ˝ ˝
0 1
  ˝ ˝ ˝
    B  ˝ ˝ ˝C
I2 .1/ I2
B C
b .2/
A WD b
A DB B  ˝ ˝ ˝C C:
P2 P2 @ ˝ ˝ ˝A
˝ ˝ ˝

.2/
In the last step, we find a Householder reflection P3 that eliminates aO 53 , i.e., such
that !  
.2/
aO 43 
P3 .2/ D 3
aO 53 0

holds for a 3 2 F. Applying this reflection to the fourth and fifth rows and columns
gives us the required Hessenberg form:

0 1
    
  B    C
I3 B C
b
A.2/ DB
B    CC;
P3 @ ˝ ˝ ˝A
0 ˝ ˝
Section 5.2 Hessenberg form 111

0 1
   ˝ ˝
    B   ˝ ˝C
I .2/ I3
B C
b
A WD 3
.3/ b
A DB
B   ˝ ˝CC:
P3 P3 @  ˝ ˝A
˝ ˝

We can “hide” this procedure in the choice of the initial basis for the QR iteration: we
choose the initial matrix as
   
.0/ 1 I2 I3
Q D
P1 P2 P3

and obtain
A.0/ D .Q.0/ / AQ.0/ D b
A.3/ :
The resulting algorithm is given in Figure 5.4.

Exercise 5.5 (Triangular form). Explain why this procedure cannot be used to
reach triangular form, i.e., to compute the Schur decomposition directly.

Exercise 5.6 (Self-adjoint matrix). Let A 2 F nn be self-adjoint. In this case, it

is customary to store only its lower or upper triangular part. Modify the algorithm
for transforming a matrix to Hessenberg form (cf. Figure 5.4) in such a way that
it works only with the lower triangular part of A.

Proposition 5.7 (Complexity). The algorithm given in Figure 5.4 requires not more
than
16 3
n operations
3
to transform A into a Hessenberg matrix.

Proof. For a given k 2 ¹1; : : : ; n  2º, computing ˛, ˇ and the Householder vector
requires
.n  k/ C 6 operations:
Applying the Householder reflection to the rows of A requires

4.n  k/2 C .n  k/ operations;

while applying it to the columns of A and the columns of Q takes

8n.n  k/ C 2.n  k/ operations:

112 Chapter 5 QR iteration

procedure hessenberg(var A, Q);

begin
for k D 1 to n  2 do begin
˛ 0;
for i 2 ¹k C 1; : : : ; nº do ˛ ˛ C jaik j2 ;
if ˛ ¤ 0pdo begin
˛ ˛; ˇ 1=.˛.˛ C jakC1;k j//; akC1;k akC1;k C sgn.akC1;k /˛;
for j 2 ¹k C 1; : : : ; nº do begin ¹ Apply to rows of A º

0;
for i 2 ¹k C 1; : : : ; nº do

C aN ik aij ;


ˇ;
for i 2 ¹k C 1; : : : ; nº do aij aij 
aik
end;
for i 2 ¹1; : : : ; nº do begin ¹ Apply to columns of A º

0;
for j 2 ¹k C 1; : : : ; nº do

C aij aj k ;


ˇ;
for j 2 ¹k C 1; : : : ; nº do aij aij  aNj k

end;
for i 2 ¹1; : : : ; nº do begin ¹ Apply to columns of Q º

0;
for j 2 ¹k C 1; : : : ; nº do

C qij aj k ;


ˇ;
for j 2 ¹k C 1; : : : ; nº do qij qij  aNj k

end
end
end
end
Figure 5.4. Transformation to Hessenberg form.

The total number of operations is given by

X
n2
.8n.n  k/ C 4.n  k/2 C 4.n  k/ C 6/
kD1
X
n2 X
n2
D .8n C 4/ .n  k/ C 4 .n  k/2 C 6.n  2/
kD1 kD1
 
n.n  1/ 4
 .8n C 4/  1 C n.n  1/.2n  1/ C 6.n  2/
2 6
Section 5.3 Shifting 113

   
2 16 4
 4n C 2 C .2n  1/ n.n  1/ D nC .n2  n/
3 3 3
16 3 4 2 16 2 4 16 3
D n C n  n  n n :
3 3 3 3 3
Exercise 5.8 (Improved computation of Q). Assume that we start the transfor-
mation to Hessenberg form in the algorithm of Figure 5.4 with Q D I . Prove that
in this case the number of operations can be reduced by 2n3 =3 without changing
the result. (Hint: It may be necessary to apply the Householder reflections in
reversed order)

5.3 Shifting
The rate of convergence observed in Example 5.3 is too slow for a practical method:
since the computational effort for each step of the iteration is proportional to n2 , we
would like the number of steps to be as small as possible. We have already seen in
Example 4.14 that shifting the spectrum can significantly improve the convergence,
so we are interested in introducing a shift strategy for the QR iteration.
Since the QR method works with a full basis of F n , we can use shift strategies
without the need for computing an inverse: if we have found a shift parameter  2
F sufficiently close to an eigenvalue n 2 .A/ and sufficiently far from all other
eigenvalues, i.e., if we have

j1  j
j2  j
  
jn1  j > jn  j;

applying (5.5) to k D n1 and the matrix AI yields that the off-diagonal elements
in the n-th row will converge to zero at a rate of jn  j=jn1  j.
We can easily introduce the shift to the QR iteration: let m 2 N0 . One step of the
simultaneous iteration for A  I is given by
.mC1/ .mC1/ .m/
Q R D .A  I /Q ;

and using again

.mC1/ .m/ b .mC1/
Q D Q Q
we obtain
.m/ b .mC1/ .mC1/ .m/
Q Q R D .A  I /Q :
.m/ 
Multiplication by .Q / yields
b .mC1/
Q 
.mC1/
R .m/ 
D .Q .m/
/ .A  I /Q .m/ 
D .Q .m/
/ AQ  I:

We denote the iteration matrices by

.m/ 
A.m/ .m/
 WD .Q / AQ for all m 2 N0
114 Chapter 5 QR iteration

and write the first half of the shifted QR step in the form
b .mC1/
Q 
.mC1/
R D A.m/
  I: (5.10)
.mC1/
In order to compute the next iteration matrix A , we consider
.mC1/  b .mC1/ .m/ b .mC1/
A.mC1/
 D .Q .mC1/
/ AQ D .Q  / .Q
.m/ 
/ AQ Q
b .mC1/
D .Q  / A.m/
 Q
b .mC1/ D .Q
b .mC1/
 / .A.m/ b .mC1/ C I
  I /Q
b .mC1/
D .Q 
b .mC1/
/ Q 
.mC1/ b .mC1/
R Q C I
.mC1/ b .mC1/
D R Q C I: (5.11)

Combining (5.10) and (5.11) yields the definition of the shifted QR iteration:
b .mC1/
Q 
.mC1/
R D A.m/
  I;
.mC1/ b .mC1/
A.mC1/
 D R Q C I for all m 2 N0 :

As we can see, the only difference between the shifted QR iteration and the original
one consists of subtracting  from the diagonal elements prior to computing the QR
factorization and adding  after the multiplication. Compared to the overall complex-
ity of the QR iteration, the additional operations required for the shift are negligible.
As in the case of the Rayleigh iteration, we can choose a different shift parameter
for each step of the iteration, and doing so promises quadratic or even cubic con-
vergence of the resulting shifted QR iteration. In the following, we will write A.m/
.m/
instead of A in order to keep the notation simple if the choice of shift parameter is
clear from the context in order to keep the notation simple.
It is customary to try to choose the shift parameter to speed up convergence of
the last row of the iteration matrices. According to (5.5), this means that we should
ensure that the quotient jn j=jn1 j is as small as possible. If the subdiagonal
.m/
elements of the row are sufficiently close to zero, the diagonal element ann is close
to the eigenvalue n , so it is a natural choice for the shift parameter.

Definition 5.9 (Rayleigh shift). The Rayleigh shift strategy is given by choosing the
.m/
shift parameter in the .m C 1/-th step of the QR iteration as  D ann .

The name “Rayleigh shift” is motivated by the close connection to the Rayleigh
quotient introduced in Definition 4.4: if we denote the last column of Q.m/ by q .m/ ,
we have
.m/
ann D ..Q.m/ / AQ.m/ /nn D hAq .m/ ; q .m/ i D ƒA .q .m/ /

due to the definition of A.m/ and kq .m/ k D 1. This equation suggests that we can
expect the same good convergence behaviour as for the Rayleigh iteration described
in Section 4.5.
Section 5.3 Shifting 115

We are interested in choosing a shift parameter that is close to an eigenvalue. In the

case of the Rayleigh quotient, we have used the eigenvalue of the lower right 1  1
principal submatrix of A.m/ as an approximation, so it is straightforward to generalize
the strategy to consider larger principal submatrices. The case of the lower right 2  2
submatrix is particularly useful: we are looking for the eigenvalues of
!
.m/ .m/
b a a
AD n1;n1
.m/
n1;n
.m/ :
an;n1 ann

They can be determined by computing the zeros of the characteristic polynomial

pb
A
.t / D det.tI  b A/ D .t  aO 11 /.t  aO 22 /  aO 12 aO 21
2
D t  .aO 11 C aO 22 /t C aO 11 aO 22  aO 12 aO 21
   
2 aO 11 C aO 22 2 aO 11 C aO 22 2
D t  .aO 11 C aO 22 /t C  C aO 11 aO 22  aO 12 aO 21
2 2
   
aO 11 C aO 22 2 aO 11  aO 22 2
D t   aO 12 aO 21 ;
2 2
which are given by
8 s 9
< aO  2 =
C aO 22 aO 11  aO 22
.b
11
A/ D ˙ C aO 12 aO 21 :
: 2 2 ;

Since we are interested in convergence of the last row, we should choose the eigen-
value in .b
A/ that is closest to the n-th diagonal element.

Definition 5.10 (Francis single shift). The Francis single shift strategy is given by
.m/
choosing the element closest to ann in the set
8 v 9
ˆ u .m/ !2 >
<a .m/ .m/ u a .m/ =
b n1;n1 C ann t n1;n1  ann .m/ .m/
.A/ D ˙ C a a
n1;n n;n1 >
:̂ 2 2 ;

as the shift parameter in the .m C 1/-th step of the QR iteration.

If we are dealing with a real matrix, the Francis shift strategy may yield a complex-
valued shift parameter, and this would make the next iteration matrix also complex-
valued. Since eigenvalues of real matrices always appear in complex-conjugate pairs,
it is possible to avoid working with complex values by first performing a QR step with
the Francis shift  and following it by a QR step with its complex conjugate . N Due to

N / D A2  . C /A
.A  I /.A  I N C jj2 I;
116 Chapter 5 QR iteration

the resulting iteration matrix is again real-valued. It is possible to arrange the com-
putation in such a way that complex numbers are avoided entirely, this results in the
Francis double shift strategy (cf. Example 5.17).

Example 5.11 (Failure to converge). As we have seen in the case of the Rayleigh
iteration (cf. Proposition 4.13), shift strategies work best if the iteration vectors or
matrices are sufficiently close to the result. The matrix
0 1
0 0 1
P WD @1 0 0A
0 1 0

taken from [32] illustrates this point: its characteristic polynomial is given by

pP .t / D t 3  1;

so its eigenvalues are the third roots of unity. Since they are equal in modulus, we can
expect no convergence unless a suitable shift parameter is chosen.
Unfortunately, both the Rayleigh and the Francis shift strategies choose the shift
parameter  D 0.

5.4 Deflation
Due to (5.5), we can expect the matrices A.m/ to converge to a matrix of the form
 
B F
A.m/  ; B 2 F kk ; D 2 F .nk/.nk/ : (5.12)
C

Once A.m/ is sufficiently close to this block-triangular form, we can stop performing
iterations for the entire matrix and focus on handling the diagonal blocks B and C
individually: if we can find unitary matrices QB 2 F kk and QC 2 F .nk/.nk/
such that QB  BQ and Q  CQ are sufficiently close to upper triangular form, the
B C C
same holds for
        
QB .m/ QB QB B F QB
A 
QC QC QC C QC
  

QB BQB QB FQC
D  CQ ;
QC C

and we have approximated the Schur decomposition.

This observation suggests a strategy known as deflation: we perform iterations until
one of the subdiagonal blocks has become sufficiently small. Then we repeat the
procedure for the individual diagonal blocks. If we can choose the shift parameters
Section 5.4 Deflation 117

correctly, we will apply the procedure to smaller and smaller diagonal blocks until we
arrive at 1  1 blocks and have reached an approximation of the Schur decomposition.
Applying the deflation approach to Hessenberg matrices is particularly simple:
the subdiagonal blocks contain only one non-zero entry, so we can easily check for
convergence. In a practical deflation approach, we first look for the largest index
˛ 2 ¹1; : : : ; nº such that
.m/
akC1;k  0 for all k 2 ¹1; : : : ; ˛  1º:

If ˛ D n, the matrix A.m/ is sufficiently close to upper triangular form and we can
.m/
stop the iteration. Otherwise, we have a˛C1;˛ 6 0 due to the maximality of ˛, and
we look for the largest ˇ 2 ¹˛; : : : ; nº such that
.m/
akC1;k 
6 0 for all k 2 ¹˛; : : : ; ˇ  1º:

.m/
If ˇ < n, we have aˇ C1;ˇ  0 due to the maximality of ˇ.
We split the matrix A.m/ in the form
0 1
H11 H12 H13 H11 2 F .˛1/.˛1/ ; H22 2 F .ˇ ˛C1/.ˇ ˛C1/ ;
A.m/  @ H22 H23 A ;
H33 H33 2 F .nˇ /.nˇ / :

Due to our choice of ˛, H11 is sufficiently close to triangular form and the block below
H11 is sufficiently close to zero. Due to our choice of ˇ, the block below H22 is also
sufficiently close to zero. H22 is a Hessenberg matrix, and we can efficiently apply the
QR iteration to try to eliminate its subdiagonal entries. Once we have succeeded, H11
and H22 are upper triangular and we can apply the same procedure to the remaining
matrix H33 .
The resulting algorithm is given in Figure 5.5. In order to decide whether a condi-
.m/
tion of the form “a˛C1;˛  0” is satisfied in practice, a condition like

.m/ .m/ .m/

ja˛C1;˛ j  .ja˛˛ j C ja˛C1;˛C1 j/

is frequently used, where > 0 denotes a relative error tolerance. A typical choice
for depends on the machine’s floating point
p accuracy mach . Common choices are
D c mach for a small constant c or D n mach , taking into account the matrix
dimension.

Example 5.12 (QR iteration). We apply the QR iteration with Francis and Rayleigh
shift to the matrix A given in Example 5.3. The program gives the following results:
118 Chapter 5 QR iteration

procedure qr_deflation(var A, Q);

begin
˛ 1; while ˛ < n and a˛C1;˛  0 do ˛ ˛ C 1;
ˇ ˛ C 1; while ˇ < n and aˇ C1;ˇ 6 0 do ˇ ˇ C 1;
Choose a shift parameter ;
for k D ˛ to ˇ do akk akk  ;
for k D ˛ to ˇ  1 do ¹ Find factorization A  I D QR b º
givens_setup(akk , akC1;k , ck , sk );
for j 2 ¹k C 1; : : : ; nº do givens_apply(ck , sk , akj , akC1;j )
end;
for k D ˛ to ˇ  1 do begin ¹ Compute RQ b and QQ bº
for i 2 ¹1; : : : ; kº do givens_conjapply(ck , sk , aik , ai;kC1 );
akC1;k sk akC1;kC1 ; akC1;kC1 cNk akC1;kC1 ;
for i 2 ¹1; : : : ; nº do givens_conjapply(ck , sk , qik , qi;kC1 )
end;
for k D ˛ to ˇ do akk akk C 
end
Figure 5.5. One step of the Hessenberg QR iteration with shift and deflation.

Rayleigh Francis
.m/ .m/ .m/ .m/ .m/ .m/
m ja21 j ja32 j ja43 j ja21 j ja32 j ja43 j
1 6:78  101 6:23  101 conv. 6:78  101 6:23  101 conv.
2 3:86  101 7:14  102 3:66  101 conv.
3 1:73  101 4:51  10 3 conv.
4 7:68  102 2:32  105
5 3:60  102 5:96  1010
6 1:74  102 conv.
7 2:73  104
8 7:27  108
9 conv.

We can see that the Rayleigh shift strategy seems to lead to quadratic convergence as
soon as the subdiagonal entries are sufficiently close to zero. With the Francis shift
strategy, the iteration requires only three steps to converge. For practical problems,
the difference between both strategies can be expected to be far less pronounced.

5.5 Implicit iteration

The QR iteration presented in Figure 5.5 is still slightly inelegant: we have to subtract
and add the shift parameter explicitly, and we have to store the values ck and sk corre-
Section 5.5 Implicit iteration 119

sponding to the Givens rotations used during the factorization process. In this section,
we derive an alternative version of the QR iteration that avoids these inconveniences
and facilitates the implementation of multiple shift strategies like the Francis double
shift or even more sophisticated techniques that can lead to significant improvements
of the basic algorithm [50, 4, 5].
We have seen that the QR iteration preserves the Hessenberg form of the iteration
matrices A.m/ : if A.m/ is a Hessenberg matrix, then so is the next iteration matrix
A.mC1/ D .Q b .mC1/ / A.m/ Q
b .mC1/ . Unitarily similar Hessenberg matrices like A.m/
and A.mC1/ have special properties that we can use to simplify the algorithm: if we
have two unitarily similar Hessenberg matrices with non-zero subdiagonal entries and
if the first column of the corresponding similarity transformation matrix is the first
canonical unit vector, then both matrices can only differ in the signs of their entries.

Theorem 5.13 (Implicit Q). Let H; J 2 F nn be Hessenberg matrices and let Q 2
F nn be unitary such that
J D Q HQ: (5.13)
Let m 2 ¹0; : : : ; n  1º satisfy

jkC1;k ¤ 0 for all k 2 ¹1; : : : ; mº:

If the first column of Q is the canonical unit vector ı1 , the first m C 1 columns of Q
are multiples of the first m C 1 canonical unit vectors.

Proof. (cf. [18, Theorem 7.4.2]) We prove by induction on k 2 ¹1; : : : ; nº that the
k-th column qk WD Qık of Q is a multiple of the k-th canonical unit vector.
For k D 1, this is given. Let now k 2 ¹1; : : : ; mº be such that q1 ; : : : ; qk are
multiples of the corresponding unit vectors. This implies

q` D ı` q`` for all ` 2 ¹1; : : : ; kº: (5.14)

We write (5.13) in the form

QJ D HQ
and multiply by the k-th canonical unit vector to obtain

X
kC1 X
kC1 X
kC1
q` j`k D Q ı` j`k D QJ ık D HQık D H qk D H ık qkk D ı` h`k qkk :
`D1 `D1 `D1

We apply (5.14) to the left-hand side to obtain

X
k X
kC1
qkC1 jkC1;k C ı` q`` j`k D ı` h`k qkk ;
`D1 `D1
120 Chapter 5 QR iteration

and rearranging the terms yields

X
k
qkC1 jkC1;k D ıkC1 hkC1;k qk C ı` .h`k qkk  q`` j`k /:
`D1

Due to jkC1;k ¤ 0, we find

0 1
1 @ X
k
qkC1 D ıkC1 hkC1;k qk C ı` .h`k qkk  q`` j`k /A ;
jkC1;k
`D1

i.e., qkC1 2 span¹ı1 ; : : : ; ıkC1 º and therefore

.qkC1 /` D 0 for all ` 2 ¹k C 2; : : : ; nº: (5.15)

Using (5.14) again yields

.qkC1 /` D hqkC1 ; ı` i D hqkC1 ; q` =q`` i D 0 for all ` 2 ¹1; : : : ; kº (5.16)

since Q is unitary. Combining (5.15) and (5.16) yields that qkC1 has to be a multiple
of ıkC1 .

We aim to use this result to simplify the QR iteration: the QR iteration takes a
Hessenberg matrix A.m/ and computes a unitarily similar Hessenberg matrix A.mC1/ .
If we can find a new algorithm that also yields a unitarily similar Hessenberg matrix
e
A.mC1/ and if the first column of both similarity transformations are equal, Theo-
rem 5.13 yields that A.mC1/ and e A.mC1/ can differ only in the signs of the coeffi-
cients.

e 2 F nn be Hessenberg matri-

Corollary 5.14 (Hessenberg similarity). Let A; B; B
e
ces. Let Q; Q 2 F nn be unitary matrices such that

B D Q AQ; eDQ
B e  AQ:
e

e be identical and let m 2 ¹0; : : : ; n  1º satisfy

Let the first columns of Q and Q

bk;kC1 ¤ 0 for all k 2 ¹1; : : : ; mº: (5.17)

Then there are a unitary diagonal matrix D 2 F .mC1/.mC1/ and a unitary matrix
b 2 F .nm1/.nm1/ such that
P
   
e D D
BD b B b :
P P
Section 5.5 Implicit iteration 121

e  Q and observe
Proof. In order to be able to apply Theorem 5.13, we define P WD Q

eB
ADQ eQ
e ;
eB
B D Q AQ D Q Q eQ
e  Q D P  BP:
e

Since the first columns of Q and Qe are identical, the first column of P has to be the
first canonical unit vector.
Theorem 5.13 yields that the first m C 1 columns of P are multiples of the first
m C 1 canonical unit vectors, so we can find a diagonal matrix D 2 F .mC1/.mC1/
b 2 F .nm1/.nm1/ and R 2 F .mC1/.nm1/ such that
and matrices P
 
D R
P D b :
P

Since P is the product of unitary matrices, we have

     
ImC1  D D R
DI DP P D b b
Inm1 R P P
  
D D DR
D b ;
bP
R D R  R C P

i.e., D  D D I , D  R D 0 and R R C P bPb D I . The first equation implies

that D is unitary. Since unitary matrices are invertible, the second equation yields
bP
R D 0. Since R is equal to zero, the third equation gives us P b D I , i.e., P
b is also
unitary.

The Hessenberg QR step discussed in Section 5.2 consists of first finding Givens
rotations G1 ; : : : ; Gn1 that render A.m/ upper triangular

Gn1 : : : G1 A.m/ D R.mC1/ ; Q.mC1/ WD G1 : : : Gn1



and then multiplying by their adjoints in reversed order to compute the next iteration
matrix

A.mC1/ D R.mC1/ Q.mC1/ D .Q.mC1/ / A.m/ Q.mC1/

D Gn1 : : : G1 A.m/ G1 : : : Gn1

:

In our construction, only the first rotation G1 can change the first component of a
vector, therefore the first column of Q.mC1/ depends only on G1 . If we can construct
e2; : : : ; G
an alternate sequence of unitary transformations G e n1 that do not change the
first component, the first columns of

Q.mC1/ D G1 G2 : : : Gn1


and Q e 2 : : : G
e .mC1/ D G1 G e n1
122 Chapter 5 QR iteration

have to be equal. If we can ensure that

e e .mC1/ / A.m/ Q
A.mC1/ WD .Q e .mC1/

is also a Hessenberg matrix, Corollary 5.14 applies and guarantees that A.mC1/ and
e
A.mC1/ differ only in the signs of the coefficients, while the convergence of the re-
sulting iteration remains unchanged. The condition (5.17) can be taken care of by
using the deflation approach introduced in Section 5.4: as soon as a subdiagonal entry
vanishes, the matrix is split into submatrices with non-zero subdiagonal entries and
the submatrices are processed independently.
Our goal is now to construct the matrices G e n1 . We illustrate the proce-
e2; : : : ; G
dure using a Hessenberg matrix of the form
0 1
    
B    C
B C
H WD A .m/
DBB    C C:
@   A
 

Applying the first Givens rotation G1 to H yields matrices of the form

0 1
˝ ˝ ˝ ˝ ˝
B˝ ˝ ˝ ˝ ˝C
B C
G1 H D BB    C C;
@   A
 
0 1
˝ ˝   
B˝ ˝   C
B C
H WD G1 HG1 D B
.1/ 
B ˝   C :
C
@   A
 

.1/
We can see that H .1/ is “almost” a Hessenberg matrix, only the coefficient h31 in
the first column of the third row poses a problem. Fortunately, we can apply a Givens
e 2 to the second and third row to eliminate this entry and obtain
rotation G
0 1
    
B˝ ˝ ˝ ˝ ˝C
B C
G 2H D B
e .1/
B 0 ˝ ˝ ˝ ˝C ;
C
@   A
 
Section 5.5 Implicit iteration 123

0 1
 ˝ ˝  
B ˝ ˝  C
B C
H .2/ WD G e 2 D B
e 2 H .1/ G ˝ ˝  C
B C:
@  ˝  A
 
The entry in the first column is gone, unfortunately we have introduced a non-zero
.2/ e3
entry h42 in the second column of the fourth row. We can apply a Givens rotation G
to the third and fourth row to get rid of this entry and find
0 1
    
B     C
B C
G 3H D B
e .2/
B ˝ ˝ ˝ ˝C C;
@ 0 ˝ ˝ ˝A
 
0 1
  ˝ ˝ 
B  ˝ ˝ C
B C
H WD G 3 H G 3 D B
.3/ e .2/ e 
B  ˝ ˝ C C:
@ ˝ ˝ A
 ˝ 
.3/
Once more only one entry causes problems, now the entry h53 in the third column of
e 4 for the fourth and
the fifth row. We already know what to do: a Givens rotation G
fifth row eliminates the offending entry:
0 1
    
B     C
B C
e4H D B
.3/
   C
G B C;
@ ˝ ˝ ˝A
0 ˝ ˝
0 1
   ˝ ˝
B   ˝ ˝C
B C
.4/
H WD G e4H G.3/ e4 D B

  ˝ ˝C
B C:
@  ˝ ˝A
˝ ˝
We can see that we have achieved our goal:
e e n1 : : : G
A.mC1/ WD H .4/ D G e 2 : : : G
e 2 G1 A.m/ G1 G e n1

is a Hessenberg matrix. This technique is called “bulge chasing”, since the matrices
H .1/ ; : : : ; H .n2/ differ from Hessenberg form only by the “bulge” marked as 
consisting of one non-zero entry and this bulge is “chased” down and to the right until
it “drops out of the matrix” and we have reached Hessenberg form.
124 Chapter 5 QR iteration

procedure qr_implicit(var A, Q);

begin
˛ 1; while ˛ < n and a˛C1;˛  0 do ˛ ˛ C 1;
if ˛ D n then return;
ˇ ˛ C 1; while ˇ < n and aˇ C1;ˇ 6 0 do ˇ ˇ C 1;
Choose a shift parameter ;
0
a˛˛ a˛˛  ; a˛C1;˛ 0 a˛C1;˛ ;
0 , a0
givens_setup(a˛˛ ˛C1;˛ , c, s);
for j 2 ¹˛; : : : ; nº do givens_apply(c, s, a˛j , a˛C1;j );
for i 2 ¹1; : : : ; ˛ C 1º do givens_conjapply(c, s, ai˛ , ai;˛C1 );
for i 2 ¹1; : : : ; nº do givens_conjapply(c, s, qi˛ , qi;˛C1 );
for k D ˛ to ˇ  2 do begin
b sakC2;kC1 ; akC2;kC1 N kC2;kC1 ;
ca ¹ Update row k C 2 º
givens_setup(akC1;k , b, c, s); ¹ Givens rotation to eliminate bulge º
for j 2 ¹k C 1; : : : ; nº do givens_apply(c, s, akC1;j , akC2;j );
for i 2 ¹1; : : : ; k C 2º do givens_conjapply(c, s, ai;kC1 , ai;kC2 );
for i 2 ¹1; : : : ; nº do givens_conjapply(c, s, qi;kC1 , qi;kC2 )
end
end
Figure 5.6. One step of the implicit QR iteration with shift and deflation.

e2; : : : ; G
Since the rotations G e n1 do not change the first row, the conditions of
Corollary 5.14 are fulfilled and we have computed an iteration matrix e A.mC1/ that
can replace A .mC1/ without changing the important properties of our algorithm (as
long as deflation is used). The resulting algorithm is called the implicit QR method
and offers several advantages, most importantly an elegant way of handling shifts: for
the shifted QR iteration, the next iteration matrix is given by

b .mC1/ R.mC1/ D A.m/  I;

Q
b .mC1/ C I D .Q
A.mC1/ D R.mC1/ Q b .mC1/ ;
b .mC1/ / A.m/ Q

and since adding and subtracting the identity does not change the Hessenberg form,
we can also apply our construction with a small change: we compute the first Givens
rotation G1 not for the matrix A.m/ , but for the shifted matrix A.m/  I . Then we
apply it to A.m/ and proceed as before to regain Hessenberg form. For the implicit
approach, the only difference between the original and the shifted iteration lies in
the choice of the first Givens rotation. Combining the shifted implicit iteration with
deflation leads to the algorithm given in Figure 5.6 that performs one step of the
practical implicit QR iteration.
Section 5.5 Implicit iteration 125

Proposition 5.15 (Complexity). The algorithm given in Figure 5.6 requires not more
than
12n2 C 8n operations
to perform a step of the implicit QR iteration. The estimate does not include the
operations required for finding an appropriate shift parameter.

Proof. We require 8 operations to compute the first Givens rotation and 6.n  ˛ C 1/
operations to apply it to the rows as well as 6.˛ C 1/ operations to apply it to the
columns, for a total of

8 C 6.n  ˛ C 1 C ˛ C 1/ D 8 C 6.n C 2/ D 20 C 6n

operations. Applying the rotation to the columns of Q takes additional

6n

operations. For each k 2 ¹˛; : : : ; ˇ  2º, updating the .k C 2/-th row takes 2 op-
erations, determining the Givens rotation to eliminate the bulge takes 6 operations,
applying this rotation to the rows requires 6.n  k/ operations, and applying it to the
column requires 6.k C 2/, for a total of

8 C 6.n  k C k C 2/ D 8 C 6.n C 2/ D 20 C 6n

operations. Applying the rotation to the columns of Q again takes additional

6n

operations. In the worst case, all subdiagonal entries are non-zero and we have ˛ D 1
and ˇ D n, so the entire algorithm requires not more than

X
ˇ 2
20 C 12n C .20 C 12n/ D .20 C 12n/.1 C ˇ  2  ˛ C 1/
kD˛
D .20 C 12n/.ˇ  ˛/
D .20 C 12n/.n  1/  20n C 12n2  20  12n
D 12n2 C 8n  20 < 12n2 C 8n

operations.

Although the number of operations for the implicit algorithm is slightly higher than
in the case of the original algorithm given in Figure 5.5, its practical performance is
frequently superior, e.g., in the self-adjoint case, A is a tridiagonal matrix, and the
bulge-chasing algorithm works its way from the upper left to the lower right along the
diagonal. If the tridiagonal matrix is stored appropriately, this operation can benefit
from the cache memory of modern processors.
126 Chapter 5 QR iteration

Remark 5.16 (Structured eigenvalue problems). Frequently, the matrix under inves-
tigation exhibits additional structure, e.g., it may be Hamiltonian, Skew–Hamiltonian
or the product of two matrices. In these cases, it is possible to derive special variants
of the QR iteration that can take advantage of the structure [24, 49].

5.6 Multiple-shift strategies 

The most important feature of the implicit QR iteration is that this approach facilitates
the use of sophisticated multiple-shift strategies: we consider two consecutive steps
of the iteration with shift values of mC1 and mC2 , given by

b .mC1/ R.mC1/ D A.m/  mC1 I;

Q b .mC1/ ;
b .mC1/ / A.m/ Q
A.mC1/ D .Q
b .mC2/ / A.mC1/ Q
b .mC2/ R.mC2/ D A.mC1/  mC2 I; A.mC2/ D .Q
Q b .mC2/ :

We are looking for a way to get from A.m/ to A.mC2/ without computing A.mC1/ .
We have

b .mC2/
b .mC2/ / A.mC1/ Q
A.mC2/ D .Q
b .mC2/ / .Q
D .Q b .mC1/ Q
b .mC1/ / A.m/ Q b .mC2/
b .mC2/ / A.m/ Q
b .mC1/ Q
D .Q b .mC1/ Q
b .mC2/ D .Q b .mC2/
b .mC2/ / A.m/ Q
dbl dbl

with the auxiliary unitary matrix

b .mC2/ WD Q
Q b .mC2/ :
b .mC1/ Q
dbl

In order to reach our goal, we have to find this matrix without computing A.mC1/
or Q.mC1/ . We know that the QR iteration is closely related to the simultaneous
iteration, so it is reasonable to consider the result of performing two steps of the
simultaneous iteration at once, i.e., to investigate the product matrix
.A.m/  mC2 I /.A.m/  mC1 I /
DQ b .mC1/ .Qb .mC1/ / .A.m/  mC2 I /Q b .mC1/ .Q
b .mC1/ / .A.m/  mC1 I /
b .mC1/ ..Q
DQ b .mC1/  mC2 I /.Q
b .mC1/ / A.m/ Q b .mC1/ / .A.m/  mC1 I /
b .mC1/ .A.mC1/  mC2 I /R.mC1/
DQ
b .mC2/ R.mC2/ R.mC1/
b .mC1/ Q
DQ
b .mC2/ R.mC2/ ;
DQ dbl dbl

where we have introduced the matrix

.mC2/
Rdbl WD R.mC2/ R.mC1/ :
Section 5.6 Multiple-shift strategies 127

.mC2/
Since R.mC2/ and R.mC1/ are upper triangular, their product Rdbl shares this prop-
b .mC2/
erty (cf. Lemma 4.29) and the matrices Qdbl
.mC2/
and Rdbl define a QR decompo-
sition
b .mC2/ R.mC2/ D .A.m/  mC2 I /.A.m/  mC1 I /
Q (5.18)
dbl dbl

of the product matrix. As long as mC1 and mC2 are not eigenvalues, the right-hand
side matrix is invertible and this equation can be used to construct a matrix that differs
from Qb .mC2/ only by the signs of its columns. If one of the shift parameters happens
dbl
to be an eigenvalue, a subdiagonal entry of A.mC2/ vanishes and we can use deflation.
We can construct Q b .mC2/ using Givens rotations or Householder reflections: we
dbl
consider again the case of 5  5 matrices. Since A.m/  mC1 I and A.m/  mC2 I
are Hessenberg matrices, their product has the form
0 1
    
B    C
B C
.0/
B WD .A .m/
 mC2 I /.A .m/
 mC1 / D B B    C :
C
@    A
  

We choose a Householder reflection M1 that changes the first three rows to eliminate
.0/ .0/
the entries b21 and b31 :
0 1
˝ ˝ ˝ ˝ ˝
B0 ˝ ˝ ˝ ˝C
B C
B .1/ WD M1 B .0/ DB
B0 ˝ ˝ ˝ ˝CC:
@    A
  

.1/
A second Householder reflection M2 works on the rows two to four to eliminate b32
.1/
and b42 :
0 1
    
B ˝ ˝ ˝ ˝C
B C
.2/
B WD M2 B D B .1/ B 0 ˝ ˝ ˝C C:
@ 0 ˝ ˝ ˝A
  
In the same way, we can construct reflections M3 and M4 acting on the rows three to
five and four to five, respectively, that complete the transformation to upper triangular
128 Chapter 5 QR iteration

form:
0 1
    
B    C
B C
B .3/ WD M3 B .2/ B
DB ˝ ˝ ˝CC;
@ 0 ˝ ˝A
0 ˝ ˝
0 1
    
B    C
B C
B .4/ WD M4 B .3/ B
DB   CC:
@ ˝ ˝A
0 ˝

We let Q b
.mC2/
WD M1 : : : Mn1
 and Rdbl WD B .n1/ and have found a QR factor-
dbl
ization.
In order to construct an implicit method, we can once again use Corollary 5.14:
we already know that A.mC2/ D .Q b .mC2/ / A.m/ Qb .mC2/ and A.m/ are Hessenberg
dbl dbl
matrices. If we can find a unitary matrix Q e .mC2/ such that eA.mC2/ WD .Q e .mC2/ /
dbl dbl
A.m/ Qe .mC2/ is a Hessenberg matrix and the first columns of Q e .mC2/ and Q b .mC2/
dbl dbl dbl
are identical, we can replace A.mC2/ by e A.mC2/ without harming the convergence of
the QR iteration.
We again employ bulge chasing to construct the matrix Q e .mC2/ : we aim for
dbl
e .mC2/ D M  M
Q f : : : M
f , where the unitary transformations M f2 ; : : : ; M
fn1 do
dbl 1 2 n1
not change the first row and ensure that Ae.mC2/ is a Hessenberg matrix. We start with
the Hessenberg matrix
0 1
    
B    C
B C
H WD A .m/
DBB    C C
@   A
 

and apply the first Householder reflection M1 . Since it works on the first three rows,
we end up with a larger bulge:
0 1
˝ ˝ ˝ ˝ ˝
B˝ ˝ ˝ ˝ ˝C
B C
M1 H D B B˝ ˝ ˝ ˝ ˝C ;
C
@   A
 
Section 5.6 Multiple-shift strategies 129

0 1
˝ ˝ ˝  
B˝ ˝ ˝  C
B C
H .1/ WD M1 HM1 D B
B ˝ ˝  C
C:
@  ˝  A
 

.1/
We can eliminate the offending entries h31 and h.1/
41 by applying a Householder re-
f2 to the rows two to four and find
flection M
0 1
    
B˝ ˝ ˝ ˝ ˝C
B C
f2 H .1/ D B 0 ˝ ˝ ˝ ˝C
M B C;
@0 ˝ ˝ ˝ ˝A
 
0 1
 ˝ ˝ ˝ 
B ˝ ˝ ˝ C
B C
H .2/ WD M f2 D B
f2 H .1/ M ˝ ˝ ˝ C
B C:
@  ˝ ˝ A
  ˝ 

As in the case of the simple implicit QR iteration, we have managed to chase the
bulge one step towards the right lower edge of the matrix. We apply a Householder
f3 to the rows three to five to eliminate h.2/ and h.2/ :
reflection M 42 52
0 1
    
B    C
B C
f3 H .2/ D B
M ˝ ˝ ˝ ˝C
B C;
@ 0 ˝ ˝ ˝A
0 ˝ ˝ ˝
0 1
  ˝ ˝ ˝
B  ˝ ˝ ˝C
B C
H .3/ WD M f3 D B
f3 H .2/ M  ˝ ˝ ˝C
B C:
@ ˝ ˝ ˝A
 ˝ ˝
130 Chapter 5 QR iteration

Part of the bulge has already disappeared, now we only have to apply one last
Householder reflection Mf4 to the fourth and fifth row to eliminate h.3/ and get
53
0 1
    
B     C
B C
f4 H .3/ D B    C
M B C;
@ ˝ ˝ ˝A
0 ˝ ˝
0 1
   ˝ ˝
B   ˝ ˝C
B C
f4 D B
f4 H .3/ M
H .4/ WD M   ˝ ˝C
B C:
@  ˝ ˝A
˝ ˝
We have constructed
e fn1 : : : M
A.mC2/ WD H .n1/ D M f2 : : : M
f2 M1 A.m/ M1 M fn1 ;

and this is indeed again a Hessenberg matrix.

Example 5.17 (Francis double shift). A double shift strategy like the one described
here is particularly useful when dealing with real-valued matrices: although all matrix
entries are real, the eigenvalues may be complex. In this case, they have to appear in
conjugate pairs, so it makes sense to perform a double shift using both numbers, i.e.,
to choose mC2 D N mC1 . This approach is particularly elegant since the matrix

.A.m/  mC2 I /.A.m/  mC1 I / D .A.m/ /2  .mC1 C mC2 /A.m/

C mC2 mC1 I
D .A.m/ /2  .mC1 C N mC1 /A.m/
C N mC1 mC1 I
D .A.m/ /2  2<.mC1 /A.m/ C jmC1 j2 I

defining the first Householder reflection M1 is real-valued, allowing us to perform the

entire computation without resorting to complex numbers.

Remark 5.18 (Multiple-shift strategies). Obviously, we do not have to stop with dou-
ble shifts, we can also apply r shifts 1 ; : : : ; r simultaneously by replacing the equa-
tion (5.18) by
Qb .mCr/ R.mCr/ D .A  r I / : : : .A  1 I /:
multi multi
The first Householder reflection M1 now has to eliminate r entries in the first column,
therefore the implicit method has to chase a bulge with r rows out of the matrix.
Sophisticated multiple-shift strategies [4] can be used to apply a large number of shifts
simultaneously.
Section 5.6 Multiple-shift strategies 131

Remark 5.19 (Aggressive deflation). During a multiple-shift algorithm, some eigen-

values tend to converge more rapidly than others, but the standard stopping criteria
fail to take advantage of this property. Aggressive deflation methods [5] can be used
to detect converged eigenvalues and apply deflation steps to improve the efficiency
significantly.
Chapter 6

Bisection methods 

Summary
If we are mainly interested in the eigenvalues of a matrix, we can look for roots of
its characteristic polynomial (cf. Proposition 2.13). In general, evaluating the char-
acteristic polynomials takes too long, but if the matrix is self-adjoint, Householder
transformations can be used to make it tridiagonal (cf. Figure 5.4), and the character-
istic polynomial of tridiagonal matrices can be evaluated efficiently. In this setting, a
bisection algorithm can be used to compute the eigenvalues. Similar to all bisection
methods, it is guaranteed to always converge at a rate of 1=2. In the case of character-
istic polynomials of self-adjoint tridiagonal matrices, it is even possible to refine the
bisection method to choose which eigenvalues are computed.

Learning targets
 Introduce the bisection algorithm for eigenvalue problems.
 Modify the algorithm to compute specific eigenvalues.
 Use Gershgorin circles to obtain a reliable initial guess for the bisection algo-
rithm.
Using the algorithm given in Figure 5.4, we can turn any matrix A 2 F nn into
a Hessenberg matrix H 2 F nn by using unitary similarity transformations, i.e., we
have
Q AQ D H
with a unitary matrix Q 2 F nn . If A is self-adjoint, i.e., if we have A D A , we find
H  D .Q AQ/ D Q A Q D Q AQ D H;

i.e., the Hessenberg matrix is also self-adjoint, and therefore tridiagonal.

In this chapter, we consider only self-adjoint tridiagonal matrices
0 1
a1 bN1
B :: C
Bb1 a2 : C
T DB B C with a 2 Rn ; b 2 F n1 : (6.1)
: : C
@ : : : N
: bn1 A
bn1 an
 133

We are looking for eigenvalues, and due to Proposition 2.13, these coincide with the
zeros of the characteristic polynomial pT given by
0 1
t  a1 bN1
B :: :: C
B b1 : : C
pT .t / D det.tI  T / D det B C for all t 2 F:
B :: :: C
@ : : bNn1 A
bn1 t  an
In order to derive an efficient algorithm for evaluating pT , we consider the character-
istic polynomials
0 1
t  a1 bN1
B :: :: C
B b1 : : C
B
pm .t / WD det B C for all m 2 ¹1; : : : ; nº; t 2 F
: : C
@ : : : : N
bm1 A
bm1 t  am
corresponding to the m-th principal submatrices of T . We are looking for a recurrence
relation that allows us to compute pn D pT .
Computing p1 and p2 is straightforward, therefore we only have to consider the
computation of pm for m 2 ¹3; : : : ; nº. We can evaluate the characteristic polynomial
pm for a given t 2 F by Laplace expansion in the last column and row:
0 1
t  a1 bN1
B :: :: C
B b1 : : C
B C
B :: :: N C
B
pm .t / D det B : :  b C
m3 C
B bm3 t  am2 bm2 N C
B C
@ bm2 t  am1 bm1 AN
bm1 t  am
0 1
t  a1 bN1
B :: :: C
B b1 : : C
B C
D .t  am / det B
B :: :: N
C
C
B
: : bm3 C
@ bm3 t  am2 bm2 AN
bm2 t  am1
0 1
t  a1 bN1
B :: :: C
B b1 : : C
B C
 .bNm1 / det B B ::
:
::
:  Nm3
b
C
C
B C
@ bm3 t  am2 bm2 AN
bm1
134 Chapter 6 Bisection methods

0 1
t  a1 bN1
B :: :: C
B b1 : : C
D .t  am / det B
B
C
C
: ::
@ :: : bNm2 A
bm2 t  am1
0 1
t  a1 bN1
B :: :: C
B b1 : : C
 jbm1 j2 det B
B
C
C
: ::
@ : : : bNm3 A
bm3 t  am2
2
D .t  am /pm1 .t /  jbm1 j pm2 .t /:

This equation allows us to compute p1 ; p2 ; : : : ; pn in not more than 6n operations by

the recurrence relation
p1 .t / D t  a1 ; (6.2a)
p2 .t / D .t  a2 /p1 .t /  jb1 j2 ; (6.2b)
2
pm .t / D .t  am /pm1 .t /  jbm1 j pm2 .t / for all m 2 ¹3; : : : ; nº; t 2 F:
(6.2c)
Provided that we know ˛; ˇ 2 R with ˛ < ˇ and pn .˛/pn .ˇ/  0, the intermediate
value theorem states that there exists a root  2 Œ˛; ˇ of pn . We can apply a bisection
method to approximate this root: we let
WD .ˇ C ˛/=2 denote the midpoint of the
interval and check whether pn .˛/pn .
/  0 holds. If this is the case, we know that
a root exists in Œ˛;
. Otherwise, we observe p.
/p.ˇ/  0 and conclude that a root
exists in Œ
; ˇ. Repeating this process leads to an arbitrarily small interval containing
a root.
This approach has two major disadvantages: we can only use it to compute one
root, and we require an initial guess for the interval Œ˛; ˇ such that the signs of pn .˛/
and pn .ˇ/ differ. The rest of this chapter is dedicated to fixing these problems and
thus providing us with a reliable method for finding the eigenvalues.

6.1 Sturm chains

We are interested in computing the roots of the characteristic polynomial pT . Assume
that pT has n simple roots 1 < 2 <    < n , and consider the derivative pT0 . For
each i 2 ¹1; : : : ; n  1º, the mean value theorem states that pT .i / D 0 D pT .i C1 /
implies that there has to be a zero 0i of pT0 in Œi ; i C1 . Since i and i C1 are
simple, we have i < 0i < i C1 . Since pT0 is a non-zero polynomial of degree n  1,
it cannot have more than these n  1 roots, and all of the roots have to be simple.
A straightforward induction yields that for m 2 ¹1; : : : ; nº, the m-th derivative
.m/
pT has exactly n  m simple roots and that these roots are situated between the
Section 6.1 Sturm chains 135

(4)
pT

(3)
pT

p"T

p'T

pT

Figure 6.1. Sign pattern for pT and its derivatives. The boxes represent the interval Œa; b,
subintervals with negative values of the polynomials are marked in blue.

.m1/
roots of pT . We can take advantage of this property to determine how many roots
.m/
of pT are located within a given interval Œa; b: since all roots of pT are simple, the
.m/
polynomial has to change its sign at each root. Since the roots of pT are interleaved
.mC1/
with the roots of pT , the signs change in a pattern like the one given in Figure 6.1.
We introduce a function s giving us the number of sign changes between pT and
its derivatives for any points  2 R
.m/ .mC1/
s./ WD #¹m 2 ¹0; : : : ; n  1º W pT ./pT ./ < 0º

and see that it decreases by one each time  passes from left to right across a root
of pT and remains otherwise constant: it can be used to count the number of roots.
Computing all derivatives of pT is possible, but not very elegant. Fortunately,
.m/
we can replace pT by the characteristic polynomial qm WD pnm of the .n  m/-
th principal submatrix without losing the important sign-change properties. Since
p1 ; p2 ; : : : ; pn are computed anyway during our algorithm for evaluating pn D pT ,
checking the signs requires almost no additional effort and we obtain a very efficient
algorithm.

Definition 6.1 (Sturm chain). Let .q0 ; q1 ; : : : ; qn / be a tuple of polynomials. We call

it a Sturm chain if the following conditions hold:
1. All zeros of q0 are simple.
2. For  2 R with q0 ./ D 0, we have q00 ./q1 ./ > 0.

3. For  2 R, m 2 ¹1; : : : ; n  1º with qm ./ D 0, we have qm1 ./qmC1 ./ < 0.

4. qn has no zeros.
136 Chapter 6 Bisection methods

Theorem 6.2 (Number of zeros). Let .q0 ; q1 ; : : : ; qn / be a Sturm chain. We define

S./ WD ¹m 2 ¹0; : : : ; n  1º W qm ./qmC1 ./ < 0 or qm ./ D 0º;

s./ WD #S./ for all  2 R:

Then we have

s.a/  s.b/ D #¹ 2 .a; b W q0 ./ D 0º for all a; b 2 R; a < b;

i.e., the function s counts the number of zeros of q0 .

Proof. We are interested in the zeros of the polynomials q0 ; : : : ; qn , i.e., in the set

Z./ WD ¹m 2 ¹0; : : : ; nº W qm ./ D 0º for all  2 R:

The function s changes it value only if the signs of the functions qm change, i.e., in
points  2 R with Z./ ¤ ;.
Let  2 R with Z./ ¤ ;. Since the set of zeros of polynomials cannot have limit
points, there is an 2 R>0 such that Z./ D ; and q00 ./ ¤ 0 for all  2 .;  C .
Let m 2 Z./.
Case 1: Assume m > 0. Condition 4 yields m ¤ n and therefore m 2 ¹1; : : : ; n  1º.
Condition 3 implies that qm1 ./ and qmC1 ./ are not zero and have opposite signs,
and due to the choice of , both qm1 and qmC1 cannot change their signs in Œ; C .
If m 2 S. C /, we have qm . C /qmC1 . C / < 0 and therefore
qm1 . C /qm . C / > 0, i.e., m  1 62 S. C /.
On the other hand if m 62 S. C /, we have qm . C /qmC1 . C / > 0 and
therefore qm1 . C /qm . C / < 0, i.e., m  1 2 S. C /.
cm WD ¹m  1; mº and b
We let M S m ./ WD S./ \ ¹m  1; mº for all  2 Œ;  C 
and summarize our findings as

#b
S m ./ D 1 for all  2 .;  C :

Since qm1 ./ is not zero, we also have b S m ./ D ¹mº and therefore #b
S m ./ D 1
for all  2 Œ;  C .
Case 2: Assume now m D 0. Due to condition 2, we have q00 ./q1 ./ > 0, and due
to our choice of , q00 cannot change its sign in .;  C .
By the fundamental theorem of calculus, we obtain
Z C
q0 . C /q1 ./ D q00 .t /q1 ./dt > 0;


and since q1 cannot change its sign in Œ;  C , we conclude

q0 . C /q1 . C / > 0;
Section 6.1 Sturm chains 137

c0 WD ¹0º and
i.e., 0 62 S. C /. By definition, we also have 0 2 S./. We let M
S 0 ./ WD S./\¹0º and summarize our result as #S 0 ./ D 1 and #b
b b S 0 .C / D 0.
Conclusion: Due to our choice of , the set
[
R WD S./ n cm
M
m2Z./

does not depend on  2 Œ;  C  and is therefore well-defined. The equation

[
S./ D R [ b
S m ./ for all  2 Œ;  C 
m2Z./

defines unions of disjoint sets, so we have

X
s./ D #S./ D #R C #b
S m ./ for all  2 Œ;  C :
m2Z./

We have already proven that #b

S m ./ D 1 holds for all  2 Œ; C , m 2 Z./n¹0º.
We also have #.S./\¹0º/ D 1 and #.S.C /\¹0º/ D 0 if 0 2 Z./ and conclude
´
s./ if 0 62 Z./;
s. C / D ;
s./  1 otherwise

i.e., s decreases by one each time we pass a zero of q0 .

In order to apply this general result to the problem of computing eigenvalues of

the tridiagonal matrix T , we have to check that the polynomials q0 WD pn ; q1 WD
pn1 ; : : : ; qn1 WD p1 ; qn WD 1 are a Sturm chain. This is not always the case: if we
choose T D I , the polynomials qm have a zero of multiplicity n  m at  D 1, so
conditions 2 and 3 are violated. Fortunately, there is a simple criterion that helps us
avoid this problem:

Definition 6.3 (Irreducible matrix). A Hessenberg matrix H 2 F nn is called irre-

ducible (or unreduced) if

hi C1;i ¤ 0 for all i 2 ¹1; : : : ; n  1º;

i.e., if all sub-diagonal elements are non-zero.

Checking whether a Hessenberg matrix is irreducible is straightforward, and if it is

not, deflation (cf. Section 5.4) can be used to split it into irreducible submatrices.
138 Chapter 6 Bisection methods

Theorem 6.4 (Sturm chain). Let T 2 F nn be an irreducible tridiagonal matrix of

the form (6.1), i.e., let

bi ¤ 0 for all i 2 ¹1; : : : ; n  1º:

Then the polynomials given by

qn WD 1; qm WD pnm for all m 2 ¹0; : : : ; n  1º

are a Sturm chain.

Proof. We are looking for eigenvectors and eigenvalues of the matrix T . Given a
t 2 R, we look for a non-zero vector e.t / 2 F n such that the first n  1 components
of .tI  T /e.t / vanish. If the last component also vanishes, e.t / is an eigenvector for
the eigenvalue t .
In order to ensure e.t / ¤ 0, we choose e1 .t / D 1. By considering the first row of
.tI  T /e.t /, we obtain

0 D .t  a1 /e1 .t /  bN1 e2 .t /;
bN1 e2 .t / D .t  a1 /e1 .t / D p1 .t /:

For m 2 ¹2; : : : ; n  1º, we have

0 D bm1 em1 .t / C .t  am /em .t /  bNm emC1 .t /;

bNm emC1 .t / D .t  am /em .t /  bm1 em1 .t /: (6.3)

The right-hand side resembles (6.2c), and we can take advantage of this resemblance:
we define p0 WD 1 and let

pm1 .t /
em .t / WD Qm1 for all m 2 ¹1; : : : ; nº; t 2 R:
bNk
kD1

By definition, we have e1 .t / D 1, and we also obtain

bm1 em1 .t / C .t  am /em .t /  bNm emC1 .t /

pm2 .t / pm1 .t / pm .t /
D bm1 Qm2 C .t  am / Qm1  bNm Qm
Nbk Nbk N
kD1 kD1 kD1 bk
jbm1 j pm2 .t /
2 C .t  am /pm1 .t /  pm .t /
D Qm1 N
kD1 bk
pm .t /  pm .t /
D Qm1 N D0 for all m 2 ¹3; : : : ; n  1º:
kD1 bk
Section 6.1 Sturm chains 139

using (6.2c) in the last step. (6.3) implies that the vector e.t / meets our requirements:
the first n  1 components of .tI  T /e.t / are equal to zero. Now we consider the
last component given by
pn2 .t / pn1 .t /

.t / WD bn1 en1 .t / C .t  an /en .t / D bn1 Qn2 C .t  an / Qn1
Nbk N
kD1 kD1 bk
jbn1 j2 pn2 .t / C .t  an /pn1 .t / pn .t /
D Qn1 N D Qn1 for all t 2 R:
kD1 bk kD1bNk

The vector e.t / is an eigenvector for an eigenvalue t 2 R if and only if

.t / D 0
holds, and this is equivalent to pn .t / D 0. In general we have
 
0
.tI  T /e.t / D for all t 2 R;

.t /

and differentiating by t yields

 
0 0
e.t / C .tI  T /e .t / D for all t 2 R: (6.4)

0 .t /

Now we can verify the conditions of Definition 6.1. We begin with condition (2). Let
2 R be a root of q0 D pn , i.e.,
. / D 0. Taking the inner product of (6.4) by the
vector e. / yields

eNn . /
0 . / D he. / C . I  T /e 0 . /; e. /i D ke. /k2 C he 0 . /; . I  T /e. /i
D ke. /k2 > 0;

and we conclude
pn1 . / pn0 . / pn1 . /pn0 . / q1 . /q 0 . /
0 < eNn . /
0 . / D Qn1 Qn1 N D Qn1 D Qn1 0 ;
kD1 jbk j kD1 jbk j
2 2
kD1 bk kD1 bk

so the condition (2) holds. It implies q00 . / ¤ 0, therefore all zeros of q0 are simple
and (1) holds as well. qn D 1 yields condition (4). This leaves only condition (3) to
consider. Let m 2 ¹1; : : : ; n  1º and let  2 R with 0 D qm ./ D pnm ./. The
recurrence relation (6.2c) yields

qm1 ./ D pnmC1 ./ D .  anmC1 /pnm ./  jbnm j2 pnm1 ./
D jbnm j2 qmC1 ./;

i.e., qmC1 ./qm1 ./ D jbnm j2 qmC1

2 ./  0. To complete the proof, we only
have to show qmC1 ./ ¤ 0. We can use (6.2c) to do this: qm ./ D 0 and qmC1 ./ D
0 would imply qmC2 ./ D 0, and a simple induction leads to qn ./ D 0. This would
contradict qn D p0 D 1, so we can conclude qmC1 ./ ¤ 0.
140 Chapter 6 Bisection methods

function sturm_evaluate(t , a, b);

p0 1; p1 t  a1 ;
c 0;
if p0 p1 < 0 or p1 D 0 then c c C 1;
for m D 2 to n do begin
pm .t  am /pm1  jbm1 j2 pm2 ;
if pm pm1 < 0 or pm D 0 then c cC1
end;
return c
end;

procedure sturm_bisection(k, a, b, var ˛, ˇ);

begin
while ˇ  ˛ > do begin

.ˇ C ˛/=2; c sturm_evaluate(
, a, b);
if k  n  c then
ˇ

else
˛

end
end
Figure 6.2. Compute an interval containing the k-th eigenvalue by a bisection method applied
to Sturm chains.

We can use Theorem 6.2 to compute the k-th eigenvalue of the matrix T : we have
´
1 if m is even;
lim pm .t / D for all m 2 ¹1; : : : ; nº;
t!1 1 otherwise

since the leading coefficients of the polynomials are always equal to one, and this
implies
lim s.t / D n:
t!1

Theorem 6.2 states that s decreases by one each time a zero of pT is passed, so for any
t 2 R, n  s.t / gives us the number of zeros less than or equal to t . If we are looking
for the k-th eigenvalue, we simply check whether k  n  s.t / holds: in this case
we continue to search in Rt , otherwise we consider R>t . The resulting algorithm is
given in Figure 6.2.
Section 6.2 Gershgorin discs 141

The bisection method, particularly the evaluation of the function s, can react
very sensitively to rounding errors, since even small perturbations of pm .t / and
pm1 .t / may change the sign and therefore the result.

Remark 6.5 (Slicing the spectrum). The key ingredient of the method is the compu-
tation of the number of eigenvalues smaller and larger than a given value t . Instead of
examining a Sturm sequence, we can also solve this problem by computing factoriza-
tions I  T D LDL of the shifted matrix and examining the number of negative
and positive diagonal elements of the matrix D [33, page 15].

Remark 6.6 (Divide-and-conquer methods). The relatively simple structure of tridi-

agonal matrices can be used to construct alternative solution algorithms. One pos-
sibility are divide-and-conquer methods [7, 12] that split the tridiagonal matrix into
submatrices, compute eigenvalues and eigenvectors of the submatrices, and then com-
bine them to construct eigenvalues and eigenvectors of the original matrix. This class
of algorithms is particularly well-suited for parallel implementations.

Remark 6.7 (MRRR). Based on sufficiently accurate approximations of the eigen-

values, the approach of multiple relatively robust representations (MRRR or MR3 )
employs multiple factorizations I  T D LDL of shifted matrices in order to
compute eigenvectors of T with high relative accuracy [10, 11].

6.2 Gershgorin discs

Let A 2 F nn . In order to use the bisection algorithm, we have to find an interval
Œ˛; ˇ that is guaranteed to contain all eigenvalues of the matrix T . Gershgorin discs
[16] provide us not only with this interval, but can also be used to determine bounds
for the eigenvalues of general matrices.
We use the maximum norm defined by

kxk1 WD max¹jxi j W i 2 ¹1; : : : ; nºº for all vectors x 2 F n

and by

kAk1 WD max¹kAxk1 W x 2 F n ; kxk1 D 1º for all matrices A 2 F nn :

These definitions ensure that we again have compatible norms, i.e.,

kAxk1  kAk1 kxk1 for all A 2 F nn ; x 2 F n : (6.5)

142 Chapter 6 Bisection methods

Combining this property with the definition of the norm yields the estimate
kABk1  kAk1 kBk1 for all A; B 2 F nn (6.6)
that leads to the following simple criterion for the invertibility of perturbed matrices.

Proposition 6.8 (Neumann series). Let X 2 F nn be a matrix satisfying kX k1 < 1.

Then I  X is invertible.
Proof. We have
X
m X
m X
mC1
.I  X / Xk D Xk  X k D I  X mC1 for all m 2 N:
kD0 kD0 kD1

Due to kXk1 < 1 and (6.6), we have

k lim X m k1 D lim kX m k1  lim kXkm
1 D 0;
m!1 m!1 m!1

i.e., limm!1 X m D 0 and therefore

1
X
.I  X / X k D I:
kD0

This implies that I  X is invertible.

In order to apply this result, it would be advantageous to be able to evaluate the
norm kXk1 for a matrix X 2 F nn explicitly. For the maximum norm, this task can
be solved directly:

Proposition 6.9 (Maximum norm). Let A 2 F nn be a matrix. Then we have

8 9
<X n =
kAk1 D
WD max jaij j W i 2 ¹1; : : : ; nº : (6.7)
: ;
j D1

Proof. The triangle inequality yields

8ˇ ˇ 9
<ˇˇ Xn ˇ
ˇ =
kAxk1 D max ˇˇ aij xj ˇˇ W i 2 ¹1; : : : ; nº
:ˇ ˇ ;
j D1
8 9
<X n =
 max jaij j jxj j W i 2 ¹1; : : : ; nº
: ;
j D1
8 9
<X n =
 max jaij j W i 2 ¹1; : : : ; nº kxk1 D
kxk1 for all x 2 F n ;
: ;
j D1

and this implies kAk1 

.
Section 6.2 Gershgorin discs 143

Let i 2 ¹1; : : : ; nº be the index for which the maximum in (6.7) is attained. We
define a vector x 2 F n by
xj WD sgn.aN ij / for all j 2 ¹1; : : : ; nº
and find kxk1 D 1 and
ˇ ˇ ˇ ˇ
ˇ n ˇ ˇ n ˇ
ˇX ˇ ˇX ˇ X n
kAxk1
ˇˇ aij xj ˇˇ D ˇˇ aij sgn.aN ij /ˇˇ D jaij j D

ˇj D1 ˇ ˇj D1 ˇ j D1

so we have also proven kAk1

and therefore kAk1 D
.

Theorem 6.10 (Gershgorin spheres). Let A 2 F nn . We define discs (or intervals)
8 9
ˆ
ˆ >
>
< X n =
Gi WD  2 F W j  ai i j  jaij j for all i 2 ¹1; : : : ; nº:
ˆ >
>
:̂ j D1 ;
j ¤i

Then we have
[
n
.A/  Gi ;
i D1
i.e., the spectrum of A is contained in discs (or intervals) centered at the diagonal
elements.
Proof. (cf. [16, Satz II]) Let  2 .A/, and let
0 1
a11
B :: C
D WD @ : A
ann
be the diagonal part of A. If I  D is not invertible, there has to be an index
i 2 ¹1; : : : ; nº such that  D ai i , therefore we have  2 Gi .
Assume now that I  D is invertible. Since  is an eigenvalue, I  A cannot be
invertible, and the same holds for
.I  D/1 .I  A/ D .I  D/1 .I  D  .A  D// D I  .I  D/1 .A  D/:
Due to Propositions 6.8 and 6.9, this implies
8 9
<X n
ja  d j =
1  k.I  D/1 .A  D/k1 D max
ij ij
W i 2 ¹1; : : : ; nº
: j  ai i j ;
j D1
8 9
ˆ
ˆ >
>
<X n
jaij j =
D max W i 2 ¹1; : : : ; nº :
ˆ j  ai i j >
>
:̂j D1 ;
j ¤i
144 Chapter 6 Bisection methods

We choose an index i 2 ¹1; : : : ; nº for which the maximum is attained and conclude

1 X
n X
n
1 jaij j; j  ai i j  jaij j;
j  ai i j
j D1 j D1
j ¤i j ¤i

i.e.,  2 Gi .

In the context of Sturm chain bisection methods, we can use this result to compute
an interval Œ˛; ˇ given by

˛ WD min¹a1  jb1 j; an  jbn1 j; ai  jbi j  jbi 1 j W i 2 ¹2; : : : ; n  1ºº;

ˇ WD max¹a1 C jb1 j; an C jbn1 j; ai C jbi j C jbi 1 j W i 2 ¹2; : : : ; n  1ºº;

that is guaranteed to include the entire spectrum of T (since T is self-adjoint, Corol-

lary 2.51 yields that all eigenvalues are real, and Theorem 6.10 provides upper and
lower bounds).
Chapter 7

Krylov subspace methods for large sparse

eigenvalue problems

Summary
Large sparse eigenvalue problems typically have an underlying matrix that has a very
large size, but also a special structure so that matrix-vector multiplications can ef-
ficiently be performed, but similarity transformations cannot, so different strategies
for the computation of eigenvalues and eigenvectors are needed. The most common
methods are Krylov subspace methods that can be used to compute some, but not
all, eigenvalues of the matrix. Among those, the Arnoldi iteration and the symmetric
Lanczos algorithm will be considered in detail in this chapter and also the conver-
gence behaviour of these methods will be investigated with the help of an important
tool: Chebyshev polynomials.

Learning targets
 Introduce projection methods for large sparse eigenvalue problems.
 Construct Krylov subspaces and investigate their basic properties.
 Introduce the Arnoldi iteration and the symmetric Lanczos algorithm as exam-
ples of Krylov subspace methods.
 Use Chebyshev polynomials to analyze the convergence behaviour of Krylov
subspaces methods.

7.1 Sparse matrices and projection methods

Many applications in engineering sciences, physics, chemistry et cetera give rise to
eigenvalue problems with matrices A 2 F nn , where n is very large, e.g., n  108
(or maybe even much larger). Typically, these matrices are sparse, i.e., most of the
entries of the matrix are zero. An example of a sparse matrix is the matrix
0 1
2 1
B C
B1 2 : : : C
ADB B C 2 Rnn (7.1)
:: :: C
@ : : 1A
1 2
146 Chapter 7 Krylov subspace methods for large sparse eigenvalue problems

500

1000

1500

2000

0 500 1000 1500 2000

nz = 11328

Figure 7.1. Depiction of a sparse matrix which represents the 5-point discrete Laplacian on a
square 50-by-50 grid using a nested dissection ordering of the gridpoints. Nonzero entries of
the matrix are plotted in blue. “nz” is the total number of nonzero entries.

from (1.2). We observe that regardless how large n is, there are at most three entries in
each row that are distinct from zero. Another more complicated example is depicted
in Figure 7.1. The matrix is of size 2304  2304, but only 11,328 of its 5,308,416
entries are nonzero.
Sparsity is a special matrix structure having two important properties that allow us
to deal with matrices of very large size n. First, it is not necessary to store the matrix
in the conventional way which would require to store all n2 entries. Instead, we only
need to store the nonzero entries and information on their positions within the matrix.
For example, instead of storing the matrix
0 1
1 2 0 0
B0 0 3 0 C
ADB @0 0 0 0 A
C

0 0 4 5
Section 7.1 Sparse matrices and projection methods 147

as a 4  4 array, we may store the three vectors

0 1 0 1 0 1
1 1 1
B2C B1C B2C
B C B C B C
l DB C B C B C
B3C ; r D B2C ; c D B3C ;
@4A @4A @3A
5 4 4

instead, where l is the list of nonzero entries of A, and r and c are vectors containing
the corresponding row and column indices of the entries, respectively. The matrix
A D .aij / can be easily reconstructed from these three vectors by setting
²
lk if rk D i and ck D j;
aij D (7.2)
0 else.

Exercise 7.1 (Storing sparse matrices). Construct vectors l, r, c, so that the ma-
trix generated by (7.2) corresponds to the matrix A in (7.1) for n D 6.

The second important property is that matrix-vector multiplications with sparse

matrices can be performed considerably faster than compared to the conventional way.
This is due to the fact that most entries of the matrix are zero. Multiplying a matrix
A 2 F nn and a vector x 2 F n , we have to multiply each of the n rows of A with x
which adds up to n  n D n2 multiplications and n  .n  1/ D n2  n additions, i.e.,
in total 2n2  n operations. On the other hand, if like in (7.1) only three entries per
row are nonzero (except for the first and last row), then multiplying A with a vector
x 2 F n costs 3n2 multiplications and 2n2 additions, i.e., 5n4 operations. Thus,
the cost for computing a matrix-vector product is growing approximately linearly in n
in this example as opposed to quadratically in the conventional case.
In some applications, it may actually be more convenient to not store the matrix A
at all, but write a procedure instead that computes Ax for a given vector x. In this
situation, we could think of our matrix A as being a black box that transforms a given
vector x to the vector y D Ax.

x ! A ! Ax (7.3)

For example, for our matrix A from (7.1) such a procedure that computes Ax from a
given vector x is presented in Figure 7.2. In this situation, we cannot apply similarity
transformations anymore, because A is not given explicitly. This also refers to the
case that we actually do store the matrix, because then similarity transformations may
(and do in general) destroy the sparsity of the matrix. We would then have to store
the matrix in the conventional way and this may not be possible because of its large
size. Therefore, the efficient QR iteration from Chapter 5 is no longer a method of
choice as it heavily manipulates the given matrix. Therefore, we have to come up with
148 Chapter 7 Krylov subspace methods for large sparse eigenvalue problems

procedure sparse_matrix_mult(x, var y);

begin
y1 2x1  x2 ;
for i 2 ¹2; : : : ; n  1º do
yi 2xi  xi 1  xi C1 ;
yn 2xn  xn1
end
Figure 7.2. Sparse matrix multiplication with the matrix A from (7.1).

a different strategy which should be based on matrix-vector multiplications, because

this is the only way to obtain information on our matrix A.
As mentioned in the beginning of this chapter, we are particularly interested in the
case that n is very large. In most applications, not all eigenvalues and eigenvectors
are needed, but just a few of them are of interest, for example, the eigenvalues that
are largest or smallest in modulus, e.g., in Example 1.1 from the introduction, we are
interested in a few of the smallest eigenvalues of A. However, one should keep in
mind that for sizes as large as 108 “a few” may still mean “a few hundred” or “a few
thousand”.
A concept commonly used for the treatment of large sparse eigenvalue problems
is the concept of projection methods. The idea is pretty simple: if A 2 F nn is a
matrix and if .; v/ 2 F  F n is an eigenpair of A, then Av D v, i.e., the vector
Av  v is the zero vector which is orthogonal to the space F n . Now let us replace
the full space F n by a subspace X  F n of smaller dimension, i.e., we will look for
pairs .; x/ 2 F  X so that Ax  x ? X, i.e., Ax  x is orthogonal to X. This
condition is called Ritz–Galerkin condition as it is analogous to the Ritz–Galerkin
conditions used in finite element methods.

Definition 7.2 (Ritz pair). Let A 2 F nn and let X  F n be a subspace. Then a
pair .; x/ 2 F  X is called a Ritz pair of A with respect to X if it satisfies the
Ritz–Galerkin condition
Ax  x ? X:
If .; x/ is a Ritz pair, then  is called a Ritz value and x is called the associated Ritz
vector.

At this point, the following questions arise naturally:

1. How do we choose a suitable subspace X?

2. How do we compute Ritz values and vectors?

3. Will the computed Ritz pairs be “good” approximations to eigenpairs?

Section 7.2 Krylov subspaces 149

We will turn to question 1. and 3. in the next section. Question 2., however, has a
fairly immediate answer if we are given an orthonormal basis of X, i.e., a basis with
pairwise perpendicular vectors that all have norm one.

Theorem 7.3 (Detection of Ritz pairs). Let A 2 F nn and let X  F n be a subspace.
Furthermore let Q 2 F nm be an isometric matrix such that X D R.Q/, i.e., the
columns of Q form a orthonormal basis of X. Then the following two conditions are
equivalent for x D Qy 2 X, where y 2 F m ,
1. .; Qy/ 2 F  X is a Ritz pair of A with respect to X;
2. .; y/ 2 F  F m is an eigenpair of the m  m matrix Q AQ.
Proof. .; y/ is an eigenpair of Q  AQ if and only if
Q AQy D y D Q Qy
which itself is equivalent to
Q .AQy  Qy/ D 0:
The latter identity means that AQy Qy is orthogonal to R.Q/ D X, i.e., .; Qy/
is a Ritz pair of A with respect to X.

7.2 Krylov subspaces

What kind of subspaces are suitable for computing Ritz pairs that are good approxi-
mations to eigenpairs of a given matrix A 2 F nn ? Recalling the power iteration from
Section 4.8, we know from Theorem 4.2 that under some mild assumptions the angles
between Am x and an eigenvector associated with the dominant eigenvalue converge to
zero. We can therefore expect that for a given starting vector x the subspace spanned
by x; Ax; A2 x; : : : ; Am x will contain at least good approximations to eigenvectors
associated with the dominant eigenvalue of A. Given the fact that multiplying a given
vector by A may be the only way to obtain information on the matrix A (cf. (7.3)), this
kind of subspaces seems to be a natural candidate for the use of projection methods.
Definition 7.4 (Krylov subspace). Let A 2 F nn and x 2 F n . Then

Km .A; x/ WD span.x; Ax; : : : ; Am1 x/

is called the Krylov subspace of order m generated by A and x.

The name refers to the Russian mathematician Alexei Krylov who used the se-
quence .x; Ax; A2 x; : : : / to find the coefficients of the characteristic polynomial of a
given matrix A, see [25]. Obviously, we always have dim Km .A; x/  m, but the di-
mension can be considerably smaller than m. For example, if x ¤ 0 is an eigenvector
of A, then we clearly have dim Km .A; x/ D 1 for all m 2 N.
150 Chapter 7 Krylov subspace methods for large sparse eigenvalue problems

Theorem 7.5 (Dimension of Krylov subspaces). Let A 2 F nn and x 2 F n . Then

there exists a number ` 2 N0 with the following properties:
1. dim Km .A; x/ D m for all m  `;
2. K` .A; x/ D Km .A; x/ and dim Km .A; x/ D ` for all `  m.
In particular, K` .A; x/ is an invariant subspace with respect to A.

Proof. 1. If x D 0, then there is nothing to show, and we have ` D 0. If x ¤ 0, then

let `
1 be the largest integer for which x; Ax; : : : ; A`1 x are linearly independent.
Then clearly `  n as dim F n D n. As long as m  `, it follows that the vectors
x; Ax; : : : ; Am1 x are also linearly independent, and we obtain dim Km .A; x/ D m.
2. Next let `  m. By the construction of `, it follows that x; Ax; : : : ; A` x are
linearly dependent, i.e., there exist scalars ˛0 ; : : : ; ˛` , not all being zero, such that

˛0 x C ˛1 Ax C    C ˛` A` x D 0:

Then ˛` ¤ 0, because otherwise the linear independence of x; Ax; : : : ; A`1 x would

imply ˛0 D    D ˛`1 D 0 in contradiction to the assumption that not all ˛i ,
i D 0; : : : ; ` are zero. But then, we have
˛0 ˛`1 `1
A` x D  x    A x; (7.4)
˛` ˛`

i.e., A` x is a linear combination of x; Ax; : : : ; A`1 x.

We now show by induction on m
` that K` .A; x/ D Km .A; x/ which by 1. im-
mediately implies dim Km .A; x/ D `. The case m D ` is clear, so for the induction
step assume ` < m. By construction of the Krylov subspaces and ` < m, it follows
that K` .A; x/  Km .A; x/. By multiplying (7.4) on both sides with Am` , we obtain
˛0 m` ˛
Am x D  A x      `1 Am1 x
˛` ˛`

and thus Am x 2 span.Am` x; : : : ; Am1 x/. By the induction hypothesis, we have

K` .A; x/ D Km1 .A; x/ which implies x; Ax; : : : ; Am1 x 2 K` .A; x/. But then
we also have Am x 2 K` .A; x/ which finally proves K` .A; x/ D Km .A; x/. The
proof that K` .A; x/ is an invariant subspace with respect to A is left as an exercise.

Exercise 7.6 (Properties of Krylov subspaces). Let A 2 F nn and x 2 F n n ¹0º.

Show that K` .A; x/ is the smallest invariant subspace (with respect to A) that
contains x, that is, K` .A; x/ is an invariant subspace containing x, and any other
invariant subspace X containing x satisfies K` .A; x/  X.
Section 7.2 Krylov subspaces 151

For any polynomial p.t / D ˇ` t ` C    C ˇ1 t C ˇ0 with coefficients in F, we can

define a polynomial map p W F nn ! F nn by

p.A/ WD ˇ` A` C    C ˇ1 A C ˇ0 In ;

where A 2 F nn . The vector space of all polynomials over F will be denoted by F Œt 
and by Fm Œt  we denote the subspace of all polynomials of degree not exceeding m.
Then we have the following characterization of Krylov subspaces.

Lemma 7.7 (Structure of Krylov subspaces). Let A 2 F nn and x 2 F n . Then

Km .A; x/ D ¹p.A/x j p 2 Fm1 Œt º : (7.5)

Proof. Let y 2 Km .A; x/ D span.x; Ax; : : : ; Am1 x/. Then there exist coefficients
ˇ0 ; : : : ; ˇm1 2 F such that

y D ˇ0 x C ˇ1 Ax C    C ˇm1 Am1 x D p.A/x; (7.6)

where p.t / D ˇm1 t m1 C    C ˇ1 t C ˇ0 which proves “”. On the other hand, if
we start with an arbitrary polynomial p.t / D ˇm1 t m1 C    C ˇ1 t C ˇ0 , then the
identity in (7.6) implies p.A/x 2 Km .A; x/.

Let A 2 F nn , x 2 F n n ¹0º, and let ` be as in Theorem 7.5. Then from (7.4)
one finds that there exist coefficients ˇ0 ; : : : ; ˇ`1 2 F such that

A` x D ˇ0 x      ˇ`1 A`1 x;

˛i
where ˇi D ˛` . This equation can also be written as

p.A/x D 0; where p.t / D t ` C ˇ`1 t `1 C    C ˇ1 t C ˇ0 : (7.7)

As x; Ax; : : : ; A`1 x are linearly independent, the coefficients ˇ0 ; : : : ; ˇ`1 2

F are uniquely determined and q.A/x ¤ 0 for all polynomials q 2 FŒt  of degree
less than `. Thus, p in (7.7) is the unique monic polynomial of minimal degree
satisfying p.A/x D 0 and it is called the minimal polynomial of x with respect
to A. (A polynomial is called monic if the leading coefficient is equal to one.)
If ` D n, then the minimal polynomial of x with respect to A is equal to
the characteristic polynomial pA of A, because the theorem of Cayley–Hamilton
implies that pA .A/ D 0. This fact follows immediately from the uniqueness of
the minimal polynomial and the characteristic polynomial being monic.
152 Chapter 7 Krylov subspace methods for large sparse eigenvalue problems

Exercise 7.8 (Minimal polynomials). Let A 2 F nn and x 2 F n . Show that

the minimal polynomial p of x with respect to A is a divisor of the characteristic
polynomial pA of A, i.e., there exists a polynomial q 2 FŒt  such that pA D p  q.
Hint: Use the remainder theorem, i.e., for nonzero polynomials p1 ; p2 , there
exist unique polynomials q; r such that p2 D p1  q C r and deg r < deg p1 .

Exercise 7.9 (Dimensions of Krylov subspaces). Consider the matrix

0 1
1 0 0
A D @0 2 0A ;
0 0 c

where c 2 R. Show that there exists vectors x such that K3 .A; x/ D R3 if

and only if c ¤ 1; 2. For these values of c, determine all vectors y for which
K3 .A; y/ ¤ R3 .

7.3 Gram–Schmidt process

In order to compute Ritz pairs with respect to a subspace X of F n , we need to compute
an orthonormal basis for X by Theorem 7.3—a problem that we already encountered
in Section 4.7, where we used Householder orthogonalization for the computation
of orthonormal bases. In our particular situation, we are interested in computing an
orthonormal basis of a Krylov subspace Kk .A; x/ D span.x; Ax; : : : ; Ak1 x/ for
many values of k. Suppose that dim Kk .A; x/ D k and that we have constructed an
orthonormal basis q1 ; : : : ; qk of Kk .A; x/. If we then consider the next larger Krylov
subspace KkC1 .A; x/ (assuming dim KkC1 .A; x/ D k C 1), we observe that

KkC1 .A; x/ D span.x; Ax; : : : ; Ak1 x; Ak x/ D span.q1 ; : : : ; qk ; Ak x/:

Thus, in order to obtain an orthonormal basis of KkC1 .A; x/, the only task is to
orthonormalize the vector Ak x against the previously constructed vectors q1 ; : : : ; qk .
If we do this for k D 1; : : : ; m (assuming dim Kk .A; x/ D k for k D 1; : : : ; m), we
will finally obtain an orthonormal basis q1 ; : : : ; qm of Km .A; x/ that satisfies

Kk .A; x/ D span.x; Ax; : : : ; Ak1 x/ D span.q1 ; : : : ; qk /; k D 1; : : : ; m: (7.8)

For this purpose, the so-called Gram–Schmidt process is more appropriate than
Householder orthogonalization.
For deriving the corresponding algorithm in general, let us assume that we have m
linearly independent vectors v1 ; : : : ; vm 2 F n . Our task is to construct an orthonormal
basis q1 ; : : : ; qm of V D span.v1 ; : : : ; vm / such that

span.v1 ; : : : ; vk / D span.q1 ; : : : ; qk /; k D 1; : : : ; m:
Section 7.3 Gram–Schmidt process 153

*

 6
v2 
  e
q 2 D v2  r12 q1

- - -
q1 v1 r12 q1

Figure 7.3. Second step in the Gram–Schmidt process interpreted geometrically.

The first step is immediate, we only have to normalize the vector v1 :

1
q1 WD v1 :
kv1 k

For the second step, we look for a vector e q 2 2 span.v1 ; v2 / D span.q1 ; v2 / that is
orthogonal to q1 . Note that eq 2 must have a component in the direction v2 (that is
e
q 2 D ˛1 q1 C ˛2 v2 , where ˛2 ¤ 0), because otherwise q1 and e q 2 would be linearly
dependent. As we have to normalize the vector e q 2 anyhow, we may assume without
loss of generality that ˛2 D 1, i.e., e
q 2 has the form

e
q 2 WD v2  r12 q1 ; (7.9)

for some r12 2 F. (The minus sign in (7.9) has been introduced for notational
convenience—this will become clear later.) Then the requirement that e
q 2 be orthogo-
nal to q1 implies

0 D he
q 2 ; q1 i D hv2  r12 q1 ; q1 i D hv2 ; q1 i  r12 ;

because hq1 ; q1 i D 1. Thus, we obtain the vector q2 by

r12 WD hv2 ; q1 i (7.10)

eq 2 WD v2  hv2 ; q1 iq1 ; (7.11)
r22 WD keq2k
1
q2 WD e
q2:
r22

The identity (7.11) can be interpreted geometrically in the following way: the vector
r12 q1 is the orthogonal projection of the vector v2 onto the subspace generated by q1 .
Thus, we obtain by v2  r12 q1 a vector that is orthogonal to q1 , see Figure 7.3.
Assuming we have already computed orthonormal vectors q1 ; : : : ; qk1 satisfying

span.v1 ; : : : ; vj / D span.q1 ; : : : ; qj /; j D 1; : : : ; k  1;
154 Chapter 7 Krylov subspace methods for large sparse eigenvalue problems

procedure gram_schmidt_procedure(v1 ; : : : ; vm , var R, q1 ; : : : ; qm );

begin
for k 2 ¹1; : : : ; mº do begin
for i 2 ¹1; : : : ; k  1º do
rik hvk ; qi i;
qk vk ;
end
for i 2 ¹1; : : : ; k  1º do
qk qk  rik qi ;
rkk kqk k;
end
if rkk D 0 then
STOP
else
1
qk rkk qk
end
end
Figure 7.4. Gram–Schmidt procedure for vectors v1 ; : : : ; vm 2 F n .

we now look for a vector e q k 2 span.v1 ; : : : ; vk / D span.q1 ; : : : ; qk1 ; vk / orthog-

onal to q1 ; : : : ; qk1 . Analogous to the second step, we may assume that e q k has the
form
eq k D vk  r1k q1      rk1;k qk1 ; (7.12)
where r1k ; : : : ; rk1;k 2 F. From he
q k ; qj i D 0 for j D 1; : : : ; k  1, we obtain

rj k D hvk ; qj i; j D 1; : : : ; k  1;

using hqi ; qj i D ıj i . Here, the vector r1k q1 C    C rk1;k qk1 can be interpreted as
the orthogonal projection of vk onto the subspace spanned by q1 ; : : : ; qk1 . We then
obtain the vector qk as follows.

rj k WD hvk ; qj i; j D 1; : : : ; k  1
q k WD vk  r1k q1      rk1;k qk1 ;
e
rkk WD keqk k
1
qk WD eqk :
rkk

The resulting algorithm is given in Figure 7.4.

The algorithm breaks down if rkk D 0 in some step. But this would mean e qk D 0
in (7.12) showing that vk 2 span.q1 ; : : : ; qk1 / D span.v1 ; : : : ; vk1 /, i.e., v1 ; : : : ; vk
Section 7.3 Gram–Schmidt process 155

are linearly dependent. Thus, in exact arithmetic a breakdown will never occur as long
as the vectors v1 ; : : : ; vm are linearly independent.

Remark 7.10 (Orthonormal bases for nested sets of subspaces). The application of
the Gram–Schmidt process to the linearly independent vectors v1 ; : : : ; vm 2 F n pro-
duces orthonormal vectors q1 ; : : : ; qm 2 F n with the additional property

span.v1 ; : : : ; vk / D span.q1 ; : : : ; qk /; k D 1; : : : ; m: (7.13)

Remark 7.11 (Gram–Schmidt and QR factorization). Rewriting identity (7.12) we

obtain

vk D r1k q1 C    C rk1;k qk1 C e

q k D r1k q1 C    C rkk qk

for k D 1; : : : ; m. Setting A D Œv1 ; : : : ; vm  2 F nm , Q D Œq1 ; : : : ; qm  2 F nm

and 0 1
r11 r12 : : : r1m
B
B 0 r22 : : : :: C
B : C
C mm
RDB : :: :: : C2F ;
@ :: : : : A
:
0 : : : 0 rmm
we obtain A D QR which is a skinny QR factorization of A as introduced in Sec-
tion 4.7. Thus, the Gram–Schmidt process is an alternative method for computing a
QR factorization of a matrix A 2 F nm if the columns of A are linearely independent.

Exercise 7.12 (Orthonormal bases for nested sets of subspaces). Let v1 ; : : : ; vm

denote the columns of the matrix A 2 F nm and let A D QR be a QR factor-
ization of A, where we denote the columns of the isometric matrix Q 2 F nm
by q1 ; : : : ; qm . Show that if the columns of A are linearly independent then the
identity (7.13) holds. What can be said if the columns of A are linearly depen-
dent?

Unfortunately, the numerical behaviour of the Gram–Schmidt procedure is not too

good: in finite precision arithmetic, the computed vectors q1 ; : : : ; qm typically show
a significant loss in orthogonality due to roundoff errors. However, a slight modi-
fication of the algorithm already improves its performance considerably. Instead of
orthogonally projecting vk onto the subspace spanned by q1 ; : : : ; qk , we first project
vk onto the subspace spanned by q1 and then subtract the projection from vk to obtain
.1/ .1/
a vector e
q k that is orthogonal with respect to q1 . Then eq k is projected onto the
.1/
subspace spanned by q2 and this projection is subtracted from e
qk to obtain a vector
.2/
e
qk which is orthogonal to q1 ; q2 . This procedure is then iterated, i.e., we compute

.j / .j 1/ .j 1/
e
qk De
qk  he
qk ; qj iqj ; j D 1; : : : ; k  1
156 Chapter 7 Krylov subspace methods for large sparse eigenvalue problems

procedure modified_gram_schmidt_procedure(v1 ; : : : ; vm , var R, q1 ; : : : ; qm );

begin
for k 2 ¹1; : : : ; mº do begin
qk vk ;
for i 2 ¹1; : : : ; k  1º do begin
rik hqk ; qi i;
qk qk  rik qi
end;
rkk kqk k;
if rkk D 0 then
STOP
else
1
qk rkk qk
end
end
Figure 7.5. Modified Gram–Schmidt procedure for vectors v1 ; : : : ; vm 2 F n .

.0/
starting with e
q k D vk . In exact arithmetic this procedure is equivalent to the Gram–
Schmidt procedure as we can show by induction that

.j /
X
j
e
qk D vk  hvk ; qj iqi ; j D 0; : : : ; k  1:
i D1

Indeed, for j D 0 there is nothing to show and using the induction hypothesis for
j  1, we obtain that

e
qk
.j /
Deq k.j 1/  he
q k.j 1/ ; qj iqj
jX1
* 1
jX
+
D vk  hvk ; qj iqi  vk  hvk ; qj iqi ; qj qj
i D1 i D1
X j
D vk  hvk ; qj iqi ;
i D1

q k.k1/ D e
because hqi ; qj i D ıij . Thus, after k  1 steps we have e q k , where e
qk
is as in (7.12). The corresponding algorithm, the so-called modified Gram–Schmidt
process, is given in Figure 7.5.
A heuristic explanation why the modified Gram–Schmidt process performs better
.j /
than the classical Gram–Schmidt process is that in the computation of e
q k the vector
.j / .j 1/
e
qk is also orthogonalized to the errors made in the computation of e
qk .
Section 7.3 Gram–Schmidt process 157

procedure mgs_with_reorthogonalization(v1 ; : : : ; vm , var R, q1 ; : : : ; qm );

begin
for k 2 ¹1; : : : ; mº do begin
qk vk ;
for i 2 ¹1; : : : ; k  1º do begin
rik hqk ; qi i;
qk qk  rik qi
end;
for i 2 ¹1; : : : ; k  1º do begin
si hqk ; qi i;
qk qk  si qi ;
rik rik C si
end;
rkk kqk k;
if rkk D 0 then
STOP
else
1
qk rkk qk
end
end
Figure 7.6. Modified Gram–Schmidt procedure with reorthogonalization.

Exercise 7.13 (Gram–Schmidt procedure). Use the Gram–Schmidt procedure

and the modified Gram–Schmidt procedure to compute the QR factorization of
the matrix 0 1
1 5 4
A D @2 4 8 A :
2 2 1

Unfortunately, even the modified Gram–Schmidt process still shows a significant

loss in orthogonality in finite precision arithmetic, in particular, if the vectors
v1 ; : : : ; vm are nearly linearly dependent. Nevertheless, the computed basis vectors
are less likely to be nearly linearly independent than the original vectors v1 ; : : : ; vm ,
so the key idea is reorthogonalization. Thus, after orthogonalizing vk against
q1 ; : : : ; qk1 , we orthogonalize the computed vector eq k again with respect to the vec-
tors q1 ; : : : ; qk1 . Thus, using the classical Gram–Schmidt procedure, we compute

q k D vk  r1k q1      rk1;k qk1 ;

e (7.14)
.2/
e
qk De
q k  s1k q1      sk1;k qk1 ; (7.15)
158 Chapter 7 Krylov subspace methods for large sparse eigenvalue problems

G.S. modified G.S. mod. G.S. with

reorthogonaliz.
nD5 5:5  1008 1:1  1011 2:8  1016
n D 10 3:0  1000 1:5  1004 4:4  1016
n D 20 12:7  1000 1:0  1000 5:9  1016

Table 7.1. Defect from orthogonality in some variants of the Gram–Schmidt process. The
results were obtained in Matlab 7.11.0 with unit roundoff u D 253  1:1  1016 .

with rik D hvk ; qi i and sik D he

q k ; qi i. Note that combining (7.14) and (7.15) yields
.2/
q k D vk  .r1k C s1k /q1      .rk1;k C sk1;k /qk1 ;
e

so the values rij have to be updated to rij C sij in the algorithm to yield the corre-
sponding QR factorization. In a similar way, reorthogonalization can be performed
in the modified Gram–Schmidt process. The corresponding algorithm is given in Fig-
ure 7.6.
One may think that reorthogonalizing the computed vector e q k multiple times
against q1 ; : : : ; qk1 would produce an even better result in finite precision arith-
metic. However, it has been observed empirically that in general it suffices to do the
reorthogonalization once. We refer the reader to [49] and [17] for details.

Example 7.14 (Loss of orthogonality). Let Hn 2 Rnn denote the n  n Hilbert

matrix given by
0 1 1 1 1
1 2 3 ::: n
B1 1 1 1 C

B2 3 4 : : : nC1 C
1 B 1 C
Hn D B
D 3
1 1 1
: : : C
nC2 C : (7.16)
i C j C 1 i;j D1;:::;n B
4 5
B :: :: :: : : C
@: : : : : : A
:
1 1 1 1
n nC1 nC2 : : : 2n1

Applying the Gram–Schmidt procedure, the modified Gram–Schmidt procedure, and

the modified Gram–Schmidt procedure with reorthogonalization to the columns of Hn
and assembling the computed orthonormal basis vectors q1 ; : : : ; qn produces a matrix
Qn D Œq1 ; : : : ; qn  2 Rnn with defect from orthogonality as given in Table 7.1. As
a measure of the defect from orthogonality, we computed kIn  Qn Qn k2 .
Section 7.4 Arnoldi iteration 159

The Hilbert matrices Hn from Example 7.14 are highly ill-conditioned in the
sense that the columns of Hn are nearly linearly dependent. As a consequence,
the inverse of Hn contains very large entries even for moderate values of n. Thus,
Hilbert matrices represent a challenge to numerical algorithms and are often used
as test matrices in numerical experiments.

Exercise 7.15 (Hilbert matrices). Compute the inverse of the Hilbert matrices
H2 and H3 from (7.16).

7.4 Arnoldi iteration

Let A 2 F nn be a matrix and let us assume that for a suitable vector x 2 F n we
have dim Km .A; x/ D m. In view of Theorem 7.3, we may then compute an or-
thonormal basis of the Krylov subspace Km .A; x/ in order to compute Ritz pairs
of A with respect to Km .A; x/. A naive approach to compute such a basis would
be to apply the Gram Schmidt process (with reorthogonalization) to the sequence
x; Ax; A2 x; : : : ; Am1 x. As we know from Remark 7.10 this will produce orthonor-
mal vectors q1 ; : : : ; qm satisfying

Kk .A; x/ D span.x; Ax; : : : ; Ak1 x/ D span.q1 ; : : : ; qk /; k D 1; : : : ; m:

(7.17)
k 1
However, we know from Chapter 4 that the sequence .A x/kD0 converges under ap-
propriate assumptions to an eigenvector e1 associated with the dominant eigenvalue,
in the sense that the corresponding sequence of angles between the iterates and e1
converges to zero. Thus, the set of vectors x; Ax; A2 x; : : : ; Am1 x may be in the-
ory linearly independent, but numerically it may be linearly dependent, because the
directions of the vectors are “not different enough”.
Fortunately, a slight modification of our problem allows us to avoid this situation of
nearly linearly independent vectors challenging our orthonormalization algorithm. To
see this, let us assume that q1 ; : : : ; qk in (7.17) have already been constructed. Thus,
it remains to orthogonalize the vector Ak x against the vectors q1 ; : : : ; qk in order to
obtain an orthonormal basis of KkC1 .A; x/. Since

Ak1 x 2 Kk .A; x/ D span.q1 ; : : : ; qk /;

there exist scalars ˛1 ; : : : ; ˛k such that

Ak1 x D ˛1 q1 C    C ˛k qk

which implies
Ak x D ˛1 Aq1 C    C ˛k Aqk :
160 Chapter 7 Krylov subspace methods for large sparse eigenvalue problems

Note that Aq1 ; : : : ; Aqk1 2 Kk .A; x/, because q1 ; : : : ; qk1 2 Kk1 .A; x/. Thus,
with Ak x also Aqk must be contained in KkC1 .A; x/ n Kk .A; x/ from which we
obtain that
KkC1 .A; x/ D span.q1 ; : : : ; qk ; Ak x/ D span.q1 ; : : : ; qk ; Aqk /:

Hence, instead of orthogonalizing Ak x we can orthogonalize the vector Aqk with

respect to the vectors q1 ; : : : ; qk . In exact arithmetic, there is no difference between
these two strategies, but numerically, the vector Aqk will be more “distinct” from the
vectors q1 ; : : : ; qk in the sense explained above which may have a significant effect
in finite precision arithmetic. Applying now one step of Gram Schmidt to Aqk , we
obtain
X k Xk
q kC1 D Aqk 
e hAqk ; qi iqi D Aqk  hik qi ; (7.18)
i D1 i D1
where we used the abbreviation hik WD hAqk ; qi i. Finally normalizing the vector
e
q kC1 by
1
qkC1 D e
q kC1 ; where hkC1;k D ke q kC1 k (7.19)
hkC1;k
we obtain an orthonormal basis q1 ; : : : ; qkC1 of KkC1 .A; x/. For the practical im-
plementation, it is advisable to prefer the Gram Schmidt procedure with reorthogo-
nalization. The corresponding algorithm is given in Figure 7.7 and is called Arnoldi
iteration [1].
Let us denote by Qm 2 F nm the isometric matrix with columns q1 ; : : : ; qm . Then
the eigenpairs of the matrix Hm WD Qm  AQ are the Ritz pairs of A with respect to
m
Km .A; x/. It turns out that this matrix Hm has a special structure.

Theorem 7.16 (Krylov subspaces and Hessenberg matrices). Let A 2 F nn , m 2 N

and let x 2 F n be such dim Km .A; x/ D m. If Qm D Œq1 ; : : : ; qm  2 F nm is an
isometric matrix such that (7.17) is satisfied, then Hm D Œhij  WD Qm  AQ is an
m
m  m Hessenberg matrix, i.e., hij D 0 for all i > j C 1, j D 1; : : : ; m  1.
Proof. Let j 2 ¹1; : : : ; m  1º. Then by (7.17) we have qj 2 Kj .A; x/ and thus
Aqj 2 Kj C1 .A; x/ D span.q1 ; : : : ; qj C1 /:

Since the vectors q1 ; : : : ; qm are orthonormal, we obtain that hij D hAqj ; qi i D 0 for
i D j C 2; : : : ; m. Thus, Hm is in Hessenberg form.

Remark 7.17 (Possible breakdown of the Arnoldi iteration). Note that the Arnoldi
iteration will break down if hkC1;k D 0 for some k < m. By (7.18), we have
hkC1;k ¤ 0 whenever Aqk 2 KkC1 .A; x/ n Kk .A; x/. If dim Km .A; x/ D m then
this is guaranteed for k < m. Under this hypothesis, the Arnoldi iteration will run
without breakdown for at least m  1 steps. For the mth step, however, there are two
possibilities:
Section 7.4 Arnoldi iteration 161

procedure arnoldi_iteration(A, x, var H , q1 ; : : : ; qm );

begin

kxk;
1
q1  x;
for k 2 ¹1; : : : ; mº do begin
qkC1 Aqk ;
for i 2 ¹1; : : : ; kº do begin
hi;k hqkC1 ; qi i;
qkC1 qkC1  hik qi
end;
for i 2 ¹1; : : : ; kº do begin
si hqkC1 ; qi i;
qkC1 qkC1  si qi ;
hi;k hi;k C si
end;
hkC1;k kqkC1 k;
if hkC1;k D 0 then
STOP
else
1
qkC1 q
hkC1;k kC1
end
end
Figure 7.7. Arnoldi iteration (using modified Gram–Schmidt with reorthogonalization).

1. If hmC1;m D 0, then the algorithm breaks down and we have

X
kC1
Aqk D hik qi ; for k D 1; : : : ; m  1;
i D1
X m
Aqm D him qi ;
i D1

and thus Aqk 2 Km .A; x/ D span.q1 ; : : : ; qk / for k D 1; : : : ; m. This implies

that Km .A; x/ is an A-invariant subspace and hence, all eigenvalues of Hm D
Qm AQ , Q D Œq ; : : : ; q  are also (exact) eigenvalues of A, see Exercise 2.44.
m 1 m
Thus, a breakdown of the Arnoldi iteration is actually a preferable situation,
because it gives us an orthonormal basis of an invariant subspace.
162 Chapter 7 Krylov subspace methods for large sparse eigenvalue problems

2. If hmC1;m ¤ 0, then we can continue to perform the mth step of the Arnoldi
iteration to produce the next iterate qmC1 . Then we have
X
kC1
Aqk D hik qi ; for k D 1; : : : ; m;
i D1

or, equivalently,
0 1
h11 ::: ::: h1n
B :: :: C
Bh21 h22 : : C
B C
B :: :: :: C
A Πq1 ; : : : ; qm  D Πq1 ; : : : ; qm ; qmC1  B 0 : : : C:
B C
B :: :: C
@ : : hm;m1 hmm A
0 ::: 0 hmC1;m
(7.20)
Note that the upper m  m block of the .m C 1/  m matrix in (7.20) is just Hm
and that the .m C 1/st row is hmC1;m ım T , where ı denotes the mth canonical
m
unit vector of length m. Thus, we can write (7.20) in the shorter form
T
AQm D Qm Hm C hmC1;m qmC1 ım (7.21)
which is sometimes called an Arnoldi configuration.
Exercise 7.18 (Arnoldi configuration). Let A 2 F nn , let q1 ; : : : ; qm ; qmC1 2
F n be linearely independent (but not necessarily orthonormal) such that (7.20)
holds with hkC1;k ¤ 0 for k D 1; : : : ; m C 1. Show that

span.q1 ; : : : ; qk / D Kk .A; q1 /

for k D 1; : : : ; m C 1.

Exercise 7.19 (Krylov subspaces and Hessenberg matrices). Let A 2 F nn and
let Q D Œq1 ; : : : ; qn  2 F nn be an invertible matrix such that H D Q1 AQ is
in Hessenberg form. Show that

span.q1 ; : : : ; qk / D Kk .A; q1 /

for k D 1; : : : ; n if and only if hkC1;k ¤ 0 for k D 1; : : : ; n  1.

Remark 7.20 (Computation of Ritz pairs). In both of the scenarios described in

Remark 7.17, we are left with the task of computing the eigenvalues of the matrix
Hm D Qm  AQ . Since H is already in Hessenberg form, those eigenvalues can be
m m
efficiently computed by the QR iteration discussed in Chapter 5. If  is a computed
eigenvalue, then we can compute a corresponding eigenvector v 2 F m by using in-
verse iteration with shift  for an arbitrary starting vector, see Section 4.4. In practice,
usually one or at most two iterations are sufficient.
Section 7.4 Arnoldi iteration 163

Once we have successfully computed Ritz pairs, the question arises whether a given
Ritz pair .; v/ 2 F  Km .A; x/ is a good approximation to an eigenpair. We answer
this question by slightly extending the notion of residual from Definition 4.7.

Definition 7.21 (Residual). Let A 2 F nn and let .; v/ 2 F  F n n ¹0º. Then the
vector
r WD v  Av
is called the residual of the pair .; v/ with respect to A.

Theorem 7.22 (Backward error from the residual). Let A 2 F nn and .; v/ 2 F 
F n be such that kvk D 1. Then there exists a matrix E 2 F nn such that

.A C E/v D v and kEk  kv  Avk:

Proof. Set E WD .v  Av/v  . Then we have

v  v D Av C v  Av D v
.A C E/v D Av C .v  Av/ „ƒ‚…
D1

and kEk D k.v  Av/v  k  kv  Avkkv  k D kv  Avk.

Thus, whenever the residual r D v  Av of a Ritz pair .; v/ is small, then we

have a small backward error in the sense that .; v/ is an exact eigenpair of a nearby
matrix. A nice property of the Arnoldi iteration is the fact that the residual can be
computed very cheaply.

Theorem 7.23 (Residual of Ritz pairs). Let A 2 F nn be a matrix and Qm 2 F nm ,
Hm 2 F mm , qmC1 2 F n , and hmC1;m 2 F be as in (7.21). If .; y/ 2 F .F m n¹0º/
is an eigenpair of Hm , then .; v/ with v D Qm y is a Ritz pair of A with respect to
Km .A; x/ satisfying
kv  Avk D jhmC1;m j  jym j;
where ym denotes the last entry of the vector y.

Proof. Let .; y/ 2 F .F m n¹0º/ be an eigenpair of Hm . By Theorem 7.3, we know

that .; Qm y/ is a Ritz pair of A with respect to Km .A; x/ D R.Qm /. Moreover,
we have

v  Av D Qm y  AQm y
(7.21) T
D Qm y  .Qm Hm C hmC1;m qmC1 ım /y
D Qm .y  Hm y/ hmC1;m qmC1 ym ;
„ ƒ‚ …
D0

where we used that ym D ım

T y. But then we obtain kv  Avk D jh
mC1;m j  jym j.
164 Chapter 7 Krylov subspace methods for large sparse eigenvalue problems

Thus, the residual of the Ritz pair .; v/ can be immediately computed from the
two scalars hmC1;m and ym without explicitly forming the Ritz vector v and without
computing the matrix-vector product Av.

7.5 Symmetric Lanczos algorithm

In the special situation that our matrix A D A 2 F nn is self-adjoint, the Arnoldi
iteration can be simplified significantly. After m steps of the Arnoldi iteration, we
have computed a matrix Qm D Œq1 ; : : : ; qm  with columns that form an orthonormal
basis of the Krylov subspace Km .A; x/ for some initial vector x 2 F n n ¹0º. With A,
also the matrix Hm D Qm  AQ is self-adjoint and has the following form:
m
0 1 0 1
h11 h12 : : : ::: h1n ˛ 1 ˇ1 0 ::: 0
B :: :: C B :: :: C
Bh21 h22 h23 : : C B : : C
B C Bˇ1 ˛2 ˇ2 C
B : : : : C B : : C
Hm D B 0 :: :: :: :: C D B 0 ˇ :: :: 0 C ;
B C B 2 C
B :: :: :: :: C B : :: :: :: C
@ : : : : hm1;m A @ :: : : : ˇm1 A
0 : : : 0 hm;m1 hmm 0 : : : 0 ˇm1 ˛m

where ˛k WD hkk 2 R, because self-adjoint matrices have real diagonal elements, and
ˇk1 WD hk;k1 D ke q k k 2 R by (7.19) which implies hk1;k D hk;k1 D ˇk1 .
Thus, even in the case that we started with a complex self-adjoint matrix A, the matrix
H m D Qm  AQ will be tridiagonal and real. With this notation and starting with an
m
arbitrary vector x 2 F n n ¹0º, the equations (7.18) and (7.19) simplify to

q kC1 DAqk  ˛k qk  ˇk1 qk1

e (7.22)
ˇk Dke
q kC1 k
1
qkC1 D e q kC1 ; k D 1; : : : ; m  1;
ˇk
1
where q0 WD 0, ˇ0 WD 0, q1 WD kxk x. This special case of the Arnoldi iteration
is called the symmetric Lanczos iteration [27]. The resulting algorithm is given in
Figure 7.8
As in the Arnoldi iteration, the Lanczos iteration may break down in some step (say
the mth step) yielding an invariant subspace given by the Krylov subspace Km .A; x/
which is spanned by the vectors q1 ; : : : ; qm . Otherwise, we may compute the eigen-
values of Hm D Qm AQm to obtain the Ritz values of A with respect to Km .A; x/.
An important feature of the symmetric Lanczos iteration is the so-called three-
term-recurrence (7.22). In each iteration, only the vectors qk and qk1 from the two
previous iterations are needed. However, the full set of vectors q1 ; : : : ; qm is needed
for the computation of Ritz vectors and for reorthogonalization in order to avoid a loss
in orthogonality of the vectors q1 ; : : : ; qm .
Section 7.6 Chebyshev polynomials 165

procedure lanczos_iteration(A, x, var q1 ; : : : ; qm , ˛1 ; : : : ; ˛m , ˇ1 ; : : : ; ˇm1 );

begin
q0 0;
ˇ0 0;

kxk;
1
q1  x;
for k 2 ¹1; : : : ; m  1º do begin
qkC1 Aqk ;
˛k hqkC1 ; qk i;
qkC1 qkC1  ˛k qk  ˇk1 qk1 ;
ˇk kqkC1 k;
if ˇk D 0 then
STOP
else
1
qkC1 q
ˇk kC1
end
end
Figure 7.8. Lanczos iteration without reorthogonalization.

If only eigenvalues are needed, then one may use the Lanczos iteration without
reorthogonalization and refrain from storing the vectors q1 ; : : : ; qm except for
those from the two previous iterations. As a consequence, the number of steps
m can be taken much higher as with reorthogonalization, but on the other hand
a loss in orthogonality of the vectors q1 ; : : : ; qm will occur. This will typically
lead to the occurrence of ghost eigenvalues. These are multiple “copies” of some
eigenvalues of Hm , typically those that are well approximating the smallest and
largest eigenvalues of A. Thus, the Ritz values give the wrong impression that
the multiplicity of the corresponding eigenvalue of A is higher than it actually
is. We refer the reader to the monograph [45] for an illustrative example and a
heuristic explanation of this observation.

7.6 Chebyshev polynomials

In the previous sections we have discussed how to compute orthonormal bases of
Krylov subspaces, how to compute the corresponding Ritz pairs, and how to check
whether these Ritz pairs are good approximations to eigenpairs. However, a question
that is still open is the following. Given a matrix A 2 F nn and a start vector x 2 F n ,
166 Chapter 7 Krylov subspace methods for large sparse eigenvalue problems

does Km .A; x/ contain any good approximations to eigenpairs of A? And if so,

which eigenpairs are most likely to be detected first? We will see that this question
can best be answered by the help of a new tool, the so-called Chebyshev polynomials
that we will introduce in this section.

Example 7.24. Let A D diag.3; 2:99; 2:98; : : : ; 2:02; 2:01; 2; 1/ 2 R102102 , i.e.,
the diagonal matrix with eigenvalues k D 3  .k  1/=100 for k D 1; : : : ; 101 and
102 D 1. Applying the Arnoldi iteration (or Lanczos iteration) with the start vector
x 2 R102 with all entries equal to one, we obtain after 10 steps the 1010 matrix H10
having the eigenvalues i , i D 1; : : : ; 10 displayed in the following table rounded to
five significant digits:

i i minj ji  j j
1 2:9840 6:2933  1004
2 2:9262 3:7955  1003
3 2:8192 7:5917  1004
4 2:6786 1:4213  1003
5 2:5180 1:9794  1003
6 2:3539 3:8811  1003
7 2:2038 3:7657  1003
8 2:0851 4:8982  1003
9 2:0130 2:9747  1003
10 1:0000 3:8769  1012
Apparently, the eigenvalue 102 D 1 of A in Example 7.24 is well approximated
by a Ritz value of A with respect to K10 .A; x/. At first, this may be a surprise,
because K10 .A; x/ is spanned by x; Ax; : : : ; A9 x and so by the results of Chapter 4
one may expect that we find a good approximation to eigenvectors associated with the
dominant eigenvalue 1 D 3 in K10 .A; x/. However, the gap between 1 and 2 is
rather small, and so the convergence of the power method for A is rather slow which
explains why we did not yet find a good approximation to an eigenvector associated
with 1 D 3 after only ten iterations. On the other hand, we see that the smallest
eigenvalue 102 is approximated with an absolute error as small as 3:8769  1012 .
How can this be explained?
For the remainder of this section, we will assume that A is self-adjoint in order to
keep the discussion simple. Thus, we may assume that A 2 F nn has the n eigen-
values 1
  
n in descending order and a corresponding orthonormal basis of
eigenvectors e1 ; : : : ; en associated with 1 ; : : : ; n . The start vector x 2 F n n ¹0º
can then be represented as a linear combination of this basis vectors, i.e., there exists
Section 7.6 Chebyshev polynomials 167

coefficients ˛1 ; : : : ; ˛n 2 F such that

X
n
xD ˛i ei :
i D1

Recall that Km .A; x/ D ¹p.A/xjp 2 Fm1 Œt º by Lemma 7.7. For an arbitrary

polynomial p the vector p.A/x then has the form given in the following remark.

Remark 7.25 (Elements of Krylov subspaces). Let A 2 F nn be self-adjoint and let
Pn of F ofn eigenvectors associated with the eigenvalues
n
e1 ; : : : ; en be a basis
1 ; : : : ; n . If x D i D1 ˛i ei 2 F , ˛1 ; : : : ; ˛n 2 F, and if p 2 FŒt , then
X
n
p.A/x D ˛i p.i /ei :
i D1

If we can now construct a polynomial p in such a way that

p.j / D 1; max jp.i /j  ";

i ¤j

then we obtain using hei ; ej i D ıij that

 2  2
 X n  X 
   
k˛j ej  p.A/xk22 D ˛j ej  ˛i p.i /ei  D  ˛ p. /e
i i

   i
i D1 2
 i ¤j 
2
* +
X X X
D ˛i p.i /ei ; ˛i p.i /ei D j˛i j2 jp.i /j2
i ¤j i ¤j i ¤j
X
 "2 j˛i j2 :
i ¤j

Assuming without loss of generality that kxk2 D 1 (we can always scale the start
vector without changing the corresponding Krylov subspace), we obtain that
X
j˛i j2 D 1  j˛j j2 D 1  jhx; ej ij D 1  cos2 †.x; ej / D sin2 †.x; ej /:
i¤j

Thus, we have obtained the following result.

Lemma 7.26 (Convergence of Krylov subspace methods). Let A 2 F nn be self-

adjoint with eigenvalues 1
  
n and let e1 ; : : : ; en be a corresponding or-
thonormal basis of eigenvectors, i.e., Aei D i ei for i D 1; : : : ; n. Furthermore let
m 2 N, " > 0 and
Xn X n
n
xD ˛i ei 2 F ; j˛i j2 D 1:
i D1 i D1
168 Chapter 7 Krylov subspace methods for large sparse eigenvalue problems

If there exists a polynomial p 2 Fm Œt  with p.j / D 1 and max jp.i /j  ", then
i ¤j

k˛j ej  p.A/xk2  " sin †.x; ej /  ":

Lemma 7.26 states that if " is small and ˛j ¤ 0, then the vector p.A/x is a good
approximation to the vector ˛j ej and hence to an eigenvector associated with the
eigenvalue j . Revisiting Example 7.24, we see that the minimal eigenvalue 102 is
well separated from the remaining eigenvalues in the interval Œ2; 3. If we choose a
polynomial
Y10
pDc .t  i / 2 F10 Œt ; c 2 R
i D1

having all zeros in the interval Œ2; 3, and if we choose c such that p.102 / D 1, we
can expect due to the continuity of polynomials that

max jp.i /j  ";

i ¤102

where " is small. Thus, K10 .A; x/ can be expected to contain good approximations
to eigenvectors associated with 102 . In order to obtain quantitative statements, it is
beneficial to construct p with the help of Chebyshev polynomials.

Definition 7.27 (Chebyshev polynomials). The polynomials Tn 2 FŒt , n 2 N that

are recursively defined by the formula

T0 WD 1;
T1 WD t;
TkC2 WD 2t TkC1  Tk for all k 2 N0 ;

are called the Chebyshev polynomials of the first kind.

It is easy to verify that Tn is a polynomial of degree n. In the following, we will

only refer to them as Chebyshev polynomials thus dropping the extension “of the first
kind”.

Exercise 7.28 (Chebyshev polynomials). Prove by induction that Tn is a polyno-

mial of degree n with leading coefficient 2n1 for n
1. Also show that

Tn .x/ D .1/n Tn .x/

for all x 2 F and all n 2 N.

Section 7.6 Chebyshev polynomials 169

Example 7.29 (Low degree Chebyshev polynomials). The first six Chebyshev poly-
nomials have the form
T0 D 1; T3 D 4t 3  3t;
T1 D t; T4 D 8t 4  8t 2 C 1;
T2 D 2t 2  1; T5 D 16t 5  20t 3 C 5t:

The Chebyshev polynomials have other representations that are quite useful for
theoretical considerations.

Theorem 7.30 (Alternative representations of the Chebyshev polynomials).

The Chebyshev polynomials Tn , n 2 N0 from Definition 7.27 satisfy the following
conditions.
1. Tn .t / D cos.n arccos t / for all t 2 Œ1; 1.
z n C z n z C z 1
2. Tn .t / D , where t D , z 2 C n ¹0º.
2 2
Proof. 1. We show that the functions fn W Œ1; 1 ! R; t 7! cos.n arccos t /
satisfy the recursive formula of Definition 7.27. This implies Tn .t / D fn .t / for all
t 2 Œ1; 1. Clearly, for all t 2 Œ1; 1 we have

f0 .t / D cos.0  arccos t / D cos.0/ D 1; and

f1 .t / D cos.arccos t / D t:

Using the addition theorem cos.x C y/ D cos x cos y  sin x sin y we then obtain for
k 2 N0 that

fkC2 .t / D cos..k C 2/ arccos t / D cos..k C 1/ arccos t C arccos t /

D cos..k C 1/ arccos t / cos.arccos t /  sin..k C 1/ arccos t / sin.arccos t /

and

fk .t / D cos.k arccos t / D cos..k C 1/ arccos t C . arccos t //

D cos..k C 1/ arccos t / cos.arccos t / C sin..k C 1/ arccos t / sin.arccos t /;

where we also used cos.x/ D cos.x/ and sin.x/ D  sin.x/. Adding both equa-
tions yields

fkC2 .t / C fk .t / D 2 cos..k C 1/ arccos t / cos.arccos t / D 2tfkC1 .t /

which implies that the functions fn satisfy the recursive formula of Definition 7.27.
2. For a fixed t 2 C, the equation

z C z 1
tD (7.23)
2
170 Chapter 7 Krylov subspace methods for large sparse eigenvalue problems

p
has two solutions z1=2 D t ˙ t 2  1 (for some branch of the complex square root)
that satisfy z2 D z11 , because
p p
z1  z2 D .t C t 2  1/.t  t 2  1/ D t 2  .t 2  1/ D 1:

Let us now consider the functions

z n C z n z C z 1
gn W C ! C; t 7! ; where t D ;
2 2
for n 2 N. Then the functions gn , n 2 N are well defined, because the expression
z n Cz n
2 is invariant under replacing z with z 1 , so it does not matter which solution
of (7.23) is used for the computation of gk .t /. Now we have
z 0 C z 0 z 1 C z 1
g0 .t / D D 1; g1 .t / D D t; and
2 2

z C z 1 z n C z n z n1 C z .n1/
2 t  gn .t /  gn1 .t / D 2   
2 2 2
z nC1 C z nC1 C z n1 C z n1  z n1  z .n1/
D
2
z nC1 C z .nC1/
D D gnC1 .t /;
2
i.e., the functions gn , n 2 N satisfy the recursive formula of Definition 7.27 which
implies Tn .t / D gn .t / for all t 2 C.

Corollary 7.31 (Zeros and extrema of Chebyshev polynomials). Let n 2 N. Then

the following conditions are satisfied:
1. If n
1, then Tn has n zeros, all being in Œ1; 1 and given by
 
2k  1
cos  ; k D 1; : : : ; n:
2n

2. max jTn .t /j D 1.
t2Œ1;1

3. Tn W Œ1; 1 ! Œ1; 1 has n C 1 local extrema given by

 
.n/ k .n/
sk D cos  ; Tn .sk / D .1/k ; k D 0; : : : ; n:
n

Of those, sk.n/ , k D 1; : : : ; n  1 are also local extrema for Tn W R ! R, while

.n/ .n/
s0 D 1 and sn D 1 are not.
Proof. The proof follows immediately from Theorem 7.30.
Section 7.6 Chebyshev polynomials 171

Exercise 7.32 (Properties of Chebyshev polynomials). Show that the Chebyshev

polynomials satisfy the following conditions.
1. Tn .Tm .t // D Tnm .t / for all t 2 F and all n; m 2 N.
2. The Chebyshev polynomials
p are orthogonal on the interval .1; 1/ with re-
spect to the weight 1= 1  t 2 , i.e.,

Z1
1
Tn .t /Tm .t / p dt D 0
1  t2
1

for all n; m 2 N with m ¤ n.

3. For all n
1, the Chebyshev polynomial Tn W R ! R satisfies the differen-
tial equation

.1  t 2 /Tn00 .t /  t Tn0 .t / C n2 Tn .t / D 0:

What makes Chebyshev polynomials so useful is the fact that they satisfy an opti-
mality condition. Recall that a polynomial is called monic if the leading coefficient
is equal to one. Note that the Chebyshev polynomials are not monic for n > 1, but
we can easily make them monic by normalizing them as T en D 21n Tn , because by
Exercise 7.28 the leading coefficient of Tn is 2n1 .

Theorem 7.33 (Optimality of Chebyshev polynomials). Let n

1,
> 1, c 2 R.
en WD 21n Tn 2 RŒt 
1. Among all monic polynomials p 2 Rn Œt  the polynomial T
has the smallest maximum norm on the interval Œ1; 1 which is given by 21n ,
i.e.,
21n D max jT en .t /j D min max jp.t /j : (7.24)
t2Œ1;1 p2Rn Œt t2Œ1;1
p monic

2. Among all p 2 Rn Œt  satisfying p.

/ D c the polynomial Tbn WD c Tn 2
Tn ./
RŒt  has the smallest maximum norm on the interval Œ1; 1 which is given by
jcj=Tn .
/, i.e.,

jcj bn .t /j D
D max jT min max jp.t /j : (7.25)
Tn .
/ t2Œ1;1 p2Rn Œt t2Œ1;1
p./Dc

Proof. 1. By Corollary 7.31, we immediately obtain that

en .t /j D 21n :
max jT
t2Œ1;1
172 Chapter 7 Krylov subspace methods for large sparse eigenvalue problems

For the second equality in (7.24), the inequality “

” is trivial, so it remains to show
the inequality “”. To this end, assume that p 2 Fn Œt  is a monic polynomial such
that
jp.t /j < 21n for all t 2 Œ1; 1:
By Corollary 7.31 the polynomial T en has n C 1 local extrema in Œ1; 1 given by
 
en .s .n/ / D 21n Tn .s .n/ / D 21n .1/k ; s .n/ D cos k  ; k D 0; : : : ; n:
T k k k n

Thus, we obtain that

²
.n/ en .s /.n/ <0 if k is even,
p.sk /  T k >0 if k is odd:

en .t / changes sign n times in the interval

for k D 0; : : : ; n, i.e., the function p.t /  T
Œ1; 1 and thus has n roots in Œ1; 1 by the intermediate value theorem. On the other
hand, as p and T en are both monic, the difference p  T en is a polynomial of degree
at most n  1 and hence it must be the zero polynomial because of having at least n
pairwise distinct roots. This implies p D T en in contradiction to the assumption.
2. The proof of 2. is quite similar to the proof of 1. and is left as an exercise.

Exercise 7.34. Prove part 2. of Theorem 7.33.

7.7 Convergence of Krylov subspace methods

Let A 2 F nn be a self-adjoint matrix with eigenvalues 1
  
n . In the follow-
ing, we will try to find approximations to eigenvectors associated with the extremal
(largest and smallest) eigenvalues of A. We will restrict our attention to the largest
eigenvalue 1 , because we can always consider A instead of A so that the smallest
eigenvalue n of A becomes the largest eigenvalue of A. By Lemma 7.26, we need
to construct a polynomial of degree at most m that has small values in modulus in
the interval Œn ; 2  in order to obtain a good approximation to an eigenvector asso-
ciated with 1 . Since the Chebyshev polynomials satisfy an optimality condition on
the interval Œ1; 1 in the sense of Theorem 7.33, it is reasonable to transform the
polynomials in such a way that this optimality is transferred to the interval Œn ; 2 .
Assuming 2 > n , this can be done by help of the transformation
2t  .2 C n /
t 7!
2  n
which maps 2 to 1 and n to 1 while 1 is mapped to
21  .2 C n / 2.1  2 / C .2  n /  1  2

WD D D2 C 1: (7.26)
2  n 2  n 2  n
Section 7.7 Convergence of Krylov subspace methods 173

Note that
> 1 if 1 > 2 . Using this transformation, we obtain the following
optimality condition.

Lemma 7.35 (Optimality of transformed Chebyshev polynomials). Let 1 > 2 >

n . Among all p 2 Rm Œt  satisfying p.1 / D 1, the polynomial pm 2 Rm Œt  with
 
1 2t  .2 C n / 1  2
pm .t / WD Tm ;
WD 2 C1
Tm .
/ 2  n 2  n

has the smallest maximum norm on the interval Œn ; 2 . In particular, setting
p
 1  n 1
 WD and % WD p ;
 1  2 C1

we have
1 2%m
min max jp.t /j D max jpm .t /j D D : (7.27)
p2Rn Œt t2Œn ;2  t2Œn ;2  Tm .
/ 1 C %2m
p.1 /D1

Proof. By applying part 2 of Theorem 7.24, we immediately obtain the first and the
second identity in (7.27). For the third identity, we have to calculate the value of
1=Tm .
/. The trick is to use the second alternative representation
p of Tn from Theo-
rem 7.30. Let us pick the smaller solution z D

 1 of the two solutions of
2

the equation
z C z 1

D :
2
(Note that both solutions are real, because of
> 1.) Then we have
 2
2 1  2 .1  2 /2 1  2

1D 2 C1 1D4 C4
2  n .2  n / 2 2  n
.1  2 /2 C .1  2 /.2  n / .1  2 /.1  n /
D4 D4 :
.2  n /2 .2  n /2

This implies
q p p
2.1  2 / C .2  n /  2 1  2 1  n


2 1D
 2  n
p p
.1  2 /  2 1  2 1  n C .1  n /
D
 2  n
p p
. 1  2  1  n /2
D p
. 2  n /2
174 Chapter 7 Krylov subspace methods for large sparse eigenvalue problems

0s s s 12
         2A
D@
1 2 1 n 1
 
2  n 2  n  1  2
0 s 12
 1  2 @  1  n A
D 1 :
2  n 1  2

Using
1  2 1  2 1
D D ;
2  n .1  n /  .1  2 / 1
we finally obtain
q p p p
.1  /2 .   1/2 1


2 1D D p p Dp D %:
1 .  C 1/.   1/ C1

Then part 2. of Theorem 7.30 implies

1 2 2%m
D m D
Tm .
/ % C %m %2m C 1

which finishes the proof of the third identity in (7.27).

Theorem 7.36 (Convergence rate for Krylov subspace methods I). Let A 2 F nn be
self-adjoint with eigenvalues 1 > 2
  
n , where 2 > n and let e1 ; : : : ; en
be a corresponding orthonormal basis of eigenvectors such that Aei D i ei for i D
1; : : : ; n. Furthermore let m 2 N and

X
n X
n
xD ˛i ei 2 F n ; j˛i j D 1:
i D1 i D1

Then with the notation of Lemma 7.35 there exists a polynomial pm 2 Fm Œt  with
2%m
k˛1 e1  pm .A/xk2  sin †.x; e1 /:
1 C %2m

Proof. The proof follows immediately by combining Lemma 7.26 and

Lemma 7.35.

Note that Theorem 7.36 simply contains an existence statement: we know about
the existence of the vector pm .A/x in the Krylov subspace Km .A; x/, but we do not
know how to construct this vector. However, concerning eigenvalues, we can now
give a relation between the largest eigenvalue 1 and the corresponding largest Ritz
value with respect to Km .A; x/.
Section 7.7 Convergence of Krylov subspace methods 175

Theorem 7.37 (Convergence rate for Krylov subspace methods II). Let A 2 F nn be
self-adjoint with eigenvalues 1 > 2
  
n , where 2 > n and let e1 ; : : : ; en
be a corresponding orthonormal basis of eigenvectors such that Aei D i ei for i D
1; : : : ; n. Furthermore let m 2 N and
X
n X
n
n
xD ˛i ei 2 F ; j˛i j D 1;
i D1 i D1

where ˛1 ¤ 0. If 1 is the largest Ritz value of A with respect to Km .A; x/, then with
the notation of Lemma 7.35 we have
 2
2%m1
1
1
1  .1  n / tan2 †.x; e1 /:
1 C %2.m1/
Proof. Without loss of generality, we may assume that dim Km .A; x/ D m. Other-
wise, the result is trivial, because Km .A; x/ is an invariant subspace which must con-
tain e1 because of ˛1 ¤ 0, see Exercise 7.38. Thus, let the columns of Qm 2 F mn
form an orthonormal basis of Km .A; x/. Then the largest Ritz value 1 of A with
 AQ and thus, by
respect to Km .A; x/ is the largest eigenvalue of the matrix Qm m
Exercise 3.9 we have
 y  Qm
 AQ y
m
1 D max hQm AQm y; yi D max :
y2F y2F n¹0º yy
kykD1

Recalling that
R.Qm / D Km .A; x/ D ¹p.A/x j p 2 Fm1 Œt º
and letting pm1 denote the transformed Chebyshev polynomial from Theorem 7.36
which satisfies pm1 .1 / D 1, we obtain
x  p.A/ Ap.A/x x  pm1 .A/ Apm1 .A/x
1 D max

p2Fm1 Œt x  p.A/ p.A/x x  pm1 .A/ pm1 .A/x

p.A/x¤0
Pn Pn
i pm1 .i /2 j˛i j2 i D1 .1 C i  1 /pm1 .i / j˛i j
2 2
D PiD1n D P n
i D1 pm1 .i / j˛i j i D1 pm1 .i / j˛i j
2 2 2 2
Pn
.1  i /pm1 .i / j˛i j
2 2
D 1  i D1 Pn
i D1 pm1 .i / j˛i j
2 2
Pn
i D2 pm1 .i / j˛i j
2 2

1  .1  n / P
j˛1 j2 C niD2 pm1 .i /2 j˛i j2
P
max¹pm1 .t /2 j t 2 Œn ; 2 º niD2 j˛i j2

1  .1  n /
j˛1 j2
  2
2%m1 1  j˛1 j2

1  .1  n / ;
1 C %2.m1/ j˛1 j2
176 Chapter 7 Krylov subspace methods for large sparse eigenvalue problems

Pn
i D2 j˛i j D 1  j˛1 j2 . The result then follows from
where we have used 2

sin2 †.x; e1 / 1  cos2 †.x; e1 /

tan2 †.x; e1 / D D
cos2 †.x; e1 / cos2 †.x; e1 /

and cos †.x; e1 / D jhe1 ; xij D j˛1 j.

Exercise 7.38 (Invariant Krylov subspace). Let A 2 F nn be self-adjoint with

eigenvalues 1 > 2
  
n and let e1 ; : : : ; en be a corresponding basis of
eigenvectors. Furthermore let m 2 N and

X
n
xD ˛i ei 2 F n n ¹0º;
i D1

where ˛1 ¤ 0. Show that if the Krylov subspace Km .A; x/ is invariant with

respect to A then e1 2 Km .A; x/.

Example 7.39 (Convergence of the largest Ritz value I). We illustrate the result of
Theorem 7.37 for the diagonal 100  100 matrix A with diagonal entries akk D
1 C k=100 for k D 1; : : : ; 99 and a100;100 D 2:2, i.e.,

A D diag.1:01; 1:02; 1:03; : : : ; 1:98; 1:99; 2:20/ 2 R100100 :

Thus, there is a sufficiently large gap between the largest eigenvalue 1 D 2:2 and the
remaining eigenvalues 2 ; : : : ; 100 2 Œ1; 1:99. Choosing the start vector
0 1
1
1 B:C 100
xDp @ :: A 2 R ;
100
1

.m/ .m/
we obtain the following error 1  1 for the largest Ritz value 1 with respect
to Km .A; x/ as plotted in Figure 7.9.
As one can see, the convergence rate corresponds quite well to the convergence
rate predicted by the bound from Theorem 7.37. In fact the error is a little bit smaller
than predicted by the bound, but this effect is due to the simplifications we performed
in the proof of Theorem 7.37. Numerical experiments show that the bound becomes
much tighter if the gap between 1 and 2 is very large.

Example 7.40 (Convergence of the largest Ritz value II). For a second test, let B be
the diagonal matrix with diagonal entries bkk D 1 C k=100 for k D 1; : : : ; 100, i.e.,

B D diag.1:01; 1:02; 1:03; : : : ; 1:98; 1:99; 2:00/ 2 R100100 :

Section 7.7 Convergence of Krylov subspace methods 177

0
10

−2
10

−4
10

−6
10

−8
10

−10
10

−12
10
0 5 10 15 20 25 30 35 40 45 50

Figure 7.9. The error 1 .m/

1 of the largest Ritz value .m/
1 of the matrix A in Example 7.39
with respect to Km .A; x/ for m D 1; : : : ; 50. The error is plotted as a solid line, the dotted
line is the bound from Theorem 7.37.

In contrast to Example 7.39, the gap between the largest and second largest eigenval-
ues 1 and 2 is now much smaller. Choosing the same start vector x as in Exam-
.m/ .m/
ple 7.39, we obtain the following error 1  1 for the largest Ritz value 1 with
respect to Km .A; x/ as plotted in Figure 7.10.
In this case, the bound is quite an overestimate of the actual error. Moreover, the
convergence seems to be superlinear at it seems that from step 25 onwards, the actual
convergence rate becomes better than the linear convergence rate predicted by the
bound from Theorem 7.37.
How can the observation in Example 7.40 be explained? Why does the convergence
rate seem to accelerate for increasing values of m? The reason is that for computing
the bound in Theorem 7.37, we used the transformed Chebyshev polynomial pm1
which has the minimal maximum norm on the interval Œn ; 2  under the constraint
pm1 .1 / D 1. However, for our particular purpose, we do not need a polynomial that
has minimal maximum norm on a whole interval, but it is sufficient if the polynomial
is small on the eigenvalues n ; : : : ; 2 . For example, we could use the polynomial
t  2
b
p m2 .t / D pm2 .t /;
1  2
where pm2 is the transformed Chebyshev polynomial of degree m  2 from Theo-
rem 7.36 with pm2 .1 / D 1 and minimal maximum norm on the interval Œn ; 3 .
178 Chapter 7 Krylov subspace methods for large sparse eigenvalue problems

0
10

−2
10

−4
10

−6
10

−8
10

−10
10

−12
10
0 5 10 15 20 25 30 35 40 45 50

.m/ .m/
Figure 7.10. The error 1  1 of the largest Ritz value 1 of the matrix B in Exam-
ple 7.40 with respect to Km .A; x/ for m D 1; : : : ; 50. The error is plotted as a solid line, the
dotted line is the bound from Theorem 7.37.

Then bp m2 is small on all eigenvalues n ; : : : ; 2 , because 2 has been chosen to be

a zero of b
p m2 . We can use this polynomial to produce a different bound.

Theorem 7.41 (Convergence rate for Krylov subspace methods III). Let A 2 F nn
be self-adjoint with eigenvalues 1
2 > 3
  
n , where 3 > n and
let e1 ; : : : ; en be a corresponding orthonormal basis of eigenvectors such that Aei D
i ei for i D 1; : : : ; n. Furthermore let m 2 N and
X
n X
n
xD ˛i ei 2 F n ; j˛i j D 1;
i D1 i D1

where ˛1 ¤ 0. If p
 1  n b1
b
 WD and b
% WD p
 1  3 b
C1
then
 2  2 Pn
n  2 %m2
2b i D3 j˛i j
2
1
1
1  .1  n / :
 1  2 1 Cb
%2.m2/ j˛1 j2
where 1 is the largest Ritz value of A with respect to Km .A; x/.
Section 7.7 Convergence of Krylov subspace methods 179

Exercise 7.42 (Convergence rate for Krylov subspace methods).

Prove Theorem 7.41.
Hint: mimic the proof of Theorem 7.37 using

t  2
p m2 .t / D
b pm2 .t /
1  2

instead of pm1 .

Note that if 2 > 3 , then b % < %. Thus, asymptotically the bound from Theo-
rem 7.41 will be smaller than the one from Theorem 7.37. However, this may only
be relevant for sufficiently large m as the new bound has a different constant now and
the exponent has reduced from m  1 to m  2.

Example 7.43 (Convergence of the largest Ritz value III). Let C 2 R100100 be the
diagonal matrix with diagonal entries

ckk D 1 C k=100; k D 1; : : : ; 98;

and c99;99 D 2:19, c100;100 D 2:2, i.e.,

C D diag.1:01; 1:02; 1:03; : : : ; 1:97; 1:98; 2:19; 2:20/ 2 R100100 :

Thus, now have a nice gap between 2 and 3 , whereas the gap between 1 and 2
is rather small. Choosing the start vector as in Example 7.39, we obtain the following
.m/ .m/
error 1  1 for the largest Ritz value 1 with respect to Km .A; x/ as plotted in
Figure 7.11.
This time, the convergence rate corresponds quite well to the convergence rate pre-
dicted by the bound from Theorem 7.41 while the bound from Theorem 7.36 is a huge
overestimate due to the fact that the gap between 1 and 2 is rather small.
The observation from Example 7.43 can be generalized to explain also the effect
observed in Example 7.40. If m is sufficiently large, then Km .A; x/ contains poly-
nomials bp mk that have zeros in 2 ; : : : ; k , that satisfy b
p mk .1 / D 1, and that
have small norm in the interval Œn ; kC1 . For such polynomials, the relevant gap
in eigenvalues is the gap between kC1 and 1 rather than 2 and 1 . Thus, the
convergence rate accelerates as m increases.

Exercise 7.44 (Convergence of the largest Ritz value). Under the hypothesis
k > kC1 > n construct polynomials b p mk as outlined above and obtain
a result analogous to the one of Theorem 7.41 by improving the bound b
% to
p
e
1 1  n
% WD p
e with e WD :
eC1 1  k
180 Chapter 7 Krylov subspace methods for large sparse eigenvalue problems

0
10

−2
10

−4
10

−6
10

−8
10

−10
10

−12
10
0 5 10 15 20 25 30 35 40 45 50

.m/ .m/
Figure 7.11. The error 1  1 of the largest Ritz value 1 of the matrix C in Exam-
ple 7.43 with respect to Km .A; x/ for m D 1; : : : ; 50. The error is plotted as a solid line,
the dotted line is the bound from Theorem 7.37, and the dashed line is the bound from Theo-
rem 7.41.

Exercise 7.45 (Convergence of the second largest Ritz value). Let A 2 F nn be
self-adjoint with eigenvalues 1
  
n , where 1 > 2 > 3 > n . Derive
a bound similar to the one in Theorem 7.37 that gives an estimate for the error
2 2 , where 2 is the second largest Ritz value of A with respect to Km .A; x/.
Hint: Construct a polynomial ep m2 with e p m2 .1 / D 0 and e
p m2 .2 / D 1
that has small norm on the interval Œn ; 3 .

By applying the results of this section to A, we obtain corresponding bounds

for the smallest Ritz values of the self-adjoint matrix A. It can be observed that
among the Ritz values computed by Krylov subspace methods one typically finds
first good approximations to the largest or smallest eigenvalues of A. Heuristi-
cally, this can be explained by the observation that for an eigenvalue b  in the
interior of the spectrum that is not well separated from the other eigenvalues, it
is quite hard to find a polynomial b p .b
p of low degree such that b / D 1 which takes
values small in modulus at all remaining eigenvalues. This observation also gen-
eralizes to the case when A is not self-adjoint. Again, typically the eigenvalues
near the boundary of the spectrum are located first.
Section 7.7 Convergence of Krylov subspace methods 181

Another important point in Krylov subspace method are restarts. At first, the
start vector x in the Arnoldi or Lanczos iteration may be chosen randomly, but
after a few iterations, we may actually find a vector b x in the computed Krylov
subspace Km .A; x/ that would have been a better start vector. We could then
restart our iteration with the new start vector bx . In the Arnoldi iteration, this
may be a huge advantage, because the summation in (7.18) becomes more and
more expensive by each step, because in each step an additional summand occurs
in the formula. A commonly used and rather effective methods is the so-called
implicitly restarted Arnoldi iteration. For further reading we refer to [2, 49].

There are other algorithms for computing eigenvalues and eigenvectors of large
sparse matrices. Two among those are the Jacobi Davidson method and Precon-
ditioned Inverse iteration (PINVIT).
The Jacobi Davidson method introduced in [40] is a projection method, but
uses other subspaces instead of Krylov subspaces. The basic idea is to enlarge the
current subspace R.Qm / in such a way that a particular eigenpair .; v/ of A that
is approximated by a Ritz pair .; Qm y/ will be approximated much better by a
Ritz pair with respect to the enlarged subspace R.QmC1 /. The resulting method
is particularly suited for the computation of eigenvalues in the interior of the
spectrum and leads to quadratic convergence of the selected Ritz pairs. However,
typically only one Ritz pair converges at a time and once it has converged one
focuses on the next Ritz pair. This is different to the case of Krylov subspaces
where convergence of several Ritz pairs takes place simultaneously. For details,
we refer the reader to [40] and [41].
The principal idea of PINVIT can be described as follows. If we consider
inverse iteration and scale the next iteration vector by the Rayleigh quotient, the
resulting iteration equation can be reformulated as

xi C1 D ƒA .xi /A1 xi D xi  A1 Axi C ƒA .xi /A1 xi

D xi  A1 .Axi  ƒA .xi /xi /:

Then A1 is replaced by an approximate inverse B 1 of A that can be easily

calculated, a so-called preconditioner. For details, we refer the reader to [29, 30,
23].
Chapter 8

Generalized and polynomial eigenvalue

problems 

Summary
Generalized eigenvalue problems are the natural extension of the standard eigenvalue
problem. Their discussion will lead to the theory of matrix pencils. Concerning the
solution of generalized eigenvalue problems, a generalization of the efficient QR iter-
ation introduced in Chapter 5, the QZ algorithm, turns out to be an efficient method.
It is based on a generalization of the Schur decomposition introduced in Chapter 2.
An even further generalization of the standard eigenvalue problem are polynomial
eigenvalue problems that can be reduced to generalized eigenvalue problems by the
concept of linearization.

Learning targets
 Introduce matrix polynomials and matrix pencils.
 Introduce the eigenvalue 1.
 Generalize the Schur decomposition to matrix pencils.
 Generalize the QR algorithm to the corresponding QZ algorithm for matrix
pencils.

8.1 Polynomial eigenvalue problems and linearization

In the introduction, we have seen that systems of linear differential equations of higher
order of the form

X̀
Ak y .k/ D A` y .`/ C A`1 y .`1/ C    C A2 y 00 C A1 y 0 C A0 y D 0; (8.1)
kD0

where A0 ; : : : ; A` 2 F nn lead to a polynomial eigenvalue problem of the form

0 1
X̀
@ k Ak A y0 D 0:
kD0
Section 8.1 Polynomial eigenvalue problems and linearization 183

Definition 8.1 (Matrix polynomials). Let ` 2 N n ¹0º. The polynomial

X̀
P .t / D t k Ak (8.2)
kD0

with matrix coefficients A0 ; : : : ; A` 2 F nn , A` ¤ 0 is called an n  n matrix

polynomial over F of degree `.

Definition 8.2 (Polynomial and quadratic eigenvalue problems). Let P be an n  n

matrix polynomial over F of degree `. Then the problem of finding scalars  2 F and
nonzero vectors x 2 F n such that

P ./x D 0 (8.3)

is called a polynomial eigenvalue problem. In this case,  is called an eigenvalue

of P and x is called an eigenvector of P for the eigenvalue . If ` D 2, then the
problem (8.3) is called a quadratic eigenvalue problem.

How can we solve the polynomial eigenvalue problem (8.3)? For a fixed  2 F , the
equation P ./x D 0 has a nontrivial solution if the matrix P ./ 2 F nn is singular,
i.e., if det P ./ D 0.

Remark 8.3 (Determinant of matrix polynomials). Let P be an n  n matrix polyno-

mial over F . Then  2 F is an eigenvalue of P if and only if det P ./ D 0.
A common method to tackle the polynomial eigenvalue problem (8.3) is the concept
of linearization, i.e., of transforming the problem into an equivalent linear eigenvalue
problem. A natural way to achieve this is based on the concept of order reduction
in the theory of differential equations. Let us return to the system of differential
equations (8.1). Introducing new variables y1 D y, y2 D y 0 , . . . , y` D y .`1/ , we
obtain the new system of linear differential equations

X
`1
y10 D y2 ; : : : ; y`1
0
D y` ; A` y`0 C Ak ykC1 D 0;
kD0

or, equivalently,
0 1 0 10 0 10 1
In 0 : : : 0 y1 0 In 0 y1
B : : : C B
: : : : :: C By2 CC B :: B
C By2 C
B0 CB C
:: ::
B CB C C B : : :
C :C
B :: : : C B :: C B
@0 0 In A B C D 0:
@: : In 0 A @ : A ::: @ :: A
0 : : : 0 A` y` A0 A1 : : : A`1 y`
184 Chapter 8 Generalized and polynomial eigenvalue problems

Similarly, introducing the new variables x1 D x, x2 D x, . . . , x` D `1 x, we

obtain from the polynomial eigenvalue problem
0 1
X̀
@ k Ak A x D 0 (8.4)
kD0

the new eigenvalue problem

0 10 1 0 10 1
In 0 : : : 0 x1 0 In 0 x1
B : : :: C B C B : C B C
B0 : : C Bx C CB x2 C
: : :: : :: : ::
B
: C B 2C D B CB C
B :: : : C B :: C B
@ A B :: C : (8.5)
@: : In 0 A @ : A 0 : : : 0 I n @ :A
0 : : : 0 A` x` A0 A1 : : : A`1 x`
Note that the matrix on the right hand side of (8.5) has a block form analogous to
the companion matrix of Exercise 2.16. This motivates the following definition.

Definition 8.4 (Companion polynomial). Let P be a matrix polynomial as in (8.2).

Then the n`  n` matrix polynomial
0 1 0 1
In 0 : : : 0 0 In 0
B C C
B 0 : : : : : : ::: C B :: :: ::
C
C.t / D t B CCB : : :
C
B :: : : C B
@ 0 In A
(8.6)
@: : In 0 A 0 :::
0 : : : 0 A` A0 A1 : : : A`1

is called the companion polynomial of P .

Exercise 8.5 (Companion polynomial). Show that z 2 F n` n¹0º is an eigenvector

of C in (8.6) associated with the eigenvalue  if and only if  is an eigenvalue
of P as in (8.2) with associated eigenvector x that equals the vector consisting of
the last n components of z.

Note that the eigenvalue problem (8.5) is linear in the sense that is depends linearly
on the eigenvalue parameter . If A` D In , then this eigenvalue problem reduces to
a standard eigenvalue problem with an n`  n` matrix, but if A` ¤ In , then it is a
special case of a so-called generalized eigenvalue problem.

Definition 8.6 (Matrix pencil). Let E; A 2 F nn . Then the matrix polynomial
L.t / D tE  A is called a matrix pencil. The corresponding polynomial eigenvalue
problem L./x D 0 is called a generalized eigenvalue problem.

The reason for introducing the minus sign in the term tE  A is the fact that the
corresponding generalized eigenvalue problem can then be written in the convenient
form Ex D Ax.
Section 8.2 Matrix pencils 185

8.2 Matrix pencils

In this section, we will investigate matrix pencils L.t / D tE A, where E; A 2 F nn
and the corresponding generalized eigenvalue problem Ex D Ax. From Remark 8.3
we know that  2 F is an eigenvalue of L if and only if det L./ D 0. It is also
well known that any matrix A 2 F nn has exactly n eigenvalues in C counted with
algebraic multiplicities. This may be different for matrix pencils which may also have
infinitely many or no eigenvalues in C.

Example 8.7 (Number of eigenvalues). Let L1 .t / D tE1 A1 and L2 .t / D tE2 A2 ,
where
       
0 1 1 0 0 1 1 0
E1 D ; A1 D ; E2 D ; A2 D :
0 0 0 0 0 0 0 1
Then for all  2 C we have
   
1  1 
det.E1  A1 / D det D 0 and det.E2  A2 / D det D 1;
0 0 0 1
thus E2  A2 is nonsingular for all  2 C and consequently L2 ./ does not have
any eigenvalues according to Definition 8.2. On the other hand, E1  A1 is singular
for all  2 C and for each  2 C, the vector
 

v D
1
is an eigenvector of L1 associated with the eigenvalue .
The pencil L1 .t / in Example 8.7 is a pathologic case and we will therefore restrict
ourselves to pencils for which the determinant is not identically zero.
Definition 8.8 (Regular pencils). An nn matrix pencil L.t / over F is called regular
if det L.t / is not the zero polynomial. Otherwise, the pencil is called singular.

Exercise 8.9 (Singular pencils). Consider the singular matrix pencil

   
0 1 0 1
L.t / D tE  A D t  :
0 0 0 0

ee
Show that for any " > 0 and any 1 ; 2 2 R, there exists a regular pencil t E A
having the eigenvalues 1 and 2 such that
e  Ek2 ; ke
max.kE A  Ak2 /  ":

Thus, singular matrix pencils are extremely ill-conditioned in the sense that arbi-
trarily small perturbations make them regular pencils with arbitrarily prescribed
eigenvalues.
186 Chapter 8 Generalized and polynomial eigenvalue problems

A key idea for solving the standard eigenvalue problem Ax D x, where A 2
F nn , was the application of similarity transformations to A. There is a similar con-
cept for matrix pencils. Let E; A 2 F nn and let x 2 F n n¹0º be an eigenvector of the
pencil tE  A associated with the eigenvalue  2 F, i.e., we have Ex D Ax. Multi-
plying this equation from the left by a nonsingular matrix P 2 F nn and introducing
x WD Q1 x, where Q 2 F nn is nonsingular, we find
b

x D PAQb
PEQb x:

b b
Thus, any scalar  is an eigenvalue of the pencil t E A WD tPEQ  PAQ if and only
bb
if it is an eigenvalue of the pencil tE  A, and any associated eigenvector of t E A
can be transformed to an eigenvector of tE  A associated with  by multiplication
with Q.

Definition 8.10 (Equivalence of pencils). Two nn matrix pencils L1 .t / D tE1 A1
and L2 .t / D tE2 A2 over F are called equivalent if there exist nonsingular matrices
P; Q 2 F nn such that

E2 D PE1 Q and A2 D PA1 Q:

Remark 8.11 (Equivalence and similarity). Let A 2 F nn . Then the standard eigen-
value problem Ax D x can also be interpreted as a generalized eigenvalue problem
of the form In x D Ax. Thus, any matrix A can be canonically identified with the
matrix pencil tIn  A. Two matrix pencils L1 .t / D tIn  A1 and L2 .t / D tIn  A2
with A1 ; A2 2 F nn are equivalent if and only if there exist nonsingular matri-
ces P; Q such that
PQ D In and A2 D PA1 Q:
The first identity immediately implies P D Q1 and thus the pencils L1 .t / and L2 .t /
are equivalent if and only if A1 and A2 are similar. Thus, Definition 8.10 is the natural
extension of Definition 2.17 to matrix pencils.

Exercise 8.12 (Invariance of regularity under equivalence). Let the n  n pencils

L1 .t / and L2 .t / over F be equivalent. Show that L1 is regular if and only if L2
is regular.

Exercise 8.13 (Pencils with commuting coefficient matrices). Let E; A 2 F nn

such that the pencil L.t / D tE  A is regular. Show that L.t / is equivalent to a
pencil with commuting coefficients as follows:
Section 8.2 Matrix pencils 187

1. Let B; C 2 F nn be such that B C C D In . Show that B and C commute,

i.e., BC D CB.
2. Let  2 F be such that E  A is nonsingular. Use the results of 1. to show
that the matrices

e WD .E  A/1 E
E and e
A WD .E  A/1 A

commute.

Remark 8.14 (Reduction to standard eigenvalue problems). Let E; A 2 F nn . If

E is nonsingular, then the pencil tE  A is equivalent to both the pencils tIn  E 1 A
and tIn  AE 1 , i.e., the generalized eigenvalue problem Ex D Ax is equivalent
to the standard eigenvalue problem with E 1 A or AE 1 .
We have seen in Example 8.7 that a regular n  n pencil tE  A may have less
than n eigenvalues over C if the matrix E is singular. This comes from the fact
that the determinant det.tE  A/ is then a polynomial in t of degree less than n.
This “defect” can be repaired by introducing the concept of infinite eigenvalues. We
begin with the following observation. Let x 2 F n n ¹0º be an eigenvector of the
pencil tA  E associated with the eigenvalue  ¤ 0, i.e., Ax D Ex. Then x is
also an eigenvector of the pencil tE  A now associated with the eigenvalue 1 ,
because
1 Ex D 1 .Ax/ D Ax:
Thus, the nonzero eigenvalues of the pencil tE  A are exactly the reciprocals of
the nonzero eigenvalues of tA  E. Extending this reciprocity to include the case
 D 0 with the convention that zero and 1 are reciprocals motivates the following
definition:

Definition 8.15 (Infinite eigenvalue). Let E; A 2 F nn . If x 2 F n n ¹0º is such that

Ex D 0, then we say that x is an eigenvector of the pencil tE  A associated with the
eigenvalue 1.
188 Chapter 8 Generalized and polynomial eigenvalue problems

A more precise definition of the eigenvalue 1 can be given by introducing ho-

mogeneous parameters. If x 2 F n n ¹0º is an eigenvector of tE  A associated
with the eigenvalue  2 F and if .˛; ˇ/ 2 F 2 , ˇ ¤ 0 is a pair satisfying  D ˇ˛ ,
then
Ex D Ax ” ˛Ex D ˇAx:
Now let us introduce the equivalence relation  on F 2 n ¹.0; 0/º given by

.˛1 ; ˇ1 /  .˛2 ; ˇ2 / W” ˛1 ˇ2 D ˛2 ˇ1 :

Then for ˛; ˇ 2 F with ˇ ¤ 0 satisfying ˛Ex D ˇAx for some nonzero vector
x 2 F n ¹0º, the equivalence class

˛ ; ě/ j . e
Œ.˛; ˇ/ WD ¹. e ˛ ; ě/  .˛; ˇ/º
˛
can be identified with the eigenvalue  D ˇ
of tE  A. Note that there is an
additional equivalence class

1 WD ¹.˛; 0/ j ˛ 2 F n ¹0ºº:

If x 2 F n n ¹0º is such that Ex D 0, then

˛Ex D 0

for all ˛ ¤ 0. Thus, we interpret the equivalence class 1 as the so-called infinite
eigenvalue of tE  A. Associated eigenvectors are all nonzero vectors from the
null space of E.

Definition 8.16 (Algebraic and geometric multiplicity). Let E; A 2 F nn such that
L.t / D tE  A is regular and let  2 F [ ¹1º be an eigenvalue of L.t /.
1. If  ¤ 1 then the multiplicity a .E; A; / of  as a zero of det.tE  A/ is
called the algebraic multiplicity of .
2. If  D 1 and if k 2 ¹0; 1; 2; : : : ; nº is the degree of the polynomial det.tE A/,
then a .E; A; 1/ D n  k is called the algebraic multiplicity of 1.
3. If  ¤ 1 then g .E; A; / WD dim N .E  A/ is called the geometric multi-
plicity of .
4. If  D 1 then g .E; A; 1/ WD dim N .E/ is called the geometric multiplicity
of 1.
Section 8.3 Deflating subspaces and the generalized Schur decomposition 189

Remark 8.17 (Number of eigenvalues). By Definition 8.16, any regular n  n matrix

pencil over F has exactly n eigenvalues over C counted with algebraic multiplicities.

Exercise 8.18 (Algebraic and geometric multiplicity). Let E; A 2 F nn such

that L.t / D tE  A is regular.
1. Prove that g .E; A; 1/  a .E; A; 1/. Hint: use that there exists k 2 N
and nonsingular matrices P; Q 2 F nn such that
 
Ik 0
PEQ D :
0 0

2. Find an example to show that g .E; A; 1/ < a .E; A; 1/ is possible.

8.3 Deflating subspaces and the generalized Schur

decomposition
In the theory of matrices, the concept of invariant subspaces is an important gen-
eralization of the concept of eigenvectors, see Section 2.5. In this section, we will
generalize this concept to the concept of deflating subspaces that play the analogous
role in the theory of matrix pencils. Recall that for A 2 F nn a subspace X  F n is
called invariant with respect to A if x 2 X implies Ax 2 X. In order to generalize
this definition to the case of matrix pencils an alternative characterization of invariant
subspaces is more suitable. If X  F n is a subspace of dimension k, we have

X is invariant w.r.t. A ” dim¹x C Ay j x; y 2 Xº D k:

Indeed, the subspace on the right hand side of the equivalence is contained in X if X
is invariant with respect to A, and hence its dimension does not exceed k. On the other
hand, the space also contains X, and therefore it must be of dimension exactly k. We
can now use this equivalence to generalize the concept of invariant subspaces to the
case of matrix pencils.

Definition 8.19 (Deflating subspace). Let tE  A be an n  n matrix pencil over F.

A subspace X  F n of dimension k is called a deflating subspace of tE  A if

dim¹Ex C Ay j x; y 2 Xº  k:

The inequality in Definition 8.19 is due to the fact that the pencil is allowed to be
singular which may lead to a drop in the dimension.
190 Chapter 8 Generalized and polynomial eigenvalue problems

Exercise 8.20 (Deflating subspaces of regular pencils). Let tE  A be a regular

n  n matrix pencil over F and let X  F n be a deflating subspace of tE  A.
Show that
dim¹Ex C Ay j x; y 2 Xº D k:

Theorem 8.21 (Alternative characterizations of deflating subspaces). Let tE  A be

an n  n matrix pencil over F and X  F n be a subspace of dimension k. Then the
following statements are equivalent:
1. X is a deflating subspace of tE  A.
2. There exists a subspace Y  F n with dim Y  k and AX; EX  Y.
3. There exist unitary matrices Z D Œz1 ; : : : ; zn ; Q 2 F nn with span.z1 ; : : : ; zk /
D X such that
   
  E11 E12 A11 A12
tQ EZ  Q AZ D t  ;
0 E22 0 A22

where E11 ; A11 2 F kk and E22 ; A22 2 F .nk/.nk/ .

Proof. “1: ) 2:”: Set Y WD ¹Ex C Ay j x; y 2 Xº. Then dim Y  k as X is a

deflating subspace of tE  A. Clearly, we have AX; EX  Y.
“2: ) 3:”: Let z1 ; : : : ; zk be an orthonormal basis of X. This basis can be com-
pleted to an orthonormal basis z1 ; : : : ; zn of F n . Similarly, let qkC1 ; : : : ; qn 2 Y ?
be a family of orthonormal vectors from the orthogonal complement of Y and let
q1 ; : : : ; qn be a completion to an orthonormal basis of F n . Set
   
Z D Z1 Z2 D Œz1 ; : : : ; zn ; Q D Q1 Q2 D Œq1 ; : : : ; qn ;

where Z1 D Œz1 ; : : : ; zk  and Q1 D Œq1 ; : : : ; qk . Then Q2 EZ1 D 0 D Q2 AZ1 by

construction which implies
     
  Q1 EZ1 Q1 EZ2 Q1 AZ1 Q1 AZ2
tQ EZ  Q AZ D t 
Q2 EZ1 Q2 EZ2 Q2 AZ1 Q2 AZ2
   
E11 E12 A11 A12
Dt  ;
0 E22 0 A22

where Eij D Qi EZj and Aij D Qi AZj for i; j D 1; 2.

“3: ) 1:”: Let Z1 D Œz1 ; : : : ; zk . Then R.Z1 / D X and by 3. we have
   
 E11  A11
Q EZ1 D ; Q AZ1 D :
0 0

From this we obtain Q ¹Ex C Ay j x; y 2 Xº  span.ı1 ; : : : ; ık /, where ı1 ; : : : ; ık

are the first k canonical unit vectors. This implies dim¹Ex C Ay j x; y 2 Xº  k as
Q is invertible and thus, X is a deflating subspace.
Section 8.3 Deflating subspaces and the generalized Schur decomposition 191

Part 3 of Theorem 8.21 is the reason for the name deflating subspace that was
introduced in [42], because the knowledge of a deflating subspace allows to deflate
the eigenvalue problem .E A/x D 0 into two smaller subproblems with the pencils
tE11  A11 and tE22  A22 .

Remark 8.22 (Left deflating subspaces). Sometimes, Y as in Theorem 8.21 is called

a left deflating subspace associated with the right deflating subspace X, so X; Y to-
gether are called a pair of right and left deflating subspaces.
A key result in Section 2 was the Schur decomposition of a matrix, because it is a
form that displays the eigenvalues of the matrix and can be achieved by applying uni-
tary similarity transformations only. There is a corresponding decomposition of equal
importance in the case of matrix pencils, called the generalized Schur decomposition.

Theorem 8.23 (Generalized Schur decomposition). Let tE  A be an n  n matrix

pencil over C. Then there exist unitary matrices Q; Z 2 C nn such that
0 1 0 1
r11 r12 : : : r1n s11 s12 : : : s1n
B : C B : C
B 0 r22 : : : :: C B 0 s22 : : : :: C

Q EZ D R WD B :B C  B C
:: :: :: C ; Q AZ D S WD B :: :: :: :: C :
@ :: : : : A @ : : : : A
0 : : : 0 rnn 0 : : : 0 snn

If there exists k 2 ¹1; : : : ; nº such that rkk D 0 D skk then tE  A is singular.

Otherwise, tE  A has the n eigenvalues
skk
k D ; k D 1; : : : ; n;
rkk

where we interpret k D 1 if rkk D 0.

Proof. The proof proceeds by induction on n. The case for n D 1 is trivial. Thus, let
us assume n > 1 and that the assertion of the theorem holds for n  1. Let  2 C and
x 2 C n n ¹0º be such that
Ex D Ax:
Such a choice is always possible. Indeed, if tE  A is regular, then let  be an
eigenvalue and x be an associated eigenvector. If, on the other hand, tE  A is
singular, then det.tE  A/ 0 and for an arbitrary  2 C choose a nonzero vector x
from the kernel of E  A. Then span.x/ is a deflating subspace of tE  A. Thus, by
Theorem 8.21, there exist unitary matrices Q; Z 2 Cnn , where the first column z1
of Z spans span.x/ such that
   
  r11 E12 s11 A12
tQ EZ  Q AZ D t  ;
0 E22 0 A22
192 Chapter 8 Generalized and polynomial eigenvalue problems

where r11 ; s11 2 C and E22 ; A22 2 C .n1/.n1/ . By the induction hypothesis, there
exist unitary matrices Q2 ; Z2 2 C .n1/.n1/ such that Q2 E22 Z2 and Q2 A22 Z2
are both upper triangular. Setting then
   
1 0 1 0
Q D Q1 ; Z D Z1 ;
0 Q2 0 Z2

we obtain the desired decomposition. The remainder of the proof then follows from

Y
n
det.tE  A/ D det.Q/ det.tR  S / det.Z  / D det.QZ  / .t rkk  skk /:
kD1

Exercise 8.24. Show that an n  n matrix pencil tE  A over C has a diago-

nal generalized Schur decomposition (i.e., R and S in Theorem 8.23 are both
diagonal) if and only if there exists an orthonormal basis of C n consisting of
eigenvectors of tE  A.

Exercise 8.25. Let E; A 2 C nn be Hermitian. Show that if the matrix pencil
tE  A has a diagonal generalized Schur decomposition, then E and A commute,
i.e., EA D AE.

8.4 Hessenberg-triangular form

In the following, we aim to numerically compute the generalized Schur decomposi-
tion. How can this be achieved? Suppose that L.t / WD tE  A is an n  n matrix
pencil over C and suppose that Q; Z 2 C nn are unitary such that

tQ EZ  Q AZ D tR  S (8.7)

is the generalized Schur decomposition of L. Suppose for the moment, that E is

invertible. Then we obtain from (8.7) that

Q .AE 1 /Q D Q AZZ  E 1 Q D SR1 :

Since the inverse of an upper triangular matrix and the product of two upper tri-
angular matrices are again upper triangular matrices, the latter identity means that
Q  .AE 1 /Q D SR1 is a Schur decomposition of the matrix AE 1 . Thus, a pos-
sible idea is to first compute a unitary matrix Q that transforms AE 1 into Schur
form and then to compute a unitary Z so that (8.7) holds. The standard method for
computing Q is, of course, the QR iteration from Chapter 5. However, E may be ill-
conditioned (or even singular), so we do not want (or are not able) to form the inverse
E 1 and the product AE 1 . Instead, we will try to develop a method that implicitly
performs the QR iteration for AE 1 , but works directly on the matrices A and E.
Section 8.4 Hessenberg-triangular form 193

Before we do so, recall that the QR iteration requires only O.n2 / operations for Hes-
senberg matrices as opposed to O.n3 / operations for arbitrary matrices. Therefore, a
related reduction for matrix pencils would be useful. Observe that if the pencil tE  A
is such that A is a Hessenberg matrix and E is upper triangular and invertible, then
also AE 1 is a Hessenberg matrix.

Definition 8.26 (Hessenberg-triangular form). Let E; A 2 F nn , where E is upper

triangular and A is a Hessenberg matrix. Then we say that the pencil tE  A is in
Hessenberg-triangular form.

In the following, we will show constructively that for arbitrary E; A 2 F nn there
exist unitary matrices Q; Z 2 F nn such that tQ EZ  Q AZ is in Hessenberg-
triangular form. We will depict the general procedure examplarily for n D 5. Thus,
we start with matrices of the form
0 1 0 1
 
B    C B    C
B C B C
EDB B      C ; A D B    C :
C B C
@    A @    A
 
In the first step, we compute the QR factorization of E (e.g., with the help of House-
holder reflections, see Section 4.7), i.e., we compute a unitary Q1 2 F nn such that
Q1 E is upper triangular. Then we have
0 1 0 1
 
B    C B    C
B C B C
.1/ 
E WD Q1 E D B B   C ; A WD Q1 A D B
C .1/ 
     C;
B C
@  A @     A
 
where blanks stand for zero entries.
In the following, we will transform A.1/ step by step to Hessenberg form and simul-
taneously keep the triangular form of E .1/ . Alternatingly we will apply elimination
steps and restoration steps. Thus, if k is odd, then after the kth step we eliminate an
element of A.k/ with the help of a Givens rotation (introduced in Section 5.2), and
in the next step, we restore the triangular form of E .kC1/ , again with the help of a
Givens rotation. We start by applying a Givens rotation G .1/ that transforms the forth
and fifth row in order to eliminate the .5; 1/-element of A.1/ :
0 1 0 1
       
B    C B    C
B C B C
.2/
E WD G E D B .1/ .1/ B    C ; A WD G A D B
C .2/ .1/ .1/
     C:
B C
@ ˝˝ A @ ˝ ˝ ˝ ˝ ˝A
˝ 0 ˝˝˝˝
194 Chapter 8 Generalized and polynomial eigenvalue problems

Note that this transformation destroys the upper triangular form of E .1/ as it intro-
duces a fill-in in the .5; 4/-position of the matrix. But we can restore the triangular
form by multiplying E .2/ with a Givens rotation G .2/ which manipulates the forth
and fifth columns of E .2/ . Application of this Givens rotation to A.2/ will keep the
zero entry in the .5; 1/-position, because the first column remains unchanged.
0 1 0 1
˝˝ ˝˝
B   ˝ ˝C B   ˝ ˝C
B C B C
.3/ .2/ .2/
E WD E G D B B  ˝ ˝C ; A WD A G D B
C .3/ .2/ .2/
   ˝ ˝C:
B C
@ ˝˝ A @    ˝ ˝A
0 ˝ ˝˝

Next, we perform an elimination step to eliminate the .4; 1/-element of A.3/ by ap-
plying a Givens rotation G .3/ to the rows three and four (this time producing a fill-in
in the .4; 3/-position of E .3/ ), followed by a restauration step to restore the triangular
form of E .3/ by applying a Givens rotation G .4/ to the columns three and four:
0 1 0 1
        
B    C B    C
B C B C
E WD G E D B
.4/ .3/ .3/
B ˝ ˝ ˝C .4/ .3/ .3/
C ; A WD G A D B˝ ˝ ˝ ˝ ˝C ;
B C
@  ˝ ˝A @ 0 ˝ ˝ ˝ ˝A
    
0 1 0 1
 ˝ ˝    ˝ ˝ 
B  ˝ ˝ C B  ˝ ˝ C
B C B C
E .5/ WD E .4/ G .4/ DB
B ˝ ˝ CC; A.5/ WD A.4/ G .4/ DB
B  ˝ ˝ C
C:
@ 0 ˝ A @  ˝ ˝ A
  ˝ ˝ 
We continue this procedure until we have annihilated all but the first two elements
of the first column of A. In the case n D 5, we need one more pair of elimination-
restauration-steps:
0 1 0 1
         
B ˝ ˝ ˝ ˝C B˝ ˝ ˝ ˝ ˝C
B C B C
E WD G E D B  ˝ ˝ ˝C ; A WD G A D B
.6/ .5/ .5/ B C .6/ .5/ .5/
B 0 ˝ ˝ ˝ ˝C ;
C
@   A @    A
    
0 1 0 1
˝˝  ˝ ˝  
B ˝ ˝  C B ˝ ˝  C
B C B C
E .7/ WD E .6/ G .6/ DB
B 0 ˝   C;
C A.7/ WD A.6/ G .6/ DB
B ˝ ˝  C
C:
@  A @ ˝ ˝  A
 ˝ ˝  
Section 8.4 Hessenberg-triangular form 195

Observe that at this stage we cannot proceed with eliminating the element in the
.2; 1/-position of A.7/ , because this would produce a fill-in in the .2; 1/-position
of E .7/ , and the following restauration step would work on the first and second
columns of our matrices, thereby destroying all the strenuously obtained zeros in A.7/ .
Instead, we now continue with the second column of A.7/ , again starting from the
bottom. Note that all our transformation will preserve the zero structure of the first
column of A.7/ :
0 1 0 1
       
B    C B     C
B C B C
.8/ B   C ; A D B
C .8/ C
E DB B    C ;
@ ˝˝ A @ ˝ ˝ ˝ ˝A
˝ 0 ˝˝˝
0 1 0 1
˝˝   ˝ ˝
B   ˝ ˝C B  ˝C
 ˝
B C B C
E DB
.9/
B  ˝ ˝CC; A .9/
DB
B  ˝C
C; ˝
@ ˝ ˝A @  ˝A
 ˝
0 ˝ ˝ ˝
0 1 0 1
        
B    C B     C
B C B C
E .10/
DB
B ˝ ˝ ˝ C;
C A.10/
DBB ˝ ˝ ˝ ˝ C;
C
@  ˝ ˝A @ 0 ˝ ˝ ˝A
   
0 1 0 1
˝˝ ˝˝
B  ˝ ˝ C B  ˝ ˝ C
B C B C
E .11/ D B
B ˝ ˝ CC; A.11/ D B
B  ˝ ˝ C :
C
@ 0 ˝ A @ ˝ ˝ A
 ˝˝
For the second column, we have to stop the procedure after eliminating all but the
first three element, because proceeding by eliminating the .3; 2/-entry of A.11/ would
lead to a fill-in in the .3; 1/-position of A.11 /, thereby destroying the zero structure of
the first column. We now continue this series of steps until A has reached Hessenberg
form. In the case n D 5 only one more pair of steps is necessary:
0 1 0 1
      
B    C B     C
B C B C
.12/ B C
  C ; A .12/
DB C
E DB B    C ;
@ ˝ ˝A @ ˝ ˝ ˝A
˝ 0 ˝˝
196 Chapter 8 Generalized and polynomial eigenvalue problems

0 1 0 1
˝˝ ˝˝
B   ˝ ˝C B   ˝ ˝C
B C B C
E .13/ DB
B  ˝ ˝CC; A.13/ DB C
B   ˝ ˝C :
@ ˝ ˝A @  ˝ ˝A
0 ˝ ˝˝

Exercise 8.27. Write a procedure hessenberg_triangular(var E; A) that com-

putes the Hessenberg-triangular form for an n  n matrix pencil tE  A
over F.

Exercise 8.28. Calculate how many operations are necessary to compute the
Hessenberg triangular form for an n  n matrix pencil tE  A over F. Also
calculate the number of operations needed to explicitly compute the transforma-
tion matrices Q and Z.

8.5 Deflation
Deflation was an important strategy in the practical use of the QR iteration as noted
in Section 5.4, and the same is true for generalized eigenvalue problems. Assume
that our n  n matrix pencil tE  A over F is regular and in Hessenberg-triangular
form. If one or more of the subdiagonal elements of A are zero (or, sufficiently small
in magnitude), then the corresponding generalized eigenvalue problem can be divided
into two smaller generalized eigenvalue problems of smaller size. Let k 2 ¹1; : : : ; nº
be an index such that akC1;k ¤ 0. Then the pencil tE  A has the form

 
tE11  A11 tE12  A12
tE  A D ;
0 tE22  A22

where tE11  A11 and tE22  A22 are two pencils of sizes k  k and .n  k/  .n  k/,
respectively, that are again in Hessenberg-triangular form. Thus, we can compute the
eigenvalues of tE  A by computing the eigenvalues of the pencils tE11  A11 and
tE22  A22 . In practice, we will deflate the problem as soon as there exists an index
k for akC1;k  0.
Next assume that all subdiagonal entries of A are nonzero and assume that the
pencil tE  A has the eigenvalue 1. Then E is singular and necessarily must have
a zero entry on the diagonal. Thus, the eigenvalue 1 obviously plays an exceptional
role, because it is automatically displayed after reduction to Hessenberg-triangular
form. Due to this exceptional role, it may not be a complete surprise that we are also
able to deflate it. The corresponding procedure is called zero-chasing, and we will
explain it exemplarily on a 5  5 matrix pencil. Thus, let us assume that e33 D 0,
Section 8.5 Deflation 197

where
D 3 is the smallest index for which ekk ¤ 0 for all k D
C 1; : : : ; n. Thus,
our pencil has the following pattern.
0 1 0 1
 
B    C B    C
B C B C
B  C ; A D B
C C
EDB B    C :
@  A @   A
 
Our aim is to chase the ‘zero’ on the diagonal of E down to the lower right corner
of E. In the first step, we apply a Givens rotation G1 to E and A acting on the third
and forth rows that will eliminate the .4; 4/-entry of E.
0 1 0 1
       
B    C B     C
B C B C
E .1/ D G1 E D B B ˝ ˝ C ; A.1/ D G1 A D B ˝ ˝ ˝ ˝C :
C B C
@ 0 ˝A @  ˝ ˝ ˝A
  
This procedure will generate a fill-in in the .4; 2/-position of A thus creating a bulge in
the Hessenberg form. However, applying a Givens rotation G2 acting on the columns
two and three, we can restore the Hessenberg form of A.1/ without destroying the
upper triangular form of0E .1/ . 1 0 1
˝˝ ˝˝
B ˝ ˝  C B ˝ ˝  C
B C B C
.2/ .1/ B  C ; A D A G2 D B
C .2/ .1/ C
E D E G2 D B B ˝ ˝  C :
@  A @ 0 ˝  A
 
Note that we chased the zero on the diagonal of E further down while the Hessenberg
form of A has been restored. (There is an additional zero entry remaining in the
.3; 3/-position of E, but this one will fill up again later, so it is out of our focus.) We
continue now to chase the zero down to the .n; n/-position.
0 1 0 1
     
B    C B     C
B C B C
.3/ .2/ B   C ; A D G3 A D B
C .3/ .2/ C
E D G3 E D B B    C ;
@ ˝ A @ ˝ ˝ ˝A
0 ˝˝
0 1 0 1
˝˝ ˝˝
B  ˝ ˝ C B  ˝ ˝ C
B C B C
E .4/ D E G4 D B
.3/
B  ˝ C;
C A.4/ D A G4 D B
.3/
B  ˝ ˝  C:
C
@ A @ ˝ ˝ A
0  
198 Chapter 8 Generalized and polynomial eigenvalue problems

Observe that the last step has finally filled up the .3; 3/-entry of E .3/ . However, as
E .3/ still has upper triangular form, this is acceptable. The pattern of the matrices
E .4/ and A.4/ is the typical pattern that we obtain after chasing down the zero on the
diagonal of E to the .n; n/-position. In order to deflate the problem, we can apply
a final Givens rotation G5 acting on columns that eliminates the .5; 4/-entry of A.4/ .
This will create a fill-in in the .4; 4/-position of E .4/ , but again this is acceptable as
the upper triangular form is not destroyed.
0 1 0 1
˝ ˝ ˝˝
B   ˝ ˝C B   ˝ ˝C
B C B C
B C
 ˝ ˝ C ; A D A G5 D B C
B   ˝ ˝C:
.5/ .4/ .5/ .4/
E D E G5 D B
@  ˝A @  ˝ ˝A
0 ˝

We can now deflate the eigenvalue 1 and continue with the .n  1/  .n  1/ ma-
trix pencil from the upper left corner of tE .5/  A.5/ which is again in Hessenberg-
triangular form.

Exercise 8.29. Write a procedure zero_chasing(var E; A) that deflates the eigen-

value 1 (which may have algebraic multiplicity larger than one) for a given nn
matrix pencil tE  A over F in Hessenberg-triangular form.

8.6 The QZ step

Let tE  A be a regular n  n matrix pencil over F in Hessenberg-triangular form. By
the previous section, we may also assume that E is invertible, because otherwise we
could deflate the eigenvalue 1 until we are left with a smaller pencil in Hessenberg-
triangular form not having the eigenvalue 1 any more. The idea is then to determine
unitary matrices Q; Z 2 F nn such that t E ee A WD Q .tE  A/Z is again in
Hessenberg-triangular form and such that e AEe is the matrix that one would obtain
1

after one step of the implicit QR iteration with multi-shift strategy outlined in Sec-
tion 5.6. We will do this exemplary for the case n D 5 and the double shift strategy
as in Example 5.17.
As in Chapter 5, the main idea is the application of Corollary 5.14. In the following
let H WD AE 1 and let 1 ; 2 2 C be two shifts. In Section 5.6, we started by
investigating the matrix B D .H  1 In /.H  2 In / which has the form
0 1

B    C
B C
BDB B    C :
C
@    A

Section 8.6 The QZ step 199

We then computed a Householder reflection M1 that eliminates the .2; 1/- and
.3; 1/-entries of B, but instead of applying M1 directly to B and computing the QR
factorization of B, we applied M1 directly to the matrix H and restored its Hessenberg
form with bulge chasing. Corollary 5.14 then implied that the resulting Hessenberg
matrix was essentially the matrix that we would have obtained after two consecutive
steps of the QR iteration with shifts 1 and 2 . Applying the same strategy here, we
will have to compute a Householder reflection that reflects the first column b1 of the
matrix B D .H  1 In /.H  2 In / to a multiple of the first canonical unit vector.
It is important to highlight that the first column of B can be easily computed without
explicitly forming E 1 and the product AE 1 , see Exercise 8.30.
Exercise 8.30 (First column of B). Let E; A 2 C nn such that E is nonsingular
and tE A is in Hessenberg-triangular form. Show that only the first three entries
of the first column

b1 D .AE 1  1 In /.AE 1  2 In /ı1

of the matrix B D .AE 1  1 In /.AE 1  2 In / are nonzero and provide a

formula for its entries in dependence of 1 , 2 and the entries of the first two
columns of A and E.
Hint: Use that E 1 is upper triangular and that you will need the entries of the
principal 2  2 submatrix only.

Once we have computed the first column b1 of B, we can compute a Householder

reflection M1 such that M1 b1 is a multiple of the first canonical unit vector. By
Exercise 8.30, only the first three entries of b1 are nonzero, so M1 will have the form
 
P 0
M1 D ;
0 In3
where P is a 3  3 Householder reflection. Thus M1 only manipulates the first three
rows of any matrix it will be applied to. Applying M1 both to E and A, we therefore
obtain matrices of the following form.
0 1 0 1
    
B     C B     C
B C B C
E D M1 E D B
.0/ C .0/ B
B    C ; A D M1 A D B    C :
C
@  A @   A
 
Thus, we have created a triangular and a Hessenberg matrix each having a bulge. If
we can now find unitary matrices Q; Z 2 C nn such that tQ M1 EZ  Q M1 AZ is
again in Hessenberg-triangular form and such that M1 Q and M1 have identical first
columns, then by Corollary 5.14 the matrix
Q M1 AZ.Q M1 EZ/1 D Q M1 AE 1 M1 Q
200 Chapter 8 Generalized and polynomial eigenvalue problems

is essentially the matrix that we would have obtained after two steps of the QR iter-
ation with shifts 1 and 2 applied to AE 1 . As in Chapter 5, we will achieve this
by bulge chasing. We start by applying a Householder reflection Z1 that changes the
.0/ .0/
first three columns to eliminate e31 and e32 . This will create fill-ins in the .4; 1/- and
.4; 2/-positions of A.0/ .
0 1 0 1
˝˝˝ ˝˝˝
B˝ ˝ ˝  C B˝ ˝ ˝  C
B C B C
E D E Z1 D B 0 0 ˝  C ; A D A Z1 D B
.1/ .0/ B C .1/ .0/
B˝ ˝ ˝  C :
C
@  A @   ˝  A
 

Next, we apply a Householder reflection Z2 that changes the first two columns to
.0/
eliminate e21 . Note that this will not create any additional fill-ins in our matrices.
0 1 0 1
˝˝ ˝ ˝   
B 0 ˝   C B˝ ˝   C
B C B C
E .2/ D E Z2 D B
.1/
B   CC; A.2/ D A.1/ Z2 D B
B˝ ˝   C
C:
@  A @˝ ˝   A
  

These two step have restored the triangular form of E. We now concentrate on the
matrix A.2/ and apply a Householder reflection Q1 changing the rows two to four to
.2/ .2/
eliminate the entries a31 and a41 .
0 1 0 1
         
B ˝ ˝ ˝ ˝C B˝ ˝ ˝ ˝ ˝C
B C B C
E .3/ D Q1 E .2/ DB
B  ˝ ˝ ˝CC; A.3/ D Q1 A.2/ DB
B0 ˝ ˝ ˝ ˝CC:
@   ˝ ˝A @0 ˝ ˝ ˝ ˝A
  

The fact that Q1 only changes the rows two to four, its conjugate transpose Q1 will
manipulate only the columns two to four if we multiply it from the right to M1 . Thus,
M1 and M1 Q1 will have identical first columns. Observe that as in Chapter 5, we
have managed to chase the bulge one step towards the right lower edge of the matrix.
We continue to chase the bulge further down and finally off the diagonal to restore the
Hessenberg-triangular form of our pencil.
0 1 0 1
˝˝˝ ˝˝˝
B ˝ ˝ ˝ C B ˝ ˝ ˝ C
B C B C
.4/ .3/ B ˝ ˝ ˝  C .4/ .3/ B C
E D E Z3 D B C ; A D A Z3 D B ˝ ˝ ˝ C ;
@ 0 0 ˝ A @ ˝ ˝ ˝ A
 ˝
Section 8.6 The QZ step 201

0 1 0 1
˝˝  ˝ ˝ 
B ˝ ˝  C B ˝ C
˝ 
B C B C
E D E Z4 D B
.5/ .4/ C
B 0 ˝  C ; A .5/
D A.4/ Z4 D B
B ˝ C
˝C; 
@  A @ ˝ A
˝ 
 ˝ ˝ 
0 1 0 1
        
B    C B     C
B C B C
E D Q2 E D B
.6/  .5/
B ˝ ˝ ˝ C;
C A D Q2 A D B
.6/  .5/
B ˝ ˝ ˝ ˝ C;
C
@  ˝ ˝A @ 0 ˝ ˝ ˝A
˝ 0 ˝˝˝
0 1 0 1
˝˝˝ ˝˝˝
B  ˝ ˝ ˝C B  ˝ ˝ ˝C
B C B C
E .7/ D E .6/ Z5 D B
B ˝ ˝ ˝CC; A.7/ D A.6/ Z5 D B
B  ˝ ˝ ˝C ;
C
@ ˝ ˝ ˝A @ ˝ ˝ ˝A
0 0 ˝ ˝˝˝
0 1 0 1
˝˝ ˝˝
B  ˝ ˝ C B  ˝ ˝ C
B C B C
E D E Z6 D B
.8/ .7/
B ˝ ˝ CC; A D A Z6 D B
.8/ .7/
B  ˝ ˝ C ;
C
@ 0 ˝ A @ ˝ ˝ A
 ˝˝
0 1 0 1
      
B    C B     C
B C B C
E D Q3 E D B
.9/  .8/
B   CC; A D Q3 A D B
.9/  .8/
B    C ;
C
@ ˝ ˝A @ ˝ ˝ ˝A
˝ 0 ˝˝
0 1 0 1
˝˝ ˝˝
B   ˝ ˝C B   ˝ ˝C
B C B C
E .10/
D E Z7 D B
.9/
B  ˝ ˝CC; A .10/
D A Z7 D B
.9/
B   ˝ ˝C :
C
@ ˝ ˝A @  ˝ ˝A
0 ˝ ˝˝
In the general n  n case, this procedure yields matrices

Q D Q0    Qn2 ; Z D Z1    Z2n3

ee
such that the pencil t E A WD Q .tE  A/Z is in Hessenberg-triangular form
 Q and M  have identical first columns. As explained above, the
and such that M1 1
matrix e
AEe 1 is essentially the Hessenberg matrix that is obtained after two steps of
the QR iteration with shifts 1 and 2 applied to the Hessenberg matrix AE 1 . We
can now repeat the steps until one or more elements on the subdiagonal of AE 1 are
202 Chapter 8 Generalized and polynomial eigenvalue problems

sufficiently small to be considered as zero. Clearly, the .k C 1; k/-element in AE 1 is

zero if and only if the .k C 1; k/-element in A is zero, since with E also E 1 is upper
triangular. In this case, we have deflation and can continue with two subproblems of
smaller size in order to compute the eigenvalues of our matrix pencil.
The algorithm outlined above is the so-called QZ-algorithm and was originally
proposed in [28]. It can be considered to be the standard algorithm for the solution of
dense (i.e., not sparse) generalized eigenvalue problems. For further reading, we refer
the reader to [28], [9], and [18].

Exercise 8.31. Write a procedure qz_step(var E; A) that performs a QZ step on

an n  n matrix pencil tE  A over F in Hessenberg-triangular form.

Exercise 8.32. Calculate how many operations are necessary to perform a QZ

step on an nn matrix pencil tE A over F in Hessenberg-triangular form. Also
calculate the number of operations needed to explicitly compute the transforma-
tion matrices Q and Z.
Bibliography

[1] W. E. Arnoldi, The principle of minimized iterations in the solution of the matrix eigen-
value problem, Quart. Appl. Math. 9 (1951), 17–29.
[2] Z. Bai, J. Demmel, J. Dongarra, A. Ruhe and H. van der Vorst, Templates for the Solution
of Algebraic Eigenvalue Problems, SIAM, Philadelphia, PA, 2000.
[3] F. L. Bauer and C. T. Fike, Norms and exclusion theorems, Numer. Math. 2 (1960),
137–141.
[4] K. Braman, R. Byers and R. Mathias, The Multishift QR Algorithm. Part I: Maintaining
Well-Focused Shifts and Level 3 Performance, SIAM Mat. Anal. Appl. 23 (2002), 929–
947.
[5] K. Braman, R. Byers and R. Mathias, The Multishift QR Algorithm. Part II: Aggressive
Early Deflation, SIAM Mat. Anal. Appl. 23 (2002), 948–973.
[6] R. Courant and D. Hilbert, Methoden der mathematischen Physik, Springer, 1924.
[7] J. J. M. Cuppen, A Divide and Conquer Method for the Symmetric Tridiagonal Eigen-
problem, Numer. Math. 36 (1981), 177–195.
[8] J. Demmel and K. Veselić, Jacobi’s method is more accurate than QR, SIAM J. Matrix
Anal. Appl. 13 (1992), 1204–1245.
[9] J. W. Demmel, Applied Numerical Linear Algebra, SIAM, 1997.
[10] I. S. Dhillon, A New O.n2 / Algorithm for the Symmetric Tridiagonal Eigen-
value/Eigenvector Problem, Ph. D. thesis, University of California, Berkeley, 1997.
[11] I. S. Dhillon and B. N. Parlett, Multiple representations to compute orthogonal eigen-
vectors of symmetric tridiagonal matrices, Lin. Alg. Appl. 387 (2004), 1–28.
[12] J. Dongarra and D. Sorensen, A fully parallel algorithm for the symmetric eigenvalue
problem, SIAM J. Sci. Statist. Comput. 8 (1987), 139–154.
[13] Z. Drmač, A Global Convergence Proof for Cyclic Jacobi Methods with Block Rota-
tions, SIAM J. Matrix Anal. Appl. 31 (2009), 1329–1350.
[14] J. G. F. Francis, The QR Transformation. A Unitary Analogue to the LR Transformation
— Part 1, The Computer Journal 4 (1961), 265–271.
[15] J. G. F. Francis, The QR Transformation — Part 2, The Computer Journal 4 (1962),
332–345.
[16] S. Gershgorin, Über die Abgrenzung der Eigenwerte einer Matrix, Izv. Akad. Nauk SSSR
Ser. Mat. 1 (1931), 749–754.
204 Bibliography

[17] L. Giraud, J. Langou, M. Rozložnik and J. van den Eshof, Rounding error analysis of the
classical Gram-Schmidt orthogonalization process, Numer. Math. 101 (2005), 87–100.
[18] G. H. Golub and C. F. Van Loan, Matrix Computations, Johns Hopkins University Press,
London, 1996.
[19] P. Henrici, On the Speed of Convergence of Cyclic and Quasicyclic Jacobi Methods
for Computing Eigenvalues of Hermitian Matrices, J. Soc. Ind. Appl. Math 6 (1958),
144–162.
[20] K. Hessenberg, Behandlung linearer Eigenwertaufgaben mit Hilfe der Hamilton-
Cayleyschen Gleichung, Institut für Praktische Mathematik, TH Darmstadt, Report
no. 1, 1940, available at http://www.hessenberg.de/numverf.pdf.
[21] A. S. Householder, Unitary Triangularization of a Nonsymmetric Matrix, Journal of the
ACM 5 (1958), 339–342.
[22] C. G. J. Jacobi, Über ein leichtes Verfahren die in der Theorie der Säcularstörungen
vorkommenden Gleichungen numerisch aufzulösen, Journal für die reine und ange-
wandte Mathematik 30 (1846), 51–94.
[23] A. Knyazev and K. Neymeyr, A geometric theory for preconditioned inverse iteration.
III: A short and sharp convergence estimate for generalized eigenvalue problems, Linear
Algebra Appl. 358 (2003), 95–114.
[24] D. Kressner, Numerical Methods for General and Structured Eigenvalue Problems,
Springer, 2005.
[25] A. N. Krylov, On the numerical solution of the equation by which, in technical matters,
frequencies of small oscillations of material systems are determined, Izv. Akad. Nauk.
S.S.S.R. Otdel. Mat. Estest. 1 (1931), 491–539.
[26] B. H. Kublanovskaja, On some algorithms for the solution of the complete problem of
proper values, J. Comput. Math. and Math. Phys. 1 (1961), 555–570.
[27] C. Lanczos, An iteration method for the solution of the eigenvalue problem of linear
differential and integral operators, J. Res. Nat. Bur. Stand. 45 (1950), 255–282.
[28] C. B. Moler and G. W. Stewart, An algorithm for generalized matrix eigenvalue prob-
lems, SIAM J. Numer. Anal. 10 (1973), 241–256.
[29] K. Neymeyr, A geometric theory for preconditioned inverse iteration. I: Extrema of the
Rayleigh quotient, Linear Algebra Appl. 322 (2001), 61–85.
[30] K. Neymeyr, A geometric theory for preconditioned inverse iteration. II: Convergence
estimates, Linear Algebra Appl. 322 (2001), 87–104.
[31] L. Page, S. Brin, R. Motwani and T. Winograd, The PageRank Citation Ranking: Bring-
ing Order to the Web, Stanford InfoLab, Report, 1999.
[32] B. N. Parlett, Global Convergence of the Basic QR Algorithm on Hessenberg Matrices,
Math. Comp. 22 (1968), 803–817.
[33] B. N. Parlett, The Symmetric Eigenvalue Problem, SIAM, 1987.
[34] O. Perron, Zur Theorie der Matrices, Mathematische Annalen 64 (1907), 248–263.
Bibliography 205

[35] E. Pohlhausen, Berechnung der Eigenschwingungen statisch-bestimmter Fachwerke,

ZAMM — Zeitschrift für Angewandte Mathematik und Mechanik 1 (1921), 28–42.
[36] H. Rutishauser, Der Quotienten-Differenzen-Algorithmus, ZAMP — Zeitschrift für
Angewandte Mathematik und Physik 5 (1954), 233–251.
[37] H. Rutishauser, Une methode pour la determination des valeurs propres d’une matrice,
Comptes Rendues de l’Academie des Sciences Paris 240 (1955), 34–36.
[38] H. Rutishauser, Solution of eigenvalue problems with the LR-transformation, National
Bureau of Standards Applied Mathematics Series 49 (1958), 47–81.
[39] A. Schönhage, Zur Konvergenz des Jacobi-Verfahrens, Numer. Math. 3 (1961), 374–
380.
[40] G. L. G. Sleijpen and H. A. A. van der Vorst, A Jacobi-Davidson iteration method for
linear eigenvalue problems, SIAM J. Matrix Anal. Appl. 17 (1996), 401–425.
[41] G. L. G. Sleijpen and H. A. A. van der Vorst, A Jacobi-Davidson iteration method for
linear eigenvalue problems, SIAM Review 42 (2000), 267–293.
[42] G. W. Stewart, On the sensitivity of the eigenvalue problem Ax D Bx, SIAM J. Numer.
Anal. 9 (1972), 669–686.
[43] G. W. Stewart, Matrix Algorithms. Volume I: Basic Decompositions, SIAM, 1998.
[44] G. W. Stewart, Matrix Algorithms. Volume II: Eigensystems, SIAM, 2001.
[45] L. N. Trefethen and D. Bau, Numerical Linear Algebra, SIAM, 1997.
[46] H. P. M. van Kempen, On the Quadratic Convergence of the Special Cyclic Jacobi
Method, Numer. Math. 9 (1966), 19–22.
[47] R. von Mises and H. Pollaczek-Geiringer, Praktische Verfahren der Gleichungsauflö-
sung, ZAMM — Zeitschrift für Angewandte Mathematik und Mechanik 9 (1929), 152–
164.
[48] D. S. Watkins, Understanding the QR algorithm, SIAM Review 24 (1982), 427–440.
[49] D. S. Watkins, The matrix eigenvalue problem, SIAM, 2007.
[50] D. S. Watkins, The QR algorithm revisited, SIAM Rev, 50 (2008), 133–145.
[51] J. H. Wilkinson, Note on the Quadratic Convergence of the Cyclic Jacobi Process, Nu-
mer. Math. 4 (1962), 296–300.
[52] J. H. Wilkinson, The Algebraic Eigenvalue Problem, Oxford University Press, 1965.
Index

Accuracy characterized by inner product 63

inverse iteration 75 characterized by projection 81
Jacobi iteration 52 Courant–Fischer–Weyl theorem 49
power iteration 71
Rayleigh quotient 68 Deflating subspace 189
Adjoint matrix 20 Deflation 116, 196
Algebraic multiplicity 14 aggressive 131
Angle 63 Determinant 12
between vector and subspace 80 Diagonalizability
characterized by projection 81 complex 29
Arnoldi configuration 162 real 31
Arnoldi iteration 160 real spectrum 30
Diagonalizable matrix 18
Bauer–Fike theorem 50 Dominant eigenvalue 62
Block-diagonalization 35
Bulge chasing 123, 200 Eigenpair 10
Eigenspace 12
Canonical unit vector 12 Eigenvalue 10
Cauchy–Schwarz inequality 19 dominant 62
Characteristic polynomial 13 infinite 187
evaluation for tridiagonal matrices 134 perturbed 50
Chebyshev polynomial 166, 168 simple 14
optimality 171 Eigenvector 10
trigonometric representation 169 generalized 37
Companion matrix 14 perturbed 51
Companion polynomial 184 Equivalence of matrix pencils 186
Complexity
Francis double shift 115, 130
Hessenberg QR step 107
Francis single shift 115
Hessenberg transformation 111
Frobenius norm 31
implicit QR iteration 124
Convergence Generalized eigenvalue problem 184
classical Jacobi iteration 58 Generalized eigenvector 37
cyclic Jacobi iteration 56 Geometric multiplicity 12
inverse iteration 73 Gershgorin spheres 143
Jacobi iteration 45 Gram–Schmidt process 152
Krylov subspace method 174, 175
power iteration 63, 82, 95 Hessenberg matrix 104
Rayleigh iteration 77 Hessenberg-triangular form 193
simultaneous iteration 88 Hilbert matrix 159
Cosine Hilbert space 19
Index 207

Householder reflection 27 Minimal polynomial 151

Minimization problem 49
Infinite eigenvalues 187 Multiplicity
Injective 9 algebraic 14, 188
Invariant subspace 25 geometric 12, 188
Inverse iteration 73
accuracy 75 Neumann series 142
convergence 73 Nil-potent matrix 11
Irreducible matrix 137 Norm
Isometric matrix 23 Frobenius 31
maximum 141
Jacobi iteration spectral 47
accuracy 52 Normal matrix 22
classical 46 Null space 9
convergence 45 Nullity 9
cyclic-by-row 47 Orthogonal iteration 86
Orthogonal matrix 23
Krylov subspace 149 Orthogonal projection 80
Orthonormal basis 149
Lanczos iteration 164
Linearization 183 Perpendicular subspaces 21
Perpendicular vectors 21
Matrices Perturbed eigenvalue 50
similar 16 Polynomial
Matrix characteristic 13
adjoint 20 Chebyshev 166, 168
˛-triangular 92 minimal 151
diagonalizable 18 Polynomial eigenvalue problem 6, 183
exponential 18 Power iteration
Hessenberg 104 accuracy 71
Hilbert 159 convergence 63, 95
irreducible 137 convergence to subspace 82
isometric 23 Projection 11
nil-potent 11 orthogonal 80
normal 22 Projection methods 148
off-diagonal part 32
QR factorization 85
orthogonal 23
existence 85
self-adjoint 21
QZ algorithm 198
sparse 145
triangular 24 Range 9
tridiagonal 104 Rank 9
unitary 23 Rayleigh iteration 77
unreduced 137 convergence 77
Matrix pencil 184 Rayleigh quotient
regular 185 accuracy 68
singular 185 definition 67
Matrix polynomial 183 Rayleigh shift 114
Metric equivalence 22 Residual 70, 163
208 Index

inverse iteration 75 characterized by minimization 68

power iteration 72 characterized by projection 81
Ritz pair 148 Spectral gap 51
computation 162 Spectral norm 47
Ritz value 148 Spectrum 10
Ritz vector 148 Sturm chain 135
Ritz–Galerkin condition 148 for tridiagonal matrix 138
Subspace
Schur decomposition 27
invariant 25
Schur decomposition, generalized 191
Surjective 9
Self-adjoint matrix 21
Sylvester’s equation 34
Shift
Francis double 115, 130
Francis single 115 Tangent
multiple 130 characterized by inner product 63
Rayleigh 114 characterized by projection 81
Shift parameter 73, 113 Triangular matrix 24
Similar matrices 16 Tridiagonal matrix 104
Simple eigenvalue 14
Simultaneous iteration 86 Unitary matrix 23
convergence 88
Sine Zero-chasing 196
characterized by inner product 63 Zero-counting 136