Lecture Notes on

Solving Large Scale Eigenvalue Problems

Prof. Dr. Peter Arbenz

Computer Science Department
ETH Zurich
Email: arbenz@inf.ethz.ch

Spring semester 2016

ii
Contents

1 Introduction 1
1.1 What makes eigenvalues interesting? . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Example 1: The vibrating string . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2.1 Problem setting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2.2 The method of separation of variables . . . . . . . . . . . . . . . . . 5
1.3 Numerical methods for solving 1-dimensional problems . . . . . . . . . . . . 6
1.3.1 Finite differences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.3.2 The finite element method . . . . . . . . . . . . . . . . . . . . . . . . 7
1.3.3 Global functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.3.4 A numerical comparison . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.4 Example 2: The heat equation . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.5 Example 3: The wave equation . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.6 The 2D Laplace eigenvalue problem . . . . . . . . . . . . . . . . . . . . . . 13
1.6.1 The finite difference method . . . . . . . . . . . . . . . . . . . . . . . 13
1.6.2 The finite element method (FEM) . . . . . . . . . . . . . . . . . . . 16
1.6.3 A numerical example . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
1.7 Cavity resonances in particle accelerators . . . . . . . . . . . . . . . . . . . 21
1.8 Spectral clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
1.8.1 The graph Laplacian . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
1.8.2 Spectral clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
1.8.3 Normalized graph Laplacians . . . . . . . . . . . . . . . . . . . . . . 27
1.9 Googles PageRank . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
1.10 Other sources of eigenvalue problems . . . . . . . . . . . . . . . . . . . . . . 30
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

2 Basics 33
2.1 Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.2 Statement of the problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
2.3 Similarity transformations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.4 Schur decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
2.5 The real Schur decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . 39
2.6 Normal matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
2.7 Hermitian matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
2.8 The Jordan normal form . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
2.9 Projections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
2.10 The Rayleigh quotient . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
2.11 Cholesky factorization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
2.12 The singular value decomposition (SVD) . . . . . . . . . . . . . . . . . . . . 50

iii
iv CONTENTS
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

3 Newton methods 53
3.1 Linear and nonlinear eigenvalue problems . . . . . . . . . . . . . . . . . . . 53
3.2 Zeros of the determinant . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
3.2.1 Algorithmic differentiation . . . . . . . . . . . . . . . . . . . . . . . . 55
3.2.2 Hymans algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
3.2.3 Computing multiple zeros . . . . . . . . . . . . . . . . . . . . . . . . 58
3.3 Newton methods for the constrained matrix problem . . . . . . . . . . . . . 58
3.4 Successive linear approximations . . . . . . . . . . . . . . . . . . . . . . . . 60
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

4 The QR Algorithm 63
4.1 The basic QR algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
4.1.1 Numerical experiments . . . . . . . . . . . . . . . . . . . . . . . . . . 64
4.2 The Hessenberg QR algorithm . . . . . . . . . . . . . . . . . . . . . . . . . 67
4.2.1 A numerical experiment . . . . . . . . . . . . . . . . . . . . . . . . . 69
4.2.2 Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
4.3 The Householder reduction to Hessenberg form . . . . . . . . . . . . . . . . 71
4.3.1 Householder reflectors . . . . . . . . . . . . . . . . . . . . . . . . . . 71
4.3.2 Reduction to Hessenberg form . . . . . . . . . . . . . . . . . . . . . 71
4.4 Improving the convergence of the QR algorithm . . . . . . . . . . . . . . . . 73
4.4.1 A numerical example . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
4.4.2 QR algorithm with shifts . . . . . . . . . . . . . . . . . . . . . . . . 75
4.4.3 A numerical example . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
4.5 The double shift QR algorithm . . . . . . . . . . . . . . . . . . . . . . . . . 77
4.5.1 A numerical example . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
4.5.2 The complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
4.6 The symmetric tridiagonal QR algorithm . . . . . . . . . . . . . . . . . . . 84
4.6.1 Reduction to tridiagonal form . . . . . . . . . . . . . . . . . . . . . . 84
4.6.2 The tridiagonal QR algorithm . . . . . . . . . . . . . . . . . . . . . . 85
4.7 Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

5 Cuppens Divide and Conquer Algorithm 91

5.1 The divide and conquer idea . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
5.2 Partitioning the tridiagonal matrix . . . . . . . . . . . . . . . . . . . . . . . 92
5.3 Solving the small systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
5.4 Deflation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
5.4.1 Numerical examples . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
5.5 The eigenvalue problem for D + vvT . . . . . . . . . . . . . . . . . . . . . 95
5.6 Solving the secular equation . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
5.7 A first algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
5.7.1 A numerical example . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
5.8 The algorithm of Gu and Eisenstat . . . . . . . . . . . . . . . . . . . . . . . 103
5.8.1 A numerical example [continued] . . . . . . . . . . . . . . . . . . . . 104
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
CONTENTS v
6 LAPACK and the BLAS 109
6.1 LAPACK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
6.2 BLAS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
6.2.1 Typical performance numbers for the BLAS . . . . . . . . . . . . . . 111
6.3 Blocking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
6.4 LAPACK solvers for the symmetric eigenproblems . . . . . . . . . . . . . . 114
6.5 Generalized Symmetric Definite Eigenproblems (GSEP) . . . . . . . . . . . 116
6.6 An example of a LAPACK routines . . . . . . . . . . . . . . . . . . . . . . . 116
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122

7 Vector iteration (power method) 125

7.1 Simple vector iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
7.2 Angles between vectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
7.3 Convergence analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
7.4 A numerical example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
7.5 The symmetric case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
7.6 Inverse vector iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
7.7 The generalized eigenvalue problem . . . . . . . . . . . . . . . . . . . . . . . 139
7.8 Computing higher eigenvalues . . . . . . . . . . . . . . . . . . . . . . . . . . 139
7.9 Rayleigh quotient iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
7.9.1 A numerical example . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144

8 Simultaneous vector or subspace iterations 145

8.1 Basic subspace iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
8.2 Angles between subspaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
8.3 Convergence of basic subspace iteration . . . . . . . . . . . . . . . . . . . . 148
8.4 Accelerating subspace iteration . . . . . . . . . . . . . . . . . . . . . . . . . 153
8.5 Relation between subspace iteration and QR algorithm . . . . . . . . . . . . 158
8.6 Addendum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161

9 Krylov subspaces 163

9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
9.2 Definition and basic properties . . . . . . . . . . . . . . . . . . . . . . . . . 164
9.3 Polynomial representation of Krylov subspaces . . . . . . . . . . . . . . . . 165
9.4 Error bounds of Saad . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171

10 Arnoldi and Lanczos algorithms 173

10.1 An orthonormal basis for the Krylov space Kj (x) . . . . . . . . . . . . . . . 173
10.2 Arnoldi algorithm with explicit restarts . . . . . . . . . . . . . . . . . . . . 175
10.3 The Lanczos basis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
10.4 The Lanczos process as an iterative method . . . . . . . . . . . . . . . . . . 178
10.5 An error analysis of the unmodified Lanczos algorithm . . . . . . . . . . . . 185
10.6 Partial reorthogonalization . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
10.7 Block Lanczos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
10.8 External selective reorthogonalization . . . . . . . . . . . . . . . . . . . . . 193
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
vi CONTENTS
11 Restarting Arnoldi and Lanczos algorithms 195
11.1 The m-step Arnoldi iteration . . . . . . . . . . . . . . . . . . . . . . . . . . 195
11.2 Implicit restart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
11.3 Convergence criterion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
11.4 The generalized eigenvalue problem . . . . . . . . . . . . . . . . . . . . . . . 199
11.5 A numerical example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
11.6 Another numerical example . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
11.7 The Lanczos algorithm with thick restarts . . . . . . . . . . . . . . . . . . . 210
11.8 KrylovSchur algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
11.9 The rational Krylov space method . . . . . . . . . . . . . . . . . . . . . . . 214
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215

12 The Jacobi-Davidson Method 217

12.1 The Davidson algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
12.2 The Jacobi orthogonal component correction . . . . . . . . . . . . . . . . . 218
12.2.1 Restarts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
12.2.2 The computation of several eigenvalues . . . . . . . . . . . . . . . . 221
12.2.3 Spectral shifts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
12.3 The generalized Hermitian eigenvalue problem . . . . . . . . . . . . . . . . 224
12.4 A numerical example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
12.5 The JacobiDavidson algorithm for interior eigenvalues . . . . . . . . . . . . 228
12.6 Harmonic Ritz values and vectors . . . . . . . . . . . . . . . . . . . . . . . . 229
12.7 Refined Ritz vectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
12.8 The generalized Schur decomposition . . . . . . . . . . . . . . . . . . . . . . 233
12.9 JDQZ: Computing a partial QZ decomposition . . . . . . . . . . . . . . . . 233
12.9.1 Restart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
12.9.2 Deflation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
12.9.3 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
12.10Jacobi-Davidson for nonlinear eigenvalue problems . . . . . . . . . . . . . . 236
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239

13 Rayleigh quotient and trace minimization 241

13.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
13.2 The method of steepest descent . . . . . . . . . . . . . . . . . . . . . . . . . 242
13.3 The conjugate gradient algorithm . . . . . . . . . . . . . . . . . . . . . . . . 243
13.4 Locally optimal PCG (LOPCG) . . . . . . . . . . . . . . . . . . . . . . . . . 247
13.5 The block Rayleigh quotient minimization algorithm (BRQMIN) . . . . . . 250
13.6 The locally-optimal block preconditioned conjugate gradient method (LOBPCG)250
13.7 A numerical example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
13.8 Trace minimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
Chapter 1

Introduction

Before we start with the subject of this notes we want to show how one actually arrives at
large eigenvalue problems in practice. In the following, we restrict ourselves to problems
from physics [7, 18, 14] and computer science.

1.1 What makes eigenvalues interesting?

In physics, eigenvalues are usually related to vibrations. Objects like violin strings, drums,
bridges, sky scrapers can swing. They do this at certain frequencies. And in some situ-
ations they swing so much that they are destroyed. On November 7, 1940, the Tacoma
narrows bridge collapsed, less than half a year after its opening. Strong winds excited the
bridge so much that the platform in reinforced concrete fell into pieces. A few years ago
the London millennium footbridge started wobbling in a way that it had to be closed. The
wobbling had been excited by the pedestrians passing the bridge. These are prominent
examples of vibrating structures.
But eigenvalues appear in many other places. Electric fields in cyclotrones, a special
form of particle accelerators, have to oscillate in a precise manner, in order to accelerate the
charged particles that circle around its center. The solutions of the Schr odinger equation
from quantum physics and quantum chemistry have solutions that correspond to vibrations
of the, say, molecule it models. The eigenvalues correspond to energy levels that molecule
can occupy.
Many characteristic quantities in science are eigenvalues:

decay factors,

frequencies,

norms of operators (or matrices),

singular values,

condition numbers.

In the sequel we give a number of examples that show why computing eigenvalues is
important. At the same time we introduce some notation.

1
2 CHAPTER 1. INTRODUCTION
1.2 Example 1: The vibrating string
1.2.1 Problem setting
Let us consider a string as displayed in Fig. 1.1. The string is fixed at both ends, at x = 0

u(x,t)
x
0 L

Figure 1.1: A vibrating string fixed at both ends.

and x = L. The x-axis coincides with the strings equilibrium position. The displacement
of the rest position at x, 0 < x < L, and time t is denoted by u(x, t).
We will assume that the spatial derivatives of u are not very large:
 
 u 
  is small.
 x 

This assumption entails that we may neglect terms of higher order.

Let v(x, t) be the velocity of the string at position x and at time t. Then the kinetic
energy of a string section ds of mass dm = ds is given by
 2
1 1 u
(1.1) dT = dm v 2 = ds .
2 2 t

2 2
From Fig. 1.2 we see that ds2 = dx2 + u
x dx and thus
s  2  
ds u 1 u 2
= 1+ =1+ + higher order terms.
dx x 2 x

Plugging this into (1.1) and omitting also the second order term (leaving just the number 1)
gives
 
dx u 2
dT = .
2 t
The kinetic energy of the whole string is obtained by integrating over its length,
Z L Z  2
1 L u
T = dT (x) = (x) dx
0 2 0 t
The potential energy of the string has two components
1.2. EXAMPLE 1: THE VIBRATING STRING 3

ds

dx

Figure 1.2: A vibrating string, local picture.

1. the stretching times the exerted strain ,

s
Z L Z L Z L  2
ds dx = 1 + u 1 dx
0 0 0 x
Z  2 !
L
1 u
= + higher order terms dx
0 2 x

2. exterior forces of density f ,

Z L
f udx.
0

Summing up, the potential energy of the string becomes

Z L   !
u 2
(1.2) V = f u dx.
0 2 x

To consider the motion (vibration) of the string in a certain time interval t1 t t2 we

form the integral
Z t2 Z Z "  2  2 #
1 t2 L u u
(1.3) I(u) = (T V ) dt = (x) + 2f u dx dt
t1 2 t1 0 t x

Here functions u(x, t) are admitted that are differentiable with respect to x and t and
satisfy the boundary conditions (BC) that correspond to the fixing,

(1.4) u(0, t) = u(L, t) = 0, t1 t t2 ,

as well as given initial conditions and end conditions,

u(x, t1 ) = u1 (x),
(1.5) 0 < x < L.
u(x, t2 ) = u2 (x),
4 CHAPTER 1. INTRODUCTION
According to the principle of Hamilton a mechanical system with kinetic energy T and
potential energy V behaves in a time interval t1 t t2 for given initial and end positions
such that
Z t2
I= L dt, L = T V,
t1

is minimized.
Let u(x, t) be such that I(u) I(w) for all w, that satisfy the initial, end, and
boundary conditions. Let w = u + v with

() v(0, t) = v(L, t) = 0, v(x, t1 ) = v(x, t2 ) = 0.

v is called a variation. We now consider I(u + v) as a function of . Then we have the

equivalence
I(u) minimal dI (u) = 0 for all admitted v.
d

Plugging u + v into eq. (1.3) we obtain

Zt2 ZL "  2  2 #
1 (u + v) (u + v)
I(u + v) = (x) + 2f (u + v) dx dt
2 t x
t1 0
(1.6)
Zt2 ZL  
u v u v
= I(u) + (x) + 2f v dx dt + O(2 ).
t t x x
t1 0

Thus, after integration by parts, exploiting the conditions in (), the equation
Z t2 Z L  
I 2u 2u
= + 2 f v dx dt = 0
t1 0 t2 x2

must hold for all admissible v. Therefore, the bracketed expression must vanish,

2u 2u
(1.7) + = 2 f.
t2 x2

This last differential equation is named Euler-Lagrange equation.

If the force is proportional to the displacement u(x, t) (like, e.g., in Hookes law) then
we get a differential equation of the form

2  
(x) u + p(x) u + q(x)u(x, t) = 0.
(1.8) t2 x x
u(0, t) = u(1, t) = 0

which is a special case of the Euler-Lagrange equation (1.7). Here, (x) plays the role of
a mass density, p(x) of a locally varying elasticity module. We do not specify initial and
end conditions for the moment. Note that there are no external forces present in (1.8).
From physics we know that (x) > 0 and p(x) > 0 for all x. These properties are of
importance also from a mathematical view point! For simplicity, we assume that (x) = 1.
1.2. EXAMPLE 1: THE VIBRATING STRING 5
1.2.2 The method of separation of variables
For the solution u in (1.8) we make the ansatz

(1.9) u(x, t) = v(t)w(x).

Here, v is a function that depends only on the time t, while w depends only on the spatial
variable x. With this ansatz (1.8) becomes

(1.10) v (t)w(x) v(t)(p(x)w (x)) q(x)v(t)w(x) = 0.

Now we separate the variables depending on t from those depending on x,

v (t) 1
= (p(x)w (x)) + q(x).
v(t) w(x)
This equation holds for any t and x. We can vary t and x independently of each other
without changing the value on each side of the equation. Therefore, each side of the
equation must be equal to a constant value. We denote this value by . Thus, from the
left side we obtain the equation

(1.11) v (t) = v(t).

This equation has the well-known solution v(t) = a cos( t) + b sin( t) where > 0
is assumed. The right side of (1.10) gives a so-called Sturm-Liouville problem

(1.12) (p(x)w (x)) + q(x)w(x) = w(x), w(0) = w(1) = 0.

A value for which (1.12) has a non-trivial (i.e. nonzero) solution w is called an eigen-
value; w is a corresponding eigenfunction. It is known that all eigenvalues of (1.12) are
positive. By means of our ansatz (1.9) we get
h i
u(x, t) = w(x) a cos( t) + b sin( t)

as a solution of (1.8). It is known that (1.12) has infinitely many real positive eigenvalues
0 < 1 2 , (k ). (1.12) has a non-zero solution, say wk (x), only for these
k
particular values k . Therefore, the general solution of (1.8) has the form

X h p p i
(1.13) u(x, t) = wk (x) ak cos( k t) + bk sin( k t) .
k=0

The coefficients ak and bk are determined by initial and end conditions. We could, e.g.,
require that

X
u(x, 0) = ak wk (x) = u0 (x),
k=0
p
X
u
(x, 0) = k bk wk (x) = u1 (x),
t
k=0
where u0 and u1 are given functions. It is known that the wk form an orthogonal basis in
the space of square integrable functions L2 (0, 1),
Z 1
wk (x)w (x)dx = k k .
0
6 CHAPTER 1. INTRODUCTION
Therefore, it is not difficult to compute the coefficients ak and bk ,
Z 1 Z 1 p
ak = u0 (x)wk (x)dx/k , bk = u1 (x)wk (x)dx/k k .
0 0

In concluding, we see that the difficult problem to solve is the eigenvalue problem (1.12).
Knowing the eigenvalues and eigenfunctions the general solution of the time-dependent
problem (1.8) is easy to form.
Eq. (1.12) can be solved analytically only in very special situation, e.g., if all coefficients
are constants. In general a numerical method is needed to solve the Sturm-Liouville
problem (1.12).

1.3 Numerical methods for solving 1-dimensional problems

In this section we consider three methods to solve the Sturm-Liouville problem.

1.3.1 Finite differences

We approximate w(x) by its values at the discrete points xi = ih, h = 1/(n + 1), i =
1, . . . , n.

x
0 xi1xi xi+1 L

Figure 1.3: Grid points in the interval (0, L).

At point xi we approximate the derivatives by finite differences. We proceed as

follows. First we write
d g(xi+ 1 ) g(xi 1 )
2 2
g(xi ) .
dx h
For g = p dw
dx we get
w(xi+1 ) w(xi )
g(xi+ 1 ) = p(xi+ 1 )
2 2 h
and, finally, for i = 1, . . . , n,
   
d dw 1 w(xi+1 ) w(xi ) w(xi ) w(xi1 )
p (xi ) p(xi+ 1 ) p(xi 1 )
dx dx h 2 h 2 h
1 h i
= 2 p(xi 1 )wi1 + (p(xi 1 ) + p(xi+ 1 ))wi p(xi+ 1 )wi+1 .
h 2 2 2 2

Note that at the interval endpoints w0 = wn+1 = 0.

We can collect all equations in a matrix equation,

p(x 1 ) + p(x 3 ) p(x 3 ) w1 w1
2 2
+ q(x 1 ) 2

h2 h2 w2 w2
p(x 3 ) p(x 3 ) + p(x 5 ) p(x 5 )

w3 = w3
,
22 2 2
+ q(x 2 ) 2
. ..
h h2 h2 . .
p(x 5 ) .. .. .
22 . . wn wn
h
1.3. NUMERICAL METHODS FOR SOLVING 1-DIMENSIONAL PROBLEMS 7
or, briefly,

(1.14) Aw = w.

By construction, A is symmetric and tridiagonal. One can show that it is positive definite
as well. Note that this matrix has just a few nonzeros: out of the n2 elements of A only
3n 2 are nonzero. This is an example of a sparse matrix.

1.3.2 The finite element method

We write (1.12) in the form
Find a twice differentiable function w with w(0) = w(1) = 0 such that
Z 1 
(p(x)w (x)) + q(x)w(x) w(x) (x)dx = 0
0

for all smooth functions that satisfy (0) = (1) = 0.

To relax the requirements on w we integrate by parts and get the new so-called weak
or variational form of the problem:

Find a differentiable function w with w(0) = w(1) = 0 such that

Z 1 
(1.15) p(x)w(x) (x) + q(x)w(x)(x) w(x)(x) dx = 0
0

for all differentiable functions that satisfy (0) = (1) = 0.

Remark: Requiring continuous differentiability is too strong and does not lead to a
mathematically suitable formulation. In particular, the test functions that will be used
below are not differentiable in the classical sense. It is more appropriate to require w and
to be weakly differentiable. In terms of Sobolev spaces: w, H01 ([0, 1]). An introduction
to Sobolev spaces is, however, beyond the scope of these notes.

i 1

x
0 xi1xi xi+1 L

Figure 1.4: A basis function of the finite element space: a hat function.

We now write w as the linear combination

n
X
(1.16) w(x) = i i (x),
i=1

where
 
|x xi | |x xi |
(1.17) i (x) = 1 = max{0, 1 },
h + h
8 CHAPTER 1. INTRODUCTION
is the function that is linear in each interval (xi , xi+1 ) and satisfies

1, i = k,
i (xk ) = ik :=
0, i 6= k.

An example of such a basis function, a so-called hat function, is displayed in Fig. 1.4.
We now replace w in (1.15) by the linear combination (1.16), and replace testing
against all by testing against all j . In this way (1.15) becomes
Z 1 n n
!
X X

p(x)( i i (x))j (x) + (q(x) ) i i (x)j (x) dx, for all j,
0 i=1 i=1

or,
n
X Z 1

(1.18) i p(x)i (x)j (x) + (q(x) )i (x)j (x) dx = 0, for all j.
i=1 0

These last equations are called the RayleighRitzGalerkin equations. Unknown are
the n values i and the eigenvalue . In matrix notation (1.18) becomes

(1.19) Ax = M x

with Z Z
1
1
aij = p(x)i j + q(x)i j dx and mij = i j dx
0 0
For the specific case p(x) = 1 + x and q(x) = 1 we get
Z kh "   #
1 x (k 1)h 2
akk = (1 + x) 2 + dx
(k1)h h h
Z (k+1)h "   #
1 (k + 1)h x 2 2 1
+ (1 + x) 2 + dx = 2(n + 1 + k) +
kh h h 3n+1
Z (k+1)h  
1 (k + 1)h x x kh 3 1 1
ak,k+1 = (1 + x) 2 + dx = n k +
kh h h h 2 6n+1

In the same way we get

4 1
..
1 1 4 .
M=
6(n + 1)
... ..
. 1
1 4
Notice that both matrices A and M are symmetric tridiagonal and positive definite.

1.3.3 Global functions

Formally we proceed as with the finite element method, i.e., we solve equation (1.18). But
now we choose the k (x) to be functions with global support1 . We could, e.g., set

k (x) = sin kx,

1
The support of a function f is the set of arguments x for which f (x) 6= 0.
1.4. EXAMPLE 2: THE HEAT EQUATION 9
functions that are differentiable and satisfy the homogeneous boundary conditions. The k
are eigenfunctions of the nearby problem u (x) = u(x), u(0) = u(1) = 0 corresponding
to the eigenvalue k2 2 . The elements of matrix A are given by
Z 1  3 1
akk = (1 + x)k2 2 cos2 kx + sin2 kx dx = k2 2 + ,
0 4 2
Z 1
 
akj = (1 + x)kj 2 cos kx cos jx + sin kx sin jx dx
0
kj(k2 + j 2 )((1)k+j 1)
= , k 6= j.
(k2 j 2 )2

1.3.4 A numerical comparison

We consider the above 1-dimensional eigenvalue problem

(1.20) ((1 + x)w (x)) + w(x) = w(x), w(0) = w(1) = 0,

and solve it with the finite difference and finite element methods as well as with the global
functions method. The results are given in Table 1.1.
Clearly the global function method is the most powerful of them all. With 80 basis
functions the eigenvalues all come right. The convergence rate is exponential.
With the finite difference and finite element methods the eigenvalues exhibit quadratic
convergence rates. If the mesh width h is reduced by a factor of q = 2, the error in the
eigenvalues is reduced by the factor q 2 = 4. There exist higher order finite elements and
higher order finite difference stencils [11, 6].

1.4 Example 2: The heat equation

The instationary temperature distribution u(x, t) in an insulated container satisfies the
equations

u(x, t)
u(x, t) = 0, x , t > 0,
t
(1.21) u(x, t)
= 0, x , t > 0,
n
u(x, 0) = u0 (x), x .

Here is a 3-dimensional domain2 with boundary . u0 (x), x = (x1 , x2 , x3 )T R3 , is a

given bounded, sufficiently smooth function.
X 2u
(1.22) u =
xi 2
u
is called the Laplace operator and n denotes the derivative of u in direction of the outer
normal vector n. To solve the heat equation the method of separation of variables is
employed. We write u in the form

(1.23) u(x, t) = v(t)w(x).

2
In the sequel we understand a domain to be bounded and simply connected.
10 CHAPTER 1. INTRODUCTION

Finite difference method

k k (n = 10) k (n = 20) k (n = 40) k (n = 80)
1 15.245 15.312 15.331 15.336
2 56.918 58.048 58.367 58.451
3 122.489 128.181 129.804 130.236
4 206.419 224.091 229.211 230.580
5 301.499 343.555 355.986 359.327
6 399.367 483.791 509.358 516.276
7 492.026 641.501 688.398 701.185
8 578.707 812.933 892.016 913.767
9 672.960 993.925 1118.969 1153.691
10 794.370 1179.947 1367.869 1420.585

Finite element method

k k (n = 10) k (n = 20) k (n = 40) k (n = 80)
1 15.447 15.367 15.345 15.340
2 60.140 58.932 58.599 58.511
3 138.788 132.657 130.979 130.537
4 257.814 238.236 232.923 231.531
5 426.223 378.080 365.047 361.648
6 654.377 555.340 528.148 521.091
7 949.544 773.918 723.207 710.105
8 1305.720 1038.433 951.392 928.983
9 1702.024 1354.106 1214.066 1178.064
10 2180.159 1726.473 1512.784 1457.733

Global function method

k k (n = 10) k (n = 20) k (n = 40) k (n = 80)
1 15.338 15.338 15.338 15.338
2 58.482 58.480 58.480 58.480
3 130.389 130.386 130.386 130.386
4 231.065 231.054 231.053 231.053
5 360.511 360.484 360.483 360.483
6 518.804 518.676 518.674 518.674
7 706.134 705.631 705.628 705.628
8 924.960 921.351 921.344 921.344
9 1186.674 1165.832 1165.823 1165.822
10 1577.340 1439.083 1439.063 1439.063

Table 1.1: Numerical solutions of problem (1.20)

1.4. EXAMPLE 2: THE HEAT EQUATION 11
If a constant can be found such that

w(x) + w(x) = 0, w(x) 6= 0, x in ,

(1.24) w(x)
= 0, x on ,
n

then the product u = vw is a solution of (1.21) if and only if

dv(t)
(1.25) + v(t) = 0,
dt

the solution of which has the form aexp(t). By separating variables, the problem (1.21)
is divided in two subproblems that are hopefully easier to solve. A value , for which (1.24)
has a nontrivial (i.e. a nonzero) solution is called an eigenvalue; w then is called a corre-
sponding eigenfunction.
If n is an eigenvalue of problem (1.24) with corresponding eigenfunction wn , then

en t wn (x)

is a solution of the first two equations in (1.21). It is known that equation (1.24) has
infinitely many real eigenvalues 0 1 2 , that tend to infinity, n as
n . Multiple eigenvalues are counted according to their multiplicity. An arbitrary
bounded piecewise continuous function can be represented as a linear combination of the
eigenfunctions w1 , w2 , . . . Therefore, the solution of (1.21) can be written in the form

X
(1.26) u(x, t) = cn en t wn (x),
n=1

where the coefficients cn are determined such that

X
(1.27) u0 (x) = cn wn (x).
n=1

The smallest eigenvalue of (1.24) is 1 = 0 with w1 = 1 and 2 > 0. Therefore we see

from (1.26) that

(1.28) u(x, t) c1 .
t

Thus, in the limit (i.e., as t goes to infinity), the temperature will be constant in the whole
container. The convergence rate towards this equilibrium is determined by the smallest
positive eigenvalue 2 of (1.24):

X
X
n t
ku(x, t) c1 k = k cn e wn (x)k |en t |kcn wn (x)k
n=2 n=2

X
2 t
e kcn wn (x)k e2 t ku0 (x)k.
n=2

Here we have assumed that the value of the constant function w1 (x) is set to unity.
12 CHAPTER 1. INTRODUCTION
1.5 Example 3: The wave equation
The air pressure u(x, t) in a volume with acoustically hard walls satisfies the equations

2 u(x, t)
(1.29) u(x, t) = 0, x , t > 0,
t2
u(x, t)
(1.30) = 0, x , t > 0,
n
(1.31) u(x, 0) = u0 (x), x ,
u(x, 0)
(1.32) = u1 (x), x .
t
Sound propagates with speed u, along the (negative) gradient from high to low pres-
sure.
To solve the wave equation we proceed as with the heat equation in section 1.4: sepa-
ration of u according to (1.23) leads again to equation (1.24) but now together with

d2 v(t)
(1.33) + v(t) = 0.
dt2
We know this equation from the analysis of the vibrating string, see (1.11). From there
we know that the general solution of the wave equation has the form

X h p p i
(1.13) u(x, t) = wk (x) Ak cos( k t) + Bk sin( k t) .
k=0

where the wk , k = 1, 2, . . ., are the eigenfunctions of the eigenvalue problem (1.24). The
coefficients ak and bk are determined by (1.31) and (1.32).
If a harmonic oscillation is forced on the system, an inhomogeneous problem

2 u(x, t)
(1.34) u(x, t) = f (x, t),
t2
is obtained. The boundary and initial conditions are taken from (1.29)(1.32). This
problem can be solved by expanding u and f in the eigenfunctions wn (x),

X
u(x, t) := vn (t)wn (x),
n=1
(1.35)
X
f (x, t) := n (t)wn (x).
n=1

With this approach, vn has to satisfy equation

d2 vn
(1.36) + n vn = n (t).
dt2
If n (t) = an sin t, then the solution becomes
p p 1
(1.37) vn = An cos n t + Bn sin n t + an sin t.
n 2
1.6. THE 2D LAPLACE EIGENVALUE PROBLEM 13
Anand Bn are real constants that are determined by the initial conditions.
If gets close
to n , then the last term can be very large. In the limit, if = n , vn gets the form
p p
(1.38) vn = An cos n t + Bn sin n t + an t sin t,

in which case, vn is not bounded in time anymore. This phenomenon is called resonance.
Often resonance is not desirable; it may, e.g., mean the blow up of some structure. In
order to prevent resonances eigenvalues have to be known. Possible remedies are changing
the domain (the structure) or parameters (the materials).
Remark 1.1. Vibrating membranes satisfy the wave equation, too. In general the boundary
conditions are different from (1.30). If the membrane (of a drum) is fixed at its boundary,
the condition

(1.39) u(x, t) = 0

is imposed. These boundary conditions are called Dirichlet boundary conditions. The
boundary conditions in (1.21) and (1.30) are called Neumann boundary conditions. Com-
binations of these two can occur.

1.6 Numerical methods for solving the Laplace eigenvalue

problem in 2D
In this section we again consider the eigenvalue problem

(1.40) u(x) = u(x), x ,

with the more general boundary conditions

(1.41) u(x) = 0, x 1 ,

u
(1.42) (x) + (x)u(x) = 0, x 2 .
n
Here, 1 and 2 are disjoint subsets of with 1 2 = . We restrict ourselves in
the following on two-dimensional domains and write (x, y) instead of (x1 , x2 ).
In general it is not possible to solve a problem of the form (1.40)(1.42) exactly (ana-
lytically). Therefore, one has to resort to numerical approximations. Because we cannot
compute with infinitely many variables we have to construct a finite-dimensional eigenvalue
problem that represents the given problem as well as possible, i.e., that yields good approx-
imations for the desired eigenvalues and eigenvectors. Since finite-dimensional eigenvalue
problem only have a finite number of eigenvalues one cannot expect to get good approxi-
mations for all eigenvalues of (1.40)(1.42).
Two methods for the discretization of eigenvalue problems of the form (1.40)(1.42) are
the Finite Difference Method [11, 16, 9] and the Finite Element Method (FEM) [6, 15, 8].
We briefly introduce these methods in the following subsections.

1.6.1 The finite difference method

In this section we just want to mediate some impression what the finite difference method
is about. Therefore we assume for simplicity that the domain is a square with sides of
14 CHAPTER 1. INTRODUCTION
length 1: = (0, 1) (0, 1). We consider the eigenvalue problem

u(x, y) = u(x, y), 0 < x, y < 1

(1.43) u(0, y) = u(1, y) = u(x, 0) = 0, 0 < x, y < 1,
u (x, 1) = 0, 0 < x < 1.
n
This eigenvalue problem occurs in the computation of eigenfrequencies and eigenmodes of
a homogeneous quadratic membrane with three fixed and one free side. It can be solved
analytically by separation of the two spatial variables x and y. The eigenvalues are
 
2 (2l 1)2
k,l = k + 2 , k, l N,
4
and the corresponding eigenfunctions are
2l 1
uk,l (x, y) = sin kx sin y.
2
In the finite difference method one proceeds by defining a rectangular grid with grid
points (xi , yj ), 0 i, j N . The coordinates of the grid points are

(xi , yj ) = (ih, jh), h = 1/N.

By a Taylor expansion one can show that for sufficiently smooth functions u
1
u(x, y) = (4u(x, y) u(x h, y) u(x + h, y) u(x, y h) u(x, y + h))
h2
+ O(h2 ).

It is therefore straightforward to replace the differential equation u(x, y) = u(x, y)

by a difference equation at the interior grid points

(1.44) 4ui,j ui1,j ui+1,j ui,j1 ui,j+1 = h2 ui,j , 0 < i, j < N.

We consider the unknown variables ui,j as approximations of the eigenfunctions at the

grid points (i, j):

(1.45) ui,j u(xi , xj ).

The Dirichlet boundary conditions are replaced by the equations

(1.46) ui,0 = ui,N = u0,i , 0 < i < N.

At the points at the upper boundary of we first take the difference equation (1.44)

(1.47) 4ui,N ui1,N ui+1,N ui,N 1 ui,N +1 = h2 ui,N , 0 i N.

The value ui,N +1 corresponds to a grid point outside of the domain! However the Neumann
boundary conditions suggest to reflect the domain at the upper boundary and to extend the
eigenfunction symmetrically beyond the boundary. This procedure leads to the equation
ui,N +1 = ui,N 1 . Plugging this into (1.47) and multiplying the new equation by the factor
1/2 gives
1 1 1
(1.48) 2ui,N ui1,N ui+1,N ui,N 1 = h2 ui,N , 0 < i < N.
2 2 2
1.6. THE 2D LAPLACE EIGENVALUE PROBLEM 15
In summary, from (1.44) and (1.48), taking into account that (1.46) we get the matrix
equation

4 1 0 1 u1,1
1 4 1 0 1
u1,2
0 1 4 0 0 1 u
1,3
1 0 0 4 1 0 1 u
2,1
1 0 1 4 1 0 1 u
2,2

1 0 1 4 0 0 1 u2,3

1 0 0 4 1 0 1 u3,1

1 0 1 4 1 0 1 u3,2

1 0 1 4 0 0 1 u3,3

1 0 0 2 21 0 u4,1

1 0 12 2 12 u4,2
1
1 0 2 2 u4,3
(1.49)
1 u1,1
1 u
1,2
1 u
1,3
1 u
2,1
1 u
2,2

1 u2,3
= h2 .
1 u3,1

1 u3,2

1 u3,3
1
u4,1
2
1
2
u4,2
1
2 u4,3

For arbitrary N > 1 we define

ui,1
ui,2

ui := .. RN 1 ,
.
ui,N 1

4 1
..
1 4
.
T :=
.. ..
R(N 1)(N 1) ,

. . 1
1 4

1
1

I := .. R(N 1)(N 1) .
.
1
16 CHAPTER 1. INTRODUCTION
In this way we obtain from (1.44), (1.46), (1.48) the discrete eigenvalue problem

T I u1 I u1
.
I T . . . ..
2
. ..

..
.


(1.50) = h
.. ..
. . I u 3 I uN 1
1
I 12 T u4 2I uN

of size N (N 1). This is a matrix eigenvalue problem of the form

(1.51) Ax = M x,
where A and M are symmetric and M additionally is positive definite. If M is the
identity matrix, we call (1.51) a special and otherwise a generalized eigenvalue problem.
In these lecture notes we deal with numerical methods, to solve eigenvalue problems like
these.
In the case (1.50) it is easy to obtain a special (symmetric) eigenvalue problem by a
simple transformation: By left multiplication by

I
I

I
2I
we obtain from (1.50)

T I u1 u1
I T I u2 u2
(1.52) = h2 .
I T 2I u3 u3
1
2I T u 1 u4
2 4 2

A property common to matrices obtained by the finite difference method are its spar-
sity. Sparse matrices have only very few nonzero elements.
In real-world applications domains often cannot be covered easily by a rectangular
grid. In this situation and if boundary conditions are complicated the method of finite
differences can be difficult to implement. Because of this the finite element method is
often the method of choice.
Nevertheless, problems that are posed on rectangular grids can be solved very effi-
ciently. Therefore, tricks are used to deal with irregular boundaries. The solution of
the problem may be extended artificially beyond the boundary, see e.g. [1, 17, 9]. Simi-
lar techiques, so-called immersed boundary conditions are applied at (irregular) interfaces
where, e.g., equations or parameters change [11].

1.6.2 The finite element method (FEM)

Let (, u) R V be an eigenpair of problem (1.40)(1.42). Then
Z
(1.53) (u + u)v dx dy = 0, v V,

where V is vector space of bounded twice differentiable functions that satisfy the boundary
conditions (1.41)(1.42). By partial integration (Greens formula) this becomes
Z Z Z
(1.54) uv dx dy + u v ds = u v dx dy, v V,
2
1.6. THE 2D LAPLACE EIGENVALUE PROBLEM 17
or

(1.55) a(u, v) = (u, v), v V

where Z Z Z
a(u, v) = u v dx dy + u v ds, and (u, v) = u v dx dy.

2

We complete the space V with respect to the Sobolev norm [8, 3]

sZ
(u2 + |u|2 ) dx dy

to become a Hilbert space H [3, 19]. H is the space of quadratic integrable functions with
quadratic integrable first derivatives that satisfy the Dirichlet boundary conditions (1.41)

u(x, y) = 0, (x, y) 1 .

(Functions in H in general do not satisfy the so-called natural boundary conditions (1.42).)
One can show [19] that the eigenvalue problem (1.40)(1.42) is equivalent with the eigen-
value problem

Find (, u) R H such that

(1.56)
a(u, v) = (u, v) v H.

(The essential point is to show that the eigenfunctions of (1.56) are elements of V .)

The RayleighRitzGalerkin method

In the RayleighRitzGalerkin method one proceeds as follows: A set of linearly indepen-
dent functions

(1.57) 1 (x, y), , n (x, y) H,

are chosen. These functions span a subspace S of H. Then, problem (1.56) is solved where
H is replaced by S.

Find (, u) R S such that

(1.58)
a(u, v) = (u, v) v S.

With the Ritz ansatz [15]

n
X
(1.59) u= xi i ,
i=1

equation (1.58) becomes

Find (, x) R Rn such that

(1.60) Pn P
n
xi a(i , v) = xi (i , v), v S.
i=1 i=1
18 CHAPTER 1. INTRODUCTION
Eq. (1.60) must hold for all v S, in particular for v = 1 , , n . But since the
i , 1 i n, form a basis of S, equation (1.60) is equivalent with
n
X n
X
(1.61) xi a(i , j ) = xi (i , j ), 1 j n.
i=1 i=1

This is a matrix eigenvalue problem of the form

(1.62) Ax = M x

where

x1 a11 a1n m11 m1n
.. .. , .. ..
(1.63) x = ... , A= . ..
. . M = . ..
. .
xn an1 ann mn1 mnn

with Z Z
aij = a(i , j ) = i j dx dy + i j ds
2
and Z
mij = (i , j ) = i j dx dy.

The finite element method (FEM) is a special case of the RayleighRitz method.
In the FEM the subspace S and in particular the basis {i } is chosen in a particularly
clever way. For simplicity we assume that the domain is a simply connected domain with
a polygonal boundary, c.f. Fig 1.5. (This means that the boundary is composed entirely
of straight line segments.) This domain is now partitioned into triangular subdomains

Figure 1.5: Triangulation of a domain

T1 , , TN , so-called elements, such that

[
(1.64) Ti Tj = for all i 6= j, and Te = .
e

Finite element spaces for solving (1.40)(1.42) are typically composed of functions that
are continuous in and are polynomials on the individual subdomains Te . Such functions
1.6. THE 2D LAPLACE EIGENVALUE PROBLEM 19
are called piecewise polynomials. Notice that this construction provides a subspace of the
Hilbert space H but not of V , i.e., the functions in the finite element space are not very
smooth and the natural boundary conditions are not satisfied.
An essential issue is the selection of the basis of the finite element space S. If S1 H
is the space of continuous, piecewise linear functions (the restriction to Te is a polynomial
of degree 1) then a function in S1 is uniquely determined by its values at the vertices of the
triangles. Let these nodes, except those on the boundary portion 1 , be numbered from
1 to n, see Fig. 1.6. Let the coordinates of the i-th node be (xi , yi ). Then i (x, y) S1 is
defined by
17 20 24 27
16 29
15 19 23 26
13 28
11 14 21
10
9 22 25
7 18
6
12
4
3 8

1 5
2

Figure 1.6: Numbering of nodes on (piecewise linear polynomials)


1 i=j
(1.65) i ((xj , yj )) := ij =
0 i 6= j
A typical basis function i is sketched in Figure 1.7.

Figure 1.7: A piecewise linear basis function (or hat function)

Another often used finite element element space is S2 H, the space of continuous,
piecewise quadratic polynomials. These functions are (or can be) uniquely determined by
their values at the vertices and edge midpoints of the triangle. The basis functions are
defined according to (1.65). There are two kinds of basis functions i now, first those
that are 1 at a vertex and second those that are 1 at an edge midpoint, cf. Fig. 1.8. One
immediately sees that for most i 6= j
(1.66) a(i , j ) = 0, (i , j ) = 0.
20 CHAPTER 1. INTRODUCTION

Figure 1.8: The piecewise quadratic basis functions corresponding to the edge midpoints [5]

Therefore the matrices A and M in (1.62)(1.63) will be sparse. The matrix M is positive
definite as
N
X N
X N
X
T
(1.67) x Mx = xi xj mij = xi xj (i , j ) = (u, u) > 0, u= xi i 6= 0,
i,j=1 i,j=1 i=1

p
because the i are linearly independent and because ||u|| = (u, u) is a norm. Similarly
it is shown that
xT Ax 0.

It is possible to have xT Ax = 0 for a nonzero vector x. This is the case if the constant
u
function u = 1 is contained in S. This happens if Neumann boundary conditions n =0
are posed on the whole boundary . Then,
X
u(x, y) = 1 = i (x, y),
i

i.e., we have xT Ax = 0 for x = [1, 1, . . . , 1].

1.6.3 A numerical example

We want to determine the acoustic eigenfrequencies and corresponding modes in the inte-
rior of a car. This is of interest in the manufacturing of cars, since an appropriate shape of
the form of the interior can suppress the often unpleasant droning of the motor. The prob-
lem is three-dimensional, but by separation of variables the problem can be reduced to two
dimensions. If rigid, acoustically hard walls are assumed, the mathematical model of the
problem is again the Laplace eigenvalue problem (1.24) together with Neumann boundary
conditions. The domain is given in Fig. 1.9 where three finite element triangulations are
shown with 87 (grid1 ), 298 (grid2 ), and 1095 (grid3 ) vertices (nodes), respectively. The
results obtained with piecewise linear polynomials are listed in Table 1.2. From the results
we notice the quadratic convergence rate. The smallest eigenvalue is always zero. The
corresponding eigenfunction is the constant function. This function can be represented
exactly by the finite element spaces, whence its value is correct (up to rounding error).
The fourth eigenfunction of the acoustic vibration problem is displayed in Fig. 1.10.
The physical meaning of the function value is the difference of the pressure at a given
location to the normal pressure. Large amplitudes thus means that the corresponding
noise is very much noticable.
1.7. CAVITY RESONANCES IN PARTICLE ACCELERATORS 21
16 16

14 14

12 12

10 10

8 8

6 6

4 4

2 2

0 0

2 2

0 5 10 15 20 25 0 5 10 15 20 25

16

14

12

10

0 5 10 15 20 25

Figure 1.9: Three meshes for the car length cut

1.7 Cavity resonances in particle accelerators

The Maxwell equations in vacuum are given by

B
curl E(x, t) = (x, t), (Faradays law)
t
D
curl H(x, t) = (x, t) + j(x, t), (MaxwellAmp`ere law)
t
div D(x, t) = (x, t), (Gausss law)
div B(x, t) = 0. (Gausss law magnetic)

where E is the electric field intensity, D is the electric flux density, H is the magnetic
field intensity, B is the magnetic flux density, j is the electric current density, and is the
electric charge density. Often the optical problem is analyzed, i.e. the situation when
the cavity is not driven (cold mode), hence j and are assumed to vanish.
Again by separating variables, i.e. assuming a time harmonic behavior of the fields,
e.g.,
E(x, t) = e(x)eit ,
and by using the constitutive relations

D = E, B = H, j = E,
22 CHAPTER 1. INTRODUCTION

Finite element method

k k (grid1 ) k (grid2 ) k (grid3 )
1 0.0000 -0.0000 0.0000
2 0.0133 0.0129 0.0127
3 0.0471 0.0451 0.0444
4 0.0603 0.0576 0.0566
5 0.1229 0.1182 0.1166
6 0.1482 0.1402 0.1376
7 0.1569 0.1462 0.1427
8 0.2162 0.2044 0.2010
9 0.2984 0.2787 0.2726
10 0.3255 0.2998 0.2927

Table 1.2: Numerical solutions of acoustic vibration problem

0.05

0.05

0.1

Figure 1.10: Fourth eigenmode of the acoustic vibration problem

1.8. SPECTRAL CLUSTERING 23
one obtains after elimination of the magnetic field intensity the so called time-harmonic
Maxwell equations

curl 1 curl e(x) = e(x), x ,

(1.68) div e(x) = 0, x ,
n e = 0, x .

Here, additionally, the cavity boundary is assumed to be perfectly electrically conduct-

ing, i.e. E(x, t) n(x) = 0 for x .
The eigenvalue problem (1.68) is a constrained eigenvalue problem. Only functions
are taken into account that are divergence-free. This constraint is enforced by Lagrange
multipliers. A weak formulation of the problem is then

Find (, e, p) R H0 (curl; ) H01 () such that e 6= 0 and

(a) (1 curl e, curl ) + (grad p, ) = ( e, ), H0 (curl; ),
(b) (e, grad q) = 0, q H01 ().

With the correct finite element discretization this problem turns in a matrix eigenvalue
problem of the form      
A C x M O x
= .
CT O y O O y
The solution of this matrix eigenvalue problem correspond to vibrating electric fields. A
possible shape of domain is given in Figure 1.11.

Figure 1.11: Comet cavity of Paul Scherrer Institute

1.8 Spectral clustering3

The goal of clustering is to group a given set of data points x1 , . . . , xn into k clusters such
that members from the same cluster are (in some sense) close to each other and members
from different clusters are (in some sense) well separated from each other.
3
This section is based on a tutorial by von Luxburg [12]. Thanks to Daniel Kressner for compiling it!
24 CHAPTER 1. INTRODUCTION
A popular approach to clustering is based on similarity graphs. For this purpose, we
need to assume some notion of similarity s(xi , xj ) 0 between pairs of data points xi
and xj . An undirected graph G = (V, E) is constructed such that its vertices correspond
to the data points: V = {x1 , . . . , xn }. Two vertices xi , xj are connected by an edge if
the similarity sij between xi and xj is sufficiently large. Moreover, a weight wij > 0 is
assigned to the edge, depending on sij . If two vertices are not connected we set wij = 0.
The weights are collected into a weighted adjacency matrix

n
W = wij i,j=1 .

There are several possibilities to define the weights of the similarity graph associated
with a set of data points and a similarity function:
fully connected graph All points with positive similarity are connected with each other
and we simply set wij = s(xi , xj ). Usually, this will only result in reasonable clusters
if the similarity function models locality very well. One example of such a similarity
kx x k2

function is the Gaussian s(xi , xj ) = exp i22j , where kxi xj k is some
distance measure (e.g., Euclidean distance) and is some parameter controlling
how strongly locality is enforced.

k-nearest neighbors Two vertices xi , xj are connected if xi is among the k-nearest

neighbors of xj or if xj is among the k-nearest neighbors of xi (in the sense of some
distance measure). The weight of the edge between connected vertices xi , xj is set
to the similarity function s(xi , xj ).

-neighbors Two vertices xi , xj are connected if their pairwise distance is smaller than
for some parameter > 0. In this case, the weights are usually chosen uniformly,
e.g., wij = 1 if xi , xj are connected and wij = 0 otherwise.
Assuming that the similarity function is symmetric (s(xi , xj ) = s(xj , xi ) for all xi , xj ) all
definitions above give rise to a symmetric weight matrix W . In practice, the choice of the
most appropriate definition depends as usual on the application.

1.8.1 The graph Laplacian

In the following we construct the so called graph Laplacian, whose spectral decomposition
will later be used to determine clusters. For simplicity, we assume the weight matrix W
to be symmetric. The degree of a vertex xi is defined as
n
X
(1.69) di = wij .
j=1

In the case of an unweighted graph, the degree di amounts to the number of vertices
adjacent to vi (counting also vi if wii = 1). The degree matrix is defined as

D = diag(d1 , d2 , . . . , dn ).

The graph Laplacian is then defined as

(1.70) L = D W.

By (1.69), the row sums of L are zero. In other words, Le = 0 with e the vector of all
ones. This implies that 0 is an eigenvalue of L with the associated eigenvector e. Since L
1.8. SPECTRAL CLUSTERING 25
is symmetric all its eigenvalues are real and one can show that 0 is the smallest eigenvalue;
hence L is positive semidefinite. It may easily happen that more than one eigenvalue is
zero. For example, if the set of vertices can be divided into two subsets {x1 , . . . , xk },
{xk+1 , . . . , xn }, and vertices from one subset are not connected with vertices from the
other subset, then  
L1 0
L= ,
0 L2
where L1 , L2 are the Laplacians of the two disconnected components. Thus L has two
eigenvectors (0e) and (0e) with eigenvalue 0. Of course, any linear combination of these two
linearly independent eigenvectors is also an eigenvector of L.
The observation above leads to the basic idea behind spectral graph partitioning: If
the vertices of the graph decompose into k connected components V1 , . . . , Vk there are k
zero eigenvalues and the associated invariant subspace is spanned by the vectors

(1.71) V1 , V2 , . . . , Vk ,

where Vj is the indicator vector having a 1 at entry i if xi Vj and 0 otherwise.

1.8.2 Spectral clustering

On a first sight, it may seem that (1.71) solves the graph clustering problem. One simply
computes the eigenvectors belonging to the k zero eigenvalues of the graph Laplacian
and the zero structure (1.71) of the eigenvectors can be used to determine the vertices
belonging to each component. Each component gives rise to a cluster.
This tempting idea has two flaws. First, one cannot expect the eigenvectors to have
the structure (1.71). Any computational method will yield an arbitrary eigenbasis, e.g.,
arbitrary linear combinations of V1 , V2 , . . . , Vk . In general, the method will compute
an orthonormal basis U with


(1.72) U = v1 , . . . , vk Q,

where Q is an arbitrary orthogonal k k matrix and vj = Vj /|Vj | with the cardinality

|Vj | of Vj . Second and more importantly, the goal of graph clustering is not to detect
connected components of a graph4 . Requiring the components to be completely discon-
nected from each other is too strong and will usually not lead to a meaningful clustering.
For example, when using a fully connected similarity graph only one eigenvalue will be
zero and the corresponding eigenvector e yields one component, which is the graph itself!
Hence, instead of computing an eigenbasis belonging to zero eigenvalues, one determines
an eigenbasis belonging to the k smallest eigenvalues.

Example 1.1 200 real numbers are generated by superimposing samples from 4 Gaussian
distributions with 4 different means:
m = 50; randn(state,0);
x = [2+randn(m,1)/4;4+randn(m,1)/4;6+randn(m,1)/4;8+randn(m,1)/4];
The following two figures show the histogram of the distribution of the entries of x and the
eigenvalues of the graph Laplacian for the fully connected similarity graph with similarity
|x x |2

function s(xi , xj ) = exp i 2 j :
4
There are more efficient algorithms for finding connected components, e.g., breadth-first and depth-first
search.
26 CHAPTER 1. INTRODUCTION

8 50

40
6
30

4 20

10
2
0

0 10
2 4 6 8 2 4 6 8 10

As expected, one eigenvalue is (almost) exactly zero. Additionally, the four smallest
eigenvalues have a clearly visible gap to the other eigenvalues. The following four figures
show the entries of the 4 eigenvectors belonging to the 4 smallest eigenvalues of L:
0.0707 0.15 0.1 0.15

0.1 0.1
0.0707 0.05

0.05 0.05
0.0707 0
0 0

0.0707 0.05
0.05 0.05

0.0707 0.1 0.1 0.1

0 50 100 150 200 0 50 100 150 200 0 50 100 150 200 0 50 100 150 200

On the one hand, it is clearly visible that the eigenvectors are well approximated by linear
combinations of indicator vectors. On the other hand, none of the eigenvectors is close to
an indicator vector itself and hence no immediate conclusion on the clusters is possible.

To solve the issue that the eigenbasis (1.72) may be transformed by an arbitrary
orthogonal matrix, we transpose the basis and consider the row vectors of U :


U T = u1 , u2 , . . . , un , ui Rk .
If U contained indicator vectors then each of the short vectors ui would be a unit vector
ej for some 1 j k (possibly divided by |Vj |). In particular, the ui would separate very
well into k different clusters. The latter property does not change if the vectors ui undergo
an orthogonal transformation QT . Hence, applying a clustering algorithm to u1 , . . . , un
allows us to detect the membership of ui independent of the orthogonal transformation.
The key point is that the short vectors u1 , . . . , un are much better separated than the
original data x1 , . . . , xn . Hence, a much simpler algorithm can be used for clustering. One
of the most basic algorithms is k-means clustering. Initially, this algorithm assigns each
ui randomly5 to a cluster with 1 k and then iteratively proceeds as follows:
1. Compute cluster centers c as cluster means:
X . X
c = ui 1.
i in cluster i in cluster

2. Assign each ui to the cluster with the nearest cluster center.

3. Goto Step 1.
The algorithm is stopped when the assigned clusters do not change in an iteration.
5
For unlucky choices of random assignments the k-means algorithm may end up with less than k clusters.
A simple albeit dissatisfying solution is to restart k-means with a different random assignment.
1.8. SPECTRAL CLUSTERING 27
Example 1.1 (contd). The k-means algorithm applied to the eigenbasis from Example
1.1 converges after 2 iterations and results in the following clustering:

3.5

3
Cluster
2.5

1.5

1
2 4 6 8
Data points

1.8.3 Normalized graph Laplacians

It is sometimes advantageous to use a normalized Laplacian

(1.73) D 1 L = I D 1 W

instead of the standard Laplacians. Equivalently, this means that we compute the eigen-
vectors belonging to the smallest eigenvalues of the generalized eigenvalue problem W x =
Dx. Alternatively, one may also compute the eigenvalues from the symmetric matrix
D 1/2 W D 1/2 but the eigenvectors need to be adjusted to compensate this transforma-
tion.

Example 1.1 (contd). The eigenvalues of the normalized Laplacian for Example 1.1
are shown below:

0.8

0.6

0.4

0.2

0.2
2 4 6 8 10

In comparison to the eigenvalues of the standard Laplacian, the four smallest eigenvalues
of the are better separated from the rest. Otherwise, the shape of the eigenvectors is
similar and the resulting clustering is identical with the one obtained with the standard
Laplacian.
28 CHAPTER 1. INTRODUCTION

Figure 1.12: Things that can go wrong with the basic model: left is a dangling node, right
a terminal strong component featuring a cyclic path. Figures are from [2]

1.9 Googles PageRank6

One of the reasons why Google is such an effective search engine is the PageRank that
determines the importance of a web page [2, 10, 13]. The PageRank is determined entirely
by the link structure of the World Wide Web. For any particular query, Google finds
the pages on the Web that match that query and lists those pages in the order of their
PageRank. Lets imagine a surfer brachiate through pages of the world wide web randomly
choosing an outgoing link from one page to get to the next. This can lead to dead ends at
pages with no outgoing links, or cycles around cliques of interconnected pages. So, every
once in a while, simply choose a random page from the Web. This theoretical random
walk is known as a Markov chain or Markov process. The limiting probability that an
infinitely dedicated random surfer visits any particular page is its PageRank. A page has
high rank if other pages with high rank link to it.
Let W be the set of (reachable) web pages and let n = |W |. On WorldWideWebSize.com7
it is estimated that Googles index contains around to 49.5 billion pages.
The elements of the connectivity matrix G Rnn is defined by
(
1 there is a hyperlink j 7 i,
gij =
0 otherwise.

Clearly, this is an extremely sparse matrix. The number of its nonzero elements nnz(G)
equals the number of hyperlinks in W . Let ri and cj be the row and column sums of G,
X X
ri = gij , cj = gij .
j i

Then rj is called the in-degree and cj is called the out-degree of the jth page. cj = 0
means a dead end.
In Fig. 1.13 we see the example of a tiny web with just n = 6 nodes. The nodes , ,
, , , correspond to labels 1 to 6 in the matrix notation, in this sequence.
6
Here we closely follow Section 2.11 in Molers Matlab introduction [13].
7
http://www.worldwidewebsize.com/ accessed on Feb. 18, 2016.
1.9. GOOGLES PAGERANK 29

Figure 1.13: A small web with 6 nodes.

Then the connectivity matrix for the small web is given by

0 0 0 1 0 1
1 0 0 0 0 0

0 1 0 0 0 0
G=
0
.
1 1 0 0 0
0 0 1 0 0 0
1 0 1 0 0 0

Notice the zero 5th column of G. This column corresponds to the dead end at the dangling
node .
Let A be the matrix with elements
(
gij /cj if cj 6= 0
aij =
1/n if cj = 0 (dead end).

In the small web example above,

1
0 0 0 1 6 1
1 0 0 0 1
0
2 6
0 1
0 0 1
0
A=
0
2 6 .

1
2
1
3 0
1
6 0
0 0 1 1
0
3 0 6
1 1 1
2 0 3 0 6 0

The entries in As column j indicate the probabilities of jumping from the jth page to the
other pages on the web. Column 3, e.g., tells that starting from node 3 (= ) nodes , ,
are chosen with equal probability 1/3. Note that we choose any page of the web with
equal probability when we land at a dead end.
30 CHAPTER 1. INTRODUCTION
To not be stuck to much in parts of the web, we follow the links only with probability .
With probability 1 we choose a random page. Therefore, we replace A by the matrix

A = A + (1 )peT ,

where p is a personalization vector and e = (1, 1, . . . , 1)T . (p has nonnegative elements

that sum to 1, kpk1 = 1.) Note that p may have zero entries indicating, e.g., uncongenial,
discredited, or discriminated web pages. We assume an innocent web and set p = e/n.
Since n 5 1010 in the real WWW, a typical entry of p is about 2 1011 .
Note that
eT A = eT .
So, 1 (AT ) = (A), i.e., 1 is an eigenvalue of A with left eigenvector e. Since the
matrix norm
Xn
kAk1 = max |aij | = 1,
1jn
i=1

A cannot have an eigenvalue larger than 1 in modulus. The PerronFrobenius theorem for
matrices with nonnegative entries states that such matrices have a simple real eigenvalue
of largest modulus [4]. Therefore, the eigenvalue 1 is in fact the largest eigenvalue of A.
We are not interested in the left eigenvector e but in the right eigenvector x,

x = Ax.

The PerronFrobenius theory confirms that x can be chosen such that all its entries are
nonnegative. If x is scaled such that
n
X
xi = 1
i=1

then x is the state vector of the Markov chain and is Googles PageRank.
The computation of the PageRank amounts to determining the largest eigenvalue and
corresponding eigenvector of a matrix. It can be determined by vector iteration. The
computation gets easier the smaller the damping factor is chosen. However, small
means small weight is given to the structure of the web. In [13] a Matlab routine
pagerankpow.m is provided to compute the PageRank exploiting the sparsity structure of
G.

1.10 Other sources of eigenvalue problems

The selection of applications above may lead to the impression that eigenvalue problems
in practice virtually always require the computation of the smallest eigenvalues of a sym-
metric matrix. This is not the case. For example, a linear stability analysis requires the
computation of all eigenvalues on or close to the imaginary axis of a nonsymmetric matrix.
Computational methods for decoupling the stable/unstable parts of a dynamical system
require the computation of all eigenvalues in the left and/or right half of the complex
plane. The principal component analysis (PCA), which plays an important role in a large
variety of applications, requires the computation of the largest eigenvalues (or rather sin-
gular values). As we will see in the following chapters, the region of eigenvalues we are
interested in determines the difficulty of the eigenvalue problem to a large extent (along
BIBLIOGRAPHY 31
with the matrix order and structure). It should also guide the choice of algorithm for
solving an eigenvalue problem.
Saad [14] discusses further interesting sources of eigenvalue problems like electronic
structure calculations, the stability of dynamical systems, or Markov chain models similar
as Googles PageRank.

Bibliography
[1] A. Adelmann, P. Arbenz, and Y. Ineichen, A fast parallel Poisson solver on ir-
regular domains applied to beam dynamics simulations, J. Comput. Phys., 229 (2010),
pp. 45544566.

[2] U. M. Ascher and C. Greif, A First Course in Numerical Methods, SIAM,

Philadelphia, PA, 2011.

[3] O. Axelsson and V. Barker, Finite Element Solution of Boundary Value Prob-
lems, Academic Press, Orlando, FL, 1984.

[4] A. Berman and R. J.Plemmons, Nonnegative Matrices in the Mathematical Sci-

ences, Academic Press, New York, 1979. (Republished by SIAM, Philadelphia, 1994.).

[5] O. Chinellato, The complex-symmetric JacobiDavidson algorithm and its

application to the computation of some resonance frequencies of anisotropic
lossy axisymmetric cavities, PhD Thesis No. 16243, ETH Z urich, 2005.
(doi:10.3929/ethz-a-005067691).

[6] P. G. Ciarlet, The Finite Element Method for Elliptic Problems, North-Holland,
Amsterdam, 1978. (Studies in Mathematics and its Applications, 4).

[7] R. Courant and D. Hilbert, Methoden der Mathematischen Physik, Springer,

Berlin, 1968.

[8] H. C. Elman, D. J. Silvester, and A. J. Wathen, Finite Elements and Fast

Iterative Solvers, Oxford University Press, Oxford, 2005.

[9] G. E. Forsythe and W. R. Wasow, Finite-difference methods for partial differ-

ential equations, Wiley, New York, 1960.

[10] A. N. Langville and C. D. Meyer, Googles pagerank and beyond: the science of
search engine rankings, Princeton University Press, Princeton, N.J., 2006.

[11] R. J. LeVeque, Finite Difference Methods for Ordinary and Partial Differential
Equations, SIAM, Philadelphia, PA, 2007.

[12] U. von Luxburg, A tutorial on spectral clustering, Stat. Comput., 17 (2007),

pp. 395416.

[13] C. B. Moler, Numerical Computing with Matlab, SIAM, Philadelphia, PA, 2004.

[14] Y. Saad, Numerical Methods for Large Eigenvalue Problems, SIAM, Philadelphia,
PA, 2011.

[15] H. R. Schwarz, Methode der finiten Elemente, Teubner, Stuttgart, 3rd ed., 1991.
32 CHAPTER 1. INTRODUCTION
[16] , Numerische Mathematik, Teubner, Stuttgart, 3rd ed. ed., 1993.

[17] G. H. Shortley and R. Weller, The numerical solution of Laplaces equation, J.

Appl. Phys., 9 (1939), pp. 334344.

[18] G. Strang, Introduction to Applied Mathematics, Wellesley-Cambridge Press,

Wellesley, 1986.

[19] H. F. Weinberger, Variational Methods for Eigenvalue Approximation, Regional

Conference Series in Applied Mathematics 15, SIAM, Philadelphia, PA, 1974.
Chapter 2

Basics

2.1 Notation
The fields of real and complex numbers are denoted by R and C, respectively. Elements
in R and C, scalars, are denoted by lowercase letters, a, b, c, . . ., and , , , . . .
Vectors are denoted by boldface lowercase letters, a, b, c, . . ., and , , , . . . We
denote the space of vectors of n real components by Rn and the space of vectors of n
complex components by Cn .

x1
x2

(2.1) x Rn x = . , xi R.
..
xn

We often make statements that hold for real or complex vectors or matrices. Then we
write, e.g., x Fn .
The inner product of two n-vectors in C is defined as
n
X
(2.2) (x, y) = xi yi = y x,
i=1

that is, we require linearity in the first component and anti-linearity in the second.
y = (y1 , y2 , . . . , yn ) denotes conjugate transposition of complex vectors. To simplify
notation we denote real transposition by an asterisk as well.
Two vectors x and y are called orthogonal, x y, if x y = 0.
The inner product (2.2) induces a norm in F,

n
!1/2
p X
(2.3) kxk = (x, x) = |xi |2 .
i=1

This norm is often called Euclidean norm or 2-norm.

The set of m-by-n matrices with components in the field F is denoted by Fmn ,

a11 a12 . . . a1n
a21 a22 . . . a2n

(2.4) A Fmn A = . .. .. , aij F.
.. . .
am1 am2 . . . amn

33
34 CHAPTER 2. BASICS
The matrix A Fnm ,

a
11 a
21 . . . a
m1
a
a m2
22 . . . a
12
(2.5) A = . .. ..
.. . .
a
1n a
2n . . . a
nm

is the Hermitian transpose of A. Notice, that with this notation n-vectors can be
identified with n-by-1 matrices.
The following classes of square matrices are of particular importance:

A Fnn is called Hermitian if and only if A = A.

A real Hermitian matrix is called symmetric.

U Fnn is called unitary if and only if U 1 = U .

Real unitary matrices are called orthogonal.

A Fnn is called normal if A A = AA . Both, Hermitian and unitary matrices

are normal.

We define the norm of a matrix to be the norm induced by the vector norm (2.3),

kAxk
(2.6) kAk := max = max kAxk.
x6=0 kxk kxk=1

The condition number of a nonsingular matrix is defined as (A) = kAkkA1 k. It is easy

to show that if U is unitary then kU xk = kxk for all x. Thus the condition number of a
unitary matrix is 1.

2.2 Statement of the problem

The (standard) eigenvalue problem is as follows.

Given a square matrix A Fnn .

Find scalars C and vectors x Cn , x 6= 0, such that

(2.7) Ax = x,

i.e., such that

(2.8) (A I)x = 0

has a nontrivial (nonzero) solution.

So, we are looking for numbers such that A I is singular.

Definition 2.1 Let the pair (, x) be a solution of (2.7) or (2.8), respectively. Then

is called an eigenvalue of A,

x is called an eigenvector corresponding to

2.2. STATEMENT OF THE PROBLEM 35
(, x) is called eigenpair of A.
The set (A) of all eigenvalues of A is called spectrum of A.
The set of all eigenvectors corresponding to an eigenvalue together with the vector
0 form a linear subspace of Cn called the eigenspace of . As the eigenspace of
is the null space of I A we denote it by N (I A).
The dimension of N (I A) is called geometric multiplicity g() of .
An eigenvalue is a zero of the characteristic polynomial
() := det(I A) = n + an1 n1 + + a0 .
The multiplicity of as a zero of is called the algebraic multiplicity m() of .
We will later see that
1 g() m() n, (A), A Fnn .

Remark 2.1. A nontrivial solution solution y of

(2.9) y A = y
is called left eigenvector corresponding to . A left eigenvector of A is a right eigenvector
A y = y.
of A , corresponding to the eigenvalue ,

Problem 2.2 Let x be a (right) eigenvector of A corresponding to an eigenvalue and

let y be a left eigenvector of A corresponding to a different eigenvalue 6= . Show that
x y = 0.
Remark 2.2. Let A be an upper triangular matrix,

a11 a12 . . . a1n
a22 . . . a2n

(2.10) A= .. .. , aik = 0 for i > k.
. .
ann
Then we have
n
Y
det(I A) = ( aii ).
i=1

Problem 2.3 Let = aii , 1 i n, be an eigenvalue of A in (2.10). Can you give a

corresponding eigenvector? Can you detect a situation where g() < m()?
The (generalized) eigenvalue problem is as follows.
Given two square matrices A, B Fnn .
Find scalars C and vectors x C, x 6= 0, such that

(2.11) Ax = Bx,

or, equivalently, such that

(2.12) (A B)x = 0

has a nontrivial solution.

36 CHAPTER 2. BASICS
Definition 2.4 Let the pair (, x) be a solution of (2.11) or (2.12), respectively. Then
is called an eigenvalue of A relative to B,
x is called an eigenvector of A relative to B corresponding to .
(, x) is called an eigenpair of A relative to B,
The set (A; B) of all eigenvalues of (2.11) is called the spectrum of A relative
to B.
Let us look at some examples.
Let B be nonsingular. Then
(2.13) Ax = Bx B 1 Ax = x

Let both A and B be Hermitian, A = A and B = B . Let further be B positive

definite and B = LL be its Cholesky factorization. Then
(2.14) Ax = Bx L1 AL y = y, y = L x.

Let A be invertible. Then Ax = 0 implies x = 0. That is, 0 6 (A; B). Therefore,

1
(2.15) Ax = Bx x = A1 Bx, =

Let A = B Rnn be invertible. Then
Ax = Bx B 1 Ax = Ix = x.
Therefore, (A; B) = {1}. The associated eigenspace is Rn . Every nonzero vector x
is an eigenvector.
Difficult situation: both A and B are singular.
1. Let, e.g.,    
1 0 0 0
A= , B= .
0 0 0 1
Then,
Ae2 = 0 = 0 Be2 = 0 e2 ,
such that 0 is an eigenvalue of A relative to B. Since
Ae1 = e1 = Be1 = 0
e1 cannot be an eigenvector of A relative to B.
As in (2.15) we may swap the roles of A and B. Then
Be1 = 0 = Ae1 = 0e1 .
So, = 0 is an eigenvalue of B relative to A. We therefore say, informally, that
= is an eigenvalue of A relative to B. So, (A; B) = {0, }.
2. Let    
1 0 1 0
A= , B= = A.
0 0 0 0
Then,
Ae1 = 1 Be1 ,
Ae2 = 0 = Be2 = 0, for all C.
Therefore, in this case, (A; B) = C.
2.3. SIMILARITY TRANSFORMATIONS 37
2.3 Similarity transformations
Definition 2.5 A matrix A Fnn is similar to a matrix C Fnn , A C, if and only
if there is a nonsingular matrix S such that

(2.16) S 1 AS = C.

The mapping A S 1 AS is called a similarity transformation.

Theorem 2.6 Similar matrices have equal eigenvalues with equal multiplicities. If (, x)
is an eigenpair of A and C = S 1 AS then (, S 1 x) is an eigenpair of C.

Proof. Ax = x and C = S 1 AS imply that

CS 1 x = S 1 ASS 1 x = S 1 x.

Hence, A and C have equal eigenvalues and their geometric multiplicity is not changed by
the similarity transformation. From

det(I C) = det(S 1 S S 1 AS)

= det(S 1 (I A)S) = det(S 1 ) det(I A) det(S) = det(I A)

it follows that the characteristic polynomials of A and C are equal and hence also the
algebraic eigenvalue multiplicities are equal.
Similarity transformations are used to transform matrices into similar matrices from
which eigenvalues can be easily read. Diagonal matrices are the preferred matrix structure.
However, not all matrices are diagonalizable. There is, e.g., no invertible matrix S that
diagonalizes the matrix  
1 1
.
0 1
In the Jordan normal form introduced in section 2.8 the transformation is into a bidiagonal
matrix. In the Schur normal form, see section 2.4 the transformation is into an upper
tridiagonal matrix, but with an unitary S.

Definition 2.7 Two matrices A and B are called unitarily similar if S in (2.16) is
unitary. If the matrices are real the term orthogonally similar is used.

Unitary similarity transformations are very important in numerical computations. Let

U be unitary. Then kU k = kU 1 k = 1, the condition number of U is therefore (U ) = 1.
Hence, if C = U 1 AU then C = U AU and kCk = kAk. In particular, if A is disturbed
by A (e.g., roundoff errors introduced when storing the entries of A in finite-precision
arithmetic) then
U (A + A)U = C + C, kCk = kAk.
Hence, errors (perturbations) in A are not amplified by a unitary similarity transformation.
This is in contrast to arbitrary similarity transformations. However, as we will see later,
small eigenvalues may still suffer from large relative errors.
Another reason for the importance of unitary similarity transformations is the preser-
vation of symmetry: If A is symmetric then U 1 AU = U AU is symmetric as well.
For generalized eigenvalue problems, similarity transformations are not so crucial since
we can operate with different matrices from both sides. If S and T are nonsingular

Ax = Bx T AS 1 Sx = T BS 1 Sx.
38 CHAPTER 2. BASICS
This sometimes called equivalence transformation of A, B. Thus, (A; B) = (T AS 1 ; T BS 1 ).
Let us consider a special case: let B be invertible and let B = LU be the LU-factorization
of B. Then we set S = U and T = L1 and obtain T BU 1 = L1 LU U 1 = I. Thus,
(A; B) = (L1 AU 1 ; I) = (L1 AU 1 ).

2.4 Schur decomposition

Theorem 2.8 (Schur decomposition) If A Cnn then there is a unitary matrix
U Cnn such that

(2.17) U AU = T

is upper triangular. The diagonal elements of T are the eigenvalues of A.

Proof. The proof is by induction. For n = 1, the theorem is obviously true.

Assume that the theorem holds for matrices of order n 1. Let (, x), kxk = 1, be
an eigenpair of A, Ax = x. We construct a unitary matrix U1 with first column x (e.g.
the Householder reflector U1 with U1 x = e1 ). Partition U1 = [x, U ]. Then
   
x Ax x AU
U1 AU1 = =
U Ax U AU 0 A

as Ax = x and U x = 0 by construction of U1 . By assumption, there exists a unitary
matrix U C(n1)(n1) such that U AU
= T is upper triangular. Setting U := U1 (1 U
),
we obtain (2.17).
Notice , that this proof is not constructive as we assume the knowledge of an eigenpair
(, x). So, we cannot employ it to actually compute the Schur form. The QR algorithm
is used for this purpose. We will discuss this basic algorithm in Chapter 4.
Let U AU = T be a Schur decomposition of A with U = [u1 , u2 , . . . , un ]. The Schur
decomposition can be written as AU = U T . The k-th column of this equation is
k1
X
(2.18) Auk = uk + tik ui , k = tkk .
i=1

This implies that

(2.19) Auk span{u1 , . . . , uk }, k.

Thus, the first k Schur vectors u1 , . . . , uk form an invariant subspace1 for A. From
(2.18) it is clear that the first Schur vector is an eigenvector of A. The other columns of U ,
however, are in general not eigenvectors of A. Notice, that the Schur decomposition is not
unique. In the proof we have chosen any eigenvalue . This indicates that the eigenvalues
can be arranged in any order in the diagonal of T . This also indicates that the order with
which the eigenvalues appear on T s diagonal can be manipulated.

Problem 2.9 Let  

1
A= .
0 2
1
A subspace V Fn is called invariant for A if AV V.
2.5. THE REAL SCHUR DECOMPOSITION 39
Find an orthogonal 2 2 matrix Q such that
 
2
Q AQ = .
0 1
Hint: the first column of Q must be a normalized eigenvector of A corresponding to
eigenvalue 2 . Why?

2.5 The real Schur decomposition

Real matrices can have complex eigenvalues. If complex eigenvalues exist, then they occur

in complex conjugate pairs! That is, if is an eigenvalue of the real matrix A, then also
is an eigenvalue of A. The following theorem indicates that complex computation can be
avoided.

Theorem 2.10 (Real Schur decomposition) If A Rnn then there is an orthogonal

matrix Q Rnn such that

R11 R12 R1m
R22 R2m

(2.20) QT AQ = .. ..
. .
Rmm
is upper quasi-triangular. The diagonal blocks Rii are either 1 1 or 2 2 matrices. A
1 1 block corresponds to a real eigenvalue, a 2 2 block corresponds to a pair of complex
conjugate eigenvalues.

Remark 2.3. The matrix  

, , R,

has the eigenvalues + i and i.
Proof. Let = + i, 6= 0, be an eigenvalue of A with eigenvector x = u + iv. Then
= i is an eigenvalue corresponding to x
= u iv. To see this we first observe that
Ax = A(u + iv) = Au + iAv,
x = ( + i)(u + iv) = (u v) + i(u + v).
Thus,
x = A(u iv) = Au iAv,
A
= (u v) i(u + v)
x.
= ( i)u i( i)v = ( i)(u iv) =
Now, the actual proof starts. Let k be the number of complex conjugate pairs. We prove
the theorem by induction on k.
First we consider the case k = 0. In this case A has real eigenvalues and eigenvectors.
It is clear that we can repeat the proof of the Schur decomposition of Theorem 2.8 in real
arithmetic to get the decomposition (2.17) with U Rnn and T Rnn . So, there are n
diagonal blocks Rjj in (2.20) all of which are 1 1.
Let us now assume that the theorem is true for all matrices with fewer than k complex
conjugate pairs. Then, with = + i, 6= 0 and x = u + iv, as previously, we have
 

A[u, v] = [u, v] .

40 CHAPTER 2. BASICS
Let {x1 , x2 } be an orthonormal basis of span([u, v]). Then, since u and v are linearly
independent2 , there is a nonsingular 2 2 real square matrix C with
[x1 , x2 ] = [u, v]C.
Now, 

A[x1 , x2 ] = A[u, v]C = A[u, v] C

 
1
= [x1 , x2 ]C C =: [x1 , x2 ]S.

 

S and are similar and therefore have equal eigenvalues. Now we construct an

orthogonal matrix [x1 , x2 , x3 , . . . , xn ] =: [x1 , x2 , W ]. Then
T
x1   
 T    S [x1 , x2 ]T AW
[x1 , x2 ], W A [x1 , x2 ], W = x2 T [x1 , x2 ]S, AW = .
O W T AW
WT
The matrix W T AW has less than k complex-conjugate eigenvalue pairs. Therefore, by
the induction assumption, there is an orthogonal Q2 R(n2)(n2) such that the matrix
QT2 (W T AW )Q2
is quasi-triangular. Thus, the orthogonal matrix
 
I2 O
Q = [x1 , x2 , x3 , . . . , xn ]
O Q2
transforms A similarly to quasi-triangular form.

2.6 Normal matrices

Definition 2.11 A matrix A Fnn is called normal if
(2.21) AA = A A.
Let A = U RU be the Schur decomposition of A. Then,
RR = U AU U A U = U AA U = U A AU = U A U U AU = R R.
Therefore, also the upper triangular R is normal. We look at the (1,1)-elements of RR
and R R that evidently must be equal. On one hand we have
(R R)11 = r11 r11 = |r11 |2 ,
on the other hand
n
X n
X
2
(RR )11 = r1j r1j = |r11 | + |r1j |2 .
j=1 j=2
Therefore, the latter sum must vanish, i.e., r1j = 0 for j = 2, . . . , n. Comparing the (2,2)-
elements, (3,3)-elements, etc., of RR and R R, we see that R is diagonal. In this way we
arrive at
Theorem 2.12 A matrix is normal if and only if it is diagonalizable by a unitary matrix.

(Note that unitarily diagonalizable matrices are trivially normal.)

2
If u and v were linearly dependent then it follows that must be zero.
2.7. HERMITIAN MATRICES 41
2.7 Hermitian matrices
Definition 2.13 A matrix A Fnn is Hermitian if
(2.22) A = A .

The Schur decomposition for Hermitian matrices is particularly simple. We first note
that A being Hermitian implies that the upper triangular in the Schur decomposition
A = U U is Hermitian and thus diagonal. In fact, because
= = (U AU ) = U A U = U AU = ,
each diagonal element i of satisfies i = i . So, has to be real. In summary have the
following result.
Theorem 2.14 (Spectral theorem for Hermitian matrices) Let A be Hermitian.
Then there is a unitary matrix U and a real diagonal matrix such that
n
X

(2.23) A = U U = i ui ui .
i=1

The columns u1 , . . . , un of U are eigenvectors corresponding to the eigenvalues 1 , . . . , n .

They form an orthonormal basis for Fn .
The decomposition (2.23) is called a spectral decomposition of A.
As the eigenvalues are real we can sort them with respect to their magnitude. We can,
e.g., arrange them in ascending order such that 1 2 n .
If i = j , then any nonzero linear combination of ui and uj is an eigenvector corre-
sponding to i ,
A(ui + uj ) = ui i + uj j = (ui + uj )i .
However, eigenvectors corresponding to different eigenvalues are orthogonal. Let Au = u
and Av = v, 6= . Then
u v = (u A)v = u (Av) = u v,
and thus
( )u v = 0,
from which we deduce u v = 0 as 6= .
In summary, the eigenvectors corresponding to a particular eigenvalue form a sub-
space, the eigenspace {x Fn , Ax = x} = N (A I). They are perpendicular to the
eigenvectors corresponding to all the other eigenvalues. Therefore, the spectral decompo-
sition (2.23) is unique up to signs if all the eigenvalues of A are distinct. In case of
multiple eigenvalues, we are free to choose any orthonormal basis for the corresponding
eigenspace.
Remark 2.4. The notion of Hermitian or symmetric has a wider background. Let hx, yi
be an inner product on Fn . Then a matrix A is symmetric with respect to this inner
product if hAx, yi = hx, Ayi for all vectors x and y. For the ordinary Euclidean inner
product (x, y) = x y we arrive at the element-wise Definition 2.7 if we set x and y equal
to coordinate vectors.
It is important to note that all the properties of Hermitian matrices that we will derive
subsequently hold similarly for matrices symmetric with respect to a certain inner product.
42 CHAPTER 2. BASICS
Example 2.15 We consider the one-dimensional Sturm-Liouville eigenvalue problem

(2.24) u (x) = u(x), 0 < x < , u(0) = u() = 0,

that models the vibration of a homogeneous string of length that is fixed at both ends.
The eigenvalues and eigenvectors or eigenfunctions of (2.24) are

k = k 2 , uk (x) = sin kx, k N.

(n)
Let ui denote the approximation of an (eigen)function u at the grid point xi ,

ui u(xi ), xi = ih, 0 i n + 1, h= .
n+1
We approximate the second derivative of u at the interior grid points by finite differ-
ences [3, 7]
1
(2.25) (ui1 + 2ui ui+1 ) = ui , 1 i n.
h2
Collecting these equations and taking into account the boundary conditions, u0 = 0 and
un+1 = 0, we get a (matrix) eigenvalue problem

(2.26) Tn x = x

where
2 1
1 2 1

2
(n + 1) 1 2 1

Tn := 2 . . .. .. Rnn .
. . .

1 2 1
1 2
The matrix eigenvalue problem (2.26) can be solved explicitly [9, p.229]. Eigenvalues and
eigenvectors are given by

(n) (n + 1)2 4(n + 1)2 k

k = (2 2 cos k ) = sin2 ,
2 2 2(n + 1)
(2.27)  1/2
(n) 2 k
uk = [sin k , sin 2k , . . . , sin nk ]T , k = .
n+1 n+1
(n)
Clearly, k converges to k as n . (Note that sin as 0.) When we identify
(n) (n)
uk with the piecewise linear function that takes on the values given by uk at the grid
points xi then this function evidently converges to sin kx.

Let p() be a polynomial of degree d, p() = 0 + 1 + 2 2 + + d d . As

Aj = (U U )j = U j U we can define a matrix polynomial as

Xd Xd Xd
(2.28) p(A) = j Aj = j U j U = U j j U .
j=0 j=0 j=0

This equation shows that p(A) has the same eigenvectors as the original matrix A. The
eigenvalues are modified though, k becomes p(k ). Similarly, more complicated functions
of A can be computed if the function is defined on spectrum of A.
2.8. THE JORDAN NORMAL FORM 43
2.8 The Jordan normal form
Theorem 2.16 (Jordan normal form) For every A Fnn there is a nonsingular
matrix X Fnn such that

(2.29) X 1 AX = J = diag(J1 , J2 , . . . , Jp ),

where

k 1
..
k .
(2.30) Jk = Jmk (k ) =
..
Fmk mk

. 1
k

are called Jordan blocks and m1 + + mp = n. The values k need not be distinct. The
Jordan matrix J is unique up to the ordering of the blocks. The transformation matrix X
is not unique.

A matrix is diagonalizable if all Jordan blocks are 1 1, i.e., mk = 1 for all k 3 . In this
case the columns of X are eigenvectors of A.
More generally, there is one eigenvector associated with each Jordan block, e.g.,
  
1 1
J2 ()e1 = = e1 .
0 0

Nontrivial Jordan blocks give rise to so-called generalized eigenvectors e2 , . . . , emk since

(Jk () I)ej+1 = ej , j = 1, . . . , mk 1.

This choice of generalized eigenvectors is not unique though, as (Jk ()I)(e2 +e1 ) = e1
for any . This is one of the reasons for the non-uniqueness of the transformation matrix
X in Theorem 2.16.
From the Jordan blocks we can read geometric and algebraic multiplicity of an eigen-
value: The number of Jordan blocks associated with a particular eigenvalue give the
geometric multiplicity; the sum of its orders gives the algebraic multiplicity.
Numerically the size of the Jordan blocks cannot be determined stably as the following
example shows. Let    
1 0 1
= J2 (0)
0 0 0
be the approximation for J2 (0) that some numerical algorithm has computed. This matrix
has two distinct eigenvalues and thus two eigenvectors,
     
1 1 1 1 1 0
= .
0 0 2 0 2 0

For small the two eigenvectors are very close. They even collaps when 0. A
numerical code cannot differ between the two cases ( = 0, 6= 0) that have a completely
different structure.
3
1 1 Jordan blocks are called trivial.
44 CHAPTER 2. BASICS
Let Y := X and let X = [X1 , X2 , . . . , Xp ] and Y = [Y1 , Y2 , . . . , Yp ] be partitioned
according to J in (2.29), meaning that Xj , Yj Fnmj . Then,
p
X p
X
(2.31) A = XJY = Xk Jk Yk = (k Xk Yk + Xk Nk Yk ),
k=1 k=1

where Nk = Jmk (0). If mk = 1 then Nk is zero. We define the matrices Pk := Xk Yk and

Dk := Xk Nk Yk . Then, since Pk2 = Pk , Pk is a projector on R(Pk ) = R(Xk ). It is called a
spectral projector. From (2.31) we immediately obtain [8]
p
X
(2.32) A= (k Pk + Dk ).
k=1

Since Imk Nk = Nk Imk = Nk , we have

Pk D = D Pk = k D ,
APk = Pk A = Pk APk = k Pk + Dk ,
Aj Pk = Pk Aj = Pk Aj Pk = Pk (k In + Dk )j = (k In + Dk )j Pk .

The Jordan normal form can be computed from the Schur decomposition A = U T U ,
see, e.g., [2], although it is not recommended in general to do so.

1. Group equal eigenvalues on the diagonal of the triangular T . This is a generalization

of the solution of Problem 2.4.

2. Let

T1 T12 T1s
T2 T2s

(2.33) T = .. ..
. .
Ts

where the s diagonal blocks Tk are related to the s distinct eigenvalues of T . The off-
diagonal blocks Tj are zeroed one after the other. Each steps requires the solution
of a Sylvester equation Tj = Tj Y Y T .
Exercise: Consider the case of two (simple or multiple) eigenvalues,
 
T1 T12
T = .
T2

Apply a similarity transformation with the matrix

 
I1 Y
X= .
I2

Determine Y ? How can this be extended to the case (2.33) with s diagonal blocks?

3. The diagonal blocks T1 , . . . , Ts are brought to Jordan form.

The Jordan normal form can be nicely employed to define matrix functions, see [5].
2.9. PROJECTIONS 45
2.9 Projections
Definition 2.17 A matrix P that satisfies

(2.34) P2 = P

is called a projection.

Obviously, a projection is a square matrix. If P is a projection then P x = x for all

x in the range R(P ) of P . In fact, if x R(P ) then x = P y for some y Fn and
P x = P (P y) = P 2 y = P y = x.
x2

x1

Figure 2.1: Oblique projection of example 2.9

Example 2.18 Let  

1 2
P = .
0 0
The range of P is R(P ) = F {0}. The effect of P is depicted in Figure 2.1: All points x
that lie on a line parallel to span{(2, 1) } are mapped on the same point on the x1 axis.
So, the projection is along span{(2, 1) } which is the null space N (P ) of P .

Example 2.19 Let x and y be arbitrary vectors such that y x 6= 0. Then

xy
(2.35) P =
y x
is a projection. Notice that the projector of the previous example can be expressed in the
form (2.35).

Problem 2.20 Let X, Y Fnp such that Y X is nonsingular. Show that

P := X(Y X)1 Y

is a projection.

Example 2.21 The spectral projectors Xk Yk introduced in (2.31) are projectors. Their
range is the span of all eigenvectors and generalized eigenvectors associated with the
eigenvalue k .
46 CHAPTER 2. BASICS
If P is a projection then I P is a projection as well. In fact, (I P )2 = I 2P + P 2 =
I 2P + P = I P . If P x = 0 then (I P )x = x. Therefore, the range of I P coincides
with the null space of P , R(I P ) = N (P ). It can be shown that R(P ) = N (P ) .
Notice that R(P )R(I P ) = N (I P )N (P ) = {0}. For, if P x = 0 then (I P )x =
x, which can only be zero if x = 0. So, any vector x can be uniquely decomposed into

(2.36) x = x1 + x2 , x1 R(P ), x2 R(I P ) = N (P ).

The most interesting situation occurs if the decomposition is orthogonal, i.e., if x1 x2 =

0 for all x.

Definition 2.22 A matrix P is called an orthogonal projection if

(i) P 2 = P
(2.37)
(ii) P = P.

Proposition 2.23 Let P be a projection. Then the following statements are equivalent.
(i) P = P ,
(ii) R(I P ) R(P ), i.e. (P x) (I P )y = 0 for all x, y.

Proof. (ii) follows trivially from (i) and (2.34).

Now, let us assume that (ii) holds. Then

x P y = (P x) y = (P x) (P y + (I P )y)
= (P x) (P y)
= (P x + (I P )x)(P y) = x (P y).

This equality holds for any x and y and thus implies (i).

Example 2.24 Let q be an arbitrary vector of norm 1, kqk = q q = 1. Then P = qq

is the orthogonal projection onto span{q}.

Example 2.25 Let Q Fnp with Q Q = Ip . Then QQ is the orthogonal projector

onto R(Q), which is the space spanned by the columns of Q.

Problem 2.26 Let Q, Q1 Fnp with Q Q = Q1 Q1 = Ip such that R(Q) = R(Q1 ).

This means that the columns of Q and Q1 , respectively, are orthonormal bases of the same
subspace of Fn . Show that the projector does not depend on the basis of the subspace,
i.e., that QQ = Q1 Q1 .

Problem 2.27 Let Q = [Q1 , Q2 ], Q1 Fnp , Q2 Fn(np) be a unitary matrix. Q1

contains the first p columns of Q, Q2 the last n p. Show that Q1 Q1 + Q2 Q2 = I. Hint:
Use QQ = I. Notice, that if P = Q1 Q1 then I P = Q2 Q2 .

Problem 2.28 What is the form of the orthogonal projection onto span{q} if the inner
product is defined as hx, yi := y M x where M is a symmetric positive definite matrix?
2.10. THE RAYLEIGH QUOTIENT 47
2.10 The Rayleigh quotient
Definition 2.29 The quotient
x Ax
(x) := , x 6= 0,
x x
is called the Rayleigh quotient of A at x.

Notice, that (x) = (x), 6= 0. Hence, the properties of the Rayleigh quotient
can be investigated by just considering its values on the unit sphere. Using the spectral
decomposition A = U U , we get
n
X

x Ax = x U U x = i |ui x|2 .
i=1
Pn
Similarly, x x = 2
i=1 |ui x| . With 1 2 n , we have
n
X n
X n
X
1 |ui x|2 i |ui x|2 n |ui x|2 .
i=1 i=1 i=1

So,
1 (x) n , for all x 6= 0.
As
(uk ) = k ,
the extremal values 1 and n are actually attained for x = u1 and x = un , respectively.
Thus we have proved the following theorem.
Theorem 2.30 Let A be Hermitian. Then the Rayleigh quotient satisfies

(2.38) 1 = min (x), n = max (x).

x x

As the Rayleigh quotient is a continuous function it attains all values in the closed interval
[1 , n ].
The next theorem generalizes the above theorem to interior eigenvalues. The following
theorems is attributed to Poincare, Fischer and Polya.
Theorem 2.31 (Minimum-maximum principle) Let A be Hermitian. Then

(2.39) p = min max (Xx)

XFnp , rank(X)=p x6=0

Fnp with full rank we can choose x 6= 0

Proof. Let Up1 = [u1 , . . . , up1 ]. For every X P
Xx = 0. Then 0 6= z := Xx = n
such that Up1 i=p zi ui . As in the proof of the previous
theorem we obtain the inequality
(z) p .
To prove that equality holds in (2.39) we choose X = [u1 , . . . , up ]. Then

1 0
.. x = 0

Up1 Xx = . . . .
1 0
implies that x = ep , i.e., that z = Xx = up . So, (z) = p .
An important consequence of the minimum-maximum principle is the following
48 CHAPTER 2. BASICS
Theorem 2.32 (Monotonicity principle) Let A be Hermitian and let q1 , . . . , qp be
normalized, mutually orthogonal vectors. Set Q := [q1 , . . . , qp ] and A := Q AQ Fpp .
Then the p eigenvalues 1 p of A satisfy

(2.40) k k , 1 k p.

Proof. Let w1 , . . . , wp Fp be the eigenvectors of A ,

(2.41) A wi = i wi , 1 i p,

with wi wj = ij . Then the vectors Qw1 , . . . , Qwp are normalized and mutually orthogo-
nal. Therefore, we can construct a normalized vector x0 with kx0 k = 1,

x0 := a1 Qw1 + + ak Qwk = Q(a1 w1 + + ak wk ) = Qa,

that is orthogonal to the first k 1 eigenvectors of A,

x0 ui = 0, 1 i k 1.

(Note, that kx0 k = 1 implies kak = 1.) Then, with the minimum-maximum principle we
get
k
X
k = min R(x) R(x0 ) = x0 Ax0
= a Q AQa =
|a|2i i k .
x6=0
x u1 ==x uk1 =0 i=1

Exercise: It is possible to prove the inequalities (2.40) without assuming that the
q1 , . . . , qp are orthonormal. But then one has to use the eigenvalues k of

A x = Bx, B = Q Q,

instead of (2.41). Prove this.

Remark 2.5. Let qi = eji , 1 i k. This means that we extract rows and columns
j1 , . . . , jk to construct A . (The indices ji are assumed to be distinct.)
Remark 2.6. Lets remove a single row/column (with equal index) from A. Then k = n 1
in Remark 2.5 and the index set j1 , . . . , jn1 contains all but one of the integers 1, . . . , n.
If we formulate a monotonicity principle based on the eigenvalues n , n1 , . . . as con-
secutive maxima of the Rayleigh quotient, then we arrive at the interlacing property

(2.42) k k k+1 , 1 k < n.

This interlacing property can be generalized, see, e.g., [6, 4].

The trace of a matrix A Fnn is defined to be the sum of the diagonal elements of a
matrix. Matrices that are similar have equal trace. Hence, by the spectral theorem,
n
X n
X
(2.43) trace(A) = aii = i .
i=1 i=1

The following theorem is proved in a similar way as the minimum-maximum theorem.

Theorem 2.33 (Trace theorem)

(2.44) 1 + 2 + + p = min trace(X AX)

XFnp , X X=Ip
2.11. CHOLESKY FACTORIZATION 49
2.11 Cholesky factorization
Definition 2.34 A Hermitian matrix is called positive definite (positive semi-definite)
if all its eigenvalues are positive (nonnegative).
For a Hermitian positive definite matrix A, the LU decomposition can be written in a
particular form reflecting the symmetry of A.
Theorem 2.35 (Cholesky factorization) Let A Fnn be Hermitian positive definite.
Then there is a lower triangular matrix L such that
(2.45) A = LL .
L is unique if we choose its diagonal elements to be positive.
Proof. We prove the theorem by giving an algorithm that computes the desired factoriza-
tion.
Since A is positive definite, we have a11 = e1 Ae1 > 0. Therefore we can form the
matrix
(1) a
l11 11
(1) a21 1

l21 1 a
L1 = =
1,1
.
.. .. .. ..
. . . .
(1) a
n1 1
ln1 1 a1,1
We now form the matrix

1 0 ... 0

0 a22 a21 a12
a11 . . . a2n a21 a1n
a11

1
A1 = L1
1 AL 1 = .. .. .. .. .
. . . .
0 an2 an1 a12
a11 . . . ann an1 a1n
a11
This is the first step of the algorithm. Since positive definiteness is preserved by a congru-
ence transformation X AX (see also Theorem 2.37 below), A1 is again positive definite.
Hence, we can proceed in a similar fashion factorizing A1 (2 : n, 2 : n), etc.
Collecting L1 , L2 , . . . , we obtain
1 1
I = L1 1 1
n L2 L1 A(L1 ) (L2 ) (Ln )1
or
(L1 L2 Ln )(Ln L2 L1 ) = A.
which is the desired result. It is easy to see that L1 L2 Ln is a triangular matrix and
that
(1)
l11
(1) (2)
l
21 l22
(1) (2) (3)
L1 L2 Ln = l31 l32 l33
. . .
. .. .. ..
. .
(1) (2) (3) (n)
ln1 ln2 ln3 . . . lnn

Remark 2.7. When working with symmetric matrices, one often stores only half of the
matrix, e.g. the lower triangle consisting of all elements including and below the diagonal.
The L-factor of the Cholesky factorization can overwrite this information in-place to save
memory.
50 CHAPTER 2. BASICS
Definition 2.36 The inertia of a Hermitian matrix is the triple (, , ) where , , is
the number of negative, zero, and positive eigenvalues.

Theorem 2.37 (Sylvesters law of inertia) If A Cnn is Hermitian and X Cnn

is nonsingular then A and X AX have the same inertia.

Proof. The proof is given, for example, in [4].

Remark 2.8. Two matrices A and B are called congruent if there is a nonsingular matrix
X such that B = X AX. Thus, Sylvesters law of inertia can be stated in the following
form: The inertia is invariant under congruence transformations.

2.12 The singular value decomposition (SVD)

Theorem 2.38 (Singular value decomposition) If A Cmn then there exist unitary
matrices U Cmm and V Cnn such that
 
diag(1 , . . . , p ) 0
(2.46) U AV = = , p = min(m, n),
0 0

where 1 2 p 0.

Proof. If A = O, the theorem holds with U = Im , V = In and equal to the m n zero

matrix.
We now assume that A 6= O. Let x, kxk = 1, be a vector that maximizes kAxk and
let Ax = y where = kAk = kAxk and kyk = 1. As A 6= O, > 0. Consider the scalar
function
kA(x + y)k2 (x + y) A A(x + y)
f () := =
kx + yk2 (x + y) (x + y)
Because of the extremality of Ax, the derivative f () of f () must vanish at = 0. This
holds for all y! We have

df (x A Ay +
y A Ay)kx + yk2 (x y + y
y)kA(x + y)k2
() =
d kx + yk4

Thus, we have for all y,


df  x A Aykxk2 x ykA(x)k2
() = = 0.
d =0 kxk4

As kxk = 1 and kAxk = , we have

(x A A 2 x )y = (A Ax 2 x) y = 0, for all y,

from which
A Ax = 2 x
follow. Multiplying Ax = y from the left by A we get A Ax = A y = 2 x from which

A y = x

and AA y = Ax = 2 y follows. Therefore, x is an eigenvector of A A corresponding to

the eigenvalue 2 and y is an eigenvector of AA corresponding to the same eigenvalue.
2.12. THE SINGULAR VALUE DECOMPOSITION (SVD) 51
Now, we construct a unitary matrix U1 with first column y and a unitary matrix V1
with first column x, U1 = [y, U ] and V1 = [x, V ]. Then
     
y Ax y AV x V 0
U1 AV1 = = =
U Ax U AV U y U AV 0 A

where A = U AV .
The proof above is due to W. Gragg. It nicely shows the relation of the singular value
decomposition with the spectral decomposition of the Hermitian matrices A A and AA ,

(2.47) A = U V = A A = V 2 V . AA = U 2 U ,

Note that the proof given in [4] is shorter and maybe more elegant.
The SVD of dense matrices is computed in a way that is very similar to the dense Her-
mitian eigenvalue problem. However, in the presence of roundoff error, it is not advisable
to make use of the matrices A A and AA . Instead, let us consider the (n + m) (n + m)
Hermitian matrix
 
O A
(2.48) .
A O

Making use of the SVD (2.46) we immediately get

     
O A U O O U O
= .
A O O V T O O V

mn and
 assume that m n. Then we write U = [U1 , U2 ] where U1 F
Now, let us
1
= with 1 Rnn . Then
O

   
O O 1 U1 O   O 1 O U1 O
O A U U2 O U O U2
= 1 O O O U2 O = 1 1 O O O V .
A O O O V O V O
1 O O O V O O O U2 O

The first and third diagonal zero blocks have order n. The middle diagonal block has
order n m. Now we employ the fact that
      
0 1 1 1 0 1 1 1
=
0 2 1 1 0 2 1 1

to obtain
" # 1 1
  O O U V
O A 1 U1 1 U1 U2 1 2 1 2
O
(2.49) = 2
1 V
2 1 O 2 U1 2 V .
1 1
A O 12 V O
2 O O O U2 O

Thus, there are three ways how to treat the computation of the singular value decompo-
sition as an eigenvalue problem. One of the two forms in (2.47) is used implicitly in the
QR algorithm for dense matrices A, see [4],[1]. The form (2.48) is suited if A is a sparse
matrix.
52 CHAPTER 2. BASICS
Bibliography
[1] E. Anderson, Z. Bai, C. Bischof, J. Demmel, J. Dongarra, J. D.
Croz, A. Greenbaum, S. Hammarling, A. McKenney, S. Ostrouchov,
and D. Sorensen, LAPACK Users Guide Release 2.0, SIAM, Philadel-
phia, PA, 1994. (Software and guide are available from Netlib at URL
http://www.netlib.org/lapack/).

[2] R. Fletcher and D. C. Sorensen, An algorithmic derivation of the Jordan canon-

ical form, Amer. Math. Monthly, 90 (1983), pp. 1216.

[3] G. E. Forsythe and W. R. Wasow, Finite-difference methods for partial differential

equations, Wiley, New York, 1960.

[4] G. H. Golub and C. F. van Loan, Matrix Computations, The Johns Hopkins
University Press, Baltimore, MD, 2nd ed., 1989.

[5] N. J. Higham, Functions of matrices, SIAM, Philadelphia, PA, 2008.

[6] R. A. Horn and C. R. Johnson, Matrix Analysis, Cambridge University Press,

Cambridge, 1985.

[7] R. J. LeVeque, Finite Difference Methods for Ordinary and Partial Differential Equa-
tions, SIAM, Philadelphia, PA, 2007.

[8] Y. Saad, Numerical Methods for Large Eigenvalue Problems, SIAM, Philadelphia, PA,
2011.
hl, Matrizen und ihre technischen Anwendungen, Springer, Berlin, 4th ed.,
[9] R. Zurmu
1964.
Chapter 3

Newton methods

3.1 Linear and nonlinear eigenvalue problems

In linear eigenvalue problems we have to find values C such that I A is singular.
Here A Fnn is a given real or complex matrix. Equivalently, we have to find values
C such that there is a nontrivial (nonzero) x that satisfies

(3.1) (A I)x = 0 Ax = x.

In the linear eigenvalue problem (3.1) the eigenvalue appears linearly. However, as the
unknown is multiplied with the unknown vector x, the problem is in fact nonlinear. We
have n+1 unknowns , x1 , . . . , xn that are not uniquely defined by the n equations in (3.1).
We have noticed earlier, that the length of the eigenvector x is not determined. This can
be rectified by adding a further equation that fixes the length of x. The straightforward
condition is

(3.2) kxk2 = x x = 1,

that determines x up to a complex scalar of modulus 1, in the real case 1. Another

condition to normalize x is by requesting that

(3.3) cT x = 1, for some c.

Eq. (3.3) is linear in x and thus simpler. However, the combined equations (3.1)(3.3) are
nonlinear anyway. Furthermore, c must be chosen such that it has a strong component in
the (unknown) direction of the searched eigenvector. This requires some knowledge about
the solution.
In nonlinear eigenvalue problems we have to find values C such that

(3.4) A()x = 0

where A() is a matrix the elements of which depend on in a nonlinear way. An example
is a matrix polynomial,
d
X
(3.5) A() = k Ak , Ak Fnn .
k=0

The linear eigenvalue problem (3.1) is a special case with d = 1,

A() = A0 A1 , A0 = A, A1 = I.

53
54 CHAPTER 3. NEWTON METHODS
Quadratic eigenvalue problems of the form

(3.6) Ax + Kx + 2 M x = 0.

Matrix polynomials can be linearized, i.e., they can be transformed in a linear eigenvalue
of bigger size. The quadratic eigenvalue problem (3.6) can be transformed in a linear
eigenvalue problem of size 2n. Setting y = x we get
     
A O x K M x
=
O I y I O y
or      
A K x O M x
= .
O I y I O y
Notice that many other linearizations are possible [2, 6]. Notice also the relation with the
transformation of high order to first order ODEs [5, p. 478].
Instead of looking at the nonlinear system (3.1) (complemented with (3.3) or (3.2)) we
may look at the nonlinear scalar equation

(3.7) f () := det A() = 0

and apply some zero finder. Here the question arises how to compute f () and in particular
d
f () = d det A().

3.2 Zeros of the determinant

We first consider the computation of eigenvalues and subsequently eigenvectors by means
of computing zeros of the determinant det(A()).
Gaussian elimination with partial pivoting (GEPP) applied to A() provides the de-
composition

(3.8) P ()A() = L()U (),

where P () is the permutation matrix due to partial pivoting, L() is a lower unit trian-
gular matrix, and U () is an upper triangular matrix. From well-known properties of the
determinant function, equation (3.8) gives

det P () det A() = det L() det U ().

Taking the particular structures of the factors in (3.8) into account, we get
n
Y
(3.9) f () = det A() = 1 uii ().
i=1

The derivative of det A() is

n
X n
Y

f () = 1 uii () ujj ()
i=1 j6=i
(3.10) n n n
X uii () Y X uii ()
= 1 ujj () = f ().
uii () uii ()
i=1 j=1 i=1

How can we compute the derivatives uii of the diagonal elements of U ()?
3.2. ZEROS OF THE DETERMINANT 55
3.2.1 Algorithmic differentiation
A clever way to compute derivatives of a function is by algorithmic differentiation, see
e.g., [1]. Here we assume that we have an algorithm available that computes the value
f () of a function f , given the input argument . By algorithmic differentiation a new
algorithm is obtained that computes besides f () the derivative f ().
The idea is easily explained by means of the Horner scheme to evaluate polynomials.
Let
n
X
f (z) = ci z i .
i=1

be a polynomial of degree n. f (z) can be written in the form

f (z) = c0 + z (c1 + z (c2 + + z (cn ) ))

which gives rise to the recurrence

pn := cn ,
pi := z pi+1 + ci , i = n 1, n 2, . . . , 0,
f (z) := p0 .

Note that each of the pi can be considered as a function (polynomial) in z. We use the
above recurrence to determine the derivatives dpi ,

dpn := 0, pn := cn ,

dpi := pi+1 + z dpi+1 , pi := z pi+1 + ci , i = n1, n2, . . . , 0,

f (z) := dp0 , f (z) := p0 .

We can proceed in a similar fashion for computing det A(). We however need to be able
to compute the derivatives aij . Then, we can derive each single assignment in the GEPP
algorithm.
If we restrict ourselves to the standard eigenvalue problem Ax = x then A() =
A I. Then, aij = ij , the Kronecker .

3.2.2 Hymans algorithm

In a Newton iteration we have to compute the determinant for possibly many values .
Using the factorization (3.8) leads to computational costs of 32 n3 flops (floating point
operations) for each factorization, i.e., per iteration step. If this algorithm was used to
compute all eigenvalues then an excessive amount of flops would be required. Can we do
better?
The strategy is to transform A by a similarity transformation to a Hessenberg ma-
trix, i.e., a matrix H whose entries below the lower off-diagonal are zero,

hij = 0, i > j + 1.

Any matrix A can be transformed into a similar Hessenberg matrix H by means of a

sequence of elementary unitary matrices called Householder transformations. The
details are given in Section 4.3.
56 CHAPTER 3. NEWTON METHODS
Let S AS = H, where S is unitary. S is the product of the just mentioned Householder
transformations. Then

Ax = x Hy = y, x = Sy.

So, A and H have equal eigenvalues (A and H are similar) and the eigenvectors are
transformed by S. We now assume that H is unreduced, i.e., hi+1,i 6= 0 for all i.
Otherwise we can split Hx = x in smaller problems.
Let be an eigenvalue of H and

(3.11) (H I)x = 0,

i.e., x is an eigenvector of H associated with the eigenvalue . Then the last component
of x cannot be zero, xn 6= 0. The proof is by contradiction. Let xn = 0. Then (for n = 4)

h11 h12 h13 h14 x1 0
h21 h22 h23 h24 x2 0

= .
h32 h33 h34 x3 0
h43 h44 0 0

The last equation reads

hn,n1 xn1 + (hnn ) 0 = 0

from which xn1 = 0 follows since we assumed hn,n1 6= 0. In the exact same procedure
we obtain xn2 = 0, . . . , x1 = 0. But the zero vector cannot be an eigenvector. Therefore,
xn must not be zero. Without loss of generality we can set xn = 1.
We continue to expose the procedure with a problem size n = 4. If is an eigenvalue
then there are xi , 1 i < n, such that

h11 h12 h13 h14 x1 0
h21 h22 h23 h24
x2 0
(3.12) = .
h32 h33 h34 x3 0
h43 h44 1 0

If is not an eigenvalue then we determine the xi such that

h11 h12 h13 h14 x1 p()
h21 h22 h24
(3.13) h23 x2 = 0 .
h32 h33 h34 x3 0
h43 h44 1 0

We determine the n 1 numbers xn1 , xn2 , . . . , x1 by

1

xi = (hi+1,i+1 ) xi+1 + hi+1,i+2 xi+2 + + hi+1,n xn , i = n 1, . . . , 1.
hi+1,i |{z}
1

The xi are functions of , in fact, xi Pni . The first equation in (3.13) gives

(3.14) (h1,1 ) x1 + h1,2 x2 + + h1,n xn = p().

3.2. ZEROS OF THE DETERMINANT 57
Eq. (3.13) can be understood by the factorization

h11 h12 h13 h14 1 x1
h21 h22 h23 h24 x2
1
h32 h33 h34 1 x3
h43 h44 1

h11 h12 h13 p()
h21 h22 h23 0
=
.
h32 h33 0
h43 0

The last column of this equation corresponds to (3.13). Taking determinants yields
n1
!
Y
det(H I) = (1)n1 hi+1,i p() = c p().
i=1

So, p() is a constant multiple of the determinant of H I. Therefore, we can solve

p() = 0 instead of det(H I) = 0.
Since the quantities x1 , x2 , . . . , xn and thus p() are differentiable functions of , we
can algorithmically differentiate to get p ().
For i = n 1, . . . , 1 we have
1

xi = xi+1 + (hi+1,i+1 ) xi+1 + hi+1,i+2 xi+2 + + hi+1,n1 xn1 .
hi+1,i
Finally,
c f () = x1 + (h1,n ) x1 + h1,2 x2 + + h1,n1 xn1 .
Algorithm 3.2.2 implements Hymans algorithm that returns p() and p () given an input
parameter [7].

Algorithm 3.1 Hymans algorithm

1: Choose a value .
2: xn := 1; dxn := 0;
3: for i = n 1 downto 1 do
4: s = ( hi+1,i+1 ) xi+1 ; ds = xi+1 + ( hi+1,i+1 ) dxi+1 ;
5: for j = i + 2 to n do
6: s = s hi+1,j xj ; ds = ds hi+1,j dxj ;
7: end for
8: xi = s/hi+1,i ; dxi = ds/hi+1,i ;
9: end for
10: s = ( h1,1 )x1 ; ds = x1 ( h1,1 )dx1 ;
11: for i = 2 to n do
12: s = s + h1,i xi ; ds = ds + h1,i dxi ;
13: end for
14: p() := s; p () := ds;

This algorithm computes p() = c det(H()) and its derivative p () of a Hessenberg

matrix H in O(n2 ) operations. Inside a Newton iteration the new iterate is obtained by
p(k )
k+1 = k , k = 0, 1, . . .
p (k )
58 CHAPTER 3. NEWTON METHODS
The factor c cancels. An initial guess 0 has to be chosen to start the iteration. It is
clear that a good guess reduces the iteration count of the Newton method. The iteration
is considered converged if f (k ) 0. The vector x = (x1 , x2 , . . . , xn1 , 1)T is a good
approximation of the corresponding eigenvector.
Remark 3.1. Higher order deriatives of f can be computed in an analogous fashion. Higher
order zero finders (e.g. Laguerres zero finder) are then applicable [3].

3.2.3 Computing multiple zeros

If we have found a zero z of f (x) = 0 and want to compute another one, we want to avoid
recomputing the already found z.
We can explicitly deflate the zero by defining a new function
f (x)
(3.15) f1 (x) := ,
xz
and apply our method of choice to f1 . This procedure can in particular be done with
polynomials. The coefficients of f1 are however very sensitive to inaccuracies in z. We
can proceed similarly for multiple zeros z1 , . . . , zm . Explicit deflation is not recommended
and often not feasible since f is not given explicitely.
For the reciprocal Newton correction for f1 in (3.15) we get
f (x) f (x)
f1 (x) xz (xz)2 f (x) 1
= f (x)
= .
f1 (x) f (x) xz
xz

Then a Newton correction becomes

1
(3.16) x(k+1) = xk
f (xk )
x 1 z
f (xk ) k

and similarly for multiple zeros z1 , . . . , zm . Working with (3.16) is called implicit defla-
tion. Here, f is not modified. In this way errors in z are not propagated to f1

3.3 Newton methods for the constrained matrix problem

We consider the nonlinear eigenvalue problem (3.4) equipped with the normalization con-
dition (3.3),
T () x = 0,
(3.17)
cT x = 1,
where c is some given vector. At a solution (x, ), x 6= 0, T () is singular. Note that x is
defined only up to a (nonzero) multiplicative factor. cT x = 1 is just a way to normalize
x. Another one would be kxk2 = 1, cf. the next section.
Solving (3.17) is equivalent with finding a zero of the nonlinear function f (x, ),
   
T () x 0
(3.18) f (x, ) = = .
cT x 1 0
To apply Newtons zero finding method we need the Jacobian of f ,
 
f (x, ) T () T ()x
(3.19) J(x, ) = .
(x, ) cT 0
3.3. NEWTON METHODS FOR THE CONSTRAINED MATRIX PROBLEM 59
Here, T () denotes the (elementwise) derivative of T with respect to . Then, a step of
Newtons iteration is given by
   
xk+1 xk
(3.20) = J(xk , k )1 f (xk , k ),
k+1 k

or, with the abbreviations Tk := T (k ) and Tk := T (k ),

    
Tk Tk xk xk+1 xk Tk xk
(3.21) = .
cT 0 k+1 k 1 cT xk

If xk is normalized, cT xk = 1, then the second equation in (3.21) yields

(3.22) cT (xk+1 xk ) = 0 cT xk+1 = 1.

The first equation in (3.21) gives

Tk (xk+1 xk ) + (k+1 k ) Tk xk = Tk xk Tk xk+1 = (k+1 k ) Tk xk .

We introduce the auxiliary vector uk+1 by

(3.23) Tk uk+1 = Tk xk .

Note that

(3.24) xk+1 = (k+1 k ) uk+1 .

So, uk+1 points in the desired direction; it just needs to be normalized. Premultiply-
ing (3.24) by cT and using (3.22) gives

1 = cT xk+1 = (k+1 k ) cT uk+1 ,

or
1
(3.25) k+1 = k .
cT uk+1
In summary, we get the following procedure.

Algorithm 3.2 Newton iteration for solving (3.18)

1: Choose a starting vector x0 Rn with cT x0 = 1. Set k := 0.
2: repeat
3: Solve T (k ) uk+1 := T (k ) xk for uk+1 ; (3.23)
4: k := cT uk+1 ;
5: xk+1 := uk+1 /k ; (Normalize uk+1 )
6: k+1 := k 1/k ; (3.25)
7: k := k + 1;
8: until some convergence criterion is satisfied

If the linear eigenvalue problem is solved by Algorithm 3.3 then T ()x = x. In each
iteration step a linear system has to be solved which requires the factorization of a matrix.
We now change the way we normalize x. Problem (3.17) becomes

(3.26) T () x = 0, kxk2 = 1,
60 CHAPTER 3. NEWTON METHODS
with the corresponding nonlinear system of equations
   
T () x 0
(3.27) f (x, ) = 1 T = .
2 (x x 1) 0

The Jacobian now is

 
f (x, ) T () T () x
(3.28) J(x, ) = .
(x, ) xT 0

The Newton step (3.20) is changed into

    
Tk Tk xk xk+1 xk Tk xk
(3.29) = 1 .
xTk 0 k+1 k T
2 (1 xk xk )

If xk is normalized, kxk k = 1, then the second equation in (3.29) gives

(3.30) xTk (xk+1 xk ) = 0 xTk xk+1 = 1.

The correction xk := xk+1 xk is orthogonal to the actual approximation. The first

equation in (3.29) is the same as in (3.21). Again, we employ the auxiliary vector uk+1
defined in (3.23). Premultiplying (3.24) by xTk and using (3.30) gives

1 = xTk xk+1 = (k+1 k ) xTk uk+1 ,

or
1
(3.31) k+1 = k .
xTk uk+1

The next iterate xk+1 is obtained by normalizing uk+1 ,

(3.32) xk+1 = uk+1 /kuk+1 k.

Algorithm 3.3 Newton iteration for solving (3.27)

1: Choose a starting vector x0 Rn with kx(0) k = 1. Set k := 0.
2: repeat
3: Solve T (k ) uk+1 := T (k ) xk for uk+1 ; (3.23)
4: k := xTk uk+1 ;
5: k+1 := k 1/k ; (3.31)
6: xk+1 := uk+1 /kuk+1 k; (Normalize uk+1 )
7: k := k + 1;
8: until some convergence criterion is satisfied

3.4 Successive linear approximations

Ruhe [4] suggested the following method which is not derived as a Newton method. It is
based on an expansion of T () at some approximate eigenvalue k .

(3.33) T ()x (T (k ) T (k ))x = 0, = k .

BIBLIOGRAPHY 61
Algorithm 3.4 Algorithm of successive linear problems
1: Start with approximation 1 of an eigenvalue of T ().
2: for k = 1, 2, . . . do
3: Solve the linear eigenvalue problem T ()u = T ()u.
4: Choose an eigenvalue smallest in modulus.
5: k+1 := k ;
6: end for

Equation (3.33) is a generalized eigenvalue problem with eigenvalue . If k is a good ap-

proximation of an eigenvalue, then it is straightforward to compute the smallest eigenvalue
of

(3.34) T (k )x = T (k )x

and update k by k+1 = k .

Remark: If T is twice continuously differentiable, and is an eigenvalue of problem
(1) such that T () is singular and 0 is an algebraically simple eigenvalue of T ()1 T (),
then the method in Algorithm 3.4 converges quadratically towards .

Bibliography
[1] P. Arbenz and W. Gander, Solving nonlinear eigenvalue problems by algorithmic
differentiation, Computing, 36 (1986), pp. 205215.

[2] D. S. Mackey, N. Mackey, C. Mehl, and V. Mehrmann, Structured polynomial

eigenvalue problems: Good vibrations from good linearizations, SIAM J. Matrix Anal.
Appl., 28 (2006), pp. 10291051.

[3] B. Parlett, Laguerres method applied to the matrix eigenvalue problem, Math. Com-
put., 18 (1964), pp. 464485.

[4] A. Ruhe, Algorithms for the nonlinear eigenvalue problem, SIAM J. Numer. Anal., 10
(1973), pp. 674689.

[5] G. Strang, Introduction to Applied Mathematics, Wellesley-Cambridge Press, Welles-

ley, 1986.

[6] F. Tisseur and K. Meerbergen, The quadratic eigenvalue problem, SIAM Rev., 43
(2001), pp. 235286.

[7] R. C. Ward, The QR algorithm and Hymans method on vector computers, Math.
Comput., 30 (1976), pp. 132142.
62 CHAPTER 3. NEWTON METHODS
Chapter 4

The QR Algorithm

The QR algorithm computes a Schur decomposition of a matrix. It is certainly one of the

most important algorithm in eigenvalue computations [9]. However, it is applied to dense
(or: full) matrices only.
The QR algorithm consists of two separate stages. First, by means of a similarity
transformation, the original matrix is transformed in a finite number of steps to Hessenberg
form or in the Hermitian/symmetric case to real tridiagonal form. This first stage of
the algorithm prepares its second stage, the actual QR iterations that are applied to the
Hessenberg or tridiagonal matrix. The overall complexity (number of floating points) of
the algorithm is O(n3 ), which we will see is not entirely trivial to obtain.
The major limitation of the QR algorithm is that already the first stage generates
usually complete fill-in in general sparse matrices. It can therefore not be applied to large
sparse matrices, simply because of excessive memory requirements. On the other hand,
the QR algorithm computes all eigenvalues (and eventually eigenvectors) which is rarely
desired in sparse matrix computations anyway.
The treatment of the QR algorithm in these lecture notes on large scale eigenvalue
computation is justified in two respects. First, there are of course large or even huge dense
eigenvalue problems. Second, the QR algorithm is employed in most other algorithms to
solve internal small auxiliary eigenvalue problems.

4.1 The basic QR algorithm

In 1958 Rutishauser [10] of ETH Zurich experimented with a similar algorithm that we are
going to present, but based on the LR factorization, i.e., based on Gaussian elimination
without pivoting. That algorithm was not successful as the LR factorization (nowadays
called LU factorization) is not stable without pivoting. Francis [5] noticed that the QR
factorization would be the preferred choice and devised the QR algorithm with many of
the bells and whistles used nowadays.
Before presenting the complete picture, we start with a basic iteration, given in Algo-
rithm 4.1, discuss its properties and improve on it step by step until we arrive at Francis
algorithm.
We notice first that

(4.1) Ak = Rk Qk = Qk Ak1 Qk ,

and hence Ak and Ak1 are unitarily similar. The matrix sequence {Ak } converges (under
certain assumptions) towards an upper triangular matrix [11]. Let us assume that the

63
64 CHAPTER 4. THE QR ALGORITHM
Algorithm 4.1 Basic QR algorithm
1: Let A Cnn . This algorithm computes an upper triangular matrix T and a unitary
matrix U such that A = U T U is the Schur decomposition of A.
2: Set A0 := A and U0 = I.
3: for k = 1, 2, . . . do
4: Ak1 =: Qk Rk ; /* QR factorization */
5: Ak := Rk Qk ;
6: Uk := Uk1 Qk ; /* Update transformation matrix */
7: end for
8: Set T := A and U := U .

eigenvalues are mutually different in magnitude and we can therefore number the eigen-
values such that |1 | > |2 | > > |n |. Then as we will show in Chapter 8 the
elements of Ak below the diagonal converge to zero like
(k)
(4.2) |aij | = O(|i /j |k ), i > j.

From (4.1) we see that

(4.3) Ak = Qk Ak1 Qk = Qk Qk1 Ak2 Qk1 Qk = = Qk Q1 A0 Q1 Qk .

| {z }
Uk

With the same assumption on the eigenvalues, Ak tends to an upper triangular matrix
and Uk converges to the matrix of Schur vectors.

4.1.1 Numerical experiments

We conduct two Matlab experiments to illustrate the convergence rate given in (4.2). To
that end, we construct a random 4 4 matrix with eigenvalues 1, 2, 3, and 4.

D = diag([4 3 2 1]);
rand(seed,0);
format short e
S=rand(4); S = (S - .5)*2;
A = S*D/S % A_0 = A = S*D*S^{-1}
for i=1:20,
[Q,R] = qr(A); A = R*Q
end

This yields the matrix sequence

A( 0) = [ -4.4529e-01 4.9063e+00 -8.7871e-01 6.3036e+00]

[ -6.3941e+00 1.3354e+01 1.6668e+00 1.1945e+01]
[ 3.6842e+00 -6.6617e+00 -6.0021e-02 -7.0043e+00]
[ 3.1209e+00 -5.2052e+00 -1.4130e+00 -2.8484e+00]

A( 1) = [ 5.9284e+00 1.6107e+00 9.3153e-01 -2.2056e+01]

[ -1.5294e+00 1.8630e+00 2.0428e+00 6.5900e+00]
[ 1.9850e-01 2.5660e-01 1.7088e+00 1.2184e+00]
[ 2.4815e-01 1.5265e-01 2.6924e-01 4.9975e-01]

A( 2) = [ 4.7396e+00 1.4907e+00 -2.1236e+00 2.3126e+01]

4.1. THE BASIC QR ALGORITHM 65
[ -4.3101e-01 2.4307e+00 2.2544e+00 -8.2867e-01]
[ 1.2803e-01 2.4287e-01 1.6398e+00 -1.8290e+00]
[ -4.8467e-02 -5.8164e-02 -1.0994e-01 1.1899e+00]

A( 3) = [ 4.3289e+00 1.0890e+00 -3.9478e+00 -2.2903e+01]

[ -1.8396e-01 2.7053e+00 1.9060e+00 -1.2062e+00]
[ 6.7951e-02 1.7100e-01 1.6852e+00 2.5267e+00]
[ 1.3063e-02 2.2630e-02 7.9186e-02 1.2805e+00]

A( 4) = [ 4.1561e+00 7.6418e-01 -5.1996e+00 2.2582e+01]

[ -9.4175e-02 2.8361e+00 1.5788e+00 2.0983e+00]
[ 3.5094e-02 1.1515e-01 1.7894e+00 -2.9819e+00]
[ -3.6770e-03 -8.7212e-03 -5.7793e-02 1.2184e+00]

A( 5) = [ 4.0763e+00 5.2922e-01 -6.0126e+00 -2.2323e+01]

[ -5.3950e-02 2.9035e+00 1.3379e+00 -2.5358e+00]
[ 1.7929e-02 7.7393e-02 1.8830e+00 3.2484e+00]
[ 1.0063e-03 3.2290e-03 3.7175e-02 1.1372e+00]

A( 6) = [ 4.0378e+00 3.6496e-01 -6.4924e+00 2.2149e+01]

[ -3.3454e-02 2.9408e+00 1.1769e+00 2.7694e+00]
[ 9.1029e-03 5.2173e-02 1.9441e+00 -3.4025e+00]
[ -2.6599e-04 -1.1503e-03 -2.1396e-02 1.0773e+00]

A( 7) = [ 4.0189e+00 2.5201e-01 -6.7556e+00 -2.2045e+01]

[ -2.1974e-02 2.9627e+00 1.0736e+00 -2.9048e+00]
[ 4.6025e-03 3.5200e-02 1.9773e+00 3.4935e+00]
[ 6.8584e-05 3.9885e-04 1.1481e-02 1.0411e+00]

A( 8) = [ 4.0095e+00 1.7516e-01 -6.8941e+00 2.1985e+01]

[ -1.5044e-02 2.9761e+00 1.0076e+00 2.9898e+00]
[ 2.3199e-03 2.3720e-02 1.9932e+00 -3.5486e+00]
[ -1.7427e-05 -1.3602e-04 -5.9304e-03 1.0212e+00]

A( 9) = [ 4.0048e+00 1.2329e-01 -6.9655e+00 -2.1951e+01]

[ -1.0606e-02 2.9845e+00 9.6487e-01 -3.0469e+00]
[ 1.1666e-03 1.5951e-02 1.9999e+00 3.5827e+00]
[ 4.3933e-06 4.5944e-05 3.0054e-03 1.0108e+00]

A(10) = [ 4.0024e+00 8.8499e-02 -7.0021e+00 2.1931e+01]

[ -7.6291e-03 2.9899e+00 9.3652e-01 3.0873e+00]
[ 5.8564e-04 1.0704e-02 2.0023e+00 -3.6041e+00]
[ -1.1030e-06 -1.5433e-05 -1.5097e-03 1.0054e+00]

A(11) = [ 4.0013e+00 6.5271e-02 -7.0210e+00 -2.1920e+01]

[ -5.5640e-03 2.9933e+00 9.1729e-01 -3.1169e+00]
[ 2.9364e-04 7.1703e-03 2.0027e+00 3.6177e+00]
[ 2.7633e-07 5.1681e-06 7.5547e-04 1.0027e+00]

A(12) = [ 4.0007e+00 4.9824e-02 -7.0308e+00 2.1912e+01]

[ -4.0958e-03 2.9956e+00 9.0396e-01 3.1390e+00]
[ 1.4710e-04 4.7964e-03 2.0024e+00 -3.6265e+00]
[ -6.9154e-08 -1.7274e-06 -3.7751e-04 1.0014e+00]

A(13) = [ 4.0003e+00 3.9586e-02 -7.0360e+00 -2.1908e+01]

[ -3.0339e-03 2.9971e+00 8.9458e-01 -3.1558e+00]
[ 7.3645e-05 3.2052e-03 2.0019e+00 3.6322e+00]
[ 1.7298e-08 5.7677e-07 1.8857e-04 1.0007e+00]

A(14) = [ 4.0002e+00 3.2819e-02 -7.0388e+00 2.1905e+01]

66 CHAPTER 4. THE QR ALGORITHM
[ -2.2566e-03 2.9981e+00 8.8788e-01 3.1686e+00]
[ 3.6855e-05 2.1402e-03 2.0014e+00 -3.6359e+00]
[ -4.3255e-09 -1.9245e-07 -9.4197e-05 1.0003e+00]

A(15) = [ 4.0001e+00 2.8358e-02 -7.0404e+00 -2.1902e+01]

[ -1.6832e-03 2.9987e+00 8.8305e-01 -3.1784e+00]
[ 1.8438e-05 1.4284e-03 2.0010e+00 3.6383e+00]
[ 1.0815e-09 6.4192e-08 4.7062e-05 1.0002e+00]

A(16) = [ 4.0001e+00 2.5426e-02 -7.0413e+00 2.1901e+01]

[ -1.2577e-03 2.9991e+00 8.7953e-01 3.1859e+00]
[ 9.2228e-06 9.5295e-04 2.0007e+00 -3.6399e+00]
[ -2.7039e-10 -2.1406e-08 -2.3517e-05 1.0001e+00]

A(17) = [ 4.0000e+00 2.3503e-02 -7.0418e+00 -2.1900e+01]

[ -9.4099e-04 2.9994e+00 8.7697e-01 -3.1917e+00]
[ 4.6126e-06 6.3562e-04 2.0005e+00 3.6409e+00]
[ 6.7600e-11 7.1371e-09 1.1754e-05 1.0000e+00]

A(18) = [ 4.0000e+00 2.2246e-02 -7.0422e+00 2.1899e+01]

[ -7.0459e-04 2.9996e+00 8.7508e-01 3.1960e+00]
[ 2.3067e-06 4.2388e-04 2.0003e+00 -3.6416e+00]
[ -1.6900e-11 -2.3794e-09 -5.8750e-06 1.0000e+00]

A(19) = [ 4.0000e+00 2.1427e-02 -7.0424e+00 -2.1898e+01]

[ -5.2787e-04 2.9997e+00 8.7369e-01 -3.1994e+00]
[ 1.1535e-06 2.8265e-04 2.0002e+00 3.6421e+00]
[ 4.2251e-12 7.9321e-10 2.9369e-06 1.0000e+00]

A(20) = [ 4.0000e+00 2.0896e-02 -7.0425e+00 2.1898e+01]

[ -3.9562e-04 2.9998e+00 8.7266e-01 3.2019e+00]
[ 5.7679e-07 1.8846e-04 2.0002e+00 -3.6424e+00]
[ -1.0563e-12 -2.6442e-10 -1.4682e-06 1.0000e+00]

Looking at the element-wise quotients of the last two matrices one recognizes the conver-
gence rates claimed in (4.2).
A(20)./A(19) = [ 1.0000 0.9752 1.0000 -1.0000]
[ 0.7495 1.0000 0.9988 -1.0008]
[ 0.5000 0.6668 1.0000 -1.0001]
[ -0.2500 -0.3334 -0.4999 1.0000]

The elements above and on the diagonal are relatively stable.

If we run the same little Matlab script but with the initial diagonal matrix D replaced
by
D = diag([5 2 2 1]);
then we obtain
A(19) = [ 5.0000e+00 4.0172e+00 -9.7427e+00 -3.3483e+01]
[ -4.2800e-08 2.0000e+00 2.1100e-05 -4.3247e+00]
[ 1.3027e-08 7.0605e-08 2.0000e+00 2.1769e+00]
[ 8.0101e-14 -2.4420e-08 4.8467e-06 1.0000e+00]

A(20) = [ 5.0000e+00 4.0172e+00 -9.7427e+00 3.3483e+01]

[ -1.7120e-08 2.0000e+00 1.0536e-05 4.3247e+00]
[ 5.2106e-09 3.3558e-08 2.0000e+00 -2.1769e+00]
[ -1.6020e-14 1.2210e-08 -2.4234e-06 1.0000e+00]

So, again the eigenvalues are visible on the diagonal of A20 . The element-wise quotients
of A20 relative to A19 are
4.2. THE HESSENBERG QR ALGORITHM 67
A(20)./A(19) = [ 1.0000 1.0000 1.0000 -1.0000]
[ 0.4000 1.0000 0.4993 -1.0000]
[ 0.4000 0.4753 1.0000 -1.0000]
[ -0.2000 -0.5000 -0.5000 1.0000]

Notice that (4.2) does not state a rate for the element at position (3, 2).
These little numerical tests are intended to demonstrate that the convergence rates
given in (4.2) are in fact seen in a real run of the basic QR algorithm. The conclusions we
can draw are the following:

1. The convergence of the algorithm is slow. In fact it can be arbitrarily slow if eigen-
values are very close to each other.

2. The algorithm is expensive. Each iteration step requires the computation of the QR
factorization of a full n n matrix, i.e., each single iteration step has a complexity
O(n3 ). Even if we assume that the number of steps is proportional to n we would
get an O(n4 ) complexity. The latter assumption is not even assured, see point 1 of
this discussion.

In the following we want to improve on both issues. First we want to find a matrix
structure that is preserved by the QR algorithm and that lowers the cost of a single
iteration step. Then, we want to improve on the convergence properties of the algorithm.

4.2 The Hessenberg QR algorithm

A matrix structure that is close to upper triangular form and that is preserved by the QR
algorithm is the Hessenberg form.

Definition 4.1 A matrix H is a Hessenberg matrix if its elements below the lower off-
diagonal are zero,
hij = 0, i > j + 1.

Theorem 4.2 The Hessenberg form is preserved by the QR algorithms.

Proof. We give a constructive proof, i.e., given a Hessenberg matrix H with QR factor-
ization H = QR, we show that H = RQ is again a Hessenberg matrix.
The Givens rotation or plane rotation G(i, j, ) is defined by

1 0 0 0
.. . . .. .. ..
. . . . .

0 c s 0
i
..
G(i, j, ) := ... ..
.
..
. .
..
.

(4.4) 0 s c 0 j

.. .. .. . . ..
. . . . .
0 0 0 1

i j

where c = cos() and s = sin(). Pre-multiplication by G(i, j, ) amounts to a counter-

clockwise rotation by radians in the (i, j) coordinate plane. Clearly, a Givens rotation is
68 CHAPTER 4. THE QR ALGORITHM
an orthogonal matrix. For a unitary version see [4]. If x Rn and y = G(i, j, ) x, then

cxi sxj , k=i
yk = sxi + cxj , k=j

xk , k 6= i, j

We can force yj to be zero by setting

xi xj
(4.5) c= p , s= p .
|xi + |xj |2
|2 |xi |2 + |xj |2

Thus, it is a simple matter to zero a single specific entry in a vector by using a Givens
rotation1 .
Now, let us look at a Hessenberg matrix H. We can show the principle procedure by
means of a 4 4 example.


G(1, 2, 1 ) 0
H=
0 0

0 0 0 0


G(2, 3, 2 ) 0 G(3, 4, 3 )
0 = R
0 0
0 0
0 0 0 0 0

So, with Gk = G(k, k + 1, k ), we get

G G G H = R H = QR.
| 3 {z2 }1
Q

Multiplying Q and R in reversed order gives

H = RQ = RG1 G2 G3 ,

or, pictorially,


0 G(1, 2, 1 )
R=
0 0 0 0

0 0 0 0 0 0


G(2, 3, 2 ) G(3, 4, 1 )

0 0
=H

0 0 0 0 0

More generally, if H is n n, n 1 Givens rotations G1 , . . . , Gn1 are needed to transform

H to upper triangular form. Applying the rotations from the right restores the Hessenberg
form.
Remark 4.1. The Hessenberg nonzero pattern isnt the only pattern that is preserved by
the QR algoritm, see [2], however it is the most simple one.
1
For a stable way to compute Givens rotations see Algorithm 5.1.3 in [6].
4.2. THE HESSENBERG QR ALGORITHM 69
4.2.1 A numerical experiment
We repeat one of the previous two Matlab experiments

D = diag([4 3 2 1]);
rand(seed,0);
S=rand(4); S = (S - .5)*2;
A = S*D/S % A_0 = A = S*D*S^{-1}
H = hess(A); % built-in MATLAB function

for i=1:30,
[Q,R] = qr(H); H = R*Q
end
This yields the matrix sequence
H( 0) = [ -4.4529e-01 -1.8641e+00 -2.8109e+00 7.2941e+00]
[ 8.0124e+00 6.2898e+00 1.2058e+01 -1.6088e+01]
[ 0.0000e+00 4.0087e-01 1.1545e+00 -3.3722e-01]
[ 0.0000e+00 0.0000e+00 -1.5744e-01 3.0010e+00]

H( 5) = [ 4.0763e+00 -2.7930e+00 -7.1102e+00 2.1826e+01]

[ 5.6860e-02 2.4389e+00 -1.2553e+00 -3.5061e+00]
[ -2.0209e-01 2.5681e+00 -2.1805e+00]
[ 4.3525e-02 9.1667e-01]

H(10) = [ 4.0024e+00 -6.2734e-01 -7.0227e+00 -2.1916e+01]

[ 7.6515e-03 2.9123e+00 -9.9902e-01 3.3560e+00]
[ -8.0039e-02 2.0877e+00 3.3549e+00]
[ -7.1186e-04 9.9762e-01]

H(15) = [ 4.0001e+00 -1.0549e-01 -7.0411e+00 2.1902e+01]

[ 1.6833e-03 2.9889e+00 -8.9365e-01 -3.2181e+00]
[ -1.2248e-02 2.0111e+00 -3.6032e+00]
[ 2.0578e-05 9.9993e-01]

H(20) = [ 4.0000e+00 -3.1163e-02 -7.0425e+00 -2.1898e+01]

[ 3.9562e-04 2.9986e+00 -8.7411e-01 3.2072e+00]
[ -1.6441e-03 2.0014e+00 3.6377e+00]
[ -6.3689e-07 1.0000e-00]

H(25) = [ 4.0000e+00 -2.1399e-02 -7.0428e+00 2.1897e+01]

[ 9.3764e-05 2.9998e+00 -8.7056e-01 -3.2086e+00]
[ -2.1704e-04 2.0002e+00 -3.6423e+00]
[ 1.9878e-08 1.0000e-00]

H(30) = [ 4.0000e+00 -2.0143e-02 -7.0429e+00 -2.1897e+01]

[ 2.2247e-05 3.0000e+00 -8.6987e-01 3.2095e+00]
[ -2.8591e-05 2.0000e+00 3.6429e+00]
[ -6.2108e-10 1.0000e-00]

Finally we compute the element-wise quotients of the last two matrices.

H(30)./H(29) = [ 1.0000 0.9954 1.0000 -1.0000]
[ 0.7500 1.0000 0.9999 -1.0000]
[ 0.6667 1.0000 -1.0000]
[ -0.5000 1.0000]

Again the elements in the lower off-diagonal reflect nicely the convergence rates in (4.2).
70 CHAPTER 4. THE QR ALGORITHM
4.2.2 Complexity
We give the algorithm for a single Hessenberg-QR-step in a Matlab-like way, see Algo-
rithm 4.2. By
Hk:j,m:n

we denote the submatrix of H consisting of rows k through j and columns m through n.

Algorithm 4.2 A Hessenberg QR step

1: Let H Cnn be an upper Hessenberg matrix. This algorithm overwrites H with
H = RQ where H = QR is a QR factorization of H.
2: for k = 1, 2, . . . , n 1 do
3: /* Generate Gk and then apply it: H = G(k, k+1, k ) H */
4: [ck , sk ] := givens(H
 k,k, Hk+1,k );
ck sk
5: Hk:k+1,k:n = Hk:k+1,k:n;
sk ck
6: end for
7: for k = 1, 2, . . . , n 1 do
8: /* Apply the rotations Gk from the right */
ck sk
9: H1:k+1,k:k+1 = H1:k+1,k:k+1 ;
sk ck
10: end for

If we neglect the determination of the parameters ck and sk , see (4.5), then each of
the two loops requires
n1
X n(n 1)
6i = 6 3n2 flops.
2
i=1

A flop is a floating point operation (+, , , /). We do not distinguish between them,
although they may slightly differ in their execution time on a computer. Optionally, we
also have to execute the operation Uk := Uk1 Qk of Algorithm 4.1. This is achieved by a
loop similar to the second loop in Algorithm 4.2. Since all the rows and columns of U are

1: for k=1,2,. . . ,n-1 do  

ck sk
2: U1:n,k:k+1 = U1:n,k:k+1 ;
sk ck
3: end for

involved, executing the loop costs

n1
X
6n 6n2 flops.
i=1

Altogether, a QR step with a Hessenberg matrix, including the update of the unitary
transformation matrix, requires 12n2 floating point operations. This has to be set in
relation to a QR step with a full matrix that costs 37 n3 . Consequently, we have gained a
factor of O(n) in terms of operations by moving from dense to Hessenberg form. However,
we may still have very slow convergence if one of the quotients |k |/|k+1 | is close to 1.
4.3. THE HOUSEHOLDER REDUCTION TO HESSENBERG FORM 71
4.3 The Householder reduction to Hessenberg form
In the previous section we discovered that it is a good idea to perform the QR algorithm
with Hessenberg matrices instead of full matrices. But we have not discussed how we
transform a full matrix (by means of similarity transformations) into Hessenberg form.
We catch up on this issue in this section.

4.3.1 Householder reflectors

Givens rotations are designed to zero a single element in a vector. Householder reflectors
are more efficient if a number of elements of a vector are to be zeroed at once. Here, we
follow the presentation given in [6].
Definition 4.3 A matrix of the form

P = I 2uu , kuk = 1,

is called a Householder reflector.

It is easy to verify that Householder reflectors are Hermitian and that P 2 = I. From this
we deduce that P is unitary. It is clear that we only have to store the Householder
vector u to be able to multiply a vector (or a matrix) with P ,

(4.6) P x = x u(2u x).

This multiplication only costs 4n flops where n is the length of the vectors.
A task that we repeatedly want to carry out with Householder reflectors is to transform
a vector x on a multiple of e1 ,

P x = x u(2u x) = e1 .

Since P is unitary, we must have = kxk, where C has absolute value one. Therefore,

x1 kxk
x kxke1 1 x2

u= = ..
kx kxke1 k kx kxke1 k .
xn

We can freely choose provided that || = 1. Let x1 = |x1 |ei . To avoid numerical
cancellation we set = ei .
In the real case, one commonly sets = sign(x1 ). If x1 = 0 we can set in any way.

4.3.2 Reduction to Hessenberg form

Now we show how to use Householder reflectors to reduce an arbitrary square matrix to
Hessenberg form. We show the idea by means of a 5 5 example. In the first step of the
reduction we introduce zeros in the first column below the second element,



P1 P1
A=
0 0 = P1 AP1 .

0 0
0 0
72 CHAPTER 4. THE QR ALGORITHM
Notice that P1 = P1 since it is a Householder reflector! It has the structure

1 0 0 0 0
0  
1 0T
P1 =
0
= .
0 0 I4 2u1 u1

0

The Householder vector u1 is determined such that

a21 u1
a u2
(I 2u1 u1 ) 31 = 0 with u1 =
a41 0 u3 .
a51 0 u4

The multiplication of P1 from the left inserts the desired zeros in column 1 of A. The
multiplication from the right is necessary in order to have similarity. Because of the
nonzero structure of P1 the first column of P1 A is not affected. Hence, the zeros stay
there.
The reduction continues in a similar way:



P2 / P2

P1 AP1 = 0 0

0 0 0
0 0 0



P3 / P3
0 = P3 P2 P1 A P1 P2 P3 .

0 0 | {z }
U
0 0 0

Algorithm 4.3 gives the details for the general n n case. In step 4 of this algorithm,
the Householder reflector is generated such that

ak+1,k u1
ak+2,k u2
0
(I 2uk uk ) . = with u k = .. and || = kxk
.
. 0 .
an,k 0 unk

according to the considerations of the previous subsection. The Householder vectors are
stored at the locations of the zeros. Therefore the matrix U = P1 Pn2 that effects the
similarity transformation from the full A to the Hessenberg H is computed after all House-
holder vectors have been generated, thus saving (2/3)n3 flops. The overall complexity of
the reduction is
P
n2
Application of Pk from the left: 4(n k 1)(n k) 34 n3
k=1

n2
P
Application of Pk from the right: 4(n)(n k) 2n3
k=1
4.4. IMPROVING THE CONVERGENCE OF THE QR ALGORITHM 73
Algorithm 4.3 Reduction to Hessenberg form
1: This algorithm reduces a matrix A Cnn to Hessenberg form H by a sequence of
Householder reflections. H overwrites A.
2: for k = 1 to n2 do
3: Generate the Householder reflector Pk ;
4: /* Apply Pk = Ik (Ink 2uk uk ) from the left to A */
5: Ak+1:n,k:n := Ak+1:n,k:n 2uk (uk Ak+1:n,k:n);
6: /* Apply Pk from the right, A := APk */
7: A1:n,k+1:n := A1:n,k+1:n 2(A1:n,k+1:n uk )uk ;
8: end for
9: if eigenvectors are desired form U = P1 Pn2 then
10: U := In ;
11: for k = n2 downto 1 do
12: /* Update U := Pk U */
13: Uk+1:n,k+1:n := Uk+1:n,k+1:n 2uk (uk Uk+1:n,k+1:n);
14: end for
15: end if

P
n2
Form U = P1 Pn2 : 4(n k)(n k) 43 n3
k=1

Thus, the reduction to Hessenberg form costs 10 3

3 n flops without forming the transforma-
14 3
tion matrix and 3 n including forming this matrix.

4.4 Improving the convergence of the QR algorithm

We have seen how the QR algorithm for computing the Schur form of a matrix A can be
executed more economically if the matrix A is first transformed to Hessenberg form. Now
we want to show how the convergence of the Hessenberg QR algorithm can be improved
dramatically by introducing (spectral) shifts into the algorithm.

Lemma 4.4 Let H be an irreducible Hessenberg matrix, i.e., hi+1,i 6= 0 for all i =
1, . . . , n 1. Let H = QR be the QR factorization of H. Then for the diagonal elements
of R we have
|rkk | > 0, for all k < n.
Thus, if H is singular then rnn = 0.

Proof. Let us look at the k-th step of the Hessenberg QR factorization. For illustration,
let us consider the case k = 3 in a 5 5 example, where the matrix has the structure

+ + + + +
0 + + + +

0 0 + + + .

0 0
0 0 0
The plus-signs indicate elements that have been modified. In step 3, the (nonzero) element
h43 will be zeroed by a Givens rotation G(3, 4, ) that is determined such that
    
cos() sin() kk
h r
= kk .
sin() cos() hk+1,k 0
74 CHAPTER 4. THE QR ALGORITHM
Because the Givens rotation preserves vector lengths, we have

kk |2 + |hk+1,k |2 |hk+1,k |2 > 0,

|rkk |2 = |h

which confirms the claim.

We apply this Lemma to motivate a further strategy to speed up the convergence of
the QR algorithm.
Let be an eigenvalue of the irreducible Hessenberg matrix H. Let us check what
happens it we perform

1: H I = QR /* QR factorization */
2: H = RQ + I

First we notice that H H. In fact,

H = Q (H I)Q + I = Q HQ.

Second, by Lemma 4.4 we have

" #
H I = QR, with R= .
0

Thus, " #
RQ =
00

and " # " #

H 1 h1
H = RQ + I = = .
0 0T
So, if we apply a QR step with a perfect shift to a Hessenberg matrix, the eigenvalue
drops out. We then could deflate, i.e., proceed the algorithm with the smaller matrix
H 1.
Remark 4.2. We could prove the existence of the Schur decomposition in the following
way. (1) transform the arbitrary matrix to Hessenberg form. (2) Do the perfect shift
Hessenberg QR with the eigenvalues which we known to exist one after the other.

4.4.1 A numerical example

We use a matrix of a previous Matlab experiments to show that perfect shifts actually
work.

D = diag([4 3 2 1]); rand(seed,0);

S=rand(4); S = (S - .5)*2;
A = S*D/S;
format short e
H = hess(A)
[Q,R] = qr(H - 2*eye(4))
H1 = R*Q + 2*eye(4)
format long
lam = eig(H1(1:3,1:3))
4.4. IMPROVING THE CONVERGENCE OF THE QR ALGORITHM 75
Matlab produces the output
H = [ -4.4529e-01 -1.8641e+00 -2.8109e+00 7.2941e+00]
[ 8.0124e+00 6.2898e+00 1.2058e+01 -1.6088e+01]
[ 4.0087e-01 1.1545e+00 -3.3722e-01]
[ -1.5744e-01 3.0010e+00]

Q = [ -2.9190e-01 -7.6322e-01 -4.2726e-01 -3.8697e-01]

[ 9.5645e-01 -2.3292e-01 -1.3039e-01 -1.1810e-01]
[ 6.0270e-01 -5.9144e-01 -5.3568e-01]
[ -6.7130e-01 7.4119e-01]

R = [ 8.3772e+00 4.6471e+00 1.2353e+01 -1.7517e+01]

[ 6.6513e-01 -1.1728e+00 -2.0228e+00]
[ 2.3453e-01 -1.4912e+00]
[ -2.4425e-14]

H1 = [ 3.9994e+00 -3.0986e-02 2.6788e-01 -2.3391e+01]

[ 6.3616e-01 1.1382e+00 1.9648e+00 -9.4962e-01]
[ 1.4135e-01 2.8623e+00 -1.2309e+00]
[ 1.6396e-14 2.0000e+00]

lam = [9.99999999999993e-01 4.00000000000003e+00 3.00000000000000e+00]

4.4.2 QR algorithm with shifts

This considerations indicate that it may be good to introduce shifts into the QR algorithm.
However, we cannot choose perfect shifts because we do not know the eigenvalues of the
matrix! We therefore need heuristics how to estimate eigenvalues. One such heuristic is
the Rayleigh quotient shift: Set the shift k in the k-th step of the QR algorithm equal
to the last diagonal element:

(4.7) k := h(k1)
n,n = en H (k1) en .

Algorithm 4.4 The Hessenberg QR algorithm with Rayleigh quotient shift

1: Let H0 = H Cnn be an upper Hessenberg matrix. This algorithm computes its
Schur normal form H = U T U .
2: k := 0;
3: for m=n,n-1,. . . ,2 do
4: repeat
5: k := k + 1;
(k1)
6: k := hm,m ;
7: Hk1 k I =: Qk Rk ;
8: Hk := Rk Qk + k I;
9: Uk := Uk1 Qk ;
(k)
10: until |hm,m1 | is sufficiently small
11: end for
12: T := Hk ;

Algorithm 4.4 implements this heuristic. Notice that the shift changes in each iteration
step! Notice also that deflation is incorporated in Algorithm 4.4. As soon as the last lower
off-diagonal element is sufficiently small, it is declared zero, and the algorithm proceeds
with a smaller matrix. In Algorithm 4.4 the active portion of the matrix is m m.
76 CHAPTER 4. THE QR ALGORITHM
Lemma 4.4 guarantees that a zero is produced at position (n, n 1) in the Hessenberg
matrix H if the shift equals an eigenvalue of H. What happens, if hn,n is a good approx-
imation to an eigenvalue of H? Let us assume that we have an irreducible Hessenberg
matrix



0 ,

0 0
0 0 0 hn,n
where is a small quantity. If we perform a shifted Hessenberg QR step, we first have to
factor H hn,n I, QR = H hn,n I. After n 2 steps of this factorization the R-factor is
almost upper triangular,
+ + + + +
0 + + + +

0 0 + + + .

0 0 0
0 0 0 0
From (4.5) we see that the last Givens rotation has the nontrivial elements

cn1 = p , sn1 = p .
||2 + ||2 ||2 + ||2

Applying the Givens rotations from the right one sees that the last lower off-diagonal
element of H = RQ + hn,n I becomes

n,n1 = 2
(4.8) h .
2 + 2
So, we have quadratic convergence unless is also tiny.
A second even more often used shift strategy is the Wilkinson shift:
" (k1) (k1)
#
hn1,n1 hn1,n
(4.9) k := eigenvalue of (k1) (k1)
that is closer to h(k1)
n,n .
hn,n1 hn,n

4.4.3 A numerical example

We give an example for the Hessenberg QR algorithm with shift, but without deflation.
The Matlab code

D = diag([4 3 2 1]);
rand(seed,0);
S=rand(4); S = (S - .5)*2;
A = S*D/S;
H = hess(A)

for i=1:8,
[Q,R] = qr(H-H(4,4)*eye(4)); H = R*Q+H(4,4)*eye(4);
end
produces the output
4.5. THE DOUBLE SHIFT QR ALGORITHM 77
H( 0) = [ -4.4529e-01 -1.8641e+00 -2.8109e+00 7.2941e+00]
[ 8.0124e+00 6.2898e+00 1.2058e+01 -1.6088e+01]
[ 0.0000e+00 4.0087e-01 1.1545e+00 -3.3722e-01]
[ 0.0000e+00 0.0000e+00 -1.5744e-01 3.0010e+00]

H( 1) = [ 3.0067e+00 1.6742e+00 -2.3047e+01 -4.0863e+00]

[ 5.2870e-01 8.5146e-01 1.1660e+00 -1.5609e+00]
[ -1.7450e-01 3.1421e+00 -1.1140e-01]
[ -1.0210e-03 2.9998e+00]

H( 2) = [ 8.8060e-01 -4.6537e-01 9.1630e-01 1.6146e+00]

[ -1.7108e+00 5.3186e+00 2.2839e+01 -4.0224e+00]
[ -2.2542e-01 8.0079e-01 5.2445e-01]
[ -1.1213e-07 3.0000e+00]

H( 3) = [ 1.5679e+00 9.3774e-01 1.5246e+01 1.2703e+00]

[ 1.3244e+00 2.7783e+00 1.7408e+01 4.1764e+00]
[ 3.7230e-02 2.6538e+00 -7.8404e-02]
[ 8.1284e-15 3.0000e+00]

H( 4) = [ 9.9829e-01 -7.5537e-01 -5.6915e-01 1.9031e+00]

[ -3.2279e-01 5.1518e+00 2.2936e+01 -3.9104e+00]
[ -1.6890e-01 8.4993e-01 3.8582e-01]
[ -5.4805e-30 3.0000e+00]

H( 5) = [ 9.3410e-01 -3.0684e-01 3.0751e+00 -1.2563e+00]

[ 3.5835e-01 3.5029e+00 2.2934e+01 4.1807e+00]
[ 3.2881e-02 2.5630e+00 -7.2332e-02]
[ 1.1313e-59 3.0000e+00]

H( 6) = [ 1.0005e+00 -8.0472e-01 -8.3235e-01 1.9523e+00]

[ -7.5927e-02 5.1407e+00 2.2930e+01 -3.8885e+00]
[ -1.5891e-01 8.5880e-01 3.6112e-01]
[ -1.0026e-119 3.0000e+00]

H( 7) = [ 9.7303e-01 -6.4754e-01 -8.9829e-03 -1.8034e+00]

[ 8.2551e-02 3.4852e+00 2.3138e+01 3.9755e+00]
[ 3.3559e-02 2.5418e+00 -7.0915e-02]
[ 3.3770e-239 3.0000e+00]

H( 8) = [ 1.0002e+00 -8.1614e-01 -8.9331e-01 1.9636e+00]

[ -1.8704e-02 5.1390e+00 2.2928e+01 -3.8833e+00]
[ -1.5660e-01 8.6086e-01 3.5539e-01]
[ 0 3.0000e+00]
The numerical example shows that the shifted Hessenberg QR algorithm can work very
nicely. In this example the (4,3) element is about 1030 after 3 steps. (We could stop
there.) The example also nicely shows a quadratic convergence rate.

4.5 The double shift QR algorithm

The shifted Hessenberg QR algorithm does not always work so nicely as in the previous
example. If in (4.8) is O() then hn,n1 can be large. (A small indicates a near
singular H1:n1,1:n1.)
Another problem occurs if real Hessenberg matrices have complex eigenvalues. We
know that for reasonable convergence rates the shifts must be complex. If an eigenvalue
It is (for rounding errors)
has been found we can execute a single perfect shift with .
unprobable however that we will get back to a real matrix.
78 CHAPTER 4. THE QR ALGORITHM
Since the eigenvalues come in complex conjugate pairs it is natural to search for a pair
of eigenvalues right-away. This is done by collapsing two shifted QR steps in one double
step with the two shifts being complex conjugates of each other.
Let 1 and 2 be two eigenvalues of the real matrix (cf. Wilkinson shift (4.9))
" (k1) (k1)
#
hn1,n1 hn1,n
G= (k1) (k1)
R22 .
hn,n1 hn,n

If 1 C \ R then 2 =
1 . Let us perform two QR steps using 1 and 2 as shifts. Setting
k = 1 for convenience we get

H 0 1 I = Q 1 R1 ,
H1 = R1 Q1 + 1 I,
(4.10)
H 1 2 I = Q 2 R2 ,
H2 = R2 Q2 + 2 I.

From the second and third equation in (4.10) we obtain

R1 Q1 + (1 2 )I = Q2 R2 .

Multiplying this equation with Q1 from the left and with R1 from the right we get

Q1 R1 Q1 R1 + (1 2 )Q1 R1 = Q1 R1 (Q1 R1 + (1 2 )I)

= (H0 1 I)(H0 2 I) = Q1 Q2 R2 R1 .

Because 2 =
1 we have

1 I) = H02 2Re()H0 + ||2 I = Q1 Q2 R2 R1 .

M := (H0 1 I)(H0

Therefore, (Q1 Q2 )(R2 R1 ) is the QR factorization of a real matrix. We can choose (scale)
Q1 and Q2 such that Z := Q1 Q2 is real orthogonal. (Then also R2 R1 is real.) By
consequence,
H2 = (Q1 Q2 ) H0 (Q1 Q2 ) = Z T H0 Z
is real.
A procedure to compute H2 by avoiding complex arithmetic could consist of three
steps:
(k1)
1. Form the real matrix M = H02 sH0 + tI with s = 2Re() = trace(G) = hn1,n1 +
(k1) (k1) (k1) (k1) (k1)
hn,n and t = ||2 = det(G) = hn1,n1 hn,n hn1,n hn,n1 . Notice that M has
two lower off-diagonals, " #
M= .

2. Compute the QR factorization M = ZR,

3. Set H2 = Z T H0 Z.

This procedure is however too expensive since item 1, i.e., forming H 2 requires O(n3 )
flops.
A remedy for the situation is provided by the Implicit Q Theorem.
4.5. THE DOUBLE SHIFT QR ALGORITHM 79
Theorem 4.5 (The implicit Q theorem) Let A Rnn . Let Q = [q1 , . . . , qn ] and
V = [v1 , . . . , vn ] be orthogonal matrices that both similarly transform A to Hessenberg
form, H = QT AQ and G = V T AV . Let k denote the smallest positive integer for which
hk+1,k = 0, with k = n if H is irreducible.
If q1 = v1 then qi = vi and |hi,i1 | = |gi,i1 | for i = 2, . . . , k. If k < n, then
gk+1,k = 0.

Proof. [6] Let W = V T Q. Clearly, W is orthogonal, and GW = W H.

We first show that the first k columns of W form an upper triangular matrix, i.e.,

(4.11) wi = W ei span{e1 , . . . , ei }, i k.

(Notice that orthogonal upper triangular matrices are diagonal with diagonal entries 1.)
This is proced inductively. For i = 1 we have w1 = e1 by the assumption that q1 = v1 .
For 1 < i k we assume that (4.11) is true for wi and use the equality GW = W H. The
(i1)-th column of this equation reads

i
X
Gwi1 = GW ei1 = W Hei1 = wj hj,i1 .
j=1

Since hi,i1 6= 0 we have

i1
X
wi hi,i1 = Gwi1 wj hj,i1 span{e1 , . . . ei },
j=1

as G is a Hessenberg matrix. So, the upper-left k k block of W is upper triangular.

Since the columns of W are orthogonal we conclude that wi = ei , i k.
Since wi = V T Qei = V T qi = ei we see that qi is orthogonal to all columns of V
except the i-th. Therefore, we must have qi = vi . Further,

hi,i1 = eTi Hei1 = eTi QT AQei1 = eTi QT V GV T Qei1 = wiT Gwi1 = gi,i1 ,

thus, |hi,i1 | = |gi,i1 |. If hk+1,k = 0 then

k
X
gk+1,k = eTk+1 Gek = eTk+1 GW ek = eTk+1 W Hek = eTk+1 wj hj,k = 0.
j=1

since eTk+1 wj = eTk+1 ej = 0 for j k.

Golub and van Loan [6, p.347] write that The gist of the implicit Q theorem is that if
QT AQ = H and Z T AZ = G are both unreduced Hessenberg matrices and Q and Z have
the same first column, then G and H are essentially equal in the sense that G = DHD
with D = diag(1, . . . , 1).

We apply the Implicit Q Theorem in the following way: We want to compute the
Hessenberg matrix Hk+1 = Z T Hk1 Z where ZR is the QR factorization of M = Hk1 2
sHk1 + tI. The Implicit Q Theorem now tells us that we essentially get Hk+1 by any
orthogonal similarity transformation Hk1 Z1 Hk1 Z1 provided that Z1 HZ1 is Hessen-
berg and Z1 e1 = Ze1 .
80 CHAPTER 4. THE QR ALGORITHM
Let P0 be the Householder reflector with

P0T M e1 = P0T (Hk1

2
2Re()Hk1 + ||2 I) e1 = e1 .

Since only the first three elements of the first column M e1 of M are nonzero, P0 has the
structure





P0 = 1 .

.
. .
1
So,



+


Hk1 := P0 Hk1 P0 =
T
+ + .





We now reduce P0T Hk1 P0 similarly to Hessenberg form the same way as we did earlier, by
a sequence of Householder reflectors P1 , . . . , Pn2 . However, P0T Hk1 P0 is a Hessenberg
matrix up to the bulge at the top left. We take into account this structure when forming
the Pi = I 2pi pTi . So, the structures of P1 and of P1T P0T Hk1 P0 P1 are

1


0

P1 =
, H = P T H P1 = 0 + .
k1 1 k1
1 + +

1
1

The transformation with P1 has chased the bulge one position down the diagonal. The
consecutive reflectors push it further by one position each until it falls out of the matrix
at the end of the diagonal. Pictorially, we have







Hk1 = P2T Hk1

P2 =
0
0 +

+ +






Hk1 = P3 Hk1 P3 =
T


0

0 +
+ +
4.5. THE DOUBLE SHIFT QR ALGORITHM 81

Hk1 = P4T Hk1

P4 =




0

0 +




Hk1 T
= P5 Hk1 P5 =




0

It is easy to see that the Householder vector pi , i < n 2, has only three nonzero elements
at position i + 1, i + 2, i + 3. Of pn2 only the last two elements are nonzero. Clearly,
P0 P1 Pn2 e1 = P0 e1 = M e1 /.
Remark 4.3. Notice that in Algorithm 4.5 a double step is taken also if the eigenvalues of
 
hqq hqp
G=
hpq hpp

are real. As in the complex case we set s = trace(G) and t = det(G).

4.5.1 A numerical example

We consider a simple Matlab implementation of the Algorithm 4.5 to compute the eigen-
values of the real matrix

7 3 4 11 9 2
6 4 5 7 1 12

1 9 2 2 9 1
A=
8

0 1 5 0 8

4 3 5 7 2 10
6 1 4 11 7 1

that has the spectrum

(A) = {1 2i, 3, 4, 5 6i}.
The intermediate output of the code was (after some editing) the following:
>> H=hess(A)

H(0) =

7.0000 7.2761 5.8120 -0.1397 9.0152 7.9363

12.3693 4.1307 18.9685 -1.2071 10.6833 2.4160
0 -7.1603 2.4478 -0.5656 -4.1814 -3.2510
0 0 -8.5988 2.9151 -3.4169 5.7230
0 0 0 1.0464 -2.8351 -10.9792
0 0 0 0 1.4143 5.3415
82 CHAPTER 4. THE QR ALGORITHM
Algorithm 4.5 The Francis double step QR algorithm
1: Let H0 = H Rnn be an upper Hessenberg matrix. This algorithm computes its
real Schur form H = U T U T using the Francis double step QR algorithm. T is a quasi
upper triangular matrix.
2: p := n; /* p indicates the active matrix size. */
3: while p > 2 do
4: q := p 1;
5: s := Hq,q + Hp,p; t := Hq,q Hp,p Hq,p Hp,q ;
6: /* compute first 3 elements of first column of M */
2 +H H
7: x := H1,1 1,2 2,1 sH1,1 + t;
8: y := H2,1 (H1,1 + H2,2 s);
9: z := H2,1 H3,2 ;
10: for k = 0 to p 3 do
11: Determine the Householder reflector P with P T [x; y; z]T = e1 ;
12: r := max{1, k};
13: Hk+1:k+3,r:n := P T Hk+1:k+3,r:n;
14: r := min{k + 4, p};
15: H1:r,k+1:k+3 := H1:r,k+1:k+3P ;
16: x := Hk+2,k+1 ; y := Hk+3,k+1 ;
17: if k < p 3 then
18: z := Hk+4,k+1 ;
19: end if
20: end for
21: Determine the Givens rotation P with P T [x; y]T = e1 ;
22: Hq:p,p2:n := P T Hq:p,p2:n;
23: H1:p,p1:p := H1:p,p1:pP ;
24: /* check for convergence */
25: if |Hp,q | < (|Hq,q | + |Hp,p |) then
26: Hp,q := 0; p := p 1; q := p 1;
27: else if |Hp1,q1 | < (|Hq1,q1 | + |Hq,q |) then
28: Hp1,q1 := 0; p := p 2; q := p 1;
29: end if
30: end while

>> PR=qr2st(H)

[it_step, p = n_true, H(p,p-1), H(p-1,p-2)]

1 6 -1.7735e-01 -1.2807e+00
2 6 -5.9078e-02 -1.7881e+00
3 6 -1.6115e-04 -5.2705e+00
4 6 -1.1358e-07 -2.5814e+00
5 6 1.8696e-14 1.0336e+01
6 6 -7.1182e-23 -1.6322e-01

H(6) =

5.0000 6.0000 2.3618 5.1837 -13.4434 -2.1391

-6.0000 5.0000 2.9918 10.0456 -8.7743 -21.0094
0.0000 -0.0001 -0.9393 3.6939 11.7357 3.8970
4.5. THE DOUBLE SHIFT QR ALGORITHM 83
0.0000 -0.0000 -1.9412 3.0516 2.9596 -10.2714
0 0.0000 0.0000 -0.1632 3.8876 4.1329
0 0 0 0.0000 -0.0000 3.0000

7 5 1.7264e-02 -7.5016e-01
8 5 2.9578e-05 -8.0144e-01
9 5 5.0602e-11 -4.6559e+00
10 5 -1.3924e-20 -3.1230e+00

H(10) =

5.0000 6.0000 -2.7603 1.3247 11.5569 -2.0920

-6.0000 5.0000 -10.7194 0.8314 11.8952 21.0142
-0.0000 -0.0000 3.5582 3.3765 5.9254 -8.5636
-0.0000 -0.0000 -3.1230 -1.5582 -10.0935 -6.3406
0 0 0 0.0000 4.0000 4.9224
0 0 0 0.0000 0 3.0000

11 4 1.0188e+00 -9.1705e-16

H(11) =

5.0000 6.0000 -10.2530 4.2738 -14.9394 -19.2742

-6.0000 5.0000 -0.1954 1.2426 7.2023 -8.6299
-0.0000 -0.0000 2.2584 -5.4807 -10.0623 4.4380
0.0000 -0.0000 1.0188 -0.2584 -5.9782 -9.6872
0 0 0 0 4.0000 4.9224
0 0 0 0.0000 0 3.0000

4.5.2 The complexity

We first estimate the complexity of a single step of the double step Hessenberg QR al-
gorithm. The most expensive operations are the applications of the 3 3 Householder
reflectors in steps 13 and 15 of Algorithm 4.5. Let us first count the flops for applying the
Householder reflector to a 3-vector,

x := (I 2uuT )x = x u(2uT x).

The inner product uT x costs 5 flops, multiplying with 2 another one. The operation
x := x u, = 2uT x, cost 6 flops, altogether 12 flops.
In the k-th step of the loop there are n k of these application from the left in step 13
and k + 4 from the right in step 15. In this step there are thus about 12n + O(1) flops to be
executed. As k is running from 1 to p 3. We have about 12pn flops for this step. Since p
runs from n down to about 2 we have 6n3 flops. If we assume that two steps are required per
eigenvalue the flop count for Francis double step QR algorithm to compute all eigenvalues
of a real Hessenberg matrix is 12n3 . If also the eigenvector matrix is accumulated the two
additional statements have to be inserted into Algorithm 4.5. After step 15 we have

1: Q1:n,k+1:k+3 := Q1:n,k+1:k+3 P ;

and after step 23 we introduce

84 CHAPTER 4. THE QR ALGORITHM

1: Q1:n,p1:p := Q1:n,p1:pP ;
which costs another 12n3 flops.
We earlier gave the estimate of 6n3 flops for a Hessenberg QR step, see Algorithm 4.2.
If the latter has to be spent in complex arithmetic then the single shift Hessenberg QR al-
gorithm is more expensive than the double shift Hessenberg QR algorithm that is executed
in real arithmetic.
Remember that the reduction to Hessenberg form costs 10 3
3 n flops without forming the
14 3
transformation matrix and 3 n if this matrix is formed.

4.6 The symmetric tridiagonal QR algorithm

The QR algorithm can be applied straight to Hermitian or symmetric matrices. By (4.1) we
see that the QR algorithm generates a sequence {Ak } of symmetric matrices. Taking into
account the symmetry, the performance of the algorithm can be improved considerably.
Furthermore, from Theorem 2.14 we know that Hermitian matrices have a real spectrum.
Therefore, we can restrict ourselves to single shifts.

4.6.1 Reduction to tridiagonal form

The reduction of a full Hermitian matrix to Hessenberg form produces a Hermitian Hes-
senberg matrix, which (up to rounding errors) is a real symmetric tridiagonal matrix. Let
us consider how to take into account symmetry. To that end let us consider the first
reduction step that introduces n 2 zeros into the first column (and the first row) of
A = A Cnn . Let
 
1 0T
P1 = , u1 Cn , ku1 k = 1.
0 In1 2u1 u1
Then,
A1 := P1 AP1 = (I 2u1 u1 )A(I 2u1 u1 )
= A u1 (2u1 A 2(u1 Au1 )u1 ) (2Au1 2u1 (u1 Au1 )) u1
| {z } | {z }
v1 v1
= A u1 v1 v1 u1 .
In the k-th step of the reduction we similarly have
Ak = Pk Ak1 Pk = Ak1 uk1 vk1

vk1 uk1 ,
where the last n k elements of uk1 and vk1 are nonzero. Forming
vk1 = 2Ak1 uk1 2uk1 (uk1 Ak1 uk1 )
costs 2(n k)2 + O(n k) flops. This complexity results from Ak1 uk1 . The rank-2
update of Ak1 ,

Ak = Ak1 uk1 vk1 vk1 uk1 ,
requires another 2(nk)2 +O(nk) flops, taking into account symmetry. By consequence,
the transformation to tridiagonal form can be accomplished in
n1
X
4
4(n k)2 + O(n k) = n3 + O(n2 )
3
k=1
4.6. THE SYMMETRIC TRIDIAGONAL QR ALGORITHM 85
floating point operations.

4.6.2 The tridiagonal QR algorithm

In the symmetric case the Hessenberg QR algorithm becomes a tridiagonal QR algorithm.
This can be executed in an explicit or an implicit way. In the explicit form, a QR step
is essentially

1: Choose a shift
2: Compute the QR factorization A I = QR
3: Update A by A = RQ + I.

Of course, this is done by means of plane rotations and by respecting the symmetric
tridiagonal structure of A.
In the more elegant implicit form of the algorithm we first compute the first Givens
rotation G0 = G(1, 2, ) of the QR factorization that zeros the (2, 1) element of A I,
    
c s a11
(4.12) = , c = cos(0 ), s = sin(0 ).
s c a21 0

Performing a similary transformation with G0 we have (n = 5)

+


G0 AG0 = A =
+



Similar as with the double step Hessenberg QR algorithm we chase the bulge down the
diagonal. In the 5 5 example this becomes

+ 0
+
G0 G1

A +
0
= G(1, 2, 0 ) = G(2, 3, 1 ) +


0

G2 G 3

+
0 = A.
= G(3, 4, 2 ) 0 = G(4, 5, 3 )
+ 0
The full step is given by

A = Q AQ, Q = G0 G1 Gn2 .

Because Gk e1 = e1 for k > 0 we have

Q e1 = G0 G1 Gn2 e1 = G0 e1 .

Both explicit and implicit QR step form the same first plane rotation G0 . By referring to
the Implicit Q Theorem 4.5 we see that explicit and implicit QR step compute essentially
the same A.
86 CHAPTER 4. THE QR ALGORITHM

Algorithm 4.6 Symmetric tridiagonal QR algorithm with implicit Wilkinson

shift
1: Let T Rnn be a symmetric tridiagonal matrix with diagonal entries a1 , . . . , an and
off-diagonal entries b2 , . . . , bn .
This algorithm computes the eigenvalues 1 , . . . , n of T and corresponding eigenvec-
tors q1 , . . . , qn . The eigenvalues are stored in a1 , . . . , an . The eigenvectors are stored
in the matrix Q, such that T Q = Q diag(a1 , . . . , an ).

2: m = n /* Actual problem dimension. m is reduced in the convergence check. */

3: while m > 1 do
4: d := (am1 am )/2; /* Compute Wilkinsons shift */
5: if d = 0 then
6: s := am |bm |;
7: else p
8: s := am b2m /(d + sign(d) d2 + b2m );
9: end if
10: x := a(1) s; /* Implicit QR step begins here */
11: y := b(2);
12: for k = 1 to m 1 do
13: if m > 2 then
14: [c, s] := givens(x, y);
15: else    
c s a1 b2 c s
16: Determine [c, s] such that is diagonal
s c b2 a2 s c
17: end if
18: w := cx sy;
19: d := ak ak+1 ; z := (2cbk+1 + ds)s;
20: ak := ak z; ak+1 := ak+1 + z;
21: bk+1 := dcs + (c2 s2 )bk+1 ;
22: x := bk+1 ;
23: if k > 1 then
24: bk := w;
25: end if
26: if k < m 1 then
27: y := sbk+2 ; bk+2 := cbk+2 ;
28: end if  
c s
29: Q1:n;k:k+1 := Q1:n;k:k+1 ;
s c
30: end for/* Implicit QR step ends here */
31: if |bm | < (|am1 | + |am |) then /* Check for convergence */
32: m := m 1;
33: end if
34: end while
4.7. RESEARCH 87
Algorithm 4.6 shows the implicit symmetric tridiagonal QR algorithm. The shifts
are chosen acording to Wilkinson. An issue not treated in this algorithm is deflation.
Deflation is of big practical importance. Let us consider the following 6 6 situation

a1 b2
b2 a2 b3

b3 a3 0
T =

.

0 a 4 b5
b5 a 5 b6
b6 a 6

The shift for the next step is determined from elements a5 , a6 , and b6 . According to (4.12)
the first plane rotation is determined from the shift and the elements a1 and b1 . The im-
plicit shift algorithm then chases the bulge down the diagonal. In this particular situation,
the procedure finishes already in row/column 4 because b4 = 0. Thus the shift which is an
approximation to an eigenvalue of the second block (rows 4 to 6) is applied to the wrong
first block (rows 1 to 3). Clearly, this shift does not improve convergence.
If the QR algorithm is applied in its explicit form, then still the first block is not treated
properly, i.e. with a (probably) wrong shift, but at least the second block is diagonalized
rapidly.
Deflation is done as indicated in Algorithm 4.6:

if |bk | < (|ak1 | + |ak |) then deflate.

Deflation is particularly simple in the symetric case since it just means that a tridiagonal
eigenvalue problem decouples in two (or more) smaller tridiagonal eigenvalue problems.
Notice, however, that the eigenvectors are still n elements long.

4.7 Research
Still today the QR algorithm computes the Schur form of a matrix and is by far the
most popular approach for solving dense nonsymmetric eigenvalue problems. Multishift
and aggressive early deflation techniques have led to significantly more efficient sequential
implementations of the QR algorithm during the last decade. For a brief survey and a
discussion of the parallelization of the QR algorithm, see [7].
The three steps of the presented symmetric QR algorithm are (1) reducion of the origi-
nal matrix to tridiagonal form, (2) computation of the eigenpairs of the tridiagonal matrix,
and (3) back-transformation of the eigenvectors. In the ELPA project the first step has
been successfully replaced by a two-stage procedure: transformation full to banded, and
banded to tridiagonal. This approach improves the utilization of memory hierarchies [8, 3].

4.8 Summary
The QR algorithm is a very powerful algorithm to stably compute the eigenvalues and (if
needed) the corresponding eigenvectors or Schur vectors. All steps of the algorithm cost
O(n3 ) floating point operations, see Table 4.1. The one exception is the case where only
eigenvalues are desired of a symmetric tridiagonal matrix. The linear algebra software
package LAPACK [1] contains subroutines for all possible ways the QR algorithm may be
employed.
88 CHAPTER 4. THE QR ALGORITHM
nonsymmetric case symmetric case
without with without with
Schurvectors eigenvectors
10 3 14 3 4 3 8 3
transformation to Hessenberg/tridiagonal form 3 n 3 n 3n 3n
20 3 50 3
real double step Hessenberg/tridiagonal QR al- 3 n 3 n 24n2 6n3
gorithm (2 steps per eigenvalues assumed)
4 3
total 10n3 25n3 3n 9n3

Table 4.1: Complexity in flops to compute eigenvalues and eigenvectors/Schur vectors of

a real n n matrix

We finish by repeating, that the QR algorithm is a method for dense matrix problems.
The reduction of a sparse matrix to tridiagonal or Hessenberg form produces fill-in, thus
destroying the sparsity structure which one almost always tries to preserve.

Bibliography
[1] E. Anderson, Z. Bai, C. Bischof, J. Demmel, J. Dongarra, J. D.
Croz, A. Greenbaum, S. Hammarling, A. McKenney, S. Ostrouchov,
and D. Sorensen, LAPACK Users Guide Release 2.0, SIAM, Philadel-
phia, PA, 1994. (Software and guide are available from Netlib at URL
http://www.netlib.org/lapack/).

[2] P. Arbenz and G. H. Golub, Matrix shapes invariant under the symmetric QR
algorithm, Numer. Linear Algebra Appl., 2 (1995), pp. 8793.

[3] T. Auckenthaler, H.-J. Bungartz, T. Huckle, L. Kra mer, B. Lang, and

P. Willems, Developing algorithms and software for the parallel solution of the sym-
metric eigenvalue problem, J. Comput. Sci., 2 (2011), pp. 272278.

[4] J. W. Demmel, Applied Numerical Linear Algebra, SIAM, Philadelphia, PA, 1997.

[5] J. G. F. Francis, The QR transformation Parts 1 and 2, Comput. J., 4 (1961-

1962), pp. 265271 and 332345.

[6] G. H. Golub and C. F. van Loan, Matrix Computations, The Johns Hopkins
University Press, Baltimore, MD, 2nd ed., 1989.

[7] B. K agstro m, D. Kressner, and M. Shao, On aggressive early deflation in

parallel variants of the QR algorithm, in Applied Parallel and Scientific Computing
(PARA 2010), K. J onasson, ed., Heidelberg, 2012, Springer, pp. 110. (Lecture Notes
in Computer Science, 7133).

[8] A. Marek, V. Blum, R. Johanni, V. Havu, B. Lang, T. Auckenthaler,

A. Heinecke, H.-J. Bungartz, and H. Lederer, The ELPA library: scalable
parallel eigenvalue solutions for electronic structure theory and computational science,
J. Phys.: Condens. Matter, 26 (2014), p. 213201.

[9] B. N. Parlett, The QR algorithm, Computing Sci. Eng., 2 (2000), pp. 3842.
BIBLIOGRAPHY 89
[10] H. Rutishauser, Solution of eigenvalue problems with the LR-transformation, NBS
Appl. Math. Series, 49 (1958), pp. 4781.

[11] J. H. Wilkinson, The Algebraic Eigenvalue Problem, Clarendon Press, Oxford, 1965.
90 CHAPTER 4. THE QR ALGORITHM
Chapter 5

Cuppens Divide and Conquer

Algorithm

In this chapter we deal with an algorithm that is designed for the efficient solution of the
symmetric tridiagonal eigenvalue problem

a1 b1
..
b1 a2 .
(5.1) T x = x, T = .. ..
.

. . bn1
bn1 an
We noticed from Table 4.1 that the reduction of a full symmetric matrix to a similar tridi-
agonal matrix requires about 83 n3 while the tridiagonal QR algorithm needs an estimated
6n3 floating operations (flops) to converge. Because of the importance of this subproblem
a considerable effort has been put into finding faster algorithms than the QR algorithms
to solve the tridiagonal eigenvalue problem. In the mid-1980s Dongarra and Sorensen [4]
promoted an algorithm originally proposed by Cuppen [2]. This algorithm was based on
a divide and conquer strategy. However, it took ten more years until a stable variant was
found by Gu and Eisenstat [5, 6]. Today, a stable implementation of this latter algorithm
is available in LAPACK [1].

5.1 The divide and conquer idea

Divide and conquer is an old strategy in military to defeat an enemy going back at least to
Caesar. In computer science, divide and conquer (D&C) is an important algorithm design
paradigm. It works by recursively breaking down a problem into two or more subproblems
of the same (or related) type, until these become simple enough to be solved directly. The
solutions to the subproblems are then combined to give a solution to the original problem.
Translated to our problem the strategy becomes
1. Partition the tridiagonal eigenvalue problem into two (or more) smaller tridiagonal
eigenvalue problems.
2. Solve the two smaller problems.
3. Combine the solutions of the smaller problems to get the desired solution of the
overall problem.
Evidently, this strategy can be applied recursively.

91
92 CHAPTER 5. CUPPENS DIVIDE AND CONQUER ALGORITHM
5.2 Partitioning the tridiagonal matrix
Partitioning the irreducible tridiagonal matrix is done in the following way. We write
(5.2)
a 1 b1
..

.

b
1 a2


.. ..

. .


b m1


bm1 am bm


T =
b a b

m m+1 m+1



. .

b a .

m+1 m+2



.. ..
. . bn1



bn1 an

a1 b1
.

a2 . .

b
1


.. ..

. .

bm1



bm1 am bm bm bm


= +


am+1 bm bm+1 bm bm



.

am+2 . .

bm+1



.. ..


. . bn1



bn1 an
   
T1 em
= + uuT with u = and = bm ,
T2 e1

where em is a vector of length m n 2 and e1 is a vector of length n m. Notice that

the most straightforward way to partition the problem without modifying the diagonal
elements leads to a rank-two modification. With the approach of (5.2) we have the original
T as a sum of two smaller tridiagonal systems plus a rank-one modification.

5.3 Solving the small systems

We solve the half-sized eigenvalue problems,
(5.3) Ti = Qi i QTi , QTi Qi = I, i = 1, 2.

These two spectral decompositions can be computed by any algorithm, in particular also
by this divide and conquer algorithm by which the Ti would be further split. It is clear
that by this partitioning an large number of small problems can be generated that can be
potentially solved in parallel. For a parallel algorithm, however, the further phases of the
algorithm must be parallelizable as well.
Plugging (5.3) into (5.2) gives
 T       
Q1 T1 T Q1 1
(5.4) + uu = + vvT
QT2 T2 Q2 2
with
     
QT1 QT1 em last row of Q1
(5.5) v= u= = .
QT2 QT2 e1 first row of Q2
5.4. DEFLATION 93
Now we have arrived at the eigenvalue problem
(5.6) (D + vvT )x = x, D = 1 2 = diag(1 , . . . , n ).
That is, we have to compute the spectral decomposition of a matrix that is a diagonal
plus a rank-one update. Let
(5.7) D + vvT = QQT
be this spectral decomposition. Then, the spectral decomposition of the tridiagonal T is
   T 
Q1 T Q1
(5.8) T = QQ .
Q2 QT2
Forming the product (Q1 Q2 )Q will turn out to be the most expensive step of the
algorithm. It costs n3 + O(n2 ) floating point operations

5.4 Deflation
There are certain solutions of (5.7) that can be given immediately, by just looking carefully
at the equation.
If there are zero entries in v then we have


(5.9) vi = 0 v T e i = 0 = (D + vvT )ei = di ei .
Thus, if an entry of v vanishes we can read the eigenvalue from the diagonal of D at once
and the corresponding eigenvector is a coordinate vector.
If identical entries occur in the diagonal of D, say di = dj , with i < j, then we can find
a plane rotation G(i, j, ) (see (4.4)) such that it introduces a zero into the j-th position
of v,

..

p .
vi 2 + vj 2
i
..
GT v = G(i, j, )T v = .

0 j

..
.

Notice, that (for any ),
G(i, j, )T DG(i, j, ) = D, di = dj .
So, if there are multiple eigenvalues in D we can reduce all but one of them by introducing
zeros in v and then proceed as previously in (5.9).
When working with floating point numbers we deflate if
(5.10) |vi | < CkT k or |di dj | < CkT k, (kT k = kD + vvT k)
where C is a small constant. Deflation changes the eigenvalue problem for D + vvT into
the eigenvalue problem for
 
D1 + v1 v1T O p
(5.11) = GT (D + vvT )G + E, kEk < C kDk2 + ||2 kvk4 ,
O D2
where D1 has no multiple diagonal entries and v1 has no zero entries. So, we have to
compute the spectral decomposition of the matrix in (5.11) which is similar to a slight
perturbation of the original matrix. G is the product of Givens rotations.
94 CHAPTER 5. CUPPENS DIVIDE AND CONQUER ALGORITHM
5.4.1 Numerical examples

Let us first consider

1 1 1 1 0
1 2 1 1 2 1 0

1 3 1 1 2 0 1 1
T =

=

+



1 4 1 0 3 1 1 1
1 5 1 1 5 1 0
1 6 1 6 0
T
1 1 0 0
1 2 1 0 0

1 2 1 1
=

+



= T0 + uuT .

3 1 1 1
1 5 1 0 0
1 6 0 0

Then a little Matlab experiment shows that

T
0.1981 0.3280 0.3280
1.5550 0.7370 0.7370

3.2470 0.5910 0.5910
QT0 T Q0 =

+





2.5395 0.9018 0.9018
4.7609 0.4042 0.4042
6.6996 0.1531 0.1531

with

0.7370 0.5910 0.3280
0.5910 0.3280 0.7370

0.3280 0.7370 0.5910
Q0 =


0.9018 0.4153 0.1200

0.4042 0.7118 0.5744
0.1531 0.5665 0.8097

Here it is not possible to deflate.

Let us now look at an example with more symmetry,

2 1 2 1 0
1 2 1 1 2 1 0

1 2 1 1 1 0 1 1
T =

=

+



1 2 1 0 1 1 1 1
1 2 1 1 2 1 0
1 2 1 2 0
T
2 1 0 0
1 2 1 0 0

1 1 1 1
=

+



= T0 + uuT .

1 1 1 1
1 2 1 0 0
1 2 0 0
5.5. THE EIGENVALUE PROBLEM FOR D + VVT 95
Now, Matlab gives
T
0.1981 0.7370 0.7370
1.5550 0.5910 0.5910

3.2470 0.3280 0.3280
Q0 T Q0 =
T

+





0.1981 0.7370 0.7370
1.5550 0.5910 0.5910
3.2470 0.3280 0.3280

with
0.3280 0.7370 0.5910
0.5910 0.3280 0.7370

0.7370 0.5910 0.3280
Q0 =


0.7370 0.5910 0.3280

0.5910 0.3280 0.7370
0.3280 0.7370 0.5910
In this example we have three double eigenvalues. Because the corresponding components
of v (vi and vi+1 ) are equal we define
G = G(1, 4, /4)G(2, 5, /4)G(3, 6, /4)

0.7071 0.7071
0.7071 0.7071

0.7071 0.7071
= 0.7071
.

0.7071
0.7071 0.7071
0.7071 0.7071

Then,

GT QT0 T Q0 G = GT QT0 T0 Q0 G + GT v(GT v)T = D + GT v(GT v)T

T
0.1981 1.0422 1.0422
1.5550 0.8358 0.8358

3.2470 0.4638 0.4638
=
0.1981 + 0.0000 0.0000

1.5550 0.0000 0.0000
3.2470 0.0000 0.0000

Therefore, (in this example) e4 , e5 , and e6 are eigenvectors of

D + GT v(GT v)T = D + GT vvT G

corresponding to the eigenvalues d4 , d5 , and d6 , respectively. The eigenvectors of T corre-
sponding to these three eigenvalues are the last three columns of

0.2319 0.4179 0.5211 0.5211 0.4179 0.2319
0.5211 0.2319 0.4179 0.4179 0.2319 0.5211

0.4179 0.5211 0.2319 0.2319 0.5211 0.4179
Q0 G =

.

0.2319 0.4179 0.5211 0.5211 0.4179 0.2319
0.5211 0.2319 0.4179 0.4179 0.2319 0.5211
0.4179 0.5211 0.2319 0.2319 0.5211 0.4179

5.5 The eigenvalue problem for D + vvT

We know that 6= 0. Otherwise there is nothing to be done. Furthermore, after deflation,
we know that all elements of v are nonzero and that the diagonal elements of D are all
96 CHAPTER 5. CUPPENS DIVIDE AND CONQUER ALGORITHM
distinct, in fact,
|di dj | > CkT k.
We order the diagonal elements of D such that

d1 < d2 < < dn .

Notice that this procedure permutes the elements of v as well. Let (, x) be an eigenpair
of

(5.12) (D + vvT )x = x.

Then,

(5.13) (D I)x = vvT x.

cannot be equal to one of the di . If = dk then the k-th element on the left of (5.13)
vanishes. But then either vk = 0 or vT x = 0. The first cannot be true for our assumption
about v. If on the other hand vT x = 0 then (D dk I)x = 0. Thus x = ek and
vT ek = vk = 0, which cannot be true. Therefore D I is nonsingular and

(5.14) x = (I D)1 v(vT x).

This equation shows that x is proportional to (I D)1 v. If we require kxk = 1 then

(I D)1 v
(5.15) x= .
k(I D)1 vk

Multiplying (5.14) by vT from the left we get

(5.16) vT x = vT (I D)1 v(vT x).

Since vT x 6= 0, is an eigenvalue of (5.12) if and only if

10

1
0

10
2 0 1 3 3.5 7 8 10

1 0.22 0.62 0.52 0.92 0.82

Figure 5.1: Graph of 1 + 0 + 1 + 3 + 3.5 + 7 + 8

P
n
vk2
(5.17) f () := 1 vT (I D)1 v = 1 = 0.
k=1
dk
5.5. THE EIGENVALUE PROBLEM FOR D + VVT 97
This equation is called secular equation. The secular equation has poles at the eigen-
values of D and zeros at the eigenvalues of D + vvT . Notice that
n
X vk2
f () = 2.
k=1
( d k )

Thus, the derivative of f is positive if > 0 wherever it has a finite value. If < 0
the derivative of f is negative (almost) everywhere. A typical graph of f with > 0 is
depicted in Fig. 5.1. (If is negative the image can be flipped left to right.) The secular
equation implies the interlacing property of the eigenvalues of D and of D + vvT ,

(5.18) d1 < 1 < d2 < 2 < < dn < n , > 0.

or

(5.19) 1 < d1 < 2 < d2 < < n < dn , < 0.

So, we have to compute one eigenvalue in each of the intervals (di , di+1 ), 1 i < n, and
a further eigenvalue in (dn , ) or (, d1 ). The corresponding eigenvector is then given
by (5.15). Evidently, these tasks are easy to parallelize.
Equations (5.17) and (5.15) can also been obtained from the relations
" #  " 1 # 
1 v T 1 0T 0T 1 vT
=
v I D v I 0 I D vvT 0 I
  " # 
1 vT (I D)1 1 vT (I D)1 v 0T 1 0T
= .
0 I 0 I D (I D)1 v I

These are simply block LDLT factorizations of the first matrix. The first is the well-known
one where the factorization is started with the (1, 1) block. The second is a backward fac-
torization that is started with the (2, 2) block. Because the determinants of the tridiagonal
matrices are all unity, we have
1 1
(5.20) det(I D vvT ) = (1 vT (I D)1 v) det(I D).

Denoting the eigenvalues of D + vvT again by 1 < 2 < < n this implies
n
Y n
Y
( j ) = (1 vT (I D)1 v) ( dj )
j=1 j=1
n
! n
X vk2 Y
(5.21) = 1 ( dj )
dk
k=1 j=1
n
Y n
X Y
= ( dj ) vk2 ( dj )
j=1 k=1 j6=k

Setting = dk gives
n
Y n
Y
(5.22) (dk j ) = vk2 (dk dj )
j=1 j=1
j6=i
98 CHAPTER 5. CUPPENS DIVIDE AND CONQUER ALGORITHM
or
Q
n Q
k1 Q
n
(dk j ) (j dk )(1)nk+1
(dk j )
1 j=1 1 j=1 j=k
vk2 = =
Q n
k1
Q Q
n
(dk dj ) (dk dj ) (dj dk )(1)nk
j=1 j=1 j=k+1
j6=i
(5.23)
k1
Q Q
n
(dk j ) (j dk )
1 j=1 j=k
= k1 > 0.
Q Q
n
(dk dj ) (dj d k )
j=1 j=k+1

Therefore, the quantity on the right side is positive, so

v
u k1
u Q Q
n
u (dk j ) (j dk )
u j=1 j=k
(5.24) u
vk = u .
t k1Q Qn
(dk dj ) (dj dk )
j=1 j=k+1

(Similar arguments hold if < 0.) Thus, we have the solution of the following inverse
eigenvalue problem:
Given D = diag(d1 , . . . , dn ) and values 1 , . . . , n that satisfy (5.18). Find a vector
v = [v1 , . . . , vn ]T with positive components vk such that the matrix D + vvT has the
prescribed eigenvalues 1 , . . . , n .
The solution is given by (5.24). The positivity of the vk makes the solution unique.

5.6 Solving the secular equation

In this section we follow closely the exposition of Demmel [3]. We consider the computation
of the zero of f () in the interval (di , di+1 ). We assume that = 1.
We may simply apply Newtons iteration to solve f () = 0. However, if we look
carefully at Fig. 5.1 then we notice that the tangent at certain points in (di , di+1 ) crosses
the real axis outside this interval. This happens in particular if the weights vi or vi+1 are
small. Therefore that zero finder has to be adapted in such a way that it captures the
poles at the interval endpoints. It is relatively straightforward to try the ansatz
c1 c2
(5.25) h() = + + c3 .
di di+1
Notice that, given the coefficients c1 , c2 , and c3 , the equation h() = 0 can easily be solved
by means of the equivalent quadratic equation
(5.26) c1 (di+1 ) + c2 (di ) + c3 (di )(di+1 ) = 0.
This equation has two zeros. Precisly one of them is inside (di , di+1 ).
The coefficients c1 , c2 , and c3 are computed in the following way. Let us assume
that we have available an approximation j to the zero in (di , di+1 ). We request that
h(j ) = f (j ) and h (j ) = f (j ). The exact procedure is as follows. We write
i
X Xn
vk2 vk2
(5.27) f () = 1 + + = 1 + 1 () + 2 ().
d d
k=1 k k=i+1 k
| {z } | {z }
1 () 2 ()
5.7. A FIRST ALGORITHM 99
1 () is a sum of positive terms and 2 () is a sum of negative terms. Both 1 () and
2 () can be computed accurately, whereas adding them would likely provoke cancellation
and loss of relative accuracy. We now choose c1 and c1 such that

c1
(5.28) h1 () := c1 + satisfies h1 (j ) = 1 (j ) and h1 (j ) = 1 (j ).
di

This means that the graphs of h1 and of 1 are tangent at = j . This is similar to
Newtons method. However in Newtons method a straight line is fitted to the given
function. The coefficients in (5.28) are given by

c1 = 1 (j )(di j )2 > 0,
i
X dk di
c1 = 1 (j ) 1 (j )(di j ) = vk2 0.
k=1
(dk j )2

Similarly, the two constants c2 and c2 are determined such that

c2
(5.29) h2 () := c2 + satisfies h2 (j ) = 2 (j ) and h2 (j ) = 2 (j )
di+1

with the coefficients

c2 = 2 (j )(di+1 j )2 > 0,
Xn
dk di+1
c2 = 2 (j ) 2 (j )(di+1 j ) = vk2 2 0.
k=i+1
(dk )

Finally, we set

c1 c2
(5.30) h() = 1 + h1 () + h2 () = (1 + c1 + c2 ) + + .
| {z } di di+1
c3

This zerofinder is converging quadratically to the desired zero [7]. Usually 2 to 3 steps
are sufficient to get the zero to machine precision. Therefore finding a zero only requires
O(n) flops. Thus, finding all zeros costs O(n2 ) floating point operations.

5.7 A first algorithm

We are now ready to give the divide and conquer algorithm, see Algorithm 5.1.
All steps except step 10 require O(n2 ) operations to complete. The step 10 costs n3 flops.
Thus, the full divide and conquer algorithm, requires
 n 3
T (n) = n3 + 2 T (n/2) = n3 + 2 + 4T (n/4)
2
(5.31)  n 3
n3 4
= n3 + +4 + 8T (n/8) = = n3 .
4 4 3

This serial complexity of the algorithm very often overestimates the computational costs
of the algorithm due to significant deflation that is observed surprisingly often.
100 CHAPTER 5. CUPPENS DIVIDE AND CONQUER ALGORITHM
Algorithm 5.1 The tridiagonal divide and conquer algorithm
1: Let T Cnn be a real symmetric tridiagonal matrix. This algorithm computes
the spectral decomposition of T = QQT , where the diagonal is the matrix of
eigenvalues and Q is orthogonal.
2: if T is 1 1 then
3: return ( = T ; Q = 1)
4: else  
T1 O
5: Partition T = + uuT according to (5.2)
O T2
6: Call this algorithm with T1 as input and Q1 , 1 as output.
7: Call this algorithm with T2 as input and Q2 , 2 as output.
8: Form D + vvT from 1 , 2 , Q1 , Q2 according to (5.4)(5.6).
9: Find the eigenvalues
  and the eigenvectors Q of D + vvT .
Q1 O
10: Form Q = Q which are the eigenvectors of T .
O Q2
11: return (; Q)
12: end if

5.7.1 A numerical example

Let A be a 4 4 matrix

0 1
2  
(5.32) A = D + vvT =

+

1 1 .

2+
5 1
In this example (that is similar to one in [8]) we want to point at a problem that the
divide and conquer algorithm possesses as it is given in Algorithm 5.1, namely the loss of
orthogonality among eigenvectors.
Before we do some Matlab tests let us look more closely at D and v in (5.32). This
example becomes difficult to solve if gets very small. In Figures 5.2 to 5.5 we see
graphs of the function f () that appears in the secular equation for = 1, = 0.1, and
= 0.01. The critical zeros move towards 2 from both sides. The weights v22 = v32 = 2
are however not so small that they should be deflated.
The following Matlab code shows the problem. We execute the commands for =
k
10 for k = 0, 1, 2, 4, 8.
v = [1 beta beta 1]; % rank-1 modification
d = [0, 2-beta, 2+beta, 5]; % diagonal matrix

L = eig(diag(d) + v*v) % eigenvalues of the modified matrix

e = ones(4,1);
q = (d*e-e*L).\(v*e); % unnormalized eigenvectors cf. (5.15)

Q = sqrt(diag(q*q));
q = q./(e*Q); % normalized eigenvectors

norm(q*q-eye(4)) % check for orthogonality

We do not bother how we compute the eigenvalues. We simply use Matlabs built-in
function eig. We get the results of Table 5.1.
5.7. A FIRST ALGORITHM 101
10

10
2 1 0 1 2 3 4 5 6 7

Figure 5.2: Secular equation corresponding to (5.32) for = 1

10

10
2 1 0 1 2 3 4 5 6 7

Figure 5.3: Secular equation corresponding to (5.32) for = 0.1

We observe loss of orthogonality among the eigenvectors as the eigenvalues get closer
and closer. This may not be surprising as we compute the eigenvectors by formula (5.15)

(I D)1 v
x= .
k(I D)1 vk
If = 2 and = 3 which are almost equal, 2 3 then intuitively one expects almost
the same eigenvectors. We have in fact

2.2204 1016 4.3553 108 1.7955 108 1.1102 1016
4.3553 108 0 5.5511 108 1.8298 108
Q T Q I4 =
1.7955 10
.
8 5.5511 108 1.1102 1016 7.5437 109
1.1102 1016 1.8298 108 7.5437 109 0

Orthogonality is lost only with respect to the vectors corresponding to the eigenvalues
close to 2.
102 CHAPTER 5. CUPPENS DIVIDE AND CONQUER ALGORITHM
10

10
1 1.5 2 2.5 3

Figure 5.4: Secular equation corresponding to (5.32) for = 0.1 for 1 3

10

10
1.9 1.95 2 2.05 2.1

Figure 5.5: Secular equation corresponding to (5.32) for = 0.01 for 1.9 2.1

Already Dongarra and Sorensen [4] analyzed this problem. In their formulation they
normalize the vector v of D + vvT to have norm unity, kvk = 1. They stated

Lemma 5.1 Let

  1/2
v1 v2 vn
(5.33) qT = , ,...,
.
d1 d2 dn f ()

Then for any , 6 {d1 , . . . , dn } we have

1 |f () f ()|
(5.34) |qT q | = .
| | [f ()f ()]1/2

Proof. Observe that

1 1
= .
(dj )(dj ) dj dj
Then the proof is straightforward.
5.8. THE ALGORITHM OF GU AND EISENSTAT 103
1 2 3 4 kQT Q Ik
1 0.325651 1.682219 3.815197 7.176933 5.6674 1016
0.1 0.797024 1.911712 2.112111 6.199153 3.4286 1015
0.01 0.807312 1.990120 2.010120 6.192648 3.9085 1014
104 0.807418 1.999900 2.000100 6.192582 5.6767 1012
108 0.807418 1.99999999000000 2.00000001000000 6.192582 8.3188 1008

Table 5.1: Loss of orthogonality among the eigenvectors computed by (5.15)

Formula (5.34) indicates how problems may arise. In exact arithmetic, if and are
eigenvalues then f () = f () = 0. However, in floating point arithmetic this values may
be small but nonzero, e.g., O(). If | | is very small as well then we may have trouble!
So, a remedy for the problem was for a long time to compute the eigenvalues in doubled
precision, so that f () = O(2 ). This would counteract a potential O() of | |.
This solution was quite unsatisfactory because doubled precision is in general very slow
since it is implemented in software. It took a decade until a proper solution was found.

5.8 The algorithm of Gu and Eisenstat

Computing eigenvector according to the formula
v1
. d1
(5.35) x = (I D)1 v =
.. ,
= k(I D)1 vk,
vn
dn
is bound to fail if is very close to a pole dk and the difference dk has an error of size
O(|dk |) instead of only O(|dk |). To resolve this problem Gu and Eisenstat [5] found
a trick that is at the same time ingenious and simple.
They observed that the vk in (5.24) are very accurately determined by the data di and
i . Therefore, once the eigenvalues are computed accurately a vector v could be computed
such that the i are accurate eigenvalues of D + v v. If v
approximates well the original v
then the new eigenvectors will be the exact eigenvectors of a slightly modified eigenvalue
problem, which is all we can hope for.
The zeros of the secular equation can be computed accurately by the method presented
in section 5.6. However, a shift of variables is necessary. In the interval (di , di+1 ) the origin
of the real axis is moved to di if i is closer to di than to di+1 , i.e., if f ((di + di+1 )/2) > 0.
Otherwise, the origin is shifted to di+1 . This shift of the origin avoids the computation
of the smallest difference di (or di+1 ) in (5.35), thus avoiding cancellation in this
most sensitive quantity. Equation (5.26) can be rewritten as

(5.36) (c + c2 i + c3 i i+1 ) (c1 + c2 + c3 (i + i+1 )) + c3 2 = 0,

| 1 i+1 {z } | {z } |{z}
b a c

where i = di j , i+1 = di+1 j , and j+1 = j + is the next approximate zero.

With equations (5.28)(5.30) the coefficients in (5.36) get

a = c1 + c2 + c3 (i + i+1 ) = (1 + 1 + 2 )(i + i+1 ) (1 + 2 )i i+1 ,

(5.37) b = c1 i+1 + c2 i + c3 i i+1 = i i+1 (1 + 1 + 2 ),
c = c3 = 1 + 1 + 2 i 1 i+1 2 .
104 CHAPTER 5. CUPPENS DIVIDE AND CONQUER ALGORITHM
If we are looking for a zero that is closer to di than to di+1 then we move the origin to j ,
i.e., we have e.g. i = j . The solution of (5.36) that lies inside the interval is [7]
p

a a2 4bc , if a 0,
2c
(5.38) =

p2b , if a > 0.
a + a2 4bc
The following algorithm shows how step 9 of the tridiagonal divide and conquer algo-
rithm 5.1 must be implemented.

Algorithm 5.2 A stable eigensolver for D + vvT

1: This algorithm stably computes the spectral decomposition of D+vvT = QQT where
D = diag(d1 , . . . dn ), v = [v1 , . . . , vn ] Rn , = diag(1 , . . . n ), and Q = [q1 , . . . , qn ].
2: di+1 = dn + kvk2 .
3: In each interval (di , di+1 ) compute the zero i of the secular equation f () = 0.
4: Use the formula (5.24) to compute the vector v such that the i are the exact
eigenvalues of D + v v.
5: In each interval (di , di+1 ) compute the eigenvectors of D + v v
according to (5.15),

(i I D)1 v

qi = .
k(i I D)1 v
k

6: return (; Q)

5.8.1 A numerical example [continued]

We continue the discussion of the example on page 100 where the eigenvalue problem of

0 1
2  
(5.39) A = D + vvT =
+

1 1 .

2+
5 1

The Matlab code that we showed did not give orthogonal eigenvectors. We show in
the following script that the formulae (5.24) really solve the problem.

dlam = zeros(n,1);
for k=1:n,
[dlam(k), dvec(:,k)] = zerodandc(d,v,k);
end

V = ones(n,1);
for k=1:n,
V(k) = prod(abs(dvec(k,:)))/prod(d(k) - d(1:k-1))/prod(d(k+1:n) - d(k));
V(k) = sqrt(V(k));
end

Q = (dvec).\(V*e);
diagq = sqrt(diag(Q*Q));
Q = Q./(e*diagq);
5.8. THE ALGORITHM OF GU AND EISENSTAT 105
for k=1:n,
if dlam(k)>0,
dlam(k) = dlam(k) + d(k);
else
dlam(k) = d(k+1) + dlam(k);
end
end

norm(Q*Q-eye(n))
norm((diag(d) + v*v)*Q - Q*diag(dlam))

A zero finder returns for each interval the quantity i di and the vector [d1
i , . . . , dn i ]T to high precision. These vector elements have been computed as (dk
di ) (i di ). The zerofinder of Li [7] has been employed here. At the end of this sec-
tion we list the zerofinder written in Matlab that was used here. The formulae (5.37)
and (5.38) have been used to solve the quadratic equation (5.36). Notice that only one of
the while loops is traversed, depending on if the zero is closer to the pole on the left or
to the right of the interval. The vk of formula (5.24) are computed next. Q contains the
eigenvectors.

Algorithm kQT Q Ik kAQ Qk

0.1 I 3.4286 1015 5.9460 1015
II 2.2870 1016 9.4180 1016
0.01 I 3.9085 1014 6.9376 1014
II 5.5529 1016 5.1630 1016
104 I 5.6767 1012 6.3818 1012
II 2.2434 1016 4.4409 1016
108 I 8.3188 1008 1.0021 1007
II 2.4980 1016 9.4133 1016

Table 5.2: Loss of orthogonality among the eigenvectors computed by the straightforward
algorithm (I) and the Gu-Eisenstat approach (II)

Again we ran the code for = 10k for k = 0, 1, 2, 4, 8. The numbers in Table 5.2
confirm that the new formulae are much more accurate than the straight forward ones.
The norms of the errors obtained for the Gu-Eisenstat algorithm always are in the order
of machine precision, i.e., 1016 .
Note that there may be errors in the following code! The vector v is squared on the
3rd line, and later (assigning values to di1, psi1, psi2) squared again!

function [lambda,dl] = zerodandc(d,v,i)

% ZERODANDC - Computes eigenvalue lambda in the i-th interval
% (d(i), d(i+1)) with Lis middle way zero finder
% dl is the n-vector [d(1..n) - lambda]

n = length(d);
di = d(i);
v = v.^2;
if i < n,
di1 = d(i+1); lambda = (di + di1)/2;
else
di1 = d(n) + norm(v)^2; lambda = di1;
106 CHAPTER 5. CUPPENS DIVIDE AND CONQUER ALGORITHM
end
eta = 1;
psi1 = sum((v(1:i).^2)./(d(1:i) - lambda));
psi2 = sum((v(i+1:n).^2)./(d(i+1:n) - lambda));

if 1 + psi1 + psi2 > 0, % zero is on the left half of the interval

d = d - di; lambda = lambda - di; di1 = di1 - di; di = 0;

while abs(eta) > 10*eps

psi1 = sum(v(1:i)./(d(1:i) - lambda));
psi1s = sum(v(1:i)./((d(1:i) - lambda)).^2);

psi2 = sum((v(i+1:n))./(d(i+1:n) - lambda));

psi2s = sum(v(i+1:n)./((d(i+1:n) - lambda)).^2);

% Solve for zero

Di = -lambda; Di1 = di1 - lambda;
a = (Di + Di1)*(1 + psi1 + psi2) - Di*Di1*(psi1s + psi2s);
b = Di*Di1*(1 + psi1 + psi2);
c = (1 + psi1 + psi2) - Di*psi1s - Di1*psi2s;
if a > 0,
eta = (2*b)/(a + sqrt(a^2 - 4*b*c));
else
eta = (a - sqrt(a^2 - 4*b*c))/(2*c);
end
lambda = lambda + eta;
end

else % zero is on the right half of the interval

d = d - di1; lambda = lambda - di1; di = di - di1; di1 = 0;

while abs(eta) > 10*eps

psi1 = sum(v(1:i)./(d(1:i) - lambda));
psi1s = sum(v(1:i)./((d(1:i) - lambda)).^2);

psi2 = sum((v(i+1:n))./(d(i+1:n) - lambda));

psi2s = sum(v(i+1:n)./((d(i+1:n) - lambda)).^2);

% Solve for zero

Di = di - lambda; Di1 = - lambda;
a = (Di + Di1)*(1 + psi1 + psi2) - Di*Di1*(psi1s + psi2s);
b = Di*Di1*(1 + psi1 + psi2);
c = (1 + psi1 + psi2) - Di*psi1s - Di1*psi2s;
if a > 0,
eta = (2*b)/(a + sqrt(a^2 - 4*b*c));
else
eta = (a - sqrt(a^2 - 4*b*c))/(2*c);
end
lambda = lambda + eta;
end

end
BIBLIOGRAPHY 107
dl = d - lambda;

return

Bibliography
[1] E. Anderson, Z. Bai, C. Bischof, J. Demmel, J. Dongarra, J. D.
Croz, A. Greenbaum, S. Hammarling, A. McKenney, S. Ostrouchov,
and D. Sorensen, LAPACK Users Guide - Release 2.0, SIAM, Philadel-
phia, PA, 1994. (Software and guide are available from Netlib at URL
http://www.netlib.org/lapack/).

[2] J. J. M. Cuppen, A divide and conquer method for the symmetric tridiagonal eigen-
problem, Numer. Math., 36 (1981), pp. 177195.

[3] J. W. Demmel, Applied Numerical Linear Algebra, SIAM, Philadelphia, PA, 1997.

[4] J. J. Dongarra and D. C. Sorensen, A fully parallel algorithm for the symmetric
eigenvalue problem, SIAM J. Sci. Stat. Comput., 8 (1987), pp. s139s154.

[5] M. Gu and S. C. Eisenstat, A stable and efficient algorithm for the rank-one
modification of the symmetric eigenproblem, SIAM J. Matrix Anal. Appl., 15 (1994),
pp. 12661276.

[6] , A divide-and-conquer algorithm for the symmetric tridiagonal eigenproblem,

SIAM J. Matrix Anal. Appl., 16 (1995), pp. 172191.

[7] R.-C. Li, Solving secular equations stably and efficiently, Technical Report UT-CS-
94-260, University of Tennessee, Knoxville, TN, Nov. 1994. LAPACK Working Note
No. 89.

[8] D. C. Sorensen and P. T. P. Tang, On the orthogonality of eigenvectors computed

by divide-and-conquer techniques, SIAM J. Numer. Anal., 28 (1991), pp. 17521775.
108 CHAPTER 5. CUPPENS DIVIDE AND CONQUER ALGORITHM
Chapter 6

LAPACK and the BLAS

6.1 LAPACK
(This section is essentially compiled from the LAPACK Users Guide [1] that is available
online from http://www.netlib.org/lapack/lug/.)
LAPACK [1] is a library of Fortran 77 subroutines for solving the most commonly
occurring problems in numerical linear algebra. It has been designed to be efficient on a
wide range of modern high-performance computers. The name LAPACK is an acronym
for Linear Algebra PACKage.
LAPACK can solve systems of linear equations, linear least squares problems, eigen-
value problems and singular value problems. LAPACK can also handle many associated
computations such as matrix factorizations or estimating condition numbers.
LAPACK contains driver routines for solving standard types of problems, compu-
tational routines to perform a distinct computational task, and auxiliary routines to
perform a certain subtask or common low-level computation. Each driver routine typically
calls a sequence of computational routines. Taken as a whole, the computational routines
can perform a wider range of tasks than are covered by the driver routines. Many of the
auxiliary routines may be of use to numerical analysts or software developers, so we have
documented the Fortran source for these routines with the same level of detail used for
the LAPACK routines and driver routines.
Dense and banded matrices are provided for, but not general sparse matrices. In all
areas, similar functionality is provided for real and complex matrices.
LAPACK is designed to give high efficiency on vector processors, high-performance
super-scalar workstations, and shared memory multiprocessors. It can also be used sat-
isfactorily on all types of scalar machines (PCs, workstations, mainframes). A distributed-
memory version of LAPACK, ScaLAPACK [2], has been developed for other types of
parallel architectures (for example, massively parallel SIMD machines, or distributed mem-
ory machines).
LAPACK has been designed to supersede LINPACK [3] and EISPACK [10, 8], princi-
pally by restructuring the software to achieve much greater efficiency, where possible, on
modern high-performance computers; also by adding extra functionality, by using some
new or improved algorithms, and by integrating the two sets of algorithms into a unified
package.
LAPACK routines are written so that as much as possible of the computation is per-
formed by calls to the Basic Linear Algebra Subprograms (BLAS) [9, 6, 5]. Highly
efficient machine-specific implementations of the BLAS are available for many modern

109
110 CHAPTER 6. LAPACK AND THE BLAS
high-performance computers. The BLAS enable LAPACK routines to achieve high per-
formance with portable code.
The BLAS are not strictly speaking part of LAPACK, but Fortran 77 code for the
BLAS is distributed with LAPACK, or can be obtained separately from netlib where
model implementations are found.
The model implementation is not expected to perform as well as a specially tuned
implementation on most high-performance computers on some machines it may give
much worse performance but it allows users to run LAPACK codes on machines that do
not offer any other implementation of the BLAS.
The complete LAPACK package or individual routines from LAPACK are freely avail-
able from the World Wide Web or by anonymous ftp. The LAPACK homepage can be
accessed via the URL http://www.netlib.org/lapack/.

6.2 BLAS
By 1976 it was clear that some standardization of basic computer operations on vectors
was needed [9]. By then it was already known that coding procedures that worked well
on one machine might work very poorly on others. In consequence of these observations,
Lawson, Hanson, Kincaid and Krogh proposed a limited set of Basic Linear Algebra
Subprograms (BLAS) to be (hopefully) optimized by hardware vendors, implemented in
assembly language if necessary, that would form the basis of comprehensive linear alge-
bra packages [9]. These so-called Level 1 BLAS consisted of vector operations and some
attendant co-routines. The first major package which used these BLAS kernels was LIN-
PACK [3]. Soon afterward, other major software libraries such as the IMSL library and
NAG rewrote portions of their existing codes and structured new routines to use these
BLAS. Early in their development, vector computers saw significant optimizations us-
ing the BLAS. Soon, however, such machines were clustered together in tight networks
and somewhat larger kernels for numerical linear algebra were developed [6, 7] to include
matrix-vector operations (Level 2 BLAS). Additionally, Fortran compilers were by then
optimizing vector operations as efficiently as hand coded Level 1 BLAS. Subsequently,
in the late 1980s, distributed memory machines were in production and shared memory
machines began to have significant numbers of processors. A further set of matrix-matrix
operations was proposed [4] and soon standardized [5] to form a Level 3. The first major
package for linear algebra which used the Level 3 BLAS was LAPACK [1] and subsequently
a scalable (to large numbers of processors) version was released as ScaLAPACK [2]. Ven-
dors focused on Level 1, Level 2, and Level 3 BLAS which provided an easy route to
optimizing LINPACK, then LAPACK. LAPACK not only integrated pre-existing solvers
and eigenvalue routines found in EISPACK [10] (which did not use the BLAS) and LIN-
PACK (which used Level 1 BLAS), but incorporated the latest dense and banded linear
algebra algorithms available. It also used the Level 3 BLAS which were optimized by much
vendor effort. Later, we will illustrate several BLAS routines. Conventions for different
BLAS are indicated by
A root operation. For example, axpy for the operation

(6.1) y := ax + y

A prefix (or combination prefix) to indicate the datatype of the operands, for example
saxpy for single precision axpy operation, or isamax for the index of the maximum
absolute element in an array of type single.
6.2. BLAS 111
a suffix if there is some qualifier, for example cdotc or cdotu for conjugated or
unconjugated complex dot product, respectively:
n1
X
cdotc(n,x,1,y,1) = xi yi
i=0
n1
X
cdotu(n,x,1,y,1) = xi y i
i=0

where both x, y are vectors of complex elements.

Tables 6.1 and 6.2 give the prefix/suffix and root combinations for the BLAS, respectively.

Prefixes:
S REAL
D DOUBLE PRECISION
C COMPLEX
Z DOUBLE COMPLEX
Suffixes:
U transpose
C Hermitian conjugate

Table 6.1: Basic Linear Algebra Subprogram prefix/suffix conventions.

6.2.1 Typical performance numbers for the BLAS

Let us look at typical representations of all three levels of the BLAS, daxpy, ddot, dgemv,
and dgemm, that perform some basic operations. Additionally, we look at the rank-1
update routine dger. An overview on the number of memory accesses and floating point
operations is given in Table 6.3. The Level 1 BLAS comprise basic vector operations. A
call of one of the Level 1 BLAS thus gives rise to O(n) floating point operations and O(n)
memory accesses. Here, n is the vector length. The Level 2 BLAS comprise operations
that involve matrices and vectors. If the involved matrix is n-by-n then both the memory
accesses and the floating point operations are of O(n2 ). In contrast, the Level 3 BLAS have
a higher order of floating point operations than memory accesses. The most prominent
operation of the Level 3 BLAS, matrix-matrix multiplication costs O(n3 ) floating point
operations while there are only O(n2 ) reads and writes. The last column in Table 6.3
shows the crucial difference between the Level 3 BLAS and the rest.
Table 6.4 gives some performance numbers for the five BLAS of Table 6.3. Notice that
the timer has a resolution of only 1 sec! Therefore, the numbers in Table 6.4 have been
obtained by timing a loop inside of which the respective function is called many times.
The Mflop/s rates of the Level 1 BLAS ddot and daxpy quite precisely reflect the ratios
of the memory accesses of the two routines, 2n vs. 3n. The high rates are for vectors that
can be held in the on-chip cache of 512 MB. The low 240 and 440 Mflop/s with the very
long vectors are related to the memory bandwidth of about 1900 MB/s.
The Level 2 BLAS dgemv has about the same performance as daxpy if the matrix can
be held in cache (n = 100). Otherwise it is considerably reduced. dger has a high volume
of read and write operations, while the number of floating point operations is limited.
112 CHAPTER 6. LAPACK AND THE BLAS

Level 1 BLAS

rotg, rot Generate/apply plane rotation

rotmg, rotm Generate/apply modified plane rotation
swap Swap two vectors: x y
scal Scale a vector: x x
copy Copy a vector: x y
axpy axpy operation: y y + x
dot Dot product: s x y = x y
nrm2 2-norm: s kxk2
asum 1-norm: s kxk1
i amax Index of largest vector element:
first i such |xi | |xk | for all k

Level 2 BLAS

gemv, gbmv General (banded) matrix-vector multiply:

y Ax + y
hemv, hbmv, hpmv Hermitian (banded, packed) matrix-vector
multiply: y Ax + y
semv, sbmv, spmv Symmetric (banded, packed) matrix-vector
multiply: y Ax + y
trmv, tbmv, tpmv Triangular (banded, packed) matrix-vector
multiply: x Ax
trsv, tbsv, tpsv Triangular (banded, packed) system solves
(forward/backward substitution): x A1 x
ger, geru, gerc Rank-1 updates: A xy + A
her, hpr, syr, spr Hermitian/symmetric (packed) rank-1 updates:
A xx + A
her2, hpr2, syr2, spr2 Hermitian/symmetric (packed) rank-2 updates:
A xy + yx + A

Level 3 BLAS

gemm, symm, hemm General/symmetric/Hermitian matrix-matrix

multiply: C AB + C
syrk, herk Symmetric/Hermitian rank-k update:
C AA + C
syr2k, her2k Symmetric/Hermitian rank-k update:
C AB + BA + C
trmm Multiple triangular matrix-vector multiplies:
B AB
trsm Multiple triangular system solves: B A1 B

Table 6.2: Summary of the Basic Linear Algebra Subroutines.

6.3. BLOCKING 113
read write flops flops / mem access
ddot 2n 1 2n 1
daxpy 2n n 2n 2/3
dgemv 2
n +n n 2n2 2
dger n2 + 2n n2 2n2 1
dgemm 2n2 n2 2n3 2n/3

Table 6.3: Number of memory references and floating point operations for vectors of length
n.

n = 100 500 2000 10000000

ddot 1480 1820 1900 440
daxpy 1160 1300 1140 240
dgemv 1370 740 670
dger 670 330 320
dgemm 2680 3470 3720

Table 6.4: Some performance numbers for typical BLAS in Mflop/s for a 2.4 GHz Pentium
4.

This leads to a very low performance rate. The Level 3 BLAS dgemm performs at a good
fraction of the peak performance of the processor (4.8Gflop/s). The performance increases
with the problem size. We see from Table 6.3 that the ratio of computation to memory
accesses increases with the problem size. This ratio is analogous to a volume to surface
area effect.

6.3 Blocking
In the previous section we have seen that it is important to use Level 3 BLAS. However, in
the algorithm we have treated so far, there were no blocks. For instance, in the reduction
to Hessenberg form we applied Householder (elementary) reflectors from left and right to
a matrix to introduce zeros in one of its columns.
The essential point here is to gather a number of reflectors to a single block transfor-
mation. Let Pi = I 2ui ui , i = 1, 2, 3, be three Householder reflectors. Their product
is

P = P3 P2 P1 = (I 2u3 u3 )(I 2u2 u2 )(I 2u1 u1 )

= I 2u3 u3 2u2 u2 2u1 u1 + 4u3 u3 u2 u2 + 4u3 u3 u1 u1 + 4u2 u2 u1 u1
+ 8u3 u3 u2 u2 u1 u1
(6.2)
2
= I [u1 u2 u3 ] 4u2 u1 2 [u1 u2 u3 ] .

4u3 u1 + 8(u3 u2 )(u2 u1 ) 4u3 u2 2

So, if e.g. three rotations are to be applied on a matrix in blocked fashon, then the three
Householder vectors u1 , u2 , u3 have to be found first. To that end the rotations are first
applied only on the first three columns of the matrix, see Fig. 6.1. Then, the blocked
rotation is applied to the rest of the matrix.
114 CHAPTER 6. LAPACK AND THE BLAS

Figure 6.1: Blocking Householder reflections

Remark 6.1. Notice that a similar situation holds for Gaussian elimination because

1 1 1
l21 1 1 l21 1

l31 1
l32 1 = l31 l32 1 .
.. .. .. .. .. .. ..
. . . . . . .
ln1 1 ln2 1 ln1 ln2 1

However, things are a complicated because of pivoting.

6.4 LAPACK solvers for the symmetric eigenproblems

To give a feeling how LAPACK is organized we consider solvers for the symmetric eigen-
problem (SEP). Except for this problem there are driver routines for linear systems, least
squares problems, nonsymmetric eigenvalue problems, the computation of the singular
value decomposition (SVD).
The basic task of the symmetric eigenproblem routines is to compute values of and,
optionally, corresponding vectors z for a given matrix A.
There are four types of driver routines for symmetric and Hermitian eigenproblems.
Originally LAPACK had just the simple and expert drivers described below, and the
other two were added after improved algorithms were discovered. Ultimately we expect
the algorithm in the most recent driver (called RRR below) to supersede all the others,
but in LAPACK 3.0 the other drivers may still be faster on some problems, so we retain
them.

A simple driver computes all the eigenvalues and (optionally) eigenvectors.

An expert driver computes all or a selected subset of the eigenvalues and (optionally)
eigenvectors. If few enough eigenvalues or eigenvectors are desired, the expert driver
is faster than the simple driver.
 6.4. LAPACK SOLVERS FOR THE SYMMETRIC EIGENPROBLEMS 115
A divide-and-conquer driver solves the same problem as the simple driver. It is much
faster than the simple driver for large matrices, but uses more workspace. The name
divide-and-conquer refers to the underlying algorithm.

A relatively robust representation (RRR) driver computes all or (in a later release)
a subset of the eigenvalues, and (optionally) eigenvectors. It is the fastest algorithm
of all (except for a few cases), and uses the least workspace. The name RRR refers
to the underlying algorithm.
This computation proceeds in the following stages:
1. The real symmetric or complex Hermitian matrix A is reduced to real tridiagonal
form T . If A is real symmetric this decomposition is A = QT QT with Q orthogonal
and T symmetric tridiagonal. If A is complex Hermitian, the decomposition is
A = QT QH with Q unitary and T , as before, real symmetric tridiagonal.

2. Eigenvalues and eigenvectors of the real symmetric tridiagonal matrix T are com-
puted. If all eigenvalues and eigenvectors are computed, this is equivalent to factor-
izing T as T = SS T , where S is orthogonal and is diagonal. The diagonal entries
of are the eigenvalues of T , which are also the eigenvalues of A, and the columns
of S are the eigenvectors of T ; the eigenvectors of A are the columns of Z = QS, so
that A = ZZ T (ZZ H when A is complex Hermitian).
In the real case, the decomposition A = QT QT is computed by one of the routines
sytrd, sptrd, or sbtrd, depending on how the matrix is stored. The complex analogues
of these routines are called hetrd, hptrd, and hbtrd. The routine sytrd (or hetrd)
represents the matrix Q as a product of elementary reflectors. The routine orgtr (or
in the complex case unmtr) is provided to form Q explicitly; this is needed in particular
before calling steqr to compute all the eigenvectors of A by the QR algorithm. The
routine ormtr (or in the complex case unmtr) is provided to multiply another matrix by
Q without forming Q explicitly; this can be used to transform eigenvectors of T computed
by stein, back to eigenvectors of A.
For the names of the routines for packed and banded matrices, see [1].
There are several routines for computing eigenvalues and eigenvectors of T , to cover the
cases of computing some or all of the eigenvalues, and some or all of the eigenvectors. In
addition, some routines run faster in some computing environments or for some matrices
than for others. Also, some routines are more accurate than other routines.
steqr This routine uses the implicitly shifted QR algorithm. It switches between the QR
and QL variants in order to handle graded matrices. This routine is used to compute
all the eigenvalues and eigenvectors.

sterf This routine uses a square-root free version of the QR algorithm, also switching
between QR and QL variants, and can only compute all the eigenvalues. This
routine is used to compute all the eigenvalues and no eigenvectors.

stedc This routine uses Cuppens divide and conquer algorithm to find the eigenvalues and
the eigenvectors. stedc can be many times faster than steqr for large matrices
but needs more work space (2n2 or 3n2 ). This routine is used to compute all the
eigenvalues and eigenvectors.

stegr This routine uses the relatively robust representation (RRR) algorithm to find eigen-
values and eigenvectors. This routine uses an LDLT factorization of a number of
 116 CHAPTER 6. LAPACK AND THE BLAS
translates T I of T , for one shift near each cluster of eigenvalues. For each
translate the algorithm computes very accurate eigenpairs for the tiny eigenvalues.
stegr is faster than all the other routines except in a few cases, and uses the least
workspace.

stebz This routine uses bisection to compute some or all of the eigenvalues. Options
provide for computing all the eigenvalues in a real interval or all the eigenvalues
from the ith to the jth largest. It can be highly accurate, but may be adjusted to
run faster if lower accuracy is acceptable.

stein Given accurate eigenvalues, this routine uses inverse iteration to compute some or
all of the eigenvectors.

6.5 Generalized Symmetric Definite Eigenproblems (GSEP)

Drivers are provided to compute all the eigenvalues and (optionally) the eigenvectors of
the following types of problems:
1. Az = Bz

2. ABz = z

3. BAz = z
where A and B are symmetric or Hermitian and B is positive definite. For all these
problems the eigenvalues are real. The matrices Z of computed eigenvectors satisfy
Z T AZ = (problem types 1 and 3) or Z 1 AZ T = I (problem type 2), where is a
diagonal matrix with the eigenvalues on the diagonal. Z also satisfies Z T BZ = I (problem
types 1 and 2) or Z T B 1 Z = I (problem type 3).
There are three types of driver routines for generalized symmetric and Hermitian eigen-
problems. Originally LAPACK had just the simple and expert drivers described below,
and the other one was added after an improved algorithm was discovered.
a simple driver computes all the eigenvalues and (optionally) eigenvectors.

an expert driver computes all or a selected subset of the eigenvalues and (optionally)
eigenvectors. If few enough eigenvalues or eigenvectors are desired, the expert driver
is faster than the simple driver.

a divide-and-conquer driver solves the same problem as the simple driver. It is much
faster than the simple driver for large matrices, but uses more workspace. The name
divide-and-conquer refers to the underlying algorithm.

6.6 An example of a LAPACK routines

The double precision subroutine dsytrd.f implements the reduction to tridiagonal form.
We give it here in full length.
SUBROUTINE DSYTRD( UPLO, N, A, LDA, D, E, TAU, WORK, LWORK, INFO )
*
* -- LAPACK routine (version 3.0) --
* Univ. of Tennessee, Univ. of California Berkeley, NAG Ltd.,
* Courant Institute, Argonne National Lab, and Rice University
* June 30, 1999
6.6. AN EXAMPLE OF A LAPACK ROUTINES 117
*
* .. Scalar Arguments ..
CHARACTER UPLO
INTEGER INFO, LDA, LWORK, N
* ..
* .. Array Arguments ..
DOUBLE PRECISION A( LDA, * ), D( * ), E( * ), TAU( * ),
$ WORK( * )
* ..
*
* Purpose
* =======
*
* DSYTRD reduces a real symmetric matrix A to real symmetric
* tridiagonal form T by an orthogonal similarity transformation:
* Q**T * A * Q = T.
*
* Arguments
* =========
*
* UPLO (input) CHARACTER*1
* = U: Upper triangle of A is stored;
* = L: Lower triangle of A is stored.
*
* N (input) INTEGER
* The order of the matrix A. N >= 0.
*
* A (input/output) DOUBLE PRECISION array, dimension (LDA,N)
* On entry, the symmetric matrix A. If UPLO = U, the leading
* N-by-N upper triangular part of A contains the upper
* triangular part of the matrix A, and the strictly lower
* triangular part of A is not referenced. If UPLO = L, the
* leading N-by-N lower triangular part of A contains the lower
* triangular part of the matrix A, and the strictly upper
* triangular part of A is not referenced.
* On exit, if UPLO = U, the diagonal and first superdiagonal
* of A are overwritten by the corresponding elements of the
* tridiagonal matrix T, and the elements above the first
* superdiagonal, with the array TAU, represent the orthogonal
* matrix Q as a product of elementary reflectors; if UPLO
* = L, the diagonal and first subdiagonal of A are over-
* written by the corresponding elements of the tridiagonal
* matrix T, and the elements below the first subdiagonal, with
* the array TAU, represent the orthogonal matrix Q as a product
* of elementary reflectors. See Further Details.
*
* LDA (input) INTEGER
* The leading dimension of the array A. LDA >= max(1,N).
*
* D (output) DOUBLE PRECISION array, dimension (N)
* The diagonal elements of the tridiagonal matrix T:
* D(i) = A(i,i).
*
* E (output) DOUBLE PRECISION array, dimension (N-1)
* The off-diagonal elements of the tridiagonal matrix T:
* E(i) = A(i,i+1) if UPLO = U, E(i) = A(i+1,i) if UPLO = L.
*
* TAU (output) DOUBLE PRECISION array, dimension (N-1)
* The scalar factors of the elementary reflectors (see Further
* Details).
118 CHAPTER 6. LAPACK AND THE BLAS
*
* WORK (workspace/output) DOUBLE PRECISION array, dimension (LWORK)
* On exit, if INFO = 0, WORK(1) returns the optimal LWORK.
*
* LWORK (input) INTEGER
* The dimension of the array WORK. LWORK >= 1.
* For optimum performance LWORK >= N*NB, where NB is the
* optimal blocksize.
*
* If LWORK = -1, then a workspace query is assumed; the routine
* only calculates the optimal size of the WORK array, returns
* this value as the first entry of the WORK array, and no error
* message related to LWORK is issued by XERBLA.
*
* INFO (output) INTEGER
* = 0: successful exit
* < 0: if INFO = -i, the i-th argument had an illegal value
*
* Further Details
* ===============
*
* If UPLO = U, the matrix Q is represented as a product of elementary
* reflectors
*
* Q = H(n-1) . . . H(2) H(1).
*
* Each H(i) has the form
*
* H(i) = I - tau * v * v
*
* where tau is a real scalar, and v is a real vector with
* v(i+1:n) = 0 and v(i) = 1; v(1:i-1) is stored on exit in
* A(1:i-1,i+1), and tau in TAU(i).
*
* If UPLO = L, the matrix Q is represented as a product of elementary
* reflectors
*
* Q = H(1) H(2) . . . H(n-1).
*
* Each H(i) has the form
*
* H(i) = I - tau * v * v
*
* where tau is a real scalar, and v is a real vector with
* v(1:i) = 0 and v(i+1) = 1; v(i+2:n) is stored on exit in A(i+2:n,i),
* and tau in TAU(i).
*
* The contents of A on exit are illustrated by the following examples
* with n = 5:
*
* if UPLO = U: if UPLO = L:
*
* ( d e v2 v3 v4 ) ( d )
* ( d e v3 v4 ) ( e d )
* ( d e v4 ) ( v1 e d )
* ( d e ) ( v1 v2 e d )
* ( d ) ( v1 v2 v3 e d )
*
* where d and e denote diagonal and off-diagonal elements of T, and vi
* denotes an element of the vector defining H(i).
6.6. AN EXAMPLE OF A LAPACK ROUTINES 119
*
* =====================================================================
*
* .. Parameters ..
DOUBLE PRECISION ONE
PARAMETER ( ONE = 1.0D+0 )
* ..
* .. Local Scalars ..
LOGICAL LQUERY, UPPER
INTEGER I, IINFO, IWS, J, KK, LDWORK, LWKOPT, NB,
$ NBMIN, NX
* ..
* .. External Subroutines ..
EXTERNAL DLATRD, DSYR2K, DSYTD2, XERBLA
* ..
* .. Intrinsic Functions ..
INTRINSIC MAX
* ..
* .. External Functions ..
LOGICAL LSAME
INTEGER ILAENV
EXTERNAL LSAME, ILAENV
* ..
* .. Executable Statements ..
*
* Test the input parameters
*
INFO = 0
UPPER = LSAME( UPLO, U )
LQUERY = ( LWORK.EQ.-1 )
IF( .NOT.UPPER .AND. .NOT.LSAME( UPLO, L ) ) THEN
INFO = -1
ELSE IF( N.LT.0 ) THEN
INFO = -2
ELSE IF( LDA.LT.MAX( 1, N ) ) THEN
INFO = -4
ELSE IF( LWORK.LT.1 .AND. .NOT.LQUERY ) THEN
INFO = -9
END IF
*
IF( INFO.EQ.0 ) THEN
*
* Determine the block size.
*
NB = ILAENV( 1, DSYTRD, UPLO, N, -1, -1, -1 )
LWKOPT = N*NB
WORK( 1 ) = LWKOPT
END IF
*
IF( INFO.NE.0 ) THEN
CALL XERBLA( DSYTRD, -INFO )
RETURN
ELSE IF( LQUERY ) THEN
RETURN
END IF
*
* Quick return if possible
*
IF( N.EQ.0 ) THEN
WORK( 1 ) = 1
120 CHAPTER 6. LAPACK AND THE BLAS
RETURN
END IF
*
NX = N
IWS = 1
IF( NB.GT.1 .AND. NB.LT.N ) THEN
*
* Determine when to cross over from blocked to unblocked code
* (last block is always handled by unblocked code).
*
NX = MAX( NB, ILAENV( 3, DSYTRD, UPLO, N, -1, -1, -1 ) )
IF( NX.LT.N ) THEN
*
* Determine if workspace is large enough for blocked code.
*
LDWORK = N
IWS = LDWORK*NB
IF( LWORK.LT.IWS ) THEN
*
* Not enough workspace to use optimal NB: determine the
* minimum value of NB, and reduce NB or force use of
* unblocked code by setting NX = N.
*
NB = MAX( LWORK / LDWORK, 1 )
NBMIN = ILAENV( 2, DSYTRD, UPLO, N, -1, -1, -1 )
IF( NB.LT.NBMIN )
$ NX = N
END IF
ELSE
NX = N
END IF
ELSE
NB = 1
END IF
*
IF( UPPER ) THEN
*
* Reduce the upper triangle of A.
* Columns 1:kk are handled by the unblocked method.
*
KK = N - ( ( N-NX+NB-1 ) / NB )*NB
DO 20 I = N - NB + 1, KK + 1, -NB
*
* Reduce columns i:i+nb-1 to tridiagonal form and form the
* matrix W which is needed to update the unreduced part of
* the matrix
*
CALL DLATRD( UPLO, I+NB-1, NB, A, LDA, E, TAU, WORK,
$ LDWORK )
*
* Update the unreduced submatrix A(1:i-1,1:i-1), using an
* update of the form: A := A - V*W - W*V
*
CALL DSYR2K( UPLO, No transpose, I-1, NB, -ONE, A( 1, I ),
$ LDA, WORK, LDWORK, ONE, A, LDA )
*
* Copy superdiagonal elements back into A, and diagonal
* elements into D
*
DO 10 J = I, I + NB - 1
6.6. AN EXAMPLE OF A LAPACK ROUTINES 121
A( J-1, J ) = E( J-1 )
D( J ) = A( J, J )
10 CONTINUE
20 CONTINUE
*
* Use unblocked code to reduce the last or only block
*
CALL DSYTD2( UPLO, KK, A, LDA, D, E, TAU, IINFO )
ELSE
*
* Reduce the lower triangle of A
*
DO 40 I = 1, N - NX, NB
*
* Reduce columns i:i+nb-1 to tridiagonal form and form the
* matrix W which is needed to update the unreduced part of
* the matrix
*
CALL DLATRD( UPLO, N-I+1, NB, A( I, I ), LDA, E( I ),
$ TAU( I ), WORK, LDWORK )
*
* Update the unreduced submatrix A(i+ib:n,i+ib:n), using
* an update of the form: A := A - V*W - W*V
*
CALL DSYR2K( UPLO, No transpose, N-I-NB+1, NB, -ONE,
$ A( I+NB, I ), LDA, WORK( NB+1 ), LDWORK, ONE,
$ A( I+NB, I+NB ), LDA )
*
* Copy subdiagonal elements back into A, and diagonal
* elements into D
*
DO 30 J = I, I + NB - 1
A( J+1, J ) = E( J )
D( J ) = A( J, J )
30 CONTINUE
40 CONTINUE
*
* Use unblocked code to reduce the last or only block
*
CALL DSYTD2( UPLO, N-I+1, A( I, I ), LDA, D( I ), E( I ),
$ TAU( I ), IINFO )
END IF
*
WORK( 1 ) = LWKOPT
RETURN
*
* End of DSYTRD
*
END

Notice that most of the lines (indicated by ) contain comments. The initial comment
lines also serve as manual pages. Notice that the code only looks at one half (upper or
lower triangle) of the symmetric input matrix. The other triangle is used to store the
Householder vectors. These are normed such that the first component is one,

I 2uu = I 2|u1 |2 (u/u1 )(u/u1 ) = I vv .

In the main loop of dsytrd there is a call to a subroutine dlatrd that generates a
block reflektor. (The blocksize is NB.) Then the block reflector is applied by the routine
122 CHAPTER 6. LAPACK AND THE BLAS
dsyr2k.
Directly after the loop there is a call to the unblocked dsytrd named dsytd2 to deal
with the first/last few (<NB) rows/columns of the matrix. This excerpt concerns the
situation when the upper triangle of the matrix A is stored. In that routine the mentioned
loop looks very much the way we derived the formulae.

ELSE
*
* Reduce the lower triangle of A
*
DO 20 I = 1, N - 1
*
* Generate elementary reflector H(i) = I - tau * v * v
* to annihilate A(i+2:n,i)
*
CALL DLARFG( N-I, A( I+1, I ), A( MIN( I+2, N ), I ), 1,
$ TAUI )
E( I ) = A( I+1, I )
*
IF( TAUI.NE.ZERO ) THEN
*
* Apply H(i) from both sides to A(i+1:n,i+1:n)
*
A( I+1, I ) = ONE
*
* Compute x := tau * A * v storing y in TAU(i:n-1)
*
CALL DSYMV( UPLO, N-I, TAUI, A( I+1, I+1 ), LDA,
$ A( I+1, I ), 1, ZERO, TAU( I ), 1 )
*
* Compute w := x - 1/2 * tau * (x*v) * v
*
ALPHA = -HALF*TAUI*DDOT( N-I, TAU( I ), 1, A( I+1, I ),
$ 1 )
CALL DAXPY( N-I, ALPHA, A( I+1, I ), 1, TAU( I ), 1 )
*
* Apply the transformation as a rank-2 update:
* A := A - v * w - w * v
*
CALL DSYR2( UPLO, N-I, -ONE, A( I+1, I ), 1, TAU( I ), 1,
$ A( I+1, I+1 ), LDA )
*
A( I+1, I ) = E( I )
END IF
D( I ) = A( I, I )
TAU( I ) = TAUI
20 CONTINUE
D( N ) = A( N, N )
END IF

Bibliography
[1] E. Anderson, Z. Bai, C. Bischof, J. Demmel, J. Dongarra, J. D.
Croz, A. Greenbaum, S. Hammarling, A. McKenney, S. Ostrouchov,
and D. Sorensen, LAPACK Users Guide - Release 2.0, SIAM, Philadel-
phia, PA, 1994. (Software and guide are available from Netlib at URL
http://www.netlib.org/lapack/).
BIBLIOGRAPHY 123
[2] L. S. Blackford, J. Choi, A. Cleary, E. DAzevedo, J. Demmel,
I. Dhillon, J. Dongarra, S. Hammarling, G. Henry, A. Petitet,
K. Stanley, D. Walker, and R. C. Whaley, ScaLAPACK Users Guide,
SIAM, Philadelphia, PA, 1997. (Software and guide are available at URL
http://www.netlib.org/scalapack/).

[3] J. J. Dongarra, J. R. Bunch, C. B. Moler, and G. W. Stewart, LINPACK

Users Guide, SIAM, Philadelphia, PA, 1979.

[4] J. J. Dongarra, J. D. Croz, I. Duff, and S. Hammarling, A proposal for a

set of level 3 basic linear algebra subprograms, ACM SIGNUM Newsletter, 22 (1987).

[5] , A set of level 3 basic linear algebra subprograms, ACM Trans. Math. Softw., 16
(1990), pp. 117.

[6] J. J. Dongarra, J. Du Croz, S. Hammarling, and R. J. Hanson, An extended

set of fortran basic linear algebra subprograms, ACM Trans. Math. Softw., 14 (1988),
pp. 117.

[7] , An extended set of fortran basic linear algebra subprograms: Model implemen-
tation and test programs, ACM Transactions on Mathematical Software, 14 (1988),
pp. 1832.

[8] B. S. Garbow, J. M. Boyle, J. J. Dongarra, and C. B. Moler, Matrix Eigen-

system Routines EISPACK Guide Extension, Lecture Notes in Computer Science
51, Springer-Verlag, Berlin, 1977.

[9] C. Lawson, R. Hanson, D. Kincaid, and F. Krogh, Basic linear algebra sub-
programs for Fortran usage, ACM Trans. Math. Softw., 5 (1979), pp. 308325.

[10] B. T. Smith, J. M. Boyle, J. J. Dongarra, B. S. Garbow, Y. Ikebe, V. C.

Klema, and C. B. Moler, Matrix Eigensystem Routines EISPACK Guide, Lec-
ture Notes in Computer Science 6, Springer-Verlag, Berlin, 2nd ed., 1976.
124 CHAPTER 6. LAPACK AND THE BLAS
Chapter 7

Vector iteration (power method)

7.1 Simple vector iteration

In this chapter we consider the simplest method to compute a single extremal eigenvalue,
called vector iteration or power method [2, 5]. Let A Fnn
. Starting with an arbitrary
initial vector x(0) Fn we form the vector sequence x(k) k=0 by defining

(7.1) x(k) := Ax(k1) , k = 1, 2, . . .

Clearly,
(7.2) x(k) := Ak x(0) .
The hope is that the x(k) converge to an eigenvector associated with the eigenvalue of
largest magnitude. As we are interested only in the direction but not in the length of the
eigenvector, there is no need to normalize the iterates in (7.1), well at least in theory. In
practice, x(k) may either underflow (if kAk < 1) or overflow (if kAk > 1) for large k. Thus,
one usually adds a normalization step to (7.1), leading to Algorithm 7.1.

Algorithm 7.1 Simple vector iteration (power method)

1: Choose a starting vector x(0) Fn with kx(0) k = 1.
2: k = 0.
3: repeat
4: k := k + 1;
5: y(k) := Ax(k1) ;
6: k := ky(k) k;
7: x(k) := y(k) /k ;
8: until a convergence criterion is satisfied

The vectors x(k) generated by Algorithm 7.1 have all norm (length) one. That is, x(k) k=0
is a sequence on the unit sphere in Fn .
Let A = XJY be the Jordan normal form of A with Y := X 1 , see Section 2.8.
Then,
(7.3) Y x(k) := JY x(k1) and Y x(k) := J k Y x(0) ,
 
respectively. If the sequence x(k) k=0 converges to x then the sequence y(k) k=0 with
y(k) = Y x(k) converges to y = Y x . By consequence, for the convergence analysis, we
may assume without loss of generality (w.l.o.g.) that A is a Jordan block matrix.

125
126 CHAPTER 7. VECTOR ITERATION (POWER METHOD)
7.2 Angles between vectors
Let q1 and q2 be unit vectors, cf. Fig. 7.1. The length of the orthogonal projection of q2

q2

q1

Figure 7.1: Angle between vectors q1 and q2

on span{q1 } is given by

(7.4) c := kq1 q1 q2 k = |q1 q2 | 1.

The length of the orthogonal projection of q2 on span{q1 } is

(7.5) s := k(I q1 q1 )q2 k.

As q1 q1 is an orthogonal projection we have by Pythagoras formula that

(7.6) 1 = kq2 k2 = kq1 q1 q2 k2 + k(I q1 q1 )q2 k2 = s2 + c2 .

Alternatively, we can conclude from (7.5) that

s2 = k(I q1 q1 )q2 k2
= q2 (I q1 q1 )q2
(7.7)
= q2 q2 (q2 q1 )(q1 q2 )
= 1 c2

So, there is a number, say, , 0 2 , such that c = cos and s = sin . We call this
uniquely determined number the angle between the vectors q1 and q2 :

= (q1 , q2 ).

The generalization to arbitrary vectors is straightforward.

Definition 7.1 The angle between two nonzero vectors x and y is given by
     
xx y |x y|
(7.8) = (x, y) = arcsin
I kxk2 kyk = arccos kxkkyk .

When investigating the convergence behaviour of eigensolvers we usually show that

the angle between the approximating and the desired vector tends to zero as the number
of iterations increases. In fact it is more convenient to work with the sine of the angle.
7.3. CONVERGENCE ANALYSIS 127
In the formulae above we used the projections P and I P with P = q1 q1 . We would
have arrived at the same point if we had exchanged the roles of q1 and q2 . As

kq1 q1 q2 k = kq2 q2 q1 k = |q2 q1 |

we get
k(I q1 q1 )q2 k = k(I q2 q2 )q1 k.
This immediately leads to

Lemma 7.2 sin (q1 , q2 ) = kq1 q1 q2 q2 k.

7.3 Convergence analysis

Let us now assume that A has Jordan block form,
 
1 0
(7.9) A=
0 J2

with eigenvalues

(7.10) |1 | > |2 | |3 | |n |.

Then, the eigenvector of A corresponding to its largest eigenvalue 1 is e1 . We will now

show that the iterates x(k) converge to e1 . More precisely, we will show that the angle
(x(k) , e1 ) between x(k) and e1 goes to zero with k . Let
(k)

x1
(k) !
(k)
x2 x1
x(k) =
..
=:
(k)
. x2
(k)
xn

with kx(k) k = 1. Then,

v
u n
(k) uX (k)
sin (k) := sin((x , e1 )) = k(I e1 e1 )x k = kx2 k = t |xi |2 .
(k) (k)

i=2

If we omit the normalization kx(k) k = 1, which we will do for convenience, then this
becomes v
(k) u Pn (k)
kx2 k u |x |2
sin := sin((x , e1 )) = (k) = t Pi=2 i(k) .
(k) (k)
kx k n 2
i=1 |xi |

This means that for the convergence analysis we look at the iteration (7.1), while the
actual implementation follows closely Algorithm 7.1.
From (7.1) we have
!   !   !
x
(k)
0 x
(k1)
0 k x
(0)
1 1
x(k) = 1
(k) = 1
(k1) = 1
(0) .
x2 0 J2 x2 0 J2 x2
128 CHAPTER 7. VECTOR ITERATION (POWER METHOD)
Defining
1 (k)
(7.11) y(k) := x
k1
we have  
(k) 1 0
y = 1 y(k1) .
0 1 J2
(0) (k)
Let us assume that y1 = 1. Then y1 = 1 for all k and

2
3
1 1 |k |
(k) (k1) .. ..
y2 = J2 y2 , J2 = . . , |k | = < 1.
1 1 |1 |
n1
n

For the sequel we need

Theorem 7.3 Let ||| ||| be any matrix norm. Then

(7.12) lim |||M k |||1/k = (M ) = max |i (M )|.

k i

Proof. See Horn-Johnson [3], pp.297-299.

Definition 7.4 (M ) in (7.12) is call spectral radius of M .

Regarding the convergence of the vector iteration, Theorem 7.3 implies that for any
> 0 there is an integer K() such that

(7.13) |||M k |||1/k (M ) + , for all k > K().

We will apply this theorem to the case M = 1 1 J2 , the matrix norm ||| ||| will be the
ordinary 2-norm. Thus, for any > 0 there is a K() N with

1 k 1/k

(7.14) J2 |2 | + , k > K().
1

We can choose such that

|2 | + < 1.
Then,
(k) (k)
(k) k(I e1 e1 )y(k) k ky k ky k
sin((y , e1 )) = (k)
= 2(k) = q 2
ky k ky k (k)
1 + ky2 k
(k) 1 (0) (0)
ky2 k k k J2k kky2 k (|2 | + )k ky2 k.
1
Thus, the angle between y(k) and e1 goes to zero with a rate 2 + for any positive .
Since x(k) is a scalar multiple of y(k) the same holds for the angle between x(k) and e1 .
Since we can choose arbitrarily small, we have proved that
 k
 2 
sin (k) = sin((x(k) , u1 )) c  
1
7.3. CONVERGENCE ANALYSIS 129
(0)
provided that x1 = e1 x(0) 6= 0.
Returning to a general matrix A Fnn with Jordan normal form A = XJY , we
employ equation (7.3). The sequence y(k) = Y x(k) converges to y = e1 with 6= 0.
Therefore, x(k) converges to a multiple of Xe1 , which is an eigenvector associated with
the largest eigenvalue 1 . The condition e1 y(0) 6= 0 translates into

e1 (Y x(0) ) = (Y e1 ) x(0) 6= 0.

The first column of Y is a left eigenvector associated with 1 . Therefore, we have proved

Theorem 7.5 Let the eigenvalues of A Fnn be arranged such that |1 | > |2 |
|3 | |n |. Let u1 and v1 be right and left eigenvectors of A corresponding to 1 ,
respectively. Then, the vector sequence generated by Algorithm 7.1 converges to u1 in the
sense that
 k
 2 
(7.15) sin = sin((x , u1 )) c  
(k) (k)
1

provided that v1 x(0) 6= 0.

Remark 7.1. The quantity k in Algorithm 7.1 converges to |1 |. The true value 1 C
can be found by comparing single components of y(k) and x(k1) . If 1 R then only the
sign of 1 is at stake.
Remark 7.2. The convergence of the vector iteration is faster the smaller the quotient
|2 |/|1 | is.
Remark 7.3. From (7.12) we see that the norm of the powers of a matrix goes to zero if all

40

35

30

25
||A ||
k

20

15

10

0
0 20 40 60 80 100
k

Figure 7.2: Norms of powers of B in (7.16).

is eigenvalues are smaller than one in modulus. For small powers the norm can initially
grow considerably. In Fig. 7.2 we have plotted the norms of B k with
 
0.9 5
(7.16) B= .
0 0.9
130 CHAPTER 7. VECTOR ITERATION (POWER METHOD)

Remark 7.4. If v1 x(0) = 0 then the vector iteration converges to an eigenvector corre-
sponding to the second largest eigenvalue. In practice, rounding errors usually prevent
this behaviour of the algorithm. After a long initial phase the x(k) turn to u1 .
Remark 7.5. In case that 1 6= 2 but |1 | = |2 | there may be no convergence at all. An
example is    
1 0 (0)
A= , x = .
0 1

7.4 A numerical example

In the following Matlab script we assume that A is upper triangular and that the largest
eigenvalue (in modulus) is at position (1,1), i.e., |a1,1 | > |aj,j | for all j > 1.
%Demo Simple Vector Iteration
%
n = 6;
randn(state,0);
A = diag([n:-1:1]) + triu(randn(n),1) % upper triangular matrix
x0 = rand(n,1); x0=x0/norm(x0) % initial vector

e = eye(n,1); % Right eigenvector corresponding to largest

% eigenvalue A(1,1)
% ----------------------------------------------------------

x=x0; ang = norm(x - e*(e*x))

hist = [ang,nan,nan];

if ~exist(tol), tol = 100*eps; end

oldang = nan;

while ang > tol

x = A*x;
mue = norm(x); % This is an approximation for the
x = x/mue; % searched eigenvalue

ang = norm(x - e*(e*x));

hist = [hist; [mue,ang,ang/oldang]];
oldang = ang;
end
Because the solution is known, we can compute the angle between iterate and true solution.
We can even use this angle in the stopping criterion. The matrix A is given by
A =
6.0000 1.1892 -0.5883 -0.0956 -0.6918 -0.3999
0 5.0000 2.1832 -0.8323 0.8580 0.6900
0 0 4.0000 0.2944 1.2540 0.8156
7.5. THE SYMMETRIC CASE 131
7

5
eigenvalue
angle(xk, u1)

4 angle/old angle

0
0 20 40 60 80 100 120 140 160 180

Figure 7.3: Plot of three important quantities: eigenvalue, angle between eigenvector
approximation and exact eigenvector, convergence rate of eigenvector

0 0 0 3.0000 -1.5937 0.7119

0 0 0 0 2.0000 1.2902
0 0 0 0 0 1.0000

The development of three important quantities is given in Fig. 7.3. In Fig. 7.4 the case
is depicted when the initial vector is chosen orthogonal to the left eigenvector correspond-
ing to 1 = 6. Initially, the approximated eigenvalue is 5. Because the stopping criterion
does not hold, the iteration continues until eventually rounding errors take effect.

7.5 The symmetric case

Let us now consider the Hermitian/symmetric case. We again assume the now real eigen-
values to be arranged as in (7.10). But now the Schur decomposition of A becomes its
spectral decomposition,

(7.17) A = U U , U = [u1 , . . . , un ], = diag(1 , . . . , n ).

For the convergence analysis, we assume that A is diagonal, and that

(7.18) 1 > 2 n 0.

Therefore, in (7.9) we have J2 = diag(2 , . . . , n ), i.e., all Jordan blocks are 1 1.

In contrast to the general case, in the Hermitian case we approximate the eigenvalue
by the Rayleigh quotient of x(k) ,

(7.19) (k) := x(k) Ax(k) , kx(k) k = 1.

The symmetric algorithm is given in Algorithm 7.2.

132 CHAPTER 7. VECTOR ITERATION (POWER METHOD)
7

eigenvalue
4 angle(xk, u1)

angle/old_angle
3

0
0 50 100 150 200 250 300 350 400

Figure 7.4: Plot of three important quantities: eigenvalue, angle between eigenvector
approximation and exact eigenvector, convergence rate of eigenvector. Here, the initial
vector is chosen orthogonal to the left eigenvector corresponding to the largest eigenvalue

Algorithm 7.2 Simple vector iteration for Hermitian matrices

1: Choose a starting vector x(0) Fn with kx(0) k = 1.
2: y(0) := Ax(0) .

3: (0) := y(0) x(0) .
4: k := 0.
5: while ky(k) (k) x(k) k > tol do
6: k := k + 1;
7: x(k) := yk1 /kyk1 k;
8: y(k) := Ax(k) ;

9: (k) := y(k) x(k) ;
10: end while

In order to investigate the convergence of the Rayleigh quotient we work with auxiliary
vectors
!
1 1
(7.20) y(k) = (k) = (k) x(k) .
y2 |x | 1

Notice, that any reasonable approximation of the first eigenvector e1 has a nonzero first
component. For the Rayleigh quotients we have

(y(k) ) = (x(k) ).

Now,
(k) (k) (k) (k)
(k) y(k) Ay(k) (e1 + y2 ) A(e1 + y2 ) 1 + y2 Ay2
(7.21) = (k) (k) = (k)
= (k)
y y 1 + ky2 k2 1 + ky2 k2
7.5. THE SYMMETRIC CASE 133
(k) (k)
where we used that e1 y2 = 0 and e1 Ay2 = 0. Because,
(k)
tan (k) := tan((y(k) , e1 )) = ky2 k
and
1
1 + tan2 () =
1 sin2 ()
we get from (7.21) that
(k) (k) (k) (k)
(7.22) (k) = (1 + y2 Ay2 )(1 sin2 (k) ) = 1 1 sin2 (k) + y2 Ay2 cos2 (k) .
Now, since 1 > 0,
(k) (k)
0 1 (k) = 1 sin2 (k) y2 Ay2 cos2 (k)
(7.23)
(k)
1 sin2 (k) n ky2 k2 cos2 (k) = (1 n ) sin2 (k) .
In summary, we have proved
Theorem 7.6 Let A be a symmetric matrix with spectral decomposition (7.17)(7.18).
Then, the simple vector iteration of Algorithm 7.2 computes sequences (k) k=0 and
 (k)
x k=0
that converge linearly towards the largest eigenvalue 1 of A and the corre-
sponding eigenvector u1 provided that the initial vector x(0) has a nonzero component in
the direction of u1 , i.e., that u1 x(0) 6= 0. The convergence rates are given by
 k  2k
 2   2 
(k)  
sin   sin , (0)
|1 | (1 n )   sin2 (0) .
(k)
1 1
where (k) = (x(k) , u1 .
Thus, the speed of convergence is determined by the ratio of the two
 eigenvalues
 largest
in
modulus and the quality of the initial guess x(0) . Both sequences (k) and x(k) con-
verge linearly, but the decisive ratio appears squared in the bound for the approximation
error in the eigenvalue. 1 n is called the spread of the spectrum of A. Its occurance
in the bound for max (k) shows that a simple scaling of the matrix does not affect the
convergence behavior of the algorithm.
Example 7.7 Lets compute the smallest eigenvalue and corresponding eigenvector of the
one-dimensional Poisson matrix T = Tn of Example 2.7 with n = 40. Let us assume that
we know an upper bound for the largest eigenvalue n of T then the transformed matrix
I T has the same eigenvectors as T and eigenvalues n < n1 < < 1 .
So, we apply vector iteration to compute the desired quantities.
We set = 4(n + 1)2 / 2 a number that is easily obtained by applying Gerschgorins
circle theorem. We performed a Matlab experiment starting with a random vector.
>> n=40;
>> T = (4*((n+1)^2/pi^2))*eye(n) - ((n+1)^2/pi^2)*p_1d(n);
>> rand(state,0); x0=rand(n,1);
>> [x,lam,nit]=vit(T,x0,1e-4);
>> tau-lam
ans =
0.9995
>> nit
nit =
1968
134 CHAPTER 7. VECTOR ITERATION (POWER METHOD)
In as many as 1968 iteration steps we arrived at an eigenvalue approximation 0.9995.
This number is correct to all digits. The difference to the eigenvalue 1 of the continuous
eigenvalue problem u (x) = u(x) is due to the discretization error. Figure 7.5 shows
the convergence history of this calculation. The straight lines show the actual angle (k)

2
10

0
10

2
10

4
10

6
10

8
10

10
10
0 200 400 600 800 1000 1200 1400 1600 1800 2000

Figure 7.5: Simple vector iteration with I40 T40

between x(k) and u1 (above) and the actual error (k) 1 . These quantities can of course
not be computed in general. In this example we know them, see Ex. 2.7. The dotted
lines show powers of q = ( 2 )/( 1 ) that indicate the convergence rates given by
Theorem 7.6. Here, q = 0.9956. Clearly, the convergence is as predicted.

Example 7.8 We mentioned that a good initial vector can reduce the number of iteration
steps. Remember that the smallest eigenfunction is sin x, a function that is positive on
the whole interval (0, ). Let us therefore set x(0) to be the vector of all ones.

>> x0 = ones(n,1);
>> [x,lam,nit]=vit(T,x0,1e-4);
>> nit
nit =
866

This is a surprisingly high reduction in the number of iteration steps. Figure 7.6 shows
the convergence history of this calculation. Again the doted lines indicate the convergence
rates that are to be expected. The actual convergence rates are evidently much better.
How can that be?
The eigenvectors of Tn resemble the eigenfunctions sin kx of the continuous eigen-
value problem. Therefore the coefficients corresponding to eigenvectors corresponding to
eigenfunctions antisymmetric with respect to the point /2 vanish. In particular x2 = 0.
Therefore the convergence rate is not q = ( 2 )/( 1 ) but q = ( 3 )/( 1 ). This
is verified by the numbers given in Fig. 7.7 where the assymptotic corrected convergence
rates q and q2 are indicated.
7.6. INVERSE VECTOR ITERATION 135
2
10

0
10

2
10

4
10

6
10

8
10

10
10
0 100 200 300 400 500 600 700 800 900

Figure 7.6: Simple vector iteration with I40 T40 and starting vector (1, 1, . . . , 1)T

Problem 7.9 When computing the smallest eigenvalue of Tn by the simple vector iter-
ation we can find a better shift than above if the extremal points of the spectrum are
known. Determine such that In Tn exhibits the optimal convergence rate. Hint: On
the one hand we would like the quotient ( n1 )/( n ) to be as small as possible. On
the other hand | 1 |/( n ) must not become to big. Hint: Equate the two quantities.

7.6 Inverse vector iteration

The previous examples have shown that the convergence of simple vector iteration is
potentially very slow. The quotient of the second largest to the largest eigenvalue are very
close to 1. We noticed this by means of a very simple and small eigenvalue problem. The
situation gets much worse if the problems are big.
We have seen in (2.28) that a polynomial in A has the same eigenvectors as A. We
therefore may try to find a polynomial that enhances the eigenvalue that we are looking
for. This approach is however not successful in the most critical case when the wanted
eigenvalue is very close to unwanted. In this situation, the shift-and-invert spectral
transformation is most appropriate. Instead of a polynomial we transform the matrix by
the rational function f () = 1/( ) where is a so-called shift that is chosen close to
the desired eigenvalue. Simple vector iteration with the matrix (A I)1 is referred to
as inverse vector iteration, see Algorithm 7.6.

(7.24) x(k) := (A I)1 x(k1) (A I) x(k) := x(k1) , k = 1, 2, . . .

The iteration converges towards the eigenvector with eigenvalue closest to . A linear
system of equations has to be solved in each iteration step. Of course only one Cholesky
or LU factorization has to be computed as the shift remains constants in all iterations.
The stopping criterion is changed into

(7.25) kx(k) y(k) /(k) k tolky(k) k

136 CHAPTER 7. VECTOR ITERATION (POWER METHOD)
2
10

0
10

2
10

4
10

6
10

8
10

10
10
0 100 200 300 400 500 600 700 800 900

Figure 7.7: Simple vector iteration with I40 T40 and starting vector (1, 1, . . . , 1)T

Algorithm 7.3 Inverse vector iteration

1: Choose a starting vector x0 Fn and a shift .
2: Compute the LU factorization of A I: LU = P (A I)

3: y(0) := U 1 L1 P x(0) . (0) = y(0) x(0) , (0) := + 1/(0) . k := 0.
4: while kx(k) y(k) /(k) k > tolky(k) k do
5: k := k + 1.
6: x(k) := yk1 /kyk1 k.
7: y(k) := U 1 L1 P x(k) .

8: (k) := y(k) x(k) , (k) := + 1/(k) .
9: end while

where we have used

 
(k) (k) (k) (k) 1
Ay y = Ay (k) y(k) = x(k) y(k) /(k)

The convergence result of Theorem 7.6 can easily be adapted to the new situation if
it is taken into account that A I has eigenpairs (i , ui ) with i = 1/( i ).
Theorem 7.10 Let A be symmetric positive definite with spectral decomposition (7.17).
Let 1 , . . . , n be a renumeration of the eigenvalues in (7.17) such that
1 1 1
(7.26) >
|1 | |2 | |n |

Then, provided
 (k) that u1 x(0) 6= 0, the inverse vector iteration of Algorithm 7.6 constructs

sequences k=0
and x(k) k=0 that converge linearly towards that eigenvalue 1 clos-
est to the shift and to the corresponding eigenvector u1 , respectively. The bounds
   
 1 k  1 2k 2 (0)
sin 
(k)  sin ,
(0)
1 
(k)  sin .
2  2 

hold with (k) = (x(k) , u1 ) and = spread(((A I)1 )).

7.6. INVERSE VECTOR ITERATION 137
(k)
If the shift approximates very well an eigenvalue of A then 1 and cover-
n
gence is very rapid.
Example 7.11 Let us now use inverse iteration to compute the smallest eigenvalue and
corresponding eigenvector of the one-dimensional Poisson matrix T = Tn of Example 2.7
with n = 40. If we assume that we know that the smallest eigenvalue 1 is around 1 then
a shift = .9 is reasonable, if we want A I to still be positive definite. Starting with
the vector of all ones three iteration steps suffice to get the desired accuracy of tol = 105 ,
see Table 7.1.
k (k) 1 sin((k) )
1 2.0188e-02 4.1954e-03
2 1.7306e-06 5.0727e-05
3 2.5289e-10 6.2492e-07

Table 7.1: Computing the lowest eigenvalue of the one-dimensinal Poisson equation by
inverse iteration

Example 7.12 We consider the problem of computing the eigenvector corresponding to a

known eigenvalue. The matrix that we consider is one of the so-called Wilkinson matrices

19 1
1 18 1

. . . . . .
. . .

1 1 1
T = .
1 1 1

.. .. ..
. . .

1 19 1
1 20
Wilkinson matrices are irreducible tridiagonal matrices that have very close eigenvalues.
This matrix has the eigenvalues
i i i i
1 -1.1254415221199814 11 5.0002362656192743
2 0.2538058170966502 12 5.9999918413270530
3 0.9475343675285830 13 6.0000083521880692
4 1.7893213526669509 14 6.9999997949295611
5 2.1302092192694015 15 7.0000002079042920
6 2.9610588806935558 16 7.9999999961918720
7 3.0430992883895192 17 8.0000000038418246
8 3.9960479973346419 18 8.9999999999455120
9 4.0043538173235769 19 9.0000000000548166
10 4.9997743198148310 20 9.9999999999996234
The following Matlab code constructs the sparse tridiagonal matrix T .
n = 40;
e = ones(n,1); f = abs([-n/2+1:n/2]);
T = spdiags([-e f -e], [-1:1], n, n);
lam = sort(eig(T));
138 CHAPTER 7. VECTOR ITERATION (POWER METHOD)
Computing the 20-th and 21-st eigenvectors could be done in the following way.

>> x = (T - lam(20)*eye(n))\e;
>> y = (T - lam(21)*eye(n))\e;
>> x = x/norm(x); y = y/norm(y);
>> x*y
ans =
0.00140329005834
>> norm((T - lam(20)*eye(n))*x)
ans =
7.325760095786749e-15
>> norm((T - lam(21)*eye(n))*y)
ans =
7.120036319503636e-15

The computed vectors x and y are good approximations in the sense that they give small
residuals. However, the two vectors are nor mutually orthogonal at all. We try to improve
orthogonality by apply a second step of inverse iteration

>> x = (T - lam(20)*eye(n))\x;
>> y = (T - lam(21)*eye(n))\y;
>> x = x/norm(x); y = y/norm(y);
>> x*y
ans =
-1.313592004487587e-05

Things have only slightly improved. Therefore, we orthogonalize y explicitely against x.

>> y = y - x*(x*y);
>> x*y
ans =
-2.155571068436496e-17
>> norm((T - lam(21)*eye(n))*y)
ans =
1.557058217172078e-15
>> norm((T - lam(20)*eye(n))*x)
ans =
4.117116818055497e-16

This helped. The two eigenvectors are now perpendicular on each other, and the residuals
are still fine.

Discussion of inverse iteration

We have seen that

we can compute eigenvectors corresponding to any (simple and well separated) eigen-
value if we choose the shift properly, and that

we have very good convergence rates, is the shift is close to an eigenvalue.

7.7. THE GENERALIZED EIGENVALUE PROBLEM 139
However, one may feel uncomfortable solving an almost singular system of equations,
after all k means that the condition of A I is very big. From the analysis of linear
systems of equations we know that this means large errors in the solution. Furtunately,
the error that is suffered from when solving with A I points in the right direction. To
see this, assume that the singular value decomposition of A I is given by

A I = U V , = diag(1 , . . . , n ), with 1 n 0.

If A I is almost singular then n 1. If even n = 0 then (A I)vn = 0, i.e.,

the last right singular vector is an eigenvector of A corresponding to the eigenvalue (the
shift).
If n = O() then
(A I)z = U V z = y.
Thus,
n
X ui y n n1 un y
z = V 1 U y = vi vn .
i n
i=1
The tiny n blows up the component in direction of vn . So, the vector z points in the
desired most singular direction.

7.7 The generalized eigenvalue problem

Applying the vector iteration (7.1) to the generalized eigenvalue problem Ax = Bx leads
to the iteration
x(k) := B 1 Ax(k1) , k = 1, 2, . . .
Since the solution of a linear system is required in each iteration step, we can execute an
inverse iteration right-away,

(7.27) (A B) x(k) := Bx(k1) , k = 1, 2, . . .

The iteration performs an ordinary vector iteration for the eigenvalue problem
1
(7.28) (A B)1 Bx := x, = .

Thus, the iteration (7.27) converges to the largest eigenvector of (7.28), i.e., the eigenvector
with eigenvalue closest to the shift .

7.8 Computing higher eigenvalues

In order to compute higher eigenvalues 2 , 3 , . . . , we make use of the mutual orthogonality
of the eigenvectors of symmetric matrices, see Theorem 2.14. (In the case of Schur vectors
we can proceed in a similar way.)
So, in order to be able to compute the second eigenpair (2 , u2 ) we have to know
the eigenvector u1 corresponding to the lowest eigenvalue. Most probably is has been
computed previously. If this is the case we can execute an inverse iteration orthogonal to
u1 .
More generally, we can compute the j-th eigenpair (j , uj ) by inverse iteration, keep-
ing the iterated vector x(k) orthogonal to the already known or computed eigenvectors
u1 , . . . , uj1 .
140 CHAPTER 7. VECTOR ITERATION (POWER METHOD)
Algorithm 7.4 Inverse vector iteration for computing (j , uj )
1: The LU factorization of A I: LU = P (A I)
and the eigenvectors u1 , . . . , uj1 are known.
2: Choose a starting vector x(0) such that uq x(0) = 0, q < j.
3: Set k := 0.
4: while kx(k) y(k) /(k) k > tolky(k) k do
5: k := k + 1;
6: x(k) := y(k1) /ky(k1) k;
7: y(k) := U 1 L1 P x(k) ;

8: (k) := y(k) x(k) , (k) := + 1/(k) .
9: end while

In exact arithmetic, the condition u1 x(0) = = uj1 x(0) = 0 implies that all x(k) are
orthogonal to u1 , . . . , uj1 . In general, however, one has to expect rounding errors that
introduce components in the directions of already computed eigenvectors. Therefore, it is
necessary to enforce the orthogonality conditions during the iteration.
Assuming exact arithmetic, Theorem 7.10 immediately implies that
 k

sin (x(k) , xj ) c1 j
 2k j

(k) j
| j | c2
j

where j is the smallest index for which j > j .

7.9 Rayleigh quotient iteration

We now assume that the matrix the eigenpairs of which we want to determine is Hermitian
(or symmetric).
We have noticed that inverse iteration is an effective way to compute eigenpairs, if a
good approximation of the desired eigenvalue is known. This approximation is used as
a shift. However, as we have seen earlier, if a good approximation of an eigenvector is
available its Rayleigh quotient gives a very good approximation of its eigenvalue.
Indeed we have the following

Lemma 7.13 Let q be any nonzero vector. The number that minimizes kAq qk is
the Rayleigh quotient

q Aq
(7.29) = .
q q

Proof. Let R be the Rayleigh quotient (7.29) of q 6= 0 and let C be any number.
Then we have

kAq ( + )qk2 = q A2 q (2 + + ) q Aq + | + |2 q q
= q A2 q 2 q Aq 2Re( ) q Aq + 2 q q + 2Re( ) q q + | |2 q q
(q Aq)2
= q A2 q + | |2 q q.
q q

The last term is smallest if = 0.

7.9. RAYLEIGH QUOTIENT ITERATION 141
Algorithm 7.5 Rayleigh quotient iteration (RQI)
1: Choose a starting vector y0 Fn with ky0 k = 1 and a tolerance .
2: for k = 1, 2, . . . do

3: (k) := y(k1) Ay(k1) .
4: Solve (A (k) I)z(k) = y(k1) for z(k) .
5: (k) = kz(k) k.
6: y(k) := z(k) / (k) .
7: if (k) > 10/ then
8: return {y(k) }
9: end if
10: end for

The following algorithm 7.9 is a modification of inverse iteration. In each iteration

step the shift is modified to be the Rayleigh quotient of the most recent eigenvector
approximation. This is not a curse but a blessing [4] as we have seen in section 7.6.
In step 4 of this algorithm a close to singular system of equation is solved. This results
in a very long solution whose norm is used a the convergence criterion.
The Rayleigh quotient iteration usually converges, however not always towards the
desired solution. Therefore, to investigate the convergence rate we make the following
Assumption: y(k) x with Ax = x.
k
This assumption garantees that there is at all convergence towards a certain eigenvector
x. Let kxk = ky(k) k = 1 and let the angle between this eigenvector and its approximation
be (k) = (x, y(k) ). Then the assumption implies that {(k) }
k=1 converges to zero. We
can write
y(k) = x cos (k) + u(k) sin (k) , kxk = ky(k) k = ku(k) k = 1.
Let
y(k) Ay(k)
(k) (k)
= (y ) = (k) (k) = y(k) Ay(k)
y y
be the Rayleigh quotient of yk . Then we have
k = cos2 k x
| {zAx} cos k sin k x Auk sin2 k uk Auk
| {z }
0
2 2
= (1 cos k ) sin k (uk )
= ( (uk )) sin2 k .
We now prove the
Theorem 7.14 (Local convergence of Rayleigh quotient iteration) With the above
assumption we have
 
 k+1 
(7.30) lim   1.
k  3 k


i.e., RQI converges cubically.

Proof. (The proof follows closely the one given by Parlett [4].) We have
zk+1 = (A k I)1 yk = x cos k /( k ) + (A k I)1 uk sin k
= x cos k /( k ) +uk+1 sin k k(A k I)1 uk k,
| {z } | {z }
kzk+1 k cos k+1 kzk+1 k sin k+1
142 CHAPTER 7. VECTOR ITERATION (POWER METHOD)
where we set

(7.31) uk+1 := (A k I)1 uk /k(A k I)1 uk k

such that kuk+1 k = 1 and uk+1 x = 0. Thus,

tan k+1 = sin k+1 /cos k+1

= sin k k(A k I)1 uk k ( k )/ cos k
= ( k ) k(A k I)1 uk k tan k
= ( (uk )) k(A k I)1 uk k sin2 k tan k .

So,
X X i
(A k I)1 uk = (A k I)1 i xi = xi
i k
i 6= i 6=

and taking norms,

X i2 1 X
(7.32) k(A k I)1 uk k2 = 2
2
i2
|i k | mini 6= |i k |
i 6= i 6=
| {z }
kuk k2 =1

We define the gap between the eigenvalue and the rest of As spectrum by

:= min |i |.
i 6=

The assumption implies that there must be a k0 N such that | k | < 2 for all k > k0 ,
and, therefore,

|i k | > for all i 6= .
2
Using this in (7.32) gives

1 2
k(A k I)1 uk k , k > k0 .
mini 6= |i k |

Because tan k sin k k if k 1 we can deduce the cubic convergence rate.

We now look at the sequence {uk } more closely. We note from (7.31) that this sequence
is obtained by inverse iteration with variable shift k . But since k with a cubic
rate of convergence we can for large k assume that k = and that uk x.
We now consider two cases, either {uk } converges, or it does not converge.

1. We assume that {uk } converges. Then the limit vector u must be an eigenvector of
A in span{x} . (In general, u
is an eigenvector corresponding to the eigenvalue
that is closest to .) Thus,

k
( (uk ))k(A k I)1 uk k | | = 1.
u/( )k
k

2. Now we assume that {uk } does not converge. Then A has two eigenvalues of equal
distance to and the cluster points of the sequence {uk } are two vectors in the plane
7.9. RAYLEIGH QUOTIENT ITERATION 143
that is spanned by two eigenvectors corresponding to these two eigenvalues ,
xp + xq , where 6= 0, 6= 0, and 2 + 2 = 1. Their Rayleigh quotients are

(xp + xq ) = 2 p + 2 q = 2 ( ) + 2 ( ) = (2 2 ).

As k the Rayleigh quotients of uk jump between these two values. Hence,

( (uk ))k(A k I)1 uk k (2 2 )/,

and, therefore,  
 k+1 
  |2 2 | < 1
 3  k
k

Remark 7.6. Notice that we have not proved global convergence. Regarding this issue
consult the book by Parlett [4] that contains all of this and more.
RQI converges almost always. However, it is not clear in general towards which
eigenpair the iteration converges. So, it is wise to start RQI only with good starting
vectors. An alternative is to first apply inverse vector iteration and switch to Rayleigh
quotient iteration as soon as the iterate is close enough to the solution. For references on
this technique see [6, 1].
Remark 7.7. The Rayleigh quotient iteration is expensive. In every iteration step another
system of equations has to be solved, i.e., in every iteration step a matrix has to be
factorized. Therefore, RQI is usually applied only to tridiagonal matrices.

7.9.1 A numerical example

The following Matlab script demonstrates the power of Rayleigh quotient iteration. It
expects as input a matrix A, an initial vector x of length one.

% Initializations
k = 0; rho = 0; ynorm = 0;

while abs(rho)*ynorm < 1e+15,

k = k + 1; if k>20, break, end
rho = x*A*x;
y = (A - rho*eye(size(A)))\x;
ynorm = norm(y);
x = y/ynorm;
end

We invoked this routine with the matrix

2 1
1 2 1

.. .. ..
A= . . . R99

1 2 1
1 2

and the initial vector x = [4, 3, . . . , 4]T . The numbers obtained are
144 CHAPTER 7. VECTOR ITERATION (POWER METHOD)
k rho ynorm
1 0.6666666666666666 3.1717e+00
2 0.4155307724080958 2.9314e+01
3 0.3820048793104663 2.5728e+04
4 0.3819660112501632 1.7207e+13
5 0.3819660112501051 2.6854e+16

The cubic convergence is evident.

Bibliography
[1] C. Beattie and D. Fox, Localization criteria and containment for Rayleigh Quotient
iteration, SIAM J. Matrix Anal. Appl., 10 (1989), pp. 8093.

[2] G. H. Golub and C. F. van Loan, Matrix Computations, The Johns Hopkins
University Press, Baltimore, MD, 4th ed., 2012.

[3] R. A. Horn and C. R. Johnson, Matrix Analysis, Cambridge University Press,

Cambridge, 1985.

[4] B. N. Parlett, The Symmetric Eigenvalue Problem, Prentice Hall, Englewood Cliffs,
NJ, 1980. (Republished by SIAM, Philadelphia, 1998.).

[5] H. R. Schwarz, Methode der finiten Elemente, Teubner, Stuttgart, 3rd ed., 1991.

[6] D. B. Szyld, Criteria for combining inverse and Rayleigh Quotient iteration, SIAM
J. Numer. Anal., 25 (1988), pp. 13691375.
Chapter 8

Simultaneous vector or subspace

iterations

8.1 Basic subspace iteration

We have learned in subsection 7.8 how to compute several eigenpairs of a matrix, one after
the other. This turns out to be quite inefficient. Some or several of the quotients i+1 /i
may be close to one. The following algorithm differs from Algorithm 7.8 in that it does
not perform p individual iterations for computing the, say, p smallest eigenvalues, but a
single iteration with p vectors simultaneously.

Algorithm 8.1 Basic subspace iteration

1: Let X Fnp be a matrix with orthnormalized columns, X X = Ip . This algorithmus
computes eigenvectors corresponding to the p largest eigenvalues 1 p of A.
2: Set X (0) := X, k = 1,

3: while k(I X (k) X (k) )X (k1) k > tol do
4: k := k + 1
5: Z (k) := AX (k1)
6: X (k) R(k) := Z (k) /* QR factorization of Z (k) */
7: end while

The QR factorization in step 6 of the algorithm prevents the columns of the X (k) from
converging all to an eigenvector of largest modulus.
If the convergence criterion is satisfied then

X (k) X (k1) (X (k1) X (k) ) = E, with kEk tol.

Therefore, for the residual,

AX (k1) X (k1) (X (k1) X (k) )R(k) = ER(k) ,

Therefore, in case of convergence, X (k) tends to an invariant subspace, say R(X ). X (k1) X (k)
Ip and AX (k) = X (k) R(k) is an approximation of a partial Schur decomposition. We will
show that the matrix X (k) converges to the Schur vectors associated with the largest
p eigenvalues of A. In the convergence analysis we start from the Schur decomposition
A = U T U of A.
It is important to notice that in the QR factorization of Z (k) the j-th column affects
only the columns to its right. If we would apply Algorithm 8.1 to a matrix X Fnq , with

145
146 CHAPTER 8. SIMULTANEOUS VECTOR OR SUBSPACE ITERATIONS
Xei = Xe i for i = 1, . . . , q then, for all k, we would have X (k) ei = X
(k) ei for i = 1, . . . , j.
(k)
This, in particular, means that the first columns X e1 perform a simple vector iteration.

Problem 8.1 Show by recursion that the QR factorization of Ak X = Ak X (0) is given by

Ak X = X (k) R(k) R(k1) R(1) .

8.2 Angles between subspaces

In the convorgence analysis of the subspace iteration we need the notion of an angle
between subspaces. Let Q1 Fnp , Q2 Fnq be matrices with orthonormal columns,
Q1 Q1 = Ip , Q2 Q2 = Iq . Let Si = R(Qi ). Then S1 and S2 are subspaces of Fn of dimension
p and q, respectively. We want to investigate how we can define a distance or an angle
between S1 and S2 [2].
It is certainly straightforward to define the angle between the subspaces S1 and S2 to
be the angle between two vectors x1 S1 and x2 S2 . It is, however, not clear right-
away how these vectors should be chosen. Let us consider the case of two 2-dimensional

q3

q1
q2

Figure 8.1: Two intersecting planes in 3-space

subspaces in R3 , cf. Fig. (8.1). Let S1 = span{q1 , q2 } and S2 = span{q1 , q3 } where we

assume that q1 q2 = q1 q3 = 0. We might be tempted to define the angle between S1 and
S2 as the maximal angle between any two vectors in S1 and S2 ,

(8.1) (S1 , S2 ) = max (x1 , x2 ).

x1 S1
x2 S2

This would give an angle of 90o as we could chose q1 in S1 and q3 in S2 . This angle
would not change as we turn S2 around q1 . It would even stay the same if the two planes
coincided.
What if we would take the minimum in (8.1)? This definition would be equally un-
satisfactory as we could chose q1 in S1 as well as in S2 to obtain an angle of 0o . In fact,
any two 2-dimensional subspaces in 3 dimensions would have an angle of 0o . Of course,
we would like to reserve the angle of 0o to coinciding subspaces.
A way out of this dilemma is to proceed as follows: Take any vector x1 S1 and
determine the angle between x1 and its orthogonal projection (I Q2 Q2 )x1 on S2 . We
now maximize the angle by varying x1 among all non-zero vectors in S1 . In the above
3-dimensional example we would obtain the angle between x2 and x3 as the angle between
8.2. ANGLES BETWEEN SUBSPACES 147
S1 and S3 . Is this a reasonable definition? In particular, is it well-defined in the sense
that it does not depend on how we number the two subspaces? Let us now assume that
S1 , S2 Fn have dimensions p and q. Formally, the above procedure gives an angle with

sin := max k(In Q2 Q2 )rk = max k(In Q2 Q2 )Q1 ak

rS1 aFp
(8.2) krk=1 kak=1

= k(In Q2 Q2 )Q1 k.

Because In Q2 Q2 is an orthogonal projection, we get

k(In Q2 Q2 )Q1 ak2 = a Q1 (In Q2 Q2 )(In Q2 Q2 )Q1 a

= a Q1 (In Q2 Q2 )Q1 a
(8.3) = a (Q1 Q1 Q1 Q2 Q2 Q1 )a
= a (Ip (Q1 Q2 )(Q2 Q1 ))a
= a (Ip W W )a

where W := Q2 Q1 Fqp . With (8.2) we obtain

sin2 = max a (Ip W W )a

kak=1
(8.4) = largest eigenvalue of Ip W W
= 1 smallest eigenvalue of W W .

If we change the roles of Q1 and Q2 we get in a similar way

(8.5) sin2 = k(In Q1 Q1 )Q2 k = 1 smallest eigenvalue of W W .

Notice, that W W Fpp and W W Fqq and that both matrices have equal rank.
Thus, if W has full rank and p < q then < = /2. However if p = q then W W and
W W have equal eigenvalues, and, thus, = . In this most interesting case we have

sin2 = 1 min (W W ) = 1 min

2
(W ),

where min (W ) is the smallest singular value of W [2, p.16].

For our purposes in the analysis of eigenvalue solvers the following definition is most
appropriate.

Definition 8.2 Let S1 , S2 Fn be of dimensions p and q and let Q1 Fnp and Q2

Fnq be matrices the columns of which form orthonormal bases of S1 and S2 , respectively,
i.e. Si = R(Qi ), i = 1, 2. Then we define the angle , 0 /2, between S1 and S2 by
q
1 2 (Q Q ) if p = q,
min 1 2
sin = sin (S1 , S2 ) =
1 if p 6= q.

If p = q the equations (8.2)(8.4) imply that

sin2 = max a (Ip W W )a = max b (Ip W W )b

kak=1 kbk=1
(8.6) = k(In Q2 Q2 )Q1 k = k(In Q1 Q1 )Q2 k
= k(Q1 Q1 Q2 Q2 )Q1 k = k(Q1 Q1 Q2 Q2 )Q2 k
148 CHAPTER 8. SIMULTANEOUS VECTOR OR SUBSPACE ITERATIONS
Let x S1 + S2 . Then x = q
1 + q i Si . We write
2 with q

1 + Q1 Q1 q
x=q 2 + (In Q1 Q1 )
q2 =: q1 + q2

with q1 = Q1 a and q2 = Q2 b = (In Q1 Q1 )Q2 b. Then

k(Q1 Q1 Q2 Q2 )xk2 = k(Q1 Q1 Q2 Q2 )(Q1 a + Q2 b)k2

= kQ1 a + Q2 Q2 Q1 a + Q2 bk2
= k(In Q2 Q2 )Q1 a + Q2 bk2
= a Q1 (In Q2 Q2 )Q1 a
+ 2Re(a Q1 (In Q2 Q2 )Q2 b) + b Q2 Q2 b
sin2 = max a Q1 (In Q2 Q2 )Q1 a,
kak=1

= max a Q1 (Q1 Q1 Q2 Q2 )Q1 a,

kak=1
x (Q1 Q1 Q2 Q2 )x
= max .
xS1 \{0} x x
Thus, sin is the maximum of the Rayleigh quotient R(x) corresponding to Q1 Q1 Q2 Q2 ,
that is the largest eigenvalue of Q1 Q1 Q2 Q2 . As Q1 Q1 Q2 Q2 is symmetric and positive
semi-definite, its largest eigenvalue equals its norm,
Lemma 8.3 sin (S1 , S2 ) = kQ2 Q2 Q1 Q1 k

Lemma 8.4 (S1 , S2 ) = (S1 , S2 ).

Proof. Because
kQ2 Q2 Q1 Q1 k = k(I Q2 Q2 ) (I Q1 Q1 )k
the claim immediately follows from Lemma 8.3.

8.3 Convergence of basic subspace iteration

In analyzing the convergence of the basic subspace iteration we proceed similarly as in
the analysis of the simple vector iteration, exploiting the Jordan normal form A = XJY
with Y := X 1 . We assume that the p largest eigenvalues of A in modulus are separated
from the rest of the spectrum,

(8.7) |1 | |p | > |p+1 | |n |.

This means that the eigenvalues are arranged on the diagonal of the Jordan block matrix
J in the order given in (8.7).
In fact as we can either analyze the original iteration {X (k) } in the canonical coordi-
nate system or the iteration {Y (k) } = {U X (k) } in the coordinate system generated by
the (generalized) eigenvectors we assume that A itself is a Jordan block matrix with its
diagonal elements arranged as in (8.7).
The invariant subspace of A associated with the p largest (or dominant) eigenvalues is
given by R(Ep ) where Ep = [e1 , . . . , ep ]. We are now going to show that the angle between
R(X (k) ) and R(Ep ), tends to zero as k goes to .
From Problem 8.1 we know that

(8.8) (k) := (R(Ep ), R(X (k) )) = (R(Ep ), R(Ak X (0) )).

8.3. CONVERGENCE OF BASIC SUBSPACE ITERATION 149
We partition the matrices A and X (k) ,
" #
(k)
(k) X1 (k)
A = diag(J1 , J2 ), X = (k) , J1 , X1 Fpp .
X2

(k)
From (8.7) we know that J1 is nonsingular. Let us also assume that X1 = Ep X (k) is
invertible. This means, that X (k) has components in the direction of the invariant subspace
associated with the p dominant eigenvalues. Then, with Problem 8.1,
" #  
(0)
(k) k (0) J1k X1 Ip k (0) (k) k (0) (0)
1 k
(8.9) X R = A X = (0) = (k) J1 X1 , S := J2 X 2 X 1 J1 .
J2k X2 S

(k) (0)
Notice that X1 is invertible if X1 is so. (8.8) and (8.9) imply that

sin (k) = k(I Ep Ep )X (k) k

    
(8.10) Ip Ip kS (k) k

= (I Ep Ep ) (k) (k)
=p .
S S 1 + kS (k) k2

Likewise, we have
1
cos (k) = kEp X (k) k = p .
1 + kS (k) k2
Since (J2 ) = |p+1 | and (J11 ) = 1/|p | we obtain with (7.13) and a few algebraic
manipulations for any > 0 that
  k
 p+1 
(8.11) tan (k)
= kS (k)
k kJ2k kkS (0) kkJ1k k 
  + tan (0) ,
p 

for k > K(). Making a transformation back to a general matrix A as before Theorem 7.5
we get

Theorem 8.5 Let Up , Vp Fnp , Up Up = Vp Vp = Ip , be matrices that span the right and
left invariant subspace associated with the dominant p eigenvalues 1 , . . . , p of A. Let
X (0) Fnp be such that Vp X (0) is nonsingular. Then, if |p | < |p+1 | and > 0, the
iterates X (k) of the basic subspace iteration with initial subpace X (0) converges to Up , and
  k
 p+1 
(8.12) tan (k) 
  + tan (0) , (k) = (R(Up ), R(X (k) ))
p 

for sufficiently large k.

If the matrix A is Hermitian or real-symmetric we can simplify Theorem 8.5.

Theorem 8.6 Let Up := [u1 , . . . , up ] be the matrix formed by the eigenvectors correspond-
ing to the p dominant eigenvalues 1 , . . . , p of A. Let X (0) Fnp be such that Up X (0) is
nonsingular. Then, if |p | < |p+1 |, the iterates X (k) of the basic subspace iteration with
initial subpace X (0) converges to Up , and
 
 p+1 k
(8.13) tan (k) 
  tan (0) , (k) = (R(Up ), R(X (k) )).
p 
150 CHAPTER 8. SIMULTANEOUS VECTOR OR SUBSPACE ITERATIONS
Let us elaborate on this result. (Here we assume that A is Hermitian or real-symmetric.
Otherwise the statements are similar modulo terms as in (8.12).) Let us assume that
not only Wp := Up X is nonsingular but that each principal submatrix

w11 w1j
.. , 1 j p,
Wj := ... .
wj1 wjj
(k) (k)
of Wp is nonsingular. Then we can apply Theorem 8.6 to each set of columns [x1 , . . . , xj ],
1 j p, provided that |j | < |j+1 |. If this is the case, then
 
(k)  j+1 k
(8.14) tan j    tan (0) ,
j  j

(k) (k) (k)

where j = (R([u1 , . . . , uj ]), R([x1 , . . . , xj ])).
We can even say a little more. We can combine the statements in (8.14) as follows.
Theorem 8.7 Let X Fnp . Let |q1 | > |q | . . . |p | > |p+1 |. Let Wq and Wp be
nonsingular. Then
(    )
 q k  p+1 k
(8.15) sin (R([xq , . . . , xp ]), R([uq , . . . , up ])) c max 
(k) (k)  ,  .
q1   p 

Proof. Recall that the sine of the angle between two subspaces S1 , S2 of equal dimension is
(k) (k)
the norm of the projection on S2 restricted to S1 , see (8.6). Here, S1 = R([xq , . . . , xp ])
and S2 = R([uq , . . . , up ]).
Let x S1 with kxk = 1. The orthogonal projection of x on S2 reflects the fact, that
y R([uq , . . . , up ]) implies that y R([u1 , . . . , up ]) as well as y R([u1 , . . . , uq ]) ,

Uq1 Uq1 x + (I Up Up )x.

To estimate the norm of this vector we make use of Lemmata 8.4 and (8.10),


1/2
kUq1 Uq1 x + (I Up Up )xk = kUq1 Uq1

xk2 + k(I Up Up )xk2
 1/2 n o
(k) (k)
sin2 q1 + sin2 (k)
p 2 max sin q1 , sin (k)
p
n o
(k)
2 max tan q1 , tan (k)
p .

Then, inequality (8.15) is obtained by applying (8.14) that we know to hold true for both
j = q1 and j = p.

Corollary 8.8 Let X Fnp . Let |j1 | > |j | > |j+1 | and let Wj1 and Wj be
nonsingular. Then
(    )
(k)  j k  j+1 k
(8.16) sin (xj , uj ) c max   ,  .
j1   j 

Example 8.9 Let us see how subspace iteration performs with the matrix

A = diag(1, 3, 4, 6, 10, 15, 20, . . . , 185)1 R4040

if we iterate with 5 vectors. The critical quotients appearing in Corollary 8.8 are
8.3. CONVERGENCE OF BASIC SUBSPACE ITERATION 151
j 1 2 3 4 5
.
|j+1 |/|j | 1/3 3/4 2/3 3/5 2/3
(k)
So, according to (8.16), the first column x1 of X (k) R405 should converge to the first
(k) (k)
eigenvector at a rate 1/3, x2 and x3 should converge at a rate 3/4 and the last two
columns should converge at the rate 2/3. The graphs in Figure 8.2 show that convergence
0
10

2
10

4
10

6
10

8
10

10
sin(1)
10 sin(2)
sin(3)
sin(4)
12
10 sin(5)

14
10
0 5 10 15 20 25

Figure 8.2: Basic subspace iteration with I40 T40

takes place in exactly this manner.

(k)
Similarly as earlier the eigenvalue approximations j approach the desired eigenvalues
more rapidly than the eigenvectors. In fact we have
(k) (k)
(k+1) 2 (k+1) 2 xj A2 xj (k) (k)
j = kzj k = (k) (k)
= xj A2 xj ,
xj xj

(k) (k)
since kxj k = 1. Let xj = u + u , where u is the eigenvalue corresponding to j . Then,
(k) (k)
since u = xj cos and u = xj sin for a (k) , we have

(k+1) 2 (k) (k)

j = xj A2 xj = u Au + u Au = 2j u u + u Au
2j kuk2 + 21 ku k2
2j cos2 (k) + 21 sin2 (k)
= 2j (1 sin2 (k) ) + 21 sin2 (k) = 2j + (21 2j ) sin2 (k) .

Thus,

(k+1) 2 ( k   )!
(k+1) 21 j j j+1 k
|j j | (k+1)
sin2 (k) = O max , .
j + j j1 j
152 CHAPTER 8. SIMULTANEOUS VECTOR OR SUBSPACE ITERATIONS

(k1) (k1) (k1) (k1) (k1)

1 1 2 2 3 3 4 4 5 5
k (k) (k) (k) (k) (k)
1 1 2 2 3 3 4 4 5 5
1 0.0002 0.1378 -0.0266 0.0656 0.0315
2 0.1253 0.0806 -0.2545 0.4017 -1.0332
3 0.1921 0.1221 1.5310 0.0455 0.0404
4 0.1940 0.1336 0.7649 -3.0245 -10.4226
5 0.1942 0.1403 0.7161 0.9386 1.1257
6 0.1942 0.1464 0.7002 0.7502 0.9327
7 0.1942 0.1522 0.6897 0.7084 0.8918
8 0.1942 0.1574 0.6823 0.6918 0.8680
9 0.1942 0.1618 0.6770 0.6828 0.8467
10 0.1942 0.1652 0.6735 0.6772 0.8266
11 0.1943 0.1679 0.6711 0.6735 0.8082
12 0.1942 0.1698 0.6694 0.6711 0.7921
13 0.1933 0.1711 0.6683 0.6694 0.7786
14 0.2030 0.1720 0.6676 0.6683 0.7676
15 0.1765 0.1727 0.6671 0.6676 0.7589
16 0.1733 0.6668 0.6671 0.7522
17 0.1744 0.6665 0.6668 0.7471
18 0.2154 0.6664 0.6665 0.7433
19 0.0299 0.6663 0.6664 0.7405
20 0.6662 0.6663 0.7384
21 0.6662 0.6662 0.7370
22 0.6662 0.6662 0.7359
23 0.6661 0.6662 0.7352
24 0.6661 0.6661 0.7347
25 0.6661 0.6661 0.7344
26 0.6661 0.6661 0.7343
27 0.6661 0.6661 0.7342
28 0.6661 0.6661 0.7341
29 0.6661 0.6661 0.7342
30 0.6661 0.6661 0.7342
31 0.6661 0.6661 0.7343
32 0.6661 0.6661 0.7343
33 0.6661 0.6661 0.7344
34 0.6661 0.6661 0.7345
35 0.6661 0.6661 0.7346
36 0.6661 0.6661 0.7347
37 0.6661 0.6661 0.7348
38 0.6661 0.6661 0.7348
39 0.6661 0.6661 0.7349
40 0.6661 0.6661 0.7350

Table 8.1: Example of basic subspace iteration.

The convergence criterion k(I X (k1) X (k1) )X (k) k < 106 was satisfied after 87 itera-
tion steps
8.4. ACCELERATING SUBSPACE ITERATION 153
A numerical example
Let us again consider the test example introduced in subsection 1.6.3 that deals with the
accustic vibration in the interior of a car. The eigenvalue problem for the Laplacian is
solved by the finite element method as introduced in subsection 1.6.2. We use the finest
grid in Fig. 1.9. The matrix eigenvalue problem
(8.17) Ax = Bx, A, B Fnn ,
has the order n = 1095. Subspace iteration is applied with five vectors as an inverse
iteration to
L1 ALT (Lx) = (Lx), B = LLT . (Cholesky factorization)
X (0) is chosen to be a random matrix. Here, we number the eigenvalues from small to big.
The smallest six eigenvalues of (8.17) shifted by 0.01 to the right are
1 = 0.01,
4 = 0.066635,

2 = 0.022690,
5 = 0.126631,

3 = 0.054385,
6 = 0.147592.

and thus the ratios of the eigenvalues that determine the rate of convergence are
1 /
( 2 )2 = 0.194, 4 /
( 5 )2 = 0.277,
2 /
( 3 )2 = 0.174, 5 /
( 6 )2 = 0.736,
3 /
( 4 )2 = 0.666.

So, the numbers presented in Table 8.1 reflect quite accurately the predicted rates. The
numbers in column 6 are a little too small, though.
The convergence criterion
(k1)
max k(I X (k) X (k) )xi k = 105
1ip

was not satisfied after 50 iteration step.

8.4 Accelerating subspace iteration

Subspace iteration potentially converges very slowly. It can be slow even it one starts with
(0) (0)
a subspace that contains all desired solutions! If, e.g., x1 and x2 are both elements in
(k)
R([u1 , u2 ]), the vectors xi , i = 1, 2, . . ., still converge linearly towards u1 und u2 although
they could be readily obtained from the 2 2 eigenvalue problem,
" #
(0) h i
x1 (0) (0)
(0) A x1 , x2
y = y
x2
The following theorem gives hope that the convergence rates can be improved if one
proceeds in a suitable way.
Theorem 8.10 Let X Fnp as in Theorem 8.5. Let ui , 1 i p, be the eigenvectors
corresponding to the eigenvalues 1 , . . . , p of A. Then we have
 
i k
min sin (ui , x) c
xR(X (k) ) p+1
154 CHAPTER 8. SIMULTANEOUS VECTOR OR SUBSPACE ITERATIONS
Proof. In the proof of Theorem 8.5 we have seen that
  
Ip
R(X (k) ) = R U , S(k) F(np)p ,
S(k)

where  k
(k) j
sij = sij , 1 i n p, 1 j p.
p+i
But we have
   
Ip
min sin (ui , x) sin ui , U ei ,
xR(X (k) ) S(k)

0

..
.

0

1  

0 Ip

= (I ui ui )U / (k) ei

.. S
.

s1i (i /p+1 )k

..
.

snp,i (i /n ) k

Xn  k
i

(I ui ui ) ui + sjp,i uj

p+j
j=p+1
v v
unp   unp
uX
t 2 2k
i i k u t
X
= sji 2k s2ji .
p+j p+1
j=1 j=1

These considerations lead to the idea to complement Algorithm 8.1 by a so-called

Rayleigh-Ritz step. Here we give an inverted algorithm to compute the smallest
eigenvalues and corresponding eigenvectors of a symmetric/Hermitian matrix. For the
corresponding nonsymmetric algorithm see [1].

Algorithm 8.2 Subspace or simultaneous inverse iteration combined with

Rayleigh-Ritz step
1: Let X Fnp with X X = Ip :
2: Set X (0) := X.
3: for k = 1, 2, . . . do
4: Solve AZ (k) := X (k1)
5: Q(k) R(k) := Z (k) /* QR factorization of Z (k) (or modified GramSchmidt) */
(k) (k) (k)
6: H := Q AQ ,
(k) Fpp */
7: H =: F (k) (k) F (k)
(k) /* Spectral decomposition of H
8: (k)
X =Q F . (k) (k)

9: end for

(k)
Remark 8.1. The columns xi of X (k) are called Ritz vectors and the eigenvalues
(k) (k)
1 p in the diagonal of are called Ritz values. According to the Rayleigh-
8.4. ACCELERATING SUBSPACE ITERATION 155
Ritz principle 2.32 we have
(k)
i i 1 i p, k > 0.

The solution of the full eigenvalue problems H (k) y = y is solved by the symmetric
QR algorithm.
The computation of the matrix H (k) is expensive as matrix-vector products have to
be executed. The following considerations simplify matters. We write X (k) in the form

X (k) = Z (k) G(k) , G(k) Fpp nonsingular

Because X (k) must have orthonormal columns we must have

(8.18) G(k) Z (k) Z (k) G(k) = Ip .

Furthermore, the columns of Z (k) G(k) are the Ritz vectors in R(Ak X) of A2 ,
2
G(k) Z (k) A2 Z (k) G(k) = (k) ,

where (k) is a diagonal matrix. Using the definition of Z (k) in Algorithm 8.2 we see that
2
G(k) X (k1) X (k1) G(k) = G(k) G(k) = (k) ,

and that Y (k) := G(k) (k) is orthogonal. Substituting into (8.18) gives
2
Y (k) Z (k) Z (k) Y (k) = (k) .

So, the columns of Y (k) are the normalized eigenvectors of H (k) := Z (k) Z (k) .
Thus we obtain a second variant of the inverse subspace iteration with Rayleigh-Ritz
step.

Algorithm 8.3 Subspace or simultaneous inverse vector iteration combined

with Rayleigh-Ritz step, version 2
1: Let X Fnp with X X = Ip .
2: Set X (0) := X.
3: for k = 1, 2, . . . do
4: AZ (k) := X (k1) ;

5: H (k) := Z (k) Z (k) /* = X (k1) A2 X (k1) */
2
6: H (k) =: Y (k) (k) Y (k) /* Spectral decomposition of H (k) */
1
7: X (k) = Z (k) Y (k) (k) /* = Z (k) G(k) */
8: end for

Remark 8.2. An alternative to Algorithm 8.3 is the subroutine ritzit, that has been
programmed by Rutishauser [5] in ALGOL, see also [3, p.293].
We are now going to show that the Ritz vectors converge to the eigenvectors, as
Theorem 8.10 lets us hope. First we prove
Lemma 8.11 ([3, p.222]) Let y be a unit vector and F. Let be the eigenvalue of A
closest to and let u be the corresponding eigenvector. Let

:= min |i (A) |
i (A)6=
156 CHAPTER 8. SIMULTANEOUS VECTOR OR SUBSPACE ITERATIONS
and let = (y, u). Then
kr(y)k kAy yk
sin := ,

where r(y, ) = Ay y plays the role of a residual.
Proof. We write y = u cos + v sin with kvk = 1. Then

r(y, ) = Ay y = (A I)u cos + (A I)v sin ,

= ( )u cos + (A I)v sin .

Because u (A I)v = 0, Pythagoras theorem implies

kr(y, )k2 = ( )2 cos2 + k(A I)vk2 sin2 2 kvk2 sin2 .

(k)
Theorem 8.12 ([3, p.298]) Let the assumptions of Theorem 8.5 be satisfied. Let xj =
 
(k) (k) I
X ej be the j-th Ritz vector as computed be Algorithm 8.3, and let yi = U ei
S(k)
(cf. the proof of Theorem 8.5). Then the following inequality holds
 
(k) (k) i k
sin (xi , yi ) c , 1 i p.
p+1
 
Ip
Proof. The columns of U form a basis of R(X (k) ). Therefore, we can write
S(k)
 
(k) Ip
xi = U ti , ti Fp .
S (k)

Instead of the special eigenvalue problem

X (k1) A2 X (k1) y = H (k) y = 2 y

in the orthonormal basis X (k) we consider the equivalent eigenvalue problem

h i   h i I 
(k) 2 Ip 2 (k) p
(8.19) Ip , S UA U t= Ip , S t.
S (k) S (k)

Let (, t) be an eigenpair of (8.19). Then we have

h i   h i I 
(k) 2 Ip 2 (k) p
0 = Ip , S UA U t Ip , S t
S (k) S (k)
   
(k) 2 (k) (k) (k)
= 21 + S 2 S t 2
I p + S S t,
 
(k)
(8.20) = (2 2
1 I) + S (2 2
2 I)S
(k)
t
 
k (0) k
= (21 2
I) + 1 S 2 ( 2
2 2
I) k (0) k
2 S 1 t
 k  k !
1 1
= (2 1 2
I) + 1 H k 1 t
p+1 p+1

with

Hk = 2k
p+1 S
(0)
k 2 2 k (0)
2 (2 I)2 S .
8.4. ACCELERATING SUBSPACE ITERATION 157
As the largest eigenvalue of 1
2 is 1/p+1 , Hk is bounded,

kHk k c1 k > 0.

Thus,
 k  k !
1 1
1 Hk 1 t 0.
p+1 p+1

Therefore, in (8.20) we can interpret this expression as an perturbation of the diagonal

matrix 2 2
2 I. For sufficiently large k (that may depend on i) there is a i that is
close to i , and a ti that is close to ei . We now assume that k is so big that
1
|2 1
i i | := min |2 2
j |
2 j 6=i i

such that 2 i is closer to 2

i than to any other 2 j , j 6= i.
We now consider the orthonormal basis
  1/2
Ip
B= (k) Ip + S (k) S (k) .
S
   
2 (k) (k) 1/2
If (i , ti ) is an eigenpair of (8.19) or (8.20), respectively, then i , Ip + S S ti
is an eigenpair of

(8.21) B A2 Bt = 2 t.

As, for sufficiently  large k, (2 2

i , ei ) is a goodapproximation of the eigenpair (i , ti )
 1/2
of (8.20), then also 2 (k) S (k)
i , Ip + S ei is a good approximation to the eigenpair
  1/2 
2 (k) S (k)
i , Ip + S ti of (8.21). We now apply Lemma 8.11 with

= , = 2
i ,
   
(k) (k)
1/2 (k) 1/2
y = Ip + S S ei / Ip + S
(k)
S ei
,
 1/2  1/2
(k) (k) (k) (k)
u = Ip + S S t i / Ip + S S ti
.

Now we have
 1/2 1/2
2 (k) (k)
 (k) (k)

2
kr(y)k = (B A B i I) Ip + S S
ei Ip + S S ei

 
 1 h i   

(k) (k) (k) Ip 1  (k) (k)

ei
2 2
Ip + S S Ip , S UA U (k) 2 Ip + S S
S i
       
(k) (k)
1/2 k k
Ip + S S 2 2 I + 1 1 Hk 1 1
1 i p+1 p+1 ei

 k  k
1

p+1 1 H k 1
p+1 1 ei

k  k  k  
1
1 ei c1 p i k
1
p+1 1 kHk k p+1 .
p+1 p+1
158 CHAPTER 8. SIMULTANEOUS VECTOR OR SUBSPACE ITERATIONS
Then, Lemma 8.11 implies that
 1/2  1/2  c  k
(k) (k) (k) (k) (k) (k) 1 i
sin (xi , yi ) = sin Ip + S S t i , Ip + S S ei .
p+1

In the proof of Theorem 8.10 we showed that

 
(k) i k
(ui , yi ) c .
p+1

In the previous theorem we showed that

 k
(k) (k) i
(xi , yi ) c1 .
p+1

By consequence,
 k
(k) i
(xi , ui ) c2
p+1
must be true for a constant c2 independent of k.
As earlier, for the eigenvalues we can show that
 2k
(k) i
|i i | c3 .
p+1

A numerical example
For the previous example that is concerned with the accustic vibration in the interior of a
car the numbers listed in Table 8.2 are obtained. The quotients 2 /
2 , that determine
i p+1
the convergence behavior of the eigenvalues are

1 /
( 6 )2 = 0.004513, 4 /
( 6 )2 = 0.2045,
2 /
( 6 )2 = 0.02357, 5 /
( 6 )2 = 0.7321.
3 /
( 6 )2 = 0.1362,

The numbers in the table confirm the improved convergence rate. The convergence rates of
the first four eigenvalues have improved considerably. The predicted rates are not clearly
visible, but they are approximated quite well. The convergence rate of the fifth eigenvalue
(k) (k)
has not improved. The convergence of the 5-dimensional subspace R([x1 , . . . , x5 ]) to
the searched space R([u1 , . . . , u5 ]) has not been accelerated. Its convergence rate is still
5 /6 according to Theorem 8.5
(k)
By means of the Rayleigh-Ritz step we have achieved that the columns xi = x(k)
converge in an optimal rate to the individual eigenvectors of A.

8.5 Relation between subspace iteration and QR algorithm

The connection between (simultaneous) vector iteration and the QR algorithm has been
investigated by Parlett and Poole [4].
Let X0 = In , the n n identity matrix.
8.5. RELATION BETWEEN SUBSPACE ITERATION AND QR ALGORITHM 159

(k1) (k1) (k1) (k1) (k1)

1 1 2 2 3 3 4 4 5 5
k (k) (k) (k) (k) (k)
1 1 2 2 3 3 4 4 5 5
1 0.0001 0.0017 0.0048 0.0130 0.0133
2 0.0047 0.0162 0.2368 0.0515 0.2662
3 0.0058 0.0273 0.1934 0.1841 0.7883
4 0.0057 0.0294 0.1740 0.2458 0.9115
5 0.0061 0.0296 0.1688 0.2563 0.9195
6 0.0293 0.1667 0.2553 0.9066
7 0.0288 0.1646 0.2514 0.8880
8 0.0283 0.1620 0.2464 0.8675
9 0.0275 0.1588 0.2408 0.8466
10 0.1555 0.2351 0.8265
11 0.1521 0.2295 0.8082
12 0.1490 0.2245 0.7921
13 0.1462 0.2200 0.7786
14 0.1439 0.2163 0.7676
15 0.1420 0.2132 0.7589
16 0.1407 0.2108 0.7522
17 0.1461 0.2089 0.7471
18 0.1659 0.2075 0.7433
19 0.1324 0.2064 0.7405
20 0.2054 0.7384
21 0.2102 0.7370
22 0.2109 0.7359
23 0.7352
24 0.7347
25 0.7344
26 0.7343
27 0.7342
28 0.7341
29 0.7342
30 0.7342
31 0.7343
32 0.7343
33 0.7344
34 0.7345
35 0.7346
36 0.7347
37 0.7348
38 0.7348
39 0.7349
40 0.7350

Table 8.2: Example of accelerated basic subspace iteration.

160 CHAPTER 8. SIMULTANEOUS VECTOR OR SUBSPACE ITERATIONS
Then we have

AI = A0 = AX0 = Y1 = X1 R1 (SV I)
A1 = X1 AX1 = X1 X1 R1 X1 = R1 X1 (QR)
AX1 = Y2 = X2 R2 (SV I)
A1 = X1 Y2 = X1 X2 R2 (QR)
A2 = R2 X1 X2 (QR)
= X2 X1 X1 X2 R2 X1 X2 = X2 AX2 (QR)
| {z }
A1

More generally, by induction, we have

AXk = Yk+1 = Xk+1 Rk+1 (SV I)

Ak = Xk AXk = Xk Yk+1 = Xk Xk+1 Rk+1
Ak+1 = Rk+1 Xk Xk+1 (QR)

= Xk+1
Xk Xk Xk+1 Rk+1 Xk Xk+1 = Xk+1 AXk+1 (QR)
| {z }
Ak
| {z }
A
Relation to QR: Q1 = X1 , Qk = Xk Xk+1 .

Ak = Ak X0 = Ak1 AX0 = Ak1 X1 R1

= Ak2 AX1 R1 = Ak2 X2 R2 R1
..
.
= Xk Rk Rk1 R1 = Xk Uk (QR)
| {z }
Uk

Because Uk is upper triangular we can write

u11 u1p
k .. ..
A [e1 , . . . , ep ] = Xk Uk [e1 , . . . , ep ] = Xk Uk (:, 1 : p) = Xk (:, 1 : p) . .
upp

This holds for all p. We therefore can interpret the QR algorithm as a nested sub-
space iteration. There is also a relation to simultaneous inverse vector iteration! Let us
assume that A is invertible. Then we have,1

AXk1 = Xk1 Ak1 = Xk Rk

Xk Rk = A Xk1 , Rk is lower triangular

k
Xk Rk Rk1

R1 = A X0
| {z }

Uk

1
Notice that A = (A1 ) = (A )1 .
8.6. ADDENDUM 161
Then,
u
,
.. ..

k
Xk [e , . . . , en ] . . = A X0 [e , . . . , en ]
u
n, u
n,n
By consequence, the last n + 1 columns of Xk execute a simultaneous inverse
vector iteration. This holds for all . Therefore, the QR algorithm also performs a nested
inverse subspace iteration.

8.6 Addendum
Let A = H be an irreducible Hessenberg matrix and W1 = [w1 , . . . , wp ] be a basis of the
p-th dominant invariant subspace of H ,

H W1 = W1 S, S invertible.

Notice that the p-th dominant invariant subspace is unique if |p | > |p+1 |.
Let further X0 = [e1 , . . . , ep ]. Then we have the

Theorem 8.13 W1 X0 is nonsingular.

Remark 8.3. If W1 X0 is nonsingular then Wk X0 is nonsingular for all k > 0.

Proof. If W1 X0 were singular then there was a vector a Fp with X0 W1 a = 0. Thus,
w = W1 a is orthogonal to e1 , . . . , ep . Therefore, the first p components of w are zero.
From, H W1 = W1 S we have that (H )k w R(W1 ) for all k.
But we have

0 0

.. .. ..
. p zeros . p 1 zeros .


..
0 0 .
w=
H
w =

(H
)k
w = .
..
.. ..
. . ..
.

These vectors evidently are linearly independent.

So, we have constructed p + 1 linearly independent vectors w, . . . , (H )p w in the p-
dimensional subspace R(W1 ). This is a contradiction.

Bibliography
[1] Z. Bai and G. W. Stewart, Algorithm 776. SRRIT A FORTRAN subroutine
to calculate the dominant invariant subspace of a nonsymmetric matrix, ACM Trans.
Math. Softw., 23 (1997), pp. 494513.

[2] G. H. Golub and C. F. van Loan, Matrix Computations, The Johns Hopkins
University Press, Baltimore, MD, 2nd ed., 1989.

[3] B. N. Parlett, The Symmetric Eigenvalue Problem, Prentice Hall, Englewood Cliffs,
NJ, 1980. (Republished by SIAM, Philadelphia, 1998.).
162 CHAPTER 8. SIMULTANEOUS VECTOR OR SUBSPACE ITERATIONS
[4] B. N. Parlett and W. G. Poole, A geometric theory for the QR, LU, and power
iterations, SIAM J. Numer. Anal., 8 (1973), pp. 389412.

[5] H. Rutishauser, Simultaneous iteration method for symmetric matrices, Numer.

Math., 16 (1970), pp. 205223. Reprinted in: Linear Algebra, J.H. Wilkinson, C.
Reinsch (eds.), pp. 284301, Springer, Berlin, 1971.
Chapter 9

Krylov subspaces

9.1 Introduction
In the power method or in the inverse vector iteration we computed, up to normalization,
sequences of the form
x, Ax, A2 x, . . .
The information available at the k-th step of the iteration is the single vector x(k) =
 (0)
A x/kA xk. One can pose the question if discarding all the previous information x , . . . , x(k1)
k k

is not a too big waste of information. This question is not trivial to answer. On one hand
there is a big increase of memory requirement, on the other hand exploiting all the infor-
mation computed up to a certain iteration step can give much better approximations to
the searched solution. As an example, let us consider the symmetric matrix

2 1
 2 1 2 1
51 .. .. ..


T = . . . R5050 .

1 2 1
1 2

the lowest eigenvalue of which is around 1. Let us choose x = [1, . . . , 1] and compute the
first three iterates of inverse vector iteration, x, T 1 x, and T 2 x. We denote their Rayleigh
(k) (k) (k)
k (k) 1 2 3
1 10.541456 10.541456
2 1.012822 1.009851 62.238885
3 0.999822 0.999693 9.910156 147.211990

(k)
Table 9.1: Ritz values j vs. Rayleigh quotients (k) of inverse vector iterates.

(k)
quotients by (1) , (2) , and (3) , respectively. The Ritz values j , 1 j k, obtained
with the Rayleigh-Ritz procedure with Kk (x) = span(x, T 1 x, . . . , T 1k x), k = 1, 2, 3,
are given in Table 9.1. The three smallest eigenvalues of T are 0.999684, 3.994943, and
(3)
8.974416. The approximation errors are thus (3) 1 0.000 14 and 1 1 0.000 009,
which is 15 times smaller.
These results immediately show that the cost of three matrix vector multiplications
can be much better exploited than with (inverse) vector iteration. We will consider in this

163
164 CHAPTER 9. KRYLOV SUBSPACES
section a kind of space that is very often used in the iterative solution of linear systems
as well as of eigenvalue problems.

9.2 Definition and basic properties

Definition 9.1 The matrix

(9.1) K m (x) = K m (x, A) := [x, Ax, . . . , A(m1) x] Fnm

generated by the vector x Fn is called a Krylov matrix. Its columns span the Krylov
(sub)space
n o
(9.2) Km (x) = Km (x, A) := span x, Ax, A2 x, . . . , A(m1) x = R (K m (x)) Fn .

The Arnoldi and Lanczos algorithms are methods to compute an orthonormal basis
of the Krylov space. Let
h i
x, Ax, . . . , Ak1 x = Q(k) R(k)

be the QR factorization of the Krylov matrix K m (x). The Ritz values and Ritz vectors
of A in this space are obtained by means of the k k eigenvalue problem

(9.3) Q(k) AQ(k) y = (k) y.
(k) (k)
If (j , yj ) is an eigenpair of (9.3) then (j , Q(k) yj ) is a Ritz pair of A in K m (x).
The following properties of Krylov spaces are easy to verify [1, p.238]

1. Scaling. Km (x, A) = Km (x, A), , 6= 0.

2. Translation. Km (x, A I) = Km (x, A).

3. Change of basis. If U is unitary then U Km (U x, U AU ) = Km (x, A).

In fact,

K m (x, A) = [x, Ax, . . . , A(m1) x]

= U [U x, (U AU )U x, . . . , (U AU )m1 U x],
= U K m (U x, U AU ).

Notice that the scaling and translation invariance hold only for the Krylov subspace, not
for the Krylov matrices.
What is the dimension of Km (x)? It is evident that for n n matrices A the columns
of the Krylov matrix K n+1 (x) are linearly dependent. (A subspace of Fn cannot have a
dimension bigger than n.) On the other hand if u is an eigenvector corresponding to the
eigenvalue then Au = u and, by consequence, K2 (u) = span{u, Au} = span{u} =
K1 (u). So, there is a smallest m, 1 m n, depending on x such that

K1 (x) 2 m
6= K (x) 6= 6= K (x) = K
m+1
(x) =

For this number m,

(9.4) Km+1 (x) = [x, Ax, . . . , Am x] Fnm+1

9.3. POLYNOMIAL REPRESENTATION OF KRYLOV SUBSPACES 165
has linearly dependant columns, i.e., there is a nonzero vector a Fm+1 such that

(9.5) Km+1 (x)a = p(A)x = 0, p() = a0 + a1 + + am m .

The polynomial p() is called the minimal polynomial of A relativ to x. By construction,

the highest order coefficient am 6= 0.
If A is diagonalizable, then the degree of the minimal polynomial relativ to x has a
simple geometric meaning (which does not mean that it is easily checked). Let

m 1
X ..
x= ui = [u1 , . . . , um ] . ,
i=1 1

where the ui are eigenvectors of A, Aui = i ui , and i 6= j for i 6= j. Notice that we

have arranged the eigenvectors such that the coefficients in the above sum are all unity.
Now we have k
m 1
X ..
k k
A x= i ui = [u1 , . . . , um ] . ,
i=1 km
and, by consequence,

1 1 21 1j1
1 2 22 2j1

K j (x) = [u1 , . . . , um ] . .. .. .. .. .
| {z } .. . . . .
Cnm j1
1 m 2m m
| {z }
Cmj

Since matrices of the form

1 1 1s1
1 2 2s1
ss
.. .. . . .. F , i 6= j for i 6= j,
. . . .
1 s s1
m

so-called
QVandermonde matrices, are nonsingular if the i are different (their determinant
equals i6=j (i j )) the Krylov matrices K j (x) are nonsingular for j m. Thus for
diagonalizable matrices A we have

dim Kj (x, A) = min{j, m}

where m is the number of eigenvectors needed to represent x. The subspace Km (x) is the
smallest invariant space that contains x.

9.3 Polynomial representation of Krylov subspaces

In this section we assume A to be Hermitian. Let s Kj (x). Then
j1
X j1
X
i
(9.6) s= ci A x = (A)x, () = ci i .
i=0 i=0
166 CHAPTER 9. KRYLOV SUBSPACES
Let Pj be the space of polynomials of degree j. Then (9.6) becomes

(9.7) Kj (x) = {(A)x | Pj1 } .

Let m be the smallest index for which Km (x) = Km+1 (x). Then, for j m the mapping
X X
P j1 ci i ci Ai x Kj (x)

is bijective, while it is only surjective for j > m.

Let Q Fnj be a matrix with orthonormal columns that span Kj (x), and let A =
Q AQ. The spectral decomposition

A X = X , X X = I, = diag(i , . . . , j ),

of A provides the Ritz values of A in Kj (x). The columns yi of Y = QX are the Ritz
vectors.
By construction the Ritz vectors are mutually orthogonal. Furthermore,

(9.8) Ayi i yi Kj (x)

because
Q (AQxi Qxi i ) = Q AQxi xi i = A xi xi i = 0.
It is easy to represent a vector in Kj (x) that is orthogonal to yi .

Lemma 9.2 Let (i , yi ), 1 i j be Ritz values and Ritz vectors of A in Kj (x), j m.

Let Pj1 . Then

(9.9) (A)x yk (k ) = 0.

Proof. = Let first Pj with (x) = (x k )(x), Pj1 . Then

yk (A)x = yk (A k I)(A)x, here we use that A = A

(9.10) (9.8)
= (Ayk k yk ) (A)x = 0.

= Let Sk Kj (x) be defined by

Sk := (A k I) Kj1 (x) = { (A)x | Pj1 , (k ) = 0} .

Each polynomial in the definition of Sk has the form (k ) = (x k )(x), Pj2 .

Therefore, dim(Sk ) = dim(Pj2 ) = j 1. As the dimension of a subspace of Kj (x) that is
orthogonal to yk is j 1, it must coincide with Sk .
Next we define the polynomials
j
Y Y j
()
() := ( i ) Pj , k () := = ( i ) Pj1 .
( k )
i=1 i=1
i6=k

Then the Ritz vector yk can be represented in the form

k (A)x
(9.11) yk = ,
kk (A)xk
9.3. POLYNOMIAL REPRESENTATION OF KRYLOV SUBSPACES 167
as k () = 0 for all i , i 6= k. According to Lemma 9.2 k (A)x is perpendicular to all yi
with i 6= k. Further,

(9.12) j := k(A)xk = min {k(A)xk | Pj monic } .

(A polynomial in Pj is monic if its highest coefficients aj = 1.) By the first part of

Lemma 9.2 (A)x Kj+1 (x) is orthogonal to Kj (x). As each monic Pj can be
written in the form
() = () + (), Pj1 ,
we have
k(A)xk2 = k(A)xk2 + k(A)xk2 ,
as (A)x Kj (x). Because of property (9.12) is called the minimal polynomial of x
of degree j. (In (9.5) we constructed the minimal polynomial of degree m in which case
m = 0.)
Let u1 , , um be the eigenvectors of A corresponding to 1 < < m that span
m
K (x). We collect the first i of them in the matrix Ui := [u1 , . . . , ui ]. Let kxk = 1. Let
:= (x, ui ) and := (x, Ui Ui x) ( ). (Remember that Ui Ui x is the orthogonal
projection of x on R(Ui ).)
Let
Ui Ui x (I Ui Ui )x
g := and h := .
kUi Ui xk k(I Ui Ui )xk
Then we have
kUi Ui xk = cos , k(I Ui Ui )xk = sin .
(j)
The following Lemma will be used for the estimation of the difference i i of the
desired eigenvalue and its approximation from the Krylov subspace.

Lemma 9.3 ([1, p.241]) For each Pj1 and each i j m the Rayleigh quotient

((A)x) (A i I)((A)x)
((A)x; A i I) = = ((A)x; A) i
k(A)xk2

satisfies the inequality

 
sin k(A)hk 2
(9.13) ((A)x; A i I) (m i ) .
cos (i )

Proof. With the definitions of g and h from above we have

x = Ui Ui x + (I Ui Ui )x = cos g + sin h.

which is an orthogonal decomposition. As R(Ui ) is invariant under A,

s := (A)x = cos (A)g + sin (A)h

is an orthogonal decomposition of s. Thus,

cos2 g (A i I) 2 (A)g + sin2 h (A i I) 2 (A)h

(9.14) ((A)x; A i I) = .
k(A)xk2

Since 1 < 2 < < m , we have

168 CHAPTER 9. KRYLOV SUBSPACES
(i) v (A i I)v 0 for all v R(Ui ),

(ii) w (A i I)w (m i )kwk2 for all w R(Ui ) .

Setting v = (A)g and w = (A)h we obtain from (9.14)

k(A)hk2
(s; A i I) sin2 (m i ) .
k(A)xk2

With
m
X
ksk2 = k(A)xk2 = 2 (l )(x ul )2 2 (i ) cos2
l=1

we obtain the claim.

9.4 Error bounds of Saad

The error bounds to be presented have been published by Saad [2]. We follow the presen-
(j)
tation in Parlett [1]. The error bounds for i i are obtained by carefully selecting the
polynomial in Lemma 9.3. Of course we would like (A) to be as small as possible and
(i ) to be as large as possible. First, by the definition of h, we have
P
k(A)(I Ui Ui )xk2
2 k(A) m
l=i+1 (ul x)ul k
2
k(A)hk = = P
k(I Ui Ui )xk2 k m
l=i+1 (ul x)ul k
2
Pm
(ul x)2 2 (l )
= l=i+1
Pm x)2
max 2 (l ) max 2 ().
l=i+1 (ul
i<lm i+1 m

The last inequality is important! In this step the search of a maximum in a few selected
points (i+1 , . . . , m ) is replaced by a search of a maximum in a whole interval containing
these points. Notice that i is outside of this interval. Among all polynomials of a given
degree that take a given fixed value (i ) the Chebyshev polynomial have the smallest
(j)
maximum. As i is a Ritz value, we know from the monotonicity principle 2.32 that

(j)
0 i i .

(j)
Further, from the definition of i (as an eigenvalue of A in the subspace Kj (x)),

(j)
i i (s, A i I) provided that s yl , 1 l i 1.

According to Lemma 9.2 s = (A)x is orthogonal on y1 , . . . , yi1 , if has the form

(j) (j)
() = ( 1 ) ( i1 )(), Pji .

With this choice of we get

(j) (j) i1 (j)

k(A)hk k(A 1 I) (A i1 I)k k(A)hk Y m l ()
(j) (j)
(j)
max .
(i ) |(i 1 )| |(i i1 )| |(i )| i+1 m (i )
l=1 i l
9.4. ERROR BOUNDS OF SAAD 169
This expression should be as small as possible. Now we have
max Tji (; [i+1 , m ])
|()| i+1 m
min max =
Pj1 i+1 m |(i )| Tji (i ; [i+1 , m ])
1
=
Tji (i ; [i+1 , m ])
1 i+1 i
= , = .
Tji (1 + 2) m i+1
Tji (1+2) is the value of the Chebyshev polynomial corresponding to the normal interval
[1, 1]. The point 1 + 2 is obtained if the affine transformation
2 i+1 m
[i+1 , m ] [1, 1]
i i+1
is applied to i .
Thus we have proved the first part of the following
(j) (j)
Theorem 9.4 [2] Let 1 , . . . , j be the Ritz values of A in Kj (x) and let (l , ul ), l =
1, . . . , m, be the eigenpairs of A (in Km (x)). Then for all i j we have
2
Q m (j)
i1
l
sin (j)
i l i+1 i
(j)
(9.15) 0 i i (m i ) l=1 , = ,
cos Tji (1 + 2) m i+1

and
Q
i1
m l
i l
sin
(9.16) tan (ui , projektion of ui on Kj ) l=1 .
cos Tji (1 + 2)
Proof. For proving the second part of the Theorem we write

x = g cos (x, Ui1 Ui1 x) + ui cos (x, ui ) +h sin (x, Ui Ui x) .
| {z } | {z }

We choose such that (1 ) = = (i1 ) = 0. Then.

s = (A)x = (i )ui cos + (A)h sin

is an orthogonal decomposition of s. By consequence,

sin k(A)hk
tan (s, ui ) = .
cos |(i )|
The rest is similar as above.
Remark 9.1. Theorem 9.4 does not give bounds for the angle between (ui , yi ), an angle
that would be more interesting than the abstract angle between ui and its projection on
Kj (x). It is possible however to show that [1, p. 246]
v
u
u j2
sin (ui , yi ) t1 + sin (ui , projection of ui onto Kj )
(j) 2
i
170 CHAPTER 9. KRYLOV SUBSPACES
j is the number that appeared earlier in the discussion after Lemma 9.2, and
(j)
i = min |i (j)
s |
s6=i

(j) (j)
Theorem 9.4 can easily be rewritten to give error bounds for m j , m1 j1 ,
etc.
We see from this Theorem that the eigenvalues at the beginning and at the end of
the spectrum are approximated the quickest. For the first eigenvalue the bound (9.15)
simplifies a little,

(j) tan2 1 2 1
(9.17) 0 1 1 (m 1 ) , 1 = , 1 = (x, u1 ).
Tji (1 + 21 )2 m 2

Analogously, for the largest eigenvalue we have

(j) 1
(9.18) 0 m j (m 1 ) tan2 m ,
Tji (1 + 2m )2
with
m m1
m = , and cos m = x um .
m1 1
If the Lanczos algorithmus is applied with (A I)1 as with the shifted and inverted
vector iteration then we form Krylov spaces Kj (x, (A I)1 ). Here the largest eigen-
i = i .
values are 1 1 1 ,
1 2 j
Eq. (9.18) then becomes
1 1
1 1 1 1 tan2 1 1 2

0 (j) ( ) 2, 1 = 1 1 .
1
1
j Tj1 (1 + 2
1 ) 2 j
j

Now, we have
1 1

2 1 1j

1 2 2 1 > 1.
1
j

1 = 2(1 + 1 ) 1 = 2
1 + 2 1 1
1=2
2 j
1 1 2
1

j

| {z }
>1

Since |Tj1 ()| grows rapidly and monotonically outside [1, 1] we have
2

1 ) Tj1 (2
Tj1 (1 + 2 1),
1

and thus
2
1 1 1
(9.19) c1
1 (j)
T (2
2

1)
1 j1 1

With the simple inverse vector iteration we had

!2(j1)
1 1 1

(9.20) (j) c2

1 2

1
BIBLIOGRAPHY 171
In Table 9.2 the numbers 2
1

Tj1 (2 2 1)
1

are compared with

!2(j1)
1

2

2 /
for 1 = 2, 1.1, 1.01. If this ratio is large both methods quickly provide the desired
results. If however the ratio tends to 1 then a method that computes the eigenvalues by
means of Ritz values of Krylov spaces shows an acceptable convergence behaviour whereas
vector iteration hardly improves with j. Remember that j is the number of matrix-vector
multiplications have been executed, or, with the shift-and-invert spectral transformation,
how many systems of equations have been solved.

2 /
1 j=5 j = 10 j = 15 j = 20 j = 25

2.0 3.0036e 06 6.6395e 14 1.4676e 21 3.2442e 29 7.1712e 37

3.9063e 03 3.8147e 06 3.7253e 09 3.6380e 12 3.5527e 15

1.1 2.7152e 02 5.4557e 05 1.0814e 07 2.1434e 10 4.2482e 13

4.6651e 01 1.7986e 01 6.9343e 02 2.6735e 02 1.0307e 02

1.01 5.6004e 01 1.0415e 01 1.4819e 02 2.0252e 03 2.7523e 04

9.2348e 01 8.3602e 01 7.5684e 01 6.8515e 01 6.2026e 01

(1/Tj1 (2 2 /
1 1))2
Table 9.2: Ratio 2 /
for varying j and ratios 1 .
1 /
( 2 ) 2(j1)

Bibliography
[1] B. N. Parlett, The Symmetric Eigenvalue Problem, Prentice Hall, Englewood Cliffs,
NJ, 1980. (Republished by SIAM, Philadelphia, 1998.).

[2] Y. Saad, On the rates of convergence of the Lanczos and the block Lanczos methods,
SIAM J. Numer. Anal., 17 (1980), pp. 687706.
172 CHAPTER 9. KRYLOV SUBSPACES
Chapter 10

Arnoldi and Lanczos algorithms

10.1 An orthonormal basis for the Krylov space Kj (x)

The natural basis of the Krylov subspace Kj (x) = Kj (x, A) is evidently {x, Ax, . . . , Aj1 x}.
Remember that the vectors Ak x converge to the direction of the eigenvector correspond-
ing to the largest eigenvalue (in modulus) of A. Thus, this basis tends to be badly
conditioned with increasing dimension j. Therefore, the straightforward procedure, the
GramSchmidt orthogonalization process, is applied to the basis vectors in their
natural order.
Suppose that {q1 , . . . , qi } is the orthonormal basis for Ki (x), where i j. We con-
struct the vector qj+1 by first orthogonalizing Aj x against q1 , . . . , qj ,
j
X
j
(10.1) yj := A x qi qi Aj x,
i=1

and then normalizing the resulting vector,

(10.2) qj+1 = yj /kyj k.

Then {q1 , . . . , qj+1 } is an orthonormal basis of Kj+1 (x), called in general the Arnoldi
basis or, if the matrix A is real symmetric or Hermitian, the Lanczos basis. The vectors
qi are called Arnoldi vectors or Lanczos vectors, respectively, see [6, 1].
The vector qj+1 can be computed in a more economical way since


Kj+1 (x, A) = R [x, Ax, . . . , Aj x] , (q1 = x/kxk) ,
j


= R [q1 , Aq1 , . . . , A q1 ] (Aq1 = q1 + q2 , 6= 0),


= R [q1 , q1 + q2 , A(q1 + q2 ), . . . , Aj1 (q1 + q2 )] ,


= R [q1 , q2 , Aq2 , . . . , Aj1 q2 ] ,
..
.
= R ([q1 , q2 , . . . , qj1 , Aqj ]) .

So, instead of orthogonalizing Aj q1 against q1 , . . . , qj , we can orthogonalize Aqj

against q1 , . . . , qj to obtain qj+1 . The component rj of Aqj orthogonal to q1 , . . . , qj
is given by
j
X
(10.3) rj = Aqj qi (qi Aqj ).
i=1

173
174 CHAPTER 10. ARNOLDI AND LANCZOS ALGORITHMS
If rj = 0 then the procedure stops which means that we have found an invariant subspace,
namely span{q1 , . . . , qj }. If krj k > 0 we obtain qj+1 by normalizing,
rj
(10.4) qj+1 = .
krj k
Since, qj+1 and rj are aligned, we have
(10.3)
(10.5) qj+1 rj = krj k = qj+1 Aqj .

The last equation holds since qj+1 (by construction) is orthogonal to all the previous
Arnoldi vectors. Let
hij = qi Aqj .
Then, (10.3)(10.5) can be written as
j+1
X
(10.6) Aqj = qi hij .
i=1

We collect the procedure in Algorithm 10.1

Algorithm 10.1 The Arnoldi algorithm for the computation of an orthonormal

basis of a Krylov space
1: Let A Fnn . This algorithm computes an orthonormal basis for Kk (x).
2: q1 = x/kxk2 ;
3: for j = 1, . . . do
4: r := Aqj ;
5: for i = 1, . . . , j do /* Gram-Schmidt orthogonalization */
6: hij := qi r, r := r qi hij ;
7: end for
8: hj+1,j := krk;
9: if hj+1,j = 0 then /* Found an invariant subspace */
10: return (q1 , . . . , qj , H Fjj )
11: end if
12: qj+1 = r/hj+1,j ;
13: end for
14: return (q1 , . . . , qk+1 , H Fk+1k )

The Arnoldi algorithm returns if hj+1,j = 0, in which case j is the degree of the
minimal polynomial of A relative to x, cf. (9.5). This algorithm costs k matrix-vector
multiplications, n2 /2 + O(n) inner products, and the same number of axpys.
Defining Qk = [q1 , . . . , qk ], equation (10.6) can be collected for j = 1, . . . , k,

(10.7) AQk = Qk Hk + [ 0, . . . , 0 , qk+1 hk+1,k ]

| {z }
k1 times

Equation (10.7) is called Arnoldi relation. The construction of the Arnoldi vectors is
expensive. Most of all, each iteration step becomes more costly as the number of vectors
against which r has to be orthogonalized increases. Therefore, algorithms based on the
Arnoldi relation like GMRES or the Arnoldi algorithm itself are restarted. This in general
means that the algorithm is repeated with a initial vector that is extracted from previous
invocation of the algorithm.
10.2. ARNOLDI ALGORITHM WITH EXPLICIT RESTARTS 175
10.2 Arnoldi algorithm with explicit restarts
Algorithm 10.1 stops if hm+1,m = 0, i.e., if it has found an invariant subspace. The vectors
{q1 , . . . , qm } then form an invariant subspace of A,
AQm = Qm Hm , Qm = [q1 , . . . , qm ].
The eigenvalues of Hm are eigenvalues of A as well and the Ritz vectors are eigenvectors
of A.
In general, we cannot afford to store the vectors q1 , . . . , qm because of limited memory
space. Furthermore, the algorithmic complexity increases linearly in the iteration number
j. The orthogonalization would cost 2nm2 floating point operations.
Often it is possible to extract good approximate eigenvectors from a Krylov space of
small dimension. We have seen, that in particular the extremal eigenvalues and corre-
sponding eigenvectors are very well approximated after a few iteration steps. So, if only a
small number of eigenpairs is desired, it is usually sufficient to get away with Krylov space
of much smaller dimension than m.
Exploiting the Arnoldi relation (10.7) we can get cheap estimates for the eigenvalue/eigen-
(k) (k) (k)
vector residuals. Let ui = Qk si be a Ritz vector with Ritz value i . Then
(k) (k) (k) (k) (k) (k) (k) (k)
Aui i ui = AQk si i Qk si = (AQk Qk Hk )si = hk+1,k qk+1 ek si .
Therefore,
(k) (k) (k)
(10.8) k(A i I)ui k2 = hk+1,k |ek si |.
(k)
The residual norm is equal to the last component of si multiplied by hk+1,k (which is
positive by construction). These residual norms are not always indicative of actual errors
(k)
in i , but they can be helpful in deriving stopping procedures.
We now consider an algorithm for computing some of the extremal eigenvalues of a
non-Hermitian matrix. The algorithm proceeds by computing one eigenvector or rather
Schur vector at the time. For each of them an individual Arnoldi procedure is employed.
Let us assume that we have already computed k1 Schur vectors u1 , . . . uk1 . To compute
uk we force the iterates in the Arnoldi process (the Arnoldi vectors) to be orthogonal to
Uk1 where Uk1 = [u1 , . . . uk1 ]. So, we work essentially with the matrix

(I Uk1 Uk1 )A
that has k 1 eigenvalues zero which we of course neglect.
The procedure is given in Algorithm 10.2. The Schur vectors u1 , . . . uk1 are kept
in the search space, while the Krylov space is formed with the next approximate Schur
vector. The search space thus is
span{u1 , . . . uk1 , uk , Auk , . . . Amk uk }.
In Algorithm 10.2 the basis vectors are denoted vj with vj = uj for j < k. The vectors
vk , . . . , vm form an orthonormal basis of span{uk , Auk , . . . Amk uk }.
The matrix Hm for k = 2 has the structure





Hm =



176 CHAPTER 10. ARNOLDI AND LANCZOS ALGORITHMS
Algorithm 10.2 Explicitly restarted Arnoldi algorithm
1: Let A Fnn . This algorithm computes the nev largest eigenvalues of A together with
the corresponding Schur vectors.
2: Set k = 1.
3: loop
4: for j = k, . . . , m do /* Execute m k steps of Arnoldi */
5: r := Aqj ;
6: for i = 1, . . . , j do
7: hij := qi r, r := r qi hij ;
8: end for
9: hj+1,j := krk;
10: qj+1 = r/hj+1,j ;
11: end for
12: Compute approximate eigenvector of A associated with k and the corresponding
residual norm estimate k according to (10.8).
13: Orthogonalize this eigenvector (Ritz vector) against all previous vj to get the ap-
proximate Schur vector uk . Set vk := uk .
14: if k is small enough then /* accept eigenvalue */
15: for i = 1, . . . , k do
16: hik := vi Avk ;
17: end for
18: Set k := k + 1.
19: if k nev then
20: return (v1 , . . . , vk , H Fkk )
21: end if
22: end if
23: end loop

where the block in the lower right corresponds to the Arnoldi process for the Krylov space
)A).
Kmk (uk , (I Uk1 Uk1
This algorithm needs at most m basis vectors. As soon as the dimension of the search
space reaches m the Arnoldi iteration is restarted with the best approximation as the
initial vector. The Schur vectors that have already converged are locked or deflated.

10.3 The Lanczos basis

We have seen that the Lanczos basis is formally constructed in the same way as the Arnoldi
basis, however with a Hermitian matrix. It deserves a special name for the simplifications
that the symmetry entails.
By multiplying (10.7) with Qk from the left we get

(10.9) Qk AQk = Qk Qk Hk = Hk .

If A is Hermitian, then so is Hk . This means that Hk is tridiagonal. To emphasize this

matrix structure, we call this tridiagonal matrix Tk . Due to symmetry, equation (10.3)
simplifies considerably,

(10.10) rj = Aqj qi (qj Aqj ) qj1 (qj1 Aqj ) = Aqj j qj j1 qj1 .

| {z } | {z }
j R j1 F
10.3. THE LANCZOS BASIS 177
Similarly as earlier, we premultiply (10.10) by qj+1 to get

krj k = qj+1 rj = qj+1 (Aqj j qj j1 qj1 )

= q Aqj = j .
j+1

From this it follows that j R. Therefore,

(10.11) j qj+1 = rj , j = krj k.

Collecting (10.10)(10.11) yields

(10.12) Aqj = j1 qj1 + j qj + j qj+1 .

Gathering these equations for j = 1, . . . , k we get

1 1
1 2 2

..
(10.13) AQk = Qk 2 3 . +k [0, . . . , 0, qk+1 ].

. .. . ..
k1
k1 k
| {z }
Tk

Tk Rkk is real symmetric. Equation (10.13) is called Lanczos relation. Pictorially,

this is

Tk

Ak Qk = Qk + O

The Lanczos algorithm is summarized in Algorithm 10.3. In this algorithm just the
three vectors q, r, and v are employed. In the j-th iteration step (line 8) q is assigned
qj and v stores qj1 . r stores first (line 9) Aqj j1 qj1 . Later (step 11), when j
is available, it stores rj = Aqj j1 qj1 j qj . In the computation of j the fact is
exploited that qj qj1 = 0 whence

j = qj Aqj = qj (Aqj j1 qj1 ).

In each traversal of the j-loop a column is appended to the matrix Qj1 to become Qj . If
the Lanczos vectors are not desired this statement can be omitted. The Lanczos vectors
are required to compute the eigenvectors of A. Algorithm 10.3 returns when j = m, where
m is the degree of the minimal polynomial of A relative to x. bm = 0 implies

(10.14) AQm = Qm Tm .
178 CHAPTER 10. ARNOLDI AND LANCZOS ALGORITHMS
Algorithm 10.3 Basic Lanczos algorithm for the computation of an orthonormal
basis for of the Krylov space Km (x)
1: Let A Fnn be Hermitian. This algorithm computes the Lanczos relation (10.13),
i.e., an orthonormal basis Qm = [q1 , . . . , qm ] for Km (x) where m is the smallest index
such that Km (x) = Km+1 (x), and (the nontrivial elements of) the tridiagonal matrix
Tm .
2: q := x/kxk; Q1 = [q];
3: r := Aq;
4: 1 := q r;
5: r := r 1 q;
6: 1 := krk;
7: for j = 2, 3, . . . do
8: v = q; q := r/j1 ; Qj := [Qj1 , q];
9: r := Aq j1 v;
10: j := q r;
11: r := r j q;
12: j := krk;
13: if j = 0 then
14: return (Q Fnj ; 1 , . . . , j ; 1 , . . . , j1 )
15: end if
16: end for

Let (i , si ) be an eigenpair of Tm ,
(m) (m) (m)
(10.15) Tm si = i si .

Then,
(m) (m) (m) (m)
(10.16) AQm si = Qm Tm si = i Qm si .

So, the eigenvalues of Tm are also eigenvalues of A. The eigenvector of A corresponding

to the eigenvalue i is
m
X
(m) (m) (m)
(10.17) yi = Qm si = [q1 , . . . , qm ] si = qj sji .
j=1

The cost of a single iteration step of Algorithm 10.3 does not depend on the index of
the iteration! In a single iteration step we have to execute a matrix-vector multiplication
and 7n further floating point operations.
Remark 10.1. In certain very big applications the Lanczos vectors cannot be stored for
reasons of limited memory. In this situation, the Lanczos algorithm is executed without
building the matrix Q. When the desired eigenvalues and Ritz vectors have been deter-
mined from (10.15) the Lanczos algorithm is repeated and the desired eigenvectors are
accumulated on the fly using (10.17).

10.4 The Lanczos process as an iterative method

The Lanczos Algorithm 10.3 essentially determines an invariant Krylov subspace Km (x)
of Fn . More precisely, it constructs an orthonormal basis {q1 , . . . , qm } of Km (x). The
10.4. THE LANCZOS PROCESS AS AN ITERATIVE METHOD 179
projection of A onto this space is a Hessenberg or even a real tridiagonal matrix if A is
Hermitian.
We have seen in section 9.4 that the eigenvalues at the end of the spectrum are ap-
proximated very quickly in Krylov spaces. Therefore, only a very few iteration steps may
be required to get those eigenvalues (and corresponding eigenvectors) within the desired
(j)
accuracy, i.e., |i i | may be tiny for j m.
(j)
The Ritz values i are the eigenvalues of the tridiagonal matrices Tj that are gener-
ated element by element in the course of the Lanczos algorithm. They can be computed
efficiently by, e.g., the tridiagonal QR algorithm in O(j 2 ) flops. The cost for computing
the eigenvalues of Tj are in general negligible compared with the cost for forming Aqj .
(j)
But how can the error |i i | be estimated? We will adapt the following more
general lemma to this end.

Lemma 10.1 (Eigenvalue inclusion of KrylovBogoliubov [5] [7, p.69]) Let A

Fnn be Hermitian. Let R and x Fn with x 6= 0 be arbitrary. Set := k(A
I)xk/kxk. Then there is an eigenvalue of A in the interval [ , + ].

Proof. Let
n
X

A = U U = i ui ui
i=1
be the spectral decomposition of A. Then,
n
X n
X
(A I)x = (i ui ui ui ui )x = (i )(ui x)ui .
i=1 i=1

Taking norms, we obtain

n
X n
X
k(A I)xk2 = |i |2 |ui x|2 |k |2 |ui x|2 = |k |2 kxk ,
i=1 i=1

where k is the eigenvalue closest to , i.e., |k | |i | for all i.

(j)
We want to apply this Lemma to the case where the vector is a Ritz vector yi
(j)
corresponding to the Ritz value = i as obtained in the j-th step of the Lanczos
algorithm. Then,
(j) (j) (j) (j) (j)
yi = Qj si , Tj si = i si .
Thus, by employing the Lanczos relation (10.13),
(j) (j) (j) (j) (j) (j)
kAyi i yi k = kAQj si i Qj si k
(j)
= k(AQj Qj Tj )si k
(j) (j) (j)
= kj qj+1 ej si k = |j ||ej si | = |j ||sji |.

(j)
sji is the j-th, i.e., the last element of the eigenvector matrix Sj of Tj ,

(j) (j)
Tj Sj = Sj j , j = diag(1 , , j ).

According to Lemma 10.1 there is an eigenvalue of A such that

(j)
(10.18) | i | j |sji |.
180 CHAPTER 10. ARNOLDI AND LANCZOS ALGORITHMS
Thus, it is possible to get good eigenvalue approximations even if j is not small! Further,
we know that [7, 11.7]

(j) |sji |
(10.19) sin (yi , z) j ,

where z is the eigenvector corresponding to in (10.18) and is the gap between and
the next eigenvalue 6= of A. In an actual computation, is not known. Parlett suggests
(j) (j) (j)
to replace by the distance of i to the next k , k 6= i. Because the i converge to
eigenvalues of A this substitution will give a reasonable number, at least in the limit.
In order to use the estimate (10.18) we need to compute all eigenvalues of Tj and the
last row of Sj . It is possible and in fact straightforward to compute this row without the
rest of Sj . The algorithm, a simple modification of the tridiagonal QR algorithm, has been
introduced by Golub and Welsch [3] in connection with the computation of interpolation
points and weights in Gaussian integration.

A numerical example
This numerical example is intended to show that the implementation of the Lanczos algo-
rithm is not as simple as it seems from the previous. Let
A = diag(0, 1, 2, 3, 4, 100000)

and
x = (1, 1, 1, 1, 1, 1)T .
The diagonal matrix A has six simple eigenvalues and x has a non-vanishing component in
the direction of each eigenspace. Thus, the Lanczos algorithm should stop after m = n = 6
iteration steps with the complete Lanczos relation. Up to rounding error, we expect that
6 = 0 and that the eigenvalues of T6 are identical with those of A. Lets see what happens
if Algorithm 10.3 is applied with these input data. in the sequel we present the numbers
that we obtained with a Matlab implementation of this algorithm.

j=1
1 = 16668.33333333334, 1 = 37267.05429136513.

j=2
2 = 83333.66652666384, 2 = 3.464101610531258.
The diagonal of the eigenvalue matrix 2 is:

diag(2 ) = (1.999959999195565, 99999.99989999799) T .

The last row of 2 S2 is

2 S2,: = (1.4142135626139063.162277655014521) .

The matrix of Ritz vectors Y2 = Q2 S2 is

0.44722 2.0000 1005
0.44722 9.9998 1006

0.44721 4.0002 1010

0.44721 1.0001 1005

0.44720 2.0001 1005
4.4723 1010 1.0000
10.4. THE LANCZOS PROCESS AS AN ITERATIVE METHOD 181
j=3
3 = 2.000112002245340 3 = 1.183215957295906.
The diagonal of the eigenvalue matrix is
diag(3 ) = (0.5857724375775532, 3.414199561869119, 99999.99999999999) T .
The largest eigenvalue has converged already. This is not surprising as 2 /1 =
4 105 . With simple vector iteration the eigenvalues would converge with the factor
2 /1 = 4 105 .
The last row of 3 S3 is


3 S3,: = 0.8366523355001995, 0.8366677176165411, 3.741732220526109 1005 .
The matrix of Ritz vectors Y3 = Q3 S3 is

0.76345 0.13099 2.0000 1010
0.53983 0.09263 1.0001 1010

0.31622 0.31623 2.0001 1010

0.09262 0.53984 1.0000 1010

0.13098 0.76344 2.0001 1010
1.5864 10 13 1.5851 1013 1.00000
The largest element (in modulus) of Y3T Y3 is 3 1012 .
The Ritz vectors (and thus the Lanczos vectors qi ) are mutually orthogonal up to
rounding error.
j=4
4 = 2.000007428756856 4 = 1.014186947306611.
The diagonal of the eigenvalue matrix is

0.1560868732577987
1.999987898940119
diag(4 ) =
3.843904656006355
.

99999.99999999999
The last row of 4 S4 is


4 S4,: = 0.46017, 0.77785, 0.46018, 3.7949 1010 .
The matrix of Ritz vectors Y4 = Q4 S4 is

0.82515 0.069476 0.40834 0.18249
0.034415 0.41262 0.40834 0.18243

0.37812 0.37781 0.40834 0.18236
.
0.41256 0.034834 0.40834 0.18230

0.069022 0.82520 0.40834 0.18223
1.3202 1004 1.3211 1004 0.40777 0.91308

The largest element (in modulus) of Y4T Y4 is 2 108 .

.
We have 4 s4,4 = 4 1010 . So, according to our previous estimates (4 , y4 ), y4 =
Y4 e4 is a very good approximation for an eigenpair of A. This is in fact the case.
Notice that Y4T Y4 has off diagonal elements of the order 108 . These elements are
in the last row/column of Y4T Y4 . This means that all Ritz vectors have a small but
not negligible component in the direction of the largest Ritz vector.
182 CHAPTER 10. ARNOLDI AND LANCZOS ALGORITHMS
j=5
5 = 2.363169101109444 5 = 190.5668098726485.
The diagonal of the eigenvalue matrix is

0.04749223464478182
1.413262891598485

diag(5 ) =
2.894172742223630 .

4.008220660846780
9.999999999999999 104

The last row of 5 S5 is

5 S5,: = 43.570 111.38134.0963.4957.2320 1013 .

The matrix of Ritz vectors Y5 is

0.98779 0.084856 0.049886 0.017056 1.1424 1017
0.14188 0.83594 0.21957 0.065468 7.2361 1018

0.063480 0.54001 0.42660 0.089943 8.0207 1018

0.010200 0.048519 0.87582 0.043531 5.1980 1018

0.0014168 0.0055339 0.015585 0.99269 1.6128 1017
4.3570 104 0.0011138 0.0013409 6.3497 104 1.0000

Evidently, the last column of Y5 is an excellent eigenvector approximation. Notice,

however, that all Ritz vectors have a relatively large ( 104 ) last component. This,
gives rise to quite large off-diagonal elements of Y5T Y5 I5 =

2.2201016 1.5871016 3.4301012 7.890109 7.780104
1.5871016 1.1101016 1.2831012 1.764108 1.740103

3.4301012 1.2831012 0 5.68001017 6.027108 .

7.890109 1.764108 5.68001017 2.2201016 4.1871016
7.780104 1.740103 6.027108 4.1871016 1.1101016

Similarly as with j = 4, the first four Ritz vectors satisfy the orthogonality condition
very well. But they are not perpendicular to the last Ritz vector.

j=6
6 = 99998.06336906151 6 = 396.6622037049789
The diagonal of the eigenvalue matrix is

0.02483483859326367
1.273835519171372

2.726145019098232
diag(6 ) =

.

3.975161765440400
9.999842654044850 10+4
1.000000000000000 10+5

The eigenvalues are not the exact ones, as was to be expected. We even have two
copies of the largest eigenvalue of A in 6 ! The last row of 6 S6 is


6 S6,: = 0.20603, 0.49322, 0.49323, 0.20604, 396.66, 8.6152 1015
10.4. THE LANCZOS PROCESS AS AN ITERATIVE METHOD 183
although theory predicts that 6 = 0. The sixth entry of 6 S6 is very small, which
means that the sixth Ritz value and the corresponding Ritz vector are good approx-
imations to an eigenpair of A. In fact, eigenvalue and eigenvector are accurate to
machine precision.
5 s6,5 does not predict the fifth column of Y6 to be a good eigenvector approximation,
although the angle between the fifth and sixth column of Y6 is less than 103 . The
last two columns of Y6 are

4.7409 104 3.3578 1017
1.8964 103 5.3735 1017

2.8447 103 7.0931 1017

1.8965 103 6.7074 1017 .

4.7414 104 4.9289 1017
0.99999 1.0000

As 6 6= 0 one could continue the Lanczos process and compute ever larger tridi-
agonal matrices. If one proceeds in this way one obtains multiple copies of certain
(j)
eigenvalues [2, 2]. The corresponding values j sji will be tiny. The corresponding
Ritz vectors will be almost linearly dependent.
From this numerical example we see that the problem of the Lanczos algorithm consists
in the loss of orthogonality among Ritz vectors which is a consequence of the loss of
orthogonality among Lanczos vectors, since Yj = Qj Sj and Sj is unitary (up to roundoff).
To verify this diagnosis, we rerun the Lanczos algorithm with complete reorthogonal-
ization. This procedure amounts to the Arnoldi algorithm 10.1. It can be accomplished
by modifying line 11 in the Lanczos algorithm 10.3, see Algorithm 10.4.

Algorithm 10.4 Lanczos algorithm with full reorthogonalization

11: r := r j q; r := r Q(Q r);

Of course, the cost of the algorithm increases considerably. The j-th step of the
algorithm requires now a matrix-vector multiplication and (2j + O(1))n floating point
operations.

A numerical example [continued]

With matrix and initial vector as before Algorithm 10.4 gives the following numbers.
j=1
1 = 16668.33333333334, 1 = 37267.05429136513.
j=2
2 = 83333.66652666384, 2 = 3.464101610531258.
The diagonal of the eigenvalue matrix 2 is:
diag(2 ) = (1.999959999195565, 99999.99989999799) T .

j=3
3 = 2.000112002240894 3 = 1.183215957295905
The diagonal of the eigenvalue matrix is
diag(3 ) = (0.5857724375677908, 3.414199561859357, 100000.0000000000) T .
184 CHAPTER 10. ARNOLDI AND LANCZOS ALGORITHMS
j=4
4 = 2.000007428719501 4 = 1.014185105707661

0.1560868732475296
1.999987898917647
diag(4 ) =
3.843904655996084

99999.99999999999
The matrix of Ritz vectors Y4 = Q4 S4 is

0.93229 0.12299 0.03786 1.1767 1015
0.34487 0.49196 0.10234 2.4391 1015

2.7058 106 0.69693 2.7059 106 4.9558 1017

0.10233 0.49195 0.34488 2.3616 1015

0.03786 0.12299 0.93228 1.2391 1015
2.7086 1017 6.6451 1017 5.1206 1017 1.00000

The largest off-diagonal element of |Y4T Y4 | is about 2 1016

j=5
5 = 2.000009143040107 5 = 0.7559289460488005

0.02483568754088384
1.273840384543175

diag(5 ) =
2.726149884630423

3.975162614480485
10000.000000000000
The Ritz vectors are Y5 =

9.91 1001 4.62 1002 2.16 1002 6.19 1003 4.41 1018
1.01 10 01 8.61 1001 1.36 1001 3.31 1002 1.12 1017

7.48 1002 4.87 1001 4.87 1001 7.48 1002 5.89 1018

3.31 1002 1.36 1001 8.61 1001 1.01 1001 1.07 1017

6.19 10 03 2.16 1002 4.62 1002 9.91 1001 1.13 1017
5.98 1018 1.58 1017 3.39 1017 5.96 1017 1.000000000000000

Largest off-diagonal element of |Y5T Y5 | is about 1016 The last row of 5 S5 is

5 S5,: = 0.20603, 0.49322, 0.49322, 0.20603, 2.8687 1015 .

j=6
6 = 2.000011428799386 6 = 4.1785508667493421028

7.9503070793407461013
1.000000000000402

2.000000000000210

diag(6 ) =

3.000000000000886
4.000000000001099
9.999999999999999104
The Ritz vectors are very accurate. Y6 is almost the identity matrix are 1.0. The
largest off diagonal element of Y6T Y6 is about 1016 . Finally,


6 S6,: = 4.991029 , 2.001028 , 3.001028 , 2.001028 , 5.001029 , 1.201047 .
10.5. AN ERROR ANALYSIS OF THE UNMODIFIED LANCZOS ALGORITHM 185
With a much enlarged effort we have obtained the desired result. Thus, the loss
of orthogonality among the Lanczos vectors can be prevented by the explicit reorthogo-
nalization against all previous Lanczos vectors. This amounts to applying the Arnoldi
algorithm. In the sequel we want to better understand when the loss of orthogonality
actually happens.

10.5 An error analysis of the unmodified Lanczos algorithm

When the quantities Qj , Tj , rj , etc., are computed numerically by using the Lanczos algo-
rithm, they can deviate greatly from their theoretical counterparts. However, despite this
gross deviation from the exact model, it nevertheless delivers fully accurate Ritz value and
Ritz vector approximations.
In this section Qj , Tj , rj etc. denote the numerically computed values and not their
theoretical counterparts. So, instead of the Lanczos relation (10.13) we write

(10.20) AQj Qj Tj = rj ej + Fj
where the matrix Fj accounts for errors due to roundoff. Similarly, we write

(10.21) Ij Qj Qj = Cj + j + Cj ,

where j is a diagonal matrix and Cj is a strictly upper triangular matrix (with zero
diagonal). Thus, Cj + j + Cj indicates the deviation of the Lanczos vectors from orthog-
onality.
We make the following assumptions
1. The tridiagonal eigenvalue problem can be solved exactly, i.e.,

(10.22) Tj = Sj j Sj , Sj = Sj1 , j = diag(1 , . . . , j ).

2. The orthogonality of the Lanczos vectors holds locally, i.e.,

(10.23) qi+1 qi = 0, i = 1, . . . , j 1, and rj qi = 0.

3. Furthermore,
(10.24) kqi k = 1.

So, we assume that the computations that we actually perform (like orthogonalizations or
solving the eigenvalue problem) are accurate. These assumptions imply that j = O and
(j)
ci,i+1 = 0 for i = 1, . . . , j 1.
We premultiply (10.20) by Qj and obtain

(10.25) Qj AQj Qj Qj Tj = Qj rj ej + Qj Fj
In order to eliminate A we subtract from this equation its transposed,
Qj rj ej ej rj Qj = Qj Qj Tj + Tj Qj Qj + Qj Fj Fj Qj ,
= (I Qj Qj )Tj Tj (I Qj Qj ) + Qj Fj Fj Qj ,
(10.21)
(10.26) = (Cj + Cj )Tj Tj (Cj + Cj ) + Qj Fj Fj Qj ,
= (Cj Tj Tj Cj ) + (Cj Tj Tj Cj ) Fj Qj + Qj Fj .
| {z } | {z }
upper triangular lower triangular
186 CHAPTER 10. ARNOLDI AND LANCZOS ALGORITHMS
Fj Qj Qj Fj is skew symmetric. Therefore we have
Fj Qj Qj Fj = Kj + Kj ,
where Kj is an upper triangular matrix with zero diagonal. Thus, (10.25) has the form


. 0 Cj Tj Tj Cj 0 Kj


O ..
= ..
.

+
..
.

.

Cj Tj Tj Cj 0 Kj 0

| {z
} 0
First j 1
components
of rj Qj .
As the last component of Qj rj vanishes, we can treat these triangular matrices sepa-
rately. For the upper triangular matrices we have
Qj rj ej = Cj Tj Tj Cj + Kj .
Multiplication by si and si , respectively, from left and right gives
si Qj rj ej si = si (Cj Tj Tj Cj )si + si Kj si .
| {z } |{z} |{z}
yi j qj+1 sji
Let Gj := Si Kj Si . Then we have
(j) (j)
(10.27) j sji yi qj+1 = sji yi rj = si Cj si i i si Cj si + gii = gii .
We now multiply (10.25) with si from the left and with sk from the right. As Qj si = yi ,
we have
yi Ayk yi yk k = yi rj ej sk + si Qj Fj sk .
Now, from this equation we subtract again its transposed, such that A is eliminated,
yi yk (i k ) = yi rj ej sk yk rj ej si + si Qj Fj sk sk Qj Fj si
!
(j) (j)
(10.27) gii g
= (j)
sjk kk (j)
sji
sji sjk
1 1
+ (si Qj Fj sk + sk Fj Qj si ) (sk Qj Fj si + si Fj Qj sk )
2 2
(j) (j)
(j) sjk (j) sji (j) (j)
= gii (j) gkk (j) (gik gki ).
sji sjk
Thus we have proved
Theorem 10.2 (Paige, see [7, p.266]) With the above notations we have
(j)
(j) gii
(10.28) yi qj+1 = (j)
j sji

(j) (j)
(j) (j) (j) (j) (j) sjk (j) sji (j) (j)
(10.29) (i k )yi yk = gii (j) gkk (j) (gik gki ).
sji sjk
10.6. PARTIAL REORTHOGONALIZATION 187

We can interpret these equations in the following way.

From numerical experiments it is known that equation (10.20) is always satisfied to

machine precision. Thus, kFj k kAk. Therefore, kGj k kAk, and, in particular,
(j)
|gik | kAk.
(j) (j)
We see from (10.28) that |yi qj+1 | becomes large if j |sji | becomes small, i.e., if
(j)
the Ritz vector yi is a good approximation of the corresponding eigenvector. Thus,
each new Lanczos vector has a significant component in the direction of converged
(good) Ritz vectors.
As a consequence: convergence loss of orthogonality .
(j) (j) (j) (j)
Let |sji | |sjk |, i.e., yi is a good Ritz vector in contrast to yk that is a bad
Ritz vector. Then in the first term on the right of (10.29) two small (O()) quantities
counteract each other such that the right hand side in (10.29) becomes large, O(1). If
the corresponding Ritz values are well separated, |i k | = O(1), then |yi yk | .
So, in this case also bad Ritz vectors have a significant component in the direction
of the good Ritz vectors.
(j) (j) (j) (j)
If |i k | = O() and both sji and sjk are of O() the sji /sjk = O(1) such that
the right hand side of (10.29) as well as |i k | is O(). Therefore, we must have
(j) (j)
yi yk = O(1). So, these two vectors are almost parallel.

10.6 Partial reorthogonalization

In Section 10.4 we have learned that the Lanczos algorithm does not yield orthogonal
Lanczos vectors as it should in theory due to floating point arithmetic. In the previous
section we learned that the loss of orthogonality happens as soon as Ritz vectors have
converged accurately enough to eigenvectors. In this section we review an approach how
to counteract the loss of orthogonality without executing full reorthogonalization [8, 9].
In [7] it is shown that if the Lanczos basis is semiorthogonal, i.e., if

Wj = Qj Qj = Ij + E, kEk < M ,

then the tridiagonal matrix Tj is the projection of A onto the subspace R(Vj ),

Tj = Nj ANj + G, kGk = O((M )kAk),

where Nj is an orthonormal basis of R(Qj ). Therefore, it suffices to have semiorthog-

onal Lanczos vectors for computing accurate eigenvalues. Our goal is now to enforce
semiorthogonality by monitoring the loss of orthogonality and to reorthogonalize if needed.
The computed Lanczos vectors satisfy

(10.30) j qj+1 = Aqj j qj j1 qj1 + fj ,

where fj accounts for the roundoff errors committed in the j-th iteration step. Let Wj =
((ik ))1i, kj . Premultiplying equation (10.30) by qk gives

(10.31) j j+1,k = qk Aqj j jk j1 j1,k + qk fj .

188 CHAPTER 10. ARNOLDI AND LANCZOS ALGORITHMS
Exchanging indices j and k in the last equation (10.31) gives

(10.32) k j,k+1 = qj Aqk k jk k1 j,k1 + qj fk .

By subtracting (10.32) from (10.31) we get

(10.33) j j+1,k = k j,k+1 + (k j )jk k1 j,k1 j1 j1,k qj fk + qk fj .

Given Wj we employ equation (10.33) to compute the j + 1-th row of Wj+1 . However,
elements j+1,j and j+1,j+1 are not defined by (10.33). We can assign values to these
two matrix entries by reasoning as follows.
We set j+1,j+1 = 1 because we explicitly normalize qj+1 .

We set j+1,j = O(M ) because we explicitly orthogonalize qj+1 and qj .

For computational purposes, equation (10.33) is now replaced by

= k j,k+1 + (k j )jk k1 j,k1 j1 j1,k ,

(10.34) j+1,k = (
+ sign(
) 2kAk )/j .
| {z }
the estimate of
qj fk + qk fj

As soon as j+1,k > M the vectors qj and qj+1 are orthogonalized against all previous
Lanczos vectors q1 , . . . , qj1 . Then the elements of last two lines of Wj are set equal to a
number of size O(M ). Notice that only the last two rows of Wj have to be stored.

Numerical example
We perform the Lanczos algorithm with matrix

A = diag(1, 2, . . . , 50)

and initial vector

x = [1, . . . , 1] .
In the first experiment we execute 50 iteration steps. In Table 10.1 the base-10 logarithms
of the values |wi,j |/macheps are listed where |wi,j | = |qi qj |, 1 j i 50 and macheps
2.2 1016 . One sees how the |wi,j | steadily grow with increasing i and with increasing
|i j|.
In the second experiment we execute 50 iteration steps with partial reorthogonalization
turned on. The estimators j,k are computed according to (10.33),

k,k = 1, k = 1, . . . , j
k,k1 = k , k = 2, . . . , j
(10.35) 1
j+1,k = [k j,k+1 + (k j )jk
j
k1 j,k1 j1 j1,k ] + i,k , 1 k j.

Here, we set j,0 = 0. The values k and i,k could be defined to be random variables of
the correct magnitude, i.e., O(k). Following a suggestion of Parlett [7] we used
p
k = kAk, i,k = kAk.
0
0 0
1 1 0
1 0 1 0

10.6. PARTIAL REORTHOGONALIZATION

1 1 0 1 0
1 1 1 0 1 0
0 1 1 0 0 0 0
1 0 1 1 0 1 1 0
1 0 0 1 0 1 1 1 0
1 0 0 1 1 1 1 0 1 0
0 1 0 1 1 1 1 0 0 0 0
0 1 0 1 1 0 1 1 0 0 0 0
1 1 1 1 1 1 0 1 1 0 1 1 0
1 1 1 1 1 1 0 0 1 0 1 0 1 0
1 1 0 1 0 1 1 1 1 0 1 1 0 1 0
1 1 1 0 1 1 1 0 1 0 1 1 1 0 1 0
0 1 1 1 1 1 0 1 1 1 0 1 1 0 0 1 0
1 1 1 1 1 0 1 1 1 1 1 1 1 1 0 1 1 0
0 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0
1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 0 0 0
1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 1 1 1 0 0 0
1 1 1 1 2 1 2 1 1 1 1 1 1 1 1 0 1 1 0 0 1 0
0 2 1 2 1 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 0 0 0
1 1 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 0 0 0
1 2 2 2 2 2 2 2 2 2 2 2 1 2 1 2 1 1 1 1 1 0 0 1 0
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 0 0 1 0
2 2 2 2 2 3 2 2 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 0 0 0 0
2 2 3 3 3 3 3 3 3 2 3 2 2 2 2 2 2 2 2 1 1 1 1 1 1 0 1 0
2 3 3 3 3 3 3 3 3 3 3 3 2 3 2 2 2 2 2 2 2 2 1 1 1 0 1 1 0
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 2 2 2 2 2 2 2 2 1 2 1 1 1 1 0
3 3 3 4 3 4 3 3 3 3 3 3 3 3 3 3 3 3 2 2 2 2 2 2 1 1 1 1 1 1 0
3 3 4 4 4 4 4 4 4 4 4 3 3 3 3 3 3 3 3 2 2 2 2 2 2 1 2 1 1 0 1 0
4 4 4 4 4 4 4 4 4 4 4 4 4 4 3 3 3 3 3 3 3 3 2 2 2 2 1 1 1 1 0 0 0
4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 3 3 3 3 3 3 2 2 2 2 1 1 1 1 1 1 0
4 5 5 5 5 5 5 5 5 5 4 5 4 4 4 4 4 4 4 4 3 3 3 3 3 2 2 2 2 2 1 1 1 1 0
5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 4 4 4 4 4 4 3 3 3 3 3 3 2 2 2 1 1 2 0 0 0
5 5 5 6 5 6 5 5 5 5 5 5 5 5 5 5 5 5 4 4 4 4 4 4 3 3 3 3 2 2 2 1 1 1 1 0 0
6 6 6 6 6 6 6 6 6 6 6 5 6 5 5 5 5 5 5 5 5 4 4 4 4 3 3 3 3 2 2 2 2 1 1 1 0 0
6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 5 5 5 5 5 5 4 4 4 4 4 3 3 3 2 2 2 2 1 1 0 0 0
6 6 7 7 7 7 7 7 7 6 7 6 6 6 6 6 6 6 6 5 5 5 5 5 5 4 4 4 4 3 3 3 2 2 2 1 1 1 1 0
7 7 7 7 7 7 7 7 7 7 7 7 7 7 6 7 6 6 6 6 6 6 5 5 5 5 4 4 4 4 3 3 3 3 2 2 1 1 1 1 0
7 7 8 8 8 8 8 7 8 7 8 7 7 7 7 7 7 7 7 6 6 6 6 6 6 5 5 5 5 4 4 4 3 3 3 2 2 1 2 0 1 0
8 8 8 8 8 8 8 8 8 8 8 8 8 8 7 8 7 7 7 7 7 7 6 6 6 6 5 5 5 5 4 4 4 4 3 3 2 1 1 1 1 1 0
8 8 9 9 9 9 9 9 9 8 9 8 8 8 8 8 8 8 8 7 7 7 7 7 7 6 6 6 6 5 5 5 4 4 4 3 3 2 2 1 2 1 1 0
9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 8 8 8 8 8 8 7 7 7 7 7 6 6 6 5 5 5 5 4 4 3 3 3 2 1 2 1 1 0
10 10 10 10 10 10 10 10 10 10 10 10 10 9 9 9 9 9 9 9 9 8 8 8 8 7 7 7 7 6 6 6 6 5 5 4 4 4 3 3 2 1 2 1 1 0
10 11 10 11 10 11 10 11 10 11 10 10 10 10 10 10 10 10 9 9 9 9 9 9 8 8 8 8 7 7 7 7 6 6 5 5 5 4 4 3 3 2 1 2 1 2 0
11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 10 10 10 10 10 10 9 9 9 9 8 8 8 8 7 7 7 6 6 6 5 5 4 4 3 1 1 2 1 1 0
12 12 12 12 12 12 12 12 12 12 12 12 12 12 11 12 11 11 11 11 11 11 10 10 10 10 9 9 9 9 8 8 8 8 7 7 6 6 5 5 5 4 3 2 1 2 2 1 0

189
13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 12 12 12 12 12 12 11 11 11 11 11 11 10 10 10 9 9 9 8 8 8 7 7 7 6 6 5 4 4 3 1 3 2 1 0
Table 10.1: Matlab demo on the loss of orthogonality among Lanczos vectors. Unmodified Lanczos. round(log10(abs(I-Q50Q50 )/eps))
190 CHAPTER 10. ARNOLDI AND LANCZOS ALGORITHMS

Reorthogonalization takes place in the j-th Lanczos step if maxk (j+1,k ) > macheps.
qj+1 is orthogonalized against all vectors qk with j+1,k > macheps3/4 . In the following
iteration step also qj+2 is orthogonalized against these vectors. In Table 10.2 the base-
10 logarithms of the values |wi,j |/macheps obtained with this procedure are listed where
|wi,j | = |qi qj |, 1 j i 50 and macheps 2.2 1016 . In Table 10.3 the base-10
logarithms of the estimates |i,j |/macheps are given. The estimates are too high by (only)
an order of magnitude. However, the procedure succeeds in that the resulting {qk } are
semi-orthogonal.

10.7 Block Lanczos

As we have seen, the Lanczos algorithm produces a sequence {qi } of orthonormal vec-
tors. These Lanczos vectors build an orthonormal basis for the Krylov subspace Kj (x) =
span{q1 , . . . , qj } Rn . The restriction of A to Kj (x) is an unreduced tridiagonal ma-
trix. However the Lanczos algorithm cannot detect the multiplicity of the eigenvalues it
computes. This limitation prompted the development of the block version of the Lanc-
zos process (Block Lanczos algorithm), which is capable of determining multiplicities of
eigenvalues up to the block size.
The idea is not to start with a single vector q1 Rn but with a set of mutually
orthogonal vectors which we take as the columns of the matrix Q1 Rnp with the block
size p > 1.
Associated with Q1 is the big Krylov subspace

(10.36) Kjp (Q1 ) = span{Q1 , AQ1 , . . . , Aj1 Q1 }.

(We suppose, for simplicity, that Aj1 Q1 has rank p. Otherwise we would have to consider
variable block sizes.)
The approach is similar to the scalar case with p = 1: Let Q1 , . . . , Qj Rnp be
pairwise orthogonal block matrices (Qi Qk = O for i 6= k) with orthonormal columns
(Qi Qi = Ip for all i j). Then, in the j-th iteration step, we obtain the matrix AQj
and orthogonalize it against matrices Qi , i j. The columns of the matrices are obtained
by means of the QR factorization or with the GramSchmidt orthonormalization process.
We obtained the following:

Algorithm 10.5 Block Lanczos algorithm

1: Choose Q1 Fnp such that Q1 Q1 = Ip . Set j := 0 and Fnp V := 0.
This algorithm generates a block tridiagonal matrix Tj with the diagonal blocks Ai ,
i j, the lower diagonal blocks Bi , i < j, and the Krylov basis [Q1 , . . . , Qj ] of Kjp (Q1 ).
2: for j 0 do
3: if j > 0 then
4: V =: Qj+1 Bj ; /* QR decomposition */
5: V := Qj Bj ;
6: end if
7: j := j + 1;
8: Aj := Qj V ;
9: V := V Qj Aj ;
10: Test for convergence (Ritz pairs, evaluation of error)
11: end for
0
0 0
1 1 0
1 0 1 0

10.7. BLOCK LANCZOS

1 1 0 1 0
1 1 1 0 1 0
0 1 1 0 0 0 0
1 0 1 1 0 1 1 0
1 0 0 1 0 1 1 1 0
1 0 0 1 1 1 1 0 1 0
0 1 0 1 1 1 1 0 0 0 0
0 1 0 1 1 0 1 1 0 0 0 0
1 1 1 1 1 1 0 1 1 0 1 1 0
1 1 1 1 1 1 0 0 1 0 1 0 1 0
1 1 0 1 0 1 1 1 1 0 1 1 0 1 0
1 1 1 0 1 1 1 0 1 0 1 1 1 0 1 0
0 1 1 1 1 1 0 1 1 1 0 1 1 0 0 1 0
1 1 1 1 1 0 1 1 1 1 1 1 1 1 0 1 1 0
0 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0
1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 0 0 0
1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 1 1 1 0 0 0
1 1 1 1 2 1 2 1 1 1 1 1 1 1 1 0 1 1 0 0 1 0
0 2 1 2 1 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 0 0 0
1 1 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 0 0 0
1 2 2 2 2 2 2 2 2 2 2 2 1 2 1 2 1 1 1 1 1 0 0 1 0
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 0 0 1 0
2 2 2 2 2 3 2 2 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 0 0 0 0
2 2 3 3 3 3 3 3 3 2 3 2 2 2 2 2 2 2 2 1 1 1 1 1 1 0 1 0
2 3 3 3 3 3 3 3 3 3 3 3 2 3 2 2 2 2 2 2 2 2 1 1 1 0 1 1 0
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 2 2 2 2 2 2 2 2 1 2 1 1 1 1 0
3 3 3 4 3 4 3 3 3 3 3 3 3 3 3 3 3 3 2 2 2 2 2 2 1 1 1 1 1 1 0
3 3 4 4 4 4 4 4 4 4 4 3 3 3 3 3 3 3 3 2 2 2 2 2 2 1 2 1 1 0 1 0
4 4 4 4 4 4 4 4 4 4 4 4 4 4 3 3 3 3 3 3 3 3 2 2 2 2 1 1 1 1 0 0 0
4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 3 3 3 3 3 3 2 2 2 2 1 1 1 1 1 1 0
4 5 5 5 5 5 5 5 5 5 4 5 4 4 4 4 4 4 4 4 3 3 3 3 3 2 2 2 2 2 1 1 1 1 0
5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 4 4 4 4 4 4 3 3 3 3 3 3 2 2 2 1 1 2 0 0 0
5 5 5 6 5 6 5 5 5 5 5 5 5 5 5 5 5 5 4 4 4 4 4 4 3 3 3 3 2 2 2 1 1 1 1 0 0
6 6 6 6 6 6 6 6 6 6 6 5 6 5 5 5 5 5 5 5 5 4 4 4 4 3 3 3 3 2 2 2 2 1 1 1 0 0
6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 5 5 5 5 5 5 4 4 4 4 4 3 3 3 2 2 2 2 1 1 0 0 0
6 6 7 7 7 7 7 7 7 6 7 6 6 6 6 6 6 6 6 5 5 5 5 5 5 4 4 4 4 3 3 3 2 2 2 1 1 1 1 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 3 2 2 1 1 1 1 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 3 3 2 2 1 2 0 1 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 3 3 3 3 2 1 1 1 1 1 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 3 3 3 3 3 3 2 2 1 2 1 0 0
1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 3 3 3 3 4 3 3 3 3 2 1 2 1 1 0
1 1 1 0 0 0 1 0 0 0 1 0 0 0 1 0 1 1 1 1 1 1 0 1 1 0 1 0 3 3 4 4 4 4 4 4 4 4 3 3 2 1 1 1 0 0
1 1 2 1 1 1 0 1 1 1 0 1 1 1 1 1 1 1 2 1 2 1 1 1 1 2 0 3 4 4 4 4 4 4 4 4 4 4 4 3 3 2 2 2 1 1 0
2 2 2 2 1 1 2 1 2 1 2 1 1 2 2 2 2 2 2 2 2 2 1 2 2 2 4 4 4 5 5 5 5 5 5 5 5 5 4 4 4 3 2 2 2 1 1 0
3 3 3 3 3 2 2 2 2 3 2 3 2 2 2 3 3 3 3 3 3 2 3 3 3 4 5 5 5 5 6 6 6 6 6 5 5 5 5 5 5 4 3 2 2 2 1 1 0
4 4 4 4 3 3 3 3 3 3 4 3 3 3 4 4 4 4 4 4 4 4 3 4 5 5 6 6 6 6 6 7 6 7 6 6 6 6 6 6 5 5 4 4 2 2 2 2 2 0

191
Table 10.2: Matlab demo on the loss of orthogonality among Lanczos vectors: Lanczos with partial reorthogonalization.
round(log10(abs(I-Q50Q50 )/eps))
 0
2 0
0 2 0
1 0 2 0

CHAPTER 10. ARNOLDI AND LANCZOS ALGORITHMS

0 1 0 2 0
1 0 1 0 2 0
0 1 0 1 0 2 0
1 0 1 0 1 0 2 0
0 1 1 1 0 1 0 2 0
1 1 1 1 1 1 1 0 2 0
1 1 1 1 1 1 1 1 0 2 0
1 1 1 1 1 1 1 1 1 0 2 0
1 1 1 1 1 1 1 1 1 1 0 2 0
1 1 1 1 1 1 1 1 1 1 1 0 2 0
1 1 1 1 1 2 1 1 1 1 1 1 0 2 0
1 1 2 1 2 1 2 1 2 1 1 1 1 0 2 0
1 2 1 2 2 2 2 2 1 2 1 1 1 1 0 2 0
1 2 2 2 2 2 2 2 2 1 2 1 2 1 1 0 2 0
1 2 2 2 2 2 2 2 2 2 2 2 1 2 1 1 0 2 0
2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 1 1 0 2 0
2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 1 1 0 2 0
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 2 1 1 0 2 0
2 2 2 3 2 3 2 3 2 2 2 2 2 2 2 2 2 2 1 2 0 2 0
2 2 3 3 3 3 3 3 3 2 3 2 2 2 2 2 2 2 2 1 2 0 2 0
2 3 3 3 3 3 3 3 3 3 3 3 2 3 2 2 2 2 2 2 1 2 0 2 0
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 2 2 2 2 2 2 1 2 0 2 0
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 2 3 2 2 2 2 1 2 0 2 0
3 3 4 3 4 3 4 3 4 3 3 3 3 3 3 3 3 3 3 2 2 2 2 1 2 0 2 0
3 4 4 4 4 4 4 4 4 4 4 4 3 3 3 3 3 3 3 3 2 2 2 2 1 2 0 2 0
4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 3 3 3 3 3 3 2 2 2 2 1 2 0 2 0
4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 3 4 3 3 3 3 2 3 2 2 1 2 0 2 0
4 4 5 5 5 5 5 5 5 4 5 4 4 4 4 4 4 4 4 3 3 3 3 3 3 2 2 1 2 0 2 0
4 5 5 5 5 5 5 5 5 5 5 5 4 5 4 4 4 4 4 4 3 3 3 3 3 3 2 2 2 2 0 2 0
5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 4 5 4 4 4 4 4 4 3 3 3 3 2 2 2 2 0 2 0
5 6 5 6 6 6 6 6 5 6 5 5 5 5 5 5 5 5 4 4 4 4 4 4 3 3 3 3 2 2 2 2 0 2 0
6 6 6 6 6 6 6 6 6 6 6 6 6 5 5 5 5 5 5 5 5 4 4 4 4 3 3 3 3 2 2 2 2 0 2 0
6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 5 5 5 5 5 5 4 4 4 4 4 4 3 3 2 3 2 2 0 2 0
6 6 7 7 7 7 7 7 7 7 7 6 6 6 6 6 6 6 6 5 5 5 5 5 5 4 4 4 4 3 3 3 3 2 2 0 2 0
7 7 7 7 7 7 7 7 7 7 7 7 7 7 6 7 6 6 6 6 6 6 5 5 5 5 4 4 4 4 3 3 3 3 2 2 0 2 0
7 7 8 7 8 8 8 7 8 7 7 7 7 7 7 7 7 7 7 6 6 6 6 5 5 5 5 5 5 4 4 3 3 3 3 2 2 0 2 0
8 8 8 8 8 8 8 8 8 8 8 8 8 8 7 7 7 7 7 7 7 7 6 6 6 6 5 5 5 5 4 4 4 4 3 3 2 2 0 2 0
7 7 8 8 8 8 8 7 8 7 7 7 7 7 7 7 7 7 7 6 6 6 6 5 6 5 5 5 5 4 4 3 3 4 4 3 3 2 2 0 2 0
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 4 4 4 3 3 2 2 0 2 0
2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 2 2 2 2 2 4 4 4 4 4 4 4 3 3 2 2 0 2 0
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 3 3 2 2 0 2 0
4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 3 4 4 5 5 5 5 5 5 5 4 4 3 4 2 3 0 2 0
4 4 5 5 5 5 5 5 5 5 5 5 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 5 4 5 5 5 5 5 5 5 5 5 5 5 4 4 3 3 0 2 0
5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 4 6 5 6 6 6 6 6 6 6 5 5 5 5 4 4 3 3 0 2 0
6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 5 6 6 6 6 7 6 7 6 7 6 6 6 6 5 5 4 4 3 3 0 2 0
192

7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 6 7 6 7 7 7 7 8 7 7 7 7 7 7 6 6 6 6 5 5 3 3 0 2 0
Table 10.3: Matlab demo on the loss of orthogonality among Lanczos vectors: Lanczos with partial reorthogonalization.
round(log10(abs(I-W50)/eps))
10.8. EXTERNAL SELECTIVE REORTHOGONALIZATION 193
Let Q j := [Q1 , Q2 , . . . , Qj ] be the Krylov basis generated by Algorithm 10.5. Then, in
this basis, the projection of A is the block tridiagonal matrix Tj

A1 B1
..
B1 A2 .
AQ
Q j = Tj = , Ai , Bi Rpp .
j .. ..

. . Bj1
Bj1 Aj

If matrices Bi are chosen to be upper triangular, then Tj is a band matrix with bandwidth
2p + 1!
Similarly as in scalar case, in the j-th iteration step we obtain the equation

O
..

AQj Q j Tj = Qj+1 Bj E + Fj ,
j Ej = . ,
O
Ip

where Fj accounts for the effect of roundoff error. Let (i , yi ) be a Ritz pair of A in
Kjp (Q1 ). Then
yi = Q j si , Tj si = i si .
As before, we can consider the residual norm to study the accuracy of the Ritz pair (i , yi )
of A

sj(p1)+1,i

kAyi i yi k = kAQ j si k kQj+1 Bj E si k =
j si i Q B


..
.
j

j .
sjp+1,i

We have to compute the bottom p components of the eigenvectors si in order to test for
convergence.
Similarly as in the scalar case, the mutual orthogonality of the Lanczos vectors (i.e.,
the columns of Q j ) is lost, as soon as convergence sets in. The remedies described earlier
are available: full reorthogonalization or selective orthogonalization.

10.8 External selective reorthogonalization

If many eigenvalues are to be computed with the Lanczos algorithm, it is usually advisable
to execute shift-and-invert Lanczos with varying shifts [4].
In each new start of a Lanczos procedure, one has to prevent the algorithm from finding
already computed eigenpairs. We have encountered this problem when we tried to compute
multiple eigenpairs by simple vector iteration. Here, the remedy is the same as there. In
the second and further runs of the Lanczos algorithm, the starting vectors are made
orthogonal to the already computed eigenvectors. We know that in theory all Lanczos
vectors will be orthogonal to the previously computed eigenvectors. However, because the
previous eigenvectors have been computed only approximately the initial vectors are not
orthogonal to the true eigenvectors. Because of this and because of floating point errors
loss of orthogonality is observed. The loss of orthogonality can be monitored similarly as
with partial reorthogonalization. For details see [4].
194 CHAPTER 10. ARNOLDI AND LANCZOS ALGORITHMS
Bibliography
[1] W. E. Arnoldi, The principle of minimized iterations in the solution of the matrix
eigenvalue problem, Quarterly of Applied Mathematics, 9 (1951), pp. 1729.

[2] J. K. Cullum and R. A. Willoughby, Lanczos Algorithms for Large Symmetric

Eigenvalue Computations, vol. 1: Theory, Birkhauser, Boston, 1985.

[3] G. H. Golub and J. H. Welsch, Calculation of Gauss quadrature rules, Math.

Comp., 23 (1969), pp. 221230.

[4] R. Grimes, J. G. Lewis, and H. Simon, A shifted block Lanczos algorithm for
solving sparse symmetric generalized eigenproblems, SIAM J. Matrix Anal. Appl., 15
(1994), pp. 228272.

[5] N. Krylov and N. Bogoliubov, Sur le calcul des racines de la transcendante de

Fredholm les plus voisines dune nombre donne par les methodes des moindres carres
et de lalgorithme variationel, Izv. Akad. Naik SSSR, Leningrad, (1929), pp. 471488.

[6] C. Lanczos, An iteration method for the solution of the eigenvalue problem of linear
differential and integral operators, J. Res. Nat. Bureau Standards, Sec. B, 45 (1950),
pp. 255282.

[7] B. N. Parlett, The Symmetric Eigenvalue Problem, Prentice Hall, Englewood Cliffs,
NJ, 1980. (Republished by SIAM, Philadelphia, 1998.).

[8] H. Simon, Analysis of the symmetric Lanczos algorithm with reorthogonalization meth-
ods, Linear Algebra Appl., 61 (1984), pp. 101132.

[9] , The Lanczos algorithm with partial reorthogonalization, Math. Comp., 42 (1984),
pp. 115142.
Chapter 11

Restarting Arnoldi and Lanczos

algorithms

The number of iteration steps can be very high with the Arnoldi or the Lanczos algorithm.
This number is, of course, not predictable. The iteration count depends on properties of
the matrix, in particular the distribution of its eigenvalues, but also on the initial vectors.
High iteration counts entail a large memory requirement to store the Arnoldi/Lanczos
vectors and a high amount of computation because of growing cost of the reorthogonal-
ization.
The idea behind the implicitely restarted Arnoldi (IRA) and implicitely restarted Lanc-
zos (IRL) algorithms is to reduce these costs by limiting the dimension of the search space.
This means that the iteration is stopped after a number of steps (which is bigger than
the number of desired eigenvalues), reduce the dimension of the search space without
destroying the Krylov space structure, and finally resume the Arnoldi / Lanczos iteration.
The implicitely restarted Arnoldi has first been proposed by Sorensen [7, 8]. It is imple-
mented together with the implicitely restarted Lanczos algorithms in the software package
ARPACK [4]. The ARPACK routines are the basis for the sparse matrix eigensolver eigs
in Matlab.

11.1 The m-step Arnoldi iteration

Algorithm 11.1 The m-step Arnoldi iteration

1: Let A Fnn . This algorithm executes m steps of the Arnoldi algorithm.
2: q1 = x/kxk; z = Aq1 ; 1 = q1 z;
3: r1 = w 1 q1 ; Q1 = [q1 ]; H1 = [1 ];
4: for j = 1, . . . , m 1 do
5: j := krj k; qj+1 = rj /j ;  
Hj
6: Qj+1 := [Qj , qj+1 ]; Hj := F(j+1)j ;
j eTj
7: z := Aqj ;
8: h := Qj+1 z; rj+1 := z Qj+1 h;
9: Hj+1 := [H j , h];
10: end for

We start with the Algorithm 11.1 that is a variant of the Arnoldi Algorithm 10.1. It

195
196 CHAPTER 11. RESTARTING ARNOLDI AND LANCZOS ALGORITHMS
executes just m Arnoldi iteration steps. We will now show how the dimension of the search
space is reduced withouth losing the information regarding the eigenvectors one is looking
for.
Remark 11.1. Step 8 in Algorithm 11.1 is classical GramSchmidt orthogonalization. As

rj+1 = z Qj+1 h = z Qj+1 Qj+1 z,

we formally have Qj+1 rj+1 = 0. However, classical GramSchmidt orthogonalization is

faster but not so accurate as modified GramSchmidt orthogonalization [1]. So, often,
Qj+1 rj+1 is quite large. Therefore, the orthogonalization is iterated to get sufficient or-
thogonality.
A possible modification of step 8 that incorporates a second iteration is

8: h := Qj+1 z; rj+1 := z Qj+1 h;

c := Qj+1 rj+1 ; rj+1 := rj+1 Qj+1 c; h = h + c;

Now we have,

rj+1 = corrected rj+1
= rj+1 Qj+1 Qj+1 rj+1
| {z }
c

= z Qj+1 Qj+1 z Qj+1 Qj+1 rj+1 = z Qj+1 (h + c)
| {z } | {z }
h c

More iterations are possible but seldom necessary.

After the execution of Algorithm 11.1 we have the Arnoldi / Lanczos relation
" #
(11.1) AQm = Qm Hm + rm em , Hm =

available with
rm = m qm+1 , kqm+1 k = 1.
If m = 0 then R(Qm ) is invariant under A, i.e., Ax R(Qm ) for all x R(Qm ).
This lucky situation implies that (Hm ) m (A). So, the Ritz values and vectors are
eigenvalues and eigenvectors of A.
What we can realistically hope for is m being small. Then,

AQm rm em = (A rm qm )Qm = Qm Hm .

Then, R(Qm ) is invariant under a matrix A + E, that differs from A by a perturbation E

with kEk = krm k = |m |. From general eigenvalue theory we know that in this situation
well-conditioned eigenvalues of Hm are good approximations of eigenvalues of A.
In the sequel we investigate how we can find a q1 such that m becomes small?

11.2 Implicit restart

Let us start from the Arnoldi relation

(11.2) AQm = Qm Hm + rm em ,
11.2. IMPLICIT RESTART 197
Algorithm 11.2 k implicit QR steps applied to Hm
1: Hm+ := H .
m
2: for i := 1, . . . , k do
Hm + := V H + V , + I =V R
where Hm
3: i m i i i i (QR factorization)
4: end for

that is obtained after calling Algorithm 11.1.

We apply k < m implicit QR steps to Hm with shifts 1 , . . . , k , see Algorithm 11.2.
Let V + := V1 V2 Vk . V + is the product of k (unitary) Hessenberg matrices whence it
has k nonzero off-diagonals below its main diagonal.

Vm+ = .

}k
We define
Q+ +
m := Qm V ,
+
Hm := (V + ) Hm V + .
Then, from (11.2) we obtain

AQm V + = Qm V + (V + ) Hm V + + rm em V + ,

or

(11.3) AQ+ + + +
m = Qm Hm + rm em V .

As V + has k nonzero off-diagonals below the main diagonal, the last row of V + has the
form
em V + = (0, . . . , 0, , . . . , ), k + p = m.
| {z } | {z }
p1 k+1
We now simply discard the last k columns of (11.3).

AQ+ + + +
m (:, 1 : p) = Qm Hm (:, 1 : p) + rm em V (:, 1 : p)
+ +
= Q+ + +
m (:, 1 : p)Hm (1 : p, 1 : p) + hp+1,p qp+1 ep + vm,p rm ep
| {z }
p+
= Q+ + + + +
m (:, 1 : p)Hm (1 : p, 1 : p) + (qp+1 hp+1,p + rm vm,p ) ep .
| {z }
r+
p

In Algorithm 11.3 we have collected what we have derived so far. We have however left
open in step 3 of the algorithm how the shifts 1 , . . . , k should be chosen. In ARPACK [4],
all eigenvalues of Hm are computed. Those k eigenvalues that are furthest away from some
target value are chosen as shifts. We have not specified how we determine convergence,
too.
One can show that a QR step with shift i transforms the vector q1 in a multiple of
(A i I)q1 . In fact, a simple modification of the Arnoldi relation (11.2) gives

(A i I)Qm = Qm (Hm i I) +rm em = Qm V1 R1 + rm em .

| {z }
V1 R1
198 CHAPTER 11. RESTARTING ARNOLDI AND LANCZOS ALGORITHMS
Algorithm 11.3 Implicitely restarted Arnoldi (IRA)
1: Let the Arnoldi relation AQm = Qm Hm + rm em be given.
2: repeat
3: Determine k shifts 1 , . . . , k ;
4: v := em ;
5: for i = 1, . . . , k do
6: Hm i I = Vi Ri ; /* QR factorization */
7: Hm := Vi Hm Vi ; Qm := Qm Vi ;
8: v := v Vi ;
9: end for
10: rp := q+ + +
p+1 p + rm vm,p ;
11: Qp := Qm (:, 1 : p); Hp := Hm (1 : p, 1 : p);
12: Starting with
AQp = Qp Hp + rp ep
execute k additional steps of the Arnoldi algorithm until

AQm = Qm Hm + rm em .

13: until convergence

Comparing the first columns in this equation gives

(1)
(A i I)q1 = Qm V1 e1 r11 + 0 = q1 r11 .

By consequence, all k steps combined give

k
Y
q1 (A)q1 , () = ( i ).
i=1

If i were an eigenvalue of A then (A i I)q1 removes components of q1 in the direction

of the corresponding eigenvector. More general, if i is close to an eigenvalue of A then
(Ai I)q1 will have only small components in the direction of eigenvectors corresponding
to nearby eigenvalues. Choosing the i equal to Ritz values far away from the desired part
of the spectrum thus enhances the desired component. Still there is the danger that in each
sweep on Algorithm 11.3 the same undesired Ritz values are recovered. Therefore, other
strategies for choosing the shifts have been proposed [2]. Experimental results indicate
however, that the original strategy chosen in ARPACK mostly works best.

11.3 Convergence criterion

Let Hm s = s with ksk = 1. Let x
= Qm s. Then we have as earlier

(11.4) kA xk = kAQm s Qm Hm sk = krm k|em s| = m |em s|.

x

In the Hermitian case, A = A , the Theorem 10.1 of KrylovBogoliubov provides an

interval that contains an eigenvalue of A. In the general case, we have

(11.5) (A + E)
x =
x, E = rm qm , kEk = krm k = m .
11.4. THE GENERALIZED EIGENVALUE PROBLEM 199
According to an earlier theorem we know that if (A) is simple and is the
eigenvalue of A + E closest to , then
kEk
(11.6) | | + O(kEk2 ).
y x
Here, y and x are left and right eigenvectors of E corresponding to the eigenvalue . A
similar statement holds for the eigenvectors, but the distance (gap) to the next eigenvalue
comes into play as well.
In ARPACK, a Ritz pair (, x ) is considered converged if

(11.7) m |em s| max(M kHm k, tol ||).

As || kHm k kAk, the inequality kEk tol kAk holds at convergence. According
to (11.6) well-conditioned eigenvalues are well approximated.

11.4 The generalized eigenvalue problem

Let us consider now the generalized eigenvalue problem

(11.8) Ax = M x.

Applying a shift-and-invert spectral transformation with shift transforms (11.8) into

1
(11.9) Sx = (A M )1 M x = x, = .

We now execute an Arnoldi/Lanczos iteration with S to obtain

(11.10) SQm = Qm Hm + rm em , Qm M Qm = Im , Qm M rm = 0.

Let s with ksk = 1 be an eigenvector of Hm with Ritz value . Let y = Qm s be the

associated Ritz vector. Then,

(11.11) SQm s = Sy = Qm Hm s + rm em s = y + rm em s.

So, y+rm em s can be considered a vector that is obtained by one step of inverse iteration.
This vector is an improved approximation to the desired eigenvector, obtained at negligible
cost. This so-called eigenvector purification is particularly important if M is singular.
Let us bound the residual norm of the purified vector. With (11.11) we have

(11.12) M y = (A M )(y + rm em s)
| {z }
y

with q
k
ykM = 2 + k2 |em s|2 .
This equality holds as y M r. By consequence,
kA
y M y
k = k(A M )
y + My
( })k
| {z
(11.13) 1

= kM y M (y + rm em s)/k = kM rk |em s|/||.

Since || is large in general, we obtain good bounds for the residual of the purified eigen-
vectors.
200 CHAPTER 11. RESTARTING ARNOLDI AND LANCZOS ALGORITHMS
EIGS Find a few eigenvalues and eigenvectors of a matrix using ARPACK
D = EIGS(A) returns a vector of As 6 largest magnitude eigenvalues.
A must be square and should be large and sparse.

[V,D] = EIGS(A) returns a diagonal matrix D of As 6 largest magnitude

eigenvalues and a matrix V whose columns are the corresponding
eigenvectors.

[V,D,FLAG] = EIGS(A) also returns a convergence flag. If FLAG is 0 then

all the eigenvalues converged; otherwise not all converged.

EIGS(A,B) solves the generalized eigenvalue problem A*V == B*V*D. B

must be symmetric (or Hermitian) positive definite and the same size as
A. EIGS(A,[],...) indicates the standard eigenvalue problem A*V == V*D.

EIGS(A,K) and EIGS(A,B,K) return the K largest magnitude eigenvalues.

EIGS(A,K,SIGMA) and EIGS(A,B,K,SIGMA) return K eigenvalues. If SIGMA is:

LM or SM - Largest or Smallest Magnitude
For real symmetric problems, SIGMA may also be:
LA or SA - Largest or Smallest Algebraic
BE - Both Ends, one more from high end if K is odd
For nonsymmetric and complex problems, SIGMA may also be:
LR or SR - Largest or Smallest Real part
LI or SI - Largest or Smallest Imaginary part
If SIGMA is a real or complex scalar including 0, EIGS finds the
eigenvalues closest to SIGMA. For scalar SIGMA, and when SIGMA = SM,
B need only be symmetric (or Hermitian) positive semi-definite since it
is not Cholesky factored as in the other cases.

EIGS(A,K,SIGMA,OPTS) and EIGS(A,B,K,SIGMA,OPTS) specify options:

OPTS.issym: symmetry of A or A-SIGMA*B represented by AFUN [{false} | true]
OPTS.isreal: complexity of A or A-SIGMA*B represented by AFUN [false | {true}]
OPTS.tol: convergence: Ritz estimate residual <= tol*NORM(A) [scalar | {eps}]
OPTS.maxit: maximum number of iterations [integer | {300}]
OPTS.p: number of Lanczos vectors: K+1<p<=N [integer | {2K}]
OPTS.v0: starting vector [N-by-1 vector | {randomly generated}]
OPTS.disp: diagnostic information display level [0 | {1} | 2]
OPTS.cholB: B is actually its Cholesky factor CHOL(B) [{false} | true]
OPTS.permB: sparse B is actually CHOL(B(permB,permB)) [permB | {1:N}]
Use CHOL(B) instead of B when SIGMA is a string other than SM.

EIGS(AFUN,N) accepts the function AFUN instead of the matrix A. AFUN is

a function handle and Y = AFUN(X) should return
A*X if SIGMA is unspecified, or a string other than SM
A\X if SIGMA is 0 or SM
(A-SIGMA*I)\X if SIGMA is a nonzero scalar (standard problem)
(A-SIGMA*B)\X if SIGMA is a nonzero scalar (generalized problem)
N is the size of A. The matrix A, A-SIGMA*I or A-SIGMA*B represented by
AFUN is assumed to be real and nonsymmetric unless specified otherwise
by OPTS.isreal and OPTS.issym. In all these EIGS syntaxes, EIGS(A,...)
may be replaced by EIGS(AFUN,N,...).

Example:
A = delsq(numgrid(C,15)); d1 = eigs(A,5,SM);

Equivalently, if dnRk is the following one-line function:

%----------------------------%
function y = dnRk(x,R,k)
y = (delsq(numgrid(R,k))) \ x;
%----------------------------%

n = size(A,1); opts.issym = 1;
d2 = eigs(@(x)dnRk(x,C,15),n,5,SM,opts);

See also eig, svds, ARPACKC, function_handle.

11.5. A NUMERICAL EXAMPLE 201
11.5 A numerical example
This example is taken from the Matlab document pages regarding eigs. eigs is the
Matlab interface to the ARPACK code, see page 200. The matrix called west0479 is a
479479 matrix originating in a chemical engineering plant model. The matrix is available
from the Matrix Market [5], a web site that provides numerous test matrices. Its nonzero
structure is given in Fig. 11.1

479x479 Matrix west0479

0

50

100

150

200

250

300

350

400

450

0 100 200 300 400

nnz = 1887

Figure 11.1: Nonzero structure of the 479 479 matrix west0479

To compute the eight largest eigenvalues of this matrix we issue the following Matlab
commands.

>> load west0479

>> d = eig(full(west0479));
>> dlm=eigs(west0479,8);
Iteration 1: a few Ritz values of the 20-by-20 matrix:
0
0
0
0
0
0
0
0
0

Iteration 2: a few Ritz values of the 20-by-20 matrix:

1.0e+03 *

-0.0561 - 0.0536i
202 CHAPTER 11. RESTARTING ARNOLDI AND LANCZOS ALGORITHMS
0.1081 + 0.0541i
0.1081 - 0.0541i
-0.1009 - 0.0666i
-0.1009 + 0.0666i
-0.0072 + 0.1207i
-0.0072 - 0.1207i
0.0000 - 1.7007i
0.0000 + 1.7007i

Iteration 3: a few Ritz values of the 20-by-20 matrix:

1.0e+03 *

-0.0866
-0.1009 - 0.0666i
-0.1009 + 0.0666i
-0.0072 + 0.1207i
-0.0072 - 0.1207i
0.1081 - 0.0541i
0.1081 + 0.0541i
0.0000 - 1.7007i
0.0000 + 1.7007i

Iteration 4: a few Ritz values of the 20-by-20 matrix:

1.0e+03 *

0.0614 - 0.0465i
-0.0072 - 0.1207i
-0.0072 + 0.1207i
0.1081 + 0.0541i
0.1081 - 0.0541i
-0.1009 + 0.0666i
-0.1009 - 0.0666i
0.0000 - 1.7007i
0.0000 + 1.7007i

Iteration 5: a few Ritz values of the 20-by-20 matrix:

1.0e+03 *

-0.0808
-0.0072 + 0.1207i
-0.0072 - 0.1207i
-0.1009 - 0.0666i
-0.1009 + 0.0666i
0.1081 + 0.0541i
0.1081 - 0.0541i
0.0000 + 1.7007i
0.0000 - 1.7007i

Iteration 6: a few Ritz values of the 20-by-20 matrix:

1.0e+03 *

0.0734 - 0.0095i
-0.0072 + 0.1207i
-0.0072 - 0.1207i
0.1081 - 0.0541i
11.5. A NUMERICAL EXAMPLE 203
2000

1500

1000

500

500

1000

1500

2000
150 100 50 0 50 100 150

Figure 11.2: Spectrum of the matrix west0479

0.1081 + 0.0541i
-0.1009 - 0.0666i
-0.1009 + 0.0666i
0.0000 - 1.7007i
0.0000 + 1.7007i

Iteration 7: a few Ritz values of the 20-by-20 matrix:

1.0e+03 *

-0.0747
-0.0072 - 0.1207i
-0.0072 + 0.1207i
0.1081 + 0.0541i
0.1081 - 0.0541i
-0.1009 + 0.0666i
-0.1009 - 0.0666i
0.0000 + 1.7007i
0.0000 - 1.7007i

The output indicates that eigs needs seven sweeps to compute the eigenvalues to
the default accuracy of machepskAk. The Ritz values given are the approximations of the
eigenvalues we want to compute. The complete spectrum of west0479 is given in Fig. 11.2.
Notice the different scales of the axes! Fig. 11.3 is a zoom that shows all eigenvalues except
the two very large ones. Here the axes are equally scaled. From the two figures it becomes
clear that eigs has computed the eight eigenvalues (and corresponding eigenvectors) of
largest modulus.
To compute the eigenvalues smallest in modulus we issue the following command.
dsm=eigs(west0479,8,sm);
Iteration 1: a few Ritz values of the 20-by-20 matrix:
0
204 CHAPTER 11. RESTARTING ARNOLDI AND LANCZOS ALGORITHMS
200

150

100

50

50

100

150

200
200 150 100 50 0 50 100 150 200

Figure 11.3: A zoom to the center of the spectrum of matrix west0479 that excludes the
largest two eigenvalues on the imaginary axis

0
0
0
0
0
0
0
0

Iteration 2: a few Ritz values of the 20-by-20 matrix:

1.0e+03 *

-0.0228 - 0.0334i
0.0444
-0.0473
0.0116 + 0.0573i
0.0116 - 0.0573i
-0.0136 - 0.1752i
-0.0136 + 0.1752i
-3.4455
5.8308

Iteration 3: a few Ritz values of the 20-by-20 matrix:

1.0e+03 *
11.5. A NUMERICAL EXAMPLE 205
-0.0228 - 0.0334i
0.0444
-0.0473
0.0116 + 0.0573i
0.0116 - 0.0573i
-0.0136 + 0.1752i
-0.0136 - 0.1752i
-3.4455
5.8308

Iteration 4: a few Ritz values of the 20-by-20 matrix:

1.0e+03 *

-0.0228 + 0.0334i
0.0444
-0.0473
0.0116 - 0.0573i
0.0116 + 0.0573i
-0.0136 + 0.1752i
-0.0136 - 0.1752i
-3.4455
5.8308

Iteration 5: a few Ritz values of the 20-by-20 matrix:

1.0e+03 *

-0.0228 + 0.0334i
0.0444
-0.0473
0.0116 - 0.0573i
0.0116 + 0.0573i
-0.0136 + 0.1752i
-0.0136 - 0.1752i
-3.4455
5.8308

>> dsm

dsm =

0.0002
-0.0003
-0.0004 - 0.0057i
-0.0004 + 0.0057i
0.0034 - 0.0168i
0.0034 + 0.0168i
-0.0211
0.0225
206 CHAPTER 11. RESTARTING ARNOLDI AND LANCZOS ALGORITHMS
0.025

0.02

0.015

0.01

0.005

0.005

0.01

0.015

0.02

0.025
0.025 0.02 0.015 0.01 0.005 0 0.005 0.01 0.015 0.02 0.025

Figure 11.4: Smallest eigenvalues of the matrix west0479

>> 1./dsm

ans =

1.0e+03 *

5.8308
-3.4455
-0.0136 + 0.1752i
-0.0136 - 0.1752i
0.0116 + 0.0573i
0.0116 - 0.0573i
-0.0473
0.0444

The computed eigenvalues are depicted in Fig. 11.4

11.6 Another numerical example

We revisit the determination the acoustic eigenfrequencies and modes in the interior of a
car, see section 1.6.3. The computations are done with the finest grid depicted in Fig. 1.9.
We first compute the lowest ten eigenpairs with simultaneous inverse vector iteration
(sivit). The dimension of the search space is 15.

>> [p,e,t]=initmesh(auto);
>> [p,e,t]=refinemesh(auto,p,e,t);
>> [p,e,t]=refinemesh(auto,p,e,t);
>> p=jigglemesh(p,e,t);
>> [A,M]=assema(p,t,1,1,0);
>> whos
Name Size Bytes Class
11.6. ANOTHER NUMERICAL EXAMPLE 207
A 1095x1095 91540 double array (sparse)
M 1095x1095 91780 double array (sparse)
e 7x188 10528 double array
p 2x1095 17520 double array
t 4x2000 64000 double array

Grand total is 26052 elements using 275368 bytes

>> sigma=-.01;
>> p=10; tol=1e-6; X0=rand(size(A,1),15);
>> [V,L] = sivit(A,M,p,X0,sigma,tol);

||Res(0)|| = 0.998973
||Res(5)|| = 0.603809
||Res(10)|| = 0.0171238
||Res(15)|| = 0.00156298
||Res(20)|| = 3.69725e-05
||Res(25)|| = 7.11911e-07
>> % 25 x 15 = 375 matrix - vektor - multiplications until convergence
>>
>> format long, L

L =

0.00000000000000
0.01269007628847
0.04438457596824
0.05663501055565
0.11663116522140
0.13759210393200
0.14273438015546
0.20097619880776
0.27263682280769
0.29266080747831

>> format short

>> norm(V*M*V - eye(10))

ans =

1.8382e-15

Then we use Matlabs solver eigs. We set the tolerance and the shift to be the same
as with sivit. Notice that ARPACK applies a shift-and-invert spectral transformation if
a shift is given.

>> options.tol=tol; options.issym=1;

>> [v,l,flag]=eigs(A,M,p,sigma,options);
Iteration 1: a few Ritz values of the 20-by-20 matrix:
0
0
0
0
0
0
0
208 CHAPTER 11. RESTARTING ARNOLDI AND LANCZOS ALGORITHMS
0
0
0

Iteration 2: a few Ritz values of the 20-by-20 matrix:

3.3039
3.5381
4.7399
6.5473
6.7754
7.8970
15.0071
18.3876
44.0721
100.0000

Iteration 3: a few Ritz values of the 20-by-20 matrix:

3.3040
3.5381
4.7399
6.5473
6.7754
7.8970
15.0071
18.3876
44.0721
100.0000

>> flag

flag =

>> l=diag(l); l=l(end:-1:1); norm(l-L)

ans =

3.7671e-14

>> norm(v*M*v - eye(10))

ans = 8.0575e-15

Clearly the eigenvectors are mutually m-orthogonal. Notice that eigs returns the
eigenvalues sorted from large to small such that they have to be reordered before comparing
with those sivit computed.
In the next step we compute the largest eigenvalues of the matrix

(11.14) S = R(A M )1 RT ,

where RT R = M is the Cholesky factorization of M . The matrix in (11.14) is transferred

to eigs as a function.
>> type afun
11.6. ANOTHER NUMERICAL EXAMPLE 209
function x = afun(x)
global RA RB

x = RB*(RA\(RA\(RB*x)));

>> global RA RB
>> RA = chol(A-sigma*M);
>> RB = chol(M);
>> [v,l1,flag]=eigs(afun,n,10,lm,options);
Iteration 1: a few Ritz values of the 20-by-20 matrix:
0
0
0
0
0
0
0
0
0
0

Iteration 2: a few Ritz values of the 20-by-20 matrix:

3.3030
3.5380
4.7399
6.5473
6.7754
7.8970
15.0071
18.3876
44.0721
100.0000

Iteration 3: a few Ritz values of the 20-by-20 matrix:

3.3040
3.5381
4.7399
6.5473
6.7754
7.8970
15.0071
18.3876
44.0721
100.0000

>> flag

flag =

>> l1 = diag(l1)

l1 =
210 CHAPTER 11. RESTARTING ARNOLDI AND LANCZOS ALGORITHMS
100.0000
44.0721
18.3876
15.0071
7.8970
6.7754
6.5473
4.7399
3.5381
3.3040

>> sigma + 1./l1

ans =

0.0000
0.0127
0.0444
0.0566
0.1166
0.1376
0.1427
0.2010
0.2726
0.2927

>> norm(sigma + 1./l1 - l)

ans =

4.4047e-14

11.7 The Lanczos algorithm with thick restarts

The implicit restarting procedures discussed so far are very clever ways to get rid of
unwanted directions in the search space and still keeping a Lanczos or Arnoldi basis. The
latter admits to continue the iteration in a known framework. The Lanczos or Arnoldi
relations hold that admit very efficient checks for convergence. The restart has the effect
of altering the starting vector.
In this and the next section we discuss algorithms that work with Krylov spaces but
are not restricted to Krylov or Arnoldi bases. Before continuing we make a step back and
consider how we can determine if a given subspace of Fn is a Krylov space at all.
Let A be an n-by-n matrix and let v1 , . . . , vk be linearly independent n-vectors. Is
the subspace V := span{v1 , . . . , vk } a Krylov space, i.e., is there a vector q V such that
V = Kk (A, q)? The following theorem gives the answer [3, 10].

Theorem 11.1 V = span{v1 , . . . , vk } is a Krylov space if and only if there is a k-by-k

matrix M such that

(11.15) R := AV V M, V = [v1 , . . . , vk ],

has rank one and span{v1 , . . . , vk , R(R)} has dimension k + 1.

11.7. THE LANCZOS ALGORITHM WITH THICK RESTARTS 211
Proof. Let us first assume that V = Kk (A, q) for some q V. Let Q = [q1 , . . . , qk ] be the
Arnoldi basis of Kk (A, q). Then Q = V S with S a nonsingular k-by-k matrix. We now
multiply the Arnoldi relation

k+1 eTk ,
AQ = QH + q Q q
k+1 = 0, H Hessenberg.

by S 1 from the right to get

AV = V SHS 1 + q
k+1 ek S 1 .

which is (11.15) with M = SHS 1 .

Let us now assume that R in (11.15) has rank 1 so that we can write

(11.16) AV = V M + R = V M + vw , M Fkk .

with some v Fn and w Fk . Let S1 , S11 = S1 , be the Householder reflector that maps
w onto a multiple of ek , S1 w = ek . Then, (11.16) becomes

AV S1 = V S1 S1 M S1 + veTk .

There is another unitary matrix S2 with S2 ek = ek that transforms S1 M S1 similarly to

Hessenberg form,
S M S = H, H Hessenberg,
where S = S1 S2 . S2 can be formed as the product of Householder reflectors. In contrast
to the well-known transformation of full matrices to Hessenberg form, here the zeros are
generated row-wise starting with the last in order not to destroy ek in the rank-1 term.
Thus,
AV S = V SH + veTk .
So, V = Kk (A, q) with q = V Se1 .
We apply this theorem to the case where a subspace is spanned by some Ritz vectors.
Let A = A and let

(11.17) AQk Qk Tk = k+1 qk+1 eTk

be a Lanczos relation. Let

(k) (k)
Tk Sk = Sk k , Sk = [s1 , . . . , sk ], k = diag(1 , . . . , k ).

be the spectral decomposition of the tridiagonal matrix Tk . Then, for all i, the Ritz vector
(k)
yi = Qk si Kk (A, q)

gives rise to the residual

(k)
ri = Ayi yi i = k+1 qk+1 ek si Kk+1 (A, q) Kk (A, q).

Therefore, for any set of indices 1 i1 < < ij k we have

(k) (k) (k)
A[yi1 , yi2 , . . . , yij ] [yi1 , yi2 , . . . , yij ] diag(i1 , . . . , ij ) = k+1 qk+1 [si1 , si2 , . . . , sij ].

By Theorem 11.1 we see that any set [yi1 , yi2 , . . . , yij ] of Ritz vectors forms a Krylov
space. Note that the generating vector differs for each set.
212 CHAPTER 11. RESTARTING ARNOLDI AND LANCZOS ALGORITHMS
Algorithm 11.4 Thick restart Lanczos
1: Let us be given k Ritz vectors yi and a residual vector rk such that Ayi = i yi + i rk ,
i = 1, . . . , k. The value k may be zero in which case r0 is the initial guess.
This algorithm computes an orthonormal basis y1 , . . . , yj , qj+1 , . . . , qm that spans a
m-dimensional Krylov space whose generating vector is not known unless k = 0.
2: qk+1 := rk /krk k.
3: z := Aqk+1 ;
4: k+1 := qk+1 z;
Pk
5: rk+1 = z k+1 qk+1 i=1 i yi
6: k+1 := krk+1 k
7: for i = k + 2, . . . , m do
8: qi := ri1 /i1 .
9: z := Aqi ;
10: i := qi z;
11: ri = z i qi i1 qi1
12: i = kri k
13: end for

We now split the indices 1, . . . , k in two sets. The first set contains the good Ritz
vectors that we want to keep and that we collect in Y1 , the second set contains the bad
ones that we want to remove. Those we put in Y2 . In this way we get
 
1
(11.18) A[Y1 , Y2 ] [Y1 , Y2 ] = k+1 qk+1 [s1 , s2 ].
2
Keeping the first set of Ritz vectors and purging (deflating) the rest yields
AY1 Y1 1 = k+1 qk+1 s1 .
We now can restart a Lanczos procedure by orthogonalizing Aqk+1 against Y1 =: [y1 , . . . , yj ]
and qk+1 . From the equation
(k)
Ayi yi i = qk+1 i , i = k+1 ek si
we get
qk+1 Ay = ,
whence
j
X
(11.19) rk+1 = Aqk+1 k+1 qk+1 i yi Kk+1 (A, q.)
i=1
From this point on the Lanczos algorithm proceeds with the ordinary three-term recur-
rence. We finally arrive at a relation similar to (11.17), however, with
Qm = [y1 , . . . , yj , qk+1 , . . . , qm+kj ]
and
1 1
.. ..
. .

j j

Tm = ..
1 j k+1 .

.. ..
. . m+kj1
m+kj1 m+kj
11.8. KRYLOVSCHUR ALGORITHM 213
This procedure, called thick restart, has been suggested by Wu & Simon [11], see Al-
gorithm 11.4. It allows to restart with any number of Ritz vectors. In contrast to the
implicitly restarted Lanczos procedure, here we need the spectral decomposition of Tm .
Its computation is not an essential overhead in general. The spectral decomposition ad-
mits a simple sorting of Ritz values. We could further split the first set of Ritz pairs into
converged and unconveregd ones, depending on the value m+1 |sk,i |. If this quantity is
below a given threshold we set the value to zero and lock (deflate) the corresponding Ritz
vector, i.e., accept it as an eigenvector.
The procedure is mathematically equivalent with the implicitely restarted Lanczos al-
gorithm. In fact, the generating vector of the Krylov space span{y1 , . . . , yj , qj+1 , . . . , qm }
that we do not compute is q1 = (A j+1 I) (A m I)q1 . This restarting procedure
is probably simpler than with IRL.
The problem of losing orthogonality is similar to plain Lanczos. Wu & Simon [11] in-
vestigate the various reorthogonalizing strategies known from plain Lanczos (full, selective,
partial). In their numerical experiments the simplest procedure, full reorthogonalization,
performs similarly or even faster than the more sophisticated reorthogonalization proce-
dures.
Remark 11.2. The thick restart Lanczos procedure does not need a Krylov basis of
span{y1 , . . . , yj } or, equivalently, the tridiagonalization of

1 1
.. ..
. .
.
j j
1 j k+1

However, at the next restart, the computation of the spectral decomposition will most
probably require it.
Question: How can the arrow matrix above be tridiagonalized economically?

11.8 KrylovSchur algorithm

The KrylovSchur algorithm introduced by Stewart [9] is a generalization of the thick-
restart procedure for non-Hermitian problems. The Arnoldi algorithm constructs the
Arnoldi relation

(11.1) AQm = Qm Hm + rm em ,
be a Schur
where Hm is Hessenberg and [Qm , rm ] has full rank. Let Hm = Sm Tm Sm
decomposition of Hm with unitary Sm and triangular Tm . Then, similarly as in the
previous section we have

(11.20) AYm = Ym Tm + rm s , Y m = Qm S m , s = em Sm .

The upper trangular form of Tm eases the analysis of the individual Ritz pairs. In par-
ticular, it admits moving unwanted Ritz values to the lower-right corner of Tm . (See the
subroutine trexc in LAPACK for details.) Similarly as in (11.18) we collect the good
and bad Ritz vectors in matrices Y1 and Y2 , respectively. In this way we get
 
T T12
(11.21) A[Y1 , Y2 ] [Y1 , Y2 ] 11 = k+1 qk+1 [s1 , s2 ].
T22
214 CHAPTER 11. RESTARTING ARNOLDI AND LANCZOS ALGORITHMS
Keeping the first set of Ritz vectors and purging the rest yields

AY1 Y1 T11 = k+1 qk+1 s1 .

In the thick-restart Lanczos procedure we have found an eigenpair as soon as k+1 |sik | is
sufficiently small. The determination of a converged subspace with the general Krylov
Schur procedure is not so easy. However, if we manage to bring s1 into the form
   
s1 0
s1 = =
s1 s1

then we found an invariant subspace.



T11 T12
A[Y1 , Y1 ] [Y1 , Y1 ] = k+1 qk+1 [0T , s1 ]
T22

i.e.,
AY1 = Y1 T11

In most cases s1 consists of a single small element or of two small elements in the case of
a complex-conjugate eigenpair of a real nonsymmetric matrix [9]. These small elements
are then declared zero and the columns in Y1 are locked, i.e., they are not altered any-
more in the future computations. Orthogonality against them has to be enforced in the
continuation of the eigenvalue computation though.

11.9 The rational Krylov space method

After having computed a number of eigenvalueeigen/Schurvector pairs in the neighbor-
hood of some shift 1 with the shift-invert Lanczos, Arnoldi, or KrylovSchur algorithm
it may be advisable to restart with a changed shift 2 . This is in fact possible with-
out discarding the available Krylov space [6]. In this section we consider the generalized
eigenvalue problem Ax = Bx.
The rational Krylov space method starts out as a shift-invert Arnoldi iteration with
shift 1 and starting vector v1 . It computes an orthonormal basis Vj using the basic
recurrence,

(11.22) j.
(A 1 B)1 BQj = Qj Hj + rj eT = Qj+1 H

or, using the Schur decomposition of Hj , cf. (11.20),

 
1 Tj
(11.23) (A 1 B) BYj = Yj Tj + rj s = Yj+1 , Yj+1 = [Yj , rj ]
s

We want to derive a KrylovSchur relation for a new shift 2 6= 1 from (11.23) for the
same space R(Yj+1 ) without accessing the matrices A or B. The tricky thing is to avoid
discard all the information gathered in the basis Yj+1 that was computed with the old
shift 1 . This is indeed possible if we replace the basis Yj+1 with a new basis Wj+1 , which
spans the same subspace as Yj+1 but can be interpreted as the orthonormal basis of a
KrylovSchur relation with the new shift 2 .
We rewrite the relation (11.23) as
   
Ij Tj
BYj = BYj+1 = (A 1 B)Y j+1 .
0 s
BIBLIOGRAPHY 215
Introducing the shift 2 this becomes
     
Ij Tj Tj
(11.24) BYj+1 + (1 2 ) = (A 2 B)Yj+1 .
0 s s

To construct a KrylovSchur relation we must get rid of the last non-zero row of the matrix
in braces in (11.24). To that end we use the QR factorization
     
Ij Tj Rj
+ (1 2 ) = Qj+1 .
0T s 0T

Using it we obtain
     
Rj Rj Tj
BYj+1 Qj+1 BW j+1 = BW R
j j = (A 2 B)W Q
j+1 j+1
0T 0T s

Multiplying with (A 2 B)1 from the left we obtain

   
Tj Rj1 Mj
(11.25) (A 2 B)1 BWj = Wj+1 Qj+1 = W j+1
s t
or

(11.26) (A 2 B)1 BWj = Wj Mj + wj+1 t .

This equation can easily been transformed into an Arnoldi or KrylovSchur relation.
All these transformations can be executed without performing any operations on the
large sparse matrices A and B.
In a practical implementation, the mentioned procedure is combined with locking,
purging, and implicit restart. First run shifted and inverted Arnoldi with the first shift
1 . When an appropriate number of eigenvalues around 1 have converged, lock these
converged eigenvalues and purge those that are altogether outside the interesting region,
leaving an Arnoldi (11.22) or KrylovSchur recursion (11.22) for the remaining vectors.
Then introduce the new shift 2 and perform the steps above to get a new basis Wj+1
that replaces Vj+1 . Start at the new shift by operating on the last vector of this new basis

r := (A 2 B)1 Bwj+1

and get the next basis vector wj+2 in the Arnoldi recurrence with the new shift 2 . Con-
tinue until we get convergence for a set of eigenvalues around 2 , and repeat the same
procedure with new shifts until either all interesting eigenvalues have converged or all the
shifts in the prescribed frequency range have been used.

Bibliography
. Bjo
[1] A rck, Numerics of GramSchmidt orthogonalization, Linear Algebra Appl.,
197/198 (1994), pp. 297316.

[2] D. Calvetti, L. Reichel, and D. C. Sorensen, An implicitely restarted Lanczos

method for large symmetric eigenvalue problems, Electron. Trans. Numer. Anal., 2
(1994), pp. 121.

[3] M. Genseberger and G. L. G. Sleijpen, Alternative correction equations in the

JacobiDavidson method, Numer. Linear Algebra Appl., 6 (1999), pp. 235253.
216 CHAPTER 11. RESTARTING ARNOLDI AND LANCZOS ALGORITHMS
[4] R. B. Lehoucq, D. C. Sorensen, and C. Yang, ARPACK Users Guide: Solution
of Large-Scale Eigenvalue Problems by Implicitely Restarted Arnoldi Methods, SIAM,
Philadelphia, PA, 1998. (The software and this manual are available at URL http:
//www.caam.rice.edu/software/ARPACK/).

[5] The Matrix Market. A repository of test data for use in comparative studies of
algorithms for numerical linear algebra. Available at URL http://math.nist.gov/
MatrixMarket/.

[6] A. Ruhe, Rational Krylov subspace method, in Templates for the solution of Algebraic
Eigenvalue Problems: A Practical Guide, Z. Bai, J. Demmel, J. Dongarra, A. Ruhe,
and H. van der Vorst, eds., SIAM, Philadelphia, PA, 2000, pp. 246249.

[7] D. C. Sorensen, Implicit application of polynomial filters in a k-step Arnoldi

method, SIAM J. Matrix Anal. Appl., 13 (1992), pp. 357385.

[8] , Implicitly restarted Arnoldi/Lanczos methods for large scale eigenvalue calcu-
lations, in Parallel Numerical Algorithms, D. E. Keyes, A. Sameh, and V. Venkatakr-
ishnan, eds., Kluwer, Dordrecht, 1997, pp. 119165. (ICASE/LaRC Interdisciplinary
Series in Science and Engineering, 4).

[9] G. W. Stewart, A KrylovSchur algorithm for large eigenproblems, SIAM J. Matrix

Anal. Appl., 23 (2001), pp. 601614.

[10] , Matrix Algorithms II: Eigensystems, SIAM, Philadelphia, PA, 2001.

[11] K. Wu and H. D. Simon, Thick-restart Lanczos method for large symmetric eigen-
value problems, SIAM J. Matrix Anal. Appl., 22 (2000), pp. 602616.
Chapter 12

The Jacobi-Davidson Method

The Lanczos and Arnoldi methods are very effective to compute extremal eigenvalues
provided these are well separated from the rest of the spectrum. Lanczos and Arnoldi
methods combined with a shift-and-invert spectral transformation are also efficient to
compute eigenvalues in the vicinity of the shift . In this case it is necessary to solve a
system of equation

(A I)x = y, or (A M )x = y,

respectively, in each iteration step. These systems have to be solved very accurately since
otherwise the Lanczos or Arnoldi relation does not hold anymore. In most cases the matrix
A I (or A M ) is LU or Cholesky factored. The JacobiDavidson (JD) algorithm is
particularly attractive if this factorization is not feasible [11].

12.1 The Davidson algorithm

Let v1 , . . . , vm be a set of orthonormal vectors, spanning the search space R(Vm ) with
Vm = [v1 , . . . , vm ]. In the Galerkin approach we are looking for vectors s Fm such that
the Galerkin condition holds,

(12.1) AVm s Vm s v1 , . . . , vm .

This immediately leads to the (small) eigenvalue problem

(12.2) Vm AVm s = Vm Vm s

(m) (m) (m) (m)

with solutions (j , sj ), j = 1, . . . , m. j is called a Ritz value and Vm sj is called
a Ritz vector. In the sequel we omit the superscript m for readability. The dimension of
the search space should become evident from the context.
Let us consider, say, the Ritz value j , its Ritz vector uj = Vm sj and their residual
rj = Auj j uj . Often we are looking for the largest or smallest eigenvalue of A in which
case j = 1 or j = m, respectively. The question immediately arises how we can improve
(j , uj ) if krj k is still too large. It is straightforward to try to find a better approximate
eigenpair by expanding the search space. Davidson, in his original paper [2], suggested to
compute a vector t from

(12.3) (DA j I)t = rj ,

217
218 CHAPTER 12. THE JACOBI-DAVIDSON METHOD
where DA is the diagonal of the matrix A. The vector t is then made orthogonal to the
basis vectors v1 , . . . , vm . The resulting vector, after normalization, is chosen as vm+1 by
which R(Vm ) is expanded, i.e., Vm+1 = [v1 , . . . , vm , vm+1 ].
This method is successful in finding dominant eigenvalues of (strongly) diagonally dom-
inant matrices. The matrix DA j I has therefore often been viewed as a preconditioner
for the matrix A j I. A number of investigations were made with more sophisticated
preconditioners M j I, see e.g. [7, 8]. They lead to the conclusion that M j I should
not be too close to A j I which contradicts the notion of a preconditioner as being an
easily invertible (factorizable) approximation of A j I.

12.2 The Jacobi orthogonal component correction

In his seminal paper, Jacobi [6] not only presented the solution of symmetric eigenvalue
problems by successive application of (later to be called) Jacobi rotations, but also pre-
sented an approach to improve an approximate eigenpair with an iterative procedure.
Here, we give Jacobis approach in a generalized form presented by Sleijpen and van der
Vorst [11]. Let uj be an approximation to the eigenvector x of A corresponding to the
eigenvalue . Jacobi proposed to correct uj by a vector t, uj t, such that

(12.4) A(uj + t) = (uj + t), uj t.

This is called the Jacobi orthogonal component correction (JOCC) by Sleijpen &
van der Vorst [11]. As t uj we may split equation (12.4) in the part parallel to uj and
in the part orthogonal to uj . If kuj k = 1 then the part parallel to uj is

(12.5) uj uj A(uj + t) = uj uj (uj + t)

which simplifies to the scalar equation

(12.6) j + uj At = .

Here j is the Rayleigh quotient of uj , j = (uj ). The part orthogonal to uj is

(12.7) (I uj uj )A(uj + t) = (I uj uj )(uj + t)

which is equivalent to

(I uj uj )(A I)t = (I uj uj )(Auj + uj )

= (I uj uj )Auj = (A j I)uj =: rj .

As (I uj uj )t = t we can rewrite this equation in symmetrized form as

(12.8) (I uj uj )(A I)(I uj uj )t = rj .

If A is symmetric then the matrix in (12.8) is symmetric as well.

Unfortunately, we do not know ! Therefore, we replace by j to get the Jacobi
Davidson correction equation

(12.9) (I uj uj )(A j I)(I uj uj )t = rj = (A j I)uj , t uj .

As rj uj (in fact rj Vm ) this equation is consistent if A j I is nonsingular.

12.2. THE JACOBI ORTHOGONAL COMPONENT CORRECTION 219
The correction equation (12.9) is, in general, solved iteratively by the GMRES or
MINRES algorithm [1]. Often, only little accuracy in the solution is required.
Once t is (approximately) known we set

(12.10) uj+1 = uj + t.

From (12.6) we may then obtain

(12.11) j+1 = j + uj At.

If A is symmetric j+1 may be set equal to the Rayleigh quotient (uj+1 ).

Let us analyze (12.9) more closely. Let us first investigate the role of the orthogonality
condition t uj . If this condition is omitted then the equation to be solved is

(12.12) (I uj uj )(A j I)t = rj = (A j I)uj .

This equation has the solution t = uj . Therefore, without the condition t uj there is
no progress in solving the eigenvalue problem Ax = x.
One can argue that this is the approach suggested by Davidson [2]. Davidson approx-
imated A on the left side of (12.12) by an approximation of it, typically the diagonal,
say DA , of A. As his matrices were diagonally dominant, he solved a reasonably good
approximation of (12.12). If DA in (12.3) is considered a preconditioner of A then any
matrix closer to A should lead to better performance of the algorithm. In extremis, A
should be a possible choice for the matrix on the left. But we have just seen that this
leads to a situation without progress. In fact the progess in the iteration deteriorates the
better the preconditioner approximates the system matrix. In consequence, DA in (12.3)
must not be considered a preconditioner.
Let us now investigate what happens if the correction equation is solved exactly. To
that end we write it as

(I uj uj )(A j I)t = rj , t uj ,

which immediately leads to

(A j I)t uj uj (A j I)t = rj ,
| {z }
F
or,
(A j I)t = uj rj .
Assuming that j is not an eigenvalue of A we get

t = (A j I)1 uj (A j I)1 rj .

The constraint uj t = 0 allows us to determine the free variable ,

0 = uj (A j I)1 uj uj (A j I)1 rj ,

whence
uj (A j I)1 rj
= .
uj (A j I)1 uj
220 CHAPTER 12. THE JACOBI-DAVIDSON METHOD
By (12.10), the next approximate is then
(12.13) uj+1 = uj + t = uj + (A j I)1 uj (A j I)1 rj = (A j I)1 uj
| {z }
uj
which is a step of Rayleigh quotient iteration! This implies a fast (quadratic in general,
cubic in the Hermitian case) convergence rate of this algorithm.
In general the correction equation
(12.14) = (I uj uj )(A j I)(I uj uj )t = rj ,
At t uj ,
is solved iteratively with a Krylov space solver like GMRES or MINRES [1]. To get
a decent performance a preconditioner is needed. Sleijpen and van der Vorst suggest
preconditioners of the form
(12.15) = (I uj u )K(I uj u ),
K K A j I.
j j

We assume that K is (easily) invertible, i.e., that it is computationaly much cheaper to

solve a system of equation with K than with A. With this assumption the system of
equation
= v,
Kz z uj ,
i.e., provided that
can be solved provided that the right-hand side v is in the range of K,
+
v uj . We formally denote the solution by z = K v. So, instead of (12.14) we solve the
equation
(12.16) + At
K = K
+ rj , t uj .
Let t0 = 0 be the initial approximation to the solution of (12.16). (Notice that t0 trivially
satisfies the orthogonality constraint.) Because of the projectors I uj uj in the definitions
of A and K all approximations are orthogonal to uj .
In each iteration step we have to compute
+ Av,
z=K z uj
where v uj . To do this we proceed as follows. First we write
= (I uj uj )(A j I)v =: y.
Av
| {z }
y
Then,
= y,
Kz z uj .
With (12.15) this becomes
(I uj uj )Kz = Kz uj uj Kz = y,
the solution of which is
z = K 1 y K 1 uj ,
where, formally, = uj Kz. Similarly as earlier, we determine the scalar by means of
the constraint z uj = 0. Thus
uj K 1 y
= 1 .
uj K uj
Remark 12.1. Since uj is fixed during the solution of the secular equation, the vector
K 1 uj has to be computed just once. Thus, if the iterative solver needs k steps until
convergence, k + 1 systems of equations have to be solved with the matrix K.
12.2. THE JACOBI ORTHOGONAL COMPONENT CORRECTION 221
Algorithm 12.1 The JacobiDavidson algorithm to compute the eigenvalue of
A closest to a target value
1: Let A, B Fnn . This algorithm computes the eigenvalue of A that is closest to .
Let t be an initial vector. Set V0 = [], V0A = [], m = 0.
2: loop
3: for i = 1, . . . , m 1 do
4: t := t (vi t)vi ;
/* t = (I Vm1 Vm1 )t */
5: end for
6: vm := t/ktk; vm A := Av ; A , vA ];
Vm := [Vm1 , vm ]; VmA := [Vm1
m m
7: for i = 1, . . . , m do
8: Mi,m := vi vm A; vA ;
Mm,i := vm i /* M = Vm AVm */
9: end for
10: Mm,m := vm vA ;
m
11: Compute the eigenvalue of M closest to and the /* Rayleigh Ritz step */
corresponding eigenvector s: M s = s; ksk = 1;
12: u := Vm s; uA := VmA s; r := uA u;
13: if krk < tol then
14: return ( = , x = u)
15: end if
16: (Approximatively) solve the correction equation for t,
(I uu )(A j I)(I uu )t = r, t u;
17: end loop

12.2.1 Restarts
Evidently, in Algorithm 12.1, the dimension m of the search space can get large. To limit
memory consumption, we limit m such that m mmax . As soon as m = mmax we restart:
Vm = Vmmax is replaced by the q Ritz vectors corresponding to the Ritz values closest to
the target . Notice that the Schur decomposition of M = Mm,m = Vm AVm is computed
already in step 11 of the Algorithm. Let M = S T S be this Schur decomposition with
|t11 | |t22 | . Then we set Vq = Vm S:,1:q , VqA = VmA S:,1:q , M = T S1:q,1:q .
Notice that the restart is easy because the JacobiDavidson algorithm is not a Krylov
space method.

12.2.2 The computation of several eigenvalues

Let x
1 , x
2 , . . . , x i x
k be already computed eigenvectors or Schur vectors with x j = ij ,
1 i, j k. Then

(12.17) AQk = Qk Tk , Qk = [
x1 , . . . , x
k ].

is a partial Schur decomposition of A [13]. We want to extend the partial Schur de-
composition by one vector employing the JacobiDavidson algorithm. Since Schur vectors
are mutually orthogonal we can apply the JacobiDavidson algorithm in the orthogonal
complement of R(Qk ), i.e., we apply the JacobiDavidson algorithm to the matrix

(12.18) (I Qk Qk )A(I Qk Qk ), Qk = [
x1 , . . . , x
k ].

The correction equation gets the form

(12.19) (I uj uj )(I Qk Qk )(A j I)(I Qk Qk )(I uj uj )t = rj , t uj , t Q k .

222 CHAPTER 12. THE JACOBI-DAVIDSON METHOD
As uj Qk we have
kQ
(I uj uj )(I Qk Qk ) = I Q , k = [
Q x1 , . . . , x
k , uj ].
k

Thus, we can write (12.19) in the form

(12.20) kQ
(I Q kQ
)(A j I)(I Q )t = rj , t = 0.
Q
k k k

The preconditioner becomes

(12.21) = (I Q
K kQ kQ
)K(I Q ), K A j I.
k k

Similarly as earlier, for solving

= Av,
Kz z = Q
Q v = 0,
k k

we execute the following steps. Since

= (I Q
Av kQ kQ
) (A j I)v =: (I Q k Q
)y =: y Q y .
k k k
| {z } |{z}
y a

we have to solve
= (I Q
Kz kQ
)Kz = (I Q
kQ
)y, k.
zQ
k k

Thus,
k a.
z = K 1 y K 1 Q
z = 0,
Similarly as earlier, we determine a by means of the constraint Q k

a = (Q k )1 Q
K 1 Q K 1 y.
k k

If the iteration has converged to the vector x

k+1 we can extend the partial Schur decom-
position (12.17). Setting
Qk+1 := [Qk , x
k+1 ],
we get

(12.22) AQk+1 = Qk+1 Tk+1

with  
Tk Qk Axk+1
Tk+1 = .
0 xk+1 A
xk+1

12.2.3 Spectral shifts

In the correction equation (12.9) and implicitely in the preconditioner (12.15) a spectral
shift j appears. Experiments show that it is not wise to always choose the Rayleigh
quotient of the recent eigenvector approximation u as the shift. In particular, far away
from convergence, i.e., in the first few iteration steps, the Rayleigh quotient may be far
away from the (desired) eigenvalue, and in fact may direct the JD iteration to an un-
wanted solution. So, one proceeds similarly as in the plain Rayleigh quotient iteration,
cf. Remark 7.6 on page 143. Initially, the shift is held fixed, usually equal to the target
value . As soon as the norm of the residual is small enough, the Rayleigh quotient of the
actual approximate is chosen as the spectral shift in the correction equation. For efficiency
12.2. THE JACOBI ORTHOGONAL COMPONENT CORRECTION 223
Algorithm 12.2 The JacobiDavidson QR algorithm to compute p of the eigen-
values closest to a target value
1: Q0 := []; k = 0. /* Initializations */
2: Choose v1 with kv1 k = 1.
3: w1 = Av1 ; H1 := v1 w1 ; V1 := [v1 ]; W1 := [W1 ];
= v1 ; = v1 w1 ; r := w1
4: q q.
5: j := 1;
6: while k < p do
7: /* Compute Schur vectors one after the other */
8: Approximatively solve the correction equation for t
kQ
(I Q
)(A I)(I kQ
Q )t = rj , t = 0.
Q
k k k

where Q k = [Qk , q ].
vj = (I Vj1 Vj1 )t/k(I V
9: j1 Vj1 )tk; Vj := [Vj1 , vj ].


Hj1 Vj1 wj
10: wj = Avj ; Hj = ; Wj = [Wj1 , wj ].
vj Wj1 vj wj
11: Compute the Schur decomposition of
Hj =: Sj Rj Sj
(j)
with the eigenvalues rii sorted according to their distance to .
12: /* Test for convergence */
13: repeat
(j)
14: = 1 ; q = Vj s1 ; w = Wj s1 ; r = w q
15: found := krk <
16: if found then
17: Qk+1 = [Qk , q ]; k := k + 1;
18: end if
19: until not found
20: /* Restart */
21: if j = jmax then
22: Vjmin := Vj [s1 , . . . , smin ]; Tjmin := Tj (1 : jmin , 1 : jmin );
23: Hjmin := Tjmin ; Sjmin := Ijmin ; J := jmin
24: end if
25: end while

reasons, the spectral shift in the preconditioner K is always fixed. In this way it has to
be computed just once. Notice that K is changing with each correction equation.

Remark 12.2. As long as the shift is held fixed JacobiDavidson is actually performing a
shift-and-invert Arnoldi iteration.

Algorithm 12.2 gives the framework for an algorithm to compute the partial Schur
decomposition of a matrix A. Qk stores the converged Schur vectors; Vj stores the active
search space. This algorithm does not take into account some of the just mentioned
issues. In particular the shift is always taken to be the Rayleigh quotient of the most
recent approximate q.
224 CHAPTER 12. THE JACOBI-DAVIDSON METHOD
12.3 The generalized Hermitian eigenvalue problem
We consider the problem

(12.23) Ax = M x,

with A and M nn Hermitian, and M additionally positive definite. Then the eigenvectors
can be chosen mutually M -orthogonal,

(12.24) xi M xj = ij , Axi = i M xi , 1 i, j n,

where ij denotes the Kronecker delta function. Then it makes sense in the Jacobi
Davidson (as in other algorithms) to keep the iterates M -orthogonal.
Let Vm = [v1 , . . . , vm ] be an M -orthogonal basis of the search space Vm . Then the
Galerkin condition

(12.25) AVm s M Vm s v1 , . . . , vm ,

leads to the eigenvalue problem

(12.26) Vm AVm s = Vm M Vm s = s.

u
Let (, = Vm
s) be a solution of (12.26). Then the correction t to u
must be M -
orthogonal,

(12.27) t M u
=0 (I u
u M )t = t.

The correction equation in turn becomes

(12.28) (I M u
u )(I u
)(A M u M )t = (I u
u M )
r, =
r, t M u
,

where
r = A u
u M . Preconditioners for the secular equation are chosen of the form

(12.29) = (I M u
K u )K(I u
u M ),

where K A M and is the target value.

12.4 A numerical example

We give a demonstration on how a full-fledged JacobiDavidson algorithm works. The
code is a Matlab implementation of a program from the PhD thesis of Geus [4]. It solves
the generalized symmetric eigenvalue problem as discussed in the previous section. The
command help jdsym provides the output given on page 225.
As the numerical example we again consider the accustic behavour in the interior
of a car. We compute the five smallest eigenvalues and associated eigenvectors. The
q
preconditioner is chosen to be the diagonal of A. An eigenpair (, is declared converged
if the residual norm kA
q M q 8
k < 10 k qk. Most of the components of options are
explained in the help text. The residual norms for each iteration step are plotted in
Fig. 12.1. As soon as an eigenpair has converged a new iteration starts. The residual
norm then increases by several orders of magnitude.
12.4. A NUMERICAL EXAMPLE 225
[Q, lambda, it] = jdsym(n, A, B, K, kmax, tau, options)

jdsym is a MATLAB implementation of the JDQR algorithm for symmetric

matrices.

jdsym returns kmax eigenvalues with corresponding eigenvectors of

the matrix A near the target tau. K is a symmetric preconditioner
for A - tau * B.

The arguments A and B both contain either n-by-n symmetric matrices

or a string containing the name of an M-file which applies a
symmetric linear operator to the columns of a given
matrix. Matrix B must be positive definite.

To solve the specialized eigenvalue problem A * x = lambda * x pass

an empty matrix [] for parameter B. If no preconditioner is used
pass an empty matrix [] for parameter K.

The options structure specifies certain parameters in the algorithm:

options.tol convergence tolerance 1e-10

options.jmax maximal dimension of search subspace V 2*kmax
options.jmin dimension of search subspace V after restart kmax
options.maxit maximum number of outer iterations max(100,2*n/jmax)
options.clvl verbosity of output (0 means no output) 1
options.eps_tr tracing parameter as described in literature 1e-4
options.toldecay convergence tolerance for inner iteration is 2
toldecay ^ (-solvestep)
options.cgmaxit maximum number of iterations in linear solver 100
options.V0 initial search subspace rand(n,1)-.5
V0 will be orthonormalized by jdsym
options.linsolv solver used for corrections equation 1
1 -- CGS
2 -- SYMMLQ
3 -- CGS_OP
4 -- CGS mit SYMOP
5 -- MINRES
6 -- QMR
7 -- QMRS
options.strategy strategy to avoid computation of zero
eigenvalues:
0 -- standard JD algorithm 0
1 -- never choose Ritz values that are close
to zero as best current approximation.
Purge Ritz values that are close
to zero when restarting
2 -- dynamically adjust tau
3 -- see (1) and set tau to last converged
eigenvalue if it was bigger than the old
tau
4 -- set tau to last converged eigenvalue if
it was bigger than the old tau

The converged eigenvalues and eigenvectors are stored in Q and lambda. The
number of outer JD iterations performed is returned in it.
226 CHAPTER 12. THE JACOBI-DAVIDSON METHOD
>> K=diag(diag(A));
>> options

options =
linsolv: 6
strategy: 0

>> options.tol=1e-8

options =

tol: 1.0000e-08
jmax: 20
jmin: 10
clvl: 1
optype: 1
linsolv: 5

>> [Q, lambda, it] = jdsym(n, A, M, K, 5, -0.01, options);

JDSYM Solving A*x = lambda*M*x with preconditioning

N= 1095 ITMAX=1.095000e+02
KMAX= 5 JMIN= 10 JMAX= 20 V0DIM= 1
TAU= -1.0000e-02 JDTOL= 1.0000e-08 STRATEGY= 0
LINSOLVER= MINRES OPTYPE= SYM
LINITMAX= 100 EPS_TR= 1.000e-04 TOLDECAY= 2.00e+00

IT K J RES CGTHET CGTOL CGIT CGERR CGFLG Ritz values 1-5

-----------------------------------------------------------------------------------
0 0 1 4.26e+00
1 0 2 9.33e-01 -1.00e-02 2.50e-01 1 9.74e-01 0
2 0 3 7.13e-02 -1.00e-02 1.25e-01 4 6.95e-02 0
3 0 4 4.14e-03 -1.00e-02 6.25e-02 10 4.04e-03 0
4 0 5 2.01e-04 -1.00e-02 3.12e-02 33 1.22e-04 0
5 0 6 4.79e-05 -1.00e-02 1.56e-02 71 3.07e-06 0
6 0 7 3.66e-07 9.33e-08 7.81e-03 88 3.53e-07 0
7 0 8 1.70e-09 6.39e-12 3.91e-03 74 1.34e-09 0
7 1 7 5.94e-03
8 1 8 4.98e-03 -1.00e-02 5.00e-01 4 2.67e-03 0
9 1 9 2.53e-03 -1.00e-02 2.50e-01 11 1.19e-03 0
10 1 10 3.38e-04 -1.00e-02 1.25e-01 18 3.06e-04 0
11 1 11 4.76e-05 -1.00e-02 6.25e-02 27 2.05e-05 0
12 1 12 1.45e-06 1.27e-02 3.12e-02 26 1.48e-06 0
13 1 13 1.87e-08 1.27e-02 1.56e-02 38 2.22e-08 0
14 1 14 9.87e-11 1.27e-02 7.81e-03 60 1.38e-10 0
14 2 13 4.75e-03
15 2 14 3.58e-03 -1.00e-02 5.00e-01 5 2.17e-03 0
16 2 15 1.16e-03 -1.00e-02 2.50e-01 9 8.93e-04 0
17 2 16 1.59e-04 -1.00e-02 1.25e-01 10 1.24e-04 0
18 2 17 1.46e-05 -1.00e-02 6.25e-02 14 8.84e-06 0
19 2 18 4.41e-07 4.44e-02 3.12e-02 21 4.29e-07 0
20 2 19 7.01e-09 4.44e-02 1.56e-02 29 6.58e-09 0
20 3 18 4.82e-03
21 3 19 3.44e-03 -1.00e-02 5.00e-01 3 2.34e-03 0
22 3 20 8.25e-04 -1.00e-02 2.50e-01 7 7.08e-04 0
12.4. A NUMERICAL EXAMPLE 227
23 3 11 1.57e-04 -1.00e-02 1.25e-01 11 8.91e-05 0
24 3 12 1.65e-05 -1.00e-02 6.25e-02 14 9.77e-06 0
25 3 13 4.77e-07 5.66e-02 3.12e-02 31 4.68e-07 0
26 3 14 6.51e-09 5.66e-02 1.56e-02 32 7.26e-09 0
26 4 13 1.28e-02
27 4 14 1.14e-02 -1.00e-02 5.00e-01 3 6.30e-03 0
28 4 15 3.54e-03 -1.00e-02 2.50e-01 6 2.45e-03 0
29 4 16 8.00e-04 -1.00e-02 1.25e-01 10 4.19e-04 0
30 4 17 1.13e-04 -1.00e-02 6.25e-02 12 4.95e-05 0
31 4 18 1.67e-05 -1.00e-02 3.12e-02 16 3.22e-06 0
32 4 19 4.23e-07 1.17e-01 1.56e-02 21 2.49e-07 0
33 4 20 3.20e-09 1.17e-01 7.81e-03 45 3.21e-09 0

JDSYM

IT_OUTER=33 IT_INNER_TOT=764 IT_INNER_AVG= 23.15

Converged eigensolutions in order of convergence:

I LAMBDA(I) RES(I)
---------------------------------------
1 9.102733263227557e-16 1.70111e-09
2 1.269007628846320e-02 9.86670e-11
3 4.438457596823515e-02 7.01153e-09
4 5.663501055565738e-02 6.50940e-09
5 1.166311652214006e-01 3.19504e-09
>>

0
10

2
10

4
10

6
10

8
10

10
10

0 5 10 15 20 25 30 35

Figure 12.1: JacobiDavidson convergence history

228 CHAPTER 12. THE JACOBI-DAVIDSON METHOD
12.5 The JacobiDavidson algorithm for interior eigenval-
ues
Interior eigenvalues are eigenvalues that do not lie at the border of the convex hull of the
spectrum, cf. Fig. 12.2

Figure 12.2: View of a spectrum (A) in the complex plane. The eigenvalues in the red
circle are to be computed

The success of the JacobiDavidson algorithm depends heavily on the quality of the
actual Ritz pair (j , q
). However, the RayleighRitz procedure can lead to problem if it is
applied to interior eigenvalues. The following simple numerical example shall demonstrate
the problem. Let
0 0 0 1 0
A= 0 1 0 , U = 0 0.5 .
0 0 1 0 0.5
Then, 

0 0
U AU = U U = I2 .
0 0
So, any linear combination of the columns of U is a Ritz vector corresponding to the Ritz
value 0, e.g.,
  0.5
0.5
U = 0.5 .
0.5
0.5
Thus, although the basis contains the correct eigenvalue associated with the eigenvalue 0,
the RayleighRitz procedure fails to find it and, instead, returns a very bad eigenvector
approximation.
This example may look contrived. So, we conduct a Matlab experiment with the
same A but with a randomly perturbed U .
12.6. HARMONIC RITZ VALUES AND VECTORS 229
>> rand(state,0)
>> U1=U+1e-4*rand(size(U)); [U1,dummy]=qr(U1,0); U1=-U1
U1 =
1.0000 -0.0001
0.0000 0.7071
0.0001 0.7071

>> B=U1*A*U1

B =
1.0e-04 *
-0.0000 -0.2656
-0.2656 0.1828

>> [X,L]=eig(B)

X =
-0.8140 -0.5808
-0.5808 0.8140
L =
1.0e-04 *
-0.1896 0
0 0.3723

>> x=U1*-X(:,1)

x =
0.8140
0.4107
0.4107

>> theta=L(1,1)

theta =
-1.8955e-05

>> norm(A*x-x*theta)

ans =
0.5808

We note that is a reasonable approximation for the eigenvalue 0. However, as the

norm of the residual indicates, the Ritz vector is a bad approximation of the eigenvector.

12.6 Harmonic Ritz values and vectors

In the shift-and-invert Arnoldi algorithm the basic operator is A I where is some
shift. The Arnoldi algorithm finds the largest eigenvalues of A I, i.e., the eigenvalues
of A closest to the shift. One of the reasons for inventing the Jacobi-Davidson algorithm
is infeasibility of the factorization of A I. Therefore, a shift-and-invert approach is not
possible.
A clever way out of this dilemma works as follows: We apply the RitzGalerkin pro-
cedure with the matrix (A I)1 and some subspace R(V ) Fn . This leads to the
230 CHAPTER 12. THE JACOBI-DAVIDSON METHOD
eigenvalues problem

(12.30) V (A I)1 V s = V V s.

The largest Ritz values j approximate the largest eigenvalues of (A I)1 , i.e.,
1 1
j j + ,
j j

where j is an eigenvalue of A close to the shift .

The trick is in the choice of V . Let us set V := (A I)U . Then (12.30) becomes

(12.31) U (A I) U s = U (A I) (A I)U s,

or, with = 1/,

(12.32) U (A I) (A I)U s = U (A I) U s.

With V = (A I)U this becomes

(12.33) V V s = V U s.

If A is nonsymmetric, we compute an orthonormal basis V of V = (A I)U . Then we

can write (12.32) in the nonsymmetric form

(12.34) V (A I)U s = V U s.

We make the following

Definition 12.1 Let (, s) be an eigenpair of (12.32)(12.34). Then the pair ( + , U s)

is called a harmonic Ritz pair of A with shift .

In practice, we are interested only in the harmonic Ritz pair corresponding to the small-
est harmonic Ritz values. In the correction equation of the JacobiDavidson algorithm the
harmonic Ritz vector is used as the latest eigenvector approximation and the harmonic
Ritz values as the shift. In the symmetric case the harmonic Ritz value is replaced by the
Rayleigh quotient of the harmonic Ritz vector x, since

kAx (x)xk kAx xk, for all .

We continue the previous numerical example regarding the computation of the eigen-
value 0 of A = diag(0, 1, 1)
>> V=(A-theta*eye(3))*U1;
>> [v,l]=eig(V*V, V*U1)
v =
-1.000000000000000 -1.000000000000000
0.000059248824925 -0.713473633096137
l =
1.0e+17 *
0.000000000000000 0
0 -1.970695224946170

>> theta + l(1,1) % Harmonic Ritz value

ans =
3.722769433847084e-05
12.7. REFINED RITZ VECTORS 231

>> x = U1*v(:,1) % Harmonic Ritz vector

x =
1.000000001402380
-0.000018783973233
0.000018783630008
>> x*A*x
ans =
1.289413628670287e-14

The above considerations affect the JacobiDavidson algorithm in the extraction phase.
Steps 11 and 14 in Algorithm 12.2 become

11: Compute the smallest eigenpair (

,
s) of

(Wj
Vj )(Wj Vj )s = (Wj
Vj )Vj s.

14: Set q
= Vj
s, w
= Wj
s. = + or = q
A q q
q/ .

To solve the eigenvalue problem (12.34) the QZ algorithm has to be employed, see
section 12.8. In the symmetric case (12.33) the symmetric QR algorithm will suffice in
general since the matrix on the left is positive definite.

12.7 Refined Ritz vectors

An alternative to harmonic Ritz vectors are refined Ritz vectors [13]. Again we start from
the observation that the Ritz values were of good quality. What we need are improved
Ritz vectors. Stewart [13] suggested the following procedure.

Definition 12.2 Let be a Ritz value of A restricted to U . A solution of the mini-

mization problem

(12.35) min kA
x x
k
U ,k
x xk=1

is called a refined Ritz vector.

How is this minimization problem solved? We write x

= U z. Then (12.35) becomes

(12.36) min k(A I)U zk.

kzk=1

This minimization problem is solved by the right singular vector corresponding to the
smallest singular value of (A I)U or, equivalently, the eigenvector corresponding to
the smallest eigenvalue of

U (A I) (A I)U z = z.

We continue the example of before.

>> [u,s,v]=svd((A - 0*eye(3))*U)

u =
232 CHAPTER 12. THE JACOBI-DAVIDSON METHOD
0 1.0000 0
-0.7071 0 0.7071
0.7071 0 0.7071

s =
1.0000 0
0 0
0 0

v =
0 1
-1 0

>> U*v(:,2)

ans =
1
0
0

>> [u,s,v]=svd((A - L(1,1)*eye(3))*U1)

u =
-0.0000 0.5810 0.8139
-0.7071 -0.5755 0.4108
0.7071 -0.5755 0.4108

s =
1.0001 0
0 0.0000
0 0

v =
-0.0001 1.0000
-1.0000 -0.0001

>> format long

>> U1*v(:,2)

ans =
1.00009500829405
-0.00001878470226
0.00001878647014

With the refined Ritz vector approach Steps 11 and 14 in Algorithm 12.2 are replaced
by

11: q
Compute the Ritzpair (, ) of A closest to the target value.
s of AVj V
Compute the smallest singular vector j.
14: Replace q
by Vj
s.
12.8. THE GENERALIZED SCHUR DECOMPOSITION 233
12.8 The generalized Schur decomposition
The QZ algorithm computes the following generalized Schur decomposition.

Theorem 12.3 (Generalized Schur decomposition) If A, B Cnn then there are

unitary matrices Q, Z Cnn such that

(12.37) Q AZ = T A , Q BZ = T B ,

are upper triangular. If for some k, tA B

kk = tkk = 0 then (A, B) = C. Otherwise

(A, B) = {tA B B
ii /tii | tii 6= 0}.

Proof. See [5]

The algorithm starts out with transforming A and B in Hessenberg and upper triangular
form, respectively. After defalting zeros in the lower offdiagonal of the Hessenberg matrix
and on the diagonal of the upper triangular matrix, the QR algorithm with implicit shifts
is applied to AB 1 . For details see [5].

Corresponding to the notion of an invariant subspace for a single matrix we have

the notion of a deflating subspace for the pencil A B. In particular, we say that
a k-dimensional subspace S Fn is deflating for the pencil A B if the subspace
{Ax + By | x, y S} has dimension k or less. Note that the columns of the matrix Z
in the generalized Schur decomposition define a family of deflating subspaces, for if Q =
[q1 , . . . , qn ] and Z = [z1 , . . . , zn ] then we have span{Az1 , . . . , Azk } span{q1 , . . . , qk }
and span{Bz1 , . . . , Bzk } span{q1 , . . . , qk }.

12.9 JDQZ: Computing a partial QZ decomposition by the

JacobiDavidson algorithm
We now consider the generalized eigenvalue problem

(12.38) Ax = Bx,

with arbitrary A and B. There is a variant of JacobiDavidson called JDQZ that com-
putes a partial QZ decomposition of the stencil (A, B). This section follows closely the
corresponding section in the eigenvalue templates [12]. Further details are found in [3].
With = /, the generalized eigenproblem (12.38) is equivalent to the eigenproblem

(12.39) (A B)x = 0,

where we denote a generalized eigenvalue of the matrix pair {A, B} as a pair (, ). The
notation (12.39) is preferred over (12.40), because underflow or overflow for = / in
finite precision arithmetic may occur when and/or are zero or close to zero. It also
emphazises the symmetry of the roles of A and B.
A partial generalized Schur form of dimension k for a matrix pair {A, B} is the
decomposition

(12.40) AQk = Zk RkA , BQk = Zk RkB ,

where Qk and Zk are unitary n k matrices and RkA and RkB are upper triangular k k
matrices. A column qi of Qk is referred to as a generalized Schur vector, and we refer to a
234 CHAPTER 12. THE JACOBI-DAVIDSON METHOD
pair ((i , i ), qi ), with (i , i ) = (RkA (i, i), RkB (i, i)) as a generalized Schur pair. It follows
that if ((, ), y) is a generalized eigenpair of (RkA , RkB ) then ((, ), Qk y) is a generalized
eigenpair of {A, B}.
From the relations (12.40) we see that

i Aqi i Bqi zi .

This somewhat resembles the Schur decomposition, where Aqi i qi qi . The zi on

the right hand side suggests that we should follow a Petrov-Galerkin condition for the
construction of reduced systems. In each step the approximate eigenvector u is selected
from a j-dimensional search subspace span(Vj ) = span{v1 , . . . , vj }. We require that the
residual Au Bu is orthogonal to some other well-chosen test subspace span(Wj ) =
span{w1 , . . . , wj },

(12.41) Au Bu span(Wj ).

Equation (12.41) leads to the projected generalized j j eigenproblem

(12.42) ( Wj AVj Wj BVj ) s = 0.

The j-dimensional pencil Wj AVj Wj BVj can be reduced by the QZ algorithm

(see 12.8) to generalized Schur form. This leads to orthogonal j j matrices S R and S L
and upper triangular j j matrices T A and T B , such that

(12.43) (S L ) (Wj AVj )S R = T A and (S L ) (Wj BVj )S R = T B .

This decomposition can be reordered such that the first column of S R and the (1, 1)-
entries of T A and T B represent the wanted Petrov solution [3]. With s := sR R
1 := S e1 and
A B
:= T1,1 , := T1,1 , the Petrov vector is defined as

u := Vj s = Vj sR
1

for the associated generalized Petrov value (, ). In an analogous way we can define a
left Petrov vector as
p := Wj sL
1 sL L
1 := S e1

If Vj and Wj are unitary, as in Algorithm 12.3, then ksR k2 = ksL k2 = 1 implies kuk2 = 1.
With the decomposition in (12.43), we construct an approximate partial generalized
Schur form (cf. (12.40)): Vj S R approximates a Qk , and Wj S L approximates the associated
Zj .
It is not yet clear how to choose the test space Wj . The equations span(Zj ) =
span(AQj ) = span(BQj ), cf. (12.40), suggest to choose Wj such that span(Wj ) coin-
cides with span(0 AVj + 0 BVj ) for some suitably chosen 0 and 0 . With the weights
0 and 0 we can influence the convergence of the Petrov values. If we want eigenpair
approximations for eigenvalues close to a target , then the choice
p
0 = 1/ 1 + | |2 , 0 = 0

is very effective [3], especially if we want to compute eigenvalues in the interior of the
spectrum of A B. We will call the Petrov approximations for this choice the harmonic
Petrov eigenpairs. The Jacobi-Davidson correction equation for the component t u for
the pencil A B becomes

(12.44) (I pp ) (A B) (I uu ) t = r, r := Au Bu.
12.9. JDQZ: COMPUTING A PARTIAL QZ DECOMPOSITION 235
Sleijpen et al. [10] have shown that if (12.44) is solved exactly, the convergence to the
generalized eigenvalue is quadratic. Usually, this correction equation is solved only ap-
proximately, for instance, with a (preconditioned) iterative solver. The obtained vector t
is used for the expansion v of Vj and 0 Av + 0 Bv is used for the expansion of Wj . For
both spaces we work with orthonormal bases. Therefore, the new columns are orthonor-
malized with respect to the current basis by a modified Gram-Schmidt orthogonalization
process.

12.9.1 Restart
Suppose that the generalized Schur form (12.43) is ordered with respect to such that
A B A B A B
|T1,1 /T1,1 | |T2,2 /T2,2 | |Tj,j /Tj,j |,

where j is the dimension of span(Vj ). Then, for i < j, the space span(Vj sR R
1 , . . . , Vj si )
R
spanned by the first i columns of Vj S contains the i most promising Petrov vectors.
The corresponding test subspace is given by span(Wj sL , . . . , W sL
i ). Therefore, in order to
reduce the dimension of the subspaces (implicit restart) to jmin , jmin < j, the columns
vjmin +1 through vj and wjmin+1 through wj can simply be discarded and the Jacobi-
Davidson algorithm can be continued with

V = [V sR R L L
1 , . . . , V sjmin ] and W = [W s1 , . . . , W sjmin ].

12.9.2 Deflation
Like in the Jacobi-Davidson algorithm for the standard eigenvalue problem, in the Jacobi-
Davidson process for the generalized eigenvalue problem found (converged) Ritz (here
Petrov) vectors can be deflated.
The partial generalized Schur form can be obtained in a number of successive steps.
Suppose that we have already available the partial generalized Schur form AQk1 =
A
Zk1 Rk1 and BQk1 = Zk1 Rk1 B . We want to expand this partial generalized Schur

form with the new right Schur vector u and the left Schur vector p to
A
 
Rk1 a
A[Qk1 u] = [Zk1 p]
0

and  
B
Rk1 b
A[Qk1 u] = [Zk1 p]
0
The new generalized Schur pair ((, ), u) satisfies

Qk1 u = 0 and (A B)u Zk1 (a b) = 0,

(A B)u,
or, since a b = Zk1



Qk1 u = 0 and
I Zk1 Zk1 (A B) I Qk1 Qk1 u = 0.

Hence, the vectors a and b can be computed from

a = Zk1 Au and b = Zk1 Bu.
236 CHAPTER 12. THE JACOBI-DAVIDSON METHOD
Furthermore, the generalized Schur pair ((, ), u) is an eigenpair of the deflated matrix
pair







I Zk1 Zk1 A I Qk1 Qk1 , I Zk1 Zk1

B I Qk1 Qk1 .

This eigenproblem can be solved again with the Jacobi-Davidson QZ process. In that
process we construct vectors vi that are orthogonal to Qk1 and vectors wi that are
orthogonal to Zk1 . This simplifies the computation of the interaction matrices M A and
M B , associated with the deflated operators

M A W (I Zk1 Zk1 ) A (I Qk1 Qk1 ) V = W AV,
M A W (I Zk1 Zk1 ) B (I Qk1 Qk1 ) V = W BV,

and M A and M B can be simply computed as W AV and W BV , respectively.

12.9.3 Algorithm
The Jacobi-Davidson algorithm to compute a partial QZ decomposition for a general
matrix pencil (A, B) is given in Algorithm 12.3 This algorithm attempts to compute the
generalized Schur pairs ((, ), q), for which the ratio / is closest to a specified target
value in the complex plane. The algorithm includes restart in order to limit the dimension
of the search space, and deflation with already converged left and right Schur vectors.
To apply this algorithm we need to specify a starting vector v0 , a tolerance , a target
value , and a number kmax that specifies how many eigenpairs near should be computed.
The value of jmax specifies the maximum dimension of the search subspace. If it is exceeded
then a restart takes place with a subspace of dimension jmin .
On completion the kmax generalized eigenvalues close to are delivered, and the cor-
responding reduced Schur form AQ = ZRA , BQ = ZRB , where Q and Z are n by kmax
orthogonal and RA , RB are kmax by kmax upper triangular. The generalized eigenvalues
are the on-diagonals of RA and RB . The computed form satisfies kAqj ZRA ej k2 = O(),
kBqj ZRB ej k2 = O(), where qj is the jth column of Q.

12.10 Jacobi-Davidson for nonlinear eigenvalue problems

Nonlinear eigenvalue problems have the form

(12.45) T ()x = 0

where the n n matrix T () has elements that depend on the scalar parameter . For the
linear eigenvalue problem T () = A B. is an eigenvalue of (12.45) if T () is singular;
a nontrivial solution x of the singular linear system is a corresponding eigenvector.
For small problems, Newton iteration is applicable. Ruhe [9] suggests to proceed as
follows. Complement (12.45) by a normalization condition

(12.46) v x = 1.

Then, we solve
     
x T ()x 0
(12.47) P = = .
v x 1 0
12.10. JACOBI-DAVIDSON FOR NONLINEAR EIGENVALUE PROBLEMS 237

Algorithm 12.3 JacobiDavidson QZ method for kmax interior eigenvalues close

to for the generalized non-Hermitian eigenvalue problem
1: Let A, B Fnn be non-Hermitian. This algorithm computes kmax interior eigenvalues
of Ax = Bx close topthe target .
2: t = v0 ; k = 0; 0 = 1/ 1 + | |2 ; 0 = 0 ; m = 0;
3: Q = []; Z = []; S = []; T = [];
4: while k < kmax do
5: Orthogonalize t := t Vm Vm t
6: m = m + 1; vm = t/ktk; vm A := Av ; vB := Bv ; w := vA + vB ;
m m m 0 m 0 m
7: Orthogonalize w := w Zk Zk w
8: Orthogonalize w := w Wm1 Wm1 w
9: wm = w/kwk;
10:    
A MA
Wm1 vmA
B MB
Wm1 B
vm
M := VA vA ; M := VB vB ;
wm m1 wm m wm m1 wm m

11: Compute the QZ decomposition M A S R = S L T A , M B S R = S L T B , such that

A /T B | |T A
|Ti,i B
i,i i+1,i+1 /Ti+1,i+1 | /* Rayleigh Ritz step */
12: R L A A R B B R A
u := V s1 ; p := Wj s1 ; u := V s1 ; u := V s1 ; = T1,1 ; = T1,1 B ;

13: r = uA uB ; = Z uB ;
a = Z uA ; b r = r Z(
a b);
14: while k rk < do
15:  A   B 
A R
a B R
b
R := ; R := ;
0T 0T
16: Q := [Q, u]; Z := [Z, p]; k := k + 1;
17: if k = kmax then
18: return (Q, Z, RA , RB )
19: end if
20: m := m 1;
21: for i = 1, . . . , m do
22: vi := V sR A A R B B R
i+1 ; vi := V si+1 ; vi := V si+1 ;
23: L R L
wi := W si+1 ; si := si := ei ;
24: end for
25: M A , M B is the lower m m block of T A , T B , resp.
26: u := u1 ; p := w1 ; uA := v1A ; uB := v1b ; = T1,1 A ; = TB ;
1,1
27: r = uA uB ; = Z uB ;
a = Z uA ; b r = r Z(
a b);
28: end while
29: if m mmax then
30: for i = 2, . . . , mmin do
31: vi := V sR A A R B B R
i ; vi := V si ; vi := V si ; wi := W si ;
L

32: end for

33: M A , M B is the leading mmin mmin block of T A , T B , resp.
34: v1 := u; v1A := uA ; v1B := uB ; w1 := p; m := mmin
35: end if
36: Q := [Q, u]; Z := [Z, p];
37: (Approximatively) solve the correction equation for t Q,
38:
(I Z Z )(A B)(I QQ )

39: end while

238 CHAPTER 12. THE JACOBI-DAVIDSON METHOD
For the derivative of P we obtain
 
T () T ()x
P =
v 0

such that the Newton iteration becomes

     1  
xs+1 xs T (s ) T (s )xs T (s )xs
(12.48) =
s+1 s vs 0 vs xs 1
or
T (s )us+1 = T (s )xs ,
(12.49) s+1 = s (vs xs )/(vs xs+1 ),
xs+1 = C us+1 .

Here, C is some normalization constant. The vector vs may depend on the iteration step.
It can be chosen in a number of ways. It could be constant, e.g., vs = ei . This amounts
to keeping one of the entries of xs constant. Another choce is

vs = T (s ) ys

where ys is an approximation to the left eigenvector y.

A Jacobi-Davidson algorithm for large nonlinear eigenvalue problems is given in Algo-
rithm 12.4. This algorithm is by Voss [14]. There are two noteworthy issues.

Algorithm 12.4 Nonlinear JacobiDavidson algorithm

1: Start with an initial basis V , V V = I; m = 1.
2: Determine a preconditioner K T (), close to the first wanted eigenvalue.
3: while m number of wanted eigenvalues do
4: Compute an approximation to the m-th wanted eigenvalue m and corresponding
eigenvector sm of the projected problem V T (m )V s = 0.
5: Determine the Ritz vector u = V sm and the residual r = T (m )u
6: if krk/kuk < then
7: Accept approximate eigenpair (m , u); m := m + 1;
8: Reduce the search space V if necessary
9: Choose an approximation (m , u) to the next eigenpair.
10: Compute the residual r = T (m )u
11: end if
12: p = T (m )x;
13: (Approximatively) solve the correction equation for t,

pu uu
(12.50) (I )T ()(I )t = r, t u.
u p u u

14: Orthogonalize t := t V V t, v := t/ktk, and expand the subspace [V, v].

15: Determine a new preconditioner K T (m ) if necessary.
16: Update the projected problem.
17: end while

1. The projected problem is the nonlinear eigenvalue problem V T (m )V s = 0 where

m is an approximation to the wanted eigenvalue.
BIBLIOGRAPHY 239
2. In the expansion of the search space, it is ensured that the Newton iterate is contained
in the expanded search space. To this end, assume that u is the Ritz vector in R(V )
obtained from the projected problem, u = V s. Set p = T (m )u. We now solve the
correction equation

pu uu
(12.50) (I )T ( m )(I )t = r = T (m )u, t u.
u p u u

This equation can be written as

1
T (m )t p = r, = u T (m )t.
u p

Using T (m )u = r we get

t = u + T (m )1 p = u + T (m )1 T (m )u.

is determined such that t u. Since u R(V ), we must have T (m )1 T (m )u

R([V, t]). This ensures the quadratic convergence rate of Newtons method.

The correction equation (12.50) in Algorithm 12.4 is typically solved to low accuracy
by a preconditioned GMRES iteration where the preconditioner has the form

(12.51) = (I pu )K(I uu ),
K K T ().
u p u u

Solving with the preconditioner amounts to solving the equation

= g,
Kt t u.

Bibliography
[1] R. Barret, M. Berry, T. F. Chan, J. Demmel, J. Donato, J. Dongarra,
V. Eijkhout, R. Pozo, C. Romine, and H. van der Vorst, Templates for the
Solution of Linear Systems: Building Blocks for Iterative Methods, SIAM, Philadel-
phia, PA, 1994. (Available from Netlib at URL http://www.netlib.org/templates/
index.html).

[2] E. R. Davidson, The iterative calculation of a few of the lowest eigenvalues and cor-
responding eigenvectors of large real-symmetric matrices, J. Comp. Phys., 17 (1975),
pp. 8794.

[3] D. R. Fokkema, G. L. G. Sleijpen, and H. A. van der Vorst, JacobiDavidson

style QR and QZ algorithms for the partial reduction of matrix pencils, SIAM J. Sci.
Comput., 20 (1998), pp. 94125.

[4] R. Geus, The JacobiDavidson algorithm for solving large sparse symmetric eigen-
value problems, PhD Thesis No. 14734, ETH Z urich, 2002. (Available at URL
http://e-collection.ethbib.ethz.ch/show?type=diss&nr=14734).

[5] G. H. Golub and C. F. van Loan, Matrix Computations, The Johns Hopkins
University Press, Baltimore, MD, 2nd ed., 1989.
240 CHAPTER 12. THE JACOBI-DAVIDSON METHOD

[6] C. G. J. Jacobi, Uber ein leichtes Verfahren die in der Theorie der S
acularst
orungen
vorkommenden Gleichungen numerisch aufzul osen, J. reine angew. Math., 30 (1846),
pp. 5195.

[7] R. B. Morgan, Davidsons method and preconditioning for generalized eigenvalue

problems, J. Comp. Phys., 89 (1990), pp. 241245.

[8] R. B. Morgan and D. S. Scott, Generalizations of Davidsons method for comput-

ing eigenvalues of sparse symmetric matrices, SIAM J. Sci. Stat. Comput., 7 (1986),
pp. 817825.

[9] A. Ruhe, Algorithms for the nonlinear eigenvalue problem, SIAM J. Numer. Anal.,
10 (1973), pp. 674689.

[10] G. L. G. Sleijpen, A. G. L. Booten, D. R. Fokkema, and H. A. van der

Vorst, JacobiDavidson type methods for generalized eigenproblems and polynomial
eigenproblems, BIT, 36 (1996), pp. 595633.

[11] G. L. G. Sleijpen and H. A. van der Vorst, A JacobiDavidson iteration method

for linear eigenvalue problems, SIAM J. Matrix Anal. Appl., 17 (1996), pp. 401425.

[12] G. L. G. Sleijpen and H. A. van der Vorst, JacobiDavidson method, in Tem-

plates for the solution of Algebraic Eigenvalue Problems: A Practical Guide, Z. Bai,
J. Demmel, J. Dongarra, A. Ruhe, and H. van der Vorst, eds., SIAM, Philadelphia,
PA, 2000, pp. 238246.

[13] G. W. Stewart, Matrix Algorithms II: Eigensystems, SIAM, Philadelphia, PA, 2001.

[14] H. Voss, A JacobiDavidson method for nonlinear eigenproblems, in Computational

Science ICCS 2004, G. D. van Albada, M. Bubak, P. M. A. Sloot, and J. J. Dongarra,
eds., Berlin, 2004, Springer, pp. 3441. (Lecture Notes in Computer Science, 3037).
Chapter 13

Rayleigh quotient and trace

minimization

13.1 Introduction
In this chapter we restrict ourselves to the symmetric/Hermitian eigenvalue problem

(13.1) Ax = M x, A = A , M = M > 0.

We want to exploit the property of the Rayleigh quotient that

x Ax
(13.2) 1 = min (x) (x) = ,
x6=0 x M x
which was proved in Theorem 2.30. The basic idea of Rayleigh quotient minimization is
to construct a sequence {xk }k=1,2,... such that (xk+1 ) < (xk ) for all k. The hope is
that the sequence {(xk )} converges to 1 and by consequence the vector sequence {xk }
towards the corresponding eigenvector.
The procedure is as follows: For any given xk let us choose a search direction pk ,
so that

(13.3) xk+1 = xk + k pk .

The parameter k is determined such that the Rayleigh quotient of the new iterate xk+1
becomes minimal,

(13.4) (xk+1 ) = min (xk + pk ).

We can write the Rayleigh quotient of the linear combination xk + pk of two (linearly
independant) vectors xk and pk as
(13.5)     
1 xk Axk xk Apk 1
xk Axk + 2xk Apk + 2 pk Apk pk Axk pk Apk
(xk + pk ) = 2 =   
  .
xk M xk + 2xk M pk + pk M pk 1 xk M xk xk M pk 1

pk M xk pk M pk

This is the Rayleigh quotient associated with the generalized 2 2 eigenvalue problem
     
xk Axk xk Apk xk M xk xk M pk
(13.6) = .
pk Axk pk Apk pk M xk pk M pk

241
242 CHAPTER 13. RAYLEIGH QUOTIENT AND TRACE MINIMIZATION
The smaller of the two eigenvalues of (13.6) is the searched value k+1 := (xk+1 ) in (13.4)
that minimizes the Rayleigh quotient. The corresponding eigenvector is normalized such
that its first component equals one1 . The second component of this eigenvector is = k .
Inserting the solution [1, k ] into the second line of (13.6) we obtain

(13.7) pk (A k+1 M )(xk + k pk ) = pk rk+1 = 0.

So, the next residual rk+1 is orthogonal to the actual search direction pk .
There are various ways how to choose the search direction pk . A simple way is to cycle
through the coordinate vectors, a method that is called coordinate relaxation [3]. It cannot
compete with the methods we discuss next; but it has some potential for parallelization.

13.2 The method of steepest descent

Let us make a detour to solving systems of equations

(13.8) Ax = b,

where A is symmetric/Hermitian positive definite. Let us define the functional

1 1 1
(13.9) (x) x Ax x b + b A1 b = (Ax b) A1 (Ax b).
2 2 2
The functional is minimized at the solution x of (13.8). (With the above definition of
the minimum is actually zero.) The negative gradient of is

(13.10) (x) = b Ax =: r(x).

It is nonzero except at x . In the method of steepest descent [2, 3] a sequence of vectors

{xk }k=1,2,... is constructed such that the relation

(13.3) xk+1 = xk + k pk

holds among any two consecutive vectors. The search direction pk is chosen to be the
negative gradient (xk ) = rk = b Axk . This is the direction in which decreases
the most. Setting xk+1 as in (13.3) we get

(xk+1 ) 
0=  = pk (Axk b) + k pk Apk = pk rk + k pk Apk .
=k

Thus,
pk rk
(13.11) k =
pk Apk
which, for steepest descent, becomes
rk rk
(13.12) k =
rk Ark
Remark 13.1. Notice that

(13.13) rk+1 = b Axk+1 = b A(xk + k pk ) = rk k Apk .

1
The first component of this eigenvector is nonzero if it has a component in the direction of the smallest
eigenvector.
13.3. THE CONJUGATE GRADIENT ALGORITHM 243
Therefore, from (13.11) we have

(13.14) pk rk+1 = pk rk k pk Apk = 0,

which corresponds to (13.7) in the linear system case.

For the eigenvalue problem we can proceed similarly by choosing pk to be the negative
gradient of the Rayleigh quotient ,
2
pk = gk = (xk ) = (Axk (xk )M xk ).
xk M xk

Notice that gk points in the same direction as the residual rk . (This is in contrast to the
linear system case!) Since in eigenvalue problems we only care about directions we can
equivalently set

(13.15) pk = rk = Axk k M xk , k = (xk ).

With this choice of search direction we immediately have from (13.7) that

(13.16) rk rk+1 = 0.

Not surprisingly, the method of steepest descent often converges slowly, as it does for
linear systems. This happens if the spectrum is very much spread out, i.e., if the condition
number of A relative to M is big.

13.3 The conjugate gradient algorithm

As with linear systems of equations a remedy against the slow convergence of steepest
descent are conjugate search directions. So, lets again first look at linear systems [5].
There, we define the search directions as2

(13.17) pk = gk + k pk1 , k > 0.

where the coefficient k is determined such that pk and pk1 are conjugate, i.e.,

(13.18) pk Apk1 = gk Apk1 + k pk1 Apk1 = 0,

such that
gk Apk1
(13.19) k = .
pk1 Apk1

Premultiplying (13.17) by gk gives

(13.14)
(13.20) gk pk = gk gk + k gk pk1 = gk gk .

Furthermore, since xk M xk = 1 we have rk = gk and

(13.14) (13.17)
0 = gk+1 pk = gk+1 gk + k gk+1 pk1
(13.13)
= gk+1 gk + k gk pk1 + k k pk Apk1 .
2
In linear systems the residual r = b Ax is defined as the negative gradient whereas in eigenvalue
computations it is defined as r = Ax (x)M x, i.e., in the same direction as the gradient. To reduce the
confusion we proceed using the gradient.
244 CHAPTER 13. RAYLEIGH QUOTIENT AND TRACE MINIMIZATION
From (13.14) we have that gk pk1 = 0 and by construction of pk and pk1 being conjugate
we have that pk Apk1 = 0. Thus,

(13.21) gk+1 gk = 0,

as with the method of steepest descent. Still in the case of linear systems, using these
identities we find formulae equivalent to (13.19),
gk Apk1 (13.13) gk (gk gk1 ) (13.14) gk (gk gk1 )
k = = =
pk1 Apk1 pk1 (gk gk1 ) pk1 gk1
(13.20) gk (gk gk1 )
(13.22) = g
gk1 k1
(13.21) gk gk
(13.23) = g .
gk1 k1

The equivalent identities (13.19), (13.22), and (13.23) can be used to define k the most
economic being (13.23).
We now look at how a conjugate gradient algorithm for the eigenvalue problem can be
devised. The idea is straightforward. The algorithm differs from steepest descent by the
choice of the search directions that are kept conjugate, i.e., consecutive search directions
satisfy pk Apk1 = 0.
The crucial difference to linear systems stems from the fact, that the functional that
is to be minimized, i.e., the Rayleigh quotient, is not quadratic anymore. (In particular,
there is no finite termination property.) The gradient of (x) is
2
g = (xk ) = (Ax (x)M x).
x M x
So, in particular, the equation (13.14), does not hold:

xk+1 = xk + k pk 6= gk+1 = gk + k Apk .

Therefore, in the context of nonlinear systems or eigenvalue problems the formulae in (13.19),
(13.22), and (13.23) that define k are not equivalent anymore! Feng and Owen [4] exten-
sively compared the three formulae and found that in the context of eigenvalue problems
the last identity (13.23) leads to the fastest convergence. So, we opt for this equation and
define the search directions according to


p0 = g0 , k = 0,

gk M gk
(13.24)
pk = gk + g M g
pk1 , k > 0,
k1 k1

where we have given the formulae for the generalized eigenvalue problem Ax = M x. The
complete procedure is given in Algorithm 13.1

Convergence
The construction of Algorithm 13.1 guarantees that (xk+1 ) < (xk ) unless rk = 0, in
which case xk is the searched eigenvector. In general, i.e., if the initial vector x0 has a
nonvanishing component in the direction of the smallest eigenvector u1 , convergence is
toward the smallest eigenvalue 1 . This assumption must also hold for vector iteration or
the Lanczos algorithm.
13.3. THE CONJUGATE GRADIENT ALGORITHM 245
Algorithm 13.1 The Rayleigh quotient algorithm
1: Let x0 be a unit vector, kx0 kM = 1.
2: v0 := Ax0 , u0 := M x0 ,
v x
3: 0 := 0 0 ,
u0 x0
4: g0 := 2(v0 0 u0 )
5: while kgk k > tol do
6: if k = 1 then
7: pk := gk1 ;
8: else
g M gk1
9: pk := gk1 + k1 p ;
gk2 M gk2 k1
10: end if
11: Determine the smallest Ritz value k and corresponding Ritz vector xk of (A, M )
in R([xk1 , pk ])
12: vk := Axk , uk := M xk
13: k := xk vk /xk uk
14: gk := 2(vk k uk )
15: end while

Let

(13.25) xk = cos k u1 + sin k zk =: cos k u1 + wk ,

where kxk kM = ku1 kM = kzk kM = 1 and u1 M zk = 0. Then we have

(xk ) = cos2 k 1 + 2 cos k sin k u1 Azk + sin2 k zk Azk

= 1 (1 sin2 k ) + sin2 k (zk ),
or,

(13.26) (xk ) 1 = sin2 k ((zk ) 1 ) (n 1 ) sin2 k .

As seen earlier, in symmetric eigenvalue problems, the eigenvalues are much more accurate
than the eigenvectors.
Let us now suppose that the eigenvalues have already converged, i.e.,
(xk ) = k
= 1 ,
while the eigenvectors are not yet as accurate as desired. Then we can write
n
X
(13.27) rk = (A k M )xk
= (A 1 M )xk = (j 1 )M uj uj M xk ,
j=1

which entails u1 rk = 0 since the first summand on the right of (13.27) vanishes. From (13.25)
we have wk = sin k zk M u1 . Thus,
(
(A 1 M )wk = (A 1 M )xk = rk u1
(13.28)
wk M u1 = 0

If 1 is a simple eigenvalue of the pencil (A; B) then A 1 M is a bijective mapping of

R(u1 )M onto R(u1 ) . If r R(u1 ) then the equation
(13.29) (A 1 M )w = r
246 CHAPTER 13. RAYLEIGH QUOTIENT AND TRACE MINIMIZATION
has a unique solution in R(u1 )M .
So, close to convergence, Rayleigh quotient minimization does nothing else but solving
equation (13.29) by the conjugate gradient algorithm. The convergence of the Rayleigh
quotient minimization is determined by the condition number of A1 M (as a mapping of
R(u1 )M onto R(u1 ) ), according to the theory of conjugate gradients for linear system
of equations. This condition number is
 n 1

(13.30) 0 = K(A 1 M ) = ,

R(u1 ) M 2 1
and the rate of convergence is given by

0 1
(13.31) .
0 + 1

A high condition number implies slow convergence. We see from (13.31) that the condition
number is high if the distance of 1 and 2 is much smaller than the spread of the spectrum
of (A; B). This happens more often than not, in particular with FE discretizations of
PDEs.

Preconditioning
In order to reduce the condition number of the eigenvalue problem we change

Ax = M x

into

(13.32) A M
x = x
,

such that

(13.33) 1M
(A ) (A 1 M ).

To further investigate this idea, let C be a nonsingular matrix, and let y = Cx. Then,

x Ax y C AC 1 y
y Ay
(13.34) (x) = = = = (y)
x M x y C M C 1 y M
y y

Thus,
A 1 M
= C (A 1 M )C 1 ,

or, after a similarity transformation,

C 1 (A 1 M
)C = (C C)1 (A 1 M ).

How should we choose C to satisfy (13.33)? Let us tentatively set C C = A. Then we

have
 
1 1 1 1
(C C) (A 1 M )uj = A (A 1 M )uj = (I 1 A M )uj = 1 uj .
j

Note that
1
0 1 < 1.
j
13.4. LOCALLY OPTIMAL PCG (LOPCG) 247
Dividing the largest eigenvalue of A1 (A1 M ) by the smallest positive gives the condition
number
   1 1 2 n 1 2
1  n
(13.35) 1 := A (A 1 M ) R(u1 )M = 1
= = 0 .
1 n 2 1 n
2

If 2 n then the condition number is much reduced. Further, 1 is bounded indepen-

dently of n,
1 1 /n 1
(13.36) 1 = < .
1 1 /2 1 1 /2
So, with this particular preconditioner, 1 does not dependent on the choice of the mesh-
width h in the FEM application.
The previous discussion suggests to choose C in such way that C C
= A. C could, for
instance, be obtained form an Incomplete Cholesky decomposition. We make this choice
in the numerical example below.
Notice that the transformation x y = Cx need not be made explicitly. In partic-
ular, the matrices A and M must not be formed. As with the preconditioned conjugate
gradient algorithm for linear systems there is an additional step in the algorithm where
the preconditioned residual is computed, see Fig. 13.1 on page 248.

13.4 Locally optimal PCG (LOPCG)

The parameter k in the RQMIN und (P)CG algorithms is determined such that
(13.37) (xk+1 ) = (xk + k pk ), pk = gk + k pk1
is minimized. k is chosen to make consecutive search directions conjugate. Knyazev [6]
proposed to optimize both parameters, k and k , at once.
(13.38) (xk+1 ) = min (xk gk + pk1 )
,

This results in potentially smaller values for the Rayleigh quotient, as

min xk gk + pk1 min xk (gk k pk ) .
,

Hence, Knyazev coined the notion locally optimal.

(xk+1 ) in (13.38) is the minimal eigenvalue of the 3 3 eigenvalue problem

xk xk
(13.39) g A[xk , gk , pk1 ] = g M [xk , gk , pk1 ]
k k
pk1 pk1
We normalize the eigenvector corresponding to the smallest eigenvalue such that its first
component becomes 1,
[1, k , k ] := [1, /, /].
These values of k and k are the parameters that minimize the right hand side in (13.38).
Then we can write
(13.40) xk+1 = xk k gk + k pk1 = xk + k (gk + (k /k )pk1 ) = xk + k pk .
| {z }
=:pk

We can consider xk+1 as having been obtained by a Rayleigh quotient minimization from
xk along pk = gk +(k /k )pk1 . Notice that this direction is needed in the next iteration
step. (Otherwise it is not of a particular interest.)
248 CHAPTER 13. RAYLEIGH QUOTIENT AND TRACE MINIMIZATION

function [x,rho,log] = rqmin1(A,M,x,tol,C)

%RQMIN1 [x,rho] = rqmin1(A,M,x0,tol,C)
% cg-Rayleigh quotient minimization for the computation
% of the smallest eigenvalue of A*x = lambda*M*x,
% A and M are symmetric, M spd. x0 initial vector
% C*C preconditioner
% tol: convergence criterium:
% ||2*(C*C)\(A*x - lam*M*x)|| < tol

% PA 16.6.2000

u = M*x;
q = sqrt(x*u);
x = x/q; u = u/q;
v = A*x;
rho = x*v;

k = 0; g = x; gnorm = 1; log=[]; % Initializations

while gnorm > tol,

k = k + 1;
galt = g;
if exist(C),
g = 2*(C\(C\(v - rho*u))); % preconditioned gradient
else
g = 2*(v - rho*u); % gradient
end
if k == 1,
p = -g;
else
p = -g + (g*M*g)/(galt*M*galt)*p;
end

[qq,ll] = eig([x p]*[v A*p],[x p]*[u M*p]);

[rho,ii] = min(diag(ll));
delta = qq(2,ii)/qq(1,ii);

x = x + delta*p;
u = M*x;
q = sqrt(x*u);
x = x/q; u = u/q;
v = A*x;
gnorm = norm(g);
if nargout>2, log = [log; [k,rho,gnorm]]; end
end

Figure 13.1: Matlab code RQMIN: Rayleigh quotient minimization

13.4. LOCALLY OPTIMAL PCG (LOPCG) 249

function [x,rho,log] = lopcg(A,M,x,tol,C)

%RQMIN1 [x,rho] = lopcg(A,M,x0,tol,C)
% Locally Optimal Proconditioned CG algorithm for
% computing the smallest eigenvalue of A*x = lambda*M*x,f
% where A and M are symmetrisch, M spd.
% x0 initial vektor
% C*C preconditioner
% tol: stopping criterion:
% (C*C)\(A*x - lam*M*x) < tol

% PA 2002-07-3

n = size(M,1);
u = M*x;
q = sqrt(x*u);
x = x/q; u = u/q;
v = A*x;
rho = x*v;

k = 0; gnorm = 1; log=[]; % initializations

while gnorm > tol,

k = k + 1;
g = v - rho*u; % gradient
gnorm = norm(g);
if exist(C),
g = (C\(C\g)); % preconditioned gradient
end
if k == 1, p = zeros(n,0); end

aa = [x -g p]*[v A*[-g p]]; aa = (aa+aa)/2;

mm = [x -g p]*[u M*[-g p]]; mm = (mm+mm)/2;
[qq,ll] = eig(aa,mm);
[rho,ii] = min(diag(ll));
delta = qq(:,ii);

p = [-g p]*delta(2:end);
x = delta(1)*x + p;
u = M*x;
q = sqrt(x*u);
x = x/q; u = u/q;
v = A*x;
if nargout>2, log = [log; [k,rho,gnorm]]; end
end

Figure 13.2: Matlab code LOPCG: Locally Optimal Preconditioned Conjugate Gradient
algorithm
250 CHAPTER 13. RAYLEIGH QUOTIENT AND TRACE MINIMIZATION
13.5 The block Rayleigh quotient minimization algorithm
(BRQMIN)
The above procedures converge very slowly if the eigenvalues are clustered. Hence, these
methods should be applied only in blocked form.
Longsine and McCormick [8] suggested several variants for blocking Algorithm 13.1.
See [1] for a recent numerical investigation of this algorithm.

13.6 The locally-optimal block preconditioned conjugate gra-

dient method (LOBPCG)
In BRQMIN the Rayleigh quotient is minimized in the 2q-dimensional subspace generated
by the eigenvector approximations Xk and the search directions Pk = Hk + Pk1 Bk ,
where the Hk are the preconditioned residuals corresponding to Xk and Bk is chosen such
that the block of search directions is conjugate. Instead, Knyazev [6] suggests that the
space for the minimization be augmented by the q-dimensional subspace R(Hk ). The
resulting algorithm is deemed locally-optimal because (x) is minimized with respect to
all available vectors.

Algorithm 13.2 The locally-optimal block preconditioned conjugate gradient

method (LOBPCG) for solving Ax = M x with preconditioner N of [1]
1: Choose random matrix X0 Rnq with X0T M X0 = Iq . Set Q := [ ].
2: Compute (X0T AX 0 )S0 = S0 0 /* (Spectral decomposition) */
where S0T S0 = Iq , 0 = diag(1 , . . . , q ), 1 . . . q .
3: X0 := X0 S0 ; R0 := AX0 M X0 0 ; P0 := [ ]; k := 0.
4: while rank(Q) < p do
5: Solve the preconditioned linear system N Hk = Rk
6: Hk := Hk Q(QT M Hk ).
7: Ae := [Xk , Hk , Pk ]T A[Xk , Hk , Pk ].
8: f := [Xk , Hk , Pk ]T M [Xk , Hk , Pk ].
M
9: Compute A eSek = M fSek ek /* (Spectral decomposition) */
where Sek M
T fSek = I3q , e k = diag(1 , . . . , 3q ), 1 . . . 3q .
10: Sk := Sek [e1 , . . . , eq ], := diag(1 , . . . , q ).
11: Pk+1 := [Hk , Pk ] Sk,2 ; Xk+1 := Xk Sk,1 + Pk+1 .
12: Rk+1 := AX k+1 M X k+1 k .
13: k := k + 1.
14: for i = 1, . . . , q do
15: /* (Convergence test) */
16: if kRk ei k < tol then
17: Q := [Q, Xk ei ]; Xk ei := t, with t a random vector.
18: M -orthonormalize the columns of Xk .
19: end if
20: end for
21: end while

If dj = [dT1j , dT2j , dT3j ]T , dij Rq , is the eigenvector corresponding to the j-th eigenvalue
of (13.1) restricted to R([Xk , Hk , Pk1 ]), then the j-th column of Xk+1 is the corresponding
13.7. A NUMERICAL EXAMPLE 251
Ritz vector

(13.41) Xk+1 ej := [Xk , Hk , Pk1 ] dj = Xk d1j + Pk ej ,

with
Pk ej := Hk d2j + Pk1 d3j .
Notice that P0 is an empty matrix such that the eigenvalue problem in step (8) of the
locally-optimal block preconditioned conjugate gradient method (LOBPCG), displayed in
Algorithm 13.2, has order 2q only for k = 0.
The algorithm as proposed by Knyazev [6] was designed to compute just a few eigen-
pairs and so a memory efficient implementation was not presented. For instance, in addi-
tion to Xk , Rk , Hk , Pk , the matrices M Xk , M Hk , M Pk and AXk , AHk , APk are also stored.
The resulting storage needed is prohibitive if more than a handful of eigenpairs are needed.
A more memory efficient implementation results when we iterate with blocks of width q
in the space orthogonal to the already computed eigenvectors. The computed eigenvectors
are stored in Q and neither M Q nor AQ are stored. Hence only storage for (p + 10q)n +
O(q 2 ) numbers is needed.
Here, the columns of [Xk , Hk , Pk ] may become (almost) linearly dependent leading to
ill-conditioned matrices A e and M
f in step (9) of the LOBPCG algorithm. If this is the case
we simply restart the iteration with random Xk orthogonal to the computed eigenvector
approximations. More sophisticated restarting procedures that retain Xk but modify Hk
and/or Pk were much less stable in the sense that the search space basis again became
linearly dependent within a few iterations. Restarting with random Xk is a rare occurrence
and in our experience, has little effect on the overall performance of the algorithm.

13.7 A numerical example

We again look at the determination the acoustic eigenfrequencies and modes in the in-
terior of a car, see section 1.6.3. The computations are done with the finest grid de-
picted in Fig. 1.9. We compute the smallest eigenvalue of the problem with RQMIN and
LOPCG, with preconditioning and without. The preconditioner we chose was the incom-
plete Cholesky factorization without fill-in, usually denoted IC(0). This factorization is
implemented in the Matlab routine ichol.

>> [p,e,t]=initmesh(auto);
>> [p,e,t]=refinemesh(auto,p,e,t);
>> [p,e,t]=refinemesh(auto,p,e,t);
>> p=jigglemesh(p,e,t);
>> [A,M]=assema(p,t,1,1,0);
>> whos
Name Size Bytes Class

A 1095x1095 91540 double array (sparse)

M 1095x1095 91780 double array (sparse)
e 7x188 10528 double array
p 2x1095 17520 double array
t 4x2000 64000 double array

Grand total is 26052 elements using 275368 bytes

>> n=size(A,1);
252 CHAPTER 13. RAYLEIGH QUOTIENT AND TRACE MINIMIZATION
>> R=ichol(A); % Incomplete Cholesky factorization
>> x0=rand(n,1)-.5;
>> tol=1e-6;
>> [x,rho,log0] = rqmin1(A,M,x0,tol);
>> [x,rho,log1] = rqmin1(A,M,x0,tol,R);
>> [x,rho,log2] = lopcg(A,M,x0,tol);
>> [x,rho,log3] = lopcg(A,M,x0,tol,R);
>> whos log*
Name Size Bytes Class

log0 346x3 8304 double array

log1 114x3 2736 double array
log2 879x3 21096 double array
log3 111x3 2664 double array

Grand total is 4350 elements using 34800 bytes

>> L = sort(eig(full(A),full(M)));
>> format short e, [L(1) L(2) L(n)], format

ans =

-7.5901e-13 1.2690e-02 2.6223e+02

>> k0= L(n)/L(2);

>> (sqrt(k0) - 1)/(sqrt(k0) + 1)

ans =

0.9862

>> l0=log0(end-6:end-1,2).\log0(end-5:end,2);
>> l1=log1(end-6:end-1,2).\log1(end-5:end,2);
>> l2=log2(end-6:end-1,2).\log2(end-5:end,2);
>> l3=log3(end-6:end-1,2).\log3(end-5:end,2);
>> [l0 l1 l2 l3]

ans =

0.9292 0.8271 0.9833 0.8046

0.9302 0.7515 0.9833 0.7140
0.9314 0.7902 0.9837 0.7146
0.9323 0.7960 0.9845 0.7867
0.9320 0.8155 0.9845 0.8101
0.9301 0.7955 0.9852 0.8508

>> semilogy(log0(:,1),log0(:,3)/log0(1,3),log1(:,1),log1(:,3)/log1(1,3),...
log2(:,1),log2(:,3)/log2(1,3),log3(:,1),log3(:,3)/log3(1,3),LineWidth,2)
>> legend(rqmin,rqmin + prec,lopcg,lopcg + prec)

The convergence histories in Figure 13.3 for RQMIN and LOPCG show that precon-
ditioning helps very much in reducing the iteration count.
In Figure 13.4 the convergence histories of LOBPCG for computing ten eigenvalues is
shown. In 43 iteration steps all ten eigenvalues have converged to the desired accuracy ( =
105 ). Clearly, the iteration count has been decreased drastically. Note however, that each
13.8. TRACE MINIMIZATION 253
0
10

1
10
rqmin
rqmin + prec
2
10 lopcg
lopcg + prec
3
10
norm of residual
4
10

5
10

6
10

7
10

8
10
0 100 200 300 400 500 600 700 800 900
iteration number

Figure 13.3: Convergence of variants of Rayleigh quotient minimization

iteration step requires solving ten systems of equation resulting in 430 system solves. (In
fact, if converged eigenvectors are locked, only 283 systems had to be solved.) Nevertheless,
when comparing with Fig. 13.3 one should remember that in the LOBPCG computation
ten eigenpairs have been computed. If a single eigenpair is required then a blocksize of 10
is too big, but a smaller blocksize may reduce the execution time. If a small number of
eigenvalues is desired then a blocksize equal or slightly bigger than theis number is certainly
advantageous. Not that in step (5) of Algorithm 13.2 q linear systems of equations are
solved concurrently. An efficient implementation accesses the preconditioner N only once.
The Matlab code does this naturally. A parallel implementation of LOBPCG can be
found in the software package Block Locally Optimal Preconditioned Eigenvalue Xolvers
(BLOPEX) [7].

13.8 Trace minimization

Remember that the trace of a matrix A Fnn is defined to be the sum of its diagonal
elements. Similar matrices have equal trace. Hence, by the spectral theorem 2.14, we have
n
X n
X
(13.42) trace(A) = aii = i .
i=1 i=1

The following theorem [11] generalizes the trace theorem 2.33 for the generalized eigenvalue
problem

(13.1) Ax = M x, A = A , M = M > 0.

Theorem 13.1 (Trace theorem for the generalized eigenvalue problem) Let A
and M be as in (13.1). Then,

(13.43) 1 + 2 + + p = min trace(X AX)

XFnp , X M X=Ip
254 CHAPTER 13. RAYLEIGH QUOTIENT AND TRACE MINIMIZATION
1
10

0
10

1
10

2
10

3
10

4
10

5
10

6
10
0 5 10 15 20 25 30 35 40 45

Figure 13.4: Convergence of 10 eigenvalues with LOBPCG preconditioned by IC(0)

where 1 , . . . , n are the eigenvalues of problem (13.1). Equality holds in (13.43) if and
only if the columns of the matrix X that achieves the minimum span the eigenspace cor-
responding to the smallest p eigenvalues.

Sameh and coworkers [11, 10] suggested an algorithm to exploit this property of
the trace, following the lines of Rayleigh quotient minimization. Let Xk Fnp with
Xk M Xk = Ip and
(k)
Xk AXk = k = diag(1 , . . . , p(k) ).
We want to construct the next iterate Xk+1 by setting

(13.44) Xk+1 = (Xk k )Sk

such that

(13.45) Xk+1 M Xk+1 = Ip ,
(k+1)
(13.46) Xk+1 AXk+1 = k+1 = diag(1 , . . . , p(k+1) ),

(13.47) trace(Xk+1 AXk+1 ) < trace(Xk AXk ).

Sk in (13.44) is needed to enforce the orthogonality of Xk+1 . We choose the correction k

to be orthogonal to Xk ,

(13.48) k M Xk = 0.

Similarly as in JacobiDavidson [13] this choice of k is no loss of generality. We first

assume that we have found the k that minimizes

(13.49) trace((Xk k ) A(Xk k ))

under the constraint (13.48). Let Zk+1 = Xk k be the solution of (13.49). Then, by
construction,

trace(Zk+1 AZk+1 ) trace(Xk AXk ).
13.8. TRACE MINIMIZATION 255
Furthermore,

(13.48)
Zk+1 M Zk+1 = Xk M Xk + k M k Xk M Xk = Ip .
MZ
From this it follows that Zk+1 has maximal rank and that all eigenvalues of Zk+1 k+1

are 1. Therefore, the spectral decomposition of Zk+1 M Zk+1 can be written in the form

Zk+1 M Zk+1 = U D 2 U , U U = Ip , D = diag(1 , . . . , p ), i 1.

This implies that the columns of Zk+1 U D 1 are M -orthogonal. Let the spectral decom-
position of D 1 U Zk+1
AZ
k+1 U D
1 be given by

D 1 U Zk+1

AZk+1 U D 1 = V k+1 V , V V = Ip .

Then,

(13.50) V D 1 U Zk+1

M Zk+1 U D 1 V = Ip ,
| {z }
Ip
(13.51) V D 1 U Zk+1

AZk+1 U D 1 V = k+1 .

So, if we have found Zk+1 = Xk k then Sk in (13.44) is given by

Sk = U D 1 V.

Thus, with Xk+1 = Zk+1 Sk we have

trace(Xk+1 AXk1 ) = trace(k+1 ) = trace(V D 1 U Zk+1

AZk+1 U D 1 V )
= trace(D 1 U Zk+1

AZk+1 U D 1 )
| {z }
p W
X
= wii /i2
i=1
Xp
wii
i=1
= trace(U Zk+1

AZk+1 U )
trace(Xk AXk )

Equality can only hold if k = 0.

To solve the minimization problem (13.49) we write
p
X

trace((Xk k ) A(Xk k )) = ei (Xk k ) A(Xk k )ei
i=1
p
X
= (xi di ) A(xi di )
i=1

where xi = Xk ei and di = k ei . Interestingly, these are p individual minimization

problems, one for each di !

(13.52) Minimize (xi di ) A(xi di ) subject to Xk M di = 0, i = 1, . . . , p.

256 CHAPTER 13. RAYLEIGH QUOTIENT AND TRACE MINIMIZATION
To solve (13.52) we define the functional

f (d, l) := (xi d) A(xi d) + l Xk M d.

Here, the vector l contains the Lagrange multipliers. A necessary condition for d to be a
solution of (13.52) is

d f = 0 A(xi d) + M Xk l = 0,

l f = 0 Xk M d = 0.

In matrix form this is     

A M Xk d Axi
= .
Xk M O l 0
We can collect all p equations in one,
    
A M Xk k AXk
(13.53) = .
Xk M O L O

Using the LU factorization

    
A M Xk I 0 A M Xk
=
Xk M O Xk M A1 I O Xk M A1 M Xk

we obtain
       
A M Xk k I 0 AXk AXk
= = .
O Xk M A1 M Xk L Xk M A1 I O Xk M Xk

Since Xk M Xk = Ip , L in (13.53) becomes

L = (Xk M A1 M Xk )1 .

Multiplying the first equation in (13.53) by A1 we get

k + A1 M Xk L = Xk ,

such that

Zk+1 = Xk k = A1 M Xk L = A1 M Xk (Xk M A1 M Xk )1 .

Thus, one step of the above trace minimization algorithm amounts to one step of
subspace iteration with shift = 0. This proves convergence of the algorithm to the
smallest eigenvalues of (13.1). Remember the similar equation (12.13) for the Jacobi
Davidson iteration and Remark 12.2.
Let P be the orthogonal projection onto R(M Xk ) ,

(13.54) P = I (M Xk )((M Xk ) (M Xk ))1 (M Xk ) = I M Xk (Xk M 2 Xk )1 Xk M.

Then the linear systems of equations (13.53) and

(13.55) P AP k = P AXk , Xk M k = 0,
 
k
are equivalent, i.e., they have the same solution k . In fact, let be the solution
L
of (13.53). Then, from Xk M k = 0 we get P k = k . Equation (13.55) is now obtained
13.8. TRACE MINIMIZATION 257
Algorithm 13.3 Trace minimization algorithm to compute p eigenpairs of Ax =
M x.
1: Choose random matrix V1 Rnq with V1T M V1 = Iq , q p.
2: for k = 1, 2, . . . until convergence do
3: Compute Wk = AVk and Hk := Vk Wk .
4: Compute spectral decomposition Hk = Uk k Uk ,
(k) (k) (k) (k)
with k = diag(1 , . . . , q ), 1 . . . q .
5: Compute Ritz vectors Xk = Vk Uk and residuals Rk = Wk Uk M Xk k
6: For i = 1, . . . , q solve approximatively
(k) (k) (k)
P (A i M )P di = P ri , di M Xk

by some modified PCG solver.

(k) (k)
7: Compute Vk+1 = [Xk k , Rk ], k = d1 , . . . , dq ], by a M -orthogonal modified
Gram-Schmidt procedure.
8: end for

by multiplying the first equation in (13.53) by P . On the other hand, let k be the
solution of (13.55). Since P (AP k AXk ) = 0 we must have AP k AXk = M Xk L for
some L. As Xk M k = 0 we get P k = k and thus the first equation in (13.53).
As P AP is positive semidefinite, equation (13.55) is easier to solve than equation (13.53)
which is an indefinite system of equations. (13.55) can be solved by the (preconditioned)
conjugate gradient method (PCG). The iteration has to be started by a vector z0 that
satisfies the constrains Xk M z0 . A straightforward choice is z0 = 0

reducion factor 104 102 0.5

#its A mults #its A mults #its A mults
59 6638 59 4263 77 4030

Table 13.1: The basic trace minimization algorithm (Algorithm 13.3). The inner systems
are solved by the CG scheme which is terminated such that the 2-norm of the residual is
reduced by a specified factor. The number of outer iterations (#its) and the number of
multiplications with matrix A (A mults) are listed for different residual reduction factors.

In practice, we do not solve the p linear systems

(k) (k) (k)
(13.56) P (A i M )P di = P ri , di M Xk

to high accuracy. In Table 13.1 the number of outer iterations (#its) are given and the
number of multiplications of the matrix A with a vector for various relative stopping
criteria for the inner iteration (reduction factor) [10].

Acceleration techniques
Sameh & Tong [10] investigate a number of ways to accelerate the convergence of the trace
minimization algorithm 13.3.

1. Simple shifts. Choose a shift 1 1 until the first eigenpair is found. Then proceed
with the shift 2 2 and lock the first eigenvector. In this way PCG can be used
to solve the linear systems as before.
258 CHAPTER 13. RAYLEIGH QUOTIENT AND TRACE MINIMIZATION
2. Multiple dynamic shifts. Each linear system (13.56) is solved with an individual
shift. The shift is turned on close to convergence. Since the linear systems are
indefinite, PCG has to be adapted.

3. Preconditioning. The linear systems (13.56) can be preconditioned, e.g., by a matrix

of the form M = CC where CC A is an incomplete Cholesky factorization. One
then solves
(k) (k) (k) = 0
(13.57) P (A i M ri ,
)P di = P M
Xk
d
i

with A = C 1 AC , M (k) = C d(k) , X

= C 1 M C , d k = C Xk , x(k) (k)
i = C xi ,
i i
and P = I M
X k (X
M
k
2X
k )1 X
M
k
.

In Table 13.2 results are collected for some problems in the HarwellBoeing collec-
tion [10]. These problems are diffcult because the gap ratios for the smallest eigenvalues
are extremely small due to the huge span of the spectra. Without preconditioning, none of
these problems can be solved with a reasonable cost. In the experiments, the incomplete
Cholesky factorization (IC(0)) of A was used as the preconditioner for all the matrices of
the form A B.

Problem Size Max # Block JacobiDavidson Davidson-type tracemin

inner its #its A mults time[sec] #its A mults time[sec]
BCSST08 1074 40 34 3954 4.7 10 759 0.8
BCSST09 1083 40 15 1951 2.2 15 1947 2.2
BCSST11 1473 100 90 30990 40.5 54 20166 22.4
BCSST21 3600 100 40 10712 35.1 39 11220 36.2
BCSST26 1922 100 60 21915 32.2 39 14102 19.6

Table 13.2: Numerical results for problems from the HarwellBoeing collection with four
processors (reproduced from [10])

The Davidson-type trace minimization algorithm with multiple dynamic shifts works
better than the block JacobiDavidson algorithm for three of the five problems. For the
other two, the performance for both algorithms is similar.

Bibliography
[1] P. Arbenz, U. L. Hetmaniuk, R. B. Lehoucq, and R. Tuminaro, A comparison
of eigensolvers for large-scale 3D modal analysis using AMG-preconditioned iterative
methods, Internat. J. Numer. Methods Eng., 64 (2005), pp. 204236.

[2] O. Axelsson and V. Barker, Finite Element Solution of Boundary Value Prob-
lems, Academic Press, Orlando FL, 1984.

[3] D. K. Faddeev and V. N. Faddeeva, Computational Methods of Linear Algebra,

Freeman, San Francisco, 1963.

[4] Y. T. Feng and D. R. J. Owen, Conjugate gradient methods for solving the smallest
eigenpair of large symmetric eigenvalue problems, Internat. J. Numer. Methods Eng.,
39 (1996), pp. 22092229.
BIBLIOGRAPHY 259
[5] M. R. Hestenes and E. Stiefel, Methods of conjugent gradients for solving linear
systems, J. Res. Nat. Bur. Standards, 49 (1952), pp. 409436.

[6] A. V. Knyazev, Toward the optimal preconditioned eigensolver: Locally optimal

block preconditioned conjugate gradient method, SIAM J. Sci. Comput., 23 (2001),
pp. 517541.

[7] A. V. Knyazev, M. E. Argentati, I. Lashuk, and E. E. Ovtchinnikov, Block

locally optimal preconditioned eigenvalue Xolvers (BLOPEX) in hypre and PETSc,
SIAM J. Sci. Comput., 29 (2007), pp. 22242239.

[8] D. E. Longsine and S. F. McCormick, Simultaneous Rayleighquotient mini-

mization methods for Ax = Bx, Linear Algebra Appl., 34 (1980), pp. 195234.

[9] A. Ruhe, Computation of eigenvalues and vectors, in Sparse Matrix Techniques, V. A.

Barker, ed., Lecture Notes in Mathematics 572, Berlin, 1977, Springer, pp. 130184.

[10] A. Sameh and Z. Tong, The trace minimization method for the symmetric gener-
alized eigenvalue problem, J. Comput. Appl. Math., 123 (2000), pp. 155175.

[11] A. H. Sameh and J. A. Wisniewski, A trace minimization algorithm for the gen-
eralized eigenvalue problem, SIAM J. Numer. Anal., 19 (1982), pp. 12431259.

[12] H. R. Schwarz, RayleighQuotientMinimierung mit Vorkonditionierung, in Nu-

merical Methods of Approximation Theory, Vol. 8, L. Collatz, G. Meinardus, and
G. N
urnberger, eds., vol. 81 of International Series of Numerical Mathematics (ISNM),
Basel, 1987, Birkhauser, pp. 22945.

[13] G. L. G. Sleijpen and H. A. van der Vorst, JacobiDavidson method, in Tem-

plates for the solution of Algebraic Eigenvalue Problems: A Practical Guide, Z. Bai,
J. Demmel, J. Dongarra, A. Ruhe, and H. van der Vorst, eds., SIAM, Philadelphia,
PA, 2000, pp. 238246.