Numerical Analysis
Numerical Analysis
1 General introduction
The overall goal of the field of numerical analysis is the
design and analysis of techniques to give approximate but
accurate solutions to hard problems, the variety of which
is suggested by the following:
1
2 2 GENERATION AND PROPAGATION OF ERRORS
great mathematicians of the past were preoccupied by nu- the same manner as for an iterative method.
merical analysis, as is obvious from the names of im-
portant algorithms like Newton’s method, Lagrange in-
terpolation polynomial, Gaussian elimination, or Euler’s 1.3 Discretization
method.
Furthermore, continuous problems must sometimes be
To facilitate computations by hand, large books were pro-
replaced by a discrete problem whose solution is known
duced with formulas and tables of data such as interpo-
to approximate that of the continuous problem; this pro-
lation points and function coefficients. Using these ta-
cess is called discretization. For example, the solution of
bles, often calculated out to 16 decimal places or more
a differential equation is a function. This function must
for some functions, one could look up values to plug into
be represented by a finite amount of data, for instance by
the formulas given and achieve very good numerical esti-
its value at a finite number of points at its domain, even
mates of some functions. The canonical work in the field
though this domain is a continuum.
is the NIST publication edited by Abramowitz and Ste-
gun, a 1000-plus page book of a very large number of
commonly used formulas and functions and their values
at many points. The function values are no longer very 2 Generation and propagation of
useful when a computer is available, but the large listing errors
of formulas can still be very handy.
The mechanical calculator was also developed as a tool The study of errors forms an important part of numerical
for hand computation. These calculators evolved into analysis. There are several ways in which error can be
electronic computers in the 1940s, and it was then found introduced in the solution of the problem.
that these computers were also useful for administrative
purposes. But the invention of the computer also influ-
enced the field of numerical analysis, since now longer 2.1 Round-off
and more complicated calculations could be done.
Round-off errors arise because it is impossible to repre-
sent all real numbers exactly on a machine with finite
1.2 Direct and iterative methods memory (which is what all practical digital computers
are).
Direct methods compute the solution to a problem in a
finite number of steps. These methods would give the
precise answer if they were performed in infinite pre- 2.2 Truncation and discretization error
cision arithmetic. Examples include Gaussian elimina-
tion, the QR factorization method for solving systems of Truncation errors are committed when an iterative
linear equations, and the simplex method of linear pro- method is terminated or a mathematical procedure is ap-
gramming. In practice, finite precision is used and the proximated, and the approximate solution differs from
result is an approximation of the true solution (assuming the exact solution. Similarly, discretization induces a
stability). discretization error because the solution of the discrete
problem does not coincide with the solution of the contin-
In contrast to direct methods, iterative methods are not
expected to terminate in a finite number of steps. Start- uous problem. For instance, in the iteration in the sidebar
to compute the solution of 3x3 + 4 = 28 , after 10 or so
ing from an initial guess, iterative methods form succes-
sive approximations that converge to the exact solution iterations, we conclude that the root is roughly 1.99 (for
example). We therefore have a truncation error of 0.01.
only in the limit. A convergence test, often involving the
residual, is specified in order to decide when a sufficiently Once an error is generated, it will generally propagate
accurate solution has (hopefully) been found. Even us- through the calculation. For instance, we have already
ing infinite precision arithmetic these methods would not noted that the operation + on a calculator (or a computer)
reach the solution within a finite number of steps (in gen- is inexact. It follows that a calculation of the type is even
eral). Examples include Newton’s method, the bisection more inexact.
method, and Jacobi iteration. In computational matrix What does it mean when we say that the truncation error is
algebra, iterative methods are generally needed for large created when we approximate a mathematical procedure?
problems. We know that to integrate a function exactly requires one
Iterative methods are more common than direct methods to find the sum of infinite trapezoids. But numerically
in numerical analysis. Some methods are direct in prin- one can find the sum of only finite trapezoids, and hence
ciple but are usually used as though they were not, e.g. the approximation of the mathematical procedure. Sim-
GMRES and the conjugate gradient method. For these ilarly, to differentiate a function, the differential element
methods the number of steps needed to obtain the exact approaches zero but numerically we can only choose a fi-
solution is so large that an approximation is accepted in nite value of the differential element.
3
2.3 Numerical stability and well-posed catastrophic cancelation) has a huge effect
problems on the results, even though both functions are
equivalent, as shown below
Numerical stability is an important notion in numerical (√ √ )
analysis. An algorithm is called numerically stable if an f (x) = x x + 1 − x
√ √
error, whatever its cause, does not grow to be much larger (√ √ ) x+1+ x
during the calculation. This happens if the problem is =x x+1− x √ √
x+1+ x
well-conditioned, meaning that the solution changes by √ √
( x + 1)2 − ( x)2
only a small amount if the problem data are changed by =x √ √
a small amount. To the contrary, if a problem is ill- x+1+ x
conditioned, then any small error in the data will grow x+1−x
= x√ √
to be a large error. x+1+ x
Both the original problem and the algorithm used to 1
= x√ √
solve that problem can be well-conditioned and/or ill- x+1+ x
conditioned, and any combination is possible. x
=√ √
x+1+ x
So an algorithm that solves a well-conditioned problem
may be either numerically stable or numerically unstable. = g(x)
An art of numerical analysis is to find a stable algorithm
for solving a well-posed mathematical problem. For in- The desired value, computed using infinite pre-
stance, computing the square root of 2 (which is roughly cision, is 11.174755...
1.41421) is a well-posed problem. Many algorithms solve
this√problem by starting with an initial approximation x0 • The example is a modification of one taken from
to 2 , for instance x0 = 1.4, and then computing im- Mathew; Numerical methods using Matlab, 3rd ed.
proved guesses x1 , x2 , etc. One such method is the fa-
mous Babylonian method, which is given by xk₊₁ = xk/2
+ 1/xk. Another method, which we will call Method X,
is given by xk₊₁ = (xk2 − 2)2 + xk.[3] We have calculated
3 Areas of study
a few iterations of each scheme in table form below, with
initial guesses x0 = 1.4 and x0 = 1.42. The field of numerical analysis includes many sub-
disciplines. Some of the major ones are:
Observe that the Babylonian method converges quickly
regardless of the initial guess, whereas Method X con-
verges extremely slowly with initial guess x0 = 1.4 and 3.1 Computing values of functions
diverges for initial guess x0 = 1.42. Hence, the Baby-
lonian method is numerically stable, while Method X is One of the simplest problems is the evaluation of a func-
numerically unstable. tion at a given point. The most straightforward approach,
of just plugging in the number in the formula is some-
Numerical stability is affected by the number times not very efficient. For polynomials, a better ap-
of the significant digits the machine keeps on, proach is using the Horner scheme, since it reduces the
if we use a machine that keeps only the four necessary number of multiplications and additions. Gen-
most significant decimal digits, a good exam- erally, it is important to estimate and control round-off
ple on loss of significance is given by these two errors arising from the use of floating point arithmetic.
equivalent functions
(√ √ )
f (x) = x x + 1 − x and g(x) =
√ x √ .
x+1+ x
3.2 Interpolation, extrapolation, and re-
If we compare the results of gression
(√ √ )
f (500) = 500 501 − 500 = 500 (22.38 − 22.36) =Interpolation
500(0.02) =solves
10 the following problem: given the
value of some unknown function at a number of points,
and what value does that function have at some other point
500
g(500) = √ √ between the given points?
501 + 500
500 Extrapolation is very similar to interpolation, except that
= now we want to find the value of the unknown function at
22.38 + 22.36
500 a point which is outside the given points.
= = 11.17
44.74 Regression is also similar, but it takes into account that
by looking to the two results above, we real- the data is imprecise. Given some points, and a measure-
ize that loss of significance (caused here by ment of the value of some function at these points (with
4 4 SOFTWARE
an error), we want to determine the unknown function. The method of Lagrange multipliers can be used to re-
The least squares-method is one popular way to achieve duce optimization problems with constraints to uncon-
this. strained optimization problems.
4 Software
3.5 Optimization
Main articles: List of numerical analysis software and
Main article: Mathematical optimization Comparison of numerical analysis software
Optimization problems ask for the point at which a given Since the late twentieth century, most algorithms are im-
function is maximized (or minimized). Often, the point plemented in a variety of programming languages. The
also has to satisfy some constraints. Netlib repository contains various collections of software
The field of optimization is further split in several sub- routines for numerical problems, mostly in Fortran and C.
fields, depending on the form of the objective function Commercial products implementing many different nu-
and the constraint. For instance, linear programming merical algorithms include the IMSL and NAG libraries;
deals with the case that both the objective function and a free alternative is the GNU Scientific Library.
the constraints are linear. A famous method in linear pro- There are several popular numerical computing applica-
gramming is the simplex method. tions such as MATLAB, TK Solver, S-PLUS, LabVIEW,
5
and IDL as well as free and open source alternatives • Higham, Nicholas J. (1996). Accuracy and Stability
such as FreeMat, Scilab, GNU Octave (similar to Mat- of Numerical Algorithms (Society for Industrial and
lab), IT++ (a C++ library), R (similar to S-PLUS) and Applied Mathematics, ISBN 0-89871-355-2).
certain variants of Python. Performance varies widely:
while vector and matrix operations are usually fast, scalar • Hildebrand, F. B. (1974). Introduction to Numeri-
loops may vary in speed by more than an order of cal Analysis (2nd ed.). McGraw-Hill. ISBN 0-07-
magnitude.[5][6] 028761-9.
Many computer algebra systems such as Mathematica • Leader, Jeffery J. (2004). Numerical Analysis and
also benefit from the availability of arbitrary precision Scientific Computation. Addison Wesley. ISBN 0-
arithmetic which can provide more accurate results. 201-73499-0.
Also, any spreadsheet software can be used to solve sim- • Wilkinson, J.H. (1965). The Algebraic Eigenvalue
ple problems relating to numerical analysis. Problem (Clarendon Press).
• Numerical Recipes
8 External links
• Symbolic-numeric computation
Journals
9.2 Images
• File:Folder_Hexagonal_Icon.svg Source: https://upload.wikimedia.org/wikipedia/en/4/48/Folder_Hexagonal_Icon.svg License: Cc-by-
sa-3.0 Contributors: ? Original artist: ?
• File:Internet_map_1024.jpg Source: https://upload.wikimedia.org/wikipedia/commons/d/d2/Internet_map_1024.jpg License: CC BY
2.5 Contributors: Originally from the English Wikipedia; description page is/was here. Original artist: The Opte Project
• File:LemonadeJuly2006.JPG Source: https://upload.wikimedia.org/wikipedia/commons/2/28/LemonadeJuly2006.JPG License: CC-
BY-SA-3.0 Contributors: ? Original artist: ?
• File:Linear-regression.svg Source: https://upload.wikimedia.org/wikipedia/commons/1/12/Linear-regression.svg License: Public do-
main Contributors: Own work Original artist: Qef
• File:Portal-puzzle.svg Source: https://upload.wikimedia.org/wikipedia/en/f/fd/Portal-puzzle.svg License: Public domain Contributors: ?
Original artist: ?
• File:Schumacher_(Ferrari)_in_practice_at_USGP_2005.jpg Source: https://upload.wikimedia.org/wikipedia/commons/9/9d/
Schumacher_%28Ferrari%29_in_practice_at_USGP_2005.jpg License: CC BY-SA 2.0 Contributors: ? Original artist: ?
• File:Text_document_with_red_question_mark.svg Source: https://upload.wikimedia.org/wikipedia/commons/a/a4/Text_document_
with_red_question_mark.svg License: Public domain Contributors: Created by bdesham with Inkscape; based upon Text-x-generic.svg
from the Tango project. Original artist: Benjamin D. Esham (bdesham)
• File:Wikibooks-logo.svg Source: https://upload.wikimedia.org/wikipedia/commons/f/fa/Wikibooks-logo.svg License: CC BY-SA 3.0
Contributors: Own work Original artist: User:Bastique, User:Ramac et al.
• File:Wikiquote-logo.svg Source: https://upload.wikimedia.org/wikipedia/commons/f/fa/Wikiquote-logo.svg License: Public domain
Contributors: Own work Original artist: Rei-artur
• File:Wind-particle.png Source: https://upload.wikimedia.org/wikipedia/commons/e/ef/Wind-particle.png License: Public domain Con-
tributors: http://en.wikipedia.org/wiki/File:Wind-particle.png Original artist: Loisel
• File:Ybc7289-bw.jpg Source: https://upload.wikimedia.org/wikipedia/commons/0/0b/Ybc7289-bw.jpg License: CC BY 2.5 Contribu-
tors: Own work Original artist: Bill Casselman