Lecture Notes 8: Dynamic Optimization

Part 2: Optimal Control

Peter J. Hammond

2018 September 21st; typeset from optControl18.tex

University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 1 of 44

Outline

Introduction
A Basic Optimal Growth Problem
Digression: Sufficient Conditions for Static Optimality

The Maximum Principle

From Lagrangians to Hamiltonians
Example: A Macroeconomic Quadratic Control Problem

Sufficient Conditions for Optimality

Finite Horizon Case
Infinite Horizon Case

Discounting and the Current Value Hamiltonian

Maximum Principle Revisited
Application to an Optimal Growth Problem

University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 2 of 44

Statement of Basic Optimal Growth Problem
A consumption path C is a mapping [t0 , t1 ] 3 t 7→ C (t) ∈ R+ .
A capital path K is a mapping [t0 , t1 ] 3 t 7→ K (t) ∈ R+ .
Given K (0) at time 0, the benevolent planner’s objective
is to choose C in order to maximize
Z t1
J(C) := e −rt u(C (t))dt
t0

subject to the continuum of equality constraints

C (t) = f (K (t)) − K̇ (t)

Introduce the Lagrange multiplier path p

as a mapping [t0 , t1 ] 3 t 7→ p(t) ∈ R+ .
Use it to define the Lagrangian integral
Z t1 Z t1
Lp (C) = e −rt u(C (t))dt − p(t)[C (t) − f (K (t)) + K̇ (t)]dt
t0 t0

University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 3 of 44

Integrate by Parts
So we have the “Lagrangian”
Z t1 Z t1
−rt
Lp (C) = e u(C (t))dt − p(t)[C (t) − f (K (t)) + K̇ (t)]dt
t0 t0

Integrating the last term by parts yields

Z t1 Z t1
t1
− p(t)K̇ (t)dt = − t0 p(t)K (t) + ṗ(t) K (t)dt
t0 t0

Hence
Z t1
e −rt u(C ) + ṗ K − p C + p f (K ) dt − t1
 
Lp (C) = t0 p(t)K (t)
t0

For the moment we ignore the last “endpoint terms”,

and consider just the integral
Z t1
 −rt 
Ip (C) := e u(C ) + ṗ K − p C + p f (K ) dt
t0

University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 4 of 44

Maximizing the Integrand

Evidently the two paths t 7→ C (t) and t 7→ K (t)

jointly maximize the integral Ip (C), with p fixed,
if and only if, for almost all t ∈ (t0 , t1 ),
the pair (C (t), K (t)) jointly maximizes w.r.t. C and K the
integrand
e −rt u(C ) + ṗ K − p C + p f (K )
The first-order conditions for maximizing this integrand,
at any time t ∈ (t0 , t1 ), are found by differentiating partially:
1. w.r.t. C (t) to obtain e −rt u 0 (C (t)) = p(t);
2. w.r.t. K (t) to obtain ṗ(t) = −p(t) f 0 (K (t));
There is also the equality constraint K̇ (t) = f (K (t)) − C (t).

University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 5 of 44

Outline

Introduction
A Basic Optimal Growth Problem
Digression: Sufficient Conditions for Static Optimality

The Maximum Principle

From Lagrangians to Hamiltonians
Example: A Macroeconomic Quadratic Control Problem

Sufficient Conditions for Optimality

Finite Horizon Case
Infinite Horizon Case

Discounting and the Current Value Hamiltonian

Maximum Principle Revisited
Application to an Optimal Growth Problem

University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 6 of 44

Statement of Sufficient Conditions
Consider the static problem of maximizing the objective function
Rn ⊇ D 3 x 7→ f (x) ∈ R
subject to the vector constraint g(x) 5 a ∈ Rm .
Definition
The pair (p, x∗ ) ∈ Rm × Rn
jointly satisfies complementary slackness just in case:

(i) p> = 0; (ii) g(x∗ ) 5 a; (iii) p> [g(x∗ ) − a] = 0

These are generally summarized as p> = 0, g(x∗ ) 5 a (comp).

Theorem
Suppose that x∗ ∈ Rn is a global maximum over the domain D
of the Lagrangian function Lp (x) = f (x) − p> [g(x) − a]
where (p, x∗ ) ∈ Rm × Rn
jointly satisfy the complementary slackness conditions.
Then x∗ is a global maximum of f (x) subject to g(x) 5 a.
University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 7 of 44
Proof of Sufficient Conditions
Proof.
By definition of the Lagrangian Lp (x) = f (x) − p> [g(x) − a],
for every x ∈ D one has

f (x) − f (x∗ ) = Lp (x) + p> [g(x) − a] − Lp (x∗ ) − p> [g(x∗ ) − a]

By hypothesis one has Lp (x) ≤ Lp (x∗ ) for all x ∈ D, so

f (x) − f (x∗ ) ≤ p> [g(x) − a] − p> [g(x∗ ) − a] = p> [g(x) − g(x∗ )]

But the complementary slackness conditions

p> = 0, g(x∗ ) 5 a (comp)

imply that for any x ∈ D satisfying the constraint g(x) 5 a

one has p> g(x) ≤ p> a, whereas p> g(x∗ ) = p> a.
Hence f (x) − f (x∗ ) ≤ p> [g(x) − g(x∗ )] ≤ p> a − p> a = 0.
University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 8 of 44
A Cheap Result on Necessary Conditions

Recall that we are considering the problem of maximizing f (x)

subject to g(x) 5 a.
Suppose we know that any solution x∗ must be unique.
This will be the case, for example, if:
1. the objective function Rn 3 x 7→ f (x) ∈ R is strictly concave;
2. and each component function Rn 3 x 7→ gj (x) ∈ R
of the vector function Rn 3 x 7→ g(x) ∈ Rm is convex.
Suppose that the pair (p, x∗ ) ∈ Rm × Rn jointly satisfy
the sufficient conditions of maximizing the Lagrangian
while also meeting the complementary slackness conditions.
Then it is necessary that the only possible maximum
satisfy these sufficient conditions!

University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 9 of 44

Outline

Introduction
A Basic Optimal Growth Problem
Digression: Sufficient Conditions for Static Optimality

The Maximum Principle

From Lagrangians to Hamiltonians
Example: A Macroeconomic Quadratic Control Problem

Sufficient Conditions for Optimality

Finite Horizon Case
Infinite Horizon Case

Discounting and the Current Value Hamiltonian

Maximum Principle Revisited
Application to an Optimal Growth Problem

University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 10 of 44

Statement of General Problem
Given the time interval [t0 , t1 ] ⊂ R,
consider the general one-variable optimal control problem
of choosing paths:
1. [t0 , t1 ] 3 t 7→ x(t) ∈ R of states;
2. [t0 , t1 ] 3 t 7→ u(t) ∈ R of controls.
The objective functional is taken to be the integral
Z t1
f (t, x(t), u(t)) dt
t0

We fix the initial state x(t0 ) = x0 , where x0 is given.

We leave the terminal state x(t1 ) free.
Finally, we impose the dynamic constraint ẋ = g (t, x, u)
at every time t ∈ [t0 , t1 ].

University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 11 of 44

The Lagrangian Integral
Consider the path [t0 , t1 ] 3 t 7→ p(t) ∈ R
of a single costate variable or shadow price p.
Here p(t) is the Lagrange multiplier
associated with the dynamic constraint at time t.
Then, after dropping the time argument from p, x and u,
the associated “Lagrangian integral” is
Z t1 Z t1
L= f (t, x, u) dt − p[ẋ − g (t, x, u)] dt
t0 t0

d
Because dt p x = ṗ x + p ẋ, integrating by parts
R t1 Rt
gives t0 p ẋ dt = − t01 ṗ x dt + tt10 p x and so
Z t1
t1
L= [f (t, x, u) + ṗ x + p g (t, x, u)] dt − t0 p x
t0

University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 12 of 44

The Hamiltonian

Definition Rt
For the problem of maximizing t01 f (t, x, u) dt
subject to ẋ = g (t, x, u),
the Hamiltonian function is defined as

H(t, x, u, p) := f (t, x, u) + p g (t, x, u)

With this definition, the integral part of the Lagrangian, which is

Z t1
[f (t, x, u) + ṗ x + p g (t, x, u)] dt
t0
R t1
can be written as t0 [H(t, x, u, p) + ṗ x] dt.

University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 13 of 44

The Maximum Principle

Recall the definition H(t, x, u, p) := f (t, x, u) + p g (t, x, u).

Definition
According to the maximum principle, for a.e. t ∈ [t0 , t1 ],
an optimal control should satisfy

u ∗ (t) ∈ arg max H(t, x, u, p) where x = x(t) and p = p(t)

u

Moreover the co-state variable p(t) should evolve

according to the vector differential equation

ṗ = −Hx0 (t, x, u, p)

where Hx0 (t, x, u, p) denotes the partial gradient vector

of the Hamiltonian H w.r.t. the state vector x.

University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 14 of 44

An Extended Maximum Principle
Definition
Define the extended Hamiltonian

H̃(t, x, u, p) := H(t, x, u, p) + ṗ x = f (t, x, u) + p g (t, x, u) + ṗ x

According to the extended maximum principle,

for a.e. (almost every) time t ∈ [t0 , t1 ], one should have

(u ∗ (t), x ∗ (t)) ∈ arg max H̃(t, x, u, p(t))

(u,x)

Remark
The first-order conditions for maximizing H̃(t, x, u, p) include

ṗ = −fx0 (t, x, u) − pgx0 (t, x, u) = −Hx0 (t, x, u, p)

as required by the maximum principle.

University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 15 of 44
Outline

Introduction
A Basic Optimal Growth Problem
Digression: Sufficient Conditions for Static Optimality

The Maximum Principle

From Lagrangians to Hamiltonians
Example: A Macroeconomic Quadratic Control Problem

Sufficient Conditions for Optimality

Finite Horizon Case
Infinite Horizon Case

Discounting and the Current Value Hamiltonian

Maximum Principle Revisited
Application to an Optimal Growth Problem

University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 16 of 44

Statement of the Problem
Let c > 0 denote an adjustment cost parameter.
Consider the problem of choosing the path t 7→ (u(t), x(t)) ∈ R2
RT
in order to minimize the quadratic integral 0 (x 2 + cu 2 ) dt
subject to the dynamic constraint ẋ = u,
as well as the initial condition x(0) = x0
and the terminal condition allowing x(T ) to be chosen freely.
The associated Hamiltonian is

H = −x 2 − cu 2 + p u

with a minus sign to convert the minimization problem

into a maximization problem.
The associated extended Hamiltonian is

H̃ = −x 2 − cu 2 + p u + ṗ x

University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 17 of 44

Example: First-Order Conditions

The first-order conditions for maximizing, at any time t ∈ [0, T ],

either the Hamiltonian or the extended Hamiltonian,
include 0 = Hu0 = H̃u0 = −2c u + p.
Either of these two equivalent conditions implies that u ∗ = p/2c.
A second first-order condition
for maximizing the extended Hamiltonian is ṗ = −Hx0 = 2x,
which is also the co-state differential equation.
Combining this with the dynamic constraint ẋ = u
leads to the following coupled pair of differential equations:

ṗ = −Hx0 = 2x and ẋ = u ∗ = p/2c

University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 18 of 44

Example: Solving the Coupled Pair

In order to solve the coupled pair

ṗ = 2x and ẋ = p/2c

I differentiate the first equation w.r.t. t to obtain p̈ = 2ẋ;

I substitute in the second equation to obtain p̈ = 2ẋ = p/c.
We need to consider the second-order differential equation

p̈ = p/c

in p, whose associated characteristic equation is λ2 − 1/c = 0.

The two roots are λ1,2 = ±c −1/2 = ±r where r := c −1/2 .
The general solution of this homogeneous equation
is p = Ae rt + Be −rt for arbitrary constants A and B.

University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 19 of 44

Explicit Solution
In addition to p = Ae rt + Be −rt with r := c −1/2 and ṗ = 2x,
we also have ẋ = p/2c, along with the initial condition x(0) = x0
and the terminal condition p(T ) = 0.
This terminal condition implies Ae rT + Be −rT = 0,
from which one obtains B = −Ae 2rT .
Also differentiating p = Ae rt + Be −rt w.r.t. t
implies ṗ = r (Ae rt − Be −rt ).
At time t = 0 one has ṗ(0) = 2x0 and so r (A − B) = 2x0 .
Substituting B = −Ae 2rT gives r (A + Ae 2rT ) = 2x0 ,
so A = 2x0 /r (1 + e 2rT ) = 2x0 e −rT /r (e −rT + e rT )
implying that B = −2x0 e rT /r (e −rT + e rT ).
So p = Ae rt + Be −rt = 2x0 (e −r (T −t) − e r (T −t) )/r (e −rT + e rT )
and x = ṗ/2 = x0 (e −r (T −t) + e r (T −t) )/(e −rT + e rT ).
Also u = ẋ = rx0 (e −r (T −t) − e r (T −t) )/(e −rT + e rT ).
University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 20 of 44
The Case of an Infinite Horizon
We multiply both numerator and denominator by e −rT
in each equation to transform the explicit solution:

2x0 e −r (T −t) − e r (T −t) 2x0 e −r (2T −t) − e −rt

   
p(t) = =
r [e −rT + e rT ] r (e −2rT + 1)
 −r (T −t)
+ e r (T −t) x0 e −r (2T −t) + e −rt
  
x0 e
x(t) = =
r (e −rT + e rT ) r (e −2rT + 1)
 −r (T −t)
− e r (T −t) x0 e −r (2T −t) − e −rt
  
x0 e
u(t) = =
e −rT + e rT e −2rT + 1
Taking the limit as T → ∞, one has p(t) → −2x0 e −rt /r .
Similarly x(t) = 12 ṗ → x0 e −rt , and u(t) = ẋ(t) → −rx0 e −rt .
Finally, (p(t), x(t), u(t)) → (0, 0, 0) as t → ∞.
See page 311 of FMEA.

University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 21 of 44

Outline

Introduction
A Basic Optimal Growth Problem
Digression: Sufficient Conditions for Static Optimality

The Maximum Principle

From Lagrangians to Hamiltonians
Example: A Macroeconomic Quadratic Control Problem

Sufficient Conditions for Optimality

Finite Horizon Case
Infinite Horizon Case

Discounting and the Current Value Hamiltonian

Maximum Principle Revisited
Application to an Optimal Growth Problem

University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 22 of 44

Mangasarian and Arrow’s Sufficient Conditions
At any fixed time t, let (x∗ (t), u∗ (t)) be a stationary point
w.r.t. (x, u) of the extended Hamiltonian

H̃(t, x, u, p(t)) := H(t, x, u, p(t)) + ṗ> (t) x

That is, suppose that the respective partial gradients satisfy

Hu0 (t, x∗ (t), u∗ (t), p(t)) = 0 and ṗ(t) = −Hx0 (t, x∗ (t), u∗ (t), p(t))

Here are two alternative sufficient conditions for (x∗ (t), u∗ (t))
to maximize the extended Hamiltonian.
1. See FMEA Theorem 9.7.1, due to Mangasarian.
Suppose that (x, u) 7→ H(t, x, u, p(t)) is concave,
which implies that (x, u) 7→ H̃(t, x, u, p(t)) is also concave.
2. See FMEA Theorem 9.7.2, due to Arrow.
Define Ĥ(t, x, p(t)) := maxu H(t, x, u, p(t)),
and suppose that x 7→ Ĥ(t, x, p(t)) is concave.
University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 23 of 44
Sufficient Conditions
Consider the single variable problem
of choosing the paths t 7→ (x(t), u(t)) ∈ R2
RT
in order to maximize 0 f (t, x, u) dt
subject to ẋ ≤ g (t, x, u) (all t ∈ [0, T ])
as well as x(0) ≤ x0 , x(T ) ≥ xT .
The extended Hamiltonian is
H̃(t, x, u, p) = f (t, x, u) + p g (t, x, u) + ṗ x
Suppose that for all t ∈ [0, T ] the path t 7→ (x ∗ (t), u ∗ (t)) ∈ R2
satisfies the extended maximization condition
(x ∗ (t), u ∗ (t)) ∈ arg max H̃(t, x, u, p(t))
x,u

as well as the three complementary slackness conditions

1. p(t) ≥ 0, ẋ ∗ (t) ≤ g (t, x ∗ (t), u ∗ (t)) (comp) (all t ∈ [0, T ]);
2. p(0) ≥ 0, x ∗ (0) ≤ x0 (comp);
3. p(T ) ≥ 0, x ∗ (T ) ≥ xT (comp).
University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 24 of 44
Proof of Sufficiency, I
Consider any alternative feasible path t 7→ (x(t), u(t))
satisfying all the constraints.
RT
Define D(x, u) := 0 [f (t, x(t), u(t)) − f (t, x ∗ (t), u ∗ (t))] dt.
After dropping the time arguments from x(t), u(t), x ∗ (t), u ∗ (t),
this difference D(x, u) equals
Z T nh i
H̃(t, x, u, p) − p g (t, x, u) − ṗ x
0
h io
− H̃(t, x ∗ , u ∗ , p) − p g (t, x ∗ , u ∗ ) − ṗ x ∗ dt
The maximization hypothesis implies that for all t ∈ [0, T ] one has
H̃(t, x(t), u(t), p(t)) ≤ H̃(t, x ∗ (t), u ∗ (t), p(t))
Also, together with feasibility and non-negativity of prices,
the complementary slackness conditions imply that
p(t) g (t, x(t), u(t)) ≥ p(t) ẋ(t);
p(t) g (t, x ∗ (t), u ∗ (t)) = p(t) ẋ ∗ (t)
University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 25 of 44
Proof of Sufficiency, II
These equalities and inequalities imply that
RT
D(x, u) ≤ 0 [−p ẋ − ṗ x + p ẋ ∗ + ṗ x ∗ ] dt
RT d
= 0 dt [−p(t) x(t) + p(t) x ∗ (t)] dt
= −p(T ) [x(T ) − x ∗ (T )] + p(0) [x(0) − x ∗ (0)]

But, together with feasibility and non-negativity of prices,

the second and third complementary slackness conditions
imply that
p(T ) x(T ) ≥ p(T ) xT ; p(T ) x ∗ (T ) = p(T ) xT ;
p(0) x(0) ≤ p(0) x0 ; p(0) x ∗ (0) = p(0) x0 .

It follows that
Z T
D(x, u) := [f (t, x(t), u(t)) − f (t, x ∗ (t), u ∗ (t))] dt ≤ 0
0

So the path t 7→ (x ∗ (t), u ∗ (t)) is optimal.

University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 26 of 44
Outline

Introduction
A Basic Optimal Growth Problem
Digression: Sufficient Conditions for Static Optimality

The Maximum Principle

From Lagrangians to Hamiltonians
Example: A Macroeconomic Quadratic Control Problem

Sufficient Conditions for Optimality

Finite Horizon Case
Infinite Horizon Case

Discounting and the Current Value Hamiltonian

Maximum Principle Revisited
Application to an Optimal Growth Problem

University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 27 of 44

The Infinite Horizon Problem
We consider the problem of choosing [0, ∞) 3 t 7→ (x(t), u(t))
to maximize the infinite horizon objective functional
Z ∞
f (t, x(t), u(t)) dt
0
subject to ẋ = g (t, x, u) at every time t ∈ [0, ∞),
as well as x(0) = x0 , where x0 is given.
As before, the extended maximum principle suggests
looking for a path [0, ∞) 3 t 7→ p(t) of co-state variables,
as well as a path [0, ∞) 3 t 7→ (x ∗ (t), u ∗ (t))
of the state and control variables
which maximizes the extended Hamiltonian
H̃(t, x, u, p) := f (t, x, u) + p(t) g (t, x, u) + ṗ(t) x
— i.e., for (almost) all t ∈ [0, ∞) one has
(x ∗ (t), u ∗ (t)) ∈ arg max H̃(t, x, u, p)
(u,x)

University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 28 of 44

Implications of the Extended Maximum Principle, I
Consider any alternative feasible path t 7→ (x(t), u(t))
satisfying all the constraints.
We start by repeating our earlier argument for a finite horizon.
RT
Define D T (x, u) := 0 [f (t, x(t), u(t)) − f (t, x ∗ (t), u ∗ (t))] dt.
After dropping the time arguments from x(t), u(t), x ∗ (t), u ∗ (t),
this difference D T (x, u) equals
Z T nh i
H̃(t, x, u, p) − p g (t, x, u) − ṗ x
0
h io
− H̃(t, x ∗ , u ∗ , p) − p g (t, x ∗ , u ∗ ) − ṗ x ∗ dt

The extended maximum principle

implies that for all t ∈ [0, T ] one has

H̃(t, x(t), u(t), p(t)) ≤ H̃(t, x ∗ (t), u ∗ (t), p(t))

University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 29 of 44

Implications of the Extended Maximum Principle, II
Arguing as before, from (x ∗ (t), u ∗ (t)) ∈ arg max(u,x) H̃(t, x, u, p)
where H̃(t, x, u, p) := f (t, x, u) + p(t) g (t, x, u) + ṗ(t) x,
it follows that for all finite T the difference D T (x, u) satisfies
RT
D T (x, u) := 0 [f (t, x(t), u(t)) − f (t, x ∗ (t), u ∗ (t))] dt
R T nh i
= 0 H̃(t, x, u, p) − p g (t, x, u) − ṗ x
h io
− H̃(t, x ∗ , u ∗ , p) − p g (t, x ∗ , u ∗ ) − ṗ x ∗ dt
RT h i
= 0 H̃(t, x, u, p) − H̃(t, x ∗ , u ∗ , p) dt
RT
− 0 [p g (t, x, u) + ṗ x − p g (t, x ∗ , u ∗ ) − ṗ x ∗ ] dt
RT
≤ − 0 [p ẋ + ṗ x − p ẋ ∗ − ṗ x ∗ ] dt
RT d
= − 0 dt [p x − p x ∗ ] dt
= −p(T ) [x(T ) − x ∗ (T )] + p(0) [x(0) − x ∗ (0)]
= p(T ) [x ∗ (T ) − x(T )] given that x(0) = x ∗ (0) = x0
University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 30 of 44
A Transversality Condition
Consider the transversality condition

lim sup p(T ) [x ∗ (T ) − x(T )] = 0

T →∞

If this were satisfied, it would imply that

0 ≥ lim supT →∞ D T (x, u)

RT
= 0 [f (t, x(t), u(t)) − f (t, x ∗ (t), u ∗ (t))] dt

In the case when

Z T Z ∞
f (t, x ∗ (t), u ∗ (t)) dt → f (t, x ∗ (t), u ∗ (t)) dt
0 0

as T → ∞, it would imply that

Z T Z ∞
lim sup f (t, x(t), u(t)) dt ≤ f (t, x ∗ (t), u ∗ (t)) dt
T →∞ 0 0

University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 31 of 44

Malinvaud’s Transversality Condition

In many economic contexts, feasibility requires that, for all t,

one has both x(t) ≥ 0 and ẋ(t) ≤ g (t, x(t), u(t)).
Then, since p(t) ≥ 0,
for any alternative feasible path x(t) and any terminal time T ,
one has p(T ) [x ∗ (T ) − x(T )] ≤ p(T ) x ∗ (T ).
Definition
The Malinvaud transversality condition
is that p(T ) x ∗ (T ) → 0 as T → ∞.
When this Malinvaud transversality condition is satisfied, evidently

lim sup p(T ) [x ∗ (T ) − x(T )] ≤ lim sup p(T ) x ∗ (T ) = 0

T →∞ T →∞

Hence, the general transversality condition is also satisfied.

University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 32 of 44

Outline

Introduction
A Basic Optimal Growth Problem
Digression: Sufficient Conditions for Static Optimality

The Maximum Principle

From Lagrangians to Hamiltonians
Example: A Macroeconomic Quadratic Control Problem

Sufficient Conditions for Optimality

Finite Horizon Case
Infinite Horizon Case

Discounting and the Current Value Hamiltonian

Maximum Principle Revisited
Application to an Optimal Growth Problem

University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 33 of 44

A Problem with Exponential Discounting
Consider the general problem of choosing paths:
1. [t0 , t1 ] 3 t 7→ x(t) ∈ R of states;
2. [t0 , t1 ] 3 t 7→ u(t) ∈ R of controls.
The objective functional is taken to be the integral
Z t1
e −rt f (x(t), u(t)) dt
t0

where: (i) f is independent of t;

(ii) there is a constant discount rate r
and associated exponential discount factor e −rt .
Assume too that the dynamic constraint is ẋ = g (x, u),
at every time t ∈ [t0 , t1 ], where g is independent of t.
Fix the initial state x(t0 ) = x0 , where x0 is given.
But leave the terminal state x(t1 ) free.
University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 34 of 44
Present versus Current Value Hamiltonian

Up to now, we have considered the present value Hamiltonian

H(t, x, u, p) := e −rt f (x, u) + p g (x, u)

We remove the discount factor e −rt

by defining the current value Hamiltonian

H C (x, u, q) := f (x, u) + q g (x, u)

with the current value co-state variable q := e rt p.

These definitions imply that

H(t, x, u, p) = e −rt [f (x, u) + e rt p g (x, u)] = e −rt H C (x, u, q)

where q = e rt p, so q̇ = re rt p + e rt ṗ = rq + e rt ṗ.

University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 35 of 44

Present and Current Value Maximum Principles
The (present value) maximum principle
states that for (almost) all t ∈ [0, ∞) one has

u ∗ (t) ∈ arg maxu H(t, x, u, p) and ṗ = −Hx0 (t, x, u, p)

By definition, one has H(t, x, u, p) = e −rt H C (x, u, q)

where q = e rt p.
Because e −rt is independent of u,
it follows that u ∗ (t) ∈ arg maxu H C (x, u, q).
Also q̇ − rq = e rt ṗ = −e rt Hx0 (t, x, u, p) = −H C 0x (x, u, q).
We have derived the current value maximum principle
states that for (almost) all t ∈ [0, ∞) one has

u ∗ (t) ∈ arg maxu H C (x, u, q) and q̇ − rq = −H C 0x (x, u, q)

University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 36 of 44

Outline

Introduction
A Basic Optimal Growth Problem
Digression: Sufficient Conditions for Static Optimality

The Maximum Principle

From Lagrangians to Hamiltonians
Example: A Macroeconomic Quadratic Control Problem

Sufficient Conditions for Optimality

Finite Horizon Case
Infinite Horizon Case

Discounting and the Current Value Hamiltonian

Maximum Principle Revisited
Application to an Optimal Growth Problem

University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 37 of 44

Statement of the Problem

The problem will be to choose:

1. a consumption stream R+ 3 t 7→ C (t) ∈ R++ ;
2. a stream R+ 3 t 7→ K (t) ∈ R++ of capital stocks.
At any time t, given capital K , output will be Y = a K − b K 2 ,
where a, b ∈ R are positive parameters, with a > r > 0.
Output is divided between consumption C and investment K̇ ,
so K̇ = Y − C ; there is no depreciation.
The planner’s objective
R ∞ is to maximize
the utility integral 0 e −rt u(C (t)) dt.
We assume that the utility function R++ 3 C 7→ u(C )
takes the isoelastic form with u 0 (C ) = C − .
The constant elasticity parameter  > 0
is a constant degree of relative fluctuation aversion.

University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 38 of 44

The Current Value Maximum Principle
R∞
The optimal growth problem is to maximize 0 e −rt u(C (t)) dt
subject to K̇ = a K − b K 2 − C where u 0 (C ) = C − .
With λ as the co-state variable, the current value Hamiltonian is

H C (K , C ) := u(C ) + λ(a K − b K 2 − C )

The first-order condition for maximizing (K , C ) 7→ H C (K , C )

w.r.t. C is u 0 (C ) = λ, which implies C − = λ and so C = λ−1/ .
Because C 7→ u(C ) is strictly concave, this is the unique maximum.
The co-state variable evolves according to the equation

λ̇ − r λ = −HKC 0 (K , C ) = −λ (a − 2 b K )

Finally, therefore, we have the coupled differential equations

K̇ = a K − b K 2 − λ−1/ and λ̇ = λ (r − a + 2 b K )

University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 39 of 44

Steady State of Coupled Differential Equations

The coupled differential equations

K̇ = a K − b K 2 − λ−1/ and λ̇ = λ (r − a + 2 b K )

have a steady state at any point satisfying K̇ = 0 and λ̇ = 0.

There is a unique steady state at the point (K , λ) = (K ∗ , λ∗ )
with K ∗ = (r − a)/2b and λ∗ = [K ∗ (a − bK ∗ )]− .

University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 40 of 44

Phase Diagram Analysis of Coupled Differential Equations
We have the coupled differential equations
K̇ = a K − b K 2 − λ−1/ and λ̇ = λ (r − a + 2 b K )
with a unique steady state at
K ∗ = (r − a)/2b, λ∗ = [K ∗ (a − bK ∗ )]−

The phase diagram on the next slide shows:

1. the two “isoclines” where K̇ = 0 and λ̇ = 0 respectively;
2. the intersection of these two isoclines
at the unique stationary point (K ∗ , λ∗ );
3. the division of the plane of (K , λ) values
into four different “phases” according as K̇ ≷ 0 and λ̇ ≷ 0,
which are marked by blue arrows
pointing in the relevant direction;
4. six possible different solutions
of the coupled differential equations
that are marked by blue curves.
University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 41 of 44
Phase Diagram

University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 42 of 44

Suboptimal Solutions to the Differential Equations
Paths of pairs (K , λ) where λ starts out too low,
and so C = λ−1/ starts out too high:
1. pass below and to the left of the steady state (K ∗ , λ∗ );
2. eventually reach the phase where K̇ < 0 and λ̇ < 0;
3. in that profligate phase, K reaches 0 in finite time,
after which there is no output
and so C = K = 0 for ever thereafter.
Such paths could be optimal for a suitable finite horizon,
but with an infinite horizon, they end in disaster.

Paths of pairs (K , λ) where λ starts out too high,

and so C = λ−1/ starts out too low:
1. pass above and to the right of the steady state (K ∗ , λ∗ );
2. eventually reach the phase where K̇ > 0 and λ̇ > 0;
3. in that phase of wasteful over-accumulation
one has K (t) → ∞ yet C (t) → 0 as t → ∞.
University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 43 of 44
Optimal Solutions to the Differential Equations

The red curve in the phase diagram

shows the unique solution curve
that passes through the steady state (K ∗ , λ∗ ).
Along this curve lies the happy medium
between profligacy and wasteful over-accumulation,
where (K , λ) → (K ∗ , λ∗ ) as t → ∞.
Furthermore, the present discounted value e −rt λ(t) K (t)
of the capital stock converges to zero.
So the Malinvaud transversality condition is satisfied,
thus completing the proof that this path solves
the infinite-horizon optimal growth problem.

University of Warwick, EC9A0 Maths for Economists Peter J. Hammond 44 of 44