Optical and Scene Flow

Bryan Chiang and Jeannette Bohg

February 14, 2022

1 Overview
Given a video, optical flow is defined as a 2D vector field describing the ap-
parent movement of each pixel due to relative motion between the camera (ob-
server) and the scene (objects, surfaces, edges). The camera or the scene or both
may be moving. Figure 1 shows a fly rotating in a counter-clockwise direction
(from the fly’s point-of-view). Although the scene is static, the 2D optical flow
of apparent motion indicates a rotation in the opposite (clockwise) direction
around the origin.

Figure 1: Optical flow example [3].

1.1 Motion field

Optical flow is not to be confused with the motion field, a 2D vector field
describing the projection of 3D motion vectors for points in the scene onto the
image plane of the observer. Figure 2 illustrates the motion field in a simple
2D case (imagine viewing a 3D scene from a top-down perspective). The 2D
object point Po is projected to a a1D point Pi in the image plane as viewed
by observer O. If the object point Po is displaced by Vo · dt (called the motion
vector), the corresponding projected 1D point moves by Vi · dt. The 1D motion
field here consists of all velocity values Vi for all i located in the image plane.

1
 Figure 2: Example of a motion field for a 2D scene [5].

Generalizing to a 3D scene, the motion field for pixel (x, y) is given by

   dx 
u dt
  =   = Mx′ (1)
dy
v dt

dy dz T
where x′ = [ dx
dt , dt , dt ] represents the motion of a 3D point and M ∈ R
2×3

contains the partial derivatives of pixel displacement with respect to the 3D

point locations.
The motion field is an ideal 2D representation of 3D motion as projected onto
the image plane. It is the “ground truth” that we cannot observe directly; we
can only estimate the optical flow (apparent motion) from our noisy observations
(video). It is important to note that the optical flow is not always the same
as the motion field. For instance, a uniform rotating sphere with a fixed light
source has no optical flow but a non-zero motion field. In contrast, a fixed
uniform sphere with a light source moving around it has a non-zero optical flow
but a zero motion field. These two cases are illustrated in Figure 3.

2
 Figure 3: Physical vs. optical correspondence [4].

2 Computing the optical flow

We define a video as an ordered sequence of frames captured over time. I(x, y, t),
a function of both space and time, represents the intensity of pixel (x, y) in the
frame at time t. In dense optical flow, at every time t and for every pixel (x, y),
we want to compute the apparent velocity of the pixel in both the x-axis and
∆y
y-axis, given by u(x, y, t) = ∆x
∆t and v(x, y, t) = ∆t , respectively. The optical
flow vector for each pixel is then given as u = [u, v]T . In the following sections,
we describe the Lucas-Kanade method, which uses a semi-local approach to
independently solve for u different pixel patches using least-squares.
From the brightness constancy assumption, we can assume that the
apparent intensity in the image plane for the same object does not change across
different frames. This is represented by Equation (2) for a pixel that moved ∆x
and ∆y in the x and y directions between times t to t + ∆t.

I(x, y, t) = I(x + ∆x, y + ∆y, t + ∆t) (2)

One common simplification is to use ∆t = 1 (consecutive frames), such that
the velocities are equivalent to the displacements u = ∆x and v = ∆y. We can
then obtain I(x, y, t) = I(x + u, y + u, t + 1) as in the lecture slides.
Next, by the small motion assumption, we assume the motion (∆t, ∆y)
is small from frame to frame. This allows us to linearize I with a first-order
Taylor series expansion, illustrated by Equation (3).

3
 ∂I ∂I ∂I
I(x + ∆x, y + ∆y, t + ∆t) = I(x, y, t) + ∆x + ∆y + ∆t + . . .
∂x ∂y ∂t
(3)
∂I ∂I ∂I
≈ I(x, y, t) + ∆x + ∆y + ∆t
∂x ∂y ∂t
The . . . represents the higher-order terms in the Taylor series expansion
which we subsequently truncate out in the next line. Substituting the result
from Equation (3) into Equation (2), we arrive at the optical flow constraint
equation:

∂I ∂I ∂I
0= ∆x + ∆y + ∆t
∂x ∂y ∂t
∂I ∆x ∂I ∆y ∂I (4)
= + +
∂x ∆t ∂y ∆t ∂t
= Ix u + Iy v + It

Ix , Iy , It are short-hand for the two spatial derivatives and time derivative,
respectively.

−It = Ix u + Iy v
= ∇I T u (5)
= ∇I · ⃗u

We recognize this as linear system in the form of Ax = b. ∇I = [Ix , Iy ]T ∈

2×1
R represents the spatial gradient of intensity, and ⃗u ∈ R2×1 is the flow vector
we want to solve for.
However, since ∇I is a fat matrix, this is an under-constrained system since
we only have a single constraint equation for our two unknowns u, v. This
constraint only takes us from two to one degree of freedom; the set of (u, v)
solutions must lie along a line given by Equation (5) and illustrated in Figure 4.
Specifically, the optical flow constraint can only give us the normal flow: the
component of u along the direction of the spatial gradient ∇I. Figure 5 visu-
alizes the spatial image gradient in the x and y direction of an example image.
You can see that the high gradient magnitudes correspond to images edges.
Recall that the dot product between two vectors a and b is a·b = ∥a∥∥b∥ cos(θ),
where θ is the angle between the two vectors. The projection of a on b can then
be obtained by dividing both sides by ∥b∥, illustrated in Figure 6.
Thus, Equation (6) shows how the optical flow constraint gives us the pro-
jection of u on ∇I, which is the normal flow.

∇I −It
·u= (6)
∥∇I∥ ∥∇I∥

4
 Figure 4: Solution set for (u, v) as a line in the form of y = mx + b.

Figure 5: On the left, an intensity image of a cat. In the center, a gradient

image in the x direction measuring horizontal change in intensity. On the right,
a gradient image in the y direction measuring vertical change in intensity. Gray
pixels have a small gradient; black or white pixels have a large gradient [1].

Figure 6: Dot product projection [2].

5
 Figure 7: Visual example of the aperture problem.

Figure 4 plots the solution space for u. However, we can also visualize the
image space on top of this. The normal flow (solution space) is in the same
direction of the spatial gradient of the image intensity at a given pixel. The
solution set (the line) has the same direction as the edge in the image space
since it is orthogonal to the spatial gradient.
Thus, while we know the component of u in direction of ∇I (normal flow),
we do not know the component of u along the edge. This is represented by the
solution set, which shows all the possible values this component could have. To
find a possible value for u, we first travel in the direction of the spatial gradient
by a distance equivalent to the magnitude of the normal flow, and then we can
travel any (unknown) distance in the direction of the edge, which represents our
one degree of freedom.
This issue where we do not know the magnitude of the pixel movement in the
edge direction is also known as the aperture problem, illustrated in Figure 7.
The blue square is opaque and has a square cutout at the center that forms a
small aperture. The light gray rectangle is our moving object. On the left, the
gray rectangle is in front of the blue square. On the right, the gray rectangle
is behind the blue square. In both cases, the rectangle moves down and to
the right (at the same time). The red arrows indicate the perceived direction
of movement in both cases. In the first case on the left, since the rectangle
is not blocked by the blue square, we can clearly perceive its true direction
of movement. However, for the case on the right, the rectangle is behind the
blue square and is thus occluded: the movement of the rectangle is in the same
direction as on the left, but we only perceive a movement directly to the right.
This aligns with what we derived earlier. We only know how much the rectangle
moved in the direction of the spatial gradient (here the x-axis), but not how
much the it traveled in the direction of the image edge (here the y-axis).
To mitigate this problem, we can use the spatial smoothness assumption:

6
neighboring points belong to the same surface in the scene, and thus share the
same optical flow u. If we define an N × N neighborhood around the current
pixel, we end up with a constraint ∇I(pi )T u = −It (pi ) for each of the N 2
pixels, where pi = [xi , yi ]T is the location of the i−th pixel. Assuming N 2 > 2,
we now have the following system of equations:

Au = b
   
Ix (p1 ) Iy (p1 )   It (p1 )
.. ..  u .. (7)
 v = −
  
 . . . 
Ix (pN 2 ) Iy (pN 2 ) It (pN 2 )

We then minimize the error ∥Au − b∥ to find the least-squares solution of

this overdetermined system, uls = (AT A)−1 AT b by multiplying both sides of
Equation (7) with AT . We consider the solvability of this system by examining
the following normal equations:

AT Au = AT b
(8)
 P 2 P  P 
P Ix PIx I2y u = − P Ix It
Ix Iy Iy Iy It

The summations are over each pixel in the neighbourhood. For the system
to be solvable, AT A should not be small due to the presence of noise. For
example, low-texture regions have small gradient magnitudes, leading to small
eigenvalues and easily corrupted u values. AT A should also be well-conditioned
such that the system is robust to measurement errors since AT b depends on all
the gradients calculated from the image intensity data. For example, at an edge
all the large gradients will be pointing in the same direction, resulting in one
eigenvalue that is significantly larger than the other (large condition number).
A good region where u can be solved unambiguously is a highly textured region
with directionally varying spatial structures; we will have large gradients in
different directions, leading to a well-conditioned AT A.
To more intuitively see why directionally varying structures helps us solve
for u, consider Figure 8. We have two cases here. The gray circle represents
our pixel neighborhood that we are optimizing over. The dashed line represents
the object at a previous time step, and and the solid line represents the object
at the current time step. This depicts the true movement of the object. In
(c), we see the aperture problem at play again. We have a single normal flow
vector (in green), so there are numerous possible solutions for u (in blue) along
the constraint line (also in green). However, in (b), we have varying spatial
structure in the form of a corner, which we can view as the two edges with
different directions. We now have two normal flow vectors and two constraint
lines (one in red, the other in green). There is only one possible solution for
the flow vector u that satisfies both constraints lines, shown in the blue. Thus,
we are incentivized to increase the neighborhood size N to increase the chances
of including varying spatial structure. However, there is a tradeoff as we might

7
Figure 8: (b) shows the importance of having varying spatial structure in light
of the aperture problem depicted in (c).

also cross motion boundaries to include pixels that belong to other surfaces and
therefore it becomes more and more likely that u is not constant across the
entire neighborhood.
In practice, Lucas-Kanade in this traditional formulation is not robust due
to large camera motions, occlusions, and violations of the aforementioned set of
assumption.

References
[1] https://commons.wikimedia.org/w/index.php?curid=10588443 By Njw000
Own work, Public Domain. Image gadient.

[2] Nykamp DQ. The dot product. https://mathinsight.org/dot_product,

2021. Accessed: 2022-01-31.
[3] Stephen J Huston and Holger G Krapp. Visuomotor transformation in the
fly gaze stabilization system. PLoS biology, 6(7):e173, 2008.

[4] Bernd Jähne, Horst Haussecker, and Peter Geissler. Handbook of computer
vision and applications, volume 2. Citeseer, 1999.
[5] Robyn Owens. Computer vision it412. https://homepages.inf.ed.ac.uk/
rbf/CVonline/LOCAL_COPIES/OWENS/LECT12/node3.html, 1997. Accessed:
2022-01-31.