Visual Object Tracking
Instructor: Seunghoon Hong
Visual object tracking
Objective: locating the object(s) over time in a video
Initial frame
Target Tracking over
Visual Tracking
Visual object tracking
Objective: locating the object(s) over time in a video
Formal definition: given an object state at the initial frame z0=(x0,y0,w0,h0),
identify z1:T={z1,z2,…,zT} over a video of length T.
Visual object tracking
Objective: locating the object(s) over time in a video
Formal definition: given an object state at the initial frame z0=(x0,y0,w0,h0),
identify z1:T={z1,z2,…,zT} over a video of length T.
In learning perspective:
● Classification problem with a single object class (= target vs distractors)
● Labeled data is given at only the initial frame
● Optionally requires online learning to adapt the variations in a video
● Online learning is driven by a self-supervision (training data = tracking results)
Visual object tracking
Objective: locating the object(s) over time in a video
Formal definition: given an object state at the initial frame z0=(x0,y0,w0,h0),
identify z1:T={z1,z2,…,zT} over a video of length T.
Two sub-categories:
● Single target tracking
○ Tracking only one object in an video
○ Single-class classification (target vs. distractors)
● Multi target tracking
○ Tracking multiple objects in a video
○ Multi-class classification (target 1 vs. target 2 vs. target 3 vs. … vs. distractors)
Visual object tracking
Objective: locating the object(s) over time in a video
Formal definition: given an object state at the initial frame z0=(x0,y0,w0,h0),
identify z1:T={z1,z2,…,zT} over a video of length T.
Two sub-categories:
● Single target tracking
○ Tracking only one object in an video
○ Single-class classification (target vs. distractors)
● Multi target tracking
○ Tracking multiple objects in a video
○ Multi-class classification (target 1 vs. target 2 vs. target 3 vs. … vs. distractors)
Approaches in single object tracking
● Probabilistic tracking
○ Formulate the localization task as a sequential probabilistic inference problem
○ Given a probability of the initial target location, propagate it over the remaining frames
Approaches in single object tracking
● Probabilistic tracking
○ Formulate the localization task as a sequential probabilistic inference problem
○ Given a probability of the initial target location, propagate it over the remaining frames
● Discriminative tracking
○ Classify the object from the distractors at every frame
○ Can be considered as sequential binary object detection (class = target, background)
Probabilistic tracking
● Tracking as a Bayesian network
Bayes Rule
z: object location (state)
x: frame (observation)
Likelihood Prior
Posterior
the measurement of The belief of object state
the probability of
how likely the without observation
object state given
observation
an observation
coincide with the
given state
Probabilistic tracking
● Tracking as a Bayesian network
Bayes Rule
z: object location (state)
x: frame (observation)
Target template
Prior
1 The belief of object state
without observation
2 3 Where is the target
likely to exist?
Probabilistic tracking
● Tracking as a Bayesian network
Bayes Rule
z: object location (state)
x: frame (observation)
Target template
Likelihood
the measurement of
how likely the
observation
coincide with the
given state
Which region of
image look similar
to the target?
Probabilistic tracking
● Tracking as a Bayesian network
Bayes Rule
z: object location (state)
x: frame (observation)
Target template
Posterior
the probability of
object state given
an observation
Where is the object
in this frame?
Probabilistic tracking
● Tracking as a Bayesian network
Bayes Rule
Sequential Bayesian filtering
z1:T: object locations in frame 1 to T
x1:T: frames 1 to T
Probabilistic tracking
● Hidden Markov Model
● Markovian assumption
Probabilistic tracking
● Sequential Bayesian filtering
Integration over all object locations!
Likelihood Prior
Likelihood Transition Posterior upto
model the previous frame
Probabilistic tracking
● Approximation by Monte Carlo sampling
where
Probabilistic tracking
● Particle filtering (Sequential Markov-Chain Monte-Carlo)
○ Approximate the prior distribution using Markov-Chain Monte Carlo (MCMC) sampling
Probabilistic tracking pipeline
Frame t-1 Frame t
2. Move samples by
1. Extract samples transition model 3. Re-evaluate likelihood
proportional to using appearance model
previous posterior
Probabilistic tracking pipeline
Frame t
Tracking procedure (simplified):
1. Sample target states near the previous
target location
2. Evaluate the likelihood based on
appearance model
Example target
appearance model
3. Select the most probable sample as the
target at the current frame
4. Update the target appearance model
using the current tracking results
Attendance check
https://forms.gle/rGpXxLKZ4jbcArid8
Discriminative tracking pipeline
Quick overview: learning tracking-by-detection
● Objective: a ridge regression
Model parameters
Training Training data
labels
Quick overview: learning tracking-by-detection
● Objective: a ridge regression
How do we solve it?
Quick overview: learning tracking-by-detection
● Objective: a ridge regression
We should update this classifier for every frames
(i.e. every time we perform tracking and
get positive/negative samples)
Can we make it faster?
Correlation filtering
● We can make it extremely fast for certain positive/negative sets!
Negative samples
(translated samples)
+30 +15 -15 -30
Base sample
(tracking results)
Correlation filtering
● Representing positive/negative images using circulant matrices
Consider base sample x as n-dimensional array
Circulant matrix
Positive sample
Negative samples
Correlation filtering
● Any circulant matrices can be made diagonal by the Discrete Fourier Transform
(DFT)
DFT matrix
(constant,
independent to x)
DFT of base sample
Correlation filtering
● Putting all together
Circulant matrix
Matrix inner-product
Plug into ridge
regression
Kernelized Correlation filtering
● Easy to extend to kernelized version
ridge regression
ridge regression with
kernel
We can do fast
computation if kernel
matrix K is circulant matrix
Fortunately, it has been
shown that most useful
kernels are circulant[1]
[1] Henriques et al., High-Speed Tracking with Kernelized Correlation Filters, In TPAMI, 2015
Challenges
● Modeling severe appearance variations in a video
figure credit: Li et al., A survey of appearance models in visual object tracking
Modeling appearance for tracking
● Classic: hand-designed features
○ Color histogram
○ Intensity
○ Object Templates
○ Key-points (SIFT)
○ …
● Issue
○ All prone to overfitting
○ Cannot generalize to various appearances
Integrating CNN for appearance modeling
● Benefits
○ Features from a pre-trained CNN can be robust against various appearance changes
○ Especially useful in tracking since we have only one target ground-truth in the initial frame
CNN-based tracking
● CNNTrack: direct application of CNN feature for tracking
CNN-based tracking
● CNNTrack: direct application of CNN feature for tracking
CNN-based tracking
● CNNTrack: direct application of CNN feature for tracking
CNN-based tracking
● CNNTrack: direct application of CNN feature for tracking
CNN-based tracking
● CNNTrack: direct application of CNN feature for tracking
Discussions
● Limitations?
Better representation learning with videos
● MDNet: learn representation for tracking with a large amount of videos
Challenges in visual object tracking
● Temporal drift (i.e. error propagation through time)
○ Drift in posterior estimation: the error in posterior propagates through time
○ Drift in appearance model: if update the appearance model in temporal failure, the error will
propagate
● But why is it so prune to temporal drift?
Summary: Visual tracking
● Object localization in a video
● Probabilistic vs. discriminative tracking
● Modeling target appearance is important
○ Essential to evaluate the affinity of samples in both tracking frameworks
○ Should be able to handle a wide range of appearance variations
○ Should be able to generalize well from a single ground-truth at initial frame
● CNN for visual tracking
○ Applying a pre-trained CNN for feature extraction
○ Training CNN with many heterogeneous videos for tracking
