Continual Learning: On Machines

that can Learn Continually

Official Open-Access Course @ University of Pisa, ContinualAI, AIDA

Lecture 5: Methodologies [Part 1]

Vincenzo Lomonaco
University of Pisa & ContinualAI
vincenzo.lomonaco@unipi.it
TABLE OF CONTENTS

01 02 03
Strategies Replay Avalanche
Categorization Strategies: Strategies &
and History Intro & Main Plugins
Approaches
 Strategy
Categorization and
History
 Possible 4-way Fuzzy Categorization

With some twists

● No formal definition

● Alternative categorizations are possible

Continual Learning for Robotics: Definition, Framework, Learning Strategies, Opportunities and Challenges, Lesort et al. Information Fusion, 2020.
A continual learning survey: Defying forgetting in classification tasks. De Lange et al, TPAMI 2021.
 Continual Learning Baselines

Common Baselines / Control Algorithms

● Naive / Finetuning (just continuing backprop)

● JointTraining / Offline (pure Multi-task

learning): The best you can do with all the data
starting from scratch

● Ensemble: one model for each experience

● Cumulative: for every experience, accumulate

all data and re-train from scratch.

A brief review on multi-task learning. Thung et al, 2018.

 Fundamental Design Choices

Strategic Choices

● Start from scratch or pre-trained?

● What model architecture to use?

● Such choices may affect the CL approach

effectiveness

Multi-Head vs Single-Head

Continual Learning for Recurrent Neural Networks: an Empirical Evaluation. Cossu et al, 2021.
Historical Trends

● Initial focus on Task Incremental (a few experiences, one for task, task labels given)

● Simple Regularization methods (L1 / L2, Dropout, Elastic Weights Consolidation, Synaptic Intelligence,
etc.)

● Simple Architectural strategies (Multi-head, Copy-Weight with Reinit, Progressive Neural Networks,
etc.)

● Simple Replay Strategies (random Replay, multi-buffer random replay, etc.)

● Current trend: more and more articulate strategies (often starting from pre-trained models), mostly
hybrid

● Mostly Heuristics, not principled methods. Very difficult to generalize to a large set of scenarios
 Effective Solutions

Good News

● Replay is a very general and

effective strategy for CL

Bad News

● Replay is approximating an
i.i.d distribution
● Can be seen as a form of
cheating
● Compute / memory
limitations

Replay-Based Methods for Continual Learning, Gabriele Merlin, MS Thesis, University of Pisa, 2021.
 Is Forgetting Solved?

Not really

● The gap with an offline strategy may be still

very large

● The accuracy improvements with respect to the

memory size is often logarithmic

● Huge buffer sizes (approximating a cumulative

strategy) may be very inefficient

○ Memory size (for imagenet 50 imgs per

class means about 7 GB memory)

○ Additional forward and backward passes

over the same examples

Latent Replay for Real-Time Continual Learning. Pellegrini et al. IROS, 2019.
Replay Strategies
 Random Replay

A basic approach

● Sample randomly from the

current experience data

● Fill your fixed Random Memory

(RM)

● Replace examples randomly to

maintain an approximate equal
number of examples for
experience

Latent Replay for Real-Time Continual Learning. Pellegrini et al. IROS, 2019.
 Many Implementation Options
…and many implications

● Fixed or “adaptive” external memory?

● Sample selection: random or representative examples only?

● Mini-batch sample selection: what examples to choose from M and to use in the current mini-batch?
What augmentations to use?

● Separate buffers per class / tasks / notable distributions?

● Sample based on time: different timescales? Uniform sampling in time?

● Sample replacement: which examples to throw away when the memory is full?

● No clear answer to all these questions: a coherent empirical evaluation still missing

● It really depends on the scenario / problem you are solving -> more engineering than science

Memory Efficient Experience Replay for Streaming Learning, Hayes et al. 2019
 GDUMB: Another Control Baseline

Greedy Sampler and Dumb Learner

● Interesting paper that sparked strong

discussions in the CL community

● Note that there’s no knowledge transfer in

this strategy (quite dumb indeed!)

● Despite its simplicity, It was shown to work

better than some existing and more complex
strategies, questioning the utility of some
benchmarks/metrics in our field

● If your strategy cannot beat GDumb there’s

something wrong about your strategy or your
evaluation setting

GDumb: A Simple Approach that Questions Our Progress in Continual Learning. Prabhu et al. ECCV, 2020.
 Maximally Interfered Retrieval (MIR)

Mini-batch Sample Selection

● Select the examples

that are more
negatively impacted
by the estimated
weights update

● May be quite slow in

practice w.r.t. the
actual accuracy gain
over random selection

Online Continual Learning with Maximally Interfered Retrieval. Aljundi et al. 2019.
 Latent Replay

Key Ideas

● Replay in the input space is inefficient and

biologically implausible

● Why not replaying in the latent activations

space?

● Good Accuracy-Memory-Computation
trade-offs are possible

Latent Replay for Real-Time Continual Learning. Pellegrini et al. IROS, 2019.
 Generative Replay
Key Ideas

● Instead of a replay memory why not generating examples?

● In theory this would be even better than replay: allowing for generating examples that were never
seen before (a form of dreaming or imagination)
● Still difficult to scale on high-dimensional data and find good accuracy-efficiency trade-offs

Continual Learning with Deep Generative Replay, Shi et al, 2017.

Replay: Summary and Next Steps

● A definitive study of replay in deep continual learning is still missing

● Replay has been shown to be an effective strategy in CL if performance is the main objective

● Replay is unlikely to be represent the main computational principle for CL in biological learning
systems (not a good efficiency-effectiveness trade-off)

● Many improvements and implementation options have been explored with different degrees of
success

● Generative / latent replay constitute an interesting future direction but quite challenge at the moment
due to the limited generative models capabilities
 Avalanche
Strategies and
Plugins
 Training: Design

Avalanche provides popular strategies already implemented and ready-to-use and easy mechanisms to
define custom strategies.

● Many strategies are already available

● Easy modification of the training loop to add logging and custom behavior (mostly trough
Polymorphism)

V. Lomonaco et al. Avalanche: an End-to-End Library for Continual Learning. CLVision Workshop at CVPR 2021.
How to: Strategy Initialization
 How to: Training & Evaluation

V. Lomonaco et al. Avalanche: an End-to-End Library for Continual Learning. CLVision Workshop at CVPR 2021.
 Training: Design

● Strategy: defines a CL strategy with two simple methods:

○ train and eval.

● Plugin: a simple interface to add custom behavior to the training and eval loops.

V. Lomonaco et al. Avalanche: an End-to-End Library for Continual Learning. CLVision Workshop at CVPR 2021.
 How to: Add Plugins

V. Lomonaco et al. Avalanche: an End-to-End Library for Continual Learning. CLVision Workshop at CVPR 2021.
 Training: Custom Strategies

How to write custom strategy

● plugin: the easiest way to customize training and define new strategies.

● strategy: override the loop methods directly.

Why should I use Avalanche to implement my own strategies?

● automatic logging & metrics evaluation.

● you write less code, and you can easily share it with the community.

V. Lomonaco et al. Avalanche: an End-to-End Library for Continual Learning. CLVision Workshop at CVPR 2021.
 BaseStrategy: Under the hood

● The base class from which to inherit

and to specialize

● Implemented as a series of callbacks as

a skeleton to the plugin system: this
means you can write plugins “by
difference” and compose plugins

V. Lomonaco et al. Avalanche: an End-to-End Library for Continual Learning. CLVision Workshop at CVPR 2021.
 Custom Plugin

V. Lomonaco et al. Avalanche: an End-to-End Library for Continual Learning. CLVision Workshop at CVPR 2021.
 Custom Strategy

V. Lomonaco et al. Avalanche: an End-to-End Library for Continual Learning. CLVision Workshop at CVPR 2021.
 Training: What’s Next?

● More Strategies & Plugins! (and make sure they can reproduce published results)

● Support for Unsupervised / Reinforcement Continual Learning (check the Avalanche ecosystem!)

V. Lomonaco et al. Avalanche: an End-to-End Library for Continual Learning. CLVision Workshop at CVPR 2021.
 Training in Avalanche Demo Session
!

https://avalanche.continualai.org/from-zero-to-hero-tutorial/04_training
Replay in
Avalanche
 Replay in Avalanche Demo Session
!

https://avalanche.continualai.org/how-tos/dataloading_buffers_replay
 Next:
Methodologies [Part 2]
 Do you have any questions?

vincenzo.lomonaco@unipi.it
vincenzolomonaco.com
University of Pisa

THANKS
CREDITS: This presentation template was created by Slidesgo,
including icons by Flaticon, and infographics & images by Freepik