Computer Vision I:

Variational Methods
Prof. Dr. Daniel Cremers
Chair of Computer Vision & Artificial Intelligence
Departments of Informatics & Mathematics
Technical University of Munich
 Exercises

Marvin Eisenberger Emanuel Laude

 Computer Vision & Driver Assistance

Daniel Cremers Computer Vision I: Variational Methods 3

 Variational Scene Flow

Wedel et al. IJCV ‘11, Wedel & Cremers, Springer 2011

Daniel Cremers Computer Vision I: Variational Methods 4
3300 delegates!
 TUM Chair of Computer Vision & AI

Daniel Cremers Computer Vision I: Variational Methods 6

 Spatially Dense 3D Reconstruction

infinite-dimensional optimization
Daniel Cremers Computer Vision I: Variational Methods 7
 Which path is the fastest?

Daniel Cremers Computer Vision I: Variational Methods 8

 Bernoulli & The Brachistochrone

Johann Bernoulli (1667-1748)

Daniel Cremers Computer Vision I: Variational Methods 9
 Optimization in Computer Vision
Image segmentation:

Geman, Geman ’84, Blake, Zisserman ‘87, Kass et al. ’88,

Mumford, Shah ’89, Caselles et al. ‘95, Kichenassamy et al. ‘95,
Paragios, Deriche ’99, Chan, Vese ‘01, Tsai et al. ‘01, …

Multiview stereo reconstruction:

Non-convex energies
Faugeras, Keriven ’98, Duan et al. ‘04, Yezzi, Soatto ‘03,
Seitz et al. ‘06, Hernandez et al. ‘07, Labatut et al. ’07, …

Optical flow estimation:

Horn, Schunck ‘81, Nagel, Enkelmann ‘86, Black, Anandan ‘93,

Alvarez et al. ‘99, Brox et al. ‘04, Baker et al. ‘07, Zach et al. ‘07,
Sun et al. ‘08, Wedel et al. ’09, …
Daniel Cremers Computer Vision I: Variational Methods 10
 Optimization and Convexity

Non-convex energy Convex energy

Daniel Cremers Computer Vision I: Variational Methods 11

 Inverse Problems: Denoising

image denoised image

What is tipical in a clean
image? ==> regularizer

Rudin, Osher, Fatemi 1992, Goldlücke, Strekalovskiy, Cremers 2012

Daniel Cremers Computer Vision I: Variational Methods 12
 Variational Methods

Joseph-Louis Lagrange
Leonhard Euler (1736 – 1813)
(1703-1783)

Daniel Cremers Computer Vision I: Variational Methods 13

 Variational Methods & PDEs

Euler-Lagrange equation as necessary condition:

PDE

If the function L is convex, then the E-L equation brings global minima, no
saddle point will occur.
Gradient descent:

iterations

Daniel Cremers Computer Vision I: Variational Methods 14

 Overview

Multiview reconstruction Super-res.textures Stereo reconstruction

Realtime dense geometry RGB-D cameras Reconstruction on the fly

Daniel Cremers Computer Vision I: Variational Methods 15
 Overview

Multiview reconstruction Super-res.textures Stereo reconstruction

Realtime dense geometry RGB-D cameras Reconstruction on the fly

Daniel Cremers Computer Vision I: Variational Methods 16
 3D Reconstruction from Multiple Views

Kolev, Klodt, Brox, Cremers, Int. J. of Computer Vision ’09:

Theorem: Globally optimal surfaces can be computed by convex optimization.

Daniel Cremers Computer Vision I: Variational Methods 17

 Evolution to Global Optimum

Kolev, Klodt, Brox, Cremers, Int. J. of Computer Vision ’09:

Theorem: Globally optimal surfaces can be computed by convex optimization.

Daniel Cremers Computer Vision I: Variational Methods 18

 Reconstruction of Fine-scale Structures

Image data courtesy of Yasutaka Furukawa.

Daniel Cremers Computer Vision I: Variational Methods 19

 Reconstructing the Niobids Statues

Kolev, Cremers, ECCV ’08, PAMI ‘11

Daniel Cremers Computer Vision I: Variational Methods 20

 Multiview Reconstruction

Kolev, Cremers, ECCV ’08, PAMI ‘11

Daniel Cremers Computer Vision I: Variational Methods 21

 Reconstructing Dynamic Scenes

Oswald, Stühmer, Cremers, ECCV ‘14

Daniel Cremers Computer Vision I: Variational Methods 22
 Action Reconstruction

Oswald, Stühmer, Cremers, ECCV ‘14

Daniel Cremers Computer Vision I: Variational Methods 23
 Single View Reconstruction

Can we recover geometry from a single image?

Yes: Shape-from-shading, shape-from-focus, shape from symmetry,…
Solution: Fixed-volume silhouette-consistent minimal surface.

Daniel Cremers Computer Vision I: Variational Methods 24

 Single View Reconstruction

Toeppe, Oswald, Rother, Cremers, ACCV 2010

Daniel Cremers Computer Vision I: Variational Methods 25

 Single View Reconstruction

Reconstruction computed in fractions of a second on GPU

Toeppe et al. ACCV 2010, Oswald et al. CVPR 2012

Daniel Cremers Computer Vision I: Variational Methods 26

 Single View Reconstruction

Toeppe, Oswald, Rother, Cremers, ACCV 2010

Daniel Cremers Computer Vision I: Variational Methods 27

 Modifying the Material Properties

Daniel Cremers Computer Vision I: Variational Methods 28

 Single View Reconstruction

Toeppe, Oswald, Rother, Cremers, ACCV 2010

Daniel Cremers Computer Vision I: Variational Methods 29

 Single View Reconstruction

Toeppe, Oswald, Rother, Cremers, ACCV 2010*

* Best Paper Honorable Mention

In collaboration with Microsoft Research

Daniel Cremers Computer Vision I: Variational Methods 30

 Overview

Multiview reconstruction Super-res.textures Stereo reconstruction

Realtime dense geometry RGB-D cameras Reconstruction on the fly

Daniel Cremers Computer Vision I: Variational Methods 31
 Evolution to Global Optimum

Kolev, Klodt, Brox, Cremers, IJCV 2009

Daniel Cremers Computer Vision I: Variational Methods 32

 Super-Resolution Texture Map

Given all images determine the surface color

blur & downsample back-projection

* Best Paper
Goldlücke, Cremers, ICCV ’09, DAGM ’09*, IJCV ‘13 Award
Daniel Cremers Computer Vision I: Variational Methods 33
 Super-Resolution Texture Map

* Best Paper
Goldlücke, Cremers, ICCV ’09, DAGM ’09*, IJCV ‘13 Award
Daniel Cremers Computer Vision I: Variational Methods 34
 Super-Resolution Texture Map

Weighted average Super-resolution texture

* Best Paper
Goldlücke, Cremers, ICCV ’09, DAGM ’09*, IJCV ‘13 Award
Daniel Cremers Computer Vision I: Variational Methods 35
 Super-Resolution Texture Map

Closeup of input image Super-resolution texture

* Best Paper
Goldlücke, Cremers, ICCV ’09, DAGM ’09*, IJCV ‘13 Award
Daniel Cremers Computer Vision I: Variational Methods 36
 Overview

Multiview reconstruction Super-res.textures Stereo reconstruction

Realtime dense geometry RGB-D cameras Reconstruction on the fly

Daniel Cremers Computer Vision I: Variational Methods 37
 From Binary to Multilabel Optimization

Example: Stereo Reconstruction

Pock, Schoenemann, Bischof, Cremers, Europ. Conf. on Computer Vision ’08:

Theorem: Stereo reconstruction can be solved by convex optimization.

Daniel Cremers Computer Vision I: Variational Methods 38

 Evolution to Global Minimum

Daniel Cremers Computer Vision I: Variational Methods 39

 Reconstruction from Aerial Images

1/2 input images (6 Mpixel) Depth reconstruction

Courtesy of H. Hirschmüller 77 seconds

Stangl, Souiai, Cremers, GCPR ‘13

Daniel Cremers Computer Vision I: Variational Methods 40

 Reconstruction from Aerial Images

One of two input images

Depth reconstruction
Courtesy of Microsoft

Daniel Cremers Computer Vision I: Variational Methods 41

 Reconstruction from Aerial Images

Daniel Cremers Computer Vision I: Variational Methods 42

 Highly accurate Stereo Reconstruction

Stereo input Depth reconstruction

Möllenhoff, Laude, Möller, Lellmann, Cremers, CVPR ’16 *

* Best Paper Honorable Mention

Daniel Cremers Computer Vision I: Variational Methods 43

 Munich from the Air

1/2 input images (1000x1000) Depth reconstruction

Kuschk, Cremers, ICCV Big Data Workshop 2013

Daniel Cremers Computer Vision I: Variational Methods 44

 Overview

Multiview reconstruction Super-res.textures Stereo reconstruction

Realtime dense geometry RGB-D cameras Reconstruction on the fly

Daniel Cremers Computer Vision I: Variational Methods 45
 From Dense Flow to Dense Geometry

Input video Optical flow field

Horn & Schunck ‘81, Zach et al. DAGM ’07, Wedel et al. ICCV ’09

Daniel Cremers Computer Vision I: Variational Methods 46

 From Dense Flow to Dense Geometry

Input video Optical flow field *

* 60 fps @ 640x480

Horn & Schunck ‘81, Zach et al. DAGM ’07, Wedel et al. ICCV ’09

Daniel Cremers Computer Vision I: Variational Methods 47

 Dense geometry from hand-held camera

Brightness constancy:

Stuehmer , Gumhold, Cremers, DAGM ’10

Daniel Cremers Computer Vision I: Variational Methods 48

 Dense geometry from hand-held camera

Stuehmer, Gumhold, Cremers, DAGM ’10

Daniel Cremers Computer Vision I: Variational Methods 49

 Dense geometry from hand-held camera

Stuehmer, Gumhold, Cremers, DAGM ’10

Daniel Cremers Computer Vision I: Variational Methods 50

 Realtime Dense Reconstruction

16.0 fps 22.0 fps 41.1 fps

Stuehmer, Gumhold, Cremers, DAGM ’10

Daniel Cremers Computer Vision I: Variational Methods 51

 Overview

Multiview reconstruction Super-res.textures Stereo reconstruction

Realtime dense geometry RGB-D cameras Reconstruction on the fly

Daniel Cremers Computer Vision I: Variational Methods 52
 RGB-D Camera Tracking

Optimize dense photo-consistency:

Steinbruecker et al. ICCV ’11, Kerl et al., ICRA ‘13

Daniel Cremers Computer Vision I: Variational Methods 53

 Realtime 3D Modeling

Color input Depth input

Daniel Cremers Computer Vision I: Variational Methods 54

 Realtime 3D Modeling

Daniel Cremers Computer Vision I: Variational Methods 55

 Realtime 3D Modeling

Daniel Cremers Computer Vision I: Variational Methods 56

 Realtime 3D Modeling

Daniel Cremers Computer Vision I: Variational Methods 57

 Realtime 3D Modeling

Daniel Cremers Computer Vision I: Variational Methods 58

 Realtime 3D Modeling

Daniel Cremers Computer Vision I: Variational Methods 59

 Realtime 3D Modeling

Daniel Cremers Computer Vision I: Variational Methods 60

 Realtime 3D Modeling

Daniel Cremers Computer Vision I: Variational Methods 61

 Full-Body Scanner

Daniel Cremers Computer Vision I: Variational Methods 62

 Overview

Multiview reconstruction Super-res.textures Stereo reconstruction

Realtime dense geometry RGB-D cameras Reconstruction on the fly

Daniel Cremers Computer Vision I: Variational Methods 63
 Reconstruction on the Fly

Bylow, Sturm, Kerl, Kahl, Cremers RSS ‘13

Daniel Cremers Computer Vision I: Variational Methods 64
 Large Scale: Loop Closure

Kerl, Sturm, Cremers ICRA ‘13

Daniel Cremers Computer Vision I: Variational Methods 65
 Large Scale: Octrees

Steinbrücker, Kerl, Sturm, Cremers ICCV ‘13

Daniel Cremers Computer Vision I: Variational Methods 66

 Realtime Large-Scale Reconstruction

Steinbrücker, Kerl, Sturm, Cremers ICCV ‘13, ICRA ‘14

Daniel Cremers Computer Vision I: Variational Methods 67

 Summary

multiview reconstruction super-res. textures action reconstruction

stereo reconstruction RGB-D modeling 3D on the fly

Daniel Cremers Computer Vision I: Variational Methods 68