Computational Imaging
Lecture 13: Computing Toolbox: Image Blurry
Q:Why are Our Images
Blurry?
GAMES204 Computational Imaging, Qilin Sun
Lens Imperfections and Physical Limit.
Camera Shake.
Scene Motion.
Today’s
Topic Depth Defocus.
Lens Imperfections and
Physical Limit
GAMES204 Computational Imaging, Qilin Sun
Lens Imperfections
Ideal lens: A point maps to a point at a certain plane.
1 1 1
+ =
𝑆𝑆 ′ 𝑆𝑆 𝑓𝑓
object distance S sensor distance S’
GAMES204 Computational Imaging, Qilin Sun
Lens Imperfections
Ideal lens: A point maps to a point at a certain plane.
Real lens: A point maps to a circle that has non-zero minimum radius among all planes.
1 1 1
+ =
𝑆𝑆 ′ 𝑆𝑆 𝑓𝑓
object distance S sensor distance S’
What is the effect of this on the images we capture?
GAMES204 Computational Imaging, Qilin Sun
Lens Imperfections
Ideal lens: A point maps to a point at a certain plane.
Real lens: A point maps to a circle that has non-zero minimum radius among all planes.
1 1 1
+ =
𝑆𝑆 ′ 𝑆𝑆 𝑓𝑓
blur kernel
object distance S sensor distance S’
Shift-invariant blur.
GAMES204 Computational Imaging, Qilin Sun
Lens Imperfections
(Important note: Oblique aberrations like coma and distortion are
What causes lens imperfections? not shift-invariant blur and we do not consider them here!)
Aberrations.
Diffraction
small large
aperture aperture
GAMES204 Computational Imaging, Qilin Sun
Lens: An Optical Low-pass Filter
Point spread function (PSF): The blur kernel of a lens.
“Diffraction-limited” PSF: No aberrations, only diffraction. Determined by aperture shape.
1 1 1
+ =
𝑆𝑆 ′ 𝑆𝑆 𝑓𝑓
blur kernel
diffraction-limited PSF
of a circular aperture
object distance S sensor distance S’ (Airy pattern)
GAMES204 Computational Imaging, Qilin Sun
Basics of Diffraction Theory
Assume that we can use:
Fraunhofer diffraction
i.e., distance of sensor and aperture is large relative to wavelength.
Incoherent illumination
i.e., the light we are measuring is not laser light.
Ignore various scale factors. Different functions are not drawn to scale.
GAMES204 Computational Imaging, Qilin Sun
Basics of Diffraction Theory
aperture:
rect 𝑥𝑥
The 1D case
?
GAMES204 Computational Imaging, Qilin Sun
Basics of Diffraction Theory
coherent point spread
function: sinc 𝑥𝑥
aperture:
rect 𝑥𝑥
The 1D case ?
GAMES204 Computational Imaging, Qilin Sun
Basics of Diffraction Theory
coherent point spread
function: sinc 𝑥𝑥
aperture:
rect 𝑥𝑥 ?
optical transfer
The 1D case
function: tent 𝑥𝑥
GAMES204 Computational Imaging, Qilin Sun
Basics of Diffraction Theory
coherent point spread
function: sinc 𝑥𝑥
aperture: incoherent point spread
rect 𝑥𝑥 function: sinc 2 𝑥𝑥
why do we get the
same result?
optical transfer
The 1D case
function: tent 𝑥𝑥
GAMES204 Computational Imaging, Qilin Sun
Basics of Diffraction Theory
coherent point spread
function: sinc 𝑥𝑥
aperture: incoherent point spread
rect 𝑥𝑥 function: sinc 2 𝑥𝑥
what happens if we
increase the aperture
size?
optical transfer
The 1D case
function: tent 𝑥𝑥
GAMES204 Computational Imaging, Qilin Sun
Basics of Diffraction Theory
coherent point spread
function: sinc 2𝑥𝑥
aperture: incoherent point spread
rect 𝑥𝑥/2 function: sinc 2 2𝑥𝑥
optical transfer
The 1D case
function: tent 𝑥𝑥/2
GAMES204 Computational Imaging, Qilin Sun
Basics of Diffraction Theory
coherent point spread
function: sinc 10𝑥𝑥
aperture: incoherent point spread
rect 𝑥𝑥/10 function: sinc 2 10𝑥𝑥
As the aperture size … point spread function
increases… becomes smaller
optical transfer
The 1D case function: tent(𝑥𝑥/
10)
GAMES204 Computational Imaging, Qilin Sun
Basics of Diffraction Theory
aperture incoherent point spread
function
As the aperture size … point spread function
increases… becomes smaller
optical transfer
The 2D case
function
GAMES204 Computational Imaging, Qilin Sun
Basics of Diffraction Theory
aperture incoherent point spread
function
As the aperture size … point spread function
increases… becomes smaller
optical transfer
The 2D case
function
GAMES204 Computational Imaging, Qilin Sun
Basics of Diffraction Theory
Why do we prefer circular apertures?
aperture incoherent point spread
function
As the aperture size … point spread function
increases… becomes smaller
optical transfer
The 2D case
function
GAMES204 Computational Imaging, Qilin Sun
Basics of Diffraction Theory
Other shapes produce very
anisotropic blur.
aperture incoherent point spread
function
As the aperture size … point spread function
increases… becomes smaller
optical transfer
The 2D case
function
GAMES204 Computational Imaging, Qilin Sun
Lens: An Optical Low-pass Filter
Point spread function (PSF): The blur kernel of a lens.
“Diffraction-limited” PSF: No aberrations, only diffraction. Determined by aperture shape.
blur kernel
aperture
diffraction-limited PSF
of a circular aperture
optical transfer function object distance S sensor distance S’ (Airy pattern)
(OTF)
GAMES204 Computational Imaging, Qilin Sun
Lens: An Optical Low-pass Filter
* =
image from a perfect lens imperfect lens PSF image from imperfect lens
x * c = b
If we know b and c, can we recover x?
GAMES204 Computational Imaging, Qilin Sun
Camera Shake
GAMES204 Computational Imaging, Qilin Sun
Camera Shake: Motion PSF
Motion PSF is a
H Function of:
Energy ~ 1/ speed
Motion path
Motion speed
Y
X
Spatial spread
GAMES204 Computational Imaging, Qilin Sun
Camera Shake as A Filter
* =
PSF
image from static camera from camera motion image from shaky camera
x * c = b
If we know b and c, can we recover x?
GAMES204 Computational Imaging, Qilin Sun
Sharp image Blur kernel
Multiple Possible Solutions
= *
= *
Blurry image
How do we
= * detect this
PSF?
GAMES204 Computational Imaging, Qilin Sun
Use Prior Information
Among all the possible pairs of images and blur kernels, select the ones where:
The image “looks like” a natural image.
The kernel “looks like” a motion PSF.
GAMES204 Computational Imaging, Qilin Sun
Shake Kernel Statistics
Gradients in natural images follow a
characteristic “heavy-tail” distribution.
sharp
natural
image
blurry
natural
image
Can be approximated by‖∇x ‖0.8
GAMES204 Computational Imaging, Qilin Sun
Use Prior Information
Among all the possible pairs of images and blur kernels, select the ones where:
The image “looks like” a natural image.
Gradients in natural images follow a
characteristic “heavy-tail” distribution.
The kernel “looks like” a motion PSF.
Shake kernels are very sparse, have
continuous contours, and are always positive
How to use this information for blind deconvolution?
GAMES204 Computational Imaging, Qilin Sun
Three Sources of Information
b = observed image c = blur kernel x = sharp image
𝑝𝑝(𝑐𝑐, 𝑥𝑥|𝑏𝑏) = 𝑘𝑘 𝑝𝑝(𝑏𝑏|𝑐𝑐, 𝑥𝑥) 𝑝𝑝(𝑥𝑥) 𝑝𝑝(𝑏𝑏)
GAMES204 Computational Imaging, Qilin Sun
Regularized Blind Deconvolution
Solve regularized least-squares optimization
minx,c ‖b – c ∗ x‖2 + ‖∇x‖0.8 + ‖ c ‖1
What does each term in this summation correspond to?
GAMES204 Computational Imaging, Qilin Sun
Regularized Blind Deconvolution
Solve regularized least-squares optimization
minx,c ‖b – c ∗ x‖2 + ‖∇x‖0.8 + ‖ c ‖1
* =
Estimated
Sharp image Blur kernel Blurry image
Note: Solving such optimization problems is complicated (no longer linear least squares).
GAMES204 Computational Imaging, Qilin Sun
A Demonstration
input deconvolved image and kernel
This image looks worse
than the original…
This doesn’t look like a
plausible shake kernel…
GAMES204 Computational Imaging, Qilin Sun
Regularized Blind Deconvolution
Solve regularized least-squares optimization
minx,c ‖b – c ∗ x‖2 + ‖∇x‖0.8 + ‖ c ‖1
cost function
GAMES204 Computational Imaging, Qilin Sun
Regularized Blind Deconvolution
Solve regularized least-squares optimization
minx,c ‖b – c ∗ x‖2 + ‖∇x‖0.8 + ‖ c ‖1
cost function
inverse loss
Where in this graph is the
solution we find?
pixel intensity
GAMES204 Computational Imaging, Qilin Sun
Regularized Blind Deconvolution
Solve regularized least-squares optimization
minx,c ‖b – c ∗ x‖2 + ‖∇x‖0.8 + ‖ c ‖1
cost function
inverse loss many plausible solutions here
Rather than keep just
maximum, do a weighted
average of all solutions
optimal solution pixel intensity
GAMES204 Computational Imaging, Qilin Sun
A Demonstration
input maximum-only average
GAMES204 Computational Imaging, Qilin Sun
Image Artifacts & Estimated Kernels
Blur kernels
Image patterns
Note: blur kernels were inferred from large image patches,NOT the image patterns shown
GAMES204 Computational Imaging, Qilin Sun
Scene Motion
GAMES204 Computational Imaging, Qilin Sun
Scene Motion Blur
Moshe Ben-Ezra and Shree K. Nayar Motion Deblurring Using Hybrid Imaging
GAMES204 Computational Imaging, Qilin Sun
Depth Defocus
GAMES204 Computational Imaging, Qilin Sun
Recall: Out of Focus Blur
Out of focus blur
circle of confusion
Focal plane
GAMES204 Computational Imaging, Qilin Sun
Recall: Out of Focus Blur
Out of focus blur
circle of confusion
Focal plane
GAMES204 Computational Imaging, Qilin Sun
Recall: Out of Focus Blur
Out of focus blur
circle of confusion
Focal plane
GAMES204 Computational Imaging, Qilin Sun
PSF Behavior of Different Depths
0.5m 0.7m 1.0m 1.5m
AL2550
0.5m 0.7m 1.0m 1.5m
ACA254
0.5m 0.7m 1.0m 1.5m
Ours
Qilin Sun et.al End-to-end Complex Lens Design with Differentiable Ray Tracing
GAMES204 Computational Imaging, Qilin Sun
Depth Defocus Examples
https://digital-photography-school.com/out-of-focus-photos/
GAMES204 Computational Imaging, Qilin Sun
Lens Imperfections and Physical Limit.
Camera Shake.
Scene Motion.
Today’s
Topic Depth Defocus.
Thank You!
