Introduction

ECE467
Image Processing & Robot Vision

Dr. Saqer S. Alja’afreh

Textbook Information

• Textbook:
Rick Szeliski, Computer Vision: Algorithms and Applications online at:
http://szeliski.org/Book/
Course Information Chapter 0

Course Information
• This course is an introduction to those areas of Artificial Intelligence that
deal with fundamental issues and techniques of computer vision and
image processing.

• The emphasis is on physical, mathematical, and information-

processing aspects of the vision.

• Topics to be covered include image formation, edge detection and

segmentation, convolution, image enhancement techniques, extraction
of features such as color, texture, and shape, object detection, 3-D vision,
and computer vision system architectures and applications.

• The material is based on undergraduate-level texts augmented with

research papers, as appropriate.
Course Information Chapter 0

• The course will move fast

• Self-discipline is important
• The emphasis of the course is to develop practical skills for solving
Computer Vision and Image Processing problems
• Fair evaluations: undergraduate and graduate students will be
scored separately
• Academic Integrity (AI) will be taken into consideration, please
refer to the course syllabus. (Work on homework and projects
independently)
Course Information Chapter 0

Preferable Skills

• Introduction to Programming Languages

• Knowledge of Linear Algebra
• Programming in Python Preferred
Course Information Chapter 0

Course Requirements
• Class attendance and participation is expected
• You are responsible for ALL materials presented in class and
assigned to read
• Quizzes will be given during class time only.
• Regular deliverables on the project will be graded during the
course
Course Information Chapter 0

Late Submission Policy

• Completed homework and project deliverables are to be
submitted by their deadline (11:59pm).
• For homework and projects, every day in delay will result in a
10% of deduction of its score.
Introduction Chapter 1

What is Computer Vision

• Computer vision is a field of computer science

• works on enabling computers to see,
• identify and process images in the same way that
human vision does, and
• then provide appropriate output.
• It is like imparting human intelligence and instincts
about vision to a computer.

23
Introduction Chapter 1

Every image tells a story

ThTS
• Goal of computer vision: perceive
the “story” behind the picture
definition
IS
a 8 - a

&
of
• Compute properties of the world the "
actual
• 3D shape definitiOh
• Names of people or objects of
Comp
• What happened? Vestor
.
Introduction Chapter 1

What is Computer Vision

• Computer vision is a field of computer science

• works on enabling computers to see,
• identify and process images in the same way that
human vision does, and
• then provide appropriate output.
• It is like imparting human intelligence and instincts
about vision to a computer.

23
Introduction Chapter 1

What is Computer Vision

• Automatic understanding of images and video

1. Computing properties of the 3D world from
visual data (measurement)
2. Algorithms and representations to allow a
machine to recognize objects, people, scenes,
and activities. (perception and interpretation)

23
Introduction Chapter 1
Visual Perception
• Definition: Process of acquiring knowledge about
environmental objects and events by extracting information
from the light they emit or reflect [Palmer, 2012].

Cognitive Acquisition of
Vision
Activity knowledge

Perception is analogous to taking a picture!

(credit: Palmer, 2012)
23
Introduction Chapter 1

Computer Vision vs Human Vision

24
Introduction Chapter 1
Computer Vision vs Human Vision
Interpreting
Sensing device device
Interpretations
Picture
Man
Thrash
Bulb
Light
…

24
Introduction Chapter 1
Can computers match human perception?
• Yes and no (mainly no)
• computers can be better at “easy”
things
• humans are better at “hard” things

• But huge progress

• Accelerating in the last five years due
to deep learning
• What is considered “hard” keeps
changing
Introduction Chapter 1
Human perception has its shortcomings

What can you see

in this picture?
Introduction Chapter 1
Human perception has its shortcomings

What can you see

in this picture?

Credit: Thompson, Basic Vision, Oxford Press, 2012.

Introduction Chapter 1
Human perception has its shortcomings

Copyright A.Kitaoka 2003

Introduction Chapter 1
But humans can tell a lot about a scene from a little
information…

Source: “80 million tiny images” by Torralba, et al.

Introduction Chapter 1

Related disciplines
Artificial
intelligence Machine
Graphics learning
Computer
Image vision Cognitive
processing science
Algorithms

25
Introduction Chapter 1
Chapter 0 Week 1

-
Al tool applied
structured (labeled
S

for
Jata .
Introduction Chapter 1

Bothfools
Suitable for

unstructured Jata
Introduction Chapter 1
Image Segmentation extraction
Manual Feature =

& L for
Structured
equires ma data

Features distinguishing
properties

annotation
one
byimage

Image Processing Important steps

computer vision
initial step : Image processing
Introduction Chapter 1
fre
high precision J useful
high accuracy structures
Jata

humans only
needed for usedfor unstructured data:
preprocessing
& implementation cunsupervised
learning]

00

neural O
network
Introduction Chapter 1
Introduction Chapter 1
Introduction Chapter 1
Image Processing vs. Computer Vision
• Image Processing
• Research area within electrical engineering/signal processing
• Focus on syntax, low level features

image image
(Denoising/inpainting)
• Computer Vision Coutcome of computer vision is story
a

• Research area within computer science/artificial intelligence

• Focus on semantics, symbolic or geometric descriptions
31
Faces, People
Chairs, etc.
(Recognition/Detection)
image
Introduction Chapter 1
21 Spatial signals
What is a (digital) Image? An image is an array of
numbers (pixels).

32

What humans see What Computer see

Introduction Chapter 1
What is a (digital) Image?
• Definition: A digital image is defined by integrating and sampling
continuous (analog) data in a spatial domain [Klette, 2014].
Left Hand

O
32

coordinate system Left hand coordinate

system
Introduction Chapter 1
What Digital Image Processing?

• To Bridge the Gap

between Pixels and
Meaning

32
Introduction Chapter 1
perinch
-
tridot
Image Types: (Gray)Scalar and Binary resizing an
image

-y changes
the Spr

• A scalar image has integer simplest images

a=8 a=3 a=2
values Not complex
• a: level (bit)
• Ex. If 8 bit (a=8)
• image spans from 0 to
255
• 0 black and 255 white
• Ex. If 1 bit (a=1) 32

• it is binary image
# of colors =
2"
• 0 and 1 only
Grayscale
112
Image ot colors 2 = =
246

# of pixels/image 512x236 131072

1266
= =

1048576 bimage
z
#
bit/image i Pixe
/*8
131 077hd
every single pixel # of
bytes 131072 bytes
=
= .

8 Gets
is represented
by
8 lors
2 Co
4 biES
#of
-
bit
bytes
Introduction Chapter 1
Image Type: RGB (red, green, blue)
• Each channel spans a-bit values. Human Cone-cells (normalized)
RGB3 image responsivity spectra

seutet

Goverlanges
into when
31
, Gina
32
Wavelength (nm)
• Some people might have 4 cone-types!
• Some might have just 2!
Each has a different array
Introduction Chapter 1
Color
• Color vision has evolved over millions of years.
visible
light Normalized
curve edium
hort ong

cone
comes
Come

C fX=

X =
f= ; ↑f + xy

wavelength
32

Shortest Longest
M M
Introduction Chapter 1
Color
• If there is no light, there is no color!
• Human vision can only discriminate a few dozens of grey levels on a
screen, but hundreds of thousands of different colors.
• RED -> ~625 to 780 nm [long wavelength]
• ORANGE -> ~ 590 to 625 nm [long wavelength]
• YELLOW -> ~565 to 590 nm [middle range wavelength]
• GREEN -> ~ 500 to 565 nm [middle range wavelength] [
• CYAN -> ~485 to 500 nm middle range wavelength]
• BLUE -> ~440 to 485 nm [short wavelength]
• VIOLET -> ~330 to 440 nm [very short wavelength] 32
Introduction Chapter 1
Retina of Human Eye

There are three different types of color-

sensitive cones corresponding to (roughly)
• RED (64% of the cones)
• GREEN (about 32%), and
↓ Photoreceptor
Gone Trols
• BLUE (about 2%).

6-7 million cones ·

responsible -Vision

120 million rods Station Jurongt

during day
Some may have only 2 cones 32

Credit: Klette, 2012.

 eyes
Photoreceptors

Efcones Cross
 Img Annotation =
Inglabeling
/
Bound Box 17
Polygon polyline
Answer this
In what I
question
cases
are we
:

going
to use
enchecase 3
adv & disadv ?
Introduction Chapter 1

Why Computer Vision?

32
Introduction Chapter 1
The goal of computer vision
• Compute the 3D shape of the world

ZED 2i
Camera

32
Introduction Chapter 1
The goal of computer vision
e
• Recognize objects and people
color

32
Introduction Chapter 1
The goal of computer vision
• “Enhance” images

32
Introduction Chapter 1
The goal of computer vision

32
Introduction Chapter 1
The goal of computer vision
• Forensics

32

Source: Nayar and Nishino, “Eyes for Relighting”

Introduction Chapter 1
The goal of computer vision

32
Introduction Chapter 1
The goal of computer vision

32
Introduction Chapter 1
The goal of computer vision

32

Source: Nayar and Nishino, “Eyes for Relighting”

Introduction Chapter 1
The goal of computer vision
• Billions of images/videos captured per day

• Huge number of potential applications 32

• The next slides show the current state of the art

Introduction Chapter 1
The goal of computer vision
• Optical character recognition (OCR)
• If you have a scanner, it probably came with OCR software

Automatic check processing

Digit recognition, AT&T labs (1990’s) License plate readers

http://yann.lecun.com/exdb/lenet/ http://en.wikipedia.org/wiki/Automatic_number_plate_recognition

32
Introduction Chapter 1
The goal of computer vision
• Face detection

• Nearly all cameras detect faces in real time

• (Why?)
Introduction Chapter 1
The goal of computer vision

• Face analysis and

recognition
Introduction Chapter 1
The goal of computer vision
• Login without a password (Face ID)

Fingerprint scanners on Face unlock on Apple iPhone X

many new smartphones and See also http://www.sensiblevision.com/
other devices
Introduction Chapter 1
The goal of computer vision
• Image synthesis

Karras, et al., Progressive Growing of GANs for Improved Quality, Stability, and Variation, ICLR 2018
Introduction Chapter 1
The goal of computer vision

• Sports and
Advertising
 Introduction Chapter 1
The goal of computer vision

• Smart cars

• Mobileye
• Tesla Autopilot
• Safety features in many cars
Introduction Chapter 1
The goal of computer vision

• Self-driving cars

Waymo
 Introduction Chapter 1
The goal of computer vision

• Robotics

Eig E NASA’s Mars Curiosity Rover Amazon Picking Challenge

http://www.robocup2016.org/en/events
https://en.wikipedia.org/wiki/Curiosity_(rover)
/amazon-picking-challenge/

computer
vision Amazon Prime Air Amazon Scout
Introduction Chapter 1
The goal of computer vision
• Medical imaging

3D imaging
(MRI, CT) Skin cancer classification with deep learning
https://cs.stanford.edu/people/esteva/nature/
Introduction Chapter 1
The goal of computer vision
• Virtual & Augmented
Reality

6DoF head tracking Hand & body tracking

3D scene understanding 3D-360 video capture

Introduction Chapter 1
Why computer vision is difficult?
Stroblems
to
fall
Ch 3
①
.

Viewpoint variation

Credit: Flickr user michaelpaul

Credit: Flickr user michaelpaul

② Illumination ⑤ Scale
Introduction Chapter 1
Why computer vision is difficult? more problems

Motion (Source: S. Lazebnik)

Blurry Zig
Intra-class variation
Different features
with the same object :
-Bearded vS unbearded
.

Nigab VS
.
Nigabless

Background clutter Occlusion (Hidden)

Introduction Chapter 1
Why computer vision is difficult?
Challenges: local
ambiguity

slide credit: Fei-Fei, Fergus & Torralba

Introduction Chapter 1
Can we manage these difficulties?
Yes, there are lots of
visual cues we can
use…
-10

• We often must use

prior knowledge
about the world’s
structure
Introduction Chapter 1
Can we manage these difficulties?

What do we lose
geometrically?

• Angles
• Distances
• and therefore Area
Introduction Chapter 1
Can we manage these difficulties?

• Vanishing points and lines

Parallel lines in the world

intersect in the image at
a “vanishing point”
Introduction Chapter 1
Can we manage these difficulties? Vertical
vanishing
Example: point
(at infinity)

Vanishi
Vanishi
ng
ng
point
point
Slide from Efros, Photo from Criminisi

1
Introduction Chapter 1
Can we manage these difficulties?
• Any two lines, parallel in
3D will meet at a unique
vanishing point in image
plane.
• All pairs of parallel lines on
the same plane in 3D will
have vanishing points on a
unique vanishing line.

1
Introduction Chapter 1
Course Overview (Tentative)

1. Low-level vision
• image processing, edge detection,
feature detection, cameras, image
formation
2. Geometry and algorithms
• projective geometry, stereo,
structure from motion,
optimization
3. Recognition
• face detection / recognition,
category recognition,
segmentation
Introduction Chapter 1
Course Overview (Tentative)
1. Low-level vision
• Basic image processing and image formation

* =
Filtering, edge detection

Image formation
Feature extraction
Introduction Chapter 1
Course Overview (Tentative)
2. Geometry

Image credit: IDS Imaging

Projective geometry Stereo vision

Multi-view stereo Structure from motion

Introduction Chapter 1
Course Overview (Tentative)
2. Recognition
“dog”

Image classification

Object detection

Convolutional Neural Networks