Lecture Notes 11

Introduction to Color Imaging

• Color filter options

• Color processing
• Color interpolation (demozaicing)
• White balancing
• Color correction

EE 392B: Color Imaging 11-1

 Preliminaries

• Up till now we have been only discussing gray scale image capture
◦ If the incident photon flux density at a pixel is f0(λ) ph/cm3.s, for
400 ≤ λ ≤ 700 nm, then the resulting photocurrent density
!
jph = q f0(λ)QE(λ)dλ A/cm2,
where QE(λ) e-/ph is the photodetector QE, which is a function of
the technology parameters
◦ Assuming constant jph over pixel area and over time (and ignoring
dark current and noise) we get a pixel output (voltage)
!
vo ∝ f0(λ)QE(λ)dλ
• To capture color images, each pixel must output more information about
the spectral distribution of the incident photon flux (f0(λ))
• A key fact from color science is that we do not need to completely know
the incident photon flux spectral distribution to faithfully reproduce color
– in fact only three values per pixel can be sufficient
EE 392B: Color Imaging 11-2
 • Reason: the human eye has three types of photodetectors (cones) L, M,
and S with different spectral sensitivities

1
S
0.9 M
L

0.8
Normalized Spectral Sensitivity

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0
350 400 450 500 550 600 650 700 750
Wavelength(nm)

EE 392B: Color Imaging 11-3

 • So under uniform illumination (photon flux density f0(λ)) the color we see
can be represented by a 3-dimensional vector
⎡ ⎤ ⎡( ⎤
L f0(λ)l(λ)dλ
⎢ ⎥ ⎢ ( ⎥
C = ⎣ M ⎦ = ⎣ f0(λ)m(λ)dλ ⎦
(
S f0(λ)s(λ)dλ
Or using discrete λ values as
⎡ ⎤
⎢ ⎥
⎡ ⎤⎢⎢ ⎥
⎥
lT (λ) ⎢ ⎥
⎢ ⎥⎢⎢ ⎥
C=⎣ mT (λ) ⎦ ⎢ F0(λ) ⎥
⎥
sT (λ) ⎢ ⎥
⎢ ⎥
⎢ ⎥
⎣ ⎦

Thus C can be expressed as a linear combination of three basis vectors

C=L·l+M ·m+S·s

EE 392B: Color Imaging 11-4

 • Note: photon flux densities with different spectral distributions can
produce the same perceived color (these are called metamers), e.g.,
Power spectral density of A Power spectral density of B
200 900

180 800

160 700

140 600

Relative Power

Relative Power
120 500

100 400

80 300

60 200

40 100

20 0
400 500 600 700 400 500 600 700
Wavelength(nm) Wavelength(nm)

• The color basis vectors are not unique — we can use different basis
vectors to represent color, e.g., RGB (but we must be careful in selecting
the spectral responses for the basis vectors), so for example we can write
C=R·r+G·g+B·b
EE 392B: Color Imaging 11-5
 • C can be transformed from one basis vector representation to another (or
from one color space to another) using a 3 × 3 matrix (more on this later)
• To get the three values from a pixel, color filters with different spectral
responses are used, e.g., R, G, B filters
• So if we denote the R filter response by φR(λ), the R output from a pixel
with photon flux density f0(λ) is
!
voR ∝ f0(λ)η(λ)φR (λ)dλ

and similarly for the other filters

• The “camera” RGB spectral responses are the products of each filter’s
response and the photodetector spectral response, i.e., φR(λ)η(λ), . . . etc.

EE 392B: Color Imaging 11-6

 • Example: RGB spectral responses for a Kodak digital camera
0.8

B
G
0.7 R

0.6

0.5
Spectral Response

0.4

0.3

0.2

0.1

0
400 450 500 550 600 650 700
Wavelength(nm)

EE 392B: Color Imaging 11-7

 Color filter options

• Use three image sensors and a beam splitter (prism)

+ Every photon finds its way to a sensor
+ High spatial resolution
− High cost, nonoverlapping color filter spectra not desirable
• Use time-switched color filter
+ High spatial resolution, each color can have different exposure time
− Longer exposure time — motion blur can be a problem
− Optical loss due to filters, high cost (rarely used)
• Use color filter array (CFA) or mosaic deposited on top of the pixel array,
so each pixel outputs only one color component, e.g., R, G, or B
+ Lowest cost option
− Lower spatial resolution, optical loss due to filters
− Processing (demozaicing) needed to reconstruct the missing color
components for each pixel
EE 392B: Color Imaging 11-8
 Color Filter Arrays

EE 392B: Color Imaging 11-9

 Color Processing

Object Display
Camera

Eye

• Color processing is needed to (i) reconstruct missing pixel color

components and (ii) to produce color (on a display device) that is close to
what the eye would perceive
EE 392B: Color Imaging 11-10
 • Typical color processing steps in a digital camera

Color White Color Gamma Color

From Interpolation Balancing Correction Correction Conversion To
ADC DSP

◦ White balance: used to adjust for illuminant so that, for example, a

white background appears white (the eye does this adaptively)
◦ Color correction: transforms the camera output to the color space
of the display, or to a standard color space
◦ Gamma correction: corrects for display nonlinearity, also needed
before image processing/compression
◦ Color conversion: needed before image processing/compression
• Color processing is performed mostly in the digital domain (but sometimes
in analog, e.g., white balancing)
• It is computationally very demanding (about 70% of processing in a
digital camera is related to color)

EE 392B: Color Imaging 11-11

 Color Interpolation (Demozaicing)

• Used to reconstruct the missing pixel color components (when a CFA is

used)
• Interpolation method must
◦ Reduce artifacts such as aliasing and color fringing (false colors)
◦ Have reasonable computational complexity
Interpolation algorithms:
• Nearest neighbor replication
◦ To reconstruct a missing color component of a pixel, simply set it
equal to the value of its nearest pixel with that color
◦ Simple and fast, but results in large artifacts especially at edges
• Bilinear interpolation
◦ Perform bilinear interpolation in each color plane

EE 392B: Color Imaging 11-12

 ◦ relatively simple, but still suffers from some edge artifacts (may not
be visible in a video sequence)
• 2-D filtering
◦ This is a generalization of bilinear interpolation
◦ The filter window size and coefficients for each color plane are
designed to reduce artifacts
◦ Artifacts will still exist around edges
• Adaptive algorithms
◦ Since most artifacts occur around edges, change (adapt) the
interpolation method when edges are present
◦ Yields better performance but requires more computations (for edge
detection)

EE 392B: Color Imaging 11-13

 Interpolation Algorithms — Examples

• Consider the Bayer pattern

B1 G2 B3 G

G4 R5 G6 R

B7 G8 B9 G
G R G R

• Bilinear interpolation
G2+G4+G6+G8
◦ G5 = 4
B1+B3+B7+B9
◦ B5 = 4
B1+B3
◦ B2 = 2

• Adaptive algorithm
◦ interpolation results in color fringing and zipper effects along edges
— most significant for luminance (green), since the eye is more
EE 392B: Color Imaging 11-14
 sensitive to spatial variation in luminance than chrominance
◦ For each pixel (with missing green) perform edge detection before
interpolation, and only use pixels along edges
◦ For example, assume that the pixels to the left of the edge have
larger pixel values than the ones on the right

R G R

G B G

R G R

Instead of using the four greens to estimate the missing green value
of the blue pixel, which would result in color fringing, we only use the
two greens along the edge
What if the edge is diagonal?
Use larger region for interpolation . . .

EE 392B: Color Imaging 11-15

 White Balancing

• Different light sources (illuminants) have different power spectral densities

(psd)
• The psd of color reflected from an object is a function of both the
object’s surface reflectance and the illuminant psd
more specifically the photon flux density at a pixel is proportional to the
product of the object surface reflectance S(λ) and the illuminant psd
E(λ), i.e., f0(λ) ∝ E(λ)S(λ)
• So, for example, a raw image taken by a camera of a white piece of paper
will look yellowish under incandescent lighting and greenish under
fluorescent lighting compared to under day light
• The eye, by comparison, would see the white paper as white almost
independent of the illuminant, and in a scene with white background it
adjusts the colors in the scene so that the background looks white
• Captured images must also be processed so that a white background
looks white — this is called white (color) balancing
EE 392B: Color Imaging 11-16
 Two Approaches to White Balancing

• Fixed white balance, i.e., with known illuminant

◦ Capture images of a white piece of paper under each potential
illuminant (the first illuminant being the standard one where the
image looks white), for each illuminant
∗ Compute the average value for each color channel (Ri, Gi, Bi)
∗ Compute the ratio between each color channel and the green
channel (luminance), i.e., Ri and Bi
Gi Gi
∗ Normalize each ratio
) * by the
) corresponding
* ratio of the first
illuminant to get Ri / R1 . . .
Gi G1
◦ To perform white balancing for a captured image with known
illuminant, divide the red and blue values by the appropriate
normalized ratios

EE 392B: Color Imaging 11-17

 • Automatic white balance is used if we do not know the illuminant
◦ Most algorithms used in cameras are proprietary, most use some
variation of the Gray World assumption
◦ The Gray World assumption is that over all scenes Ravg = Gavg = Bavg
◦ Simple GrayWorld Algorithm: equalize the averages for the three
R
color channels by dividing each red value by ( Gavg
avg
) and each blue by
B
( Gavg
avg
) (this works ok except when we have “atypical” scenes, e.g., a
forest with mostly bright green leaves — the image will look grayish
after white balancing)
◦ Another approach is to use the color information to estimate (or
decide on) the illuminant

EE 392B: Color Imaging 11-18

 Color Correction

• Color filter technology and photodetector spectral response determine the

camera color space
• To ensure that color from the camera looks the same on a display, the
camera output must be transformed to the display color spectral response
space
• Since there may be many display types used to render a captured image, it
is customary to transform the camera output to a standard color space,
e.g., corresponding to the LMS spectral responses, in the camera —
correction for each display type is performed outside the camera
• To transform the camera output to a standard color space we use a 3 × 3
matrix D, thus if C is the color from a pixel, the corrected color
Co = DC

EE 392B: Color Imaging 11-19

 • So how do we find D ?
If A1 is the camera spectral response matrix (3 × n) and A2 is the LMS
spectral response matrix (3 × n), then we can select D such that DA1 is
as close to A2 as possible, which can be done, for example, using least
squares
• This seems to work well, the following is the corrected RGB (of the Kodak
camera) compared to LMS spectral responses

EE 392B: Color Imaging 11-20

 Gamma Correction

• Intensity of the light generated by a display device Z is not linear in its

input Y , e.g., Z ∝ Y 2.22
• Must prewarp the image sensor output X so that the output of the
display is linear in the illumination at the camera — done using a
companding function, e.g., Y ∝ X 0.45
• Also needed prior to image enhancement and compression
◦ Most image processing algorithms assume pixel values proportional
to perceptual brightness, which is close to the prewarped value Y
• Typically implemented using 3 lookup tables, one for each color
component

EE 392B: Color Imaging 11-21

 Color Space Conversion

• Transform RGB to YCbCr, or to YUV using 3 × 3 matrix

⎡ ⎤ ⎡ ⎤⎡ ⎤
Y a11 a12 a13 R
⎢ ⎥ ⎢ ⎥⎢ ⎥
⎣Cb⎦ = ⎣a21 a22 a23⎦ ⎣G⎦
Cr a31 a32 a33 B
• Most image enhancement and compression are performed on luminance
and chrominace values separately
◦ Eye is more sensitive to luminance than to chrominance
◦ Preserve color before and after processing

EE 392B: Color Imaging 11-22