Video Fundamentals: Signal Processing For Digital TV

Video Fundamentals
Signal Processing for Digital TV

Presented to IEEE OC Computer Society by Dwight Borses, MTS FAE National Semiconductor, Irvine
Original Presentation Materials Developed by: Dr. Nikhil Balram, CTO and Dr. Gwyn Edwards, TME
National Semiconductor Displays Group
Tonights Presentation
Digital Television architecture and functionality NTSC Background (with a touch of PAL and SECAM) Major video processing building blocks Application examples
2
2004 National Semiconductor Corporation
Definition of Digital Television

Any display system that
Has digital processing of one or more of its inputs Includes the function of showing video
Divide into 4 segments

Monitor/TV
TV/Monitor
Digital monitor with video function 15 XGA LCD Monitor/TV HD-Ready TV / EDTV / SDTV / 100 Hz TV digital displays often include monitor as additional function Integrated HDTV (USA); iDTV [Integrated Digital TV] (Europe); BS Digital (Japan) Internet connectivity, built-in hard-drive (PVR), interactivity etc
MPEG TV
Smart (IP) TV
3
Digital Television
Smart TV
iPTV
+ Media Communication Processor
MPEG TV
HDTV
+ ATSC tuner + VSB/QAM/ QPSK Receiver
+ MPEG Processor + Transport Demux + Multiple Stream Decoder
TV/Monitor
SDTV/EDTV/HDTV-READY
+ 3D Decoder
+ 3D Deinterlacer + Dual Scalers + Intelligent Color mgmt + FRC Display Processor 2D Deinterlacer Scaling Color Mgmt OSD
4
Monitor/TV
RF front end (type A) (NTSC/PAL/SECAM)
Baseband Front-end ADC DVI-HDCP 2D Video Decoder
The Modern HD Ready TV Set

RF Source(s)
Sound IF (SIF) 4.5 to 6.5 MHz Other Video Sources CVBS Tuner & Demodulator Video Decoder YCbCr RGB Other Audio Sources Display Electronics (Display Dependent) Audio Decoder & Amplifiers Display e.g., CRT, LCD, DLP, LCOS, Plasma
Video Deinterlacer, Scaler & CSC
The tuner extracts one TV channel at a time from many, then downconverts and demodulates the signal to baseband The video (or color) decoder separates the colors from the composite (CVBS) signal The deinterlacer and scaler converts the format of the picture to match that of the display type (optional for CRT TVs) The display electronics converts the signal format to match that of the display type: e.g. analog for CRT, LVDS for LCD panel
5
Functional System Architecture for MPEG TV

ATSC/NTSC/PAL Tuners VSB/QAM Rcvr MPEG Decoder Audio IR/Keypad CPU
System I/F
CVBS
VBI/CC Teletext
Y/C
YUV
3D Decoder
OSD
3D Decoder S E L E C T O R
CVBS Y/C YPrPb (HD)
3D Deinterlacer & NR
Display I/F
Scaler
Blending & Color Mgt. Output Format
RGB/YUV TTL LVDS Analog
ADC/Sync
RGB (VGA)
DVI-HDCP
DVI / HDCP Receiver
3D Deinterlacer & NR
Scaler
Frame Buffer (SDRAM)
6
System Interfaces
RF NTSC RF ATSC Baseband analog NTSC
Composite (CVBS) S-Video (Y/C) Component (YUV)
Analog HD component (YPbPr) Analog PC graphics (VGA) Digital PC graphics (DVI-HDCP) Digital HD
DVI-HDCP [High Definition Content Protection] from PC space used by STBs and current generation of HD-Ready TV HDMI - New CE version of DVI adds audio, video formats, control functions
7
High Definition Multimedia Interface (HDMI)

HDMI is DVI plus
Audio Support for YCbCr video CE control bus Additional control and configuration capabilities Small CE-friendly connector
Supported video and audio formats
Video and audio stream information
HDMI enables device communication

To source To display
Developed by the HDMI Working Group 1.0 specification released Dec. 2002
Hitachi, Panasonic, Philips, Silicon Image, Sony, Thomson, Toshiba
Information courtesy of Silicon Image Inc.
8
Color Television Systems

Color TV systems developed in 50s (NTSC) and 60s (PAL) Backward compatibility with monochrome TVs more important than color quality! Basic parameters of signal (carrier frequencies, bandwidths, modulation format, etc.) had to remain unchanged NTSC and PAL systems added chrominance (color) information to luminance (brightness) signal in a manner transparent to monochrome TVs
11
NTSC Fundamentals
Approved in US by FCC in 1953 as color system compatible with existing 525 line, 60 fields/sec, 2:1 interlace monochrome system Color added to existing luminance structure by interleaving luma and chroma in frequency domain Basic properties
525 lines/frame 2:1 interlace 2 fields/frame with 262.5 lines/field Field rate 59.94 Hz Line frequency (fh) = 15.734 KHz Chroma subcarrier frequency (fsc) = 3.58MHz = 227.5 fh = 119437.5 fv
chosen so that consecutive lines and frames have opposite (180o) phase
Luma: Y = 0.299R + 0.587 G + 0.114 B, where R, G, B are gamma-corrected R, G, B Chroma: I (In-phase) and Q (Quadrature) used instead of color difference signals U, V Composite = Y + Q sin(wt) + I cos(wt) + sync + blanking + color burst, where w = 2 pi fsc
12
U = 0.492 (B-Y), V = 0.877 (R-Y) I = V cos33o - U sin33o, Q = V sin33o + U cos33o
Monochrome TV Signals (NTSC)

Video Carrier Audio Sub-Carrier
Video Signal
Audio Signal
4.5 MHz
In the NTSC monochrome system the luminance signal is AM/VSB (Amplitude Modulation/Vestigial Sideband) modulated onto the video carrier The sound signal is FM modulated onto the Audio Sub-Carrier located 4.5 MHz from the video carrier
13
Spectrum of Monochrome TV Signal (NTSC)

Video Carrier = Frame freq. (30 Hz) = Line freq. (15.734 kHz)
Detail
Spectrum of the video extends from just below the video
carrier frequency to just below the sound carrier

Repetitive nature of the signal from line to line and frame to frame results in a picket-fence, or comb, spectrum
14
Color in Video
In PC space, R+G+B signals generate color images
In video space, color signals developed for backward

compatibility with monochrome TVs Image brightness represented by luma signal (Y),
equivalent of monochrome TV signal

Color added with color difference signals: Cb and Cr Matrix equation translates color spaces Y (luma) = 0.299R' + 0.587G' + 0.114B' Cb (blue chroma) = 0.492(B'-Y) Cr (red chroma) = 0.877(R'-Y)
15
Principles of NTSC Color System

Takes advantage of spectral nature of luminance signal Recognizes human eye is less sensitive to color changes than luma changes Low bandwidth chrominance information is modulated onto a Color Sub-Carrier and added to the luma signal The chroma signal has a picket-fence spectrum sub-carrier frequency very carefully chosen so of the chroma signal pickets are interlaced between those of the luma signal fSC = 227.5 x fH = 3.579545 MHz
16
Why a weird number like 59.94 Hz?

Early TV systems used local power line frequency as the field rate reference Europe used 50 Hz, the USA used 60 Hz With the introduction of color, audio subcarrier frequency required integer relationship to color subcarrier to prevent interference
Nearest value to the original 4.500 MHz was 4.5045 MHz, too large a difference for backward compatibility
Reducing field rate from 60 to 59.94 Hz, allowed integer value of 4.49999 MHz possible for audio subcarrier
This is close enough, solving the problem

17
The Result
Chroma Components Luma Components
Low-Level Mixing
The chroma components can be mostly separated from the luma with a comb filter Note the mixing of lower-level luma and chroma components, resulting in residual cross-luma and cross-color artifacts
18
Implementation of NTSC Color
Gamma correction applied to adjust for CRT non-linearity of Component color signals (R', G' and B') are converted to luma and chroma-difference signals with a matrix circuit: Cb and Cr are lowpass filtered, then quadrature modulated (QAM) onto the chroma sub-carrier Signal Amplitude represents the color saturation of video Phase represents the hue Chroma levels chosen such that peak level of composite signal does not exceed 100 IRE with 75% color bars
R G B Gamma Correction R' G' Matrix B' Cr Y Cb LP Filter Composite
LP Filter cos sin
Sub-carrier Generator
19
Spectrum of the NTSC Color Signal

Video Carrier Cr BW = 0.6 MHz Cb BW = 0.6 MHz Cr BW = 1.3 MHz Cb BW = 0.6 MHz 0 3.58 MHz 4.5 MHz Full chroma signal bandwidth, 1.3 MHz around sub-carrier, too wide for transmission within channel allocation Usually, both Cb and Cr bandwidths are reduced to 600 kHz Reduces cross-color and cross-luma in TV Alternatively, compliant to true NTSC specification:
Cb component (only) can be band-limited to 600 kHz Phase of sub-carrier rotated by 33, puts flesh tones at around 0 Results in asymmetrical signal (shown) The rotation aligns flesh tones to the I axis and is transparent to demodulator since color-burst is rotated by same amount
20
Color Audio Sub-Carrier Sub-Carrier
The NTSC Color Video signal (EIA 75% color bar signal)
=R+B+G =R+G =B+G =G =R+B =R =B =0
21
NTSC Color Video Signal

EIA 75% Color Bar Signal
100
Phase=241
White level
Phase=61 Phase=103
80
60
Phase=347
40
Color burst Phase=0 Phase=167
20 IRE 0 Magenta
Phase=283
Black level Blank level

Green Black Sync level
22
White
Cyan
9 cycles
- 40
Backporch
Blanking Interval Visible Line Interval
Red
Blue
-20
Yellow
PAL Fundamentals
European standard with many flavors - broadcasting begun in 1967 in Germany and UK. Similar in concept to NTSC, except that line and field timings are different, and the phase of the V (chroma) component is reversed every line to allow color phase errors to be averaged out Basic properties (except for PAL-M which has NTSC like rates)
625 lines/frame 2:1 interlace 2 fields/frame with 312.5 lines/field Field rate 50 Hz Line frequency (fh) = 15.625 KHz Chroma subcarrier frequency (fsc) = 4.43MHz = (1135/4 + 1/625) fh
Luma: Y = 0.299R + 0.587 G + 0.114 B, where R, G, B are gamma-corrected R, G, B Chroma: Usual color difference signals U, V Composite = Y + U sin(wt) +/- V cos(wt) + sync + blanking + color burst, where w = 2 pi fsc
U = 0.492 (B-Y), V = 0.877 (R-Y)
consecutive lines and frames have 90o phase shift, so 2 lines or 2 frames required for opposite phase
23
SECAM Fundamentals
Developed in France - broadcasting begun in 1967 Basic timing is identical to PAL but chroma is handled differently from NTSC/PAL Basic properties
Only one chroma component per line FM modulation is used to transmit chroma
625 lines/frame 2:1 interlace 2 fields/frame with 312.5 lines/field Field rate 50 Hz Line frequency (fh) = 15.625 KHz Luma: Y = 0.299R + 0.587 G + 0.114 B, where R, G, B are gamma-corrected R, G, B Chroma: Scaled color difference signals U, V
Db = 1.505 (B-Y), Dr = -1.902 (R-Y) only one chroma component per line, alternating between Dr, Db separate subcarriers for Dr, Db
24
Composite Video: NTSC / PAL / SECAM

Long-time world television standards Basic properties
Analog interlaced scanning
3D (H,V,T) information expressed as a 1D (temporal) raster scanned signal Each picture (frame) displayed as 2 interleaved fields - odd + even
Luminance (Y) and Chrominance (R-Y, B-Y), sync, blanking, and color reference information all combined into one composite signal
25
Luma/Chroma Separation
Many approaches trading off complexity, cost and performance
Basic approaches (historical)
Low pass luma / high pass chroma Notch luma / bandpass chroma 2D passive line comb filter 2D adaptive line comb filter 3D (spatio-temporal) comb filter
Advanced approaches (commonly used in most systems today)
Decoding artifacts
Loss of resolution Dot-crawl Cross-color
Good decoding requires some black magic (art) because luma and chroma spectrums overlap in real motion video
26
S-Video Signals
S-Video was developed in conjunction with the S-VHS VCR standard, where the luma and chroma signals are kept separate after initial Y/C separation Keeping the signals separate, i.e. never adding the luma and chroma back together, eliminates the NTSC artifacts Since video sources are generally composite (NTSC), the full benefit is not realized Keeping the signals separate after playback with the VCR does help, especially because of the timing jitter
27
Luma and Chroma Signal Separation (Y/C Separation)

The chroma signal (C) is separated from the composite signal by filtering
Adaptive comb filtering is required for high quality
Low cost TVs use a bandpass filter, resulting in incomplete separation and bad cross luma and chroma artifacts Non-adaptive comb filters introduce problems at edges
The luma signal (Y) may be derived by subtracting the chroma from the composite signal
Only works well if the chroma was separated well Low cost TVs use a bandstop filter to eliminate the
chroma, resulting in poor luma bandwidth
29
NTSC Color Video signal after Y/C Separation (EIA 75% color bar signal)
700mV
White level
0.961V
0.793V Blank level

Black level 0mV Blank level
0.507V
0.339V
-300mV Sync level
Luma
Chroma
30
The NTSC Color Video signal after chroma demodulation (EIA 75% color bar signal)
Cb
Cr
31
Chroma Demodulation
The Cb and Cr color difference signals are recovered by coherent demodulation of the QAM chroma signal
An absolute phase reference is provided to facilitate the process A color burst - 9 cycles of unmodulated color sub-carrier is added between the horizontal sync pulses and the start of the active video (the backporch)
32
Notch/LPF versus Comb Filtering

Comb filtering allows full bandwidth decoding
Notch filter: loss of information in 2-4 MHz region
Comb filter : full horizontal resolution
33
Comb Filtering (cont.)

Use 1 or more lines for each delay element, e.g., for NTSC, D = 1 line = 910 z-1 Apply cos version with positive coefficients to extract chroma or sin version with negative center coefficient to extract luma
34
HDTV Technical Overview

Video:
Audio: Dolby AC-3 Transport:
MPEG2 Main Profile @ High Level (MP@HL) 18 formats: 6 HD, 12 SD
RF/Transmission:
Terrestrial:
Subset of MPEG2 Fixed length 188-byte packets

8-VSB (Vestigial Side Band) with Trellis coding effective payload of ~19.3 Mb/s (18.9 Mb/s used for video) Uses QAM instead of VSB effective payload of ~38.6 Mb/s
Cable:
35
HDTV/DTV Overview
ATSC Formats
Vertical Horizontal Aspect Ratio Picture Rate 1080 1920 16:9 60I, 30P, 24P HDTV 720 1280 16:9 60P, 30P, 24P 480 704 16:9 & 4:3 60P, 60I, 30P, 24P SDTV 480 640 4:3 60P, 60I, 30P, 24P
18 formats: 6 HD, 12 SD 720 vertical lines and above considered High Definition Choice of supported formats left voluntary due to disagreement between broadcasters and computer industry Computer industry led by Microsoft wanted exclusion of interlace and initially use of only those formats which leave bandwidth for data services - HD0 subset Different picture rates depending on motion content of application 24 frames/sec for film 30 frames/sec for news and live coverage 60 fields/sec, 60 frames/sec for sports and other fast action content 1920 x 1080 @ 60 frames/sec not included because it requires ~100:1 compression to fit in 19.3 Mb/s terrestrial channel, which cannot be done at high quality with MPEG2
36
HDTV/DTV Overview
HDTV/DTV System Layers

layered system with header/descriptors
Picture Layer
996 Mb/s
1920 x 1080 @60I
Multiple Picture Formats and Frame Rates
Compression Layer
Data Headers
Chroma and Luma DCT Coefficients Variable Length Codes Packet Headers
Motion Vectors
MPEG-2 video and Dolby AC-3 compression syntax
Transport Layer
Flexible delivery of data and future extensibility

Aux data
Video packet
Audio packet
Video packet
MPEG-2 packets
Transmission Layer
19.3 Mb/s
6 MHz
8-VSB
37
Source:Sarnoff Corporation
HDTV/DTV Overview
Aspect Ratios
Two options: 16:9 and 4:3 4:3 standard aspect ratio for US TV and computer monitors HD formats are 16:9
better match with cinema aspect ratio better match for aspect ratio of human visual system better for some text/graphics tasks
allows side-by-side viewing of 2 pages
800
800
600
600
450
4:3 aspect ratio
16:9 aspect ratio

40
Additive White Gaussian Noise

Ubiquitous in any electronics systems where analog is present
Central Limit Theorem explains the underlying cause
Noise can be dramatically reduced by motion-adaptive recursive filtering (3D NR)
Basic equation:
Yi = X + Zi where Zi = measurement at time I, X = original data Wi = noise at time i = Gaussian white noise with zero mean
MMSE estimate for N measurements = (Yi)/N
Compute Average over same pixel location in each frame Noise averages to zero over a period of time Since averaging pixels that are in motion produces tails, we need reduce or stop averaging when there is motion
41
AWGN - Example
Original
After noise removal
With Gaussian noise

42
Impulse Noise Reduction

Use nonlinear spatial filtering to remove impulsive noise without reducing resolution
Original
After noise removal
With impulse noise
43
Digital (MPEG) Noise
Block Noise
Tiling effect caused by having different DC coefficients for neighboring 8x8 blocks of pixels
Mosquito Noise
Ringing around sharp edges caused by removal of highfrequency coefficients

Different choice of filters across block boundaries versus within blocks
Noise reduction is achieved by using adaptive filtering
44
Deinterlacing (Line Doubling)

Conversion of interlaced (alternate line) fields into progressive (every line) frames Required to present interlaced TV material on progressive display Odd Even
CRT-TV uses Interlaced scanning, with odd lines first followed by even lines
PC Monitor and all digital displays are Progressive - scanning all lines in consecutive order
45
Vertical-Temporal Progression
1 2 3 4 5 6 7 8 Lines = missing line
= original line (current field)

= original line (adjacent fields)
t-1
Current field
t+1
Time
47
Interlacing and Deinterlacing Artifacts

Interlacing artifacts
Twitter Wide-area flicker Temporal aliasing Line Crawl
Deinterlacing artifacts
Feathering/ghosting Jaggies/stepping Twitter Loss of vertical detail Motion judder Motion blur Specialized artifacts
48
Methods of Deinterlacing
Spatial interpolation (Bob) Temporal interpolation (Weave) Spatio-temporal interpolation Median filtering Motion-adaptive interpolation Motion-compensated interpolation Inverse 3-2 and 2-2 pulldown (for film) Other (statistical estimation, modelbased etc)
49
Film vs. Video

Nature of content is the most important factor Fundamentally two types Progressive and Interlaced Progressive content is content that was originally acquired in progressive form but converted to fit into an interlaced standard
Film-to-video (Telecin) process is used to convert film to the desired interlaced video format
24 frames/sec 50 fields/sec PAL by running film at 25 fps and doing 2:2 pulldown 24 frames/sec 60 fields/sec NTSC by doing 3:2 pulldown
Most common form of such content is film 24 frames/sec or 30 frames/sec Other forms include computer graphics/animation
50
Film-to-Video Transfer (NTSC)

1/24
second Movie frame 2
1/24
second
1/6
second
Movie frame 3
1/24
second Movie frame 4
1/24
second
Movie frame 1
Odd field
Even field Video frame 1
Odd field
Even field
Odd field
Odd field
Odd field
Video frame 2 Real time
Conversion of 24 frames/sec into 60 fields/sec: 4 movie frames mapped to 5 video frames In this process, one movie frame is mapped into 3 video fields, the next into 2, etc... Referred to as 3:2 Pulldown Similar process used to convert 25 frames/sec to 50 fields/sec and 30 frames/sec to 60 fields/sec (2:2 pulldown)
51
De-Interlacing of Film-Originated Material

Incorrect field pairing
Correct field pairing
52
De-Interlacing of Film-Originated Material

Without Film Mode
With Film Mode

53
Video
Odd field No motion Even field Odd + Even
Motion
Odd and even lines are in different places when there is motion
54
Video Deinterlacing Artifact Feathering

Feathering caused by improper handling of motion
55
Moving Edges in Video

Hardest problem in de-interlacing because odd and even lines are in different places Combining odd and even lines causes feathering Using spatial interpolation causes jaggies/staircasing
Angled Line
Line Doubled using Vertical Interpolation
56
Video Deinterlacing Artifact Jaggies / Staircasing

Jaggies/staircasing
Caused by vertical interpolation across lines in same field
Typical
57
Optimal Deinterlacing
Content-adaptive
Film vs. Video detect film and use inverse 3-2 (NTSC) or inverse 2-2 (PAL) pulldown Bad edit detection/compensation need to detect and compensate for incorrect cadence caused by editing Detect amount of motion and use appropriate mix of spatial and temporal processing
Motion-adaptive
Edge-adaptive
Highest resolution for still areas with no motion artifacts in moving areas
Interpolate along edge to get smoothest/most natural image
58
Film-Mode: Inverse Pulldown

From movie frame 1 Odd field Even field Odd field From movie frame 2 Even field Odd field Even field From movie frame 3 Odd field Even field From movie frame 4 Odd field Even field
Odd 1+ Even 1
Odd 1+ Even 1
Odd 1+ Even 1
Odd 2+ Even 2
Odd 2+ Even 2
Odd 3+ Even 3
Odd 3+ Even 3
Odd 3+ Even 3
Odd 4+ Even 4
Odd and even fields generated from the same original movie frame can be combined with no motion artifacts 3:2 Pulldown sequence detection is necessary Done by analysis of motion content
59
Bad Edit Detection and Correction

There are 25 potential edit breaks
2 Good edits 23 distinct disruptions of the film chain that cause visual bad edits Edit
Odd field Even field
Sequence has to be continuously monitored

From movie frame 1 From movie frame 3
From movie frame 4
Odd field
Even field
Odd field
Even field
Odd field
Even field
Odd 1+ Even 1
Odd 1+ Even 1
Odd 1+ Even 1
Odd 3+ Even 3
Odd 3+ Even 3
Odd 4+ Error Even 3
Odd 4+ Even 4
Film to video transitions - commercial insertion or news flashes

60
Motion-Adaptive Deinterlacing
1 2 3 4 5 6 7 8 Lines = missing line (m)(S) + (1-m)(T)
= original line (current field)

= original line (adjacent fields) m = motion S = spatial interpol. T = temporal interpol. t-1 t t+1 Current field Time
61
Motion-Adaptive Deinterlacing
Estimate motion at each pixel Use Motion value to cross-fade spatial and temporal interpolation at each pixel
Quality of motion detection is the differentiator

Motion window size Vertical detail Noise
Low motion means use more of temporal interpolation High motion means use more of spatial interpolation
62
Edge-Adaptive Deinterlacing
Moving edges are interpolated cleanly by adjusting the direction of interpolation at each pixel to best match the predominant local edge One Field One Field
Angled Line Angled Line
Line Doubled using Vertical Interpolation
Line Doubled using Edge-adaptive Interpolation
63
Scaling
Linear Scaling
Resolution conversion PIP/PAP/POP Aspect ratio conversion
Nonlinear scaling
Variable scaling Warping
Keystone correction
Resampling based on a mapping function
64
Upscaling
Input signal Intermediate signal
F(nT/2)
F(nT)
nT
12 3 456 7 8
Output signal
nT
Interpolating low-pass filter
F(nT/2) nT
1 2 3 456 7 8
nT
65
Downscaling
Input signal
F(nT)
Decimating low-pass filter prevents alias at lower rate

4 nT
Output signal
F(2nT)
2
66
Practical Scaling
Textbook scaling implies you need a very large filter when dealing with expanded signal In practice you only need a small number of filter coefficients (taps) at any particular interpolation point because of all the zero values The interpolation points are called phases
e.g., scaling by 4/3 requires 4 interpolation locations (phases) that repeat 0, 0.25, 0.5, 0.75 Pre-compute and store one set of filter coefficients for each phase Use DDA to step across the input space using step size = (input size / output size)
Practical scalers use polyphase interpolation
For each location, use filter coefficients corresponding to the current phase and compute the interpolated value
67
Xi = Xi-1 + Step Fractional portion of Xi represents the filter phase for current location
Upscaling Comments
Theoretically simpler than downscaling
often mistakenly referred to as aliasing.
However, poor reconstruction filter can introduce jaggies and Moir
Fixed length filter can be used since there is no concern about aliasing
Quarter zone plate upscaled using replication - shows jaggies
Quarter zone plate upscaled using interpolation smooth

68
Upscaling Comments (cont.)

Moir
Introduced by beating of high frequency content with first harmonic that is inadequately suppressed by the reconstruction filter.
Original sampled image: 1D sine wave grating - cos (2*pi*(0.45)x) - visible Moir
Upscaled 2X horizontally using linear interpolation - visible Moir Upscaled 2X horizontally using a 16-tap reconstruction filter - negligible Moir
69
Downscaling Comments
More difficult than upscaling
Each new scaling factor needs cutoff frequency of reconstruction filter to be altered. Inverse relationship between time (space) and frequency requires filter length to grow proportionately to shrink factor. Aliasing and lost information can be very visible when a fixed low-order filter is used
Grid downscaled using fixed 2-tap filter
Grid downscaled using filter with dynamic taps

70
Scaling for PIP/PAP/POP
Main
PIP
Main
PIP
Main
PIP
PIP Mode
PAP Mode
POP Mode
Main
TeleText
Live
PAT Mode
Mosaic Mode
71
Linear Scaling State of the Art

Polyphase interpolation Separate H and V scaling Typical number of phases from 8 to 64 Typical number of taps from 2 to 8 (H and V can be different), usually more than 2 (linear)
Keep in mind the fundamental differences between graphics and video Watch out for marketing gimmicks Total # of effective filter taps is NOT #taps x #phases, it is just #taps Correct definition of #Taps is how many input samples are used to compute an output sample
Graphics is non-Nyquist
72
HDTV/DTV Overview
Nonlinear Scaling for Aspect Ratio Conversion
Aspect ratio conversion is required for going between 4:3 and Widescreen Several options (shown below)
Video Transmission Format
16 9
4:3 material on 16:9 monitor 16:9 material on 4:3 monitor
16 x 9 Display Modes
Full Zoom Squeeze Variable Expand Full
4 x 3 Display Modes
Zoom Squeeze Variable Shrink
(a)
4 3 (e) (f) (g) (i)
(b)
(c)
(d)
(j)
(h)
73
Nonlinear Scaling Example Panoramic Mode

input
Non-linear 3 Zone scaling

The input aspect ratio is preserved in the middle zone of the output image while scaling. output
Aspect ratio slowly changes in the tail zones to accommodate rest of the input picture.
Horizontal nonlinear scaling

74
Non-linear 3 Zone Scaling Example
Linear Scaling 16:9
Original 4:3 image Nonlinear scaling 16:9

75
Vertical Keystone Correction
Image with vertical keystone correction and aspect ratio correction
Projection of the image
77
Edge Enhancement
Adaptive peaking
Extract high-pass filtered version of signal Apply gain Add back to original Compute derivative of signal Use shaped version of derivative to sharpen the transient without introducing undershoot or overshoot
Transient improvement
78
Picture Enhancement and Controls

Standard Picture Controls
Brightness, Contrast, Saturation, Hue or Tint New 6-point controls R, G, B, Cy, Mag, Yellow Intelligent Colour Remapping (ICRTM) produces more pleasing vivid images Locally Adaptive Contrast Enhancement (ACETM) expands the dynamic range of the scene to provide more detail sRGB Color space for internet
Advanced Picture Controls
Automatic contrast and colour enhancements
Color Management
91
Standard Global Picture Controls

Typically comprises of a fully programmable [(3x3) matrix + (3x1)
vector] Color-Space-Converter (CSC) and Look-Up-Table (LUT) Can be used to do linear color space transformations, standard picture controls (hue, saturation, brightness, contrast) and gamma correction
Hue Saturation
Brightness
Contrast
Original
92
New 6-point Control

Separate controls for 6 chroma channels R, G, B, Cyan, Magenta and Yellow Red
Yellow Magenta
Green
Blue
93
Cyan
Intelligent Color Remapping (ICRTM)

Example of automatic setting to enhance specific colour regions green grass
Greener grass
94
Intelligent Color Remapping (ICRTM)

Example of automatic setting to enhance specific colour regions blue sky
Bluer Sky
95
Locally Adaptive Contrast Enhancement (ACETM)
Original
Contrast enhanced
96
Application Examples
Basic LCD-TV/Monitor Fully Featured LCD-TV/Monitor Fully Featured MPEG-TV
97
Pixelworks Examples
Application Example 1: LCD TV/Monitor without PIP

Tuner Board
Inputs: Standard TV
HDTV (480p/720p/1080i)
AV-AL / AV-AR HD-AL / HD-AR PC-Audio
Audio Decoder MSP3450
Audio Amp TDA1517
Output: VGA WXGA

4:3 & 16:9, Progressive
TV-In
Tuner FI12X6
SDRAM
Key Features:
Motion Adaptive I/P Film Mode (3:2 & 2:2) Noise Reduction CC/V-Chip/Teletext Multi-Language UI IR Remote
AV-IN (composite) S-VID (s-video)
Video Decoder VPC3230
PW1230
Flash PromJet
YUV
LVDS V-chip / CC Z86129 PW113 90C383
Basic LCD-TV/Monitor
TTL
HD-Y/HD-Pb/HD-Pr (HD)
MUX (330) VGA-In ADC AD9883 keypad IR
Reference design courtesy of Pixelworks Inc.
98
Pixelworks Examples
Application example 2: LCD TV/Monitor with PIP

AV-AL / AV-AR
Inputs: Standard TV
HDTV (480p/720p/1080i)
HD-AL / HD-AR PC-Audio
Audio Decoder MSP3450
Audio Amp TDA8944J
Output: VGA WXGA

4:3 & 16:9, Progressive
TV-In AV-IN (composite) Tuner FI12X6
V-chip / CC / Teletext SAA5264 SDRAM
Key Features:
Motion Adaptive I/P Film Mode (3:2 & 2:2) Noise Reduction Multi-regional scaling PIP/split screen/POP CC/V-Chip/Teletext Multi-Language UI IR Remote
S-VID (s-video) YUV
Video Decoder SAA7118
PW1230
Flash PromJet
90C383 TV-In AV-In (composite) S-VID (s-video) YUV keypad IR Tuner FI12X6 PW181 Video Decoder SAA7118 Sil164
LVDS
TMDS
TTL
Fully Featured LCD-TV/Monitor
V-chip / CC / Teletext SAA5264 HD-Y/HD-Pb/HD-Pr (HD) VGA-In DVI Rx SiI161 ADC AD9888
DVI-In
99
Reference design courtesy of Pixelworks Inc.
Pixelworks Examples
Application example 3: Reference Design for MPEG-TV

MPEG Decoder Audio Decoder
3D Y/C Video Switch 2D Video Decoder 3D Y/C
Fully Featured MPEG-TV

ADC
Muxes
Muxes Deinterlacer Video Decoder + ADC
Dual-channel Scaler
100
Reference design courtesy of Pixelworks Inc. 2004 National Semiconductor Corporation
Acknowledgements
Speaker gratefully acknowledges material and information provided by
Dr. Nikhil Balram, Chief Technical Officer Dr. Gwyn Edwards, Technical Marketing Engineer National Semiconductor Displays Group
101
References
Image/Video/Television
Fundamentals of Video N. Balram, Short Course S-4, SID International Symposium, 2000. Video Demystified: A Handbook for the Digital Engineer, K. Jack, HighText Publications, 1993. The Art of Digital Video, J. Watkinson, Focal Press, 1994. Digital Television, C. P. Sandbank (editor), John Wiley & Sons, 1990. Video Processing for Pixellized Displays, Y. Faroudja, N. Balram, Proceedings of SID International Symposium, May, 1999. Principles of Digital Image Synthesis, Vols 1 & 2, A. Glassner, Morgan Kaufmann Publishers, 1995. Digital Image Warping, G. Wolberg, IEEE Computer Society Press, 1994 Fundamentals of Digital Image Processing, A. Jain, Prentice Hall, 1989 Sampling-Rate Conversion of Video Signals, Luthra, Rajan, SMPTE J. Nov. 1991.
102
References
Temporal Rate Conversion
IC for Motion Compensated 100 Hz TV with a Smooth Motion Movie-Mode, G. de Haan, IEEE Transactions on Consumer Electronics, vol. 42, no. 2, May 1996
HDTV Status and Prospects, B. Lechner, SID 1997 Seminar M-10. detailed history of development of HDTV
HDTV/DTV
www.atsc.org
www.teralogic-inc.com www.fcc.gov/mmb/vsd
web site for Advanced Television Systems Committee
white papers on set-top box and PC implementations of DTV

web site for FCC - up-to-date information on TV stations DTV transition
Modeling Display Systems
Multi-valued Modulation Transfer Function, Proceedings of SID International Symposium, May, 1996. Vertical Resolution of Monochrome CRT Displays, Proceedings of SID International Symposium, May, 1996.
103
References
Human Visual Systems
Linear Systems HDMI
Visual Perception, Cornsweet, 1970 Signals and Systems, Oppenheim, Willsky, Young, Prentice Hall.
MPEG2
www.hdmi.com
An Introduction to MPEG-2 B. Haskell, A. Puri, A. Netravali, Chapman & Hall, 1997
Video2000 Benchmark
www.madonion.com
104
Thank You!
105

Video Fundamentals: Signal Processing For Digital TV

Uploaded by

Copyright:

Available Formats

Video Fundamentals: Signal Processing For Digital TV

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Video Fundamentals: Signal Processing For Digital TV

Uploaded by

Copyright:

Available Formats

Video Fundamentals

Signal Processing for Digital TV

National Semiconductor Displays Group

Definition of Digital Television

Divide into 4 segments

+ Media Communication Processor

+ ATSC tuner + VSB/QAM/ QPSK Receiver

+ MPEG Processor + Transport Demux + Multiple Stream Decoder

RF front end (type A) (NTSC/PAL/SECAM)

Baseband Front-end ADC DVI-HDCP 2D Video Decoder

The Modern HD Ready TV Set

Video Deinterlacer, Scaler & CSC

Functional System Architecture for MPEG TV

CVBS Y/C YPrPb (HD)

DVI / HDCP Receiver

Frame Buffer (SDRAM)

High Definition Multimedia Interface (HDMI)

HDMI enables device communication

Hitachi, Panasonic, Philips, Silicon Image, Sony, Thomson, Toshiba

Information courtesy of Silicon Image Inc.

Color Television Systems

U = 0.492 (B-Y), V = 0.877 (R-Y) I = V cos33o - U sin33o, Q = V sin33o + U cos33o

Monochrome TV Signals (NTSC)

Spectrum of Monochrome TV Signal (NTSC)

Spectrum of the video extends from just below the video

carrier frequency to just below the sound carrier

In video space, color signals developed for backward

equivalent of monochrome TV signal

Principles of NTSC Color System

Why a weird number like 59.94 Hz?

This is close enough, solving the problem

Implementation of NTSC Color

LP Filter cos sin

Spectrum of the NTSC Color Signal

Color Audio Sub-Carrier Sub-Carrier

NTSC Color Video Signal

Black level Blank level

2004 National Semiconductor Corporation

Composite Video: NTSC / PAL / SECAM

Advanced approaches (commonly used in most systems today)

Loss of resolution Dot-crawl Cross-color

Luma and Chroma Signal Separation (Y/C Separation)

Adaptive comb filtering is required for high quality

0.793V Blank level

Notch/LPF versus Comb Filtering

Notch filter: loss of information in 2-4 MHz region

Comb filter : full horizontal resolution

Comb Filtering (cont.)

HDTV Technical Overview

Audio: Dolby AC-3 Transport:

MPEG2 Main Profile @ High Level (MP@HL) 18 formats: 6 HD, 12 SD

Subset of MPEG2 Fixed length 188-byte packets

HDTV/DTV System Layers

Multiple Picture Formats and Frame Rates

MPEG-2 video and Dolby AC-3 compression syntax

Flexible delivery of data and future extensibility

4:3 aspect ratio

16:9 aspect ratio