High speed serial links: High-speed Design Trends and Challenges
Vladimir Stojanovi j
Integrated Systems Group Massachusetts Institute of Technology
Backbone router lots of high-speed links
source: Alcatel, Tyco source: Juniper Networks
State-of-the art up to 1 Tb/s throughput Lots of linecards power constrained system
What matters is energy cost per bit
Integrated Systems Group 2
Inside the router
Line Cards: 8 to 16 per System Passive Switch Cards: Backplane 2 to 4 per System
MEM
MEM
MEM
MEM
SerDes
Crossbar
Optics
SerDes
MAC
NPU
TM/ Fabric IF
SerDes
OC-192 12 5Gb/s 12.5Gb/s Laser driver link
4x3.125 Gb/s XAUI Serial Links (chip-to-chip)
3.125-12.5Gb/s Backplane Serial Links
Regardless of where the links are, there is a constant desire to signal faster and with less power
Integrated Systems Group 3
Scaling the throughput to 100 Tb/s
Electrical I/O Challenges
100 Tb/s I/O throughput With 10Gb/s per link 10000 transceivers 20000 high-speed I/O pairs 2 10000 mm in 0 0.13 13 m technology Power 4kW 40 mW/Gb/s energy cost per bit
Integrated Systems Group
Scaling the throughput to 100 Tb/s
Density issues
Connectors 50 diff pairs/inch 400 400 long connector Trace routing 50mils pitch 250 wide 4-signal layer line-card Backplane less critical Package Package/Chip ball pitch (1mm / 200um) 2 2 4000 mm / 160mm
source: Teradyne, Rambus Integrated Systems Group 5
Design challenge
Goal
Fit 100 Tb/s on a 100 W crossbar chip Reasonable system/rack y size
Need
Power
Reduce energy/bit to 1mW/Gb/s Increase data rate per link by 10-15x
Density
Integrated Systems Group
What makes it challenging
High speed link chip
> 2 GHz signals g
Now, the bandwidth limit is in wires
Integrated Systems Group
source: Rambus
High-speed link efficiency energy cost per bit
Energy cost per b bit [mW/Gb/s]
How efficient are high-speed links?
1000000 100000
18 16 14 12 10 8 6 4 2 0
PAM4 PAM2 5.5 4
Energy co ost per bit mW/(G Gb/s)
5.9 8 1.5 1 0.45 03 0.3 RxTap 2.2
11
10000
1000
TxTap
RxSamp
PLL
CDR
En nergy cost per bit t [mW/Gb/s]
100
140 120 100 80 60 40 20 0 0
PAM2 PAM2 PAM2 PAM4 PAM4 Tx5 Rx20 Tx5 Rx1+20 Tx50 Rx80 Tx5 Rx20 Tx50 Rx80
10
1 56Kb/s V.92 modem 12x12Mb/s ADSL modem Gigabit Ethernet 10Gb/s High-speed link
2-3 orders more energy-efficient
Than traditional wireline systems
8 10 12 14 Data rate [Gb/s]
16
18
20
Starting to pa pay the price for band band-limited limited channels
Integrated Systems Group 8
Outline
Show the path to efficient 100 Tb/s systems
Look at all aspects of system design
High-speed link environment Improving the channel What can chips do?
Integrated Systems Group
Backplane environment
Package On-chip parasitic Line card trace
(termination resistance and device loading capacitance)
Package via
Back plane trace
Back plane connector
Line card via
Backplane via
Line attenuation Reflections from stubs (vias)
Integrated Systems Group 10
Backplane channel
Loss is variable
Same backplane p Different lengths Different stubs
Atte enuation [d dB]
0 -10 -20 -30 -40 -50 50 -60 9" FR4, via i stub t b 26" FR4, via stub 2 4 6 8 10 frequency [GHz] 26" FR4 9" FR4
Top p vs. Bot
Attenuation is large
>30dB @ 3GHz But is that bad?
Required signal amplitude set by noise
Integrated Systems Group
11
Interference
Attenu uation [dB] 0
pulse re esponse
-10 -20 20 -30 -40 -50 50 -60
THROUGH
1 0.8 0.6 0.4 0.2 0 Tsymbol=160ps
NEXT
FEXT
6 8 10 frequency [GHz] f [GH ]
3 ns
Inter-symbol interference
Dispersion spe s o (s (skin-effect, e ec , d dielectric e ec c loss) oss) - s short o latency a e cy Reflections (impedance mismatches connectors, via stubs, device parasitics, package) long latency
Co-channel Co channel interference (Far-End (Far End & Near-End Near End Crosstalk)
Integrated Systems Group 12
Reflections and Crosstalk
Dont just receive the signal you want
Get versions of signals close to you Vertical connections have worst coupling
Close in these vertical connection regions
Far-end XTALK (FEXT) Desired signal g
Sercu, DesignCon03
Reflections Near-end XTALK (NEXT)
Integrated Systems Group
13
A complex system
PCB only
PCB + Connectors
PCB, Connectors, Via stubs & Devices
Integrated Systems Group
14
Outline
Show the path to efficient 100 Tb/s systems
Look at all aspects of system design
High-speed link environment Improving the channel What can chips do?
Integrated Systems Group
15
Dispersion: material loss
FR4 dielectric, 8 mil wide and 1m long 50 Ohm strip line 1 Atten nuation 0.8 0.6 0.4 0.2 0 1.0E+06 1.0E+07 1.0E+08 Frequency, Hz
Total loss Conductor loss Dielectric loss
1.0E+09
1.0E+10
Kollipara DesignCon03
PCB Loss : skin & dielectric loss Skin Loss f Dielectric loss f : a bigger issue at higher f
Integrated Systems Group 16
Better dielectric
Rogers
FR4 FR4+stubs
source: Alcatel, Tyco
Rogers is expensive but smallest loss
Integrated Systems Group 17
Minimizing reflections - the vias
Minimizing g via stubs
Thinner PCBs are better but sometimes impossible Counter-boring Counter boring Blind vias SMT technology All are costly 1.1x - 2x
plated through hole
counter-bored Integrated Systems Group
blind via 18
Connector technologies
LC BP
microvia
trace
Standard - Press-Fit Side-Interface (Tyco)
Surface-mount + microvia Orthogonal - Teradyne (Differential Plated Through Hole)
Stubs big problem in standard press press-fit fit connectors Side-Interface eliminates DC stubs and diff-pair length mismatch Orthogonal interconnect DPTH eliminates the backplane Surface-mount
Integrated Systems Group 19
Eliminating the backplane - orthogonal interconnect
source: Teradyne
No backplane trace No backplane via-stub Coax-like shielding and diff-pair matching in DPTH mid-plane
M. Cartier et al Optimized Signal Path for Orthogonal System Architectures, DesignCon 2005. Integrated Systems Group 20
DPTH connector performance
No shared vias (non-DPTH) Shared vias (DPTH)
Insertion Loss of DPTH very small Reflections minimized NEXT and FEXT minimized
Integrated Systems Group 21
Outline
Show the path to efficient 100Tb/s systems
Look at all aspects of system design
High-speed link environment Improving the channel What can chips do?
Integrated Systems Group
22
New link design
Dealing with bandwidth limited channels
This is an old research area
Textbooks on digital communications Thi k modems, Think d DSL Standard approach requires high-speed high speed A/Ds and digital signal processing 20Gs/s A/Ds are expensive
But cant directly apply their solutions
(Un)fortunately need to rethink issues
Integrated Systems Group
23
Baseline Channels
0
Short ATCA BP, 3
-20 20
S21 [dB]
-40
N6K BP, 26
-60
Legacy FR4 BP 26, via stub
-80
-100
5 10 frequency [GHz]
15
Legacy (FR4) - lots of reflections Microwave engineered (N6K) Emerging standards (IEEE 802.3ap, ATCA)
Integrated Systems Group 24
Capacity and MT data rates the impact of noise
Capacity thermal and phase noise
220 200 180 160 Capacity y [Gb/s] 140 120 100 80 60 40 20 0 0 5 10 15 Noise factor [dB] 20
Uncoded MT thermal and phase noise
220 200
Short ATCA BP
180 160
N6K BP
Data rate e [Gb/s]
140 120 100 80 60 40 20 0 0
Short ATCA BP N6K BP
Legacy FR4 BP
Legacy FR4 BP
5 10 15 Noise factor [dB] 20
Capacity
Uncoded MT
Much higher than data rates in todays links Noise
Half the capacity
BER target of 10-15 Peak-power constraint
Thermal - 50Ohm termination Phase noise best LC PLL (0.14%UI rms)
Coding can help
Integrated Systems Group
25
Removing ISI baseband link
Linear transmit equalizer
Tx Data
Anticausal taps
Sampled Data
Deadband
Feedback taps
Channel
50 50 outP outN
Causal taps
d
TapSel TapSel Logic
I eq 0
Decision-feedback equalizer
Transmit and Receive Equalization
Changes signal to correct for ISI Often easier to work at transmitter
DACs easier than ADCs
Integrated Systems Group 26
J. Zerbe et al, "Design, Equalization and Clock Recovery for a 2.5-10Gb/s 2-PAM/4-PAM Backplane Transceiver Cell," IEEE Journal Solid-State Circuits, Dec. 2003.
Pulse amplitude modulation
Binary (NRZ)
1 bit / symbol Symbol rate = bit rate
PAM4
2 bits / symbol Symbol rate = bit rate/2
00 01 11 10
1 0
Integrated Systems Group
27
Multi-level: offset and jitter are crucial
th thermal l noise i
45 Data rat te [Gb/s]
Data rat te [Gb/s] 30 25 20 15
PAM4 PAM2
PAM8
thermal noise + offset ff t
30 Data rate [Gb/s] 25 20 15 10
thermal noise + offset+ jitt jitter
40 35 30 25 20 15 10 5 0 0 2 4 6 8 10 12 14 16 18 20 S Symbol b l rate t [Gs/s] [G / ]
PAM2 PAM16 PAM8
PAM16 PAM4
PAM4 PAM2 PAM8
10 5 0 0 2 4 6 8 10 12 14 16 18 20 S b l rate [Gs/s] Symbol [G / ]
5 0 0 2 4 6 8 10 12 14 16 18 20 Symbol rate [Gs/s]
To make better use of available bandwidth, need better circuits PAM2/PAM4 2/ robust candidate for f next generation links
Integrated Systems Group 28
Full ISI compensation too costly
thermal noise
Data a rate [Gb/s] 20 18 16 14 12 10 8 6 4 2 0 0 2 4 6 8 10 12 14 16 Symbol rate [Gs/s]
PAM2 PAM8
PAM4
thermal noise + offset
20
Data rate [Gb/s]
thermal noise + offset+ jitter
20 Data rate [Gb/s] 18 16 14 12 10
PAM4
18 16 14
12 PAM16 10 PAM8 8 6 4 2 0 0 2 4 6 8 10 12 14 16 Symbol rate [Gs/s]
PAM4 PAM2
8 6 4 2
PAM8
PAM2
0 0 2 4 6 8 10 12 14 16 Symbol rate [Gs/s]
Todays links cannot afford to compensate all ISI
Too much power Limits todays maximum achievable data rates
Integrated Systems Group 29
Capacity Bit Loading
Excess Noise factor 0dB
8 7 6 # bits pe er dimension 5 4 3 2 1 0 Legacy FR4 BP 0 5 10 Frequency [GHz] 4.5
Excess Noise factor 20dB
4 3.5
Short ATCA BP
N6K BP
# bits pe er dimension 15
3 2.5 2 1.5 1 05 0.5 0 0 5 10 Frequency [GHz] 15
B d idth i Bandwidth is li limited it d b by attenuation tt ti and d noise i
Cant just keep increasing the signaling frequency Need to focus on available bandwidth (at most 10-20GHz)
Need circuits that can create/sense 4-8 bits/dim
Integrated Systems Group 30
Uncoded Multi-tone Bit Loading
Excess Noise factor 0dB
5 4.5 4 # bits per d dimension # bits per d dimension 3.5 3 25 2.5 2 1.5 1 0.5
Excess Noise factor 20dB
2 1.8
1.6
Short ATCA BP
N6K BP
1.4 1.2 1 0.8 0.6 0.4 0.2
Legacy FR4 BP
0 0 5 10 Frequency [GHz] 15
0 0
5 10 Frequency [GHz]
15
Integer constellations and target BER=10-15
Bandwidth not affected much (still 10-20GHz)
In high-noise g case - less advantage g over baseband With coding can improve by up to 2x closer to capacity
Integrated Systems Group 31
Impact of jitter on baseband
Legacy FR4 BP
0 -2 12Gb/s
Short ATCA BP
0 -2
-4 log10BER -6 -8 -10 -12 -14 0
10Gb/s
25Gb/s 8Gb/s
log10 BER 0 -4 -6 -8
20Gb/s
-10 -12
15Gb/s
6Gb/s
5 10 Jitter Factor [dB] 15 20
-14 0 5 10 Jitter Factor [dB]
10Gb/s
15 20
With proper coding
Increase data rate Relax PLL jitter spec save power
Integrated Systems Group 32
Original jitter rms = 1.4%UI (ring oscillator based PLL)
BER vs. hardware complexity
Legacy FR4 BP
0
0
Short ATCA BP
12Gb/s
-5 log10B ER
log10BE ER
-5
-10
10Gb/s
-10
25Gb/s
6Gb/s
-15 0 10 20
8Gb/s
30 40 50 60 # feedback eq q taps p 70 80
-15 0
15Gb/s
10
20Gb/s
20 30 # feedback eq taps 40
Partially eliminate ISI (leave most of the reflections) Let simple code take care of the rest
5 Can recover from raw BER of 10-5 And save up to 50 feedback taps - up to 15mW/Gb/s in 0.13m
Integrated Systems Group
33
But, need to be careful
Always now what youre optimizing
Powerful coders/encoders often costly
Example - fastest RS (255,239) implementation 10 40 Gb/s throughput Energy cost - 12mW/Gb/s 50x area of the high-speed link (extensive parallelism)
Need to include the energy cost per bit in the code d d design i spec
L. Song, M-L Yu, M.S. Shaffer, 10- and 40-Gb/s Forward Error Correction Devices for Optical Communications, IEEE Journal of Solid-State Circuits, vol. 37, no. 11, Nov. 2002.
Integrated Systems Group
34
Opportunity for coding
Break the coding/equalization/modulation hierarchy Goal to minimize overall energy cost per bit Proper coding can be more energy-efficient in achieving the low BER than modulation/equalization
Especially with lots of crosstalk and numerous small reflections
Need new paradigms in code development to specification
Non Gaussian (system) noise Non-Gaussian Circuit non-idealities Crosstalk and residual channel memory (ISI) E Energy cost t constraint t i t on code d performance f
Integrated Systems Group 35
Bridging the gap: Multi-tone link
10
Multi-tone data rates with thermal noise
8
#bits/H Hz
Nelco 64Gb/s FR4 38Gb/s
0 0
10
12
14
frequency [GHz]
A. Amirkhany, V. Stojanovic, M.A. Horowitz, Multi-tone Signaling for High-speed Backplane Electrical Links, IEEE Global Telecommunications Conference, November 2004.
Integrated Systems Group
36
Bridging the gap: Multi-tone link
10
Multi-tone data rates with thermal noise
8
#bit ts/Hz
Nelco 64Gb/s FR4 38Gb/s
data0 data1 LPF
LPF BPF
LPF
2 4 6 8 10 12 14
frequency [GHz]
data0 d t 1 data1
0 0
BPF
LPF
dataN LPF BPF
# level ls
ejw1t
ejw1t
BPF
LPF
dataN
ejwNt f ejwNt Challenge balancing the inter-symbol and inter-channel interference
Microwave filter techniques Custom signal processing
Integrated Systems Group 37
Conclusions
Interfaces are challenging system designs
Good space to explore system level optimization 2-3x improvement in data rate possible
Better backplanes are around the corner
State-of-the-art baseband links (chips)
Far from utilizing the capacity of the channels
10-20x difference in data rates
Looking into multi-tone and coding to bridge the gap Useful channel bandwidth 10-20 GHz
F Focus on lower-speed l d precision i i circuits i it f for hi higher h order d constellations t ll ti
Coding
If careful, can lower the energy cost per bit for the whole system Problem formulation different in so many ways
Integrated Systems Group 38
Acknowledgments
MARCO Interconnect Focus Center Jared Zerbe and Ravi Kollipara - Rambus John DAmbrosia Tyco IEEE 802.3ap, p, ATCA forum Alcatel, Teradyne, Juniper Networks
Integrated Systems Group
39
