CUMULUS NETWORKS NETWORKING TERMINOLOGY CHEAT SHEET

Networking Terminology Cheat Sheet

YOUR REFERENCE SHEET FOR GOING WEB-SCALE WITH CUMULUS LINUX

With Cumulus Linux, you can build a web-scale data center with all the THIS DOCUMENT INCLUDES:

scalability, efficiency and automation available to the largest data center Data Center Switches
operators in the world. Oversubscription Calculations
Common Host to ToR Network Types
You have the ability to: Terminology
Customize your network applications Clos Architecture
Incremental Growth
Automate your configurations Optics and Cables
Choose whatever whitebox hardware you want based on your budget TIA Standard Color Codes
and your needs 40G and 100G as Leafs
Port Configurations
Build a web-scale data center like the worlds largest operators at a
fraction of the cost

As the new, modern way to build your network, web-scale networking gives
you access to intelligent software, its open and modular, and it allows you to
automate and scale with ease.

This short guide provides you with a condensed cheat sheet of things you
should consider when designing your web-scale network.

Data Center Switches TYPICAL 1RU 10G ToR SWITCH

This section includes a sampling of common whitebox eth0 RJ45 1Gbps OOB port

form-factors used in data center deployments.

RJ45 Console

USB
48 x 10Gbps SFP+ (4 or 6) x 40Gbps QSFP+

TYPICAL 1RU 1G ToR SWITCH TYPICAL 1RU 40G (LEAF/SPINE) SWITCH

eth0 RJ45 1Gbps OOB port eth0 RJ45 1Gbps OOB port

RJ45 Console

USB
48 x 1Gbps RJ45 4 x 10Gbps SFP+ USB
32 x 40Gbps QSFP+ RJ45 Console

TYPICAL 1RU 10G-T ToR SWITCH TYPICAL 1RU 100G (LEAF/SPINE) SWITCH

eth0 RJ45 1Gbps OOB port RJ45 Console

RJ45 Console

USB USB
48 x 10G-T RJ45 (4 or 6) x 40Gbps QSFP+ eth0 RJ45 1Gbps OOB port 32 x 100Gbps QSFP28
CUMULUS NETWORKS NETWORKING TERMINOLOGY CHEAT SHEET

Oversubscription Calculation THIS DOCUMENT INCLUDES:

Data Center Switches

Oversubscription is the difference in host bandwidth (downlinks) vs. network Oversubscription Calculations
capacity (uplinks). These calculations are shown from the point of view of a
Common Host to ToR Network Types
ToR Switch. A 1:1 oversubscription ratio ensures no network bottlenecks, but
may result in excess capacity during non-peak times. The ideal design tries to Terminology
approach 1:1 oversubscription but entirely depends on the applications and Clos Architecture
capacity needed by the administrator. Incremental Growth
Optics and Cables
TIA Standard Color Codes
10G ToR 40G and 100G as Leafs
Port Configurations

48 x 10 = 480 Gbps 4 x 40 = 160 Gbps

480 Gbps downlinks to servers / 160 Gbps uplinks to network

= 480 / 160
=3

Which is represented as 3:1 oversubscription ratio. If every server sends at

line rate, only one third of all traffic will make it into the network.

10G ToR W/6QSFP+

48 x 10 = 480 Gbps 6 x 40 = 240 Gbps

480 Gbps downlinks to servers / 240 Gbps uplinks to network

= 480 / 240
=2
480 / 240 = 2

Which is represented as 2:1 oversubscription ratio. If every server sends at

line rate, only one half of all traffic will make it into the network. This is better
than only one third of traffic making it into the network.
CUMULUS NETWORKS NETWORKING TERMINOLOGY CHEAT SHEET

Common Host to ToR Network Types THIS DOCUMENT INCLUDES:

Data Center Switches

MLAG Oversubscription Calculations
leaf01 leaf02 Common Host to ToR Network Types
Terminology
swp49-50
Clos Architecture
bridge bridge
swp1 swp1
Incremental Growth
Optics and Cables
TIA Standard Color Codes
40G and 100G as Leafs
eth1 eth2
Port Configurations

server01

MLAG is when two switches can create a single LACP bond despite coming
from two independent devices. VRR (Virtual Router-Redundancy) gives the
ability for a pair of switches to act as gateways simultaneously for HA (High
Availability) and Active-Active mode.

REDISTRIBUTE NEIGHBOR

Routing Fabric

leaf01 leaf02

swp1 swp1

eth1 eth2

server01

Redistribute neighbor daemon dynamically monitors ARP entries to

redistribute these IP addresses into the fabric.

RoH (ROUTING ON THE HOST)

Routing Fabric

leaf01 leaf02

swp1 swp1

eth1 eth2

server01

VM

Routing on the host is a solution where Cumulus Networks Quagga is installed

on either the bare metal host or the hypervisor.
CUMULUS NETWORKS NETWORKING TERMINOLOGY CHEAT SHEET

Common Terminology THIS DOCUMENT INCLUDES:

Data Center Switches

Oversubscription Calculations
ToR PEERLINK Common Host to ToR Network Types
Terminology
Clos Architecture
Top of Rack switch, where Link or bonded links used to Incremental Growth
servers connect to the network. connect two switches Optics and Cables
In an MLAG pair. TIA Standard Color Codes
40G and 100G as Leafs
Port Configurations

LEAF MLAG

Also referred to as a ToR or Multi-Chassis Link Aggregation.

Access Switch. Used typically Ability for a pair of Switches to
when referring to Spine-Leaf or act redundantly in an active-
Clos topology. active architecture and appear
as a single, logical switch.

EXIT-LEAF ECMP

A leaf connected to services Equal-cost multi-path routing.

outside the datacenter, including Allows load-balancing Across
firewalls, load balancers and multiple paths.
internet routers.

SPINE OOB

Also referred to as an aggregation Out of Band Management.

switch, end-of-row Switch or Indicating both the port that is
distribution switch. Typically used for management and the
referred to as a Spine switch in network that is dedicated to
a Spine-Leaf or Clos topology. manage the infrastructure.

CLOS LAYER 3 FABRIC

A multi-stage network Layer 3 adjacency between

architecture that optimizes multiple Leafs and Spines using
resource allocation for bandwidth. OSPF or BGP. Allows ECMP to
take place.
CUMULUS NETWORKS NETWORKING TERMINOLOGY CHEAT SHEET

Clos Architecture THIS DOCUMENT INCLUDES:

Data Center Switches

Oversubscription Calculations
Common Host to ToR Network Types
Terminology
Spine Clos Architecture
Incremental Growth
IP Fabric Optics and Cables
TIA Standard Color Codes
Leaf
40G and 100G as Leafs
Port Configurations

Routing fabric is used so aggregation/spine layer can have 3 or more

aggregation switches. A MLAG environment will always have a maximum
redundancy of 2 and burn ports for the peerlink.

Incremental Growth
Cost-effective whitebox switches coupled with a Clos design can lead to
highly redundant networks where the aggregation/spine
Layer is no longer restricted to 2 devices.

Spine Switch 01 Spine Switch 02 Spine Switch 03 Spine Switch 04

uplink 1 uplink 2 uplink 3 uplink 4

Add lead switches incrementally

Leaf Switch 01
CUMULUS NETWORKS NETWORKING TERMINOLOGY CHEAT SHEET

Optics and Cables THIS DOCUMENT INCLUDES:

Data Center Switches

This section includes common optics and cabling terminology used Oversubscription Calculations
within data centers. Common Host to ToR Network Types
Terminology
Clos Architecture
Incremental Growth
Optics and Cables
TIA Standard Color Codes
40G and 100G as Leafs
Port Configurations

FORM FACTORS

RJ45 SFP+ AND SFP28 QSFP+ AND QSFP28

8 position 8 contact (8P8C) for Small form-factor pluggables. Quad small form-factor pluggables.
termination Ethernet over twisted SFP+ supports 10 Gigabit QSFP+ supports four channels of
pair. Standard termination for Cat5/ Ethernet and SFP28 supports 10 Gigabit Ethernet QSFP28 for
Cat6 cables. 25 Gigabit Ethernet. 40 Gigabit Ethernet links. QSFP28
supports four channels of 25 Gigabit
Ethernet for 100 Gigabit Ethernet links.

LC

RJ-45

AOC

DAC

TYPES OF MODULES

DAC AOC OPTICAL TRANSCEIVER

Directly Attached Copper. A type of Active Optical Cable,a type of Uses lasers to transmit and receive
cable that consi sts of a transceiver cable similar to a DAC where the data over a pluggable optical fiber.
and cable combined together. transceiver and cable are combined The fiber can be multi-mode or
except it uses multimode or single-modedepending on the type
single-mode fiber instead of copper. of transceiver.
CUMULUS NETWORKS NETWORKING TERMINOLOGY CHEAT SHEET

TYPES OF OPTICAL TRANSCEIVERS

SR AND SR4 LR AND LR4 ER AND ER4

Short range transceivers use 850 10GBASE-LR Long reach 10GBASE-ER extended range
nm lasers on multi-mode fiber. SR4 transceivers use 1310 nm lasers transceivers use 1550 nm lasers
uses four 850 ns lasers on four on single-mode fiber. 40GBASE- on single-mode fiber. 40GBASE-
parallel multi-mode fibers (four LR4 and 100GBASE-LR4 use ER4 and 100GBASE-ER4 use
fibers in each direction). four different wavelengths of light four different wavelengths of light
centered around 1300 nm and centered around 1300 nm and
multiplex them onto a single strand multiplex them onto a single strand
of single-mode fiber (one fiber in of single-mode fiber (one fiber in
each direction). Some LR optics can each direction). Some ER optics can
reach 10 km in distance. reach 40 km in distance.

TYPES OF FIBER CONNECTORS

MPO LC
Multi-Fiber Termination Push-On. Little Connector or Local Connector.
Support for both multi-mode and A high density fiber connector found
single-mode fiber. on pluggable modules. They usually
MTP exist in pairs, one for transmit and
Brand of MPO, sometimes used together one for receive.
or interchangeably.

Female Male
CUMULUS NETWORKS NETWORKING TERMINOLOGY CHEAT SHEET

TIA Standard Color Codes THIS DOCUMENT INCLUDES:

Data Center Switches

Telecommunication Industry Association (TIA) is accredited by the American Oversubscription Calculations
National Standards Institute (ANSI) to develop voluntary, consensus-based Common Host to ToR Network Types
industry standards. Many data centers use cables of varying color and dont
Terminology
adhere to a standard.
Clos Architecture
Incremental Growth
Optics and Cables
COLOR OPTIC
TIA Standard Color Codes
40G and 100G as Leafs
Port Configurations
Yellow Single Mode Fiber (OS1 or OS2)

Orange Multi-Mode Fiber (OM1 or OM2)

Aqua Multi-Mode Fiber (OM3)

Aqua OR Multi-Mode Fiber (OM4)

Erika Violet Multi-Mode Fiber (OM4)

Black DAC Ethernet Cables are typically black

CUMULUS NETWORKS NETWORKING TERMINOLOGY CHEAT SHEET

40G and 100G as Leafs THIS DOCUMENT INCLUDES:

Data Center Switches

Oversubscription Calculations
40G HIGH PERFORMANCE LEAF Common Host to ToR Network Types
Terminology
Clos Architecture
Incremental Growth
Optics and Cables
TIA Standard Color Codes
40G and 100G as Leafs
Port Configurations
64 x 10G host ports = 640 Gbps 16 x 40 = 640 Gbps

640/640 = 1, Which is represented at 1:1 oversubscription

100G HIGH PERFORMANCE LEAF

64 x 25 host ports = 1600 Gbps 16 x 100 = 1600 Gbps

1600/1600 = 1. Which is represented at 1:1 oversubscription

100G (1/2 RU) SPECTRUM

32 x 25 host ports = 800 Gbps 8 x 100 = 800 Gbps

800/800 = 1. Which is represented at 1:1 oversubscription

CUMULUS NETWORKS NETWORKING TERMINOLOGY CHEAT SHEET

Port Configurations THIS DOCUMENT INCLUDES:

Data Center Switches
Oversubscription Calculations
BREAKOUT CABLE MATRIX
Common Host to ToR Network Types
Terminology
SFP+ Not Supported Clos Architecture
Incremental Growth
QSFP 4 x 10 Gbps Optics and Cables
TIA Standard Color Codes
2 x 50 Gbps 40G and 100G as Leafs
40 Gbps Port Configurations
QSFP28
4 x 25 Gbps
4 x 10 Gbps

BREAKOUT CABLE MAXIMUM

Broadcom Trident2: 104

Broadcom Tomahawk: 130

Mellanox Spectrum: 64*

*can fit 2 switches in 1RU for total of 128

CABLE SUPPORTABILITY

SFP+ SFP28 QSFP QSFP28

1GB yes yes* yes yes*

10G yes yes* yes yes*

25G no yes no yes

40G no no yes yes*

100G no no no yes

*dependent on Transceiver Compliance Codes

ABOUT CUMULUS NETWORKS

Cumulus Networks helps customers realize cost-effective, high capacity networking for modern data centers. Linux transformed the
economics and innovation for data center compute, and Cumulus Linux is doing the same for the network. It radically reduces the costs
and complexities of operating modern data center networks for businesses of all sizes. Cumulus Networks has received venture funding
from Andreessen Horowitz, Battery Ventures, Sequoia Capital, Peter Wagner and four of the original VMware founders.
For more information visit cumulusnetworks.com, or follow @cumulusnetworks.

2017 Cumulus Networks. All rights reserved. CUMULUS, the Cumulus Logo, CUMULUS NETWORKS, and the Rocket Turtle Logo (the Marks) are trademarks
and service marks of Cumulus Networks, Inc. in the U.S. and other countries. You are not permitted to use the Marks without the prior written consent of Cumulus
Networks. The registered trademark Linux is used pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a worldwide
basis. All other marks are used under fair use or license from their respective owners.
03012017