#CLUS

Carrier Grade
Disaggregation
with IOS XR
Mike Korshunov
Technical Marketing @ Service Provider
BRKSPG-2015

#CLUS
Session Abstract
• Session focused on IOS-XR based systems.
• We will go through main Disaggregation concepts and will cover
Cisco’s approach for it.
• Demonstration during the talk: IOS-XR running on 3rd party
platform!
• We’ll conclude different aspects for white box integration and
discuss multiple operation considerations, which they introduce.

#CLUS BRKSPG-2015 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Agenda
• Introduction to White Box & Disaggregation
• Current Market State
• Delayering IOS-XR Software
• ONIE – New Installer
• Demo
• Summary

#CLUS BRKSPG-2015 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Introduction to
White Box &
Disaggregation
 The Market is Dramatically Evolving
Bandwidth Growth Finally “Breaking the Bank” Traffic is Moving Closer
to the End User
1,400,000 30

1,200,000 Cross-country
25 Delivered
1,000,000 58% in 2016
Spending in Millions

Bandwidth in Petabits
20 41% by 2021
800,000
15 Regional
600,000 Delivered
10 20% in 2016
400,000 23% by 2021

200,000 5
Metro
0 0 Delivered
22% in 2016
2013 2014 2015 2016 2017
35% by 2021
SP Total OPEX SP Total CAPEX SP Bandwidth
Sources: IHS Technology, “Service Provider CapEx, OpEx, Revenue, and Subscribers Database, Q1 2017”;
Cisco Visual Networking Index (VNI)

#CLUS BRKSPG-2015 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
 73%
73%

82%
82%

BRKSPG-2015 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
What is a Whitebox ?
• White box refers to the ability to use ‘generic,’
off-the-shelf switching and routing hardware
Same Hardware – Different NOS
• White box switches are really just that ‘blank’ Same NOS – Different Hardware
hardware.

• They rely on an Network Operating System

(NOS) to perform switching/routing functions.
• The NOS may come already installed or be
purchased from a software vendor or
assembled from different Open Source projects.
• NOS are often Linux based (any flavor) to
benefit from the many open and free tools
available but also to align more closely with
servers operating models.

#CLUS BRKSPG-2015 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
 What Do you get ?
Switching Hardware

• Generally, HW Manufacturer do not provide any

Switching/Routing Stack.
• ONIE is the only SW available on the box.

• End-User need to shop around for a NOS or an

NOS Installer OS + switching stack.
• NOS is usually provided as a ONIE boot image

List Price for a Wedge-100 Broadcom Tomahawk 32x100G: $11,275.00

#CLUS BRKSPG-2015 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
What is Disaggregation?
• … an "unbundling," whereby the network operating system that runs
on an ethernet switch is not integrated with or baked into the
device.
• Instead, it is provided by a third-party network operating system
vendor or by an open source project that provides a network
operating system.
• In that sense, the network operating system becomes decoupled
from the switch hardware … giving customers choice in selecting
the network hardware and software best suited for their application
environments.
Source: IDC

#CLUS BRKSPG-2015 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
The White Box Myth ?
Pricing between a white box switch and
OS vs. a Cisco merchant silicon based
platform over a 6.6 year period is less
than $900 apart.

The cost of operational complexity may further narrow the $900 Gap
#CLUS BRKSPG-2015 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
Current Market State
New Industry Landscape
More choices offered to end-customers

Vertically Integrated Systems market HW/SW Disaggregation Market

SW-only vendor &

Open Source SW
IOS XR
cEOS JunOS
NX OS

Set of Set of Set of Original Design Manufacturers

qualified qualified qualified
3rd-party 3rd-party 3rd-party
HW devices HW devices HW devices

• Mainstream market • Nascent market

• Networking vendors-led • SW-only/Open Source-led
• Market value: $$$ Networking vendors • Market value: $
• Market growth – low to moderate entry to market • Market growth – strong

#CLUS BRKSPG-2015 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
Customer profiles

Adopters Explorers Uninterested

• Unsatisfied with • See potential for • Uninterested in
vendor s/w velocity & savings in hardware disaggregated
flexibility and software price solutions

• Have use cases • Don’t understand what • Happy with integrated

outside of vendor disaggregation means offers
capabilities to them necessarily

• Long-term vision
around
commoditization

#CLUS BRKSPG-2015 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
Delayering IOS-XR
Software
IOS XR: Built For All Deployments

Elastic Cost Optimized Ultra-high Density Critical Infrastructure Programmable and

Automatable

VNF Qualified third- Cisco HW with Cisco HW with

party hardware Merchant Si Custom Si

Fixed hardware, Fixed and modular hardware, Fixed, modular, and multi-chassis
x86 data plane merchant data plane merchant data plane hardware; Cisco data plane

Common SP Operating System Across Physical and Virtual Data Planes

#CLUS BRKSPG-2015 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
Introducing Flexible Deployment with IOS XR

IOS XR integration with community

SW Delayering tools and open-source software
innovations via object-modeled APIs

IOS XR integration on OCP-compliant

HW/SW Disaggregation
hardware enabling custom, bespoke
solutions

#CLUS BRKSPG-2015 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
API-Driven, Layered SW Architecture
3rd Party Agent + Telemetry OSS

1 Management API
NBI Leverages an extensive set of YANG
FCAPS Management Layer (native, OC) data models to enable
SW Mgmt etc. CLI Netconf … programmatic configuration
CFG Mgr
Application Layer API*
Admin Layer

APL
2
Offers direct programmatic access to the
Application/Protocol Layer protocol applications – BGP, IGP, etc.

BGP ISIS …
SAL
3 Service Layer API
Service Adaptation Layer Offers a scalable and convenient
integration point to build/extend devices’
RIB LM L2 control plane functionality
IFM QoS …

Hardware Adaptation Layer 4 Open Forwarding Abstraction

Abstraction
Platform

API
Provides easier and rapid enablement of the
OS Abstraction Chip SDK software on new platforms and silicon, while
ensuring performant forwarding operation

OS/BSP HW/Chipset * The use case & implementation of the Application Layer API is
currently under evaluation & external availability of the API is TBD

#CLUS BRKSPG-2015 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
IOS XR SW Architecture – White Box Touchpoints
IOS-XR
Platform MGMT FWD
FIB
Shelf mgr QoS
Env mon L2
WB Platform Inv mon
Agents PM Optics Port LED OFA

Platform Abstraction Optics Abstraction BCM SDK

WB Agents

BSP
BMC CPU Host
WRL
CPU x86

FAN, PS, EEPROM, Jericho, Qumran,

Optics, buses
PCIe, i2c, serial Jericho2
#CLUS BRKSPG-2015 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
IOS XR on 3rd Party HW – Guiding Principles

Cisco IOS XR will be offered as a Software offering for

qualified 3rd party hardware platforms
Target customer-specific feature validation cycles will be
implemented
IOS XR will be certified with a specific version of OCP software
(ONIE, hardware drivers, ONLP, etc.)
The feature set supported on the 3rd party HW will be
dependent on the customer use case & capabilities of the
underlying ASIC

#CLUS BRKSPG-2015 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
IOS XR on 3rd Party HW – BMC Support
• BMC manages the system environments, power on/off of x86, sets up the
x86 for OS installation, provides access to serial on x86, etc.
• IOS XR is agnostic to the presence of a BMC and integrates seamlessly with
or without a BMC in the system
• All platform CLIs, SNMP MIBs and Yang models provide data in the same
format for both BMC and non-BMC systems.
• BMC can be accessed via the IOS XR management ethernet port
• Hardware access drivers (Proxy vs Actual driver in x86) vary based on BMC
presence
• Hardware access latency through BMC requires fine-tuning or
customization in IOS XR

#CLUS BRKSPG-2015 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
IOS XR on 3rd Party HW - ONLP Application APIs
• ONLP application APIs are used to create an application that builds in ONL on top of
the platform, the "application to platform" APIs are found in:
• https://github.com/opencomputeproject/OpenNetworkLinux/tree/master/packages/base/any/onl
p/src/onlp/module/inc/onlp

• Each hardware device is given a unique OID.

• Each type of device has a number of different properties that
is querable/programmable (e.g. read/set fan speed, query an SFP+ port status, etc.)
and a mechanism for negotiating hardware capabilities (e.g. is the fan speed
setable? reversible? does the SFP have a hardware interrupt bit on link down?).
• The ONLP APIs have their own internal locking mechanism and support multiple
concurrent callers.
• No inter-application internal contention resolution (last caller wins).

#CLUS BRKSPG-2015 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
ONIE – New
Installer
IOS XR on 3rd Party HW – Installation & SW
Provisioning
1. ONIE is used as the default bootloader
2. IOS XR SW image can be downloaded using DHCP or through manual transfer of
the image followed by the use of ONIE commands for installation
3. ONIE also supports upgrade of the firmware – CPLD, BIOS and BMC (if present).
ONIE can also upgrade ONIE.
4. ONIE supports all downloads via the management ethernet port.
5. ONIE can also access other shared partitions on the disk to access images. This
can be used when images are downloaded by data ports when IOS-XR is running.
6. IOS XR is provisioned using ZTP (Zero Touch Provisioning) - ZTP supports IPv4,
IPv6 DHCP requests through Mgmt. and Data-ports

#CLUS BRKSPG-2015 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
ONIE Options

#CLUS BRKSPG-2015 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
 ONIE initial System Boot

• Low level boot loader, configures

Boot Loader CPU complex
(HW Vendor Supplied) • Loads and boots ONIE from flash
• Available from network or USB
• Linux executable
• Installs IOS-XR into storage
• Linux OS with Busybox
ONIE • Configures mgmt. interface
• Locates & executes installer from the
(HW Vendor Supplied)
network
• Provides tools & env for installer
Installer
(Cisco Supplied)

IOS-XR
(Cisco Supplied)

#CLUS BRKSPG-2015 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
ONIE Subsequent System Boots

• Low level boot loader, configures

Boot Loader CPU complex
(HW Vendor Supplied) • Loads and boots IOS-XR

ONIE • Still exists in the flash, but is not

(HW Vendor Supplied) used
• Available for uninstall / re-install ops

• Configures Switching ASIC

IOS-XR • Runs Network Protocols
(Cisco Supplied) • Provides CLI

#CLUS BRKSPG-2015 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
Summary
Currently Supported Third-Party HW*

AS7816-64x AS5916-54XKS
Tomahawk II Qumran-MX

*limited availability
#CLUS BRKSPG-2015 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
HW Optimizations For Vertically Integrated Platforms
Improved Route Scale: Improved route management through sorting routes in internal routing database
Usage of external TCAM for large-scale routing (faster storing and lookup of information)

Dedicated FPGA for statistics: NCS 55A2-mod-SE has dedicated FPGA statistics enabling higher
interface counts, etc.

Timing - NCS 55A2-mod-SE & NCS 540 have a separate board (GNSS) for timing functionality - PTP

MACSec: Specialized internal chipsets (ES200) are used for MACSec capabilities

DWDM: Line card dedicated for DWDM & support for DCO optics

End-to-End Distributed System: Support for distributed platforms (NCS 5508, NCS 5516) – LC, Fabric,
Chassis et al
#CLUS BRKSPG-2015 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
 Operational Differences between a Vertically
Integrated and Disaggregated Solution

Feature Disaggregated Solution (Cisco IOS XR on 3 rd Party

Vertically Integrated Platforms (NCS5500, Spitfire, etc.)
Deliverable HW)

NOS installation on
iPXE boot followed by IOS XR Install (SMU, RPM repo, etc.) ONIE is used for image discovery and installation
the device

• Use IOS XR Install commands (install add <source>, install • Upgrade using new image based on ONIE install
activate, install commit) for software upgrades – may or may (device reboots)
Software Upgrade
not require reboot of the device • Configuration is wiped off during ONIE install.
post FCS
• Config persists post upgrade. Upgrade rollback can be ZTP or Ansible can be used to save & reapply
utilized existing config during upgrade

• Cisco hardware Root-of-Trust (Cisco controls trust • Cisco does not control the root-of-trust, OEM
guarantee of the entire stack) vendor does
Security • Cisco Secure Boot (HW TAM module), Secure storage • Might support UEFI Secure Boot – no hardware
with HW root-of-trust, HW support for remote integrity root-of-trust
and attestation • Support for runtime protections only

#CLUS BRKSPG-2015 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
Cisco Webex Teams
Questions?
Use Cisco Webex Teams to chat
with the speaker after the session

How
1 Find this session in the Cisco Live Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space

Webex Teams will be moderated cs.co/ciscolivebot# BRKSPG-2015

by the speaker until June 16, 2019.

#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
Complete your
online session • Please complete your session survey
evaluation after each session. Your feedback
is very important.
• Complete a minimum of 4 session
surveys and the Overall Conference
survey (starting on Thursday) to
receive your Cisco Live water bottle.
• All surveys can be taken in the Cisco Live
Mobile App or by logging in to the Session
Catalog on ciscolive.cisco.com/us.
Cisco Live sessions will be available for viewing
on demand after the event at ciscolive.cisco.com.

#CLUS BRKSPG-2015 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
Continue your education

Demos in the
Walk-in labs
Cisco campus

Meet the engineer

Related sessions
1:1 meetings

#CLUS BRKSPG-2015 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
Thank you

#CLUS
#CLUS