Cisco MDS 32G FC SAN Solutions

MDS 32G FC module & Entry Level 32G FC SAN Switch

Bhavin Yadav
Technical Marketing Engineer
May 15, 2018
Guidelines and Disclaimers

• All roadmaps are displayed in calendar year (CY) quarters.

• A roadmap presentation slide may not show a complete list of new features;
contact the account team for more details.

• Information contained in this document regarding future technology, future

features, future quality standards, or product or release availability is subject
to change and is not a binding commitment by Cisco.

• The information contained in this document is Cisco Partner Confidential

information and is provided to the recipient subject to the terms and
conditions of the applicable nondisclosure agreement between Cisco and
the recipient.

- For Partners and Customers with Nondisclosure Agreements

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Cisco MDS 32G FC Module

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Agenda

Use case
Brief overview Hardware features Software features
topologies

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Cisco MDS 32-Gbps Fibre Channel Module
Lock Screws

Eject Push Eject Lever

Button Port LEDs

• 48 x 32-Gbps Fibre Channel line-rate ports

• Common module for Cisco® MDS 9700 Series platform
• 4/8/16/32-Gbps FC supported speeds in SFP+ form
• 16 ports per port group
• Up to 1.5 Tbps front-panel bandwidth
• Cisco Part number: DS-X9648-1536K9

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Port Numbering and Specifications

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47

2 4 6 8 10 12 14 16 18 20 22 24 10 28 30 32 34 36 38 40 42 44 46 48

• Top row: Odd-numbered ports

• Bottom row: Even-numbered ports
• Supports F, FL, E, TE, SD, and ST modes
• Supports FC TrustSec
• No support for 2G FC, 10G FC and Arbitrated Loop (AL) ports
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Buffer to Buffer Credit Allocations

16-Member Port Group

Per Port
• Default 500 buffer credits per port for “E” port
• Default 32 buffer credits per port for “F” port

Port groups
• Three Port Groups, 16 ports in each port group
• Up to 8300 buffer credits among a single port group
• Up to 8270 buffer credits can be assigned to a port in port group
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
 B2B Credits, Speed, and Distance Reference
Per-kilometer B2B credit requirements at different speeds and frame sizes

Frame Size 1 Gbps 2 Gbps 4 Gbps 8 Gbps 10 Gbps 16 Gbps 32 Gbps

512 bytes 2 B2B/km 4 B2B/km 8 B2B/km 16 B2B/km 24 B2B/km 32 B2B/km 64 B2B/km

1024 bytes 1 B2B/km 2 B2B/km 4 B2B/km 8 B2B/km 12 B2B/km 16 B2B/km 32 B2B/km

2112 bytes 0.5 B2B/km 1 B2B/km 2 B2B/km 4 B2B/km 6 B2B/km 8 B2B/km 16 B2B/km

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
B2B Credits, Speed, and Distance Reference
With default buffer credit of 500 per port for 32-Gbps module

Speed and Frame Size 512 Bytes 1024 Bytes 2112 Bytes

2 Gbps 125 km 250 km 500 km

4 Gbps 62 km 125 km 250 km

8 Gbps 31 km 62 km 125 km

16 Gbps 15 km 31 km 62 km

32 Gbps 7 km 15 km 31 km

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
 B2B Credits, Speed, and Distance Reference
Using 8270 buffer credits per port for 32-Gbps module (with Enterprise license)

Speed and Frame Size 512 Bytes 1024 Bytes 2112 Bytes

2 Gbps 2050 km 4100 km 8200 km

4 Gbps 1025 km 2050 km 4100 km

8 Gbps 512 km 1025 km 2050 km

16 Gbps 256 km 512 km 1025 km

32 Gbps 128 km 256 km 512 km

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Line-rate Port Configuration
Number of Fabric Modules in Front-Panel FC Front Panel FCoE Maximum Number of
Cisco® MDS 9700 Series Bandwidth per Slot Bandwidth per Slot Line-Rate Ports per Module

24 x 32G FC
3 768 Gbps 660 Gbps 48 x 16G FC
48 x 10GE FCoE

32 x 32G FC
4 1024 Gbps 880 Gbps 48 x 16G FC
48 x 10GE FCoE

40 x 32G FC
5 1280 Gbps 1100 Gbps 48 x 16G FC
48 x 10GE FCoE

48 x 32G FC
6 1536 Gbps 1320 Gbps 48 x 16G FC
48 x 10GE FCoE
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
 SFP+ Optical Modules
Wavelength Cable Core Size Modal Bandwidth
SFP Cable Distance
(nm) Type (microns) (MHz-km)

32GFC - LW
• 3200-SM-LC-L
1310 SMF G.652 - 10 km
• 1600-SM-LC-L
• 800-SM-LC-L

32GFC - SW
• 3200-M5-SN-S 500 (OM2) 20m (OM2)
850 MMF G.651.1 2000 (OM3) 70m (OM3)
• 1600-M5-SN-S
4700 (OM4) 100m (OM4)
• 800-M5-SN-S

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Functional Block Diagram
Central
Arbiter Spine 1 Spine 2 Spine 3 Spine 4 Spine 5 Spine 6

Crossbar ASIC -1 Crossbar ASIC -2

F32 ASIC -1 F32 ASIC -2 F32 ASIC -3

48 Ports

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Temperature Sensors

• F32 ASIC has internal temperature sensor circuitry that is

accessed through internal registers
• No separate inlet or outlet temperature sensors
• Two external temperature sensors exist on module
- One for AXE3506 processor
- One for PEX8713 FPGA

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
LEDs
Module LED Description

Green • Operation is normal.

• The module is booting or running diagnostics.

Orange
• A minor temperature condition occurred.

• The diagnostics test failed.

• The module is not operational.
Red
• A fault occurred during the initialization sequence.
• An over-temperature condition occurred.

• The module is resetting.

• A major temperature threshold has been exceeded.
Blinking Red
• The module failed to download code and configuration information successfully
during the initial reset.

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
LEDs
Interface LEDs Description

Green The port is active.

Orange The module or port is disabled through the CLI command, or the module is initializing.

Blinking Orange The port is faulty and has been disabled.

Off The port is not active or the link is not connected.

ID LEDs Description

Blinking Blue The module is selected for service from the CLI.

Off The module is not selected.

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Comparison: 16G and 32G FC Modules
16G FC (DS-X9448- 32G FC (DS-X9648-
Description
768K9) 1536K9)
Total Bandwidth 768G FC 1536G FC

Port-group size 4 ports 16 ports

Ports per ASIC 8 ports 16 ports

ASICs per module 6 3

Analytics capability Absent Present

Anti-Counterfeit Technology Absent Present

Secure Boot Absent Present

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Use Case Topologies

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
 32-Gbps Storage Edge
16- and 32-Gbps Flash
Storage Arrays
16- and 32-Gbps FC

Cisco® MDS 9700 Series Directors

with 32-Gbps FC Modules

16-Gbps FC
Cisco MDS 9700 Series Directors
with 16-Gbps FC Modules
2-, 4-, 8-, and
16-Gbps FC

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
 32-Gbps Inter-Switch Links
16- and 32-Gbps Flash
Storage Arrays

16- and 32-Gbps FC

Cisco® MDS 9700 Series Directors

with 32-Gbps FC Modules

32-Gbps FC ISL
Cisco MDS 9700 Series Directors
with 16- and 32-Gbps FC Modules
2-, 4-, 8-, 16-, and
32-Gbps FC

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
 High-Performance Storage
16- and 32-Gbps FC and NVMe
FC-SCSI NVMe-FC
Flash Storage
Array Array
16- and 32-Gbps FC

Cisco® MDS 9700 Series Directors

with 32-Gbps FC Modules

16- and 32-Gbps FC

FC-SCSI Servers NVMe-FC Servers

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
 NVMe over FC and FCoE
Cisco UCS® C-Series Rack Servers

HBA
Traditional FC-SCSI-capable initiator
Traditional FC-SCSI-capable target

HBA
FC-NVMe-capable initiator FC or FCoE fabric built using Cisco® MDS FC-NVMe-capable target
9000 Family switches

• Increased performance: Unleash the power of all-flash arrays using NVMe over FC or FCoE
• Seamless insertion: Support on all existing 16-Gbps FC and new 32-Gbps FC Cisco MDS 9000 Family switches
• Phased transition: SCSI and NVMe initiators and targets continue to share same SAN
• Multiprotocol flexibility: SCSI or NVMe traffic over FC or FCoE transport
• Ecosystem support: Fully validated solution on Cisco UCS C-Series servers, Cisco MDS 9000 Family switches, and
Broadcom and Avago HBAs

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Cisco MDS 9700 Series Modules
Bringing FC, FCoE, and FCIP Together

48-Port
32-Gbps
FC Module

• Flexibility

• Redundancy
24/10 SAN 48-Port
Extension 16-Gbps
Module FC Module
• Interchangeable
z
Cisco® Cisco Cisco
• Multiprotocol MDS MDS MDS
9710 9718 9706

• Licensing
48-Port 10 24-Port 40
GE FCoE GE FCoE
Module Module

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
 Cisco MDS 9700 Series Directors Reference

Winning Points

• Multiprotocol flexibility
• Single OS (Cisco® NX-OS)
• Single management (Cisco DCNM)
14RU
• Line-rate performance 9RU
• Better Cisco UCS® interoperability

Hardware Feature Cisco MDS 9706 Cisco MDS 9710

Cisco part number DS-C9706-1K9 DS-C9710-1K9
Line-card slots 4 8
Line-rate 32-Gbps FC ports (DS-X9648-1536K9) 192 384
Line-rate 16-Gbps FC ports (DS-X9448-768K9) 192 384
Line-rate 10 GE FCoE ports (DS-X9848-480K9) 192 384
Line-rate 40 GE FCoE ports (DS-X9848-960K9) 96 192
Line-rate 2/4/8/10/16-Gbps FC ports (DS-X9334-K9) 96 192
Line-rate 1/10/40 GE FCIP ports (DS-X9334-K9) 32 8 64 16
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Cisco MDS 9132T
Overview

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Cisco MDS 9132T Overview

• Base Switch: 16 x 32G FC line-rate ports with one expansion slot

• 8-port active and 16-port active options available
• Logical Expansion Module (LEM): 16 x 32G FC line-rate add-on ports
• Supports 4/8/16/32G FC speeds
• Port side intake and Port side exhaust air-flow options
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Cisco MDS 9132T Overview

• Dedicated Network Processor for SAN Analytics*

• Base chassis contains one IO-ASIC, CPU, NPU processor

• LEM module contains additional dedicated IO-ASIC

• Required NX-OS: 8.2(1) or later

• DCNM Management: Required DCNM version: 10.4(1) or later

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential

*: Future Software Support

Physical Dimensions
17.3 inches

1.72 inches

Chassis dimensions:
Height: 1RU ; 1.72 inches
Width: 17.3inches Logical Expansion Module (LEM)
Depth: 20.12 inches

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
MDS 9132T Rear view

• Hot swappable PSUs

• 1 PSU option for 8-port base switch
• 1-2 PSU option for 16-32 port configurations
• Hot swappable redundant fan modules
• 2 Fan modules for 8-port base switch
• 2-4 Fan modules for 16 – 32 port configuration with 3+1 redundancy
• Color coded FAN and PSUs:
• Blue: Port side exhaust airflow
• Red: Port side intake airflow
• PSU and FAN status LEDs
© 2017 Cisco and/or its affiliates. All rights reserved.
Cisco Partner Confidential
Cisco MDS 9132T Chassis LEDs

• Switch Status (tri-color)

• Power Supply Status (tri-color)

• Fan Status (tri-color)

• Ethernet Management Port Link (bi-color)

• Ethernet management Port Activity (bi-color)

• Fibre Channel Port Link (one tri-color LED per port)

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Management Ports
• One console port

• 2x 10/100/1000M-baseT Ethernet management ports

• Mgmt0 is a standard management port
• Mgmt1 is reserved for future use

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Port Options
• Supports 4/8/16/32G FC speed

• Supported port modes: F, E, TE, SD, ST

• Up to 1024G FC front panel bandwidth

• Incremental port options

• 8-Port à 16-Port à 32-Port
• 8-Port à 24-Port à 32-port
• Supported SFPs: 32G FC SW/LW, 16G FC SW/LW, 8G FC SW/LW optics

• Based on same IO ASIC as 32G module of MDS 9700 platform

• SAN Analytics* supported on ANY port

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential

* Future Software Support

 Cisco MDS 9132T Port Numbering
Top Row: Odd number of ports
Bottom Row: Even number of ports
Console Port
Mgmt1 Eth Port*

1 15 17 31

2 16 18 32

Status Mgmt0 Eth Port

USB Port LEM Holding Screw
LEDs

PSU 1 FAN 1 FAN 2 FAN 3 FAN 4 PSU 2

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential

*: For future use

Cisco MDS 9132T Rear LEDs
FAN Status Led
PSU Status Leds

• Power supply LEDs FAN handles

• Top LED (tri-color): Power Supply AC Status (1 tri-color per PS)

• Bottom LED (tri-color): PS Failure (AMBER), Temperature, Over Voltage, Fan Failure)
• Fan Status LEDs (tri-color)
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Cisco MDS 9132T Port Options
Any 8 Ports

8 Port
+16 Port LEM +8 Ports on Demand

16 Port

Any 16 Ports

+16 Port LEM

24 Port

Expansion Module Slot +8 Ports on Demand

32 Port
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Feature support
• POAP (Power On Auto Provisioning)

• Secure boot

• ISSU and ISSD

• FC Redirect capability

• Active power measurement

• Link level encryption (Cisco FC Trustsec) with AES-128 bit encryption

• Interoperability with Smartoptics xDWDM 8G/16G transceivers

• Per VL credit slicing and buffer allocation

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Buffer to Buffer Credits
• Default buffer credits per port: 500

• 8300 buffer credits per 16-port port group

• Extended buffer credits:

• Maximum of 8270 credits on a single port
• Supports ~512km @ 32G FC Speed

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Supported Optics
Part Number Product Name

DS-SFP-FC32G-SW 32G FC Shortwave Optics

DS-SFP-FC32G-LW 32G FC Longwave Optics

DS-SFP-FC16G-SW 16G FC Shortwave Optics

DS-SFP-FC16G-LW 16G FC Longwave Optics

DS-SFP-FC8G-SW 8G FC Shortwave Optics

DS-SFP-FC8G-LW 8G FC Longwave Optics

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Power Supply Unit (PSU) and Fan Redundancy
• 650W Power supply unit

• N:N PSU redundancy (2 PSUs)

• Redundant Fan modules

• Airflow options (PSU and Fan):

• Port-Side Intake (PSI)
• Port-Side Exhaust (PSE)

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Built-in PSU mis-match airflow protection
• Automatic detection of PSU hot-swap and airflow directions

• If PSU airflow is different from Fan-tray, a fatal error will be generated

• Switch will perform software initiated power down procedure in10 minutes

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Fan-tray mismatch airflow protection
• Automatic detection of Fan during hot-swap and installation

• In case of mismatch airflow detection:

• Switch will power down the board in 20 seconds
• Light front panel status LED in slow flashing red
• Default shutdown timer: 20s

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Troubleshooting

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Front Panel LED Troubleshooting

LED Color State Description

Interface LED (Link) Solid green Link is up

Steady flashing green Link is up (beacon used to identify port)

Intermittent flashing green Link is up (traffic on port)

Solid orange Link is disabled by software

Flashing orange A fault condition exists

Off No Link
Ethernet LED
MGMT Eth1 (Left)
Solid Green Physical link

Off No Activity
Ethernet LED
MGMT Eth0 (Right)
BlinkingCisco
Green
© 2017 Cisco and/or its affiliates. All rights reserved. Partner Confidential Activity
LED Troubleshooting

Indicator Location Function Color Status State

Solid On Power supply is OK

Green
Front panel Chassis
Power LED Off The module is not receiving power
of chassis Power/Health
Red On Power supply failed

Green Solid on All diagnostics pass. The module is operational

The switch is booting or running diagnostics.
Orange
An over temperature condition occurred (minor)
Front panel
Status LED System status
of chassis Red Blinking Unmatched airflow direction for PSU/Fan-tray
The diagnostic test failed during initialization
Red Solid on sequence
An over temperature condition occurred (major)

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
LED Troubleshooting
Indicator Location Function Color Status State

Green Solid on Fan operating normally

Fan status Front panel Fan health
Red Solid on Fan failure

OFF No AC power to power supply

Green Solid On Power supply on and DC output OK

Blinking AC Power OK, DC output not enabled

PSU Status Faceplate of each
PSU Health
Indicators PSU Critical power supply failures, over voltage,
Solid On
over current, over temperature or fan failure
Amber Blinking Power supply warning, PSU is still operating

OFF Operating normally

Faceplate of each Fan tray heath Green Solid On Fan tray operating normally
Fan Status
Fan Module indicator
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential Amber Solid On Fan failure within the fan tray
Use Cases

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Scalable Standalone Switch
Pay-as-You-Grow Flexibility

Fl
sha
St
or
ag
e
Backup
Storage
Application &
© 2017 Cisco and/or its affiliates. All rights reserved. Database Servers
Cisco Partner Confidential
Enterprise Core-Edge SAN
MDS 9132T as Middle of Row / Top of Rack

• MDS 97xx as Core switches Flash Storage

• Analytics visibility across Fabric (NVMe/SCSI)

• Easy to scale, Easy to Manage

Core MDS 9700s

ToR/MoR MDS 9132T

Compute Servers
(NVMe/SCSI)
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Business Continuity in Metro-SAN
Business Continuity and Consolidation

Flash Storage
MAN / WAN
xWDM circuit

Backup Site
Application &
Database Servers

Primary Data Center

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Software Features

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Congestion Isolation and Slow Flow

• Credit-based flow control scheme using R_RDY protocol

• R_RDY protocol informs the transmitter that a new buffer is available at the
receiver end

• Normal flows and slow flows both consume the buffer credits on an ISL

• Slow flows hog the link credits on the ISL, reducing credits available for
normal flows leading to under-utilization of the ISL
PMON: Port monitor
VL: Virtual Link
R_RDY: Receiver Ready
ER_RDY: Extended Receiver Ready
HOL: Head On Line

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Congestion-isolation Benefits

• Automatic detection of congestion at an interface level using PMON policy

• Configurable option to mark a device slow
• Configurable option to exclude a slow device
• Include and exclude config is supported for offline devices
• There is a de-isolate CLI option available at an interface level

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Current Congestion Isolation overview

Credit-based flow control scheme using R_RDY protocol

Normal flows and slow flows both consume the buffer credits on an ISL, leading under-
utilization of the ISL
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Improved Congestion Isolation overview

Automatic isolation of slow flows to a low priority VL

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Congestion Isolation
Congestion Isolation using the ER_RDY & VL capability

• Supported on MDS 16G and 32G platform

• In a mixed fabric (8G Platform + 16G Platform + 32G module), feature will
not work effectively (Not disabled but will not function as expected)

• Allows different flows to be routed to different virtual links based on QoS

• Provides:
• A mechanism to isolate slow flows to a specified VL, there by the line rates of
normal flows are not affected.

• Automatic isolation of a slow flow to a low priority VL

PMON: Port monitor
• De-isolation of a slow flow using CLI mechanism VL: Virtual Link
R_RDY: Receiver Ready
ER_RDY: Extended Receiver Ready
HOL: Head On Line
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Configuration Phase 1: Enabling ER_RDY
• Prepare fabric with all ISLs to come up in ER-RDY flow control mode
• CLI Command: “system fc flow-control er_rdy” (default is r_rdy (disabled))

• Followed by link flap for ISLs to to have VL(Virtual Links) configured

• Port-channel members can be flapped individually (less disruptive)
• Check the operational mode of the ISLs (R-RDY/ER-RDY)
• Confirm ER-RDY mode using CLI:
• CLI: show flow-control er_rdy

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Configuration Phase 2: Congestion-Isolation
• Enabling indicates it will honor the slow-interface detected by
PMON
• PMON isolates the flow destined to the hosts behind slow-interface
• Active policy with port-guard action as ”cong-isolate” is needed
• PMON sends updates of the slow-interfaces to FCNS
• PMON policy monitors the counters in the policy
• When threshold is hit, PMON policy sends an update of the interface
to FCNS process

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Configuration guidelines

Disabled by default
Turn ON before adding any new switch to the fabric working in R-RDY mode
Members of a PC can be flapped individually to change to ER_RDY mode
All configurations are per switch, not applied to the entire fabric

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Slowdrain counters monitored by PMON

TX Credit Not Available: Transmit credit not available is the continuous no-
transmit of credit for 100ms
Credit Loss Recovery: When credits are not available for 1s or 1.5 sec (F or
E)
TX-Slowport-Oper-Delay: The total amount of time the port was in the zero-
transmit-credits, or the average credit delay experienced by the port
TXWait : Counter increments whenever TX B2B credits = 0 and frame(s) are
waiting in the egress queue

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Design Considerations

Congestion-detection is based on the PMON counters which are monitored

at the interfaces level
Any device behind an interface, upon slow detection will be marked as a
slow-device
Upon logging back in, will detected as a slow device again on same interface
NPV/ FDISC: Devices connected to a slow interface will be marked slow
TF port-channel: All devices behind the TF port-channel will be marked slow
One of the VL is reserved for QoS based HBA, Zone QoS
Devices involved with IVR flows, FCR, FCIP and FCoE are not honored by this
feature

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Anti-Counterfeit Technology

• Built-in ACT2 Chip to identify genuine Hardware

• Based on Secure Unique Device Identifier certificate

• SUDI (Secure Unique Device Identifier) is the combination of PID (Product ID)
+ SN (Serial Number) of module

• NX-OS reads PID and SN from ID-SPROM of module during boot up

• NX-OS authenticates the module’s PID & SN in ID-SPROM against the SUDI
in ACT2 chip

• Upon authentication failure, module will fail to load with error message:
ACT2_AUTH_FAIL: ACT2 test has failed on module 9 with
error : ACT2 authentication failure

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
 Secure Boot
• Traditional mode: Boot Code Integrity not Validated
• Cisco Secure Boot: Boot Code Integrity is validated by Hardware

Traditional Boot Cisco Secure Boot

Step 1 Step 2 Step 1 Step 2 Step 3 Step 4

Boot Code CPU CPU Boot Code Hardware Anchor CPU CPU CPU
Bootloader NX-OS Micro Micro Bootloader NX-OS
Loader Loader

Bootloader OS Launched : Image Signing Micro Loader Bootloader OS Launched

Checks OS
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
checks boot loader Checks OS
: Integrity Checks
Secure Boot

• BIOS Chip is programmed with Secure Boot Key

• During Module boot up, board FPGA validates SB-Key

• Upon failure of SB-Key validation, Module will fail to boot

• BIOS tampering will invalidate the module

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Diagnostic Enhancements

• Generate test traffic pattern from at MAC level

• Generate loopback at destination port to support D-port

and SYM-port functionality

• Ability to track packet drops on a SID/DID/VSAN basis

• Ability to track the packet latency in a switch

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Smart Arbiter

• Enhances arbiter chip function

• Smart arbiter will implement a “smart” request-generation feature
in which generation of request to central arbiter is delayed until a
superframe forms or programmable time

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
RDP: Read Diagnostics
Parameter

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
What is Read Diagnostics Parameter (RDP) ?
• It is an ELS command which requests for diagnostics parameters for
an interface were the specified FCID is connected
• Communicates with LC module in identifying the SFP parameters for
the interface were the specified FCID is connected/logged in
• It consists of following parameters
• Control Frame
• ELS Frame
• Request-Response
• Diagnostics parameters for an N_Port_ID (FCID)

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Read Diagnostics Parameter: The need
• Helps proactively identify
• Link Diagnostics
• Link Errors
• Degraded conditions
• Exchange issues

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
 RDP Parameters Checklist
Port Speed
Port speed capabilities SFP diagnostics
Port operating speed Optical Elements Vcc
(Voltage, Temp, Current, Temp
Link Errors
Tx power, Rx Power) Tx Bias
Link failure
LOSync High Alarm/Warning Tx power

LOS Low Alarm/Warning Rx power

Primitive sequence FEC status
Optical Product Data
Invalid transmission word Wavelength
Vendor name
Invalid CRC SFP type
Model Number
Serial Number Port type
Buffer Credits
Switch B2B credit Revision Number FEC
Attached Device B2B Manufacture date Corrected blocks
credit Uncorrected blocks
Return Trip Time (RTT)
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Use Cases
RDP
RDP
Case #2
FCID: Case #1
XX1122 S1 S2 FCID:
Dom: XX Dom: YY
ISL YY1177
FLOGI
H
PLOGI RDP Case #3
PRLI T
Case #1 Case#2
Case#3
(Local) (Remote) (Target)
SID XX1122 SID XX1122 SID XX1122
DID FFFFFE DID FFFCYY DID YY1177
Payload XX1122 Payload YY1177 Payload YY1177
PLOGI Local PLOGI Remote PLOGI End Device

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Deployment Recommendations
• Included with Base License, enabled by default
• Supported on MDS 16G and 32G Platforms
• No Support for MDS 95xx, 9124, 9132, 9148, 9222i

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
RDP Control Flow Get
RDP

Duplicate Yes is PC Yes

Request ? or NPV?

No No

Send Get Err

is Device RJT Details Timeout
Logged? No

Yes Send
Resp
Get Err
Ifindex Timeout

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Limitations
• HBA PLOGI is required before an RDP
• Not Supported:
• Port-Channel
• NPV Uplink
• FCoE / FCIP ports

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Virtual Machine ID (VMID)

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
 VM- VM- VM-

Why VMID ?
VM-
1 2 3 4

VN_Po
VN_Port Hypervisor rt
PN_Port

• Multiple VM’s share the same VN_port

• The fabric only knows the FCID associated with the VN_port
• Advantages:
• When VM moves, no new FCID is needed
• Possible Multipathing situation
• Granular VM visibility:
• Provide VM visibility within SAN fabric for VM’s behind hypervisor
• Enforce VM specific policies

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
VMID Overview
• Based on T11 standard
• Allows to mark VM originating from HBA
• CS_CTL field in the fc2 frame is leveraged to carry the 8 bit tag
• MDS 32G FC module can look up acl based on the incoming tag

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
VMID Flow process
• CS_CTL will be used with sid,did and vsan in the acl flow key
• HBA’s register the VM’s hosted behind a given FCID on the
hypervisor
• VMIS service on the switch maintains this per VM information as a
database
• Each VM is identified using a combination of Global VE ID, FCID and
Local VE ID
• Per FCID VM database is distributed across the fabric

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Supported hardware and software platforms
• Supported across following platforms
• MDS 9700
• MDS 9132T
• MDS 9250i
• MDS 9148S

• Minimum NX-OS required: NX-OS 8.2(1)

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Enabling VMID Feature
• Enable / Disable:
switch(config)# feature vmis
• Configure VMID range per vsan basis
vmis range <1-255> vsan <vsan-id>
show flogi database details
show flogi database details
show tech-support vmis
show vmis database [ domain | fcid | global-vmid |
interface | local | vem | vmotion | vsan ]

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Important notes
• VMID range configuration is available only at per vsan basis
• Persistence is available only for local entries
• If vmis process crashes/restarts, will have to re-learn from the
switches in remote domains
• If FC Management security is enabled management station pwwn
needs to be configured for Unzoned name server read operation
• When a vmotion occurs VM de-instantiation will not occur
immediately

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Buffer credit recovery

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
What is it - Buffer credit recovery?
• Lost credit:
• SoF (Start of Frame) is corrupted
• R_RDY is corrupted

• Detects and corrects the lost credits, non-disruptively

• Negotiated between two directly attached peer ports
• Enabled automatically during initial negotiation
• Per-hop feature

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Functionality
• Can be used on any non-arbitrated loop link
• Most useful on unreliable links, such as MANs or WANs
• Can also help on shorter, high loss links, link with a faulty fiber
connection
• Process
• Local and port peer send checkpoint primitives during initial negotiation
• If frame loss is detected, sends R_RDY to replenish lost credits
• If R_RDY loss is detected, port internally replenishes lost credits

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Implementation
• Recovery implementation
• Buffer-to-buffer state change SoF (BB_SCs) are transmitted every 2
BB_SC_N frames sent
• Enables attached port to determine if any frames are lost
• Upon frame loss detection, the receiver transmits the appropriate number
of R_RDYs to compensate for the lost frames
• Buffer-to-buffer state change R_RDY (BB_SCs) are transmitted every 2
R_RDY primitives
• Enables port to determine if any R_RDY primitives are lost
• If R_RDY primitive loss is detected, the receiver increments the number of
transmit credits to compensate for the lost R_RDYs
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Limitations
• E Ports:
• Enabled by default on ISLs
• Works on an ISL between a Cisco MDS switch and a peer switch from
any vendor, provided this feature is supported on the peer switch.
• Supported only on links that are in R_RDY flow control mode

• F Port:
• Enabled by default
• Enabled on an F port between a Cisco MDS switch and a peer device
from any vendor
• N-Port ID Virtualization (NPIV) ports do not support buffer-to-buffer credit
recovery for Cisco N-Port Virtualizer (Cisco NPV) switch logins
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Configuration
• Configuration example
• switch# configure terminal
• switch(config)# interface fc 1/1
• switch(config-if)# switchport fcbbscn
• This example shows how to configure default credits on an interface
• switch# configure terminal
• switch(config)# interface fc 1/1
• switch(config-if)# switchport fcrxbbcredit default
• switch(config-if)# switchport fcrxbbcredit 2000

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Verification
switch# show flow-control r_rdy module 3
fc3/17
Fc3/18

switch# show interface fc2/17 bbcredit

fc2/17 is trunking
Transmit B2B Credit is 255
Receive B2B Credit is 12
Receive B2B Credit performance buffers is 375
12 receive B2B credit remaining
255 transmit B2B credit remaining
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Troubleshooting
• Use the show logging onboard interrupt-stats command to view the
number of times a port sent extra R_RDYs or incremented transmit
buffer to buffer credits to restore credit count
switch# show logging onboard interrupt-stats

-------------------------------------------------------------------------------
INTERRUPT COUNTS INFORMATION FOR DEVICE: FCMAC

-------------------------------------------------------------------------------
Interface|| Interrupt Counter Name | Count | Time Stamp MM/DD/YY HH:MM:SS | | |

-------------------------------------------------------------------------------

fc1/1 |IP_FCMAC_INTR_ERR_BB_SCR_INCREMENT |1 |01/01/17 20:00:00

fc1/1 |IP_FCMAC_INTR_ERR_BB_SCS_RESEND |1 |01/01/17 10:00:00

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
Forward Error Correction
(FEC) enhancements

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
 FEC/TTS description
• A technology used to correct errors in data transmission over unreliable or noisy environment

• Can correct upto 11bits in a 2112 bits frame

• Implemented at hardware level in MDS

• Optional @ 16G FC, Must for 32G FC

• Advantages of use:
• Allows limited error correction
• Reduce number of packet drops
• Proactively identify physical layer issues
• No Performance impact / No bit-rate change

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
 FEC counters
• Noisy media increases FEC counters

• Indicates corruption either in FC primitives (or) FC frames.

• Counters to monitor:
• FCP_PCS_CNT_STATS_ERRORS_RX_FEC_CORRECTED_BLOCKS
• 32-bit counter that counts the number of blocks corrected by the FEC Decoder. (Hardware counter name:
fec_corrected_block_counter)
• FCP_PCS_CNT_STATS_ERRORS_RX_FEC_UNCORRECTED_BLOCKS
• 32-bit counter that counts the number of blocks the FEC Decoder was unable to correct (Hardware counter name:
fec_uncorrected_block_counter)

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
 FEC configuration
configure t
interface fc1/1
switchport mode E (or) F
switchport speed 16000
switchport speed fec
switchport speed fec tts // FEC encryption/decryption enabled after TTS negotiated

Caveats:
• FEC doesn’t work with “switchport speed auto” configuration
• TTS-nego is mandatory with HBAs supporting FEC
• FEC & TTS-nego are optional on ISL ports

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
 FEC Verification
switch(config-if)# sh int fc1/1 | i fc|fec|speed
fc1/1 is up
Speed is 16 Gbps
admin fec state is up
oper fec state is up

switch(config-if)# show interface fc7/15 cou details | i fc|fec

fc7/15
0 fec corrected blocks
0 fec uncorrected blocks

module-7# show hardware internal statistics device fcmac all port 17 | i FEC
FCP_PCS_CNT_STATS_ERRORS_RX_FEC_CORRECTED_BLOCKS 0000000000000002 17 -
FCP_PCS_CNT_STATS_ERRORS_RX_FEC_UNCORRECTED_BLOCKS 0000000000000007 17 –

switch(config-if)# show interface fc7/15 cou details | i fc|fec

0 fec corrected blocks
0 fec uncorrected blocks

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
 FEC cli: counters
module-7# show hardware internal statistics device fcmac all port 17 | i FEC
FCP_PCS_CNT_STATS_ERRORS_RX_FEC_CORRECTED_BLOCKS 0000000000000002 17 -
FCP_PCS_CNT_STATS_ERRORS_RX_FEC_UNCORRECTED_BLOCKS 0000000000000007 17 –

module-1# sh ha int fcmac port 1 port-info | i FEC

FEC admin/oper mode : ADMIN UP/SUCCESS
FEC tts cfg/sta : ADMIN UP/SUCCESS

module-1# show port-config internal link-events

Time PortNo Speed Event Reason
---- ------ ----- ----- ------
Feb 6 04:39:04 2014 00018794 fc7/17 16G LR Success
Feb 6 04:39:03 2014 00258935 fc7/17 16G UP Not FL
Feb 6 04:39:03 2014 00258583 fc7/17 --- FEC_UP neg-succ
Feb 6 04:38:59 2014 00799923 fc7/17 --- DOWN Disable

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
 Supported hardware and software releases
• 16G FC Module / MDS 24/10G FCIP Module
• Switchport fec
• Integrated from 6.2(7)
• Switchport fec tts
• Introduced in 6.2(11c) (Not available in 6.2(13))
• Available in all releases from 6.2(15)

• 32G FC Module
• Switchport fec [tts] integrated in NX-OS release 8.1(1) onwards
• MDS 32G 32-Port Switch
• NX-OS 8.2(1) onwards
• Supported HBAs
• Qlogic, Emulex, FICON host

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
 HPE Product Offerings
Cisco MDS 32Gb FC Module & 32Gb SFPs
HPE P/N HPE Product Names (Long, Short) Cisco PN/PID Cisco Description
• HPE C-series 32 Gb Fibre Channel Short Wave SFP+ Transceiver
Q9D30A DS-SFP-FC32G-SW= 32 Gbps Fibre Channel SW SFP+, LC
• HPE C-series 32Gb FC SW SFP+ Transceiver

• HPE C-series 32 Gb Fibre Channel Long Wave SFP+ Transceiver

Q9D31A DS-SFP-FC32G-LW= 32 Gbps Fibre Channel LW SFP+, LC
• HPE C-series 32Gb FC LW SFP+ Transceiver

• HPE StoreFabric SN8500C 48-port 32Gb Fibre Channel Director Module MDS 9700 48-Port 32-Gbps Fibre
Q9D32A DS-X9648-1536HK9=
• HPE SN8500C 48-port 32Gb FC Dir Module Channel Switching Module, spare

Please refer to the HPE StoreFabric SN8500C Director QuickSpec for additional information
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
 HPE Product Offerings – Cisco MDS 9132T Switch
(HPE StoreFabric SN6610C 32Gb FC Switch)
HPE P/N HPE Product Names (Long, Short) Cisco PN/PID Cisco Description
• HPE StoreFabric SN6610C 32Gb 8-port 16Gb Short Wave SFP+ Fibre MDS 9132T 32G FC switch, w/ 8 active ports
Q9D34A Channel Switch DS-C9132T-8PMESK9 + 8x16G SW Optics, 2 Fans, 1 PSUs, Port Side
• HPE SN6610C 32Gb 8p 16Gb SFP+ FC Swch Exhaust
• HPE StoreFabric SN6610C 32Gb 8-port 32Gb Short Wave SFP+ Fibre MDS 9132T 32G FC switch, w/ 8 active ports
Q9D35A Channel Switch DS-C9132T-8PMETK9 + 8x32G SW Optics, 2 Fans, 1 PSUs, Port Side
• HPE SN6610C 32Gb 8p 32Gb SFP+ FC Swch Exhaust
• HPE StoreFabric SN6610C 32Gb 24-port 16Gb Short Wave SFP+ Fibre MDS 9132T 32G FC Enterprise switch, w/ 24
Q9D36A Channel Enterprise Switch DS-C9132T-24PESK9 active ports + 24x16G SW Optics, Enterprise
• HPE SN6610C 32G 24p 16G SFP+ FC Ent Swch License, 4 Fans, 2 PSUs, Port Side Exhaust
• HPE StoreFabric SN6610C 32Gb 8-port Fibre Channel Upgrade E-LTU MDS 9132T 32G FC switch 8 Port Activation
Q9Z41AAE M9132T-PL8=
• HPE SN6610C 32Gb 8-port FC Upgrade E-LTU License for Base eDelivery, spare
• HPE StoreFabric SN6610C 16-port Fibre Channel Expansion Module MDS 32G FC Expansion module, w/ 16 active
Q9D33A M9XT-FC1632H=
• HPE SN6610C 16-port FC Expansion Module ports, Spare
• HPE StoreFabric SN6610C 650W Power Supply
Q9D37A DS-CAC-650W-E= AC PSU Port side Exhaust, Spare
• HPE SN6610C 650W Power Supply
• HPE StoreFabric SN6610C Fan Tray
Q9D38A DS-C32S-FAN-E= MDS 9132 FAN tray, port side Exhaust Spare
• HPE SN6610C Fan Tray

Please refer to the HPE StoreFabric SN6610C 32Gb FC Switch QuickSpec for additional information
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential
 HPE Product Spares – SFPs, Director Module, Switch*
HPE FRU P/N HPE FRU Description Cisco PN/PID Cisco Description
P06958-001 SPS-MDS 32Gb FC SW SFP+ Transceiver DS-SFP-FC32G-SW= 32 Gbps Fibre Channel SW SFP+, LC

P06959-001 SPS-MDS 32Gb FC LW SFP+ Transceiver DS-SFP-FC32G-LW= 32 Gbps Fibre Channel LW SFP+, LC
MDS 9700 48-Port 32-Gbps Fibre Channel Switching Module,
P06960-001 SPS-MDS SN8500C 48-p. 32Gb FC Dir Module DS-X9648-1536K9=
spare

P06993-001 SPS-MDS SN6610C 16-port FC Expan. Module M9XT-FC1632= MDS 32G FC Expansion module, w/ 16 active ports, Spare
MDS 9132T 32G 1 RU FC switch, w/ 8 active FC ports, 2
P06992-001 SPS-MDS SN6610C 32Gb 8a. ports FC Switch DS-C9132T-MEK9=
Fans, 1 PSU, Port Side Exhaust, Spare

P06994-001 SPS-MDS SN6610C 650W Power Supply DS-CAC-650W-E= AC PSU Port side Exhaust, Spare

P06995-001 SPS-MDS SN6610C Fan Tray DS-C32S-FAN-E= MDS 9132 FAN tray, port side Exhaust Spare

* All switch bundles share the same spare

p/n
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Partner Confidential