NX-OS Routing and

Layer 3 Switching
Arkadiy Shapiro
Technical Marketing Engineer, NX-OS and Nexus 7000
arshapir@cisco.com

BRKARC-3472
Session Objectives

At the end of the session, the participants should:

 Understand NX-OS Layer 3 software architecture
and features
 Understand configuration highlights and best
practices associated with NX-OS Layer 3 IP
routing features
 Understand how NX-OS and Nexus switches can
be used to build scalable and highly available
Layer 3 networks

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Session Non-objectives

This session does not include:

 Deep dive on any Nexus switch hardware or NX-OS
software architectures
 Technical discussion on Layer 3 protocols,
technologies and associated merits
 Detailed roadmap discussion for Layer 3 features on
NX-OS
 Detailed configuration and troubleshooting
information for NX-OS Layer 3 features

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Notes

For purposes of this presentation:

 All features are mentioned with assumption of latest
NX-OS software release and all licenses installed
 NX-OS – IOS comparisons refer to most commonly
deployed configuration on Catalyst 6500 which may not
be the latest IOS version for that platform
 “Graceful Restart” (“GR”) and “Non-Stop Forwarding”
(“NSF”) are complementary technologies (not the
same) to achieve protocol high availability in NX-OS
and Nexus 3000/5000/7000
 “NSR”, “Non-stop routing” and “stateful restart” will
refer to the same technology and behavior in NX-OS

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Related Cisco Live 2011 events
Breakout sessions
Session-ID Session Name
BRKARC-3470 Cisco Nexus 7000 Switch Architecture
BRKARC-3452 Cisco Nexus 5000/5500 and 2000 Switch Architecture
BRKARC-3471 Cisco NX-OS Software – Architecture
Data Center deployments with MPLS on NX-OS (Nexus
BRKMPL-2107
7000)
BRKCRS-3145 Troubleshooting Cisco Nexus 5000/2000 Series Switches
BRKCRS-3144 Troubleshooting Cisco Nexus 7000 Series Switches
BRKDCT-2951 Deploying Nexus 7000 in Data Center Networks
Virtual Device Context (VDC) Design and Implementation
BRKDCT-2121
Considerations with Nexus 7000
TECRST-3190 Advanced IP Routing Fast Convergence
TECCRS-2001 Enterprise High Availability Design and Architecture

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Related Cisco Live 2011 events
Labs
Session-ID Session Name
LTRDCT-4047 Deploying Nexus 7000/NX-OS Hands-on Lab
LTRCRT-5205 Configuring Nexus 7000 Virtualization

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Agenda

 Layer 3 Requirements
 NX-OS Layer 3 Software Architecture
 Unicast Routing Protocols
 Multicast Protocols
 First Hop Redundancy Protocols
 Routing Policy and Policy-based Routing (PBR)
 Layer 3 High Availability and Fast Convergence
 Summary

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Agenda

 Layer 3 Requirements
 Network Design Requirements
 System Requirements
 NX-OS Layer 3 Software Architecture
 Unicast Routing Protocols
 Multicast Protocols
 First Hop Redundancy Protocols
 Routing Policy and Policy-based Routing (PBR)
 Layer 3 High Availability and Fast Convergence
 Summary

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
 Layer 3 Requirements

Typical Enterprise Network Deployment

Branch
Campus Core

Enterprise Core

DC Core

DC Aggregation DC Core /
Aggregation

DC Access DC Access
DC 1 DC 2
BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
 Layer 3 Requirements

Layer 3 Features by Place in the Network

Enterprise L3: Routing Table Scale or Lean Core, BGP, IGP, BFD
Core MPLS P/PE

DC Core L3: Routing Table Scale, BGP, IGP, BFD, Dual Stack, IPv6/v4 translation
MPLS P/PE

L2 / L3 Boundary: FHRP, ARP / ND / IGMP / MLD Handling

DC Aggregation L3: OSPF, IS-IS, EIGRP, BGP, VRF-lite, MPLS PE, PIM, Dual stack,
summarization, BFD
DCI: OTV, LISP

Routed Access Layer:

DC Access Basic Layer 3, OSPF Stub, EIGRP Stub, Static / Default Routes, Dual
Stack

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
 Layer 3 Requirements

Layer 3 Access Considerations

 Advantages:
 Lower L2 table utilization
 Smaller L2 domain  reduce
STP impact
 Drawbacks:
 More subnets to manage that are
smaller
 More L3 configuration points
 VM mobility constrained to a
smaller L2 domain
 Use OSPF stub area, / EIGRP
stub, default routes
 Fabric Extender technology Servers Servers
enables the expansion of L2
domains without adding STP
hops
BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
 Layer 3 Requirements

Aggregation Switch Role

L3 links northbound Network

Stress
Aggregation L3

L2

Access

Pod: scope of L2 domain

Aggregation Switch provides much more than

feeds and speeds
BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
 Layer 3 Requirements

Control Plane at Layer 3 Aggregation

What‟s really happening behind the curtain
I‟m still here… I‟m still here too…
I‟m the active (vPC state sync., (vPC state sync.,
FHRP hello, FHRP hello,
default gateway Routing Protocol hello, Routing Protocol hello,
(FHRP active) BFD…) BFD…)

Aggregation
L3 links northbound

L2 links southbound Remote

MAC Updates
(DCI)

Access
Process
multicast state
(Multicast DR)
Relay DHCP query
Listen to DHCP reply
(DHCP request) My MAC address Who is my default
is 11:22:33:44:55:66. gateway?
What‟s my IP address? (ARP request)
(DHCP request)

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
 Layer 3 Requirements

Layer 3 Core

 100% Layer 3
 Network backbone
 MPLS P / PE functionality
 Fast Convergence
 ISP multi-homing
 Routing table scale
 Allows growth in aggregation
PODs

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
Agenda

 Layer 3 Requirements
 Network Design Requirements
 System Requirements
 NX-OS Layer 3 Software Architecture
 Unicast Routing Protocols
 Multicast Protocols
 First Hop Redundancy Protocols
 Routing Policy and Policy-based Routing (PBR)
 Layer 3 High Availability and Fast Convergence
 Summary

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
 Layer 3 Requirements

Nexus 7000 Layer 3 / MPLS Support

Hardware and Software
 Layer 3 / MPLS support on all existing and future M-series I/O
modules
 F1 modules support Layer 3 and MPLS via proxy-mode
 Layer 3 support: NX-OS 4.0(1) onwards
 MPLS support: NX-OS 5.2(1) onwards

N7K-M132XP-12 N7K-M108X2-12L
N7K-M132XP-12L

N7K-M148GS-11 N7K-M148GT-11
N7K-M148GS-11L N7K-M148GT-11L
BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
 Layer 3 Requirements

Nexus 7000 Layer 3 / MPLS Licensing

5.2
Transport
MPLS XL
Services 5.2 5.2
Enterprise Advanced Enhanced L2 FCoE SAN
Enterprise

Base
 Enterprise Services License - all routing protocols except RIP
 MPLS License – MPLS features
 XL License – higher FIB table sizes (optional)
 Base License - all other Layer 3 features (SVIs, Layer 3 ports,
FHRP, IGMP etc)

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
 Layer 3 Requirements

Nexus 5500 Layer 3 Support

Hardware and Software
 Layer 3 support: NX-OS 5.0(3) N1(1) onwards
 Enable Layer 3 on all ports
 Enable SVIs for all switch ports (inc. FEX ports)

N55-D160L3
 Single daughter-card
 Nexus 5548P / 5548UP only
 In-rack upgradeable for deployed units

N55-M160L3
 Expansion module
 Nexus 5596UP only
 No front-panel ports

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
 Layer 3 Requirements

Nexus 5500 Layer 3 Licensing

Layer 3 Requirements
 Layer 3 Base License
 $0 with daughter card / expansion module purchase
 Installation required
 SVIs, L3 routed ports on non-FEX interfaces, routed ACLs and uRPF
 Static Routing, RIPv2, OSPFv2, EIGRP-stub
 HSRP, VRRP
 IGMP v2/v3, PIMv2
 Maximum 256 non-connected OSPF routes

 Layer 3 Enterprise Services

 Layer 3 Base license must be present
 Full EIGRP, BGP
 VRF-Lite
 Scalable to 8K routes

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
Agenda

 Layer 3 requirements
 NX-OS Layer 3 Software Architecture
 Unicast Routing and Forwarding
 Multicast Routing and Forwarding
 Layer 3 Virtualization
 Unicast Routing Protocols
 Multicast Protocols
 First Hop Redundancy Protocols
 Routing Policy and Policy-based Routing (PBR)
 Layer 3 High Availability and Fast Convergence
 Summary

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
 NX-OS Layer 3 Software Architecture

IP Unicast Routing and Forwarding

 NX-OS and Nexus switches strongly decouple

control plane and data plane
 Routing and forwarding tables built on control plane
using routing protocols or static configuration
OSPF, EIGRP, IS-IS, RIP, BGP for dynamic routing

 Tables downloaded to forwarding engine hardware

for data plane forwarding

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
 NX-OS Layer 3 Software Architecture

NX-OS and Nexus 7000 • Neighbor management

• Protocol database
Unicast Routing Architecture • Add/Delete prefixes

RIP IS-IS EIGRP Static OSPF v2 OSPF v3 BGP ARP

AM
u4RIB u6RIB
mRIB
Unicast Routing Information Base (uRIB)
Reverse Path Forwarding
(RPF) updates
• Common API to clients
• Digest routing info
• Push routes to platform
• Select best-nexthop(s) per prefix uFDM • Route download
• Apply routing policy
• Build final routing table
Supervisor

FIB Manager

• Translate routes to hardware format

• Program hardware forwarding engine

Forwarding Hardware
I/O Module
BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
 NX-OS Layer 3 Software Architecture

Unicast Routing Information Base (URIB)

 IPv4 (U4RIB) and IPv6 (U6RIB)
 Client-driven and designed for easy insertion of new clients

u4RIB u6RIB
Unicast Routing Information Base (uRIB)

 Adds and stores:

 Best paths and next hops
 Backup paths and next hops
 Remote MPLS labels with routes

 Resolves Recursive Next Hops

 Downloads routes and remote MPLS labels to FIB
 Table-based (multi-topology “capable”)
 No direct Interface Manager or IP event handling
BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
 NX-OS Layer 3 Software Architecture

NX-OS and Nexus 7000

Unicast Routing Architecture
RIP IS-IS EIGRP Static OSPF v2 OSPF v3 BGP ARP

AM
u4RIB u6RIB
mRIB
Unicast Routing Information Base (uRIB)

uFDM
Supervisor

FIB Manager

Forwarding Hardware
I/O Module
BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
 NX-OS Layer 3 Software Architecture

Forwarding Information Base (FIB)

 Receives route updates from URIB with outgoing MPLS labels:

 BGP (VPNv4 and VPNv6) / static routes + VPN labels
 IGP / static routes + LDP labels
 Programs routes in the hardware and passes label information to
LFIB for programming
 Prefixes chained to table and identified by (VRF, topology)
 Each table is tied to a VDC descriptor block
 FIBs on I/O modules belonging to same VDC are programmed
identically

FIB Manager

Forwarding Hardware

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
 NX-OS Layer 3 Software Architecture

URIB and FIB

Management and troubleshooting highlights
 show routing [ipv4|ipv6] [<prefix>] [vrf <vrf>]
 Displays software routing (URIB) information
 Can also use IOS-like show ip route command
 show forwarding [ipv4|ipv6] route module <mod>
[vrf <vrf>]
 Hardware forwarding (FIB) information on per-module basis
 show forwarding adjacency module <mod>
 Displays hardware adjacency table information on
per-module basis

 IOS CEF based commands are not supported

(show ip cef …)
 NX-OS refers to connected routes as “direct”
(show routing direct)
BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
 NX-OS Layer 3 Software Architecture

URIB
Management and troubleshooting
switch# show routing ipv4 direct
31.3.3.3/32, ubest/mbest: 2/0, attached
*via 31.3.3.3, Lo0, [0/0], 1w5d, direct

switch# show routing ipv4 10.13.34.4 detail

10.13.34.4/30, ubest/mbest: 1/0
*via 10.13.32.6, Eth8/2, [110/6], 01:19:22, ospf-1, intra
via 34.34.34.34, [200/0], 01:19:06, bgp-100, internal, tag 100
recursive next hop: 34.34.34.34/32

switch# show routing ipv6 2142:142:34:109::1:1 detail

2142:142:34:109::1:0/112, ubest/mbest: 1/0
cand ubest/mbest: 1/0, ufdm in/update: 1/0
*via ::ffff:34.34.34.34%default:IPv4, [200/0], 01:20:18, bgp-100,
internal, tag 300 (mpls-vpn)
recursive next hop: ::ffff:34.34.34.34/128

!Recently learned routes

switch# show routing updated since 03:35
!Routes pointing to specific neighbor
switch# show routing next-hop 10.1.1.1

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
 NX-OS Layer 3 Software Architecture

FIB
Management and troubleshooting
switch# show forwarding vrf default route 10.13.34.4 module 8
Prefix Next-hop Interface
10.13.34.4/30 10.13.32.6 Ethernet8/2

switch# show forwarding vrf default ipv6 route 2142:142:34:109::1:1

module 8
Prefix Next-hop Interface
*2142:142:34:109::1:0/112 10.13.32.6 Ethernet8/2

switch# show forwarding vrf default adjacency module 8

next-hop rewrite info interface
10.13.32.2 0026.980a.2943 Ethernet8/1
10.13.32.6 001a.a223.a034 Ethernet8/2

switch# show forwarding vrf default ipv6 adjacency module 8

next-hop rewrite info interface
2015:32:48:109::2 001c.b0cb.3580 Eth4/5.109
fe80::21c:b0ff:fecb:3580 001c.b0cb.3580 Eth4/5.109

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
Agenda

 Layer 3 requirements
 NX-OS Layer 3 Software Architecture
 Unicast Routing and Forwarding
 Multicast Routing and Forwarding
 Layer 3 Virtualization
 Unicast Routing Protocols
 Multicast Protocols
 First Hop Redundancy Protocols
 Routing Policy and Policy-based Routing (PBR)
 Layer 3 High Availability and Fast Convergence
 Summary

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
 NX-OS Layer 3 Software Architecture

IP Multicast Routing and Forwarding

 Forwarding tables built on control plane using

multicast protocols
PIM-SM, PIM-SSM, PIM-Bidir, IGMP, MLD, MSDP

 Tables downloaded to:

 Forwarding engine hardware for data plane forwarding
 Replication engines for data plane packet replication

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
 NX-OS Layer 3 Software Architecture

NX-OS and Nexus 7000

Multicast Routing Architecture
IGMP MSDP PIM PIM6 ICMPv6 / MLD
Add (*,G) & (S,G) from Join/Prune
Add (*,G) & (S,G) from reports Add (S,G) from SAs & Register/Assert Add (*,G) & (S,G) from reports

m4RIB m6RIB
uRIB
Multicast Routing Information Base (mRIB)
RPF updates
• Add routes, OIFs
• Update when RPF changes • Push routes to platform
mFDM • Route download

Supervisor

FIB Manager
• Translate routes to hardware format
• Program hardware forwarding and
replication engines

Forwarding / Replication Hardware

I/O Module
BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
 NX-OS Layer 3 Software Architecture

Multicast Routing Information Base (MRIB)

Highlights
 IPv4 (M4RIB) and IPv6 (M6RIB)
 Interface with various client processes to provide/learn
multicast routes
 Aggregate route-information provided by multiple protocols
into single multicast routing table
 RPF services via U4RIB / U6RIB
 Client-driven and designed for easy insertion of new clients
 Detailed traffic statistics

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
 NX-OS Layer 3 Software Architecture

MRIB and FIB

Management and troubleshooting highlights
 show routing [ipv4|ipv6] multicast
[vrf <vrf>] [<source-ip>] [<group-ip>]
[summary]
 Displays software multicast routing (MRIB) information
 Can also use IOS-like show ip mroute command
 show forwarding [ipv4|ipv6] multicast route
[source <ip>] [group <ip>]
[vrf <vrf>] module <mod>
 Displays hardware multicast forwarding (FIB) information on per-
module basis

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
 NX-OS Layer 3 Software Architecture

MRIB and FIB

Management and troubleshooting
switch# show routing multicast 239.1.1.1 summary
Source packets bytes aps pps bit-rate oifs
(*,G) 767 84370 110 0 0 bps 2
10.1.1.2 9917158 1269395810 127 4227 4 mbps 2

switch# show ip mroute detail

(150.1.1.1/32, 226.1.2.1/32), uptime: 1d01h, mrib pim ip
Stats: 10369/1910536 [Packets/Bytes], 171.200 bps
Incoming interface: port-channel 17, RPF nbr: 20.1.17.1
Outgoing interface list: (count: 4)
port-channel 57, uptime: 1d01h, pim

switch# show forwarding multicast route group 226.1.2.1/32 source

150.1.1.1/32 mod 1
(150.1.1.1/32, 226.1.2.1/32), RPF Interface: port-channel 17, flags:
Received Packets: 0 Bytes: 0
Number of Outgoing Interfaces: 4
Outgoing Interface List Index: 43
port-channel 27 Outgoing Packets:41505884 Bytes:4087857190
port-channel 37 Outgoing Packets:0 Bytes:0

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
Agenda

 Layer 3 requirements
 NX-OS Layer 3 Software Architecture
 Unicast Routing and Forwarding
 Multicast Routing and Forwarding
 Layer 3 Virtualization
 Unicast Routing Protocols
 Multicast Protocols
 First Hop Redundancy Protocols
 Routing Policy and Policy-based Routing (PBR)
 Layer 3 High Availability and Fast Convergence
 Summary

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
 NX-OS Layer 3 Software Architecture

Virtual Device Contexts (VDC)

Highlights
Layer 2 Protocols Layer 3 Protocols

VDC 4 STP SPAN OSPF GLBP

Carve a single Nexus 7000 CTS … PIM …

switch into four logical

network entities Layer 2 Protocols Layer 3 Protocols

 Flexible separation of VDC 3 UDLD STP BGP MSDP

LACP … PIM …
hardware and software
resources
Layer 2 Protocols Layer 3 Protocols
 Isolate software faults and VDC 2 CDP STP BGP VRRP

reduce fate sharing LACP … EIGRP …

 Securely delineate
Layer 2 Protocols Layer 3 Protocols
administrative domains Default VDC VLAN STP OSPF HSRP

 All MPLS and Layer 3 CDP … PIM …

features are VDC aware

System Infrastructure

Linux Kernel

More details: BRKDCT-2121 - Virtual Device Context (VDC) Design and Implementation with Nexus 7000
BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
 NX-OS Layer 3 Software Architecture

Layer 3 Virtualization (VRF)

Highlights

 Virtual Routing and Forwarding

(VRF) provides logical network
segmentation
 Virtualizes IP routing control and
data plane functions
 For each VRF:
 Independent routing and
forwarding decisions
 IPv4 and IPv6 ucast/mcast
tables are created
automatically
 VRF membership of each
interface dictates which
forwarding table to use

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
 NX-OS Layer 3 Software Architecture

Layer 3 Virtualization (VRF)

Pre-defined VRFs
 Management VRF
 For management purposes only
mgmt 0
 Only mgmt 0 interface can be in management VRF
 mgmt 0 can not be assigned to another VRF

 Default VRF
 All Layer-3 interfaces are in the default VRF
 Default routing context for all show commands
 Routing protocols run in default VRF context unless
other VRF context is specified
 Similar to IOS global routing table

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
 NX-OS Layer 3 Software Architecture

VRF Awareness

 In NX-OS, everything is VRF aware!

(IPv4 and IPv6)

Service NX-OS Service NX-OS

Support Support

VRF Specific Static ARP  VRF Aware Ping 

 

VRF_C

VRF_E
VRF Aware SCP

VRF_D
VRF_B
PBR-set VRF

VRF_A
VRF Aware AAA (RADIUS)  VRF Aware SNMP agent 
VRF Aware AAA (TACACS+)  VRF Aware SSH 
VRF Aware BGP  VRF Aware Syslog 
VRF Aware DHCP  VRF Aware Telnet 
VRF Aware DNS  VRF Aware Traceroute 
VRF Aware FTP / TFTP  VRF Aware uRPF 
VRF Aware NTP  VRF Aware FHRP protocols 

Server
Server
Server

Server
Server
Server
Server
Server

Server
Server
Server
Server

Server
VRF Aware OSPF  VRF Aware WCCP 

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
 NX-OS Layer 3 Software Architecture

Layer 3 VPN

 Unicast - IPv4 and IPv6

(6VPE) DC Edge PE PE
 Multicast - IPv4 (MVPN)
 Most widely deployed MPLS
P

VRF D
P

VRF C
VRF B
DC Core
application to provide

VRF A
scalable segmentation
 Supports network
consolidation and access to PE PE
Agg /Access
shared services (L2/L3 boundary)

 Provides IPv6 connectivity

over the same IPv4/MPLS

Server

Server

Server
Server
Server
Server
Server
Server
L3VPN infrastructure Server
Server
Server
Server
Server
Server
Server
Server
Server

Server
Server
Server

Server
 Easier IPv4 to IPv6 migration
More details: BRKMPL-2107 - Data Center deployments with MPLS on NX-OS (Nexus 7000)
BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
 NX-OS Layer 3 Software Architecture

Layer 3 Unicast VPN and VRF-lite

VPN features
 Comprehensive VPN feature set for VRF-lite and MPLS VPNs:

Sub-feature area Details

Route-leaking Yes – MPLS VPN and VRF-lite
(Import/Export)
Route filtering and limits Yes – per-VRF configurable max route
limit in RIB
PE-CE routing protocols OSPF v2, EIGRP for IPv4, BGP for
IPv4 and IPv6, static
IPv6 6VPE with Multipathing

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
 NX-OS Layer 3 Software Architecture

Layer 3 Virtualization (VRF)

Management and Troubleshooting
 Consistent placement of “vrf” option in the end
 “routing-context” mode
 Set VRF scope for show and exec commands
 By default, CLI exec mode operates in VRF default

Without “routing-context” With “routing-context”

switch#
routing-context vrf management

switch# switch%management#
ping 172.26.242.1 vrf management ping 172.26.242.1

switch# switch%management#
show routing vrf management show routing

switch# switch%management#
show ip arp vrf management show ip arp

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
 NX-OS Layer 3 Software Architecture

NX-OS - IOS differences

Layer 3 VPN and VRF-lite Configuration
IOS NX-OS
ip vrf vrf-1 vrf context vrf-1
rd 1:1 [shutdown]
route-target import 1:1 ip route 10.0.0.0/8 1.1.1.1
route-target export 1:1 rd 1:1
ip route vrf vrf-1 10.0.0.0 address-family ipv4 unicast
255.0.0.0 1.1.1.1 route-target import 1:1
route-target export 1:1
!OR

vrf definition vrf-1

rd 1:1
address-family ipv4
route-target import 1:1
route-target export 1:1
ip route vrf vrf-1 10.0.0.0
255.0.0.0 1.1.1.1

int GigEthernet2/1 int Ethernet2/1

vrf forwarding vrf-1 vrf member vrf-1

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
 NX-OS Layer 3 Software Architecture

Unicast Routing Architecture

with VDCs and VRFs
Supervisor Engine

VDC 1 VDC 2 VDC 3

OSPF OSPF OSPF OSPF OSPF
VRF 1 VRF 2 VRF 1 VRF 2 VRF 1 VRF 2 VRF 3

VRF 1 VRF 2 VRF 1 VRF 2 VRF 1 VRF 2 VRF 3

uRIB uRIB uRIB

uFDM uFDM uFDM

IP FIB IP FIB IP FIB

Hardware Hardware Hardware

I/O Module I/O Module I/O Module
BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
Agenda

 Layer 3 requirements
 NX-OS Layer 3 Software Architecture
 Unicast Routing Protocols
 Multicast Routing
 First Hop Redundancy Protocols
 Routing Policy and Policy-based Routing (PBR)
 Layer 3 High Availability and Fast Convergence
 Summary

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
 For Your
Reference
NX-OS Unicast Routing Protocols
Platform support
Nexus 7000 Nexus 5500 Nexus 3000
Protocol
IPv4 IPv6 IPv4 IPv4
RIPv2  -  
IS-IS  - - -
OSPFv2  N/A  
OSPFv3 -  - -
EIGRP    
BGPv4    
Static    

Notes:
• Nexus 3000 / 5000 support IPv6 in hardware, software support in future release
• Check release notes and configuration guides for latest support information
BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
 Unicast Routing Protocols

NX-OS Unicast Routing Protocols

Highlights
 Modular architecture
 Extensive High Availability features
 IPv4 and IPv6 support
 CLI/features similar to IOS (Catalyst 6500)

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
 Unicast Routing Protocols

IGP Virtualization

 Multiple processes (4 per VDC)

 Multiple VRFs within a process (best practice)
 Forward referencing of VRFs
Protocol IOS NX-OS
OSPF router ospf 10 vrf 1 router ospf 10
router-id 1.1.1.1 router-id 1.1.1.1
vrf 1
router-id 1.1.1.2
IS-IS router isis 1 router isis 1
net <net name> net <net name 1>
!only 1 VRF per process vrf 1
vrf 1 net <net name 2>
EIGRP router eigrp 1 router eigrp 1
eigrp router-id 1.1.1.1 router-id 1.1.1.1
address-family ipv4 vrf 1 vrf 1
autonomous-system 200 router-id 2.2.2.2
eigrp router-id 2.2.2.2 autonomous-system 200

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
 Unicast Routing Protocols

NX-OS Unicast Routing Protocols

Configuration highlights
 Interface centric model for IGPs
 Simplified config viewing
 Less error prone
 Backward compatible with “network” statements (hidden command)
router ospf 1 interface Ethernet 1/1
passive-interface GigEthernet1/1 ip ospf passive-interface
network 10.0.0.1 0.0.0.0 area 0 ip router ospf 1 area 0
router is-is 1 interface Ethernet 1/2
network 10.0.0.1 0.0.0.0 ip router isis 1
router eigrp 1 interface Ethernet 1/3
network 10.0.0.1 0.0.0.0 ip router eigrp 1

 Enable feature first to be able to configure:

feature ospf

 Protocol shutdown option (all protocols):

router ospf foo
shutdown

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
 Unicast Routing Protocols

NX-OS Unicast Routing Protocols

Management and troubleshooting highlights
 Modular approach
show run ospf

 Controlled restart
restart ospf 1

 Record adjacency changes to logfile

router ospf foo
log-adjacency-changes

 Debug-filters
 Restrict the amount of debug output: neighbor, interface, VRF etc.
 Each protocol has its own set of debug-filters
 Use debug-filter <protocol> vrf all to apply to all
VRFs

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
Agenda

 Layer 3 requirements
 NX-OS Layer 3 Software Architecture
 Unicast Routing Protocols
 OSPF
 IS-IS
 EIGRP
 BGP
 Multicast Protocols
 First Hop Redundancy Protocols
 Routing Policy and Policy-based Routing (PBR)
 Layer 3 High Availability and Fast Convergence
 Summary

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
 Unicast Routing Protocols

OSPF in NX-OS
Highlights
 OSPF v2 (RFC 2328 with RFC 1583 compatibility)
 OSPF v3 (RFC 5340, IPv6 only)
 Dynamic and modern link-state protocol
 Extensive High-Availability features
 Graceful Restart (Nexus 3000 / 5500 in helper mode only)
 Stateful Restart
 SSO / ISSU

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
 Unicast Routing Protocols

For Your
OSPF in NX-OS Reference

Key features
Sub-feature area Details (defaults in bold)
Areas Multi-area
Regular, Stub, NSSA (RFC 3101)
LSAs Type 3 (summary), Type 5 (redistributed), Type 9
and 10 (opaque)
Multi-pathing 8 (up to 16)
Metric manipulation Static cost
Auto-cost reference bandwidth
Link types Broadcast, Point-to-point, Virtual link, Sham link
Summarization and Area range; summary-address
filtering Filter lists (using route-maps)
Authentication Clear text, MD5, keychains
Traffic Engineering Yes – TE extentions (RFC3630) for OSPF v2

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
 Unicast Routing Protocols

OSPF in NX-OS
Configuration and management highlights
 Area ID for OSPF v2 and v3:
 Configure as single decimal or dotted decimal
switch(config-if)# ip router ospf 1 area ?
A.B.C.D or <0-4294967295> Area Id as an integer or ip address
switch(config-if)# ip router ospf 1 area 2

 For consistency, always display in dotted decimal

switch(config-if)# show run ospf
router ospf 1
interface Ethernet2/4
ip router ospf 1 area 0.0.0.2

 OSPF v3 address family mode configuration for

summarization / redistribution:
router ospfv3 1
address-family ipv6 unicast
area 4 range 2006:2000::/32

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
 Unicast Routing Protocols

OSPF in NX-OS
Configuration examples
 OSPF v2
router ospf foo
passive-interface default
network 10.0.2.0/24 area 0.0.0.2

interface Ethernet2/5
ip router ospf 100 area 0.0.0.0
interface Ethernet2/6
no ip ospf passive-interface
ip router ospf 100 area 0.0.0.1

 OSPF v3
router ospfv3 foo
router-id 60.60.60.60
area 4 virtual-link 50.50.50.50
address-family ipv6 unicast
area 4 range 2006:2000::/32

interface Ethernet3/1
ipv6 address 2006:5101:0100::2003/33
ipv6 router ospfv3 foo area 0
BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
 Unicast Routing Protocols

OSPF in NX-OS
Management and troubleshooting
switch# show ip ospf
Routing Process p1 with ID 40.40.40.40 VRF default
Stateful High Availability enabled
Graceful-restart is configured
grace period: 60, state: (null)
Last graceful restart exit status: None
Supports opaque LSA
This router is an area border and autonomous system boundary.
Redistributing External Routes from
isis-one
Administrative distance 110
Reference Bandwidth is 40000 Mbps
<…>
Maximum paths to destination 8
Number of external LSAs 6, checksum sum 0x2d81e
Number of areas is 3, 2 normal, 0 stub, 1 nssa
Number of active areas is 2, 2 normal, 0 stub, 0 nssa
Area BACKBONE(0)
Area has existed for 1d01h
Interfaces in this area: 5 Active interfaces: 5
Passive interfaces: 0 Loopback interfaces: 0
No authentication available
SPF calculation has run 18 times
BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
 Unicast Routing Protocols

NX-OS / IOS OSPF Differences

Feature IOS NX-OS

Graceful Restart Yes (Disabled) Yes (Enabled)
IETF (RFC3623) and/or IETF (RFC3623)
Cisco
Auto-cost Reference 100 Mbps 40 Gbps
Bandwidth default
iSPF Yes (Disabled) No
Partial SPF Yes (Always Enabled) Yes (Always Enabled)
Flood Reduction Yes (Disabled) No
Neighbor logging Yes (Enabled) Yes (Disabled)
LSA for loopback IP Advertise subnet Advertise /32 route

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
Agenda

 Layer 3 requirements
 NX-OS Layer 3 Software Architecture
 Unicast Routing Protocols
 OSPF
 IS-IS
 EIGRP
 BGP
 Multicast Protocols
 First Hop Redundancy Protocols
 Routing Policy and Policy-based Routing (PBR)
 Layer 3 High Availability and Fast Convergence
 Summary

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
 Unicast Routing Protocols

IS-IS in NX-OS
Highlights
 ISO 10589 Intermediate system to intermediate
system intra-domain routing exchange protocol
 IPv4 routing support only, IPv6 on roadmap
 Extensive High-Availability features
Graceful Restart
Stateful Restart
SSO / ISSU

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
 Unicast Routing Protocols

For Your
IS-IS in NX-OS Reference

Key Features
Sub-feature area Details (defaults in bold)
Multi-pathing 8 (up to 16)
Authentication Clear text, MD5, keychains
Wide Metric Yes
Traffic Engineering Yes – TE extentions (RFC 3784)

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
 Unicast Routing Protocols

IS-IS in NX-OS
Configuration highlights
 IS-IS point-to-point link configuration:
switch(config)# interface Ethernet 2/4
switch(config-if)# medium p2p

 Metric style – “transition” option for interoperability:

switch(config)# router isis 1
switch(config-router)# metric-style ?
transition Use both narrow and wide metric style

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
 Unicast Routing Protocols

IS-IS in NX-OS
Configuration
router isis foo
mpls ldp autoconfig level-1-2
net 49.0000.0032.0001.0001.00
is-type level-1-2
metric-style transition
log-adjacency-changes
address-family ipv4 unicast
default-information originate

interface Vlan90
ip router isis 1
isis passive level-1-2
interface loopback0
ip router isis 1
interface Ethernet1/8
ip router isis 1

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
 Unicast Routing Protocols

NX-OS / IOS IS-IS Differences

Feature IOS NX-OS

Graceful Restart Yes (Disabled) Yes (Enabled)
IETF (RFC 5306) or IETF (RFC 5306)
Cisco
Metric style Narrow Wide
options Wide Transition
Transition
Neighbor logging Yes (Enabled) Yes (Disabled)

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
Agenda

 Layer 3 requirements
 NX-OS Layer 3 Software Architecture
 Unicast Routing Protocols
 OSPF
 IS-IS
 EIGRP
 BGP
 Multicast Protocols
 First Hop Redundancy Protocols
 Routing Policy and Policy-based Routing (PBR)
 Layer 3 High Availability and Fast Convergence
 Summary

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
 Unicast Routing Protocols

EIGRP in NX-OS
Highlights
 Single process instance for IPv4 and IPv6 routing
 Support for only IP protocols
 Wide metric support with interoperability
 High-Availability features
Graceful Restart (Nexus 3000 / 5500 in helper mode only)
SSO / ISSU

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
 Unicast Routing Protocols

For Your
EIGRP on NX-OS Reference

Key features
Sub-feature area Details (defaults in bold)
Summarization and Summary address (on interface)
route-filtering Distribute lists (using prefix list or route
map)
Multi-pathing 8 (up to 16)
Authentication MD5, keychains

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
 Unicast Routing Protocols

EIGRP in NX-OS
Configuration highlights
 “distribute-list” configuration is interface-centric only:
feature eigrp

interface Ethernet2/1
ip address 10.1.48.4/24
ip router eigrp 1
ip distribute-list eigrp 1 route-map test in

router eigrp 1
network 10.1.48.0 0.0.0.255

 Use “default-information originate” to accept and

advertise default routes:
switch(config)# router eigrp 1
switch(config-router)# default-information originate ?
<CR>
always Always advertise default route
route-map Use a route-map for default route metrics

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
 Unicast Routing Protocols

EIGRP in NX-OS
Configuration
router eigrp foo
router-id 31.3.3.3
vrf 100
router-id 31.3.3.3
address-family ipv4 unicast
address-family ipv6 unicast
address-family ipv4 unicast
address-family ipv6 unicast
redistribute bgp 1 route-map test

interface Vlan100
ip router eigrp 1
ip passive-interface eigrp 1

interface port-channel10.100
ip router eigrp 1
ip summary-address eigrp 1 101.0.0.0/16 255

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
 Unicast Routing Protocols

EIGRP in NX-OS
Management and troubleshooting highlights
switch# show ip eigrp topology detail-links vrf 100
IP-EIGRP Topology Table for AS(1)/ID(31.1.1.1) VRF 100
Codes: P - Passive, A - Active, U - Update, Q - Query, R - Reply,
r - reply Status, s - sia Status
P 11.0.15.0/30, 6 successors, FD is 768, serno 603
via 11.0.9.2 (768/320), Ethernet1/16.100
via 11.0.6.2 (768/320), Ethernet1/13.100
via 11.0.21.2 (768/320), Ethernet1/1.100
via 11.0.22.2 (768/320), Ethernet1/2.100
via 11.0.8.2 (768/320), Ethernet1/15.100
via 11.0.7.2 (768/320), Ethernet1/14.100
via 11.0.10.2 (1024/768), port-channel10.100
<…>

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
 Unicast Routing Protocols

NX-OS / IOS EIGRP Differences

Feature IOS NX-OS

Graceful Restart Yes (Disabled) - Cisco Yes (Enabled) - Cisco
Auto-summary Yes (Disabled) No (Always Disabled)
Neighbor logging Yes (Enabled) Yes (Disabled)
Bandwidth / delay No Yes
configuration on
interface

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
Agenda

 Layer 3 requirements
 NX-OS Layer 3 Software Architecture
 Unicast Routing Protocols
 OSPF
 IS-IS
 EIGRP
 BGP
 Multicast Protocols
 First Hop Redundancy Protocols
 Routing Policy and Policy-based Routing (PBR)
 Layer 3 High Availability and Fast Convergence
 Summary

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
 Unicast Routing Protocols

BGP
Highlights
 Full MP-BGP support
 BGP version 4 (RFC 4271)
 Multi-Protocol Extensions (RFC 2858)

 Integrated implementation for IPv4 and IPv6

 IPv6 support
 IPv6 peers and prefixes
 6PE / 6VPE
 4 byte ASN support and interoperability (RFC 4893)
 1 BGP instance per VDC

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
 Unicast Routing Protocols

For Your
BGP in NX-OS Reference

Key Features
 BGP and MP-BGP features:
Sub-feature area Details (defaults in bold)
Address Families IPv4/ IPv6 unicast and multicast, MDT (MVPN),
VPNv4, VPNv6 (6VPE), IPv6 labeled-unicast (6PE)
Load-balancing Label, IP src-dst, TCP/UDP
Multipath Yes – up to 16 for iBGP / eBGP
Route reflector Yes for all AFs
Route Manipulation Conditional Advertisement, routing policies
Hub and spoke Yes
HA and Graceful Restart (RFC 4724)
Convergence PIC Core
Next Hop Tracking
Low Memory Alert handling
Authentication MD5
BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
 Unicast Routing Protocols

BGP in NX-OS
Configuration highlights
 Neighbor-centric nested configuration model
 Address-family configuration required per neighbor
feature bgp
router bgp 1
address-family ipv4 unicast
network 20.20.20.0/24
neighbor 2.2.2.2 remote-as 2
update-source loopback 0
address-family ipv4 unicast
 Peer-templates (similar to peer-groups)
 Can inherit session and policy templates
template peer-session SESSION1
timers 30 90
template peer-policy POLICY1
send-community
default-originate
template peer CUSTOMER-PEERS
inherit peer-session SESSION1
address-family ipv4 unicast
inherit peer-policy POLICY1
neighbor 192.168.2.1 remote-as 10
BRKARC-3472
inherit© 2011
peer CUSTOMER-PEERS
Cisco and/or its affiliates. All rights reserved. Cisco Public 81
 Unicast Routing Protocols

MP-BGP in NX-OS
Configuration for L3 Unicast VPN
vrf context test
rd 118.48.20.162:100
address-family ipv4 unicast
route-target import 300:30
route-target export 100:10
router bgp 65000
address-family vpnv4 unicast
router-id 118.48.20.162
address-family ipv4 unicast
neighbor 118.8.20.164 remote-as 65000
update-source loopback10
address-family vpnv4 unicast
send-community extended
!Configuration under router bgp mode
vrf Prod
address-family ipv4 unicast
network 18.26.16.0/24
maximum-paths ibgp 4

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
 Unicast Routing Protocols

For Your
BGP in NX-OS Reference

Management and troubleshooting

 Session setup
show ip bgp summary
show ip bgp neighbor
show bgp sessions
show tcp connection
show tcp statistics

 BRIB (routes)
show ip bgp [<prefix>]
show ip bgp nexthop-database
show ip route [<prefix>]
show bgp paths

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
 Unicast Routing Protocols

BGP in NX-OS
Management and troubleshooting
show ip bgp neighbor 10.103.72.72
BGP neighbor is 10.103.72.72, remote AS 65001, ebgp link, Peer index 2
BGP version 4, remote router ID 3.3.3.3
BGP state = Established, up for 5d22h
Peer is directly attached, interface Ethernet2/1
Last read 00:00:21, hold time = 180, keepalive interval is 60 seconds
Last written 00:00:16, keepalive timer expiry due 00:00:43
Connections established 1, dropped 0
Last reset by us 00:00:10, due to bad peer AS error
Reset error value 6501
Last reset by peer 1d20h, due to holdtimer expired error
Neighbor capabilities:
4-Byte AS capability: advertised
Address family IPv4 Unicast: advertised received
Message statistics: Sent Rcvd
Opens: 1 1
Notifications: 0 0
Updates: 2 3
Keepalives: 8555 8558
Total: 8558 8562
Total bytes: 162604 162773
Bytes in queue: 0 0
For address family: IPv4 Unicast
BGP table version 20, neighbor version 20
6 accepted paths consume 216 bytes of memory
Local host: 10.103.72.3, Local port: 20582
Foreign host: 10.103.72.72, Foreign port: 179
BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 84
 Unicast Routing Protocols

NX-OS / IOS BGP Differences

Feature IOS NX-OS

Graceful Restart Yes (Disabled) – Yes (Enabled) –
IETF (RFC 4724) IETF (RFC 4724)
Synchronization Yes (Disabled) No (Always Disabled)
Auto-summary Yes (Disabled) No (Always Disabled)
Redistribution of Disabled Disabled
iBGP to IGP (use “redistribute (use route-map match
internal” to enable) statement to enable)
Neighbor logging Yes (Enabled) Yes (Disabled)

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 85
Agenda

 Layer 3 requirements
 NX-OS Layer 3 Software Architecture
 Unicast Routing Protocols
 Multicast Protocols
 First Hop Redundancy Protocols
 Routing Policy and Policy-based Routing (PBR)
 Layer 3 High Availability and Fast Convergence
 Summary

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 86
 For Your
Reference
NX-OS Multicast Protocols
Platform support
Nexus 7000 Nexus 5500 Nexus 3000
Protocol
IPv4 IPv6 IPv4 IPv4
IGMP v1 / v2 / v3  N/A  
PIM SM    
PIM SSM*    
PIM BiDir*   - -
MSDP  N/A  
MLD v1 / v2 N/A  N/A N/A

* Not supported in VPC mode

Nexus 3000 / 5000 support IPv6 in hardware, software support in future release
BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 87
 Multicast Protocols

PIM in NX-OS
Highlights
 PIM process responsibilities: Source Shared Tree
a.k.a RPT,
Source Tree rooted at the
 Form PIM relationship with peers a.k.a SPT,
rooted at the
rendezvous point

 Processes inbound and outbound PIM

source RP

protocol packets
 Interface with MRIB to provide/learn
multicast routes
 PIM support
 IPv4 (PIM) and IPv6 (PIM6)
 Sparse Mode - RFC 4601
 SSM - RFC 3569 / RFC 4607
 Bidir
Receivers
 Configurable ASM/SSM/Bidir group-ranges
 One single PIM process runs all PIM
flavors (SSM, BiDir, ASM) for VDC
BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 88
 Multicast Protocols

For Your
PIM in NX-OS Reference

Key features
 Configurable options for when Auto-RP should
listen versus forward Auto-RP messages.
 Configurable options for when BSR should listen
versus forward Bootstrap and Candidate-RP
messages

Sub-feature area Details (defaults in bold)

RP selection and  Static
redundancy  Boot Strap Router (BSR)
 Auto-RP listener and forwarder
 Anycast-RP (with PIM or MSDP) –
RFC 4610 / 3436
Troubleshooting mping, mtrace
Authentication Clear text, MD5

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 89
 Multicast Protocols

PIM in NX-OS
Configuration highlights
 Multicast routing enabled by default, no commands needed
such as “ip multicast-routing”, only enable PIM with
“feature pim”
 Group range taken inline without ACL (can supply a route-map)
 Configuring PIM Sparse Mode:
switch(config)# ip pim rp-address 1.1.1.1 group-list 224.0.0.0/4
switch(config)# interface Vlan101
switch(config-if)# ip pim sparse-mode

 Configuring PIM SSM:

switch(config)# interface Vlan101
switch(config-if)# ip pim sparse-mode
switch(config-if)# ip igmp version 3

 Configuring PIM BiDir:

switch(config)# ip pim rp-address 1.1.1.1 group-list 224.0.0.0/4 bidir
switch(config)# interface Vlan101
switch(config-if)# ip pim sparse-mode
BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 90
 Multicast Protocols

NX-OS / IOS Multicast Routing Differences

Feature IOS NX-OS

PIM version ver 1, 2 ver 2 only
PIM modes Sparse mode Sparse mode
Dense mode
Sparse-dense mode
IGMP ver 1, 2, 3 ver 2, 3

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 91
Agenda

 Layer 3 requirements
 NX-OS Layer 3 Software Architecture
 Unicast Routing Protocols
 Multicast Protocols
 First Hop Redundancy Protocols
 Routing Policy and Policy-based Routing (PBR)
 Layer 3 High Availability and Fast Convergence
 Summary

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 93
 First Hop Redundancy Protocols

First Hop Redundancy Protocols in NX-OS

Platform support
 FHRPs - family of protocols (HSRP, VRRP and
GLBP) designed to allow redundancy of first-hop IP
gateway

Nexus 7000 Nexus 5500 Nexus 3000

Protocol
IPv4 IPv6 IPv4 IPv4
HSRP v1  N/A  
HSRP v2    
VRRP v2  N/A  
VRRP v3 - - - -
GLBP  - - -

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 94
 First Hop Redundancy Protocols

First Hop Redundancy Protocols in NX-OS

Key Features
Sub-feature area Details
Protocols HSRP v1/v2, VRRP, GLBP
HA Object tracking
Stateful restart / ISSU / SSO

Authentication Clear text, MD5 (not VRRP)

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 95
 First Hop Redundancy Protocols

FHRPs with vPC

Active/Active mode HSRP/VRRP HSRP/VRRP
“Active”: “Standby”:
 All FHRP protocols in vPC Active for Active for
environment operate in shared L3 MAC shared L3 MAC

Active/Active mode
 No additional
configuration required
 General VRRP / HSRP best L3
practices still apply, except: L2
 Since running in active/active
mode, aggressive timers can
be relaxed
 No need to manipulate
priorities / preemption on
different devices to achieve
load-balancing

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 96
 First Hop Redundancy Protocols

FHRPs in NX-OS
Configuration and management highlights
 HSRP - use “hsrp” keyword instead of “standby”
 Config commands exist under HSRP/VRRP/GLBP sub-mode
switch(config)# int Ethernet 1/1
switch(config-if)# hsrp 1
switch(config-hsrp)# ?
authentication Authentication
ip Enable HSRP IPv4 and set the virtual IP address
mac-address Virtual MAC address
preempt Overthrow lower priority Active routers
priority Priority level
timers Hello and hold timers
track Associates track object to HSRP group
…
switch(config)# int Ethernet1/1
switch(config-if)# glbp 1
switch(config-glbp)# ?
ip Set Virtual IP address
…
switch# show hsrp brief
Interface Grp Prio P State Active addr Standby addr Group addr
Vlan100 100 120 P Active local 101.0.100.4 101.0.100.1
Vlan110 110 120 P Active local 101.0.110.4 101.0.110.1
BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 97
Agenda

 Layer 3 requirements
 NX-OS Layer 3 Software Architecture
 Unicast Routing Protocols
 Multicast Protocols
 First Hop Redundancy Protocols
 Routing Policy and Policy-based Routing (PBR)
 Layer 3 High Availability and Fast Convergence
 Summary

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 98
 Routing Policy and PBR

Routing Policy in NX-OS

Highlights
 NX-OS makes extensive use of route-maps
 Route-maps are defined as in IOS
 Redistribution requires a route-map to be
configured
 BGP regular expressions in policies undergo basic
CPU hog check during configuration

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 99
 Routing Policy and PBR

Routing Policy options in NX-OS

Key features
 Route-maps utilize:
 MAC list
 IPv4 and IPv6 Prefix List
 BGP AS-Path Access List
 Standard Community List
 Expanded Community List
 Access List (for PBR only)

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 100
 Routing Policy and PBR

Routing Policy
Configuration highlights
 Route map configuration and use in redistribution
 Classless redistribution by default (no “subnets” option)
router ospf 1
redistribute static route-map STATIC_TO_IGP

route-map STATIC_TO_IGP permit 10

match interface Vlan122

 BGP community is always configured and displayed in

new format (aa:nn):
switch(config)# ip community-list expanded test2 permit “1:1”

switch# show ip community-list

Expanded Community List test
permit "1:1“

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 101
 Routing Policy and PBR

Policy Based Routing

Highlights
 IPv4 and IPv6 support
 PBR allows routing redirection based on
configured rules (i.e. route-maps)
 PBR is a function of the uRIB applying
administratively defined rules for the final
information to be written to hardware
 PBR is implemented in hardware on
Nexus 7000 with ECMP support
 VRF-select option
 Forward referencing for route-maps used
in policies
BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 102
 Routing Policy and PBR

Policy Based Routing in NX-OS

Configuration
 Configure a route map to implement PBR:
switch(config)# feature pbr
switch(config)# route-map SELECT-PROVIDER permit 10
switch(config-route-map)# match ip address CUSTOMER-A
switch(config-route-map)# set ip next-hop 10.1.1.1
switch(config-route-map)# route-map SELECT-PROVIDER permit 20
switch(config-route-map)# match ip address CUSTOMER-B
switch(config-route-map)# set ip next-hop 10.2.2.2 10.2.2.3 load-share

switch(config)# interface ethernet 1/1

switch(config-if)# ip policy route-map SELECT-PROVIDER

 Note: Packets that are denied by the route map, or packets for which no
active next-hop can be found in the route map, will be forwarded through the
normal destination-based routing process.

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 103
Agenda

 Layer 3 requirements
 NX-OS Layer 3 Software Architecture
 Unicast Routing Protocols
 Multicast Protocols
 First Hop Redundancy Protocols
 Routing Policy and Policy-based Routing (PBR)
 Layer 3 High Availability and Fast Convergence
 Summary

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 105
 Layer 3 High Availability and Fast Convergence

Layer 3 High Availability

NX-OS and Nexus switches
 Robust features and best practices:
 System HA
 Data Plane HA
 Software HA
 Protocol Fast Convergence

More details: BRKARC-3471 – Cisco NX-OS Software Architecture

Building Highly Available Layer 3 Networks with NX-OS and Nexus 7000
BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 106
Agenda

 Layer 3 requirements
 NX-OS Layer 3 Software Architecture
 Unicast Routing Protocols
 Multicast Protocols
 First Hop Redundancy Protocols
 Routing Policy and Policy-based Routing (PBR)
 Layer 3 High Availability and Fast Convergence
 Software High Availability
 Protocol Fast Convergence
 Summary

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 107
 Layer 3 High Availability and Fast Convergence

Software High Availability

Design goal
 Non-Stop Forwarding (NSF) for all protocols and all
address families:
 Active supervisor component crash
 Supervisor Switchover (SSO) (initiated by user or HA policy)
 In-Service Software Upgrade (ISSU)

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 108
 Layer 3 High Availability and Fast Convergence

Software High Availability

 Protocols and services utilize HA mechanisms:

 Non-stop routing (NSR) / Stateful Restart
 Graceful Restart (GR)
 Periodic refresh (PR)
Protocol NSR GR PR
OSPF v2   N/A
OSPF v3   N/A
IS-IS   N/A
EIGRP -  N/A
BGP -  N/A
PIM / PIM6 - - 
RIP - - 
FHRPs  - -
BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 109
 Layer 3 High Availability and Fast Convergence

Stateful Process Restart

IS-IS IS-IS
STP STP

OSPF OSPF

PSS
No interaction with the
neighbor to recover state

PSS = Persistent Storage Service

PSS provides reliable persistent storage for the
software components to „checkpoint‟ their run-time
state enabling stateful restart
More details: BRKARC-3471 – Cisco NX-OS Software Architecture
BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 110
 Layer 3 High Availability and Fast Convergence

Graceful Restart

 Controlled restart of operations and signaling between

protocol neighbors to recover a control plane process
 Network device notifies neighbors of its “GR” capability prior to
event. That allows data to be forwarded during a restart
operation and neighbor re-establishment.
 NX-OS IETF Graceful Restart is NOT compatible with
Cisco NSF (OSPF, IS-IS)
NX-OS
Protocol
GR implementation
OSPF v2 RFC 3623
OSPF v3 RFC 5187
IS-IS RFC 3847
EIGRP Cisco
BGP RFC 4724
LDP RFC 3478
BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 111
 Layer 3 High Availability and Fast Convergence

Graceful Restart
EIGRP EIGRP
BGP BGP

OSPF OSPF

OSPF has already restarted

Graceful Restart requires once in last 4 min. Let‟s do
a “Graceful Restart”
interaction with the neighbors
to recover

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 112
 Layer 3 High Availability and Fast Convergence

Layer 3 Process Fault Recovery Logic

Example: OSPF on Nexus 7000
Monitor
service
process

Process
freezes/crashes

Process
Process
restarted Stateless Restart
restarted
Yes twice in 4
No (use GR extentions)
recently?
min?

No Yes

Initiate Stateful Switchover

(reload if single supervisor)
Process
supports
Stateful Restart
stateful
Yes No
restart
BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 113
 Layer 3 High Availability and Fast Convergence

Software High Availability

Best Practices and Recommendations
 Use routing protocols such as OSPF and IS-IS, which support
stateful process restart
 Make sure Graceful Restart is always enabled on both peers
 Regardless of stateful restart capability, keep GR extensions
!NX-OS
enabled
switch# on both
sh run ospf peers
all (default in NX-OS, not default in IOS)
router ospf 1
…
 In heterogeneous network, make sure that peer supports
standards-based GR (except EIGRP)
graceful-restart
…

!IOS
switch(config-t)# router ospf 1
switch(config-router)# nsf ietf

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 115
Agenda

 Layer 3 requirements
 NX-OS Layer 3 Software Architecture
 Unicast Routing Protocols
 Multicast Protocols
 First Hop Redundancy Protocols
 Routing Policy and Policy-based Routing (PBR)
 Layer 3 High Availability and Fast Convergence
 Software High Availability
 Protocol Fast Convergence
 Summary

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 119
 Layer 3 High Availability and Fast Convergence

Protocol Fast Convergence

What is it?
 Collection of features to improve network
convergence after various failures:
 Fast Failure Detection (FFD) – how quickly can I detect a
failure (neighbor down event)
 Fast Failure Reaction (FFR)– once I know about it, how
quickly can I react (notify peers and recalculate SPF)

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 120
 Layer 3 High Availability and Fast Convergence

Is FFD tuning needed beyond defaults?

 FFD tuning is needed when:

 Intermediate L2 hop over L3 link
 Concerns over protocol software failures
 Concerns over unidirectional failures on point-to-point
physical L3 links

 FFD tuning may not be needed when:

 Point-to-point physical L3 links with no concerns over
unidirectional failures
 Enough software redundancy to account for protocol
software failures
 FHRPs are running in active-active mode (VPC)

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 121
 Layer 3 High Availability and Fast Convergence

What are FFD timers?

 Layer 3 protocols may use hellos / keepalives

which are sent at a defined interval for failure
detection
 Dead / hold timer
 Lowering hello interval and dead timeout
values may speed up failure detection

“FFD timers”

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 122
 Layer 3 High Availability and Fast Convergence

FFD Challenges

Considerations when tuning down FFD timers:

 Single interface may have 2-3+ protocols establishing
adjacency
E.g., HSRP, PIM, OSPF on each SVI

 Lowering hello interval for every protocol increases supervisor

CPU utilization and wastes link bandwidth
 Configuration complexity
 NX-OS does not support non-disruptive software upgrade and
supervisor switchover with tuned down protocol timers

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 123
 Layer 3 High Availability and Fast Convergence

Protocol Fast Convergence

Fast Failure Detection with BFD
 Bidirectional Forwarding Detection (BFD) –
recommended fast failure detection mechanism
 Advantages of BFD vs lowered FFD timers:
 Reduced control plane load and link bandwidth usage
 Sub-second failure detection
 Stateful restart / ISSU
 Distributed implementation – hellos sent from I/O module

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 124
 Layer 3 High Availability and Fast Convergence

BFD in NX-OS
Highlights
 Software support: NX-OS 5.0(2a) onwards
 RFC 5880
 Version 1 only
 Single-hop only
 Distributed packet processing
 BFD packets sent with DSCP CS6 / CoS 6

OSPF HSRP Default timers OSPF HSRP

BGP PIM BGP PIM

Link state Link state

notifications BFD BFD
notifications

msec timer

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 126
 Layer 3 High Availability and Fast Convergence

For Your
BFD Clients and Interface Types Reference

 Supported BFD clients (IPv4 only):

OSPFv2
EIGRP
IS-IS
PIM
BGP
HSRP
VRRP
Static routes
MPLS TE FRR
 Supported interface types:
Switched Virtual Interface (SVI)
Physical Port
Port Channel
Sub-interface
Note: check configuration guides and release notes for latest supported configuration
BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 127
 Layer 3 High Availability and Fast Convergence

BFD Architecture in NX-OS (Nexus 7000)

Distributed Processing
 SUP-BFD - BFD process
running on Supervisor Engine Supervisor Engine
 Interfaces with LC-BFD OSPF HSRP PIM BGP IS-IS Etc.
processes
 Interfaces with BFD clients SUP-BFD
(OSPF, BGP etc)
EOBC
 LC-BFD – BFD process running
on CPU of each I/O module LC-BFD LC-BFD LC-BFD
 Communicates with SUP-BFD Module
process Inband
 Generates BFD hellos Hardware Hardware Hardware
 Support for stateful process I/O Module I/O Module I/O Module
restart, SSO and ISSU
 BFD is VRF and VDC aware

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 129
 Layer 3 High Availability and Fast Convergence

Fast Convergence and ISSU/SSO

 FFD - non-disruptive NX-OS software upgrades and

supervisor switchovers are not supported with lowered FFD
timers
 System allocates time for processes to come up after supervisor
switchover
 Until processes are up, no hellos are sent
 Potential false positive – dead timer expires before protocol
process comes up after SSO/ISSU

 FFR - non-disruptive NX-OS software upgrades and

supervisor switchovers are supported with FFR features
 After SSO, CPU is busy bringing up processes
 ISSU assumes no network events happening
 Network events with modified FFR timers may add extra CPU
load

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 131
 Layer 3 High Availability and Fast Convergence

Protocol Fast Convergence

Best Practices and Recommendations
1) Decide whether FFD tuning is needed. If not – do
not use it!
2) If FFD tuning is needed, use BFD for all protocols!
3) If BFD can‟t be used:
a) Prior to NX-OS 5.2, tuning down FFD timers is not
supported under any circumstances
b) From NX-OS 5.2, specific profiles tested to support tuned
FFD timers without ISSU/SSO

4) Keep consistent FFR configuration on all network

devices

More details on tested FFD timers:

http://www.cisco.com/en/US/docs/switches/datacenter/sw/5_x/nx-os/unicast/configuration/guide/l3_limits.html
BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 132
Agenda

 Layer 3 requirements
 NX-OS Layer 3 Software Architecture
 Unicast Routing Protocols
 Multicast Protocols
 First Hop Redundancy Protocols
 Routing Policy and Policy-based Routing (PBR)
 Layer 3 High Availability and Fast Convergence
 Summary

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 133
 Summary

Summary

 NX-OS is a modular operating system that decouples control

and data planes
 NX-OS Layer 3 implementation has numerous feature and CLI
usability enhancements
 NX-OS supports a variety of IPv4 and IPv6 routing, multicast
and gateway redundancy protocols to meet data center and
enterprise needs
 NX-OS and Nexus switches provide numerous features for
Layer 3 protocol high availability and fast convergence
 VDCs, VRFs, MPLS VPN and multiple process instances
support virtualization for Layer 3 protocols

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 134
 Summary

Acronyms

 ACL – Access Control List  OSPF – Open Shortest Path First

 API – Application Programming  OTV – Overlay Transport Virtualization
Interface  PIC – Protocol-Independent
 BFD – Bidirectional Forwarding Convergence
Detection  QoS – Quality of Service
 BGP – Border Gateway Protocol  RNH – Recursive Next Hop
 CNH – Connected Next Hop  RPF – Reverse Path Forwarding
 DCI – Data Center Interconnect  SSO – Stateful SwitchOver
 DSCP – Differentiated Services Code  URIB – Unified Routing Information
Point Base
 EIGRP – Enhanced Interior Gateway  UFIB – Unified Forwarding Information
Routing Protocol Base
 FEX – Fabric Extender  VDC – Virtual Device Context
 GR – Graceful Restart  VoQ – Virtual Output Queue
 MPLS – Multi-Protocol Label Switching  VPC – Virtual Port-Channel
 NSF – Non-Stop Forwarding  VPN – Virtual Private Network
 NSR – Non-Stop Routing

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 136
Complete Your Online
Session Evaluation

 Receive 25 Cisco Preferred Access points for each session

evaluation you complete.
 Give us your feedback and you could win fabulous prizes.
Points are calculated on a daily basis. Winners will be notified
by email after July 22nd.
 Complete your session evaluation online now (open a browser
through our wireless network to access our portal) or visit one
of the Internet stations throughout the Convention Center.
 Don‟t forget to activate your Cisco Live and Networkers Virtual
account for access to all session materials, communities, and
on-demand and live activities throughout the year. Activate
your account at any internet station or visit
www.ciscolivevirtual.com

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 137
 Visit the Cisco Store for
Related Titles
http://theciscostores.com

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 138
BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 139
Thank you.

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 140
 For Your
Layer 3 Functional Comparison Reference

Nexus 5000 / 7000

Nexus 5500 +
L3 Functional Areas Nexus 7000 / M1 Modules
L3 Card / Module
OSPFv2, EIGRP, RIPv2, BGP, IS-IS, Static, OSPFv2, RIPv2, PIM,
Routing Protocols PIM, IGMP IGMP, BGP, EIGRP
Dual Stack, OSPFv3, EIGRP, BGP,
IPv6 HSRPv6
For Management only

VRF Lite, VRF-aware features, VRF

L3 Segmentation Import/Export, MPLS VPNs
VRF-Lite, VRF-aware features

NSF/SSO/ISSU, Graceful Restart, ISSU – L2 STP edge ports only

High Availability NSR Graceful Restart – helper only
BFD, BGP NHT and PIC Core, MPLS
Fast Convergence TE FRR
No

Flexible Netflow, Sampled Netflow,

Monitoring ERSPAN
No

Layer 2 DCI Overlay Transport Virtualization (OTV) No

Policy-Based Routing, VRF Select,

Traffic Steering WCCPv2, Static Multicast MAC, No
MPLS Traffic Engineering

Tunneling / Mobility Unicast over GRE, LISP No

 Assumes all licenses present

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 142
 For Your
Layer 3 Scale Comparison Reference

Nexus 5000 / 7000 hardware and software

Nexus 7000 / M1-XL Nexus 5500 +
L3 Functional Areas
Modules L3 Card / Module
L3 Interfaces 4K 4K

IPv4 Unicast FIB 900K 8K

IPv4 Multicast FIB 32K 4K

L3 ECMP 16 Way 16 Way

ARP 50K 8K

Routing Adjacency 400K verified, 1M maximum 8K maximum

4K HSRP Groups 4K HSRP Groups

FHRP 255 VRRP Groups 255 VRRP Groups
Ingress: 2K
L3 ACL entries 128K
Egress: 1K

Segmentation 1K VRFs 1K VRFs

FEX Scale with L3

32 8
features enabled
 Numbers are subject to change with future software releases
 Assumes all licenses present
BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 143
Switching Platform Comparison

 More information on platform differences can be found here:

http://www.cisco.com/en/US/prod/switches/ps5718/ps708/net
working_solutions_products_genericcontent0900aecd805f095
5.pdf

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 144
 For Your
Nexus 7000 M1 Forwarding Engines Reference

Hardware Scalability comparison

 Hardware forwarding engine(s)  RACL/VACL/PACLs
integrated on every M-series I/O
module  QoS remarking and policing
policies
 IPv4 and IPv6 multicast support
(SM, SSM, BiDir)  Policy-based routing (PBR)

 Ingress and egress NetFlow (full  Unicast RPF check and IP source
and sampled) guard

Hardware Table M1 Modules M1-XL Modules M1-XL Modules

without XL License with XL License
FIB TCAM 128K 128K 900K
Classification TCAM (ACL/QoS) 64K 64K 128K
MAC Address Table 128K 128K 128K
NetFlow Table 512K 512K 512K

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 145
 Unicast Routing Protocols

Router-id selection process

 Best practice for IGPs and BGP – specify explicitly using

“router-id” command
 If not specified:

IOS NX-OS
1) Highest loopback interface IP 1) IP from loopback 0
2) Highest IP from any other 2) IP from first loopback interface
interface 3) IP from first physical interface

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 146
 Unicast Routing Protocols

Routing and vPC

Unicast Routing Protocols
 Don‟t use L2 port channel to attach routers to a vPC domain
 If both routed and bridged traffic is required:
 Individual L3 links for routed traffic
 L2 port-channel (vPC) for bridged traffic

Switch Switch

P
P
P P L3
7k1 7k2 ECMP
P Routing Protocol Peer

Dynamic Peering Relationship P P Router

P P Router
More details: BRKDCT-2048 - Deploying Virtual Port Channel in NX-OS
BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 147
 Layer 3 High Availability and Fast Convergence

Protocol Fast Convergence

Fast Failure Reaction with BGP Prefix Independent Convergence
 Before NX-OS 5.2, change in the reachability of the
BGP routes results in update of all the BGP prefixes
from RIB to FIB. This impacts convergence.
 From NX-OS 5.2:
 RIB/FIB/BGP support PIC Core
 Support for AFs: IPv4, VPNv4, VPNv6 (6VPE), 6PE
 Per-VRF label allocation required for PIC Core in MPLS
VPN scenario
 Virtual Objects (VOBJs) in FIB used to track Recursive Next
Hops (RNH) and associated VPN labels for BGP prefixes
 Whenever the IGP reachability to the RNH changes, single
update of VOBJ is done for all the BGP prefixes
 Result: significant convergence improvements
BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 148
 Layer 3 High Availability and Fast Convergence

BGP Prefix Independent Convergence

Without PIC Core
BGP Prfx4: 104.0.0.0/24
Supervisor
Prfx3: 103.0.0.0/24
IGP
Prfx2: 102.0.0.0/24
via 1.1.1.1 label 101 Prfx0: 1.1.1.1/32
Prfx1: 100.0.0.0/24
via 10.1.1.1
via 10.1.1.1
Prfx1: label 100 1
100.0.0.0/24
via 20.1.1.1
via via
20.1.1.1 label
1.1.1.1 100100
label 2

RIB
Prfx1: 100.0.0.0/24 Prfx0: 1.1.1.1/32
via 10.1.1.1 label 100 1 via 10.1.1.1 label 1
via 20.1.1.1 label 100 2 via 20.1.1.1 label 2

I/O Module
FIB
Prfx1: 100.0.0.0/24 Prfx2 Prfx3 Prfx0
via 10.1.1.1 label 100 1
via 20.1.1.1 label 100 2
via …
via …
via …
via … … via …
via …

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 150
 Layer 3 High Availability and Fast Convergence

BGP Prefix Independent Convergence

With PIC Core
BGP Prfx4: 104.0.0.0/24
Supervisor
Prfx3: 103.0.0.0/24
IGP
Prfx2: 102.0.0.0/24
via 1.1.1.1 label 101 Prfx0: 1.1.1.1/32
via 10.1.1.1
Prfx1: 100.0.0.0/24
via 20.1.1.1
via 1.1.1.1 label 100

RIB RNH Prfx0: 1.1.1.1/32

table via 10.1.1.1 label 1
Prfx: 100.0.0.0/24
via 1.1.1.1 label 100 via 20.1.1.1 label 2

I/O Module FIB

Prfx1
via … Prfx0
Prfx2
via …
Prfx3 VOBJ via …
via … [RNH, label] via …

BRKARC-3472 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 151