VXLAN Deployment Use Cases and Best

Practices
BRKDCT-1301

Azeem Suleman

Solutions Architect Cisco Advanced Services

Contributions
Thanks to the team:

Abhishek Saxena
Mehak Mahajan
Lilian Quan
Bradley Wong
Mike Herbert
Sandeep Subramaniam

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Abstract
With growing adoption of virtualization in customer environment and large
number of workload mobility requirements in data center; overlays are
becoming key technology. VXLAN is one the overlay technology.
This session will discuss about VXLAN design, deployment and best
practices in Data Center environment. It will also cover:

1) Explain VXLAN deployment steps on top design use cases

2) Configuration and key troubleshooting tips needed during deployment

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Agenda
Overlays
Introduction to VXLAN
VXLAN Design
Deployment Steps
Troubleshooting
Key Takeaways
References

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Overlays

Why Overlays?

Robust Underlay/Fabric

Flexible Overlay Virtual Network

High Capacity Resilient Fabric

Intelligent Packet Handling
Programmable & Manageable

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Mobility Track end-point attach at edges

Scale Reduce core state
Distribute and partition state to network
edge
Multi-tenancy Share Network resources
Flexibility/Programmability
Reduced number of touch points

Cisco Public

Overlay Taxonomy
Overlay Control Plane

Service = Virtual Network (VN)

Encapsulation

Edge Device

Edge Devices
Hosts
(end-points)

Underlay Network
Underlay Control Plane

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Types of Overlay Service

Layer 2 Overlays

Layer 3 Overlays

Emulate a LAN segment

Transport Ethernet Frames (IP and non-IP)
Single subnet mobility (L2 domain)
Exposure to open L2 flooding
Useful in emulating physical topologies

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Abstract IP based connectivity

Transport IP Packets
Full mobility regardless of subnets
Contain network related failures (floods)
Useful in abstracting connectivity and policy

Cisco Public

Types of Overlay Edge Devices

Network Overlays

Host Overlays

Integrated Overlays
Fabric DB

V
M
O
S

Physical

Physical

V
M
O
S

Virtual

V
M
O
S

V
M
O
S

A
p

A
p

p
O
S

p
O
S

Virtual

Virtual

Physical

Router/switch end-points

Virtual end-points only

Physical and Virtual

Protocols for resiliency/loops

Single admin domain

Resiliency + Scale

Traditional VPNs

VXLAN, NVGRE, STT

x-organizations/federation

OTV, VXLAN, VPLS, LISP

Open Standards

Tunnel End-points

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Introduction to VXLAN

VXLAN Overview
Challenges VXLAN addresses:

VLAN Scalability (4K) VXLAN extends the L2 Segment ID field to 24-bits,

potentially allowing up to 16 million unique L2 Segments over the same
network

VM mobility restricted within a VLAN VXLAN encapsulates L2 frame in IPUDP header allowing L2 adjacency across router boundaries

VXLAN Technology Overview:

MAC-in-UDP encapsulation

Leverages multicast in the transport network to simulate flooding behavior for

broadcast, unknown unicast and multicast in the layer 2 segment

Leverage ECMP to achieve optimal path usage over the transport network
BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VXLAN Overview
VXLAN can be implemented on both Hypervisor-based Virtual Switches to allow for scalable VM deployments,
as well as on Physical switches, which provides the ability to bridge VXLAN segments back into VLAN
segments. In these cases, the Physical Switch instantiates a VTEP, and function as a VXLAN Gateway

VNI 1000

VLAN 10

vSwitch
VTEP

Switch
VTEP
vSwitch
VTEP

VLAN 20

VNI 2000
BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Virtual eXtensible LAN (VXLAN)

Virtual eXtensible LAN (VXLAN) is a Layer 2 overlay scheme over a Layer 3

network.

A 24-bit VXLAN Segment ID or VXLAN Network Identifier (VNI) is included in

the encapsulation to provide up to 16M VXLAN segments for traffic isolation /
segmentation, in contrast to 4K segments achievable with VLANs.

Each of these segments represents a unique Layer 2 broadcast domain, and can be administered
in such a way that it can uniquely identify a given tenants address space or subnet.

Outer
Ethernet

Outer
IP

Outer
UDP

VXLAN

Ethernet
Header

Payload

FCS

Inner
Ethernet

Payload

New
FCS

8 Bytes
Flags

Reserved

1 Byte
Rsvd
BRKDCT-1301

Rsvd

Instance ID

Reserved

Outer UDP Destination Port = VXLAN (originally 8472, recently updated to 4789)
Outer UDP Source Port = Hash of Inner Frame Headers (optional)
2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VXLAN Frame Format

IP header, allowing
transport across any
IP network

Transport VLAN

Outer
MAC
DA

Outer
MAC
SA

Outer
802.1Q

Outer
IP DA

A Layer-2 Gateway
bridges traffic to VLAN
based on MAC DA
A Layer-3 Gateway
routes traffic to VLAN
based on IP DA

Allows for possible

16M segments

Outer
IP SA

Outer
UDP

VXLAN ID
(24 bits)

Inner Inner
MAC MAC
SA
DA

Identifies packet as a
VXLAN packet
BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Optional Inner Inner

Original IP
Inner IP DA IP SA
Payload
802.1Q

Original Ethernet Frame

Cisco Public

CRC

VXLAN Frame Format (Cont.)

Outer
IP Header

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

VNID

Reserved

32

Reserved

32

VXLAN
RRRR1RRR

16

Checksum
0x0000

Outer
Dst. IP

UDP
Length

Outer
Src. IP

72

8 Bytes

VXLAN
Port

Header
Checksum

16

FCS

8 Bytes

Protocol
0x11

16

Original
FCS L2 Frame

20 Bytes
IP Header
Misc Data

16

Ether
Type
0x0800

VLAN ID
Tag

Src.
MAC
Addr.

48

VLAN Type
0x8100

Dst.
MAC Addr.

14 Bytes
(4 bytes optional)

48

VXLAN
Header

UDP Header

UDP
Src. Port

Outer
Mac Header

16

16

16

16

24

24

Cisco Public

VXLAN Key Terminology

VTEP
NVE (Network Virtual Endpoint)
VNI (VXLAN Network Identifier or VXLAN Segment ID)
VXLAN Gateway
Transit
Remote VTEP
Delivery Group (DG)

BUM
BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VXLAN VTEP
VXLAN terminates its tunnels on VTEPs (Virtual Tunnel End Point).
Each VTEP has two interfaces - one to provide bridging function for local hosts, the
other has an IP identification in the core network for VxLAN encapsulation/deencapsulation.
Transport IP Network

VTEP

BRKDCT-1301

VTEP
IP Interface

IP Interface

Local LAN Segment

Local LAN Segment

End System

End System

End System
2014 Cisco and/or its affiliates. All rights reserved.

End System
Cisco Public

Handling of Multi-Destination Traffic

Since a control/signaling protocol has not been defined, emulation of Multi-Destination
traffic (Broadcast, Multicast, Unknown Unicast) is handled through the VXLAN IP underlay
through the use of segment control multicast groups
End
System

End
System

VTEP-3

VTEP 3
IP-3

Mcast Group
VTEP-1

VTEP-2

IP Network
End System A
MAC-A
IP-A
BRKDCT-1301

VTEP 1
IP-1

2014 Cisco and/or its affiliates. All rights reserved.

VTEP 2
IP-2

End System B
MAC-B
IP-B
Cisco Public

VXLAN Overview
VTEP Discovery & Address Learning
End System

End System

3
VTEP3

Outer S-IP: IP-1

Outer D-IP: 239.1.1.1

VTEP 3
IP-3

S-MAC: MAC-1
D-MAC:
00:01:5E:01:01:01

ARP Request for IP B

Src MAC: MAC-A
Dst MAC: FF:FF:FF:FF:FF:FF
MAC
Address
MAC-A

VxLAN ID

Remote VTEP

10

IP-1

UDP
VXLAN VNID: 10

7
ARP Response from IP
B
Src MAC: MAC-B
Dst MAC: MAC-A
End System A
MAC-A
IP-A
ARP Request for IP B
Src MAC: MAC-A
Dst MAC: FF:FF:FF:FF:FF:FF

BRKDCT-1301

ARP Request for IP B

Src MAC: MAC-A
Dst MAC:
FF:FF:FF:FF:FF:FF

VTEP 1
IP-1
MAC-1
MAC

6
VxLAN ID

Remote VTEP

10

IP-2

Address
MAC-B

VTEP 2
IP-2
MAC-2

Mcast Group
2 239.1.1.1

VTEP-1

ARP Response from IP

B
Src MAC: MAC-B
Dst MAC: MAC-A

VTEP-2

S-MAC: MAC-2
D-MAC: MAC-1
Outer S-IP: IP-2
Outer D-IP: IP-1
UDP
VXLAN VNID: 10

ARP Response
from IP B
Src MAC: MAC-B
Dst MAC: MAC 2014 Cisco and/or its affiliates. All rights reserved.
A

End System B
MAC-B
IP-B

ARP Request for IP B

Src MAC: MAC-A
Dst MAC: FF:FF:FF:FF:FF:FF
MAC
Address

VxLAN ID

Remote VTEP

10

IP-1

MAC-A

Cisco Public

VXLAN Overview
Unicast Forwarding Packet Flow
Outer S-MAC: MAC-3
Outer D-MAC: MAC-4

Outer S-MAC: MAC-1

Outer D-MAC: MAC-2

Routed Based on
Outer IP header

Outer S-IP: IP-1

Outer D-IP: IP-4
UDP

Router-1

S-MAC: MAC-A
D-MAC: MAC-B

VTEP-1

S-MAC: MAC-A
D-MAC: MAC-B
S-IP: IP-A
D-IP: IP-B

UDP
VXLAN VNID: 10

IP Network

VXLAN VNID: 10

S-IP: IP-A
D-IP: IP-B

Outer S-IP: IP-1

Outer D-IP: IP-4

MAC-2
IP-2:
165.123.1.2

S-MAC: MAC-A
D-MAC: MAC-B

Router-2
MAC-3
IP-3:
140.123.1.2

MAC-1
IP-1:
165.123.1.1

S-IP: IP-A
D-IP: IP-B

4
MAC-4
IP-4:
140.123.1.1

S-IP: IP-A
D-IP: IP-B
Host-A

MAC-A
IP-A:
10.1.1.100
BRKDCT-1301

VTEP-2

S-MAC: MAC-A
D-MAC: MAC-B

Host-B

MAC-B
IP-B:
10.1.1.101

VXLAN VNID 10 (Tenant Blue)

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Nexus 9000 Series

VXLAN Support
VXLAN is supported across the Nexus 9000 series platforms. The VXLAN
Gateway functionality is supported across all form factors and line cards.
Integrated routing functionality is only supported on ACI-enabled Modules

Nexus 9300 Series

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Nexus 9500 Series

Cisco Public

Supported Platforms
Platform

NX-OS version

Nexus 9500
Nexus 9300
Nexus 3100 (3132/3172)
Nexus 6000
Nexus 7000 with F3

Minimum

Recommended

6.1.2I3.1.x
6.1.2I2.1.x
6.x
7.0(0)N1x)
7.0.x

6.1.2.I3.1.x
6.1.2.I2.1.x
Q2 CY14
Q3 CY14
Q4 CY14

* There is no licensing cost for VXLAN Enhance Layer 3

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Nexus 9000 Series

VXLAN Gateway
VXLAN gateway bridges traffic between VXLAN segment and another
physical / logical layer 2 domain (such as a VLAN)
VLAN ID

VXLAN ID

10

1010

20

1020

VxLAN VTEP

VLAN 10

VLAN 20

(VxLAN Gateway)

L3
Network
VxLAN
VTEP

VxLAN
VTEP

VNI
1010

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

VNI
1020

Cisco Public

Nexus 9000 Series

VXLAN Gateway
The Nexus 9000 series supports VXLAN Gateway function, allowing VLANs to be
bridged/mapped to VXLAN Segments and vice versa
feature nv overlay
feature vn-segment-vlan-based

VTEP

interface et4/13
switchport
switchport access vlan 10
no shut
interface nve1
no shutdown
source-interface loopback0
overlay-encapsulation vxlan
member vni 1010 mcast-group 230.1.1.1

VXLAN Forwarding Table

MAC Address

VxLAN ID

Remote
VTEP

AA:AA:AA:AA:AA:AA

1010

10.1.1.2

BB:BB:BB:BB:BB:BB

1020

10.1.1.3

VLAN to VXLAN Mapping

VLAN ID

vlan 10
vn-segment 1010
switch# show nve vni
Interface
VNI
Multicast-group
VNI State
--------------------------------------------nve1
1010
230.1.1.1
up
switch# show nve peers
Interface
Peer-IP
VNI
Up Time
----------------------------------------nve1
10.1.1.2
1010
00:52:24
switch#
BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

VXLAN Encap

VXLAN ID

10

1010

20

1020

Ethernet/ 802.1Q

Cisco Public

Nexus 9000 Series

VXLAN Bridging
VXLAN Bridging bridges traffic between VXLAN segments
VLAN ID

VXLAN ID

10

1010

20

1020

VxLAN VTEP

VLAN 10

VLAN 20

(VxLAN Bridging)

L3
Network
VxLAN
VTEP

VxLAN
VTEP

VNI
1010

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

VNI
1020

Cisco Public

VXLAN Forwarding in 9300

VXLAN Bridging and Gateway

VXLAN

VLAN

VXLAN Encapsulation and Deencapsulation occur on NFE

VLAN -> VXLAN (gateway)

VXLAN - VLAN (gateway)

VXLAN VXLAN (bridging)

ALE

Bridging and Gateway are independent of

the port type (1/10/40G ports)

NFE
T2-96
Encap/Decap
ASIC

VLAN

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

VXLAN

Cisco Public

Bridging and
Gateway
mode
support
Any to Any
Behavior

VXLAN Forwarding Nexus 9500

VXLAN Bridging and Gateway

VXLAN Encapsulation and De-encapsulation occur on NFE

VLAN -> VXLAN (gateway)

VXLAN - VLAN (gateway)

VXLAN VXLAN (bridging)

Fabric 1
NFE
NFE

Fabric 2
NFE
NFE

Fabric 3
NFE
NFE

Fabric 4
NFE
NFE

Fabric 5
NFE
NFE

Fabric 6
NFE
NFE

ALE

ALE

ALE

ALE

ALE

ALE

NFE
ASIC

T2
Decap

NFE
ASIC

Encap

T2

NFE
ASIC

Decap
T2

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Nexus 9000 Series

VXLAN Routed Mode
VXLAN routed mode routes traffic between VXLAN segments and
between VXLAN another physical / logical layer 2 domain (such as a
VLAN)
VxLAN VTEP
(VxLAN Routed
Mode)

VLAN 20

VLAN 10

L3
Network
VxLAN
VTEP

VxLAN
VTEP

VNI
1010

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

VNI
1020

Cisco Public

VXLAN Forwarding Nexus 9000

VXLAN Routing
VXLAN Routing is not supported currently on Packet Processor
Additional recirculation required for VXLAN routing through Extended
Packet Processor
ALE

ALE

Insieme
Recirculate

Insieme
Recirculate

NFE

Route
Packet

VLAN
Subnet 10.20.20.0/24
BRKDCT-1301

NFE
Route
Packet

Encap/
Decap

VXLAN
Subnet 10.10.10.0/24

VLAN
Subnet 10.20.20.0/24

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Encap/
Decap

VXLAN
Subnet 10.10.10.0/24

VXLAN Forwarding Nexus 9500

VXLAN Routing
VXLAN Encapsulation and De-encapsulation occur on NFE
Decap on ingress line card, Encap on egress
Routing occurs on the ingress NFE and if required on fabric NFE
Routed VXLAN is recirculated by ALE ASIC
Fabric 1
NFE
NFE

Fabric 3
T2
T2

Fabric 2
NFE
NFE

LPM IP Route

ALE

IP

ASIC
Route

VLAN
BRKDCT-1301

ALE

NFE

IP
Route

ALE

VXLAN
Route

Encap

Decap

VXLAN

Fabric 6
Fabric 5
NFE
NFE LPM
T2 IP Route T2

Fabric 4
NFE
NFE

VXLAN

2014 Cisco and/or its affiliates. All rights reserved.

ALE

IP
Route

NFE

VLAN

VXLAN
Route

Decap
VLAN

Cisco Public

If required LPM
route lookup on
Fabric

/32 route
lookup on
ingress T2

VXLAN Design

VXLAN Designs
Explore a variety of VXLAN designs and evaluate how they meet key design
criteria
Assumption is the choice to go Overlay/VXLAN has already been made
Objective is not to debate "Layer 2 vs. Layer 3" or "why overlay in the Data Center?"

Introduce concepts / design building-blocks to help you build a design that

meets your requirements

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VXLAN Designs
High-level design options considered in this presentation are in the following areas:
Routed Access + IP Mobility
L2 extension across Pod / Multi-tenancy
Datacenter Interconnect (DCI)

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Classic Design
Simplest design option
Extension of traditional aggregation/access designs in
spine / leaf

L3

SVIs

SVIs

Immediate benefits:
L2/L3 boundary
Simplified configuration
Extension of VLANS
Traffic distribution using ECMP

IP Network

IP Mobility

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VXLAN Gateway Routed Access + IP Mobility

VXLAN Gateway defined at access layer

(leaf) Nexus 9000

Multicast needs to be enabled for VXLAN

to work on the source interface

Next hop of VTEP needs to be Layer 3

vPC needs peer gateway

Only 1:1 mapping is allowed for VXLAN

to VLAN

VTE
P

VTEP

VXLAN
Enabled
Hypervisor

VXLAN
Enabled
Hypervisor

Recommended N9K to be configured as

STP root switch in each L2 network

Link discovery protocols like CDP, LLDP

will not discover neighbors on the remote
VTEPs

Virtual to physical migration (P2V)

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VXLAN Forwarding
vPC

When vPC is enabled an anycast VTEP

address is programmed on both vPC
peers

Symmetrical forwarding behavior on both

peers provides

Multicast topology prevents BUM traffic

being sent to the same IP address across
the L3 network (prevents duplication of
flooded packets)

vPC peer-gateway feature must be

enabled on both peers

VXLAN header is not carried on the vPC

Peer link (MCT link)
BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

VXLAN
AnyCast
VTEP

AnyCast
VTEP

VLAN

Cisco Public

VXLAN Forwarding
Design Considerations

VXLAN
VTEP

VLAN

When VXLAN is being routed the next hop for VXLAN

encapsulated frames needs to be over an L3 interface

VXLAN VTEP downstream

of a Nexus 2000 FEX is not
supported

Alternatively, all SVIs from a VXLAN Gateway must point

to the same physical next hop
[same VXLAN header MAC DA for all VXLAN encapsulated
packets sent from the same physical port]
BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VXLAN Design with VXLAN Bridging only

L2 Extension across Pods
L2 Link
L3 Link

L3 Core
VTEP
(Layer-2 only)

VTEP
(Layer-2 only)
VXLAN Overlay
(VLAN Extension)

Pod 2

Pod 1
IP GW

IP GW
Layer-2 VLAN Domain

Layer-2 VLAN Domain

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VXLAN Design with VXLAN Bridging only

STP/VPC Replacement - Routing off-box
L2 Link
L3 Link

Nexus

VXLAN Overlay
Routing Between VXLAN
VLAN Subnets

(Bridging)

VTEP
(Layer-2 only)
VTEP
(Layer-2 only)

VTEP
(Layer-2 only)

VTEP
(Layer-2 only)

IP GW for ALL VXLAN VLANs

(HSRP, VRRP)

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VXLAN Design with VXLAN Bridging only

Spine-Leaf Deployment

Border Routing Leaf

VXLAN Routing
IP Gateway for Tenants
subnets
WAN Routing

Spine

VXLAN Overlay
(Routing)
(Bridging)

Leaf
VTEP
(Layer-2 only)

VTEP
(Layer-2 only)

VTEP
(Layer-2 only)

VTEP
(Layer-2 only)

VXLAN Bridging only

IP GW for VXLAN VLANs
(HSRP, VRRP)

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VXLAN Design with VXLAN Bridging + Routing

STP/VPC Replacement
Option A: Centralized IP Gateway and Inter-VXLAN Routing
L2 Link
L3 Link

Nexus

VXLAN Overlay
Routing Between VXLAN
Extended VLANs

(Bridging)

VTEP
(Layer-2 only)

VTEP
(Layer-2 only)

VTEP
(Bridging
+
Routing)

VTEP
(Layer-2 only)

IP GW for All VXLAN VLANs

(HSRP, VRRP)

VLAN BLUE
VLAN GREEN
VLAN GREEN
VLAN BLUE

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VXLAN Design with VXLAN Bridging + Routing

STP/VPC Replacement
Option B: Distributed IP Gateway and Inter-VXLAN Routing
L2 Link
L3 Link

Nexus

VXLAN Overlay
Routing Between
VLAN A & B Subnets
(Bridging)

VTEP
(Layer-2 only for VLAN A & B))

VTEP
(IP GW for VLAN A
Learn other VLAN/VXLAN subnets)

VTEP
(IP GW for VLAN B
Learn other VLAN/VXLAN subnets)

VXLAN Extended VLANs: VLAN A, B

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VXLAN Design with VXLAN Bridging + Routing

Spine-Leaf Deployment
Option A: Centralized IP Gateway and Inter-VXLAN Routing
Border Routing Leaf
VXLAN Routing
IP Gateway for Tenants
subnets
WAN Routing

Spine

VXLAN Overlay
(Routing)

(Bridging)

Leaf
VTEP
(Layer-2 only)

VTEP
(Layer-2 only)

VTEP
(Layer-2 only)

VTEP
(Layer-2 only)

VTEP for all Extended VLANs

IP GW for all Extended VLANs
(HSRP, VRRP)

VXLAN Bridging only

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VXLAN Design with VXLAN Bridging + Routing

Spine-Leaf Deployment
Option B: Distributed IP Gateway and Inter-VXLAN Routing

Spine

VXLAN Overlay
(Routing)

(Bridging)

Leaf

VTEP
(IP GW for VLAN A
L2 for Other VLANs)

VTEP
(IP GW for VLAN B
L2 for Other VLANs)

VTEP
(IP GW for VLAN C
L2 for Other VLANs)

VTEP
(IP GW for VLAN D
L2 for Other VLANs)

VXLAN Extended VLANs: VLAN A, B, C, D

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Border Routing Leaf

For WAN Routing
Not a VTEP

Cisco Public

VXLAN Design with VXLAN Bridging + Routing +

Anycast IP GW - STP/VPC Replacement
L2 Link
L3 Link

Nexus

Routing to Outside

VXLAN Overlay
Routing Between
VXLAN VLAN Subnets
(Bridging)

VTEP
(IP GW for All VXLAN VLANs)
VTEP
(IP GW for All VXLAN VLANs)

VTEP
(IP GW for All VXLAN VLANs)

VXLAN Extended VLANs: VLAN A, B

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VXLAN Design with VXLAN Bridging + Routing +

Anycast IP Gateway - Spine-Leaf Deployment

Spine

VXLAN Overlay
Leaf

(Bridging)

Routing to Outside

Routing Between
VXLAN VLAN Subnets

VTEPs
(IP GW for All VXLAN VLANs)

Border Routing Leaf

For WAN Routing
Not a VTEP

VXLAN Extended VLANs: VLAN A, B, C, D

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VXLAN Unsupported Bud-node Topology

This is unsupported Bud-node topology
HOST

IP
Transport

N9000
VTEP

VTEP

VTEP

HOST

HOST

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VXLAN Supported and Unsupported FEX Topology

N9000
VTEP

N9000
VTEP

FEX

FE
X

Hypervisor

Hypervisor

Hypervisor
VTEP

Hypervisor

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

N9000
VTEP

N9000
VTEP

FEX

FE
X

Hypervisor
VTEP

VXLAN host behind FEX is not supported.

Cisco Public

Hypervisor
VTEP

VXLAN Deployment Steps

VXLAN Configuration
Enabling VXLAN feature
switch(config)# [no] feature nv overlay

Enable VLAN model of VXLAN bridging

switch(config)# [no] feature vn-segment-vlan-based

Enable Multicast
switch(config)# [no] feature pim

Make sure source interface and next hop L3 interface should have multicast enabled
switch(config)# interface loopback 0
switch(config-if)# ip pim sparse-mode

Creating NVE (Overlay) interface

switch(config)# interface nve <x>
switch(config-if-nve)# source-interface <src-if>
switch(config-if-nve)# overlay-encapsulation <vxlan|nvgre>
switch(config-if-nve-si)# member vni [<range>| auto] [mcast-group <start addr> [<end addr>]]

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VXLAN Configuration
Default UDP Port is 4789
Configure VXLAN UDP Port
switch(config)# [no] vxlan udp port <number>

Creating VXLAN VNIs (optional)

switch(config)# [no] vni <range>

VLAN to VXLAM Mapping configuration

switch(config)# vlan <number>
switch(config-vlan)# vn-segment <vnid>

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VXLAN vPC Configuration

Loopback Interface Configuration
interface loopback0
no ip redirects
ip address 1.1.1.1/24
ip address 1.1.1.3/24 Secondary
ip pim sparse-mode

The secondary ip address is used by the emulated VTEP for VXLAN

Just make sure all the configs are identical between vPC Primary and vPC Secondary

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VXLAN Sample Topology - VTEP

L3 Transport Network
(OSPF and IP PIM)
Mcast grp:
230.1.1.1
RP: 10.1.1.1
.2

Lpbk0:
100.100.100.1/32

.1
e2/1

.2
20.1.1.0/30

Nexus 9000 VTEP-1

30.1.1.0/30

.1
e2/1

Nexus 9000 VTEP-2

e1/1

e1/1

Host A
(VLAN 10)

BRKDCT-1301

Host B
(VLAN 10)

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Lpbk0:
100.100.100.2/32

n9k-vtep-1(config)# feature ospf

n9k-vtep-1(config)# feature pim
n9k-vtep-1(config)# router ospf 1
n9k-vtep-1(config-router)# router-id 100.100.100.1
n9k-vtep-1(config)# ip pim rp-address 10.1.1.1 group-list 224.0.0.0/4
n9k-vtep-1(config)# interface loopback0
n9k-vtep-1(config-if)# ip address 100.100.100.1/32
Nexus 9000 VTEP-1
n9k-vtep-1(config-if)# ip router ospf 1 area 0.0.0.0
n9k-vtep-1(config-if)# ip pim sparse-mode
Host A
n9k-vtep-1(config)# interface e2/1
(VLAN 10)
n9k-vtep-1(config-if)# ip address 20.1.1.1/30
n9k-vtep-1(config-if)# ip router ospf 1 area 0.0.0.0
n9k-vtep-1(config-if)# ip pim sparse-mode

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

L3 Transport Network
(OSPF and IP PIM)
Multicast group:
230.1.1.1
RP: 10.1.1.1

Nexus 9000 VTEP-2

Host B
(VLAN 10)

n9k-vtep-1(config)# feature nv overlay

n9k-vtep-1(config)# feature vn-segment-vlan-based
n9k-vtep-1(config)# interface e1/1
n9k-vtep-1(config-if)# switchport
n9k-vtep-1(config-if)# switchport access vlan 10
n9k-vtep-1(config-if)# no shutdown
n9k-vtep-1(config)# interface nve1
n9k-vtep-1(config-if)# no shutdown
n9k-vtep-1(config-if)# source-interface loopback0
n9k-vtep-1(config-if)# overlay-encapsulation vxlan
n9k-vtep-1(config-if)# member vni 1010 mcast-group 230.1.1.1
n9k-vtep-1(config)# vlan 10
n9k-vtep-1(config-vlan)# vn-segment 1010

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

L3 Transport Network
(OSPF and IP PIM)
Multicast group:
230.1.1.1
RP: 10.1.1.1

Nexus 9000 VTEP-1

Host A
(VLAN 10)

Cisco Public

Nexus 9000 VTEP-2

Host B
(VLAN 10)

n9k-vtep-2(config)# feature ospf

n9k-vtep-2(config)# feature pim
n9k-vtep-2(config)# router ospf 1
n9k-vtep-2(config-router)# router-id 100.100.100.2
n9k-vtep-2(config)# ip pim rp-address 10.1.1.1 group-list 224.0.0.0/4
n9k-vtep-2(config)# interface loopback0
n9k-vtep-2(config-if)# ip address 100.100.100.2/32
Nexus 9000 VTEP-1
n9k-vtep-2(config-if)# ip router ospf 1 area 0.0.0.0
n9k-vtep-2(config-if)# ip pim sparse-mode
Host A
n9k-vtep-2(config)# interface e2/1
(VLAN 10)
n9k-vtep-2(config-if)# ip address 30.1.1.1/30
n9k-vtep-2(config-if)# ip router ospf 1 area 0.0.0.0
n9k-vtep-2(config-if)# ip pim sparse-mode

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

L3 Transport Network
(OSPF and IP PIM)
Multicast group:
230.1.1.1
RP: 10.1.1.1

Nexus 9000 VTEP-2

Host B
(VLAN 10)

n9k-vtep-2(config)# feature nv overlay

n9k-vtep-2(config)# feature vn-segment-vlan-based
n9k-vtep-2(config)# interface e1/1
n9k-vtep-2(config-if)# switchport
n9k-vtep-2(config-if)# switchport access vlan 10
n9k-vtep-2(config-if)# no shutdown
n9k-vtep-2(config)# interface nve1
n9k-vtep-2(config-if)# no shutdown
n9k-vtep-2(config-if)# source-interface loopback0
n9k-vtep-2(config-if)# overlay-encapsulation vxlan
n9k-vtep-2(config-if)# member vni 1010 mcast-group 230.1.1.1
n9k-vtep-2(config)# vlan 10
n9k-vtep-2(config-vlan)# vn-segment 1010

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

L3 Transport Network
(OSPF and IP PIM)
Multicast group:
230.1.1.1
RP: 10.1.1.1

Nexus 9000 VTEP-1

Host A
(VLAN 10)

Cisco Public

Nexus 9000 VTEP-2

Host B
(VLAN 10)

VXLAN Sample topology with vPC

L3 Transport Network
(OSPF and IP PIM)
Multicast group:
230.1.1.1
20.1.1.0/30
.1
e2/1

.2

.6
20.1.1.4/30

Nexus 9000 VTEP-2

e1/1

e1/1
vPC

Loop back 0:
200.200.200.2/32
100.100.100.1/32 (Secondary)

Host A
(VLAN 10)

BRKDCT-1301

.2

.5

Nexus 9000 VTEP-1

Loop back 0:
200.200.200.1/32
100.100.100.1/32 (Secondary)

30.1.1.0/30

2014 Cisco and/or its affiliates. All rights reserved.

.1
e2/1

Nexus 9000 VTEP-3

e1/1
Host B
(VLAN 10)

Cisco Public

Loop backk0:
100.100.100.2/32

n9k-vtep-1(config)# feature ospf

n9k-vtep-1(config)# feature pim
n9k-vtep-1(config)# router ospf 1
n9k-vtep-1(config-router)# router-id 200.200.200.1
n9k-vtep-1(config)# ip pim rp-address 10.1.1.1 group-list 224.0.0.0/4
n9k-vtep-1(config)# interface loopback0
Nexus 9000 VTEP-1
n9k-vtep-1(config-if)# ip address 200.200.200.1/32
n9k-vtep-1(config-if)# ip address 100.100.100.1/32 secondary
n9k-vtep-1(config-if)# ip router ospf 1 area 0.0.0.0
n9k-vtep-1(config-if)# ip pim sparse-mode
n9k-vtep-1(config)# interface e2/1
n9k-vtep-1(config-if)# ip address 20.1.1.1/30
n9k-vtep-1(config-if)# ip router ospf 1 area 0.0.0.0
n9k-vtep-1(config-if)# ip pim sparse-mode

L3 Transport Network
(OSPF and IP PIM)
Multicast group:
230.1.1.1

Nexus 9000 VTEP-2

vPC
Host A
(VLAN 10)

n9k-vtep-1(config)# feature nv overlay

n9k-vtep-1(config)# feature vn-segment-vlan-based
BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Nexus 9000 VTEP-3

Host B
(VLAN 10)

n9k-vtep-1(config)# interface port-channel 10

n9k-vtep-1(config-if)# vpc 10
n9k-vtep-1(config-if)# switchport
n9k-vtep-1(config-if)# switchport mode access
n9k-vtep-1(config-if)# switchport access vlan 10
n9k-vtep-1(config-if)# no shutdown
n9k-vtep-1(config)# interface e1/1
n9k-vtep-1(config-if)# channel-group 10 mode active
n9k-vtep-1(config-if)# no shutdown

L3 Transport Network
(OSPF and IP PIM)
Multicast group:
230.1.1.1

Nexus 9000 VTEP-1

Nexus 9000 VTEP-2

vPC
Host A
(VLAN 10)

n9k-vtep-1(config)# interface nve1

n9k-vtep-1(config-if)# no shutdown
n9k-vtep-1(config-if)# source-interface loopback0
n9k-vtep-1(config-if)# overlay-encapsulation vxlan
n9k-vtep-1(config-if)# member vni 1010 mcast-group 230.1.1.1
n9k-vtep-1(config)# vlan 10
n9k-vtep-1(config-vlan)# vn-segment 1010

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Nexus 9000 VTEP-3

Host B
(VLAN 10)

n9k-vtep-2(config)# feature ospf

n9k-vtep-2(config)# feature pim
n9k-vtep-2(config)# router ospf 1
n9k-vtep-2(config-router)# router-id 200.200.200.2
n9k-vtep-2(config)# ip pim rp-address 10.1.1.1 group-list 224.0.0.0/4
n9k-vtep-2(config)# interface loopback0
Nexus 9000 VTEP-1
n9k-vtep-2(config-if)# ip address 200.200.200.2/32
n9k-vtep-2(config-if)# ip address 100.100.100.1/32 secondary
n9k-vtep-2(config-if)# ip router ospf 1 area 0.0.0.0
n9k-vtep-2(config-if)# ip pim sparse-mode
n9k-vtep-2(config)# interface e2/1
n9k-vtep-2(config-if)# ip address 20.1.1.5/30
n9k-vtep-2(config-if)# ip router ospf 1 area 0.0.0.0
n9k-vtep-2(config-if)# ip pim sparse-mode

L3 Transport Network
(OSPF and IP PIM)
Multicast group:
230.1.1.1

Nexus 9000 VTEP-2

vPC
Host A
(VLAN 10)

n9k-vtep-2(config)# feature nv overlay

n9k-vtep-2(config)# feature vn-segment-vlan-based
BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Nexus 9000 VTEP-3

Host B
(VLAN 10)

n9k-vtep-2(config)# interface port-channel 10

n9k-vtep-2(config-if)# vpc 10
n9k-vtep-2(config-if)# switchport
n9k-vtep-2(config-if)# switchport mode access
n9k-vtep-2(config-if)# switchport access vlan 10
n9k-vtep-2(config-if)# no shutdown
n9k-vtep-2(config)# interface e1/1
n9k-vtep-2(config-if)# channel-group 10 mode active
n9k-vtep-2(config-if)# no shutdown

L3 Transport Network
(OSPF and IP PIM)
Multicast group:
230.1.1.1

Nexus 9000 VTEP-1

n9k-vtep-2(config)# interface nve1

n9k-vtep-2(config-if)# no shutdown
n9k-vtep-2(config-if)# source-interface loopback0
n9k-vtep-2(config-if)# overlay-encapsulation vxlan
n9k-vtep-2(config-if)# member vni 1010 mcast-group 230.1.1.1
n9k-vtep-2(config)# vlan 10
n9k-vtep-2(config-vlan)# vn-segment 1010

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Nexus 9000 VTEP-2

vPC
Host A
(VLAN 10)

Cisco Public

Nexus 9000 VTEP-3

Host B
(VLAN 10)

n9k-vtep-3(config)# feature nv overlay

n9k-vtep-3(config)# feature vn-segment-vlan-based
n9k-vtep-3(config)# interface e1/1
n9k-vtep-3(config-if)# switchport
n9k-vtep-3(config-if)# switchport access vlan 10
n9k-vtep-3(config-if)# no shutdown
Nexus 9000 VTEP-1
n9k-vtep-3(config)# interface nve1
n9k-vtep-3(config-if)# no shutdown
n9k-vtep-3(config-if)# source-interface loopback0
n9k-vtep-3(config-if)# overlay-encapsulation vxlan
n9k-vtep-3(config-if)# member vni 1010 mcast-group 230.1.1.1
n9k-vtep-3(config)# vlan 10
n9k-vtep-3(config-vlan)# vn-segment 1010

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

L3 Transport Network
(OSPF and IP PIM)
Multicast group:
230.1.1.1

Nexus 9000 VTEP-2

vPC
Host A
(VLAN 10)

Cisco Public

Nexus 9000 VTEP-3

Host B
(VLAN 10)

VXLAN Troubleshooting

Troubleshooting
NVE Interface State

NVE source interface is configured

The source interface has IPv4 address is configured
Source interface is OPER up
NVE interface is admin up and has multicast group associated to it.

Restrictions
Only 1 NVE local interface is supported
VXLAN gateway shouldnt be connected to multiple core routers via a layer 2 switch
Support for downstream multicast receivers

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Troubleshooting
Verify Overlay Peer
switch# show nve peers
Interface
Peer-IP
---------------------------nve1
10.1.1.2

VNI
-------10000

Up Time
------00:52:24

switch# show nve peers interface nve1

Interface
Peer-IP
---------------------------nve1
10.1.1.2

VNI
-------10000

Up Time
------00:52:24

switch# show nve peers vni 1010

Interface
Peer-IP
---------------------------nve1
10.1.1.2

VNI
-------10000

Up Time
------00:52:24

It displays VTEP peers and the

associated VNI with them. The output
displays time since the peer was last
detected / learnt.

List of all VNIs that are associated with

various nve interfaces and the associated
multicast IP address.

Verify Overlay interface

switch# show nve vni
Interface
VNI
----------------------nve1
10000

BRKDCT-1301

Multicast-group
--------------230.1.1.1

VNI State
--------up

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Troubleshooting (Cont.)
Verify MAC address is learnt

switch# show mac address-table

Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O Overlay MAC
age - seconds since last seen,+ - primary entry using vPC
Peer-Link,
(T) - True, (F) - False
VLAN
MAC Address
Type
age
Secure NTFY Ports
---------+-----------------+--------+---------+------+----+----------------* 100
0000.aa01.0001
dynamic 0
F
F
nve1
* 100
0000.bb01.0001
dynamic 0
F
F
nve1
* 100
0000.bb01.0001
dynamic 0
F
F
Eth1/46

List of all VNIs that are associated with

various nve interfaces and the associated
multicast IP address.

Few Other Commands

switch# show logging level nve
switch# show tech-support nve
switch# show runn interface nve 1

switch# show nve interface

switch# show nve vxlan-params
BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Troubleshooting (Cont.)
Few Debug commands
switch# debug nve errors
switch# debug nve events
switch# debug nve all
switch# debug nve pim-library

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VXLAN Control Plane Alternatives Future

Challenges:

Data Center Core may not support multicast.

Scale challenges with multicast flooding.
VXLAN Membership / Auto-discovery of VTEP.
Support of Gateway multi-homing.
VNI to IP multicast group mapping has to be managed.

Alternatives:
BGP Control Plane
LISP Control Plane
API

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Key Takeaways VXLAN Technology

VXLAN is simple
Keeps the attractive aspects of Layer 2 No re-addressing, simple configuration and
deployment
Integrates stability and scale of Layer 3

VXLAN is efficient
Proper utilization of ECMP
Optimal path between any two nodes

VXLAN is scalable
Can extend a bridged domain without extending the risks generally associated with
Layer 2 and beyond 4K VLAN limit

VXLAN Control Plane (Future)

BGP and LISP

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Key Takeaways VXLAN Design

You can deploy VXLAN today, with traditional network designs
VXLAN introduces immediate, tangible benefits to any design:
Simple configuration, leverage parallel network paths, extend VLANs safely etc.

Provides multiple design options to help you build a network that meets your
requirements

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

References
VXLAN Overview: Cisco Nexus 9000 Series Switches
http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps13386/whitepaper-c11-729383_ns1261_Networking_Solutions_White_Paper.html
* Cisco Nexus 9000 Series NX-OS VXLAN Configuration Guide
VXLAN: A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3
Networks
http://www.ietf.org/id/draft-mahalingam-dutt-dcops-vxlan-07.txt

* Will be available at the time of FCS of the software code

BRKDCT-1301

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Call to Action
Visit the World of Solutions: Cisco Campus
Walk-in Labs
Technical Solutions Clinics
Meet the Engineer

Lunch Time Table Topics, held in the main Catering Hall

Recommended Reading: For reading material and further resources for this
session, please visit www.pearson-books.com/CLMilan2014
Presentation_ID

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

74

Complete Your Online Session Evaluation

Complete your online session
evaluation
Complete four session evaluations
and the overall conference evaluation
to receive your Cisco Live T-shirt

Presentation_ID

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

75