0% found this document useful (0 votes)

1K views

Data Center Design Power Session

Cisco Public Infrastructure Design LAN Switching Analysis Recap on Current Trends New Layer 2 Technologies Fabric Extender Deep dive and Design with virtual Port Channeling. Cisco Public Data Center Layout Better to move the Horizontal Distribution closer to the servers to reduce the cable length.

Uploaded by

Jarod Ahlgren

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

1K views

Data Center Design Power Session

Uploaded by

Jarod Ahlgren

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

You are on page 1/ 192

Data Center Power Session

TECDCT-3873

Presentation_ID

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Agenda
Infrastructure Design LAN Switching Analysis
Recap on Current Trends New Layer 2 Technologies Fabric Extender Deep dive and Design with virtual Port Channeling

Break Demos: vPC Designs with Server Virtualization 10 Gigabit Ethernet to the Server Break Demo: Nexus1kv Blade Servers
Blade Switching LAN Blade Switching SAN Unified Compute System

Break Demo: UCS SAN Switching Analysis

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Infrastructure Design

Presentation_ID

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Data Center Layout

Better to move the Horizontal distribution closer to the servers to reduce the cable length

Main Distribution Area

Horizontal Distribution Area

Equipment Distribution Area

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Horizontal Distribution at Each Row (aka End of the Row Design)

From Direct connect to End of the Row

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Datacenter Building Block: the POD

HDA

Physical Pod

Defines a discrete amount of physical infrastructure

Racks + Power Distribution + CRAC

Pay-as-you-grow modularity - Predictable, Scalable & Flexible Pod server density affected by power & cooling, cabling & server connectivity
TECDCT-3873_c2 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Overall DC Layout

HDA MDA

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Mapping Between Physical Layout and Network Topology: HDA

Equipment Distribution Area (EDA) Single POD HDA

Acc1 Acc2

336 Servers

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Mapping Between Physical Layout and Network Topology: MDA

Core 1 Core 2

Additional Equipment: Core Routing\Firewalls LAN Appliances SAN Directors

Agg1

Agg2

Agg3

Agg4

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

10 Gigabit Ethernet
for Server Connectivity
Mid 1980s 10Mb
UTP Cat 3

Mid 1990s 100Mb

UTP Cat 5

Early 2000s 1Gb

UTP Cat 5 MMF, SMF

Late 2000s 10Gb

UTP Cat6a MMF, SMF TwinAx, CX4

10G Options
Connector (Media)
SFP+ CU*
copper

Cable
Twinax Twinax MM OM2 MM OM3 MM OM2 MM OM3
Cat6 Cat6a/7 Cat6a/7

Distance
<10m 15m 10m 100m 82m 300m
55m 100m 30m
** Draft 3.0, not final

In-rack X-rack Power

(each side)
~ 0.1W 4W 1W

Transceiver Latency (link)

~ 0.1s ~ 0.1s ~0 ~0
2.5s 2.5s 1.5s

Standard
SFF 8431** IEEE 802.3ak

X2 CX4
copper

SFP+ USR
MMF, ultra short reach

SFP+ SR
MMF,short reach

Across racks 1W
~ 6W*** ~ 6W*** ~ 4W***

~50% power none savings with EEE IEEE 802.3ae

RJ45 10GBASE-T
copper
* Terminated cable

IEEE 802.3an

*** As of 2008; expected to decrease over time

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Twisted Pair Cabling For 10GBASE-T (IEEE 802.3an)

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

10G Copper Infiniband 10GBase-CX4

IEEE 802.3ak Supports 10G up to 15 meters Quad 100 ohm twinax, Infiniband cable and connector Primarily for rack-to-rack links Low Latency

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

10G SPF+ Cu
SFF 8431 Supports 10GE passive direct attached up to 10 meters Active cable options to be available Twinax with direct attached SFP+ Primarily for in rack and rack-to-rack links Low Latency, low cost, low power

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

10GBase-*X (IEEE 802.3ae)

The 802.3ae 10GbE standard defines 3 MM and 1 SM fiber category based on the maximum transmission reach as shown below (ISO 11801 Standard defines the following MM and SM ( g fiber types):
REACH SPEED
300m 500m 200m

100Mb/s 1,000Mb/s 10Gb/s

OM1 OM1 OM3

OM1 OM2 OS1

OM1 OS1 OS1

OM1 is equivalent to standard 62.5/125m MM fiber OM2 is equivalent to standard 50/125m fiber. OM3 is laser enhanced 50/125m fiber 10gig OS1 is equivalent to SM 8/125m fiber.

Not all laser optimized 10Gig fiber cable is the same.

10Gig

150M OM2 Plus

300M OM3

550M OM3 Plus

Cisco Public

Optics Positioning for Data Centers

1G Optics Type
1000 BASE-LX 1000 BASE-SX 1000 BASE T BASE-T

Max PMD Distance (m) 10G Optics Type

10GBASE-LR 10GBASE-LRM

100

500

~10000

Require OM3 MMF

10GBASE-SR 10GBASE-T 10GBASE-USR 10GBASE-CX4 10GBASE-CX1

30M/100M OM3 MMF Only

Max PMD Distance (m)

26-82

100

220

300

~10000

In Rack X-rack
<10M
TECDCT-3873_c2 2009 Cisco Systems, Inc. All rights reserved.

Mid to End of Rack

<100 M
Cisco Public

Across Aisles
<300 M

Across Sites
<10 KM
15

Cost Effective 10G Server Connectivity Today

SFP+ USR Ultra Short Reach

100M on O OM3 fiber, 30M on OM2 fiber f O f Support on all Cisco Catalyst and Nexus switches Low Cost Target FCS: Q1 CY09

SFP+ Direct Attach

1, 3, 1 3 5 and 7M on Twinax 0.1W Power Support across all Nexus Switches Low Cost

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Cabling Infrastructure Patch Panels for End of the Row or Middle of the Row
Category 6A (Blue) with OM3 MM (Orange) per Rack, terminating in patch rack at EoR Cable count varies based on design requirement

Fiber for SAN or for TOR switches Copper for EoR server connectivity
TECDCT-3873_c2 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

HDA Photos for End or Middle of the Row

cables on the back go to the TOR patch panels

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

End of the Row Connectivity Details

End of Row End of Row
Traditionally used Copper from server to access switches
Patch P t h panel l X-connect Patch P t h panel l X-connect Patch panel server Patch panel server

Common Characteristics
Typically used for modular access Cabling is done at DC build-out Model evolving from EoR to MoR Lower cabling distances (lower cost) Allows denser access (better flexibility) 6-12 multi-RU servers per Rack 4-6 kW per server rack, 10Kw-20Kw per network rack Subnets and VLANs: one or many per switch. Subnets tend to be medium and large
TECDCT-3873_c2 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Network Access Point A-B

Network Access Point C-D server server

Fiber Copper

Middle of Row

Patch panel server Patch panel X-connect Patch panel X-connect

Patch panel server

Network Access Point A-B server

Network Access Point C-D server

Top of the Rack Connectivity Details

ToR
Used in conjunction with dense access racks(1U servers) Typically one access switch per rack Some customers are considering two + cluster Typically: ~10-15 server per rack (enterprises) ~15-30 server per rack (SP) Use of either side of rack is gaining traction Cabling: Within rack: Copper for server to access switch Outside rack (uplink): Copper (GE): needs a MoR model for fiber aggregation Fiber (GE or 10GE):is more flexible and also requires aggregation model (MoR)
TECDCT-3873_c2 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Patch panel Top of Rack p server Patch panel X-connect Patch panel X-connect

Patch panel Top of Rack server

server

Network Aggregation Point A-B

Network Aggregation Point A-B server

To network core

Patch panel Top of Rack Top of Rack server Patch panel X-connect Patch panel X-connect

Patch panel Top of Rack Top of Rack server

Network Access Point A-B server

Network Access Point C-D server

Blade Chassis Connectivity Details

End of Row (Switch to Switch)
Scales well for blade server racks ( (~3 blade chassis per rack) p ) Most current uplinks are copper but the NG switches will offer fiber
Patch panel Patch panel X-connect Patch panel X-connect Patch panel

sw1

sw2

sw1

sw2

Blade Chassis sw1 sw2 Network Aggregation Point ABC-D

Blade Chassis sw1 Network Aggregation Point AB-C-D sw2

Blade Chassis

Blade Chassis sw1 sw2

End of Row (Pass-through)

Scales well for pass-through blade racks Copper from servers to access switches

sw1

sw2

Blade Chassis

Patch panel

Patch panel Patch panel X-connect Top of Rack Patch panel X-connect Pass-through Blade Chassis Pass-through Network Aggregation Point ABC-D Network Aggregation Point AB-C-D Blade Chassis Pass-through Blade Chassis

ToR
Pass-through

Viiable option on pass-through environments is the access port count is right

Blade Chassis Pass-through Blade Chassis Pass-through Blade Chassis

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Final Result
12 Server PODs Consists of the following: 4 Switch Cabinets for LAN & SAN 32 S Server C bi t Cabinets 12 Servers per Server Cabinet
Core 1 Core 2

Servers: 4032 6509 Switches: 30 Server/Switch Cabinets: 399 Midrange/SAN Cabinets Allotted For: 124 Mid /SAN C bi t All tt d F

Agg1

Agg2

Agg3

Agg4

Acc1

Acc2
6 Pair Switches

Acc11

Acc12

Acc13

Acc14
6 Pair Switches

Acc23

Acc24

336 Servers
TECDCT-3873_c2 2009 Cisco Systems, Inc. All rights reserved.

336 Servers
Cisco Public

336 Servers

336 Servers 22

LAN Switching in the Datacenter

Presentation_ID

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Icons and Associated Product

=
Nexus 7000 Nexus 5000 Catalyst 4948-10GE Catalyst 4900M

with Service Modules

=
Nexus 1000v Catalyst 6500 with VSS = Nexus 2148T CBS 3100 Blade Switches

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

LAN Switching
Evolution of Data Center Architectures New Layer 2 Technologies Fabric Extender Deep dive and Design with virtual Port Channeling Break Demo: vPC Designs with Server Virtualization 10 Gigabit Ethernet to the Server Break Demo: Nexus1kv
TECDCT-3873_c2 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Data Center Architecture

Existing Layer 2 STP Design Best Practices
Rapid PVST+ UDLD Global Spanning Tree Pathcost Method=Long

L3+ L4 CEF Hash

LACP+L4 Port Hash Dist EtherChannel for FT and Data VLANs FT Data

Agg1: STP Primary Root HSRP Primary HSRP Preempt and Delay Dual Sup with NSF+SSO
LACP+L4 Hash Dist EtherChannel Min-Links

Agg2: STP Secondary Root HSRP Secondary HSRP Preempt and Delay Single Sup

Rootguard LoopGuard Portfast + BPDUguard

Blade Chassis with Integrated Switch

Rapid PVST+: Maximum Number of STP Active Logical Ports- 8000 and Virtual Ports Per Linecard-1500
TECDCT-3873_c2 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Migration from Inline Services

The Need: Higher performance/ scalability required in aggregation and/or core The Migration: Move Catalyst 6500 chassis with service modules to an on-the-stick configuration and re-use high speed links to connect to the aggregation Layer

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

VSS Allows a Migration to a L2 Topology Based on Etherchannels

IETF NSF 10 Gig uplinks

STP Root

6500 with VSS IETF NSF-capable 10 Gig uplinks

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

VSS Is Currently Being Used Also for Data Center Interconnect

nPE nPE

VSS system

Main benefits Loop Avoidance Load balancing Failover

Nexus-Based Datacenters High Density 10 Gigabit and Unified IO readiness

10G Core Performance
WAN
Nexus Core Layer Nexus Agg Layer Nexus Agg Layer

10G Aggregation Density

Core

Access 1G/10G to the Host

Core

Blade Servers Top of Rack

Nexus

Access Layer

High performance, Hi h f highly available 10GE core connectivity

Full F F ll Featured 10G d Density for aggregating 10G Top of Rack and 10G Blade Servers

As i A virtualization li i drives host I/O utilization, 10G to the host requirements are becoming reality

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

New Aggregation Layer Nexus 7k in the Aggregation

4*10GbE

8*10GbE 8 10GbE

4*10GbE

Optional dedicated links in case 6k is deployed as VSS

Nexus-based Aggregation Layer with VDCs, CTS and vPCs d PC Catalyst 6500 services chassis with Firewall Services and ACE Module provides Advanced Service delivery Possibility of converting the Catalyst 6500 in VSS mode

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

New Access Layer Options

New Options Include: Nexus 7k, FEX, 10 GigE TOR
1GbE End-of-Row

New Options highlighted in red 10 Gigabit Top of the Rack Connectivity with the Nexus 5k Fabric Extender (Nexus2k) Server Virtual Switching (Nexus1kv)
Nexus 7018 Catalyst 6500 1GbE Top-of-Rack Nexus 2148T Nexus 5000

10GbE End-of-Row

Nexus 7000 Nexus 1000v 10GbE Top-of-Rack Nexus 5000

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

New Layer 2 Technologies

Applicability Link Layer Encryption Virtual Domain Contexts vPC Nexus 7k and future on other Nexus platforms Nexus 7k Nexus 7k Nexus 5k Catalyst 6500 (as VSS) MAC pinning L2MP VNTAG Datacenter Ethernet OTV
TECDCT-3873_c2 2009 Cisco Systems, Inc. All rights reserved.

Fabric Extender (Nexus2148T) Nexus1kv Future on Nexus products Nexus5k, Nexus2k, SR-IOV 10 Gigabit Adapters Nexus 5k and future linecards, Converged Network Adapters Layer 2 extension
Cisco Public

Cisco TrustSec
TrustSec Linksec (802.1ae) Frame Format
The encryption used by TrustSec follows IEEE Standards-based LinkSec (802.1ae) encryption, where the upper layers are unaware of the L2 header/encryption.

CMD E_TYPE

Version

Length

SGT Option Length & Type

SGT Value

Variable

DMAC

SMAC

802.1ae Header (16 Octets)

.1Q (4)

CMD (8 Octets)

ETH_TYPE ETH TYPE

Payload P l d

ICV (16 Octets)

CRC

Encrypted Authenticated

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Nexus 7000 TrustSec

Sample Config Manual 802.1AE Symmetric Configuration
DC1

Encrypted Traffic Public Soil

DC2

Nexus-7000-1(config)# interface ethernet 2/45 Nexus-7000-1(config-if)# cts manual Nexus-7000-1(config-if-cts-manual)# sap pmk 12344219 Nexus-7000-1(config-if-cts-manual)# exit Nexus-7000-1# show cts CTS Global Configuration ============================== CTS support : enabled CTS device identity : test1 CTS caching support : disabled Number of CTS interfaces in DOT1X mode : 0 Manual mode : 1

Nexus-7000-2(config)# interface ethernet 2/3 Nexus-7000-2(config-if)# cts manual Nexus-7000-2(config-if-cts-manual)# sap pmk 12344219 Nexus-7000-2(config-if-cts-manual)# exit Nexus-7000-2# show cts CTS Global Configuration ============================== CTS support : enabled CTS device identity : test2 CTS caching support : disabled Number of CTS interfaces in DOT1X mode : 0 Manual mode : 1

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Nexus 7000 TrustSec

Interface Verification

Nexus-7000-1# show cts interface e 2/3 CTS Information for Interface Ethernet2/3: CTS i enabled, mode: is bl d d CTS_MODE_MANUAL CTS MODE MANUAL IFC state: CTS_IFC_ST_CTS_OPEN_STATE Authentication Status: CTS_AUTHC_SKIPPED_CONFIG Peer Identity: Peer is: Not CTS Capable 802.1X role: CTS_ROLE_UNKNOWN Last Re-Authentication: Authorization Status: CTS_AUTHZ_SKIPPED_CONFIG PEER SGT: 0 Peer SGT assignment: Not Trusted Global policy fallback access list: SAP Status: CTS_SAP_SUCCESS g p p Configured pairwise ciphers: GCM_ENCRYPT Replay protection: Disabled Replay protection mode: Strict Selected cipher: GCM_ENCRYPT Current receive SPI: sci:225577f0860000 an:1 Current transmit SPI: sci:1b54c1a7a20000 an:1

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Virtual Domain Contexts

Horizontal Consolidation
Objective: Consolidate lateral infrastructure that delivers similar roles for separate operational or administrative domains. Benefits: R d B fit Reduced power and space requirements, can maximize d d i t i i density of the platform, easy migration to physical separation for future growth. Considerations: Number of VDCs (4), Four VDCs != Four CPU Complexes, does not significantly reduce cabling or interfaces needed.
core 1 core 2 core core

Core Devices

Core

Aggregation Devices

agg1

agg2

agg3

agg4

agg VDC 1 agg VDC 2

Aggregation VDCs

acc1

acc2

accN

accY

acc1

acc2

accN

accY

Admin Group 1 TECDCT-3873_c2

Admin Group 2 Cisco Public

agg VDC 1 Admin Group 1

agg VDC 2 Admin Group 2

2009 Cisco Systems, Inc. All rights reserved.

Default VDC
The default VDC (VDC_1) is different from other configured VDCs.

Default VDC
vrf
VDC Admin

Can create/delete VDCs Can allocate/de-allocate resources to/ from VDCs Can intercept control plane and potentially some data-plane traffic from all VDCs (using wireshark) Has control over all global resources and p parameters such as managment0 g interface, console, CoPP, etc.

Network Admin

Can have the network-admin role which network admin has super-user priviledges over all VDCs

VDC2
VDC Admin

vrf

VDC3
VDC Admin

vrf

VDC4

vrf

With this in mind for high-security or critical environments the default VDC should be treated differently. It needs to be secured.

mgmt port (mgmt0)

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

VDC Best Practices

For high-security environments the Default VDC is really the Master VDC: Reserve the master VDC for VDC and resource administration when deploying a multi VDC environment Avoid multi-VDC environment. running data-plane traffic via the master VDC. Protect the Master VDC: Restrict access to the master VDC to the absolute minimum required to support VDC and overall global system administration. Default HA policy (2-Sups) is switchover: For enhanced VDC independence in dual supervisor configurations, explicitly set the HA polic for VDCs to restart or bringdo n policy bringdown. CoPP is global: Review CoPP policies to ensure that limits are inline with collective requirements of all VDCs. In multi-administrative environments make sure co-ordinate potential service or outage windows with administrative groups across VDCs.
TECDCT-3873_c2 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Resource Scalability Limits

Some resource scalability is limited per system, others are per VDC
16,000 maximum Logical Interfaces (RPVST+) TOTAL for all configured VDCs* 75,000 maximum Logical Interfaces (MST) TOTAL for all configured VDCs* 256 per configured VDC* 4096 VLANs per configured VDC*

FIB TCAM can be scaled by planning interface allocations

FIB is per I/O module and is only populated with entries for VDCs assigned on a module You can optionally maximize this by using the following rule: Assign 1 VDC per module (slot), with 2 modules minimum per VDC on a single system (to preserve redundancy)
*

for 4.0(3)
41

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

VDC Granularity for Current 10 GigE Ports

VDC A Ports are assigned on a per VDC basis and cannot be shared across VDC s VDCs VDC C

32 port 10GE module Once a port has been assigned to a VDC, ll b t fi ti done f from all subsequent configuration iis d within that VDC On 32-port 10GE module ports must be assigned to a VDC by 4-block groups.

VDC B

VDC C

http://www.cisco.com/en/US/docs/switches/datacenter/sw/4_1/nx-os/virtual_device_context/configuration/guide/ vdc_overview.html#wp1073104
TECDCT-3873_c2 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

VDC Granularity for 10/100/1000 Ports

On the 10/100/1000 card each port can be on a different VDC regardless of the adjacent ports (limited of course by the total of 4 VDCs) Using VDC it is possible to move servers seamlessly from a staging environment for example, to a production environment in the topology without having to re-cable the servers

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Virtual Device Contexts

VDC Resource Utilization (Layer 3)
VDC 10
Linecard 1
FIB TCAM

VDC 20
Linecard 2
FIB TCAM

VDC 30
Linecard 4
FIB TCAM

FIB and ACL TCAM resources are more effectively utilized

Linecard 3
FIB TCAM

Linecard 5
FIB TCAM

Linecard 6
FIB TCAM

Linecard 7
FIB TCAM

Linecard 8
FIB TCAM

128K ACL TCAM

64K

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Layer 2 Links, All Forwarding and No Loops

vPC
LAN
Virtual Switch

MAC Pinning
LAN

L2MP
LAN

vPC/MEC

MAC A

MAC B Active-Active

L2 ECMP

MAC A

MAC B

L2 ECMP

Virtual Switch (VSS on C6K, ( , vPC on Nexus 7K) Virtual port channel mechanism is transparent to hosts or switches connected to the virtual switch STP as fail-safe mechanism to prevent loops even in the case of control plane failure

Host Mode Eliminates STP on Uplink Bridge Ports Allows Multiple Active Uplinks Switch to Network Prevents Loops by Pinning a MAC Address to Only One Port Completely Transparent to Next Hop Switch

Uses ISIS based topology Up to 16 way ECMP Eliminates STP from L2 domain Preferred path selection

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

vPC Terminology
STP Root vPC FT link

vPC peer a vPC switch, one of a pair vPC member port one of a set of ports (port channels) that form a vPC vPC the combined port channel between the vPC peers and the downstream device vPC peer-link Link used to synchronize state between vPC peer devices, must be 10GbE vPC ft-link the fault tolerant link p , , between vPC peer devices, i.e., backup to the vPC peer-link

vPC Peer link STP Secondary Root vPC peer

vPC member Ports

10 Gig uplinks

CFS Cisco Fabric Services protocol, used for state synchronization and configuration validation between vPC peer devices

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

vPC Layer 2 Processing (i.e. Etherchannel)

Notice that the Peer-link is almost unutilized

Etherchanneling modified to keep traffic local Downstream Switch runs LACP

hashing enhanced to keep traffic local

LACP Unmodified Portchanneling

10 Gig uplinks

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

vPC: Layer 3 Traffic Processing

Notice that the Peer-link is almost unutilized
HSRP active process communicates the active MAC to its neighbor. Only the HSRP active process responds to ARP requests

HSRP MAC populated in the Layer 2 table with the R flag

HSRP primary HSRP standby y

10 Gig uplinks

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

vPC Versus VSS

vPC Control Plane SSO HSRP Etherchannel to prefer local links Failover time Configuration synchroniziation Split Brain detection
TECDCT-3873_c2

VSS Unified Yes 1 single IP address, i.e. NO HSRP yes subsecond Yes, automatically done because of the unified CP Yes via BFD and PagP+
49

Separated Yes (2 sups per chassis) 2 entities yes In the order of seconds in the current release CFS to verify configurations and warn about mismatches Yes via the Fault Tolerant link
Cisco Public

2009 Cisco Systems, Inc. All rights reserved.

Pinning
1 Border interface 2 3 4

Server interface (SIF)

A
TECDCT-3873_c2

B
Cisco Public

F
50

2009 Cisco Systems, Inc. All rights reserved.

Outgoing Traffic Known Unicast

1 2 3 4

Traffic sourced by a station y connected to a SIF can go to one of the locally connected servers Or, if no local match is found, goes out of its pinned border interface

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Outgoing Traffic Multicast/Broadcast

1 2 3 4

Local replication to all SIFs is done by the End Host Virtualizer switch One copy of the packet is sent out of the source SIFs pinned border interface

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Incoming Traffic Reverse Path Forwarding

1 2 3 4

Reverse Path Forwarding protects from Loops Packets destined to a station behind a SIF are accepted only by the SIF pinned border interface

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Incoming Traffic Multicast/ Broadcast Portal

1 2 3 4

Multicast/Broadcast Portal protects from Loops One border interface is elected to receive broadcast, multicast and unknown unicast traffic for all the SIFs

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Incoming Traffic Deja-vu Check

1 2 3 4

The Deja-vu check prevents Loops If the source MAC belongs to a local station
The multicast/broadcast portal drops the packet The pinned port accepts the packet, but no replication is done This is regardless of the destination MAC (known/unknown unicast, multicast or broadcast)

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Pinning Configurations (1)

correct configuration

Border interface

Server interface (SIF)

incorrect configuration

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Pinning Configurations (2)

all Border Interfaces of the same subnet must be in the same L2 domain

Border interface

Server interface (SIF)

Virtual Switching can be connected to End Host Virtualizer

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Layer 2 Multipathing
Clos Networks

Layer 2 MultiPathing enables designs that up until today were only possible with Infiniband
TECDCT-3873_c2 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Layer 2 Multipathing
Edge switches
Determine which Edge id can reach a given MAC address Set the destination id IS-IS computes shortest path to id

Core switches
Forward from Edge switch to Edge switch based on destination id IS-IS computes shortest path to id

Source MAC sends to Destination MAC Edge switch does lookup for id attached to Destination MAC
If found, forward based on id If not found, flood on broadcast tree

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Core Forwarding Table

FORWARDING TABLE on 3

Core l1

3 l2

Destination Switch 1

Link L1 L2 N/A L1,L2 L1,L2

Edge

Switch 2 Switch 3 Switch 4

M A C
TECDCT-3873_c2

Switch 5

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Edge Forwarding Table

FORWARDING TABLE on 1

Core l1

3 l2

Destination
l3 2

Link Directly Switch 2

MAC A, B, C MAC D, E, F

Edge

M A C
TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Server Connectivity Evolution Present

Management Challenges Shift towards server virtualization Multiple VMs inside each physical server, connected by virtual switches Rapid proliferation of logical elements that need to be managed Feature parity issues between virtual and physical elements

VSwitch

vNICs

VMs

Separate management of physical (

) and logical (

) elements
62

Server Connectivity Evolution Future

Future with Network Interface Virtualization and VNTAG: Consolidated Management
Virtual Interfaces within VMs are now visible to the switch Both network configuration and policy enforcement for these interfaces can now be driven from the switch This allows consolidated management of physical and virtual elements

VSwitch

vNICs

VMs

Consolidated management of physical (

) and logical elements

Interface Virtualizer (IV) Architecture

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

VNTAG

VNTAG Format
VNTAG Ethertype l source virtual interface
SA[6]

d p

destination virtual interface

DA[6]

direction indicates to/from adapter source virtual interface indicates frame source
looped indicates frame came back to source adapter

VNTAG[6] 802.1Q[4]

destination virtual interface dictates forwarding

pointer helps pick specific destination vNIC or vNIC list

Frame Payload

Link local scope

Rooted at Virtual Interface Switch 4096 virtual interfaces 16,384 Virtual interface lists

CRC[4]

Coexists with VLAN (802.1Q) tag

802.1Q tag is mandatory to signal data path priority
TECDCT-3873_c2 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

VNTAG Processing (1)

SAN LAN

Interface Virtualizer adds VNTAG

Unique source virtual interface for each vNIC d (direction) = 0 p (pointer), l (looped), and destination virtual interface are undefined (0) Frame is unconditionally sent to the Switch

Virtual Interface Switch

Interface Virtualizer v v OS v v OS v v OS

Application Payload P l d TCP VNTAG Ethertype l source virtual interface d p destination virtual interface IP VNTAG Ethernet

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

VNTAG Processing (2)

SAN LAN

Virtual Interface Switch ingress processing

Extract VNTAG Ingress policy based on port and source virtual interface Access control and forwarding based on frame fields and virtual interface policy Forwarding selects destination p ( ) and g port(s) destination virtual interface(s) VIS adds a new VNTAG
access control & forwarding policy

Virtual Interface Switch

Interface Virtualizer v v OS v v OS v v OS

Application Payload P l d TCP IP Ethernet

VNTAG

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

VNTAG Processing (3)

SAN LAN

Virtual Interface Switch egress processing

Features from port and destination virtual interface Insert VNTAG(2)

Virtual Interface Switch

Interface Virtualizer

direction is set to 1 destination virtual interface and pointer select a single vNIC or list source virtual interface and l (looped) filter a single vNIC if sending frame to source adapter

v v OS

Application Payload P l d TCP VNTAG Ethertype l source virtual interface d p destination virtual interface IP VNTAG(2) Ethernet

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

VNTAG Processing (4)

SAN LAN

Interface Virtualizer (IV) forwards based on VNTAG

Extract VNTAG Upper layer protocol features from frame fields destination virtual interface and pointer select vNIC(s) source virtual interface and looped filter a single vNIC if source and destination are same IV
ULP features v v OS v v OS vNIC forwarding

Virtual Interface Switch

Interface Virtualizer v v OS v v OS v v OS

Application Payload P l d TCP IP Ethernet

x
v v OS v v OS v v OS

x
v v OS

VNTAG(2)

Unicast (single vNIC)

Multicast (vNIC list)

Cisco Public

VNTAG Processing (5)

SAN LAN

OS stack formulates frames traditionally Interface Virtualizer adds VNTAG Virtual Interface Switch ingress processing Virtual Interface Switch egress processing Interface Virtualizer forwards based on VNTAG OS stack receives frame as if directly connected to Switch

Virtual Interface Switch

Interface Virtualizer v v OS v v OS v OS

Application Payload P l d TCP IP Ethernet

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

VNTAG + MAC Pinning

Interface Virtualizers connect to the network in a redundant fashion Redundancy can be addressed using MAC pinning: each downlink port is associated with an uplink port Forwarding is based on a VIF forwarding table which is made of 1024 entries For multicast traffic, a VIF_LIST table is indexed by a VIF_LIST_ID and the result is a bitmask indicating which SIF ports should the frames be sent to.

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Nexus 2000 Fabric Extender

Network Topology Physical vs. Logical Physical Topology
Core Layer

Logical Topology
Core Layer

VSS

L3 L2
4x10G uplinks from each rack
FE

L3 L2

Nexus 5020

Nexus 5020 Nexus 5020 Nexus 5020

12 FEX

FEX

12 FEX

Servers
Servers

Rack-1

Rack-N

Rack-1

Rack-N

Rack-1

Rack-2

Rack-3

Rack-4

Rack-5

Rack-12

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Data Center Access Architecture

Distributed Access Fabric
De-Coupling of the Layer 1 and Layer 2 Topologies Optimization of both Layer 1 (Cabling) and Layer 2 (Spanning Tree) Designs Mixed cabling environment (optimized as required) Flexible support for Future Requirements

...
Nexus 5000/2000 Mixed ToR & EoR
Combination of EoR and ToR cabling
TECDCT-3873_c2 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Cabling Design for FEX

Copper Connectivity

Top of Rack Fabric Extenders provide 1G server connectivity Nexus 5000 in Middle of Row connects to Fabric Extenders with CX1 copper 10G pp between racks Suitable for small server rows where each FEX is no longer than 5 meters from the 5Ks CX1 copper between racks is not patched Middle of Row Nexus 5000 can also provide 10G server connectivity within their rack

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

FEX Inner Functioning

Inband Management Model Fabric extender is discovered by switch using an L2 Satellite Discover Protocol (SDP) that is run on the uplink port of fabric extender NX5K checks software image compatibility, assign an IP address and upgrade the fabric extender if necessary N5K pushes programming data to Fabric Extender Fabric Extender updates the N5K with its operational status and statistic. Extension to existing CLI on N5K is used for Fabric Extender CLI information
TECDCT-3873_c2 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

FEX Design Considerations

Uplink Dual Homing
Without vPC support N5K-A N5K-B With vPC support N5K A N5K-A N5K B N5K-B

SDP exchange

Err-disable

Static pinning is not supported in a redundant supervisor mode Server ports appear on both N5K Currently configuration for all ports must be kept in sync manually on both N5Ks
TECDCT-3873_c2 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

FEX Design Considerations

Server Dual Homing
vPC provides two redundancy designs for the virtualized access switch Option 1 - MCEC connectivity from the server Two virtualized access switches bundled into a vPC pair Logically a similar HA model to that currently provided by VSS
Two Virtualized access switches Each with a Single Supervisor

vPC peers

MCEC from server to the access switch

FEX Design Considerations

NIC Teaming with 802.3ad Across Two FEX Devices

N5KA N5K

N5KB

By leveraging vPC it is possible to create 802.3ad configurations with dual-homed servers

FEX Design Considerations

MAC Pinning on Fabric Extender (FEX)
Static Pinning Fabric Extender associates (pins) a server side (1GE) port with an uplink (10GE) port Server ports are either individually pinned to specific uplinks (static pinning) or all interfaces pinned to a single logical port channel Behavior on FEX uplink failure depends on the configuration g p pinned to Static Pinning Server ports p the specific uplink are brought down with the failure of the pinned uplink Port Channel Server traffic is shifted to remaining uplinks based on port channel hash
Server Interface stays active
TECDCT-3873_c2 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

NIC teaming required

Port Channel

FEX Design Considerations

N2K/N5K Spanning Tree Design Considerations
Root Bridge HSRP Active Secondary Root Bridge HSRP Standby y

BPDU Guard

Bridge Assurance

Global BPDU Filter reduces the spanning tree load (BPDUs generated on a Host Port) VMWare S VMW Server Trunk T k Needs to Carry Multiple VLANs which can increase the STP load
VSwitch
VM #1 VM #2 VM VM #3 #4

UDLD

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

FEX Design Considerations

vPC - Spanning Tree Design Considerations
Enabling vPC on the access to aggregation links improves y y layer 2 scalability Removing physical loops out of the layer 2 topology Reducing the STP state on the access and aggregation layer The use of vPC does result in a reduction of logical port count on the aggregation but does involve CFS synchronization of state between the two aggregation nodes
Fabric Links (No Spanning p g Tree) vPC
Single Logical Link to STP Both vPC Peers Act as the default GW

Server Ports BPDU Guard

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

vPC Configuration Commands

Configure vPC, and start the ft-link on both peers:
(config)# feature vpc (config)# vpc domain 1 (config-vpc-domain)# peer-keepalive destination x.x.x.x source x.x.x.y (conifg)# int port-channel 10 (config-int)# vpc peer-link

Move any port-channels into appropriate vPC groups

(config)# int port-channel 20 (config-int)# vpc 20

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

vPC Control Plane vPC Role

vPC domain is identified by a configured ID, and after successful establishment of peer-link peer link adjacency, vPC domain is operationally enabled.
MAC-address derived from domainID is used for link-specific protocol operations (LACP lag-id for vPC, designated bridge-id for STP)
vPC primary vPC secondary

vPC domain

vPC election generates vPC role (primary/secondary) for each switch. vPC role is used only when dual-active topology is detected.

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

vPC Control Plane FT Link

FT link (can be routed)

vPC FT (fault-tolerant) link is an additional mechanism to detect liveness of the peer. can peer use any L3 port. By default, will use management network.
used only when peer-link is down does NOT carry any state information

VRF FT

VDC A (e.g. 2)

Peer-link

Rare lik lih d of d l R likelihood f dualactive topology vPC is within the context of a VDC

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

vPC Deployment
Recommended Configurations
vPC is a Layer 2 feature Port has to be in switchport mode before configuring vPC vPC/vPC peer link support following port/ peer-link port-channel modes Port Modes: Access or Trunk Port-channel Modes: On mode or LACP (active/passive) mode Recommended port mode Trunk vPC peer-link should support multiple VLANs and should trunk the access VLANs Recommended port-channel mode is Link Aggregation Control Protocol (LACP). Dynamically react to runtime changes and failures Lossless membership change Detection of mis-configuration Maximum 8 ports in a port-channel in on-mode and 16 ports with 8 operational ports in a LACP port-channel
TECDCT-3873_c2 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

VRF FT

VDC A (e.g. 2)

Peer-link

LACP

vPC Control Plane

CFSoE
l2fm igmp stp vpcm { opcode, payload} vpcm stp igmp l2fm vpc-transport-api vpc-transport-api

cfs

cfsoe
netstack

sw-1

sw-2

cfs

cfsoe
netstack

CFS (Cisco Fabric Service), over Ethernet (CFSoE), provides a reliable transport layer to all applications that need to co-operate with peer vPC switch. CFSoE
uses retransmissions & acknowledgements per segment transmitted. supports fragmentation and re-assembly for payloads more than MTU uses BPDU class address, and is treated with highest QoS/drop-thresholds.

Each component has (one or more) request-response handshakes (over CFSoE) with its peer. Protocols (STP/IGMP/FHRP) continue to exchange regular protocol BPDUs. In addition, theyll use CFS for state synchronization

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

CFS Distribution
CFS only checks that the VLANs assigned to a vPC are the same on both devices that are on the same vPC This warns the person on the other 7k that he has to make configuration changes to include the same exact VLANs Distribution is automatically enabled b enabling vPC by PC (config)#cfs distribute enable ( (config)#cfs ethernet g) distribute enable tc-nexus7k01-vdc3# show cfs status Distribution: Enabled Distribution over IP: Disabled IPv4 multicast address: 239.255.70.83 IPv6 multicast address: ff15::efff:4653 Distribution over Ethernet: Enabled
TECDCT-3873_c2 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

CFSoIP vs CFSoE
vPC uses CFSoE, Roles Leverage CFSoIP
vPC domain (CFSoE) CFSoIP Cloud

Role Defintion

The user creates new Role User commits the changes Role get automatically propagated to the other switches
TECDCT-3873_c2 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Type-1 Compatibility Parameters

Port Channel is disabled if one of the following parameters is mismatched
Port-channel Speed (10M, 100M, 1000M or 10G) Port-channel Duplex (half or full) Port Mode (access or trunk) Port-channel MTU Port-channel Native VLAN Port-channel mode (on, active or passive)

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Detecting Mis-Configuration
Sw1 (config)# show vpc brief VPC domain id : Peer status : VPC keep-alive status : Configuration consistency status: 1 peer adjacency formed ok Disabled success

VPC status --------------------------------------------------id Port Consistency Reason ---- -------------- ----------- ---------------1 Po2 success success 2 Po3 failed vpc port channel mis-config due to vpc links in the 2 switches connected to different partners

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

vPC Failure Scenarios Peer-Link Failure (Link Loss)

vPC primary vPC secondary

In case vPC peer-link fails Check active status of remote vPC peer via vPC ft-link (heartbeat) If both peers are active, then Secondary will disable all vPC ports to prevent loops Data will automatically forward down remaining active port channel ports Failover gated on CFS message F il t d failure, or UDLD/Link state detection

CFSoE

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

vPC Failure State Diagram

Start CFS message delivery failure? vPC secondary peer? Yes Suspend vPC member ports Recover vPC member ports No

No vPC Peer link failed? (UDLD/Link state) Yes vPC ft-link heartbeat detect? No Other processes take over based on priority (STP root, HSRP active, PIM DR) Yes

Yes

vPC peer recovered?

Yes

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

vPC Between Sites and Within Each DC

CFSoE Region 1 DC1 N7kA-DC1 Eth2/3 access Eth2/26 Eth2/9 Po60 Eth2/25 Eth8/40 Eth8/4 vPC between sites DC2 N7kC-DC2 N7kC DC2 Eth7/3 Eth7/9 Eth8/5 Eth7/25 Po50 CFSoE Region 2

Peer link
Eth2/9

Peer link
Eth7/9 Eth7/25 Eth8/5

Eth2/25

Eth2/26

Eth2/3 Po30

Eth7/3 N7kD-DC2

N7kB-DC1

Links Protected by IEEE 802.1ae FT link

Routing Design
for the Extended VLANs
DC1 gw 1.1.1.1 DC2 gw 1.1.1.2 Failover direction SRP Group 2 (e.g. 1.1.1.2) for HS

150, 150 120

Failover di irection for HSRP Group 1 (e.g. 1.1.1.1) G

120, 120 150

HSRP Group 1

HSRP Group 2

140, 140 130

130, 130 140

G 60 0000.0c07.ac3c static << group that is active or standby * 60 0000.0c07.ac3d static << group that is listen mode G 60 0000.0c07.ac3d static << group that is active or standby * 60 0000.0c07.ac3c static << group that is listen mode
TECDCT-3873_c2 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

LAN Switching
Evolution of Data Center Architectures New Layer 2 Technologies Fabric Extender Deep dive and design with virtual Port Channeling Break Demo: vPC Designs with Server Virtualization
Nexus1kv Components Operational benefits VEM Forwarding: NIC Teaming and Etherchannels LAN switching infrastructure requirements Designs with Blade Servers

10 Gigabit Ethernet to the Server Break Demo: Nexus1kv

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Why Is a Virtual Switch Needed in the First Place

Forwarding Table

Destination MAC MAC1

DMAC = MAC2 Ethernet1/1 DMAC = MAC2

Port 1/1 1/1

MAC2

MAC1

MAC2

?
VM2
Cisco Public

VM1

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Virtual Switching Virtualized Servers Need VN-link Technology

Forwarding Table

Destination MAC MAC1

Ethernet1/1

Port 1/1 1/1

MAC2

vSwitch or Nexus 1000v

=
MAC1 MAC2 Nexus1kv

VM1

VM2
99

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

ESX Server Components

VMware ESX is a bare-metal hypervisor that partitions physical servers in multiple virtual machines Virtual Machine
App App App

vnics S Software virtual switch

vmnics

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

100

Nexus 1000v
Distributed Virtual Switch
N1k-VSM# sh module

Linecards Equivalent

Mod Ports Module-Type Model Status 1 1 Supervisor Module Cisco Nexus 1000V active * 2 1 Supervisor Module Cisco Nexus 1000V standby 3 48 Virtual Ethernet Module ok 4 48 Virtual Ethernet Module ok

App

Hypervisor

vCenter

Virtual Ethernet Module

Fabric Function
TECDCT-3873_c2 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Virtual Supervisor Module

101

Nexus 1000V
Virtual Interface
veth = Virtual Machine port (vnic)
Hypervisor
App App App App OS OS OS OS

veth3

veth7 veth68

N1k-VSM# sh interface virtual Port Adapter Owner Veth3 Veth7 Veth68

Mod Host

Net Adapter 1 Ubuntu VM 1 pe-esx1 Net Adapter 1 Ubuntu VM 2 pe-esx1 Net Adapter 1 Ubuntu VM 3 pe-esx1

Cisco VSMs

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

102

Nexus 1000v
Ethernet Interface

App

Eth = uplink port on the ESX Server

eth3/1 th3/1

eth3/2

Hypervisor

WS-C6504E-VSS#sh cdp neighbors Device ID Local Intrfce Platform N1k-VSM N1k-VSM N1k-VSM N1k-VSM Gig Gig Gig Gig 1/1/1 2/1/2 1/8/1 2/8/2 Nexus1000 Nexus1000 Nexus1000 Nexus1000

Port ID Eth Eth Eth Eth 3/1 3/2 4/1 4/2

eth4/1

App

eth4/2
TECDCT-3873_c2 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Hypervisor
103

What Is a Port Profile?

n1000v# show port-profile name WebProfile port profile port-profile WebProfile description: status: enabled capability uplink: no system vlans: port-group: WebProfile config attributes: switchport mode access switchport access vlan 110 no shutdown evaluated config attributes: switchport mode access it h t d switchport access vlan 110 no shutdown assigned interfaces: Veth10

Support Commands Include: Port management VLAN PVLAN Port-channel ACL Netflow Port Security QoS

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

104

Port-Profile as Viewed from the Network Administrator and Server Administrator

Network Administrator view
N1k-VSM# sh port-profile name Ubuntu-VM port-profile Ubuntu-VM description: status: enabled capability uplink: no capability l3control: no system vlans: none port-group: Ubuntu-VM max-ports: 32 inherit: config attributes: switchport mode access switchport access vlan 95 no shutdown assigned interfaces: Vethernet2 Vethernet4
TECDCT-3873_c2 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Server admin view

105

What Makes the Virtual Switch Distributed?

ESX servers that are under the same Nexus 1kv VSM share the same PortProfile Configuration When a new Port-Profile is defined it gets automatically propagated to all the ESX servers (VEMs) that are the VSM In this example ESX1 and ESX2 are under VSM1 and share the green and red Port-Profile ESX3 and ESX4 are under VSM2 and share the Blue and Yellow Port Profile
1 2 3 4

VSM1
Cisco VSMs

VSM2
Cisco VSMs

Port Profiles

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

106

Prior to DVS Ensuring Port-Group Consistency Was a Manual Process

Each ESX host is configured individually for Networking

107

VMotion Requires the Destination vSwitch to Have the Same Port Groups/Port-Profiles as the Originating ESX Host
Rack1 Rack10

c0 vmnic0 ESX Host 1 vSwitch

vmnic1

Prior to DVS you had to manually ensure that the same Port-Group existed on ESX Host 1 as ESX Host 2

vmnic0 ESX Host 2

vmnic1

vSwitch
App OS

App OS

VM1
TECDCT-3873_c2

VM2

VM3
Cisco Public

VM4

VM5

VM6
108

2009 Cisco Systems, Inc. All rights reserved.

Distributed Virtual Switching Facilitates VMotion Migration

Port Profiles

Server 1
VM #1 VM #2 VM #3 VM #4 VM #1

Server 2
VM #2 VM #3 VM #4

VEM VMW ESX

VMs Need to Move

VMotion DRS SW Upgrade/Patch Hardware Failure
TECDCT-3873_c2 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

109

10 Gigabit Ethernet to the Server Break Demo: Nexus1kv

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

110

Configuring Access-Lists, Port Security, SPAN, etcWithout Nexus1kv Is Complicated

Is VM#1 on Server 1? Or on which server, on which switch do I put the ACL?
VM #4

Server 1
VM #1 VM #2 VM #3

vSwitch
VMW ESX

ACL need to be specify the IP address of the VM else you risk to drop both VM1 and VM3 traffic SPAN will get all traffic from VM1, VM2, VM3, VM4!! You need to filter that!! Port Security CANT be used

ACLs (complicated)

SPAN (realistically cant be used)

Port Security needs to be disabled

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

111

You Can Use Access-Lists, Port Security, SPAN, etcWITH Nexus1kv

ACLs specific to a Port-Group Server 1
VM #1 VM #2 VM #3 VM #4

Is VM#1 on Server 1? It doesnt matter ACL follows the VM SPAN will get only the traffic from the virtual Ethernet Port Port Security ensures that VMs wont generate fake make addresses

VEM VMW ESX Port Security

SPAN on a virtual ethernet port

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

112

vNIC Security
Server

VMs can be secured in multiple ways:

VLANs ACLs Private VLANs Port-Security

VM #1

VM #2

VM #3

VM #4

vnics i
Nexus 1000 DVS

vmnic IEEE 802.1q trunk

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

113

Private VLANs Can Be Extended Across ESX Servers by Using the Nexus1kv
Promiscuous ports receive and transmit to all hosts Communities allow communications between groups Isolated ports talk to promiscuous promisc o s ports only
App App App App App App App

Promiscuous Port

Only One Subnet

Promiscuous Port

x x
App

Primary VLAN
OS OS OS OS OS OS OS OS

Community VLAN Community VLAN Isolated VLAN

.11

.12

.13

.14

.15

.16

.17

.18

Community A
Cisco Public

Community B

Isolated Ports
114

Tracing Virtual Ethernet Ports

show interface VEthernet Vethernet2 is up Hardware is Virtual, address is 0050.5675.26c5 Owner is VMware VM1, adapter is vethernet1 Active on module 8, host tc-esx05.cisco.com VMware DVS port 16777215 Port-Profile is MyApplication Port mode is access Rx 444385 Input Packets 444384 Unicast Packets 0 Multicast Packets 1 Broadcast Packets 572675241 Bytes Tx 687655 Output Packets 687654 Unicast Packets 0 Multicast Packets 1 Broadcast Packets 1 Flood Packets 592295257 Bytes 0 Input Packet Drops 0 Output Packet Drops
TECDCT-3873_c2 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

115

SPAN Traffic to a Catalyst 6500 or a Nexus 7k Where You Have a Sniffer Attached
Capture here
App App App App App App App App App App App App

Hypervisor Virtual Ethernet Module

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

116

Ease of Provisioning
Plug-and-play designs with VBS
1 Add or replace a VBS Switch to the Cluster 2 Switch config and code automatically propagated Virtual Ethernet Module

3 Add a blade Server 4 Its always booted from the same LUN

Virtual Ethernet Module

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

117

Ease of Provisioning
Making Blade Servers Deployment Faster
1 Physically Add a new blade (or replace an old one)

2 Go to vCenter, add host to cluster

Virtual Ethernet Module

Nexus 1000v

3 Done: the new blade is in production All port-groups appear

Virtual Ethernet Virtual Ethernet Module Module

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

118

10 Gigabit Ethernet to the Server Break Demo: Nexus1kv

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

119

Cisco Nexus 1000V Switch Interfaces

Ethernet Port (eth)
1 per physical NIC interface Specific to each module vmnic0 = ethx/1 Up to 32 per host
Eth3/1 Po1

Eth3/2

Port Channel (po)

Aggregation of Eth ports Up to 8 Port Channels per host
VM1 VM2

Veth1

Veth2

Virtual Ethernet Port (veth)

1 per VNIC (including SC and VMK) Notation is Veth(port number). 216 per host

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

120

Loop Prevention without STP

Eth4/1 Cisco VEM Cisco VEM

Eth4/2

X
Cisco VEM

X
VM1 VM2 VM3 VM4 VM5 VM6 VM7 VM7 VM9 VM10 VM11 VM12

BPDU are dropped

No Switching From Physical NIC to NIC

Local MAC Address Packets Dropped on Ingress (L2)

121

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

MAC Learning
Each VEM learns independently and maintains a separate MAC table VM MACs are statically mapped
Other vEths are learned this way (vmknics and vswifs) No aging while the interface is up
Eth3/1 Cisco VEM Eth4/1 Cisco VEM

Devices external to the VEM are learned dynamically

VM1

VM2

VM3

VM4

VEM 3 MAC Table

VM1 VM2 VM3 VM4 Veth12 Veth23 Eth3/1 Eth3/1 Static Static Dynamic Dynamic VM1 VM2 VM3 VM4

VEM 4 MAC Table

Eth4/1 Eth4/1 Veth8 Veth7 Dynamic Dynamic Static Static

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

122

Port Channels

Standard Ci St d d Cisco Port Channels P t Ch l

Behaves like EtherChannel

Link Aggregation Control Protocol (LACP) Support 17 hashing algorithms available

Selected either system wide or per module Default is source MAC
VM1 VM2 VM3 VM4

Po1

Cisco VEM

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

123

Virtual Port Channel Host Mode

Allows a VEM to span multiple upstream switches using subgroups Forms up to two subgroups based on Cisco Discovery Protocol (CDP) or manual configuration Does not support LACP veths are associated in a round robin to a subgroup and then hashed within a subgroup Does not require a port channel p g g upstream when using single link in each sub-group Required when connecting a port channel to multiple switches unless MCEC is configured on the access side
CDP received from the same switch creates the sub-group bundle
TECDCT-3873_c2 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

N5K View

SG0

Po1

SG1

VEM View

VM #2

VM #3

VM #4

124

Automated Port Channel Configuration

Port channels can be automatically formed using port profile Interfaces belonging to different modules cannot be added t same I t f b l i t diff t d l t b dd d to channel-group. E.g. Eth2/3 and Eth3/3 auto keyword indicates that interfaces inheriting the same uplink port-profile will be automatically assigned a channel-group.
n1000v(config)# port-profile Uplink n1000v(config-port-prof)# channel-group auto

Each interface in the channel must have consistent speed/duplex Channel-group does not need to exit and will automatically be created

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

125

Uplink Port Profiles

Special profiles that define physical NIC properties Usually configured as a trunk Defined by adding capability uplink to a port profile Uplink profiles cannot be applied to vEths Non-uplink profiles cannot be applied to NICs Only l t bl in C t O l selectable i vCenter when adding a h ddi host or additional NICs
n1000v(config)# port-profile DataUplink n1000v(config-port-prof)# switchport mode trunk n1000v(config-port-prof)# switchport trunk allowed vlan 10-15 n1000v(config-port-prof)# system vlan 51, 52 n1000v(config-port-prof)# channel-group mode auto sub-group cdp n1000v(config-port-prof)# capability uplink n1000v(config-port-prof)# no shut
TECDCT-3873_c2 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Cisco VEM

VM1

VM2

VM3

VM4

126

System VLANs
System VLANs enable interface connectivity before an interface is programmed
i.E VEM cant communicate with VSM during boot

Cisco VSM

Required System VLANs

Control
L2 Cloud

Packet

Recommended System VLANs y

IP Storage Service Console VMKernel Management Networks
Cisco VEM C P

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

127

Four NIC Configuration

Access Layer Configuration
Trunk port

No EtherChannel

N1KV Port Channel 1

vPC-HM VM Data
SG0 SG1 Po1

N1KV Port Channel 2

vPC-HM Service Console, VM Kernel, Control and Packet

SG0 SG1 Po2

Cisco VEM Ci
C P

VEM Configuration
Source Based Hashing

Use Case
Medium 1Gb servers (rack or blade) Need to separate VMotion from Data
TECDCT-3873_c2 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

SC
VM Data

VMK

128

Four NIC Configuration with N2k w/o vPC

In a Four NIC implementation Access switch configured with Trunk g ports (no Etherchannel) VEM Configured with SRC based hashing N1KV Port Channel 1 (vPC-HM)
VM Data
Trunk Edge Port supporting only the VM VLANs

N1KV Port Channel 2 (vPC-HM) ( )

Service Console, VM Kernel, VEM Control and Packet

SG0

SG1

SG0

SG1

VM traffic carried on a second vPCHM uplink bundle

VMK

SC and VMK traffic carried on one upstream vPC-HM uplink bundle

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

129

Four NIC Configuration with vPC

Using 2 Separate Regular Port-Channels
Access switch configured with two server vPC MCEC trunk ports VEM C fi Configured with L3/L4 b d ith based d hashing N1KV Port Channel 1
VM Data

N1KV Port Channel 2

Service Console, VM Kernel, VEM Control and Packet

vPC MCEC Bundles

VM traffic carried on a second uplink bundle

VMK

SC and VMK traffic carried on one upstream uplink bundle

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

130

Four NIC Configuration with vPC

Using a Single vPC-HM of Four Ports
Combine vPC-HM and MCEC vPC to load share traffic across four NICs Access switch configured with t A it h fi d ith two server vPC MCEC trunk ports VEM Configured with SRC based hashing N1KV Port Channel 1 (vPC-HM)
VM Data
vPC MCEC Bundles

Do not use CDP to create the sub-groups in this type of topology (manually configure the sub-groups)

SG0

SG1

VM 1

VM 2

VM 3

Single shared upstream vPCHM comprised of four links

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

131

Cisco Nexus 1000V Scalability

A single Nexus 1000V supports:
2 Virtual Supervisor modules (HA) 64* Virtual Ethernet modules 512 Active VLANs 2048 Ports (Eth + Veth) 256 Port Channels
Nexus 1000V

A single Virtual Ethernet module supports:

216 Ports Veths 32 Physical NICs 8 Port Channels
* 64 VEMs pending final VMware/Cisco scalability testing ** Overall system limits are lower than VEM limit x 64
132 Cisco VEM

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

10 Gigabit Ethernet to the Server Break Demo: Nexus1kv

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

133

Virtual Machine Considerations

Hardware MAC learning Large HW-based MAC address Tables Control plane policing Layer 2 trace Broadcast and Storm Control Private VLAN integration Unified I/O ready
Virtual Servers

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

134

10 Gigabit Server Connectivity

VNTAG / Nexus 1000v

FCoE

10 Gigabit Ethernet

Class-Based Bandwidth Allocation

DCE

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

135

LAN Switching
Evolution of Data Center Architectures New Layer 2 Technologies Fabric Extender Deep dive and design with virtual Port Channeling Break Demo: vPC Designs with Server Virtualization
Nexus1kv Components Operational benefits VEM Forwarding: NIC Teaming and Etherchannels Scalability Considerations LAN switching infrastructure requirements Designs with Blade Servers

10 Gigabit Ethernet to the Server Break Demo: Nexus1kv

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

136

With Nexus1kv the Switch Just a Plug-and-Play Fabric

With the Nexus1kv the Profiles are defined on the Nexus1kv The Mapping is p pp g performed on the Virtual Center The Switch provides simply the Switching Fabric and trunks all necessary VLANs.

Nexus1kv Mapping of servers t M i f to VLANs/Port Profiles Profile Definition vCenter C t Nexus1kv CLI

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

137

Switching Fabric With Virtualized Servers

You have Virtualized Servers on the Blades You are better off using clustered Cisco VBS g Cisco VBS Network Management Model Stacking Capability Etherchanneling Server Identity Equivalent to a 3750 stackable: plug-andplay Up to 8 Blade Switches Switches, i.e. single config point Across switches in the stack Flexattach

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

138

Nexus 1000v With Blade Enclosures

Port-Profile Definition

App App App

App

App App App App

App App App

App

App App App App

OS OS OS OS

Fabric Function 10 Gigabit Uplinks

139

LAN Switching
Evolution of Data Center Architectures New Layer 2 Technologies Fabric Extender Deep dive and design with virtual Port Channeling Break Demo: vPC Designs with Server Virtualization
Nexus1kv Components Operational benefits VEM Forwarding: NIC Teaming and Etherchannels Scalability Considerations LAN switching infrastructure requirements Designs with Blade Servers

10 Gigabit Ethernet to the Server Break Demo: Nexus1kv

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

140

Todays Data Center Networks

LAN HPC

SAN A

SAN B

Ethernet FC High Perf. Comp. (HPC)

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

141

Consolidation Vision
Why? VM integration Cable Reduction Power Consumption reduction Foundation for Unified Fabrics IPC
(*) RDMA = Remote Direct Memory Access (**) iWARP = Internet Wide Area RDMA Protocol
TECDCT-3873_c2 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

FCoE

142

LAN Switching
Evolution of Data Center Architectures New Layer 2 Technologies Fabric Extender Deep dive and Design with virtual Port Channeling Designs with Server Virtualization Break Demo: vPC Designs with Server Virtualization 10 Gigabit Ethernet to the Server
10 Gigabit Ethernet Performance Considerations 10 Gigabit Performance in Virtualized Environments Datacenter Ethernet

Break Demo: Nexus1kv

143

10 Gigabit Adapters Typical Features

MSI-X (Message Signaled Interrupt) Support PCIe 8x for 10 Gigabit Performance g TCP Offload (TOE) in Hardware
Configurable TCP SACK (Selective Acknowledgement) (not really configurable) Checksum offload

Large Send Offload (LSO): allows the TCP layer to build a TCP message up to 64KB and send it in one call down the stack through the device driver. Segmentation is handled by the Network Adapter Receive Side Scaling queues: 2 4 or disabled. Allows distributing incoming traffic to the available cores. VLAN offload in Hardware NetDMA support
TECDCT-3873_c2 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

144

OS Enablers
TCP Chimney Offload Receive Side Scaling ( RSS g (+ capable NIC) In Windows 2003 this requires the Scalable Networking Pack (SNP). (SNP) In Windows 2008 this is already part of the OS.

Do not enable TSO in HW And disable TCP Chimney Or vice-versa!

Make sure to apply changes in:

DRIVER ADVANCED CONFIGURATIONS (which controls the 10 GigE Adapter HW) REGISTRY EDITOR

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

145

Evaluating 10 GigE Performance

The following distinctions need to be made to evaluate the 10 GigE adapter impact on the applications
TSO cards without proper OS support dont yield more than 3-4Gbps Throughput tests stress vs Transaction/s tests use different HW features You must distinguish TX performance vs RX performance TCP and UDP traffic are handled very differently in the HW y y TCP Checksum Offload and Large Segment Offload provide different functionalities.

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

146

Preliminary Tests
Maximum Throughput Is ~3.2 Gbps

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

147

Why?
Only 1 core is dealing with TCP/IP processing The OS doesnt know that the Adapter is TOE capable so it doesnt really use it A lot of memory copies between user space and kernel space Is the card plugged in the p gg PCIe x8? Solution:
Make sure that the OS uses TCP offloading in Hardware Enable Large Segment Offload Enable TCP/IP distribution to all available cores

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

148

Engaging More Than 1 Core: Receive Side Scaling (RSS)

Core 1 Core 2 Core 3 Core 4

CPU 1

CPU 2

RSS Capable NIC p Incoming Packets

Hash Interrupt Logic

Receive FIFOs

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

149

Processing W/O Large Segment Offload (LSO)

Core 1

application user I/O library

MSS

Data record
100%
MSS Data MSS MSS record MSS MSS

% CORE overhead

OS kernel device driver

MSS

TCP/IP
MSS

transport processing Intermediate buffer copies

40% 20%

I/O Adapter

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

150

Large Send Offload

V1 (Scalable Networking Pack):
allows the TCP layer to build a TCP message up to 64KB and send it in one call down the stack through the device driver. Segmentation is handled by the Network Adapter

V2 (Windows 2008):
allows the TCP layer to build a TCP message up to 256KB and send it in one call down the stack through the device driver. Segmentation is handled by the Network Adapter Supports IPv4/IPv6

Main Benefit: Reduces CPU utilization Key Use Cases: Large I/O applications such as Storage, backup, and ERP.
TECDCT-3873_c2 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

151

Processing With Large Segment Offload (LSO)

Core 1

application user I/O library

Data record
100%

% CORE overhead

OS kernel device driver

MSS MSS MSS MSS

Intermediate buffer copies

20%

I/O Adapter

TCP/IP

MSS

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

152

Registry Configuration (for Windows 2003)

In Windows 2008 Just Use netsh cmd

Set to 1

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

153

W/o LSO Checksum Offload Alone Doesnt Do Much

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

154

LSO Combined With TCP Offload Is Better

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

155

But the RX Side Cannot Keep Up With the TX Hence You Need to Enable SACK in HW

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

156

Enabling Jumbo Frames

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

157

Network Design Considerations for HPC Parallel Applications

Latency has an important effect on messaging between the nodes What matters is end-to-end application messaging, as opposed to network latency There is a big difference between regular TCP/IP stack, TCP/IP with TCP offloading (TOE), and RDMA (Remote Direct Memory Access) accelerated
Speedup

Key measurement factor: speedup Relevant protocols:

Message Passing Interface (MPI) MPI over Ethernet uses TCP

10 GigE with iWARP RDMA GigE Number of Nodes

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

158

Sources of Overhead in Datacenter Servers

Sources of Overhead in Server Networking Transport Processing Intermediate Buffer Copying Application Context Switches CPU Overhead 40% 20% 40% User Solutions for Overhead in Server Networking Transport Offload Engine (TOE) Moves Transport processor cycles to the NIC o es C / protocol stac buffer copies Moves TCP/IP p otoco stack bu e cop es from system memory to the NIC memory RDMA Eliminates intermediate and application buffer copies (memory bandwidth consumption) Kernel
TCP/IP

CPU

System Memory App Buffer

TCP Buffer Pool

s/w h/w
NIC

Kernel Bypass direct user-level access to hardware Dramatically reduces application context switches

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

159

iWARP
The Internet Wide Area RDMA Protocol (iWARP) is an Internet Engineering Task Force (IETF) update of the RDMA Consortium's RDMA over TCP standard. iWARP is a superset of the Virtual Interface Architecture that permits zero-copy transmission over legacy TCP. It may be thought of as the features of InfiniBand (IB) applied to Ethernet. http://www.openfabrics.org/ http://www openfabrics org/ runs on top of iWARP

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

160

Latency on the Switch

Latency of modular 1GbE sw can be quite high (>20us)
store & fwd many hops line serialization

Nexus 5k TOR fixes this

Cut through implementation 3.2 us latency

A single frame dropped in a sw or adapter causes significant impact on performance:

TCP NACK delayed by up to 125us with NIC with interrupt throttling enabled TCP window shortened (burst of traffic, lost of frame, slowdown most of traffic and brings burst again..etc.. for financial customer such as trading companies may suffer)
TECDCT-3873_c2 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

161

Latency Fundamentals
What matters is the application-to-application latency and jitter
Driver/Kernel software Adapter Network components Kernel NIC NIC Kernel Application
Data Packet

Application

Latencies of 1GbE switches can be quite high (>20ms)

Store and forward Multiple hops Line serialization delay

N5000 Switch 3.2 s 3 2 s

Nexus 5000 Solution

Cut through implementation 3.2 ms latency (port to port with features turned on) End to End latency

Protocol processing dominates latency

Nexus 5000 in latency optimized application

162

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Latency

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

163

What Helps Where

Checksum Offload RSS LSO iWARP

TX RX CPU % TCP workload Transactions/s TCP workload Throughput UDP throughput Latency

+ + ++ + + + ++ ++
(4)

+++ +++ + +
(1,2) +++

+
Cisco Public

+++
164

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Break Demo: Nexus1kv

165

How Much Traffic Can a Single VM Generate? (TX, aka Virtual-to-Physical)

A single VM can drive alone more than 1Gbps worth of bandwidth (in the tested configuration a single VM can drive up to 3.8 Gbps of traffic) Even if the Guest OS displays Network Adapter of 1Gbps, the performance is not gated at 1 Gbps!

ESX 3.5 U2 CPU 2 x dual core Xeon5140 Guest OS Windows 2003 R2 SP2 Memory 8 GB

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

166

Traffic Flow VM-to-Physical (V2P) With Quad-GigE Cards

Catalyst C t l t 6500

GigE 2/13 - 16

vNIC vNIC vNIC vNIC

TECDCT-3873_c2 TECDCT-3873_c2

vmnic0 vmnic1 vmnic2 vmnic3

1 GigE 1 GigE 1 GigE 1 GigE Te4/3

10 GigE
client 1

Traffic Flow VM-to-Physical (V2P) With 10 GigE Cards

ESX 1 ESX 1

Te4/4 GigE 2/17 - 20 10 GigE

client 2

1 GigE

1 GigE 1 GigE 1 GigE

vmnic1 vmnic2 vmnic3

vmnic0

vNIC

ESX 2
2009 Cisco Systems, Inc. All rights reserved. Cisco Public

167

Catalyst C t l t 6500

vNIC vNIC vNIC vNIC

2009 Cisco Systems, Inc. All rights reserved.

vmnic0 vmnic1 vmnic2 vmnic3

10 GigE Te4/3

10 GigE
client 1

Te4/4 GigE 2/17 - 20 1 GigE 1 GigE 1 GigE 1 GigE

vmnic1 vmnic2 vmnic3

client 2

10 GigE

vmnic0

vNIC

ESX 2
Cisco Public

168

How Much Traffic Can 4 VMs Generate? (TX aka V2P)

A typical configuration made of 4 VMs could drive up to ~8-9 Gbps worth of traffic, which means that an ESX server equipped with a Quad-GigE adapter throttles the VMs performance of a typical ESX implementation

ESX 3.5 U2 CPU 2 x dual core Xeon5140 Guest OS Windows 2003 R2 SP2 Memory 8 GB

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

169

P2V (RX) vs V2P (RX) Throughput With 10 GigE NICs to 4 VMs

RX: ~4.3 Gbps TX: ~ 8Gbps

ESX 3.5 U2 CPU 2 x dual core Xeon5140 Guest OS Windows 2003 R2 SP2 Memory 8 GB

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

170

How to Improve VMWARE Performance in RX?

VMWARE Solution: Netqueue What is Netqueue? Netqueue is the equivalent of Receive Side Scaling in VMWARE, i.e. it helps distributing incoming traffic to the available cores.

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

171

P2V With Netqueue Disabled

Maximum Throughput is ~3.9 Gbps CPU goes all the way to 100%

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

172

P2V With Netqueue Enabled

Maximum Throughput is ~4.2Gbps All cores are below 100%

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

173

Tips for Tuning VMWARE With 10 GigE (Courtesy of Intel)

Set CPU affinity for virtual machines:
In the vCenter (VC) console select a Virtual Machine (VM), right click and select Edit Settings. In the VM Properties dialog box select the Resources tab. Click on the Advanced g p g CPU object and in the right pane of the window click on the Run on processor(s) radio button. Select a processor core for the VM to run on and click OK to close the window. Repeat for all VMs.

Turn on NetQueue support in ESX

On the vCenter management console select the host to be configured and click the configuration tab. In the Software box select and open Advanced Settings. Find the parameter labeled VMkernel.Boot.netNetqueue and check the box to enable it. Reboot the system.

Load the driver with multiple queue support:

After the driver rpm has been installed and the machine has rebooted, the driver will have initialized in its default single queue mode. Unload the driver with the command vmkload_mod u ixgbe. Reload the driver and set it in multiple queue mode with the command vmkload_mod ixgbe VMDQ=X,X InterruptType=2,2 (where the comma separated parameter value is repeated for each physical port installed in the machine which uses the ixgbe driver and the value X is the desired number of queues. For a configuration with 8 VMs I use VMDQ=9. This gives 8 dedicated Rx queues to assign to the VMs plus the default TxRx queue.
174

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Break Demo: Nexus1kv

175

I/O Consolidation
I/O consolidation supports all three types of traffic onto a single network Servers have a common interface adapter that supports all three types of traffic

IPC: Inter Process Communication

176

Data Center Ethernet Summary

Feature
Priority-based Flow Control (PFC) CoS Based BW Management Congestion Notification (BCN/QCN) Data Center Bridging g g Exchange L2 Multi-path for Unicast & Multicast

Benefit
Provides class of service flow control Ability to support control. storage traffic Grouping classes of traffic into Service Lanes IEEE 802.1Qaz, CoS based Enhanced Transmission End to End Congestion Management for L2 network

Auto negotiation Auto-negotiation for Enhanced Ethernet capabilities DCBX (Switch to NIC) Eliminate Spanning Tree for L2 topologies Utilize full Bi-Sectional bandwidth with ECMP

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

177

SAN Switching

Presentation_ID

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

178

Complete Your Online Session Evaluation

Give us your feedback and you could win fabulous prizes. Winners announced daily. Receive 20 Passport points for each session evaluation you complete. Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.

Dont forget to activate your Cisco Live Virtual account for access to all session material, communities, and on-demand and live activities throughout the year. Activate your account at the Cisco booth in the World of Solutions or visit www.ciscolive.com.
179

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

180

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

181

TECDCT-3873_c2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

182

Data Center Power Session

TECDCT-3873

Presentation_ID

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Agenda
Infrastructure Design (Mauricio Arregoces) LAN Switching Analysis (Maurizio Portolani)
Recap on Current Trends and Past Best Practices New Layer 2 Technologies Fabric Extender Deep dive and Design with virtual Port Channeling

Break Demos: vPC, OTV (Maurizio Portolani)

Designs with Server Virtualization 10 Gigabit Ethernet to the Server

Break Demo: Nexus1kv (Maurizio Portolani) Blade Servers (Carlos Pereira)

Blade Switching LAN Blade Switching SAN Storage Networking with VMware ESX / vSphere

Break
Unified IO Unified Compute System

Demo: UCS (Carlos Pereira)

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Blade Switching - LAN

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

What Are Going to Talk About ?

Cisco Catalyst Virtual Blade Switches (VBS)

Cisco Bl d S it h Ci Blade Switch Entry Level GE switch GE VBS

Cisco Part Number CBS30x0

OEM

CBS31x0X

10G VBS

CBS31x0G

x = 1 for IBM, 2 for HP and 3 for Dell

Setting the Stage

On this session of the Data Center techtorial the maximum number of enclosures per rack will be considered for the SAN design calculations.

Nevertheless, power and cooling constraints needs to be considered on a case by case basis when implementing blade servers.

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Design with Pass-Thru Module and Modular Access Switch

Cable density Rack example:
Four Enclosures Per Rack Up to 16 servers per enclosure 32 1GE LOMs + 2 Management interfaces per enclosure. 136 available 1GE access ports Requires structured cabling to support 136 1GE connections/rack

Modular Access Switches

Blade Server Rack

Pair of Cat 6513 Nexus 7010

Supports up to pp p 28 enclosures (7 racks) 10 x 6748 cards per each switch

19 enclosures (5 racks) 7 x 48 1GE cards + 1 x 10GE card per each switch 42 enclosures (11 racks) 15 x 48 1GE cards + 1 x 10GE card per each switch
Cisco Public

Nexus 7018
TECDCT-3873

Gigabit Ethernet Connections

2009 Cisco Systems, Inc. All rights reserved.

Design with Pass-Thru Module and Modular Access Switch

Does this look Manageable?

How to you find and replace a bad cable?

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Design with Pass-Thru Module and Top of the Rack (TOR) Switches
High Cable density within the rack High capacity uplinks p provide g p y p aggregation layer connectivity Rack example:
Up to Four blade enclosures/rack Up to 128 cables for server traffic Up to 8 cables for Server management p pp Up to four rack switches support local blade servers Additional switch for server management Requires up to 136 cables within the rack 10 GigE Uplinks Aggregation Layer

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Design with Blade Switches

Reduces cables within the rack High capacity uplinks p provide g p y p aggregation layer connectivity Rack example:
Up to four enclosures per rack Two switches per enclosure Either 8 GE or one 10GE uplink per switch Between 8 and 64 cables/fibers per rack Reduces number of cables within the rack but increases the number of uplinks compared to ToR solution Based on cable cost 10GE from Blade Switch is a better option.
TECDCT-3873 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Aggregation Layer

10 GigE Or GE Uplinks

Design with Virtual Blade Switches (VBS)

Removes Cables from Rack High capacity uplinks p provide g p y p aggregation layer connectivity Rack example:
Up to Four blade enclosures/rack Up to 64 Servers per rack Two switches per enclosure p One/Two Virtual Blade Switch per rack Two or Four 10GE uplinks per Rack Reduces number of Access Layer switches by factor of 8 Allows for local Rack traffic to stay within the Rack
TECDCT-3873 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Aggregation Layer

10 GigE Or GE Uplinks

Cisco Virtual Blade Switch (VBS)

Physical Connections

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Cisco Virtual Blade Switch (VBS)

Logical View

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Multiple Deployment Options for Customers

Caters to Different Customer Needs
Benefits Common Scenario Single Virtual Blade switch per rack Entire rack can be deployed with as little as two 10 GE uplinks or two GE Etherchannels Allows for Active/Active NIC teams Creates a single router for entire rack if deploying L3 on the edge d l i h d Keeps Rack traffic in the Rack Design Considerations Ring is limited to 64 Gbps May cause Oversubscription
TECDCT-3873 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Multiple Deployment Options for Customers

Caters to Different Customer Needs
Benefits Sepa ate Separate VBS d de Left/Right s tc es S divide e t/ g t switches More resilient Provides more Ring capacity since two rings per Rack Design Considerations Requires more Uplinks per Rack Servers can not form A/A NIC teams

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Multiple Deployment Options for Customers

Caters to Different Customer Needs
Benefits Allows for 4 NICs per server p Can Active/Active Team all 4 NICs More Server Bandwidth Useful for highly virtualized environments Design Considerations Creates smaller Rings Requires more Uplinks May Increase Traffic on each Ring

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Additional Options
By combining above three scenarios, the user can:
Deploy up to 8 switches per enclosure Build smaller Rings with fewer Switches Split VBS between LAN on Motherboard (LOM) and Daughter Card Ethernet NICs Split VBS across racks Connect unused uplinks to other Devices such as additional Rack Servers or Appliances such as storage

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Plug-and-Play Designs with VBS and Nexus1000v

1 Add or replace a VBS Switch to the Cluster 2 Switch config and code automatically propagated Virtual Ethernet Module

3 Add a blade Server 4 Its always booted from the same LUN

Virtual Ethernet Module

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Cisco Virtual Blade Switch (VBS)

Scalability

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Proper VBS Ring Configuration

Each offer a full ring, could be built with 1 meter cables, and looks similar But: Certain designs could lead to a split ring if an entire enclosure is powered down
For No example, in the 4 enclosure example, if enclosure 3 had p power removed y would end up with two rings, one made up of you p g , p the switches in enclosures 1 and 2, and one made up of the switches in enclosure 4. This, at a minimum would leave each VBS contending for the same IP address, and remote switch No Yes management would become difficult The Yes examples also have a better chance of maintaining connectivity for the servers in the event a ring does get completely split due to multiple faults
Cable Lengths are 0.5, 1.0 and 3.0 Meter. The 1.0 Meter cable ships standard
No Yes

ENC 4

ENC 3 No Yes

ENC 2

ENC 1

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Virtual Blade Switch Across Racks

VBS cables are limited to max of 3 meters Insure that switches are not isolated in case of failure of switch or enclosure May require cutting holes through side walls of Cabinets/Racks

~2 FT

~2 FT
TECDCT-3873 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Deployment Scenario without vPC / VSS

Straight forward configuration
Ensure uplinks are spread across switches and enclosures If using EtherChannel (EC), make sure members are not in same enclosure By using RSTP and EC, recovery time on failure is minimized Make Master Switch (and Alternate) are not Uplink switches Use FlexLinks if STP is not desired

Aggregation Layer

Core Layer

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Deployment Scenario without vPC / VSS

Access Layer (Virtual Blade Switch) Aggregation Layer

Single Switch / Node (for Spanning Tree or Layer 3 or Management)

Spanning-Tree Blocking

Deployment Scenario without vPC / VSS

Aggregation Layer Access Layer (Virtual Blade Switch)

Single Switch / Node (for Spanning Tree or Layer 3 or Management)

Spanning-Tree Blocking

Deployment Example
Switch Numbering 1 to 8, left to Right, Top to Bottom Master Switch is Member 1 Alternate Masters will be 3,5,7 Uplink Switches will be Members 2,4,6,8
1 2

10 GE ECs from 2,4 and 6,8 will be used

3 4

RSTP will be used User Data VLANs will be interleaved

7 5 6

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Configuration Commands
switch 1 priority 15 switch 3 priority 14 switch 5 priority 13 switch 7 priority 12 spanning-tree mode rapid-pvst vlan 1-10 state active g g g g g interface range gig1/0/1 gig1/0/16 switchport access vlan xx Assign ports to VLANs Sets Sw 1 to pri master Sets Sw 3 to sec master Sets Sw 5 to 3rd master Sets Sw 7 to 4th Master Enables Rapid STP Configures VLANs

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Configuration Commands
interface range ten2/0/1, ten4/0/1 switchport mode trunk switchport trunk allowed vlans 1 10 1-10 channel group 1 mode active interface range ten6/0/1, ten8/0/1 switchport mode trunk switchport trunk allowed vlans 1-10 channel group 2 mode active interface po1 spanning-tree vlan 1 3 5 7 9 port-priority 0 i t l 1,3,5,7,9 t i it spanning-tree vlan 2,4,6,8,10 port-priority 16 interface po2 spanning-tree vlan 1,3,5,7,9 port-priority 16 spanning-tree vlan 2,4,6,8,10 port-priority 0

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Deployment Scenario with vPC / VSS

Access Layer (Virtual Blade Switch) Aggregation Layer Nexus vPC, Cat6k VSS

Single Switch / Node (for Spanning Tree or Layer 3 or Management)

All Links Forwarding

Deployment Scenario with vPC / VSS

Aggregation Layer (Nexus vPC) Access Layer (Virtual Blade Switch)

Single Switch / Node (for Spanning Tree or Layer 3 or Management) All Links Forwarding
TECDCT-3873 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Deployment Scenario with vPC / VSS

Physical View

VBS 1

VBS 2

VBS 3

VBS 4

VSS vPC

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Deployment Scenario with vPC / VSS

Logical View

VBS 1

VBS 2

VSS vPC

VBS 3
TECDCT-3873 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

VBS 4
30

Rules to Live by for EC / MCEC

1. Split links across line cards on Catalyst 6500 / Nexus 7000 side prevents against Line Card Outage 2. Split across pair of Catalyst 6500 or across pair of Nexus 7000 prevents against aggregation switch outage 3. Split links across members on blade side if using VBS - prevents against blade switch outage 4. Split links across Blade Enclosures if possible prevents against enclosure outage 5. Split VLANs 5 S lit VLAN across f l d b l for load balancing prevents idle EC i t idl ECs. 6. Chose appropriate EC load balancing algorithm example: Blade servers generally have even number MAC addresses. Consider the hashing algorithms enhancements with MCEC 7. Last but Not least, monitor your ECs - Only way to know if you need more BW or Better MCEC load balance
TECDCT-3873 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Further Points to Consider on Layer 2:

When is Layer 2 Adjacency Required?
Clustering: applications often execute on multiple servers clustered to appear as a single device. Common for HA, Load Balancing and High Performance computing requirements. MS-Windows Advanced Server Clustering Linux Beowulf or proprietary clustering (HPC)

NIC teaming software typically req ires la er t picall requires layer 2 adjacency

AFT SFT ALB

Blade NIC Teaming Configurations

Network Fault Tolerance (NFT)
Typical referred to as Active/Standby Used when server sees two or more upstream switches NIC connectivity is PREDEFINED with built-in switches and may limit NIC configuration options

Transmit Load Balancing (TLB)

Primary adapter transmit and receives Secondary adapters transmit only Rarely used

Switch Assisted Load Balancing (SLB)

Often referred to as Active/Active Server must see same switch on all member NICs GEC/802.3ad Increased throughput Available with VBS switches

Active Standby
TECDCT-3873 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Blade Server Access Topologies

Different Uplinks Possibilities
V-Topology U-Topology Trunk-Failover Topology

Very Popular Topology Some Bandwidth not available

Not as Popular

Maximum Bandwidth available Needs NIC Teaming

Cisco Public

Layer 2 Trunk Failover

Typical Blade Network Topologies
L3 Switches

Link State Group 1

Cisco Blade Switches

Link State Group 1

Blade Server Chassis

FEATURE Map Uplink EtherChannel to downlink ports (Link State Group) If all uplinks fail, instantly shutdown downlink ports Server gets notified and starts using backup NIC/switch
TECDCT-3873 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Blade Server Chassis

CUSTOMER BENEFIT Higher Resiliency / Availability Reduce STP Complexity
35

Flexlink Overview
Achieve Layer 2 resiliency without using STP Access switches have backup links to Aggregation switches p gg g Target of sub-100msec convergence upon forwarding link failover Convergence time independent of #vlans and #mac-addresses Interrupt based link-detection for Flexlink ports. Link-Down detected at a 24msec poll. No STP instance for Flexlink ports. Forwarding on all vlans on the <up> flexlink port occurs with a single update operation low cost.

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

(Mac address Move Notification) MMN Overview

Achieve near sub-100 msec downtime for the downstream traffic too, upon flexlink switchover. Lightweight protocol : Send a MMN packet to [(Vlan1, Mac1, Mac2..) (Vlan2, Mac1, Mac2..) ..] distribution network. Receiver parses the MMN packet and learns or moves the contained mac-addresses. Alternatively, it can flush the mac-address table for the vlans vlans. Receiver forwards packet to other switches.

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Flexlink Preemption
Flexlink enhanced to :
provide flexibility in choosing FWD link, optimizing available bandwidth utilization

User can configure Fl li k pair when previous FWD li k comes b k up : U fi Flexlink i h i link back
Current FWD link continues Preemption mode Off Previous FWD link preempts the current and begins FWD instead Preemption mode Forced Higher bandwidth interface preempts the other and goes FWD Preemption mode Bandwidth

Note: By default, flexlink preemption mode is OFF default When configuring preemption delay:
user can specify a preemption delay time (0 to 300 sec) default preemption delay is 35 secs

Preemption Delay Time :

Once the switch identifies a Flexlink preemption case, it waits an amount of <preemption delay> seconds before preempting the currently FWD Flexlink interface.
TECDCT-3873 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Flexlink Configuration Commands

CBS3120-VBS-TOP#config t Enter configuration commands, one per line. CBS3120-VBS-TOP(config)#int po1 CBS3120-VBS-TOP(config-if)#switchport backup int po 2 CBS3120-VBS-TOP(config-if)# CBS3120-VBS-TOP#show interface switchport backup detail End with CNTL/Z.

Switch Backup Interface Pairs:

Active Interface

Backup Interface

State

-----------------------------------------------------------------------Port-channel1 Preemption Mode Port-channel2 : off p/ p Active Up/Backup Down

Bandwidth : 20000000 Kbit (Po1), 10000000 Kbit (Po2) Mac Address Move Update Vlan : auto

CBS3120-VBS-TOP#

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Management Screenshot
Topology View

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Management Screenshot
Front Panel View

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Blade Switching - SAN

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

What are Going to Talk About ?

Cisco MDS 4Gb Fibre Channel Blade Switches

16 internal copper 1/2/4-Gbps Fibre Channel connecting to blade servers through blade chassis backplane Up to 8 SFP uplinks Offered in 12-port and 24-port configurations via port licensing

14 internal copper 1/2/4-Gbps Fibre Channel connecting to blade servers through blade chassis backplane Up to 6 SFP uplinks Offered in 10-port and 20-port configurations via port licensing

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Virtual Storage Area Network Deployment

Consolidation of SAN islands
Increased utilization of fabric ports with JustIn-Time provisioning p g
Department A

SAN Islands

Deployment of large fabrics

Dividing a large fabric in smaller VSANs Disruptive events isolated per VSAN RBAC for administrative tasks Zoning is independent per VSAN
Department B Department C

Advanced traffic management

Defining the paths for each VSAN VSANs may share the same EISL Cost effective on WAN links
Department A Department B Department C

Virtual SANs (VSANs)

Resilient SAN Extension Standard solution (ANSI T11 FC-FS-2 section 10)
TECDCT-3873 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Understanding VSANs (or Virtual Fabrics)

Production SAN
FC FC FC
FC FC

Tape SAN
FC
FC

Test SAN
FC
FC

SAN A DomainID=1 DomainID=7

SAN B DomainID=2 DomainID=8

SAN C DomainID=3

SAN D DomainID=4

SAN E DomainID=5

SAN F Domain ID=6

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

VSAN Technology
The Virtual SANs Feature Consists of Two Primary Functions:
Hardware-based isolation of tagged traffic belonging to different VSANs Create independent instance of Fibre Channel services for each newly created VSAN VSAN services include:
Fibre Channel Services for Blue VSAN Fibre Channel Services for Red VSAN

VSAN Header Is Removed at Egress Point Cisco MDS 9000 Family with VSAN Service Enhanced ISL (EISL) Trunk Carries Tagged Traffic from Multiple VSANs VSAN Header Is Added at Ingress Point Indicating Membership No Special Support Required by End Nodes

Trunking E_Port (TE_Port)

Fibre Channel Services for Blue VSAN Fibre Channel Services for Red VSAN

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Enhanced vs. Basic Zoning

Basic Zoning Enhanced Zoning Enhanced Advantages

Administrators can make simultaneous configuration changes

All configuration changes fi ti h are made within a single session. Switch locks entire fabric to implement change

One configuration session for entire fabric to ensure consistency within fabric

If a zone is a member References to the zone are Reduced payload size as the zone is referenced. of multiple zonesets , used by the zonesets as The size is more required once you define the an instance is pronounced with bigger zone. created per zoneset. database Default zone policy is defined per switch.
Enforces and exchanges default zone setting throughout the fabric Fabric-wide policy enforcement reduces troubleshooting time.

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Enhanced vs. Basic Zoning

Basic Zoning Enhanced Zoning Enhanced Advantages

Managing switch provides combined status about activation. Will not identify a failure switch. To distribute zoneset must re-activate the same zoneset. During D i a merge MDS specific types can be misunderstood by noncisco switches.

Retrieves th activation R ti the ti ti results and the nature of the problem from each remote switch.

Enhanced error E h d reporting reduces troubleshooting process.

Implements changes to the This avoids hardware zoning database and changes for hard distributes it without zoning in the switches. activation. Provides a vendor ID along with a vendor-specific type value to uniquely identify a member type Unique Vendor type

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Inter VSAN Routing

Similar to L3 interconnection between VLAN Allows sharing of centralized storage services such as tape libraries and disks across VSANswithout merging separate fabrics (VSANs) Network address translation allow interconnection of VSANs without a predefined itho t addressing schema
Engineering VSAN_1 VSAN-Specific Disk

IVR

IVR Tape VSAN_4 (Access via IVR) HR VSAN_3

Marketing VSAN_2

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Quick Review 1
VSANs enable creation of multiple virtual fabrics on top of a consolidated physical SAN infrastructure; Enhanced Zoning recommended and helpful from both scalability and troubleshooting standpoints; Inter VSAN Routing (IVR) required when selective communication between shared devices on distinct fabrics is needed.

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

N-Port ID Virtualization (NPIV)

Mechanism to assign multiple N_Port_IDs to a single N_Port Allows all the Access control, Zoning, Port Security ( g y (PSM) be ) implemented on application level So far, multiple N_Port_IDs are allocated in the same VSAN

Application Server

FC Switch

Email I/O N_Port_ID 1 Web I/O N_Port_ID 2 File Services I/O N_Port_ID 3 F_Port

Web

File Services

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

NPIV Configuration Example

NPIV Is Enabled Switchwide with the C th Command: d
npiv enable Notice that a F-port supports multiple logins

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

NPIV Usage Examples

Virtual Machine Aggregation Intelligent Pass-thru

NPV Edge Switch

NP_Port

NPIV enabled HBA F_Port F_Port

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

N-Port Virtualizer (NPV)

Enabling Large-Scale Blade Server Deployments
Deployment Model - FC Switch Mode
Blade System
Blade N Blade 2 Blade 1 E-Port
TECDCT-3873

Deployment Model HBA Mode

Blade System Blade System
Blade N Blade 2 Blade 1

Blade System

Blade Switch configured as NPV (i.e. HBA mode)

Blade N

Blade 2

Blade 1

FC Switch

NPV

NPV enables large scale Blade Server deployments by:

E-Port

N-Port

SAN

- Reducing Domain ID usage - Addressing switch interop issues - Simplifying management

F-Port

Storage

Blade Switch Attribute FC Switch Mode (E-Port) One per FC Blade Switch Yes Medium Deployment Model # of Domain IDs Used Interoperability issues with multi-vendor Core SAN switch Level of management coordination between Server and SAN Administrators
Cisco Public

HBA Mode (N-Port) None (uses Domain ID of core switch) No Low

NPV is also available on the MDS 9124 & 9134 Fabric Switches
2009 Cisco Systems, Inc. All rights reserved.

N-Port Virtualizater (NPV): An Overview

NPV-Core Switch (MDS or 3rd party switch with NPIV support)
FC FC

Solves the domain-id explosion problem l i bl

10.1.1 10 1 1

20.2.1 20 2 1

F-port

NP-port
MDS 9124 MDS 9134

Can have multiple uplinks, on different VSANs (port channel and trunking in a later release)

Up to 100 NPV switches

Cisco MDS in a Blade Chassis

Blade Server 1 Blade Server 2 Blade Server n

server port (F)

10.5.2
FC

10.5.7 20.5.1
Initiator (no FL ports)

NPV Device
Uses the same domain(s) as the NPV-core switch(es)

Target

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

NPV FLOGI/FDISC Login Process

NPV Core Switch

When NP port comes up on a NPV edge switch, it first FLOGI and PLOGI into the core to register into the FC Name Server End Devices connected on NPV edge switch does FLOGI but NPV switch converts FLOGI to FDISC command, creating a virtual PWWN for the end device and allowing to login using the physical NP port. All I/O of end device will always flow through same NP port
F

NP P1 NP P2

NPV Edge Switch

P4 = vP2

P5 = vP3

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

FlexAttach
Because Even Physical Devices Move How it works ?
Based on WWN NAT of Servers Server s WWN
Bl lade 1

Blade Server

.
NPV

Re eplaced Blade B

Bl lade N

Key Benefit:
Flexibility for Server Mobility - Adds, Moves and Changes Eliminates need for SAN and server team to coordinate changes Two modes: Lock identity to port Identity follows physical PWWN

No Blade Switch Config Change

Flex Attach

No Switch Zoning Change g

SAN

No Array Configuration Change

Storage

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Flex Attach
Example
Creation of virtual PWWN (vPWWN) on NPV switch F-port Zone vPWWN to storage LUN masking is done on vPWWN Can swap Server or replace physical HBA
No need for zoning modification No LUN masking change required

Automatic link to new PWWN

no manual re-linking to new PWWN is needed

Before: switch 1

After: switch 2

1
FC1/1 vPWWN1 FC1/6 vPWWN1

PWWN 1

Server 1
TECDCT-3873 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Server 1 58

PWWN 2

Whats Coming:
Enhanced Blade Switch Resiliency
F-Port Port Channel
F-Port Port Channel Blade N Blade 2 Blade 1

Core Director Storage g

Blade System

SAN

N-Port

F-Port

F-Port Trunking
Core Director Storage Blade System
Blade N VSAN 1

F-Port Trunking

Blade 2 Blade 1

VSAN 2

SAN

VSAN 3 N-Port F-Port

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Whats Coming:
F-Port Trunking for the End-Host / Storage
Hardware-based isolation of tagged traffic belonging to different VSANs up to Servers or Storage Devices
Non VSANTrunking capable end node
Fibre Channel Services for Blue VSAN Fibre Channel Services for Red VSAN

VSAN-trunking-enabled drivers required for end nodes (for example, Hosts)

VSAN Header removed at egress point

Trunking E_Port

Implementation example: traffic tagged in Host depending on the VM

VSAN-trunking support required by end nodes VSAN Hdader added by the HBA driver indicating Virtual Machine membership
TECDCT-3873 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Enhanced ISL (EISL) Trunk carries tagged traffic from multiple VSANs Trunking E_Port
Fibre Channel Services for Blue VSAN Fibre Channel Services for Red VSAN

Trunking F_Port
60

Quick Review 2
NPIV standard mechanism enabling F-port (switches and HBAs) virtualization NPV allows a FC switch to work on HBA mode. The switch behaves like a proxy of WWN and doesnt consume a Domain ID, enhancing SAN scalability (mainly on blade environments) Flex-Attach adds flexibility to server mobility allowing the server FC identity to follow the physical pWWN (for blades and rack mount servers) F-port port-channel on NPV scenarios, the ability to bundle p p , y multiple physical ports in to 1 logical link F-port trunking extend VSAN tagging to the N_Port to F_Port connection. Works between switches together with NPV. For host, needs VSAN support on the HBA and allows per-VM VSAN allocation.
TECDCT-3873 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

SAN Design: Initial Considerations

Requirements
-F Fan-out maintenance t i t - Dual physical fabrics

SAN Design

Factors to Consider
- Topologies - Bandwidth reservation - Networking / gear capacity
TECDCT-3873 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Parameters
- Number of end devices - Speed variation

SAN Design: Initial Considerations

Requirements:
1. Fan-out ratio needs to be maintained to have a predictable and scalable SAN. 2. Dual physical fabric (Fabric A, Fabric B) are identical

Parameters:
1. Number of end-devices (servers, storage and tape) 2. Speed: Majority of end device connection speeds will be primarily 1G, 2G or 4G

Factors to consider:
1. Required topology (core-edge, colapsed core-edge, edge-coreedge, etc.) 2. Bandwidth reservation versus Oversubscription 3. Networking capacity needed (VSANs, ISL, fabric logins, zones, NPIV instances, etc.)
TECDCT-3873 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

SAN Design: Initial Considerations

Requirements:
1. Fan-out ratio needs to be maintained to have a predictable and scalable SAN. 2. Dual physical fabric (Fabric A, Fabric B) are identical

Parameters:
1. Number of end-devices (servers, storage and tape) 2. Speed: Majority of end device connection speeds will be primarily 1G, 2G or 4G

SAN FAN-OUT Ratio: What is That ?

Fan-out ratio represents the number of hosts that are connected to a single port g p of a storage array
SAN Fan-out needs to be maintained on the whole SAN design SAN Fan-out defines the SAN oversubscription. Its fixed on blades! Oversubscription is introduced at multiple points Switches are rarely the bottleneck in SAN implementations Must consider oversubscription during a network failure event Remember, all traffic flows towards targets main bottlenecks

Disk Oversubscription Disk do not sustain wire-rate I/O with realistic I/O mixtures. A major vendor promotes 12:1 host:disk fan-out.

Tape O Oversubscription Low sustained I/O rates. All technologies currently have max theoretical native transfer rate << wire-speed FC (LTO, SDLT, etc)

ISL Oversubscription Typical oversubscription in two-tier design can approach 8:1, some even higher

8:1 O.S. (common)

Host Oversubscription Most hosts suffer from PCI bus limitations, OS, and application limitations thereby limiting maximum I/O and bandwidth rate

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

SAN FAN-OUT How to Calculate ?

Simple math with physical hosts only. Clusters, VMs and LUN/server ratio should be considered too. Three variables not to be exceeded:
Port queue depth: both storage and HBA; IOPS: to avoid port saturation Throughput: port speed versus sustained traffic.

Design by the maximum values leads to over engineered and underutilized SANs. Oversubscription helps to achieve best cost / performance ratio. Rule of thumb: limit the number of hosts per storage port based on the array fan-out. For instance, 10:1 or 12:1.

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

SAN Design: Initial Considerations

Premises:
1. Fan-out ratio needs to be maintained to have a predictable and scalable SAN. 2. Dual physical fabric (Fabric A, Fabric B) are identical

Parameters:
1. Number of end-devices (servers, storage and tape) 2. Speed: Majority of end device connection speeds will be primarily 1G, 2G or 4G

Cisco MDS 9000 Line Cards Detailed

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

SAN Design: Initial Considerations

Premises:
1. Fan-out ratio needs to be maintained to have a predictable and scalable SAN. 2. Dual physical fabric (Fabric A, Fabric B) are identical

Parameters:
1. Number of end-devices (servers, storage and tape) 2. Speed: Majority of end device connection speeds will be primarily 1G, 2G or 4G

Core-Edge
Traditional SAN design for growing SANs High density di t Hi h d it directors i in core and, on the edge:
Unified IO (FCoE) switches [1]; Directors [2] , Fabric Switches [3] or Blade switches [ 4 ]
A A B B A B A B A B

Predictable P di bl performance f Scalable growth up to core and ISL capacity

[1]

[2]

[3]

[4]
70

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

SAN Design: Initial Considerations

Premises:
1. Fan-out ratio needs to be maintained to have a predictable and scalable SAN. 2. Dual physical fabric (Fabric A, Fabric B) are identical

Parameters:
1. Number of end-devices (servers, storage and tape) 2. Speed: Majority of end device connection speeds will be primarily 1G, 2G or 4G

Cisco MDS 9000 Capacity

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Blade Servers Fibre Channel Integration Challenges

Domain ID scalability limits the maximum number of FC switches to 239 devices per VSAN as per the Standard Standard.
Resellers today do not support more than ~40-75 devices EMC: 40 domains HP: 40 domains

Being able to remove and reinsert a new blade without having to change Zoning Configurations VMWare Integration (discussed later on
this Techtorial)

Up to 8 FC switches per rack (4 Blade Servers x 2) 8 bits 8 bits

Area

8 bits
Device
73

Switch Topology Model

Switch Domain
Cisco Public

IBM BladeCenter H Core-Edge Design:

Fibre Channel with Cisco MDS FC Switch Module
MDS as FC blade switch
(1100+ usable ports per fabric, all VSAN enabled) BladeCenter Design i Bl d C t H D i using 2 x 4G ISL per blade switch. Oversubscription can be reduced for individual blade centers by adding additional ISLs as needed. VSAN supported.
[A] Storage Ports 240 (2G dedicated): or [B] Storage Ports (4G dedicated): 120 Host Ports (4G HBAs): 1152 ISL Oversubscription (ports): 7.5 : 1 Disk Oversubscription (ports): 10 : 1 Core-Edge Design 8.4 : 1 Oversubscription:
9 racks 56 dual attached servers/rack Each Cisco MDS FC blade switch: 2 ISL to core @ 4G 14 host ports @ 4G 7.5:1 oversubscription 504 total servers 1008 HBAs Storage Array 10:1 oversubscription (fan-out)

[A] 120 storage ports @ 2G or [B] 60 storage ports @ 4G 72 ISL to edge @ 4G

NPIV

NPV + Flex Attach

Cisco MDS 9513 as SAN Aggregation Directors

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Blade Server FC Attached Storage:

Fibre Channel with Cisco MDS FC Switch Module HP cClass
MDS as FC blade switch
(1200+ usable ports per fabric, all VSAN enabled)
Storage Array 10:1 oversubscription (fan-out)

Blade S Bl d Server Design using 2 D i i x 4G ISL per blade switch. Oversubscription can be reduced for individual blade centers by adding additional ISLs as needed.
[A] Storage Ports 240 (2G dedicated): or [B] Storage Ports (4G dedicated): 120 Host Ports (4G HBAs): 1152 ISL Oversubscription (ports): 8 : 1 Disk Oversubscription (ports): 10 : 1 Core-Edge Design 9.6 : 1 Oversubscription:

[A] 120 storage ports @ 2G or [B] 60 storage ports @ 4G 72 ISL to edge @ 4G

NPIV

NPV + Flex Attach

Cisco MDS 9513 as SAN Aggregation Directors

9 racks 64 dual attached servers/rack

Each Cisco MDS FC blade switch (02 switches per HP c-Class enclosure): 2 ISL to core @ 4G 16 host ports per HP c-Class enclosure @ 4G 8:1 oversubscription

576 total servers 1152 HBAs

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Storage Networking with VMWare ESX

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Virtual Machines (VM) @

Storage Networking with Blade Servers
Virtual Machines pose new requirements for SANs Switching Performance Support complex, unpredictable, dynamically changing traffic patterns d i ll h i ffi Provide fabric scalability for higher workload Differentiate Quality of Service on a per VM basis Deployment, Management, Security Create flexible and isolated SAN sections, support management Access ti t tA Control Support performance monitoring, trending, and capacity planning up to each VM Allow VM mobility without compromising security
TECDCT-3873 2009 Cisco Systems, Inc. All rights reserved. Cisco Public
Virtualized Servers Virtualized Servers Virtualized Servers Virtualized Servers

Virtual Machines

Fabric

Storage Array Storage Array

Tier 1

Tier 2

Tier 3

VMware ESX Storage Options

FC iSCSI/NFS DAS

SCSI

iSCSI is popular in SMB market DAS is not popular because it prohibits VMotion

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Virtual Servers Share a Physical HBA

A zone includes the physical hba and the storage array
Virtual Serve ers

Access control is demanded to storage array LUN masking and mapping, it is based on the physical HBA pWWN and it is the same for all VMs The hypervisor is in charge of the mapping, errors may be disastrous

Hypervisor

MDS9124e Mapping

Storage Array (SAN A or B) (LUN Mapping and Masking)

pWWN-P

Zone
TECDCT-3873

Single Login on a Single Point-to-Point Connection

FC Name Server
80

Virtual Server Using NPIV and Storage Device Mapping

Virtual HBAs can be zoned individually LUN masking and mapping is based on the virtual HBA pWWN of each VMs Very safe with respect to configuration errors Only supports RDM Available since ESX 3.5
MDS9124e
Mapping Mapping Mapping Mapping

Virtual Servers s

Hypervisor

Storage Array (SAN A or B)

To pWWN-1
pWWN-1 pWWN-2 pWWN-3 pWWN-4

To pWWN-2 pWWN-P pWWN-1 pWWN-2 pWWN-3 pWWN-4 To pWWN-3 To pWWN-4

pWWN-P

Multiple Logins on a Single Point-to-Point Connection

FC Name Server
81

QoS for Individual Virtual Machines

Zone-Based QoS: VM-1 has Priority; VM-2 and any Additional Traffic has Lower Priority VM-1 Reports Better Performances than VM-2
Virtual Machines s

VM-1

VM-2

Congested Link Cisco MDS 9124e Multilayer Fabric Switch Cisco MDS 9000 Multilayer Fabric Switch Storage Array (SAN A or B)
Storage Array

H Hypervisor

pWWN-V2 Low Priority

QoS
FC

Q QoS IVR
pWWN-T

pWWN-V1 High Priority

pWWN-P

Low-Priority Traffic

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Routing Virtual Machines Across VSANs Using NPIV and IVR

Targets are in different VSANs Inter VSAN Routing Zoning:
Virtual Machines s

VM-1

VM-2

IVR-Zone-P includes the physical devices pWWN-P and pWWN-T IVR-Zone-Vx includes the virtual machine x and the physical target only p y g y

LUN Mapping and Masking

Each LUN x is exposed to the physical initiator pWWN-P and to virtual machine x pWWN-Vx only
Raw Device Mapping Raw Device Mapping

IVR-Zone-V2 MDS9124e

MDS9000 VSAN-20
FC

ESX Hypervisor X

pWWN-V2

pWWN-T2 WWN T2
FC

IVR-Zone-V1
pWWN-V1

VSAN-1
HW
pWWN-P
FC

IVR VSAN-1 VSAN-10 VSAN-20

VSAN-10

IVR

pWWN-T1

IVR-Zone-P
TECDCT-3873 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

VMotion LUN Migration without NPIV

VM1 VM2 VM3 VM1 VM2 VM3 VM1 VM2 VM3

Standard HBAs
WWPN

W S-X9 01 6

1 STAT S U

1/2 Gbp s FC Mod ule

All configuration parameters are based on the World Wide Port Name (WWPN) of the physical HBA

All LUNs must be exposed to p every server to ensure disk access during live migration (single zone)

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

VMotion LUN Migration with NPIV

VM1 VM2 VM3

HBAs with NPIV

WWPN1 WWPN2 WWPN3
W S-X9 01 6 1 STAT S U 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

1/2 Gbp s FC Mod ule

No need to reconfigure zoning or LUN masking Dynamically reprovision VMs without impact to existing infrastructure
FC

Centralized management of VMs and resources Redeploy VMs and support live migration

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

VMotion: Switch Name Server - Before

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

VMotion: Switch Name Server - After

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Virtualization Infrastructure and Management

Example: Mapping vCenter Data Centers to VSAN
Data Center Red
Storage Array

Frame Tagged on Trunk Cisco MDS 9124e Blade Data CenterGreen VSAN-10 VSAN-20 VSAN-30 Cisco MDS 9000 Family

VSAN-10 VSAN 10

Storage Array

VSAN-20

Storage Array

VSAN VSAN30

Data Center Yellow Administrator Privileges Admininistrative Team Red Green Yellow
TECDCT-3873 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Virtual Machines Data Center Red Data Center Green Data Center Yellow

Storage Network VSAN-10 VSAN-20 VSAN-30

Storage Array Red Array Green Array Yellow

In Summary: Blade Servers w/ Cisco LAN & SAN

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Unified IO (FCoE)

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

What Is Data Center Ethernet (DCE)?

Data Center Ethernet is an architectural collection of Ethernet extensions designed to improve Ethernet networking and management in the Data Center.

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Whats the Difference Between DCE, CEE and DCB ?

Nothing! All 03 acronyms describe the same thing, meaning the architectural collection of Ethernet extensions (based on open standards) Cisco has co-authored many of the standards associated and is focused on providing a standards-based solution for a Unified Fabric in the data center The IEEE has decided to use the term DCB (Data Center Bridging) to DCB describe these extensions to the industry. http://www.ieee802.org/1/pages/dcbridges.html

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Data Center Ethernet Standards and Features

Overview
Feature
Priority-based Flow Control (PFC) - 802.1Qbb Enhanced Transmission Selection - 802.1Qaz Congestion Notification (BCN/QCN) - 802.1Qau Data Center Bridging p y g Capability Exchange Protocol - 802.1AB (LLDP) L2 Multi-path for Unicast & Multicast Lossless Service

Benefit
Provides class of service flow control. Ability to support storage traffic Grouping classes of traffic into Service Lanes IEEE 802.1Qaz, CoS based Enhanced Transmission End to End Congestion Management for L2 network

Auto-negotiation for Enhanced Ethernet capabilities DCBX Eliminate Spanning Tree for L2 topologies Utilize full Bi-Sectional bandwidth with ECMP Provides ability to transport various traffic types (e.g. Storage, RDMA)

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Data Center Ethernet Standards and Features

Overview
Feature
Priority-based Flow Control (PFC) - 802.1Qbb

Benefit
Provides class of service flow control. Ability to support storage traffic

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Data Center Ethernet Features - PFC

Priority-Based Flow Control (PFC)

Enables lossless Fabrics for each class of service PAUSE sent per virtual lane when buffers limit exceeded Network resources are partitioned between VLs (E.g. input buffer and output queue) The switch behavior is negotiable per VL
TECDCT-3873 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Data Center Ethernet Standards and Features

Overview
Feature
Priority-based Flow Control (PFC) - 802.1Qbb Enhanced Transmission Selection - 802.1Qaz

Benefit
Provides class of service flow control. Ability to support storage traffic Grouping classes of traffic into Service Lanes IEEE 802.1Qaz, CoS based Enhanced Transmission

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Data Center Ethernet Features - ETS

Enhanced Transmission Selection (ETS)

Enables Intelligent sharing of bandwidth between traffic classes control of bandwidth Being Standardized in IEEE 802.1Qaz Also known as Priority Grouping
TECDCT-3873 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Data Center Ethernet Standards and Features

Overview
Feature
Priority-based Flow Control (PFC) - 802.1Qbb Enhanced Transmission Selection - 802.1Qaz Congestion Notification (BCN/QCN) - 802.1Qau

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Data Center Ethernet Features

Congestion Management

Moves congestion out of the core to avoid congestion spreading Allows End-to-End congestion management Standards track in 802.1Qau
TECDCT-3873 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Data Center Ethernet Standards and Features

Auto-negotiation for Enhanced Ethernet capabilities DCBX

TECDCT-3873

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

100

Data Center Ethernet Features - DCBX

Data Center Bridging Capability eXchange Protocol

Data Center Ethernet

Handshaking Negotiation for: CoS BW Management Class Based Flow Control Congestion Management (BCN/QCN) Application (user_priority usage) Logical Link Down
TECDCT-3873 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

101

Data Center Ethernet Standards and Features

Auto-negotiation for Enhanced Ethernet capabilities DCBX Eliminate Spanning Tree for L2 topologies Utilize full Bi-Sectional bandwidth with ECMP

TECDCT-3873

Cisco Public

102

Data Center Ethernet Features L2MP

Layer 2 Multi-Pathing
Phase 1
LAN

Phase 2
LAN
Virtual Switch

Phase 3
LAN

MAC A

MAC B Active-Active

L2 ECMP vPC

MAC A

MAC B

L2 ECMP

We are here
Eliminates STP on Uplink Bridge Ports Allows Multiple Active Uplinks Switch to Network Prevents Loops by Pinning a MAC Address to Only One Port Completely Transparent to Next Hop Switch Virtual Switch retains physical switches independent control and data planes Virtual port channel mechanism is transparent to hosts or switches connected to the virtual switch STP as fail-safe mechanism to prevent loops even in the case of control plane failure Uses ISIS based topology Eliminates STP from L2 domain Preferred path selection TRILL is the work in progress standard

TECDCT-3873

Cisco Public

103

Data Center Ethernet Standards and Features

TECDCT-3873

Cisco Public

104

Virtual Links
An Example
Up to 8 VLs per physical link Ability to support QoS queues within the lanes
VL2 - No Drop Service - Storage
DCE
CNA

VL1 LAN Service LAN/IP LAN/IP Gateway

DCE
CNA

VL1 VL2 VL3

DCE
CNA

Campus Core/ Internet

VL3 D l Delayed Drop S i - IPC dD Service

Storage Gateway

Storage Area Network

105

TECDCT-3873

Cisco Public

Fibre Channel over Ethernet How it Works

Direct mapping of Fibre Channel over Ethernet
FC-4 FC-3 FC-2 FC-1 FC-0 FC-4
CRC C
SOF

FC-3 FC-2
FCoE Mapping MAC PHY

FC Frame
Ethernet Payload

EOF

Ethernet Header

Ethernet FCS

(a) Protocol Layers

(b) Frame Encapsulation

Leverages standards-based extensions to Ethernet (DCE) to provide reliable I/O delivery

FCoE Traffic Other Networking Traffic

106

FCoE Enablers
10Gbps Ethernet Lossless Ethernet
Matches the lossless behavior guaranteed in FC by B2B credits

Ethernet jumbo frames

Max FC frame payload = 2112 bytes
Normal et e et frame, et e type = FCoE o a ethernet a e, ethertype Co Same as a physical FC frame
Ethernet Header FCoE Header FC Header CRC EOF

FC Payload

Control information: version, ordered sets (SOF, EOF)

FCS
107

Encapsulation Technologies

Operating System / Applications SCSI Layer FCP iSCSI TCP IP FC

1, 2, 4, 8, 10 Gbps

FCP FCIP TCP IP

FCP iFCP TCP IP

FCP

SRP

FCoE IB
10, 20 Gbps

Ethernet
1, 10 . . . Gbps

TECDCT-3873

Cisco Public

108

Encapsulation Technologies
FCP layer is untouched
OS / Applications SCSI Layer FCP

Allows same management tools for Fibre Channel Allows same Fibre Channel drivers Allows same Multipathing software

FCoE E. Ethernet
1, 10 . . . Gbps

Simplifies certifications with ith OSMs Evolution rather than Revolution

TECDCT-3873

Cisco Public

109

Unified I/O (FCoE) Why ?

Fewer CNAs (Converged Network adapters) instead of NICs, HBAs and HCAs Limited number of interfaces for Blade Servers
FC HBA FC HBA NIC NIC NIC NIC HCA
TECDCT-3873 2009 Cisco Systems, Inc. All rights reserved.

FC Traffic FC Traffic
CNA

LAN Traffic LAN Traffic Mgmt Traffic Backup Traffic IPC Traffic
Cisco Public

CNA

All traffic goes over 10GE

110

Unified I/O: What Changes on the Network ?

Today: LAN
Management

SAN A
FC HBA FC HBA NIC

SAN B

Core switches

NIC

Access Top of the Rack switches Servers

111

Unified I/O: Just the Access Layer

Today Unified I/O LAN
Management

Unified I/O
SAN B
Reduction of server adapters Fewer Cables Simplification of access layer & cabling Gateway free implementation - fits in installed base of existing LAN and SAN L2 Multipathing Access Distribution Lower TCO Investment Protection (LANs and SANs) Consistent Operational Model One set of ToR Switches

SAN A

FCoE Switch

112

Converged Network Adapters (CNA)

TECDCT-3873

Cisco Public

113

CNA View on Host

10 GE/FCoE

Cisco ASIC

10 GE

PCIe Bus

TECDCT-3873

Cisco Public

114

CNA View on VMware ESX Fibre Channel

Emulex

Qlogic

TECDCT-3873

Cisco Public

115

CNA View on VMware ESX 10 GE

116

Disk Management
Storage is zoned to FC initiator f h t i iti t of host.

TECDCT-3873

Cisco Public

117

Example: CNA Configuration

TECDCT-3873

Cisco Public

118

Common SAN/LAN Architecture

Administrative Boundaries
LAN SAN A SAN B

Network Admin
Login: Net_admin Password: abc1234

SAN Admin
Login: SAN_admin Password: xyz6789

119

Common SAN/LAN Architecture

Administrative Boundaries
LAN SAN A SAN B

Network Admin
Login: Net_admin Password: abc1234

SAN Admin
Login: SAN_admin Password: xyz6789

NX5000

CNA CNA

DataCenter Ethernet with FCoE

Ethernet

FC
120

Unified IO Deployment - Unified IO

Core
Storage Arrays
Fabric A

SAN Fabric
Fabric B

L3
Aggregation

N7K

N7K MDS9500 MDS9500 MDS9500 MDS9500

L3 L2

C6K N7K N7K

C6K
SAN Edge A SAN Edge B

Access

LAN Access

N5K
A

N5K
VF_Ports
B

N5K
VN_Ports
D

N5K
E

CNA Enet FC FCoE

Unified IO Server Farm Pod Environment

Converged Edge Infrastructure: Unified/IO using ToR at the edge, and CNA at the hosts ToR 10GE Unified/IO Server Environments Leverage Ethernet and Storage Clouds to reach traditional LAN/SAN services
TECDCT-3873 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

121

Unified IO Farm - Phase 1:

vPC @ Aggregation
Core
Storage Arrays
Fabric A

SAN Fabric
Fabric B

L3
Aggregation

N7K
4 4

N7K MDS9500 MDS9500 C6K

SAN Edge A SAN Edge B

MDS9500 MDS9500

L3 L2

C6K

N7K

Access
4
LAN Access

4 4 4 4

N5K
A

N5K
B

N5K
D

N5K
E

CNA Enet FC FCoE

Unified IO Server Farm using vPC at Aggregation LAN cloud

Access Switches remain as single logical instance Storage connectivity is unchanged

TECDCT-3873

Cisco Public

122

Unified IO Farm - Phase 2:

vPC @ Aggregation and Access
Core
Storage Arrays
Fabric A

SAN Fabric
Fabric B

L3
Aggregation

N7K
4 4

N7K MDS9500 MDS9500 C6K

SAN Edge A SAN Edge B

MDS9500 MDS9500

L3 L2

C6K

N7Ks

Access
4
LAN Access

N5Ks
A B D

N5Ks
E

CNA Enet FC FCoE

Unified IO Server Farm using vPC at Aggregation LAN cloud

Access Switches provide vPC for LAN connectivity Storage connectivity is unchanged (different physical paths for SAN Fabric A and B)

TECDCT-3873

Cisco Public

123

Nexus 5000 on the Aggregation Layer VE Interfaces are NOT Supported so Far

TECDCT-3873

Cisco Public

124

Cisco Unified Computing System (UCS)

TECDCT-3873

Cisco Public

125

The Unified Computing Journey

Unified Fabric
Wire once infrastructure Low-latency lossless Virtualization aware
TECDCT-3873

Unified Virtual Machines

VN - Link Application Mobility

Unified Computing
Consolidated Fabric & I/O Stateless Vn-tagging Management

Data Center 3.0

Business service focused Resilient Distributed Standardsbased

Cisco Public

126

Unified Computing Building Blocks

Unified Fabric Introduced with the Cisco Nexus Series
Physical
Wire once infrastructure (Nexus 5000) Fewer switches, adapters, cables

Ethernet

Fibre Channel

TECDCT-3873

Cisco Public

127

Unified Computing Building Blocks

Unified Fabric Introduced with the Cisco Nexus Series
Physical
Wire once infrastructure (Nexus 5000) Fewer switches, adapters, cables

Virtual

Ethernet

Fibre Channel

Virtual
VN-Link (Nexus 1000v) Manage virtual the same as physical

Physical

TECDCT-3873

Cisco Public

128

Unified Computing Building Blocks

Unified Fabric Introduced with the Cisco Nexus Series
Physical
Wire once infrastructure (Nexus 5000) Fewer switches, adapters, cables

Virtual

Ethernet

Fibre Channel

Virtual
VN-Link (Nexus 1000v) Manage virtual the same as physical

Scale Physical
Fabric Extender (Nexus 2000) Scale without increasing points of management

TECDCT-3873

Cisco Public

129

Cisco Unified Computing Solution

Mgmt Server

Embed management Unify fabrics Optimize virtualization p Remove unnecessary

switches, adapters, management modules

Mgmt Server

Less than 1/2 the support pp infrastructure for a given workload

TECDCT-3873

Cisco Public

130 130

Cisco Unified Computing Solution

Mgmt system A single Server that encompasses:

Network: Unified fabric Compute: Industry standard x86 Storage: Access options Virtualization optimized

Unified management model

Dynamic resource provisioning

Efficient Scale
Cisco network scale & services Fewer servers with more memory

131 131

Cisco Unified Computing Solution

TECDCT-3873

Cisco Public

132 132

Cisco Unified Computing Solution

Single, scalable integrated system Network + compute virtualization Dynamic resource provisioning
Mgmt LAN SAN A SAN B

TECDCT-3873

Cisco Public

133 133

UCS Building Blocks

UCS Manager Embedded manages entire system UCS Fabric Interconnect 20 Port 10Gb FCoE 40 Port 10Gb FCoE UCS Fabric Extender Remote line card

UCS Blade Server Chassis Flexible bay configurations UCS Blade Server Industry-standard architecture UCS Virtual Adapters Choice of multiple adapters
TECDCT-3873 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

134

Cisco UCS and Nexus Technology

UCS Components
UCS Manager Embedded Manages entire system UCS Fabric Interconnect 20 Port 10Gb FCoE 40 Port 10Gb FCoE UCS Fabric Extender Remote line card UCS Blade Server Chassis Flexible bay configurations UCS Blade Server Industry-standard architecture UCS Virtual Adapters Choice of multiple adapters
TECDCT-3873 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Nexus Products

Nexus 5000 Unified Fabric Nexus 2000 Fabric Extender

VN-Link Nexus 1000V CNAs with FCoE

135

Cisco Unified Computing System (UCS) Physical

TECDCT-3873

Cisco Public

136

Cisco Unified Computing System (UCS) Physical

SAN LAN MGMT SAN

Top of Rack Interconnect

G A G G G S S G G Fabric Interconnect G Fabric A Interconnect G G G

(40 or 20 10GE ports) + (2 or 1 GEM uplink slots) l t )

Chassis
Blade Enclosure Fabric I R Extender
x8 x8

Up to 8 half width blades or 4 full width blades

I
x8

R
x8

Fabric Extender

Fabric Extender
Host to uplink traffic engineering
M Adapter B P Adapter B P Adapter

Up to 80Gb Flexible bandwidth allocation

X X x86 Computer

X X X X x86 Computer

Adapter 3 options
Cisco Virtualized adapter Compatibility CNAs (Emulex and QLogic) Native FC + Intel Oplin Intel Oplin - (10GE only)

Compute Blade (Half slot)

TECDCT-3873

Compute Blade (Full slot)

Cisco Public

Compute Blade
137

Enclosure, Fabric Switch, and Blades (Front)

Redundant, Hot Swap Power Supply Redundant, Hot Swap Fan 1U or 2U Fabric Switch

Half width server blade Up to eight per enclosure Hot Swap SAS drive (Optional)

Full width server blade Up to four per enclosure Mix blade types

6U Enclosure

Redundant, Hot Swap Power Supply

Cisco Public

138

Rear View of Enclosure and Fabric Switch

10GigE Ports Expansion Bay

Redundant Hot Swap Fan Module

Redundant Fabric Extender

Fan Handle

TECDCT-3873

Cisco Public

139

UCS Adapters Options

Virtualization
Virtual Machine Aware: Virtualization and Consolidation

Compatibility
Existing Driver Stacks

Cost
Proven 10GbE Technology T h l

140

UCS Adapters: Interface Views

10 GigE Backplane interfaces to IOMs Physical Interfaces vHBAs & vNICs will be bound to these physical interface Intel Oplin will not have HBA component. Could run FCoE software stack

TECDCT-3873

Cisco Public

141

UCS Adapters: CLI View

Required to scope to correct chassis/blade/adaptor
rtp 6100 B# rtp-6100-B# scope adapter 1/5/1

142

UCS Adapters: vHBA Detail Identification

Vendor

Provisioned WWN and if bound to Profile

TECDCT-3873

Cisco Public

143

UCS Adapters: Ethernet vNIC Details

Ethernet stats

TECDCT-3873

Cisco Public

144

Cisco Unified Computing System (UCS) Logical

TECDCT-3873

Cisco Public

145

Unified Computing Key Value Propositions:

Drivers for Use Cases
Hardware State Abstraction Service Profiles

Unified Fabric - FCOE

Virtualized Adapter

Expanded Memory Server

Unified Management

TECDCT-3873

Cisco Public

146

Server Attributes / Configuration Points 1/3

Server
Identity (UUID) Adapters Number Type: FC, Ethernet Identity Characteristics Firmware Revisions Configuration settings

TECDCT-3873

Cisco Public

147

Server Attributes / Configuration Points 2/3

Server
Identity (UUID) Adapters Number Type: FC, Ethernet Identity Characteristics Firmware Revisions Configuration settings Uplinks

Network
LAN settings vLAN, QoS, etc SAN settings vSAN Firmware Revisions

TECDCT-3873

Cisco Public

148

Server Attributes / Configuration Points 3/3

Storage
Optional Disk usage SAN settings LUNs Persistent Binding Firmware Revisions Adapters

Server
Identity (UUID) Uplinks

Network
LAN settings vLAN, QoS, etc SAN settings vSAN Firmware Revisions

Number Type: FC, Ethernet Identity Characteristics Firmware Revisions Configuration settings

TECDCT-3873

Cisco Public

149

Traditional Server Deployment

Storage Administrator: Configure LUN access C fi
Masking, binding, boot LUN

Server Administrator: Configure management LAN C fi t Upgrade firmware versions

Chassis, BMC, BIOS, adapters

Network Administrator: Configure LAN access

Uplinks, VLANs

Configure switch
Zoning, VSANs, QoS

Configure policies
QoS, ACLs

Configure BIOS settings Configure NIC settings Configure HBA settings Configure boot parameters

Perform tasks for each server Inhibits pay-as-you-grow incremental deployment pay-as-you-grow
Needs admin coordination every time May incur downtime during deployments

Complex server replacement, upgrade, migration process

Most of these tasks need to be performed for replacement server

TECDCT-3873

Cisco Public

150

UCS Server Profile Opt-in Choices

Fixed Attributes
Processors Memory Capacity Bandwidth Capacity

Definable Attributes
Disks & usage Network Type: FC, Ethernet, etc. Number Identity Characteristics LAN settings vLAN, QoS, etc SAN settings g LUNs vSAN & Persistent Binding Firmware Revisions Configuration settings Identity (BIOS)

TECDCT-3873

Cisco Public

151

UCS Service Profile

Storage
Optional Disk usage SAN settings LUNs Persistent Binding SAN settings vSAN Firmware Revisions

Server
Identity (UUID) Adapters Number Type: FC, Ethernet y Identity Characteristics Firmware Revisions Configuration settings

Network
Uplinks LAN settings vLAN QoS etc Firmware Revisions

TECDCT-3873

Cisco Public

152

UCS Service Profiles

Hardware State Abstraction
LAN Connectivity OS & Application SAN Connectivity

MAC Address NIC Firmware NIC Settings

Drive Controller F/W Drive Firmware

UUID BIOS Firmware BIOS Settings Boot Order

BMC Firmware

WWN Address HBA Firmware HBA Settings

State abstracted from hardware

Separate firmware, addresses, and parameter settings from server hardware Separate access port settings from physical ports Physical servers become interchangeable hardware components Easy to move OS & applications across server hardware
TECDCT-3873 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

154

Dont I Get this Already from VMware?

Hypervisors & Hardware State
Virtual Machine Virtual Machine Virtual Machine Virtual Machine Virtual Machine Virtual Machine

Server Virtualization (VMware, Xen, HyperV, etc.) HYPERVISOR

MAC Address NIC Firmware NIC Settings

Drive Controller F/W Drive Firmware

UUID BIOS Firmware BIOS Settings Boot Order

BMC Firmware

WWN Address HBA Firmware HBA Settings

Hardware State Virtualization

Server virtualization & hardware state abstraction are independent of each other Hypervisor (or OS) is unaware of underlying hardware state abstraction

TECDCT-3873

Cisco Public

155

UCS Service Profiles

End to End Configure of Full UCS HW Stack

TECDCT-3873

Cisco Public

156

Server Upgrades:
Within a UCS

Server Name: finance-01 UUID: 56 4d cd 3f 59 5b 61 MAC : 08:00:69:02:01:FC WWN: 5080020000075740 Boot Order: SAN, LAN Firmware: xx.yy.zz

Old Server

New Server

Disassociate server profile from old server Associate server profile to new server Old server can be retired or re-purposed

TECDCT-3873

Cisco Public

157

Server Upgrades:
Across UCS Instances
Old UCS System New UCS System

Server Name: finance-01 UUID: 56 4d cd 3ffinance-01 Server Name: 59 5b 61 MAC :Server4d cd 3ffinance-01 08:00:69:02:01:FC UUID: 56 Name: 59 5b 61 WWN: 5080020000075740 5b 61 MAC : 08:00:69:02:01:FC UUID: 56 4d cd 3f 59 Boot Order: 08:00:69:02:01:FC WWN: 5080020000075740 MAC : SAN, LAN Firmware: xx.yy.zz LAN Boot Order: SAN, WWN: 5080020000075740 Firmware: xx.yy.zz LAN Boot Order: SAN, Firmware: xx.yy.zz

1. Disassociate server profiles from servers in old UCS System

2. 3. Migrate server profiles to new UCS system Associate server profiles to hardware in new UCS system

TECDCT-3873

Cisco Public

158

Server Upgrades:
Across UCS Instances
Old System New System
Server Name: finance-01 UUID: 56 Name:3f 59 5b 61 Server 4d cd finance-01 Server Name: finance-01 MAC : 08:00:69:02:01:FC 61 UUID: 56 4d cd 3f3f 59 5b 61 UUID: 56 4d cd 59 5b WWN: 5080020000075740 MAC : 08:00:69:02:01:FC MAC : 08:00:69:02:01:FC Boot Order: SAN, LAN WWN: 5080020000075740 WWN: 5080020000075740 Firmware: xx.yy.zz LAN Boot Order: SAN, LAN Boot Order: SAN, Firmware: xx.yy.zz Firmware: xx.yy.zz

Disassociate server profiles from servers in old UCS system

2. Migrate server profiles to new UCS system

3. Associate server profiles to hardware in new UCS system

TECDCT-3873

Cisco Public

159

Server Upgrades:
Across UCS Instances
Old System New System
Server Name: finance-01 UUID: 56 4d cd 3ffinance-01 Server Name: 59 5b 61 MAC :Server4d cd 3ffinance-01 08:00:69:02:01:FC UUID: 56 Name: 59 5b 61 WWN: 5080020000075740 5b 61 MAC : 08:00:69:02:01:FC UUID: 56 4d cd 3f 59 Boot Order: 08:00:69:02:01:FC WWN: 5080020000075740 MAC : SAN, LAN Firmware: xx.yy.zz LAN Boot Order: SAN, WWN: 5080020000075740 Firmware: xx.yy.zz LAN Boot Order: SAN SAN, Firmware: xx.yy.zz

1. 2. 3.

Disassociate server profiles from servers in old UCS system Migrate server profiles to new UCS system Associate server profiles to hardware in new UCS system
160

TECDCT-3873

Cisco Public

Dynamic Server Provisioning

Profiles for Web Servers
Server Name: web-server-01 UUID: 56 4d cd 3f 59 5b 61 MAC : 08:00:69:02:01:FC WWN: 5080020000075740 Boot Order: SAN, LAN Firmware: xx yy zz xx.yy.zz

Profiles for App Servers

Server Name: app-server-01 UUID: 65 d4 cd f3 59 5b 16 MAC : 08:00:69:02:01:16 WWN: 5080020000076789 Boot Order: SAN, LAN Firmware: xx.yy.zz yy

Apply appropriate profile to provision a specific server type Same hardware can dynamically be deployed as different server types No need to purchase custom configured servers for specific applications

TECDCT-3873

Cisco Public

161

Server Profiles - Reduce Overall Server CAPEX

Todays Deployment:
Provisioned for peak capacity Spare node per workload
Oracle RAC
Blade Blade Blade Blade Blade

With Server Profiles:

Resources provisioned as needed Same availability with fewer spares
Oracle RAC
Blade Blade Blade

Web Servers
Blade Blade Blade Blade Blade Blade

Web Servers
Blade Blade Blade Blade Blade

VMware
Blade Blade Blade

VMware
Blade Blade Blade Blade Blade

Blade Blade

Burst Capacity
Blade

Total Servers: 18
Normal use Burst Capacity Spare Hot Spare

HA Spare
Blade

Blade

Total Servers: 14

TECDCT-3873

Cisco Public

162

Unified Computing Key Value Propositions:

Drivers for Use Cases
Hardware State Abstraction Service Profiles

Unified Fabric - FCOE

Virtualized Adapter

Expanded Memory Server

Unified Management

TECDCT-3873

Cisco Public

165

Unified Fabric
SAN IPC LAN

Todays Approach
All fabric types have switches in each chassis Repackaged switches Complex to manage Blade-chassis configuration Bl d h i fi ti dependency Costly Small network domain Blade Chassis

Unified Fabric
Fewer switches Fewer adapters 10GE/FCoE Blade Blade Blade All I/O types available in each chassis
10GE & FCoE LAN, SAN, IPC

Blade

Easier to manage Blades can work with any chassis Small network domain

TECDCT-3873

Cisco Public

166

Backplane and Fabric Extender

High performance backplane
2x 40G total bandwidth per half slot - 8 lanes of 10G (half-slot) (full slot) - 16 lanes of 10G (full-slot) Redundant data and management paths Support auto discover of all component

Compute blade

Backplane

Fabric Extender

Compute blade Compute blade Compute blade Compute blade

Fabric extender
Manage oversubscription 2:1 to 8:1 FCoE from blade to fabric switch Customizable bandwidth
Compute blade Compute blade Compute blade
TECDCT-3873 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

167

UCS: Overall System (Rear)

Uplinks

TECDCT-3873

Cisco Public

168

Unified Computing Key Value Propositions:

Drivers for Use Cases
Hardware State Abstraction Service Profiles

Unified Fabric - FCOE

Virtualized Adapter

Expanded Memory Server

Unified Management

TECDCT-3873

Cisco Public

169

What Is SR-IOV About?

Single Root IO Virtualization (SR-IOV) allows virtualizing the 10 GigE link (via the PCI-Express bus) into multiple virtual links. SR-IOV is a PCI-Sig standard In other words you can create multiple vmnics each with its own bandwidth allocation This could be Nexus 1000v
Server VM1 vnic VM2 vnic Virtual Switch VM3 vnic VM4 vnic Virtual Switch

vmnic

This is what SR-IOV enables

pNIC: 10 Gbps

TECDCT-3873

Cisco Public

170

UCS Adapters Options

Virtualization
VM I/O Virtualization and Consolidation

Compatibility
Existing Driver Stacks

Cost
Free SAN Access for Any Ethernet Equipped Host

10GbE/FCoE

Eth

QP FC FC C Eth

vNICs
0 1 2 3 127

10GbE

Software FCoE

PCIe x16 PCIe Bus

171

TECDCT-3873

Cisco Public

Cisco UCS Virtualized Adapter

Virtualized adapter designed for both single-OS and VM-based deployments P id mobility, isolation, and management from the network bilit i l ti d tf th t k Provides
Secure Transparent to hosts
10GE/FCoE
MAC 0 MAC 1

Cut-through architecture High Performance

2x 10Gb Low latency High BW IPC support
User Defineable vNICs
Eth
0

FC
1

SCSI
2

FC
3

Eth
127

PCIe x16
172

Enables Passthrough I/O

Guest OS
Device Driver

vNICs appear as independent PCIe devices

Centrally manageable and configurable Hot-pluggable Virtual NICs

Virtualization Layer Host

Device Manager

Different types: Eth, FC, SCSI, IPC

IOMMU

Guest drives device directly

vNIC

173

Cisco UCS Virtualized Adapter

FC Eth

Eth FC

SCSI Eth

Eth

IPC

NIV Adapter

OS Compute Blade

Network Interface Virtualization adapter Vary nature and number of PCIe interfaces
Ethernet, FC, SCSI, IPC

Up to 128 different PCIe devices

Hot-pluggable - only appear when defined PCI-Sig IOV compliant

Part of Server Array fabric

174

User Configuration Example

Global System Class Definitions Class Name
COS Value Drop/No-Drop Strict Priority Bandwidth/Weight

FC
3 No-Drop No 1 (20%)

Gold
1 Drop No 3 (60%)

Ethernet BE
0 Drop No 1 (20%)

FC Traffic

High Priority Ethernet

Best Effort Ethernet

vNIC1
Class Rate Burst FC 4000 300

vNIC2
FC 4000 400

vNIC3
Eth. BE 5000 100 Class Rate Burst

vNIC1
Gold 600 100

vNIC2
Eth. BE 4000 300

TECDCT-3873

Cisco Public

175

Unified Computing Key Value Propositions:

Drivers for Use Cases
Hardware State Abstraction Service Profiles

Unified Fabric - FCOE

Virtualized Adapter

Expanded Memory Server

Unified Management

TECDCT-3873

Cisco Public

178

Blade Overview
Half-width blade Common Attributes 2 x Intel Nehalem-EP processors 2 x SAS hard drives (optional) Blade Service processor Blade and HDD hot plug support Stateless blade design 4x the memory Full-width blade 10Gb CNA and 10GbE adapter options Differences Half-width blade 4x memory 12 x DIMM slots 2x I/O bandwidth
TECDCT-3873 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Full-width blade 48 x DIMM slots

2 x dual port adapters
179

1 x dual port adapter

Full-Height Blade
2 socket Nehalem-EP blade 48 x DDR3 DIMMs 2 x Mezzanine Cards 2 x Hot swap disk drives Up to 384GB per 2 socket blade Transparent to OS and applications

Reduced server costs

Purchase fewer servers for memory bound applications memory-bound

Reduced power and cooling costs Reduced software costs

180

Expanded Memory Blade

Nehalem-EP Processor

Physical View
8GB

Nehalem-EP Processor

Logical View
Slot 23

Slot 23 Slot 22 Slot 21 Slot 20 Slot 19 Slot 18 Slot 17 Slot 16 Slot 15 Slot 14 Slot 13 Slot Sl t 12 Slot 11 Slot 10 Slot 9 Slot 8 Slot 7 Slot 6 Slot 5 Slot 4 Slot 3 Slot 2 Slot 1 Slot 0
Cisco Public

8GB

32GB
Channel 2
(red)

Slot 22 Slot 21 Slot 20 Slot 19

8GB

Channel 2
(red)

8GB

32GB

Slot 18 Slot 17 Slot 16 Slot 15

8GB

32GB
Channel 1
(blue)

Slot 14 Slot 13 Slot Sl t 12 Slot 11

8GB

Channel 1
(blue)
8GB 8GB

8GB

32GB

Slot 10 Slot 9 Slot 8 Slot 7

8GB

32GB
Channel 0
(green)

Slot 6 Slot 5 Slot 4 Slot 3

8GB

Channel 0
(green)

8GB

32GB

Slot 2 Slot 1 Slot 0

8GB

TECDCT-3873

181

Expanded Memory Architecture

Increases number of DIMMs the system can use
Makes the system think it has high-capacity DIMMs when using larger number of lower-capacity lower capacity DIMMs

TECDCT-3873

Cisco Public

182

I/O

CPU Memory

TECDCT-3873

Cisco Public

183

Unified Computing Key Value Propositions:

Drivers for Use Cases
Hardware State Abstraction Service Profiles

Unified Fabric - FCOE

Virtualized Adapter

Expanded Memory Server

Unified Management

TECDCT-3873

Cisco Public

184

Unified Management (1/2)

SAN A SAN B LAN

Infrastructure Management Centralize chassis management

Intrinsic system management

Two Failure Domains Separate fabrics Central supervisor, forwarding logic Distributed Fabric Extenders

Single management domain Scalable architecture

10GE/FCoE
Blade Chassis Blade Chassis Blade Chassis Blade Chassis

Traffic isolation Oversubscription

Chassis Management

TECDCT-3873

Cisco Public

185

Unified Management (2/2)

Single point of device management
View 1 View 2 Adapters, blades, chassis, LAN & SAN connectivity Embedded manager g GUI & CLI Systems Management Software

Custom Portal

GUI

Standard APIs for systems management

XML, SMASH-CLP, WSMAN, IPMI, SNMP SDK for commercial & custom implementations

CLI

XML API

Standard APIs

UCS Manager

Designed for multi-tenancy

RBAC, organizations, pools & policies

TECDCT-3873

Cisco Public

186

UCS Conceptual Overview

TECDCT-3873

Cisco Public

187

UCS Resources - Example

Physical
Server Blades Adapters

Logical
UUIDs VLANs IP Address MAC Address VSANs WWNs

TECDCT-3873

Cisco Public

188

Resource Pools - Example

Blades blade 3 blade 2 blade 1 blade 0 Blade pool

MACs
01:23:45:67:89:0d 01:23:45:67:89:0c 01:23:45:67:89:0b 01:23:45:67:89:0a

MAC pool

WWN pool WWNs

05:00:1B:32:00:00:00:04 05:00:1B:32:00:00:00:03 05:00:1B:32:00:00:00:02

05:00:1B:32:00:00:00:01

TECDCT-3873

Cisco Public

189

How They Work Together

UCS Server

TECDCT-3873

Cisco Public

190

Profiles Example
Servers Virtual Machines Ethernet Adapters Fibre Channel Adapters IPMI Profiles

TECDCT-3873

Cisco Public

191

Out-of-the-Box Protocol Support

SNMP SMASH CLP

IPMI

CIM XML

Remote KVM

UCS CLI and GUI

Serial Over LAN

UCS XML API

TECDCT-3873

Cisco Public

192

UCS Manager Loaded from 6100 Switch

Point a browser at IP Address of Switch

TECDCT-3873

Cisco Public

193

UCS Graphical Interface

Top directory map tells you where you are in tree

CONTENT PANE
194

Navigation Pane Tabs

Equipment | Servers | LAN | SAN | VM | Admin

TECDCT-3873

Cisco Public

195

Creation Wizards

TECDCT-3873

Cisco Public

196

Multi-Tenancy Model (Opt-In)

Company Finance

Network Management

Policies

Server Server Server

Policies

Compute Blade

Blade Chassis

Fabric Extender

Compute Blade Compute Blade Compute Blade Compute Blade Compute Blade Compute Blade Compute Blade

Fabric Extender

197

Tenant Portal for Multi-Tenant Deployment

Server Array Manager supports
Multiple hierarchical server organizations Network organization Infrastructure organization
Cisco UCS GUI

Custom Portal

RBAC and object-level security XML API

Cisco UCS GUI

Designed for enterprise deployment Provides a global view

Single tenant custom views

Through custom portals Typically as plugin of an existing data center infrastructure

Server Array

TECDCT-3873

Cisco Public

198

Unified Compute Integration in the Data Center:

Use Cases
Hardware State Abstraction Service Profiles Unified Fabric - FCOE

Virtualized Adapter

Expanded Memory Server

Unified Management

UCS Integration

TECDCT-3873

Cisco Public

199

UCS and Nexus in the Data Center

Nexus 7010

Core Layer

Nexus 7010

Distribution Layer

10GE
Access Layer

Nexus 5000
GigE 10GE

GigE 10GE

FEX
10GE Servers

Rack 1 Rack 1 1GE to Servers Rack 1 Row 1 / Domain 1 / POD 1

...

..
Rack 12 200

TECDCT-3873

Cisco Public

UCS and Nexus in the Data Center

Nexus 7010

Core Layer

Nexus 7010

Distribution Layer

10GE
Access Layer

Nexus 5000
GigE 10GE

GigE 10GE

UCS 6100

FEX
10GE Servers
blade1 slot 1 blade2 slot 2 blade3 slot 3 blade4 slot 4 blade5 slot 5 blade6 slot 6 blade7 slot 7 blade8 slot 8 blade1 slot 1 blade2 slot 2 blade3 slot 3 blade4 slot 4 blade5 slot 5 blade6 slot 6 blade7 slot 7 blade8 slot 8 blade1 slot 1 blade2 slot 2 blade3 slot 3 blade4 slot 4 blade5 slot 5 blade6 slot 6 blade7 slot 7 blade8 slot 8

Rack 1 Rack 1 1GE to Servers Rack 1

blade1 slot 1 blade2 slot 2 blade3 slot 3 blade4 slot 4 blade5 slot 5 blade6 slot 6 blade7 slot 7 blade8 slot 8

blade1 slot 1 blade2 slot 2

blade3 slot 3 blade4 slot 4 blade5 slot 5 blade6 slot 6 blade7 slot 7 blade8 slot 8

blade1 slot 1 blade2 slot 2 blade3 slot 3 blade4 slot 4 blade5 slot 5 blade6 slot 6 blade7 slot 7 blade8 slot 8 blade1 slot 1 blade2 slot 2 blade3 slot 3 blade4 slot 4 blade5 slot 5 blade6 slot 6 blade7 slot 7 blade8 slot 8

Row 1 / Domain 1 / POD 1

...

..
Rack 12

blade1 slot 1 blade2 slot 2 blade3 slot 3 blade4 slot 4 blade5 slot 5 blade6 slot 6 blade7 slot 7 blade8 slot 8

blade1 slot 1 blade2 slot 2

blade3 slot 3 blade4 slot 4 blade5 slot 5 blade6 slot 6 blade7 slot 7 blade8 slot 8

Cisco Public

201

TECDCT-3873

Cisco Public

202

Interested in Data Center?

Discover the Data Center of the Future
Cisco booth: #617 See a simulated data center and discover the benefits including investing to save, energy efficiency and innovation.

Data Center Booth

Come by and see whats happening in the world of Data Center demos; social media activities; bloggers; author signings Demos include: Unified Computing Systems Cisco on Cisco Data Center Interactive Tour Unified Service Delivery for Service Providers Advanced Services
TECDCT-3873 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

203

Interested in Data Center?

Data Center Super Session
Data Center Virtualization Architectures, Road to Cloud Computing (UCS) Wednesday, Ju y 1, 2:30 3:30 p , Hall D ed esday, July , 30 3 30 pm, a Speakers: John McCool and Ed Bugnion

Panel: 10 Gig LOM

Wednesday 08:00 AM Moscone S303

Panel: Next Generation Data Center

Wednesday 04:00 PM Moscone S303

Panel: Mobility in the DC Data

Thursday 08:00 AM Moscone S303
204

TECDCT-3873

Cisco Public

Please Visit the Cisco Booth in the World of Solutions

See the technology in action
Data Center and Virtualization
DC1 Cisco Unified Computing System p g y DC2 Data Center Switching: Cisco Nexus and Catalyst DC3 Unified Fabric Solutions DC4 Data Center Switching: Cisco Nexus and Catalyst DC5 Data Center 3.0: Accelerate Your Business, Optimize Your Future DC6 Storage Area Networking: MDS DC7 Application Networking Systems: WAAS and ACE

TECDCT-3873

Cisco Public

205

Cisco Public

206

Complete Your Online Session Evaluation

TECDCT-3873

Cisco Public

TECDCT-3873

Cisco Public

208

TOR Starter Set The Rules
100% (11)
TOR Starter Set The Rules
28 pages
DATA Center Design
91% (11)
DATA Center Design
129 pages
Bicsi 002 Data Center Design and Implementation Best Practices
100% (9)
Bicsi 002 Data Center Design and Implementation Best Practices
50 pages
Data Center Cooling
100% (7)
Data Center Cooling
28 pages
Design and Build Modern Datacentres, A to Z practical guide
From Everand
Design and Build Modern Datacentres, A to Z practical guide
Engineer Said AL Hosni
3/5 (2)
BICSI Data Center Standard: Stephen Banks, RCDD CDCDP
100% (3)
BICSI Data Center Standard: Stephen Banks, RCDD CDCDP
26 pages
ISX12 CM2350 X102 Wiring Diagram: Positive Negative
71% (7)
ISX12 CM2350 X102 Wiring Diagram: Positive Negative
1 page
Data Center Calculations
100% (5)
Data Center Calculations
12 pages
DC110 Data Center Design and Best Practices Version 1.4 Student Guide September 2010
90% (10)
DC110 Data Center Design and Best Practices Version 1.4 Student Guide September 2010
253 pages
Data Center Energy Handbook
100% (4)
Data Center Energy Handbook
118 pages
Choosing The Optimal Data Center Power Density PDF
100% (1)
Choosing The Optimal Data Center Power Density PDF
14 pages
Data Center Best Practices Ebook
100% (2)
Data Center Best Practices Ebook
62 pages
Data Center Design Guide
100% (3)
Data Center Design Guide
123 pages
Data Center Infrastructure Resource Guide
89% (9)
Data Center Infrastructure Resource Guide
64 pages
Data Center Design Playbook
100% (1)
Data Center Design Playbook
240 pages
Data Center Design Best Practices
100% (14)
Data Center Design Best Practices
41 pages
1030hrs Panduit-NingLiu PDF
100% (1)
1030hrs Panduit-NingLiu PDF
37 pages
Google Data Centers
0% (2)
Google Data Centers
23 pages
Data Center Guide
No ratings yet
Data Center Guide
49 pages
Power and Cooling Capacity Management For Data Centers
No ratings yet
Power and Cooling Capacity Management For Data Centers
0 pages
Ebook Colocation 101 Guide
No ratings yet
Ebook Colocation 101 Guide
14 pages
19 07 31 DCDC Cred Handbook
No ratings yet
19 07 31 DCDC Cred Handbook
24 pages
520 KW, Tier 2, Chilled Water, 4823 FT: Reference Design 73)
No ratings yet
520 KW, Tier 2, Chilled Water, 4823 FT: Reference Design 73)
7 pages
Colo Design - Supplement
100% (1)
Colo Design - Supplement
16 pages
High Performance Data Centers: A Design Guidelines Sourcebook
100% (20)
High Performance Data Centers: A Design Guidelines Sourcebook
63 pages
An Introduction To Data Center Design
No ratings yet
An Introduction To Data Center Design
70 pages
Data Center Power and Cooling and Virtualization Trends in ASEAN
No ratings yet
Data Center Power and Cooling and Virtualization Trends in ASEAN
59 pages
Tier Standards - Topology
100% (2)
Tier Standards - Topology
14 pages
Data Center Infrastructure Management
100% (2)
Data Center Infrastructure Management
7 pages
How To Calculate Cooling Requirements For A Data Center
100% (1)
How To Calculate Cooling Requirements For A Data Center
10 pages
Data Center
100% (2)
Data Center
12 pages
Intel Data Center Design
100% (1)
Intel Data Center Design
21 pages
Data Center Baseline
100% (6)
Data Center Baseline
61 pages
Data Center Power Management 8.2 - Design Guide
100% (1)
Data Center Power Management 8.2 - Design Guide
118 pages
Calculating Total Power Requirements For Data Centers-En
No ratings yet
Calculating Total Power Requirements For Data Centers-En
10 pages
Whitepaper Evaluate Data Centers With Confidence
No ratings yet
Whitepaper Evaluate Data Centers With Confidence
19 pages
Data Center Design PDF
No ratings yet
Data Center Design PDF
8 pages
Data - Centers - Design Consideration PDF
100% (1)
Data - Centers - Design Consideration PDF
56 pages
Data Center Design and The ANSI - TIA-942 Standard (PDFDrive)
100% (2)
Data Center Design and The ANSI - TIA-942 Standard (PDFDrive)
61 pages
Tiers - Facts Vs - Myths and Misconceptions V2e
100% (1)
Tiers - Facts Vs - Myths and Misconceptions V2e
38 pages
Part 6 - 2016-05 BICSI 002
No ratings yet
Part 6 - 2016-05 BICSI 002
44 pages
Data Center Design
100% (1)
Data Center Design
43 pages
Data Center Site Infrastructure
100% (1)
Data Center Site Infrastructure
13 pages
30 Data Centre Cooling Templates Vincent Byrne Consulting
100% (4)
30 Data Centre Cooling Templates Vincent Byrne Consulting
29 pages
490 KW, Tier III, Chilled Water, Prefabricated, 697 M: Reference Design 86)
No ratings yet
490 KW, Tier III, Chilled Water, Prefabricated, 697 M: Reference Design 86)
7 pages
Greening the Data Center: A Pocket Guide
From Everand
Greening the Data Center: A Pocket Guide
George Spafford
No ratings yet
DC 3.0 Next Generation Data Center Design: © 2008 Cisco Systems, Inc. All Rights Reserved. Cisco Confidential
No ratings yet
DC 3.0 Next Generation Data Center Design: © 2008 Cisco Systems, Inc. All Rights Reserved. Cisco Confidential
47 pages
FCoE Bootcamp
No ratings yet
FCoE Bootcamp
51 pages
DWDM / DCI - Discovery: SP Optical
No ratings yet
DWDM / DCI - Discovery: SP Optical
21 pages
Cisco Glcgbicscompatibility Ds
No ratings yet
Cisco Glcgbicscompatibility Ds
8 pages
white-paper-c11-CISCO IMP
No ratings yet
white-paper-c11-CISCO IMP
20 pages
Transforming Data Center Core With Dce Cisco Nexus
No ratings yet
Transforming Data Center Core With Dce Cisco Nexus
27 pages
Infocom03 Rajiv PDF
No ratings yet
Infocom03 Rajiv PDF
14 pages
Cisco The Future of OT - March 2012
No ratings yet
Cisco The Future of OT - March 2012
14 pages
IP Over DWDM
No ratings yet
IP Over DWDM
25 pages
DataCenter - Overview
100% (1)
DataCenter - Overview
62 pages
Panduit Cisco Nexus 7009 Application Note PDF
No ratings yet
Panduit Cisco Nexus 7009 Application Note PDF
56 pages
Network Design Considerations in A Blade Server Environment - LAN & Storage
No ratings yet
Network Design Considerations in A Blade Server Environment - LAN & Storage
31 pages
Data Center Transformation: Patrick Mckinnis Systems Engineer Cisco Systems, Inc
No ratings yet
Data Center Transformation: Patrick Mckinnis Systems Engineer Cisco Systems, Inc
38 pages
1.6 Next Generation Server Access Architecture With Cisco Nexus 6000 Eddie Tan1 2
No ratings yet
1.6 Next Generation Server Access Architecture With Cisco Nexus 6000 Eddie Tan1 2
44 pages
Icon Library: Current As of 3-15-2007
0% (1)
Icon Library: Current As of 3-15-2007
69 pages
Technical Data: Specifications & Capacities
No ratings yet
Technical Data: Specifications & Capacities
56 pages
Tutorial 6 Problem Statement
No ratings yet
Tutorial 6 Problem Statement
4 pages
Soal Latihan Ujian Written Aecc
No ratings yet
Soal Latihan Ujian Written Aecc
26 pages
Chap-9 - Maps
No ratings yet
Chap-9 - Maps
39 pages
Luobinsen High Power Charger 150-240kW
No ratings yet
Luobinsen High Power Charger 150-240kW
1 page
Trade Briefing BIM-AM SGnHandover Guide
No ratings yet
Trade Briefing BIM-AM SGnHandover Guide
86 pages
MY23 WRX Accessories Brochure
No ratings yet
MY23 WRX Accessories Brochure
31 pages
Falling Cat Questions
No ratings yet
Falling Cat Questions
13 pages
BS7121-2-5 2012 Tower Cranes
No ratings yet
BS7121-2-5 2012 Tower Cranes
42 pages
Seeking Justice in an Energy Sacrifice Zone Standing on Vanishing Land in Coastal Louisiana 1st Edition Julie K. Maldonado - The full ebook with all chapters is available for download
100% (1)
Seeking Justice in an Energy Sacrifice Zone Standing on Vanishing Land in Coastal Louisiana 1st Edition Julie K. Maldonado - The full ebook with all chapters is available for download
78 pages
Compressor Symbol Centrifugal Compressor
No ratings yet
Compressor Symbol Centrifugal Compressor
3 pages
Gyda-Sc: System Controller For Flashlink
No ratings yet
Gyda-Sc: System Controller For Flashlink
36 pages
NTU Sample Physics
No ratings yet
NTU Sample Physics
5 pages
Food and Beverage Management
No ratings yet
Food and Beverage Management
246 pages
WRLD - Orbit
0% (1)
WRLD - Orbit
4 pages
Chapter 50
No ratings yet
Chapter 50
20 pages
Nursing Care of Death and Dying Patients
100% (2)
Nursing Care of Death and Dying Patients
71 pages
सरकारी नौकरी - Government Jobs India - Sarkari Naukri - www.SarkariNaukriBlog.com - Engineering Jobs raining through GATE-2013
No ratings yet
सरकारी नौकरी - Government Jobs India - Sarkari Naukri - www.SarkariNaukriBlog.com - Engineering Jobs raining through GATE-2013
8 pages
HY Syllabus Class 12 - 2024-25
No ratings yet
HY Syllabus Class 12 - 2024-25
4 pages
Dykem Marking Products Brochure
No ratings yet
Dykem Marking Products Brochure
2 pages
Chapter 3 Example
No ratings yet
Chapter 3 Example
15 pages
SB 10054723 6187
No ratings yet
SB 10054723 6187
19 pages
Service Manual: XM-2165GTX
No ratings yet
Service Manual: XM-2165GTX
24 pages
Asessment 1 Science Form 2
No ratings yet
Asessment 1 Science Form 2
5 pages
AMUL Presentation
No ratings yet
AMUL Presentation
20 pages
The Korean Economy from a miraculous past to a sustainable future Lim instant download
100% (1)
The Korean Economy from a miraculous past to a sustainable future Lim instant download
59 pages
Hairstyles Splatoon Wiki Fandom
No ratings yet
Hairstyles Splatoon Wiki Fandom
1 page
Tutorial - DPV - 2015
No ratings yet
Tutorial - DPV - 2015
5 pages

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.