Module 06
Module 06
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 1
Module 6: IP SAN and FCoE
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 2
Module 6: IP SAN and FCoE
Lesson 1: IP SAN
During this lesson the following topics are covered:
• Drivers for IP SAN
• IP SAN Protocols: iSCSI and FCIP
• Components, topologies, and protocol stack for iSCSI and FCIP
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 3
Drivers for IP SAN
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 4
Drivers for IP SAN (contd.)
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 5
IP SAN Protocol: iSCSI
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 6
iSCSI Implementation
iSCSI iSCSI Gateway
Storage Array
encapsulates Server
SCSI IP
commands FC Port
and data into
iSCSI HBA
an IP packet
and
transports
iSCSI Port
them using
TCP/IP
iSCSI Implementation Storage Array
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 7
Components of iSCSI
iSCSI initiator iSCSI target IP network
• Hardware or software • The iSCSI-enabled device • Interconnected Ethernet
residing on a computer to which you will be switches and/or routers
that handles storing data
communications with an • Storage array with iSCSI
iSCSI array (storage port
devices) • iSCSI gateway – enables
• Software-based iSCSI communication with FC
initiator, such as storage array
Microsoft's free iSCSI • Common deployment
initiator - uses the scenarios for an iSCSI
existing network target include: Storage
card (NIC) array, disk drives
• Hardware-based iSCSI
initiator – such as iSCSI
HBA (Host Bus
Adapter)
• Does not consume
CPU resources to
handle iSCSI
commands
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE
iSCSI Host Connectivity Options
• Simple, least expensive • Moves TCP processing load • Offloads both iSCSI and
• NIC provides network off the host CPU onto the TCP/IP processing from host
interface NIC card CPU
• Software initiator provides • Software initiator provides • Simplest option for boot
iSCSI functionality iSCSI functionality from SAN via iSCSI
• Requires host CPU cycles • Requires host CPU cycles
for iSCSI and TCP/IP for iSCSI processing
processing
• Encapsulation of SCSI into
IP packets &
decapsulation are carried
out by host CPU
• Additional overhead on
the host CPU
• Heavy I/O load -> host
CPU become bottleneck
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 9
iSCSI Topologies
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 10
iSCSI Topologies: (1) Native iSCSI
• iSCSI initiators are either directly attached to storage
array or connected through IP network
No FC component
• Storage array has iSCSI port
• Each iSCSI port on the array is configured with an IP
address and port number.
IP
Server Storage Array
iSCSI HBA iSCSI Port
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 11
iSCSI Topologies: (1) Native iSCSI (contd.)
• The array has one or more iSCSI ports configured with an IP
address and connected to a standard Ethernet switch.
• After an initiator is logged on to the network, it can access the
available LUNs on the storage array.
• A single array port can service multiple hosts or initiators as long
as the array port can handle the amount of storage traffic that
the hosts generate.
IP
Server Storage Array
Initiator iSCSI Port
e.g: iSCSI HBA
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 12
iSCSI Topologies: (2) Bridged iSCSI
• iSCSI gateway is used to enable communication between iSCSI
host and FC storage
• iSCSI gateway works as bridge between FC and IP network
Converts IP packets to FC frames and vice versa
• iSCSI initiator is configured with gateway’s IP address as its target
• iSCSI gateway is configured as FC initiator to storage array
iSCSI Gateway
IP
Server
iSCSI HBA
FC SAN
Storage Array
Server FC HBA FC Port
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 13
iSCSI Topologies: (2) Bridged iSCSI (contd.)
• In the figure illustrates an iSCSI host connectivity to an FC storage array
• In this case, the array does not have any iSCSI ports
• An external device, called a gateway or a multiprotocol router, must be
used to facilitate the communication between the iSCSI host and FC
storage
• Bridge devices contain both FC and Ethernet ports to facilitate the
communication between the FC and IP environments
iSCSI Gateway
IP
Server
iSCSI HBA
FC SAN
Storage Array
Server FC HBA FC Port
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 14
iSCSI Topologies: (3) Combining FC and Native iSCSI
Connectivity
• Array provides both FC and iSCSI ports
Enable iSCSI and FC connectivity in the same environment
No bridge devices needed
iSCSI Port
IP
Server
iSCSI HBA
FC SAN
Storage Array
FC Port
Server FC HBA
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 15
iSCSI Protocol Stack
• SCSI is the command protocol that works at the application layer of the Open
System Interconnection (OSI) model
• Initiators and targets use SCSI commands and responses to talk to each other
• Commands, data, and status messages are encapsulated into TCP/IP and
transmitted across the network between the initiators and targets
• iSCSI is the session-layer protocol that initiates a reliable session between
devices that recognize SCSI commands and TCP/IP
• Responsible for handling login, authentication, target discovery, and
session management
OSI Model iSCSI Initiator iSCSI Target
Layer 7 Application SCSI Commands and Data SCSI
Figure: iSCSI protocol
Layer 5 Session iSCSI Login and Discovery iSCSI
layers and the
encapsulation order of the Layer 4 Transport TCP Windows and Segments TCP
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 16
iSCSI Protocol Stack (contd.)
• TCP is used with iSCSI at the transport layer to provide reliable transmission
• TCP controls message flow, windowing, error recovery, and
retransmission.
• Rely upon the network layer of the OSI model to provide global
addressing and connectivity.
• The Layer 2 protocols at the data link layer of this model enable node-to-
node communication through a physical network.
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 17
iSCSI Protocol Stack (contd.)
OSI Model iSCSI Initiator iSCSI Target
Interconnect
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 18
iSCSI Discovery
• For iSCSI communication, initiator must discover location and
name of target on a network
• iSCSI discovery takes place in two ways:
Internet Storage Name
SendTargets discovery
Service (iSNS)
• Initiator is manually • Initiators and targets
configured with the register themselves with
target’s network portal iSNS server
• Initiator issues • Initiator can query iSNS
SendTargets command; server for a list of
target responds with available targets
required parameters
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 19
iSCSI Name
• iSCSI name is a unique iSCSI identifier that is used to identify
initiators and targets within an iSCSI network
• Two common types of iSCSI names are:
IQN: iSCSI Qualified Name
To use IQN, the company must own a registered domain name
A date is included in the name to avoid potential conflicts caused by
the transfer of domain names
iqn.2008-02.com.example:optional_string
▪ Example: iqn.1992-05.com.emc:apm000339013630000-10
The optional_string can be the name of the host, serial number,
asset number or any other storage identifier
EUI: Extended Unique Identifier
Use the WWN (World Wide Name)
eui.0300732A32598D26
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 20
IP SAN Protocol: FCIP
FCIP
FCIP is an IP-based protocol that is used to connect distributed
FC SAN islands
• Combines advantages of Fibre Channel and IP
• Enables distributed FC SAN islands to be interconnected over the
existing IP-based networks
• Creates virtual FC links over existing IP network that is used to
transport FC data between different FC SANs
• Encapsulates FC frames onto IP packet
• Extensively used in disaster recovery implementations in which
data is duplicated to the storage located at a remote site
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 21
FCIP Topology
• FCIP gateway is connected to each fabric via a standard FC connection
• FCIP gateway at one end of the IP network encapsulates the FC frames into IP
packets
• FCIP gateway at the other end removes the IP wrapper and sends the FC data to
the layer 2 fabric
• Fabric treats these gateways as layer 2 fabric switches
• IP address is assigned to the port on the gateway, which is connected to an IP
network.
• After the IP connectivity is established, the nodes in the two independent fabrics can
communicate with other
Servers Servers
Server Server
IP
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 23
FCIP Frame
• Encapsulation of FC frame on to IP packet could cause the IP
packet to be fragmented
• When an IP packet is fragmented, the required parts of the
header must be copied by all fragments
• When a TCP packet is segmented, normal TCP operations are
responsible for receiving and re-sequencing the data prior to
passing it on to the FC processing portion of the device
FC
FC Frame SOF SCSI Data CRC EOF
Header
FCIP Encapsulation
IP TCP FCIP
FCIP Frame IP Payload
Header Header Header
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 24
Module 6: IP SAN and FCoE
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 25
Drivers for FCoE
• Data centers have multiple networks to handle various types of I/O
traffic
Ethernet network for TCP/IP communication: used for client-server
communication, data backup, infrastructure management
communication, and so on.
FC network for FC communication: used for moving block-level
data between storage and servers.
• To support multiple networks, servers in a data center are equipped
with multiple redundant physical network interfaces—for example,
multiple Ethernet and FC cards/adapters.
• To enable the communication, different types of networking switches
and physical cabling infrastructure are implemented in data centers.
• The need for two different kinds of physical network infrastructure
increases the overall cost and complexity of data center operation.
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 26
Drivers for FCoE (contd.)
• FCoE is a protocol that transports FC data over Ethernet network
(Converged Enhanced Ethernet)
• FCoE is being positioned as a storage networking option
because:
Reduces the number Reduces cost and
Reduced power and
Enables consolidation of adapters, switch eases data center
cooling requirement
ports, and cables management
• Enables • Lower capital • Lower Total Cost of • Reduces power and
consolidation (merge) expenditure Ownership (TCO) cooling cost, and
of FC SAN traffic and • Potentially lower floor space
Ethernet traffic onto administration cost,
a common Ethernet with convergence of
infrastructure LAN and SAN
• Effective sharing of
high-bandwidth links
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 27
Data Center Infrastructure – Before Using FCoE
Servers Servers
• LAN and SAN are two
different networks. Server Server
• Storage resources are
accessible using HBAs
and network
resources that are
accessible using NICs
by hosts. FC IP
Switches Switches
• In a data center, a
server is configured FC LAN
with 2 to 4 NIC cards Switches
and redundant HBA
cards.
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 28
Data Center Infrastructure – Before Using FCoE (contd.)
Servers Servers
• If the data center has
hundreds of servers, Server Server
it would require a
large number of
adapters, cables, and
switches.
• Complex
environment, difficult FC IP
Switches Switches
to manage and scale.
• The cost of power, FC LAN
cooling, and floor Switches
space further adds to
the challenge
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 29
Data Center Infrastructure – After Using FCoE
• I/O consolidation Servers Servers
with FCoE using FCoE
switches and Server Server
Converged Network
Adapters (CNAs)
• CNA replaces both
HBAs and NICs in the
server and
FCoE
consolidates both the Switches
IP and FC traffic
• The FCoE switch FC LAN
passes Fibre Channel Switches
traffic to the SAN,
and the Ethernet
traffic to an attached
Ethernet network.
Storage Array Storage Array
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 30
Data Center Infrastructure – After Using FCoE
Servers Servers
• The cable
requirements Server Server
from host to FCoE
switches can be
reduced, which in
turn reduces the
cooling costs, FCoE
Switches
management
requirements, FC LAN
Switches
and the overall
operational cost.
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 31
Components of an FCoE Network
(1) Converged
(3) FCoE
Network (2) Cable
switch
Adapter (CNA)
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 32
Components of an FCoE Network: (1) Converged
Network Adapter (CNA)
• Provides functionality of both – a
standard NIC and an FC HBA
Eliminates the need to deploy
separate adapters and cables for FC
and Ethernet communications
• Contains separate modules for 10
Gigabit Ethernet, FC, and FCoE
Application Specific Integrated
Circuits (ASICs)
FCoE ASIC encapsulates FC frames
into Ethernet frames
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 33
Components of an FCoE Network: (2) Cable
• Two options are available for FCoE cabling
Copper based Twinax cable
Standard fiber optical cable
Twinax Cable Fiber Optical Cable
Suitable for shorter distances (up to 10 Can run over longer distances
meters)
Requires less power and are less expensive Relatively more expensive than Twinax
than fiber optical cable cables
Uses Small Form Factor Pluggable Plus Uses Small Form Factor Pluggable Plus
(SFP+) connector (SFP+) connector
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 34
Components of an FCoE Network: (3) FCoE Switch
Forwarder (FCF),
Ethernet bridge, and set Ethernet Bridge
of Ethernet ports and FC
ports (optional) Ethernet Ethernet Ethernet Ethernet
Port Port Port Port
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 35
Components of an FCoE Network: (3) FCoE Switch
(contd.)
• Forwards frames based on
Ethertype FC Port FC Port FC Port FC Port
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 36
FCoE – Frame Structure
Header
Header
Header
Fibre Channel
FCoE
EOF
CRC
FCS
FC
Payload
24 Bytes 4 Bytes
Up to 2112 Bytes
16 Bytes
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 37
FCoE Frame Mapping
• The FCoE protocol specification replaces the FC-0 and FC-1
layers of the FC stack with Ethernet
• This provides the capability to carry the FC-2 to the FC-4
layer over the Ethernet layer.
OSI Stack
7 - Application
FCoE Protocol Stack FC Protocol Stack
4 - Transport FC - 2 FC - 2 Framing
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 38
Converged Enhanced Ethernet (CEE)
• Conventional Ethernet is lossy in nature, which means that
frames might be dropped or lost during transmission
• Converged Enhanced Ethernet (CEE) provides lossless Ethernet
Eliminates the lossy nature of Ethernet
• Lossless Ethernet requires following functionalities:
1. Priority-based flow control (PFC)
2. Enhanced transmission selection (ETS)
3. Congestion notification (CN)
4. Data center bridging exchange protocol(DCBX)
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 39
Lossless Ethernet : (1) Priority-Based Flow Control (PFC)
• Traditional FC manages congestion
through the use of a link-level,
credit-based flow control that
guarantees no loss of frames.
Typical Ethernet, coupled with
TCP/IP, uses a packet drop flow
control mechanism. The packet
drop flow control is not lossless
• PFC provides a link level flow
control mechanism
• Creates eight virtual links on a
single physical link
• Uses PAUSE capability of
Ethernet for each virtual link • A virtual link can be paused and
restarted independently
• PAUSE mechanism is based on
user priorities or classes of service
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 40
Lossless Ethernet : (2) Enhanced Transmission Selection
(ETS)
• Allocates bandwidth to different traffic classes such as LAN, SAN,
and Inter Process Communication (IPC)
• Provides available bandwidth to other classes of traffic when a
particular class of traffic does not use its allocated bandwidth
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 41
Lossless Ethernet : (3) Congestion Notification (CN)
• Provides a mechanism for detecting congestion and notifying the
source
Enables a switch to send a signal to other ports that need to stop or
slow down their transmissions
Rate limiting to avoid
packet loss
FCoE
Switch
FCoE
Switch
FCoE
Switch
Host
(Node A)
Congestion
Notification Message
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 42
Lossless Ethernet : (4) Data Center Bridging Exchange
Protocol (DCBX)
• Enables Converged Enhanced Ethernet (CEE) devices to convey
and configure their features with other CEE devices in the
network
Allows a switch to distribute configuration values to attached
adapters
• Ensures consistent configuration across network
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 43
Module 6: Summary
Key points covered in this module:
• IP SAN protocols, their components, and topologies
• FCoE protocol, its components, and topology
EMC Proven Professional. Copyright © 2012 EMC Corporation. All Rights Reserved. Module 6: IP SAN and FCoE 44