Scale Up Network Availability and Performance With Multi-Port Gigabit Adapters
Scale Up Network Availability and Performance With Multi-Port Gigabit Adapters
April 2005
Contents
The Increasing Demand for Network Speed, Reliability, and Availability ..........................................................2
Expand Bandwidth with the Right Server I/O Architecture and Gigabit Adapters ..............................................3
Conclusion ...........................................................................................................................................11
Not only is there an ever-broadening demand for services, This white paper provides an overview of how Intel PRO/1000
but the required services are in themselves becoming more Server Adapters and LAN-on-motherboard (LOM) connections
complex and demanding. Global weather modeling is perhaps can be used to optimize network performance and reliability.
one of the more esoteric examples of a highly complex In particular, dual-port LOM and quad-port GbE server adapters
and demanding application. A more mundane but equally are emphasized because of their space efficiency in high-
demanding example is debit card purchases. Debit card density server applications.
use is now eclipsing both cash and check writing as the
preferred consumer-transaction method. It works because Table 1. Intel® Architecture I/O configurations
it is quick, convenient, reliable, and available just about Raw Bus Raw Bus
Bus Bandwidth Bandwidth
anywhere we go. Interconnect Bus Width Frequency (Bytes per sec.) (Bits per sec.)
2
Expand Bandwidth with the
Right Server I/O Architecture
and Gigabit Adapters
As a first step in enhancing network performance, make sure In addition to deploying servers for the optimal bus bandwidth
all network servers have the right input/output (I/O) architecture to meet performance needs, IT managers should also plan
for optimum bandwidth. Doing so ensures delivery of the for the best performance attainable from the server adapters
best server performance to the end-users on the network. that will occupy the server’s PCI-X slot(s). For example, by
placing an Intel® PRO/1000 MT Quad Port Server Adapter
Intel® Architecture-based servers have various internal I/O
into a PCI-X 133-MHz slot, all four Ethernet ports of the
capabilities, depending on the specific I/O slot configuration.
adapter will interface with the PCI-X bus at 133 MHz. By
The configurations range from 1 Gbps of raw bus bandwidth
comparison, you can provide the same number of GbE
delivered by a Peripheral Component Interconnect (PCI)
ports with two Intel® PRO/1000 MT Dual Port Server
2.3-compliant, 32-bit, 33-MHz bus to 8 Gbps of bandwidth
Adapters in two PCI-X 100-MHz slots. However, in this
delivered by a Peripheral Component Interconnect Extended
two-slot configuration, all four ports interface with the
(PCI-X*) 1.0-compliant 64-bit, 133-MHz bus, as shown in
PCI-X bus at 100 MHz, which is a 25-percent decrease
Table 1.
in bandwidth—providing less raw bandwidth capacity.
As shown in Figure 1, the amount of I/O bandwidth that a
Another factor to consider when selecting a server’s I/O
server can support depends on the PCI-X bus speed. These
capability is the number of I/O slots provided. The range
bandwidths are:
varies from rack servers and appliances that do not have
• One slot at 133 MHz for 8 Gbps total raw bandwidth PCI slots, to servers that have up to 12 slots. The number
of I/O slots in a server typically follows server cost; but
• Two slots at 100 MHz for 6.4 Gbps total raw bandwidth
high-capacity adapters like the Intel PRO/1000 MT Quad
• Four slots at 66 MHz for 4.2 Gbps total raw bandwidth
Port Server Adapter allow high-bandwidth capacity—even
in servers with fewer available I/O slots, such as the rack-
mount servers typical in datacenters.
Figure 1. PCI-X* 1.0 specifies total slot capacity per bus speed
100 MHz
100 MHz
66 MHz
66 MHz
66 MHz
66 MHz
PCI-X
PCI-X
PCI-X
PCI-X
PCI-X
PCI-X
PCI-X
OR OR
3
The Present and Future of Interrupt Moderation
Advanced interrupt-moderation technology is here Now that interrupt moderation is available on all
today. Interrupt moderation reduces host processor Intel® PRO/1000 MT Server Adapters, here is an
interrupts, thereby enabling technologies such as Gigabit important tip for all network managers interested
EtherChannel* to deliver more of their 16-Gbps bandwidth in taking advantage of this new technology: Look
potential (8 Gbps x full duplex). As a result, performance for interrupt-moderation methods that use dynamic
gains from load-balancing used in adapter teaming will timing adjustment, not just sampling—the more
become a smaller part of overall performance as compared granular the timing, the better the performance.
to the performance gains achieved through the increased
For most networks, these technologies will provide the
server CPU headroom provided by interrupt moderation.
multi-Gigabit bandwidth needed for server and backbone
Host processor interrupts are generated by the adapter connections, helping support today’s movement to Gigabit
in order to request cycles for packet processing. These Ethernet (GbE) at the desktop. Now that 10-Gigabit
interrupts need to be controlled to achieve optimum Ethernet (10GbE) server adapters are entering the
throughput. Too few interrupts can lead to latencies and marketplace for campus-to-campus connectivity, these
too many can unduly burden the server’s processor. By technologies for link aggregation, load balancing, and
bundling an appropriate number of packets before issuing interrupt moderation will continue to provide value to
an interrupt to the host (see figure), the adapter “tunes” GbE deployed in the Local Area Network (LAN). At the
interrupt frequency to match traffic conditions while very least, they enable organizations to implement a
maintaining packet flow. smoother and more cost-effective migration to 10GbE
networking speeds.
Low-volume traffic
PS stack
Hardware
Hardware
Interrupt
Interrupt
Packet
Interrupt
Packet Packet Packet Packet Packet
Packet
Server Adapter Server Adapters response and availability. The effect is analogous to adding
a new lane to a crowded highway—traffic jams are alleviated
1000 Mbps 1000 Mbps 1000 Mbps by dispersing the traffic over more lanes, allowing all traffic
to travel at a higher speed.
5
Figure 3. Three-tiered infrastructure separates and secures networks
Firewall Firewall
Appliances Appliances
Tier 1
Front-End
Network Network
Appliances Appliances
Web
Servers
Intel PRO
®
Server Adapters
Tier 2
Mid-Tier Aggregating Switches
Application
Servers
Aggregating Switches
Firewall
Appliance
Tier 3
Back-End Database
Servers
Aggregating Switches
6
connected server to the network—for a total of six subnets greatly improved. For example, by putting the two LOM
(three per server). This approach is useful in networking ports of the Intel® 82546EB Dual Port Gigabit Ethernet
diverse work groups or applications on a single LAN and Controller on one network segment and the two ports of the
for providing additional security between work groups.1 Intel PRO/1000 MT Dual Port Server Adapter on another,
redundant links are then available for the network segments
Network segmentation can also be implemented as a
of a given server. With such link redundancy, a failure of any
multi-tiered infrastructure. Figure 3, on the previous page,
one link will not result in a loss of connectivity and crippling
depicts a three-tiered implementation, which separates and
downtime. Instead, the traffic fails over to the segment’s
secures logical networks such as front-end Web servers,
redundant link and business continues uninhibited.
application servers, and back-end databases. This three-
tiered approach is effective for scaling distinct parts of With today’s organizations increasingly relying on powerful
Internet-based applications required for e-Business. It is client systems and network-intensive applications, end-users
also the first recommended step in datacenter centralization and IT staff will benefit from deploying multiple GbE server
prior to architecting server consolidation and virtualization. adapters for their critical connections. GbE server adapters
maintain higher network performance and reduce
Just as in simple segmentation, a multi-tiered approach also
degradation in the event of a network failure, resulting in
requires multiple network connections in each server, with
improved availability. To expand the server network resources
each tier associated with an individual connection. To better
required from the increasingly complex network architectures
balance network loads using segmentation in subnets and
of today’s enterprises, IT managers can deploy multi-port
multi-tiered infrastructures, the Intel PRO/1000 MT Dual Port
Intel PRO/1000 Server Adapters to relieve bottlenecks and
Server Adapter provides two GbE ports in a cost-effective,
boost system reliability.
single-slot solution. Furthermore, by deploying multiple
connections on each network segment, availability can be
Adapter Teaming Optimizes
Performance and Reliability
Figure 4. Adapter teaming
The additional GbE ports and links used in segmentation
and multi-tiered approaches require additional management
Network Segmentation capability. This added management capability is provided
with all Intel PRO/1000 Server Adapters via Intel ANS software.
Segment 1
Not only does Intel ANS manage multiple Intel adapters
by teaming them, but it allows adapters from other vendors
to be included in the adapter teams. This latter feature,
Segment 2 multi-vendor compatibility, allows existing non-Intel server
LOMs to be included in adapter teaming for higher network
performance and reliability.
Link Redundancy
Intel ANS provides the following hierarchy of three adapter-
Segment 1 teaming categories:
• Adapter Fault Tolerance
7
with two ports provided by LOM or server adapters. There is The Link Aggregation feature of Intel ANS can team up to
one GbE port and one link per segment. Adding redundancy eight GbE ports and links into a single, larger pipe for even
with another port and link for each segment increases network higher traffic capacity and reliability. In the case of Link
reliability and availability. In the case of Link Redundancy Aggregation in Figure 4, an Intel PRO/1000 MT Quad Port
(Figure 4 on the previous page), Segment 1 is set up on the Server Adapter connects each segment to the server with four
server’s dual-port LOM and an Intel PRO/1000 MT Dual Port teamed connections. All connections in the same team have
Server Adapter has been added to serve the two links on the same address and appear as a single link from the switch
Segment 2. Each segment now has a primary link and a perspective. Essentially, this is a four-lane freeway with traffic
secondary link. If the primary adapter or link fails, the equally balanced across all lanes. And, if a lane gets shut
Adapter Fault Tolerance feature automatically detects the down for any reason, three open traffic lanes still remain.
connection failure and switches primary link traffic to the
For the most demanding situations, Intel ANS can team
secondary link.
as many as eight GbE ports and links into a single team.
Adaptive Load Balancing allows redundant port and link Figure 5 shows teaming of eight ports on two Intel
teaming for higher traffic capacity as well as fault tolerance. PRO/1000 MT Quad Port Server Adapters using link
In this mode, Intel ANS automatically analyzes segment traffic aggregation. In practice, however, Intel ANS can team
and balances it across the primary and secondary links to any mix of LOM, single-port or multi-port adapters as long
provide higher overall throughput along with the higher as one of the adapters is an Intel® adapter. The teamed
reliability and availability provided by automatic fault tolerance. adapter ports allow transmit and receive on all eight channels,
In essence, another lane is added to the data highway. and cross-connected switches provide switch fault tolerance—
all supported and managed by Intel ANS.
Application B
All 8 channels
transmit and receive
Application B
Application B
Server
Application B
Teaming of eight ports on two
®
Intel PRO/1000 MT Quad Port Server Adapters
8
Several technologies enable the approach shown in Figure 5 In general, the four stages or steps to a well-executed
on the previous page, including IEEE 802.3ad* link consolidation effort are:
aggregation, Adaptive Load Balancing from Intel and Gigabit
1. Centralization
EtherChannel* from Cisco. All these technologies require
support by both the GbE adapters and the switches. Intel 2. Data consolidation
PRO/1000 Server Adapters support the IEEE 802.3ad 3. Physical consolidation (server consolidation)
industry standard for link aggregation. The IEEE standard 4. Application consolidation
allows for balancing traffic among multiple server adapters
and switches that also support the standard (Figure 5). In all of these stages, server adapters play a key role
because server-to-server and server-to-client connections
Developed by Intel to improve network performance, Adaptive must still occur. In fact, multi-port adapters become even
Load Balancing increases server bandwidth by automatically more important as a means of providing high port-to-slot
balancing traffic across as many as eight server adapters. densities on consolidated servers.
Because the distribution of traffic among the adapters is
automatic, further segmenting or reconfiguration of the As an example of server adapter importance, consider
network is not needed. The existing Internet Protocol (IP) the first stage of consolidation, which is centralization.
address is shared by all of the adapters; and traffic is always Centralization is simply the process of moving dispersed or
balanced among them. Developed by Cisco and supported far-flung servers to a centralized location for more efficient
by Intel, Gigabit Etherchannel provides scalable bandwidth management. The three-tiered datacenter, discussed
for up to eight adapters at full duplex. It can potentially previously and illustrated in Figure 3 on page 6, is a prime
deliver well in excess of 8 Gbps. 2 example of centralization and server adapter importance
in providing higher reliability and availability.
Consolidation and Virtualization The second stage, data consolidation, involves moving data
Improve Agility and Reliability onto fewer, centralized databases and storage systems. This
typically involves clustered or consolidated servers with Storage
As businesses expand, their IT infrastructure also expands to
Area Networks (SANs) or Network Attached Storage (NAS).
meet increasing network end-user demands. Historically, such
infrastructure expansion resulted in a proliferation of far-flung Figure 6, on the previous page, illustrates the third stage,
servers. For example, in retail clothing, Gap Inc. traditionally which is physical consolidation or server consolidation.
added a new server each time they added a new project or This process consolidates multiple servers running the
application. This one-application-per-server approach to same application onto a single, large platform. Server
growth ultimately resulted in Gap’s IT group managing some consolidation, when done properly, can significantly reduce
450 servers. Such server proliferation includes low utilization management costs and increase performance. This works
of server resources—as low as 10 percent for more than particularly well in a three-tiered environment with separate
300 of the servers—and increased management complexity. Web servers, application servers, and database servers
because each tier can be physically consolidated and scaled
Server consolidation and virtualization allowed Gap Inc.
independently to match specific workload requirements.4
to consolidate one-third of its existing servers to just three
In virtually all cases, realizing the full performance potential
8-way Intel® XeonTM processor MP-based servers. This high
of such consolidation requires high-performance, highly
consolidation ratio provided significant savings in equipment
reliable network connections such as those provided by
and system-administration costs and, through virtualization,
teaming multi-port adapters for the much higher traffic
allowed configuration for high application availability with the
capacity offered by link aggregation.
potential to boost uptime to greater than 99.99 percent.3
Application consolidation, the fourth and most complex
Whether or not consolidation should be used, and to
stage, involves hosting multiple, diverse applications on a
what degree, needs to be determined strategically on a
single platform. This is done with virtualization software that
case-by-case basis. Moreover, consolidation should be
partitions the platform into multiple virtual machines running
executed as a well-planned migration that does not interrupt
concurrently on a single platform. For example, VMware ESX
day-to-day IT infrastructure operations.
Server* virtualization software supports up to 64 concurrent
virtual machines with half the available partitions live at any
9
Figure 7. LAN with 10GbE backbone
Building 1
Work Group A
Work Group B
Tier 1
Web Servers
Core
Switches
Tier 2
Application
Servers
Tier 3
Database
Servers
Datacenter
Building 2
Campus A
Campus
Building 10 GbE
Datacenter 10 GbE
10 GbE
Campus
Building Server
Farm
10 GbE
10
given time. The other 32 virtual machines remain available These larger, more-spread-out networks may require a
on fail-over standby in case of a hardware or software network infrastructure based on a 10GbE backbone between
problem on any of the live virtual machines. The benefits buildings with GbE distribution to the segments or subnets
of virtualization include smoothing of workload demands for within each building. On the previous page, Figure 7
higher system utilization, server redundancy without investing illustrates such an infrastructure based on a 10GbE
in additional servers, and the ability to create and assign backbone.
virtual machines for high agility in meeting changing
For applications requiring the added bandwidth of 10GbE, Intel
application demands.
offers two 10GbE server adapters. The Intel® PRO/10GbE SR
Again, GbE server adapter connectivity is a key element Server Adapter provides cost-effective 10-Gigabit performance
in making virtualization work. With as many as 32 virtual over distances up to 300 meters in multi-mode fiber-optic
machines running on a single high-performance platform, media, making it the ideal backbone choice for campus-wide
a substantial pool of GbE ports becomes necessary. Intel LANs. The Intel® PRO/10GbE LR Server Adapter has a much
multi-port GbE server adapters allow such port count longer reach—to 10 kilometers in single-mode fiber—and is
requirements to be met while conserving server PCI slots. an ideal choice for use in expanded LANs serving multiple
Additionally, Intel ANS support for link aggregation and fault campuses such as shown in Figure 8 on the previous page.6
tolerance provides the connectivity resources necessary to
support the higher performance, agility, and reliability levels
Conclusion
achieved through server consolidation and virtualization.5
Multiple GbE adapters in servers enable greater network
capacity, performance, and availability. With its maximum
10GbE as a High-Performance 8-Gbps throughput, the PCI-X specification provides enhanced
Network Backbone network functionality when a multi-port Gigabit adapter, such as
Some application environments may be better served by the the Intel PRO/1000 MT Quad Port Server Adapter, is installed
higher bandwidth of 10GbE connectivity. Such applications in a PCI-X 133-MHz slot. The Intel PRO/1000 MT Quad Port
typically generate extremely large files and rely on fast Server Adapter quickly increases network capacity while
transfers of such files between various work groups and minimizing impact on available server slots that might be
one or more centralizing database(s). Weather modeling is required for other critical I/O functions.
one such example. Motion picture production involving
While network segmentation provides greater flexibility and
animated special effects or feature-length animation is another
availability in enterprise application deployment, both
example. In this latter case, multiple animator workstations
industry-standard and advanced technologies improve
must all have quick access to huge animation files in a
network performance. Teaming the multi-port Intel
collaborative process that creates and manipulates film
PRO/1000 MT Server Adapters with GbE LOM components
scenes and transfers them to a film production datacenter.
that are embedded on motherboards provides the level of
Yet another example is Computer Aided Design (CAD) work
throughput, reliability, and availability demanded in
groups involved in creating and simulating complex,
a connected enterprise—from simple, small-office LANs to
interactive processes.
multi-tiered datacenters incorporating server consolidation
In cases involving only one or two small work groups and virtualization techniques.
demanding super-high performance, it may be possible
to put those work groups on their own 10GbE network
segments. In many cases, though, the situation is one of
multiple work groups in widely separated locations, such
as design in one building and production in another building.
11
When designing servers with multiple GbE adapters, be For high-speed backbones, campus LANs, and expanded LAN
sure to specify the following types of Intel® PRO Gigabit environments, be sure to specify the following types of Intel®
Server Adapters: PRO/10GbE Server Adapters:
• Intel® PRO/1000 MT Dual Port Server Adapter • Intel® PRO/10GbE LR Server Adapter
LAN-on-Motherboard Connections
• Intel® 82546EB Dual Port Gigabit Ethernet Controller
1
For more information on segmentation, refer to Using Segmentation to Increase Network Performance, an Intel white paper available
at www.intel.com/network/connectivity/resources/doc_library/white_papers/305141.htm.
2
For more information on adapter teaming, refer to Intel® Advanced Network Services Software, an Intel white paper available at
www.intel.com/network/connectivity/resources/doc_library/white_papers/254031.pdf.
3
For more information on consolidation and virtualization at Gap Inc., refer to Intel® Solution Services Bridges the Gap to
Server Consolidation, an Intel Business Center Engagement Synopsis available at
http://cache-www.intel.com/cd/00/00/14/85/148583_148583.pdf.
4
For more information on server consolidation, refer to Strategic Consolidation on Intel® Architecture-Based Servers, an Intel white
paper available at www.intel.com/business/bss/products/server/strategic_consolidation.pdf.
5
For more information on multiport server adapters and their role in virtualization, refer to Virtual Infrastructure: Multiport NICs Are
Critical When Consolidating Servers and Improving IT Management Through Virtualization, an Intel and VMware Inc. white paper
available at www.vmware.com/partners/hw/intel.html.
6
For more information on 10 Gigabit Ethernet, refer to 10 Gigabit Ethernet Technology Overview, an Intel white paper available at
www.intel.com/network/connectivity/resources/doc_library/white_papers/pro10gbe_lr_sa_wp.pdf.
This document and related materials and information are provided “as is” with no warranties, express or implied, including but not
limited to any implied warranty of merchantability, fitness for a particular purpose, non-infringement of intellectual property rights, or
any warranty otherwise arising out of any proposal, specification, or sample. Intel assumes no responsibility for any errors contained
in this document and has no liabilities or obligations for any damages arising from or in connection with the use of this document.
Intel, the Intel logo, and Intel Xeon are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United
States and other countries.
*Other names and brands may be claimed as the property of others.
Copyright © 2005 Intel Corporation. All rights reserved.