Wi-Fi Performance Benchmark Testing:: Aruba Networks AP-135 and Cisco AP3602i
Wi-Fi Performance Benchmark Testing:: Aruba Networks AP-135 and Cisco AP3602i
BENCHMARK TESTING:
Aruba Networks AP-135 and
Cisco AP3602i
Table of Contents
Executive Summary ..................................................................................................................4
Test Enviroment ........................................................................................................................5
Test Bed Setup ................................................................................................................................... 5
Real World Test Scenarios....................................................................................................... 7
1. Rate vs. Range ................................................................................................................................ 7
2. Client Density Test for 2Mbps and 5Mbps Multicast HD Video Stream .................................. 10
3. iPad Client Density Test for 1Mbps TCP HD Video Stream with AirPlay ................................ 11
4. Video Performance with Mixed Clients and Background Traffic ............................................. 12
5. Battery Test: Impact on Battery Life of Mobile Devices While Downloading Large Files..... 15
What Do The Tests Reveal?................................................................................................... 15
Appendix.................................................................................................................................. 16
Executive Summary
The Aruba Networks proof-of-concept lab is a clean RF environment dedicated to showcasing
complex networking solutions in a real world setup. Aruba customers, partners and prospects
rely on the lab to validate interoperability with other vendorsʼ products as well as test
uncommon deployment scenarios. The lab is also fully equipped to conduct feature and
performance benchmark testing for customer evaluations.
This report focuses on performance and functionality testing to compare and benchmark Wi-Fi
solutions using Aruba AP-135 access point and the Cisco AP3602i access point. The results
from the tests are summarized in Table 1 below and explained in detail in the report.
The test results indicate a significant variation in performance between Aruba and Cisco Wi-Fi
products. Customers are advised to exercise their own judgment before making a vendor
decision for their Wi-Fi network. The rest of the document provides comprehensive details of
the test cases, test bed setup, observations and results collected. Detailed configuration for
both vendors is also included in the appendix for easy reference.
Test Environment
Mobility Controller 3600 AOS 6.1.3.2
5508 controller 7.2.103.0
The following table shows the detailed information on various network components that were
part of the infrastructure used for performance tests.
1. TCP throughput
The throughput tests were performed to understand how the distance of a client from the AP
affects the clientʼs performance. It is expected that the performance of the client will degrade
as it moves away from the access point. MIMO effects also impact the throughput along with
the channel model for the specific environment used for the test. In the actual test, a single
client was tested for maximum throughput at three different non-line of site (NOS) locations: 30
feet, 70 feet and 120 feet from the AP. This was done to measure the benefits of ClientLink 2.0,
if any, and compare the benefits of 4x4 MIMO vs. 3x3 MIMO at difficult-to-reach locations. TCP
download traffic was used to highlight ability of the AP to transmit effectively to the client, and
upload traffic was used to highlight the receive sensitivity of the AP. Additionally, client
orientation was tested with a tablet (iPad), and the benefit of ClientLink 2.0 was isolated when
enabled and disabled for Cisco.
Tools used for testing Ixia Chariot, throughput script
5‐GHz radio, same 40‐MHz wide channel used
AP operation mode
for Aruba and Cisco
TCP Download
250
200
13%
Better
150
100
0
30 feet (NOS)
70 feet (NOS)
120 feet (NOS)
TCP Upload
250
11%
200
Better
150
100
Cisco canʼt connect
50
0
30 feet (NOS)
70 feet (NOS)
120 feet (NOS)
0
Download Performance
2. Video density tests for 2 Mbps and 5 Mbps multicast HD video stream
For the laptop density test, 2 Mbps and 5Mbps HD video streams were multicast from the VLC
server (one at a time). Devices were added to the network one at a time till the picture quality
started to degrade due to the load on the network.
For the iPad density test, a 1 Mbps TCP HD video was streamed from the Air Video server to
Air Video clients installed on iPads. Initially, only one iPad was added onto the network, which
was streaming 1 Mbps video while mirroring its screen to an Apple TV using AirPlay. Then
more iPads were added to the network one at a time, until either video freezes were observed
or the connection to Apple TV was lost.
5‐GHz radio, same 40 MHz wide channel used for
AP operation mode
both Aruba and Cisco AP
Test cases
50
16%
Better
48
46
44
42
40
Max # of Clients
40
38
25%
Better
36
34
32
30
28
Max # of Clients
22
20
31%
Better
18
16
14
12
Aruba
Cisco
3. Wi-Fi client density and video performance with mixed clients and
background traffic
This test showcases a mix of applications (video, data), client types (laptops, tablets, Apple
TVs), and client capabilities (3x3, 2x2, and 1x1) in a high-density enterprise environment. In
this test, a variety of laptops (3x3 as well as 2x2 capable) were used to stream 5 Mbps and 2
Mbps live HD video from the VLC server. Multicast HD video was used to highlight
downstream video performance. There were several iPads (1x1) streaming 1 Mbps video from
the Air Video server with one iPad mirroring its screen to an AppleTV using AirPlay. The iPad
mirror generated upstream TCP video traffic. In the background there was a client transferring
an 11 GB file, adding background data traffic to further saturate the channel. The quantity of
various devices was adjusted such there was no degradation in video quality to reach an
optimal mix of clients.
The objective of this test was to observe the impact of increasing load on a network that has a
variety of different clients with different types of traffic.
VLC server streaming 5 Mbps and 2 Mbps HD video using
Tools used for testing active transcoding; Air Video server streaming 1 Mbps HD
video
5‐GHz radio, same 40‐MHz wide channel used for Aruba and
AP operation mode
Cisco AP
Results
Intel 6300 2 Mbps Video
Lenovo ThinkPad 9 AP ‐> Client
(3x3:3) (Downstream UDP)
Broadcom 5 Mbps Video
MacBook Pro 10 AP ‐> Client
(3x3: 3) (Downstream UDP)
Broadcom 1 Mbps Video
iPad 12 AP ‐> Client
(1x1:1) (Downstream TCP)
Intel 6205 FTP
Lenovo ThinkPad 1 AP ‐> Client
(2x3:2) (Downstream TCP)
Total Clients 37
The mixed traffic, mixed client-density test case reveals the architectural deficiencies of Cisco
WLAN controller. Since Arubaʼs solution has a stateful firewall built in, the controller is able to
detect and prioritize not only multicast video but also TCP video (Air Video) over best effort
FTP traffic.
In the Aruba case, high quality video (both 2 and 5 Mbps video) was observed without any
artifacts, including pixilation or video freezes. The iPad videos did not exhibit any buffering or
video quality issues. The Apple TV stream was low-latency, reflecting what was seen on both
the iPad and the projected screen simultaneously. The 11 GB FTP file download never timed
out.
For Cisco, the HD video (both 2 and 5 Mbps) experienced a significant amount of pixilation
artifacts and video freezes across all laptops. There was noticeable buffering on the iPads
playing TCP video. The Apple TV had difficulty mirroring, and there was significant latency
between the iPad and the projected monitor. Finally, the 11 GB FTP file download timed out in
the middle of the test.
Ciscoʼs solution can prioritize multicast video traffic, but has no mechanism to identify TCP
video used for streaming to iPads in a mixed traffic environment and hence the video
performance takes a hit. Cisco is clearly not the better solution for a highly dense mix of
laptops and tablets, and is not optimized for high-performance BYOD.
IxChariot, FTPget script to simulate 11GB file transfer for
Tools used for testing MacBook Pro and Filesnds script to simulate 3 GB file
transfer for iPad.
5‐GHz radio, same 40‐MHz wide channel used for Aruba
AP operation mode
and Cisco AP
Results
5
4
3
2
1
0
Laptop (Macbook)
Tablet (iPad)
The battery test reveals that there is almost no difference in the performance of mobile device
battery life when running on either the Aruba AP-135 or Cisco AP3602i.
Common devices like the Apple MacBook are unable to get a connection to the AP3602i at
distances greater than 100 feet. We also observed that the Cisco AP3602i is very sensitive to
the orientation of the device as evidenced by iPad throughput dropping by 67% at 100 feet
when its orientation is changed.
The client density tests for laptops as well as iPads both reveal that the Aruba AP-135 scales
far better than the Cisco AP3602i. The mixed traffic, mixed client-density tests reveal the
inherent architectural deficiencies of Cisco WLAN controller.
With an integrated stateful firewall built into the Aruba controller, it is able to detect and
prioritize not only multicast video but also TCP video (Air Video) over best effort FTP traffic.
Ciscoʼs solution does prioritize multicast video traffic, but has no mechanism to identify TCP
video used for streaming to iPads in a mixed traffic environment and hence the video
performance takes a hit.
The battery drain test reveals that there is almost no difference in the performance of mobile
device battery life when running on either AP-135 or AP3602i.
State of 802.11b Network......................... Disabled --More or (q)uit current module or <ctrl-z> to abort
State of 802.11a Network......................... Enabled Spectrum Expert connection............... Enabled
Number of WLANs.................................. 1 CleanAir NSI Key....................... 0
Number of Active Clients......................... 2 Spectrum Expert Connections counter.... 0
Burned-in MAC Address............................ 0 CleanAir Sensor State.................... Configured
Power Supply 1................................... Absent
Power Supply 2................................... Present, OK Radio Extended Configurations
Maximum number of APs supported.................. 12 Beacon period.............................. 100
Press Enter to continue or <ctrl-z> to abort milliseconds
Beacon range............................... AUTO
WLAN ID Interface Network Admission Multicast buffer........................... AUTO
Control Radio Policy Multicast data-rate........................ AUTO
------- ----------- -------------------------- --- RX SOP threshold........................... AUTO
--------- CCA threshold.............................. AUTO
1 clients Disabled
None
2 ciscoclients Disabled 802.11a Configuration
None 802.11a Network.................................. Enabled
3 multicast-vlan Disabled
None Default Tx Power Level........................... 1
4 management Disabled DTPC Status..................................... Enabled
None Fragmentation Threshold.......................... 2346
TI Threshold..................................... -50
AP Name Slots AP Model Ethernet Legacy Tx Beamforming setting....................
MAC Location Port Country Priority Enabled
------------------ ----- ------------------- ----------------- ---- Traffic Stream Metrics Status.................... Disabled
------------ ---- ------- -------- Expedited BW Request Status......................
AP3600 2 AIR-CAP3602I-A-K9 Disabled
default location 1 US 1 World Mode....................................... Enabled
EDCA profile type................................ default-wmm
CleanAir Management Information Voice MAC optimization status.................... Disabled
CleanAir Capable......................... Yes Call Admission Control (CAC) configuration
Listed on the NASDAQ and Russell 2000® Index, Aruba is based in Sunnyvale, California, and
has operations throughout the Americas, Europe, Middle East, and Asia Pacific regions. To
learn more, visit Aruba at http://www.arubanetworks.com. For real-time news updates follow
Aruba on Twitter and Facebook.
© 2012 Aruba Networks, Inc. Aruba Networksʼ trademarks include AirWave®, Aruba Networks®, Aruba Wireless Networks®, the
registered Aruba the Mobile Edge Company logo, Aruba Mobility Management System®, Mobile Edge Architecture®, People Move.
Networks Must Follow®, RFProtect®, and Green Island®. All rights reserved. All other trademarks are the property of their respective
owners. WP_Wi-Fi_Benchmark_062812