Architected for Performance

NVMe™ and NVMe-oF™ in Enterprise Arrays

Sponsored by NVM Express® organization, the owner of NVMe™, NVMe-oF™ and NVMe-
MI™ standards
Speakers

Brandon Hoff Clod Barrera

Mike Kieran

2
NVM Express Sponsored Track for Flash Memory Summit 2018
Track Title Speakers
NVMe-101-1

8/7/18 NVM Express: NVM Express roadmaps and market data for NVMe, NVMe-oF, Janene Ellefson, Micron Amber Huffman, Intel
8:30-9:35 and NVMe-MI - what you need to know the next year. J Metz, Cisco David Allen, Segate
8/7/18 NVMe architectures for in Hyperscale Data Centers, Enterprise Data Centers, Janene Ellefson, Micron Andy Yang, Toshiba
9:45-10:50 and in the Client and Laptop space. Chris Peterson, Jonmichael Hands, Intel
Facebook
3:40-4:45 NVMe Drivers and Software: This session will cover the software and drivers Uma Parepalli, Cavium Suds Jain, VMware
NVMe-102-1

8/7/18 required for NVMe-MI, NVMe, NVMe-oF and support from the top operating Austin Bolen, Dell EMC David Minturn, Intel
systems. Myron Loewen, Intel James Harris, Intel
Lee Prewitt, Microsoft
4:55-6:00 NVMe-oF Transports: We will cover for NVMe over Fibre Channel, NVMe over Brandon Hoff, Emulex Curt Beckmann, Brocade
8/7/18 RDMA, and NVMe over TCP. Fazil Osman, Broadcom Praveen Midha, Marvell
J Metz, Cisco
8/8/18 NVMe-oF Enterprise Arrays: NVMe-oF and NVMe is improving the Brandon Hoff, Emulex Mike Kieran, NetApp
NVMe-201-1

8:30-9:35 performance of classic storage arrays, a multi-billion dollar market. Clod Barrera, IBM Brent Yardley, IBM

8/8/18 NVMe-oF Appliances: We will discuss solutions that deliver high-performance Jeremy Warner, Toshiba Nishant Lodha, Marvell
9:45-10:50 and low-latency NVMe storage to automated orchestration-managed clouds. Manoj Wadekar, eBay Lior Gal, Excelero
Kamal Hyder, Toshiba
NVMe-202-1

8/8/18 NVMe-oF JBOFs: Replacing DAS storage with Composable Infrastructure Bryan Cowger, Praveen Midha, Marvell
3:20-4:25 (disaggregated storage), based on JBOFs as the storage target. Kazan Networks Fazil Osman, Broadcom
8/8/18 Testing and Interoperability: This session will cover testing for Conformance, Brandon Hoff, Emulex Jason Rusch, Viavi
4:40-6:45 Interoperability, Resilience/error injection testing to ensure interoperable Tim Sheehan, IOL Nick Kriczky, Teledyne
solutions base on NVM Express solutions. Mark Jones, FCIA 3
Abstract and Agenda
• Abstract:
• Enterprise Arrays: NVMe-oF™ and NVMe™ is improving the performance of classic
storage arrays, a multi-billion dollar market.

• NVMe-oF Panel
• Storage Segmentation – Brandon Hoff, Emulex
• NVMe over Fabrics Overview – Clod Barrera, IBM
• NVMe over Fabrics on Enterprise Arrays, ANA, and more – Mike Kieran, NetApp
• Performance Improvements at the Storage Array
• Performance improvements in NVMe over Fabrics at the initiator and end-to-end –
Brandon Hoff, Emulex
• Performance Improvements in the Sever and End-to-End
• Q&A

4
NVMe™ over Fabrics – Storage Architectures
Enterprise Arrays - Traditional SAN Benefits:
• Storage services (dedup, compression, thin
APPs provisioning)
• High availability at the array
APPs NVMe-oF Enterprise • Fully supported from the array vendor
Arrays • Example: NetApp/IBM
APPs
Benefits:
Server SAN/Storage Appliances • High performance storage
• Lower cost that storage arrays, minimal storage
APPs
services
APPs e.g. • Roll-your-own support model
NVMe-oF
Rows of • Ex. SUSE on Servers configured to be storage
APPs servers
targets
Benefits:
JBOF/Composable Storage • Very low latency
• Low cost
APPs
• Great for a single rack/single switch
APPs NVMe-oF Blocks of • Leverages NICs, smart NICs, and HBAs for
Storage NVMe-oF to PCIe®/NVMe™ translation
APPs

5
Enterprise Storage Market Block Storage Capacity Shipped
60,000
• Fibre Channel storage shows strong
Fibre Channel

Petabytes Shipped
growth in capacity 50,000
– Fibre Channel Storage capacity shipped
is larger than all other types of external 40,000
storage combined
30,000
• The adoption of All Flash Arrays and
iSCSI
NVMe™ storage will drive the need 20,000
for faster networks
10,000
• iSCSI is the dominate technology Others
block over Ethernet -
• The only RDMA market for block 2017 2018 2019 2020
storage is Infiniband Other Includes: FICON, FCoE,
Infiniband, External SAS
IDC WW Capacity Shipped, 2016

6
Three Areas of Performance Improvement
End to End Performance Improvements

Enterprise Arrays - Traditional SAN

APPs
APPs NVMe-oF Enterprise
Arrays
APPs

Server Front side of the Back side of the

Performance Storage Array Storage Array
Improvement is from a Performance Performance
shorter path through the Improvement a shorter improvement by moving
OS storage stack with path through the target from SAS/SATA drives
NVMe™ & NVMe-oF™ stack to NVMe

7
Flash Memory Summit 2018
Santa Clara, CA
8

Architected for Performance

NVMe™ over Fabric for Enterprise Arrays

Clodoaldo Barrera and Brent Yardley, IBM

Storage System Models for Flash

All-Flash Arrays
Hybrid Arrays

Server Server
Direct Attached Flash Server Server
Hyperconverged

Server Server

Flash Flash
Flash Flash
Flash Flash Flash Flash
Flash Flash
Flash Flash
Flash Flash

Disk Storage (object store)

Storage
Control
Cold Storage 9
Directions in Storage Networking

• 10GE ->100GE dominates the Cloud

infrastructure
• CSPs adopt new Ethernet technology faster
than Enterprise
• Less constrained by legacy install base. Storage Networking
$30

• FC continues link speed generations $25

$20
(now on Gen 6 at 32Gbps
$B $15
• Expect gradual decline in FC SAN share of
$10
storage attachment
$5
• Storage fabrics for new workloads, CSPs, Cold $0
storage all favor IP storage attach – iSCSI, 2012 2013 2014 2015 2016 2017 2018

NAS, and REST Object Storage APIs. FC IP Storage Other

10
NVMe™ and NVMe-oF™
• NVMe protocol enables native parallelism within SSDs and All Flash Arrays (AFA)
• NVMe allows more efficient host software stacks for lower latency at application
• User-space drivers for selected software (e.g. In-memory DB) for maximum benefit

SCSI SAN/Local Storage NVMe SAN/Local Storage New Paradigm

NVMe oF PCIe NVMe AFA SSD

AFA / SSD
“IBM Storage and the NVM Express
Revolution” Koltsidas & Hsu 2017
– IBM Redpaper

11
 NVMe-oF™ Performance Benefits

AFA SSD
NVM PCIe
e oF NVMe

AFA / SSD

• NVMe™ and NVMe-oF have new kernel driver • From IBM Research – Spark application with
stacks in hosts to reduce lock contention and RDMA connection to storage from user space
increase parallelism. Improved throughput and showed up to 5X improvement in performance.
lower latency.
• Requires complete re-structure of I/O system
• For I/O-bound workloads, NVMe-oF lowers server and application awareness/modification
I/O load and wait times.

• IBM benchmark on 16Gb FC and IBM FlashSystem

AFA showed 30% lower CPU utilization from I/O
12
NVMe™ and NVMe™ over Fabric

13
The Benefits of Continuity

• Storage Fabrics are a significant Host … Host

client investment
NVMe-oF™
• Management of full storage path (NVMe™ between
SAN Fabric
• Performance and availability management hosts and storage)

• Audit controls
• Upgrade migration process Storage Array

• Application and middleware compatibility F F

C … C NVMe
testing M M
(Within storage
• Security verification array)
• Etc.…

14
Value of NVMe™ and NVMe-oF™

Host … Host
• Optimized for Flash
NVMe-oF
• Fast and Getting Faster (NVMe between
SAN Fabric
hosts and storage)
• Reduce Application License costs
• Future proof investment Storage Array

• NVMe end-to-end strategy F F NVMe

C
M
… C
(Within storage
M
array)

15
 Architected for Performance

NVMe and NVMe-oF in Enterprise Arrays

Mike Kieran, Technical Marketing Engineer, NetApp
Real-Time Applications: The Next Phase of Digital Transformation
In-memory technologies will grow to ~$13B by 2020*

Artificial Intelligence Machine Learning Real-Time Analytics

All demand lower latency and higher performance

from faster fabrics and faster media
* Gartner, Inc., Market Guide for In-Memory Computing Technologies, 16 January 2017

17 © 2018 NetApp, Inc. All rights reserved. --- NETAPP CONFIDENTIAL --- 17
Impact of NVMe™ For Media Access
NVMe useful for SSDs but required for the next generation of solid state
Drive
~ 10 ms Latency

IO Controller
Latency

Software
Latency
~ 80 µs

~ 80 µs

~ 25 µs ~ 20 µs
~ 10 µs ~ 2 µs ~ 6 µs
~ 10 µs ~ 5 µs ~ 5 µs ~ 5 µs
SAS TLC NVME TLC NVME SCM NVME-oF
HDD
NAND SSD NAND SSD SSD SCM SSD
(Local) (Remote)

18 © 2018 NetApp, Inc. All rights reserved. --- NETAPP CONFIDENTIAL --- 18
NextGen Blocks - NVMe™

What are NVMe-oF™ and FC-NVMe?

• FCP - SCSI-3 command set encapsulated in an FC frame • Replaces SCSI-3 CDBs in a FC

Frame
FC SCSI-3 • Substantial performance boost
FC Frame Command
Data
Header because of:
• Command streamlining
• Reduced context switches
Replaced • Increased multithreading -
64,000 queues with a
maximum queue depth of
FC NVMe
64,000
FC Frame Command
Data
Header
• FC-NVMe - NVMe command set encapsulated in an FC
frame
19 © 2018 NetApp, Inc. All rights reserved. --- NETAPP CONFIDENTIAL --- 19
NetApp’s NVMe™ Vision
Driving real value out of new technologies requires significant investment on multiple fronts from a market leader

Server Storage
CPU Persistent Memory CPU Storage Class Memory
(PMEM) (SCM) as Cache

HBA/ HBA/
HCA HCA
–OR– –OR–
FC, Infiniband, Ethernet
RNIC RNIC

NVMe over Fabrics SSDs attached

via NVMe

20 © 2018 NetApp, Inc. All rights reserved. --- NETAPP CONFIDENTIAL --- 20
FCP (SCSI) vs. NVMe™/FC Performance and Latency

21
NVMe™ Vocabulary Update

Getting used to new terminology as we migrate from SCSI to NVMe-oF™

FC FC-NVMe

LUN Namespace

WWPN NQN

igroup Subsystem
* Asymmetric Namespace Access (NetApp defined multipathing
ALUA ANA* protocol for NVMe. Currently out for ratification by NVM Express®
organization.

22 © 2018 NetApp, Inc. All rights reserved. --- NETAPP CONFIDENTIAL --- 22
Ratified: Asymmetric Namespace Access

• Concept: Namespaces with multiple paths

may have asymmetric properties

• Base protocol is ratified

• Domains and partitioning work is next

23
NVMe™ over Fibre Channel Performance Test

24
NVMe™ over Fibre Channel Performance on a A700s single node

25
 Architected for Performance

Performance Improvements at the Initiator, and

general storage performance improvements with
NVMe over Fabrics
Server Test Configuration – Initiator performance
Target Servers – Qty 2
– Dual CPU - Purley Target Servers
– 32G Dual-Port LPe32002 – 1 Port in use
– RHEL7.4 w/OCS-RAMd (SCSI Target)
– SLES12SP3 w/LPFC-T (NVMe Target)
Initiator
Initiator
– Dual CPU - Purley
– 32G Dual-Port LPe32002 – 1 Port in use
– SLES12SP3 w/LPFC Driver
(v.12.0.141.2)
Test Parameters: 32 threads and queue
depth = 32
27
NVMe-oF™: Lean Stack Delivers more IOPs with less CPU
Average IOPs
1200000
Customer Comments
1000000
– “NVMe™ over Fabrics delivers more 800000

IOPs
transactions on the same storage 600000
SCSI over FC
footprint” 400000
NVMe over FC

– “Our storage strategy going forward is 200000

based on NVMe over Fabrics,” Large 0
Health Care provider 512 1k 2k 4k 8k

Performance Benefits Average CPU Consumption per Thread

40%
– On average 2x-3x more IOPs at the
30%
same CPU consumption

CPU %
SCSI over FC
– At 4k, we see 2x the IOPs at 50% of the 20%
NVMe over FC
CPU consumption 10%

0%
512 1k 2k 4k 8k
28
NVMe-oF™: Just runs faster
Application Latency: Response time Average Application Latency
as seen by the server application 4000

– A function of the number of 3000

outstanding Ios SCSI over FC

us
2000
NVMe over FC
– For this example, 32 (QD) x 32 1000

threads, which means 1024 0

outstanding IOs 512 1k 2k 4k 8k

Single IO Latency: Function of what

the hardware can do
NVMe™ benefits from increased
parallelization

29
Performance Improvement of NVMe™ over Fabrics – End to End
NVMe/FC Vs. SCSI/FC Performance Improvement on the same hardware
Simulated OLTP Data Warehouse Batch Transaction
Workload IOPS1 IO Throughput2 Latency Test3

IO Average Latency
200 4000 200
IOPS

150 3000 150

MB/s
100 2000 100
50 1000 50
0 0 0
NVMe/FC SCSI FCP NVMe/FC SCSI FCP NVMe/FC SCSI FCP

3.6x More 2.7x Higher ½

Transactions Throughput The Latency

14K 264K Random Read IOs, 16 34K Random Read IOs, 8 Threads,
Random Read IOs, 16
Threads, Queue Depth of 16 Threads, Queue Depth of 16 Queue Depth of 1
30
31
Contact Information
For more information please contact the following:

Brandon Hoff brandon.hoff@broadcom.com

Clod Berrera barrerac@us.ibm.com
Mike Kieran Michael.Kieran@netapp.com

32
Architected for Performance