Broadcom 9400 Tri-Mode Controller,

Anwendungsgebiete und Tipps zum

NVMe-Einsatz
Dietmar Hinze
Field Application Engineer
20.02.2019

Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Limited and/or its subsidiaries.
Overview

• Broadcom Overview (in short)

• Tri-mode Controller (Overview)
• Eco-System Update

2 | Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Limited and/or its subsidiaries.
Broadcom Overview

Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Limited and/or its subsidiaries.
Global Technology Leadership

Data Center Broadband Mobile Industrial

• Enterprise • DSL • Wi-Fi/Bluetooth • Automotive

• Cloud • Cable • GNSS • Factory automation
• Telecom • PON • RF Filters • Alternative energy

5 | Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Limited and/or its subsidiaries.
Data Center: Networking & Storage Connectivity

Server Connectivity

Networking

Storage

Ethernet Connectivity Custom Optical NVMe/SAS/SATA/PCIe Fibre Channel SAN SSD/HDD

(Switching, Controller, PHY) ASICs Components Connectivity Switches & HBAs Controller

6 | Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Limited and/or its subsidiaries.
Tri-Mode Controller

Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Limited and/or its subsidiaries.
PCIe/SAS/SATA Host Bus & MegaRAID Adapter

Application Application
General purpose Servers Software defined Storage
High Storage Demand Attaching external RAID Devices
High Data Protection Demand Attaching Tape Storage Drives

8 | Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Limited and/or its subsidiaries.
 Board Name Decoder
Connectors
“X”i = # of Internal ports
Brand Name
“X”e = # of External ports
MegaRAID SAS = MegaRAID Entry, Value, Feature
“X”i”X”e = # of Internal and External ports
BCL SAS = Host Bus Adapters

Technology
MegaRAID SAS
9 = SATA+SAS+PCIe
94 6 0 - 16i
Generation
Host Interface Storage

1 PCIe Gen 1 3 Gb/s SAS / SATA

2 PCIe Gen 2/3 6 Gb/s SAS / SATA

3 PCIe Gen 3 12 Gb/s SAS / SATA Subcategory

4
PCIe Gen 3
Tri-mode 0 = HBA Initiator / Target Linear Variant Digit
12 Gb/s SAS/ SATA/ PCIe 3
1 = HBA Integrated RAID Starts at “0”; This digit is used for changes that are meant
5 Tri-mode
PCIe Gen 4
12 Gb/s SAS/ SATA/ PCIe 4 4 = MegaRAID Entry (iMR) show succession from an older version to a newer version.
6
PCIe Gen 4
Tri-mode 6 = MegaRAID Value
24 Gb/s SAS/ SATA/ PCIe 4
8 = MegaRAID Feature
9 | Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Limited and/or its subsidiaries.
Introducing 9400 Family HBA & MegaRAID

• New Chipdesign utilizing ARM A15 1.2Ghz architecture

• Larger L2 Cache and OCM
• TRI-Mode SerDes supporting SAS/SATA/NVMe Media

• Updated Memory Controller with 16% increased bandwith

• Higher Memory Capacity with up to 4GB of DRAM
• Onboard flash memory for automated Cache offload

• Improvements to Fastpath
• Minimized Firmware based IO processing
• Parallel IO Processing

10 | Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Limited and/or its subsidiaries.
HBA 9400 Series Tri-Mode Storage Adapters
9400-16i, 9400-16e, 9400-8i, 9400-8e, 9405W-16i, 9405W-16e
Applications
– High-port count SAS/SATA/NVMe adapters for direct attached high
connectivity applications
– Tri-Mode connectivity enabling maximum data center flexibility
16-port Tri-Mode HBAs
9405W-16i (05-50047-00) – Flexible solutions for cloud computing
9405W-16e (05-50044-00) – External storage requiring high connectivity SAS/SATA interface for host or
drive side connect

Key Features
– Tri-Mode Storage Interface Ports
• SFF-8680 Bay
16-port Tri-Mode HBAs  x1 SAS
9400-16i (05-50008-00)  x1 SATA
9400-16e (05-50013-00)  x2 SAS (Multi Link)
9400-8i8e(05-50031-02)  Two x1 SAS (Dual port usingMPIO)
• SFF-8639 (U.2) Bay
 x2, x4 NVMe
– Supports 12, 6, and 3Gb/s SAS and 6, 3Gb/s SATA data transfer rates
– Up to 8 storage interface PCIe links.
– Each link supporting x4 or x2 link widths up to 8.0 GT/s (PCIe Gen3) per lane
8-port Tri-Mode HBAs – PCIe Gen3 x16 Host Interface for max. throughput
9400-8i (05-50008-01)
9400-8e (05-50013-01)
11 | Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Limited and/or its subsidiaries.
MegaRAID 9400 Series Tri-Mode Storage Adapters
9460-16i, 9460-8i, 9440-8i, 9480-8i8e

Applications
– High-port count SAS/SATA/NVMe controllers for direct attached high
connectivity applications
– Tri-Mode connectivity enabling maximum data center flexibility
– Flexible solutions for cloud computing
16-port Tri-Mode RAID – External storage requiring high connectivity SAS/SATA interface for host or
9460-16i (05-500011-00) drive side connect

Key Features
– Tri-Mode Storage Interface Ports
• SFF-8680 Bay
 x1 SAS
 x1 SATA
 x2 SAS (Multi Link)
8-port Tri-Mode RAID/iMR  Two x1 SAS (Dual port usingMPIO)
9460-8i (05-50011-02) & 9440-8i (05-50008-02) • SFF-8639 (U.2) Bay
 x2, x4 NVMe
– Supports 12, 6, and 3Gb/s SAS and 6, 3Gb/s SATA data transfer rates
– Up to 8 storage interface PCIe links.
– Each link supporting x4 or x2 link widths up to 8.0 GT/s (PCIe Gen3) per lane

8-internal / 8-external -port Tri-Mode RAID

9480-8i8e (05-500031-00)
12 | Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Limited and/or its subsidiaries.
Let SATA SSDs rock!
Improved
Fastpath engine shows best
SSD performance

• 9400 HBA Family performs 74%

higher in Seq. Writes
• 9400 HBA Family increases
bandwith by up to 7% in Writes
• 9400 MegaRAID Family perfoms
up to 44% higer in Seq. Writes
(RAID 5)
• 9400 MegaRAID Family
increases bandwith by up to
13% in Writes (RAID 5)

13 | Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Limited and/or its subsidiaries.
Reduced Power

26% 25%
18 25 Lower in Power
Lower in Power
16

20
14 17% 13%
12
15
10

Competitor

Competitor
8
10
6 9400 9400
9300 9305 9300 9300
4
5
HPC
2

0 0
HBA MegaRAID
Lower is Better Lower is Better

16 | Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Limited and/or its subsidiaries.
Place it right

SAS 9300 family SAS 9305 family SAS 9400 family

• Any standard 8 Port Opportunity • Need for high direct attached • High Performance needs
– Backup/Tape possibility
• Low Power / Heat needs
– External SAS Storage – Software Defined Storage
– Block Storage • NVMe Connection
– Low Latency
– Multiple Expander Systems

17 | Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Limited and/or its subsidiaries.
Major differentiators - HBAs

9300 family 9305 family 9400 family

(Fury) (Cutlass) (Tomcat/Crusader)
Port Count 8 16-24 8-16

Connectivity SAS/SATA SAS/SATA SAS/SATA/NVMe

IR support Yes No No

Performance 1.45M IOPs / 6.000 MB/s 1.5M IOPs / 6.400 MB/s 1.7M IOPs / 6.840 MB/s

Power Consumption 14.5W max. 16.2W max. 11.95W max

Cooling Requirements 200 LFM @ 55°C 200 LFM @ 55°C 200 LFM @ 55°C

Price $ $$ $$$

18 | Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Limited and/or its subsidiaries.
Place it right

MegaRAID 9300 family MegaRAID 9300-16i/24i/8i8e family MegaRAID 9400 family

• Any standard 8 Port Opportunity • Need for high direct attached • High Performance needs
possibility
• SSD Caching • Low Power / Heat needs
• Eliminating Expander issues
• NVMe Connection
– Connectivity issues
– Latency • Flash/SSDs as Primary Storage
• SSD Caching

19 | Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Limited and/or its subsidiaries.
Major differentiators - MegaRAID

9300 family 9300 family 9400 family

(Invader) (Intruder) (Harpoon/Ventura)
Port Count 8 16-24 8-16

Connectivity SAS/SATA SAS/SATA SAS/SATA/NVMe

Cache 1-2GB 2-4GB 2-4GB

CacheVault onboard No Yes (Supercap option) Yes (Supercap option)
Cache Cade support Yes Yes No
LSA support (WebGUI) Yes Yes Yes

Performance 0.9M IOPs / 6.000 MB/s 1M IOPs / 6.400 MB/s 1.4M IOPs / 6.840 MB/s

Power Consumption 20.5W max. 23.7W max. 17.7W max

Cooling Requirements 200 LFM @ 55°C 300 LFM @ 55°C 250 LFM @ 55°C

Price $ $$ $$$

20 | Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Limited and/or its subsidiaries.
Introducing next Generation Management
OS GUI - LSI Storage Authority
• Webbased GUI
• Same Feature-Set like MSM
• No client side installation needed
• Compare to MSM
– Significantly better performance
– Highly secure (as secure as browser)
– Less backend traffic to server
– Can be scripted using ReST
– Has potential to run on BMC

Command Line - Unified StorCLI PreBoot - MegaRAID Human Interface Infrastructure (HII)
• Merge of SASxFlash/SASxIRCU & StorCLI • Integration in servers UEFI BIOS
• Single Point of Management • Single Point of Configuration
• Use of common StorCLI command set • Additional functions (e.g. FW update)
• Enhanced command set

21 | Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Limited and/or its subsidiaries.
SAS / SATA – Just Better!

Lower Power  Up to 25% lower in power consumption

 Reduced cooling requirement

 Up to 247% performance improvement over previous

High
Performance Generation
 Best solution to work with SSD

Updated
Management  Unified StorCLI combines CLI for HBA and MegaRAID
& Monitoring  Webbased GUI LSA

backwards compatible for generations!

22 | Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Limited and/or its subsidiaries.
NVMe Ecosystem Update

Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Limited and/or its subsidiaries.
U.2 Enabler Cable
• x4 NVMe drive connect requires a special cable S1 P1

• Cable re-orders lanes and effectively de-swizzles S0

the adapter layout P0

Min SAS HD to Mini SAS HD
• U.2 Enabler Cable also necessary for maximizing Grantley
card flexibility when connecting to x2 and x1
NVMe drives
P1
P3
P2 S1
P3 P1
P2 P0
P1
P0
Clk S0
P0
Clk
P3 Min SAS HD to SlimLine
P2
P1
P0
Clk
P3
P2
P1
P0
Clk

These cables are for NVMe usage only – unproper

cabling can cause damage of Controller and Media
Min SAS HD to OcuLink
24 | Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Limited and/or its subsidiaries.
Purley
Adapter Cable Information
Ventura/Crusader Adapter Cables
MPN Description Where Used Status
Cable, U.2 Enabler, HD to HD(W) 1M Intel Grantley or backplanes using legacy PCIe Mini SAS HD wiring. Production
05-50061-00
Also works with SMC backplanes, but without LED functionality.

05-50062-00 Cable, U.2 Enabler, HD to OCuLink 1M Intel, SMC Oculink based backplanes Production

05-50063-00 Cable, U.2 Enabler, HD to SlimLine 1M Backplanes with 2x4 Simline connectors Production

05-50065-00 Cable, U.2 Enabler, HD to SFF8639 0.5M Direct connect to SFF-8639 drives. 0.5M cable. Production

05-50064-00 Cable, U.2 Enabler, HD to SFF8639 1M Direct connect to SFF-8639 drives. 1M cable. Production

05-60008-00 Cable, U.2 Enabler, HD to HD(W), SMC 1M Targeted for SMC Purley backplanes and supports LED functionality. Prototype

25 | Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Limited and/or its subsidiaries.
Usage Scenarios – NVMe getting started

• Single 16 Port MegaRAID / HBA

– 2 pcs. x4 NVMe Media
– 8 pcs. SAS/SATA Media

 Best of both Worlds – Performance & Capacity

 Single Monitor and Management
 Saving PCIe Lanes by using single Controller
 NVMe Data Protection utilizing HW MegaRAID

× PCIe x8 Host Connection potential

Performance bottleneck

26 | Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Limited and/or its subsidiaries.
Usage Scenarios – Get the max out of it
D23
D22
D21

• 1x 16 Port MegaRAID / HBA

Avago

LSISAS3516
Ventura
27x27mm D20
ALL
C
RIGHTS
RESERVED
20
14
D19

– 4 pcs. x4 NVMe Media

D18
D17
D16

• 1x 8/16 Port MegaRAID / HBA Avago

LSISAS3516
Ventura
D15
D14

– 8 pcs. SAS/SATA or
27x27mm

C
D13
ALL RIGHTS
20
14
RESERVED

D12
D11
D10

– 20 pcs. SAS/SATA Media D9

D8
D7
D6
 Max of both worlds D5
D4
 Single Monitor and Management D3
D2

Avago

NVMe Data Protection utilizing HW MegaRAID LSISAS3516

Ventura
27x27mm

ALL RIGHTS
20
C
D1
14
RESERVED


D0
Bandwith optimization by using multiple PCIe
x8 Slots

× Multiple Controllers have to be managed

× Higher PCIe lane usage

27 | Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Limited and/or its subsidiaries.
Configuration considerations
Item Answers

• Has the backplane a SAS

Determine what type of NVMe enabled system/backplane will be used.
expander and NVMe slots
• Must use U.2 Enabler Cable
Determine how to properly connect the adapter to NVMe drives. • Option without backplane - direct
attached cables
• SAS/SATA Only Mode(default)
Determine how to set the MegaRAID or IT Profiles to enable NVMe. • NVMe Only Mode
• Mixed Mode
• SMC issues with LED control (on
white MiniSAS HD backplanes)
Status LED’s on SMC slots • FW fix for OCulink backplanes is
available
• Backplane drives activity LED
• NVMe will show up as SCSI device
How will the OS show the NVMe devices
• No differences for the tools (LSA)

28 | Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Limited and/or its subsidiaries.
NVMe / Mixed Mode support
SAS / SATA NVMe only Mixed Mode
only Considerations
MegaRAID 9440-8i   ×
using U.2/NVMe x4 • U.2 Enable Cable occupies
MegaRAID 9460-8i   × 2 Wide Ports (unswizzle)
using U.2/NVMe x4
• NVMe is not supported for
MegaRAID 9460-16i    external connectivity (today)
 
MegaRAID 9480-8i8e  For internal Ports NVMe supported on
only internal Ports only

SAS 9400-8i   ×
using U.2/NVMe x4

SAS 9400-16i   

SAS 9400-8e  × ×

SAS 9400-16e  × ×

SAS 9405W-16e  × ×

SAS 9405W-16i   

29 | Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Limited and/or its subsidiaries.
NVMe Deployment Model

• Profile driven
– MR: Profiles are hold in FW and need to be activated
– HBA: Profiles requires flashing the proper FW
– Possible Profiles:
– SAS/SATA only
– NVMe only
– SAS / SATA / NVMe mixed

• Set profile either by

– StorCLI
– HII (EFI BIOS)

• We ship with SAS/SATA profile as default

– Customer can enable NVMe mixed mode with
– Storcli (profile 13)

30 | Broadcom Proprietary and Confidential. © 2017 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Limited and/or its subsidiaries.