PCI-SIG®

Architecture Overview
Richard Solomon
LSI Corporation

Copyright © 2009, PCI-SIG, All Rights Reserved 1

 What’s all this PCI stuff anyway?
ƒ Presentation will cover basic concepts and their
evolution from PCI™ through PCI-X™ to PCI
Express®
9 Specs written assuming designers have these key
background concepts
9 High level overview of PCI, PCI-X, PCI Express, and
I/O Virtualization
9 Brief description of compliance program

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 2

 PCI Background

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 3

 Revolutionary AND Evolutionary
ƒ PCI
9 Revolutionary
– Plug and Play jumperless configuration (BARs)
– Unprecedented bandwidth
• 32-bit / 33MHz – 133MB/sec
• 64-bit / 66MHz – 533MB/sec
– Designed from day 1 for bus-mastering adapters

9 Evolutionary
– System BIOS maps devices then operating systems boot and
run without further knowledge of PCI
– PCI-aware O/S could gain improved functionality

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 4

 Revolutionary AND Evolutionary
ƒ PCI-X
9 Revolutionary
– Unprecedented bandwidth
• Up to 1066MB/sec with 64-bit / 133MHz
– Registered bus protocol
• Eased electrical timing requirements
– Brought split transactions into PCI “world”

9 Evolutionary
– PCI compatible at hardware *AND* software levels
– PCI-X 266/533 added as “mid-life” performance bump
• 2133MB/sec at PCI-X 266 and 4266MB/sec at PCI-X 533

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 5

 Revolutionary AND Evolutionary
ƒ PCI Express (aka PCIe®)
9 Revolutionary
– Unprecedented bandwidth
• x1: 500MB/sec in *EACH* direction
• x16: 8000MB/sec in *EACH* direction
– “Relaxed” electricals due to serial bus architecture
• Point-to-point, low voltage, dual simplex with embedded clocking

9 Evolutionary
– PCI compatible at software level
• Configuration space, Power Management, etc
• Of course, PCIe-aware O/S can get more functionality
– Transaction layer familiar to PCI/PCI-X designers
– System topology matches PCI/PCI-X
– PCIe 2.0 doubled bandwidth from 250MB/s/lane to 500MB/s/lane
– PCIe 3.0 will double again to 1GB/s/lane!

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 6

 PCI Concepts

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 7

 PCI Concepts
ƒ Address spaces
9 Memory – 64-bit
9 I/O – 32-bit (non-burstable since PCI-X)
9 Configuration (“Config”) – Bus/Device/Function

ƒ Key configuration space regs/concepts

9 Base Address Registers (BARs)
– 64-bit vs 32-bit addressing
9 Linked list of capabilities

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 8

 Address Spaces – Memory & I/O
ƒ Memory space mapped cleanly to CPU semantics
9 32-bits of address space initially
9 64-bits introduced via Dual-Address Cycles (DAC)
– Extra clock of address time on PCI/PCI-X
– 4DWORD header in PCI Express
9 Burstable
ƒ I/O space mapped cleanly to CPU semantics
9 32-bits of address space
– Actually much larger than CPUs of the time
9 Non-burstable
– Most PCI implementations didn’t support
– PCI-X codified
– Carries forward to PCI Express

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 9

 Address Spaces – Configuration
ƒ Configuration space???
9 Allows control of devices’ address decodes without conflict
9 No conceptual mapping to CPU address space
– Memory-based access mechanisms introduced with PCI-X and
PCIe
9 Bus / Device / Function (aka BDF) form hierarchy-based address
– “Functions” allow multiple, logically independent agents in one
physical device.
• E.g. combination SCSI + Ethernet device
• 256 bytes or 4K bytes of configuration space per device
– PCI/PCI-X bridges form hierarchy
– PCIe switches form hierarchy
• Look like PCI-PCI bridges to software
9 “Type 0” and “Type 1” configuration cycles
– Type 0: to same bus segment
– Type 1: to another bus segment

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 10

 Configuration Space (cont’d)
Processor Processor Processor Processor

Address Port Data Port Address Port Data Port

Main Host/PCI Bridge

Host/PCI Bridge
Memory Bus = 4
Bus = 0
Subord = 3 Subord = 5

PCI Bus 0 PCI Bus 4

PCI-to-PCI PCI-to-PCI
Bridge Bridge

Primary = 0 Primary = 4
Secondary = 1 Secondary = 5
Subord = 3 Subord = 5

PCI Bus 1 PCI Bus 5

PCI-to-PCI PCI-to-PCI
Bridge Bridge

Primary = 1 Primary = 1
Secondary = 2 Secondary = 3
Subord = 2 Subord = 3

PCI Bus 2

PCI Bus 3

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 11

 Using Configuration Space
Doubleword
Number

ƒ Device Identification 3 2
Byte
1 0
(in decimal)

9 VendorID: PCI-SIG assigned Device

ID
Vendor
ID
00
Status Command 01

9 DeviceID: Vendor self-assigned Register

Class Code
Register
Revision
ID
02

9 Subsystem VendorID: PCI-SIG BIST Header

Type
Latency Cache
Timer Line
Size
03

04
Base Address 0
9 Subsystem DeviceID: Vendor Base Address 1 05

ƒ Address Decode controls Base Address 2 06

Base Address 3 07

9 Software reads/writes BARs to Base Address 4 08

09
determine required size and maps Base Address 5
10
appropriately CardBus CIS Pointer

Subsystem ID Subsystem 11
Vendor ID
9 Memory, I/O, and bus-master Expansion ROM
Base Address
12

enables Reserved Capabilities

Pointer
13

14

ƒ Other bus-oriented controls

Reserved

Max_Lat Min_Gnt Interrupt Interrupt

15
Pin Line

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 12

 Using Configuration Space –
Capabilities List Doubleword
Number
(in decimal)
Byte
3 2 1 0
Bit 4 Device
ID
Vendor
ID
00
Status Command 01
Register Register
Class Code Revision 02
ID
Header Latency Cache 03
BIST Type Timer Line
Size

Base Address 0 04

Base Address 1 05

Base Address 2 06

Base Address 3 07

Base Address 4 08

Base Address 5 09

CardBus CIS Pointer 10

Subsystem ID Subsystem 11
Vendor ID
Expansion ROM 12
Base Address
Reserved Capabilities
Pointer
13 Capabilities List “Head”
Reserved 14

Max_Lat Min_Gnt Interrupt Interrupt

15
Pin Line

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 13

 Using Configuration Space –
Capabilities List (cont’d)
ƒ Linked list
9 Follow the list! Cannot assume fixed location of any
given feature in any given device
9 Features defined in their related specs:
– PCI-X
– PCIe
– PCI Power Management
– Etc…
31 16 15 8 7 0
Pointer to
Feature-specific Next Capability Capability ID Dword 0
Dword 1
Configuration Registers
Dword n

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 14

 Using Configuration Space –
Extended Capabilities List
ƒ PCI Express only
ƒ Linked list
9 Follow the list! Cannot assume fixed location of any
given feature in any given device
9 First entry in list is *always* at 100h
9 Features defined in PCI Express specification

31 20 19 16 15 8 7 0
Pointer to Next
Version Capability ID Dword 0
Capability
Dword 1
Feature-specific Configuration Registers
Dword n

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 15

 Interrupts
ƒ PCI introduced INTA#, INTB#, INTC#, INTD# -
collectively referred to as INTx
9 Level sensitive
9 Decoupled device from CPU interrupt
9 System controlled INTx to CPU interrupt mapping
9 Configuration registers
– report A/B/C/D
– programmed with CPU interrupt number
ƒ PCI Express mimics this via “virtual wire”
messages
9 Assert_INTx and Deassert_INTx

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 16

 What are MSI and MSI-X?
ƒ Memory Write replaces previous interrupt
semantics
9 PCI and PCI-X devices stop asserting INTA, INTB,
INTC, INTD once MSI or MSI-X mode is enabled
9 PCI Express devices stop sending Assert_INTx and
Deassert_INTx TLPs once MSI or MSI-X mode is
enabled

ƒ NOTE: Boot devices and any device intended for

a non-MSI operating system generally must still
support the appropriate INTx signaling!

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 17

 MSI vs MSI-X
ƒ MSI uses one address with a variable data value
indicating which “vector” is asserting
ƒ MSI-X uses a table of independent address and
data pairs for each “vector”
9 Allows software to control aliasing (when fewer
vectors are allocated than requested)
9 Table size supports more vectors than MSI structure
allowed

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 18

 PCI-X Explained

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 19

 What is PCI-X?
ƒ “PCI-X is high-performance backward
compatible PCI”
9 PCI-X uses the same PCI architecture
9 PCI-X leverages the same base protocols as PCI
9 PCI-X leverages the same BIOS as PCI
9 PCI-X uses the same connector as PCI.
9 PCI-X and PCI products are interoperable
9 PCI-X uses same software driver models as PCI
ƒ PCI-X is faster PCI
9 PCI-X 533 is up to 32 times faster than the original
version of PCI
9 PCI-X protocol is more efficient than conventional PCI

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 20

 PCI-X Modes and Speeds
64-Bit 32-Bit Error Conf
16-Bit DIM
Mode VI/O Slots* MB/s Slots* MB/s Prot Bytes

PCI 33 5V/3.3V 266 133 N/A par 256 N/A

PCI 66 3.3V 533 266 N/A par 256 N/A

par or
PCI-X 66 3.3V 533 266 N/A 256 yes
ECC

Mode 1 PCI-X 133

(operating at 3.3V 800 400 N/A
par or
256 yes
ECC
100 MHz)

par or
PCI-X 133 3.3V 1066 533 N/A 256 yes
ECC

PCI-X 266 1.5V 2133 1066 533 ECC 4K yes

Mode 2
PCI-X 533 1.5V 4266 2133 1066 ECC 4K yes

* For lower bus speeds, # slots / bus is implementation choice to share bandwidth

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 21

 Registered Bus Protocol
ƒ PCI @ 33MHz
1 2

PCI Clock, 33 MHz

Propagation delay across bus

930 ns period
Sender

97 ns setup time
Receiver Decodes Receiver
Asserts Logic Responds
Signal

1 2 3
ƒ PCI @ 66MHz
PCI Clock, 66 MHz
9 15 ns period
Sender
9 3ns setup time
Asserts Receiver

ƒ PCI-X registered
Signal Responds

Propagation delay
across bus
Receiver Decodes Logic protocol allocates
3

PCI -X Clock
1 2
a full clock period
for logic decision
Sender
Asserts
Signal
Receiver
registers Receiver Receiver 9 @ 66MHz - 15ns
signal Decodes Logic Responds

9 @ 133MHz - 7.5ns
Propagation delay across bus

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 22

 PCI 2.x/3.0 vs. PCI-X Mode 1
ƒ Same bus and control signals
ƒ Evolutionary protocol changes New “Attribute”
phase for
ƒ Clock frequency up to 133 MHz enhanced features
1 2 3 4 5 6 7 8 9 10 11 12
1 2 3 4 5 6 7 8 9 10 11 12
PCI_CLK
PCI_CLK
ADDRESS ATTR DATA-0 DATA-1 DATA-2 DATA-3 DATA-4 DATA-5
AD
AD ADDRESS DATA-0 DATA-1 DATA-2 DATA-3 DATA-4 DATA-5

C/BE# BUS CMD ATTR BE#'s-0 BE#'s-1 BE#'s-2 BE#'s-3 BE#'s-4 BE#'s-5
C/BE# BUS CMD BE#'s-0 BE#'s-1 BE#'s-2 BE#'s-3 BE#'s-4 BE#'s-5

FRAME#

Data Transfer

Data Transfer

Data Transfer

Data Transfer

Data Transfer

Data Transfer
FRAME#
Data Transfer

Data Transfer

Data Transfer

Data Transfer

Data Transfer

Data Transfer

IRDY#
IRDY#
TRDY#
TRDY#

DEVSEL# DEVSEL#

Bus Transaction Bus Transaction

(Common clock)

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 23

 PCI-X 66/133 (Mode 1) vs.
PCI-X 266/533 (Mode 2)
ƒ Same bus and control signals
ƒ PCI-X 266 moves 2x the data
PCI-X 533 moves 4x the data 4 transfers per
ƒ Clock frequency up to 133 MHz clock cycle
1 2 3 4 5 6 7 8 9 10 11 1 2 3 4 5 6 7 8 9 10 11 12 1
PCI_CLK
PCI_CLK
(Data)

AD ADDRESS ATTR DATA-0 DATA-1 DATA-2 DATA-3 DATA-4 DATA-5 AD ADDR ATTR 0 1 2 3 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

(Strobe)
C/BE# BUS CMD ATTR BE#'s-0 BE#'s-1 BE#'s-2 BE#'s-3 BE#'s-4 BE#'s-5
C/BE# BUS CMD ATTR

FRAME#
Data Transfer

Data Transfer

Data Transfer

Data Transfer

Data Transfer

Data Transfer

FRAME#
IRDY#

Data Transfer

Data Transfer

Data Transfer

Data Transfer

Data Transfer

Data Transfer
IRDY#
TRDY#
TRDY#
DEVSEL#
DEVSEL#
Bus Transaction

PCI-X 66/133 (Mode 1) PCI-X 533 (Mode 2)

source-
synchronous
data strobes
PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved
share C/BE pins 24
 Transaction Attributes
Requester Attributes for Burst Transactions
35 32 31 30 29 28 24 23 16 15 11 10 08 07 00
Requester Requester
Upper N R Requester Lower
R Tag Bus Function
Byte Count S O Device Number Byte Count
Number Number
C/BE[3-0]# AD[31:0]

Requester Attributes for DWORD Transactions

35 32 31 30 29 28 24 23 16 15 11 10 08 07 00
Requester Requester
N R Requester
Byte Enables R Tag Bus Function Reserved
S O Device Number
Number Number
C/BE[3-0]# AD[31:0]

RO -- Relax ordering
NS -- No Snoop
R -- Reserved

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 25

 Transaction Attributes
Split Completion Address
3 0 31 30 29 28 24 23 16 15 11 10 08 07 06 00

Requester Requester Requester

R Tag
BUS CMD R R Bus Device Function R Lower Address [6:0]
O
Number Number Number
C/BE[3-0]# AD[31:0]

RO -- Relaxed ordering

Completer Attributes
3 0 31 30 29 28 24 23 16 15 11 10 08 07 00
B S S Completer Completer Completer
Upper Lower
C C C R Bus Device Function
Byte Count Byte Count
M E M Number Number Number
C/BE[3:0]# AD[31:0]

SCM -- Split Completion Message

SCE -- Split Completion Error
BCM -- Byte Count Modified
R -- Reserved

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 26

 Split Transactions
ƒ Bus efficiency of Read almost as good as Write
ƒ Split Completion routed back to requester across
bridges using initiator’s number and bus number
ƒ Split Transaction components
9 Step 1. Requester requests bus and arbiter grants bus
9 Step 2. Requester initiates transaction
9 Step 3. Target (completer) communicates intent with new
target termination, Split Response
9 Step 4. Completer executes transaction internally
9 Step 5. Completer requests bus and arbiter grants bus
9 Step 6. Completer initiates Split Completion

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 27

 Efficient PCI-X Protocol
Bandwidth Usage with Conventional PCI Protocols Bandwidth Usage with PCI-X Protocols,
included in PCI-X 2.0
275 275
100% 100%
Idle Time System Overhead
250 Idle Time 250 -- Scheduling
-- Unused BW
-- Unused BW 90% 90%
225 225 Transaction Overhead
80% 80% -- Addressing and Routing

Percent of Total Bandwidth

Bandwidth MegaBytes/sec

Bandwidth MegaBytes/sec
200 200

Percent of Total BandWidth

System Overhead 70% 70%
-- Scheduling
175 175
60% 60%
Transaction Overhead
150 150
-- Addressing and Routing 50% 50%

125 125
40% 40%

100 30% 100 30%

Transaction
Data Payload
50 Transaction Data Payload 20% 50 -- Actual user 20%
-- Actual user data data
25 10% 25 10%

1 2 3 4 5 1 2

Number of Load Exerciser Cards Number of Load Exerciser Cards

The PCI-X protocol is more efficient than traditional PCI.

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 28

 PCI Express Overview

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 29

 PCIe Architecture Features
ƒ PCI Compatibility ƒ Protocol
9 Configuration and PCI software 9 Fully packetized split-transaction
driver model 9 Credit-based flow control
9 PCI power management 9 Hierarchical topology support
software compatible 9 Virtual channel mechanism

ƒ Performance ƒ Advanced Capabilities

9 Scalable frequency (2.5-5GT/s) 9 CRC-based data integrity, hot
9 Scalable width (x1, x4, x8, x16) plug, error logging
9 Low latency and highest
utilization (Bandwidth/pin)
ƒ Enhanced Configuration
Space
ƒ Physical Interface 9 Extensions and bridges into
9 Point-to-point, dual-simplex other architectures
9 Differential low voltage signaling
9 Embedded clocking
9 Supports connectors, modules,
cables

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 30

 PCIe Protocol Overview
ƒ PCI-X Address/Attribute phases:

ƒ Evolved into the PCIe Packet Header:

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 31

 PCIe Protocol Overview
ƒ The packet bytes get converted to 8b/10b
and serialized

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 32

 PCIe Protocol Overview
Header and Payload from
Device Application/Transaction
Layer

Start Sequence Header Data Payload ECRC LCRC End

1B 2B 3-4 DW 0-1024 DW 1DW 1DW 1B

Created by Transaction Layer

Appended by Data Link Layer

Appended by Physical Layer

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 33

 PCIe Protocol Overview
ƒ Framing varies depending on link width
9 x1

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 34

 PCIe Protocol Overview
ƒ Framing varies depending on link width
9 x4

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 35

 PCIe Power Management
ƒ PCI Power Management – Device Layer
9 Introduced with Conventional PCI
9 Software-driven, addresses devices not in use

9 D0 – operating normally
9 D1 – reduced power
9 D2 – reduced power
– In both D1/D2, the device can’t *do* anything on the
bus except respond to configuration cycles
9 D3cold – device powered off
9 D3hot – device all but powered off

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 36

 PCIe Power Management
ƒ Link Layer
9 L0 – operating normally
9 L0s – one or both sides in low power mode
9 L1 – lower powered mode
9 L2 – deep power save mode but Vaux provided
9 L3 – powered off
ƒ Active State Power Management (ASPM)
9 Software-invisible, addresses idle devices
9 L0s required – enter on timer, exit on activity
9 L1 optional – enter on timer or other conditions,
exit on activity
PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 37
 PCIe Power Management
ƒ Dynamic Power Allocation (DPA)
9 New configuration space reporting and selecting
different power “envelopes” for each device
function
– ECN against PCIe 2.0 (appears in PCIe 2.1 and 3.0)
– Endpoint-only
9 Addresses fully functioning devices
9 Software-set
– Allows trading off performance for power
– Exact “cost” of reduced power is device-dependent

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 38

 PCIe Specifications
Software B – Base
9 Electrical

Transaction a 9
9
Protocol
Configuration

Form Factors: Data Link s – Bridge

– I/O Virtualization
– Card (CEM) e
– Mini Card (MiniCEM) Physical
– ExpressModule Mechanical
– Cable

ƒ Layered, scalable architecture

ƒ Performance matched to applications

ƒ Innovative form factors

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 39
 I/O Virtualization Overview

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 40

 I/O Virtualization – What?
From an adapter point of view:
ƒ One physical device looks like multiple devices
ƒ Virtual devices appear completely independent
9 May occupy different PCI memory ranges
9 May have different settings for PCI Configuration registers
From a system point of view:
ƒ “System Image” is a real or virtual system of CPU(s),
Memory, O/S, I/O, etc
9 Multiples may run on one or more sets of hardware
ƒ Each “System Image” (SI) needs to “see” it’s own PCI
hierarchy
9 Even if NO end devices are actually shared
9 Only its “portion” of shared end devices
PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 41
 I/O Virtualization – How?
ƒ Attachment of existing PCIe 1.x Base
components
9 Root Complexes, Switches, Endpoints, and
Bridges.
Multi-Root
ƒ A solution to use a combination of existing
Single Root
base and IOV-aware components:
Base 9 Single Root capabilities are a superset of the
PCIe 1.x Base specification.
9 Multi-Root capabilities build upon the Base and
Single Root capabilities.
ƒ IOV-capable components are backwards
“Concentric Circles” model
compatible with existing software.
9 Although some or all of the new IOV capabilities
may not be supported in these circumstances.

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 42

 I/O Virtualization “Flavors”
ƒ Single Root (SR IOV)
9 Fits into existing PCI hierarchies today, with address
space partitioned/allocated “above” the Root Complex
– Uses RoutingID (formerly the Bus/Device/Function field) in
packets to track back to appropriate System Image
9 Existing or absolutely minimally changed Root
Complex silicon
9 Existing or minimally changed Switch silicon
9 New Endpoint silicon
9 Presumes existence of a Virtualization Intermediary
– Opens market to lots of existing or simply-derived systems
– Shifts substantial burden to software

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 43

 I/O Virtualization “Flavors”
ƒ Single Root Hierarchy CPU #1
CPU #2
CPU #3
CPU #n

Chipset

PCI Express
“Root Complex”
Port(s)

PCI Express PCI Express

PCI Express
Endpoint Endpoint
Switch
Device Device

PCI Express PCI Express

PCI Express PCI Express
Endpoint Endpoint
Switch Switch
Device Device

PCI Express PCI Express PCI Express PCI Express PCI Express
Endpoint Endpoint Endpoint Endpoint Endpoint
Device Device Device Device Device

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 44

 I/O Virtualization “Flavors”
(cont’d)
ƒ Multi-Root (MR IOV)
9 Most obvious example is a blade server with a PCIe
“backplane”
9 New PCIe hierarchy construct
– Effectively a (mini) fabric
– Logically partitions the hierarchy into multiple Virtual Planes
(VPs) all sharing the same physical hierarchy
9 Existing or absolutely minimally changed Root
Complex (i.e. chipset) silicon
9 New Switch silicon
– Allows for use of existing or minimally changed switches in a
reduced capacity in certain places
9 New Endpoint silicon
PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 45
 I/O Virtualization “Flavors”
(cont’d)
ƒ Multi-Root Hierarchy

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 46

 I/O Virtualization “Flavors”
(cont’d)
ƒ Address Translation Services (ATS)
9 Defines a set of transactions PCIe components can
use to exchange and share translated addresses
– With an I/O MMU, bus addresses map to different system
addresses based on the “identity” of the agent using them
• Allows each SI to appear to use the entire address space
• System’s I/O MMU does translation of “normal” addresses
• Expensive in performance terms
• Impossible to size I/O MMU’s TLB or cache for all applications
• ATS-aware devices can translate an address range and bypass
I/O MMU
– New PCIe commands for translation Requests, Completions,
and Invalidations
9 Implementation is optional even for IOV devices

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 47

 Thank you for attending the
PCI-SIG Developers Conference 2009

For more information please go to

www.pcisig.com

PCI-SIG Developers Conference Copyright © 2009, PCI-SIG, All Rights Reserved 48