CHAPTER 3

INTERCONNECTION STRUCTURES
OUTLINE
 Introduction
 Bus Interconnection Structures
 Background
 Evolution of Buses
 Operation of the Bus
 Classification of Bus Lines Based on Functions
 Elements of Bus Design
 PCI Bus
 Point-to-point Interconnection Structures
 Background
 PCIe
2
INTRODUCTION
 A computer consists of a set of components or modules
of three basic types, that communicate with each other
 Processor, memory, I/O modules

 All the units must be connected, and the collection of

paths connecting various units is called
 Interconnection structure

 In effect interconnection structures are the glue that

holds computer system together

3
INTRODUCTION…
 Connection requirements of the three basic components
of a computer system (below)

4
INTRODUCTION…
 CPU
 Reads instruction and data
 Writes out data (after processing)
 Sends control signals to other units
 Receives (& acts on) interrupts

5
INTRODUCTION…
 Memory
 Receives and sends data
 Receives addresses (of locations)
 Receives control signals
 Read
 Write

 Timing

6
INTRODUCTION…
 Input/Output Module
 Similar to memory from computer’s viewpoint
 Output
 Receive data from computer
 Send data to peripheral

 Input
 Receive data from peripheral
 Send data to computer

7
INTRODUCTION…
 The interconnection structure must support the following types of transfers
 Memory to processor
 The processor reads an instruction or a unit of data from memory
 Processor to memory
 The processor writes a unit of data to memory
 I/O to processor
 The processor reads data from an I/O device via an I/O module
 Processor to I/O
 The processor sends data to the I/O device
 I/O to or from memory
 An I/O module is allowed to exchange data directly with memory, without going through the
processor, using direct memory access

 The most common interconnection structures are

 The bus and various multiple-bus structures
 E.g. PCI bus (many PCs), ISA bus (PC/AT), EISA (80386), SCSI bus (PCs and workstations), Nubus
(Macintosh), IBM PC bus (PC/XT), Universal Serial Bus (modern PCs), and FireWire (consumer
electronics)
 Point-to-point interconnection structures
 E.g. PCI Express (PCIe), Quick Path Interconnect (QPI)
8
BUS INTERCONNECTION STRUCTURE
BACKGROUND
 What is a BUS ?
 A shared communication pathway connecting multiple (two
or more) devices
 Consists of multiple lines, each line capable of transmitting
signals representing binary 1 or binary 0
 Allowing parallel movement of information
 Physically buses are a little more than bunches of wires

 Key characteristic of a bus

 It is a shared transmission medium
 Multiple devices connect to the bus, and a signal transmitted
by any one device is available for reception by all other
9
devices attached to the bus
EVOLUTION OF BUSES
 Early personal computers had a single external bus, called system bus
 Connecting major components of the computer: CPU, Memory and I/O

 As computer components (CPU, memory, I/O devices) got faster, a singe

bus could no longer handle the load, hence
 Various types of buses have been proposed
 Each having its own speed and performance characteristics

 A hierarchy of buses employed to connect subset of computer components

 Multiples buses laid out in hierarchy
10
EVOLUTION OF BUSES…
 Single Bus Problems
 If many number of devices are connected to a single bus, the
computer system performance will be poor, due to
 More devices means, the greater the bus length, the greater the
propagation delay
 Co-ordination of bus use can adversely affect performance

 The bus becomes a bottleneck as the aggregate data transfer

approaches the capacity of bus, solution

 Increasing the data rate that the bus can carry

 Using the wider bus

 Most early systems use multiple buses, laid out in hierarchy,

to overcome these problem 11
EVOLUTION OF BUSES…
A Traditional Bus Architecture
 A local bus
 Directly connects CPU to cache via cache controller
 The cache controller connects the cache to the system bus also
 A system bus
 Connects main memory, CPU, and some I/O
 An expansion bus
 Ties the system bus to I/O devices

12
EVOLUTION OF BUSES…
 An Enhanced Traditional Bus Architecture
 Incorporates a high speed bus that brings bandwidth intensive
devices into closer integration with the processor via the bridge

13
EVOLUTION OF BUSES…
 Architecture of early Pentium Buses (Mid/late
1990s)

14
EVOLUTION OF BUSES…
 Architecture of Intel PC buses(Mid 2000s)
 North bridge (Memory controller hub)
 Contains high-bandwidth interfaces, connecting the
CPU, memory, and PCIe bus
 Provides a fast communication pathway to the CPU
through the front-side bus
 Provides a fast connection to main memory
 South bridge (I/O controller hub)
 Connects to slower I/O buses, like SATA, USB, and so
forth, that connects slower I/O devices to the
computer system
 Contains legacy interfaces and devices:
 ISA bus (audio, LAN), interrupt controller, DMA
controller, time/counter

15
BUS TERMINOLOGY
 Bus Transaction
 A sequence of bus operations that include a request and may
include a response, either of which may carry data
 A transaction is initiated by a single request and may take many
individual bus operations
 The complete activity of doing either of the following
 Memory Read/Write, I/O Read/Write
 Bus Cycle Time
 The time between two consecutive ticks of the bus clock
 Clock Skew
 Difference in propagation time of signals sent on parallel paths
 Drift in the clock, occurs when signals on different lines travel at slightly
different speed
 The longer the bus and the faster the clock speed/the bus, the
more the skew 16
OPERATIONS OF THE BUS
 If one module wishes to send data to another, it must
 Obtain the use of the bus
 Transfer data via the bus

 If one module wishes to request data from another, it

must
 Obtain the use of the bus
 Transfer a request to the other module over the appropriate
control and address lines
 Wait for the second module to send the data

17
OPERATIONS OF THE BUS…
 Devices attached to the bus are classified into Master
and Slave categories
 Bus masters are active and initiate bus transfers
 Bus slaves are passive, wait for bus transfer requests
 Memory is always a bus slave
 Possible bus master and slave configuration

18
CLASSIFICATION OF BUS LINES
BASED ON FUNCTIONS
 On any bus, the lines can be classified into three
functional groups
 Data, address, and control lines

19
DATA BUS
 The data lines provide a path for moving data among
system modules
 Remember that there is no difference between “data” and
“instruction” at this level
 These lines, collectively, are called the data bus
 The data bus may consist of 32, 64, 128 or more
separate lines
 The number of lines being referred to as width of the data
bus
 Width is a key determinant of performance

20
ADDRESS BUS
 Identify the source or destination of data on the data
bus
 If the processor wishes to read a word (8, 16, or 32…bits) of
data from memory, it puts the address of the desired word
on the address lines

 Address bus width determines maximum memory

capacity of the system
 Example
Intel Processors Address Bus Width, bits Maximum Memory Capacity

8080 16 64K
8086 20 1M
80286 24 16M
21
80386 32 4G
Pentium 4/Core 2 40 1T
CONTROL BUS
 Transmit both command and timing information
 Timing signals indicate the validity of data and address
information
 Command signals specify operations to be performed

 Control the access to and the use of the data and

address lines
 As the data and address lines are shared by all components,
there must be a means of controlling their use

22
CONTROL BUS…
 Lines forming the control bus can be roughly grouped
into the following major categories
 Bus Control, Interrupts, Bus Arbitration, Co-processor
signaling, Status, Miscellaneous

23
CONTROL BUS…
 Bus control
 Memory write
 Causes data on the bus to be written into the addressed location
 Memory read
 Causes data from the addressed location to be placed on the bus
 I/O write
 Causes data on the bus to be output to the addressed I/O port
 I/O read
 Causes data from the addressed I/O port to be placed on the bus
 Transfer ACK
 Indicates that data have been accepted from or placed on the

bus
24
CONTROL BUS…
 Interrupts
 Interrupt request
 Indicates that an interrupt is pending
 Interrupt ACK
 Acknowledges that the pending interrupt has been recognized
 Bus arbitration
 Bus request
 Indicates that a module needs to gain control of the bus
 Bus grant
 Indicates that a requesting module has been granted control of the bus
 Coprocessor signaling
 Status
 Clock
 Used to synchronize operations
 Miscellaneous 25
ELEMENTS OF BUS DESIGN
 Basic design elements that serve to classify and
differentiate buses
 Bus Types
 Dedicated, Multiplexed
 Method of Arbitration
 Centralized, Decentralized
 Timing
 Synchronous, Asynchronous
 Bus Width
 Address, Data
 Data Transfer Types
 Read, Write, Read-modify-write, Read-after-write , Block
26
ELEMENTS OF BUS DESIGN…
 Bus Types
 Dedicated Bus
 Functionally dedicated bus
 A bus line that is permanently assigned to one
function
 E.g. separate data and address lines
 Physically dedicated bus
 Refers to use of multiple buses, each of which
connects only a subset of modules
 E.g. Use of an I/O bus to interconnect all I/O
modules
 Advantage
 High throughput --- less contention
 Disadvantage
 Increases size and cost of the system 27
ELEMENTS OF BUS DESIGN…
 Bus Types
 Multiplexed Bus
 A bus that uses the same lines for multiple purposes at different
times, using time multiplexing
 E.g. Address and data information may be transmitted over the

same set of lines using Address Valid control line (8086)

 Advantage

 Uses fewer lines, which saves space and usually cost

 Disadvantage
 More complex circuitry needed within each module
 Reduction in performance
• Certain events that share the same lines cannot takes place in
parallel

28
ELEMENTS OF BUS DESIGN…
 Method of Arbitration
 In systems with more than one potential bus master device,
What happens if two or more devices all want to become bus
master at the same time ?
 Solution ---- some bus arbitration mechanism is needed to prevent
chaos

 Potential bus master devices include

 CPU, I/O controllers, Coprocessors

 Bus arbitration schemes must

 Provide priority to certain master devices and, at the same time
 Make sure that low priority devices are not starved out

29
ELEMENTS OF BUS DESIGN…
 Method of Arbitration...
 Broadly classified into
 Centralized
 Decentralized

 Centralized
 A single hardware device (bus controller/arbiter) controlling bus access (allocating
time on the bus)
 It may be part of the processor or separate module

 Further divided into: Daisy-Chain arbitration, Centralized parallel arbitration

 Distributed
 There is no central controller
 Each module contains access control logic
 The module act together to share the bus
 Further divided into: Distributed arbitration using self-selection, Distributed
30
arbitration using token passing
ELEMENTS OF BUS DESIGN…
 Centralized Method of Arbitration
 Daisy-Chain Arbitration
 Bus request line can be asserted by one/more devices at any time
 When the arbiter sees a bus request, it issues a grant by asserting

the bus grant line. This line is wired through all of the I/O devices in
series
 Devices are effectively assigned priorities depending on how close
to the arbiter they are. The closest device wins

31
ELEMENTS OF BUS DESIGN…
 Centralized Method of Arbitration...
 Centralized Parallel Arbitration
 Uses multiple request/grant lines, one for each priority level
 Solves, daisy chained arbitration’s implicit priorities,
based on distance from the arbiter
 But, Grant line is daisy chained among devices of same priority level

32
ELEMENTS OF BUS DESIGN…
 Distributed Method of Arbitration
 Distributed Arbitration Using Self-Selection
 Each device has its own request line, which is prioritized
 All devices monitor all the request lines, so at the end of each bus

cycle
 The devices themselves determine who has highest priority and
who should be permitted to use the bus during the next cycle
 Requires more bus lines but avoids the potential cost of the arbiter

 Limits the number of devices to the number of request lines

33
ELEMENTS OF BUS DESIGN…
 Distributed Method of Arbitration...
 Distributed Arbitration Using Token passing
 Uses only three lines, no matter how many devices are present
 The BUSY line is asserted by the current bus master

 The arbitration line is daisy chained through all the devices, passes

(grants)/denies token
 A device holding token has been given exclusive access to the bus

34
ELEMENTS OF BUS DESIGN…
 Timing
 Refers to the way in which events are coordinated on the bus
 Buses use either synchronous timing or asynchronous timing

 Synchronous Bus
 Occurrence of events on the bus is determined by a master bus clock
 All bus activities take an integral number of bus clock cycles/Bus cycles
 All devices on the bus can read clock line
 All events start at the beginning of a clock cycle; Usually a single bus
cycle for an event
 Drawbacks
 Everything works in multiples of the bus cycles
 The bus has to be geared to the slowest one and the fast ones
cannot use their full potential
 When heterogeneous collection of devices, some fast and some
slow are located on the bus 35
 Difficult to take advantage of future improvements in technology
ELEMENTS OF BUS DESIGN…
 Timing...
 Synchronous Bus…
 A simplified timing diagram for synchronous read and write

36
ELEMENTS OF BUS DESIGN…
 Timing...
 Asynchronous Bus
 Control lines coordinate the bus operations/transaction, and a
complex handshaking protocol used to enforce timing
 It does not tie everything to the clock
 Each event is caused by a prior event, not by a clock pulse
 The occurrence of one event on the bus follows and depends on the
occurrence of a previous events
 If a particular master/slave pair is slow, there is no way a subsequent
master/slave pair, that is much faster, is affected
 Scales better with technology and can support a wider variety of
devices (as protocols, not the clock is coordinating transactions)

37
ELEMENTS OF BUS DESIGN…
 Timing...
 Asynchronous Bus…
 A simplified timing diagram for asynchronous write

38
ELEMENTS OF BUS DESIGN…
 Bus Width
 Has an impact on system performance
 The wider the data bus
 The greater the number of bits transferred at one time
 Example

Intel Processors Data Bus Width, bits

8080 8
8086 16
80286 16
80386 32
Pentium 4/Core 2 64

 The wider the address bus

 The greater the range of locations that can be referenced 39
ELEMENTS OF BUS DESIGN…
 Data Transfer Types/ BUS operation types
 Write
 Master to slave
 Read
 Slave to master
 Read-modify-write
 A read followed immediately by a write to the same address
 An indivisible operation, to prevent any access to the data element by
other potential bus masters
 Read-after-write
 An indivisible operation consisting of a write followed immediately by a
read from the same address
 The read operation may be performed for checking purposes
 Block
 One address cycle is followed by n data cycles
 The first data item is transferred to or from the specified address; the 40
remaining data items are transferred to or from subsequent addresses
PCI - BUS
 Introduced by intel in 1993 to succeed older buses, like
ISA, EISA…
 An upgrade to the older ISA bus with higher speeds and
more bits transferred in parallel
 Can be configured as a 32 or 64 bit bus
 Successive generations operate at 33MHz, 66MHz
 Superseded by PCI-X (PCI eXtended) in 2004
 PCI-X basically doubled the bandwidth of regular PCI
 Operates at 133MHz
 Every Intel-based computer since the Pentium has a PCI
bus
 PCI bus can be used in many configurations, (Top-right)
41
PCI - BUS…
 PCI BUS Operation
 A synchronous bus, using centralized arbitration
 Multiplexed Address and data lines, to keep low pin count
 Slave can insert “wait states”, when it is not ready to supply
the requested data, by activating appropriate control line
 Different kinds of bus cycles possible
 Block transfers, ….

42
PCI - BUS… [READING]
 PCI Bus Signals
 PCI Bus Commands and

 PCI BUS Transactions

43
POINT-TO-POINT INTERCONNECTION STRUCTURES
BACKGROUND
 Bus was the dominant means of computer system component
interconnection for decades
 For general-purpose computers, it has gradually given way to various
point-to-point interconnection structures

 Reasons, Why BUS did not rise up to the challenge ?

 Many I/O devices become increasingly too fast for PCI bus
 Increasing further the bus clock frequency not a solution, due to electrical constraints
 Problems with bus skew, crosstalk between the wires, and
capacitance effects just get worse
 At higher and higher data rates, it becomes increasingly difficult

to perform the synchronization and arbitration functions in a

timely fashion
 The advent of multicore chips, with multiple processors and significant memory
on a single chip
 Use of a conventional shared bus on the same chip magnified the difficulties of
increasing bus data rate and reducing bus latency to keep up with the processors 44
POINT-TO-POINT INTERCONNECTION STRUCTURES
BACKGROUND…
 Point to point interconnection structures uses a high speed serial
connection
 As there is no clock skew, a serial connection at much higher speed offsets by big
margin, the loss of parallelism
 Popular Point to point interconnection structures include
 PCIe (PCI Express), QPI (Intel’s Quick Path Interconnect)

 Key characteristics of point-to-point interconnect schemes

 Multiple direct connections
 Multiple components within the system enjoy direct pairwise connections to other
components
 This eliminates the need for arbitration found in shared transmission systems
 Layered protocol architecture
 The processor-level interconnects use a layered protocol architecture, as in TCP/IP-
based data networks, than control signals as found in shared bus arrangements
 Packetized data transfer
 Data sent not as a raw bit stream, rather as a sequence of packets
 Each including control headers and error control codes
45
POINT-TO-POINT INTERCONNECTION
PCIE
 Introduced in 2004, to replace PCI and its successor PCI-X
 Progressively replaced the AGP (accelerated graphics port) graphics
interface designed by Intel specifically for 3D graphics; PCIe generation
depicted (below)
 Represents a radical change from the PCI bus
 In fact, it is not even a bus at all
 It is point-to-point network using bit-serial lines and packet switching, more like
the Internet than like a traditional bus
PCIe Year Transfer Effective one-way
Generation Introduced Rate data rate
1.0a 2003 2.5GT/s 250 MB/s
2.0 2007 5GT/s 500 MB/s
3.0 2010 8GT/s 985 MB/s
4.0 2017 16GT/s 1.969 GB/s
5.0 2019 32GT/s 3.938 GB/s 46
6.0 2022 64GT/s 7.563 GB/s
7.0 2025 (planned) 128 GT/s 15.125 GB/s
END OF CHAPTER 3

47