Linux Clusters Institute:

Introduction to
High Performance Computing
University of Wyoming
May 22 – 26, 2017

Irfan Elahi, National Center for Atmospheric Research

1
What is Supercomputing or High
Performance Computing?
• The definition of supercomputing is constantly changing.
Supercomputers can perform up to quadrillions of FLOPS or PFLOPS.
• High-performance computing (HPC) utilizes parallel processing for
running large and advanced application programs efficiently. The term
applies especially to systems that function above a hundred teraflops.
The Top500 list has several multi-petaflop systems in the top 50.
• HPC aggregates computing power in a way that delivers much higher
performance than one could get out of a typical desktop computer or
workstation in order to solve large problems in science, engineering, or
business.
• Supercomputers were introduced in the 1960s, and initially created by
Seymour Cray at Control Data Corporation who led the HPC industry for
decades.
• To me, personally, it is an eco-system that provides users with a high
performance computational, networking, storage, and analysis
platform, and the necessary software stack to stitch these resources.

2
Fastest Supercomputer vs. Moore

100000000
Year
1993
Fastest
59.7
Moore
60
1994 143.4
1995 170.4

10000000
1996
1997
220.4
1068
240

1998 1338
1999 2121.3 960

1000000
2000 2379
2001 7226
2002 35860 3840
2003 35860 18000000

100000
2004 35860
2005 136800 15360 16000000
2006 280600 14000000
2007 280600
2008 1375780 61440 12000000
2009
2010
10000
1456700
1759000
10000000
Fastest Fastest
8000000
Moore
Moore
2011 8162000 245760
2012 16324750 6000000

1000 4000000

GFLOPs:
2000000
0
100 1990 1995 2000 2005 2010 2015

10
1993: 1024 CPU cores billions of
calculations
1 per second
1990 1995 2000 2005 2010 2015 www.top500.org

Year http://www.mooreslaw.org/

3
What is Supercomputing About?

Size
Speed

4
What is Supercomputing About?

• Size: Many problems that are interesting to scientists and engineers

can’t fit on a single personal computer system – usually because they
need more RAM and/or more disk storage.

• Speed: Many problems that are interesting to scientists and engineers

would take a very, very long time to run on a single personal computer
system - months or even years. But a problem that would take a month
on a PC might take only an hour on a supercomputer.

5
What is HPC Used For?
• Simulation of physical phenomena by developing a model
that represents the key characteristics of the selected
physical or abstract system or process. Areas where [1]
simulation is heavy used:
• Weather forecasting
• Galaxy formation
• Oil reservoir management

• Data mining: finding needle(s) in a haystack. It is the [2]

process of analyzing data from different perspectives and
summarizing it into useful and sometimes new information: Moore, OK
• Gene sequencing Tornadic
• Signal processing Storm
• Detecting storms that might produce tornados May 3, 1999

• Visualization: turning a vast sea of data into pictures that a

scientist can understand and analyze. Any technique for
creating images, diagrams, or animations to communicate a [3]
message is called Visualization.

6
Supercomputing Issues

• Storage hierarchy or storage tiers

• Parallelism: doing multiple things at the same time

• Scaling issues

• High-speed interconnect

• Software stack

• Facility

7
What is a “Cluster”?

• A cluster needs a collection of small computers, called

nodes, hooked together by an interconnection
network (or interconnect for short).
• It also needs software that allows the nodes to
communicate over the interconnect.
• A cluster is all of these components working together
as if they’re one big computer ... a super computer.

8
What Does a Cluster
Look Like?
Network View

9
10
What Does a Cluster Look Like?

11
Cluster Components
All Components Working Together

• Computational resources

• Storage and file system

• Management infrastructure

• High-speed interconnect

• HPC software stack

• HPC applications and workflow

12
Cluster Components
All Components Working Together

User Applications

HPC Software Stack: OS, Compilers, Libraries,

MPI, Programming Tools, Debuggers,
Scheduler, etc.

Network: Load balancing, discovery, failover

File System Cluster: Provisioning, cluster commands, etc.

Server: Discovery, remote power, diskless
support

Management/Service Compute Nodes Network (IHPI) Storage

13
Computational Resources

• Compute nodes and software

• Compute node has:

• Processor (CPU)
• Memory
• Networking
• Software
• Access to file system for long or short term retention

14
Processor Types Examples

• X-86 Architecture:
• The instruction set architecture (ISA) is Intel's most successful line of processors.
• Xeon & Xeon Phi - Many-Core (Intel) and Opteron (AMD)
• GPGPU or GPU:
• General-purpose computing on graphics processing units is the use of a graphics processing
unit (GPU), which typically handles computation only for computer graphics, to perform
computation in applications traditionally handled by the central processing unit (CPU).
• NVIDIA, AMD, ASUS, etc., manufacture GPGPU/GPU.
• POWER: (Power Optimization with Enhanced RISC)
• IBM has a series of high performance microprocessors called POWER.
• IBM launched OpenPOWER Foundation for collaboration on their Power Architecture in
2013. Google, Tyan, Nvidia, and Mellanox are founding members.
• ARM (Advanced RISC Machines)
• CPUs based on the RISC (reduced instruction set computer) architecture developed by
Advanced RISC Machines (ARM).
• Companies using ARM cores on their chips are Qualcomm, Samsung Electronics, Texas
Instruments, and Cavium among others.

15
Storage and File System

• Used to control how data is stored and retrieved

• Short and long term retention

• HPC or Cluster’s require shared FS:

• Storage (Disk)
• Software (Lustre/GPFS)
• Transport/Networking

16
Management Infrastructure

• Cluster Management

• Cluster Network Management

• Service Nodes

• Head/Login Nodes

• Facility

17
Cluster Management

• Node provisioning, hardware/power control, discovery,

and OS disk-based/disk-less deployment
• Monitoring and log management

• Fabric management

• Cluster startup and shutdown

• Parallel shell
• One ring to rule them all, one ring to find them, one ring to
bring them all, and in the darkness bind them.

18
Facility

• Power
• 120 V, 280 V, 440V, etc.
• 3-Phase, DC
• N + N or N+1 Redundancy
• UPS, Generators

• Cooling
• Water cooled or air cooled

• Racks 19
HPC Software Stack

• Cluster Management Software

• Operating System

• Compilers, Programming Tool, and Libraries

• File System

• Scheduler/Resource Manager

20
High-Speed Interconnect

• Low latency, high-bandwidth

• Library support, FCA, MPI offload, RDMA, etc.
• Fabric management
• Examples:
• Ethernet: performance is expected to hit 400 Gbps soon
• Infiniband: EDR = ~100 Gbps, HDR = ~200 Gbps and NDR
= ~400 Gbps
• OPA: Gen1 =~100 Gbps

21
HPC Applications and Workflow

• Parallelism:
• Speedup is not linear

• Dependencies

• Tuning:
• Race conditions
• Mutual exclusions
• Synchronization

22