Parallel and

Distributed Computing
Lecture # 01

Spring 2021
 Agenda
2

 Motivating Parallelism

 Computing vs Systems

 Parallel vs Distributed Computing

 Practical Applications of P&D Computing

 Organization and Contents of the Course

CS443
3
Motivating Parallelism

CS443
 Motivating Parallelism
4

 Developing parallel hardware and software has traditionally been

time and effort intensive.

 If one is to view this in the context of rapidly improving

uniprocessor speeds, one is tempted to question the need for parallel
computing.
 Latest trends in hardware design indicate that uni-processors may not be able
to sustain the rate of realizable performance increments in the future .

 This is the result of a number of fundamental physical and computational

limitations.

 The emergence of standardized parallel programming environments, libraries,

and hardware have significantly reduced time to develop (parallel) solution.
CS443
 Motivating Parallelism
5

Moore’s Law
 Proposed by Gorden E. Moore in 1965 and revised in 1975.

 It states that [Simplified Version]

“Processing speeds, or overall processing power for
computers will double every18 months.”

 A more technically correct interpretation

“The number of transistors on an affordable CPU doubles
after every two year [18 months].”

CS443
 Motivating Parallelism
6

Moore’s Law
 More computational power implicitly means more transistors.
 Then why need second interpretation?
 Let’s have a look on empirical data from 1970 to 2009
 In 1970’s (i.e., from 1970 to 1979), processor speeds ranged from 740 KHz to
8 Mhz. Difference shows that both the interpretations are correct.
 From 2000 to 2009, Speeds ranged from 1.3 GHz to 2.8 GHz.
 Relative speed difference is too low but, number of integrated transistors
ranged from 37.5 million to 904 million.
 So, second interpretation is more accurate.

CS443
 Motivating Parallelism
7

Moore’s Law
 Why doubling the transistors does not doubles the speed?
 Increase in number of transistor per processor is due to multi-core CPU’s.
 It means, to follow Moore’s law, companies had to:
 Introduce ULSI( ultra large-scale integrations)
 And multi-core processing era.
 Will Moore’s law hold forever?
 Adding multiple cores on single chip causes heat issues.
 Furthermore, increasing the number of cores, may not be able to increase
speeds [Due to inter-process interactions].
 Moreover, transistors would eventually reach the limits of miniaturization
at atomic levels

CS443
 Motivating Parallelism
8

Moore’s Law
 So, we must look for efficient parallel [multi-core] software solutions
to fulfill our future computational needs.

 As stated earlier, number of cores on a single chip also have some

restrictions.

 Solution[s]?
 Need to find more scalable distributed and hybrid solutions

CS443
 Motivating Parallelism
9

The Memory/Disk Speed Argument

 While clock rates of high-end processors have increased at roughly
40% per year over the past decade, DRAM access times have only
improved at the rate of roughly 10% per year over this interval.
 This mismatch in speeds causes significant performance bottlenecks.
 Distributed platforms provide increased bandwidth to memory of
the system.
 Parallel platforms also provide higher caches [aggregated].
 Some of the fastest growing applications of parallel computing
utilize not their raw computational speed, rather their ability to pump
data to memory and disk faster.

CS443
 Motivating Parallelism
10

The Data Communication Argument

 As the network evolves, the vision of the Internet as one large
computing platform has emerged.

 In many applications like databases and data mining problems, the

volume of data is such that it (sometimes) cannot be moved.

 Any analyses on this data must be performed over the network using
parallel computing techniques

CS443
 Computing vs Systems
11

Distributed Systems
 A collection of autonomous computers, connected through a network
and distribution middleware.
 This enables computers to coordinate their activities, and to share the resources
of the system.
 The system is usually perceived as a single integrated computing facility.
 Mostly concerned with the hardware-based accelerations
Distributed Computing
 A specific use of distributed systems to split a large and complex
processing into subparts, and then execute the parts in parallel to
increase the productivity.
 Computing mainly concerned with software-based accelerations (i.e.,
designing and implementing parallel algorithms)
CS443
 Parallel vs Distributed Computing
12

Parallel (shared-memory) Computing

 The term is usually used for developing concurrent solutions on the
following two types of the systems:
1. Multi-core Architecture
2. Many-core architectures (i.e., GPU’s)

Distributed Computing
 This type of computing is mainly concerned with developing
algorithms for the distributed cluster systems.
 Here distributed means a geographical distance between the
computers without any shared-Memory.

CS443
 Practical Applications of P&D Computing
13

Scientific Applications
 Bioinformatics
 DNA sequence alignment (fine-grained, Coarse-grained, and hybrid solutions)
 Applications in astrophysics have explored the evolution of galaxies,
thermonuclear processes, and applications in the analysis of
extremely large datasets from telescope.
 Advances in computational physics and chemistry have explored
new materials, understanding of chemical pathways, and more
efficient processes
 e.g., Large Hydron Collider (LHC) at European Organization for Nuclear
Research (CERN) generates petabytes of data for a single collision.

CS443
 Practical Applications of P&D Computing
14

Scientific Applications

 Weather modeling for simulating the track of natural hazards like the
extreme cyclones (storms).
 Flood prediction

CS443
 Practical Applications of P&D Computing
15

Commercial Applications
 Some of the largest parallel computers power the wall street.

 Data mining-analysis for optimizing business and marketing

decisions.

 Large scale servers (mail and web servers) are often implemented
using parallel platforms.

 Applications such as information retrieval and search are typically

powered by large clusters.

CS443
 Practical Applications of P&D Computing
16

Other Applications
 Network intrusion detection: A large amount of data needs to be
analyzed and processed

 Cryptography (the art of writing or solving codes) employs parallel

infrastructures and algorithms to solve complex codes.

 Graphics processing

 Embedded systems increasingly rely on distributed control

algorithms.

CS443
 Limitations of Parallel Computing:
17

 Exploring the proper parallelism from a problem is a hectic

process.

 It requires designing the proper communication and

synchronization mechanisms between the processes and sub-
tasks.
 The program’s subtasks must have low coupling and high
cohesion. But it’s difficult to create such programs.

 It needs relatively more technical skills to code a parallel

program.

CS443
 References
18
1. Kumar, V., Grama, A., Gupta, A., & Karypis, G. (1994). Introduction to
parallel computing (Vol. 110). Redwood City, CA: Benjamin/Cummings.

2. Quinn, M. J. Parallel Programming in C with MPI and OpenMP,(2003).

CS443
Questions
19

CS443