UNIT – II

PARALLEL PROGRAM CHALLENGES 

1. Performance
2. Scalability
3. Synchronization and Data sharing - Data races
4. Synchronization Primitives
(Mutexes, Locks, Semaphores, Barriers)
5. Deadlocks and Live locks
6. Communication between threads
(Condition variables, Signals, Message queues
and Pipes)
 1. PERFORMANCE
There are two common metrics for performance:

• Items per unit time

 This is a measure of bandwidth.
 Transactions per second, jobs per hour.

• Time per item

 A measure of the time to complete a single task.
 A measure of latency or response time.
 QoS Metric
• Specifies the expectations of the users of the system
as well as penalties if the system fails to meet these
expectations.

• Number of transactions of latency greater than some

threshold.

• The amount of time that the system is unavailable,

typically called downtime or availability.
 Uses of defining the Critical Metrics
• Clearly specified requirements can be used to drive
design decisions, both for selecting appropriate
hardware and in determining the structure of the
software.

• Knowing what metrics are expected enables the

developers to be confident that the system they
deliver fulfills the criteria.

• Defining the metrics should also define the expected

inputs.
 Algorithmic Complexity
• A measure of how much computation a program will
perform when using a particular algorithm.

• A measure of how this time will change as the size of the

input changes.

• A measure of the efficiency and estimate of operation

count.

• An algorithm with low algorithmic complexity is likely to be

more difficult to implement than an algorithm with higher
algorithmic complexity
 Example – Algorithmic Complexity
Sum of the First N Numbers
void sum(int N)
{
int total=0;
for (int i=1; i<=N; i++)
{
total += i;
}
printf( "Sum of first %i integers is %i\n", N, total );
}
 Explanation
• For a given input value N, the code will take N trips around the loop
and do N additions.

• The algorithmic complexity focuses on the number of operations,

which in this case are the N additions.

• The time it would take to complete this calculation is some cost per
addition, k, multiplied by the number of additions, N. So, the time
would be k ∗ N.

• It is quite acceptable to ignore the (constant) scaling factor and say

that the calculation will take of the order of N computations.

• This is typically written O(N).

 The Role of the Compiler Optimization
Two Types of Compiler Optimization:
• Elimination of work -
Eg: Empty Loop
for (int i=0; i<1000; i++)
{
}

• Restructuring of work –
Eg: Loop Containing Floating-Point Arithmetic
double total=0.0;
for (int i=0; i<1000; i++)
{
total = total + 1.0;
}
 Strength Reduction

• Improvement in the efficiency of the

operations.

• One operation can be replaced by a less

expensive one.

• This is called strength reduction.

 Cross-File Optimization
 Cross-file optimization is a final step after the
compiler has produced object files for all the
source files in an application.

At this step, the compiler reads all the object files
and looks for optimizations it can perform using
full knowledge of the entire application.
• Function A() calls function B(), but function B() is
defined in the file b.c, and function A() is defined in
the file a.c
 2. SCALABILITY

• Using Multiple Processes to Improve System

Productivity

• Multiple Users Utilizing a Single System

 3. SYNCHRONIZATION AND
DATA SHARING
• Sharing of data or states between the threads.
• The degree of sharing depends on the task.
• A method for sharing data between threads is a
Data race
(Multiple threads are updating the same data in
an unsafe way)
• Data races are the most common programming error
found in parallel code.
• Avoiding data races is by utilizing proper synchronization
between threads.
 DATA RACE
• A data race occurs when multiple threads use the
same data item and one or more of those threads are
updating it.

• Tools to Detect Data Races / synchronization errors :

Helgrind - A Valgrind (Debugging Tool)

Thread Analyzer – An Oracle GUI tool

 Avoiding Data Races
• Make sure that only one thread can update the
variable at a time.

• The easiest way to do this is to place a

synchronization lock called mutex lock around all
accesses to that variable.

• Ensuring that before referencing the variable, the

thread must acquire the lock.
Code Containing Data Race
Code Modified to Avoid Data Races
4. SYNCHRONIZATION PRIMITIVES
Most operating systems provide a rich set of
synchronization primitives.

1. Mutex and Critical Regions

2. Spin Locks
3. Semaphores
4. Readers-Writer Locks
5. Barriers
 MUTEX AND CRITICAL REGIONS

• Simplest form of synchronization

• Only one thread at a time can acquire a mutex lock

• Ensures that the data structure is modified by only

one thread at a time.
Placing Mutex Locks Around Accesses to
Variables
 Contended Mutex
If multiple threads are attempting to acquire the
same mutex at the same time, then only one thread
will succeed, and the other threads will have to wait.

This situation is known as a Contended Mutex.

 Critical Section / Critical Region

The region of code between the acquisition and

release of a mutex lock is called a critical section or
critical region.

Code in this region will be executed by only one

thread at a time.
 SPIN LOCKS
• Spin locks are essentially mutex locks.

• The difference between a mutex lock and a spin

lock is that a thread waiting to acquire a spin lock will
keep trying to acquire the lock without sleeping.

• In comparison, a mutex lock may sleep if it is unable

to acquire the lock.
 Pros and Cons of Spin Lock
Advantage of using spin locks :
They will acquire the lock as soon as it is released,
whereas a mutex lock will need to be woken by the
operating system before it can get the lock.

Disadvantage :
A spin lock will spin on a virtual CPU monopolizing that
resource.
In comparison, a mutex lock will sleep and free the
virtual CPU for another thread to use.
 SEMAPHORES
• Semaphores are counters that can be either incremented or
decremented.

• Used in situations where there is a finite limit to a resource

and a mechanism is needed to impose that limit.

• Example : A buffer that has a fixed size.

Every time an element is added to a buffer, the number of
available positions is decreased.

Every time an element is removed, the number available is

increased.
• Semaphores will also signal or wake up threads that
are waiting on them to use available resources; hence,
they can be used for signaling between threads.

• The method that acquires a semaphore might be

called wait, down, or acquire.

• The method to release a semaphore might be called

post,up, signal, or release.

• When the semaphore no longer has resources

available, the threads requesting resources will block
until resources are available.
 READERS - WRITER LOCKS

• A readers-writer lock allows many threads to read

the shared data but can then lock the readers
threads out to allow one thread to acquire a writer
lock to modify the data.

• A writer cannot acquire the write lock until all the

readers have released their reader locks.
Using a Readers-Writer Lock
 BARRIERS
• There are situations where a number of threads have
to all complete their work before any of the threads
can start on the next task.

• In these situations, it is useful to have a barrier

where the threads will wait until all are present.

• One common example of using a barrier arises when

there is a dependence between different sections of
code.
Using a Barrier to Order Computation
Use of Multiple Barriers
 5. DEADLOCK
Deadlock occurs whenever a thread or a process is blocked
waiting for a resource of another thread or process that
will never become available.

A set of processes is in a deadlock state :

 where two or more threads cannot make progress
because the resources that they need are
held by the other threads.

 when every process in the set is waiting for an event

that can only be caused by another process in the same
set.
 Deadlock Necessary Conditions
If a deadlock occurs, then each of the following four conditions
must hold.

Mutual Exclusion: At least one resource must be held in a

non-sharable way.

Hold and Wait: A process must be holding a resource and

waiting for another.

No Preemption: Resource cannot be preempted.

Circular Wait: P1 waits for P2 , P2 waits for P3 , …,

Pn-1 waits for Pn , and Pn waits for P1 .
 Note that the conditions are necessary.

 This means if a deadlock occurs ALL conditions are

met.

As long as one of the four conditions fails there will

be no deadlock.
 Types of Dead locks

Three types of deadlocks:

Self deadlock
Recursive deadlock
Lock - Ordering deadlock

In most instances deadlock means , it is only

Lock - Ordering deadlock only.
 Self Deadlock
 A condition or a instance when a thread wants to
acquire a lock that is already owned by another thread .
 Recursive Deadlock
 When a wake-up of a thread resides in another thread then
the condition is referred as recursive deadlock.
 Lock-Ordering Deadlock

 Here , a thread T1 locks a resource r1 and waits for a

resource r2, which is being locked by another thread T2.
Example – Two threads in a deadlock
Thread 1 Thread 2
void update1( ) void update2( )
{ {
acquire(A); acquire(B);
acquire(B); acquire(A); variable1++;
variable1++;
release(B); release(B);
release(A); release(A);
} }
 Explanation
Suppose two threads need to acquire mutex locks –
A and B to complete some task.

If thread 1 has already acquired lock A and thread 2

has already acquired lock B, then A cannot make
forward progress because it is waiting for lock B, and
thread 2 cannot make progress because it is waiting
for lock A.

The two threads are deadlocked.

 Deadlock Avoidance
Resources can be ordered in a hierarchical way.

A process must acquire resources in this particular

order.

As a result, no deadlock can happen.

 In the given example, if thread 2 acquired the locks
in the order A and then B, it would stall while waiting
for lock A without having first acquired lock B.

This would enable thread 1 to acquire B and then

eventually release both locks, allowing thread 2 to
make progress.
 LIVELOCK
 Livelock is a special case of resource starvation.

Livelock occurs , If two or more processes continually

repeat the same interaction in response to changes
in the other processes without doing any useful work.

These processes are not in the waiting state, and they

are running concurrently.

 This is different from a deadlock because in a

deadlock all processes are in the waiting state.
A livelock traps the threads in an unending loop
releasing and acquiring locks.

Livelocks can be caused by code to back out of

deadlocks.

In the example we have discussed, If the thread

cannot obtain the second lock it requires, it releases
the lock that it already holds.
Example - Two threads in a Livelock
 Explanation
Each thread acquires a lock and then attempts to
acquire the second lock that it needs.

If it fails to acquire the second lock, it releases the lock

it is holding, before attempting to acquire both locks
again.

The thread exits the loop when it manages to

successfully acquire both locks, which will eventually
happen, but until then, the application will make no
forward progress.
 If two threads encounter this code at the same time,
they will be trapped in a livelock of constantly
acquiring and releasing mutexes, but it is very
unlikely that either will make progress.
 6. COMMUNICATION BETWEEN
THREADS/PROCESSES
All parallel applications require some element of
communication between either the threads or the processes.

There is usually an implicit or explicit action of one thread

sending data to another thread.

These mechanisms usually require operating system support

to mediate the sending of messages between threads or
processes.

Various mechanisms to enable communication between

threads are - condition variables , signals , message queues
and pipes.
 CONDITION VARIABLES
Condition variables communicate readiness between
threads .

Puts a thread to sleep until a condition is true or until a

message is received.

Enables a thread to be woken up when a condition

becomes true.

Without condition variables, the waiting thread would

have to use some form of polling to check whether the
condition had become true.
Condition variables work in conjunction with a mutex.

The mutex is there to ensure that only one thread at a

time can access the variable.
 Example : Producer-Consumer Problem

 Suppose an application has one producer thread and one

consumer thread.

 The producer adds data onto a queue, and the consumer

removes data from the queue.

 If there is no data on the queue, then the consumer needs

to sleep, until it is signaled that an item of data has been
placed on the queue.
 Producer Thread Adding an Item to the
Queue
Acquire Mutex();
Add Item to Queue();
If ( Only One Item on Queue )
{
Signal Conditions Met( );
}
Release Mutex();
 Explanation
The producer thread needs to signal a waiting consumer
thread only if the queue was empty and it has just added a
new item into that queue.

If there were multiple items already on the queue, then

the consumer thread must be busy processing those items
and cannot be sleeping.

 If there were no items in the queue, then it is possible

that the consumer thread is sleeping and needs to be
woken up.
 Consumer Thread Removing Items from Queue
Acquire Mutex();
Repeat
Item = 0;
If ( No Items on Queue() )
{
Wait on Condition Variable();
}
If (Item on Queue())
{
Item = remove from Queue();
}
Until ( Item!= 0 );
Release Mutex();
 Explanation
The consumer thread will wait on the condition variable if
the queue is empty.

When the producer thread signals it to wake up, it will first

check to see whether there is anything on the queue.

If there is an item on the queue, then the consumer thread

can handle that item

otherwise, it returns to sleep.

 Limitations
The producer thread can use two types of wake-up calls.

Either it can wake up a single thread or it can broadcast to all

waiting threads.

The process of waking all the waiting threads is serial; only a

single thread can be woken at a time.

When a wake-up call is broadcast to all threads, some of them

may be woken when there is no work for them to do.

When the signal to wake up the waiting thread is sent before

the thread is ready to receive it. This situation is known as lost
wake-up.
 SIGNALS AND EVENTS
Signals are software equivalent of hardware interrupts

A signal is an asynchronous notification sent to a process or

to a specific thread within the same process in order to notify
it of an event that occurred.

When a signal is sent, the operating system interrupts the

target process’ normal fow of execution to deliver the signal.  

Signals are a UNIX mechanism which have a handler in the

receiving process perform some task upon the receipt of the
Message
 If the process has previously registered a signal handler,
that routine is executed. Otherwise, the default signal
handler is executed.

 Signal handling is vulnerable to race conditions. As

signals are asynchronous, another signal can be delivered
to the process during execution of the signal handling
routine.

 Eg: SIGKILL , SIGABRT, SIGALRM, SIGCHLD, SIGCONT

Windows has a similar mechanism called Events.

The handling of keyboard presses and mouse moves

are performed through the event mechanism.

Signals and events are really optimized for sending

limited or no data along with the Signal or an Event.

But they are probably not the best mechanism for

communication when compared to other options.
 Handling Signals

• Signal handlers can be installed with the signal()  system

call.
• If a signal handler is not installed for a particular signal,
the default handler is used.
• Otherwise the signal is intercepted and the signal handler
is invoked.
• The process can also specify two default behaviors,
without creating a handler: ignore the signal (SIG_IGN)
and use the default signal handler (SIG_DFL).
• There are two signals which cannot be intercepted and
handled: SIGKILL and SIGSTOP.
 Installing and Using a Signal Handler
void signalHandler(void *signal)
{
...
}

int main()
{
installHandler( SIGNAL, signalHandler );
sendSignal( SIGNAL );
}
 MESSAGE QUEUES
A message queue is a data structure that can be shared
between multiple processes.

Messages can be placed into the queue and will be

removed in the same order in which they were added.

Constructing a message queue looks like constructing a

shared memory segment.
The first thing needed is a descriptor, typically the location
of a file in the file system.

This descriptor can either be used to create the message

queue or be used to attach to an existing message queue.

Once the queue is configured, processes can place

messages into it or remove messages from it.

Once the queue is finished, it needs to be deleted .

Creating and Placing Messages into a Queue

ID = Open Message Queue ( Descriptor );

Put Message in Queue( ID, Message );
...
Close Message Queue( ID );
Delete Message Queue( Description );
Opening a Queue and Receiving Messages
ID = Open Message Queue ID(Descriptor);
Message = Remove Message from Queue(ID);
...
Close Message Queue(ID);
 PIPES
Pipes represent a channel for Communication between
Processes.

One process will write to the pipe, while another

process will read from the pipe.

Data is written to one end of the pipe and read from

the other end.

The method for using pipe is much like the method for
using a file – Make, Open ,Read, Write, Close, Delete.
The pipe is opened, data is written into it or read from it,
and then the pipe is closed.

A pipe exists until it is closed in all processes.

For example, the output from the command ls, which lists
all the files in a directory, could be piped into the wc
command, which counts the number of lines, words, and
characters in the input.
$ ls | wc

Can be used either in Half-duplex or Full duplex mode.

 Half Duplex Communication using Pipe
• The pipe is represented in an array of 2 file
descriptors fd(0) and fd(1).
Writing process Reading process
write fd1 read fd0

Pipe

Flow of Data
 Full Duplex Communication via Two Pipes

Two separate pipes, say p0 and p1

Process A Process B

write p01 write p11

read p10 read p01

p0

p1
 Setting Up and Writing into a Pipe

Make Pipe( Descriptor );

ID = Open Pipe( Descriptor );
Write Pipe( ID, Message, sizeof(Message) );
...
Close Pipe( ID );
Delete Pipe( ID );
 Opening an Existing Pipe to Receive
Messages
ID = Open Pipe( Descriptor );
Read Pipe( ID, buffer, sizeof(buffer) );
...
Close Pipe( ID );
 Types of Pipes

Two types:
Unnamed pipes – Stream/Traditional pipe
Named pipes – FIFO pipe
 UNNAMED PIPES
• Unnamed pipes can only be used between related
process, such as parent/child or child/child process.

• Unnamed pipes can exist only as long as the

processes using them.

• An unnamed pipe is constructed with the pipe

system call.
 NAMED PIPES
 Named Pipes is an extension to the traditional   unnamed
pipe concept.

 A named pipe can last as long as the system is up, beyond

the life of the process.

Named pipes are file-like objects that are given a specific

name that can be shared between processes.

The method for using a named pipe is much like the method
for using a file –
CreateFile,  OpenFile, ReadFile,  WriteFile, Close File
• When generated, named pipes have a directory
entry.

• With the directory entry are file access permissions

and capability for unrelated processes to use the pipe
file.

• Named pipes can be created at the shell level or

within a program using mkfifo command.
 Difference Between Pipes And Named Pipes
PIPES NAMED PIPES
They are unidirectional.  They are Bi-directinoal.

created by the shell created programatically using the

automatically. command mkfifo.

exists in the kernel.  exists in the file system with a

given file name.
 opened at the time of creation  not opened while creation.
only.

Used for communication Used for communication

between parent and child between processes on same
process computer or different computer