Introduction System model and definitions Snapshot algorithm for FIFO channels Non-FIFO channels

Distributed and Parallel Systems

M. T. Bennani
Assistant Professor, FST - El Manar University, LISI-INSAT

-Academic Year 2018-2019-

M. T. Bennani Assistant Professor, FST - El Manar University, LISI-INSAT

Chapter II. Global State
Introduction System model and definitions Snapshot algorithm for FIFO channels Non-FIFO channels

Challenges

I Recording the global state of a distributed system on-the-fly is

an important paradigm to handel services like
I Managing distibuted production sites
I Managing distributed banking operations
I Analyzing
I Testing
I Verifying properties

I To resolve this problem is non-trivial

I The lack of globally shared memory and global clock
I The message transfer delays are finite but unpredictable

2 / 18
Introduction System model and definitions Snapshot algorithm for FIFO channels Non-FIFO channels

Example
I Two distincts sites of distributed
banking: S1, S2
I Two different accounts: A, B
I C12 : Communication channel from
S1 to S2
I C21 : Communication channel from
S2 to S1
I t0 :A=600, B=200, C12 =0, C21 =0
I t1 :A=550, B=200, C12 =50, C21 =0
I t2 :A=550, B=120, C12 =50, C21 =80
I t3 :A=630, B=120, C12 =50, C21 =0
I t4 :A=630, B=170, C12 =0, C21 =0
If we record the global state of the application using:
I Local state of A at t0
I Local state of B, C12 and C21 at t2
An extra $50 appears in the system
3 / 18
Introduction System model and definitions Snapshot algorithm for FIFO channels Non-FIFO channels

System model
I The system consists of a colection of n processes p1 , .., pn connected
by channels.
I At any instant, the state of process pi , denoted by LSi , is a result
of the sequence of all the events executed by pi till that instant.
I For an event e and a process state LSi , e ∈ LSi iff e belongs to the
sequence of events that have taken process pi to state LSi .
I For an event e and a process state LSi , e ∈
/ LSi iff e does not belong
to the sequence of events that have taken process pi to state LSi .
I For a channel Cij , the following set of messages can be defined
based on the local states of the processes pi and pj
Transit: transit(LSi , LSj ) = {mij / send(mij ) ∈ LSi ∧ rec(mij ) ∈
/ LSj }
Note: The global state isn’t only the union of the local states but also it
should include the channel state.

4 / 18
Introduction System model and definitions Snapshot algorithm for FIFO channels Non-FIFO channels

Communications model

I In FIFO model, each channel acts as a first-in first-out

message queue and thus, message ordering is preserved by a
channel.
I In non-FIFO model, a channel acts like a set in which the
sender process adds messages and the receiver process
removes messages from it in a random order.
I A system that supports causal delivery of messages satisfies
the following property: ”For any two messages mij and mkj , if
send(mij ) → send(mkj ), then rec(mij ) → rec(mkj )”.

5 / 18
Introduction System model and definitions Snapshot algorithm for FIFO channels Non-FIFO channels

Consistant global state

I The global state of a distributed system is a collection of the

local states of the processes and the channels.
I Notationally, global state GS is defined as: GS={ ∪i LSi , ∪i,j SCij }
I A global state GS is a consistent global state iff it satisfies the
following two conditions:

C1 send(mij ) ∈ LSi ⇒ mij ∈ SCij ⊕ rec(mij ) ∈ LSj

C2 send(mij ) ∈
/ LSi ⇒ mij ∈
/ SCij ∧ rec(mij ) ∈
/ LSj

p ⊕ q = (p ∨ q) ∧ ¬(p ∧ q) = (p ∧ ¬q) ∨ (¬p ∧ q)

6 / 18
Introduction System model and definitions Snapshot algorithm for FIFO channels Non-FIFO channels

Interpretation in terms of cuts

I A cut in a space-time diagram is a line joining an arbitrary point on

each process line that slices the space-time diagram into a PAST
and a FUTURE.
I A consistent global state corresponds to a cut in which every
message received in the PAST of the cut was sent in the PAST of
that cut.
I Such a cut is known as a consistent cut.

7 / 18
Introduction System model and definitions Snapshot algorithm for FIFO channels Non-FIFO channels

Interpretation in terms of cuts : Example

I C1 is inconsistent because message m1 is flowing from the FUTURE

to the PAST
I C2 is consistent and message m4 must be captured in the state of
channel C21

8 / 18
Introduction System model and definitions Snapshot algorithm for FIFO channels Non-FIFO channels

Issues in recording a global state

The following two issues need to be addressed

I I1 How to distinguish between the messages to be recorded in
the snapshot from those not to be recorded?
I Any message that is sent by a process before recording its
snapshot, must be recorded in the global snapshot (from C1).
I Any message that is sent by a process after recording its
snapshot, must not be recorded in the global snapshot (from
C2).
I I2 How to determine the instant when a process takes its
snapshot?
I A process pj must record its snapshot before processing a
message mij that was sent by process pi after recording its
snapshot.

9 / 18
Introduction System model and definitions Snapshot algorithm for FIFO channels Non-FIFO channels

Chandy-Lamport algorithm

I The Chandy-Lamport algorithm uses a control message, called

a marker whose role in a FIFO system is to separate messages
in the channels.
I After a site has recorded its snapshot, it sends a marker, along
all of its outgoing channels before sending out any more
messages.
I A marker separates the messages in the channel into those to
be included in the snapshot from those not to be recorded in
the snapshot.
I A process must record its snapshot no later than when it
receives a marker on any of its incoming channels.

10 / 18
Introduction System model and definitions Snapshot algorithm for FIFO channels Non-FIFO channels

Chandy-Lamport algorithm
I The algorithm can be initiated by any process by executing
the ”Marker Sending Rule” by which it records its local state
and sends a marker on each outgoing channel.
I A process executes the ”Marker Receiving Rule” on receiving
a marker. If the process has not yet recorded its local state, it
records the state of the channel on which the marker is
received as empty and executes the ”Marker Sending Rule” to
record its local state.
I The algorithm terminates after each process has received a
marker on all of its incoming channels.
I All the local snapshots get disseminated to all other processes
and all the processes can determine the global state.

11 / 18
Introduction System model and definitions Snapshot algorithm for FIFO channels Non-FIFO channels

Chandy-Lamport algorithm
1: Process pi records its state
2: for each outgoing channel C on which a marker has not been
sent do
3: pi sends a marker along C before pi sends further messages
along C
4: end for
5: Marker receiving rule for process pj
6: On receiving a marker along channel C:
7: if Pj has not recorded its state then
8: Record the state of C as the empty set
9: Execute the ”marker sending rule”
10: else
11: Record the state of C as the set of messages received along
C after pj ’s state recorded and before pj received the marker
along C.
12: end if
12 / 18
Introduction System model and definitions Snapshot algorithm for FIFO channels Non-FIFO channels

Example

I Global state1: Total amount

800
A=550, B=170, C12=0,
C21=80
I Global state2: Total amount
800
A=600, B=120, C12=0,
C21=80

13 / 18
Introduction System model and definitions Snapshot algorithm for FIFO channels Non-FIFO channels

Correctness and Complexity

Correctness
I Due to FIFO property of channels, it follows that no message sent
after the marker on that channel is recorded in the channel state. Thus,
condition C2 is satisfied.
I When a process pj receives message mij that precedes the marker on
channel Cij , it acts as follows: If process pj has not taken its snapshot
yet, then it includes mij in its recorded snapshot. Otherwise, it records
mij in the state of the channel Cij . Thus, condition C1 is satisfied.

Complexity
I The recording part of a single instance of the algorithm requires O(e)
messages and O(d) time, where e is the number of edges in the network
and d is the diameter of the network.

14 / 18
Introduction System model and definitions Snapshot algorithm for FIFO channels Non-FIFO channels

Properties of the recorded global state

I The recorded global state may not correspond to any of the

global states that occurred during the computation.
I This happens because a process can change its state
asynchronously before the markers it sent are received by
other sites and the other sites record their states.
I But the system could have passed through the recorded global
states in some equivalent executions.
I The recorded global state is a valid state in an equivalent
execution and if a stable property (i.e., a property that
persists) holds in the system before the snapshot algorithm
begins, it holds in the recorded global snapshot.
I Therefore, a recorded global state is useful in detecting stable
properties.

15 / 18
Introduction System model and definitions Snapshot algorithm for FIFO channels Non-FIFO channels

Snapshot algorithms for non-FIFO channels

I In a non-FIFO system, a marker cannot be used to delineate

messages into those to be recorded in the global state from
those not to be recorded in the global state.
I In a non-FIFO system, either some degree of inhibition or
piggybacking of control information on computation messages
to capture out-of-sequence messages.

16 / 18
Introduction System model and definitions Snapshot algorithm for FIFO channels Non-FIFO channels

Lai-Yang algorithm (1/2)

The Lai-Yang algorithm fulfills this role of a marker in a non-FIFO
system by using a coloring scheme on computation messages that
works as follows:
1. Every process is initially white and turns red while taking a
snapshot. The equivalent of the ”Marker Sending Rule” is
executed when a process turns red.
2. Every message sent by a white (red) process is colored white
(red).
3. Thus, a white (red) message is a message that was sent before
(after) the sender of that message recorded its local snapshot.
4. Every white process takes its snapshot at its convenience, but
no later than the instant it receives a red message.

17 / 18
Introduction System model and definitions Snapshot algorithm for FIFO channels Non-FIFO channels

Lai-Yang algorithm (2/2)

5. Every white process records a history of all white messages

sent or received by it along each channel.
6. When a process turns red, it sends these histories along with
its snapshot to the initiator process that collects the global
snapshot.
7. The initiator process evaluates transit(LSi , LSj ) to compute
the state of a channel Cij as given below:
SCij = white messages sent by pi on Cij - white messages
received by pj on Cij
SCij = {send(mij )/send(mij ) ∈ LSi } − {rec(mij )/rec(mij ) ∈ LSj }

18 / 18