Ch-8.4.

Synchronization

8.4.1. Introduction

Synchronization is the coordination of the events to operate a system in unison. Systems

operating with all their parts in synchrony are said to be synchronous or in sync.
Synchronization in multimedia systems refers to the temporal relations between media
objects in the multimedia system. Synchronization between media objects comprises
relations between time-dependent media objects and time-independent media objects. It
is addressed and supported by many system components including the operating system,
communication system, databases, and documents and even often by applications.

8.4.2. Notion of Synchronization

Using the number of media criterion, even a document processing application that
supports text and graphics can be regarded as a multimedia system.

The types of supported media are an additional criterion. In this case we distinguish
between time-dependent and time-independent media. A time-independent media
object is usually presented using one presentation unit (e.g. bitmap graphics). Time-
dependent media objects are presented by a sequence of presentation units (e.g. a video
sequence).

The degree of media integration is the third criterion. In this case, integration means that
the different types of media remain independent but can be processed and presented
together.
Combining all three criteria, we propose the following definition of a multimedia system:
a system or application that supports the integrated processing of several media types
with at least one time-dependent medium.

Integral digital systems can support all types of media, and due to digital processing, may
provide a high degree of media integration. Systems that handle time-dependent analog
media objects and time-independent digital media objects are called hybrid systems. The
disadvantages of hybrid systems is that they are restricted with regard to the integration
of time-dependent and time-independent media, for example, audio and video are stored
on different devices than time-independent media objects.
The audio/video applications are time-dependent media objects. In addition, they often
do not support the separate handling of audio and video media objects. They are
supported by audio and video server.
Traditional desktop-publishing systems are examples of integrated processing of time-
independent media objects supported by the text editors.
Basic Synchronization Issues

The word synchronization refers to time. In a more general and wide sense, we use
synchronization in multimedia systems as comprising content, spatial and temporal
relation s between media objects.

Content Relations
Content relations define a dependency of media objects from some data. Example:
dependency between spreadsheet and graphics that represent data listed in spreadsheet.
Another example is two graphics that are based on the same data but different
interpretations of the data.

Spatial Relations
The spatial relations that are usually known as layout relationship define the space used
for the presentation of a media object on an output device at a certain point of time in a
multimedia presentation. Example: desktop-publishing.
Layout frames
 Placed on output device and content assigned to frame
 Positioning of layout frames:
 Fixed to position of document
 Fixed to position on page
 Relative to position of other frame
Example: in window-based system, layout frames correspond to windows and
video can be positioned in window
Temporal Relations
Temporal relations define the temporal dependencies between media objects. They are
of interest whenever time-dependent media objects exist.
An example of temporal relations is the relation between a video and an audio object that
are recorded during a concert. If these objects are presented, the temporal relation
during the presentations of the two media objects must correspond to the temporal
relation at the recording moment.

Synchronization Classification
 Intra-object Synchronization
 Time relation between various presentation units of one time-dependent
media stream.
Figure: Intra-object synchronization between frames of a video sequence showing a jumping ball

 Inter-object Synchronization
 Time relation between media objects belonging to two time dependent
media streams.

Figure: Inter- object synchronization showing temporal relations in a multimedia presentation

including audio, video, animation and picture objects.

 Live Synchronization
 Goal: exactly reproduce at presentation temporal relations as they existed
during capturing process
 Requirement: must capture temporal relation information during media
capturing
 Example: video conference, phone service
 Synthetic Synchronization
 Goal: arrange stored data objects to provide new combined multimedia
objects via artificial temporal relations
 Requirements: support flexible synchronization relations between media
 Example: authoring, tutoring systems

Two phases:
 Specification phase – define temporal relations
 Presentation phase – present data in sync mode
8.4.3. Presentation requirements

 For intra-object synchronization

 Need accuracy concerning delays in presentation of LDUs
 For inter-object synchronization
 Need accuracy in parallel presentation of media objects
For intra-object synchronization, the processes can wait for each other, using the method
of blocking; a data stream of time-independent LDUs can also be stopped.

There is a different situation when looking at sequence of audio and moving pictures:

1. What does the blocking of a stream of moving pictures mean for the connected output
device?
2. Should and can the last picture of a stream be shown during blocking?
3. Should, in the case of speech or music, a previous part be repeated during the blocking?
4. How long can such a gap exist?

This situation is known as the gap problem.

Solution 1: in case of moving pictures, existing systems are solving the problem by
switching the output device to dark or white.
Solution 2: showing the last moving picture as a still picture.
Solution 3: Switch output device to alternative presentation if gap between late video and
audio exceeds during blocking.
Solution 4: it is significant, whether the duration of such a gap is a couple of milliseconds,
a couple of seconds or even a couple of minutes. So alternatives must be
available that are selected independently of the expected blocking time.
For inter-object synchronization, more detailed results in the lip synchronization and
pointer synchronization areas are described to make clear the importance if user
perception aspects for presentation accuracy.

Lip synchronization requirements

 Temporal relation between audio and video

Synchronization skew
 Time difference between related audio and video LDUs
 Streams in sync: if skew = 0 or skew ≤bound
 Negative skew: video before audio
 Positive skew: Audio before video
  In sync: -80ms ≤ skew ≤ 80ms
 Out of sync: Skew < -160ms, Skew > 160ms
 Transient: -160ms ≤ skew < -80ms, 80ms < skew ≤ 160ms

Pointer synchronization requirements

Negative skew: pointer before audio

Positive skew: pointer after audio

 In sync: -500ms ≤ skew ≤ 750ms

 Out of sync: Skew < -1000ms, Skew > 1250ms
 Transient sync situation: -1000ms ≤ skew < -500ms, 750ms < skew ≤ 1250ms
8.4.4. Reference model for multimedia synchronization
We need reference models to
 Understand various requirements for multimedia sync
 Identify and structure run-time mechanisms to support execution of sync
 Identify interface between run-time mechanisms
 Compare system solutions for multimedia sync

Figure: Four-layer reference model.

8.4.5. Synchronization Specification

The synchronization specification of a multimedia object describes all temporal

dependencies of the included objects in the multimedia object. Because the
synchronization specification determines the whole presentation, it is a central issue in
multimedia systems. A synchronization specification should be comprised of:

 Intra-object synchronization specifications for the objects of the presentation.

 QoS descriptions for intra-object synchronization.
 Inter-object synchronization specification for media objects of the presentation.
 QoS description for inter-object synchronization.

Interval-based specifications

In interval-based synchronization specification, the presentation duration of an object is

regarded as interval.

Figure: Types of temporal relations between two objects

Complex specification comprises:

• Multiple object synchronization
• User interactions

Requirements for a specification method:

• Support of object consistency and for maintenance of specifications
• Media objects should be kept as one logical unit in the specification
• All types of synchronization relations should be easily describable
• Support for the integration of time-dependent and time-independent media objects
• Quality of service requirements must be describable
• Support of hierarchical levels of a synchronization