Multimedia

Lecture One

What is Multimedia?
Multimedia can have many definitions these include:
 A computer system perspective or
 Multimedia means that computer information can be represented through audio, video,
and animation in addition to traditional media (i.e., text, graphics/drawings, images).
 Multimedia is the field concerned with the computer controlled integration of text,
graphics, drawings, still and moving images (Video), animation, audio, and any other
media where every type of information can be represented, stored, transmitted and
processed digitally.
Multimedia Application:
A Multimedia Application is an application which uses a collection of multiple media sources e.g.
text, graphics, images, sound/audio, animation and/or video.
Hypertext
Hypertext is a text which contains links to other texts. The term was invented by Ted Nelson
around 1965.

0
Hypertext Navigation
Traversal through pages of hypertext is therefore usually non-linear (as indicated below).

This has implications in layout and organization of material |and depends a lot on the
application at hand.
Hypermedia
Hypermedia is not constrained to be text-based. It can include other media, e.g., graphics,
images, and especially continuous media { sound and video}
Example Hypermedia Applications?

•The World Wide Web (WWW) is a clear example of a

•hypermedia application.
•Powerpoint
•Adobe Acrobat (or other PDF software)
•Adobe Flash
•Video-on-demand
•Interactive TV
•Computer Games
•Virtual reality
•Digital video editing and production systems
•Multimedia Database systems
•Many many others?
1
Multimedia Systems
A Multimedia System is a system capable of processing multimedia data and applications.

A Multimedia System is characterized by the processing, storage, generation, manipulation and

rendition of Multimedia information.
Characteristics of a Multimedia System
A Multimedia system has four basic characteristics:

Multimedia systems must be computer controlled.

Multimedia systems are integrated.

The information they handle must be represented digitally.

The interface to the final presentation of media is usually interactive.

Challenges for Multimedia Systems
 Distributed Networks
 Temporal relationship between data
o Render deferent data at same time | continuously.
o Sequencing within the media:
o playing frames in correct order/time frame in video
o Synchronization | inter-media scheduling
o E.g. Video and Audio | Lip synchronization is clearly important for humans to
watch playback of video and audio and even animation and audio. Ever tried
watching an out of (lip) sync lm for a long time?
Key Issues for Multimedia Systems
The key issues multimedia systems need to deal with here are:

•How to represent and store temporal information.

•How to strictly maintain the temporal relationships on play back/retrieval
•What process are involved in the above.
•Data has to represented digitally | Analog{Digital Conversion, Sampling etc.
•Large Data Requirements | bandwidth, storage,
Data compression is usually mandatory
Desirable Features for a Multimedia System
Given the above challenges the following feature a desirable (if not a prerequisite) for a
Multimedia System:

•Very High Processing Power | needed to deal with large data processing and real time
delivery of media.

2
 Special hardware commonplace.

•Multimedia Capable File System | needed to deliver real-time media | e.g. Video/Audio
Streaming.

•Special Hardware/Software needed { e.g. RAID technology.}

•Data Representations | File Formats that support multimedia should be easy to handle yet
allow for compression/decompression in real-time.

•Efficient and High I/O | input and output to the fille subsystem needs to be efficient and
fast. Needs to allow for real-time recording as well as playback of data. e.g. Direct to Disk
recording systems.

•Special Operating System | to allow access to file system and process data efficiently and
quickly. Needs to support direct transfers to disk, real-time scheduling, fast interrupt
processing, I/O streaming etc.

•Storage and Memory | large storage units (of the order of hundreds of Tb if not more) and
large memory (several Gb or more). Large Caches also required and high speed buses for
effecient management.

•Network Support | Client-server systems commoncas distributed systems common.

•Software Tools | user friendly tools needed to handle media, design and develop
applications, deliver media.
Components of a Multimedia System
 Now let us consider the Components (Hardware and Software) required for a multimedia
system:
 Capture devices | Video Camera, Video Recorder, Audio Microphone, Keyboards, mice,
graphics tablets, 3D input devices, tactile sensors, VR devices.
Digitising Hardware
 Storage Devices | Hard disks, CD-ROMs, DVD-ROM, etc
 Communication Networks | Local Networks, Intranets, Internet, Multimedia or other special
high speed networks.
 Computer Systems | Multimedia Desktop machines, Workstations, MPEG/VIDEO/DSP
Hardware
 Display Devices | CD-quality speakers, HDTV,SVGA, Hi-Res monitors, Color printers etc.
Text and Static Data
•Source: keyboard, speech input, optical character recognition, data stored on disk.
•Stored and input character by character:
•Storage of text is 1 byte per char / more bytes for Unicode.
•For other forms of data (e.g. Spreadsheet les). May store format as text (with
3
 formatting) others may use binary encoding.

•Format: Raw text or formatted text e.g HTML, Rich Text Format (RTF), Word or a program
language source (Java, Python, MATLAB etc.)

•Not temporal | BUT may have natural implied sequence e.g. HTML format sequence,
Sequence of C program statements.

•Size Not significant w.r.t. other Multimedia data.

Graphics [Vector]
•Format: constructed by the composition of primitive objects such as lines, polygons, circles,
curves and arcs.

•Input: Graphics are usually generated by a graphics editor program (e.g. Illustrator) or
automatically by a program (e.g. Postscript).

•Graphics are usually editable or revisable (unlike Images).

•Graphics input devices: keyboard (for text and cursor control), mouse, trackball or graphics
tablet.

•graphics standards : OpenGL, PHIGS, GKS

•Graphics files usually store the primitive assembly
•Do not take up a very high storage overhead.
Images
•Still pictures which (uncompressed) are represented as a bitmap (a grid of pixels).
•Input: digitally scanned photographs/pictures or direct from a digital camera.
•Input: May also be generated by programs \similar" to graphics or animation programs.
•Stored at 1 bit per pixel (Black and White), 8 Bits per pixel (Grey Scale, Colour Map) or 24
Bits per pixel (True Color)

•Size: a 512x512 Grey scale image takes up 1/4 MB, a 512x512 24 bit image takes 3/4 MB
with no compression.

•This overhead soon increases with image size | modern high digital camera 20+
Megapixels 60MB uncompressed!

•Compression is commonly applied.

Audio
•Audio signals are continuous analog signals.
•Input: microphones and then digitised and stored
4
 •CD Quality Audio requires 16-bit sampling at 44.1 KHz Even higher audiophile rates (e.g.
24-bit, 96 KHz)

•1 Minute of Mono CD quality (uncompressed) audio requires 5 MB.

•1 Minute of Stereo CD quality (uncompressed) audio requires 10 MB.
•Usually compressed (E.g. MP3, AAC, Flac, Ogg Vorbis).
Video
•Input: Analog Video is usually captured by a video camera and then digitized.
•There are a variety of video (analog and digital) formats
•Raw video can be regarded as being a series of single images. There are typically 25, 30 or
50 frames per second.

•E.g. A 512 512 size monochrome video images take 25 0:25 = 6.25MB for a second to
store uncompressed.

•Typical PAL digital video (720 576 pixels per colour frame)1:24 25 = 31MB for a second to
store uncompressed.

•High Denition video on Blu-ray (up to 1920 1080 = 2 Megapixels per frame) 6:2 25 =
155MB for a second to store uncompressed. (There are higher possible frame rates!)

•Digital video clearly needs to be compressed for most times.

Multimedia Data Compression
•How can we compress data?
Lossy v Lossless :

•Lossless : Ideal (e.g. zip, unix compress) not good enough for MM data!
•Lossy :Throw away nonessential (perceptually less relevant) parts of the data stream FILTER
the data somehow.

•Examples: MP3, JPEG, MPEG Video/Audio

Digital Signal Processing and Digital Audio
What is Digital Signal Processing (DSP)?
•DSP includes many dierent topics, such as:
•Digital lters
•Analysis of signals and systems (especially in terms of frequency)
•Synthesis of signals
•Detection of signals and estimation of signal and system
5
 •parameters
•Data compression
•and on and on ...
DSP Related Subjects
•DSP is the intersection of a number of deferent areas of study:
•Mathematics
•Electrical engineering
•Signals and systems
•Analog circuit theory
•Computer architecture, (and more)
•Probability and statistics
•Computer programming
Strong Link to Image and Video Processing | more soon

Digital Audio Applications

•Music Production
•Hard Disk Recording
•Sound Synthesis
•Samplers
•Effects Processing
•Video
•Audio Important Element: Music and Effects
•Web
•Many uses on Web:
•Streaming Audio
•Spotify
•Listen to Web Radio
•Element of a Web Page

6
What is Sound?

•Sound Generation
•Source | Generates Sound
•Air Pressure changes
•Electrical | Loud Speaker
•Acoustic | Direct Pressure Variations
•Sound Reception
•Destination | Receives Sound
•Electrical | Microphone produces electric signal
•Ears | Responds to pressure hear sound (MPEG Audio | exploits this fact)

7
Digitizing Sound

•Microphone:
•Receives sound
•Converts to analog signal.
•Computer like discrete entities
•Need to convert Analog-to-Digital | Dedicated Hardware (e.g. Soundcard)
•Also known as Digital Sampling
Computer Manipulation of Sound
Digital Audio Examples

•Digital Signal Processing routines range from being trivial to highly complex :
•Volume
•Fading in/out, Cross-fading
•Reverse
•Repeating/Looping
•Echo/Reverb/Delay
•Filtering
•Signal Analysis
•Change Pitch and Tempo
Sample Rates and Bit Size
How do we store each sample value (Quantization)?

•8 Bit Value (0-255)

•16 Bit Value (Integer) (0-65535)
8
Sample Rate
•How many Samples to take?
•11.025 KHz | Speech (Telephone 8 KHz)
•22.05 KHz | Low Grade Audio (WWW Audio, AM Radio)
•44.1 KHz | CD Quality
Digital Sampling

Sampling process basically involves:

•Measuring the analog signal at regular discrete intervals

•Recording the value at these points

Digital Sampling
Digital sampling refers to a technique of taking a recorded passage from an already existing or
recorded musical, spoken or other work and then adding it into a new recording.

9
Nyquist's Sampling Theorem
•The Sampling Frequency is critical to the accurate reproduction of a digital version of an
analog waveform

•The Sampling frequency for a signal must be at least twice the highest frequency
component in the signal.

Sampling at Signal Frequency

10
Sampling at Twice Nyquist Frequency

Sampling at above Nyquist Frequency

If you get Nyquist Sampling Wrong?

11
 If you get Nyquist Sampling Wrong?

Decibel :(dB) a measurement of sound level

Implications of Sample Rate and Bit Size

•Ears do not respond to sound in a linear fashion
•Decibel (dB) a logarithmic measurement of sound level
12
 •16-Bit has a signal-to-noise ratio of 98 dB | virtually inaudible
•8-bit has a signal-to-noise ratio of 50 dB
•Therefore, 8-bit is roughly 8 times as noisy
•6 dB increment is twice as loud
File Type 44.1 KHz 22.05 KHz 11.025 KHz

16 Bit Stereo 10.1 Mb 5.05 Mb 2.52 Mb

16 Bit Mono 5.05 Mb 2.52 Mb 1.26 Mb

8 Bit Mono 2.52 Mb 1.26 Mb 630 Kb

Memory Required for 1 Minute of Digital Audio Practical Implications of Nyquist Sampling
Theory

Practical Implications of Nyquist Sampling Theory, Must (low pass) filter signal before sampling:

Otherwise strange artifacts from high frequency (above Nyquist Limit)signals would appear in
the sampled signal.
Why are CD Sample Rates 44.1 KHz?
Because of the Nyquist theorem, the maximum frequency that can be represented at any
given sampling rate is half the sampling rate; thus a 44.1 kHz CD can capture tones up to
22.05 kHz, while a 48 kHz DVD can capture tones up to 24 kHz.
Basic Digital Audio Signal Processing
In this section we look at some basic aspects of Digital Audio
•Signal Processing:

13
 •Some basic definitions and principles Filtering
•Basic Digital Audio Effects
Decibel and acoustics
•dB is commonly used to quantify sound levels relative to some 0 dB reference.
•The reference level is typically set at the threshold of human perception
•Human ear is capable of detecting a very large range of sound pressures.
Examples of dB measurement in Sound
Threshold of Pain

•The ratio of sound pressure that causes permanent damage from

•short exposure to the limit that (undamaged) ears can hear is above a million:
•The ratio of the maximum power to the minimum power is above one (short scale) trillion
(1012).

•The log of a trillion is 12, so this ratio represents a dierence of 120 dB.
•120 dB is the quoted Threshold of Pain for Humans.
Speech Sensitivity

•Human ear is not equally sensitive to all the frequencies of sound

•within the entire spectrum:
•Maximum human sensitivity at noise levels at between 2 and 4 kHz (Speech)
•These are factored more heavily into sound descriptions using a process called
frequency weighting.
•Filter (Partition) into frequency bands concentrated in this range.
•Used for Speech Analysis
•Mathematical Modelling of Human Hearing
•Audio Compression (E.g. MPEG Audio)

14