Scribd
Upload
80 views11 pages

Information Theory and Coding: - Entropy - Probabilistic Information Measure

Information theory is a mathematical approach to quantify and study the coding of information for storage, communication, and transmission. It examines concepts like entropy, uncertainty, and probabilistic measures of information. Key aspects include defining information as organized messages that can be interpreted, quantifying the information content of messages using probabilities, and measuring information in bits based on the uncertainty or probability of messages. Information theory establishes properties for quantifying information like the direct relationship between uncertainty and information.

Uploaded by

Shawn Moyo
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
80 views11 pages

Information Theory and Coding: - Entropy - Probabilistic Information Measure

Information theory is a mathematical approach to quantify and study the coding of information for storage, communication, and transmission. It examines concepts like entropy, uncertainty, and probabilistic measures of information. Key aspects include defining information as organized messages that can be interpreted, quantifying the information content of messages using probabilities, and measuring information in bits based on the uncertainty or probability of messages. Information theory establishes properties for quantifying information like the direct relationship between uncertainty and information.

Uploaded by

Shawn Moyo
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 11

INFORMATION

THEORY AND
CODING
• Entropy
• Probabilistic information measure
INFORMATION?

• Information - organised acts that can e interpreted as a message, idea or intelligence in


inromation theory.
• is the source of a communication system, whether it is analog or digital.

• message - electrical signal, speech/voice, text


WHAT IS INFORMATION THEORY?

• Is a mathematical approach to study the coding of information along with the quantification
,storage and communication of information
INFORMATION THEORY: UNCERTAINITY

•consider:
•m = {mo , m1 , m2 ..... mn }
•p = {po , p1 , p2 ....... pn } n

•total probability is given by :


i 1
pi

•when an event has a chance of occuring, then it has an uncertainity e.g p= 0.5
•if an event does not occur or is certain then then there is a lot of information and minimum probability
INFORMATION THEORY: MEASURE OF
INFORMATION
• Measure of information
• it is the infomation content of a message

• consider source emitting independent messages with probability as follows

m = {mo , m1 , m2 ..... mn }

p = {po , p1 , p2 ....... pn }

n
total probability is given by : p
i 1
i
INFORMATION THEORY: MEASURE OF INFORMATION

• formula :

1 base units
I k  log 2 ( ) 2 bits
pk e nat
10 decit
1
log 2 ( )
pk
Ik 
log 2
INFORMATION THEORY: PROPERTIES
• uncertanity is directly prpotional to information

• if receiver knows what is being transmitted then information is zero

• total information is equal to sum o indepedant information

• if there are M=2N equiproable messages, then the amount o infromation carried by each message
will be N bits
ENTROPY(H)

•  Defined as the average information per symbol


• Entropy, H =
• log() bits/symbol
PROPERTIES

•1.   Entropy is zero if event is certain.


2. If symbols are equally probable (equiprobable) then entropy is maximum and it is given by H
=M
SOURCE EFFICIENCY AND REDUNDANCY

•  Source efficiency, η =
• Redundancy, Re = 1 - η
INFORMATION RATE

• The rate of transfer of a message


• R = H×r
• Where H = entropy
r = sampling rate

You might also like

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy