Theory of Computation

Muzurura O
Computer Science Department
MSU
 What is this course about?

 Thismodule is on the Theory of

Computation, tries to answer the following
questions:
– What are the mathematical properties of
computer hardware and software?
– What is a computation and what is an algorithm ?
Can we give rigorous mathematical definitions of
these notions?
– What are the limitations of computers? Can
“everything” be computed? (As we will see, the
answer to this question is “no”.)
Theory of Computation

Purpose of the Theory of Computation:

• Develop formal mathematical models of
computation that reflect real-world
computers.
• Theory of Computation can be divided into
the following three areas: Complexity Theory,
Computability Theory, and Automata Theory.
Complexity Theory
 Central Question in Complexity Theory:
– Classify problems according to their degree of “difficulty”. Give a rigorous
proof that problems that seem to be “hard” are really “hard”.
– “What makes some problems computationally hard and other problems
easy ?”
 Informally, a problem is called “easy”, if it is efficiently solvable.
Examples of “easy” problems are
– sorting a sequence of, say, 1,000,000 numbers,
– searching for a name in a telephone directory, and
– computing the fastest way to drive from Gweru to Harare.
 On the other hand, a problem is called “hard”, if it cannot be
solved efficiently, or if we don’t know whether it can be solved
efficiently. Examples of “hard” problems are
– time table scheduling for all modules at MSU,
– computing a layout for TRANSISTORS in a chip
Computability theory

 Central Question in Computability Theory:

– Classify problems as being solvable or
unsolvable.
 The theoretical models that were proposed in
order to understand solvable and unsolvable
problems led to the development of real
computers.
Automata theory
 Automata Theory deals with definitions and properties of
different types of “computation models”. Examples of such
models are:
– Finite Automata. These are used in text processing, compilers,
and hardware design.
– Context-Free Grammars. These are used to define programming
languages and in Artificial Intelligence.
– Turing Machines. These form a simple abstract model of a “real”
computer, such as your PC.
 Central Question in Automata Theory: Do these models have
the same power, or can one model solve more problems than
the other?
Why should one study Theory of Computation

 Thereare a number of reasons for you to study

theory of computation if you are in the field of
computer science. Some of those reasons are :
– It plays an important role when making software for
designing and checking the behavior of digital
circuits.
– It is used in the design of lexical analyzer phase and
parser (syntax analyzer). The techniques used in
there design can also be used for designing other
software component.
Why should one study Theory of Computation

– It is also used in developing the Scanning Software

for large body of text to find occurrence of words,
phrases or other patterns.
– Regular Expressions are used in pattern recognition.
– Automata theory is very useful for natural language
processing, computer security, artificial intelligence,
etc., etc.
– This module helps you to learn problem solving
skills. TOC teaches you how to think, prove, argue,
solve problems, express, and abstract.
Why should one study Theory of
Computation
 The study of computation aims at providing an insight
into the characteristics of computations.
 Such an insight can be used for predicting the
complexity of desired computations, for choosing the
approaches they should take, and for developing tools
that facilitate their design.
 The study of computation reveals that there are
problems that cannot be solved.
 And of the problems that can be solved, there are
some that require infeasible amount of resources.
 Algorithm
 A finitesequence of simple instructions that is
guaranteed to halt in a finite amount of time.
 This is a very abstract definition, since:
– We didn’t specify the nature of this simple
instructions.
 Forexample an instruction can be “increment a number by
one” or “Calculate the triple integral”
– We didn’t specify the entity which can execute these
instructions.
 For example is this entity a person OR a computer, …
 If it is a computer what is the processor type? How much
memory does it have? …. ?
Computation

 Computation is a general term for any type of

information processing that can be represented
as an algorithm precisely (mathematically).
Examples:
 Adding two numbers in our brains, on a piece of paper or
using a calculator.
 Converting a decimal number to its binary presentation or
vise versa.
 Finding the greatest common divisors of two numbers.
 Computations are designed for processing
information.
 An Abstract Machine
 To make a more solid definition of algorithm we need to
define an abstract (general) machine which can perform
any algorithm that can be executed by any computer.
 Then, We need to show that indeed this machine can run
any algorithm that can be executed by any other
computer. Then,
– We can associate the notion of algorithm with this abstract
machine.
– We can study this machine to find the limitations of
computations. (Problems with no computation available to
solve.)
Turing Machine
 A conceptual model for general purpose
computers proposed by Alan Turing in 1936.
 A Turing machine has an unlimited and
unrestricted amount of memory.
 A Turing machine can do everything a real
computer can do.
 Nevertheless there are problems that a Turing
machine cannot solve.
 In a real sense, these problems are beyond
the theoretical limits of computations.
 Strings

1. Alphabet - a finite set of symbols.

– Notation:  .
– Examples: Binary alphabet {0,1},
English alphabet {a,...,z,!,?,...}
2. String over an alphabet  - a finite sequence of
symbols from .
– Notation: (a) Letters u, v, w, x, y, and z denote strings.
(b) Convention: concatenate the symbols. No
parentheses or commas used.
– Examples: 0000 is a string over the binary alphabet.
a!? is a string over the English alphabet.
 Definitions

3. Empty string: e or  denotes the empty

sequence of symbols.
4. Language over alphabet  - a set of strings
over .
– Notation: L.
– Examples:
 {0, 00, 000, ...} is an "infinite" language over the binary
alphabet.
 {a, b, c} is a "finite" language over the English alphabet.
 Definitions (contd.)

5. Empty language - empty set of strings.

Notation: .
6. Binary operation on strings:
Concatenation of two strings u.v -
concatenate the symbols of u and v.
– Notation: uv
– Examples:
 00.11 = 0011.
 .u = u. =  for every u. (identity for concatenation)

– Reverse, Permutation, ….
 Binary relations on strings

1. Prefix - u is a prefix of v if there is a w such that v

= uw.
– Example:
 01 is a prefix of 011
2. Proper Prefix – the prefix u is said to be a
proper prefix of v if w ≠ ε.
– Example: apple is a prefix of appleton since appleton =
apple.ton
 Binary relations (contd.)

3. Suffix - u is a suffix of v if there is a w such that

v = wu.
– Examples: ba is a suffix of abba
4. Proper Suffix – the suffix u of v is said to be a
proper suffix of v if w ≠ ε.
– Examples:
 ton is a suffix of appleton since ?

ton= apple.ton
 Binary relations (contd.)

5. Substring - u is a substring of v if there are

x and y such that v = xuy.
– Examples:
 let is a substring of appleton since appleton =
app.let.on

NB: Observe that prefix and suffix are special

cases of substring.
Ordering of Strings
 Searching is probably the most commonly applied
operation on information.
 Due to the importance of this operation, approaches for
searching information and for organizing information to
facilitate searching, receive special attention.
 Sequential search, binary search, insertion sort, quick
sort, and merge sort are some examples of such
approaches.
 These approaches rely in most cases on the existence of
a relationship that defines an ordering of the entities in
question.
Ordering of Strings - Alphabetical ordering

 A stringu is said to be alphabetically smaller in

Σ* than a string v, or equivalently, v is said to
be alphabetically bigger in Σ* than u if u and v
are in Σ* and either of the following two cases
holds:
1. u is a proper prefix of v.
2. For some w in Σ* and some a and b in Σ such that
a precedes b in Σ, the string wa is a prefix of u and
the string wb is a prefix of v.
Ordering of Strings - Alphabetical ordering

 Example: Let Σ be the binary alphabet {0, 1}.

The string 01 is alphabetically smaller in Σ* than
the string 01100, because 01 is a proper prefix of
01100.
 On the other hand, 01100 is alphabetically
smaller than 0111, because both strings agree in
their first three symbols and the fourth symbol in
01100 is smaller than the fourth symbol in 0111.
Ordering of Strings - Canonical ordering

 A string u is said to be canonically smaller or

lexicographically smaller in Σ* than a string v,
or equivalently, v is said to be canonically
bigger than u or lexicographically bigger in Σ*
than u if either of the following two cases holds:
1. u is shorter than v.
2. u and v are of identical length but u is
alphabetically smaller than v.
Ordering of Strings - Canonical ordering

 Example:
 Consider the alphabet Σ = {0, 1}. The string 11
is canonically smaller in Σ* than the string 000,
because 11 is a shorter string than 000.
 On the other hand, 00 is canonically smaller
than 11, because the strings are of equal length
and 00 is alphabetically smaller than 11.
Relevance of strings and languages for
this course.

 Consider decision problems. Decision problems

are problems with Yes/No answers.
 Examples: (a) Is an array sorted?

(b) Is x in that array A?

 Relevance (contd.)

 We can encode instances of any problem by

strings.
 Associate language with decision problem D
by taking all the strings that encode its yes
instances.
Assigment 1
1. Find all the pairs of strings and over the alphabet {a, b} that satisfy =
abb. [10]
2. For each of the following cases give the first six strings in {a, b, c}*.
3. In alphabetical ordering. [10]
4. In canonical ordering. [10]

UofH - COSC 3340 - Dr. Verma

Applications of the Concepts

 Are vast, and include:

– Text processing (editors, etc.)
– Compiler design
– Verification through model checking
 Real world examples are all around you and
not just on the desk!