LANGUAGE PROCESSORS

A language processor, or translator, is a computer program that

translates source code from one programming language to another.
They also identify errors during translation.
Computer programs are usually written in high-level programming
languages (like C++, Python, and Java). Further, to make them
understandable by the computer, a language processor needs to translate the
source code into machine code (also known as object code, which is made
up of ones and zeroes).
There are three types of language processors: assembler, compiler, and
interpreter.
1. Assembler
The assembler translates a program written in assembly language into
machine code.
Assembly language is a low-level, machine-dependent symbolic code that
consists of instructions (like ADD, SUB, MUX, MOV, etc.):

2. Compiler
A compiler reads an entire source code and then translates it into machine
code. Further, the machine code, aka the object code, is stored in an object
file.
If the compiler encounters any errors during the compilation process, it
continues to read the source code to the end and then shows the errors and
their line numbers to the user.
Compiled programming languages are high-level and machine-
independent. Moreover, examples of compiled programming languages are
C, C++, C#, Java, Rust, and Go:
3. Interpreter
An interpreter receives the source code and then reads it line by line,
translating each line of code to machine code and executing it before
moving on to the next line.
If the interpreter encounters any errors during its process, it stops the
process and shows an error message to the user.
Interpreted programming languages are also high-level and machine-
independent. Python, Javascript, PHP, and Ruby are examples of
interpreted programming languages:
 STRUCTURE OF COMPILER

• In a compiler,
o linear analysis
▪ is called LEXICAL ANALYSIS or SCANNING and
▪ is performed by the LEXICAL ANALYZER or LEXER,
o hierarchical analysis
▪ is called SYNTAX ANALYSIS or PARSING and
▪ is performed by the SYNTAX ANALYZER or PARSER.
• During the analysis, the compiler manages a SYMBOL TABLE by
o recording the identifiers of the source program
o collecting information (called ATTRIBUTES) about them: storage
allocation, type, scope, and (for functions) signature.
• When the identifier x is found by the lexical analyzer
o generates the token id
o enters the lexeme x in the symbol-table (if it is not already there)
o associates to the generated token a pointer to the symbol-table entry x.
This pointer is called the LEXICAL VALUE of the token.
• During the analysis or synthesis, the compiler may DETECT ERRORS and
report on them.
o However, after detecting an error, the compilation should proceed
allowing further errors to be detected.
o The syntax and semantic phases usually handle a large fraction of the
errors detectable by the compiler.
 PHASES OF COMPILER
1. Lexical Analysis
In the first phase in the compiler, lexical analysis receives as input the
source code of the program. Lexical analysis is also referred to as linear
analysis or scanning. It's the process of tokenizing.
Lexer scans the input source code, one character at a time. The instant it
identifies the end of a lexeme, it transforms the lexeme into a token. The
input is transformed in this manner into a sequence of tokens. A token is
a meaningful group of characters from the source which the compiler
recognizes. The lexical analyzer then passes these tokens to the next
phase in the compiler. Scanning only eliminates the non-token structures
from the input stream, such as comments, unnecessary white spaces, etc.
The program that implements lexical analysis is known as a lexer,
lexical analyzer, or scanner.
2. Syntax Analysis
The second phase in the compiler, receives as input the stream of tokens
from the previous phase are used to create an intermediate tree-like data
structure known as the parse tree in this phase. The parse tree is
generated with the help of pre-determined grammar rules of the
language that the compiler targets. The syntax analyzer checks whether
or not a given program follows the rules of context-free grammar. If it
does, then the syntax analyzer creates the parse tree for the input source
program. If the syntax is incorrect, it generates a syntax error. The phase
of syntax analysis is also known as hierarchical analysis, or parsing. The
program that is responsible for performing syntax analysis is referred to
as a parser.
3. Semantic Analysis
Semantic Analysis is the third phase of a compiler, coming after
syntax analysis. While syntax analysis checks whether the source
code follows the grammatical structure of the programming language,
semantic analysis ensures that the code is meaningful and logically
correct. It looks at the meaning of the code to find errors that grammar
checks can’t catch. If something doesn’t make sense in the code, it
gives a semantic error. So, semantic analysis helps ensure that the
program is not just written correctly, but also works correctly. It
makes sure that all variables and functions are properly declared and
used, and that the types of data being used are correct and compatible
with each other.
4. Intermediate Code Generation
Intermediate Code Generation (ICG) in compiler design is the fourth
phase in the process of compilation which involves converting high-
level source code into an intermediate representation (IR). This step
improves portability and efficiency, acting as a bridge between source
code and machine code. The IR is independent of machine architecture,
facilitating optimization and easier translation into target machine code
across different platforms. It can be represented using various notations
techniques such as postfix notation, directed acyclic graph, syntax tree,
three-address codes, quadruples, triples, etc.
5. Code Optimization
Code optimization is a program transformation approach that aims to
enhance code by reducing resource consumption (i.e., CPU and
memory) while maintaining high performance. In code optimization,
high-level generic programming structures are substituted with low-
level programming codes. The three guidelines for code optimization
are as follows:
• In no way should the output code alter the program's meaning.
• The program's speed should be increased, and it should use fewer
resources if at all feasible.
• The optimization step should be quick and not hinder the
compilation process.
6. Code Generation
In the sixth and the final phase of the compiler, code generation receives
as input the optimized intermediate code and translates the optimized
intermediate code into into machine code or assembly code that the
computer’s hardware can understand and execute. The main goal of this
phase is to produce efficient and correct machine-level instructions that
performs exactly what the source code was intended to do. It converts
each part of the intermediate code into low-level instructions and
assigns variables to physical memory locations or registers.
7. Error Handler
Error Handling in a compiler is the process of detecting, reporting, and
recovering from errors in a program. The compiler’s job is to find these
errors and tell you about them clearly, so you can fix them. Error
handling is done in almost every phase of the compiler. The compiler
tries not to stop immediately after finding one error. Instead, it continues
checking the rest of the code to find more errors, so the programmer can
fix them all at once. This is called error recovery. A good compiler
doesn't just say “there’s an error” — it also gives helpful messages like
where the error happened and what kind of mistake it is. This makes it
easier for programmers to understand and correct their code. Error
handling helps make programming easier by catching and explaining
mistakes during the compilation process.
8. Symbol Table

A Symbol Table is like a dictionary or a list that the compiler uses to

keep track of important information about your program. When you
write code, you create things like variables, functions, classes, and
objects. The compiler needs to remember details about all these things
such as their names, types (like integer or string), where they are used,
and where they are stored in memory. The symbol table stores this
information in an organized way so the compiler can quickly find it
whenever needed during different stages of compilation. This table is
built and updated during the early stages of compilation, especially
during lexical and semantic analysis. It plays a big role in making sure
the program follows all the rules of the programming language.
 COMPILER CONSTRUCTION TOOLS
The whole compilation process is divided into different phases like Lexical
Analysis, preprocessing, parsing, Semantic Analysis, code generation, and
Code optimisation. There are some specialized tools present that help in the
implementation of various Phases of Compiler known as Compiler
Construction Tools.
Compiler Construction Tools are specialized tools that help in the
implementation of various phases of a compiler. These tools help in the
creation of an entire compiler or its parts.
Some of the commonly used compiler construction tools are:-
• Parser Generator
This tool creates the parser, which checks if the sequence of tokens
follows the grammatical rules of the programming language. It
builds a tree-like structure called a parse tree or syntax tree.
• Scanner Generator
A scanner generator creates the lexical analyzer or lexer. It breaks
the source code into tokens such as keywords, operators, and
identifiers. It reads the code character by character and groups them
into meaningful parts.
• Automatic Code Generators
These tools convert intermediate code into target machine code or
assembly code. They understand the structure of the target computer
and generate correct, working code from higher-level instructions.
They save effort in writing low-level code manually and ensure
better performance and compatibility.
• Data-Flow Analysis Engines
These tools help in analyzing how data moves and changes
throughout the program. They are useful for optimization by finding
things like unused variables, constant values, or repeated
expressions. Data-flow analysis helps make the final code more
efficient by reducing unnecessary instructions or memory use.
 ROLE OF LEXICAL ANALYSIS
1. Reads Source Code Character by Character
The lexical analyzer scans the source code one character at a time from left
to right. It groups characters into meaningful units (lexemes) like variable
names, keywords, or numbers. This helps break the continuous stream of
characters into structured pieces.
2. Generates Tokens
It groups characters into meaningful words called lexemes and converts
them into tokens. Tokens represent categories like keywords, operators,
identifiers, and literals. These tokens make it easier for the parser to
understand the program's structure.
3. Removes Unnecessary Characters
It filters out white spaces, tabs, newline characters, and comments from the
input. These are not needed for the meaning of the program and can be
ignored during parsing. This cleaning process makes the code easier to
process for the next phase.
4. Provides Input to the Parser
It passes the token stream to the syntax analyzer (parser) in a well-
structured format. Each token includes enough information for the parser to
build the syntax tree. Without this, the parser would have to handle raw,
unstructured code, which is inefficient.
5. Maintains Symbol Table
It may store names of variables, functions, and constants in the symbol
table along with their details. This information is used later for type
checking, memory allocation, and code generation. The table helps the
compiler remember what each identifier refers to.
6. Identifies Errors in Lexemes
If a lexeme is not recognized or does not match any valid pattern, the
lexical analyzer flags it as an error. This allows early detection of mistakes,
like invalid symbols or misplaced characters, which can be reported to the
programmer for correction.
7. Tracks Line Numbers and Positions
The lexical analyzer keeps track of the position of each token within the
source code, including line numbers and column positions. This is helpful
when reporting errors or warnings, as it provides the exact location of the
issue, making it easier for the programmer to fix it.
8. Improves Efficiency
By handling the task of breaking down source code into tokens, the lexical
analyzer reduces the complexity for subsequent stages of the compiler. It
ensures that the parser and later phases focus only on essential information,
improving the overall speed and efficiency of the compilation process.

INPUT BUFFERING
Input Buffering in Compiler is a method used to increase the speed of
reading of the source code by reducing the number of times the compiler
needs to access the source file. Without input buffering, the compiler needs
to read each character from the file one at a time, which is slow and time
consuming. Input buffering solves this problem by reading large blocks of
characters into memory at once, thus least the number of input operations.
How Input Buffering Works?
The basic idea of input buffering is to use a buffer, which is a block of
memory where the source code is temporarily stored. There are typically
two types of buffers used:
1. Single Buffer:
- A single large block of memory that holds part of the source code.
2. Double Buffer: Two blocks of memory, used alternately, to ensure that
while one buffer is being processed, the other can be filled with new
characters from the source file.
Single Buffer
In a single buffer system, the compiler reads a large block of the source file
into a buffer. The lexical analyser then processes this buffer character by
character to identify tokens.
When the buffer is exhausted, the next block of characters is read into the
same buffer, and the process repeats. While simple, this method can be
inefficient because the processing has to stop every time the buffer needs to
be refilled.
Here's how single buffering works in detail:
- One buffer is used named Buffer A.
- The compiler fills Buffer A with characters from the source file.
- The lexical analyser starts processing characters from Buffer A.
- When the buffer is completely processed and is empty, the compiler starts
filling Buffer A with the next set of characters.
- This process continues after refilling, until the end of the file.

Double Buffer
A more efficient approach is double buffering. In this system, there are two
buffers. While the lexical analyser processes characters from one buffer, the
other buffer can be filled with the next block of characters from the source
file. This overlapping of processing and reading helps in maintaining a
continuous flow of characters and reduces the waiting time.
Here's how double buffering works in detail:
- Buffer A and Buffer B are the two buffers.
- The compiler fills Buffer A with characters from the source file.
- The lexical analyser starts processing characters from Buffer A.
- When Buffer A is half-processed, the compiler starts filling Buffer
B with the next set of characters.
- Once Buffer A is completely processed, the lexical analyser switches
to Buffer B.
- This process continues, keeping the flow smooth and uninterrupted.
Sentinels in Input Buffering
To make the process of input buffering fast, sentinels can be used. Sentinels
are special characters placed at the end of each buffer to signify the end.
This eliminates the need for checking the buffer's end condition repeatedly,
which can slow down the process.
Advantages of Input Buffering
1. Efficiency:
By reading large blocks of data at once, input buffering reduces the number
of input operations, making the process faster.
2. Reduced Latency:
Double buffering ensures that while one buffer is being processed, the other
is being filled, reducing waiting time and increasing the overall speed of
the lexical analysis.
3. Smooth Processing:
The use of sentinels helps in seamless buffer transitions, avoiding constant
end-of-buffer checks.
 REGULAR EXPRESSIONS
A regular expression can also be described as a sequence of pattern that
defines a string. In Compiler Design, regular expressions are concise
notations used to define and recognize patterns in source code. They play a
crucial role in lexical analysis, where they are used to identify and extract
tokens (e.g., keywords, identifiers) from code. Regular expressions help in
defining the syntax of programming languages, facilitating the
transformation of human-readable code into machine-readable forms
during compilation .
For instance:
• In a regular expression, x* means zero or more occurrence of x. It
can generate {ε, x, xx, xxx, xxxx, .....}
• In a regular expression, x+ means one or more occurrence of x. It can
generate {x, xx, xxx, xxxx, .....}
Here are some examples of regular expressions commonly used in compiler
design, particularly for lexical analysis:

1. Identifiers:
• Regex: ^[a-zA-Z_][a-zA-Z0-9_]*$
• Explanation: This regular expression matches valid identifiers in
programming languages. It ensures that an identifier starts with a
letter or underscore, followed by any combination of letters, digits,
or underscores. This pattern is used to recognize variable names,
function names, and other identifiers.
2. Numeric Literals:
• Regex: ^(\d+(\.\d+)?|\.\d+)$
• Explanation: This regex matches numeric literals, including both
integers and floating-point numbers. It allows for optional decimal
points, ensuring that integers can be expressed as whole numbers and
floating-point numbers can start with digits, end with digits, or begin
with a decimal point.
3. String Literals:
• Regex: ^"([^"\\]|\\.)*"$
• Explanation: This regular expression matches string literals
enclosed in double quotes. It allows for any characters except for
unescaped double quotes, and it accommodates escape sequences
(like \" or \\). This pattern is essential for recognizing string data
types in programming languages.
 ROLE OF PARSERS

A parser is a crucial component of a compiler responsible for checking the

syntax of the source code and constructing a data structure known as a
parse tree.
After the lexical analysis phase, where the source code is broken into
tokens (such as keywords, identifiers, and symbols), the parser takes over.
It analyzes these tokens to ensure that they follow the grammatical rules of
the programming language. The grammar of a programming language
defines how tokens can be combined to form valid statements and
expressions. The parser checks whether the token sequence adheres to these
rules. If the source code violates any of them, the parser generates an
appropriate syntax error message, and the compilation process is stopped.
If the code is syntactically correct, the parser constructs a parse tree—a
hierarchical data structure that represents the syntactic structure of the
source code. This parse tree provides the foundation for later stages of
compilation.
By organizing the code into structured trees, parsers help streamline later
stages of compilation, making it easier to analyze, optimize, and translate
code efficiently.
There are two ways of identifying an elementary subtree:
1. By deriving a string from a non-terminal
2. By reducing a string of symbol to a non-terminal.
The parser serves as a bridge between raw token streams and meaningful
program structures. It ensures syntactic correctness, facilitates detailed
analysis, supports error handling, and lays the groundwork for efficient
code generation, making it one of the most essential phases in the entire
compilation process.
 CONTEXT FREE GRAMMAR

A Context-Free Grammar (CFG) is a set of recursive rules used to generate

patterns of strings. It is a formal system used to define the syntactical
structure of a language. It consists of a set of production rules, where each
rule specifies how a symbol or group of symbols can be replaced by other
symbols.
A context-free grammar G=(V,T,P,S) is composed of
• V : a set of variables (also known as non-terminals), each denoting a
set of strings.
• T : a set of terminal symbols (“terminals” for short) that constitutes
the alphabet over which the strings in the language are composed.
• P : a set of productions, rules that recursively define the structure of
the language.
A production has the form A→α where
o A is a variable (one of the symbols in VV).
o α is a string of zero or more symbols that may be either
terminals or variables.
• S : a starting symbol. This is a variable that denotes the set of strings
comprising the entire language.
Example
Construct a CFG for language L = {wcwR | where w € (a, b)*}.
The production rules can be
• S → aSa rule 1
• S → bSb rule 2
• S→c rule 3.
Now, we can use it to derive different strings. Let’s take an example of the
string “abbcbba”
• S → aSa
• S → abSba from rule 2
• S → abbSbba from rule 2
• S → abbcbba from rule 3

Applications
CFG has great practical importance. Some of the applications are given
below:
• For defining programming languages.
• For the construction of compilers
• For describing Arithmetic expressions
• For the transition of programming languages.
 SHIFT REDUCING PARSING

A shift-reduce parser is a bottom-up parsing technique that uses a stack. It

shifts input symbols onto the stack and reduces them based on grammar
rules until the input is completely parsed. It continuously reduces or shifts
symbols until a valid parse is achieved.
The Shift reduce parsing is a type of bottom-up parsing as it generates a
parse tree from the leaves (bottom) to the root(up).
• In shift-reduce parsing, the input string is reduced to the starting
symbol.
• This reduction can be achieved by directly handling the rightmost
derivation from the starting symbol to the input string.
• Two Data Structures are required to perform shift-reduce parsing-
- An input buffer to hold the input string.
- A stack to keep the grammar symbols for accessing the
production rules.
Basic Operations in Shift-Reduce Parsing
There are four basic operations a shift-reduce parser can perform:
1. Shift- This operation involves moving the current symbol from the
input buffer onto the stack.
2. Reduce- When the parser knows the right hand of the handle is at the
top of the stack, the reduce operation applies the applicable
production rules, i.e., pops out the RHS of the production rule from
the stack and pushes the LHS of the production rule onto the stack.
3. Accept- After repeating the shift and reduce operations, if the stack
contains the starting symbol of the input string and the input buffer is
empty, i.e., includes the $ symbol, the input string is said to be
accepted.
4. Error- If the parser cannot perform the shift or the reduce operation,
also the string is not accepted, then it is said to be in the error state.
Rules for Shift Reduce
• Rule 1: If the priority of the incoming operator is higher than the
operator's priority at the top of the stack, then we perform the shift
action.
• Rule 2: If the priority of the incoming operator is equal to or less
than the operator's priority at the top of the stack, then we perform
the reduce action.
Example
 OPERATOR PRECEDANCE PARSING
Operator precedence parsing is a type of Shift Reduce Parsing. In operator
precedence parsing, the shift and reduce operations are done based on the
priority between the symbol at the top of the stack and the current input
symbol.
In operator precedence parsing, an operator grammar and an input string
are fed as input to the operator precedence parser, which may generate a
parse tree
Operator Grammar
A grammar is said to be an operator grammar if it follows these two
properties:
1. There should be no ε (epsilon) on the right-hand side of any
production.
2. There should be no two non-terminals adjacent to each other.
Operator Precedence Table
A precedence table is used in operator precedence parsing to establish the
relative precedence of operators and to resolve shift-reduce conflicts during
the parsing process. The table instructs the parser when to shift (consume
the input and proceed to the next token) and when to reduce (apply a
production rule to reduce a set of tokens to a non-terminal symbol). It is an
essential Data Structure for building a shift-reduce parser.
A precedence table is commonly expressed as a two-dimensional matrix,
with rows and columns corresponding to grammatical operators. The table's
elements define the order of precedence between the operators.
Example
 TOP DOWN PARSING
Top-down parsing means parsing the input and constructing the parse tree,
starting from the root and going down to the leaf. It uses left most
derivation to build a parse tree.
A top-down parser builds the leftmost derivation from the grammar’s start
symbol. Then it chooses a suitable production rule to move the input string
from left to right in sentential form.
Leftmost derivation:
It is a process of exploring the production rules from left to right and
selecting the leftmost non-terminal in the current string as the next symbol
to expand. This approach ensures that the parser always chooses the
leftmost derivation and tries to match the input string. If a match cannot be
found, the parser backtracks and tries another production rule. This process
continues until the parser reaches the end of the input string or fails to find
a valid parse tree.

Top-down parsing in compiler design is a software application included

with the compiler, and parsing is a step in the compilation process. Parsing
occurs during the compilation’s analysis stage. Parsing is the process of
taking code from the preprocessor, breaking it down into smaller pieces,
and analyzing it so that other software can understand it.
Parsing

Parsing is the process of converting information from one type to another.

The parser is a component of the translator in the organization of linear text
structure according to a set of defined rules known as grammar.
Types of the Parser:
Parser is divided into two types:
• Bottom-up parser
• Top-down parser

Bottom-Up Parser
A bottom-up parser is a type of parsing algorithm that starts with the input
symbols to construct a parse tree by repeatedly applying production rules in
reverse until the start symbol is reached. Bottom-up parsers are also known
as shift-reduce parsers because they shift input symbols onto the parse
stack until a set of consecutive symbols can be reduced by a production
rule.
Top-Down Parser
A top-down parser in compiler design can be considered to construct a
parse tree for an input string in preorder, starting from the root. It can also
be considered to create a leftmost derivation for an input string. The
leftmost derivation is built by a top-down parser. A top-down parser builds
the leftmost derivation from the grammar’s start symbol. Then it chooses a
suitable production rule to move the input string from left to right in
sentential form.
Leftmost derivation:
It is a process of exploring the production rules from left to right and
selecting the leftmost non-terminal in the current string as the next symbol
to expand. This approach ensures that the parser always chooses the
leftmost derivation and tries to match the input string. If a match cannot be
found, the parser backtracks and tries another production rule. This process
continues until the parser reaches the end of the input string or fails to find
a valid parse tree.
Example of Top-Down Parsing
Consider the lexical analyzer’s input string ‘acb’ for the following grammar
by using left most deviation.

Classification of the Top-Down Parser:

Top-down parsers can be classified based on their approach to parsing as
follows:
• Recursive-descent parsers: Recursive-descent parsers are a type of
top-down parser that uses a set of recursive procedures to parse the
input. Each non-terminal symbol in the grammar corresponds to a
procedure that parses input for that symbol.
• Backtracking parsers: Backtracking parsers are a type of top-down
parser that can handle non-deterministic grammar. When a parsing
decision leads to a dead end, the parser can backtrack and try another
alternative. Backtracking parsers are not as efficient as other top-
down parsers because they can potentially explore many parsing
paths.
• Non-backtracking parsers: Non-backtracking is a technique used
in top-down parsing to ensure that the parser doesn’t revisit already-
explored paths in the parse tree during the parsing process. This is
achieved by using a predictive parsing table that is constructed in
advance and selecting the appropriate production rule based on the
top non-terminal symbol on the parser’s stack.
• Predictive parsers: Predictive parsers are top-down parsers that use
parsing to predict which production rule to apply based on the next
input symbol. Predictive parsers are also called LL parsers because
they construct a left-to-right, leftmost derivation of the input string.
 PREDICTIVE PARSING
Predictive parsing is a form of recursive descent parsing, in which no
backtracking is needed, so it can predict which products to use as the
replacement of the input string.
Predictive parsing is a parsing technique used in compiler construction and
syntax analysis. It helps compilers analyze and understand the structure of
code by predicting the part that comes next depending on what's already
there.
Predictive parsing is a parsing technique used in compiler design to analyze
and validate the syntactic structure of a given input string based on a
grammar. It predicts the production rules to apply without backtracking,
making it efficient and deterministic.

https://www.naukri.com/code360/library/predictive-parsing
LR(0) Parser
SLR(0) Parser
CLR(1)
LALR(1)