Unit I Fundamentals of Natural Language (06 Hours)

Processing
History of NLP, Generic NLP system, levels of NLP, Knowledge in language processing, Ambiguity
in Natural language, stages in NLP, challenges of NLP, Applications of NLP, Approaches of NLP:Rule
based, Data Based, Knowledge Based approaches

History of NLP :
NLP stands for Natural Language Processing, which is a part of Computer Science, Human
language, and Artificial Intelligence. It is the technology that is used by machines to
understand, analyse, manipulate, and interpret human's languages. It helps developers to
organize knowledge for performing tasks such as translation, automatic summarization,
Named Entity Recognition (NER), speech recognition, relationship extraction, and topic
segmentation.

1. Early Beginnings (1940s–1950s)

• Alan Turing's Influence(1950):

Turing’s paper "Computing Machinery and Intelligence" introduced the idea of
machines processing language and proposed the Turing Test to measure a machine’s
ability to exhibit human-like intelligence.
 • First Experiments:
Early NLP efforts focused on machine translation during the Cold War. The
Georgetown-IBM experiment (1954) demonstrated automatic translation of 60 Russian
sentences into English, sparking interest in computational linguistics.

2. Rule-Based Systems (1950s–1970s)

• Symbolic NLP:
Early systems relied on handcrafted rules and grammars. These approaches used
syntax-driven parsing and context-free grammars (e.g., Chomsky’s linguistic theories).
• Key Systems:
o ELIZA (1966): A rule-based chatbot that mimicked a psychotherapist by
pattern-matching and generating responses.
o SHRDLU (1970): A system that interacted with users in a simulated blocks
world using semantic parsing.
• Limitations:
Rule-based systems struggled with ambiguity, context, and the complexity of natural
languages.

3. Statistical Methods (1980s–1990s)

• Shift to Data-Driven Approaches:

With increasing computational power and the availability of text corpora, researchers
began using probabilistic models like Hidden Markov Models (HMMs) and n-grams
for tasks like speech recognition and part-of-speech tagging.
• Notable Milestones:
o Introduction of the Penn Treebank (1993): A resource for syntactic and semantic
annotation of text.
o IBM’s statistical machine translation system (1980s): Pioneered the use of
statistical methods for translating text.
• Limitations:
These methods often required large datasets and lacked deeper linguistic understanding.
4. Rise of Machine Learning (1990s–2010s)

• Supervised Learning Models:

Algorithms like Support Vector Machines (SVMs) and decision trees were applied to
tasks such as sentiment analysis, named entity recognition, and document classification.
• Emergence of Word Embeddings:
o Latent Semantic Analysis (LSA) and later, word2vec (2013), introduced ways
to represent words in continuous vector spaces, capturing semantic
relationships.
• Growth of NLP Libraries:
Tools like NLTK and Stanford NLP became widely used in academia and industry.

5. Deep Learning Revolution (2010s–Present)

• Neural Networks for NLP:

Deep learning models, particularly Recurrent Neural Networks (RNNs) and Long
Short-Term Memory (LSTM) networks, significantly improved performance in
sequential tasks like machine translation and text generation.
• Attention Mechanism and Transformers:
o The Transformer architecture (2017), introduced in the "Attention is All You
Need" paper, revolutionized NLP by allowing parallelization and improved
handling of long-range dependencies.
o Models like BERT (2018), GPT (2018–), and T5 became state-of-the-art,
achieving breakthroughs in tasks such as question answering, summarization,
and conversational AI.
• Pre-trained Models and Transfer Learning:
Large-scale pre-trained language models have become dominant, enabling fine-tuning
for various downstream tasks with minimal labelled data.
6. Recent Trends and the Future (2020s)

• Large Language Models(LLMs):

Models like GPT-4, PaLM, and ChatGPT demonstrate human-like text generation,
conversation, and reasoning capabilities.
• Multimodal Models:
Integration of text, image, and audio processing (e.g., CLIP, DALL-E) has broadened
NLP’s applications.
• Ethics and Bias in NLP:
Growing concerns about fairness, accountability, and transparency have led to research
on mitigating biases in language models.
• Low-Resource NLP:
Efforts are underway to make NLP tools accessible for low-resource languages and
underserved communities.

Applications of NLP

• Machine Translation (e.g., Google Translate)

• Speech Recognition (e.g., Siri, Alexa)
• Sentiment Analysis
• Chatbots and Virtual Assistants
• Text Summarization
• Information Retrieval (e.g., search engines)

Advantages of NLP

1. NLP helps users to ask questions about any subject and get a direct response within
seconds.
2. NLP offers exact answers to the question means it does not offer unnecessary and
unwanted information.
3. NLP helps computers to communicate with humans in their languages.
4. It is very time efficient.
5. Most of the companies use NLP to improve the efficiency of documentation processes,
accuracy of documentation, and identify the information from large databases.
Disadvantages of NLP
A list of disadvantages of NLP is given below:

1. NLP may not show context.

2. NLP is unpredictable
3. NLP may require more keystrokes.
4. NLP is unable to adapt to the new domain, and it has a limited function that's why NLP
is built for a single and specific task only.

Generic NLP system

Working in natural language processing (NLP) typically involves using computational

techniques to analyse and understand human language. This can include tasks such as
language understanding, language generation, and language interaction.
1. Text Input and Data Collection
• Data Collection: Gathering text data from various sources such as websites, books, social
media, or proprietary databases.
• Data Storage: Storing the collected text data in a structured format, such as a database
or a collection of documents.
2. Text Pre-processing
Pre-processing is crucial to clean and prepare the raw text data for analysis. Common pre-
processing steps include:
• Tokenization: Splitting text into smaller units like words or sentences.
• Lowercasing: Converting all text to lowercase to ensure uniformity.
• Stop word Removal: Removing common words that do not contribute significant
meaning, such as “and,” “the,” “is.”
• Punctuation Removal: Removing punctuation marks.
• Stemming and Lemmatization: Reducing words to their base or root forms. Stemming
cuts off suffixes, while lemmatization considers the context and converts words to their
meaningful base form.
• Text Normalization: Standardizing text format, including correcting spelling errors,
expanding contractions, and handling special characters.
3. Text Representation
• Bag of Words (BoW): Representing text as a collection of words, ignoring grammar and
word order but keeping track of word frequency.
• Term Frequency-Inverse Document Frequency (TF-IDF): A statistic that reflects the
importance of a word in a document relative to a collection of documents.
• Word Embeddings: Using dense vector representations of words where semantically
similar words are closer together in the vector space (e.g., Word2Vec, GloVe).
4. Feature Extraction
Extracting meaningful features from the text data that can be used for various NLP tasks.
• N-grams: Capturing sequences of N words to preserve some context and word order.
• Syntactic Features: Using parts of speech tags, syntactic dependencies, and parse trees.
• Semantic Features: Leveraging word embeddings and other representations to capture
word meaning and context.
5. Model Selection and Training
Selecting and training a machine learning or deep learning model to perform specific NLP
tasks.
• Supervised Learning: Using labelled data to train models like Support Vector Machines
(SVM), Random Forests, or deep learning models like Convolutional Neural Networks
(CNNs) and Recurrent Neural Networks (RNNs).
• Unsupervised Learning: Applying techniques like clustering or topic modelling (e.g.,
Latent Dirichlet Allocation) on unlabelled data.
• Pre-trained Models: Utilizing pre-trained language models such as BERT, GPT, or
transformer-based models that have been trained on large corpora.
6. Model Deployment and Inference
Deploying the trained model and using it to make predictions or extract insights from new
text data.
• Text Classification: Categorizing text into predefined classes (e.g., spam detection,
sentiment analysis).
• Named Entity Recognition (NER): Identifying and classifying entities in the text.
• Machine Translation: Translating text from one language to another.
• Question Answering: Providing answers to questions based on the context provided by
text data.
7. Evaluation and Optimization
Evaluating the performance of the NLP algorithm using metrics such as accuracy, precision,
recall, F1-score, and others.
• Hyperparameter Tuning: Adjusting model parameters to improve performance.
• Error Analysis: Analysing errors to understand model weaknesses and improve
robustness.
8. Iteration and Improvement
Continuously improving the algorithm by incorporating new data, refining pre-processing
techniques, experimenting with different models, and optimizing features.
Levels of NLP

1. Phonology Level

• At this level basically, it deals with pronunciation.

• It deals with the interpretation of speech sound across words

2. Morphological Level

• It deals with the smallest words that convey meaning and suffixes and prefixes.
• Morphemes mean studying the words that are built from smaller meanings.

• E.g So the rabbit word has single morphemes while the rabbits have two morphemes.

• The ‘s’ denotes the singular and plural concepts.

3. Lexical Level

• This deals with the study at the level of words with respect to their lexical meaning and
Part of speech (POS)

• It uses the lexicon that is collection of lexemes.

• A Lexeme is a basic unit of lexical meaning which is an abstract unit of morphological

analysis

4. Syntactic Level

• This level deals with the grammar and structure pf sentences

• It studies the proper relationships between the words.

• The POS tagging output of lexical analysis can be used at the syntactic level of two group
words into the phrase and clause brackets.

5. Semantics Level

• This level deals with the meaning of words and sentences

• There are different two approaches:

o 1) Syntax driven semantic analysis

o 2) Semantic grammar

• Its a study of meaning of words that are associated with grammatical structure.
6. Disclosure Level

• This level deals with the structure of different kinds of text.

• There are 2 types of discourse:

o 1) Anaphora resolution

o 2) Discourse/ text structure recognition

• Words are replaced in anaphora resolution.

7. Pragmatic Level

• This level deals with the use real world knowledge and understanding of how this
influences the meaning of what is being communicated.

• Pragmatics identifies the meaning of words and phrases based on how language is used to
communicate.

Knowledge required in NLP

• Phonetic and Phonological knowledge

• Morphological Knowledge

• Syntactic Knowledge

• Semantic knowledge

• Pragmatic Knowledge

• Discourse Knowledge

• Word knowledge

Phonetic and Phonological Knowledge

1. Phonetics is the study of language at the level of sounds while phonology is the study
of the combination of sounds into organized units of speech.
2. Phonetic and Phonological knowledge is essential for speech-based systems as they deal
with how words are related to the sounds that realize them.

Morphological Knowledge

1. Morphology concerns word-formation.

2. It is a study of the patterns of formation of words by the combination of sounds into

minimal distinctive units of meaning called morphemes.

3. Morphological Knowledge concerns how words are constructed from morphemes.

Syntactic Knowledge

1. The syntax is the level at which we study how words combine to form phrases, phrases
combine to form clauses and clauses join to make sentences.

2. The syntactic analysis concerns sentence formation.

3. It deals with how words can be put together to form correct sentences.

Semantic Knowledge

1. It concerns the meaning of the words and sentences.

2. Defining the meaning of a sentence is very difficult due to the ambiguities involved.

Pragmatic Knowledge

1. Pragmatics is the extension of the meanings or semantics.

2. Pragmatics deals with the contextual aspects of meaning in particular situations.

3. It concerns how sentences are used in different situations.

Discourse Knowledge

1. Discourse concerns connected sentences. It includes the study of chunks of language

which are bigger than a single sentence.
2. Discourse language concerns inter-sentential links that is how the immediately
preceding sentences affect the interpretation of the next sentence.

3. Discourse language is important for interpreting pronouns and temporal aspects of the
information conveyed.

Word Knowledge

1. Word knowledge is nothing but everyday knowledge that all the speakers share about
the world.

2. It includes the general knowledge about the structure of the world and what each
language user must know about the other user’s beliefs and goals.

3. This is essential to make the language understanding much better.

Ambiguity in Natural language:

Ambiguity in Natural Language Processing (NLP) refers to situations where a word, phrase, or
sentence can be interpreted in more than one way. Ambiguity arises because natural languages
are inherently flexible and context-dependent. Resolving ambiguity is one of the key challenges
in NLP.

Types of Ambiguity in NLP:

1. Lexical Ambiguity
o Occurs when a word has multiple meanings.
o Example:
▪ "Bank" can mean a financial institution or the side of a river.
2. Syntactic Ambiguity
o Happens when a sentence can be parsed in multiple ways due to its structure.
o Example:
▪ "I saw the man with a telescope."
(Did I use a telescope to see the man, or does the man have a telescope?)
3. Semantic Ambiguity
 o Arises when the meaning of a sentence is unclear even after syntactic structure
is determined.
o Example:
▪ "Visiting relatives can be fun."
(Does it mean the act of visiting them is fun, or that relatives who visit
are fun?)
4. Pragmatic Ambiguity
o Happens when the context of the statement leaves its intent unclear.
o Example:
▪ "Can you pass the salt?"
(Is this a request or a question about capability?)
5. Anaphoric Ambiguity
o Occurs when a pronoun or reference in a sentence could refer to multiple
antecedents.
o Example:
▪ "John met Bill, and he gave him a book."
(Who gave the book to whom?)

Addressing Ambiguity in NLP:

• Contextual Models: Use machine learning models like transformers (e.g., BERT,
GPT) to capture the context.
• Rule-Based Systems: Develop syntactic or semantic rules to resolve specific
ambiguities.
• Word Sense Disambiguation (WSD): Techniques to determine which sense of a word
is being used in a given context.
• Coreference Resolution: Identify entities or pronouns and determine their references.
• Domain-Specific Knowledge: Use specialized knowledge to reduce possible
interpretations.

Stages in NLP:

Natural Language Processing (NLP) involves several stages that transform raw text data into a
form that machines can process and understand. These stages are often categorized as a
pipeline. Below are the primary stages in NLP:

1. Text Pre-processing

Preparing raw text for further processing and analysis.

• Tokenization: Splitting text into smaller units (e.g., words or sentences).

Example: "Hello, world!" → ["Hello", ",", "world", "!"]
• Stop word Removal: Removing common words (e.g., "is", "the", "and") that do not
carry significant meaning.
• Stemming and Lemmatization: Reducing words to their base or root form.
Example: "Running" → "Run" (stemming), "better" → "good" (lemmatization).
• Lowercasing: Converting all text to lowercase for uniformity.
• Noise Removal: Eliminating punctuation, special characters, or irrelevant data (e.g.,
HTML tags).

2. Lexical Analysis

Understanding the meaning of individual words.

• Part-of-Speech (POS) Tagging: Assigning grammatical tags to words (e.g., noun,

verb,adjective).
Example: "She runs fast" → [She/PRONOUN, runs/VERB, fast/ADVERB].
 • Named Entity Recognition (NER): Identifying entities such as names, dates, or
locations.
Example: "John lives in New York." → [John/Person, New York/Location].

3. Syntactic Analysis (Parsing)

Analysing sentence structure and grammar to form parse trees.

• Dependency Parsing: Determining relationships between words.

Example: "The cat chased the mouse." → "chased" is the root, "cat" is the subject,
"mouse" is the object.
• Constituency Parsing: Breaking a sentence into phrases (noun phrases, verb phrases).

4. Semantic Analysis

Understanding the meaning of sentences or phrases.

• Word Sense Disambiguation (WSD): Determining the correct meaning of a word

based on context.
• Semantic Role Labelling (SRL): Identifying the roles played by entities in a sentence.
Example: "John gave Mary a book." → John (giver), Mary (recipient), book (object).

5. Pragmatic Analysis

Interpreting language based on context and external knowledge.

• Understanding implied meanings, tone, or intent (e.g., sarcasm, politeness).

• Resolving ambiguities or understanding reference (e.g., coreference resolution).

6. Discourse Analysis

Studying text at a multi-sentence or document level to understand context and coherence.

• Identifying relationships between sentences (e.g., cause-effect, elaboration).

• Example: Summarizing a document or extracting key themes.
7. Text Representation

Converting processed text into numerical formats for machine learning.

• Bag of Words (BoW): Representing text as word frequency counts.

• TF-IDF (Term Frequency-Inverse Document Frequency): Weighing words by
importance in the document.
• Word Embeddings: Representing words in vector spaces (e.g., Word2Vec, GloVe).
• Contextual Embeddings: Using models like BERT or GPT to capture word meanings
in context.

Challenges of NLP:
Ambiguity and Polysemy
One of the fundamental challenges in NLP is dealing with the ambiguity and polysemy inherent
in natural language. Words often have multiple meanings depending on context, making it
challenging for NLP systems to accurately interpret and understand text.
Data Sparsity and Quality
NLP models require large amounts of annotated data for training, but obtaining high-quality
labeled data can be challenging. Furthermore, data sparsity and inconsistency pose significant
hurdles in building robust NLP systems, leading to suboptimal performance in real-world
applications.
Context and Understanding
Understanding context is crucial for NLP tasks such as sentiment analysis, summarization, and
language translation. However, capturing and representing context accurately remains a
challenging task, especially in complex linguistic environments.
Multilingualism and Language Variations
NLP systems must be able to handle multiple languages and dialects to cater to diverse user
populations. However, language variations, slang, and dialectical differences pose challenges
in developing universal NLP solutions that work effectively across different linguistic contexts.
Lack of Domain-Specific Data
Many Natural Language Processing applications require domain-specific knowledge and
terminology, but obtaining labeled data for specialized domains can be difficult. This lack of
domain-specific data limits the performance of NLP systems in specialized domains such as
healthcare, legal, and finance.
Semantic Understanding and Reasoning
NLP systems often struggle with semantic understanding and reasoning, especially in tasks that
require inferencing or common sense reasoning. Capturing the subtle nuances of human
language and making accurate logical deductions remain significant challenges in NLP
research.
Handling Noise and Uncertainty
Natural language data is often noisy and ambiguous, containing errors, misspellings, and
grammatical inconsistencies. NLP systems must be robust enough to handle such noise and
uncertainty while maintaining accuracy and reliability in their outputs.
Ethical and Bias Concerns
NLP models can inadvertently perpetuate biases present in the training data, leading to unfair
or discriminatory outcomes. Addressing ethical concerns and mitigating biases in NLP systems
is crucial to ensuring fairness and equity in their applications.
Scalability and Performance
Scalability is a critical challenge in NLP, particularly with the increasing complexity and size
of language models. Building scalable NLP solutions that can handle large datasets and
complex computations while maintaining high performance remains a daunting task.
Interdisciplinary Collaboration
NLP research requires collaboration across multiple disciplines, including linguistics,
computer science, cognitive psychology, and domain-specific expertise. Bridging the gap
between these disciplines and fostering interdisciplinary collaboration is essential for
advancing the field of NLP and addressing NLP challenges effectively.

In short:
Ambiguity: Lexical, syntactic, semantic, and pragmatic ambiguities complicate language
understanding.
Language Diversity: Handling multiple languages, dialects, and code-switching.
Context Understanding: Difficulty in resolving pronouns, references, and discourse
relations.
Figurative Language: Challenges with metaphors, idioms, and sarcasm.
Domain-Specific Knowledge: Poor performance without fine-tuning for specialized areas.
Low-Resource Languages: Insufficient datasets for many languages.
Sentiment Detection: Interpreting sarcasm, mixed emotions, and nuanced sentiments.
Resource Intensity: Large computational requirements for training and deploying models.
 Bias and Ethics: Addressing biases in data, privacy concerns, and misuse of models.
Speech-Text Integration: Challenges with accents, dialects, and noisy environments.
Evaluation Metrics: Lack of universally effective methods for assessing models.
Language Evolution: Keeping models updated with new words and trends

https://www.jellyfishtechnologies.com/natural-language-processing-challenges-and-
applications/

Applications of NLP:
1. Chatbots: Chatbots are AI-powered programs designed to interact with humans or other
machines. They help improve user experiences on websites by providing instant responses.
The first chatbot, ELIZA, was created in 1966 to simulate human conversations. Today,
chatbots are widely used, even for psychological support.
2. Text Classification: Text classification helps categorize and organize unstructured text data
efficiently. It reduces human effort in sorting texts, like emails or customer reviews, by
using machine learning and deep learning models. Techniques like CNN and RNN improve
accuracy, and applications range from spam filtering to brand monitoring.
3. Sentiment Analysis: This technique analyzes people's emotions towards products, movies,
or brands. It often uses the "Bag of Words" method, where words are grouped based on their
importance, ignoring word order. Businesses use sentiment analysis to understand customer
feedback and improve their services.
4. Machine Translation: This AI-powered tool translates text between languages, making
communication easier. Early systems had limited vocabulary, but modern neural networks
have greatly improved translation quality. Google Translate now supports over 100
languages and can even translate text from images.
5. Virtual Assistants: These AI-powered helpers, like Siri and Alexa, perform tasks based on
voice commands, such as setting alarms or playing music. They use NLP and deep learning
to understand human speech and improve interactions. Over time, they learn user
preferences and can handle more complex tasks.
6. Speech Recognition: NLP can be used to recognize speech and convert it into text. This can be
used for applications such as voice assistants, dictation software, and speech-to-text transcription.
7. Text Summarization: NLP can be used to summarize large volumes of text into a shorter, more
manageable format. This can be useful for applications such as news articles, academic papers,
and legal documents.
8. Named Entity Recognition: NLP can be used to identify and classify named entities, such as
people, organizations, and locations. This can be used for applications such as search engines,
chatbots, and recommendation systems.
9. Question Answering: NLP can be used to automatically answer questions posed in natural
language. This can be used for applications such as customer service, chatbots, and search engines.
10. Language Modeling: NLP can be used to build models of natural language that can generate new
text. This can be used for applications such as chatbots, virtual assistants, and creative writing.
In Short:
Machine Translation: Translating text between languages (e.g., Google Translate).
Sentiment Analysis: Analysing opinions and emotions in text (e.g., social media
monitoring).
Chatbots and Virtual Assistants: Conversational AI for customer support (e.g., Siri, Alexa).
Text Summarization: Generating concise summaries from long documents.
Speech Recognition: Converting spoken words into text (e.g., dictation software).
Text-to-Speech (TTS): Generating human-like speech from text.
Information Retrieval: Finding relevant information (e.g., search engines).
Named Entity Recognition (NER): Identifying entities like names, dates, and locations in
text.
Spam Filtering: Detecting and blocking unwanted or malicious messages (e.g., in emails).
Recommendation Systems: Personalizing suggestions based on user preferences (e.g.,
Netflix).
Sentiment and Opinion Mining: Understanding public opinion for market research.
Document Analysis: Automating legal and medical document processing.
Language Modelling: Enhancing applications like autocomplete and text generation.
Approaches of NLP:
• Rule based
• Data Based
• Knowledge Based approaches

Rule-based approach in NLP

Rule-based approach is one of the oldest NLP methods in which predefined linguistic rules
are used to analyse and process textual data. Rule-based approach involves applying a
particular set of rules or patterns to capture specific structures, extract information, or
perform tasks such as text classification and so on. Some common rule-based techniques
include regular expressions and pattern matches.

Steps in Rule-based approach in NLP:

1. Rule Creation: Based on the desired tasks, domain-specific linguistic rules are created
such as grammar rules, syntax patterns, semantic rules or regular expressions.
2. Rule Application: The predefined rules are applied to the inputted data to capture
matched patterns.
3. Rule Processing: The text data is processed in accordance with the results of the matched
rules to extract information, make decisions or other tasks.
4. Rule refinement: The created rules are iteratively refined by repetitive processing to
improve accuracy and performance. Based on previous feedback, the rules are modified
and updated when needed.

Advantages of the Rule-based approach:

• Easily interpretable as rules are explicitly defined
• Rule-based techniques can help semi-automatically annotate some data in domains where
you don’t have annotated data (for example, NER(Named Entity Recognization) tasks in
a particular domain).
• Functions even with scant or poor training data
• Computation time is fast and it offers high precision
• Many times, deterministic solutions to various issues, such as tokenization, sentence
breaking, or morphology, can be achieved through rules (at least in some languages).
Disadvantages of the Rule-based approach:
• Labor-intensive as more rules are needed to generalize
• Generating rules for complex tasks is time-consuming
• Needs regular maintenance
• May not perform well in handling variations and exceptions in language usage
• May not have a high recall metric

Applications

1. Spell Checkers: Detecting misspelled words using predefined dictionaries and

phonetic rules.
2. Grammar Correction: Correcting grammatical errors based on syntactic rules.
3. Information Extraction: Extracting structured data (e.g., dates, phone numbers) from
unstructured text using regex.
4. Chatbots (Early Systems): Responding to user queries with hardcoded rules.

Data-Based Approach in NLP

The data-based approach in NLP focuses on using data-driven methods, particularly machine
learning (ML) and deep learning, to analyze and process natural language. Instead of relying
on manually crafted rules, this approach learns patterns and features directly from large
datasets.

The data-based approach typically follows these steps:

1. Data Collection:
o Gather raw text data from sources like social media, articles, or chat logs.
2. Data pre-processing:
 o Cleaning the text (e.g., removing stop words, special characters).
o Tokenization (splitting text into words or phrases).
o Encoding text into numerical formats (e.g., word embeddings).
3. Model Selection:
o Choose a machine learning or deep learning algorithm.
o Examples: Logistic Regression, Support Vector Machines (SVM), Neural
Networks.
4. Training:
o Train the model on labelled datasets (supervised learning) or unlabelled datasets
(unsupervised learning).
5. Evaluation:
o Test the model’s performance on unseen data using metrics like accuracy, F1
score, or BLEU score.
6. Prediction:
o Deploy the trained model to process and analyse new text.

Types of Data-Based Approaches

1. Statistical Methods:
o Use probabilities and statistical models to predict text patterns.
o Examples: Naive Bayes, N-grams, Hidden Markov Models (HMM).
2. Machine Learning:
o Algorithms learn patterns from labeled data.
o Examples: Decision Trees, SVMs, Random Forest.
3. Deep Learning:
o Neural networks with multiple layers model complex language patterns.
o Examples: Recurrent Neural Networks (RNNs), Transformers (BERT, GPT).

Advantages

• Scalability: Handles large-scale tasks with diverse data.

• Adaptability: Generalizes well to new data.
• Automated Feature Learning: Reduces the need for manual feature engineering.
• Performance: Delivers state-of-the-art results in many NLP tasks.
Challenges

• Data Dependency: Requires large, high-quality datasets.

• Resource Intensive: Needs significant computational power for training.
• Bias: Models can inherit biases present in the data.
• Interpretability: Complex models (e.g., deep learning) are often hard to explain.

Examples

Sentiment Analysis: Classifying text as positive, negative, or neutral.

1. Machine Translation: Translating text between languages using models like Google
Translate.
2. Named Entity Recognition (NER): Identifying entities such as names, dates, and
locations.

Knowledge-Based Approach in NLP

The knowledge-based approach in NLP utilizes structured knowledge sources such as

ontologies, knowledge graphs, dictionaries, and rule-based systems to process and understand
natural language. Unlike data-based approaches, it focuses on incorporating pre-existing
knowledge about language and the world to derive insights from text.

1. Input Text Processing

• The input is raw text, such as sentences, paragraphs, or queries.

• Techniques Used:
o Tokenization: Breaks text into words or phrases.
o POS Tagging: Identifies parts of speech (e.g., noun, verb).
o Syntactic Parsing: Analyzes grammatical structure (e.g., subject-verb-object).

Example:
Input: "What is the capital of France?"
Processed Tokens: ["What", "is", "the", "capital", "of", "France"]
2. Entity Recognition and Linking

• Identifies named entities in the text and links them to corresponding entities in a
knowledge base.
• Techniques Used:
o Named Entity Recognition (NER): Identifies entities such as "France."
o Entity Linking: Maps "France" to a specific entity in a knowledge base (e.g.,
France -> Q142 in Wikidata).

Example:
Recognized Entity: France
Linked Entity: France -> Knowledge Base _ID: Q142

3. Semantic Representation

• Converts natural language into a structured representation using knowledge bases.

• Techniques Used:
o Ontologies: Represent relationships between concepts.
o Knowledge Graphs: Represent entities and their relationships as graphs.
o Semantic Triples: Subject-Predicate-Object format (e.g., Paris -> isCapitalOf -
> France).

Example:
Query: "What is the capital of France?"
Representation: ?x -> isCapitalOf -> France

4. Reasoning and Inference

• Applies rules and logical reasoning to the structured representation to derive answers
or insights.
• Techniques Used:
o Logical Rules: Use predefined if-then rules (e.g., If x -> isCapitalOf -> France,
then x = Paris).
o Ontological Reasoning: Leverages hierarchies and relationships (e.g., France is
a country in Europe).
Example:
Reasoning Rule: isCapitalOf (France, ?x)
Result: x = Paris

5. Output Generation

• Provides a meaningful response or insight based on the reasoning process.

• Techniques Used:
o Natural Language Generation (NLG): Converts structured data into human-
readable text.
o Direct Display: Outputs structured data (e.g., a graph or table).

Example:
Output: "The capital of France is Paris."

Advantages of Knowledge-Based Approach in NLP

1. High Accuracy: Provides reliable results in well-defined domains.

2. Transparency: Outputs and reasoning are interpretable and explainable.
3. No Large Data Requirement: Works well without needing extensive labeled datasets.
4. Domain Expertise: Easily incorporates domain-specific knowledge.

Disadvantages of Knowledge-Based Approach in NLP

1. Scalability Issues: Hard to cover all language variations and contexts.

2. Maintenance Effort: Knowledge bases require regular updates to remain relevant.
3. Limited Ambiguity Handling: Struggles with informal or context-dependent
language.
4. Resource-Intensive: Building and curating knowledge bases is time-consuming and
costly.