Introduction to Computational Linguistics

Assoc. Prof. Dr. Nguyen Ngoc Vu

30/11/23 1
What is Computational Linguistics
• Computational Linguistics Evolution:
Transition from academic theory to forefront of
tech innovation.
• TARI Chatbot by HUFLIT University: A
2023 milestone, combining OpenAI's
capabilities for dynamic, contextually relevant
interactions.
• Bridge Concept: Links human language's
intricacies with computational efficiency.
• Challenges and Potential: Enormous
complexity in modeling human language for
computational processing.

2
Central Pursuit of Computational Linguistics
• Understanding Human Language: Analysis and
extraction of meaning, identifying language
components, and interpreting complex structures
like idioms and metaphors.
• Sentiment Detection: Discerning overall tone
(positive, negative, neutral) from text complexities.
• Interpreting Abstract Ideas: Grasping non-
physical concepts (e.g., freedom, love, justice)
beyond quantifiable definitions.
• Generating Human-like Text: Creating natural,
engaging text that mirrors human language, from
chatbot responses to detailed articles.

3
Levels of Language
Comprehension

4
Speech Recognition
• Evolution: From rudimentary systems to
advanced technology recognizing diverse accents
and languages.
• Modern Examples: Siri, Alexa, Google Assistant
revolutionizing human-digital device interaction.
• Principles: Algorithms mapping speech sounds to
written words using complex mathematical
models.
• Applications: Virtual assistants, healthcare
transcription, educational tools, hands-free
navigation in transportation.
• Accessibility Impact: Enhanced ease of use for
individuals with visual impairments or motor
difficulties.

30/11/23 5
Syntactic Parsing
• Syntax: Study of sentence structure rules, varying
significantly across languages (e.g., SVO in
English vs. SOV in Japanese).
• Syntactic Parsing Process: Identifying
grammatical categories of words, determining
syntactic roles, and establishing inter-word
relationships.
• Parsing Algorithms: Either rule-based
(following pre-defined grammatical rules) or
statistical (learning from large text datasets).
• Importance in NLP: Critical for tasks like
machine translation and information extraction,
enabling understanding of sentence meaning.

30/11/23 6
Semantic Understanding
• Definition: Study of meaning in language,
covering literal definitions, metaphors,
connotations, and cultural references.
• Role in Language: Relates words to objects and
ideas, decodes signs and symbols to construct
shared understanding.
• Computational Linguistics Approach:
Designing algorithms for meaningful human
language interpretation, considering context.
• Applications: Crucial for machine translation,
information extraction, text summarization, and
dialog systems like Siri, Alexa, Google Assistant.

30/11/23 7
Discourse and Contextual Interpretation
• Discourse Analysis: Examines language beyond the sentence
level, analyzing choices in expression and their contribution to
overall meaning.
• Contextual Interpretation: Understanding how immediate
and broader contexts (situational, social, cultural) influence
language interpretation.
• Algorithm Design: Development of algorithms to comprehend
the broader context, including speaker’s intent and dialogue
history.
• Approaches: Use of rule-based systems, statistical methods,
and deep learning techniques like recurrent neural networks
(RNNs).
• Applications: Vital for machine translation, information
extraction, text summarization, and dialog systems for accurate
and context-aware responses.

30/11/23 8
History of Computational
Linguistics

9
Early Developent

10
Birth of an idea
• Beginnings: Emerged in the mid-20th century
with the rise of mainframe computers.
• 1950s Technology Advancements: Digital
computers opened new opportunities in various
domains, including linguistics.
• Initial Focus: Machine translation, translating
text from one language to another automatically.
• Early Challenges: Language complexity, context-
dependence, and cultural influences.
• Significance: Marked the beginning of
computational linguistics, aiming to create
machines understanding human language.

30/11/23 11
 Geopolitical Climate and the Need for
Machine Translation
Background of Machine Translation's Inception
United States and Soviet Union's dominance post-war
Intense ideological rivalry impacting scientific research.
Surge in Scientific Literature and Translation Challenges
Significant advancements in science and technology by the U.S.
and Soviet Union.
Increase in technical literature, especially in Russian.
Need for Efficient Translation Solutions
Inadequacy of human translators for the volume of documents.
Necessity for quick translation to maintain competitive edge.
U.S. Government's Role and Strategic Importance
Funded research into machine translation.
Aimed to maintain national security and technological superiority
during the Cold War.

30/11/23 12
 Early Approaches to Machine Translation
Early Machine Translation Methods
Initially based on direct word-for-word substitution using dictionaries.
Treated translation as a direct mapping of words from one language to
another.
Limitations and Challenges
The approach failed due to language complexity and intricacies.
Translations often had incorrect grammar and awkward phrasing.
Literal translation of idiomatic expressions led to nonsensical results.
Beyond Simple Word Substitution
Recognized need for understanding syntax, semantics, and pragmatics.
Syntax: Understanding grammatical relationships in sentences.
Semantics: Comprehending meanings of words and sentence meanings.
Pragmatics: Considering context, cultural nuances, and speaker intentions.
Significance in Machine Translation Evolution
Marked a shift from basic substitution to comprehensive language
understanding.
Acknowledged complexities and computational challenges of natural
languages.

30/11/23 13
 The ALPAC Report and Its Impact
ALPAC Report's Impact on Machine Translation
Late 1960s: High hopes for machine translation.
ALPAC Committee's critical 1966 report marked a pivotal moment.
Report's Findings and Consequences
Machine translation was slow, less accurate, and costlier than human
translation.
Cast doubt on the future of machine translation.
Led to reduced U.S. government funding, contributing to the "AI winter."
Silver Lining and New Directions
The report recognized computers' potential in text processing and
information retrieval.
Laid the foundation for search engine development.
Shifted focus from translation to other areas of computational linguistics.
Broader Implications for Scientific Progress
Serves as a reminder of the continuous evolution of scientific inquiry and
technology.

30/11/23 14
 Chomsky's Theories and Their Influence
Chomsky's Transformational-Generative Grammar
Introduced the concept of 'universal grammar' underlying all languages.
Challenged the structuralist view, suggesting an innate language capacity.
Impact on Linguistics and Language Acquisition
Shifted focus to syntax and abstract language rules.
Opened new avenues in understanding language acquisition and variation.
Influence on Computational Linguistics
Offered a theoretical framework for language modeling in computers.
Shifted focus from practical applications to theoretical language studies.
Evolution in Computational Linguistics
Transition from a practical discipline to a theoretically grounded science.
Focus on understanding complex mechanisms of human language.
Greater emphasis on developing models and algorithms for language
processing.

30/11/23 15
 The Birth of Corpora and Corpus Linguistics
Pivotal Role of 1960s and 70s in Computational Linguistics
Emergence of the concept of linguistic corpora.
Focus on real-world language data in linguistic research.
Emergence of Corpus Linguistics
New subfield focusing on language study within corpora.
Bridged theoretical linguistics with computational applications.
Shift to Empirical, Data-Driven Approach
Marked a new direction in computational linguistics.
Underpinned development of accurate computational models.
Integration of Machine Learning
Utilization of machine learning to analyze large datasets.
Enabled the training of sophisticated language models.
New Era in Computational Linguistics
Emphasis on empirical research using authentic language data.
Highlighted the synergy between linguistics and machine learning.

30/11/23 16
Modern Advances

30/11/23 17
 The Evolution of Machine Learning
Growth in Computational Power in Late 20th and Early 21st Century
Significant increase in computational capabilities.
Enabled advancements in artificial intelligence and machine learning.
DeepMind's AlphaGo Achievement in 2016
Defeated Go grandmaster Lee Sedol.
Showcased rapid evolution of machine learning techniques.
Demonstrated parallels between game strategies and computational linguistics.
Machine Learning Development
Algorithms learn and improve from experience.
Diverges from traditional programming.
Suited for complex tasks like natural language processing.
Application in Natural Language Processing
Shift towards data-driven computational linguistics in the 1990s and 2000s.
Use of decision trees, support vector machines, naïve Bayes classifiers.
Improvements in Language Processing Tasks
Enhanced accuracy in part-of-speech tagging and named entity recognition.
Significant advancements in text classification.
Automated systems can now classify text based on content

30/11/23 18
 The Rise of Neural Networks
Rise of Neural Networks in Computational Linguistics (Mid-2010s)
Neural networks inspired by the human brain's structure.
Consist of interconnected nodes processing and passing data.
Breakthroughs with RNNs and LSTMs
Introduced ability to process sequential data.
Internal memory allows influence of past inputs on current output.
Suited for language processing due to the significance of sequence in
language.
Overcoming Limitations of RNNs
RNNs faced the "vanishing gradients" issue in long sequences.
LSTMs introduced mechanisms for selective memory retention.
Advancements in Computational Linguistics Applications
Machine translation improved with better context understanding.
Text generation achieved syntactic correctness and semantic coherence.
Speech recognition enhanced by sequential processing capabilities

30/11/23 19
 Deep Learning and Language Processing
Emergence of Transformer Models (Late 2010s)
Introduction of a new neural network architecture.
Advanced beyond RNNs and LSTMs in language comprehension and
generation.
Breakthroughs with BERT and GPT-3
Google's BERT: Enhanced understanding of context in language.
OpenAI's GPT-3: One of the largest language models with 175 billion
parameters.
Attention Mechanisms in Transformer Models
Allows the model to focus on relevant parts of data.
Enables understanding of context and relationships in language.
BERT's Bidirectional Processing
Processes text in both directions for full context comprehension.
Significant improvements in various natural language processing
tasks.
GPT-3's Few-Shot Learning Capability
Ability to perform tasks without extensive training.
Represents a step towards general artificial intelligence

30/11/23 20
 Looking to the future
Continued Progress in Computational Linguistics
Rapid evolution of machine learning models.
Enhanced natural language understanding and generation.
Abundance of Textual Data
Daily production of vast textual data feeds machine learning growth.
Development of Multimodal Models
Ability to process text, images, and sound.
Applications in image captioning, visual question answering, and art creation.
Advancements in Transfer Learning
Training models on one task, then fine-tuning for related tasks.
Increases efficiency and versatility of machine learning models.
Ethical Challenges and Transparency
Need for guidelines on ethical use and data security.
Addressing the 'black box' problem for understanding model decisions.
Ensuring fairness and unbiased outputs.
Accessibility of Computational Linguistics Technologies
Making technologies affordable and understandable for all.
Bridging the gap between technical expertise and general usability

30/11/23 21
 Core Techniques

30/11/23 22
 Parsing
The Underpinnings of Parsing
Uses algorithms for syntactic analysis in sentences.
Various parser types like top-down, bottom-up, and probabilistic parsers.
Syntactic Analysis and Parse Trees
Identifies word functions and constructs parse trees.
Parses sentences based on language grammar rules.
Disambiguation and Grammatical Relationships
Resolves ambiguities in sentence structure.
Understands relationships like subject-object and modifier-head.
Applications in Information Extraction and Text-to-Speech
Crucial for identifying key information in texts.
Assists in creating natural-sounding speech in text-to-speech systems.
Parsing in Sentiment Analysis and Search Engines
Helps identify sentiments in texts for digital marketing and social media.
Evolving search engines to understand grammatical structure of queries

30/11/23 23
 Parts of Speech Tagging
Part-of-Speech Tagging (POS) in Computational Linguistics
Identifies and categorizes words based on grammatical roles.
Foundational to parsing, setting the stage for syntactic analysis.
Roles and Functions of Words
Classifies words as nouns, verbs, adjectives, adverbs, etc.
Provides fundamental linguistic information for sentence structure.
Resolving Ambiguities in Word Functions
Essential for disambiguating words with multiple part-of-speech roles.
Employs rule-based, statistical, and machine learning methods.
Beyond Parsing: Wider Applications
Vital in Named Entity Recognition (NER) for identifying specific entities.
Enhances clarity in sentiment analysis, machine translation, text summarization.
POS Tagging in NLP Ecosystem
Foundational role across various sub-domains of NLP.
Facilitates understanding of syntactic roles in sentiment analysis.
Aids in ensuring contextual accuracy in machine translation

30/11/23 24
 Named Entity Recognition
Named Entity Recognition (NER) in Computational Linguistics
Identifies and categorizes specific data points like names, places, and
dates.
Fundamental in information extraction from unstructured text.
Targeting Identification and Categorization of Named Entities
Classifies entities into pre-defined categories.
Identifies concrete or abstract elements in text that can be named.
NER's Process and Steps
Involves identifying entity boundaries and determining their types.
Fundamental for organizing unstructured text data.
NER in Language Processing and Applications
Core of information extraction systems.
Important in machine translation and information retrieval.
News aggregation: Identifies key players, locations, and themes.
Event extraction: Isolates important figures and places.
Customer service: Focuses on specific issues and products.
Healthcare: Identifies medical codes, drug names, and conditions.
Legal domain: Distinguishes legal codes and relevant entities

30/11/23 25
Machine Learning Algorithms
Machine Learning in Computational Linguistics
Key to understanding and processing natural language.
Empowers systems to learn from data, identify patterns, and
make decisions.
Transformative Impact of Machine Learning Algorithms
Facilitates a range of tasks like parsing, POS tagging, NER, and
semantic analysis.
Drives chatbots, virtual assistants, and dialogue systems.
Future of Machine Learning and Computational Linguistics
Combines language understanding and AI for sophisticated
communication systems.
Uses supervised learning for spam detection, sentiment
analysis, and more.
Deep learning for complex tasks like language translation and
speech recognition.

30/11/23 26
 Statistical Models
Underpinnings in Mathematical Constructs
Rooted in statistical models, offering structured data interpretation.
Shift from algorithmic learning to mathematical reasoning.
Statistical Models at Computational Linguistics' Core
Translate raw data into meaningful linguistic analysis.
Make probabilistic decisions based on data.
Applications of Statistical Models
Language modeling: Calculate probability of word sequences.
Machine translation: Improved quality through bilingual text learning.
Speech recognition: Decoding acoustic signals into words.
Advancements and Future of Statistical Models
Integral to machine translation, speech recognition, information extraction.
Expected to deliver more nuanced predictions with growing data.
Complemented by deep learning techniques for complex task

30/11/23 27
Areas of Computational
Linguistics

30/11/23 28
Phonetics
Computational Phonetics: Intersection with Technology
Involves modeling and analyzing speech sounds computationally.
Used in automatic speech recognition (ASR) and speech synthesis
systems.
Importance in Speech Recognition and Synthesis
Extracts features from speech, categorizes them phonetically.
Generates human-like speech from text, focusing on articulation,
acoustics, and prosody.
The Future of Phonetics in Computational Linguistics
Expected to evolve with more sophisticated models and techniques.
Potential applications in virtual avatars, hearing aids, and language
learning.
Challenges and Opportunities
Navigating speech's complexity and diversity.
Balancing data privacy and bias concerns.

30/11/23 29
 Phonology
Phonology in Computational Linguistics
Explores abstract, systemic aspects of sounds within languages.
Critical for understanding rules and patterns in sound use and
distribution.
Phonology in Real-World Contexts
Influential in language acquisition, second language learning, and
sociolinguistics.
Provides insights into phonological processes and variations.
Computational Phonology and Technology Integration
Models phonological principles with computer algorithms.
Enhances transcription accuracy in ASR systems.
Aids in the segmentation and analysis of spoken language in NLP.
Future Trends in Computational Phonology
Deep learning and big data analytics offer new modeling
capabilities.
Potential for more nuanced representations of language sounds.

30/11/23 30
 Morphology
The Role of Morphemes
Investigates the smallest meaningful units in language.
Categorizes morphemes into free and bound types.
Morphology in Machine Translation
Facilitates accurate translation by understanding word structure.
Addresses grammatical differences between languages.
Information Retrieval Systems
Improves search accuracy through morphological understanding.
Handles language complexity, including irregular forms and variants.
Text-to-Speech Systems
Employs morphology for accurate pronunciation and natural speech
generation.
Manages stress placement and pronunciation rules.
Languages with Rich Morphology
Essential for handling languages with complex word structures.
Aids in accurate translation and information retrieval in morphologically
rich languages

30/11/23 31
 Syntax
Syntactic Analysis in Linguistics
Studies rules for constructing sentences and relationships between sentence
parts.
Involves phrase structure and dependencies within sentences.
Parsing: A Key Application in Computational Linguistics
Dissects sentences into constituent elements, determining syntactic roles.
Utilizes parsers to construct parse trees, revealing sentence structure.
Types of Parsers
Constituency-based parsers focus on phrase structure.
Dependency-based parsers emphasize relationships between individual words.
Importance in Machine Translation
Parsing ensures grammatically coherent translations.
Crucial for understanding grammatical structures of source and target languages.
Syntactic Parsing in Information Extraction
Extracts structured information from unstructured text.
Enhances understanding of context and relationships within sentences.
Practical Applications of Parsing
Fundamental in machine translation and information extraction.
Helps translate text coherently and extract relevant information accurately

30/11/23 32
 Semantics
Semantics in Computational Linguistics
Explores meanings of words, phrases, and sentences.
Focuses on denotations, connotations, and relationships between words.
Semantic Analysis in Language
Decodes meaning, considering context and abstract subtleties in sentences.
Utilizes formal systems for interpreting semantic representations.
Role in Machine Translation
Requires understanding the semantic content of source text.
Aims for translations that are accurate and idiomatic.
Semantic Analysis in Information Extraction
Identifies and extracts specific information from text data.
Enhances accuracy and contextual appropriateness of extracted information.
Computational Semantics Tools
Employs computational algorithms for complex semantic structure analysis.
Bridges human language and machine understanding.
Advancements in Semantic Analysis
Leads to deeper linguistic and cognitive science insights.
Improves machine translation and information extraction technologies

30/11/23 33
 Pragmatics
Pragmatics in Computational Linguistics
Focuses on how context influences language interpretation.
Explores speech acts, implicature, presupposition, and deixis.
Sentiment Analysis
Improves accuracy by understanding conversational tone, cultural norms,
and irony.
Machine Translation
Assists in translating idioms, cultural references, and indirect speech acts.
Captures intent, tone, and nuances for natural translation.
Information Extraction
Identifies relevant information based on speaker intentions and
conversational context.
Application in Computational Linguistics
Vital for natural language understanding, machine translation, sentiment
analysis, and more.
Enables more nuanced and contextually aware language processing systems

30/11/23 34
 Discourse Analysis
Discourse Analysis in Computational Linguistics
Examines how sentences and utterances form complex ideas and
maintain coherence.
Focuses on larger structures of communication and societal/cultural
contexts.
Discourse Analysis in Automatic Summarization
Essential for creating concise yet comprehensive summaries of text.
Involves understanding overall meaning, structure, and key ideas.
Coherence and Key Idea Identification
Analyzes how parts of text logically connect to form a narrative.
Identifies central themes and arguments for effective summarization.
Machine Translation and Discourse Analysis
Ensures translated text maintains coherence and intent of the original.
Focuses on preserving semantic and pragmatic fabric across languages.
Preserving Tone and Style
Recognizes and replicates rhetorical devices and stylistic elements in
translation.

30/11/23 35
Interdisciplinary Connections

30/11/23 36
 Computer Science
Core Techniques from Computer Science
Machine learning: Empowers models to learn from data for
prediction and decision-making.
Natural Language Processing (NLP): Analyzes, understands, and
generates human language.
Data mining: Uncovers patterns and relationships in large volumes
of linguistic data.
Integration of Computer Science and Linguistics
Merges analytical thinking of computer science with empirical
approach of linguistics.
Facilitates investigation of human language from a computational
perspective.
Advancements Triggered by the Collaboration
Significant strides in speech recognition, machine translation, and
information retrieval.
Symbiotic relationship enhances computational linguistics research
and applications

30/11/23 37
 Artificial Intelligence
Artificial Intelligence (AI) in Computational Linguistics
AI provides theoretical and technical frameworks for language model
design and learning algorithms.
Techniques like neural networks, deep learning, and reinforcement
learning enhance language processing tasks.
AI Techniques in Language Processing
Neural networks: Applied in machine translation, sentiment analysis,
and entity recognition.
Deep learning: Understands complex language patterns for various
tasks.
Reinforcement learning: Enhances performance of dialogue systems.
Advancements in Computational Linguistics due to AI
Natural language understanding: Systems can interpret language with
context and subtlety.
Dialogue systems: Chatbots generate more appropriate, contextually
relevant responses.
Language generation: Models like GPT-3 exhibit an ability to generate
human-like text.

30/11/23 38
 Cognitive Science
Cognitive Science in Computational Linguistics
Investigates mental processes in language comprehension, production,
and acquisition.
Aims to understand how humans perceive, process, and represent
language.
Intersection with Computational Linguistics
Enhances computational models to be more refined and human-like.
Draws from understanding human cognition to improve machine
language processing.
Key Areas of Cognitive Science in Language
Perception: Interpreting auditory or visual language signals.
Processing: Understanding grammar, syntax, and speaker/writer
intentions.
Representation: Exploring how language is stored and accessed in the
mind.
Mutual Enhancement of Both Fields
Cognitive science provides empirical data and theoretical frameworks.
Computational linguistics offers models and algorithms to test cognitive
theories.

30/11/23 39
 Data Science
Data Science in Computational Linguistics
Utilizes data mining, statistical analysis, and machine
learning.
Extracts meaningful insights from large, complex linguistic
datasets.
Identifies patterns and trends in language data for
hypothesis testing.
Role in Linguistic Data Processing
Organizes vast amounts of linguistic data, including text and
spoken language transcripts.
Enables quantitative analysis of linguistic phenomena.
Accounts for variability and complexity in language.
Integration with Computational Linguistics
Assists in the creation of computational language models.
Applies data mining for efficient feature extraction from
linguistic datasets.
Uses machine learning algorithms for pattern recognition
and model improvement.

30/11/23 40
 Linguistics
Interconnection of Linguistics and Computational Linguistics:
Linguistics provides a theoretical framework for computational
models.
Computational linguistics relies on linguistic theories and
concepts.
Core Principles of Linguistics in Computational Approaches:
Guides the structure, grammar, semantics, and other language
facets.
Forms the foundation for computational models and algorithms.
Role in Computational Linguistics:
Enables computers to interact with humans using natural
language.
Draws upon linguistic theories to design and build computational
models.
Linguistic methodologies assist in feature extraction and analysis.

30/11/23 41
 Sociolinguistics
Core Tenets of Sociolinguistics
Language variation across regions, social classes, ethnic
groups.
Language change over time and multilingualism.
Impact of societal factors like regional dialects, sociolects,
ethnolects, and genderlects on language.
Sociolinguistics and Human Language Processing
Aims to make computational systems more culturally aware
and reflective of language variations.
Importance in areas like speech recognition and sentiment
analysis for diverse user groups.
Mutual Enrichment of Computational Linguistics and
Sociolinguistics
Computational linguistics benefits from sociolinguistic
insights for more accurate and inclusive technologies.
Computational methods provide new ways to analyze
sociolinguistic data, revealing large-scale language patterns
and trends.

30/11/23 42
 Psycholinguistics
Overview of Psycholinguistics
Investigates psychological processes linked to language
comprehension, production, and acquisition.
Focuses on real-time language processing and representation in
the human mind.
Interplay with Computational Linguistics
Provides insights for developing algorithms that mirror human
language processing.
Psycholinguistic theories inform parsing algorithm design in
computational linguistics.
Research on word recognition guides optimization of spell
checking, disambiguation, and information retrieval algorithms.
Applications in Computational Linguistics
Enhancement of Natural Language Processing (NLP)
applications: machine translation, speech recognition,
information extraction.
Contribution to cognitive modeling – simulating cognitive
processes in computational models.
Allows comparison between computational model behavior and
human performance for refinement and accuracy enhancement

30/11/23 43
 Comparative Linguistics
Language Variation and Change
Utilizes computational methods to analyze historical language data and linguistic evolution.
Tracks changes in vocabulary, grammar, and phonetics across time.
Identifies linguistic innovations reflecting cultural, societal, and technological influences.
Employs machine learning for predicting future language changes.
Language Contact and Bilingualism
Investigates code-switching, language borrowing, and language interference in multilingual
contexts.
Analyzes patterns in bilingual speech, revealing sociolinguistic trends.
Studies cognitive aspects of bilingualism using computational models.
Translation Studies
Revolutionized by computational linguistics, enhancing analysis and efficiency.
Uses translation corpora for detailed comparative studies and pattern identification.
Develops computer-aided translation tools and advances machine translation systems.
Shift from rule-based to statistical and neural machine translation models.
Impact and Applications
Enhances understanding of language interaction and development.
Improves speech recognition, sentiment analysis, and other computational linguistics
applications.
Contributes to more effective language teaching methods and translation services.
Offers insights into sociocultural influences on language evolution and usage

30/11/23 44