Scribd
Upload
5 views3 pages

Gitika Mandal BE4 A 17 NLP EXP1

The document outlines the importance of text preprocessing techniques in Natural Language Processing (NLP), specifically focusing on tokenization, filtration, and script validation. It describes various types of tokenization, including word, subword, sentence, and character tokenization, and their applications in tasks like sentiment analysis and machine translation. Additionally, it discusses the significance of filtering unwanted text elements and ensuring script validation for accurate language processing and security.

Uploaded by

Gitika Mandal
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
5 views3 pages

Gitika Mandal BE4 A 17 NLP EXP1

The document outlines the importance of text preprocessing techniques in Natural Language Processing (NLP), specifically focusing on tokenization, filtration, and script validation. It describes various types of tokenization, including word, subword, sentence, and character tokenization, and their applications in tasks like sentiment analysis and machine translation. Additionally, it discusses the significance of filtering unwanted text elements and ensuring script validation for accurate language processing and security.

Uploaded by

Gitika Mandal
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 3

Experiment:01 Name: Gitika Mandal

PRN: 21UF16885CM030
Roll No: 17
Batch: A
AIM: Apply various text preprocessing techniques for any given text :
Tokenization and Filtration & Script Validation.
INPUT/OUTPUT:

WRITE UP:
1. What is Meant by Word Tokenization?
Tokenization is a fundamental process in Natural Language Processing (NLP) that involves
breaking down a text into smaller units, known as tokens. These tokens can be words, phrases,
subwords, or even characters, depending on the level of granularity required. Tokenization
serves as a preliminary step in text processing and is crucial for various NLP applications such
as text analysis, machine translation, and information retrieval.
There are two primary types of tokenization:
● Word Tokenization: This involves splitting a sentence or text into individual words. For
instance, the sentence "NLP is fascinating." would be tokenized into ["NLP", "is",
"fascinating", "."].
● Subword Tokenization: This breaks words into smaller meaningful units, which is useful
for handling complex words, rare words, and morphological variations. An example is
theuse of Byte-Pair Encoding (BPE) in transformer-based models like BERT, where the
word “unhappiness” may be split into ["un", "happiness"].
● Sentence Tokenization: This divides a text into separate sentences. For instance, the text
"Hello! How are you? I’m fine." would be tokenized into ["Hello!", "How are you?", "I’m
fine."].
● Character Tokenization: This involves splitting a word into individual characters, useful
for languages without clear word boundaries, such as Chinese or Japanese.
Tokenization is more complex than simple splitting based on spaces or punctuation. Many
languages have unique challenges:
In Chinese, Japanese, and Thai, words are not always separated by spaces, requiring specialized
algorithms like Jieba or MeCab. In German, compound words like
"Donaudampfschifffahrtsgesellschaftskapitän" need to be split carefully. In Arabic,
tokenization must handle root-based morphology, where words undergo inflectional changes.
Modern NLP models often use subword tokenization methods such as WordPiece (used in
BERT), SentencePiece (used in T5 and ALBERT), or Unigram Language Model (used in
XLNet) to handle rare words effectively. By breaking words into smaller meaningful parts,
these models achieve better generalization in handling vocabulary.
2.What Are the Uses of Tokenization, Filterization, and Script Validation?
Uses of Tokenization
Tokenization is a crucial step in various NLP tasks, including:
● Text Preprocessing: Before applying machine learning algorithms, text needs to be
converted into a structured format. Tokenization enables this by breaking text into
analyzable units.
● Sentiment Analysis: Tokenizing reviews or social media posts helps identify individual
words or phrases that contribute to sentiment classification.
● Machine Translation: Tokenization allows translation models to handle words efficiently,
especially in morphologically rich languages.
● Speech Recognition: Tokenized words or subwords are mapped to phonemes, improving
transcription accuracy.
● Search Engines: Search algorithms tokenize user queries to match relevant documents in
databases.
● Chatbots and Virtual Assistants: Tokenization helps break down user input into
understandable components for response generation.
● Named Entity Recognition (NER): Tokenization assists in identifying names, locations,
and organizations from unstructured text.

Uses of Filterization
Filterization (or filtering) refers to the process of removing unwanted words, characters, or
symbols from a text. It enhances the quality of text processing by eliminating irrelevant data.
Some common applications include:
● Stopword Removal: In search engines and NLP models, frequent but unimportant words
like “the,” “is,” and “of” are removed to improve efficiency.
● Noise Reduction in Data Processing: Text documents often contain unnecessary
punctuation, special characters, or repeated words that need filtering.
● Spam Detection: Filtering out spam keywords helps classify emails or messages as spam
or non-spam.
● Normalization: Inconsistent text formatting, such as varied capitalization or redundant
whitespace, can be filtered to improve text uniformity.
● Profanity Filtering: Social media platforms use filtering techniques to remove offensive
words from user-generated content.

Uses of Script Validation


Script validation ensures that a text adheres to predefined language constraints and is written
in the correct script. This is useful in multiple contexts:
Multilingual Text Processing: Validating the script of input text helps in language
identification and ensures that an NLP system processes the text correctly.
Security and Fraud Prevention: Many cyberattacks involve script-based manipulations, such
as homoglyph attacks (where similar-looking characters from different scripts are used to
deceive users). Script validation helps detect such attempts.
OCR (Optical Character Recognition) Systems: When digitizing printed or handwritten text,
script validation ensures that the extracted characters belong to the intended language.
Text Standardization: Ensuring that user-generated content follows a single script helps in
document consistency and readability.
AI Model Training: Script validation prevents models from learning from incorrectly labeled
data, improving accuracy in NLP applications.

You might also like

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy