EXPERIMENT-1

Aim:- Implement various term weighting schemes (e.g., TF-IDF, binary weighting,
logarithmic weighting) and evaluate their impact on retrieval performance.

Theory:-
Implementing various term weighting schemes and evaluating their impact on retrieval performance
involves several steps. Here's a high-level overview of how you can approach this task:

1. Data Preparation: You'll need a dataset to work with, typically a collection of documents. Ensure
the data is preprocessed, including steps like tokenization, lowercasing, removing stop words, and
stemming/lemmatization.

2. Term Frequency (TF): Calculate the frequency of each term in each document. This is the
simplest form of term weighting.

3. Inverse Document Frequency (IDF): Calculate the IDF of each term, which measures how
important a term is across the entire corpus. IDF is calculated as the logarithm of the total number
of documents divided by the number of documents containing the term.

4. TF-IDF: Multiply the TF of each term in a document by its IDF to get the TF-IDF weight. This
weight reflects both the importance of the term in the document and its rarity in the corpus.

5. Binary Weighting: Assign a binary weight of 1 to each term in a document if it is present, and 0
if it is absent.

6. Logarithmic Weighting: Apply a logarithmic function to the term frequency to diminish the
impact of very frequent terms.

7. Retrieval Model: Implement a retrieval model such as Vector Space Model (VSM) or Okapi
BM25, which utilizes the term weights to rank documents for a given query.

8. Evaluation Metrics: Choose appropriate evaluation metrics such as precision, recall, and F1-
score to evaluate the retrieval performance of each weighting scheme.

9. Experimentation and Analysis: Run experiments using different weighting schemes and
evaluate their impact on retrieval performance using the chosen metrics. Compare the results to
identify which weighting scheme performs better in terms of retrieval effectiveness.

10. Visualization (Optional): Visualize the results using graphs or charts to make it easier to
interpret and compare the performance of different weighting schemes.

Implementation:-

1
2
3
Conclusion: The experiment using TF-IDF weighting for document retrieval demonstrates
its effectiveness in ranking documents based on relevance to the query. By considering both
term frequency and inverse document frequency, TF-IDF accurately identifies relevant
documents. Further exploration of alternative weighting schemes could enhance retrieval
performance.

4
 EXPERIMENT-2
Aim:- Assess the effectiveness of query expansion techniques (e.g., pseudo- relevance
feedback) in improving retrieval results in a vector space model.

Theory:-
In information retrieval, query expansion techniques aim to improve retrieval results by
enhancing the original query with additional terms. Pseudo-relevance feedback is a common
approach where top-ranked documents for an initial query are used to extract relevant terms,
which are then added to the query.

1. Initial Query Processing: The retrieval process begins with an initial query provided by the
user. This query is represented as a vector in the vector space model, typically using TF-IDF
weights for each term.

2. Document Retrieval: Using the initial query, the retrieval system ranks documents in the
corpus based on their similarity to the query vector. This initial retrieval provides a set of top-
ranked documents.

3. Pseudo-Relevance Feedback: Pseudo-relevance feedback utilizes the top-ranked

documents to extract additional terms that are likely to be relevant to the user's information
needs. Commonly used methods include term frequency in top-ranked documents, inverse
document frequency, and relevance feedback models.

4. Expanded Query Construction: The extracted terms from the top-ranked documents are
incorporated into the original query to create an expanded query. This expanded query is now
richer in terms and potentially more representative of the user's information needs.

5. Re-Retrieval: The expanded query is used to retrieve documents from the corpus again,
employing the same retrieval model as in the initial step. This re-retrieval process aims to
improve the relevance of the retrieved documents by leveraging the additional terms from
pseudo-relevance feedback.

6. Evaluation: The effectiveness of query expansion techniques, such as pseudo-relevance

feedback, is assessed based on retrieval performance metrics such as precision, recall, and F1-
score. A comparison is made between the retrieval results obtained with the original query and
those obtained with the expanded query.

7. Analysis: By comparing the retrieval results before and after query expansion, the impact of
the technique on retrieval effectiveness is analyzed. An improvement in retrieval performance
indicates the effectiveness of query expansion in enhancing the relevance of retrieved
documents.

Implementation:-

5
Conclusion: The experiment on pseudo-relevance feedback in a Vector Space Model for
document retrieval concludes that the technique effectively expands queries, improving
retrieval by integrating relevant terms. Evaluation metrics validate enhanced precision and
recall, suggesting potential to refine information representation and elevate user satisfaction in
retrieval tasks.

6
 EXPERIMENT-3
Aim:- Assess the effectiveness of query expansion techniques (e.g., pseudo- relevance
feedback) in improving retrieval results in a vector space model.

Theory:-
To measure the impact of document normalization techniques such as stemming and stop words
removal on retrieval performance in a vector space model, we can conduct the following
experiment:

1. Data Preparation: Preprocess the document collection by tokenizing each document,

converting text to lowercase, and removing punctuation.

2. Normalization: Implement stemming to reduce words to their root forms and remove stop
words to filter out common but less informative terms.

3. Vector Space Model: Represent each document as a vector in the vector space model using
techniques like TF-IDF weighting.

4. Document Retrieval: Retrieve documents using a retrieval algorithm such as cosine

similarity or Okapi BM25.

5. Evaluation: Evaluate retrieval performance metrics such as precision, recall, and F1-score
before and after applying normalization techniques.

6. Comparison: Compare the retrieval performance metrics obtained with and without
normalization techniques to measure their impact on retrieval effectiveness.

7. Analysis: Analyze the results to determine the extent to which stemming and stop words
removal improve retrieval performance. Consider factors such as the size and nature of the
document collection, the specificity of the queries, and the chosen retrieval algorithm.

Implementation:-

7
8
Conclusion: The experiment concluded that document normalization techniques, including
stemming and stop words removal, significantly enhanced retrieval performance in the vector space
model. Improved relevance, precision, and recall post-normalization underscored the critical role of
preprocessing in optimizing information retrieval systems across various domains.

9
10
 Experiment 4
Aim: Evaluate the performance of an IR system using standard evaluation
metrics (e.g., precision, recall, F1-score).
Theory:
1. Precision:
• Precision measures the proportion of retrieved documents that are relevant. It is
calculated as the number of true positives divided by the total number of retrieved
documents.
• Precision = TP / (TP + FP)
2. Recall:
• Recall measures the proportion of relevant documents that are retrieved. It is
calculated as the number of true positives divided by the total number of relevant
documents in the dataset.
• Recall = TP / (TP + FN)
3. F1-score:
• F1-score is the harmonic mean of precision and recall. It provides a balance
between precision and recall.
• F1-score = 2 * (precision * recall) / (precision + recall)

Implementation:
Output:
 Experiment 5
Aim:
Implement an LSI-based search engine and compare its performance with a traditional vector
space model on real-world datasets.
Theory:
1. Vector Space Model (VSM):
• VSM represents documents and queries as vectors in a high-dimensional space,
where each dimension corresponds to a unique term in the vocabulary.
• The similarity between documents and queries is calculated using measures such as
cosine similarity.
• VSM treats terms as independent, ignoring the relationships between them.
2. Latent Semantic Indexing (LSI):
• LSI is a technique that analyzes the relationships between terms and documents by
decomposing the term-document matrix using singular value decomposition (SVD).
• It discovers the latent (hidden) semantic structure in the text corpus by reducing the
dimensions of the term-document matrix.
• LSI can capture the underlying concepts or topics in the text data, allowing for more
accurate retrieval of relevant documents.
Implementation:
Output:
 Experiment 6
Aim:
Investigate the effectiveness of link analysis algorithms (e.g., PageRank, HITS) in ranking web
pages and improving search engine result quality.
Theory:
1. PageRank Algorithm:
• PageRank measures the importance of web pages by analyzing the link structure of
the web. It assigns a numerical weighting to each element of a hyperlinked set of
documents, with the purpose of "measuring" its relative importance within the set.
• The algorithm works by counting the number and quality of links to a page to
determine a rough estimate of the website's importance.
2. HITS Algorithm (Hypertext Induced Topic Search):
• HITS also evaluates the importance of web pages but differs from PageRank in that
it considers both the authority and the hub pages.
• Authority pages are those with valuable content, while hub pages are those that link
to many authority pages.
• It iteratively computes authority and hub scores for each page based on the links
pointing to and from them.
Implementation:
Output:
 Experiment 7
Aim: Compare the effectiveness of different retrieval models (e.g., Vector Space Model,
BM25, LSI) using benchmark datasets.
Theory:
Retrieval models are fundamental components of information retrieval systems that
determine how documents are ranked and retrieved in response to user queries.
1. Vector Space Model (VSM):
● VSM represents documents and queries as vectors in a high-dimensional
space, where each dimension corresponds to a term in the vocabulary.
● Documents and queries are compared using similarity measures such as cosine
similarity, which calculates the cosine of the angle between the vectors.
● VSM does not consider the semantics or relationships between terms and relies
solely on term frequency-inverse document frequency (TF-IDF) weights for ranking.
2. BM25 (Best Matching 25):
● BM25 is a probabilistic retrieval model that improves upon the weaknesses of the TF-
IDF approach by considering document length normalization and term saturation.
● It computes the relevance score between a document and a query based on the frequency
of query terms in the document and the document length.
● BM25 is widely used in modern search engines due to its effectiveness and efficiency.
3. Latent Semantic Indexing (LSI):
● LSI is a dimensionality reduction technique that captures the latent semantic structure
of the document collection.
● It represents documents and queries in a lower-dimensional space by performing
singular value decomposition (SVD) on the term-document matrix.
● LSI can capture latent relationships between terms and documents, making it useful
for capturing semantic similarity.
4. Deep Learning-based Models:
● Deep learning models, such as neural networks, can be applied to information retrieval
tasks. These models often involve learning distributed representations of words and
documents in continuous vector spaces.
● Effectiveness: Deep learning models have shown promising results in various information
retrieval tasks, including document ranking and question answering. They can
automatically learn complex patterns and relationships from data, but they require large
amounts of labeled data and computational resources.
 Comparing the effectiveness of these models often involves using benchmark datasets and
evaluation metrics such as precision, recall, F1-score, mean average precision (MAP), and
normalized discounted cumulative gain (NDCG). The choice of the most suitable model
depends on factors such as the size and nature of the dataset, the specific retrieval task, and
computational resources available.
Code:

Output:
 Experiment 8
Aim: Implement a meta-crawler that aggregates search results from multiple search engines and
evaluates its performance in providing comprehensive and diverse search results.
Theory:
A meta-crawler enhances the search experience by aggregating search results from multiple
search engines. This approach offers several advantages:
Comprehensive Coverage:
Different search engines use varying algorithms and indexes, resulting in unique sets of
search results.By aggregating results from multiple search engines, a meta-crawler can
provide a more comprehensive coverage of the web, capturing a wider range of relevant
information.
Diverse Perspectives:
Each search engine may prioritize different websites, sources, or types of content based on
its ranking algorithms and user behavior analysis.A meta-crawler can incorporate results from
diverse sources, offering users a broader perspective and potentially uncovering valuable
resources that may not appear prominently in any single search engine's results.
Reduced Bias and Manipulation:
Single search engines may be susceptible to bias or manipulation, either unintentionally due
to algorithmic limitations or deliberately through optimization efforts by webmasters.By
combining results from multiple search engines, a meta-crawler can mitigate the impact of
bias or manipulation present in any individual search engine's results.
Enhanced Relevance and Quality:
Aggregating results from multiple search engines allows the meta-crawler to filter out
irrelevant or low-quality results by considering the consensus across different sources.By
prioritizing results that appear in multiple search engine results pages (SERPs), the meta-
crawler can improve the overall relevance and quality of the presented search results.
Increased Efficiency:
Users can save time and effort by using a meta-crawler to retrieve search results from multiple
search engines simultaneously, eliminating the need to manually visit each search engine's
website and conduct separate searches.
Code:

Output:
 Experiment 9
Aim:
Develop information extraction pipelines to extract structured data (e.g., entities,
relationships) from unstructured text sources (e.g., news articles, web pages).

Theory:
Information extraction (IE) pipelines aim to transform unstructured text data into structured
formats by identifying and extracting relevant entities, relationships, and other structured
information. This process involves several key steps:
Text Preprocessing:
Remove noise and irrelevant information from the text, such as HTML tags, special characters,
and punctuation.Tokenize the text into words or phrases, and normalize the text by converting
it to lowercase and removing stopwords.
Named Entity Recognition (NER):
Identify and classify named entities in the text, such as persons, organizations, locations, dates,
and numerical expressions.Utilize pre-trained NER models or train custom models using
annotated datasets to recognize named entities accurately.
Relation Extraction:
Identify relationships or associations between entities mentioned in the text.Use techniques
such as dependency parsing, pattern matching, or machine learning models to extract semantic
relationships from the text.
Entity Linking:
Resolve entity mentions in the text to corresponding entities in a knowledge base or reference
database.Disambiguate ambiguous entity mentions and link them to the most appropriate entity
in the knowledge base.
Structured Data Representation:
Organize the extracted entities and relationships into a structured format, such as tables, graphs,
or knowledge graphs.Enrich the structured data with additional metadata or annotations for
better understanding and interpretation.
Data Collection: Gather a diverse set of unstructured text sources, such as news articles, web
pages, and social media posts, to serve as input data for the information extraction pipeline.
Annotation and Training: Annotate a subset of the data to create training datasets for NER and
relation extraction models. Train custom models if needed to improve extraction accuracy on
specific domains or languages.
 Evaluation Metrics: Define metrics such as precision, recall, and F1-score to
evaluate the performance of the information extraction pipeline in accurately
identifying entities and relationships.
Cross-Validation: Perform cross-validation experiments to assess the
generalization performance of the information extraction pipeline on unseen
data.
Error Analysis: Analyze errors and limitations of the information extraction
pipeline to identify areas for improvement and optimization.
Code:

Output:

1
 Experiment 10
Aim: Collect and integrate specialized information from the web (e.g., product
specifications, reviews) to build domain-specific knowledge bases or
recommendation systems.

Theory:

1) Data Collection and Integration :- Gather product specifications,

descriptions, and reviews from relevant sources such as manufacturer
websites, retailer platforms, and review websites. Integrate collected data
into a structured format suitable for analysis and retrieval.

2) Data Processing and Analysis:- Cleanse and preprocess collecteddata

to ensure consistency and accuracy. Perform sentiment analysis onuser
reviews to extract insights into product perceptions.

3) Knowledge Base Construction:- Design and implement a database

schema to store product information, attributes, and sentiment analysis
results. Populate the database with processed data and establish
relationships between tables for efficient data retrieval.

4) Recommendation System Implementation:- Select and implement

appropriate recommendation algorithms such as collaborativefiltering or
content-based filtering. Train recommendation models using historical
user-item interaction data and evaluate their performance.

5) User Interface Development:- Develop a user-friendly interface (web

or mobile application) to interact with the knowledge base and
recommendation system. Design intuitive features for users to search for
products, explore recommendations, and provide feedback.

6) Testing and Deployment:- Conduct rigorous testing including unit

testing, integration testing, and user testing to ensure system reliability
and usability. Deploy the knowledge base and recommendation system on
a suitable platform, ensuring scalability and accessibility.

2
Implementation:
Step 1: Data Collection

Step 2: Data Processing and Analysis

3
Step 3: Knowledge Base Construction

Step 4: Recommendation System Implementation

3
Output:-

4
 Experiment 11
Aim:

The aim of this project is to analyze customer reviews to understand thefactors

influencing purchasing decisions and to derive actionable insights for improving
product offerings and marketing strategies.

Theory:

1) Data Collection:- Gather customer reviews from various sources suchas

e-commerce platforms, social media, and review websites. Ensure a
diverse and representative dataset by collecting reviews across different
products and categories.

2) Data Preprocessing:- Clean the collected data by removing noise,

including special characters, emojis, and HTML tags. Tokenize the text
into words and remove stopwords to focus on meaningful content.

3) Sentiment Analysis: Analyze the sentiment of each review to

determine whether it's positive, negative, or neutral. Utilize sentiment
analysis techniques such as lexicon-based analysis or machine
learning-based approaches.

4) Identify Key Factors: Extract key phrases or topics from the reviews
that influence purchasing decisions. Use techniques like keyword
extraction or topic modeling to identify these factors.

5) Analysis and Visualization: Visualize the sentiment distribution of

reviews to understand the overall sentiment of customers. Plot the
frequency of key factors to identify the most influential aspects affecting
purchasing decisions.

5
Implementation:-

6
Output:-

Conclusion:
By implementing sentiment analysis on customer reviews and identifying key
factors influencing purchasing decisions, businessescan gain valuable insights
into customer preferences and sentiments.These insights can inform strategic
decision-making processes, leading to improved products, better customer
satisfaction, and increased sales.

7
 Experiment 12
Aim:

Use web scraping techniques to collect data on product prices, reviews,

and availability from e-commerce websites. Deploy the trained model as
an APIendpoint and make predictions.

Theory:

Identify Target Websites: Choose e-commerce websites from which you

wantto scrape data. Popular choices might include Amazon, eBay,
Walmart, etc.

Inspect Website Structure: Use developer tools in your browser to

inspect thestructure of the website. Identify the HTML elements
containing the information you need such as product names, prices,
reviews, and availability.

Write a Web Scraper: Use a programming language like Python along

with libraries like Beautiful Soup or Scrapy to write a web scraper. This
scraper willsend HTTP requests to the target website, parse the HTML
response, and extract the relevant data.

Handle Dynamic Content: Some websites may load content dynamically

usingJavaScript. You may need to use tools like Selenium to simulate a
browser and interact with the page to access this content.

Store the Data: Save the scraped data in a structured format such as
CSV,JSON, or a database.

Develop API Endpoint: Create an API endpoint using a web framework

likeFlask or Django. This endpoint will handle incoming requests, use
the trainedmodel to make predictions, and return the results.

Deploy the API: Deploy the API to a server or cloud platform such as
AWS,Google Cloud Platform, or Heroku.

Security and Scalability: Ensure the API endpoint is secure and can
handlemultiple requests concurrently.

Testing: Test the API endpoint thoroughly to ensure it's functioning

8
correctlyand providing accurate predictions.

Monitoring and Maintenance: Monitor the API endpoint for any issues
andperform regular maintenance as needed.

Implementation:-

9
Output:

Conclusion: This experiment aims to demonstrate the deployment of a

Flask web application and serve a JSON response successfully. By
following the outlined steps, the experimental aim can be achieved, and
insights can be gainedfor future development and deployment projects.
This output provides a comprehensive guide for conducting the
experiment, including all necessary steps, descriptions, and considerations.
It can be used as a reference document throughout the experimental
process.

10
11