[Document title]

CHAPTER 1
PREAMBLE
1.1 Introduction

The House Price Prediction project is a practical application of machine learning that
aims to estimate the market value of residential properties. This project utilizes
historical data and various factors influencing house prices, such as location, size,
number of rooms, amenities, and proximity to key infrastructure, to create a predictive
model.
In this project, we focus on data from Metropolitan city, known for its dynamic real
estate market and diverse property features. The predictive model is built using the
XGBoost and linear regression algorithm, a powerful and efficient gradient boosting
framework known for its high accuracy and ability to handle complex datasets.
XGBoost’s strength lies in uncovering intricate patterns in data, making it an excellent
choice for predicting house prices.
This project not only provides valuable insights into the real estate market but also
serves as a decision-making tool for buyers, sellers, and investors. By leveraging
advanced machine learning techniques, the project demonstrates the potential of data-
driven approaches in addressing real-world challenges, especially in rapidly growing
urban areas.

1.2 Literature survey

Real Estate Property is not only a person's primary desire, but it also reflects a person's
wealth and prestige in today's society. Real estate investment typically appears to be
lucrative since property values do not drop in a choppy fashion. Changes in the value of
the real estate will have an impact on many home investors, bankers, policymakers, and
others. Real estate investing appears to be a tempting option for investors. As a result,
anticipating the important estate price is an essential economic indicator. According to
the 2011 census, the Asian country ranks second in the world in terms of the number of
households, with a total of 24.67 crores. However, previous recessions have
demonstrated that real estate costs cannot see the expenses of significant estate property
are linked to the state's economic situation. Regardless, we don't have accurate
standardized approaches to live the significant estate property values. First, I looked at
different articles and discussions about machine learning for housing price prediction.
The title of the article is house price prediction, and it is based on machine learning and
neural networks.
The publication's description is minimal error and the highest accuracy. The
aforementioned title of the paper is Hedonic models based on price data from Belfast
infer that submarkets and residential valuation this model is used to identify over a
larger spatial scale and implications for the evaluation process related to the selection of
comparable evidence and the quality of variables that the values may require.
Understanding current developments in house prices and homeownership are the
subject of the study. In this article, they utilized a feedback mechanism or social
pandemic that fosters a perception of property as an essential market investment.

1.3 Problem Statement

Consumers face challenges in making informed purchasing decisions due to the
overwhelming number of options in any given product category. This issue is heightened in
the e-commerce space, where numerous online stores offer a wide range of products.
Consequently, consumers often spend significant time researching and comparing products
across various sites, which can be both time-consuming and confusing.
1.4 Objectives of work
The main objectives are,
1. Scrape product data from multiple e-commerce websites.
2. Convert the scraped data into a structured format and store it in a database.
3. Analyze the data using machine learning algorithms to generate actionable insights.
 4. Use linear regression to predict future trends in product pricing and ratings.
5. Present the analysis and predictions through an easy-to-use interface for users.
1.5 Scope of work
• Gather product data from various e-commerce platforms .
• Provide users with a platform to compare product prices, specifications, and trends
easily.
• Use machine learning to predict future prices and analyze product trends.
• Display insights through interactive graphs for better decision-making.
• Help consumers find the best deals and assist businesses with market analysis.
• Expand to include more platforms, product categories, and global markets in the
future.
1.6 Organization of Report
This report contains a total of 6 chapters. First chapter is Preamble which contains
introduction, Literature survey, Problem Statement, Objectives of Work, Scope of the Work
and Organization of the Report. Second chapter contains briefing about technologies that are
used in our project like machine Learning, Django framework, Web scraping and Data
Visualization. Third chapter contains the detailed information on the design and
implementation of our project. Fourth chapter contains the results and snapshots of the
project. Fifth chapter has the conclusion and future scope of our project. Sixth chapter is the
references used for our project.

CHAPTER 2
INTRODUCTION TO WEB SCRAPING AND
MACHINE LEARNING
2.1 Web Scraping
Web scraping, also referred to as web extraction or harvesting, is a technique for
extracting data from the World Wide Web (WWW) and storing it in a structured format, such
as a file system or database, for later retrieval or analysis. This process involves utilizing
Hypertext Transfer Protocol (HTTP) or web browsers to fetch data either manually by a user
or automatically through a bot or web crawler. Given the immense amount of heterogeneous
data continuously generated on the WWW, web scraping has become an indispensable and
efficient method for collecting and analysing big data.
2.1.1 Key Characteristics of Web Scraping
• Automation: Allows for the systematic extraction of large datasets in minimal
time.
• Versatility: Can extract data from multiple formats, such as HTML, JSON, XML,
and multimedia files like images and videos.
• Scalability: Adaptable for tasks ranging from small-scale, ad hoc data collection to
large-scale automated systems.
 2.1.2 The Web Scraping Process
The process of web scraping can be divided into two sequential steps:
Acquiring Web Resources:
• A web scraper sends an HTTP request to the target website.
• The request may be a GET query (URL-based) or a POST query (message-based).
• Once processed, the requested resource (HTML, XML, JSON, or multimedia data)
is retrieved from the website.
Extracting Desired Information:
• The downloaded web data is parsed and organized into a structured format.
• Tools such as Beautiful Soup and Pyquery are commonly used to extract
information from raw HTML or XML.
2.2 Web Scraping with BeautifulSoup4 (bs4)
BeautifulSoup4 (bs4) is a powerful and easy-to-use Python library designed for web
scraping. It allows users to parse HTML and XML documents, navigate through the
document tree, and extract specific data. It is widely used in web scraping applications due to
its simplicity and flexibility. It helps automate the process of extracting data from websites,
which is an essential aspect of many data-driven applications.
2.2.1 Core Components:
The core components of bs4 are
• HTML Parsing: BeautifulSoup4 parses HTML and XML documents and
allows for easy navigation and data extraction.
• Search and Extraction: It provides methods like find(), find_all(), and select()
to locate and extract data from the document using CSS selectors, tags,
attributes, etc.
• Data Cleanliness: BeautifulSoup automatically handles poorly formatted
HTML and still provides accurate results.
2.2.2 Benefits of Using BeautifulSoup4:
• Ease of Use: BeautifulSoup4's API is designed to be easy to understand, even
for beginners in Python and web scraping.
• Compatibility: It works well with different parsers, including html.parser, lxml,
and html5lib.
• Handles Malformed HTML: BeautifulSoup can handle poorly formatted or
malformed HTML, which makes it highly useful when scraping real-world
web data.
• Search Capabilities: BeautifulSoup provides robust methods to search for and
extract elements by tag name, CSS class, ID, and attributes.
2.3 Web Scraping with Selenium
Selenium is a powerful and widely-used web scraping tool that is primarily designed
for automating web browsers. Unlike traditional scraping tools that deal with static HTML
content, Selenium enables interaction with dynamic content that is generated by JavaScript.
This makes it ideal for scraping web pages where content is loaded asynchronously or
requires user interactions like form submissions, button clicks, or scrolling.
2.3.1 Core Components of Selenium:
• WebDriver: The WebDriver is the core component of Selenium that interacts with
the browser. It can simulate user actions like navigating between pages, clicking
on buttons, or filling out forms.
• Browser Automation: Selenium supports multiple web browsers including
Chrome, Firefox, Safari, and Edge, allowing cross-browser scraping.
• Element Interaction: Selenium allows you to locate and interact with page
elements using various locators such as XPath, CSS Selectors, or element IDs.
Once an element is located, you can perform actions like clicking, typing, or
submitting.
• JavaScript Rendering: Selenium works with real web browsers, allowing it to
render JavaScript and interact with dynamically loaded content.
2.3.2 Benefits of Using Selenium:
• JavaScript Rendering Support: Selenium is particularly useful for scraping content
that is dynamically generated using JavaScript or AJAX. While libraries like
BeautifulSoup only scrape static HTML, Selenium can interact with web pages
that require JavaScript to load content. It can render the page, execute JavaScript,
 and extract the necessary data.
• Browser Automation: Selenium can fully automate browser tasks, making it
possible to interact with websites that require form submissions, user
authentication, or complex actions like dropdown selections. This allows for
automated data collection from websites that require these interactions.
• Simulates Real User Behavior: Selenium mimics the way a real user would interact
with a webpage, including clicking buttons, filling out forms, scrolling, and even
handling user input. This capability is especially useful for scraping websites that
detect and block non-human traffic, such as bot protections and CAPTCHA
challenges.
• Cross-Browser Compatibility: Selenium supports several browsers, including
Chrome, Firefox, Safari, and Edge. This cross-browser compatibility ensures that
your scraping scripts will work consistently across different platforms and
environments.
• Handles Complex Websites: Selenium is well-suited for scraping complex websites
that use dynamic content, infinite scrolling, or multiple page interactions. It can
simulate user behaviors like scrolling or clicking through multiple pages to load
additional content, making it an effective tool for scraping modern web
applications.
• Headless Mode: Selenium can run browsers in headless mode, which means the
browser operates without a graphical user interface (GUI). This is useful when
running scraping scripts in a server environment or when you don't need to see the
browser window. Running in headless mode can improve performance and reduce
resource usage.
• Element Location and Interaction: Selenium supports multiple strategies for
locating elements, including using XPath, CSS Selectors, and IDs. This flexibility
allows you to write reliable scraping scripts even for complex and dynamic web
pages.
• Real-Time Debugging: Since Selenium operates with a real browser, it provides the
benefit of real-time debugging. You can visually observe how the browser
interacts with the page, making it easier to troubleshoot issues and understand the
behavior of the webpage.

2.3.3 Purpose of using Selenium for Web Scraping

Selenium is ideal for websites where content is dynamically loaded via JavaScript,
AJAX, or requires complex interactions such as login forms or buttons. It can handle
modern web applications that traditional scraping libraries like BeautifulSoup may not be
able to manage due to their lack of JavaScript rendering capabilities. However, Selenium
tends to be slower than other scraping tools because it controls an actual browser. It is
best used when scraping websites with heavy dynamic content or those that require user-
like interactions. Selenium’s flexibility and robust automation features make it a go-to
tool for web scraping projects where user behaviour needs to be simulated, dynamic
content must be rendered, or multiple page interactions are necessary.
2.4 Machine Learning
Machine learning is a subset of artificial intelligence (AI) that enables systems to
learn from data and improve their performance without being explicitly programmed. It
involves using algorithms to identify patterns and make decisions or predictions based on
data. Machine learning techniques have a wide range of applications, including data analysis,
automation, predictive modeling, recommendation systems, and more.
Machine learning can be divided into three primary types:
• Supervised Learning: The model is trained on labeled data, where the output is
known. The algorithm learns the relationship between input data and output to predict
the results for new data.
• Unsupervised Learning: The model is provided with unlabeled data and must find
patterns or structures within the data, such as grouping similar data points together
(e.g., clustering).
• Reinforcement Learning: The model learns by interacting with its environment and
receiving feedback through rewards or penalties.
2.4.1 Linear Regression
Linear regression is one of the simplest and most widely used algorithms in
machine learning. It is a supervised learning technique used for predicting a
continuous target variable (dependent variable) based on one or more input features
(independent variables). The goal of linear regression is to model the relationship
 between the target variable and the predictors using a linear equation.
2.4.1.1 Key Characteristics of Linear Regression:
• Simple and interpretable: The relationship between input features and the
output is directly represented by a linear equation, making it easy to
interpret.
• Assumptions: Linear regression assumes a linear relationship between input
and output variables, which may not always hold in real-world datasets.
• Use cases: It is commonly used for predicting continuous values, such as
predicting sales, stock prices, or even the temperature.
2.4.1.2 Benefits of Linear Regression:
• It is computationally efficient and easy to implement.
• It provides insight into the relationship between variables.
• It is well-suited for tasks like trend analysis and forecasting.
• When to use Linear Regression: Linear regression is ideal when you have a
linear relationship between input features and the output variable. It works
well for tasks like predicting numerical outcomes and can be extended to
multiple variables (multiple linear regression).
2.4.2 K-Means Clustering
K-Means clustering is an unsupervised machine learning algorithm used for
clustering similar data points into groups, known as clusters. It is one of the most
popular clustering algorithms due to its simplicity and efficiency. The goal of K-
Means clustering is to divide a dataset into K clusters such that the data points within
each cluster are as similar as possible and as different as possible from those in other
clusters.
2.4.2.1 Key Characteristics of K-Means Clustering:
• Unsupervised learning: K-Means does not require labeled data and instead
groups data based on similarities.
• Centroid-based: The algorithm uses centroids to represent clusters, and each
data point is assigned to the closest centroid.
• Distance metric: K-Means relies on a distance metric (usually Euclidean
distance) to determine the closeness of data points to centroids.
2.4.2.2 Benefits of K-Means Clustering:
• Efficiency: K-Means is computationally efficient, especially with large
datasets.
• Simplicity: The algorithm is easy to implement and understand.
• Scalability: K-Means scales well with larger datasets and can handle many
data points effectively.
2.5 Web Scraping with Django
Django is a high-level Python web framework that encourages rapid development and
clean, pragmatic design. While Django itself is not a web scraping tool, it can be seamlessly
integrated with web scraping libraries like BeautifulSoup4, Selenium, and Scrapy to store and
display scraped data on a web interface. Using Django in web scraping projects allows
developers to create scalable, dynamic websites that serve the extracted data efficiently and
interactively.
2.5.1 Core Components of Django
The Core Components of Django is
• Models: In Django, data is stored in a database using Models, which define the
structure of your database tables. After scraping data from websites, you can
save it into Django models for easy management and retrieval.
• Views: Django views are responsible for retrieving and displaying the scraped
data on web pages. You can create views that query the database and present
the scraped data in an HTML template.
• Templates: Templates in Django are used to define how the scraped data is
displayed to users in a browser. Django's templating language allows you to
create dynamic HTML pages that can present data in tables, lists, or charts.
• Admin Interface: Django comes with a built-in admin interface that allows
you to manage and view the data stored in the database. After scraping and
storing data, the admin interface offers an easy way to manage it without
writing custom views.
2.5.3 Benefits of Using Django for Web Scraping Projects
 • Scalability: Django is built for scalability and can handle a large volume of
data, making it ideal for web scraping projects where the data collected is
significant. Its ORM (Object-Relational Mapping) allows easy interaction with
databases, which is essential when dealing with scraped data.
• Database Integration: Django provides an easy-to-use ORM to handle database
operations. Once data is scraped, it can be stored and queried directly in the
database without having to worry about raw SQL queries.
• Easy Integration with Scraping Libraries: Django can be integrated with
popular web scraping libraries like BeautifulSoup4, Selenium, or Scrapy to
collect data. Once the data is collected, Django can serve it through views and
templates, making it accessible through a web interface.
• Web Interface for Display: Django provides a powerful templating engine that
allows you to dynamically display the scraped data in a user-friendly interface.
You can create dashboards, charts, or tables to visualize the data.
• Built-in User Authentication: If your scraping project requires user accounts or
access control (such as limiting who can view or submit scraped data),
Django’s built-in authentication system is a helpful feature. You can easily
implement user logins, signups, and permissions.
• Rapid Development: Django’s architecture allows for quick setup and
development, meaning you can focus on scraping the data and serving it
through a robust web interface without worrying about the underlying backend
complexities.
• Extensive Documentation and Community Support: Django has extensive
documentation and an active community. If you run into problems during your
scraping project, you can easily find solutions and tutorials to guide you.
2.6 Pandas
Pandas is a powerful, open-source data analysis and manipulation library for Python.
It is widely used in machine learning and data science workflows due to its ability to
efficiently handle and manipulate structured data. Pandas provides data structures such as
Series and DataFrame that allow for fast and flexible data manipulation and analysis.
2.6.1 Key Features of Pandas:
The features of Pandas are its efficient handling of large datasets, flexible data
manipulation capabilities, support for missing data handling, powerful data
aggregation and grouping functions, seamless integration with other Python libraries,
and its ability to provide quick exploratory data analysis through built-in functions.
• DataFrame: The core data structure in Pandas, which is essentially a 2D table
(similar to a spreadsheet or SQL table) where data is organized in rows and
columns.
• Series: A one-dimensional array-like object in Pandas, which can store a single
column of data. A Series is essentially a labeled list.
• Handling Missing Data: Pandas provides several methods for detecting,
removing, and replacing missing values (NaN) in datasets.
• Data Alignment: Pandas automatically aligns data when performing operations
on different DataFrames or Series.
• Label-based Indexing: Data can be accessed and manipulated using labels (row
and column names), making it more intuitive and readable.
• GroupBy: This powerful feature allows users to group data based on certain
criteria and apply aggregate functions (such as sum, mean, etc.) to those
groups.
• Merging and Joining: Pandas supports powerful merging and joining
operations, allowing data from different sources to be combined easily, similar
to SQL joins
2.6.2 Benefits of Pandas
• Efficiency: Pandas is highly optimized for performance, enabling fast handling
and manipulation of large datasets.
• Flexibility: It provides a wide range of functions for reshaping, merging,
aggregating, and filtering data.
• Integration: Pandas works seamlessly with other Python libraries, such as
NumPy, Matplotlib, and Scikit-Learn, making it an essential tool in data
science and machine learning workflows.
 • Data Cleaning: Pandas simplifies data cleaning and preparation tasks, such as
handling missing data, removing duplicates, and performing transformations.
2.7 Matplotlib
Matplotlib is a widely used Python library for creating static, interactive, and
animated visualizations. It is particularly useful for creating high-quality charts, graphs, and
plots, making it an essential tool for data analysis and machine learning tasks. With
Matplotlib, users can visualize their data in a variety of formats, helping to uncover trends,
patterns, and insights.
2.7.1 Key Features of Matplotlib:
• Wide Range of Plot Types: Matplotlib supports numerous plot types, including
line plots, bar charts, histograms, scatter plots, pie charts, and more.
• Customization: It provides extensive customization options for plots, including
control over colors, labels, titles, grid lines, axis ticks, and more.
• Integration with Pandas and NumPy: Matplotlib integrates seamlessly with
other popular libraries like Pandas and NumPy, enabling the visualization of
data stored in DataFrames or arrays.
• Interactive Plots: While primarily used for static plots, Matplotlib also supports
interactive plotting, which is useful for exploring data and adjusting the
visualizations dynamically.
• Subplots: It allows for the creation of multiple plots within a single figure,
making it easy to compare different visualizations side by side.
• Saving Plots: Plots created using Matplotlib can be saved in various formats
such as PNG, PDF, SVG, and more, making it suitable for report generation or
sharing results.
2.7.2 Benefits of Matplotlib:
• Versatile: Matplotlib supports a wide variety of plot types and is capable of
producing both simple and complex visualizations.
• Customization: The library allows for detailed customization of plots, enabling
users to tailor visualizations to their specific needs.
• Wide Adoption: Matplotlib is a widely adopted library with a large community,
making it easy to find resources and support.
• Integration: It works well with other libraries, such as Pandas for data
manipulation and NumPy for numerical operations, making it an essential tool
for data analysis and machine learning workflows.
• Publication-Quality Graphics: Matplotlib is capable of creating high-quality,
publication-ready graphics, suitable for presentations, reports, and academic
papers.

2.7 Tools Used

In this project, various tools and technologies were used to carry out the web scraping and
data presentation tasks efficiently. Below are the key tools employed:
2.7.1 VS Code (Visual Studio Code)
Visual Studio Code (VS Code) is a lightweight and powerful code editor that
is highly popular among developers for its rich feature set. It provides features such as
intelligent code completion, debugging support, version control integration, and
customizable themes. For this project, VS Code was used as the primary Integrated
Development Environment (IDE) for writing and editing Python scripts, HTML
templates, and Django configuration files. The extension support for Python and
Django, along with Git integration, made it an ideal choice for development.
2.7.2 Python
Python is a high-level programming language that emphasizes readability and
simplicity, making it an excellent choice for web scraping and data manipulation.
Python was the primary language used for the entire web scraping process in this
project. Libraries such as BeautifulSoup4, Selenium, and Requests were utilized for
scraping data from websites, while Django was used to manage the backend, handle
database interactions, and present the scraped data through a web interface. Python's
rich ecosystem of libraries and frameworks played a critical role in simplifying the
scraping and data management tasks.
CHAPTER 3
DESIGN & IMPLEMENTATION
3.1 High Level Design

Figure 3.1
In the Figure 3.1 you can go through the high level design
of the project and below given are the main components
1. User Interface Layer

• Purpose: Collect user input and display results.

• Components:

o Frontend:

▪ Web application built using Django Templates.

▪ Input fields for user queries (e.g., product search).

▪ Output sections for best products, price predictions, and graphs.

o API Endpoint:

▪ Backend API to process user requests.

2. Data Collection Layer

• Purpose: Scrape product data from multiple e-commerce websites.

• Components:

o Scraping Engine:

▪ Libraries like BeautifulSoup and Selenium for scraping product data

(price, title, rating).

▪ Modular scraping functions for different platforms (Amazon, eBay, Ajio,

Snapdeal).

o Data Preprocessing:
 ▪ Handle missing or incorrect data.

▪ Convert scraped data into a structured format (Pandas DataFrame).

3. Data Analysis Layer

• Purpose: Analyze scraped data and rank products.

• Components:

o K-Means Clustering:

▪ Normalize price and rating data using MinMaxScaler.

▪ Apply clustering to group products based on similarity.

▪ Compute distances from centroids to rank products.

o Product Scoring:

▪ Combine clustering results and ratings to assign scores to products.

4. Prediction Layer

• Purpose: Predict future price trends based on historical data.

• Components:

o Linear Regression:

▪ Model future price trends using historical price data.

▪ Predict prices for upcoming intervals.

o Visualization:

▪ Generate price trend graphs using libraries like Matplotlib.

▪ Encode graphs as Base64 for embedding in the web page.

3.2 Module Wise Explanation

The Project is divided into the three modules. The first module includes collection of product
details from different websites and to convert it into structured format. The second module
includes cleaning and pre processing of data to analyse using Machine Learning Algorithm.
The third chapter includes predicting future trends and predicting the future prices using
linear regression.
3.2.1 Module 1
The module is structured to systematically handle user input, scrape data from
various e-commerce platforms, process and validate the extracted data, and store it in
a unified database. In the Flowchart Figure 3.1 you can see that it begins with User
Input Handling, where the user provides a query (e.g., a product name). This query is
processed to generate platform-specific search URLs, enabling tailored scraping for
each e-commerce website.
The Web Scraping Logic employs a hybrid approach to accommodate the diverse
nature of website structures. For dynamic websites like Amazon and Ajio, where
content is rendered using JavaScript, the module uses Selenium WebDriver to interact
with page elements and load the data dynamically. On the other hand, for static
websites like eBay and Snapdeal, the module utilizes BeautifulSoup for efficient
HTML parsing and extraction of product details. This combination of Dynamic
Scraping (Selenium) and Static Scraping (BeautifulSoup) ensures compatibility and
efficiency in handling varied platforms. Once the data is scraped, it undergoes Data
Cleaning and Validation to maintain its integrity. This includes validating critical
fields like price and ratings, handling missing values, and ensuring consistency in
currency formats. For instance, prices extracted from eBay in USD are converted to
INR using a predefined conversion rate, ensuring uniformity across platforms.
 Figure 3.2
Finally, robust Error Handling mechanisms are implemented to address potential
challenges during scraping. This includes handling missing data fields, invalid ratings,
navigation failures during pagination, and structural changes in the target websites.
These mechanisms ensure that the module can gracefully recover from unexpected
errors without disrupting the data extraction process. This structured and modular
approach ensures scalability, reliability, and accuracy in scraping and storing product
data.

3.2 .2 Module 2
In the Figure 3.2 you can see that the process begins when the user submits a
search query (e.g., a product name) through the web interface. Based on this query,
the system scrapes data from four e-commerce websites. For Amazon and Ajio, which
have dynamic content, the system uses Selenium WebDriver to interact with the page
elements and capture the product details such as title, price, rating, and product link.
For Snapdeal and eBay, which feature static content, BeautifulSoup is used to parse
and extract the required information more efficiently.
Once the data is gathered, it undergoes a cleaning process. Prices are cleaned by
removing any unwanted characters (such as the ₹ symbol and commas) and are then
converted to float values. Ratings are converted to numeric values, with any missing
ratings replaced by a default value (e.g., 0). The next step involves data normalization
using the MinMaxScaler. This ensures that both the price and rating are scaled to a
range between 0 and 1, making the data comparable across products.

Figure 3.3
The system then applies K-Means clustering to group products based on their
normalized price and rating. Each product is assigned to one of three clusters, and the
system calculates the distance to the centroid of each cluster. This distance, along with
the product’s rating, is used to generate a predicted score for each product. The
predicted score helps rank the products, with those having the highest scores
appearing first.
Finally, the best three products from each platform are displayed to the user. For each
platform (Amazon, Ajio, eBay, and Snapdeal), the system shows the product title,
price, rating, and a link to the product. This ranking process allows users to view the
best products across different e-commerce platforms based on price and rating,
optimized using machine learning techniques.
 3.2.3 Module 3
In the Figure 3.3 you can go through the systematic process for predicting
future price trends using web scraping and linear regression. This method involves
distinct stages, each playing a crucial role in. Linear regression, a simple yet powerful
machine learning technique, is employed to uncover patterns and relationships in
historical price data. By analyzing past trends, the model determines how prices have
fluctuated over time.

Figure 3.4
The insights generated from these predictions are invaluable. These predictions can be
displayed as graphs, dashboards, or interactive interfaces, making them easy to
interpret and utilize. Visualization tools help highlight key trends, such as price drops,
surges, or seasonal patterns, allowing users to explore and understand the data.
In summary, this workflow combines the automation of web scraping, the analytical
power of linear regression, and the clarity of visualizations to create a robust system
for price trend prediction. This approach not only saves time and effort but also
empowers users to make informed decisions based on reliable, data-driven insights.
3.3 Overall Flow of the Project
In the Figure 3.4 you can go through the project begins with user input, where the
user provides a product query, such as a product name or category. Using web scraping tools
like BeautifulSoup and Selenium, the system retrieves unstructured product data, including
details like titles, prices, and ratings, from e-commerce platforms. This unstructured data is
then converted into structured form for further processing. The structured data undergoes
cleaning and preprocessing using Pandas to handle missing values and ensure consistency.
Once cleaned, the data is analyzed using machine learning algorithm K-Means Clustering
Algorithm to extract insights, such as clustering products based on price and ratings. Finally,
linear regression is applied to predict future prices, enabling refined product rankings and
predictive insights. The process concludes by presenting the results to the user.

Figure 3.5
3.4 Implementation
• User Input
The project begins with the user providing the product title they wish to
analyze. This input is sent to the backend for further processing.
• Scraping Product Data
Based on the product title, relevant product data such as price, ratings, and
reviews are extracted from e-commerce websites. BeautifulSoup is used to
 parse static HTML pages for data extraction, while Selenium handles
dynamically rendered pages. Selenium automates browser interactions to
retrieve fully rendered HTML, which is then processed using BeautifulSoup.
• Data Structuring and Cleaning
The scraped data is initially in an unstructured format and is converted into a
structured tabular form using Pandas. Fields such as product name, price,
rating, and reviews are organized for analysis. A data cleaning process is
performed to handle missing or inconsistent values, normalize currencies, and
remove redundant or irrelevant entries.
• Analysis Using KMeans Clustering
The cleaned data is analyzed using KMeans clustering to group similar
products based on attributes like price and rating. This process identifies
distinct market segments, such as budget-friendly, mid-range, and premium-
priced products.
• Price Prediction Using Linear Regression
Linear Regression is applied to predict future prices for the specified product
title. Historical pricing data is analyzed, with time as the independent variable
and price as the dependent variable. The model forecasts potential price
trends, providing insights into future pricing.
• Frontend Display:
The results are presented on a user-friendly front end built with HTML, CSS,
and JavaScript. The front end accepts the product title as input and displays
the clustering insights and price predictions in a visually appealing manner.

3.5 Pseudocodes
3.5.1 Module 1 :- Scraping the Data using bs4 and Selenium
IMPORT necessary libraries (requests, BeautifulSoup, selenium, etc.)
DEFINE constants for user-agent headers
DEFINE initialize_driver() -> return WebDriver for headless browsing

DEFINE scrape_amazon(query):
- Replace spaces with '+'
- Initialize driver and load Amazon search page
- Extract product details (title, price, rating, link)
- Move to next page if available
- Save scraped products to database

DEFINE scrape_ebay(query):
- Replace spaces with '+'
- Send request to eBay search URL
- Parse response and extract product details (title, price, rating, link)
- Convert USD price to INR using usd_to_inr()
- Save scraped products to database

DEFINE scrape_snapdeal(query):
- Replace spaces with '+'
- Send request to Snapdeal search URL
- Parse response and extract product details (title, price, rating, link)
- Save scraped products to database

DEFINE scrape_ajio(query):
- Replace spaces with '+'
- Initialize driver and load Ajio search page
- Wait for products to load and extract product details
- Save scraped products to database

DEFINE save_products(products, source):

- For each product:
- Check if the rating is valid using is_valid_rating()
- Save to the respective database model (AmazonProduct, eBayProduct, etc.)
- Save to general Product model

DEFINE is_valid_rating(rating):
 - Try to convert rating to float
- If valid, return True, else return False

3.5.2 Module 2:- Product Analysis using K-Means Clustering

Algorithm
BEGIN
FUNCTION rank_products_ml(products):
- Convert list of products into a DataFrame
- Clean and preprocess 'price' and 'rating' columns:
- Remove symbols from 'price'
- Convert 'price' to float and 'rating' to numeric values
- Normalize 'price' and 'rating' columns using MinMaxScaler
- Apply KMeans clustering to group products based on normalized values
- Calculate the distance of each product from the cluster centroid
- Compute a predicted score for each product based on distance and rating
- Sort products by predicted score in descending order
- RETURN sorted products

FUNCTION scrape_products(request):
- Retrieve the search query from the request

IF query is provided:
- Scrape products from different platforms (Amazon, eBay, Snapdeal, Ajio)
- Rank the scraped products from each platform using rank_products_ml
function
- Extract the best 3 products from each platform

- Prepare context with:

- All scraped products from Amazon, eBay, Snapdeal, Ajio
- best 3 products from each platform (ranked by predicted score)

- Return the rendered page with the context containing product data

ELSE:
- Return a rendered page with an error message ("No query provided")
END
END
3.5.3 Module 3 Future Price Prediction using Linear Regression
FUNCTION visualize_price_data(prices, labels=None, future_steps=10):
- Create a DataFrame using the `prices` list
- Add an additional column:
- Use `labels` for X-axis if provided
- Otherwise, use default indices as labels
- Train a Linear Regression model:
- Use the `label` column as features (X)
- Use the `price` column as the target (y)
- Predict future prices:
- Generate future labels starting from the last index
- Use the model to predict prices for the next `future_steps`
- Plot the data:
- Plot actual prices vs. labels as a blue line with markers
- Plot predicted future prices as a red dashed line
- Add labels, a title, and rotate X-axis labels for readability
- Save the plot:
- Save the plot image to a temporary buffer in PNG format
- Encode the image in Base64 format
- RETURN:
- Base64-encoded image string of the plot
- Predicted future prices
CHAPTER 4
RESULTS AND SNAPSHOTS

Figure 4.1
This page depicts the home page of our project which includes the description of the project .
 Figure 4.2
This image shows different cities covered with some description in home page

Figure 4.3
This graph shows the overall analysis of the preferred cities.
 Figure 4.4
This graph shows the overall analysis of cost of living of the cities .

Figure 4.5
This graph shows the overall analysis of spaciousness of the cities covered.

Figure 4.6
This graph shows the overall analysis of affordability of the cities covered.
 Figure 4.7
This image shows the analysis and graph of the Mumbai city.

Figure 4.8
This image shows the analysis and graph of the Kolkata city
 Figure 4.9
This image shows the analysis and the graph of the Delhi city
 Figure 4.10
This image shows the analysis and graph of the Pune city
 Figure 4.11
This image shows the analysis and graph of the Chennai city

Figure 4.12
This image shows the analysis and graph of the Bangalore city
 Figure 4.13
This image shows the analysis and graph of the Ahmedabad city
 Figure 4.14
This image displays the predict page ,where it takes the input from the user.

Figure 4.15
This image is the result page ,displays the prediction of the city we choose
CHAPTER 5
CONCLUSION & FUTURE SCOPE
5.1 Conclusion
House price prediction using machine learning is a valuable tool for the real estate market.
By analyzing factors like location, size, and property features, machine learning models can
accurately estimate property prices. This helps buyers, sellers, and investors make better
decisions based on data. The success of the system depends on using good quality data,
selecting the right features, and updating the model regularly to reflect market changes.
Overall, it shows how technology can make price estimation more efficient, reliable, and
useful for everyone involved.
5.2 Future Scope
• Offer personalized product recommendations based on user behaviour.
• Provide real-time price tracking, stock updates, and trending product insights.
• Predict market trends and optimal pricing using advanced analytics.
• Launch a mobile app with features like voice and image search for convenience.
• Expand to support international markets with multi-currency options.
• Deliver competitive analysis and insights to help businesses make strategic decisions.
• Use interactive dashboards for clear and actionable data visualization.

CHAPTER 6
REFERENCES
[1] Aswad Shaikh, Aniket Sonmali, and Soham Wakade, "Product Comparison Website using
Web Scraping and Machine Learning," Student at Department of Information Technology,
Atharva College of Engineering, Mumbai, Maharashtra, India.
[2] V. Srividhya and P. Megala, "Scraping and Visualization of Product Data from E-
commerce Websites," Dept of Computer Science, Avinashilingam Institute for Home Science
and Higher Education for Women, Coimbatore, Tamilnadu, India.
[3] C. Lotfi, S. Srinivasan, M. Ertz, and I. Latrous, "Web Scraping Techniques and
Applications: A Literature Review," Labo NFC, University of Quebec at Chicoutimi, 555
Boulevard de l’Université, Saguenay (QC), Canada.
[4] M. Fraňo, "Web Scraping as a Data Source for Machine Learning Models and the
Importance of Preprocessing Web Scraped Data," University of Twente, The Netherlands.
[5] B. Zhao, "Web Scraping," College of Earth, Ocean, and Atmospheric Sciences, Oregon
State University, Corvallis, OR, USA.
[6] S. C. M. de S. Sirisuriya, "Importance of Web Scraping as a Data Source for Machine
Learning Algorithms - Review," Department of Computer Science, Faculty of Computing,
General Sir John Kotelawala Defence University, Sri Lanka.
[7] R. Praba, G. Darshan, K. T. Roshanraj, and B. Surya Prakash, "Study On Machine
Learning Algorithms," Department of Information Technology, Dr. N.G.P. Arts and Science
College, Coimbatore, Tamil Nadu, India.
[8] T. S. Singh and H. Kaur, "Review Paper on Django Web Development," Department of
Computer Application, Rayat Bahra University.
[9] Data Visualization workshop, Tim Grobmann and Mario Dobler, Packt Publishing, ISBN
9781800568112
[10] https://docs.python.org/3/
[11] https://pypi.org/project/beautifulsoup4/
[12] https://pypi.org/project/requests/
[13] https://www.djangoproject.com/
[14] https://www.selenium.dev/documentation/
[15] https://docs.djangoproject.com/en/stable/
[16] Adrian Holovaty, Jacob Kaplan Moss, The Definitive Guide to Django: Web
Development Done Right, Second Edition, Springer-Verlag Berlin and Heidelberg GmbH &
Co. KG Publishers, 2009
[17] S. Sridhar, M Vijayalakshmi “Machine Learning”,Oxford,2021.
[18] https://scikit-learn.org/stable/documentation.html
[19] https://matplotlib.org/stable/contents.html