UNIT - 1

BIG DATA
Study4sub

SYLLABUS:
Introduction to Big Data: Types of digital data, history of Big Data innovation, introduction to
Big Data platform, drivers for Big Data, Big Data architecture and characteristics, 5 Vs of Big
Data, Big Data technology components, Big Data importance and applications, Big Data
features – security, compliance, auditing and protection, Big Data privacy and ethics, Big Data
Analytics, Challenges of conventional systems, intelligent data analysis, nature of data, analytic
processes and tools, analysis vs reporting, modern data analytic tools.
What is Big Data?
Big Data refers to extremely large and complex data sets that are difficult to store, manage,
and process using traditional data processing tools and methods.
In Simple Words:
• Big Data means “a huge amount of data” — so big and fast that traditional software (like
Excel or simple databases) can't handle it.
Key Points:
• It includes structured, semi-structured, and unstructured data.
Study4sub

• The size of Big Data is usually in terabytes, petabytes, or more.

• Big Data is not only about size but also speed, variety, and complexity.
Example:
• Imagine YouTube — every minute, people upload 500+ hours of videos. All that video
content, user data, likes/dislikes — this is Big Data. Analyzing it helps YouTube recommend
videos, detect spam, and show relevant ads.
1. Types of Digital Data
Digital data is the foundation of Big Data. It is generally categorized into three types:
1. Structured Data
• Definition: Data organized in a fixed schema (rows and columns).
• Storage: Stored in traditional databases (RDBMS).
• Examples:
• Employee records in Excel
• SQL databases like MySQL, Oracle
• Features:
• Easy to enter, store, query, and analyze.
• Machine-readable and well-formatted. Study4sub
2. Semi-Structured Data
• Definition: Data that does not follow a strict structure but still contains tags or markers for easy separation of
elements.
• Storage: Stored in formats like XML, JSON, NoSQL databases.
• Examples:
• Emails (to, from, subject, body)
• Web pages with HTML tags
• Sensor data in JSON format
• Features:
• More flexible than structured data.
• Requires custom parsers or semi-structured data tools.
3. Unstructured Data
• Definition: Data that lacks any predefined format or organization.
• Storage: Data lakes, distributed file systems (e.g., HDFS).
• Examples:
• Social media posts, images, videos, PDFs
• Surveillance footage, audio files
• Features: Study4sub
• Most abundant form of data today.
• Hard to analyze without specialized tools like Hadoop/Spark.
History of Big Data Innovation
The evolution of Big Data spans across decades, and it is driven by advances in storage,
processing power, and the internet.
Timeline Overview:

Early development of databases and data storage systems like IBM’s IMS.
1960s–70s
Data stored on magnetic tapes.

Relational databases (RDBMS) like Oracle, DB2 emerged. Structured data

1980s
dominated.
Study4sub
Explosion of the internet. Web data starts growing. Data Warehousing
1990s
concepts introduced.
Gartner analyst Doug Laney introduces the 3Vs model (Volume, Velocity,
2001
Variety) of Big Data.
Google publishes MapReduce and Google File System (GFS) papers. These
2003–2004
inspired Hadoop.
Apache Hadoop is born (by Doug Cutting). Handles distributed processing
2005
of massive data sets.
Growth of social media, mobile apps, cloud computing – huge surge in
2010s unstructured data. Spark, NoSQL, and cloud-based big data platforms
evolved.

AI, ML, and IoT generate and consume massive real-time data. Focus on
2020 onwards
data privacy, ethics, and advanced analytics tools.
Key Drivers of Big Data
• The term “drivers” refers to the factors or reasons behind the growth and importance
of Big Data. These are the main forces that have pushed Big Data to become essential
in today’s world.
1. Rapid Growth of Internet and Social Media
• Billions of users are active on platforms like Facebook, YouTube, Instagram, and
Twitter.
• Every second, people are uploading photos, videos, comments, likes, etc.
• This creates huge volumes of data every Study4sub
day.
Example: YouTube gets more than 500 hours of video uploads per minute.
2. Increased Use of Smartphones and IoT Devices
• Every smartphone, smartwatch, and smart device (like Alexa, fitness bands, smart
TVs) collects and sends data.
• These devices generate real-time data from various sensors.
• Example: A smart home system collects data about temperature, lights, and
energy usage.
3. Cheap Storage and Cloud Computing
• Earlier, storing large data was expensive.
• Now, cloud services like AWS, Google Cloud, and Microsoft Azure offer cheap and scalable storage.
• This allows companies to collect and store massive amounts of data easily.
• Point to remember: Cloud storage is flexible, cost-effective, and accessible from anywhere.
4. Advancements in Data Processing Technologies
• Tools like Hadoop, Spark, NoSQL databases allow fast and distributed processing of large data.
• These technologies help in handling structured and unstructured data with ease.
• Example: Hadoop breaks big data into smaller parts and processes it in parallel.
Study4sub
5. Need for Real-Time Decision Making
• Companies need to make quick decisions to stay competitive.
• Big Data helps in analyzing trends, customer behavior, and business performance in real-time.
• Example: E-commerce sites suggest products based on what users just searched.
6. Growth of AI and Machine Learning
• AI and ML need huge amounts of data to learn and make accurate predictions.
• Big Data provides the fuel for these smart systems.
• Example: Netflix uses ML and Big Data to recommend shows based on your watch history.
1. Introduction to Big Data Platform
Definition:
A Big Data Platform is an integrated system that combines various tools and
technologies to manage, store, and analyze massive volumes of data efficiently.
It provides the infrastructure and environment required for:
• Ingesting data (bringing data from sources)
• Storing data (on distributed systems)
Study4sub
• Processing data (in batch or real-time)
• Analyzing and visualizing data
Benefits of Big Data Platforms:
• Scalability: Easily handle growing data
• Flexibility: Supports all types of data (structured, semi-structured, unstructured)
• Real-Time Processing: Immediate insights and decisions
• Cost-Effective: Cloud and open-source tools reduce expenses
Main Components of a Big Data Platform:
1.Data Ingestion Tools
1. Used to collect and import data from different sources
2. Examples: Apache Kafka, Apache Flume, Sqoop
2.Data Storage Systems
1. Store large datasets reliably
2. Examples: HDFS (Hadoop Distributed File System), NoSQL (MongoDB, Cassandra)
3.Processing Engines
Study4sub
1. Perform computations and analytics on data
2. Examples: Hadoop MapReduce (batch), Apache Spark (real-time)
4.Data Management
1. Tools to organize, clean, and maintain data quality
2. Examples: Hive, HBase
5.Analytics & Visualization Tools
1. Help in generating reports and dashboards
2. Examples: Tableau, Power BI, Apache Pig, R, Python
Examples of Big Data Platforms:
• Apache Hadoop Ecosystem
• Apache Spark Framework
• Google Cloud BigQuery
• Amazon EMR (Elastic MapReduce)
• Microsoft Azure HDInsight
Big Data Architecture and Characteristics
Study4sub

What is Big Data Architecture?

• Big Data Architecture is the framework that defines how Big Data is collected,
stored, processed, and accessed.
It uses distributed computing — meaning the work is shared across many
machines to handle large-scale data.
Main Components of Big Data Architecture:
1. Data Sources
These are the origins where data is generated. Data can come from websites, social
media, IoT devices, or business applications.
2. Storage Layer
Stores large volumes of data. It uses distributed systems like HDFS (Hadoop) and
NoSQL databases (MongoDB, Cassandra) to store structured and unstructured data.
3. Batch Processing
Study4sub
Processes large sets of data in chunks at specific intervals. Used when real-time
processing isn’t necessary. Tools: Hadoop MapReduce, Hive.
4. Real-Time Message Ingestion
Collects data as it’s generated. Allows for immediate processing. Tools: Apache Kafka,
Flume.
5. Stream Processing
Processes real-time data continuously as it comes in. Ideal for immediate insights like
fraud detection. Tools: Apache Storm, Apache Flink.
6. Analytical Data Store
Stores processed data in a format optimized for analysis. Examples: Amazon
Redshift, Google BigQuery.
7. Analysis & Reporting
Tools used for generating reports and dashboards. Helps users make data-driven
decisions. Examples: Tableau, Power BI.
8. Orchestration
Study4sub
• Automates workflows and tasks across different components. Ensures data
processing runs smoothly. Tools: Apache Oozie, Apache Airflow.
Architecture Flow (Text Diagram):
5 Vs of Big Data
The 5 Vs represent the core characteristics that define Big Data:
1. Volume
• Refers to the massive amount of data generated every day.
• Data can range from terabytes (TB) to petabytes (PB).
• Example: Social media platforms generate billions of posts, tweets, and images daily.
2. Velocity
• The speed at which data is generated and processed.
Study4sub

• Data flows in real-time or near real-time, requiring quick analysis.

• Example: Stock market transactions or sensor data from IoT devices.
3. Variety
• Refers to the different types of data.
• Data can be structured (databases), semi-structured (XML, JSON), or unstructured
(videos, emails).
• Example: Social media posts, images, and structured data from databases.
4. Veracity
• The uncertainty of data; how reliable or accurate data is.
• Deals with data quality, consistency, and correctness.
• Example: Fake news or noisy sensor data.
5. Value
• The usefulness of the data in deriving insights.
• The value comes from processing and analyzing
Study4sub
the data to extract meaningful
patterns.
• Example: Business recommendations based on customer behavior analysis.
Big Data Technology Components
1. Cloud Computing
• What it is: Storing and processing data on remote servers via the internet.
• Why it's important: Offers scalability, flexibility, and cost-effectiveness for Big Data.
• Example: Amazon Web Services (AWS), Google Cloud, Microsoft Azure.
2. Machine Learning (ML)
• What it is: A type of AI where machines learn from data to make predictions without
explicit programming.
• Why it's important: ML helps in pattern recognition, predictions, and automating
decisions using Big Data.
• Example: Recommendation systems, fraud detection.
3. Natural Language Processing (NLP)
• What it is: AI technology for understanding and generating human language from text
data.
Study4sub
• Why it's important: NLP analyzes unstructured text data like customer reviews and
social media posts.
• Example: Sentiment analysis, chatbots.
4. Business Intelligence (BI)
• What it is: Tools that help businesses analyze data to make better decisions through
visualizations and reports.
• Why it's important: BI turns Big Data into actionable insights for businesses.
• Example: Tableau, Power BI, Qlik.
Big Data Importance and Applications
Importance of Big Data:
• Informed Decision Making: Big data allows businesses to base decisions on data-driven insights rather than
intuition or assumptions.
• Competitive Advantage: Organizations can analyze trends, customer behavior, and market shifts to stay ahead.
• Efficiency and Cost Reduction: Automating processes and analyzing data can reduce operational costs and
increase productivity.
Applications of Big Data
1. Business Intelligence
1. Helps businesses analyze customer behavior and trends to improve marketing strategies and make data-
driven decisions.
Study4sub
2. Healthcare
1. Analyzes patient data to predict diseases, personalize treatments, and improve diagnostics.
3. Retail & E-commerce
1. Optimizes inventory, personalizes shopping experiences, and improves supply chain management.
4. Finance
1. Detects fraud, assesses credit risk, and analyzes market trends for better financial decision-making.
5. Social Media Analytics
1. Analyzes user data to understand sentiments, trends, and influencer impact.
6. Smart Cities
1. Monitors and manages traffic, energy usage, and pollution to make cities more efficient and sustainable.
1. Security in Big Data
What is it?
• Security in Big Data refers to protecting data from unauthorized access, theft, or
damage.
Why is it important?
• As more data is collected, it becomes a target for hackers and malicious entities.
Protecting it ensures that sensitive information (like customer details, financial
records, etc.) stays safe.
Study4sub
How is it done?
• Encryption: Data is converted into a code that can only be unlocked with a key.
• Access control: Only authorized users are allowed to access sensitive data.
• Firewalls: These act like a shield, blocking unauthorized access to your data.
Example :
• Imagine you have a bank account. Encryption ensures that only you (with your
password) can access your account, while a firewall blocks hackers from trying to
steal your money.
2. Compliance in Big Data
What is it?
• Compliance means following laws and regulations regarding how data should be
collected, stored, and used.
Why is it important?
• Every country has laws to protect people’s data. Companies must follow these laws
to avoid fines and legal issues. For example, businesses must make sure they don’t
misuse customer data.
How is it done?
Study4sub
• Companies follow guidelines like GDPR (General Data Protection Regulation) in
Europe, which gives people control over their data.
• In healthcare, HIPAA (Health Insurance Portability and Accountability Act) ensures
that patient data is handled properly.
Example :
• If a company collects data about its customers, it must get their permission (consent)
to use it. For example, a website asking for your email address must tell you how they
will use it, and you must agree.
3. Auditing and Protection in Big Data
What is it?
• Auditing is tracking who accesses and uses the data, while protection refers to
preventing misuse or data loss.
Why is it important?
• Auditing helps to detect malicious behavior like unauthorized access or data
breaches.
• Protection makes sure that data cannot be tampered with or lost.
Study4sub
How is it done?
• Audit Trails: Companies keep a record of every time someone accesses the data.
• Intrusion Detection Systems: These tools watch out for unusual activities, like
unauthorized access, and alert security teams.
Example:
• If someone tries to break into your bank account, the bank can track the attack and
block it. Similarly, if you use a company's service, they can track who is accessing
your personal details to ensure no one is misusing it.
4. Big Data Privacy
What is it?
• Privacy in Big Data means making sure that personal data (like names, addresses, or
social security numbers) is protected from being shared without consent.
Why is it important?
• People have a right to keep their personal information private, and companies must
ensure they protect this data from exposure or misuse.
Study4sub
How is it done?
• Anonymization: Removing personal identifiers (like names or IDs) from data to keep
it private.
• Encryption: Protecting data by making it unreadable without a special key.
Example:
• When you sign up for a website, your name and email address are personal data. The
company must protect your information and not share it without your permission.
5. Big Data Ethics
What is it?
• Ethics refers to using Big Data in a way that is fair, responsible, and
transparent.
Why is it important?
• Ethical use of Big Data ensures that companies don’t take advantage of people
by using their data for unethical purposes like discrimination or exploitation.
How is it done? Study4sub

• Companies must be transparent about how they use your data.

• Data should not be used to create biased algorithms that unfairly target certain
groups (e.g., biased hiring algorithms).
Example :
• Imagine a company using data to decide which applicants to hire. If the
algorithm is biased, it might unfairly reject people based on gender or
ethnicity. Ethical Big Data ensures that decisions are based on fairness.
Big Data Analytics
What is Big Data Analytics?
• Big Data Analytics refers to the process of examining large and complex datasets (often called Big
Data) to uncover hidden patterns, correlations, trends, and useful business insights.
Why is it important?
• In today’s world, businesses and organizations gather massive amounts of data. Big Data Analytics
helps them to make informed decisions by analyzing data to uncover valuable insights that weren’t
obvious at first glance.
• It’s used to improve efficiency, optimize operations, and predict future trends.
How is Big Data Analytics performed?
Study4sub
Big Data Analytics involves using a combination of advanced technologies, techniques, and tools to
process and analyze large volumes of data. Here’s how it works:
1.Data Collection: Collecting data from various sources like social media, customer interactions, IoT
devices, etc.
2.Data Processing: Using tools to process the data. Since Big Data can be structured, semi-structured,
or unstructured, special tools are used to organize it into usable formats.
3.Data Analysis: Applying various statistical and machine learning algorithms to analyze the data and
uncover patterns or trends.
4.Data Visualization: Presenting the findings in a visual format like graphs, dashboards, and reports so
businesses can understand the results and act on them.
Advantages of Big Data Analytics:
1.Informed Decision-Making:
1. Businesses can make data-driven decisions based on real-time insights.
2. Example: A retail company can analyze customer preferences and adjust its stock accordingly.
2.Improved Customer Insights:
1. Helps businesses understand customer behavior, preferences, and needs.
2. Example: E-commerce companies use Big Data to personalize product recommendations for
customers.
3.Cost Reduction:
1. Big Data can help businesses optimize operations
Study4sub
and reduce waste, leading to cost savings.
2. Example: Predicting maintenance needs of machinery can prevent breakdowns and lower repair
costs.
4.Better Operational Efficiency:
1. Identifies inefficiencies and helps streamline business processes.
2. Example: Manufacturing companies use Big Data to monitor production lines and improve
efficiency.
5.Competitive Advantage:
1. Companies using Big Data can stay ahead of competitors by leveraging insights for better
strategies.
2. Example: Financial institutions use Big Data to detect fraud faster than traditional methods.
Disadvantages of Big Data Analytics:
1.Data Privacy Concerns:
1. Collecting and analyzing large amounts of personal data can lead to privacy issues.
2. Example: If customer data is not handled securely, it can be exploited for malicious purposes.
2.High Costs:
1. The infrastructure and tools required for Big Data Analytics can be expensive.
2. Example: Setting up cloud storage, advanced software, and hiring specialized talent can be costly.
3.Data Complexity:
1. Analyzing unstructured and diverse data (like images, videos, and social media content) can be difficult.
2. Example: A company might struggle to extractStudy4sub
meaningful insights from text-heavy social media data.
4.Lack of Skilled Professionals:
1. The demand for professionals who can handle Big Data is high, and there is often a shortage of qualified
personnel.
2. Example: It may be challenging to find data scientists who are skilled in handling Big Data tools and
algorithms.
5.Data Security Risks:
1. The larger the dataset, the greater the risk of data breaches and hacking attacks.
2. Example: If a healthcare company experiences a data breach, it could compromise sensitive patient
information.
Challenges of Conventional Systems
• Conventional systems are traditional data processing systems that were designed for
small, structured datasets. As data grows and becomes more complex, these systems
face several key challenges:
1. Limited Data Handling Capacity
• Traditional systems struggle to handle massive volumes of data (Big Data).
• Problem: They may slow down, crash, or fail when handling large datasets.
2. Inability to Handle Unstructured DataStudy4sub
• Conventional systems are built to process structured data (e.g., tables in databases).
• Problem: They struggle with unstructured data like text, videos, and social media
posts.
3. Slower Data Processing
• These systems often rely on batch processing, which is slow.
• Problem: Real-time data analysis is difficult, making it hard to make quick decisions.
4. Scalability Issues
• Traditional systems can't easily scale up or expand to handle larger data.
• Problem: As data grows, these systems cannot adapt efficiently.
5. High Costs
• Maintaining conventional systems with physical servers is expensive.
• Problem: Storing and processing large amounts of data becomes costly.
6. Difficulty in Real-Time Decision Making
Study4sub

• These systems can't process data fast enough for real-time decisions.
• Problem: Businesses miss out on opportunities that require immediate action.
7. Limited Flexibility and Integration
• Traditional systems don’t integrate well with modern technologies like cloud or
machine learning.
• Problem: It's hard to use new tools alongside old systems.
8. Data Quality Issues
• Conventional systems struggle with ensuring clean and consistent data.
• Problem: Data errors or inconsistencies can affect decision-making.
Intelligent Data Analysis
Intelligent Data Analysis refers to using advanced techniques, algorithms, and tools to analyze large
datasets and extract meaningful patterns, insights, and predictions. It involves the application of
artificial intelligence (AI), machine learning (ML), and statistical models to make smarter decisions
based on data.
Key Points:
1.AI & Machine Learning: These technologies help in learning from data and predicting future trends
or behaviors without human intervention. Study4sub
2.Pattern Recognition: Intelligent data analysis identifies hidden patterns in data that are not
immediately obvious.
3.Automation: It automates data analysis processes, making it faster and more efficient.
4.Predictive Analytics: It helps forecast future events, trends, or behaviors based on historical data.
5.Real-time Insights: Intelligent analysis can provide real-time insights, helping businesses to make
quicker, more informed decisions.
Example:
• In retail, intelligent data analysis can be used to predict which products will sell best in the future by
analyzing past sales data, customer preferences, and trends.
Nature of Data
Nature of Data refers to the different forms, types, and characteristics of data that affect how
it is stored, processed, and analyzed.
Types of Data:
1.Structured Data:
1. What it is: Data that is organized into tables, rows, and columns, typically in relational databases (e.g.,
customer records, sales data).
2. Example: A database of employee information where each row represents an employee with columns like
name, age, salary, etc.
2.Unstructured Data:
1. What it is: Data that doesn't have a predefined structure, making it difficult to analyze with traditional
Study4sub
methods (e.g., text, images, audio, video).
2. Example: Social media posts, customer reviews, or video files.
3.Semi-structured Data:
1. What it is: Data that doesn't have a rigid structure but contains tags or markers that make it easier to
analyze (e.g., XML, JSON).
2. Example: A log file that contains a mixture of structured data (timestamps) and unstructured data (event
descriptions).
4.Big Data:
1. What it is: Extremely large datasets that require advanced tools and techniques for storage, processing,
and analysis. Big Data is often characterized by the 5 Vs: Volume, Velocity, Variety, Veracity, and Value.
2. Example: Data from IoT sensors, social media platforms, and web logs.
Characteristics of Data:
1.Volume: The amount of data being generated. It can be terabytes or even
petabytes.
2.Velocity: The speed at which data is generated and needs to be processed
(e.g., real-time data).
3.Variety: The different types of data (structured, unstructured, semi-
structured).
Study4sub
4.Veracity: The quality and accuracy of the data.
5.Value: The usefulness of the data for decision-making or gaining insights.
Nature of Data in Big Data:
• Big Data contains data from multiple sources that vary in type, speed, and
structure. Processing this data requires advanced technologies like Hadoop,
Spark, and machine learning to handle its complexity.
Analytic Processes and Tools
Analytic Process
The analytic process in Big Data involves several steps to extract meaningful insights
from large datasets. These steps are essential for data analysis and decision-making.
1.Data Collection:
1. Collect data from various sources such as sensors, databases, social media, and logs.
2. Example: Collecting sales data from e-commerce websites.
2.Data Cleaning:
1. Remove errors, duplicates, and irrelevant Study4sub
information to ensure high-quality data.
2. Example: Removing duplicate customer entries from a database.
3.Data Analysis:
1. Apply statistical methods, machine learning models, and algorithms to analyze the data and
uncover patterns.
2. Example: Analyzing customer behavior patterns using machine learning.
4.Interpretation of Results:
1. After analysis, interpret the results to make informed decisions.
2. Example: Predicting future sales trends based on past data.
Tools : Excel , R and Python, Hadoop and Spark, Tableau/Power BI , SQL
Analysis vs Reporting
While both analysis and reporting involve working with data, they serve different
purposes.
Analysis:
• Goal: To explore data, find patterns, and make predictions.
• Process: Involves using statistical models, machine learning, and algorithms.
• Outcome: Provides insights that can guide strategic decision-making.
• Example: Using customer data to predict future purchase behavior.
Study4sub

Reporting:
• Goal: To present data in a simple, understandable format.
• Process: Involves summarizing data in charts, graphs, and tables.
• Outcome: Provides an overview of performance or trends, typically for monitoring
purposes.
• Example: A monthly sales report showing the total revenue, top-selling products, and
key metrics.
Key Differences:
• Analysis is more about understanding and extracting insights from data, while reporting is about summarizing and
presenting data for easy consumption.
• Analysis typically involves advanced methods, while reporting is more about presenting results in an understandable way.
Modern Data Analytic Tools (Short Notes - AKTU Oriented)
1. Hadoop
1. Open-source framework for storing and processing large data sets in a distributed manner.
2. Handles structured and unstructured data.
2. Apache Spark
1. Fast in-memory data processing tool.
2. Suitable for real-time analytics.
3. Power BI
Study4sub
1. Microsoft’s tool for creating interactive dashboards and reports.
2. Easy to use and integrates with various data sources.
4. Tableau
1. Data visualization tool.
2. Helps in making graphs, charts, and dashboards for better understanding.
5. Python & R
1. Programming languages for data analysis, visualization, and machine learning.
2. Python is widely used due to its simplicity and libraries like Pandas, NumPy.
6. SQL
1. Language used to query and manage structured data in databases.
2. Essential for data extraction and manipulation.
7. Google Analytics
1. Used to track and report website traffic and user behavior.
THANKS FOR WATCHING

BEST OF LUCK
Study4subFOR EXAM