Big Data Analytics(BDA)

JIT #CMIT-5125

Unit-3
Big Data
Analytics
Admas Abtew
Faculty of Computing and Informatics
Jimma Technology Institute, Jimma University
Admas.abtew@ju.edu.et
+251-912499102
 Outline
Looping
• An overview of Big Data Analytics
• Big Data Analytics Techniques
• Graph Database and Analytics
• Big Data Taxonomy
Chapter Three: Big Data
Analytics
3.1 An overview of Big Data Analytics

 Big Data Analytics refers to the process of extracting valuable insights, patterns, and trends
from large and complex datasets, known as Big Data. It involves using advanced analytical
techniques and technologies to analyze vast volumes of structured, semi-structured, and
unstructured data to uncover hidden patterns, make predictions, and drive informed decision-
making.
 The process of converting large amounts of unstructured raw data, retrieved from different
sources to a data product useful for organizations forms the core of Big Data Analytics.

#(BDA)  Unit:3 –Big Data Analytics 4

Big Data Analytics - Data Life Cycle
Traditional Data Mining Life Cycle
In order to provide a framework to organize the work needed by an organization and deliver
clear insights from Big Data, it’s useful to think of it as a cycle with different stages. It is by
no means linear, meaning all the stages are related with each other. This cycle has
superficial similarities with the more traditional data mining cycle as described in CRISP
methodology.
CRISP-DM Methodology
The CRISP-DM methodology that stands for Cross Industry Standard Process for Data
Mining, is a cycle that describes commonly used approaches that data mining experts use to
tackle problems in traditional BI data mining. It is still being used in traditional BI data
mining teams.
SEMMA Methodology

#(BDA)  Unit:3 –Big Data Analytics 5

Traditional Data Mining Life Cycles…..
The stages involved in the CRISP-DM life cycle
 Business Understanding(What does the business need?) −
This initial phase focuses on understanding the project
objectives and requirements from a business perspective, and
then converting this knowledge into a data mining problem
definition.
 A preliminary plan is designed to achieve the objectives. A
decision model, especially one built using the Decision
Model and Notation standard can be used.
 Data Understanding(What data do we have / need? Is it
clean?) − This phase starts with an initial data collection and
proceeds with activities in order to get familiar with the data,
to identify data quality problems, to discover first insights
into the data, or to detect interesting subsets to form
hypotheses for hidden information.

#(BDA)  Unit:3 –Big Data Analytics 6

Traditional Data Mining Life Cycles…..
 Data Preparation(How do we organize the data for modeling?) − This phase covers all activities to
construct the final dataset (data that will be fed into the modeling tool(s)) from the initial raw data.
 Data preparation tasks are likely to be performed multiple times, and not in any prescribed order. Tasks
include table, record, and attribute selection as well as transformation and cleaning of data for modeling
tools.
 Modeling(What modeling techniques should we apply?) − In this phase, various modeling techniques
are selected and applied and their parameters are calibrated to optimal values. Typically, there are several
techniques for the same data mining problem type. Some techniques have specific requirements on the form
of data. Therefore, it is often required to step back to the data preparation phase.
 Evaluation(Which model best meets the business objectives?) − At this stage in the project, you have
built a model (or models) that appears to have high quality, from a data analysis perspective. Before
proceeding to final deployment of the model, it is important to evaluate the model thoroughly and review
the steps executed to construct the model, to be certain it properly achieves the business objectives.
 A key objective is to determine if there is some important business issue that has not been sufficiently
considered. At the end of this phase, a decision on the use of the data mining results should be reached.

#(BDA)  Unit:3 –Big Data Analytics 7

Traditional Data Mining Life Cycles…..
 Deployment(How do stakeholders access the results?) − Creation of the model is
generally not the end of the project. Even if the purpose of the model is to increase
knowledge of the data, the knowledge gained will need to be organized and presented
in a way that is useful to the customer.
 Depending on the requirements, the deployment phase can be as simple as
generating a report or as complex as implementing a repeatable data scoring (e.g.
segment allocation) or data mining process.
 In many cases, it will be the customer, not the data analyst, who will carry out the
deployment steps. Even if the analyst deploys the model, it is important for the
customer to understand upfront the actions which will need to be carried out in order
to actually make use of the created models.

#(BDA)  Unit:3 –Big Data Analytics 8

Traditional Data Mining Life Cycles…..
SEMMA Methodology
SEMMA is another methodology developed by SAS
for data mining modeling. It stands
for Sample, Explore, Modify, Model, and Asses. Here
is a brief description of its stages
 Sample − The process starts with data
sampling, e.g., selecting the dataset for
modeling. The dataset should be large
enough to contain sufficient information to
retrieve, yet small enough to be used
efficiently. This phase also deals with data
partitioning.
 Explore − This phase covers the
understanding of the data by discovering
anticipated and unanticipated relationships
between the variables, and also
abnormalities, with the help of data
visualization.
#(BDA)  Unit:3 –Big Data Analytics 9
Traditional Data Mining Life Cycles…..
 Modify − The Modify phase contains methods to select, create and transform variables in
preparation for data modeling.
 Model − In the Model phase, the focus is on applying various modeling (data mining)
techniques on the prepared variables in order to create models that possibly provide the
desired outcome.
 Assess − The evaluation of the modeling results shows the reliability and usefulness of
the created models.
 The main difference between CRISM–DM and SEMMA is that SEMMA focuses on the
modeling aspect, whereas CRISP-DM gives more importance to stages of the cycle prior to
modeling such as understanding the business problem to be solved, understanding and
preprocessing the data to be used as input, for example, machine learning algorithms.

#(BDA)  Unit:3 –Big Data Analytics 10

Big Data Life Cycle
In today’s big data context, the previous approaches are either
incomplete or suboptimal. For example, the SEMMA methodology
disregards completely data collection and preprocessing of different
data sources. These stages normally constitute most of the work in a
successful big data project.
A big data analytics cycle can be described by the following stage
 Business Problem Definition
 Research
 Data Acquisition
 Data Extraction and cleansing
 Data Storage
 Data Analysis
 Modeling and Assessment
 Implementation

#(BDA)  Unit:3 –Big Data Analytics 11

Business Problem Definition
This is a point common in traditional BI and big data analytics life cycle. Normally it is a
non-trivial stage of a big data project to define the problem and evaluate correctly how much
potential gain it may have for an organization. It seems obvious to mention this, but it has to
be evaluated what are the expected gains and costs of the project.
Research
Analyze what other companies have done in the same situation. This involves looking for
solutions that are reasonable for your company, even though it involves adapting other
solutions to the resources and requirements that your company has. In this stage, a
methodology for the future stages should be defined.

#(BDA)  Unit:3 –Big Data Analytics 12

 Data acquisition
This section is key in a big data life cycle; it defines which type of profiles would be needed to
deliver the resultant data product. Data gathering is a non-trivial step of the process; it normally
involves gathering unstructured data from different sources. To give an example, it could involve
writing a crawler to retrieve reviews from a website. This involves dealing with text, perhaps in
different languages normally requiring a significant amount of time to be completed.
 Data Extraction and cleansing
Now the data is filtered, but there might be a possibility that some of the entries of the data
might be incompatible, to rectify this issue, a separate phase is created, known as the data
extraction phase. In this phase, the data, which don’t match with the underlying scope of the
analysis, are extracted and transformed in such a form.

#(BDA)  Unit:3 –Big Data Analytics 13

Data Storage
Once the data is processed, it sometimes needs to be stored in a database. Big data
technologies offer plenty of alternatives regarding this point. The most common alternative is
using the Hadoop File System for storage that provides users a limited version of SQL, known
as HIVE Query Language. This allows most analytics task to be done in similar ways as would
be done in traditional BI data warehouses, from the user perspective. Other storage options to
be considered are MongoDB, Redis, and SPARK.
Exploratory Data Analysis
Once the data has been cleaned and stored in a way that insights can be retrieved from it, the
data exploration phase is mandatory. The objective of this stage is to understand the data, this
is normally done with statistical techniques and also plotting the data. This is a good stage to
evaluate whether the problem definition makes sense or is feasible.

#(BDA)  Unit:3 –Big Data Analytics 14

Modeling and Assessment
This stage involves reshaping the cleaned data retrieved previously and using statistical
preprocessing for missing values imputation, outlier detection, normalization, feature
extraction and feature selection.
Modelling
The prior stage should have produced several datasets for training and testing, for example, a
predictive model. This stage involves trying different models and looking forward to solving
the business problem at hand. In practice, it is normally desired that the model would give
some insight into the business. Finally, the best model or combination of models is selected
evaluating its performance on a left-out dataset.

#(BDA)  Unit:3 –Big Data Analytics 15

Implementation
In this stage, the data product developed is implemented in the data pipeline of the company.
This involves setting up a validation scheme while the data product is working, in order to track
its performance. For example, in the case of implementing a predictive model, this stage would
involve applying the model to new data and once the response is available, evaluate the model.

#(BDA)  Unit:3 –Big Data Analytics 16

Types of Analytics
 There are three types of analytics:
Descriptive analytics is the analysis of historical data to determine what happened, what
changed, and what patterns can be identified. Descriptive analytics involves the breaking down
of big data into smaller chunks of usable information so that companies can understand what
happened with a specific operation, process, or set of transactions.
Descriptive analytics can provide insight into current customer behaviors and operational
trends to support decisions about resource allocations, process improvements, and overall
performance management. Most industry observers believe it represents most of the analytics
used by companies today.

#(BDA)  Unit:3 –Big Data Analytics 17

Key Facts of Descriptive Analytics
The following points are showing the key facts of Descriptive analytics -
 It is the most fundamental and widely used sort of analytics in businesses. It summarizes
and draws attention to patterns in both current and historical data sources.
 It is used to generate reports, key performance indicators (KPIs), and business metrics
that allow businesses to track and analyze performance and other patterns.
 It assists firms in understanding what has occurred so far in their operations. When
descriptive analytics and diagnostic, predictive, and prescriptive analytics are used
together, firms can better explain why something happened while also predicting
probable future outcomes and actionable insights.
 It presents crucial information in a simple and easy-to-understand manner. In the future,
descriptive analytics will continue to be required. On the other hand, more work is being
directed towards other domains of analytics, such as predictive analytics and prescriptive
analytics.

#(BDA)  Unit:3 –Big Data Analytics 18

The Procedure For Using Descriptive Analytics
 Data aggregation and data mining are two strategies that are used in descriptive analytics
to obtain historical information about a given set of variables. Before any analysis can
begin, data is gathered and sorted according to data aggregation in order to make the
datasets more manageable for analysts.
 Data mining is the next phase in the analysis process, and it entails searching through large
amounts of data in order to find patterns and meaning. The patterns that have been identified
are next evaluated in order to determine the exact methods in which learners engaged with
the learning content and within the learning environment.

#(BDA)  Unit:3 –Big Data Analytics 19

What Is Prescriptive Analytics
Prescriptive analytics is concerned with determining the most appropriate course of action in
a particular situation based on the information provided. It is related to both descriptive analytics
and predictive analytics, but it places a greater emphasis on actionable insights rather than data
monitoring and analysis.
In the field of data analytics, predictive analytics is a sort of data analysis that involves the
use of technology to assist organizations in making better decisions based on the examination of
raw data.
Prescriptive analytics, in particular, takes into account information about probable events or
scenarios, available resources, prior performance, and current performance, and then
recommends a course of action or strategy. The method can be used to make judgments across
any time horizon, from the immediate to the long term.

#(BDA)  Unit:3 –Big Data Analytics 20

How Prescriptive Analytics Works?
In order to be effective, predictive analytics must rely on artificial intelligence techniques such as machine
learning—the ability of a computer program, without the assistance of a person, to understand and progress from
the data it collects, while adjusting as it does so. With the help of machine learning, it is now possible to process the
massive amount of data that exists today.
Prescriptive analytics is embedded into the majority of modern business intelligence solutions, allowing users to
receive actionable insights that enable them to make better decisions. In the oil and gas industry, one of the more
fascinating applications of prescriptive analytics is in the control of oil and gas prices, which are continually
fluctuating due to constantly changing political, environmental, and demand variables.
It is possible for a wide range of data-intensive enterprises and government agencies to benefit from the use of
prescriptive analytics, including those in the financial services and health care sectors, where the cost of human
error is particularly high.
When used in conjunction with another sort of data analytics, such as predictive analytics, which involves the use
of statistics and modeling to forecast future performance based on current and historical data, prescriptive analytics
can be very effective. However, it goes even beyond that: Based on the predictive analytics' prediction of what is
likely to happen, it makes recommendations on how to proceed in the future.

#(BDA)  Unit:3 –Big Data Analytics 21

What Is Predictive Analytics?
Predictive analytics is the process of using data analytics to make predictions based on
data. This process uses data along with analysis, statistics, and machine learning
techniques to create a predictive model for forecasting future events.
Predictive analytics refers to the use of statistics and modeling techniques to make
predictions about future outcomes and performance. Predictive analytics looks at current
and historical data patterns to determine if those patterns are likely to emerge again.
Typically, historical data is used to build a mathematical model that captures important
trends. That predictive model is then used on current data to predict what will happen next
or to suggest actions to take for optimal outcomes.
Overall, predictive analytics is the use of data, statistical algorithms, and machine
learning techniques to identify the likelihood of future outcomes based on historical data.
The goal is to go beyond knowing what has happened to provide the best assessment of
what will happen in the future.

#(BDA)  Unit:3 –Big Data Analytics 22

Why Predictive Analytics Is Important?
Predictive analytics is often discussed in the context of big data, engineering data, for
example, comes from sensors, instruments, and connected systems out in the world.
Business system data at a company might include transaction data, sales results, customer
complaints, and marketing information. Increasingly, businesses make data-driven
decisions based on this valuable trove of information. With increased competition,
businesses seek an edge in bringing products and services to crowded markets. Data-driven
predictive models can help companies solve long-standing problems in new ways.

#(BDA)  Unit:3 –Big Data Analytics 23

How to Use Predictive Analytics?
Predictive analytics techniques can broadly be classified as regression techniques or machine learning techniques.
Regression Techniques
Regression models focus on establishing a mathematical equation as a method to represent the interactions
between the different variables. Predictive analytics software relies heavily on a wide variety of regression
models, including linear regression models, discrete choice models, logistic regression, time series models,
survival or duration analysis, and decision tree learning.
Machine Learning
Machine learning predictive analytics is a category of algorithm that can receive input data and use statistical
analysis to predict outputs while updating outputs as new data becomes available. This allows software
applications to become more accurate in predicting outcomes without being explicitly programmed. Examples of
machine learning techniques include neural networks, multilayer perceptron, radial basis functions, support
vector machines, Naïve Bayes, and geospatial predictive modeling.
Deep Learning
Deep Learning techniques are in trend nowadays due to their capability to produce fast and accurate results.
Many researchers have developed methods for predictive analytics in the area of data science. As deep learning
techniques have an advantage over conventional learning techniques, these techniques are in demand in finding
insights into the data.
#(BDA)  Unit:3 –Big Data Analytics 24
1. Advanced Analytics Techniques: Big Data Analytics employs various advanced
analytical techniques, including:
 Data Mining: Data mining involves discovering patterns and relationships in large
datasets to extract valuable insights and make predictions.
 Machine Learning: Machine learning algorithms enable computers to learn from data
and make predictions or take actions without being explicitly programmed.
 Text Mining and Natural Language Processing (NLP): Text mining and NLP
techniques are used to extract information and insights from unstructured text data,
such as customer reviews, social media posts, and emails.
 Sentiment Analysis: Sentiment analysis involves determining the sentiment or
opinion expressed in text data, helping organizations understand customer feedback,
public perception, and brand reputation.
 Social Network Analysis: Social network analysis analyzes relationships and
interactions within social networks to uncover influential nodes, communities, and
patterns of information flow.
#(BDA)  Unit:3 –Big Data Analytics 25
1. Technologies for Big Data Analytics: To handle the volume, velocity, and variety of Big
Data, specialized technologies are used, including:
 Distributed Computing: Distributed computing frameworks like Apache Hadoop and
Spark distribute data and processing across clusters of computers, enabling parallel
processing and scalability.
 NoSQL Databases: NoSQL databases like MongoDB and Cassandra provide flexible
and scalable storage solutions for handling large volumes of unstructured and semi-
structured data.
 Data Visualization Tools: Data visualization tools help in presenting complex data in a
visual and intuitive manner, facilitating understanding and exploration of insights.
#(BDA)  Unit:3 –Big Data Analytics 26
1. Business Applications of Big Data Analytics: Big Data Analytics finds applications across
industries and departments, such as:
 Customer Analytics: Analyzing customer data to understand behavior, preferences, and
sentiment, and personalize marketing efforts.
 Risk and Fraud Analytics: Detecting anomalies, identifying potential risks, and
preventing fraudulent activities.
 Supply Chain Optimization: Analyzing supply chain data to optimize inventory
management, logistics, and demand forecasting.
 Healthcare Analytics: Analyzing patient data, medical records, and clinical research to
improve healthcare outcomes and personalize treatment plans.
 Smart Cities and IoT Analytics: Analyzing data from sensors and connected devices to
optimize urban infrastructure, energy consumption, and public services.

#(BDA)  Unit:3 –Big Data Analytics 27

Big Data Analytics provides organizations with the ability to extract meaningful insights and
drive data-driven decision-making. It enables businesses to uncover hidden patterns, gain a
competitive edge, and capitalize on the vast amounts of data generated in today's digital age.

#(BDA)  Unit:3 –Big Data Analytics 28

3.2 Big Data Analytics Techniques:

Big Data Analytics involves the application of various techniques and methodologies to extract
valuable insights from large and complex datasets. These techniques help organizations uncover patterns,
trends, and relationships within the data, enabling data-driven decision-making. Here are some key Big
Data Analytics techniques:
1. Data Mining: Data mining is the process of discovering patterns, relationships, and trends in large
datasets. It involves applying statistical and machine learning algorithms to identify hidden patterns
and make predictions. Data mining techniques include clustering, classification, association rule
mining, and anomaly detection.
2. Machine Learning: Machine Learning (ML) algorithms enable computers to learn from data and
make predictions or take actions without being explicitly programmed. ML techniques are used in Big
Data Analytics to identify patterns, build predictive models, and automate decision-making.
#(BDA)  Unit:3 –Big Data Analytics 29
3. Natural Language Processing (NLP): Natural Language Processing techniques are
used to analyze and understand human language data, such as text documents, social
media posts, and customer reviews. NLP algorithms enable sentiment analysis, topic
modeling, text classification, entity recognition, and language translation. These
techniques help organizations gain insights from unstructured text data.
4. Sentiment Analysis: Sentiment analysis, also known as opinion mining, involves
determining the sentiment or opinion expressed in text data. It helps organizations
understand customer feedback, public perception, and brand reputation. Sentiment
analysis techniques use NLP algorithms to classify text as positive, negative, or neutral,
and quantify sentiment scores.
#(BDA)  Unit:3 –Big Data Analytics 30
5. Social Network Analysis: Social Network Analysis (SNA) examines the relationships and
interactions within social networks. It helps uncover influential nodes, identify communities, and
analyze patterns of information flow. SNA techniques include measuring centrality, detecting
communities, and visualizing network structures. Organizations use SNA to understand social
dynamics, influence networks, and target influential individuals.
6. Time Series Analysis: Time Series Analysis involves analyzing data collected at regular intervals
over time. It helps identify trends, seasonality, and patterns in time-dependent data. Time series
analysis techniques include moving averages, exponential smoothing, autoregressive integrated
moving average (ARIMA) models, and forecasting methods. Time series analysis is commonly used
in demand forecasting, financial analysis, and predictive maintenance.

#(BDA)  Unit:3 –Big Data Analytics 31

7. Predictive Analytics: Predictive Analytics uses historical data and statistical modeling
techniques to make predictions about future events or outcomes. It helps organizations
anticipate customer behavior, forecast demand, identify potential risks, and optimize
business processes. Predictive analytics techniques include regression analysis, decision
trees, logistic regression, and time series forecasting.
8. Text Mining: Text Mining techniques involve extracting insights and meaning from
unstructured text data. It includes tasks such as information extraction, text categorization,
topic modeling, and named entity recognition. Text mining enables organizations to analyze
large volumes of text data, such as customer reviews, survey responses, and social media
posts, to gain valuable insights.
#(BDA)  Unit:3 –Big Data Analytics 32
9. Data Visualization: Data Visualization techniques help present complex data in a visual
and intuitive manner. Visualizations include charts, graphs, maps, and interactive
dashboards. Data visualization enhances data exploration, understanding, and
communication. It enables users to identify patterns, spot trends, and make informed
decisions based on visual insights.
10.Optimization Algorithms: Optimization algorithms are used to find the best solutions to
complex problems with multiple constraints. These algorithms help optimize business
processes, resource allocation, and decision-making. Examples include linear
programming, integer programming, genetic algorithms, and simulated annealing.

#(BDA)  Unit:3 –Big Data Analytics 33

 These are just some of the key techniques used in Big Data Analytics. The choice of
techniques depends on the nature of the data, the problem at hand, and the desired insights.
Organizations leverage these techniques to unlock the value hidden within large and diverse
datasets, enabling data-driven decision-making and gaining a competitive advantage.

#(BDA)  Unit:3 –Big Data Analytics 34

3.3 Graph Database and Analytics

A graph database is a specialized type of database that is designed to store and manage
data using graph structures. It represents data as nodes (vertices) and relationships (edges)
between those nodes. Graph databases excel at capturing and representing complex
relationships, making them ideal for scenarios where relationships and connections between
entities are crucial. Here are some key points about graph databases and their use in analytics:
1. Graph Data Model: In a graph database, data is modeled using nodes, edges, and
properties. Nodes represent entities, such as people, products, or locations, while edges
define the relationships between those entities. Properties provide additional information
about nodes and edges. This flexible and intuitive data model allows for the representation
of complex, interconnected data structures.
#(BDA)  Unit:3 –Big Data Analytics 35
2. Relationship-Focused: Graph databases excel at managing and analyzing relationships
between entities. Relationships can have properties, directionality, and various types,
enabling rich and expressive modeling. This makes graph databases particularly well-
suited for scenarios such as social networks, recommendation systems, fraud detection,
network analysis, and knowledge graphs.
3. Traversal and Querying: Graph databases provide powerful traversal and querying
capabilities to navigate and analyze the data. Traversal allows users to traverse the graph
along paths, exploring relationships and identifying patterns. Query languages, such as
Cypher (used in Neo4j) and Gremlin, enable users to query and retrieve data based on
relationships, properties, and patterns.
#(BDA)  Unit:3 –Big Data Analytics 36
4. Graph Analytics: Graph analytics is the process of analyzing graph-structured data to extract
meaningful insights. It involves applying algorithms and techniques to uncover patterns, identify
communities, calculate centrality measures, detect anomalies, and perform pathfinding. Graph
analytics helps reveal hidden relationships, influence networks, and provides contextual insights
that are not easily discovered with traditional database models.

#(BDA)  Unit:3 –Big Data Analytics 37

5. Use Cases: Graph databases and analytics find applications in various domains,
including:
 Social Network Analysis: Analyzing social networks to understand influence,
community structures, and information flow.
 Recommendations: Personalizing recommendations by analyzing user
preferences, connections, and similarity.
 Fraud Detection: Identifying patterns and anomalies in interconnected data to
detect fraudulent activities.
 Knowledge Graphs: Building knowledge graphs to organize and connect
structured and unstructured data for semantic search and knowledge discovery.
 Supply Chain Optimization: Analyzing supply chain networks to optimize
logistics, identify bottlenecks, and improve efficiency.
 IoT and Network Analysis: Analyzing sensor data and network connections to
optimize performance, identify vulnerabilities, and improve network efficiency.

#(BDA)  Unit:3 –Big Data Analytics 38

6. Scalability and Performance: Graph databases are designed to handle large-scale, interconnected
datasets efficiently. They provide high-performance query execution and scalability by leveraging
indexing, caching, and parallel processing techniques. Additionally, graph databases can take
advantage of distributed computing frameworks to handle massive graphs and perform distributed
graph analytics.
7. Integration with Other Technologies: Graph databases can be integrated with other technologies and
data sources. They can be combined with traditional databases, data lakes, and data warehouses to
provide a comprehensive view of the data landscape. Integration with analytics and visualization tools
allows users to derive insights and present the results effectively.

#(BDA)  Unit:3 –Big Data Analytics 39

 Graph databases and analytics offer a unique and powerful approach to managing and
analyzing interconnected data. They excel at capturing relationships, uncovering patterns, and
providing context-rich insights. As organizations continue to deal with increasingly complex
and interconnected data, graph databases and analytics will play a crucial role in enabling
advanced data-driven decision-making and unlocking valuable insights.

#(BDA)  Unit:3 –Big Data Analytics 40