Introduction to Big Data

Topics
• Scope: Big Data & Analytics
• Topics:
– Foundation of Data Analytics and Data Mining
– Hadoop/Map-Reduce Programming and Data
Processing & BigTable/Hbase/Cassandra
– Graph Database and Graph Analytics

2
 What’s Big Data?
No single definition:
• Big data is the term for a collection of data sets so large and
complex that it becomes difficult to process using on-hand
database management tools or traditional data processing
applications.
• The challenges include capture, curation, storage, search,
sharing, transfer, analysis, and visualization.
• The trend to larger data sets is due to the additional
information derivable from analysis of a single large set of
related data, as compared to separate smaller sets with the
same total amount of data, allowing correlations to be found
to "spot business trends, determine quality of research,
prevent diseases, link legal citations, combat crime, and
determine real-time roadway traffic conditions.”

3
Big Data: 3V’s

4
 Volume (Scale)

• Data Volume
– 44x increase from 2009 2020
– From 0.8 zettabytes to 35zb
• Data volume is increasing exponentially

Exponential increase in
collected/generated data

5
 4.6
30 billion RFID billion
tags today
12+ TBs (1.3B in 2005)
camera
of tweet data phones
every day world wide

100s of
millions
data every day

of GPS
? TBs of

enabled
devices sold
annually

25+ TBs of 2+
log data
every day billion
people on
the Web
76 million smart meters by end
in 2009… 2011
200M by 2014
 CERN’s Large Hydron Collider (LHC) generates 15 PB a year
Maximilien Brice, © CERN
 The Earthscope
• The Earthscope is the world's
largest science project. Designed to
track North America's geological
evolution, this observatory records
data over 3.8 million square miles,
amassing 67 terabytes of data. It
analyzes seismic slips in the San
Andreas fault, sure, but also the
plume of magma underneath
Yellowstone and much, much more.
(http://www.msnbc.msn.com/id/44
363598/ns/technology_and_science
-future_of_technology/#.TmetOdQ-
-uI)
 Variety (Complexity)
• Relational Data (Tables/Transaction/Legacy Data)
• Text Data (Web)
• Semi-structured Data (XML)
• Graph Data
– Social Network, Semantic Web (RDF), …

• Streaming Data
– You can only scan the data once

• A single application can be generating/collecting

many types of data

• Big Public Data (online, weather, finance, etc)

To extract knowledge➔ all these types of

data need to linked together 9
 A Single View to the Customer

Social Banking
Media Finance

Our
Gaming
Customer Known
History

Purchas
Entertain
e
 Velocity (Speed)

• Data is begin generated fast and need to be

processed fast
• Online Data Analytics
• Late decisions ➔ missing opportunities
• Examples
– E-Promotions: Based on your current location, your purchase history,
what you like ➔ send promotions right now for store next to you

– Healthcare monitoring: sensors monitoring your activities and body ➔

any abnormal measurements require immediate reaction

11
 Real-time/Fast Data

Mobile devices
(tracking all objects all the time)

Social media and networks Scientific instruments

(all of us are generating data) (collecting all sorts of data)

Sensor technology and networks

(measuring all kinds of data)

• The progress and innovation is no longer hindered by the ability to collect data
• But, by the ability to manage, analyze, summarize, visualize, and discover
knowledge from the collected data in a timely manner and in a scalable fashion

12
 Real-Time Analytics/Decision Requirement

Product
Recommendations Learning why Customers
Influence
that are Relevant Behavior Switch to competitors
& Compelling and their offers; in
time to Counter

Friend Invitations
Improving the Customer to join a
Marketing Game or Activity
Effectiveness of a that expands
Promotion while it business
is still in Play
Preventing Fraud
as it is Occurring
& preventing more
proactively
Some Make it 4V’s

14
 Harnessing Big Data

• OLTP: Online Transaction Processing (DBMSs)

• OLAP: Online Analytical Processing (Data Warehousing)
• RTAP: Real-Time Analytics Processing (Big Data Architecture & technology)

15
 The Model Has Changed…
• The Model of Generating/Consuming Data has Changed

Old Model: Few companies are generating data, all others are consuming data

New Model: all of us are generating data, and all of us are consuming data

16
What’s driving Big Data

- Optimizations and predictive analytics

- Complex statistical analysis
- All types of data, and many sources
- Very large datasets
- More of a real-time

- Ad-hoc querying and reporting

- Data mining techniques
- Structured data, typical sources
- Small to mid-size datasets

17