BIG DATA

DEFINITION

Big data is a term that describes the large volume of data – both structured and
unstructured – that inundates a business on a day-to-day basis.

Big data is a term for data sets that are so large or complex that traditional data
processing application software is inadequate to deal with them. Big data challenges
include capturing data, data storage, data analysis, search, sharing, transfer, visualization,
quering, updating and information privacy.

APPLICATIONS OF BIG DATA

1. BANKING AND SECURITIES

Industry-Specific big data challenges

A study of 16 projects in 10 top investment and retail banks  shows that the
challenges in this industry include: securities fraud early warning, tick analytics, card
fraud detection, archival of audit trails, enterprise credit risk reporting, trade
visibility, customer data transformation, social analytics for trading, IT operations
analytics, and IT policy compliance analytics, among others.
Applications of big data in the banking and securities industry
The Securities Exchange Commission (SEC) is using big data to monitor
financial market activity. They are currently using network analytics and natural
language processors to catch illegal trading activity in the financial markets.
Retail traders, Big banks, hedge funds and other so-called ‘big boys’ in the
financial markets use big data for trade analytics used in high frequency trading, pre-
trade decision-support analytics, sentiment measurement, Predictive Analytics etc.
This industry also heavily relies on big data for risk analytics including; anti-
money laundering, demand enterprise risk management, "Know Your Customer", and
fraud mitigation.

2. COMMUNICATIONS, MEDIA AND ENTERTAINMENT

Industry-Specific big data challenges

Since consumers expect rich media on-demand in different formats and in a
variety of devices, some big data challenges in the communications, media and
entertainment industry include:

 Collecting, analyzing, and utilizing consumer insights

 Leveraging mobile and social media content
 Understanding patterns of real-time, media content usage
 Applications of big data in the Communications, media and entertainment
industry
Organizations in this industry simultaneously analyze customer data along with
behavioral data to create detailed customer profiles that can be used to:

 Create content for different target audiences

 Recommend content on demand
 Measure content performance

A case in point is the Wimbledon Championships (YouTube Video) that

leverages big data to deliver detailed sentiment analysis on the tennis matches to TV,
mobile, and web users in real-time.
Spotify, an on-demand music service, uses Hadoop big data analytics, to collect
data from its millions of users worldwide and then uses the analyzed data to give
informed music recommendations to individual users.
Amazon Prime, which is driven to provide a great customer experience by
offering, video, music and Kindle books in a one-stop shop also heavily utilizes big
data.

3. HEALTHCARE PROVIDERS

Industry-Specific challenges
The healthcare sector has access to huge amounts of data but has been plagued
by failures in utilizing the data to curb the cost of rising healthcare and by inefficient
systems that stifle faster and better healthcare benefits across the board.
This is mainly due to the fact that electronic data is unavailable, inadequate, or
unusable. Additionally, the healthcare databases that hold health-related information
have made it difficult to link data that can show patterns useful in the medical field.
Applications of big data in the healthcare sector
Some hospitals, like Beth Israel, are using data collected from a cell phone app,
from millions of patients, to allow doctors to use evidence-based medicine as opposed
to administering several medical/lab tests to all patients who go to the hospital. A
battery of tests can be efficient but they can also be expensive and usually ineffective.
Free public health data and Google Maps have been used by the University of
Florida to create visual data that allows for faster identification and efficient analysis
of healthcare information, used in tracking the spread of chronic disease.
4. EDUCATION

Industry-Specific big data challenges

From a technical point of view, a major challenge in the education industry is
to incorporate big data from different sources and vendors and to utilize it on
platforms that were not designed for the varying data.
From a practical point of view, staff and institutions have to learn the new data
management and analysis tools.
Politically, issues of privacy and personal data protection associated with big
data used for educational purposes is a challenge.
Applications of big data in Education
Big data is used quite significantly in higher education. For example, The
University of Tasmania. An Australian university with over 26000 students, has
deployed a Learning and Management System that tracks among other things, when a
student logs onto the system, how much time is spent on different pages in the system,
as well as the overall progress of a student over time.
On a governmental level, the Office of Educational Technology in the U. S.
Department of Education, is using big data to develop analytics to help course correct
students who are going astray while using online big data courses. Click patterns are
also being used to detect boredom.

5. MANUFACTURING AND NATURAL RESOURCES

Industry-Specific challenges
Increasing demand for natural resources including oil, agricultural products,
minerals, gas, metals, and so on has led to an increase in the volume, complexity, and
velocity of data that is a challenge to handle.
Applications of big data in manufacturing and natural resources
In the natural resources industry, big data allows for predictive modeling to
support decision making that has been utilized to ingest and integrate large amounts of
data from geospatial data, graphical data, text and temporal data. Areas of interest
where this has been used include; seismic interpretation and reservoir characterization.

6. GOVERNMENT

Industry-Specific challenges
In governments the biggest challenges are the integration and interoperability
of big data across different government departments and affiliated organizations.
Applications of big data in Government
In public services, big data has a very wide range of applications including:
energy exploration, financial market analysis, fraud detection, health related research
and environmental protection.
7. INSURANCE

Industry-Specific challenges
Lack of personalized services, lack of personalized pricing and the lack of
targeted services to new segments and to specific market segments are some of the
main challenges.
Applications of big data in the insurance industry
Big data has been used in the industry to provide customer insights for
transparent and simpler products, by analyzing and predicting customer behavior
through data derived from social media, GPS-enabled devices and CCTV footage. The
big data also allows for better customer retention from insurance companies.
When it comes to claims management, predictive analytics from big data has
been used to offer faster service since massive amounts of data can be analyzed
especially in the underwriting stage. Fraud detection has also been enhanced.
Through massive data from digital channels and social media, real-time
monitoring of claims throughout the claims cycle has been used to provide insights.

8. RETAIL AND WHOLE SALE TRADE

Industry-Specific challenges
From traditional brick and mortar retailers and wholesalers to current day e-
commerce traders, the industry has gathered a lot of data over time. This data, derived
from customer loyalty cards, POS scanners, RFID etc. is not being used enough to
improve customer experiences on the whole. Any changes and improvements made
have been quite slow.
Applications of big data in the Retail and Wholesale industry
Big data from customer loyalty data, POS, store inventory, local demographics
data continues to be gathered by retail and wholesale stores.
In New York’s Big Show retail trade conference in 2014, companies like Microsoft,
Cisco and IBM pitched the need for the retail industry to utilize big data for analytics
and for other uses including:

 Optimized staffing through data from shopping patterns, local events,

and so on
 Reduced fraud
 Timely analysis of inventory

Social media use also has a lot of potential use and continues to be slowly but surely
adopted especially by brick and mortar stores. Social media is used for customer
prospecting, customer retention, promotion of products, and more.
9. TRANSPORTATION

Industry-Specific challenges
In recent times, huge amounts of data from location-based social networks and
high speed data from telecoms have affected travel behavior. Regrettably, research to
understand travel behavior has not progressed as quickly.
In most places, transport demand models are still based on poorly understood
new social media structures.
Applications of big data in the transportation industry
Some applications of big data by governments, private organizations and
individuals include:

 Governments use of big data: traffic control, route planning, intelligent

transport systems, congestion management (by predicting traffic conditions)
 Private sector use of big data in transport: revenue management, technological
enhancements, logistics and for competitive advantage (by consolidating
shipments and optimizing freight movement)
 Individual use of big data includes: route planning to save on fuel and time, for
travel arrangements in tourism etc.

10. ENERGY AND UTILITIES

Applications of big data in the energy and utilities industry

Smart meter readers allow data to be collected almost every 15 minutes as
opposed to once a day with the old meter readers. This granular data is being used to
analyze consumption of utilities better which allows for improved customer feedback
and better control of utilities use.
 DATA MINING VS BIG DATA

Big data and data mining are completely different concepts. However, both concepts
involve the use of large data sets to handle the collection or reporting of data that helps
businesses or clients make better decisions. However, the two concepts are used in two
different elements of this operation.

The term big data can be defined simply as large data sets that outgrow simple
databases and data handling architectures. For example, data that cannot be easily handled in
Excel spreadsheets may be referred to as big data.

Data mining relates to the process of going through large sets of data to identify
relevant or pertinent information. Businesses often collect large data sets that may be
automatically collected. However, decision makers need access to smaller, more specific
pieces of data and use data mining to identify specific data that may help their businesses
make better leadership and management decisions.

Various software packages and analytical tools can be used for data mining. The
process can be automated or be done manually. Data mining allows individual workers to
send specific queries for information to archives and databases so that they can obtain
targeted or specific results.