BIG DATA/DATA LAKE/DATA WAREHOUSE / AI / MACHINE LEARNING

Definition Big Data

Big data refers to the large, diverse sets of information that grow at ever-increasing rates. It
encompasses the volume of information, the velocity or speed at which it is created and collected,
and the variety or scope of the data points being covered. Big data often comes from multiple
sources and arrives in multiple formats.

Big data can be categorised as unstructured or structured.

Structured data consists of information already managed by the organisation in databases and
spreadsheets; it is frequently numeric in nature.

Unstructured data is information that is unorganized and does not fall into a pre-determined model
or format. It includes data gathered from social media sources, which help institutions gather
information on customer needs.

What is structured data?

Structured data is most often categorized as quantitative data, and it's the type of data most of us
are used to working with. Think of data that fits neatly within fixed fields and columns in relational
databases and spreadsheets.

Examples of structured data include names, dates, addresses, credit card numbers, stock
information, geolocation, and more.

Structured data is highly organized and easily understood by machine language. Those working
within relational databases can input, search, and manipulate structured data relatively quickly using
a relational database management system (RDBMS). This is the most attractive feature of structured
data.

What is unstructured data?

Unstructured data is most often categorized as qualitative data, and it cannot be processed and
analyzed using conventional data tools and methods.

Examples of unstructured data include text, video files, audio files, mobile activity, social media
posts, satellite imagery, surveillance imagery – the list goes on and on.

Unstructured data is difficult to deconstruct because it has no predefined data model, meaning it
cannot be organized in relational databases. Instead, non-relational or NoSQL databases are the best
fit for managing unstructured data.

Another way to manage unstructured data is to have it flow into a data lake, allowing it to be in its
raw, unstructured format.

For example, data mining techniques applied to unstructured data can help companies learn
customer buying habits and timing, patterns in purchases, sentiment toward a specific product, and
much more.

Semi-structured data
Semi-structured data is a type of structured data that lies midway between structured and
unstructured data. It doesn't have a specific relational or tabular data model but includes tags and
semantic markers that scale data into records and fields in a dataset.

Common examples of semi-structured data are JSON and XML. Semi-structured data is more
complex than structured data but less complex than unstructured data. It's also relatively easy to
store than unstructured data and bridges the gap between the two data types.

Metadata - the master data

Metadata is often used in big data analysis and is a master dataset that describes other data types. It
has preset fields that contain additional information about a specific dataset.

Metadata has a defined structure identified by a metadata markup schema that includes metadata
models and metadata standards. It contains valuable details to help users better analyze a data item
and make informed decisions.

For example, an online article can display metadata such as a headline, a snippet, a featured image,
image alt-text, slug, and other related information. This information helps differentiate one piece of
content from several other similar pieces of content on the web. Metadata is, therefore, a handy set
of data that acts as the brain for all different types of data.

The future of data

The volume of big data is continuing to rise, but soon, the importance of big data storage will cease
to exist.

Regardless if data is structured or unstructured, having the most accurate and relevant data sources
at hand will be key for companies looking to gain an advantage over their competitors.

Utilizing the right data management will allow companies to:

• Reduce operational costs

Big data is nothing but the collection of unstructured data. This data is not in a particular format and
because of this its datasets sizes are generally huge — measuring tens of terabytes — and
sometimes crossing the threshold of petabytes. The term big data was preceded by very large
databases (VLDBs) which were managed using database management systems (DBMS).

Big data analytics is the process of examining large and varied data sets — i.e., big data — to uncover
hidden patterns, unknown correlations, market trends, customer preferences and other useful
information that can help organizations make more-informed business decisions. Driven by
specialized analytics systems and software, big data analytics can point the way to various business
benefits, including new revenue opportunities, more effective marketing, better customer service,
improved operational efficiency and competitive advantages over rivals.

The top 10 sectors that could benefit the most if they make use of Big data analytics:
(1) Banking and Securities : For monitoring financial markets through network activity monitors
and natural language processors to reduce fraudulent transactions. Exchange Commissions
or Trading Commissions are using big data analytics to ensure that no illegal trading happens
by monitoring the stock market.
(2) Communications and Media: For real-time reportage of events around the globe on several
platforms (mobile, web and TV), simultaneously. Music industry, a segment of media, is
using big data to keep an eye on the latest trends which are ultimately used by autotuning
softwares to generate catchy tunes.
(3) Sports: To understand the patterns of viewership of different events in specific regions and
also monitor the performance of individual players and teams by analysis. Sporting events
like Cricket world cup, FIFA world cup and Wimbledon make special use of big data analytics.
(4) Healthcare: To collect public health data for faster responses to individual health problems
and identify the global spread of new virus strains such as Ebola. Health Ministries of
different countries incorporate big data analytic tools to make proper use of data collected
after Census and surveys.
(5) Education: To update and upgrade prescribed literature for a variety of fields which are
witnessing rapid development. Universities across the world are using it to monitor and
track the performance of their students and faculties and map the interest of students in
different subjects via attendance.
(6) Manufacturing: To increase productivity by using big data to enhance supply chain
management. Manufacturing companies use these analytical tools to ensure that are
allocating the resources of production in an optimum manner which yields the maximum
benefit.
(7) Insurance: For everything from developing new products to handling claims through
predictive analytics. Insurance companies use business big data to keep a track of the
scheme of policy which is the most in demand and is generating the most revenue.
(8) Consumer Trade: To predict and manage staffing and inventory requirements. Consumer
trading companies are using it to grow their trade by providing loyalty cards and keeping a
track of them.
(9) Transportation: For better route planning, traffic monitoring and management, and logistics.
This is mainly incorporated by governments to avoid congestion of traffic in a single place.
(10)Energy: By introducing smart meters to reduce electrical leakages and help users to manage
their energy usage. Load dispatch centers are using big data analysis to monitor the load
patterns and discern the differences between the trends of energy consumption based on
different parameters and as a way to incorporate daylight savings.
BIG DATA: THE 3 VS

There are three defining properties that can help break down the term. Dubbed the three Vs;
volume, velocity, and variety, these are key to understanding how we can measure big data and just
how very different ‘big data’ is to old fashioned data.

Volume

The most obvious one is where we’ll start. Big data is about volume. Volumes of data that can reach
unprecedented heights in fact. It’s estimated that 2.5 quintillion bytes of data is created each day,
and as a result, there will be 40 zettabytes of data created by 2020 – which highlights an increase of
300 times from 2005. As a result, it is now not uncommon for large companies to have Terabytes –
and even Petabytes – of data in storage devices and on servers. This data helps to shape the future
of a company and its actions, all while tracking progress.

Velocity

The growth of data, and the resulting importance of it, has changed the way we see data. There once
was a time when we didn’t see the importance of data in the corporate world, but with the change
of how we gather it, we’ve come to rely on it day to day. Velocity essentially measures how fast the
data is coming in. Some data will come in in real-time, whereas other will come in fits and starts,
sent to us in batches. And as not all platforms will experience the incoming data at the same pace,
it’s important not to generalise, discount, or jump to conclusions without having all the facts and
figures.

Variety

Data was once collected from one place and delivered in one format. Once taking the shape of
database files - such as, excel, csv and access - it is now being presented in non-traditional forms, like
video, text, pdf, and graphics on social media, as well as via tech such as wearable devices. Although
this data is extremely useful to us, it does create more work and require more analytical skills to
decipher this incoming data, make it manageable and allow it to work.
Big Data is much more than simply ‘lots of data’. It is a way of providing opportunities to utilise new
and existing data, and discovering fresh ways of capturing future data to really make a difference to
business operatives and make it more agile.

What Is Data Veracity?

Data veracity, in general, is how accurate or truthful a data set may be. In the context of big data,
however, it takes on a bit more meaning. More specifically, when it comes to the accuracy of big
data, it’s not just the quality of the data itself but how trustworthy the data source, type, and
processing of it is. Removing things like bias, abnormalities or inconsistencies, duplication, and
volatility are just a few aspects that factor into improving the accuracy of big data.

Why It’s Important

Big data is highly complex, and as a result, the means for understanding and interpreting it are still
being fully conceptualized. While many think machine learning will have a large use for big data
analysis, statistical methods are still needed in order to ensure data quality and practical application
of big data for market researchers. For example, you wouldn’t download an industry report off the
internet and use it to take action. Instead you’d likely validate it or use it to inform additional
research before formulating your own findings. Big data is no different; you cannot take big data as
it is without validating or explaining it. But unlike most market research practices, big data does not
have a strong foundation with statistics.

Big Data – Application examples in CRM

o Due to the change in shopping behaviour, supermarkets can usually detect whether a
customer is pregnant faster than the social environment (the father?) can find out.
o Supermarkets can adapt entire location concepts based on the shopping behaviour of specific
residential areas.
o With SalesForce.com, Coca Cola in the USA has developed Coke vending machines where
customers use IDs on which the customer profile is stored. Coca Cola can thus offer "heavy
users" a Coke with a special flavour, for example.
o With an app, customers can "conjure up" armchairs, sofas or tables from the printed Ikea
catalogue into their own homes (correctly proportioned and placed at the right angle).
o In Tokyo, waddling penguins greet visitors to the Sunshine Aquarium at the underground exit
as soon as they point their mobile phones at the pedestrian zone.
o The car supplier Continental will launch a navigation system in which arrows pointing in the
direction appear to be directly in the lane. If the driver drifts off, the white page stripe lights
up red.
 o Google can predict the spread of flu faster than the world's health authorities can.
o Engine manufacturers can replace parts of a turbine before a critical part will break
(predictive maintenance)
o Health insurance companies can calculate whether a client will become terminally ill and
therefore refuse them.
o In 30 states in the USA, the probability is calculated whether prisoners are involved in a
violent death in the next twelve months after possible release.
o The police calculate in which street crimes are most likely to be committed at what time and
plan operations accordingly.
o Big Data can provide solutions to traffic problems: we can unlock cars that are parked on the
roadside, get in and drive off. The dashboard shows where the next charging station is,
where there are free parking spaces or whether it would be better to change to the train.
o Augmented reality manuals that can be projected onto the glasses of service technicians
enable them to work 30 to 40% faster and with a lower error rate.

What is a data lake?

A data lake is a centralized repository that allows you to store all your structured and unstructured
data at any scale. You can store your data as-is, without having to first structure the data, and run
different types of analytics—from dashboards and visualizations to big data processing, real-time
analytics, and machine learning to guide better decisions.

What is the primary purpose of a data lake?

Data Lakes allow you to store relational data like operational databases and data from line of
business applications, and non-relational data like mobile apps, IoT devices, and social media. They
also give you the ability to understand what data is in the lake through crawling, cataloging, and
indexing of data.

Data Lakes compared to Data Warehouses – two different approaches

Depending on the requirements, a typical organization will require both a data warehouse and a
data lake as they serve different needs, and use cases.
A data warehouse is a database optimized to analyze relational data coming from transactional
systems and line of business applications. The data structure, and schema are defined in advance to
optimize for fast SQL queries, where the results are typically used for operational reporting and
analysis. Data is cleaned, enriched, and transformed so it can act as the “single source of truth” that
users can trust.

A data lake is different, because it stores relational data from line of business applications, and non-
relational data from mobile apps, IoT devices, and social media. The structure of the data or schema
is not defined when data is captured. This means you can store all of your data without careful
design or the need to know what questions you might need answers for in the future. Different
types of analytics on your data like SQL queries, big data analytics, full text search, real-time
analytics, and machine learning can be used to uncover insights.

As organizations with data warehouses see the benefits of data lakes, they are evolving their
warehouse to include data lakes, and enable diverse query capabilities, data science use-cases, and
advanced capabilities for discovering new information models. Gartner names this evolution the
“Data Management Solution for Analytics” or “DMSA.”

The value of a Data Lake

The ability to harness more data, from more sources, in less time, and empowering users to
collaborate and analyze data in different ways leads to better, faster decision making. Examples
where Data Lakes have added value include:

Improved customer interactions

A Data Lake can combine customer data from a CRM platform with social media analytics, a
marketing platform that includes buying history, and incident tickets to empower the business to
understand the most profitable customer cohort, the cause of customer churn, and the promotions
or rewards that will increase loyalty.

Improve R&D innovation choices

A data lake can help your R&D teams test their hypothesis, refine assumptions, and assess results—
such as choosing the right materials in your product design resulting in faster performance, doing
genomic research leading to more effective medication, or understanding the willingness of
customers to pay for different attributes.

Increase operational efficiencies

The Internet of Things (IoT) introduces more ways to collect data on processes like manufacturing,
with real-time data coming from internet connected devices. A data lake makes it easy to store, and
run analytics on machine-generated IoT data to discover ways to reduce operational costs, and
increase quality.

The challenges of Data Lakes

The main challenge with a data lake architecture is that raw data is stored with no oversight of the
contents. For a data lake to make data usable, it needs to have defined mechanisms to catalog, and
secure data. Without these elements, data cannot be found, or trusted resulting in a “data swamp."
Meeting the needs of wider audiences require data lakes to have governance, semantic consistency,
and access controls.
Characteristics of a Data Lake

• Centralised, singular, schema-less data storage with raw data (actual data) and mass data
• Mechanism for fast ingestion of structured and unstructured data
• Ability to map data across sources and provide visibility and security to users
• Catalogue for finding and retrieving data
• Calculation model of the centralised service
• Ability to manage security, permissions and data masking
• Supports self-provisioning of compute nodes, data and analytics tools without IT
intervention

Artificial Intelligence

What is Artificial intelligence?

Artificial intelligence leverages computers and machines to mimic the problem-solving and decision-
making capabilities of the human mind.

Types of artificial intelligence—weak AI vs. strong AI

Weak AI—also called Narrow AI or Artificial Narrow Intelligence (ANI)—is AI trained and focused to
perform specific tasks. Weak AI drives most of the AI that surrounds us today. ‘Narrow’ might be a
more accurate descriptor for this type of AI as it is anything but weak; it enables some very robust
applications, such as Apple's Siri, Amazon's Alexa, IBM Watson, and autonomous vehicles.

Strong AI is made up of Artificial General Intelligence (AGI) and Artificial Super Intelligence (ASI).
Artificial general intelligence (AGI), or general AI, is a theoretical form of AI where a machine would
have an intelligence equaled to humans; it would have a self-aware consciousness that has the
ability to solve problems, learn, and plan for the future. Artificial Super Intelligence (ASI)—also
known as superintelligence—would surpass the intelligence and ability of the human brain. While
strong AI is still entirely theoretical with no practical examples in use today, that doesn't mean AI
researchers aren't also exploring its development. In the meantime, the best examples of ASI might
be from science fiction, such as HAL, the superhuman, rogue computer assistant in 2001: A Space
Odyssey.

Artificial intelligence applications

There are numerous, real-world applications of AI systems today. Below are some of the most
common examples:

Speech recognition: It is also known as automatic speech recognition (ASR), computer speech
recognition, or speech-to-text, and it is a capability which uses natural language processing (NLP) to
process human speech into a written format. Many mobile devices incorporate speech recognition
into their systems to conduct voice search—e.g. Siri—or provide more accessibility around texting.

Customer service: Online virtual agents are replacing human agents along the customer journey.
They answer frequently asked questions (FAQs) around topics, like shipping, or provide personalized
advice, cross-selling products or suggesting sizes for users, changing the way we think about
customer engagement across websites and social media platforms. Examples include messaging bots
on e-commerce sites with virtual agents, messaging apps, such as Slack and Facebook Messenger,
and tasks usually done by virtual assistants and voice assistants.
Computer vision: This AI technology enables computers and systems to derive meaningful
information from digital images, videos and other visual inputs, and based on those inputs, it can
take action. This ability to provide recommendations distinguishes it from image recognition tasks.
Powered by convolutional neural networks, computer vision has applications within photo tagging in
social media, radiology imaging in healthcare, and self-driving cars within the automotive industry.

Recommendation engines: Using past consumption behavior data, AI algorithms can help to
discover data trends that can be used to develop more effective cross-selling strategies. This is used
to make relevant add-on recommendations to customers during the checkout process for online
retailers.

Automated stock trading: Designed to optimize stock portfolios, AI-driven high-frequency trading
platforms make thousands or even millions of trades per day without human intervention.

Deep learning vs. machine learning

What is machine learning?

Machine learning refers to the study of computer systems that learn and adapt automatically from
experience, without being explicitly programmed.

What is deep learning?

Deep learning is a machine learning technique that layers algorithms and computing units—or
neurons—into what is called an artificial neural network.

Since deep learning and machine learning tend to be used interchangeably, it’s worth noting the
nuances between the two. As mentioned above, both deep learning and machine learning are sub-
fields of artificial intelligence, and deep learning is actually a sub-field of machine learning.

Deep learning is actually comprised of neural networks. “Deep” in deep learning refers to a neural
network comprised of more than three layers—which would be inclusive of the inputs and the
output—can be considered a deep learning algorithm. This is generally represented using the
following diagram:
The way in which deep learning and machine learning differ is in how each algorithm learns. Deep
learning automates much of the feature extraction piece of the process, eliminating some of the
manual human intervention required and enabling the use of larger data sets. You can think of deep
learning as "scalable machine learning" as Lex Fridman noted in same MIT lecture from above.
Classical, or "non-deep", machine learning is more dependent on human intervention to learn.
Human experts determine the hierarchy of features to understand the differences between data
inputs, usually requiring more structured data to learn.

"Deep" machine learning can leverage labeled datasets, also known as supervised learning, to inform
its algorithm, but it doesn’t necessarily require a labeled dataset. It can ingest unstructured data in
its raw form (e.g. text, images), and it can automatically determine the hierarchy of features which
distinguish different categories of data from one another. Unlike machine learning, it doesn't require
human intervention to process data, allowing us to scale machine learning in more interesting ways.

deep learning is a subset of machine learning, and machine learning is a subset of artificial
intelligence.

Prerequisites for Machine Learning are Artificial Neural Networks (ANN)

• An important area of research in AI is neural networks. These are inspired by the natural
neural network of the human nervous system.
• The inventor of the first neurocomputer, Robert Hecht-Nielsen, defines a neural network as:
• "...a computer system consisting of a number of simple, highly interconnected processing
elements that process information through their dynamic state response to external inputs."

What is natural language processing (NLP)?

Natural language processing (NLP) is another branch of machine learning that deals with how
machines can understand human language. You can find this type of machine learning with
technologies like virtual assistants (Siri, Alexa, and Google Assist), business chatbots, and speech
recognition software.

• Natural Language Processing (NLP) refers to the AI method of communicating with an intelligent
system using a natural language such as German or English.

• Natural language processing is required when you want an intelligent system such as a robot to
work according to your instructions, when you want to hear decisions from a dialogue-based expert
clinical system, etc.

• The field of NLP involves the production of computers that perform useful tasks with the natural
languages that people use. The input and output of an NLP system can be:

• Language
• Written text
How machine learning is transforming CRMs

Below we’ll dive into three specific changes machine learning has brought to CRM software:

(1) Get a better ROI with predictive analytics

1. Predictive analytics leads to better ROI

Predictive analytics have been one of the key differentiators for CRM solutions in the past few years,
and Gartner found that there’s growing interest around predictive functionality from marketing
technology (full content available to Gartner clients).

With all the major players in the industry buying into this technology, even smaller businesses get to
reap the benefits. The most common predictive analytics you’ll find in CRMs is predictive lead
scoring.

Essentially, it uses machine learning to sift through your contact database to give you more
structured information about client behaviors and trends that would take you much longer to
manually do.

If you’ve always manually scored your leads and don’t trust the robots to give you accurate
information, you can always use what the predictive models give you and go through the results
with a fine-toothed comb the first few times to go further into your scoring.

Key takeaway: A CRM with predictive analytics and lead scoring will increase your revenue by arming
you with better information. If you’re in marketing, you’ll be able to more accurately target the right
consumers where they are during their buyer’s journey.

2. Connect disparate customer data points to understand customer decisions

You’ve probably gotten a call from a client wanting to cancel or move to another system before.
Sure, most of the time that might have to do with cost (which is mostly outside of your control), but
other times your rep might be blindsided by a list of complaints that you weren’t aware of.

Take this as an example: After the fact, you go through and check their profile, noticing they’ve
submitted a few support tickets in the last month. That alone isn’t enough to cause concern, but you
dig a little deeper and find their average use has gone down. Again, by itself, this isn’t a huge
concern. After a little more digging you see your sales rep wrote down they had a new chief
technical officer, which may not be a huge concern by itself either.

But put together, that paints a pretty clear picture. One you didn’t notice.

Key takeaway: A CRM with machine learning capabilities is able to go through each customer’s data
and find these pain points or concerning behaviors and alert you before you receive the call asking
to cancel, giving you time to be proactive and talk to the client before they hit their breaking point.

3. Give value to unstructured qualitative data with ease

One of the benefits of using a CRM is the ability to write notes about each individual client or
customer so you can refer back to them before you contact that person. While useful for
maintaining a good level of personalization, it’s difficult to use that data for anything else.
A CRM has a plethora of structured data that’s easily accessible, including revenue, location, or job
title, but more qualitative data is harder to parse out and draw meaningful conclusions from without
spending an unrealistic amount of time going through the data.

It’s, frankly, almost never worth the time spent by a human—and that’s where a CRM with machine
learning capabilities comes in. With a CRM with natural language processing (NLP), you can leverage
all of that unstructured textual information. It will crawl through all the text your CRM has natively
and will even go through any other contact you have with customers via email or social media to
give you a clearer picture into a customer’s identity.

Key takeaway: A CRM with machine learning and NLP can crawl through all your unstructured text to
give value to all that qualitative data. By combining both qualitative and quantitative data, you’ll
have a much greater understanding of your overall sales landscape, and you can draw more
meaningful conclusions for your business.

Risks of AI?

Artificial intelligence (AI) carries with it the risk of unintended consequences and potential negative
impacts on society. Some specific risks of AI include:

(1) Job displacement: As AI systems become increasingly sophisticated, they may be able to
perform tasks that were previously done by humans, potentially leading to job
displacement.
(2) Bias: AI systems can be biased if they are trained on biased data. This can lead to unfair and
discriminatory outcomes, particularly in areas such as hiring, lending, and criminal justice.
(3) Security and privacy: AI systems can be vulnerable to hacking and other forms of
cyberattacks, which can compromise sensitive data and personal privacy.
(4) Lack of accountability: It can be difficult to hold AI systems accountable when things go
wrong, as it may be unclear who is responsible for any negative outcomes.
(5) Loss of control: As AI systems become more sophisticated, there is a risk that they could
develop capabilities that are beyond our ability to control or understand, potentially leading
to unforeseen and potentially harmful consequences.
(6) Misuse: AI systems could be used for nefarious purposes, such as creating fake news,
cyberbullying, or spreading propaganda.

To mitigate these risks, it is important for organizations and policymakers to carefully consider the
potential impacts of AI and put appropriate safeguards in place. This might involve establishing
ethical guidelines for the development and use of AI, as well as creating regulatory frameworks to
ensure that AI is used responsibly.