Artificial Intelligence: Core Concepts

WHAT AI IS, HOW IT WORKS, AND WHY AND WHEN TO USE IT

1
 Overview
▸ What is artificial intelligence?
▸ What is machine learning?
▸ Why is data important?
▸ An introduction to the myriad aspects of data.
▸ How AI works: process, tasks, algorithms.
▸ A brief introduction to deep learning.
▸ The structure of an AI solutions team.
▸ AI post-deployment.

2
Artificial intelligence vs
machine learning
Artificial intelligence (AI) is an umbrella
term that encompasses several
subfields.

Machine learning (ML) is one of the ML DL RULES RL

subfields of AI.

3
data analysis algorithms
ml software engineering
artificial intelligence
prediction automation
big data ethics privacy4
Artificial intelligence (AI)
is a branch of computer science
concerned with enabling machines
to perform specific types of tasks
that typically require a significant
amount of human intervention, or to
discover patterns that humans cannot.

5
Contrary to popular belief,
‘AI’ does not imply or refer to a
machine or program that thinks.
It involves building computer systems to solve complex
problems, using correlations within data or otherwise,
which can often give the illusion of thinking.

6
AI is driven by machine learning,
a set of methods used to train
computers to learn from data.
Machine learning (ML) provides systems the ability to
automatically learn and improve from experience, without
being explicitly programmed.
In the modern context, the terms ‘ML’ and ‘AI’ are used
interchangeably, because the intelligence of ‘machines’
has become synonymous with the ability to learn from data.

7
data analysis algorithms
ml software engineering
machine learning
prediction automation
big data ethics privacy
8
Essential terms ALGORITHM
A systematic procedure for performing a task or solving a
problem, based on conducting a logical sequence of
Let’s take a moment to consider specified actions that is often implemented by a computer.
a few important machine
learning terms we’re going to
keep running into.
TRAINING DATA
Data that is used to develop an ML model, which will find
patterns and relationships in training data, used to define
rules for new predictions.

MODEL
An ML model consists of a set of parameters that are
learned from the training data, and rules to use these
parameters. When an algorithm is combined with training
data, we get an ML model.

PREDICTION
In ML, ‘prediction’ involves estimating unknown attributes or
qualities based on known and measurable information, and is
not necessarily related to the future. Prediction is useful when
direct measurement is difficult, dangerous, or expensive.

9
An algorithm is a procedure or
formula for solving a problem,
based on conducting a sequence
of specified actions.
Think of it as following a recipe.

10
 Time for khichdi!
What would an algorithm to cook dal khichdi look like?

Sauté ingredients in Turn down the heat

a pressure cooker for 5m, and then
(Rice, lentils, salt,
spices, water, etc.)
turn off the heat

Number of whistles = 3
Place the pressure Wait a few minutes
cooker on a stove, for the pressure to
and turn up the heat release naturally

Transfer the khichdi

Listen for pressure Three Eat the
from the cooker to a
cooker whistles whistles khichdi
plate, add ghee

Number of whistles < 3

11
A machine learning model is an
algorithm that has been learned by a
computer system.
The algorithm is a set of rules and numerical parameters that
are a part of these rules. These parameters are arrived at
after exposure to data, a process called “training”.

Let’s take a look at how ML models are “trained” on data.

12
Training a ML model MACHINE LEARNING: TRAINING

requires a computer
system and some data
for it to learn from. DATA

The training data is accompanied by

the associated, known outcome we Computer System
are looking to predict, called a label.
Known Outcomes
The computer system finds patterns (Labels)
in the training data that correspond
to the outcome, and it outputs an ML
model that captures these patterns
in the data. ML Model

13
A ML model represents what was
learned by the computer system
from the data.
Once developed, ML models can be used to make
predictions based on new and unlabeled data.

14
Using the ML model MACHINE LEARNING: INFERENCE

is called inference.
This involves exploiting the
correlations that the ML model NEW
has learned. DATA

During inference, the ML model is

fed with new, unlabeled data and
predicts the outcome (label.) ML Model PREDICTIONS

15
 CLASSICAL MACHINE LEARNING IN ACTION

The price is right.

Let’s consider a situation where
we want to predict the price of
new houses using ML, based on
some data we’ve acquired.

We have house sizes on the

HOUSE PRICES (in lakhs)

x-axis, and house prices on the
y-axis. Given a new house with
a certain size, we can predict
the price of the house using this
line (the regression line).

The model being learnt here is

the line.

HOUSE SIZES (in sqm)

16
ML/AI models can be used to
replicate the decision process a
human expert could be expected
to follow to make a prediction…
…if human experts had the ability to review the same
volumes and types of data.

17
 TRADITIONAL RULE-BASED SYSTEMS

Rules-based vs machine
learning systems DATA

Domain Handcrafted
Knowledge PREDICTIONS
Models
Example of rules based system:
considering a patient as
high-risk for non-adherence if
they are alcoholic, have low MACHINE LEARNING SYSTEMS
family support or are migrants.

An ML system for the same

purpose would learn DATA

correlations between Computer

ML Model TRAINING
characteristics of patients and System

their adherence behaviour Known Outcomes

(Labels)

What advantages does machine NEW

DATA
learning offer over traditional
rule-based methods?
INFERENCE ML Model PREDICTIONS

18
An automated MACHINE LEARNING SYSTEMS

learning system PROS CONS

offers considerable They are self-learning, i.e., they Need large amounts of input data to
require a much lower level of learn from, to make accurate
advantages. human intervention. predictions.

Can handle complex problems and They can only learn from data they
ML systems do not explicitly
are able to detect patterns and have seen before, and can only adapt
know which aspects of the data correlations in the data that are to situations similar to those they
are essential, but can learn from beyond human capabilities. have been trained for.
the data and predict future
behavior or patterns based on Can learn continuously from data There is no room for human intuition.
historical information. over time and be made to adapt to
changes in data and other external Not intelligent in the way that humans
factors. are, and do not “understand” their
function in any meaningful way.
Better performance and accuracy
when multidimensional or varied Can amplify biases in the training
data is involved. data in dangerous and unanticipated
ways.
Capable of rapid automation of
decision-making, at scale.

19
What types of While machine learning is a powerful tool, there are
some situations or problems that it is best suited for.
problems are best
solved using ML? ▸ Problems for which existing solutions require a lot of
hand-tuning or long lists of rules.

▸ Complex problems for which there is no good solution at all

using a traditional approach.

▸ Fluctuating environments in which rules change: a machine

learning system can adapt to new data.

▸ Getting insights about complex problems and large amounts

of data, especially using deep learning techniques.

▸ Pattern recognition and interpretation problems, e.g.,

abnormalities in x-rays, street signs in an image, pests on
trap images, etc.

20
data analysis algorithms
ml software engineering
data
prediction automation
big data ethics privacy
21
All ML/AI systems are built on data.
Data can be defined as the digital
representation of the world.

22
METADATA Author, file size, language, Height, width, resolution, Duration, album name, date Number of rows and
DATA ABOUT THE DATA publication date, etc. image type, format, etc. of recording, bit rate, etc. columns, data source, etc.

DATA
AS PERCEIVED BY COMPUTERS

8-bit binary Sequence of RGB values Matrix of sound intensities Numeric data in cells

COMMON FORMATS

TEXT IMAGES AUDIO TABULAR DATA

23
PREDICTING COVID-19

FEATURES LABEL

FEVER
YES COUGH
YES

COVID-19
CONTACT COVID-19 POSITIVE
SORE
YES THROAT
NO

RT-PCR
POSITIVE

24
PREDICTING COVID-19

FEATURES LABEL

FEVER
YES COUGH
YES

COVID-19
CONTACT COVID-19 NEGATIVE
SORE
NO THROAT
NO

RT-PCR
NO

25
A label is an outcome to be
eventually predicted by the ML
model. It typically correlates with
the features in the data.
The purpose of an ML model is to learn the relation
between features and labels so when it is given new
and unseen data, it can predict its label.
Labels can be complex (e.g. translating from one
sentence to the other) or simple (is the patient likely to
get a disease?).
What would the labels look like in these cases?

26
Features are the specific properties,
characteristics, or classifications in a
given set of data that can be
analyzed to enable prediction.
Features are a collection of variables in the data that are used by
the ML model to predict an outcome.
Features can be extracted from an image, video, voice clip, text in
a message, and tabular data from databases in existing systems.

27
WHAT IS A FEATURE SET?

DIGITAL IMPRINT
Information has been converted to a machine-
A feature set is a digital imprint of a readable form that can be consumed by a
computer. This gives rise to requirements and
collection of features, containing controls as to how such digital imprints are

information that we assume is created and managed.

sufficient with regards to solving a COLLECTION

problem using AI to a certain Often, multiple features may be used in solving an

degree of accuracy. AI problem.

We can attempt to logically deduce if the feature set provided ASSUME

may have information to solve the problem.
The feature set is informative in a direct (human-
Most surprises in AI come from the latent aspects. understandable) or a latent way, where new
patterns are unearthed (often loosely termed
as intelligence).

28
 Data quality

High quality data Low quality data

▸ It accurately represents a real-world phenomenon ▸ Poor-quality data: Data that is full of errors, outliers,
or entity. and noise.

▸ It is collected, stored, and used responsibly. ▸ Insufficient data: The amount of data collected is not
enough. Low amounts of training data will produce
▸ It is accurate, complete, consistent, reliable and
predictions that are inaccurate or too biased.
up-to-date.
▸ Non-representative data: The data collected is not
▸ It is reproducible and can be maintained over time,
sufficiently representative of the cases you are
with well-documented collection processes.
looking to generalize to.
▸ There is a potential for it to be reused across
▸ Irrelevant data: Data that does not correlate with the
applications.
label(s) of interest.
▸ As data is collected and processed, it is important
to consider relevance, fairness, privacy, and
security, and to prevent bias from creeping in.

29
ABC, Inc. XYZ Pvt. Ltd.

30
What kinds of Here are a few examples of health-related data that are often
used in in AI-based healthcare solutions.
data are useful
▸ Administrative and claims data
for developing e.g., hospital discharge summaries, payment and insurance records

AI solutions ▸ Clinical data

in healthcare?
e.g., clinical studies and trials data, electronic health records

▸ Imaging and scanning data

e.g., MRI scans, CT scans, x-rays

▸ Genomic data
e.g., DNA sequencing data, factor-based DNA variance data

▸ Patient-generated data
e.g., data from smartphone apps or voice assistants, social media

▸ Surveillance data
e.g., registry data, survey data, vitals data

▸ Data related to social determinants of health

e.g., access to food, education, job opportunities, housing options

31
Data Acquisition PRIMARY DATA
Data that is generated by clinicians, researchers or
program people, through surveys, interviews,
Two basic types of data are
usually collected to build an AI experiments, clinical studies, etc. that are specially
healthcare solution. designed for understanding and solving the problem at
hand.

SECONDARY DATA
Data that is generated by large government institutions,
healthcare facilities, national health programs, etc. as part
of regular or special programs and organizational
recordkeeping.

32
data analysis algorithms
ml software engineering
how ai works
prediction automation
big data ethics privacy
33
 How AI Works

Characteristics of real-world entities.

FEATURES AI uses features, which are present in the data provided to
the AI algorithm.

The science and art of problem-solving.

ALGORITHM Created by machine learning scientists, who are technicians
involved in developing solutions using AI.

Software application used to implement the solution.

AI MODEL
A file that embeds the “intelligence” of the AI solution.

34
 What types of tasks are ML models used for?

CLASSIFICATION REGRESSION CLUSTERING

Predicting whether an outcome is Predicting the numerical value of Grouping data points that have
one of a discrete set (yes/no, some aspect of a real-world entity, similar properties and/or features,
A/B/C, etc.), based on learning using patterns in the data. where the semantics of the cluster
from past observations. are understood or defined by
Example: Predicting the weight (in
human beings.
Example: Predicting whether a patient grams) of a baby based on the
will adhere to TB treatment or not, patterns identified from the features Example: Detecting subgroups of
based on a set of features associated of a 10-second video clip. patients in Electronic Health Records,
with them. based on some human-defined
characteristics in the data.

35
 What are the different types of machine learning?

SUPERVISED UNSUPERVISED SEMI-SUPERVISED REINFORCEMENT

In supervised learning, the In unsupervised learning, the Some algorithms can deal with Reinforcement learning is based
training data you feed to the training data is unlabeled. The partially labeled training data. on rewarding desired behaviours
algorithm includes the desired system tries to learn without a and/or punishing undesired ones.
A typical semi-supervised
solutions, called labels. teacher. A typical The learning system can observe
learning task is to use large
unsupervised learning task is the environment, select and
Example: An email spam filter is a amounts of unlabeled data in
clustering. perform actions, and get rewards
good example of this: it is trained order to augment the or penalties. It must then learn the
with many example emails along Example: A common use case for performance of a model built best strategy by itself to get the
with their class (spam or ham), a clustering algorithm is to try to with a small amount of labeled most reward over time.
and it must learn how to classify detect groups of similar visitors to data.
new emails. a website or blog. Example: Self-driving cars are
Example: Some cloud storage often trained using reinforcement
applications, such as Google learning algorithms.
Photos, are good examples of this.

36
Deep learning is a subset of
machine learning that attempts
to mimic the complexity of the
neural networks present in the
human brain.
Deep learning (DL) uses algorithms called artificial
neural networks. DL models enable multiple layers of
processing to extract progressively higher-level
attributes from data.

37
 AN ARTIFICIAL NEURAL NETWORK IN ACTION

Image Recognition

10% WOLF

INPUT LAYER FIRST LAYER HIGHER LAYER TOP LAYER

Simple shapes Complex structures Object verified
90% DOG
identified. identified. against others.

38
ENABLING MORE COMPLEX TASKS AND DECISIONS.

When is deep DEEP LEARNING

learning used?

PERFORMANCE
Deep learning is often referred to
as ‘scalable machine learning,’ and CLASSICAL
MACHINE
excels on problem domains where LEARNING
large amounts of data are available
and the inputs are all of a single
type rather than mixed, for
example, images, text documents,
or audio files.
AMOUNT OF DATA

39
What are some of Some problems are intrinsically difficult to solve using
AI even when high-quality data is available. A few
the challenges common challenges include:
typically involved in ▸ Data quality, quantity, completeness, and obtaining sufficiently
solving problems large data sets.

using AI? ▸ The difficulty of labeling training data, especially when labeling
requires domain expertise.

▸ Difficulty explaining results from large, complex AI systems, like

neural networks.

▸ Difficulties with domain adaptation and generalizing.

▸ Privacy concerns and the risk of discrimination and bias.

▸ Model/data drift and labelling errors; insufficiently

representative data.

40
An ML model is usually integrated
into a software system so that the
model can carry out inference
continuously, as data is fed into it.
This is called deployment.
The applied work of any AI solution occurs when
deployment starts and it begins to scale.

41
Scaled deployment Integration and
The applied work of the solution occurs when monitoring
deployment starts and scales up. This is because
The AI solution may be implemented as a
real-world data is often different from the
separate solution, or integrated into an
relatively small amount of sample data used to
application used by the organization. In
build AI models. either case, the AI in the software is
▸ The following can happen: transparent, enabling capability. This is
▹ The solution works very well (unlikely) done by:
▹ The solution works well in some cases, and not so well in others
▸ Ensuring that the AI solution integrates with the
(likely)
technology stack of the organization.
▹ The solution does not work at all (unlikely)
▸ Building effective, minimal changes in the
▸ Ideally, every decision made by a deployed solution must working style of the organization.
come back to the AI scientists for analysis. They may
decide to retrain and release a new model if the ▸ Looking for efficiency gains in approach.
performance of the old model dips.
▸ Ensuring that all predictions made are collected
▸ In the early stages, a human-in-the-loop (human-assisted and fed back for analysis.
or -validated) approach may be required. 42
Finally, what is The AI solution goes into maintenance mode after
the performance of the solution is stable, and within
maintenance? acceptable limits. During maintenance:
▸ Reporting of decisions made by the solution are stored and
viewed by designated domain specialists.

▸ Decisions made by the solution are sent periodically to the AI

scientists, who review it to ensure that the performance is in
line with expectations.

▸ The AI scientists may also retrain and release a new model if

the performance of the old model dips.

43
The performance of an ML model,
i.e., how good it is at predicting an
outcome, is evaluated by monitoring
its performance.
Accuracy tells you how many times the ML model was correct overall
(classification task).
Root mean square error (RMSE) measures the deviation from the
actual value (regression task).
Other metrics: Precision, Recall / Sensitivity, Specificity, etc.

44
AI solutions require a radically collaborative approach.

ML Modeling Epidemiologists
Engineering Doctors
User Research Technologists Domain Experts Entomologists
Agronomists

Govt. Ministries Users

Officials Administrators Implementers Beneficiaries
Extension Officers
Decision-makers
Frontline Workers

45
Questions?
Let’s talk about AI.

46