Unit-1

1
Syllabus

4
Cognitive computing

• Technology approach that enables humans to collaborate with machines.

• As an analog to the human brain.
• Analyze in context all types of data
• Machines that operate at a different level than traditional IT systems.
• Analyze and learn from this data.
• This approach to create solutions that change as more data is added (ingested) and
as the system learns
• Artificial intelligence, cognitive science, and computer science have led to the
development of cognitive computing.
Principles of Cognitive System
• Learn—A cognitive system learns. The system leverages data to make
inferences about a domain, a topic, a person, or an issue based on training and
observations from all varieties, volumes, and velocity of data.
• Model—To learn, the system needs to create a model or representation of a
domain (which includes internal and potentially external
data) and assumptions that dictate what learning algorithms are used.
Understanding the context of how the data fits into the model is key to
a cognitive system.
• Generate hypotheses—A cognitive system assumes that there is not a
single correct answer. The most appropriate answer is based on the data
itself. Therefore, a cognitive system is probabilistic. A hypothesis is a candidate
explanation for some of the data already understood. A cognitive
system uses the data to train, test, or score a hypothesis.
Cognitive Computing as a New Generation

• Evolution of technology that attempts to make sense of a complex

world
• Transform the way humans collaborate with machines to gain
actionable insights
• Volume and diversity of data has increased exponentially, many
organizations cannot turn that data into actionable knowledge
• Next generation of solutions combines some traditional technology
techniques with innovations to solve complex problems
• Cognitive systems are still in the early days of evolution
 Uses of Cognitive Systems

• Cognitive capabilities built into many different applications and systems

• Cognitive system is designed to build a dialog between human and machine so
that best practices are learned by the system as opposed to being programmed as a set
of rules.
• Industry‐specific problems-anticipate retail customer requirements and increase
sales
• Diagnose an illness.
• Anticipate when traffic will be disrupted by weather events and reroute that traffic to
avoid problems.
• Hospital’s electronic medical records to test for omissions and improve accuracy
• To teach new physicians medical best practices and improve clinical decision making
• Security issues
88
Uses of Cognitive Systems (contd)
• Potential uses of a cognitive computing approach continue to grow over time
• Frontier in cognitive computing development has been in the area of
healthcare because it is
rich in text‐based data sources
• patient outcomes are often dependent on care providers
• complete, accurate, up‐to‐date understanding of patient problems
• Medical cognitive applications can enable physicians and care providers to
better understand treatment options through continuous learning, the
ability to treat patients could be dramatically improved.
• Many other industries are testing and developing cognitive applications as well.
• Smarter city applications enable managers to plan the next best action to
control pollution, improve the traffic flow, and help fight crime.
• Traditional customer care and help desk applications can be dramatically
improved if systems can learn and help provide fast resolution of customer
problems.
 What Makes a System Cognitive?
• 1.Contextual insight from the model,
• 2. Hypothesis generation (a proposed explanation of a phenomenon),
• 3. Continuous learning from data across time
• Enables the examination of a wide variety of diverse types of data and the interpretation
of that data to provide insights and recommend actions.
• Essence of cognitive computing is the acquisition and analysis of the right
amount of information in context with the problem being addressed
• Must be aware of the context that supports the data to deliver value
• After acquiring data, curated, and analyzing, it must identify and remember patterns and
associations in the data
• It is Iterative process- enables the system to learn and deepen its scope so that
understanding
of the data improves over time.
• Capability to provide the knowledge seeker with a series of alternative answers along
with an explanation of the rationale or evidence supporting each answer
 What Makes a System Cognitive? (contd)
• Tools and techniques- Big Data and analytics, machine learning, Internet of Things (IoT),
Natural Language Processing (NLP), causal induction, probabilistic reasoning, and data
visualization.
• Cognitive systems have the capability to learn, remember, provoke, analyze, and
resolve in a manner that is contextually relevant to the organization or to the
individual user.
• Highly complex problems solution - require the integration of all sorts of data and
knowledge
• Data- available from a variety of structured, semi‐structured, and unstructured sources
including, journal articles, industry data, images, sensor data, and structured data from
operational and transactional databases.
• It employs sophisticated continuous learning techniques to understand and organize
information.
Understanding Cognition
• Cognition: from Latin base cognitio “know together”

• Cognition is a word that dates back to the 15th century, when it meant "thinking and
awareness“

• Cognition is a mental action or process of acquiring knowledge and understanding through

thoughts, experience and the senses.
• The (interdisciplinary) study of mind and intelligence.

• The study of cognitive processes involved in the acquisition, representation and use of
human knowledge.

• The scientific study of the mind, the brain, and intelligent behaviour, whether in humans,
animals, machines or the abstract.
CognitiveSkills
• What cognitive skills are required to answer a telephone call?
• Answer
❑perception (hearing the ring tone),
❑decision taking (answering or not),
❑motor skill (lifting the receiver),
❑language skills (talking and understanding language),
❑social skills (interpreting tone of voice and interacting properly
with another human being)
Knowledge, Experience & Creativity

Source: LinkedIn
Knowledge, Experience & Creativity (contd)

Source: LinkedIn
Knowledge, Experience & Creativity (contd)

Source: LinkedIn

16
 Cognitive Computing
• Cognitive Science – it is study of mind and intelligence
• Cognitive computing is simulation of human thought process in a computerised model.
• Understands Natural language and human communication style, generates and evaluates
evidence based hypothesis, adopts and learns from training and interactions.
• Cognitive Computing refers to the new hardware and / or software that mimics the
functioning of human brain, and helps to improve human decision making.
• Cognitive Computing – Cognitive Science + Computer Science
• Cognitive science—The science of the mind.
•
Computer science—The scientific and practical approach to computation and its
applications. It is the systematic technique for translating this theory into practice.

• https://www.youtube.com/watch?v=1tsFTBqXDdI
• https://www.youtube.com/watch?v=K6eZtvNUDsk
 Cognitive science

• Cognitive Computing – cognitive science + computer science

• Thinking
• Inference
• Learning Observe

• Perception
Evaluate Interpret

Decide
-Analysis
-Model Building
-Knowledge representation
-HCI (Interaction)
Intellectual functions
Cognitive science Cognitive computing

• Attention – observation, listening

• Formation of knowledge – understanding, hypothesizing
• Memory – remembering
• Judgement and Evaluation – conclusion, Inference
• Reasoning and Computation – analysis
• Problem solving and decision making – applying knowledge / data analytics
• Comprehension and Production of Language – Interface / interactive dashboards
 Features of Cognitive System
• Learn from experience with data/evidence and improve its own knowledge and
performance
without reprogramming.
• Generate and/or evaluate conflicting hypotheses based on the current state of its
knowledge.
• Report on findings in a way that justifies conclusions based on confidence in the evidence.
• Discover patterns in data, with or without explicit guidance from a user regarding the
nature of the pattern.
• Emulate processes or structures found in natural learning systems (that is, memory
management,
knowledge organization processes, or modeling the neurosynaptic brain structures and
processes).
• Use NLP to extract meaning from textual data and use deep learning tools to extract
features from images, video, voice, and sensors.
• Use a variety of predictive analytics algorithms and statistical techniques.
 Characteristics of a Cognitive System
• A cognitive system is probabilistic
• The cognitive system is based on “human-machine collaboration” to gain insight from data.
• Cognitive systems should be able to handle Big data
• Cognitive systems should be context-aware and able to infer contextual insights from
data.
• Cognitive systems should identify and remember the patterns / associations/
inferences generated out of the data.
• Cognitive systems are adaptive in nature.
• Cognitive systems should generate and evaluate multiple ( and / or conflicting)
hypotheses based on the data
• Cognitive systems should emulate (or imitate) processes or structures found in natural
learning system.
 Characteristics of a Cognitive System (contd)
• Cognitive systems should be able to converse like human.
• Cognitive systems use NLP techniques to capture the meaning of unstructured text from
documents or communications generated by users.
• Cognitive systems use Deep learning to capture meaning from non-text based sources
like videos and sensor data.
• Probabilistic system
- multiple / variety of answers i)based on
circumstances or contexts
ii)confidence level or probability based on the system’s current knowledge
• Deterministic system
-single answer based on the evidence
-no answer if there is a condition of uncertainty
 Gaining Insights from Data
• For relevant and useful system , it must continuously learn and adapt as new information
is ingested and interpreted.
• To gain insight and understanding of this information requires that a variety of tools
understand data.
• Much of the data is text based.
• Natural Language Processing (NLP) techniques -to capture the meaning of unstructured text.
• NLP is the primary tool to interpret text.
• Deep learning tools are required to capture meaning from non text‐based sources such as
videos and sensor data.
• For example, time series analysis analyzes sensor data, whereas a variety of image analysis
tools interpret images and videos.
• All these various types of data have to be transformed so that they can be understood
and processed by a machine.
 Gaining Insights from Data
• Transformations of data must be presented to allow the users to understand the
relationships
between a variety of data sources.
• Visualization tools and techniques will be critical ways for making this type of
complex data accessible and understandable.
• Visualization is one of the most powerful techniques to make it easier to recognize
patterns in massive and complex data.
• It needs to bring together structured, semi‐structured, and unstructured sources to
continuously
learn and gain insights from data.
• How these data sources are combined with processes for gaining results is key to
cognitive
computing.
• Therefore, the cognitive system offers its users a different experience in the way it
interacts with data and processes.
Where Cognitive systems are suitable

• Domains where a single query or set of data may result in hypothesis that
yields more than one answers
eg. Medical diagnosis

• When system domain is probabilistic and not deterministic

• When the domain is complex and conclusions depend on who is

asking the
question (contextual)

• Continuous learning from changing data (adaptive)

Elements of Cognitive System
• Consists of Elements ranging from
Hardware to Machine Learning
• Different architectures are available with
few common elements
• flexible and agile infrastructure to support
applications that continue to grow over time
• Private and Data Access, Metadata, and
Management Services
• Public Data and Its management with SaaS
• The Corpus, Taxonomies, and Data Catalogs
• Data Analytics Services
• Continuous Machine Learning -Hypothesis
Generation and Evaluation
• The Learning Process
• Presentation and Visualization Services
• Cognitive Applications
• The continuous machine learning procedure embraced by Cognitive
Systems typically involves two sets of dynamics:
• (a) Hypotheses Generation and
• (b) Hypotheses Evaluation.
• A hypothesis is a testable assertion based on evidence that explains
some observed phenomenon; goal here is to look for evidence to
support (or refute) hypotheses, this is typically accomplished through
an iterative process of training the data.
 Analytics
• Descriptive Analytics tells you what happened in the past.
• Diagnostic Analytics helps you understand why something happened
in the past.
• Predictive Analytics predicts what is most likely to happen in the
future.
• Prescriptive Analytics recommends actions you can take to affect
those outcomes.
 Analytics (contd)
EXAMPLES
• Descriptive analytics looks at data statistically to tell you what
happened in the past. Descriptive analytics helps a business understand
how it is performing by providing context to help stakeholders interpret
information. This can be in the form of data visualizations like graphs,
charts, reports, and dashboards.
• In a healthcare setting, for instance, say that an unusually high number
of people are admitted to the emergency room in a short period of
time. Descriptive analytics tells you that this is happening and provides
real-time data with all the corresponding statistics (date of occurrence,
volume, patient details, etc.).
Analytics (contd)

• Diagnostic analytics takes descriptive data a step further and

provides deeper analysis to answer the question: Why did this
happen? Often, diagnostic analysis is referred to as root cause
analysis. This includes using processes such as data discovery, data
mining, and drill down and drill through.
• In the healthcare example mentioned earlier, diagnostic analytics
would explore the data and make correlations. For instance, it may
help you determine that all of the patients’ symptoms—high fever,
dry cough, and fatigue—point to the same infectious agent. You now
have an explanation for the sudden spike in volume
 Analytics (contd)

• Predictive analytics takes historical data and feeds it into a machine

learning model that considers key trends and patterns. The model is
then applied to current data to predict what will happen next.
• Back in our hospital example, predictive analytics may forecast a
surge in patients admitted to the ER in the next several weeks. Based
on patterns in the data, the illness is spreading at a rapid rate.

32
 Analytics (contd)

• Prescriptive analytics takes predictive data to the next level. Now

that you have an idea of what will likely happen in the future, what
should you do? It suggests various courses of action and outlines
what the potential implications would be for each.
• Ex: now that you know the illness is spreading, the prescriptive
analytics tool may suggest that you increase the number of staff on
hand to adequately treat the influx of patients.

33
Ontology, Taxonomy.

• Taxonomy identifies hierarchical relationships within a

category. Taxonomies are useful for organizing information for both
internal and external consumption. Within a company, they can be
used for classifying documents into categories such as proposals,
contracts, letters, and briefings.
• Each taxonomy is designed to categorize items within just one
dimension

34
 Ontology, Taxonomy (contd)

• Ontologies achieve a higher level of sophistication by providing richer

information, including information about the relationships among
entities.
• Ex: In addition to a taxonomy of products, the store might
have other taxonomies, such as categories of customers. This could
include customers who are consumers and contractors, and sub-
categories within those, such as contractors who are roofers and
electricians. By relating categories of users with products sold in
stores, an ontology would present a list of products relevant to those
users.

35
Tools

• Tools are a basic condition to enable learning from data, by

processing it in both structured and unstructured form.
• For unstructured textual data, NLP (Natural Language Processing)
services can interpret and detect patterns to support a cognitive
system.
• Unstructured data such as images, videos, and sound requires DL
(Deep Learning) tools.

36
• Data Visualization →
• Last but not least, outcomes and
results need to be
represented visually to help make
sense of recommendations/ findings.

• Patterns and relationships in

data are easier to
identify and understand when
visualized with structure, color, and
such.

37
• On top of it all, the Cognitive system must leverage all these
underlying facilities to enable the creation of applications that
address business problems in vertical domains.
• These apps may need to infuse [fill / soak ] processes to gain insight
about a complex business (ex: healthcare, logistics, preventive
maintenance, etc).

38
Iterative Process

39
Components of a Cognitive System

• Model refers to the corpus and the set of assumptions and algorithms that
generate and score hypotheses to answer questions, solve problems, or discover
new insights.
• Model determines what kind of predictions that can make, patterns and
anomalies that can detect, and actions that can take.
• Cognitive system will update the model and use the model to answer questions
or provide insights.
• The corpus is the body of knowledge that machine learning algorithms use to
continuously update that model based on its experience, which may include user
feedback

40
Steps in Cognitive System Design-
1) It requires an understanding of the available data,
2) The types of questions that need to be asked,
3) The creation of a corpus comprehensive enough to support the
generation of hypotheses about the domain based on observed facts.

41
The design of a cognitive system needs to support the following
differentiating characteristics:

Access, manage, and analyze data in context.

Generate and score multiple hypotheses based on the system’s accumulated

knowledge. A cognitive system may generate multiple possible solutions to
every problem it solves and deliver answers and insights with associated
confidence levels.

The system continuously updates the model based on user interactions and
new data.

A cognitive system gets smarter over time in an automated way 42

https://www.youtube.com/watch?v=1tsFTBqXDdI

https://www.youtube.com/watch?v=K6eZtvNUDsk

43
 Two Systems of Judgement and Choice

• Complicated to translate the complexity of human thought and actions into systems.
• It is often influenced by emotion, instinct, habits, and subconscious assumptions about the
world.
• Cognition is a foundational approach that leverages not just how we think, but also
how we act and how we make decisions.
• E.g one doctor recommend one treatment whereas another doctor recommends a
completely different approach to the same disease
• To understand how to apply cognition to computer science, it is helpful to understand
Kahneman’s
theory about how we think.
• System 1—Automatic Thinking: Intuition and Biases

• System 2—Controlled, Rule‐Centric, and Concentrated Effort

44
 Psychology for Cognitive Computing

• One of the most influential thinkers on the topic is Dr. Daniel

Kahneman,
-Israeli-American psychologist and winner of the 2002 Nobel
Memorial Prize in Economic Sciences.

• He is well known for his research and writing in the field

• of
Onetheofpsychology
his greatestof judgment and
contributions to decision
cognitivemaking.
computing is his
research on the cognitive basis for common human errors that
Daniel Kahneman arise from heuristic and biases
who won the 2002 Nobel
Memorial Prize in
Economics for his work • The central thesis is dichotomy between two modes of
developing prospect theory. thought:
• System 1 is fast, instinctive and emotional;
Thinking, Fast and Slow
is a best-selling book published in 2011• by
System2 is slower, more deliberative and more logical.
Nobel Memorial Prize in Economic
Sciences laureate Daniel Kahneman. 45
System 1 (Fast): Automatic Thinking

• Based on Intuition and Biases

• Thinking that happens automatically in our brain
• We see objects / learn concepts and establish relationship with them (eg.
Mother’s voice and safety, loud noise and danger)
• Context (surrounding) in which our thinking has developed - introduces
biases (eg. Loud noise may be fun)
• System 1 is important for Cognitive systems as it uses sparse
information about events and observations and come to rapid
conclusion.
• Adv: Fast decision making
• Disadv: System 1 type of intuitive thinking can sometimes be
inaccurate and prone to error.

• https://www.youtube.com/watch?v=Ta9KSN7hhog
46
System 2(Slow): Controlled, Rule Centric,
Concentrated Effort
Observes and tests assumption
Do not come directly to conclusion
It uses simulation to take an assumption and look at the
implications of that assumption
Collect a lot of data and build model that tests System 1 intuition
Eg. Drug trial – cancer treatment, Covid 19 treatment -

Hydroxychloroquine
Adv – System 2 is more accurate as it is based on deep thinking
and analysis approach
Disadv – Slow

47
48
Types of Intelligence
• Linguistic – narrators, orators
• Musical – musicians, composers, singers
• Logical / mathematical – mathematicians, scientists
• Spatial – map readers, astronauts
• Interpersonal – mass communicators, interviewers

61
Artificial Intelligence

53
What is intelligence?
❖ Linguistic intelligence
❖ Problem solving
❖ Perception
❖ Learning
❖ Reasoning

54
Which tasks are easy for human / machine?

• Recognizing people and objects

• Medical diagnosis
• Navigating around obstacles on the
streets
• Communicating through natural
language
• Playing chess
• Mathematical problem solving
• Planning and scheduling
55
 What is AI?
• Design of intelligence in an artificial device
• Thinking and acting humanly
• Thinking and acting rationally
• Turing Test approach
✓ NLP
✓ Knowledge representation
✓ Automated reasoning
✓ Machine learning

56
Is this statement true?

• “Surely, computers cannot be intelligent – they can do only what

their programmers tell them”

• https://www.youtube.com/watch?v=Zsl7ttA9Kcg

57
 Foundations of AI
1. Philosophy
1. How does mind arise from a physical brain?
2. Where does knowledge come from and how does it lead to action?
3. Goal based analysis
2. Mathematics
1. Formal rules to draw valid conclusions
2. How do we reason with uncertain information?
3. NP-Completeness
3. Economics
1. Making decisions to maximize payoff
2. Dealing with adversaries
3. Game theory, preferred outcome – utility
4. Neuroscience
1. Study of nervous system and brain
2. Collection of neurons lead to thought, action, consciousness
55
Foundations of AI (continued…)
5. Psychology
1. How do humans and animals think and act?
2. Cognitive Science
6. Computer Engineering
1. Building efficient computers – speed, OS, programming languages, tools,
storage
7. Control theory
1. Maximize objective function over time, self-control
8. Linguistics
1. How does language relate to thought? (Verbal Behavior)
2. NLP

56
Applications of Cognitive Systems

• Healthcare – diagnose a problem, clinical decision making

• Business - Increase of sales by anticipating customer requirements
• Smart city – traffic management, prevent pollution
• Robotics – dialog between human and robots
• Any other?
 Current Applications of AI
1. Robotics – Industrial, humanoid, medical
2. Driverless car
3. Speech recognition (vs voice recognitio n??)
4. Autonomous planning and scheduling
5. Game playing
6. Logistics planning
7. Machine Translation
8. Healthcare – Prediction and diagnosis
9. Chatbots
10. Face recognition, Video analysis

62
 Pepper Robot

1. Developed by Aldeberan Softbank Robotics

2. Semi-humanoid robot
3. 4 microphones, 2 HD cameras , 3D dept sensor,
gyroscope and touch sensors, 2 sonars, 6 lasers,
3 bumper sensors
4. Used for facial recognition, emotion detection,
employed at banks and restaurants
5. Weight 28 Kg, 10.1 inch touch display, Naoqi
OS
6. Programmable
7. USD 1600 + maintenance contract

63
Daksha – developed by DRDO

▪ Developed by Defence Research and Development

Organization, operated by Indian Army
▪ Battery operated, remote controlled robot on
wheels
▪ Can navigate staircases, negotiate steep slopes to
reach hazardous materials
▪ Robotic arm to lift suspect object and scan using
portable X-ray device. If bomb, diffuse with water
jet disrupter

64
 Nao Robot

https://www.youtube.com/watch?v=2STTNYNF4lk

61
Expert System
• Emulates decision making ability of human expert
• Uses reasoning and if-then rules along with knowledge base to solve problems
• Capable of advising, instructing, assisting, diagnosing, predicting, interpreting,
justifying
• Incapable of refining its own knowledge
• Environment
• Workstations, Mainframes
• Programming languages – LISP, PROLOG (IBM Watson uses Prolog)
• Use cases – Clinical expert system, Space Shuttle Mission Control

66
 Architecture of Expert System
• Knowledge based architecture
• Components
• Knowledge base
• Inference engine
• User Interface
• Knowledge base
➢domain specific, accurate, precise
knowledge
➢data, information, past experience
➢factual or heuristic
• Knowledge representation – rules
• Inference engine – applies rules to the facts,
resolves rule conflicts
• Forward chaining, Backward chaining –
strategies to recommend a solution
• User interface – interaction between user
and ES, using NLP
67
Forward chaining vs Backward chaining

FORWARD CHAINING BACKWARD CHAINING

➢ Bottom-up model ➢ Top-down approach
➢ What will happen next? ➢ Why did this happen?
➢ Example: weather forecasting system ➢ Example: diagnosing cancer in humans
➢ Used for predicting future outcomes ➢ Analyzing historical data
➢ Data driven systems ➢ Goal driven (to prove a hypothesis)
➢ Results in new knowledge

68
Benefits & limitations of Expert System
❑ Benefits
• Availability
• Less production cost
• Speed
• Less error rate
• Steady response as compared to humans
❑ Limitations
• They can’t substitute human decision makers
• Their accuracy depends on adequate knowledge base
• They can’t refine their own knowledge

69
Machine Learning

66
Machine Learning
• Machine learning algorithms aim to learn a set of tasks T, based on experience E, to
achieve the performance criterion P.
• Performance is measured in terms of accuracy (and sometimes speed) of the
learning model.
• Design of ML system involves four components:
-training experience
-target function to be learned
-representation of target function
-algorithm for learning the target function from training examples
• The process of learning is “searching through the space of possible hypotheses to
find the hypothesis that best fits the available training experience / examples.

[Hypothesis in ML is some function of predictors ]

70
Machine Learning

71
Machine Learning

72
Supervised vs. Unsupervised Learning

• Supervised learning (classification)

• Supervision: The training data (observations, measurements, etc.)
are accompanied by labels indicating the class of the observations
• New data is classified based on the training set
• Unsupervised learning (clustering)
• The class labels of training data is unknown
• Given a set of measurements, observations, etc. with the aim of establishing
the existence of classes or clusters in the data

73
Supervised Learning

74
Unsupervised Learning

75
Semi-supervised learning

• In this type of learning, the algorithm is trained upon a combination of

labeled and unlabeled data.

• Typically, this combination will contain a very small amount of labeled

data and a very large amount of unlabeled data.

• The basic procedure involved is that first, the programmer will cluster
similar data using an unsupervised learning algorithm and then use the
existing labeled data to label the rest of the unlabeled data.

• The typical use cases of such type of algorithm have a common property
among them – The acquisition of unlabeled data is relatively cheap while
labeling the said data is very expensive.

76
ML Model building based on statistical model building process

Dataset

Training data
Testing data
2/3 rd
1/3 rd

78
Process (1): Model Construction

Classification
Algorithms
Training
Data

Classifier
NAME RANK YEARS TENURED
(Model)
Mike Assistant Prof 3 no
Mary Assistant Prof 7 yes
Bill Professor 2 yes
Jim Associate Prof 7 yes
Dave Assistant Prof 6 no RULE ??
Anne Associate Prof 3 no

79
Process (1): Model Construction

Classification
Algorithms
Training
Data

Classifier
NAME RANK YEARS TENURED
(Model)
Mike Assistant Prof 3 no
Mary Assistant Prof 7 yes
Bill Professor 2 yes
Jim Associate Prof 7 yes
IF rank = ‘professor’
Dave Assistant Prof 6 no OR years > 6
Anne Associate Prof 3 no THEN tenured = ‘yes’

80
Process (2): Using the Model in Prediction

Classifier

Testing
Data Unseen Data

(Jeff, Professor, 4)

NAME RANK YEARS TENURED

Tom Assistant Prof 2 no Tenured?
Merlisa Associate Prof 7 no
George Professor 5 yes
Joseph Assistant Prof 7 yes

81
Classification Techniques

• Decision tree based methods

• Rule based methods
• Memory based reasoning
• Neural Networks
• Genetic algorithms
• Bayesian networks
• Support vector machines

85
Decision Tree

Outlook

Sunny Overcast
Overcast Rain

Humidity Ye Wind
s

High Normal Strong Weak

Yes No Yes
N
o

87
Reinforcement Learning

• Reinforcement is the action or process of strengthening

• In this approach, occasional positive and negative feedback is used to
Reinforce behaviors.

• Ex. Think of it like training a dog, good behaviors are rewarded with a treat
and become more common. Bad behaviors are punished and become less
common.

• This reward-motivated behavior is key in reinforcement learning.

90
Agent — the learner and the decision maker.
Environment — where the agent learns and decides what actions to perform.
Action — a set of actions which the agent can perform.
Reward — for each action selected by the agent the environment provides a reward.
Usually a scalar value.

91
 Performance of Model
• Generalization error [generalization error (also known as the out-of-sample error or the
risk) is a measure of how accurately an algorithm is able to predict outcome values for
previously unseen data.]

• Underfitting – Learned model is too general and fails to find interesting patterns
– result of not including all important attributes. [performs poorly on training data]

• Overfitting – Learned model is too specific or too sensitive to training data – result of
including too many predictors. [performs well on training data, but not on evaluation
set]

• Learned model should avoid both underfitting or overfitting.

89
Accuracy and Error Measures in Classification and Prediction

• Accuracy of a classifier on a given test set is percentage of test set tuples

that are correctly classified by the classifier

• It is termed as overall Recognition rate in Pattern Recognition Literature

• Error rate / Misclassification rate of Classifier M

= 1 – acc(M)

• Confusion Matrix: method to analyze how well classifier can recognize

tuples of different classes

92
Factors for success of ML model

• Nature of dataset
• Number of records / tuples for training
• Number of features
• Feature extraction
• Feature reduction

130
Data Visualization

85
86

Data Visualization

• Data visualization is the representation of data through use of

common graphics, such as charts, plots, infographics, and even
animations.

• These visual displays of information communicate complex

data relationships and data-driven insights in a way that is easy
to understand.
87

Data Visualization (contd)

• Data visualization is a technique to present the data in a pictorial

or graphical format.
• Visuals are presented in the form of Graphs, Images, Diagrams,
Animation
• Visualization is an excellent medium to analyze, comprehend,
and share information

Rolls-Royce believes visualising data is as important as manipulating it

88

Data Visualization (contd)

• Data Visualization can be categorized into four key purposes:

• idea generation,
• idea illustration,
• visual discovery, and
• everyday data
89

Advantages of Data Visualization

■ Helps in decision making

■ Finding solution to the problems
■ For understanding data clearly
■ To find relationship among the data
■ Comparative analysis
90
Data Visualization Considerations

• Clarity Ensure the dataset is complete and relevant. This enables the Data
Scientist to use the new patterns obtained from the data in the relevant
places
• Accuracy Ensure you use appropriate graphical representation to convey
the intended message
• Efficiency Use efficient visualization techniques that highlight all the data
points
91
Types of Data Visualization
1. Tables: This consists of rows and columns used to compare variables. Tables can show a
great deal of information in a structured way, but they can also overwhelm users that are
simply looking for high-level trends.

2. Pie charts and stacked bar charts: These graphs are divided into sections that represent
parts of a whole. They provide a simple way to organize data and compare the size of each
component to one other.

3. Line charts and area charts: These visuals show change in one or more quantities by
plotting a series of data points over time and are frequently used within predictive
analytics. Line graphs utilize lines to demonstrate these changes while area charts connect
data points with line segments, stacking variables on top of one another and using color to
distinguish between variables.
 Area Chart

An area chart or area graph

displays graphically
quantitative data. It is based on
the line chart. The area
between axis and line are
commonly emphasized with
colors, textures and hatchings.

92
93
Types of Data Visualization (contd)
4. Histograms: This graph plots a distribution of numbers using a bar chart (with no spaces
between the bars), representing the quantity of data that falls within a particular range. This
visual makes it easy for an end user to identify outliers within a given dataset.

5. Scatter plots: These visuals are beneficial in reveling the relationship between two
variables, and they are commonly used within regression data analysis. However, these can
sometimes be confused with bubble charts, which are used to visualize three variables via the
x-axis, the y-axis, and the size of the bubble.

6. Heat maps: These graphical representation displays are helpful in visualizing behavioral
data by location. This can be a location on a map, or even a webpage.

7. Tree maps, which display hierarchical data as a set of nested shapes, typically rectangles.
Treemaps are great for comparing the proportions between categories via their area size.
94
95
96
97
Summary

98
99
100

Data Visualization Best Practices

101

Data Visualization best practices (contd)

1) Know your audience(s):

• Think about who your visualization is designed for and then make sure
your data visualization fits their needs.
• What is that person trying to accomplish?
• What kind of questions do they care about?
• Does your visualization address their concerns?
• You’ll want the data that you provide to motivate people to act within
their scope of their role.
• If you’re unsure if the visualization is clear, present it to one or two
people within your target audience to get feedback, allowing you to make
additional edits prior to a large presentation.
102
Data Visualization best practices (contd)

2) Choose an effective visual:

• Specific visuals are designed for specific types of datasets.

• For instance, scatter plots display the relationship between two variables
well, while line graphs display time series data well.
• Ensure that the visual actually assists the audience in understanding your
main takeaway.
• Misalignment of charts and data can result in the opposite, confusing your
audience further versus providing clarity.
103
Data Visualization best practices (contd)
3) Keep it simple:
• Data visualization tools can make it easy to add all sorts of information to
your visual.
• However, just because you can, it doesn’t mean that you should!
• In data visualization, you want to be very deliberate about the additional
information that you add to focus user attention.
• Design your data visualization for maximum impact by eliminating
information that may distract your target audience.

For example,
• Do you need data labels on every bar in your bar chart? Perhaps you only
need one or two to help illustrate your point.
• Do you need a variety of colors to communicate your idea?
• Are you using colors that are accessible to a wide range of audiences (e.g.
accounting for color blind audiences)?
104

Building the Corpus

105

What is corpus ?

• A corpus can be defined as a collection of text documents. It can be thought

as just a bunch of text files in a directory, often alongside many other
directories of text files. NLTK already defines a list of data paths or directories
in NLTK

• A corpus is a large and structured set of machine-readable texts that have

been produced in a natural communicative setting. Its plural is corpora. They
can be derived in different ways like text that was originally electronic,
transcripts of spoken language and optical character recognition, etc.
10
6
Elements of Corpus Design
• Language is infinite but a corpus has to be finite in size.

• For the corpus to be finite in size, we need to sample and

proportionally include a wide range of text types to ensure a
good corpus design.

10
7
1. Corpus Representativeness
According to Leech (1991),
“A corpus is thought to be representative of the language variety it is
supposed to represent if the findings based on its contents can be
generalized to the said language variety”.

According to Biber (1993),

“Representativeness refers to the extent to which a sample includes
the full range of variability in a population”.

10
8
Corpus Representativeness (contd)

Representativeness of a corpus are determined by the

following two factors −

• Balance − The range of genre include in a corpus

• Sampling − How the chunks for each genre are
selected.
 a) Corpus Balance
• corpus balance – the range of genre included in a corpus.

• Representativeness of a general corpus depends upon how

balanced the corpus is.

• A balanced corpus covers a wide range of text categories, which

are supposed to be representatives of the language.

• We do not have any reliable scientific measure for balance but the
best estimation and intuition works in this concern.

• The accepted balance is determined by its intended uses only.

b) Corpus Sampling
According to Biber (1993), “Some of the first
considerations in constructing a corpus concern the overall
design: for example, the kinds of texts included, the number
of texts, the selection of particular texts, the selection of text
samples from within texts, and the length of text samples.

Each of these involves a sampling decision, either

conscious or not.”
11
1
 Corpus Sampling (contd)
While obtaining a representative sample, we need to consider the
following −
Sampling unit − It refers to the unit which requires a sample. For
example, for written text, a sampling unit may be a newspaper, journal
or a book.
Sampling frame − The list of al sampling units is called a sampling
frame.
Population − It may be referred as the assembly of all sampling units.
It is defined in terms of language production, language reception or
language as a product.
11
2
2. Corpus Size
How large the corpus should be?
• There is no specific answer to this question.
• The size of the corpus depends upon the purpose for
which it is intended as well as on some practical
considerations as follows −
✓ Kind of query anticipated from the user.
✓ The methodology used by the users to study the data.
✓ Availability of the source of data.
11
3
Corpus Size (contd)

11
4
3. Sources of Corpus
• LP libraries (NLTK, Gensim, spaCy, etc.) come with their own
corpora, though you generally have to download them separately.

• You can make a corpus out of webscrapings.

• Or you can compile a folder of documents on your computer and

turn it into a corpus.

• Corpora can be composed of a wide variety of file types — .yaml,

.pickle, .txt, .json, .html — even within the same corpus, though
one generally keeps the file types uniform. 11
5
116

Hypothesis Generation and Scoring

 Hypothesis generation
• Hypothesis generation is a pre-decisional process by which we
formulate explanations and beliefs regarding the occurrences
we observe in our environment.
• Hypothesis generation is an educated “guess” of various factors that
are impacting the business problem that needs to be solved using
machine learning.
• In short, you are making wise assumptions as to how certain factors
would affect our target variable and in the process that follows, you
try to prove and disprove them using various statistical and
graphical tools. 11
7
11
8
• Hypothesis generation is a process
beginning with an educated guess
whereas hypothesis testing is a
process to conclude that the
educated guess is true/false or the
relationship between the variables is
statistically significant or not.

• This latter part could be used for

further research using statistical
proof.
11
9
How to Write Hypothesis
What is basic hypothesis format?

A hypothesis often follows a basic format of

"If {this happens} then {this will happen}."

One way to structure your hypothesis is to describe what will happen

to the dependent variable if you change the independent variable
12
0
Characteristics of Hypothesis
• Predicts the relationship and outcome.
• Simple and concise – avoid wordiness.
• Clear with no ambiguity or assumptions about the readers'
knowledge.
• Observable and testable results.
• Relevant and specific to the research question or problem.

12
1
Hypothesis Testing
• Hypothesis testing or significance testing is a method for
testing a claim or hypothesis about a parameter in a
population, using data measured in a sample.

• In this method, we test some hypothesis by determining the

likelihood that a sample statistic could have been selected, if
the hypothesis regarding the population parameter were true.
12
2
What is null vs alternative hypothesis?

Null and alternative hypotheses are used in statistical hypothesis

testing.

The null hypothesis of a test always predicts no effect or no

relationship between variables, while the alternative hypothesis
states your research prediction of an effect or relationship

12
3
• Normal distribution of data ( mean vs Standard Deviation)
4 step process for Hypothesis Generation and Testing
12
9
• A p-value is the probability of obtaining a sample outcome, given
that the value stated in the null hypothesis is true. The p-value for
obtaining a sample outcome is compared to the level of significance.

• When the p-value is less than 5% (p < .05), we reject the null
hypothesis. Thus p < .05 can be considered as the criterion for
deciding to reject the null hypothesis, although note that when p =
.05, the decision is also to reject the null hypothesis.
• When the p value is greater than 5% (p > .05), we retain the null
hypothesis.
• The decision to reject or retain the null hypothesis is called
significance
13
1
 What is a Test Statistic

Specifically, a test statistic tells us how far, or how many standard

deviations, a sample mean is from the population mean.
Example:
• Suppose we measure a sample mean equal to 4 hours per week
that children watch TV.
• To make a decision, we need to evaluate how likely this sample
outcome is, if the population mean stated by the null hypothesis (3
hours per week) is true.
• We use a test statistic to determine this likelihood.
• The larger the value of the test statistic, the further the distance, or
number of standard deviations, a sample mean is from the
population mean stated in the null hypothesis.
• The value of the test statistic is used to make a decision
Make a decision.

• We use the value of the test statistic to make a decision about the
null hypothesis.
• The decision is based on the probability of obtaining a sample
mean, given that the value stated in the null hypothesis is true
Make a decision (contd)

• Significance, or statistical significance, describes a decision made

concerning a value stated in the null hypothesis.

• When the null hypothesis is rejected, we reach significance.

• When the null hypothesis is retained, we fail to reach significance.

References

https://www.ibm.com/in-en/topics/data-visualization
Unit 1 ends