MODULE-3

chapter 3
BASICS OF LEARNING THEORY
Prepared By: Swathi B V
Assistant Professor
Dept of CSD
 CONTENTS
• Introduction to learning and its types
• Introduction to computation learning theory
• Design of a learning system
• Introduction to concept learning
Introduction to machine learning and its
types
• The process of acquiring knowledge and expertise through study,
experience or being taught is called as learning.
• To make machines learn need to simulate the strategies of human
learning in machine. But, will the computers learn?
• First let us address the question-what sort of tasks can the computers
learn? This depends on the nature of problems that the computers
can solve.
• There are two kinds of problems- well posed and ill-posed
• Computers can solve only well-posed problems as these have
well-defined specifications.
 Introduction to machine learning and its
types
• Following components inherent to it.
Class of learning tasks (T)
A measure of performance (P)
A source of experience (E)
standard definition of learning proposed by Tom Mitchell’s
A program can learn from E for the task T and P improves with
experience E.
Learning Model = Hypothesis Set + Learning Algorithm
LEARNING ENVIRONMENT
 Classical machine learning systems
• Classical machine learning system has components such as input,
process, output.
• The input values are taken from the environment directly. These
values are processed and a hypothesis is generated as output model.
• This model is then used for making predictions.
 Adaptive machine learning systems
• Here direct inputs are not available. this process is called
reinforcement learning.
• Here learning agents interact with environment and in return gets
feedback. Based on the feedback, the learning agent generates input
samples for learning.
• Such learning agents are not static and change their behaviour
according to the external signal received from the environment.
• The feedback is known as reward and learning here is the ability of
the learning agent adaptive to the environment based on the reward.
 Learning types
1. learn by memorization: memorizing without understanding the logic
or concept.
2. learn by examples also called as learn by experience. which means
performing inductive learning from observation that formulate a
general concept also called as discovery learning.
3. learn by being taught by an expert or a teacher called as passive
learning. there is special kind of learning called active learning where
the learner can interactively query a teacher to label unlabelled data
instances with the desired outputs.
4. learning by critical thinking also called as deductive learning,
deduces new facts or conclusion from related known facts and
information.
 Learning types
5. self learning also called as reinforcement learning is a self-directed
learning that normally learns from mistakes punishments and rewards.
6. learning to solve problem is a type of cognitive learning. Here the
learner initially is not aware of the solution or the way to achieve the
goal but only knows the goal.
7. learning by generalizing explanations, also called as
explanation-based learning, is another learning method that exploits
domain knowledge from experts to improve the accuracy of learned
concepts by supervised learning.
Introduction to computation learning theory
• There are many questions that have been raised by mathematicians
and logicians over the time taken by computers to learn
Some of the questions are
1. How can a learning system predict an unseen instance?
2. How do the hypothesis h is close to f, when hypothesis f itself is
unknown?
3. How many samples are required?
4. Can we measure the performance of a learning system?
5. Is the solution obtained local or global?
Computational Learning Theory
• These questions are the basis of a field called ‘Computational
Learning Theory’ (COLT).
• COLT deals with formal methods used for learning systems. It deals
with frameworks for quantifying learning tasks and learning
algorithms.
 Design of a learning system

• A system that is built around a learning algorithm is called learning

system.
The design of systems focuses on these steps:
1. Choosing a training experience
2. Choosing a target function
3. Representation of a target function
4. Function Approximation
 Design of a learning system
Step 1) Choosing the Training Experience: The very important and
first task is to choose the training data or training experience which will
be fed to the Machine Learning Algorithm.
• It is important to note that the data or experience that we fed to the
algorithm must have a significant impact on the Success or Failure of
the Model.
• So Training data or experience should be chosen wisely.
• for example: While Playing chess the training data will provide
feedback to itself like instead of this move if this is chosen the chances
of success increases.
 Design of a learning system
• Step 2-Determine the target function: The next step is the
determination of a target function. In this step, the type of knolwdge
that needs to be learnt is determined.
• In direct experience, a board move is selected and is determined
whether it is a good move or not against all other moves. If it is the
best move, then it is chosen as Legal moves.
• In indirect experience, all legal moves are accepted and a score is
generated for each. The move with largest score is then chosen and
executed.
 Design of a learning system
• Step 3- Determine the Target function Representation
The representation of knowledge may be a table, collection of rules or
a neural network.
The linear combination of these factors can be coined as:
V=w0+w1x1+w2x2+w3x3
• Where x1, x2 and x3 represent different board features and w0, w1,
w2and w3 represent weights.
 Design of a learning system
• Step 4- Choosing an Approximation Algorithm for the Target
Function : The focus is to choose weights and fit the given training
samples effectively. The aim is to reduce the error .
• The approximation is carried out as:
1. Computing the error as the difference between trained and
expected hypothesis.
2. Then for every board feature xi, the weights are updated
wi = wi + µ*error(b)* xi
µ is the constant that moderates the size of the weight update
 Design of a learning system
Thus, the learning system has the following components.
1. A performance system to allow the game to play against itself.
2. A critic system to generate samples.
3. A Generalizer system to generate a hypothesis bases on samples.
4. An Experimenter system to generate a anew system based on the
currently learnt functions.
 Introduction to Concept learning
• Concept learning is a learning strategy of acquiring abstract knowledge or
inferring a general concept or deriving a category from the given training
samples.
• It is the process of abstraction and generalization from the data.
• Concept learning helps to classify an object that has a set of common,
relevant features.
• The learner tries to simplify by observing the common features from the
training samples and then apply this simplified model to the future
samples. This task is also known as Learning from Experience.
• Each concept or category obtained by learning is a Boolean valued function
which takes a true or false value.
• For Example, humans can identify different kinds of animals based on
common relevant features and categorize all animals based on specific set
of features.
 Concept learning
• The feature that distinguish one animal from another can be called as
a concept.
• This way of learning categories for object and to recognize new
instances of those categories is called concept learning.
Concept learning requires three things
• Input – training dataset which is a set of training instances, each
labelled with the name of s concept or category to which it belongs.
• Output - target concept or target function f.it is a mapping function
f(x) from input x to output y.
example: the specific set of features to identify an elephant from the all
animals.
• Test - new instances to test the learned model
 Concept learning
• “given a set of hypotheses, the learner searches through the
hypothesis space to identify the best hypothesis that matches the
target concept”. ------- concept learning
 Representation of a hypothesis
• A hypothesis ‘h’ approximates a target function ‘f’ to represent the
relationship between the independent attributes and the dependent
attribute of the training instances.
• The hypothesis is the predicted approximate model that best maps the
inputs to outputs.
• Each hypothesis is represented as a conjunction of attribute conditions in
the antecedent part.
• For example, (tail=short)^(color=black)……
• The set of hypothesis in the search space(The space of all feasible solutions)
is called as hypotheses.
• Hypotheses are the plural form of hypothesis.
• Generally ‘H’ represent the hypothesis and ‘h’ is used to represent a
candidate hypothesis (a candidate model that approximates a target
function for mapping inputs to outputs.).
 Representation of a hypothesis
• Each attribute can take value as either ‘?’ or ‘ϕ’ or can hold a single
value.
• “?” denotes that the attribute can take any value . Eg : color=?
• “ϕ” can not take any value i.e it represents a null value. Eg: horns= ϕ
• Single value denotes the a specific single value. Eg: tail=short
Representation of a hypothesis
 Hypothesis space
• Hypothesis space is the set of all possible hypotheses that
approximates the target function f.
Or
• The set of all possible approximation of the target function can be
defined as hypothesis space.
• The subset of hypothesis space that is consistent with all-observed
training instances is called as version space
Hypothesis space
Hypothesis space
 Heuristic space search
• Is a search strategy that finds an optimized hypothesis/solution to a
problem by iteratively improving the hypothesis/solution based on a
given heuristic function or a cost measure.

• Heuristic search methods will generate a possible hypothesis that can

be a solution in the hypothesis space or a path from the initial state.

• This hypothesis will be tested with the target function or the goal state
to see if it is a real solution. if the tested hypothesis is a real solution,
then it will be selected. this method generally increases the efficiency
because it is guaranteed to find a better hypothesis but may not be the
best hypothesis.
 Heuristic space search
• It is useful for solving tough problems which could not solved by any
other method. Example: travelling salesman problem.

• Several commonly used heuristic search methods are hill climbing

methods, constraint satisfaction problems, best-first search, A*
Algorithm, genetic algorithms.
 Generalization and specialization
Searching the hypothesis space
There are two ways of learning the hypothesis

1. Specialization-general to specific learning

2. Generalization-specific to general learning

 Generalization-specific to general learning
• This learning methodology will search through the hypothesis space
for an approximate hypothesis by generalizing the most specific
hypothesis.
Example: consider the training instances shown below and illustrate
specific to general learning
Vanishri S DEPT OF
ISE
Specialization-general to specific learning
Hypothesis space search by FIND-S Algorithm

• This algorithm considers only the positive instances and eliminates

negative instances wile generating the hypothesis.
• It finds most specific hypothesis.
• It initially starts with the most specific hypothesis.
 Find S Algorithm limitations
1. Find s algorithm tries to find a hypothesis that is consistent with
positive instances, ignoring all negative instances.
2. The algorithm finds only one unique hypothesis, wherein there may
be many other hypotheses that are consistent with the training
dataset.
3. Vulnerability to Noise The algorithm is sensitive to noisy training
data, and outliers or errors in the examples can lead to the generation
of inaccurate hypotheses.
4. inconsistent training sets can actually mislead the find-s algorithm,
since it ignores the negative examples.
to overcome the limitations of find s algorithm, candidate elimination
algorithm was proposed to output the set of all hypotheses consistent
with the training dataset.
 Version spaces
• The version space contains the subset of hypotheses from the
hypothesis space that is consistent with all training instances in the
training data set.
List then eliminate algorithm
• The principle idea of this learning algorithm is to initialize the version
space to contain all hypotheses and then eliminate the any hypothesis
that is found inconsistent with any training instances.
• Initially the algorithm starts with version space to contain all
hypothesis scanning each training instance the hypothesis that are
inconsistent with the training instance are eliminated,finally,
algorithm outputs the list of remaining hypothesis that are all
consistent.
 List then eliminate algorithm

This algorithm works fine if the hypothesis space is finite but practically it is
difficult to deploy this algorithm.
Hence, a variation of this idea is introduced in the candidate
elimination algorithm.
 Version space and candidate elimination
algorithm
Version space learning is to generate all consistent hypothesis around
this algorithm computes the version space by the combination of two
cases namely,

1. Specific to general learning ----- generalize S to include the positive

example

2. General to specific learning----- specialize G to exclude the negative

example
 Version space and candidate elimination
algorithm
• Using candidate elimination algorithm, we can compute the version
space containing all (and only those) hypothesis from H that are
consistent with given observed sequence of training instance.
• The algorithm defines to boundaries called ‘general boundary’
which is set of all hypothesis that are the most general and ‘specific
boundary’ which is the set of all hypothesis that are the most specific.
Thus, the algorithm limits the version space to contain only those
hypothesis that are most general and most specific thus, it provides
compact representation of List-then algorithm.
Version space and candidate elimination
algorithm
Version space and candidate elimination
algorithm
Solve candidate elimination
candidate elimination
MODULE-3 CHAPTER-2
SIMILARITY BASED LEARNING
 Content
• Introduction to similarity or instance based learning
• Nearest-neighbor learning
• Weighted K-Nearest Neighbor Algorithm
• Nearest centroid classifier
 Similarity or Instance-based Learning
• Similarity-based classifiers use similarity measures to locate the
nearest neighbors and classify a test instance.
• Also called as Instance-based learning/just-in time learning hence it
does not build an abstract model of the training instances and
performs lazy learning when classifying a new instance.
• This learning mechanism simply stores all data and uses it only when
it needs to classify an unseen instance.
• The advantage of using this learning is that processing occurs only
when a request to classify a new instance is given.
• This methodology is useful when the whole dataset is not available in
the beginning but collected in an incremental manner.
• The drawback of this learning is that it requires a large memory to
store the data since a global abstract model is not constructed initially
with the training data.
Difference between Instance-and Model-based Learning
Some examples of Instance-based Learning
algorithms
1. K-Nearest Neighbor(KNN)
2. Variants of nearest neighbor learning
3. Locally weighted regression
4. Learning vector quantization (LVQ)
5. Self-organizing maps (SOM)
6. Radial Basis Function (RBF) networks
 Nearest-neighbor learning
∙ A powerful classification algorithm used in pattern recognition.
∙ K nearest neighbors stores all available cases and classifies new cases
based on a similarity measure (e.g distance function)
∙ A non-parametric lazy learning algorithm (An Instance based Learning method).
∙ That predicts the category of the test instance according to the ‘k’ training
samples which are closer to the test instance and classifies it to that category
which has the largest probability.

Here, 2 classes of objects called C1 and C2. When

given a test instance T, the category of this test
instance is determined by looking at the class of
k=3 nearest neighbors. Thus, the class of this test
instance T is predicted as C2.
Nearest-neighbor learning
 Nearest-neighbor learning
• The most popular distance measure such as Euclidean distance is
used in K-NN to determine the ‘k’ instances which are similar to the
test instance, the value of ‘k’ is best determined by tunning with
different ‘k’ values and choosing the ‘k’ which classifies the best test
instance more accurately.
K-Nearest-neighbor learning
K-Nearest-neighbor learning
K-Nearest-neighbor learning
 Weighted k-Nearest-Neighbor Algorithm

• The weighted KNN is an extension of k-NN.It chooses the neighbors

by using the weighted distance.
• In weighted kNN, the nearest k points are given a weight using a
function called as the kernel function(similarity based learning).
• The intuition behind weighted kNN, is to give more weight to the
points which are nearby and less weight to the points which are farther
away.
Weighted k-Nearest-Neighbor Algorithm
Weighted k-Nearest-Neighbor Algorithm
Weighted k-Nearest-Neighbor Algorithm
Nearest centroid classifier
Nearest centroid classifier
Nearest centroid classifier
 MODULE-3
CHAPTER-5
REGRESSION ANALYSIS
 CONTENTS
•Introduction to regression
•Introduction to linearity, correlation, and causation
•Introduction to linear regression
• In causality analysis, the interaction between variables can be
determined. While x determines y, y can determine x. In regression
analysis, there is a one-sided interaction. There are dependent
variable and independent variables.
 Introduction to regression
• Regression analysis is a fundamental concept that consists of a
set of machine learning methods that predict a continuous
outcome variable (y) based on the value of one or multiple
predictor variables (x).
or
• Regression analysis is a statistical method to model the
relationship between a dependent (target) and independent
(predictor) variables with one or more independent variables.
• Regression is a supervised learning technique which helps in
finding the correlation between variables.
 Introduction to regression
• Function of regression analysis is given by:
y=f(x)
Here, x is called as Explanatory variable, exploratory variable, a
predicator variable, independent variable, a covariate or a domain
point.
y is called dependent variable, labels, target variables or response
variables.
 Introduction to regression
Regression is used to predict continuous variables or quantitative
variables such as price and revenue.
The primary concern of regression analysis is to find answer to
questions such as
• What is the relationship between the variables?
• What is the strength of the relationship?
• what is the nature of the relationship such as linear or non-linear?
• What is the relevance of the attributes?
• What is the contribution of the each attribute?
 Applications of Regression Analysis
❑ Sales of a goods or services
❑ Premium on insurance companies
❑ Yield of crop in agriculture
❑ Prices of real estate
 INTRODUCTION TO LINEARITY, CORRELATION AND
CAUSATION
The quality of the regression analysis is determined by the factors such as
correlation and causation.
A correlation is the statistical summary of the relationship between two sets of
variables. It is a core part of data exploratory analysis, and is a critical aspect of
numerous advanced machine learning techniques.
• Correlation between two variables can be found using a scatter plot. The x-axis
represents input or independent variable, predicator variable.
• The y-axis represents output or dependent or predicted variables.
There are different types of correlation:
• Positive Correlation: Two variables are said to be positively correlated when their
values move in the same direction. For example, in the image below, as the value
for X increases, so does the value for Y at a constant rate.
• Negative Correlation: Finally, variables X and Y will be negatively correlated when
their values change in opposite directions, so here as the value for X increases, the
value for Y decreases at a constant rate.
• Neutral Correlation: No relationship in the change of variables X and Y. In this
case, the values are completely random and do not show any sign of correlation,
as shown in the following image:
INTRODUCTION TO LINEARITY, CORRELATION AND
CAUSATION
 Causation
• Causation is about relationship between two variables as x causes y.
This is called x implies y.
• Regression is different from causation. Causation indicates that one
event is the result of the occurrence of the other event; i.e. there is a
causal relationship between the two events.
• Regression is not same as causation relationship
• Example, the correlation between economical background and marks
scored does not imply that economic background causes high marks.
 Linear and Non-Linear Relationships
The relationship between input features (variables) and the output (target) variable
is fundamental. These concepts have significant implications for the choice of
algorithms, model complexity, and predictive performance.
Linear relationship creates a straight line when plotted on a graph. Non-Linear
relationship does not create a straight line but instead creates a curve.
Example:
• Linear-the relationship between the hours spent studying and the grades
obtained in a class.

• Non-Linear- GPS Signal.

 Linearity
• Linear Relationship: A linear relationship between variables means that a
change in one variable is associated with a proportional change in another
variable. Mathematically, it can be represented as y = a * x + b, where y is
the output, x is the input, and a and b are constants.
• Linear Models: Goal is to find the best-fitting line (plane in higher
dimensions) to the data points. Linear models are interpretable and work
well when the relationship between variables is close to being linear.
• Limitations: Linear models may perform poorly when the relationship
between variables is non-linear. In such cases, they may underfit the data,
meaning they are too simple to capture the underlying patterns.
 Non-Linearity
• Non-Linear Relationship: A non-linear relationship implies that the change
in one variable is not proportional to the change in another variable.
Non-linear relationships can take various forms, such as quadratic,
exponential, logarithmic, or arbitrary shapes.
• Non-Linear Models: Machine learning models like decision trees, random
forests, support vector machines with non-linear kernels, and neural
networks can capture non-linear relationships. These models are more
flexible and can fit complex data patterns.
• Benefits: Non-linear models can perform well when the underlying
relationships in the data are complex or when interactions between
variables are non-linear. They have the capacity to capture intricate
patterns.
Types of Regression
 Types of Regression
1. Single Independent Variable: Linear regression, also known as simple linear
regression, is used when there is a single independent variable (predictor) and
one dependent variable (target).
2. Multiple Independent Variables: Multiple regression, as the name suggests, is
used when there are two or more independent variables (predictors) and one
dependent variable (target).
3. Polynomial regression: is an extension of multiple regression used when the
relationship between the independent and dependent variables is non-linear.
4. Logistic regression: is used when the dependent variable is binary (0 or 1). It
models the probability of the dependent variable belonging to a particular
class.
5. Lasso and Ridge Regression :these are special variant of regression method
where regularization(strategies ) methods are used to limit the number and size of
coeffients of the independent variables.
Limitations of Regression
 INTRODUCTION TO LINEAR REGRESSION

• The linear regression model can be created by fitting a line among the
scattered data points.
y=(a0+a1x)+e
Here a0 is the intercept which represents the bias and a1 represents
the slope of the line .these are called regression coefficients, e is the
error in prediction.
INTRODUCTION TO LINEAR REGRESSION
 Ordinary Least Square Approach
• The idea of linear regression is based on Ordinary Least Square(OLS)
approach.
• In this method, the data points are modelled using a straight line. any
arbitrarily draw line is not an optimal line.
• The vertical distance between each point and the line is called error.
• These individual error are added to compute the total error of the
predicted er ror.
• This is called sum of residuals
 Ordinary Least Square Approach

Mathematically the line of equations for points are:

y1=(a0+a1x1)+e1
y2=(a0+a1x2)+e2 and so on
……. yn=(a0+a1xn)+en.
In general
ei= yi - (a0+a1x1)
Vanishri S DEPT OF
ISE
Linear Regression Example
Linear Regression Example
Linear Regression Example
Linear Regression Example