UNIT II: Automated Learning-1 and 2

Automated Learning-1: Supervise vs. unsupervised learning, Decision Tree

induction, rule induction algorithms.
Automated Learning-2: Bayesian network learning algorithms, Evolutionary
algorithms.
Q) What is Supervise vs. unsupervised learning?
Supervised and Unsupervised learning are the two techniques of machine
learning. But both the techniques are used in different scenarios and with
different datasets.
Supervised Machine Learning:
Supervised learning is a machine learning method in which models are trained
using labeled data. In supervised learning, models need to find the mapping
function to map the input variable (X) with the output variable (Y).

Supervised learning needs supervision to train the model, which is similar to as

a student learns things in the presence of a teacher. Supervised learning can be
used for two types of problems: Classification and Regression.
Example: Suppose we have an image of different types of fruits. The task of
our supervised learning model is to identify the fruits and classify them
accordingly. So to identify the image in supervised learning, we will give the
input data as well as output for that, which means we will train the model by the
shape, size, color, and taste of each fruit. Once the training is completed, we
will test the model by giving the new set of fruit. The model will identify the
fruit and predict the output using a suitable algorithm.
Unsupervised Machine Learning:
Unsupervised learning is another machine learning method in which patterns
inferred from the unlabeled input data. The goal of unsupervised learning is to
find the structure and patterns from the input data. Unsupervised learning does
not need any supervision. Instead, it finds patterns from the data by its own.
Unsupervised learning can be used for two types of
problems: Clustering and Association.
Example: To understand the unsupervised learning, we will use the example
given above. So unlike supervised learning, here we will not provide any
1
 supervision to the model. We will just provide the input dataset to the model and
allow the model to find the patterns from the data. With the help of a suitable
algorithm, the model will train itself and divide the fruits into different groups
according to the most similar features between them.

The main differences between Supervised and Unsupervised learning are

SL. Supervised Learning Unsupervised Learning

NO

1 Supervised learning algorithms are Unsupervised learning algorithms are

trained using labeled data. trained using unlabeled data.

2 Supervised learning model takes

Unsupervised learning model does not
direct feedback to check if it is
take any feedback.
predicting correct output or not.

3 Supervised learning model predicts Unsupervised learning model finds the

the output. hidden patterns in data.

4 In supervised learning, input data is

In unsupervised learning, only input
provided to the model along with the
data is provided to the model.
output.

5 The goal of supervised learning is to The goal of unsupervised learning is to

train the model so that it can predict find the hidden patterns and useful
the output when it is given new data. insights from the unknown dataset.

6 Supervised learning needs Unsupervised learning does not need

supervision to train the model. any supervision to train the model.

7 Supervised learning can be Unsupervised Learning can be

categorized classified
in Classification and Regression pr in Clustering and Associations proble
oblems. ms.

Supervised learning can be used for Unsupervised learning can be used for

2
8 those cases where we know the input those cases where we have only input
as well as corresponding outputs. data and no corresponding output data.

Unsupervised learning model may give

Supervised learning model produces
less accurate result as compared to
9 an accurate result.
supervised learning.

Supervised learning is not close to

Unsupervised learning is more close to
true Artificial intelligence as in this,
the true Artificial Intelligence as it
we first train the model for each
learns similarly as a child learns daily
10 data, and then only it can predict the
routine things by his experiences.
correct output.

It includes various algorithms such

as Linear Regression, Logistic It includes various algorithms such as
11
Regression, Support Vector Clustering, KNN, and Apriori
Machine, Multi-class Classification, algorithm.
Decision tree, Bayesian Logic, etc.

Note: The supervised and unsupervised learning both are the machine learning
methods, and selection of any of these learning depends on the factors related to
the structure and volume of your dataset and the use cases of the problem.

Regression Analysis in Machine learning

3
Regression analysis is a statistical method to model the relationship between a
dependent (target) and independent (predictor) variables with one or more
independent variables. More specifically, Regression analysis helps us to
understand how the value of the dependent variable is changing corresponding
to an independent variable when other independent variables are held fixed. It
predicts continuous/real values such as temperature, age, salary, price, etc.
Example: Suppose there is a marketing company A, who does various
advertisement every year and get sales on that. The below list shows the
advertisement made by the company in the last 5 years and the corresponding
sales:

Now, the company wants to do the advertisement of $200 in the year 2019 and
wants to know the prediction about the sales for this year. So to solve such
type of prediction problems in machine learning, we need regression analysis.
Some examples of regression can be as:
1. Prediction of rain using temperature and other factors
2. Determining Market trends
3. Prediction of road accidents due to rash driving.

Why do we use Regression Analysis?

4
As mentioned above, Regression analysis helps in the prediction of a continuous
variable. There are various scenarios in the real world where we need some
future predictions such as weather condition, sales prediction, marketing trends,
etc.
1. Regression estimates the relationship between the target and the
independent variable.
2. It is used to find the trends in data.
3. It helps to predict real/continuous values.
4. By performing the regression, we can confidently determine the most
important factor, the least important factor, and how each factor is
affecting the other factors.
Types of Regression
There are various types of regressions which are used in data science and
machine learning. Each type has its own importance on different scenarios, but
at the core, all the regression methods analyze the effect of the independent
variable on dependent variables. Here we are discussing some important types
of regression which are given below:
1. Linear Regression
2. Logistic Regression
3. Polynomial Regression
4. Support Vector Regression
5. Decision Tree Regression
6. Random Forest Regression
7. Ridge Regression
8. Lasso Regression:

5
Decision Tree induction
6
Decision Tree Induction in Soft Computing involves integrating techniques from
soft computing (such as fuzzy logic, genetic algorithms, and neural networks) with
decision tree algorithms to handle imprecision, uncertainty, and complex problem
spaces more effectively.
1. Overview of Decision Tree Induction
 Decision Trees are a supervised learning method used for classification and
regression tasks.
 The tree structure:
o Nodes: Represent attributes.
o Edges: Represent attribute values.
o Leaves: Represent class labels (or predicted outcomes).
 Advantages:
o Simple to understand and interpret.
o Handles both categorical and numerical data.
o Requires minimal data preprocessing.

2. Role of Soft Computing in Decision Tree Induction

7
Soft computing enhances decision trees by introducing flexibility, tolerance for
uncertainty, and better generalization. Techniques include:
a. Fuzzy Logic Integration
 Fuzzy decision trees handle imprecise or vague data.
 Key Features:
o Continuous attributes are represented as fuzzy sets (e.g., "high,"
"medium," "low").
o Decision boundaries are soft and non-crisp.
 Benefits:
o Improved handling of noisy or uncertain data.
o Better performance with real-world, imprecise datasets.
b. Genetic Algorithms
 Genetic algorithms optimize decision tree construction by:
o Selecting the best attributes for splitting.
o Pruning unnecessary branches to prevent overfitting.
 Process:
o Trees are represented as chromosomes.
o Fitness functions evaluate their accuracy and complexity.
o Crossover and mutation generate diverse solutions.
c. Neural Networks and Hybrid Models
 Neural networks can complement decision trees by:
o Preprocessing data for attribute extraction.
o Combining rule-based tree structures with continuous learning.
 Hybrid systems integrate neural trees with soft computing techniques for
complex decision-making scenarios.
d. Rough Set Theory
 Used for attribute reduction and rule extraction.
 Enhances decision-making by identifying significant attributes without losing
essential information.
3. Applications
8
  Medical Diagnosis: Handling uncertain symptoms or lab results.
 Financial Analysis: Risk assessment with vague or incomplete market data.
 Industrial Automation: Managing noisy sensor inputs in control systems.
 Pattern Recognition: Dealing with overlapping class boundaries.
4. Challenges and Considerations
 Scalability: Soft computing techniques may increase computational
complexity.
 Interpretability: Combining soft computing with decision trees may make the
model harder to interpret.
 Parameter Tuning: Fuzzy sets and genetic algorithm parameters require
careful calibration.

5. Example Framework
A fuzzy decision tree induction algorithm might include:
1. Fuzzification: Convert crisp inputs into fuzzy sets.
2. Tree Construction: Split nodes using fuzzy information gain or entropy.
3. Pruning: Use genetic algorithms to simplify the tree.
4. Defuzzification: Map fuzzy outputs back to crisp decisions.
************************************
Decision Tree Algorithm
The Decision Tree Algorithm is a popular machine learning technique used for both
classification and regression tasks. It works by recursively splitting the dataset into
subsets based on specific feature values, ultimately forming a tree structure. The goal
is to partition the data such that the subsets are as homogeneous as possible with
respect to the target variable.
1. Root Node:
 Represents the entire dataset.
 Splits into child nodes based on the best feature and value.
2. Internal Nodes:
 Represent decision points on features.
 Each node corresponds to a condition (e.g., "Is feature x≤cx \leq cx≤c?").
9
3. Leaf Nodes:
 Represent the outcome or prediction (e.g., a class label or a regression value).
4. Splitting Criterion:
 Determines how the dataset is divided at each node.
 Common criteria:
o Gini Impurity: Measures the probability of misclassification.
o Entropy (Information Gain): Measures the reduction in randomness.
o Variance Reduction: Used in regression tasks to minimize the variance
within subsets.
5. Pruning:
 Reduces the size of the tree to avoid overfitting.
 Can be done pre-emptively (before the tree grows) or post-hoc (by trimming
branches).
Advantages
 Easy to Interpret: Intuitive structure that can be visualized.
 Non-Parametric: Makes no assumptions about data distribution.
 Feature Importance: Provides insight into which features are most important
for predictions.
Disadvantages
 Overfitting: Fully grown trees may overfit the training data.
 Bias to Dominant Features: May favor features with more levels/categories.
 Instability: Small changes in data can lead to different trees.
Common Variants
1. CART (Classification and Regression Trees):
o Used for both classification and regression.
o Binary splits at each node.
2. ID3 (Iterative Dichotomiser 3):
o Uses entropy for splitting and creates multi-way splits.
3. C4.5:

10
 o An improvement of ID3, handling both continuous and categorical
features.
4. Random Forest:
o An ensemble method that builds multiple decision trees and averages
their predictions.
5. Gradient Boosted Trees:
o Builds trees sequentially, optimizing errors of previous trees.

*************

Rule induction algorithms

Rule Induction Algorithms are a set of methods in machine learning that extract
interpretable rules from data. These algorithms aim to discover patterns or
relationships in data in the form of if-then rules, which are particularly useful in
classification, prediction, and decision support systems.

Key Concepts of Rule Induction

11
  A rule has two parts:
o Antecedent (Condition): The "if" part (e.g., IF age > 30 AND income < 50k).
o Consequent (Result): The "then" part (e.g., THEN class = Low_Risk).

Types of Rule Induction Algorithms

1. Separate-and-Conquer Algorithms
 Work by repeatedly finding rules that cover subsets of data (conquer) and
removing the covered data (separate).
 Examples:
o RIPPER (Repeated Incremental Pruning to Produce Error
Reduction):
 Efficient and handles noisy data well.
 Creates rules iteratively, prunes them for generalization, and
optimizes error reduction.
o CN2 Algorithm:
 Focuses on discovering concise and interpretable rules.
 Uses a beam search and evaluates rules with metrics like entropy
or statistical significance.
2. Decision Tree-Based Rule Induction
 Extract rules directly from decision trees by tracing paths from the root to the
leaves.
 Examples:
o C4.5 (Quinlan):
 Decision tree induction algorithm.
 Rules are extracted post tree-construction, and pruning is applied
to avoid overfitting.
o CART (Classification and Regression Trees):
 Similar to C4.5 but focuses on binary splits for simplicity.

3. Association Rule Learning

12
  Identifies rules in the form of IF antecedent THEN consequent based on
support and confidence thresholds.
 Examples:
o Apriori Algorithm:
 Generates frequent itemsets and derives association rules.
 Iterative approach with candidate generation and pruning.
o FP-Growth (Frequent Pattern Growth):
 More efficient than Apriori for large datasets.
 Builds a prefix tree structure (FP-tree) to discover patterns
without candidate generation.
4. Fuzzy Rule Induction
 Generates rules with fuzzy antecedents to handle imprecision.
 Example:
o Fuzzy Rule-Based Systems (FRBS):
 Utilizes fuzzy logic to allow rules like IF temperature is high
THEN fan_speed is fast.
5. Genetic Algorithms for Rule Induction
 Uses evolutionary computation to evolve and optimize a population of rules.
 Advantages:
o Can handle non-linear and complex relationships.
o Supports multi-objective optimization (e.g., accuracy vs. simplicity).
6. Bayesian Rule Induction
 Combines probability theory and rule induction.
 Example:
o Bayesian Rule Sets (BRS):
 Derives probabilistic rules by balancing accuracy and model
complexity.

Advantages of Rule Induction Algorithms

13
 1. Interpretability: Rules are easy to understand and explain.
2. Actionable Knowledge: Rules directly suggest decision-making strategies.
3. Flexibility: Can be used for classification, regression, and clustering tasks.
Applications
1. Healthcare: Rule-based diagnosis and treatment recommendations.
2. Marketing: Customer segmentation and targeting strategies.
3. Finance: Credit risk assessment and fraud detection.
4. Industrial Systems: Control systems and fault diagnosis.

Bayesian Network Learning Algorithms

14
Bayesian networks are probabilistic graphical models that represent a set of variables
and their conditional dependencies using a directed acyclic graph (DAG). Learning a
Bayesian network from data involves two primary tasks:
1. Structure Learning: Determining the structure of the graph (i.e., which
variables are connected and the direction of the edges).
2. Parameter Learning: Estimating the conditional probability distributions
(CPDs) for the variables given the structure.
There are several algorithms to learn Bayesian networks from data. These can be
divided into constraint-based, score-based, and hybrid approaches.
1. Structure Learning Algorithms
Constraint-Based Algorithms
 Use statistical tests to infer dependencies and independencies in the data.
 Construct the graph based on these relationships.
Examples:
 PC Algorithm: Iteratively tests conditional independencies to prune edges, and
then orients edges to ensure acyclicity.
 Grow-Shrink Algorithm: Starts with an empty network and adds/removes
edges based on dependency tests.
Advantages:
 Theoretical guarantees when data is sufficient.
 Clear relationship with statistical properties.
Disadvantages:
 Sensitive to errors in dependency tests.
 Computationally expensive for large networks.
Score-Based Algorithms
 Define a scoring function to evaluate the quality of a graph structure given the
data.
 Use search strategies to find the structure that maximizes the score.

Examples:
 Greedy Hill Climbing: Starts with an empty or random graph and iteratively
adds/removes/reverses edges to maximize the score.
15
  Simulated Annealing: Searches the space of structures while avoiding local
maxima by allowing occasional downhill moves.
 Exact Search (Dynamic Programming): Searches exhaustively for the best
structure (feasible only for small networks).
Common Scoring Functions:
 Bayesian Information Criterion (BIC)
 Akaike Information Criterion (AIC)
 Bayesian Dirichlet Equivalent (BDe)
Advantages:
 Flexibility in combining prior knowledge.
 Can work with partial or noisy data.
Disadvantages:
 Computationally expensive for large networks.
 Risk of overfitting with insufficient data.

Hybrid Algorithms
 Combine constraint-based and score-based approaches to leverage the strengths
of both.
16
Examples:
 Max-Min Hill Climbing (MMHC): Uses constraint-based methods to identify
a candidate set of edges and then applies score-based methods to refine the
structure.
Advantages:
 Balances computational efficiency and accuracy.
 Reduces sensitivity to errors in dependency tests.
2. Parameter Learning Algorithms
Once the structure is learned or predefined, the next step is to estimate the CPDs for
each variable. Parameter learning depends on whether the data is complete or
incomplete.
For Complete Data:
 Maximum Likelihood Estimation (MLE): Estimates parameters directly
from the data.
 Bayesian Estimation: Incorporates prior distributions to estimate CPDs.
For Incomplete Data:
 Expectation-Maximization (EM): Iteratively estimates missing data and
updates parameters to maximize the likelihood.

Q . Evolutionary algorithms in Soft Computing

Evolutionary algorithms (EAs) are a prominent subfield of soft computing, which
focuses on solving complex real-world problems where traditional methods may fall
17
short. Soft computing emphasizes flexibility, approximation, and robustness, and
evolutionary algorithms fit well due to their nature of iterative improvement and
adaptability.
Overview of Evolutionary Algorithms
Evolutionary algorithms are optimization and search techniques inspired by the
principles of natural selection and genetics. They emulate processes observed in
biological evolution, such as reproduction, mutation, recombination, and selection, to
evolve solutions to problems over generations.
Key Concepts in Evolutionary Algorithms
1. Population: A set of candidate solutions (individuals) to the problem.
2. Chromosomes: Representation of candidate solutions, typically as strings
(binary, real-valued, or symbolic).
3. Fitness Function: A measure of how good a candidate solution is at solving the
given problem.
4. Selection: A process of choosing individuals from the population to create
offspring, favoring those with higher fitness.
5. Crossover (Recombination): Combining parts of two or more parent solutions
to create offspring.
6. Mutation: Introducing small random changes to individuals to maintain
diversity in the population.
7. Generations: Iterative cycles where populations evolve toward better
solutions.
Types of Evolutionary Algorithms
1. Genetic Algorithms (GAs):
o Use binary or real-valued encoding of solutions.
o Employ selection, crossover, and mutation extensively.
o Widely used for optimization and search problems.
2. Evolution Strategies (ESs):
o Focus on optimizing real-valued parameters.
o Emphasize mutation and self-adaptation of strategy parameters.
o Commonly applied to continuous optimization.
3. Genetic Programming (GP):

18
 o Evolve computer programs or symbolic expressions to solve problems.
o Represent solutions as tree structures.
4. Differential Evolution (DE):
o Specializes in handling continuous optimization problems.
o Operates on vector-based representations and relies on mutation and
crossover.
5. Particle Swarm Optimization (PSO):
o Inspired by social behavior of animals like birds and fish.
o Solutions (particles) adjust their positions based on individual and
collective experiences.
6. Memetic Algorithms:
o Combine evolutionary algorithms with local search techniques to refine
solutions.
Applications of Evolutionary Algorithms in Soft Computing
1. Optimization Problems:
o Scheduling (e.g., job-shop scheduling, task allocation).
o Network design (e.g., routing, topology optimization).
o Machine learning (e.g., hyperparameter tuning, feature selection).
2. Engineering Design:
o Structural optimization.
o Control systems design.
3. Robotics:
o Path planning.
o Evolution of autonomous behaviors.
4. Data Mining and Machine Learning:
o Rule extraction.
o Model optimization.
5. Game Development:
o Strategy optimization.
o Procedural content generation.
19
 6. Bioinformatics:
o Protein structure prediction.
o Gene sequencing.
Advantages of Evolutionary Algorithms
 Flexibility: Can handle a wide variety of problem types.
 Global Optimization: Capable of avoiding local minima.
 Scalability: Applicable to high-dimensional and complex problems.
 Robustness: Performs well under noisy or uncertain environments.
Challenges in Evolutionary Algorithms
 Computational Cost: May require significant computational resources for
large-scale problems.
 Parameter Tuning: Performance depends on setting appropriate parameters
like population size, mutation rate, etc.
 Premature Convergence: Risk of converging to suboptimal solutions.
 Representation Dependency: Effectiveness can depend heavily on the
representation of solutions.

20