UNIT III

UNIT III - PREDICTIVE MODELING AND MACHINE LEARNING (9Hrs)

Linear Regression – Polynomial Regression – Multivariate Regression – Multi Level
Models – Data warehousing overview – Bias / variance trade off – K Fold cross
validation – Data Cleaning and Normalization – Cleaning web log Data – Normalizing
numerical Data – Detecting Outliers – Introduction to Supervised and Unsupervised
learning.

What is Regression?

Regression analysis is a form of predictive modelling technique which

investigates the relationship between a dependent and independent
variable.

The above definition is a bookish definition, in simple terms the

regression can be defined as, “Using the relationship between variables
to find the best fit line or the regression equation that can be used to
make predictions”.

There are many types of regressions such as ‘Linear

Regression’, ‘Polynomial Regression’, ‘Logistic regression’ and
others but in this blog, we are going to study “Linear Regression”
and “Polynomial Regression”.
Linear Regression

Linear regression is a basic and commonly used type of predictive

analysis which usually works on continuous data. We will try to
understand linear regression based on an example:

Aarav is a trying to buy a house and is collecting housing data so that

he can estimate the “cost” of the house according to the “Living area”
of the house in feet.

He observes the data and comes to the conclusion that the data is
linear after he plots the scatter plot. For his first scatter plot, Aarav
uses two variables: ‘Living area’ and ‘Price’.
As soon as he saw a pattern in the data, he planned to make a
regression line on the graph so that he can use the line to predict the
‘price of the house’.
Using the training data i.e ‘Price’ and ‘Living area’, a regression line
is obtained which will give the minimum error. To do that he needs
to make a line that is closest to as many points as possible. This
‘linear equation’ is then used for any new data so that he is able to
predict the required output.

Here, the β1 it’s are the parameters (also called weights) βo is the y-
intercept and Єi is the random error term whose role is to add bias.
The above equation is the linear equation that needs to be obtained
with the minimum error.

The above equation is a simple “equation of a line” that is

Y(predicted) = (β1*x + βo) + Error value

Where ‘β1’ is the slope and ‘βo’ is the y-intercept similar to the
equation of a line. The values ‘β1’ and ‘βo’ must be chosen so that
they minimize the error. To check the error we have to calculate the
sum of squared error and tune the parameters to try to reduce the
error
Error = Σ (actual output — predicted
output)²

Key:
1. Y(predicted) is also called the hypothesis function.
2. J(θ) is the cost function which can also be called the error
function. Our main goal is to minimize the value of the cost.
3. y(i) is the predicted output.
4. hθ(x(i)) is called the hypothesis function which is basically
the Y(predicted) value.

Now the question arises, how do we reduce the error value. Well,
this can be done by using Gradient Descent. The main goal of
Gradient descent is to minimize the cost value. i.e. min
J(θo, θ1)
Choosing a perfect learning rate is a very important task as it
depends on how large of a step we take downhill during each
iteration. If we take too large of a step, we may step over the
minimum. However, if we take small steps, it will require many
iterations to arrive at the minimum.

Polynomial Linear Regression

In the last section, we saw two variables in your data set were
correlated but what happens if we know that our data is correlated,
but the relationship doesn’t look linear? So hence depending on
what the data looks like, we can do a polynomial regression on the
data to fit a polynomial equation to it.
Hence If we try to use a simple linear regression in the above graph
then the linear regression line won’t fit very well. It is very difficult
to fit a linear regression line in the above graph with a low value of
error. Hence we can try to use the polynomial regression to fit a
polynomial line so that we can achieve a minimum error or
minimum cost function. The equation of the polynomial regression
for the above graph data would be:

Advantages of using Polynomial Regression:

 Polynomial provides the best approximation of the relationship
between the dependent and independent variable.
 A Broad range of function can be fit under it.
 Polynomial basically fits a wide range of curvature.
Disadvantages of using Polynomial Regression
 The presence of one or two outliers in the data can seriously
affect the results of the nonlinear analysis.
 These are too sensitive to the outliers.
 In addition, there are unfortunately fewer model validation tools
for the detection of outliers in nonlinear regression than there
are for linear regression.

multivariate regression

To measure the relationships between these multidimensional

variables, multivariate regression is used.

Multivariate regression is a technique used to measure the degree to

which the various independent variable and various dependent
variables are linearly related to each other. The relation is said to be
linear due to the correlation between the variables. Once the
multivariate regression is applied to the dataset, this method is then
used to predict the behaviour of the response variable based on its
corresponding predictor variables.

Multivariate regression is commonly used as a supervised algorithm in

machine learning, a model to predict the behaviour of dependent
variables and multiple independent variables.

Characteristics of multivariate regression

 Multivariate regression allows one to have a different view of the
relationship between various variables from all the possible angles.
 It helps you to predict the behaviour of the response variables
depending on how the predictor variables move.
 Multivariate regression can be applied to various machine learning
fields, economic, science and medical research studies.

Example of multivariate regression

An agriculture expert decides to study the crops that were ruined in a certain
region. He collects the data about recent climatic changes, water supply,
irrigation methods, pesticide usage, etc. To understand why the crops are
turning black, do not yield any fruits and dry out soon.

In the above example, the expert decides to collect the mentioned data, which
act as the independent variables. These variables will affect the dependent
variables which are nothing but the conditions of the crops. In such a case,
using single regression would be a bad choice and multivariate regression
might just do the trick.

Steps to achieve multivariate regression

Step 1: Select the features
First, you need to select that one feature that drives the multivariate
regression. This is the feature that is highly responsible for the change in
your dependent variable.
Step 2: Normalize the feature
Now that we have our selected features, it is time to scale them in a certain
range (preferably 0-1) so that analysing them gets a bit easy.

To change the value of each feature, we can use:

Step 3: Select loss function and formulate a hypothesis
A formulated hypothesis is nothing but a predicted value of the response
variable and is denoted by h(x).

A loss function is a calculated loss when the hypothesis predicts a wrong

value. A cost function is a cost handled for those wrongly predicting
hypotheses.
Step 4: Minimize the cost and loss function
Both cost function and loss function are dependent on each other. Hence, in
order to minimize both of them, minimization algorithms can be run over the
datasets. These algorithms then adjust the parameters of the hypothesis.

One of the minimization algorithms that can be used is the gradient descent
algorithm.
Step 5: Test the hypothesis
The formulated hypothesis is then tested with a test set to check its accuracy
and correctness.
Advantages:
 The multivariate regression method helps you find a relationship
between multiple variables or features.
 It also defines the correlation between independent variables and
dependent variables.
Disadvantages:
 Multivariate regression technique requires high-level mathematical
calculations.
 It is complex.
 The output of the multivariate regression model is difficult to analyse.
 The loss can use errors in the output.
 Multivariate regression yields better results when used with larger
datasets rather than small ones.
Explore all the survey question types

What’s the Difference Between OLTP & OLAP

What is OLAP?
Online Analytical Processing, a category of software tools which provide
analysis of data for business decisions. OLAP systems allow users to
analyze database information from multiple database systems at one
time.

The primary objective is data analysis and not data processing.

What is OLTP?
Online transaction processing shortly known as OLTP supports
transaction-oriented applications in a 3-tier architecture. OLTP
administers day to day transaction of an organization.

The primary objective is data processing and not data analysis

Example of OLAP
Any Datawarehouse system is an OLAP system. Uses of OLAP are as
follows

 A company might compare their mobile phone sales in September

with sales in October, then compare those results with another
location which may be stored in a sperate database.
 Amazon analyzes purchases by its customers to come up with a
personalized homepage with products which likely interest to their
customer.

Example of OLTP system

An example of OLTP system is ATM center. Assume that a couple has a
joint account with a bank. One day both simultaneously reach different
ATM centers at precisely the same time and want to withdraw total
amount present in their bank account.

However, the person that completes authentication process first will be

able to get money. In this case, OLTP system makes sure that
withdrawn amount will be never more than the amount present in the
bank. The key to note here is that OLTP systems are optimized
for transactional superiority instead data analysis.

Benefits of using OLAP services

 OLAP creates a single platform for all type of business analytical

needs which includes planning, budgeting, forecasting, and
analysis.
 The main benefit of OLAP is the consistency of information and
calculations.
 Easily apply security restrictions on users and objects to comply
with regulations and protect sensitive data.

Benefits of OLTP method

 It administers daily transactions of an organization.

 OLTP widens the customer base of an organization by simplifying
individual processes.
 Below is the difference between OLAP and OLTP in Data
Warehouse:

Difference Between OLTP and OLAP

OLTP and OLAP both are the online processing

systems. OLTP is a transactional processing while OLAP is an analytical
processing system. OLTP is a system that manages transaction-oriented
applications on the internet for example, ATM. OLAP is an online system
that reports to multidimensional analytical queries like financial reporting,
forecasting, etc.

The basic difference between OLTP and OLAP is that OLTP is an online
database modifying system, whereas, OLAP is an online database query
answering system.
Comparison Chart

BASIS FOR
OLTP OLAP
COMPARISON

Basic It is an online It is an online data

transactional system and retrieving and data analysis

manages database system.

modification.

Focus Insert, Update, Delete Extract data for analyzing

information from the that helps in decision

database. making.

Data OLTP and its Different OLTPs database

transactions are the becomes the source of data

original source of data. for OLAP.

Transaction OLTP has short OLAP has long

transactions. transactions.

Time The processing time of a The processing time of a

transaction is transaction is comparatively

 BASIS FOR
OLTP OLAP
COMPARISON

comparatively less in more in OLAP.

OLTP.

Queries Simpler queries. Complex queries.

Normalization Tables in OLTP Tables in OLAP database

database are normalized are not normalized.

(3NF).

Integrity OLTP database must OLAP database does not

maintain data integrity get frequently modified.

constraint. Hence, data integrity is not

affected.

What is the tradeoff between Bias and Variance?

divide reducible errors into two: Bias and Variance.
 Figure 1: Errors in Machine Learning

Bias

Figure 2: Bias

The difference between the average value predicted by our Machine

Learning model and the correct target value is known as Bias. A model
that makes incorrect predictions about a dataset is called a biased
model. This model oversimplifies the target function to make it easier to
learn.

Characteristics of a High Bias Model

1. Underfitting: A model with High Bias tends to underfit the data as

it oversimplifies the solution by failing to learn how to train the data
efficiently. This results in a linear function.
2. Oversimplification: Due to the model being too simple, the biased
model is unable to learn complex features of a training data, thus,
making it inefficient when solving complex problems.
 3. Low Training Accuracy: Due to the inability to correctly process
training data, the biased model shows high-training loss resulting
in low-training accuracy.

Solution to High Bias

High bias happens because of a high training error. There are multiple
ways to reduce the bias of a model, such as:

1. By adding more features from the data to make the model more
complex.
2. By increasing training iterations so that more complex models
and relevant data can be learned.
3. Replacing current model with more complex model can reduce
the bias.
4. Using non-linear algorithms
5. Using non-parameterized algorithms
6. By decreasing regularization on inputs at different levels, the
model can learn the training set more efficiently and prevent
underfitting.
7. By using a new model architecture. However, this should only be
used as a last resort if none of the methods above give satisfactory
results.

Variance

The amount of variability in the target function in response to a change in

the training data is known as Variance. When a model takes into
consideration the noise and fluctuation in the data, it is said to be of High
Variance.

High Variance / Overfitting

Characteristics of a High Variance Model

1. Overfitting: A model with High Variance tends to overfit the data

as it overcomplicates the solution and fails to generalize new test
data. This results in a non-linear function.
2. Overcomplication: Due to the model being too complex, the
model learns a much more complex curve and fails to work
efficiently on simple problems.
 3. Low Testing Accuracy: Although these models tend to work well
on training data with high accuracy, they fail to efficiently work on
test data where they will show a huge test data loss.

Solution to High Variance

High variance is due to a high validation error. There are multiple ways
of reducing the variance of a model such as:

1. By reducing features from the data to make the model less

complex
2. By increasing training data, complex models can be simplified in
order to balance out the model complexity and amount of datasets.
3. Replacing current model with a simpler model can reduce the
variance.
4. Using lower degree algorithms.
5. Using hyper parameter tuning to avoid overfitting
6. By increasing regularization on inputs at different levels, the
model complexity will be reduced. This will prevent overfitting.
7. Use a new model architecture. However, this should only be
used as a last resort if none of the methods above give satisfactory
results.

Bias-Variance tradeoff
The goal of any supervised machine learning algorithm is to achieve low
bias and low variance. In turn the algorithm should achieve good
prediction performance.

You can see a general trend in the examples above:

 Linear machine learning algorithms often have a high bias but a

low variance.
 Nonlinear machine learning algorithms often have a low bias but a
high variance.

As seen above, if the algorithm is too simple, it will have a high bias and
a low variance. Similarly, if the algorithm is too complex, it will have a
high variance and a low bias. Therefore, it is clear that:
“Bias and variance are complements of each other” The increase of one
will result in the decrease of the other and vice versa. Hence, finding the
right balance of values is known as the Bias-Variance Tradeoff.
Target Function

An ideal algorithm should neither underfit nor overfit the data. The end
goal of all Machine Learning Algorithms is to produce a function that has
both low-bias and low-variance.

Bias-Variance Graph

Hypothetically, the dotted line above is the required optimal solution. But,
in the real world, it is very difficult to achieve due to an unknown best
target function. The goal is to find an iterative process through which we
can keep on improving our Machine Learning Algorithm so that its
predictions will improve.

The parameterization of machine learning algorithms is often a battle to

balance out bias and variance.

Below are two examples of configuring the bias-variance trade-off for

specific algorithms:

 The k-nearest neighbors algorithm has low bias and high variance,
but the trade-off can be changed by increasing the value of k
which increases the number of neighbors that contribute t the
prediction and in turn increases the bias of the model.
 The support vector machine algorithm has low bias and high
variance, but the trade-off can be changed by increasing the C
parameter that influences the number of violations of the margin
allowed in the training data which increases the bias but decreases
the variance.
There is no escaping the relationship between bias and variance in
machine learning.

 Increasing the bias will decrease the variance.

 Increasing the variance will decrease the bias.
There is a trade-off at play between these two concerns and the
algorithms you choose and the way you choose to configure them are
finding different balances in this trade-off for your problem

In reality, we cannot calculate the real bias and variance error terms
because we do not know the actual underlying target function.
Nevertheless, as a framework, bias and variance provide the tools to
understand the behavior of machine learning algorithms in the pursuit of
predictive performance.

K-Fold Cross-Validation

K-fold cross-validation approach divides the input dataset into K groups

of samples of equal sizes. These samples are called folds. For each
learning set, the prediction function uses k-1 folds, and the rest of the
folds are used for the test set. This approach is a very popular CV
approach because it is easy to understand, and the output is less biased
than other methods.

The steps for k-fold cross-validation are:

o Split the input dataset into K groups

o For each group:
o Take one group as the reserve or test data set.
o Use remaining groups as the training dataset
o Fit the model on the training set and evaluate the
performance of the model using the test set.

Let's take an example of 5-folds cross-validation. So, the dataset is

grouped into 5 folds. On 1st iteration, the first fold is reserved for test the
model, and rest are used to train the model. On 2 nd iteration, the second
fold is used to test the model, and rest are used to train the model. This
process will continue until each fold is not used for the test fold.

Stratified k-fold cross-validation

This technique is similar to k-fold cross-validation with some little

changes. This approach works on stratification concept, it is a process of
rearranging the data to ensure that each fold or group is a good
representative of the complete dataset. To deal with the bias and
variance, it is one of the best approaches.

It can be understood with an example of housing prices, such that the

price of some houses can be much high than other houses. To tackle
such situations, a stratified k-fold cross-validation technique is useful.

Holdout Method

This method is the simplest cross-validation technique among all. In this

method, we need to remove a subset of the training data and use it to
get prediction results by training it on the rest part of the dataset.

The error that occurs in this process tells how well our model will perform
with the unknown dataset. Although this approach is simple to perform, it
still faces the issue of high variance, and it also produces misleading
results sometimes.

Comparison of Cross-validation to train/test split in Machine Learning

o Train/test split: The input data is divided into two parts, that are
training set and test set on a ratio of 70:30, 80:20, etc. It provides a
high variance, which is one of the biggest disadvantages.
o Training Data: The training data is used to train the model,
and the dependent variable is known.
o Test Data: The test data is used to make the predictions
from the model that is already trained on the training data.
This has the same features as training data but not the part
of that.
o Cross-Validation dataset: It is used to overcome the
disadvantage of train/test split by splitting the dataset into groups
of train/test splits, and averaging the result. It can be used if we
want to optimize our model that has been trained on the training
dataset for the best performance. It is more efficient as compared
to train/test split as every observation is used for the training and
testing both.

Limitations of Cross-Validation

There are some limitations of the cross-validation technique, which are

given below:
 o For the ideal conditions, it provides the optimum output. But for the
inconsistent data, it may produce a drastic result. So, it is one of
the big disadvantages of cross-validation, as there is no certainty
of the type of data in machine learning.
o In predictive modeling, the data evolves over a period, due to
which, it may face the differences between the training set and
validation sets. Such as if we create a model for the prediction of
stock market values, and the data is trained on the previous 5
years stock values, but the realistic future values for the next 5
years may drastically different, so it is difficult to expect the correct
output for such situations.

Applications of Cross-Validation

o This technique can be used to compare the performance of

different predictive modeling methods.
o It has great scope in the medical research field.
o It can also be used for the meta-analysis, as it is already being
used by the data scientists in the field of medical statistics.