Diagnosing bias vs variance

We will continue to look at some methodologies to diagnose our hypothesis function. This time,
it will be about bias and variance.

Bias and Variance

So what does bias and variance represent?
Bias represents the distance of our predicted values with the original values.
Variance represents the distance of the predicted values, which means how much the predicted
values are scattered from each other.
Let’s see this intuitively through a picture with my explanations.

Errors due to the bias means that the model is too simple to represent a certain situation, it
cannot catch the details. If the bias is high, it will cause underfitting. This means that our model
is missing something important.
Errors due to the variance means that the model is not generalized enough to apply to other
situations or datasets other than our training set. If the model has high variance, this means that
the model is overfitted.
High bias & low variance, Good compromise, Low bias & high variance

Bias & Variance trade-off

Bias and variances have a trade-off relationship.

 If our model gets simple, the bias would get higher, and the variance would get
lower.

If our model gets complicated, the bias would get lower, and the variance would
get higher.

So we must find the appropriate point where the bias and the variance values are
well compromised with each other. We cannot just lower both at once.

In

machine learning, the bias–variance tradeoff is the property of a set of predictive

models whereby models with a lower bias have a higher variance of the parameter
across dataset, and vice versa.

To understand this clearly, Let’s see what bias and variance means.
What is Bias?

● Bias is an error introduced in the model due to oversimplification of the

machine learning algorithm.
● It can lead to under-fitting. When you train your model at that time model
makes simplified assumptions to make the target function easier to
understand.
● Bias is basically the difference between your model’s expected
predictions and the true values.
Low bias machine learning algorithms — Decision Trees, k-NN and SVM

High bias machine learning algorithms — Linear Regression, Logistic Regression

1. Low Bias: Suggests less assumptions about the form of the target function.
2. High-Bias: Suggests more assumptions about the form of the target
function.

Refer this article to understand what over-fitting and under-fitting models mean.

What is Variance?

● Variance is error introduced in your model due to complex machine learning

algorithm, your model learns noise also from the training dataset and
performs badly on the test dataset.
 ● It can lead to high sensitivity and over-fitting.

low-variance machine learning algorithms include: Linear Regression, Linear Discriminant

Analysis and Logistic Regression.

high-variance machine learning algorithms include: Decision Trees, k-Nearest Neighbors

and Support Vector Machines.

1. Low Variance: Suggests small changes to the estimate of the target function
with changes to the training dataset.
2. High Variance: Suggests large changes to the estimate of the target
function with changes to the training dataset.

Normally, as you increase the complexity of your model, you will see a reduction in error
due to lower bias in the model. However, this only happens until a particular point. As
you continue to make your model more complex, you end up over-fitting your model and
hence your model will start suffering from high variance.

Bias, Variance trade-off:

The goal of any supervised machine learning algorithm is to have low bias and low
variance to achieve good prediction performance.However,you cannot reduce both at
once.

● Increasing the bias will decrease the variance.

● Increasing the variance will decrease the bias.

But it can be tackled by few algorithms-

1. The k-nearest neighbor’s algorithm has low bias and high variance, but this
can be changed by increasing the value of k which increases the number of
 neighbors that contribute to the prediction and in turn increases the bias of
the model.
2. The support vector machine algorithm has low bias and high variance, but
this can be changed by increasing the C parameter that influences the
number of violations of the margin allowed in the training data which
increases the bias but decreases the variance.

So,What is Over-fitting?

● Over-fitting in a statistical model is that which describes random

error or noise instead of the underlying relationship.
● Over-fitting occurs when a model is very complex, such as having too
many parameters relative to the number of observations.
● A model that has been over-fitted, has poor predictive performance,
as it overreacts to minor fluctuations in the training data.
● Over-fitting is basically a modeling error which occurs when a function
is too closely fit to a limited set of data points.

And,What is Under-fitting?

● Under-fitting occurs when a statistical model or machine learning

algorithm cannot capture the underlying trend of the data.
● Under-fitting would occur, for example, when fitting a linear model to
non-linear data. Such a model too would have poor predictive
performance.
● Intuitively, under-fitting occurs when the model or the algorithm does
not fit the data well enough.
● Under-fitting occurs if the model or algorithm shows low variance but
high bias.
How to combat Over-fitting and Under-fitting?

To combat over-fitting:

● Resampling the data to estimate the model accuracy (k-fold

cross-validation) and by having a validation dataset to evaluate the
model.
● Early Stopping is a form of regularization used to avoid overfitting
when training a model with an iterative method, such as gradient
descent.However,a small drawback with the early stopping is that it
simultaneously try’s not to over-fit the model as well as optimize the
cost function,which leads to not so optimized cost function as it
stopped early.(To avoid this L2 regularization is used).
● Pruning is extensively used while building decision tree models. It
simply removes the nodes which add little predictive power for the
problem in hand.However,it is not needed in RandomForest algorithm
as in the algorithm random trees uses random features and so the
individual trees are strong but not so correlated with each other.
● Regularization, It introduces a cost term for bringing in more features
with the objective function. Hence it tries to push the coefficients for
many variables to zero and hence reduce cost term.
To combat under-fitting:

● Under-fitting can be avoided by using more data and also reducing

the features by feature selection.
● Increase the size or number of parameters in the ML model.
● Increase the complexity or type of the model.
● Increasing the training time until cost function of the Model is
minimized.

Good Fit in a Statistical Model:

● Ideally, the case when the model makes the predictions with 0 error, is
said to have a good fit on the data.
● This situation is achievable at a spot between overfitting and
underfitting. In order to understand it we will have to look at the
performance of our model with the passage of time, while it is learning
from training dataset.
With the passage of time, our model will keep on learning and thus the error for the
model on the training and testing data will keep on decreasing. If it will learn for too
long, the model will become more prone to overfitting due to presence of noise and
less useful details. Hence the performance of our model will decrease.

● In order to get a good fit, we will stop at a point just before where the
error starts increasing. At this point the model is said to have good
skills on training dataset as well our unseen testing dataset.As shown
in the picture below.

Here's a breakdown of the formula and its components:

 ● E(f(x)): This represents the expected value of the estimator's prediction for a
given input x. In simpler terms, it's the average prediction the estimator would
make for x across many different training datasets.
● f(x): This refers to the actual prediction made by the estimator for input x on a
specific training dataset.
● (f(x) - E(f(x))): This term captures the variance of the estimator's predictions. It
measures how much the predictions vary around the expected value, considering
different training datasets.
● 0^2: This represents the irreducible error. This is the error that remains even
with a perfect estimator, due to inherent noise or randomness in the data.

The entire formula basically calculates the average squared difference between the
estimator's predictions and the true values it's trying to predict. This difference is
squared to emphasize larger errors more than smaller ones.

Here's how the formula connects to the bias-variance tradeoff:

● Bias: A high bias estimator makes consistently wrong predictions, even on

average (E(f(x)) is far from the true value). This contributes to the MSE because
the squared difference in the formula becomes larger.
● Variance: An estimator with high variance has predictions that vary greatly
across different training datasets. This also increases the MSE because the
squared differences in the formula become larger and more erratic.

The goal is to minimize the MSE, which means finding an estimator that balances bias
and variance. A good estimator makes predictions that are close to the true values on
average (low bias), and also makes predictions that are consistent across different
training datasets (low variance).