Bias and Variance

Bias enters as is not a good approximation for

e.g. trying to learn a linear model to predict a
Where to put blame transformer’s output
• Variance enters as we were not able to learn
We have regression the bestdata where
parameters for , due
sayto poor training
etc.
We have learnt a model
• Bayes with params
error is unavoidable to approximate
– perhaps labels are
depends on That(train set,
is neat
inherently– inotherorrandom
practical
noisy elseterms, choices suchthat
this is saying
the features are as
initialization etc)
errors are either
insufficient duethe
to solve to calling
task the incorrect
Let – will write just to simplify
constructor in sklearnnotation
(or passing incorrect
arguments to the constructor) or issues in how the
Its average testfit() error is not
function alljob,
did its that nice issue
or some – whatin is to
blame? features/labels themselves!
The expectation over means
Variance aka Bias akathe bad performance was
that Bayes
As and
Estimation Approximation
not due to unluckyError
choices e.g.
Error Error train-test split etc.
Bias vs Variance
Similar to the distinction between accuracy and
precision
Variance comes in due to dependence of on random
choices
Training set, choice of mini-batches in (MB)SGD, random
initialization, etc

Low Bias Low Bias High Bias High Bias

Low Variance High Variance Low Variance High Variance
Bias Variance Tradeoff
Easiest way to reduce bias
Test is to make model more
powerful
Error
Non-linear models, universal
models, NN with lots of
layers
Bias
Powerful models have their
Varian own problems 
ce Parameter estimation is
often (NP)hard
Low Medium High Brittle – changing training
Model Complexity data slightly changes
parameters a lot
More prone to overfitting too
Overcoming Bias, Variance,
ChallengingBayes
detect Bayes
to

Error error as it may

hide inside bias

Test error can be blamed on

Bias: lowest error allowed by model (cant get better without changing
model)
Variance: how well are we able to achieve the lowest error our model
allows
Bayes Error: something wrong with how problem itself is setup
Low bias, low variance models usually exist only for specific
domains
To lower bias, make model more powerful (variance may go up)
To lower variance, use more data or better learning algorithm
To lower Bayes error, add more (informative) features
Can also lower bias but can also increase variance
 The presence of an (over) abundance

BV Tradeoff in the Age of LLMsof parameters and data seems to

makes it possible to discover good
local minima which not only do well
on training data but on test data as
Test well
Error Still an active area of
research

Bias
Varian
ce

Low Medium High Overparametrized regime

Model Complexity
 Generalization error (just like
Generalization Error variance) can usually be brought
down by using more data points or
choosing Gap
modelsbetween train and
that are simpler test
error
Test Error Difference between variance and gen error is
How well is model able to
subtle. Model complexity and train set
“generalize” to size
unseen data
affect both. However, variance can alsooften
Underfitting: be high
due to high
because of using an bias,
improper
poorlearning
training
algorithm or not optimizing properly e.g.
Overfitting: oftensmall
due to high
Underfitting Overfitting
mini-batches
variance, over training
Gen Gen error small if small bias
and small variance
Training Error
Error Note: large bias may have
good gen error yet high
Low Mediu High
Model Complexity
m
test error
Test error may be close to
train error even if both are
 Adding more data cannot
Detecting Over/underfitting decrease bias. The chosen
model just sucks  Adding more
data can decrease variance
Low training error but high test error??
thoughis memorizing
You may have overfit (high variance) – your model
training data Sometimes may need to iterate
Your model is clearly powerfulthrough
enoughthe
– does notexperiences
above seem to be a bias
problem (experience high bias, reduce it only
Use more data/better optimizer/simpler model (or
to increase variance, all) decrease
then to decrease
variance
variance etc) before reaching a sweet
High training error and high test error??spot
You may have underfit (high bias) – model incapable of handling learning
task
Increase model class complexity, add better features, to decrease bias
Use more data, better ML algo to address any underlying variance issues
Low training error and low test error
Show-off !!! 
High training error and low test error