Scribd
Upload
11 views14 pages

P&AD Lect 17 1 Unit2

The document discusses bias-variance tradeoffs in linear regression models. It explains how regularization can control overfitting by balancing bias and variance, and gives an example using 25 datasets from a sinusoidal function with varying regularization. Bayesian linear regression is also covered, where a prior over the weights is combined with the likelihood to obtain the posterior distribution over weights.

Uploaded by

RAHUL DHANOLA
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
11 views14 pages

P&AD Lect 17 1 Unit2

The document discusses bias-variance tradeoffs in linear regression models. It explains how regularization can control overfitting by balancing bias and variance, and gives an example using 25 datasets from a sinusoidal function with varying regularization. Bayesian linear regression is also covered, where a prior over the weights is combined with the likelihood to obtain the posterior distribution over weights.

Uploaded by

RAHUL DHANOLA
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 14

Pattern and Anomaly Detection

B. Tech., CSE + AI/ML


Dr Gopal Singh Phartiyal
16/09/2021

Source: Edureka
Bias- Variance
• So far in linear models for regression: We have assumed that the form
and number of basis functions are both fixed

Limiting the number of basis functions in order to avoid over-fitting


Vs
limiting the flexibility of the model to capture interesting and
important trends in the data
• The introduction of regularization terms can control over-fitting for
models with many parameters
• New question: Suitable values of these parameters
Bias- Variance
• Recall the expected squared loss,(not sum of squared error)

• where

• The second term of E[L] corresponds to the noise inherent in the


random variable t.
• What about the first term?
Bias- Variance

• Thus we can write

• where
Bias- Variance : Example
• Example: 25 data sets from the sinusoidal, varying the degree of
regularization,
Bias- Variance : Example
• Example: 25 data sets from the sinusoidal, varying the degree of
regularization,
Bias- Variance: Example
• Example: 25 data sets from the sinusoidal, varying the degree of
regularization,
Bias- Variance: Trade-off
• Example: 25 data sets from the sinusoidal, varying the degree of
regularization,
•From these plots, we note that an over-
regularized model (large ) will have a high
bias, while an under-regularized model (small
¸) will have a high variance.
So far in Linear Regression Models
Goal: Find w?
• Why linear model?
• Simple linear regression
• Basis functions
• Solving for w using maximum likelihood and least squares
• For multiple output
• Regularize the model (different regularizers)
• Managing over-fitting via the concept of bias-variance decomposition

So which model to choose?


Remember model selection and cross-validation
Next: Bayesian Linear Regression
• We know - Posterior = likelihood x prior
• Likelihood
• Define a conjugate prior over w

•Combining and using results for marginal and conditional Gaussian


distributions, gives the posterior (Refer Chapter-2, Bishop)

• where
Bayesian Linear Regression
• A common choice for the prior is (zero-mean isotropic Gaussian)

for which

•The w may vary from wML to wprior

•Maximization of this posterior distribution with respect to w is therefore


equivalent to the minimization of the sum-of-squares error
Example Likelihood Posterior Data Space

X = U(x| (-1,1), 20 observations


W0 = 0.3, w1 = 0.5
Noise = Gaussian (sigma = 0.2)
Alpha = 2 for prior
Prior Data Space

0 data points observed


Thank You

You might also like

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy