Multiple Linear Regression

Multiple Linear Regression (MLR)

• We extend the Simple Linear Regression (SLR) model to accommodate multiple
predictors.
• The resulting model is known as Multiple Linear Regression (MLR), which is
given by
𝑌 = 𝛽0 + 𝛽1 𝑋1 + 𝛽2 𝑋2 + ⋯ + 𝛽𝑝 𝑋𝑝 + 𝜖,
where 𝑋1 , 𝑋2 , … , 𝑋𝑝 are 𝑝 predictors.
• We interpret 𝛽𝑗 as the average effect on 𝑌 of a one unit increase in 𝑋𝑗 , holding
all other predictors fixed.
Multiple Linear Regression (MLR)
• In the advertising example, the MLR model becomes

𝑠𝑎𝑙𝑒𝑠 = 𝛽0 + 𝛽1 × 𝑇𝑉 + 𝛽2 × 𝑟𝑎𝑑𝑖𝑜 + 𝛽3 × 𝑛𝑒𝑤𝑠𝑝𝑎𝑝𝑒𝑟 + 𝜖.

Regression Summary Table
Coefficients Std. error t-statistic p-value
Intercept 2.939 0.312 9.422 <0.0001
TV 0.046 0.001 32.809 <0.0001
Radio 0.189 0.009 21.893 <0.0001
Newspaper -0.001 0.006 −0.177 0.860
Interpretation
• For
a given amount of TV and newspaper advertising, spending an
additional $1,000 on radio advertising leads to an increase in sales
by approximately 189 units.
• The MLR coefficient estimates for TV and radio are pretty similar to
the SLR coefficient estimates.
Interpretation
• However, while the newspaper regression coefficient estimate in the SLR model
was significantly non-zero, the coefficient estimate for newspaper in the MLR
model is close to zero, with corresponding p-value around 0.86.
Interpretation
• In SLR, the slope term represents the average effect of a $1,000 increase in
newspaper advertising, ignoring other predictors such as TV and radio.
• In contrast, in the multiple regression setting, the coefficient for newspaper
represents the average effect of increasing newspaper spending by $1,000 while
holding TV and radio fixed.
Correlation Matrix
TV Radio Newspaper Sales
TV 1.000 0.055 0.057 0.782
Radio 1.000 0.354 0.576
Newspaper 1.000 0.228
Sales 1.000
Interpretation
• The correlation between radio and newspaper is 0.35.
• This suggests that there is a tendency to spend more on newspaper advertising
in markets where more is spent on radio advertising.
• Suppose that the MLR model is correct and newspaper advertising has no direct
impact on sales, but radio advertising does increase sales.
• Then in markets where we spend more on radio our sales will tend to be higher,
and as our correlation matrix shows, we also spend more on newspaper
advertising in those same markets.
Interpretation
• In a SLR model that only checks sales versus newspaper, higher values of
newspaper tend to be linked with higher values of sales, even though
newspaper advertising does not affect sales.
• So newspaper gets “credit” for the effect of radio on sales.
Some Important Questions
1. Is at least one of the predictors 𝑋1 , … , 𝑋𝑝 useful in predicting the response?
2. How well does the model fit the data?
3. Given a set of predictor values, what response value should we predict, and
how accurate is our prediction?
Is at least one of the predictors useful in
predicting the response?
• We test the null hypothesis
𝐻0 : 𝛽1 = 𝛽2 = ⋯ = 𝛽𝑝 = 0
vs the alternative
𝐻𝑎 : 𝑎𝑡 𝑙𝑒𝑎𝑠𝑡 𝑜𝑛𝑒 𝛽𝑗 𝑖𝑠 𝑛𝑜𝑛 − 𝑧𝑒𝑟𝑜.
• The observed 𝑝-value is 𝑃 𝐹𝑝. 𝑛−𝑝−1 > 𝐹𝑜𝑏𝑠 . Reject 𝐻0 if 𝑝-value is small.
• For the Advertising data set, the observed 𝑝-value is very low. Thus we reject 𝐻0 .
Model Fit
• The quality of a linear regression fit is assessed using following quantities:
➢𝑅2 statistic
➢Adjusted 𝑅2 statistic
➢Residual Standard Error (𝑅𝑆𝐸)
𝑅 2 and Adjusted-𝑅 2 Statistic
• 𝑅2 close to 1 indicates that the model explains the large portion of the variance
in the response variable.
• However, 𝑅2 always increases with the addition of every new variable.
• This is remedied using
𝑅𝑆𝑆ൗ
2 𝑛−𝑝−1
Adjusted − 𝑅 = 1 − ,
𝑇𝑆𝑆ൗ
𝑛−1
where 𝑝 is the number of predictors.
• A model with more variables can have lower Adjusted-𝑅2 .
 Residual Standard Error (𝑅𝑆𝐸)
• The RSE is defined as
1
𝑅𝑆𝐸 = 𝑅𝑆𝑆.
𝑛−𝑝−1

• A model with more variables can have higher 𝑅𝑆𝐸 if the decrease in 𝑅𝑆𝑆 is small relative to the
increase in the number of variables (𝑝).
Model Fit for Advertising Data Set
Model Predictors 𝑅2 Adjusted 𝑅𝑆𝐸
− 𝑅2
1 TV 0.61 0.61 3.26
2 Radio 0.33 0.33 4.28
3 Newspaper 0.05 0.05 5.09
4 TV & Radio 0.90 0.90 1.68
5 TV & 0.65 0.64 3.12
Newspaper
6 Radio & 0.33 0.33 4.28
Newspaper
7 TV, Radio & 0.90 0.90 1.69
Newspaer
Two Types of Predictions
• We will develop two types of interval estimates regarding 𝑦:
1. A confidence interval for the expected value of 𝑦.
2. A prediction interval for an individual value of 𝑦.
• It is common to refer to the first as a confidence interval and the
second as a prediction interval.
Confidence Interval vs Prediction Interval
• In order to assess the uncertainty associated with the predicted response,
consider the following two cases:
➢ How should we quantify the uncertainty associated with the average sales
over a number of markets, given that $100,000 is spent on TV advertising,
$20,000 is spent on Radio advertising, and $10,000 is spent on Newspaper
advertising in each market?
➢How should we quantify the uncertainty associated with the sales of a
particular market, given that $100,000 is spent on TV advertising, $20,000 is
spent on Radio advertising, and $10,000 is spent on Newspaper advertising in
that market?
The Confidence Interval
• The point estimate of 𝐸(𝑦 0 ) is just the 𝑦ො 0 value:
𝑦ො 0 = 𝑏0 + 𝑏1 𝑥10 + 𝑏2 𝑥20 + ⋯ + 𝑏𝑝 𝑥𝑝0 .
• For specific values of 𝑥1 , … , 𝑥𝑘 , denoted by 𝑥10 , 𝑥20 , … , 𝑥𝑝0 , the 100 1 − 𝛼 %
confidence interval for the expected value of 𝑦 is computed as
𝑦ො 0 ± 𝑡𝛼Τ2,𝑑𝑓 se 𝑦ො 0 ,
where 𝑑𝑓 = 𝑛 − 𝑝 − 1.
Computation of Confidence Interval in Excel
• Many statistical programs provide confidence intervals directly. Excel
does not provide it directly.
• However one can still compute it using a simple trick.
• To derive together with se 𝑦ො 0 , we first estimate a modified regression
model where 𝑦 is the response variable and the explanatory variables are
defined as 𝑥1∗ = 𝑥1 − 𝑥10 , 𝑥2∗ = 𝑥2 − 𝑥20 , … , 𝑥𝑘∗ = 𝑥𝑘 − 𝑥𝑘0 .
• The resulting estimate of the intercept and its standard error equal 𝑦ො 0
and se 𝑦ො 0 , respectively.`
Advertising Data Set
For the Advertising Data, we first shift the values of the predictors by our
hypothesized value:
Obs. Sales TV Radio Newspaper 𝐓𝐕 − 𝟏𝟎𝟎 𝑹𝒂𝒅𝒊𝒐 − 𝑵𝒆𝒘𝒔𝒑𝒂𝒑𝒆𝒓
20 − 𝟏𝟎
1 22.1 230.1 37.8 69.2 130.1 17.8 59.2
2 10.4 44.5 39.3 45.1 −55.5 19.3 35.1
3 9.3 17.2 45.9 69.3 −82.8 25.9 59.3
 Advertising Data Set
• Estimating the modified regression now reveals the confidence interval:
Coefficients Std. t-statistic p-value Lower Upper
error 95% 95%
Intercept 11.276 0.175 64.348 0.000 10.930 11.621

TV* 0.046 0.001 32.809 0.000 0.043 0.049

Radio* 0.189 0.009 21.893 0.000 0.172 0.206

Newspaper* −0.001 0.006 −0.177 0.860 −0.013 0.011

• Here TV*=TV−100, Radio*=Radio−20, Newspaper*=Newspaper−10

• So 95% confidence interval for expected sales over a number of markets, given that TV, Radio, and
Newspaper advertising budgets are $100,000, $20,000, and $10,000, respectively, is 10.93,11.62 in
thousands of units.
Prediction Interval
• For specific values of 𝑥1 , … , 𝑥𝑘 , denoted by 𝑥10 , 𝑥20 , … , 𝑥𝑘0 , the 100 1 − 𝛼 %
prediction interval for an individual value of 𝑦 is computed as
2
0
𝑦ො ± 𝑡𝛼Τ2,𝑑𝑓 se 𝑦ො 0 + 𝑠𝑒2 ,
where 𝑑𝑓 = 𝑛 − 𝑘 − 1, se(𝑦ො 0 ) is the standard error of 𝑦ො 0 , and 𝑠𝑒 is the
standard error of the estimate.
Advertising Data Set
• The 95% prediction interval for Sales in a particular market where TV, Radio,
and Newspaper advertising budgets are $100,000, $20,000, and $10,000,
respectively, is given by
2
𝑦ො 0 ± 𝑡0.025,196 se 𝑦ො 0 + 𝑠𝑒2

= 11.276 ± 1.972 0.175 2 + 1.686 2

= 11.276 ± 3.343 = 7.934, 14.618 in thousands of units.

Marketing Plan
• Is there a relationship between sales and advertising budget?
Marketing Plan
• Is there a relationship between sales and advertising budget?
➢We fitted a regression model with sales as response variable and TV, radio,
and newspaper as predictors.
➢We test the hypothesis 𝐻0 : 𝛽𝑇𝑉 = 𝛽𝑅𝑎𝑑𝑖𝑜 = 𝛽𝑁𝑒𝑤𝑠𝑝𝑎𝑝𝑒𝑟 = 0.
➢We use the 𝐹-statistic to determine whether or not we should reject 𝐻0 .
➢The observed 𝑝-value is very low, indicating that there is a relationship
between sales and advertising budget.
Marketing Plan
• How strong is the relationship?
Marketing Plan
• How strong is the relationship?
➢The observed value for 𝑅𝑆𝐸 is 1686 units, while the mean value for the
response is 14,023.
➢This indicates a percentage error of roughly 12%.
➢The observed value for 𝑅2 statistic is around 90%.
➢ This indicates that the predictors explain almost 90% of the variability
in 𝑠𝑎𝑙𝑒𝑠.
➢This indicates that there is a strong relationship between sales and
advertising budget.
Marketing Plan
• Which media contribute to sales?
Marketing Plan
• Which media contribute to sales?
➢We observe that the 𝑝-values for 𝑇𝑉 and radio are very low, but the 𝑝-
value for newspaper is not.
➢This suggests that only TV and radio are related to sales.
Marketing Plan
• How accurately can we predict future sales?
Marketing Plan
• How accurately can we predict future sales?
➢ The response can be predicted using the formula
෣ = 2.939 + 0.046TV + 0.189Radio − 0.001Newspaper.
sales
➢The accuracy associated with this estimate depends on whether we are
interested in individual sales for a particular city or the average sales of a
number of cities.
➢In the former case, we use a prediction interval, and in the latter case, we use
a confidence interval.
➢Prediction intervals are always wider than the confidence interval.
Model Assumptions
• Conditional on 𝑥1 , … , 𝑥𝑘 , the variance of the error term 𝜀 is the
same for all observations; or in statistical terminology, there is no
heteroskedasticity.

• There is no exact linear relationship among the explanatory

variables; or in statistical terminology, there is no perfect
multicollinearity.
Model Assumptions
• Conditional on 𝑥1 , … , 𝑥𝑘 , the error term 𝜀 is uncorrelated across observations;
or in statistical terminology, there is no serial correlation. The assumption is
violated if observations are correlated.

• Conditional on 𝑥1 , … , 𝑥𝑘  , the error term 𝜀 is normally distributed. This

assumption allows us to construct interval estimates and conduct tests of
significance. If 𝜀 is not normally distributed, the interval estimates and the
hypothesis tests are valid only for large sample sizes.