Notes on Multiple Linear Regression

Note on adjusted R-square instead of multiple R-square.

The R-squared value is a good measure of the model's predictive power, but it
might not be sufficient for generalization. It's always essential to assess the model's
performance on unseen data (e.g., test data) to ensure its generalizability to new
observations.

The adjusted R-squared value is used when comparing models with different numbers
of predictors. It adjusts the R-squared value based on the number of predictors in the
model. In case, the adjusted R-squared value is slightly lower than the R-squared
value, suggesting that the model's predictive power might be slightly overestimated
when considering the number of predictors.

R-squared value, suggesting that the model has good predictive power.
However, the Adjusted R-squared value is also important, especially when dealing
with multiple predictor variables. It takes into account the number of predictor
variables and adjusts the R-squared value accordingly. If the Adjusted R-squared
value is lower than the R-squared value, this indicates that some of the predictor
variables might not be adding significant explanatory power to the model.
When deciding whether to use the R-squared or the Adjusted R-squared value, it's
generally a good practice to consider the Adjusted R-squared when dealing with
multiple predictor variables, as it penalizes the model for including irrelevant
variables.

More on adjusted R-square

Adjusted R-squared is a modification of the regular R-squared (coefficient of determination)
in the context of linear regression. While R-squared measures the proportion of the variance
in the dependent variable that is explained by the independent variables in the model,
adjusted R-squared considers the number of predictors in the model and adjusts the R-
squared value accordingly. This adjustment is important because as you add more predictors
to a model, the R-squared value tends to increase even if the added predictors do not
contribute significantly to explaining the variation in the dependent variable. Adjusted R-
squared attempts to address this issue by penalizing models with more predictors.
The formula for adjusted R-squared is:
Adjusted R2=1− ( (1−R2 )×(n−1)
n−k−1 )
Where:
 R2 is the regular R-squared value.
 n is the number of observations.
 k is the number of predictors in the model.
Interpreting Adjusted R-squared:
1. Range of Values: Adjusted R-squared always falls between 0 and 1. It can be equal
to or lower than the regular R-squared, and it's often used to compare different models
to see which provides a better balance between model complexity and explanatory
power.
 2. Improvement over Random Model: A higher adjusted R-squared indicates that a
larger proportion of the total variance in the dependent variable is explained by the
model's predictors compared to a random (intercept-only) model.
3. Model Fit: Adjusted R-squared is a measure of how well the model fits the data. It
considers both the goodness of fit and the number of predictors used. As the number
of predictors increases, adjusted R-squared will only increase if the new predictors
improve the model's explanatory power significantly.
4. Penalizing Complexity: Adjusted R-squared penalizes the inclusion of unnecessary
predictors that do not contribute much to explaining the dependent variable. It helps
guard against overfitting, where a model captures noise in the data rather than true
relationships.
5. Model Comparison: When comparing different models with differing numbers of
predictors, adjusted R-squared is often preferred over regular R-squared. It provides a
more accurate assessment of the model's ability to generalize to new data by
considering the trade-off between model complexity and goodness of fit.
6. Limitations: While adjusted R-squared provides valuable insights, it doesn't tell you
whether the chosen predictors are causally related to the dependent variable. It's also
important to use other diagnostic tools and domain knowledge to ensure the model's
validity.
In summary, adjusted R-squared is an important tool in model evaluation that helps strike a
balance between model complexity and the explanatory power of the predictors. It aids in
selecting models that are both parsimonious (not overly complex) and capable of capturing
meaningful relationships in the data.

Note on Degrees of Freedom

Degrees of freedom (df) play a role in determining the significance of the F-statistic.
In the ANOVA table, the degrees of freedom for the regression model is equal to the
number of regressors k (independent variables), and for the residual (error) it is N-k-1
These values are used to calculate the F-statistic and its associated p-value.

Standard Error
The standard error is a measure of the variability of the residuals around the regression line.
Standard error represents the average deviation of the observed values from the regression
line. The lower the standard error the better the regression line fit. In other words, a lower
standard error indicates that data points are closer to the regression line suggesting a better fit
of the line.

Assumptions while conducting Multiple Linear Regression

When conducting multiple linear regression, several assumptions need to be satisfied to
ensure the validity and reliability of the results. These assumptions are essential to interpret
the regression coefficients precisely and make important inferences from the model. The
main assumptions for multiple linear regression include:
1. Linearity: The relationship between the independent variables (predictors) and the
dependent variable (response) is assumed to be linear. This means that the change in
the response variable for a unit change in an independent variable is constant
regardless of the levels of other variables.
 2. Independence: The residuals (the differences between observed and predicted values)
should be independent of each other. This assumption is often violated when dealing
with time series data or spatial data, as there can be progressive or spatial
autocorrelation.
3. Homoscedasticity: The variance of the residuals should be constant across all levels
of the independent variables. In other words, the spread of the residuals should be
approximately the same throughout the range of the predictors. If the residuals exhibit
a funnel-like pattern (heteroscedasticity), it can affect the accuracy of coefficient
estimates and hypothesis tests.
4. Normality of Residuals: The residuals should follow a normal distribution. This
assumption is important for valid hypothesis testing and confidence interval
construction. Deviation from normality might not be a big concern for large sample
sizes due to the central limit theorem, but severe deviations can still impact the
results.
5. No or Little Multicollinearity: The independent variables should be modestly
correlated with each other. High multicollinearity can make it difficult to determine
the individual effect of each predictor on the response variable and can lead to
unstable coefficient estimates.
6. No Perfect Multicollinearity: Perfect multicollinearity, where one independent
variable is a linear combination of others, must be avoided as it makes it impossible to
estimate individual coefficients accurately.
7. No Outliers or Influential Observations: Outliers or influential data points can
distort the regression line and affect the coefficient estimates and standard errors. It's
important to identify and handle outliers appropriately.

Violations of these assumptions can lead to biased, inefficient, or misleading results. It's
important to assess these assumptions before interpreting the results of a multiple linear
regression analysis. Various diagnostic tests and graphical techniques are available to help
check the assumptions and address any issues if they arise.

Normality Assumption if violated

The error term does not seem to be normally distributed for all independent
variables: If the residuals do not follow a normal distribution, it may indicate that the
model assumptions are violated, and the results might be unreliable. This issue could
potentially affect the model's predictive performance.
It is essential to address the assumption violations before drawing final conclusions
and making decisions based on the model. Further investigation and data support is
required to validate the assumptions and ensure the model's reliability.

Violation of Multicollinearity
Homoscedasticity assumption if violated.
If the assumption of homoscedasticity is violated in a multiple linear regression analysis, it
can have several important impacts on the validity and reliability of your regression results:
1. Incorrect Standard Errors and Confidence Intervals: Homoscedasticity is a key
assumption for estimating the standard errors of the regression coefficients. When
heteroscedasticity is present, the standard errors will be biased, which can lead to
incorrect p-values and confidence intervals. This, in turn, affects the validity of
hypothesis tests and the accuracy of inferences about the significance of predictor
variables.
2. Biased Coefficient Estimates: Heteroscedasticity can lead to biased coefficient
estimates. In the presence of heteroscedasticity, the model may give too much weight
to observations with higher variability and too little weight to observations with lower
variability. This can distort the relationships between the independent variables and
the dependent variable.
3. Inefficient Estimates: Heteroscedasticity can lead to inefficiency in parameter
estimation. Inefficient estimates can have wider confidence intervals, reducing the
precision of your results.
4. Incorrect Model Fit and Prediction: The presence of heteroscedasticity can indicate
that the model does not adequately capture the underlying data-generating process. As
a result, the model might not provide accurate predictions for cases with different
levels of the predictor variables.
5. Inaccurate Hypothesis Testing: Violation of homoscedasticity assumptions can lead
to incorrect hypothesis testing outcomes. Variables that are important may be deemed
insignificant, or vice versa.
To address the issue of heteroscedasticity, the following approaches may be adopted:
1. Transforming Variables: Sometimes transforming the dependent variable or
predictor variables can help stabilize the variance and mitigate heteroscedasticity.
Common transformations include taking the logarithm, square root, or inverse of the
variables.
2. Weighted Least Squares (WLS): WLS is a regression technique that assigns
different weights to observations based on their estimated variances. This can help
down weight observations with higher variability, effectively mitigating the impact of
heteroscedasticity.
3. Robust Standard Errors: When dealing with large samples, robust standard errors
can be used to provide valid p-values and confidence intervals even in the presence of
heteroscedasticity. Robust standard errors adjust for heteroscedasticity and other
potential issues.
4. Data Trimming or Winsor zing: Removing or capping extreme values in the dataset
can sometimes help mitigate heteroscedasticity.
5. Model Specification: Reconsidering the model specification, including adding or
removing variables, can also be helpful in addressing heteroscedasticity.
It's important to diagnose and address heteroscedasticity to ensure the reliability of your
regression results and the validity of the conclusions you draw from your analysis.
Multicollinearity assumption if violated.
When multicollinearity is violated in a multiple linear regression analysis, it can have several
significant impacts on the interpretation and reliability of your regression results:
1. Unreliable Coefficient Estimates: Multicollinearity makes it difficult to separate the
individual effects of correlated predictor variables on the response variable. The
estimated coefficients can become unstable and have large standard errors, making it
difficult to determine the true relationship between each predictor and the response.
2. Inflated Standard Errors: High multicollinearity leads to inflated standard errors for
the coefficient estimates. Larger standard errors mean that the estimates are less
precise, which can result in wider confidence intervals and reduced ability to detect
statistically significant effects.
3. Uninterpretable Coefficients: Multicollinearity can lead to counterintuitive or
absurd coefficient estimates. For example, a positive correlation between two
predictors might lead to a negative coefficient estimate for one of them due to the
shared influence on the response variable.
4. Difficulty in Identifying Important Predictors: Multicollinearity can mask the true
importance of individual predictors. Even if a predictor has a strong overall effect on
the response, its coefficient might appear insignificant or have the wrong sign due to
multicollinearity.
5. Reduced Model Generalizability: A model affected by multicollinearity might
perform well on the training data but struggle to generalize to new, unseen data. The
model might become overly sensitive to small changes in the training data, leading to
poor out-of-sample performance.
6. High Sensitivity to Small Changes: Multicollinearity can cause the regression
coefficients to change severely with small changes in the data or model specification.
This makes the results unreliable and difficult to replicate.
7. Inaccurate Hypothesis Testing: Hypothesis tests for individual coefficients might
yield incorrect results due to multicollinearity. Variables that are jointly significant
might appear individually insignificant, and vice versa.
To address the issue of multicollinearity, you can consider several approaches:
1. Variable Selection: Remove one or more of the highly correlated predictors from the
model. This might involve using domain knowledge, stepwise regression, or
automated feature selection techniques.
2. Combine Variables: Create new variables by combining or transforming correlated
predictors, effectively reducing the multicollinearity.
3. Ridge Regression: Ridge regression is a regularization technique that can help
mitigate multicollinearity by adding a penalty term to the coefficients. This technique
can help stabilize coefficient estimates and improve model performance.
4. Principal Component Analysis (PCA): PCA is a dimensionality reduction technique
that can be used to create uncorrelated linear combinations of the original predictors.
These components can be used as inputs in the regression analysis.
5. Collect More Data: Sometimes, collecting more data can help relieve
multicollinearity by providing a more diverse range of observations.
 6. Domain Knowledge: If multicollinearity arises due to conceptual overlap between
predictors, consulting domain experts can help decide which variables to retain or
modify.
It's important to identify and address multicollinearity to ensure that your regression analysis
provides reliable and meaningful insights into the relationships between variables.