Econometrics Cheat Sheet Simple Regression Model Multiple Regression Model

by Tyler Ransom, University of Oklahoma Regression is useful because we can estimate a ceteris paribus Multiple regression is more useful than simple regression
@tyleransom relationship between some variable x and our outcome y because we can more plausibly estimate ceteris paribus
relationships (i.e. E (u|x) = E (u) is more plausible)
y = β0 + β1 x 1 + · · · + βk x k + u
Data & Causality y = β0 + β1 x + u
β̂1 , . . . , β̂k : partial effect of each of the x’s on y
Basics about data types and causality. We want to estimate β̂1 , which gives us the effect of x on y.
β̂0 = y − β̂1 x1 − · · · − β̂k xk
Types of data OLS formulas d (y, residualized xj )
Cov
β̂j =
Experimental Data from randomized experiment To estimate β̂0 and β̂1 , we make two assumptions: Vd
ar (residualized xj )
Observational Data collected passively where “residualized xj ” means the residuals from OLS
1. E (u) = 0
Cross-sectional Multiple units, one point in time regression of xj on all other x’s (i.e. x1 , . . . , xj−1 , xj+1 , . . . xk )
Time series Single unit, multiple points in time 2. E (u|x) = E (u) for all x
Longitudinal (or Panel)Multiple units followed over multiple Gauss-Markov Assumptions
time periods When these hold, we get the following formulas:
1. y is a linear function of the β’s
Experimental data β̂0 = y − β̂1 x 2. y and x’s are randomly sampled from population
d (y, x) 3. No perfect multicollinearity
• Correlation =⇒ Causality Cov
β̂1 = 4. E (u|x1 , . . . , xk ) = E (u) = 0 (Unconfoundedness)
• Very rare in Social Sciences Vd
ar (x)
5. V ar (u|x1 , . . . , xk ) = V ar (u) = σ 2 (Homoskedasticity)
fitted values (ŷi ) ŷi = β̂0 + β̂1 xi When (1)-(4) hold: OLS is unbiased; i.e. E(β̂j ) = βj
Statistics basics residuals (ûi ) ûi = yi − ŷi When (1)-(5) hold: OLS is Best Linear Unbiased Estimator
P
We examine a random sample of data to learn about the Total Sum of Squares SST = N (y − y)2 Variance of u (a.k.a. “error variance”)
Pi=1 i
population Expl. Sum of Squares SSE = N i=1 (ŷi − y)
2
PN 2 SSR
Resid. Sum of Squares SSR = i=1 ûi σ̂ 2 =
Random sample Representative of population R-squared (R2 ) R2 = SSE ; N −K−1
SST
Parameter (θ) Some number describing population “frac. of var. in y explained by x” XN
1
Estimator of θ Rule assigning value of θ to sample = û2
1 PN Algebraic properties of OLS estimates N − K − 1 i=1 i
e.g. Sample average, Y = N i=1 Yi
PN
Estimate of θ What the estimator spits out
i=1 ûi = 0 (mean & sum of residuals is zero) Variance and Standard Error of β̂j
for a particular sample (θ̂) PN σ2
Sampling distribution Distribution of estimates i=1 xi ûi = 0 (zero covariance bet. x and resids.) V ar(β̂j ) = , j = 1, 2, ..., k
across all possible samples SSTj (1 − Rj2 )
The OLS line (SRF) always passes through (x, y) where
Bias of estimator W E (W ) − θ
f) N
X
Efficiency W efficient if V ar(W ) < V ar(W SSE + SSR = SST
SSTj = (N − 1)V ar(xj ) = (xij − xj )
Consistency W consistent if θ̂ → θ as N → ∞ 0 ≤ R2 ≤ 1 i=1

Hypothesis testing Rj2 = R2 from a regression of xj on all other x’s

Interpretation and functional form √
The way we answer yes/no questions about our population Standard deviation: pV ar
Our model is restricted to be linear in parameters
using a sample of data. e.g. “Does increasing public school Standard error: Vd
ar
But not linear in x s
spending increase student achievement?” σ̂ 2
se(β̂j ) = , j = 1, . . . , k
Other functional forms can give more realistic model SSTj (1 − Rj2 )
null hypothesis (H0 ) Typically, H0 : θ = 0
alt. hypothesis (Ha ) Typically, H0 : θ 6= 0 Classical Linear Model (CLM)
significance level (α) Tolerance for making Type I error; Model DV RHS Interpretation of β1 Add a 6th assumption to Gauss-Markov:
(e.g. 10%, 5%, or 1%)

Level-level y x ∆y = β1 ∆x 6. u is distributed N 0, σ 2
test statistic (T ) Some function of the sample of data
critical value (c) Value of T such that reject H0 if |T | > c; Level-log y log(x) ∆y ≈ (β1 /100) [1%∆x] Need this to know what the distribution of β̂j is
c depends on α; Log-level log(y) x %∆y ≈ (100β1 ) ∆x Otherwise, need asymptotics to do hypothesis testing of β’s
c depends on if 1- or 2-sided test Log-log log(y) log(x) %∆y ≈ β1 %∆x
p-value Largest α at which fail to reject H0 ; Quadratic y x + x2 ∆y = (β1 + 2β2 x) ∆x Testing Hypotheses about the β’s
reject H0 if p < α Note: DV = dependent variable; RHS = right hand side
Under A (1)–(6),1 can test hypotheses about the β’s
1 Or, A (1)–(5) combined with asymptotic properties
t-test for simple hypotheses Interaction terms TS Forecasting
To test a simple hypothesis like interaction term: When two x’s are multiplied together A good forecast minimizes forecasting error fˆt :

h i
H 0 : βj = 0 y = β0 + β1 happy + β2 age + β3 happy × age + u min E e2t+1 |It = E (yt+1 − ft )2 |It
ft
Ha : βj 6= 0
β3 : difference in age slope for those who are happy where It is the information set
use a t-test: compared to those who are unhappy
β̂j − 0 RMSE measures forecast performance (on future data):
t=   Linear Probability Model (LPM) v
se β̂j u m−1
u1 X
When y is a dummy variable, e.g. Root Mean Squared Error = t ê2
where 0 is the null hypothesized value. m h=0 T +h+1
happy = β0 + β1 age + β2 income + u
Reject H0 if p < α or if |t| > c (See: Hypothesis testing)
β’s are interpreted as change in probability: Model with lowest RMSE is best forecast
F -test for joint hypotheses
• Can choose ft in many ways
Can’t use a t-test for joint hypotheses, e.g.: ∆ Pr(y = 1) = β1 ∆x
• Basic way: ŷT +1 from linear model
H0 : β3 = 0, β4 = 0, β5 = 0 By definition, homoskedasticity is violated in the LPM
• ARIMA, ARMA-GARCH are cutting-edge models
Ha : β3 6= 0 OR β4 6= 0 OR β5 6= 0
Instead, use F statistic:
Time Series (TS) data Granger causality
• Observe one unit over many time periods z Granger causes y if, after controlling for past values of y,
(SSRr − SSRur )/(dfr − dfur ) (SSRr − SSRur )/q past values of z help forecast yt
F = = • e.g. US quarterly GDP, 3-month T-bill rate, etc.
SSRur /dfur SSRur /(N − k − 1)
where • New G-M assumption: no serial correlation in ut CLM violations
Heteroskedasticity
SSRr = SSR of restricted model (if H0 true) • Remove random sampling assumption (makes no sense)
• Test: Breusch-Pagan or White tests (H0 : homosk.)
SSRur = SSR of unrestricted model (if H0 false)
Two focuses of TS data • If H0 rejected, SEs, t-, and F-stats are invalid
q = # of equalities in H0 ?
1. Causality (e.g. ↑ taxes =⇒ ↓ GDP growth) • Instead use heterosk.-robust SEs and t- and F-stats
N − k − 1 = Deg. Freedom of unrestricted model
2. Forecasting (e.g. AAPL stock price next quarter?) Serial correlation
Reject H0 if p < α or if F > c (See: Hypothesis testing) • Test: Breusch-Godfrey test (H0 : no serial corr.)
Requirements for TS data
• If H0 rejected, SEs, t-, and F-stats are invalid
Note: F > 0, always To properly use TS data for causal inf / forecasting,
need data free of the following elements: • Instead use HAC SEs and t- and F-stats
Qualitative data • HAC: “Heterosk. and Autocorrelation Consistent”
• Can use qualitative data in our model Trends: y always ↑ or ↓ every period
• Must create a dummy variable
Seasonality: y always ↑ or ↓ at regular intervals Measurement error
Non-stationarity: y has a unit root; i.e. not stable • Measurement error in x can be a violation of A4
• e.g. “Yes” represented by 1 and “No” by 0
Otherwise, R2 and β̂j ’s are misleading
• Attenuation bias: βj biased towards 0
dummy variable trap: Perfect collinearity that happens
when too many dummy variables are included in the model AR(1) and Unit Root Processes Omitted Variable Bias
y = β0 + β1 happy + β2 not happy + u AR(1) model (Auto Regressive of order 1): When an important x is excluded: omitted variable bias.
The above equation suffers from the dummy variable trap yt = ρyt−1 + ut Bias depends on two forces:
because units can only be “happy” or “not happy,” so 1. Partial effect of x2 on y (i.e. β2 )
including both would result in perfect collinearity with the Stable if |ρ| < 1; Unit Root if |ρ| ≥ 1
intercept “Non-stationary,” “Unit Root,” “Integrated” are all 2. Correlation between x2 and x1
synonymous Which direction does the bias go?
Interpretation of dummy variables
Interpretation of dummy variable coefficients is always relative Correcting for Non-stationarity
Corr(x1 , x2 ) > 0 Corr(x1 , x2 ) < 0
to the excluded category (e.g. not happy):
Easiest way is to take a first difference:
β2 > 0 Positive Bias Negative Bias
y = β0 + β1 happy + β2 age + u β2 < 0 Negative Bias Positive Bias
First difference: Use ∆y = yt − yt−1 instead of yt
β1 : avg. y for those who are happy compared to those who Test for unit root: Augmented Dickey-Fuller (ADF) test Note: “Positive bias” means β1 is too big;
are unhappy, holding fixed age H0 of ADF test: y has a unit root “Negative bias” means β1 is too small
 How to resolve E (u|x) 6= 0 Panel data quis, viverra ac, nunc. Praesent eget sem vel leo ultrices
How can we get unbiased β̂j ’s when E (u|x) 6= 0? Follow same sample of units over multiple time periods bibendum. Aenean faucibus. Morbi dolor nulla, malesuada eu,
pulvinar at, mollis ac, nulla. Curabitur auctor semper nulla.
• Include lagged y as a regressor yit = β0 + β1 xit1 + · · · + βk xitk + ai + uit Donec varius orci eget risus. Duis nibh mi, congue eu,
| {z }
accumsan eleifend, sagittis quis, diam. Duis eget orci sit amet
• Include proxy variables for omitted ones νit
orci dignissim rutrum.
• Use instrumental variables • νit = composite error
• Use a natural experiment (e.g. diff-in-diff) • ai = unit-specific unobservables Nam dui ligula, fringilla a, euismod sodales, sollicitudin vel,
• Use panel data wisi. Morbi auctor lorem non justo. Nam lacus libero, pretium
• uit = idiosyncratic error
at, lobortis vitae, ultricies et, tellus. Donec aliquet, tortor sed
Instrumental Variables (IV) • Allow E (a|x) 6= 0 accumsan bibendum, erat ligula aliquet magna, vitae ornare
A variable z, called the instrument, satisfies: • Maintain E (u|x) = 0 odio metus a mi. Morbi ac orci et nisl hendrerit mollis.
Suspendisse ut massa. Cras nec ante. Pellentesque a nulla.
1. cov (z, u) = 0 (not testable) Four different methods of estimating βj ’s: Cum sociis natoque penatibus et magnis dis parturient
2. cov (z, x) 6= 0 (testable) montes, nascetur ridiculus mus. Aliquam tincidunt urna.
1. Pooled OLS (i.e. ignore composite error)
Nulla ullamcorper vestibulum turpis. Pellentesque cursus
z typically comes from a natural experiment 2. First differences (FD): luctus mauris.
∆yi = β1 ∆xi1 + · · · + ∆βk xik + ∆ui
cov (z, y) Nulla malesuada porttitor diam. Donec felis erat, congue non,
β̂IV = estimated via Pooled OLS on transformed data volutpat at, tincidunt tristique, libero. Vivamus viverra
cov (z, x)
3. Fixed effects (FE): fermentum felis. Donec nonummy pellentesque ante. Phasellus
• SE’s much larger when using IV compared to OLS adipiscing semper elit. Proin fermentum massa ac quam. Sed
• Be aware of weak instruments yit − y i = β1 (xit1 − xi1 ) + · · · diam turpis, molestie vitae, placerat a, molestie nec, leo.
When there are multiple instruments: + βk (xitk − xik ) + (uit − ui ) Maecenas lacinia. Nam ipsum ligula, eleifend at, accumsan
nec, suscipit a, ipsum. Morbi blandit ligula feugiat magna.
• use Two-stage least squares (2SLS) estimated via Pooled OLS on transformed data Nunc eleifend consequat lorem. Sed lacinia nulla vitae enim.
• exclude at least as many z’s as endogenous x’s 4. Random effects (RE): Pellentesque tincidunt purus vel magna. Integer non enim.
1st stage: regress endogenous x on z’s and exogenous x’s Praesent euismod nunc eu purus. Donec bibendum quam in
yit − θy i = β0 (1 − θ) + β1 (xit1 − θxi1 ) + · · · tellus. Nullam cursus pulvinar lectus. Donec et mi. Nam
2nd stage: regress y on x̂ and exogenous x’s
+ βk (xitk − θxik ) + (νit − θν i ) vulputate metus eu enim. Vestibulum pellentesque felis eu
Test for weak instruments: Instrument is weak if massa.
estimated via FGLS, where
• 1st stage F stat < 10 s
√ 2 Quisque ullamcorper placerat ipsum. Cras nibh. Morbi vel
• or 1st stage |t| < 10 ≈ 3.2 σu
θ =1− 2
justo vitae lacus tincidunt ultrices. Lorem ipsum dolor sit
Difference in Differences (DiD) σu + T σa2
amet, consectetuer adipiscing elit. In hac habitasse platea
Can get causal effects from pooled cross sectional data β̂RE → β̂F E as θ → 1 dictumst. Integer tempus convallis augue. Etiam facilisis.
Nunc elementum fermentum wisi. Aenean placerat. Ut
A nat. experiment divides units into treatment, control groups β̂RE → β̂P OLS as θ → 0 imperdiet, enim sed gravida sollicitudin, felis odio placerat
quam, ac pulvinar elit purus eget enim. Nunc vitae tortor.
yit = β0 + δ0 d2t + β1 dTit + δ1 d2t × dTit + uit Proin tempus nibh sit amet nisl. Vivamus quis tortor vitae
RE assumes E (a|x) = 0
where risus porta vehicula.
• d2t = dummy for being in time period 2 Cluster-robust SEs Fusce mauris. Vestibulum luctus nibh at lectus. Sed
• dTit = dummy for being in the treatment group • Serial correlation of νit is a problem bibendum, nulla a faucibus semper, leo velit ultricies tellus, ac
• δ̂1 = difference in differences venenatis arcu wisi vel nisl. Vestibulum diam. Aliquam
• Use cluster-robust SEs


pellentesque, augue quis sagittis posuere, turpis lacus congue
δ̂1 = y treat,2 − y control,2 − y treat,1 − y control,1 • These correct for serial corr. and heterosk. quam, in hendrerit risus eros eget felis. Maecenas eget erat in
• Cluster at the unit level sapien mattis porttitor. Vestibulum porttitor. Nulla facilisi.
Extensions:
Sed a turpis eu lacus commodo facilisis. Morbi fringilla, wisi in
• Can also include x’s in the model Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Ut
dignissim interdum, justo lectus sagittis dui, et vehicula libero
purus elit, vestibulum ut, placerat ac, adipiscing vitae, felis.
• Can also use with more than 2 time periods dui cursus dui. Mauris tempor ligula sed lacus. Duis cursus
Curabitur dictum gravida mauris. Nam arcu libero, nonummy
• δ̂1 has same interpretation, different math formula enim ut augue. Cras ac magna. Cras nulla. Nulla egestas.
eget, consectetuer id, vulputate a, magna. Donec vehicula
Curabitur a leo. Quisque egestas wisi eget nunc. Nam feugiat
augue eu neque. Pellentesque habitant morbi tristique
Validity: lacus vel est. Curabitur consectetuer.
senectus et netus et malesuada fames ac turpis egestas. Mauris
• Need y changing across time and treatment for reasons ut leo. Cras viverra metus rhoncus sem. Nulla et lectus
only due to the policy vestibulum urna fringilla ultrices. Phasellus eu tellus sit amet
• a.k.a. parallel trends assumption tortor gravida placerat. Integer sapien est, iaculis in, pretium Layout by Winston Chang, http://wch.github.io/latexsheet/