Chapter 3: Dierence in Dierences and Policy

Evaluation
Lu Liu

Contents
1 Introduction 2

2 The Dierence Method 3

3 The Dierence-in-Dierences Method 4

4 Checking Internal Validity 8

4.1 Graphical analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

4.2 Placebo test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

4.3 Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

4.4 Dierence-in-Dierence-in-Dierences (DDD) . . . . . . . . . . . . . 11

5 Example: Troubled Asset Relief Program (TARP) 13

1
1 Introduction
Financial markets and market participants are subject to government policies,

rules and regulations. Policy makers need to evaluate the eect of the policies

to improve the implementation process. This chapter introduces some important

methods for policy evaluation.

Banking is a subject that concerns a lot with government regulations and

interventions. Examples include restrictions on the entry of new banks, limitations

of the free deployment of competitive tools by banks, restrictions on banking

activities in space and scope. In the last two-three decades, several countries have

implemented a series of deregulatory changes to stimulate competition. There

have been a lot of studies working on the impact of (de)regulation on market

structure, bank conduct, bank strategy, nancial stability and development, etc.

Moreover, the recent decade has witnessed governments' capital injection into

banking industry to address the nancial crisis. It is of crucial interest for the

regulators and general public to understand whether the primary purpose of the

intervention is fullled and if there are side-eects.

Policy evaluation is also important in many other branches in nance. For

example, extensive literature in market microstructure has studied how regulatory

changes in tick size shift the nature of trading, the structure of the markets, and

the roles of market participants.

To evaluate the eect of policy programs, we need to choose appropriate

evaluation methods in order to nd the eect that is not attributable to something

else. In the rest of this chapter, we will start by studying a simple method,

the dierence approach. Then, we will study the more advanced method, the

dierence-in-dierences approach, which better identify the impact of a policy

program.

2
2 The Dierence Method
Let's start with a simple evaluation method, namely, the dierence approach.

Provided with data on a bunch of entities right before the policy is enacted and on

the same group of entities after it is enacted, we can try to identify the eect. The

policy is enacted at time t = τ. We observe the periods of data before and after

t = τ. We can try to identify the eect by simply looking at the outcome variable y
before and after the policy. We can identify the eect as E[y|t >= τ ] − E[y|t < τ ].
We can formally justify this with a panel data model

yit = α + βTt + uit , (2.1)


0

if t < τ
Tt =
1

if t >= τ

where uit is the error term, which is orthogonal to Tt . The expected value of y
before and after the policy, respectively, are

E[y|t < τ] = α + β × 0 = α (2.2)

E[y|t >= τ ] = α + β × 1 = α + β.

Taking the dierence between the two expected values, we get the eect of the

policy, β. This is why this evaluation method is named as the dierence approach.

The model in (2.1) can be estimated using OLS.

Let's look at an example about the eect of the bank deregulation. In the US,

intrastate branching regulations imposed by state legislatures used to restrict a

bank from making state-wide branching expansions, and a bank holding company

from holding its subsidiaries in dierent counties into a single operation entity.

Beginning in the mid-1970s, individual states started to lift these restrictions at.

Bank deregulation may have impact on bank performance. Therefore, the

dependent variable yit can be the performance (measured by, e.g., prot, operating

3
costs and loan losses) of bank i at year t. Tt is a dummy variable with a value of

one when the deregulation becomes enacting.

The problem with the dierence method is that it attributes any changes in

time to the policy. Suppose something else happened at time t other than just the
program. We will attribute whatever that is to the program.

3 The Dierence-in-Dierences Method

To solve the problem of the dierence model, we compare the group that the

policy is enacted on with the group that are not aected by the policy, instead of

only comparing before and after of the former group. The former group is named

the treated group, and the latter is the control group. This method is called the

dierence-in-dierences (DD) method.

An important assumption of DD is that the treated group would follow the

same time trend as the control group if the treated group had not been treated

(i.e. aected by the policy). Therefore, the dierence before and after the policy

enactment for the control group can be used to pick up the time change of the

treated group. The idea is to correct the dierence before and after for the

treatment group by subtracting the dierence before and after for the control

group. This way, the time trend that is not attributed to the policy is removed.

The name of the method comes from taking the dierence in the before and after

dierences.

The formal formulation of the DD estimate of policy eect is

tr
|t >= τ ] − E[y tr |t < τ ] − (E[y c |t >= τ ] − E[y c |t < τ ])


β̂ = E[y (3.1)

where y tr is the outcome of the treated group, and yc is the outcome of the control

group.

4
 Figure 3.1:

As we can see in Figure 3, the DD estimator β DD corrects the dierence

estimator βD for time change that is not attributable to the policy.

We can write the DD regression as

yit = α + λDi + δTt + βDi Tt + uit , (3.2)


0

if i is in the control group
Di =
1

if i is in the treated group

0

if t < τ
Tt =
1

if t >= τ

5
 Now we illustrate that the OLS estimate of β in (3.5) is equivalent to that

given in (3.1). According to (3.5)

tr
E[y |t >= τ ] = α + λ + δ + β (3.3)

tr
E[y |t < τ ] = α + λ
c
E[y |t >= τ ] = α + δ
c
E[y |t < τ ] = α
tr
|t >= τ ] − E[y tr |t < τ ] − (E[y c |t >= τ ] − E[y c |t < τ ]) = β


E[y

(3.4)

tr
E[y |t < τ ] − E[y c |t < τ ] = λ,reecting the dierence between the treated and the
control group before τ.
Let's now return to the example about the eect of the bank deregulation

on bank performance. Using the DD method, we investigate the change in the

performance of banks before and after the reform in the states that relaxed

restrictions on intrastate branching relative to a control group of banks in the

states that were unaected by the reform.

DD with xed eects

Di and Tt on the RHS of (3.5) control for heterogeneity between groups and

between before and after. A natural extension to the DD model is including

both individual- and time-xed eects. The individual- and time- xed eects

encapsulate individual-specic and time-specic heterogeneity, which encompass

heterogeneity between groups and between before and after. This means that

Di and Tt are a linear combination of the individual-xed eects and a linear

combination of the time-xed eects, respectively. Therefore, in the DD model

with xed eects, Di and Tt become redundant and should be dropped from the

6
regression. Further, we can include other explanatory variables Xit in the DD

model. The DD regression with xed eects and control variables is formulated as

yit = βDi Tt + Xit θ + µi + κt + uit (3.5)

where µi and κt stand for individual-xed eects and time-xed eects, respec-

tively.

Treatment in a piecemeal fashion

Policy may enact on individuals in the treatment group at dierent t. We can

generalize eq(3.5) to

yit = β D̃it + Xit θ + µi + κt + uit . (3.6)

D̃it equals one if i at t has been treated and zero otherwise. In this model, the

control group at t may include individuals that get treated after t.

An example study is Jayaratne and Strahan (1998), which identies 35 states

that introduced the reform by relaxing banking restrictions. They estimate a DD

model with xed eects

P erfit = β1 Branchit + β2 Bankit + θXit + κt + µi + uit , (3.7)

where P erfit , equals one of several measures of performance (i.e., the weighted

average of commercial bank protability, aggregate nonperforming loan amount,

etc) in state i at t. Branchit is equal to one if state i does not impose restrictions

on branching via M&A at t and zero otherwise. Bankit , is equal to one if state

i is in an interstate banking agreement at t and zero otherwise. The time-xed

eect κt encompasses the national business cycle at time t, the state-xed eectµi

encompasses the state-specic component of banking performance. β1 is the DD

parameter of the eect of removing restrictions on branching, and β2 is the DD

parameter for the eect of imposing interstate banking agreement.

7
4 Checking Internal Validity
Internal validity is the extent to which you can be condent that the causal

relationship established in a study is not explained by other factors. Internal

validity in DD fails if the control group is not a valid estimate of the counter-

factual for the treatment group, in other words, if the treated group does not

follow the same trend as the treated group in the absence of the treatment. This

is true if there are confounding variables that correlate with the treatment and

drive the outcome variable. Selection bias arises if Individuals are assigned to the

treated and the control group due to dierences in confounding variables prior to

policy enactment. The DD estimator will reect dierences in the confounding

factors of the treated and the control group rather than the eect of the policy per

se.

In the example of bank deregulation, Kroszner and Strahan (1999) nd that

the relative strength of potential winners (large banks and small, bank-dependent

rms) and losers (small banks and the rival insurance rms) in bank deregulation

can explain the timing of branching deregulation across states. Therefore, the DD

estimator of bank deregulation may reect the winner-loser dierence rather than

the eect of bank deregulation per se.

Therefore, you need to check internal validity to convincingly demonstrate a

causal link between the treatment and observed outcomes. There are a set of

tools for checking internal validity. These tools are not exclusive to each other but

rather often used in combination.

4.1 Graphical analysis

One can perform a graphical analysis of the trend of the outcome variable across

the treatment and control groups during pre- and post- treatment periods. For

8
internal validity, the outcome variable of the control group should have a similar

trend with the treatment group in the pre-treatment periods. Causal eect of the

treatment may be established if the trends diverge after the treatment. See Figure

4.1.

Figure 4.1: Trend plot

4.2 Placebo test

To statistically test if the control group has a similar trend with the treatment

group, one can use data in the pre-treatment era and perform DD with a placebo

rather than the actual treatment. See Figure 4.2.

4.3 Matching
We can match individuals in the control group to those in the treated groups

based on the characteristics that aect policy enactment. For every individual

in the treated group, we should nd one (or more) control individual(s) with

similar observable characteristics against whom the eect of the treatment can be

9
 Figure 4.2: Placebo test

assessed. By matching treated individuals to similar non-treated individuals, we

reduce potential selection bias.

There exist dierent types of matching techniques. The rst generation of

matching methods paired the treated with the control based on either a single

variable or a weighting of several variables. However, when there are many

variables, it is dicult to determine along which dimensions to match units or

which weighting scheme to adopt. The propensity score matching method is

especially useful under such circumstances. A propensity score is the probability of

a individual being assigned to a policy program given a set of observed covariates.

The rst step of the propensity score matching involves estimating the propensity

score using a logit/probit model. In the second step, a treated individual is

matched to one (or) more control individual(s) whose propensity score is the closest

to the treated individual's.

Most matching techniques (like propensity score matching) match treated

groups to control groups based on observed characteristics only. Still, there

may be unobserved characteristics that still result in selection bias. There is no

10
standard solution for the unobserved confounding problem. There are specic

solutions given certain situations. For example, when examining the eect of

intrastate branching restriction on local economic growth, Huang (2008) points out

that banking deregulation could be induced by an expectation of future growth

opportunities (unobservable to econometricians), which could create a spurious

correlation between banking deregulation and future growth accelerations. To

mitigate the potential bias caused by the unobserved confounding factor (i.e.

expectation of future growth opportunities), Huang (2008) creates a geographic-

matching method. The method compares economic performance of contiguous

counties on opposite side of state borders, where only one state experienced the

deregulation. This geographic-matching procedure mitigates bias caused by both

observable and unobservable characteristics because contiguous countries are less

likely to dier from each other.

4.4 Dierence-in-Dierence-in-Dierences (DDD)

Dierence-in-dierence-in-dierences (DDD) approach builds on rening the de-

nition of treatment and control groups. It can be used to control for heterogeneity

that confounds the treatment eect.

Suppose a state implements a program that aims at regulating big banks' risk

taking behaviour. The outcome variable y measures the level of risk taking. A

DD strategy is to use big banks in the program-state as the treated group and big

banks in non-program states as the control group. The DD estimator is therefore

   
P,B P,B N P,B N P,B
βB = ȳt≥τ − ȳt<τ − ȳt≥τ − ȳt<τ (4.1)

P NP
where superscript standards for the programme state, for non-program

B
states, for big banks.

11
 A potential problem of this strategy is that the DD estimate does not take

account of non-program factors that make banks in dierent states dier. For

example, dierent industrial structure across states may induce dierent risk

taking behaviors. To control for the heterogeneity across states that confound

the treatment eect, we can estimate another DD, the DD of small banks across

the program and non-program states. Since small banks are not aected by the

program, the DD estimate for small banks in the program state and the non-

program state provides an estimate of the non-program factors that aect states

dierentially.

   
P,S P,S N P,S N P,S
βS = ȳt≥τ − ȳt<τ − ȳt≥τ − ȳt<τ (4.2)

S
where ȳ stands for expected value of y . superscript standards for small banks.

Subtracting βS from βB results in a DDD estimate. It adjusts the before/after

change of big banks in the program for both general trends aecting big banks

(by the rst DD) and for other factors dierentially aecting states (by the second

DD).

β DDD = β B − β S
       
P,B P,B N P,B N P,B P,S P,S N P,S N P,S
= ȳt≥τ − ȳt<τ − ȳt≥τ − ȳt<τ − ȳt≥τ − ȳt<τ − ȳt≥τ − ȳt<τ

(4.3)

However, if βS is insignicant, β DDD is inecient, and βB is a preferred

estimator of the program eect on big banks.

To estimate the DDD estimator with a regression, let P be a dummy variable

equal to one for the banks in the program state, and T be a dummy variable equal

to one for time after the treatment, B be a dummy variable for big banks.

12
 Then, we have

yit = β0 + β1 Pi + β2 Bi + β3 Tt + β4 Pi Bi + β5 Pi Tt + β6 Bi Tt + β7 Pi Bi Tt + uit .
(4.4)

The coecient β7 on the triple interaction term Pi Bi Tt is the DDD estimator.

P,B
ȳt≥τ = β0 + β1 + β2 + β3 + β4 + β5 + β6 + β7
P,B
ȳt<τ = β0 + β1 + β2 + β4
N P,B
ȳt≥τ = β0 + β2 + β3 + β6
N P,B
ȳt<τ = β0 + β2
P,S
ȳt≥τ = β0 + β1 + β3 + β5
P,S
ȳt<τ = β0 + β1
N P,S
ȳt≥τ = β0 + β3
N P,S
ȳt<τ = β0

5 Example: Troubled Asset Relief Program (TARP)

One of the most notable program on the banking industry in the recent years is

the Troubled Asset Relief Program (TARP), one of the largest intervention
by the US government to address the subprime mortgage crisis. The Capital

Purchase Program (CPP) is the main component of TARP. Rather than purchasing

troubled assets, the CPP authorized the U.S. Treasury to invest up to $250

billion in preferred equity of selected nancial institutions to enhance their capital

ratios. Initial receipts were nine large involuntary participants (Citigroup, Bank of

America, JPMorgan Chase,Wells Fargo, Goldman Sachs Group, Morgan Stanley,

Wachovia Corporation, State Street Corporation, and Merrill Lynch) with $125

billion invested. The rest of the receipts followed the formal CPP process and

13
applied for CPP funds from the U.S. Treasury. Approval to receive TARP took

into account the health of the banking organizations, with the viable, healthier

ones being more likely to receive capital. During 2008:Q4-2009:Q4, TARP infused

$204.9 billion into 709 banking organizations.

The primary purposes of TARP were to improve nancial stability and to

encourage banks to increase lending. Economists have paid much attention on

evaluating the eects of TARP. For example, TARP may also have had unintended

eects on bank competition and resource allocation. For example, using the DD

method, Berger and Roman (2015) nd that TARP recipients received competitive

advantages and increased both their market shares and market power.

yit = λ · TARP_recipienti + β · TARP_recipienti × POST_TARPt + Xit θ + κt + uit

(5.1)

yit is a competitive advantage indicator (market share or market power), TARP_recipient

is a dummy variable equal to 1 if the bank was provided TARP capital

support. POST_TARPt is a dummy equal to 1 after the capital injection.

TARP_recipienti × POST_TARPt is the DD coecient and captures the eect

of the treatment (TARP) on the treated (TARP recipients) compared to the

untreated (non-TARP banks) after treatment. Xit are control variables, κt is

a series of time xed eects.

Black and Hazelwood (2013) show that large TARP banks did not increase

lending but the their risk of loan increased relative to non-TARP peers.This is

suggestive of moral hazard. Further, TARP may also have had unintended eects

on bank competition and resource allocation.

14
References
Berger, A. N. and Roman, R. A. (2015). Did TARP banks get competitive

advantages? Journal of Financial and Quantitative Analysis, 50(06):11991236.

Black, L. K. and Hazelwood, L. N. (2013). The eect of TARP on bank risk-taking.

Journal of Financial Stability, 9(4):790803.

Huang, R. R. (2008). Evaluating the real eect of bank branching deregulation:

Comparing contiguous counties across us state borders. Journal of Financial

Economics, 87(3):678705.
Jayaratne, J. and Strahan, P. E. (1998). Entry restrictions, industry evolution,

and dynamic eciency: Evidence from commercial banking. The Journal of

Law and Economics, 41(1):239274.
Kroszner, R. S. and Strahan, P. E. (1999). What drives deregulation? economics

and politics of the relaxation of bank branching restrictions. The Quarterly

Journal of Economics, 114(4):14371467.

15