05 Generalizing from data

Gábor Békés

Data Analysis 1: Exploration

2020
Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

Slideshow for the Békés-Kézdi Data Analysis textbook

I Cambridge University Press, 2021

I gabors-data-analysis.com
I Download all data and code:
gabors-data-analysis.com/data-
and-code/

I This slideshow is for Chapter 05

05 Generalizing from data 2 / 42 Gábor Békés

Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

Motivation

I How likely is it that we shall experience losses on our investment portfolio? To

answer this, you have collected and analyzed past financial information. To predict
the frequency of a loss of certain magnitude for the coming calendar year, you will
need to make an inference and think hard about what can be different in the future.

05 Generalizing from data 3 / 42 Gábor Békés

Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

Generalization

I Sometimes we analyze a dataset with the goal of learning about patterns in that
dataset alone.
I In such cases there is no need to generalize our findings to other datasets.
I Example: We search for a good deal among offers of hotels, all we care about are
the observations in our dataset.

I Often we analyze a dataset in order to learn about patterns that may be true in
other situations.
I We are interested in finding it the relationship between
I Our dataset
I The situation we care about

05 Generalizing from data 4 / 42 Gábor Békés

Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

Generalization

I Generalize the results from a single dataset to other situations.

I The act of generalization is called inference: we infer something from our data
about a more general phenomenon because we want to use that knowledge in
some other situation.

I Aspect 1: statistical inference

I Aspect 2: external validity

05 Generalizing from data 5 / 42 Gábor Békés

Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

Statistical inference

I Uses statistical methods to make inference.

I Well-developed and powerful toolbox that helps generalizing to situations similar to
our data.
I Similar to ours = general pattern represented by our dataset.

I The general pattern is an abstract thing that may or may not exist.
I If we can assume that the general pattern exists, the tools of statistical inference
can be very helpful.

05 Generalizing from data 6 / 42 Gábor Békés

Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

General patterns 1: Population and representative sample

I The cleanest example of representative data is a representative sample of a

well-defined population.
I A sample is representative of a population if the distribution of all variables is very
similar in the sample and the population.
I Random sampling is the best way to achieve a representative sample.

05 Generalizing from data 7 / 42 Gábor Békés

Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

General patterns 2: No population but general pattern

The concept of representation is less straightforward in other setups.

I Using data with observations from the past to uncover a pattern that may be true
for the future.
I Generalizing patterns observed among some products to other, similar products.
There isn’t necessarily a “population” from which a random sample was drawn on purpose.
Instead, we should think of our data as one that represents a general pattern.
I There is a general pattern, each year is a random realization.
I There is a general pattern, each product is a random version, all represented by the
same general pattern.

05 Generalizing from data 8 / 42 Gábor Békés

Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

External validity

I Assessing whether our data represents the same general pattern that would be
relevant for the situation we truly care about.
I Externally valid case: the situation we care about and the data we have represent
the same general pattern
I With external validity, our data can tell what to expect.
I No external validity: whatever we learn from our data, may turn out to be not
relevant at all.

05 Generalizing from data 9 / 42 Gábor Békés

Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

The process of inference

The process of inference

1. Consider a statistic we may care about, such as the mean.
2. Compute its estimated value from a dataset
3. Infer the value in the population / in the general pattern, that our data represents.

It is good practice to divide the inference problem into two.

1. Use statistical inference to learn about the population, or general pattern, that our
data represents.
2. Assess external validity: define the population, or general pattern we are interested
in and assess how it compares to the population, or general pattern, that our data
represents.

05 Generalizing from data 10 / 42 Gábor Békés

Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

Stock market returns: Inference

I Task: Assess the likelihood of experiencing a loss of certain magnitude on an

investment portfolio from one day to the next day
I Predict the frequency of a loss of certain magnitude for the coming calendar year

I The investment portfolio is the S&P 500, a US stock market index

I Data: day-to-day returns on the S&P 500, defined as percentage changes in the
closing price of the index between two consecutive days
I 11 years: 25 August 2006 to 26 August 2016. It includes 2,519 days.

05 Generalizing from data 11 / 42 Gábor Békés

Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

Histogram of daily returns

Note: S&P 500 market index. Day to day (gaps

ignored) changes, in percentage. From August 25
2006 to August 26 2016.
05 Generalizing from data 12 / 42 Gábor Békés
Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

Stock market returns: Inference

I To define "loss", we take a day-to-day loss exceeding 5 percent.

I "loss" is a binary variable, taking 1 when the day-to-day loss exceeds 5 percent
and zero otherwise.
I The statistic in the data is the proportion of days with such losses.
I It is 0.5 percent in this dataset
I the S&P500 portfolio lost more than 5 percent of its value on 0.5 percent of the
days between August 25 2006 and August 26 2016.

I Inference problem: How can we generalize this finding? What can we infer from
this 0.5 percent chance for the next calendar year?

05 Generalizing from data 13 / 42 Gábor Békés

Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

Repeated samples

I Repeated samples - the conceptual background to statistical inference

I Our data - one example of many datasets that could have been observed.
I Each datasets can be viewed as samples drawn from the population (general
pattern)

I Easier concept: When our data is sample from a well-defined population - many
other samples could have turned out instead of what we have.
I Harder concept: no clear definition of population. We think of a general pattern
we care about.

05 Generalizing from data 14 / 42 Gábor Békés

Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

Repeated samples

I The goal of statistical inference is learning the value of a statistic in the

population, or general pattern, represented by our data.
I The statistic has a distribution: its value may differ from sample to sample.
I The distribution of the statistic of interest is called its sampling distribution

05 Generalizing from data 15 / 42 Gábor Békés

Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

Repeated samples

I Standard deviation in this distribution: spread across repeated samples

I The standard error (SE) of the statistic = the standard deviation of the sampling
distribution
I Any particular estimate is likely to be an erroneous estimate of the true value. The
magnitude of that typical error is one SE.

05 Generalizing from data 16 / 42 Gábor Békés

Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

Repeated samples properties

The sampling distribution of a statistic is the distribution of this statistic across repeated
samples.
The sampling distribution has three important properties
1. Unbiasedness: The average of the values in repeated samples is equal to its true
value (=the value in the entire population / general pattern).
2. Asymptotic normality: The sampling distribution is approximately normal. With
large sample size, it is very very close.
3. Root-n convergence: The standard error (the standard deviation of the sampling
distribution) is smaller the larger the samples, with a proportionality factor of the
square root of the sample size.

05 Generalizing from data 17 / 42 Gábor Békés

Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

Stock market returns: A simulation

I We can not rerun history many many times...

I Simulation exercise - to better understand how repeated samples work
I Suppose the 11-year dataset is the population - the fraction of days with 5%+
losses is 0.5% in the entire 11 years’ data. That’s the true value.
I Assume we have only three years (900 days) of daily returns in our dataset.
I Task: estimate the true value of the fraction in the 11-year period from the data
we have using a simulation exercise.
1. many data table with three years’ worth of observations may be created from the 11
years’ worth of data,
2. compute the fraction of days with 5%+ losses in data tables
3. learn about the true value

05 Generalizing from data 18 / 42 Gábor Békés

Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

Stock market returns: A simulation

I Do simple random sampling: days are considered one after the other and are
selected or not selected in an independent random fashion.
I This sampling destroys the time series nature
I This is OK because daily returns are (almost) independent across days in the original
dataset
I We do this 10,000 times....

05 Generalizing from data 19 / 42 Gábor Békés

Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

Stock market returns: A simulation

I percent of days with losses of 5% of

more.
I histogram created from the 10,000
random samples, each w/ 900 obs,
drawn from entire dataset
I distribution has some spread:
smallest realization is 0.1 %, while
the largest is smaller than 1.25 %

Histogram of the proportion of days with losses of 5 percent or more, across repeated samples of
size n=900. 10,000 random samples. Source: sandp-stocks data. S&P 500 market index.

05 Generalizing from data 20 / 42 Gábor Békés

Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

Stock market returns: Sampling distributions

I Proportion of days with losses

of 5 percent or more
I Repeated samples in two
simulation exercises, with
n=500 and n=1,000. (10,000
random samples)
I Kernel density (goes to minus /
can cut it at 0)
I Role of sample size: smaller
sample: skewed; higher
standard deviation

05 Generalizing from data 21 / 42 Gábor Békés

Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

The standard error and the confidence interval

I Confidence interval (CI) - measure of statistical inference.

I Recall: Statistical inference - we analyze a dataset to infer the true value of a
statistic: its value in the population, or general pattern, represented by our data.
I The CI defines a range where we can expect the true value in the population, or
the general pattern.
I CI gives a range for the true value with a probability
I Probability tells how likely it is that the true value is in that range
I Probability - data analysts need to picks it, such as 95%

05 Generalizing from data 22 / 42 Gábor Békés

Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

The standard error and the confidence interval

I The “95 percent CI” gives the range of values where we think that true value falls
with a 95 percent likelihood.
I Viewed from the perspective of a single sample, the chance (probability) that the
truth is within the CI measured around the value estimated from that single sample
is 95 percent.

I Also: we think that with 5 percent likelihood, the true value will fall outside the
confidence interval.

05 Generalizing from data 23 / 42 Gábor Békés

Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

The standard error and the confidence interval

I Confidence interval - symmetric range around the estimated value of the statistic
in our dataset.
I Get estimated value.
I Define probability
I Calculate CI with the use of SE
I 95 percent CI is the ±1.96SE (but we use ±2SE ) interval around the estimate
from the data.
I 90% CI is the ±1.6SE interval, the 99 % CI is the ±2.6SE

05 Generalizing from data 24 / 42 Gábor Békés

Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

Calculating the standard error

An important consequence of evidence from the repeated sample exercise:

I In reality, we don’t get to observe the sampling distribution. Instead, we observe a
single dataset
I That dataset is one of the many potential samples that could have been drawn
from the population, or general pattern
I Good news: We can get a very good idea of how the sampling distribution would
look like - good estimate of the standard error - even from a single sample.
I Getting SE – Option 1: Use a formula
I Getting SE – Option 2: Simulate by a new method, called ’bootstrapping’

05 Generalizing from data 25 / 42 Gábor Békés

Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

Calculating the standard error

Consider the statistic of the sample mean.
I Assume the values of x are independent across observations in the dataset.
I x̄ is the estimate of the true mean value of x in the general pattern/population.
(LLN)
I Sampling distribution is approximately normal, with the true value as its mean.
(CLT)  
a 1
x̄ ∼ N E [x] , Var [x̄]
n
The standard error formula for the estimated x̄ is
1
SE (x̄) = √ Std[x] (1)
n
where Std[x] is the standard deviation of the variable x in the data and n is the number
of observations in the data.
05 Generalizing from data 26 / 42 Gábor Békés
Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

The standard error formula

I The standard error is larger...

I the larger the standard deviation of the variable.
I the smaller the sample and
I For intuition, consider SE (x̄) vs Std[x].
I Think back to the repeated samples simulation exercise:
I SE (x̄) = the standard error of x̄ is the standard deviation of the various x̄ estimates
across repeated samples.
I The larger the standard deviation of x itself, the more variation we can expect in x̄
across repeated samples.

05 Generalizing from data 27 / 42 Gábor Békés

Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

Stock market returns: The standard error formula

Let’s consider our example of 11-years’ of data on daily returns on the S&P 500 portfolio.

I The calculated statistics, P(loss > 5%) = 0.5%

I The SE [P(loss > 5%)] is calculated by,

√
I The size of the sample is n = 2,519 so that 1/ n = 0.02.
I The standard deviation of the fraction of SD [P(loss > 5%)] = 0.07.
I So the SE = 0.07 ∗ 0.02 = 0.0014 (0.14 percent).
I Can calculate the 95 percent CI:
I CI = [0.5 − 2 ∗ SE , 0.5 + 2 ∗ SE ] = [0.22, 0.78]

I This means that in the general pattern represented by the 11-year history of
returns in our data, we can be 95 percent confident that daily losses of more than
5 percent occur with a 0.2 to 0.8 percent chance.
05 Generalizing from data 28 / 42 Gábor Békés
Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

Take a quick stop to summarize the idea of CI

I We are interested in generalizing from our data. Statistical inference.

I Consider a statistic such as the sample mean x̄
I Take a 95% confidence interval - where we can expect to see the true value
I CI=statistic +/−2SE .
I We have a formula for the SE calculated from our the data only using the standard
deviation and sample size.
I Using the CI, we can now do statistical inference, generalize for the population /
general pattern we care about.

05 Generalizing from data 29 / 42 Gábor Békés

Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

The bootstrap

I Bootstrap is a method to create synthetic samples that are similar but different
I An method that is very useful in general.
I It is essential for many advanced statistics application such as machine learning

I More in Chapter 05

05 Generalizing from data 30 / 42 Gábor Békés

Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

The bootstrap

I The bootstrap method takes the original dataset and draws many repeated
samples of the size of that dataset.
I The trick is that the samples are drawn with replacement.
I The observations are drawn randomly one by one from the original dataset; once
an observation is drawn it is “replaced” to the pool so that it can be drawn again,
with the same probability as any other observation.
I The drawing stops when it reaches the size of the original dataset.
I The result is a sample of the same size as the original dataset, yielding a single
bootstrap sample.

05 Generalizing from data 31 / 42 Gábor Békés

Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

The bootstrap

I A bootstrap sample is always the

same size the original
I it includes some of the original
observations multiple times,
I it does not include some of other
original observations.
I We typically create 500 - 10,000
samples
I Computationally intensive but
feasible, relatively fast.

05 Generalizing from data 32 / 42 Gábor Békés

Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

The bootstrap

I We have a dataset
(the sample), can
compute a statistic
(e.g. mean)
I Create many
bootstrap samples,
and get a mean value
for each sample
I Bootstrap estimate
of SE = standard
deviation of statistic
based on bootstrap
samples’ estimates.
05 Generalizing from data 33 / 42 Gábor Békés
Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

The bootstrap SE

I The bootstrap method creates many repeated samples that are different from each
other, but each has the same size as the original dataset.
I Bootstrap gives a good approximation of the standard error, too.
I The bootstrap estimate (or the estimate from the bootstrap method) of the
standard error is simply the standard deviation of the statistic across the bootstrap
samples.

05 Generalizing from data 34 / 42 Gábor Békés

Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

Stock market returns: The Bootstrap standard error

I We estimate the standard error by The process

bootstrap. 1. Take the original dataset and draw a
I Let’s consider our example of bootstrap sample.
11-years’ of data on daily returns on 2. Calculate the proportions of days with
the S&P 500 portfolio. 5%+ loss in that sample.
I Do the process —————–> 3. Save that value.
I End up with a new a dataset: one 4. Then go back to the original dataset
observations / bootstrap sample. and take another bootstrap sample.
Only variable is the estimated 5. Calculate the proportion of days with
proportion in a sample 5%+ loss and save that value, too.
I The SE is simply the standard 6. And so on, repeated many times.
deviation of those estimated
values in this new dataset.
05 Generalizing from data 35 / 42 Gábor Békés
Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

Stock market returns: The Bootstrap standard error

I 10,000 bootstrap samples with

2,519 observations
I The proportion of days with 5+
percent loss.
I Varied 0.1 percent to 1.2
percent. Mean=Median= 0.5
I Standard deviation across the
bootstrap samples = 0.14
I CI: the 95 percent CI is [0.22,
0.78].

05 Generalizing from data 36 / 42 Gábor Békés

Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

Stock market returns: The Bootstrap standard error

I This means that in the general pattern represented by the 11-year history of
returns in our data, we can be 95 percent confident that daily losses of more than
5 percent occur with a 0.22 to 0.78 percent chance.
I SE formula and bootstrap gave the same exact answer
I Under some conditions, this is what we expect
I Large enough sample size
I Observations independent

I ... (other we overlook now)

05 Generalizing from data 37 / 42 Gábor Békés

Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

External validity

I We discussed statistical inference: CI - uncertainty about the true value of the

statistic in the population / general pattern that our data represents.
I What is the population, or general pattern, we care about?
I How close is our data to this?
I External validity is the concept that captures the similarity of our data to the
population/general pattern we care about.
I High external validity: if our data is close to the population or the general pattern
we care about.
I External validity is as important as statistical inference. However, it is not a
statistical question.

05 Generalizing from data 38 / 42 Gábor Békés

Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

External validity

I The most important challenges to external validity may be collected in three

groups:

I Time: we have data on the past, but we care about the future
I Space: our data is on one country, but interested how a pattern would hold
elsewhere in the world
I Sub-groups: our data is on 25-30 year old people. Would a pattern hold on
younger / older people?

05 Generalizing from data 39 / 42 Gábor Békés

Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

External validity

I Daily 5%+ loss probability - 95 percent CI [0.2, 0.8] in our sample. This captures
uncertainty for samples like ours.
I If the future one year will be like the past 11 years in terms of the general pattern
that determines returns on our investment portfolio.
I However, external validity may not be high - not sure what the future holds.
I Our data: 2006-2016 dataset includes the financial crisis and great recession of
2008-2009. It does not include the dotcom boom and bust of 2000-2001. We have
no way to know which crisis is representative to future crises to come.
I Hence, the real CI is likely to be substantially wider.

05 Generalizing from data 40 / 42 Gábor Békés

Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

External validity in Big Data

I Big data: very large N

I Statistical inference not really important - CI becomes very narrow
I External validity remains as important

I 1.) Large sample DOES NOT mean representative sample

I 2.) Big data as result of actions - nature of things may change as people alter
behavior, outside conditions change

05 Generalizing from data 41 / 42 Gábor Békés

Inference CS: A1 Repeated samples CS: A2-A3 The CI Calculating the SE CS:A4 The bootstrap SE CS:A5 External validity Summary

Generalization - Summary

I Generalization is a key task - finding beyond the actual dataset.

I This process is made up of discussing statistical inference and external validity.
I Statistical inference generalizes from our dataset to the population using a variety
of statistical tools.
I External validity is the concept of discussing beyond the population for a general
pattern we care about; an important but typically somewhat speculative process.

05 Generalizing from data 42 / 42 Gábor Békés