Errors in Hypothetical Testing Basic
Errors in Hypothetical Testing Basic
Statisticians follow a formal process to determine whether to reject a null hypothesis, based on sample
data. This process, called hypothesis testing, consists of four steps.
State the hypotheses. This involves stating the null and alternative hypotheses. The hypotheses
are stated in such a way that they are mutually exclusive. That is, if one is true, the other
must be false.
Formulate an analysis plan. The analysis plan describes how to use sample data to evaluate the
null hypothesis. The evaluation often focuses around a single test statistic.
Analyze sample data. Find the value of the test statistic (mean score, proportion, t-score, z-
score, etc.) described in the analysis plan.
Interpret results. Apply the decision rule described in the analysis plan. If the value of the test
statistic is unlikely, based on the null hypothesis, reject the null hypothesis.
Decision Errors
Type I error. A Type I error occurs when the researcher rejects a null hypothesis when it is
true. The probability of committing a Type I error is called the significance level. This
probability is also called alpha, and is often denoted by α.
Type II error. A Type II error occurs when the researcher fails to reject a null hypothesis that is
false. The probability of committing a Type II error is called Beta, and is often denoted by β.
The probability of not committing a Type II error is called the Power of the test.
When using probability to decide whether a statistical test provides evidence for or
against our predictions, there is always a chance of driving the wrong conclusions.
Even when choosing a probability level of 95%, there is always a 5% chance that one
rejects the null hypothesis when it was actually correct. This is called Type I error,
represented by the Greek letter .
It is possible to err in the opposite way if one fails to reject the null hypothesis when it
is, in fact, incorrect. This is called Type II error, represented by the Greek letter .
These two errors are represented in the following chart.
A related concept is power, which is the probability of rejecting the null hypothesis
when it is actually false. Power is simply 1 minus the Type II error rate, and is usually
expressed as 1-.
When choosing the probability level of a test, it is possible to control the risk of
committing a Type I error by choosing an appropriate .
This also affects Type II error, since they are are inversely related: as one increases,
the other decreases. To appreciate this in a diagram, follow this link:
Choice of a
There is little control on the risk of committing Type II error, because it also depends
on the actual difference being evaluated, which is usually unknown. The following
link leads to a diagram that illustrates how at a fixed value, the value changes
according to the actual distribution of the population:
Changes in b
The consequences of these different types of error are very different. For example, if
one tests for the significant presence of a pollutant, incorrectly deciding that a site is
polluted (Type I error) will cause a waste of resources and energy cleaning up a site
that does not need it. On the other hand, failure to determine presence of pollution
(Type II error) can lead to environmental deterioration or health problems in the
nearby community.