Unit 8: Type I and Type II errors and Power of the test

Unit 8: Type I and Type II errors and Power of the test

Objectives
Type-I & Type-II Error
P value and its applications
1. Define type-I & II errors
2. Explain power of the test
3. Define P-value and real its application in research
4. Interpret different p-values

Type-I Error (False Positive)

Definition:
A Type-I error occurs when the null hypothesis (H0H_0H0) is true, but we mistakenly reject it. This
means we conclude that there is an effect or difference when, in reality, there isn't.
Comparison: It is often referred to as a false positive because it's analogous to detecting a signal or
effect that doesn't exist.
Symbol: The probability of making a Type-I error is denoted by alpha (α), which is also called the
significance level of the test.
Significance Level: The significance level (α) represents the threshold for deciding when to reject the
null hypothesis.
For example:
If α=0.05, (5%), it means there is a 5% chance of rejecting H0 when it is actually true.
Real-world Example: In a medical test, a Type-I error could occur if a test incorrectly indicates that a
person has a disease (when they don't), leading to unnecessary treatment or further tests.

Example:
Medical Test Scenario
Let’s consider a medical test scenario where you are testing whether a new diagnostic tool is better
than a placebo.
Null Hypothesis (H0): The diagnostic tool has no effect (i.e., it performs no better than the placebo).
Alternative Hypothesis (HA): The diagnostic tool performs better than the placebo.
If you conduct the test and reject H0 (based on the data), this means you believe the diagnostic tool is
effective. However, a Type-I error would occur if H0 is actually true (the tool does no better than the
placebo), but you incorrectly reject H0 and conclude that the tool is effective.
False positive: The test indicates that the tool is effective when, in fact, it isn't.
Consequences: This might lead to unnecessary adoption of the tool, patients being treated based on
false results, or even resources being wasted on further studies and treatments that are not beneficial.

Visualizing Type-I Error

Imagine a scenario where you’re testing the effectiveness of a new drug. You can think of the testing
procedure as involving a decision rule that depends on the data you collect:
If the data (e.g., test statistic) falls beyond a certain critical value, you reject the null hypothesis.
If it falls within the range of expected values under the null hypothesis, you fail to reject H0.
Now, a Type-I error happens if:

The true state of nature is that the null hypothesis is correct (i.e., the drug has no effect).
But your data suggests otherwise, leading you to reject H0 and conclude that the drug has an effect
(even though it doesn’t).

Relationship Between Type-I and Significance Level (α)

The significance level (α) is the probability of making a Type-I error, and it is set by the researcher
before the test. This means that if you choose a significance level of 0.05, you are willing to accept a 5%
chance of making a Type-I error. More formally:

P(Type-I error)=α
Unit 8: Type I and Type II errors and Power of the test

Example: If α = 0.05, you are allowing a 5% risk of incorrectly rejecting the null hypothesis when it is
true.
Lower α (e.g., 0.01): More stringent, fewer Type-I errors, but increases the risk of a Type-II error (failing
to detect a true effect).
Higher α (e.g., 0.10): Less stringent, higher risk of Type-I error, but might reduce Type-II errors
(increasing power).

Managing Type-I Errors

To control the risk of Type-I errors, researchers do the following:
Set an appropriate α level: The choice of α depends on the context. For example, in clinical trials,
researchers often use a lower α (e.g., 0.01 or 0.05) to minimize the risk of false positives, because the
consequences of Type-I errors (false positives) can be severe, such as approving an ineffective drug.

Use a larger sample size: Larger samples generally provide more reliable data and reduce the variability
of test statistics, which in turn can help avoid making a Type-I error.

Replicate the study: Repeating experiments and ensuring that results are consistent across different
samples and studies can help confirm whether the effect (if any) is real or just due to random chance.

Adjust for multiple comparisons: If you perform multiple tests, the chances of making a Type-I error
across all tests increase. Researchers use correction methods (like the Bonferroni correction) to adjust
the significance level when performing multiple tests.

Practical Example of a Type-I Error in Research

Let’s say a researcher is testing the hypothesis that a new educational program improves student
performance. The hypotheses are:
H0: The program does not improve student performance.
HA: The program improves student performance.
After analyzing the data, the researcher finds that the p-value is less than 0.05 and rejects H0,
concluding that the program is effective. However, a Type-I error would occur if, in reality, the program
has no effect (i.e., the null hypothesis is true), but the statistical test led the researcher to mistakenly
reject it.
In this case, the Type-I error could lead to:
Unjustified investment in the educational program.
Misleading conclusions that the program is effective when it is not.

Type-II Error (False Negative)

Definition
A Type-II error (also known as an error of the second kind) occurs when a false null hypothesis is not
rejected. In other words, it happens when you fail to detect an effect or difference that actually exists.
This is often referred to as a false negative, as you mistakenly conclude that there is no effect or no
difference when, in fact, one does exist.

Key Characteristics:
Null hypothesis (H0) is false, but we fail to reject it.
In practical terms, this means accepting the null hypothesis when it should have been rejected.
The probability of committing a Type-II error is denoted by beta (β).

Example:
Consider a clinical trial where you are testing a new drug:
Null hypothesis (H0): The new drug has no effect (it is the same as the placebo).
Alternative hypothesis (HA): The new drug has an effect (it is different from the placebo).
Unit 8: Type I and Type II errors and Power of the test

If the new drug does have an effect, but your test fails to detect it and you fail to reject the null
hypothesis (incorrectly concluding that the drug has no effect), this would be a Type-II error.

Impact:
Type-II errors are significant because they can lead to missed opportunities to detect real effects,
which can result in incorrect conclusions and poor decision-making.

Power of a Test
The power of a statistical test is a critical concept in hypothesis testing. It is defined as the probability
of correctly rejecting the null hypothesis when it is false (i.e., detecting a true effect when it exists). In
other words, power reflects the ability of a test to avoid Type-II errors (false negatives).
Formula for Power
The power of a test is defined as:
Power = 1− β

Where:
β is the probability of a Type-II error (failing to reject the null hypothesis when it is false).
Power represents the probability of rejecting the null hypothesis correctly when it is false, i.e., correctly
identifying a true effect.
Thus, the power of the test is 1 minus the probability of making a Type-II error.

Interpretation of Power
Power = 0.80 (80%) means there is an 80% chance of correctly rejecting the null hypothesis when it is
false (detecting a true effect).
Power = 0.90 (90%) means there is a 90% chance of correctly rejecting the null hypothesis when it is
false.

Why Power is Important

The power of a test is an important factor because it helps you understand how likely your test is to
detect an effect, if one truly exists. In research, we aim for tests with high power to minimize the
chance of making Type-II errors.
Low power means you might fail to detect a real effect (Type-II error).
High power means you are more likely to detect a true effect and avoid false negatives.

Factors that Influence Power

Several factors influence the power of a statistical test:

Sample Size (n): Larger sample sizes generally increase the power of a test. A larger sample provides
more reliable data, making it easier to detect a true effect if it exists.

Effect Size: The larger the true effect (i.e., the larger the difference between the null hypothesis and the
alternative hypothesis), the greater the power. A larger effect is easier to detect.

Significance Level (α): The power of a test is also related to the chosen significance level (α). By
increasing α (e.g., from 0.01 to 0.05), you increase the power of the test (since it becomes easier to
reject H0), but you also increase the risk of a Type-I error (false positive).

Variability of the Data: Less variability (or noise) in the data leads to higher power. High variability
makes it harder to detect a true effect.

Common Power Values

Power = 0.80: This is a commonly desired power level in many fields of research. It means there is an
80% chance of detecting a true effect, and a 20% chance of making a Type-II error.
Unit 8: Type I and Type II errors and Power of the test

Power = 0.90: Some fields, particularly those involving high-stakes decisions (like medical research), aim
for even higher power, such as 90%. This means there is a 90% chance of detecting an effect if one truly
exists.

Example of Power in Action

Let’s consider a clinical trial testing the effectiveness of a new drug:
Null Hypothesis (H0): The new drug has no effect.
Alternative Hypothesis (HA): The new drug has a positive effect.
If the power of the test is 80%, this means that:
There is an 80% probability that the test will correctly reject the null hypothesis if the drug actually has
an effect.
There is a 20% chance of failing to reject the null hypothesis, even though the drug does have a positive
effect (i.e., making a Type-II error).

Power Analysis
Before conducting a study, researchers often perform a power analysis to determine the sample size
needed to achieve a desired power level (typically 0.80 or 0.90). This helps ensure that the study has
enough power to detect a meaningful effect, if one exists.
Unit 8: Type I and Type II errors and Power of the test

Comparison between Type-I and Type-II errors in table format:

Aspect Type-I Error (False Positive) Type-II Error (False Negative)

Occurs when the null hypothesis is Occurs when the null hypothesis is
true, but it is rejected. In other false, but we fail to reject it. In other
Definition words, we incorrectly conclude words, we incorrectly conclude that
that there is an effect or there is no effect or relationship when
relationship when there is none. there actually is one.
False positive (incorrectly finding a False negative (failing to find a
Also Known As
relationship or effect) relationship or effect)
Denoted by alpha (α\alphaα), the Denoted by beta (β\betaβ), the
Notation
significance level of the test. probability of a Type-II error.
The probability of a Type-I error is
Probability of The probability of a Type-II error is
equal to the significance level (α\
Occurrence equal to beta (β\betaβ).
alphaα) of the test.
Rejecting a true null hypothesis. Failing to reject a false null
Concluding that a treatment or hypothesis. Concluding that a
Error Made
effect exists when it actually treatment or effect does not exist
doesn’t. when it actually does.
Consequences - Can lead to false claims of - Can lead to missing a true effect.
Unit 8: Type I and Type II errors and Power of the test

Aspect Type-I Error (False Positive) Type-II Error (False Negative)

discovery.
- In research, may result in failing to
- In research, may result in falsely
detect a real effect that could have
concluding that there is an effect
practical importance.
when in reality there is none.
A test showing that a new drug is A test failing to show that a new drug
effective when it is actually not is effective when it actually is (e.g.,
Example
(e.g., concluding a drug works concluding a drug doesn’t work when
when it doesn’t). it does).
Type-I errors can be particularly Type-II errors can be problematic in
problematic in areas like medical contexts where failing to detect a true
Impact research, where false positives effect may result in missed
might lead to the approval of opportunities (e.g., overlooking a life-
ineffective treatments. saving treatment).
The probability of a Type-II error
The probability of a Type-I error is
Control via depends on factors like sample size
controlled by setting a significance
Significance Level and the effect size. Increasing the
level (α\alphaα), which is typically
(α\alphaα) sample size or effect size reduces the
set at 0.05 (5%).
probability of a Type-II error.
Reducing α\alphaα (e.g., using a Increasing sample size or choosing a
stricter significance level) lowers larger effect size can reduce the risk of
Risk Management
the risk of Type-I errors, but Type-II errors, but may increase the
increases the risk of Type-II errors. risk of Type-I errors.

P value
The area that falls in the tail beyond the value of the test statistic. A p value tells you the chance of
getting a statistics as extreme or more extreme that the one calculated for the sample.

-Z cal
Unit 8: Type I and Type II errors and Power of the test

For left tail test

Z cal
For right tail test

Z cal
-Z cal

For a two tail hypothesis

Unit 8: Type I and Type II errors and Power of the test

Test Decisions with P-Values

In hypothesis testing, the p-value helps us decide whether to reject or fail to reject the null
hypothesis (H0) based on the evidence provided by the sample data. The p-value measures the
probability of obtaining test results at least as extreme as the results actually observed, under the
assumption that the null hypothesis is true.

If p value ≤ α,reject the null hypothesis

If p value> α,fail to reject the null hypothesis

Decision Rule for Hypothesis Testing with P-Values

Condition Action Interpretation

p-value ≤ α (where α There is sufficient evidence to support the alternative
Reject the null
is the significance hypothesis. The observed data is unlikely to have
hypothesis
level) occurred if the null hypothesis were true.
There is not enough evidence to support the
Fail to reject the
p-value > α alternative hypothesis. The observed data could
null hypothesis
reasonably occur if the null hypothesis were true.

Example : Clinical Trial for a Drug

Hypotheses:

Null hypothesis (H0): The new drug has no effect on patients' recovery time.
Alternative hypothesis (HA): The new drug reduces recovery time.

Test Decision:
Significance level (α): 0.05

p-value: 0.03 (calculated from the test)

Since p-value (0.03) ≤ α (0.05), we reject the null hypothesis, meaning we have enough evidence to
conclude that the drug does have an effect on recovery time.

Understanding p value
The p value can also be thought of as the probability of obtaining a result as extreme as or more
extreme than the actual sample value obtained given that the null hypothesis is true.

Example 1
A topic of recent clinical interest is the possibility of using drugs to reduce infarct size in those patients
who have had a MI within the past 24 hrs. Suppose we know that in untreated patients the mean
infarct size is 25 with a standard deviation of 10. Furthermore, in 50 patients treated with the drug, the
mean infarct size is 16. Is the drug effective in reducing infarct size?
Unit 8: Type I and Type II errors and Power of the test

Solution:

Clinical Trial on Drug Effectiveness in Reducing Infarct Size

In this example, we are testing whether a drug reduces infarct size in patients who have had a
myocardial infarction (MI) within the last 24 hours.

Given Data:
Mean infarct size in untreated patients (μ0) = 25 (population mean)
Standard deviation (σ) = 10
Sample size (n) = 50 (treated patients)
Sample mean (Xˉ) = 16 (treated patients)

We need to determine if the drug is effective in reducing infarct size, which means testing if the mean
infarct size for treated patients is less than 25.

Step 1: State the Hypotheses

Null hypothesis (H0): μ≥25 (The mean infarct size for treated patients is greater than or equal to 25)
Alternative hypothesis (HA): μ<25 (The mean infarct size for treated patients is less than 25, indicating
the drug has reduced infarct size)

Step 2: Choose Significance Level (α)

The significance level is α=0.05\alpha = (5%).

Step 3: Test Statistic Calculation

We are using a Z-test because the population standard deviation is known, and the sample size is
sufficiently large (n=50).
The formula for the Z-test statistic is:
Z = Xˉ−μ0/ σ/√ n

Where:
Xˉ=16 (sample mean)
μ0=25 (population mean)
σ=10 (standard deviation)
n=50 (sample size)
Substitute the given values into the formula:

Z = 16−26/ 10 /√ 50

First, calculate the denominator: 10 /√ 50

10/7.071= 1.414
Now calculate the Z-score:
Z = −9/1.414= −6.36

Step 4: Decision Rule

We compare the calculated Z-value to the critical value for a one-tailed test at α=0.05. For a left-
tailed test (since we are testing if the mean is less than 25), the critical value from the Z-distribution is:
Zα=−1.645

Step 5: Conclusion
Calculated Z-value = -6.36
Critical Z-value = -1.645
Since -6.36 < -1.645, we reject the null hypothesis.
There is enough evidence at the 5% significance level to conclude that the drug is effective in reducing
infarct size. The mean infarct size in treated patients is significantly smaller than the mean infarct size in
untreated patients, suggesting that the drug has a statistically significant effect in reducing infarct size.
Unit 8: Type I and Type II errors and Power of the test

So;
 Null Hypothesis: H0:μ≥25
 Alternative Hypothesis: HA:μ<25
 Significance level: α=0.05
 Z-value: -6.36
 Critical Z-value: -1.645
 Conclusion: Reject H0, indicating the drug is effective in reducing infarct size.


Applications of the P-Value in Hypothesis Testing

The p-value plays a crucial role in various applications across different fields such as medicine, social
sciences, economics, business, and engineering.

1. Medical Research / Clinical Trials

Example: Testing the effectiveness of a new drug.
Null Hypothesis (H0): The new drug has no effect on patient recovery.
Alternative Hypothesis (HA): The new drug reduces recovery time.

In this case, researchers want to determine if the new drug has a statistically significant effect on
patient recovery time. If the p-value is low (e.g., p-value=0.02), it suggests the observed effect is
unlikely to have occurred by chance, and they would reject H0, concluding the drug is effective.

2. Social Sciences / Psychology

Example: Testing a new teaching method.
Null Hypothesis (H0): The new teaching method does not improve student performance.
Alternative Hypothesis (HA): The new teaching method improves student performance.
Unit 8: Type I and Type II errors and Power of the test

Researchers in psychology or education might conduct experiments to evaluate if a new educational

intervention improves students' scores. If the p-value is low (e.g., p-value=0.03, the researchers would
reject H0 and conclude that the teaching method has a significant effect.

3. Business / Marketing Research

Example: Testing the effectiveness of a new advertising campaign.
Null Hypothesis (H0): The new advertising campaign has no effect on sales.
Alternative Hypothesis (HA): The new advertising campaign increases sales.

Marketers may use hypothesis testing to evaluate if a new advertising campaign has had an impact
on product sales. If the p-value is below 0.05, it suggests that the observed increase in sales is
statistically significant, and the company may decide to continue with or scale up the campaign.

4. Quality Control / Engineering

Example: Testing whether a new manufacturing process improves product quality.
Null Hypothesis (H0): The new process does not improve product quality.
Alternative Hypothesis (HA): The new process improves product quality.

Engineers might test a new process or change to improve manufacturing efficiency or quality. If the p-
value from the testing is low (e.g., p-value=0.01, this suggests the process change led to significant
improvements, prompting adoption.

5. Environmental Science / Ecology

Example: Testing whether a new environmental regulation reduces pollution levels.
Null Hypothesis (H0): The new regulation has no effect on pollution levels.
Alternative Hypothesis (HA): The new regulation reduces pollution levels.

In environmental studies, researchers test the effects of new regulations, conservation strategies, or
pollution-control measures. A small p-value (e.g., p-value=0.03 might suggest that the regulation
significantly reduces pollution levels, supporting its implementation.

Implications of P-Value in Biostatistics

Context Implications
A p-value helps determine if a treatment is statistically effective, but must
Clinical Trials
be interpreted alongside clinical significance and practical considerations.
P-values help assess associations between exposures and outcomes, but
Epidemiology
must account for confounding factors and potential biases.
Multiple The p-value can increase the risk of Type I errors when multiple tests are
Comparisons conducted, requiring adjustments (e.g., Bonferroni correction).
P-values alone do not measure the size or precision of an effect; they must
Effect Size and CI be considered with effect size and confidence intervals for a complete
interpretation.
Type I and Type II The p-value is directly related to the probability of Type I errors, and must
Errors be balanced with Type II error (power of the test).

Limitations of P-Value
While p-values are powerful tools in statistical testing, they do have limitations and can sometimes
lead to misinterpretations:
Unit 8: Type I and Type II errors and Power of the test

 P-value is not the probability that H0 is true:

 The p-value only tells us how likely the observed data is, assuming the null hypothesis is true, but it
does not tell us the probability that the null hypothesis itself is true.

 P-value is not a measure of effect size:

 A small p-value might indicate statistical significance, but it does not provide information about the
magnitude or importance of the effect. A large effect can have a small p-value with a large sample
size, but the effect might not be practically significant.

 Dependence on Sample Size:

 The p-value can be heavily influenced by the sample size. For very large samples, even small
differences between groups can result in very small p-values, even though those differences may
not be practically meaningful.

 Multiple Comparisons:
 When multiple hypotheses are tested, the chance of finding a significant result by chance increases.
This is known as the multiple testing problem.