4th Unit - Statistics

Correlation and Regression in Statistics

Correlation and regression are two important statistical techniques used to

understand the relationship between two variables. While they are closely related,
they have distinct purposes and applications.

Correlation:

● Measures the strength and direction of a linear relationship between two

quantitative variables.
● Quantified by a correlation coefficient (r), which ranges from -1 (perfect
negative correlation) to +1 (perfect positive correlation). A value of 0 indicates
no linear relationship.
● Does not imply causation.

Regression:

● Models the relationship between two variables as a mathematical equation.

● Used to predict the value of one variable (dependent variable) based on the
value of the other variable (independent variable).
● Provides insight into the direction and strength of the relationship.
● Can be used to make predictions about future values of the dependent
variable.

Here's a table summarizing the key differences between correlation and regression:
Choosing between correlation and regression:

The choice between using correlation or regression depends on your specific

research question and goals.

● Use correlation if you want to:

○ Measure the strength and direction of a linear relationship between two
quantitative variables.
○ Understand the general nature of the relationship between two
variables.
● Use regression if you want to:
○ Predict the value of one variable based on the value of another
variable.
○ Model the relationship between two variables as a mathematical
equation.
○ Make inferences about causation, but with caution.

Population and Sample

Population: In statistics, a population refers to the entire collection of individuals or

objects that we are interested in studying. It includes all the elements that meet the
criteria of our study.

Sample: A sample is a subset of the population that is chosen for study. We use the
data collected from the sample to make inferences about the entire population.

Relationship: The sample is drawn from the population and is used to represent the
characteristics of the population. Ideally, the sample should be representative of the
population so that the conclusions drawn from the sample can be accurately applied
to the population.

Example: Imagine you want to measure the average height of all adults in India. The
population in this case is all adults in India. It would be impossible to measure the
height of every adult, so you would need to draw a sample of adults and measure
their heights. You would then use the data from the sample to estimate the average
height of all adults in India.

Sampling Types
Sampling is the process of selecting a subset of individuals from a larger population
for the purpose of making inferences about the entire population. There are different
types of sampling, each with its own advantages and disadvantages.
1. Probability Sampling:

● Simple random sampling: Each individual in the population has an equal

chance of being selected. This is the most basic type of probability sampling
and is often used when the population is relatively small.
● Stratified random sampling: The population is divided into groups (called
strata) based on some characteristic, and then a random sample is drawn
from each stratum. This is used to ensure that the sample is representative of
the population in terms of the important characteristics.
● Cluster sampling: The population is divided into groups (called clusters), and
then a random sample of clusters is drawn. All individuals within the selected
clusters are included in the sample. This is used when it is impractical or
expensive to sample all individuals in the population.
● Systematic sampling: Every nth individual is selected from the population,
where n is the sampling interval. This is used when the population is ordered
in some way.

2. Non-probability Sampling:

● Convenience sampling: The sample is selected based on convenience,

such as using students in a statistics class as a sample of the population of all
students.
● Purposive sampling: The sample is selected based on the judgment of the
researcher, such as selecting individuals who are thought to be representative
of the population.
● Quota sampling: The population is divided into groups, and then a
non-random sample is drawn from each group until a certain quota is met.
This is used to ensure that the sample is representative of the population in
terms of the important characteristics.

Concepts of Sampling Distribution:

What is a Sampling Distribution?

A sampling distribution refers to the probability distribution of a statistic calculated

from repeated random samples of a given size drawn from a population. In other
words, it shows the possible values of a statistic and their corresponding probabilities
across all possible samples of a certain size.

Importance of Sampling Distribution:

Sampling distributions are crucial for inferential statistics because they allow us to
understand a single sample statistic in the context of the wider population. They help
in:

● Drawing conclusions about the population based on limited sample data.

● Estimating population parameters, such as mean and variance.
● Quantifying the uncertainty associated with sample statistics.
● Performing hypothesis testing to make inferences about the population.

Key Characteristics of a Sampling Distribution:

● Shape: The shape of the sampling distribution depends on the population

distribution, the statistic being calculated, and the sample size. Common
shapes include normal, skewed, and uniform.
● Center: The center of the sampling distribution, usually represented by the
mean, tends to be closer to the population parameter as the sample size
increases.
● Spread: The spread of the sampling distribution, represented by the standard
error, decreases as the sample size increases. This is a property known as
the Central Limit Theorem.
● Sampling Error: The difference between a sample statistic and its
corresponding population parameter is called sampling error. Sampling
distributions help quantify this error and estimate its range.

Types of Sampling Distributions:

There are different types of sampling distributions depending on the statistic being
calculated and the sample size. Here are three common examples:

● Sampling distribution of the mean: This is the most common type,

representing the distribution of means calculated from repeated samples of
the same size.
● Sampling distribution of the proportion: This shows the distribution of
proportions calculated from repeated samples of the same size, often used in
situations with categorical data.
● Sampling distribution of the difference between two means: This
compares the means of two different populations based on samples drawn
from each.

Applications of Sampling Distributions:

Sampling distributions are used in various fields, including:

 ● Statistics: Hypothesis testing, confidence intervals, estimation of population
parameters.
● Finance: Risk analysis, portfolio optimization, pricing of financial instruments.
● Engineering: Quality control, reliability testing, design optimization.
● Medicine: Clinical trials, drug development, disease diagnosis.

1. Standard Error:

The standard error (SE) is a measure of the variability in a statistic (e.g., mean,
proportion) due to random sampling. It tells us how much we can expect the statistic
to vary from sample to sample, drawn from the same population.

● Lower standard error indicates less variability and a more precise estimate of
the population parameter.
● Higher standard error indicates greater variability and less precision in the
estimate.

2. Significance Levels:

The significance level (α) is the probability of rejecting a true null hypothesis in a
hypothesis test. It represents the risk of making a Type I error (concluding a
difference exists when it doesn't).

● Commonly used significance levels are 0.05, 0.01, and 0.001.

● Lower significance levels indicate a stricter requirement for rejecting the null
hypothesis, meaning we need stronger evidence for a difference.

3. Confidence Limits:

A confidence interval (CI) is a range of values within which we are confident that the
true population parameter lies. It is calculated based on the sample statistic,
standard error, and chosen confidence level (1 - α).

● Confidence intervals provide a more nuanced interpretation of results

compared to just a single point estimate.
● Wider confidence intervals indicate greater uncertainty about the population
parameter, and vice versa.

Relationship between these concepts:

 ● Standard error: Used to calculate confidence intervals and plays a role in
hypothesis testing by determining the critical value for rejection.
● Significance level: Determines the width of the confidence interval. Lower
significance levels result in narrower intervals and vice versa.
● Confidence level: Directly related to the significance level (1 - α). Higher
confidence levels lead to wider confidence intervals and a lower risk of Type II
error (failing to reject a false null hypothesis).

In summary:

● Standard error measures the variability of a statistic due to sampling.

● Significance level indicates the risk of rejecting a true null hypothesis.
● Confidence intervals provide a range of plausible values for the population
parameter.

Hypothesis Testing and the Null Hypothesis

Hypothesis testing is a fundamental statistical concept used to draw conclusions

about a population based on sample data. It involves formulating two opposing
hypotheses:

1. Null Hypothesis (H0): There is no significant difference between the

observed data and what is expected.
2. Alternative Hypothesis (Ha): There is a significant difference between the
observed data and what is expected.

The null hypothesis is typically the default assumption, while the alternative
hypothesis represents the specific research question or claim being investigated.

The goal of hypothesis testing is to determine whether to reject the null hypothesis or
fail to reject it. This decision is based on a statistical test, which involves calculating
the probability of obtaining the observed data if the null hypothesis were true (known
as the p-value).

The decision-making process in hypothesis testing involves comparing a calculated

test statistic to a critical value based on a chosen significance level (α). Here's a
table summarizing the possible outcomes

Steps involved in Hypothesis Testing:

1. Formulate the null and alternative hypotheses: Clearly define what you are
trying to test and what would constitute evidence against your null hypothesis.
2. Collect data: Gather a representative sample from the population of interest.
3. Choose a statistical test: Select an appropriate test based on the type of
data and research question.
4. Calculate the test statistic and p-value: Use the collected data to calculate
the statistic and its corresponding p-value.
5. Make a decision: Compare the p-value to a predetermined significance level
(usually 0.05).
○ If the p-value is less than the significance level, reject the null
hypothesis.
○ If the p-value is greater than or equal to the significance level, fail to
reject the null hypothesis.

Key Points about the Null Hypothesis:

● The null hypothesis is always stated in the negative form.

● It represents the status quo or the default assumption.
● It is not necessarily the truth, but rather a starting point for the analysis.
● The burden of proof lies in rejecting the null hypothesis.

Examples of Null Hypotheses:

● There is no difference in the average heights of men and women.

● There is no relationship between exercise and heart disease.
● A new drug has no effect on pain relief.

Importance of Null Hypothesis Testing:

● It helps researchers draw objective conclusions about their data.

● It provides a framework for testing specific claims and research questions.
 ● It helps to control Type I and Type II errors.

Type I and Type II errors in Statistics

In statistics, hypothesis testing is a crucial technique for drawing conclusions about a

population based on data from a sample. However, this process is inherently
susceptible to two types of errors: type I errors and type II errors.

Type I error (false positive)

A type I error occurs when the null hypothesis (H0) is rejected even though it is
actually true. In simpler terms, it means wrongly concluding that there is a significant
effect or difference when there isn't. This can be likened to accusing someone of a
crime they didn't commit.

Example: A medical researcher conducts a study to test the effectiveness of a new

drug for lowering blood pressure. The researcher concludes that the drug is
effective, but in reality, the observed difference in blood pressure is simply due to
chance.

Consequences of a Type I error:

● Wasted resources: Implementing unnecessary interventions based on false

conclusions.
● Ethical concerns: In medical research, a type I error could lead to patients
receiving ineffective or harmful treatments.
● Damage to reputation: If the false conclusion is widely published, it can
damage the reputation of the researcher and the research institution.

Type II error (false negative)

A type II error occurs when the null hypothesis (H0) is not rejected even though it is
actually false. In simpler terms, it means failing to detect a real effect or difference.
This can be likened to acquitting someone guilty of a crime.

Example: A study is conducted to investigate the link between smoking and lung
cancer. The study fails to find a significant association, but in reality, smoking does
increase the risk of lung cancer.

Consequences of a Type II error:

 ● Missed opportunities: Failing to identify a real effect can impede progress in
scientific research and prevent the development of beneficial interventions.
● Harm to individuals: In medical research, a type II error could lead to
patients not receiving potentially life-saving treatments.
● Misallocation of resources: Resources might be directed towards
interventions that are not effective.

Minimizing errors

Researchers can take several steps to minimize the risk of making type I and type II
errors:

● Choosing an appropriate sample size: A larger sample size will provide

more accurate data and reduce the probability of both types of errors.
● Setting a suitable significance level: The significance level represents the
likelihood of rejecting the null hypothesis when it is actually true. A lower
significance level (e.g., 0.01) will reduce the risk of a type I error, but it will
also increase the risk of a type II error.
● Conducting a power analysis: A power analysis can help researchers
determine the appropriate sample size necessary to achieve a desired level of
power, which is the probability of correctly rejecting the null hypothesis when it
is false.
● Employing appropriate statistical tests: Choosing the correct statistical test
for the research question and data can prevent errors in analysis.

Understanding type I and type II errors is essential for interpreting the results of
statistical tests and drawing accurate conclusions about research findings. By being
aware of these potential errors and taking steps to minimize them, researchers can
ensure that their findings are reliable and valid.

T-Test, Chi-Square Test, and F-Test

These are three commonly used statistical tests in various fields, including research,
medicine, business, and social sciences. Each test has its unique purpose and
applicability depending on the type of data and research question.

1. T-Test:

● Used to compare the means of two independent groups or two related groups
(paired samples).
 ● Can be used for both parametric and non-parametric data.
● Parametric t-test assumes normally distributed data and small sample sizes
require special considerations.
● Non-parametric t-tests (e.g., Mann-Whitney U test) do not require normality
assumptions.
● Example: Comparing the average height of males and females in a
population.

2. Chi-Square Test:

● Used to determine whether there is a statistically significant association

between two categorical variables.
● Applicable for nominal and ordinal data.
● Helps assess if the observed frequencies of data points in different categories
differ significantly from what would be expected by chance alone.
● Example: Testing if there is a relationship between smoking habits and lung
cancer.

3. F-Test:

● Used to compare the variances of two or more groups.

● Often used in conjunction with ANOVA (analysis of variance) to compare the
means of more than two groups.
● Determines whether the observed differences in variances between groups
are statistically significant.
● Example: Comparing the variability of exam scores across different teaching
methods.

Here's a table summarizing the key differences between these tests:

Principles of Experimental Design

Experimental design is the process of planning a scientific experiment in a way that

will allow you to collect valid and reliable data. There are many different principles of
experimental design, but some of the most important ones include:

● Randomization: This means that you should randomly assign treatments to

experimental units. This is important because it helps to control for bias and
ensure that any differences you observe between groups are due to the
treatments themselves and not to other factors.
● Replication: This means that you should repeat your experiment multiple
times. This is important because it helps to reduce the impact of random error
and increase the accuracy of your results.
● Local control: This means that you should control for any factors that could
affect your results. This might involve things like using the same type of
materials for all of your experimental units, or taking measurements at the
same time of day.

In addition to these three main principles, there are a number of other important
considerations when designing an experiment. These include:

● The type of experiment you are conducting: There are many different
types of experiments, each with its own set of design considerations. For
example, a factorial experiment will require a different design than a simple
randomized experiment.
● The number of treatments you are comparing: The number of treatments
you are comparing will affect the sample size you need for your experiment.
● The type of data you are collecting: The type of data you are collecting will
affect the statistical analysis you need to use.

ANOVA: One-Way and Two-Way

ANOVA stands for Analysis of Variance. It is a statistical test used to compare the
means of two or more groups to determine if there is a statistically significant
difference between them.

There are two main types of ANOVA:

● One-way ANOVA: This type of ANOVA compares the means of three or more
groups when there is only one independent variable. It is used to answer
questions like:
 ○ Is there a significant difference in the average height of plants grown
with different types of fertilizer?
○ Do students perform better on exams when they are given more time to
study?
○ Is there a difference in the average income of people with different
levels of education?
● Two-way ANOVA: This type of ANOVA compares the means of three or more
groups when there are two independent variables. It is used to answer
questions like:
○ Is there a significant difference in the average yield of corn plants
grown with different types of fertilizer and under different watering
conditions?
○ Does the effectiveness of a new medication depend on the patient's
age and gender?
○ Do students perform better on exams when they are given longer study
time and when the teacher uses a different teaching method?

Here's a table summarizing the key differences between one-way and two-way
ANOVA:

Analogy:

Imagine you have a group of students and you want to compare their exam scores.
You could conduct a one-way ANOVA to see if the average score is different for
students who studied more than 10 hours compared to students who studied less
than 10 hours. However, you could also conduct a two-way ANOVA to see if the
average score is different for students who studied more than 10 hours and who
used a particular study method, compared to students who studied less than 10
hours and who used a different study method.

Here are some additional points to note about ANOVA:

● The assumptions of ANOVA include normality of the data and homogeneity of

variance. These assumptions should be checked before conducting an
ANOVA test.
● There are several different types of ANOVA tests, such as factorial ANOVA
and repeated measures ANOVA. These tests are used in more complex
situations.
● ANOVA is a powerful tool for comparing the means of multiple groups, but it is
important to interpret the results carefully and consider other factors that may
be affecting the data.