𝛂 and critical values,

Hypothesis Testing, Errors,

tests, ANOVA

- Dr.Nikita Mishra
nikita.mishra@spit.ac.i
Todays Specials

● Hypothesis Testing
● Alpha and Critical Values
● Errors in Hypothesis Testing
● Independent and dependent t-tests

● Chi-Square Tests
● Goodness of Fit test
● Test of Independence
● Anova - one-way ANOVA, two-way ANOVA
Hypothesis Testing

● One of the most important concepts in analytics.

● Hypothesis → claim made by a
person/organization.
● The claim is usually about population parameters
such as mean or proportion and we seek evidence
from a sample for the support of the claim.
● Example:
The average salary of analytics experts is at least
USD 100,000. Hypothesis testing is a process used
for either rejecting or retaining a null hypothesis.
Hypothesis Testing: Objective

● Either reject or retain a null hypothesis

● In many cases, for example, in regression models,

one would like to reject the null hypothesis to
establish statistically significant relationship
between the dependent and the independent
variables

● However, in goodness of fit tests, that are used for

checking whether the data follows a specific
Hypothesis Testing: Steps
1. Describe the Hypothesis in words

2. Define Null and Alternative Hypothesis

3. Identify the test statistic to validate the hypothesis

4. Decide Decision Criteria- Significance Value (𝜶)

5. Calculate p-value

6. Reject/Retain the null hypothesis based on p-value

&𝜶
Describing Hypothesis
Defining Null & Alternative Hypothesis
● Null hypothesis → H0
○ There is no relation between different groups wrt
the value of a population parameter
○ Assume it’s true and retain it till strong evidence is
found against it

● Alternative hypothesis → HA or H1
○ Complement of H0
○ The claim that the researcher believes in
Identifying Test Statistic
● Test statistic: standardized difference between estimated sample
parameter & the hypothesis value.
● Purpose: used to calculate the p-value, supporting evidence for the
null hypothesis.
● A large standardized distance indicates a lower p-value, suggesting
stronger evidence against the null hypothesis.
● Example: H0: μ ≤ 100,000 vs. X (sample estimate) = 110,000.
● Std distance calculation: (110,000 - 100,000) / 5000 = 2 SD’s
● p-value = P(Observing test statistics value | null hypothesis is true)
● A smaller p-value implies stronger evidence against the null
hypothesis.
● In this case, a p-value of 0.02275 supports the statement in the null
hypothesis with a certain level of significance.
Decision Criteria - Significance Value
● Significance level (𝛂) - Criteria used for taking the decision regarding the
null hypothesis. 𝛂 = P(Rejecting a null hypothesis | null hypothesis is
true)

● 𝛂 → maximum threshold for p-value

● Usually 𝛂 = 0.05. Other values: 0.1, 0.02, 0.01

probability = 𝛂
● Critical value - value of statistic in the distribution for which the

● Right-tailed test: if calculated statistic value > critical value → reject

H0

● Left-tailed test: if calculated statistic value < critical value → reject

 One-Tailed Test: Right-sided

Claim: Salary of machine learning experts on average is at least

US $100,000

H0 : 𝛍m ≤ 100,000
HA : 𝛍m > 100,000

Right-tailed
hypothesis tests
rejection region
 One-Tailed Test: Left-sided

Claim: Avg wait time at the London Heathrow airport security

check < 30 mins

H0 : 𝛍w ≥ 30
HA : 𝛍w < 30

Left-tailed
hypothesis tests
rejection region
 Two-Tailed Test

Claim: Avg annual salaries of male & female MBA students are different at the
time of graduation

H0 : 𝛍m = 𝛍f
HA : 𝛍m ≠𝛍f

Two-tailed
hypothesis tests
rejection region
Errors: Type I & Type II
Errors: Type I & Type II

● Type I error → False Positive; Type II error → False Negative

● Type I Error = 𝛂 = P(Rejecting null hypothesis | H0 is true)

● Type II Error = 𝝱 = P(Retain null hypothesis | H0 is false)

● Power of Hypothesis test = 1 - 𝝱 = 1 − P(Retain null hypothesis | H 0 is

false)
= P(Reject
null hypothesis|H0 is false)
Z-Test: For 𝝻 with known variance
● Z-test is used when a claim (hypothesis) is made about the
population parameter such as population mean or proportion when
population variance is known.

● Since the hypothesis test is carried out with just one sample, this
test is also known as one-sample Z-test

● As per CLT for sampling distribution of mean:

sampling distribution of mean from an IID population for large
sample follows a normal distribution with mean 𝝻 and standard
deviation 𝞂/√n
 Example 1

An agency based out of Bangalore claimed that the

average monthly disposable income of families living
in Bangalore is greater than INR 4200 with a standard
deviation of INR 3200. From a random sample of
40,000 families, the average disposable income was
estimated as INR 4250. Assume that the population

appropriate hypothesis test at 95% confidence level (𝛂 =

standard deviation is INR 3200. Conduct an

0.05) to check the validity of the claim by the agency

Example 1: Solution
Claim: Average disposable income is more than INR 4200
Let 𝛍 and 𝞂 denote the mean and standard deviation in the population.

H0: 𝛍 ≤ 4200
HA: 𝛍 > 4200

Since we know the population standard deviation, we can use the Z-test. The
corresponding Z-statistic is given by

Z = (4250 -4200)/ 3200 /√40000 = 3.125

● Z-critical value at 𝛂 = 0.05 for right-tailed test is approximately 1.64

● Reject H0
Z-statistic measures the standardized difference between estimated value of mean and the hypothesis value of
mean. Z = 3.125 implies that the sample mean is at 3.125 standard deviations away from the hypothesized
population mean given that the null hypothesis is true
Example 2

A passport office claims that the passport applications are

processed within 30 days of submitting the application form
and all necessary documents. Table 6.6 shows processing time
of 40 passport applicants. The population standard deviation
of the processing time is 12.5 days. Conduct a hypothesis test
at significance level a = 0.05 to verify the claim made by the
passport office.
Example 3

According to the company IQ Research, the average Intelligence

Quotient (IQ) of Indians is 82 derived based on a research carried
out by Professor Richard Lynn, a British Professor of Psychology,
using data collected from 2002 to 2006 (Source: IQ Research1). The
population standard deviation of IQ is estimated as 11.03. Based
on a sample of 100 people from India, the sample IQ was estimated
as 84.

(a) Conduct an appropriate hypothesis test at 𝛂 = 0.05 to validate the

claim of IQ Research (that average IQ of Indians is 82).

(b) Ministry of education believes that the IQ is more than 82. If the actual
IQ (population mean) of Indians is 86, calculate the Type II error and the
power of hypothesis test.
Power of Test: 1 - 𝛃
 Z-Test for Proportion

CLT of proportions:
the sampling distribution of proportions p∧ for a large sample follows an approximate
normal distribution with mean p (the population proportion) and standard deviation
 Example

According to a study exactly 12% of gift cards purchased from e-

commerce portals are never used. The manager of an e-commerce
company wanted to test whether this claim is true. She collected
data of 250 gift card purchases and found that 22 gift cards were
not used till its expiry date.
(a) Conduct an appropriate hypothesis test at 5% significance to check whether the claim that
exactly 12% gift cards are never used is true or not.

(b) Calculate the 95% confidence interval for the proportion of gift cards that are not used.
Solution
Solution
 t-Test: Population mean under
unknown population variance
We use the fact that a sampling distribution of a sample from a population
that follows normal distribution with unknown variance follows a t-distribution
with (n − 1) degrees of freedom. In cases where population variance is
unknown, we will have to
estimate the variance using the sample itself.

Let S be the standard deviation estimated from the sample of size n.

Then the statistic will follow a t-distribution with (n − 1) degrees of

freedom if the
sample is drawn from a population that follows a normal distribution
Example

Aravind Productions (AP) is a newly formed movie production house

based out of Mumbai, India. AP was interested in understanding the
production cost required for producing a Bollywood movie. The industry
believes that the production house will require at least INR 500
million (50 crore) on average. It is assumed that the Bollywood
movie production cost follows a normal distribution. Production cost of

Conduct an appropriate hypothesis test at 𝛂 = 0.05 to check whether

40 Bollywood movies in millions of rupees are shown in Table 6.7.

the belief about average production cost is correct.

Example: Solution

It is given that the production cost of Bollywood movies follows a

normal distribution; however, the standard deviation of the population
is not known and we need to estimate the standard deviation value
from the sample. Thus, we have to use the t-test for testing the
hypothesis. From the sample data in Table 6.7 we get the following
values:

The null and alternate hypothesis Corresponding test-statistic

are:
Example: Solution (contd)

𝛂 = 0.05 under
right-tailed test, tcritical = 1.6848 [in Excel TINV(2 𝛂, df) will return right-tailed critical
Note that this is a one-tailed test (right-tailed) and the critical t-value at

value at significance of 𝛂, in this example 𝛂 = 0.05, the corresponding critical t-value

using Excel function is TINV(0.1, 39) = 1.6848, that is the critical value is 1.6848]. Since
t-statistic value is less than the critical t-value, we retain the null hypothesis. The t-
statistic value and critical value for the t-test are shown in Fig 6.10.
Paired sample t-Test

In a paired t-test, the data related to the parameter is captured twice from the
same subject, once before the intervention and once after intervention.
Alternatively, the paired t-test can be used for comparing two different
interventions such as two different promotion strategies applied on the same
subject (price discount versus bundling of items).

Assume that the mean difference in the estimated parameter value before and
after the treatment is D, and the corresponding standard deviation of difference is
Sd. Let 𝝻d be the hypothesized mean difference. Then the statistic defined in Eq.
below follows a t-distribution with (n − 1) degrees of freedom.
 Example

A researcher believes that people drink more coffee on Mondays than other days of the week.
Based on a sample of 50 coffee drinkers, the mean difference was estimated as 14 ml

test at 𝛂 = 0.1 to check the claim that people drink on average 10 ml more coffee on
and the corresponding standard deviation was 8.5 ml. Conduct an appropriate hypothesis

Mondays compared to other days of the week.

Example: Solution
 Two-sample Z- and t-Test

Used for comparing parameters of two different populations. Three scenarios:

1. Population standard deviations are known

→ Two-sample Z-test

2. Population std deviations are unknown but believed to be equal

→ Two-sample t-Test

3. Population std deviations are unknown and unequal

→ Two-sample t-Test with unequal variance
 Two-sample Z-Test

1. The sample sizes (say n1 and n2) of two samples drawn from two

deviations 𝛔1 and 𝛔2 are known.

populations are large (say at least 30) and the corresponding standard

corresponding standard deviations 𝛔1 and 𝛔2 known.

2. The samples are drawn from two normally distributed populations with
 Two-sample Z-Test: Problem
The Dean of St Peter School of Management Education (SPSME) believes that
the
graduating students with specialization in Marketing earn at least INR 5000
more
per month than the students with specialization in Operations Management.
To
verify his belief, the Dean collected a sample data from his graduating

given in the table. Conduct an appropriate hypothesis test at 𝛂 = 0.05 to

students,

check
whether the difference in monthly salary is at least 5000 more for students
with
marketing specialization compared to operations specialization. Assume that
the
salary of students with marketing specialization and operations specialization
follow normal distribution.
 Two-sample t-Test (equal variance)

1. Standard deviations of the populations are unknown, and require

estimation from the samples drawn from these two populations

2. We assume that the standard deviation of two populations are equal (but
unknown)

Then the sampling distribution of the difference in estimated means (X1 - X2 ) follows a t-
distribution with (n1 + n2 – 2) degrees of freedom with mean (𝝻1 – 𝝻2) and SD
 Two-sample t-Test (EV): Problem

A company makes a claim that children (in the age group between 7 and 12)
who drink their health drink will grow taller than the children who do not drink
that
health drink. Data in Table 6.10 shows average increase in height over one-
year
period from two groups: one drinking the health drink and the other not

the health drink. At 𝛂 = 0.05, test whether the increase in height for the
drinking

children who
drink the health drink is at least 1.2 cm.
 Two-sample t-Test (Unequal
variance)
1. Standard deviations of the populations are unknown, and require
estimation from the samples drawn from these two populations

2. The standard deviation of two populations are unequal (and unknown)

Then the sampling distribution of the difference in estimated means (X1 - X2 ) follows a t-
distribution with (n1 + n2 – 2) degrees of freedom with mean (𝝻1 – 𝝻2) and SD

df → Corresponding degrees of
freedom
Two-sample t-Test (UV): Problem
Two-sample Z-Test for proportions
Problem
Effect Size: Cohen’s D

● Effect size → measure of magnitude or strength of

relationship between 2 or more groups in a
population

● 𝞂 → pooled std deviation

● 0.25 → small (proportions of explained variance of
1%)
0.5 → medium (proportions of explained variance of
5.9%)
Equality of population Variances

● If S2 → variance estimated from a sample of size n

from a population that follows a normal distribution,
then

OR

Test statistic for testing equality of

variances
Problem
 Non-Parametric Tests: 𝛘2 tests

Used when One or more of the following exists:

● The test is not about the population parameter such
as mean and std deviation

● The method does not require assumptions about

population distribution (such as population follows
normal distribution).
𝛘2 Goodness of Fit tests

● H0: There is no statistically significant difference

between the observed frequencies and the expected
frequencies from a hypothesized distribution

● HA: There is a statistically significant difference

between the observed frequencies and the expected
frequencies from a hypothesized distribution.
𝛘2 Goodness of Fit tests

● Z → Std normal distribution

● Z2 → follows a 𝛘2 distribution with 1 dof

● For k-variables, 𝛘2 distribution with k-dof is given as:

 𝛘2 Goodness of Fit tests

For binomial random variable with parameter p

(probability of success) and number of trials n, the
standardized RV follows a standard normal distribution
(CLT for proportions)
𝛘2 Goodness of Fit tests
Problem
 𝛘2 Goodness of Fit tests: Intervals

Mann & Wald (1942) suggested the following equation

to calculate the optimal number of bins for conducting
a chi-square goodness of fit test:

K → Number of bins (intervals)

N → Sample size
C → critical value under std normal distribution for given significance
𝛘2 Test of Independence

● Tests whether two (or more) groups are statistically

independent or not
● For example, assume that a telecom company is
interested in checking whether or not the customer
churn depends on the customer segment.
Table above is often called as the contingency table.
1. basic concept of independent events. From basic theory of
probability, we know that if two events A and B are
independent then P(A ∩ B) = P(A) P(B).

2. Let Oij = Observed number of cases in customer segment i (i

= 1, 2, 3) and classification j (1 = churned and 2 = retained)
3. Let Eij = Expected number of cases in customer segment i (i
= 1, 2 , 3) and classification j (1 = churned and 2 = retained)
𝛘2 Test of Independence
 𝛘2 Test of Independence

The chi-square test statistic for test of independence is

same as the chi-square statistic for goodness of fit test.
 𝛘2 Test of Independence
For our churn-retain example:

The corresponding degrees of freedom is (r − 1) × (c − 1), where r is the number of

rows and c is
the number of columns in the contingency table.
Degree of freedom
𝛘2 Test of Independence
 Problem 2
A college administrator wants to know if students' participation in
extracurricular activities is independent of their year of study. A random
sample of 300 students is taken, and the data is tabulated below.
● Expected Frequency & Chi-Square Calculation
Conclusion: Participation in extracurricular activities depends on the year of study.
 ANOVA: Introduction
● The objective of ANOVA is to check simultaneously
whether population mean from more than 2 populations
are different

● Consider a retail store which would like to study the

impact of different levels of price discounts (factor) on
sales (outcome variable) of a specific product or brand.
Price discount can range from 0% to 100%
(theoretically). For easier understanding, assume that
the levels of discounts are 0%, 10%, and 20%.
 Multiple t-tests
For more than two levels of discounts, compare the
population parameters two at a time (two discount values)

1. Test between 0% and 10%

2. Test between 0% and 20%
3. Test between 10% and 20%

when we test the hypothesis simultaneously, the Type I &

Type II errors will not be same if we conduct the 3 different
tests above
 Multiple t-tests
Assume that the mean sale (population mean) at 0%, 10%,
and 20% discount is 𝝻0, 𝝻10, and 𝝻20, respectively. Consider
the following:

1. P(A) = P(Retain H0 in test A|H0 in test A is true)

2. P(B) = P(Retain H0 in test B|H0 in test B is true)
3. P(C) = P(Retain H0 in test C|H0 in test C is true)

Note that values of P(A) = P(B) = P(C) = 1 – 𝛂 = 1 – 0.05 =

0.95
 Multiple t-tests
The conditional probability of simultaneously retaining all 3
null hypotheses when they are true is P(A ∩ B ∩ C) = 0.8573

H0: 𝝻0 = 𝝻10 = 𝝻20

significance or Type I error is not 𝛂-value, but much higher

If we retain H0 based on the 3 individual t-tests, then the

than 𝛂

For this reason, we use ANOVA whenever we need to

compare 3 or more groups for population parameter values
 One-way ANOVA
This applies when:

● We would like to study the impact of a single treatment

(factor) at different levels on a continuous response
variable

● In each group, the population response variable follows a

normal distribution and the sample subjects are chosen
using random sampling

● variability in the response variable values within different

groups is same
 Setting up ANOVA

Different values of mean (𝝻1, 𝝻2, and 𝝻3) imply statistically

significant impact of factor levels on the response variable
 Setting up ANOVA (contd)
To arrive at the statistic we calculate the following:

● Sum of Squares of Total Variation (SST)

● Sum of Squares of Between (SSB) Group Variation

● Sum of Squares of Within (SSW) Group Variation

 Cochran’s Theorem
If Y1, Y2, …, Yn are drawn from a normal distribution with
mean 𝝻 and standard deviation 𝞂 and SST is decomposed
into k sum of squares (SSr) with degrees of freedom dfr, then
the ratio (SSr/𝞂2) are independent χ2 variables with dfr
degrees of freedom if
 F-Test
If H0 is true, then there will be no difference in the mean
values which will result in no difference between MSB and
MSW

Following Cochran’s theorem MSB/MSW is a ratio of two chi-

square variate which is an F-distribution
Two Way ANNOVA
 Similarly calculate for strength and Yoga for
male and Female

3.56
ANNOVA BASED PROBLEMS
Problem 1
Problem
Grand Mean (μ) = Total Sum / Total Count = (960 + 1163 + 1392) / 90 = 39.05
Step 3: Between Group Variation (SSB)
Step 4: Within Group Variation
(SSW)
Since the calculated F-statistic is much higher than the critical F-value, we reject the null hypothesis
and conclude that the mean sales quantity values under different discounts are different.
This Example is an experimental study in which the marketer was trying to study the
impact of discounts on sales.
Problem 2
 Two-way ANOVA
To understand the impact of 2 factors simultaneously on a
response variable by trying to answer to the following
questions:

1. Are there differences in the avg sales quantity with

different levels of price discounts?
2. Are there differences in the avg sales quantity with
respect to different locations?
3. Are there interactions between price discounts and
location with respect to average sales quantity?
Two-way ANOVA
Two-way ANOVA: Hypothesis tests
Two-way ANOVA: Hypothesis tests
Two-way ANOVA: Hypothesis tests

For equal sample sizes:

SST = SSA + SSB + SSAB +

SSE
Two-way ANOVA: Summary
Problem
A researcher wants to study the effects of advertising strategy and
region on the sales of a product. Three types of advertising
strategies (A, B, and C) are tested across two regions (North and
South) over 5 different weeks. The sales quantities (in units)
recorded are shown below:
●Chapter 3 Completed