Validity

Session Objectives
At the conclusion of this class, you will be able to:

 define assessment validity

 differentiate between the types of validity

 evaluate Pros and Cons of Portfolio Assessment

 employ test-development procedures carefully

 develop bias-free assessment

 How valid are our tests?

 Are we testing what we say we would test?

 Is our test mirroring the emphasis of our

instruction?

 Are we asking enough questions to make

informed judgments about student performance?

 What is Validity?

• Validity means you are measuring what you are trying to

measure
• Educators used test to be able to make inferences about
students’ status
• We want the inferences we make with assessment
results to be valid
• We want valid arguments about the results of an
assessment
 Validity

• Teachers assess students to determine their status with

respect to educationally relevant variables

• Teachers use the results of assessment to make

decisions about students

• Appropriate educational decisions depend on the

accuracy of educational assessment. Why?

• This is where validity comes into play

 Validity Evidence
• The major contribution of classroom assessment to a

teacher’s decision making is it provides reasonably

evidence about students’ status.

• Systematic assessment-based inference are more valid

than on-the-fly inferences made while the teacher is

focusing on other concerns.

• Informal inferences about students’ knowledge, skills or

attitudes may lead to invalid inferences.

 Types of Validity Evidence

Validity

Content Criterion Construct

validity validity validity
Content Related Evidence of Validity

 The more evidence of validity we have, the better we will

know how much confidence to place in our score-based

inferences.

 Concern is the representativeness and relevance of

assessment content to content domain

 Most important type of evidence for classroom

assessments
Content Related Evidence of Validity

• Decades ago educators focus was dominantly on

achievement examinations and content has been looked
at as factual knowledge.

• The more a student is able to list facts in a test the more

evidence there was of content validity

• Today the notion of content of curricular aims is broaden

to include beside knowledge, skills and attitudes

• Curricular aims for classroom test consist of knowledge

and skills that were included in the intended learning
outcomes.
 Blueprint
Course Action Research Term First
Level 4 No. Questions 30
classes

Taxonomy of Cognitive Objectives

# of

Content Remember Understan Apply Analyze Synthesis Evaluation

Outline d Percentage # of Qs

1st Chapter 2 1 3 0 0 0 0 14.04 4

2nd Chapter 2 1 3 1 0 0 0 14.04 5

3rd Chapter 4 1 5 1 0 1 0 28.07 8

4th Chapter 2 1 3 0 0 0 0 14.04 4

5th Chapter 3 1 3 1 0 0 0 15.79 5

6th Chapter 2 1 3 0 0 0 0 14.04 4

Percentage 16.67 64.58 10.42 0.00 8.33 0.00 100%

Total 6 20 3 1 0 0 10 30 30
Name of course: Action Research

Learning Outcomes:
- define what reliability and validity means.
- correctly identify the types of reliability and validity
evidence given a situation.
- demonstrate an understanding of the concept of a null
hypothesis.
- use SPSS to analyze given data.
- evaluate the use of appropriate parametric statistics for
give situation.
- describe experimental and quasi-experimental studies.
- …………
 Taxonomy of Cognitive Objectives
Content
Remember Understand Apply Analyze Synthesis Evaluation
Outline
Validity and “..define what “..correctly identify
Reliability reliability and the types of
validity means” reliability and
validity evidence
given a situation”
5% 2 50% 1 50% 1
Inferential “..demonstrate an “..use SPSS to “..evaluate the use
Statistics understanding of analyze given of appropriate
the concept of a null data” parametric
hypothesis” statistics for give
situation”
10% 4 50% 2 25% 1 25% 1

Experimental “..describe “..perform an

Research experimental and experimental
quasi-experimental study and
studies” submit it end of
term
20% 8 75% 6 25% 2

Correlational “..describe “..apply the use “..determine “..come up with “..give conclusion
different terms of of regression the a research about a study
Research significance of
correlational equation” question, given a given a research
a study, given
variables” a situation” situation” situation”
12
40% 16 40% 6 25% 4 5% 1 5% 1 25 4
Types of Content-Related Evidence

 Developmental Care

 Set of procedures to assure assessment reflects

content
 Include Table of Specifications

 External Reviews

 External experts review content

 Focus is on the match of the assessment to the

content standards
 Criterion-Related Evidence of Validity

 The degree to which student’s performance on a test

accurately predicts the student’s performance on an
external criterion.
 Compare scores with another measure of performance
(criterion variable)
 Often use correlation coefficient to measure strength
of the association between variables
 If the predictor test is working pretty well, we can
use its result to make educational decisions about
students.
Construct-Related Evidence of Validity

What is a construct?
 A construct is an attribute, proficiency, ability, or skill that
happens in the human brain and is defined by established
theories. For example, "overall English language
proficiency" is a construct.
What is Construct Validity?
 Construct validity has traditionally been defined as the
experimental demonstration that a test is measuring the
construct it claims to be measuring
 Other Validities

Consequential
Face Validity
Validity
 Face Validity

• Is the appearance of a test seems to coincide with the use to

which the test is being put.

• For example, if an assessment instrument is needed to

measure students’ ability to work collaboratively in groups,
then a true-false test that focuses inclusively on abstract
group dynamic wouldn’t appear to be face valid.

• However, appearance may be misleading, so you need to

assemble evidence that let us put confidence in the score-
based inferences we arrive at by using the test.
 Consequential Validity

• It refers to whether the uses of test results are valid.

• If a test’s results are employed inappropriately (i.e.

denying a student to advance to the next grade), the
test said to be consequentially invalid.

• While “what a test score mean?” is an important

question to draw valid inferences, it is equally important
to ask the question “how test results are used?”
Explain!!
 Assessment Bias

 Reliability, validity, and assessment bias are the three

main criteria for evaluating assessments.

 Assessment bias refers to the parts of an assessment

that can offend or penalize a certain group of students.

 Assessment Bias Groups

 Gender
 Ethnicity

 Socioeconomic status

 Religion
 Language

 Country of origin (cultural questions)

 Disabilities
 Any other group-defining characteristic
 The Nature of Bias

 Overall assessment bias is when grades are distorted

due to students personal characteristics

 It constitutes a distinctive threat to the validity of test-

based inferences.

 There are two types of assessment bias!

 Offensiveness

 Unfair Penalization
 Offensiveness

 An assessment is biased if its content is considered

offensive to a subgroup of students.

 The offensiveness often comes from negative

stereotypes about how members of particular ethnic

or religious group behave.

 The bias can distract the student by making them

focusing more on the bias than the actual content of

the exam.
 Unfair Penalization
 Unfair penalization is when a student will be at a
disadvantage on a certain test item due to their
ethnicity, race, language, etc.

 It doesn’t mean they are offended, but they are not

equally prepared to answer the question as the majority
group.

 For ex. Asking questions about sports may give an

advantage to male students.

 Let’s look at the list again…

 Unfair Penalization

 Note: Assessment bias and the groups it affects will be

different in each country and culture!!

Which groups are at a disadvantage in Bahrain?

 If a student scores badly on your exam, it doesn’t mean

it is always due to assessment bias.

 The student may deserve the low grade due to laziness!

But always check your exams to make sure you have
tried your best.
 Example Question: Find the Bias!

In a Math class:
Last year Tariq’s parents got divorced. He spends half of
his time at his father’s house. How many days out of the
year does Tariq spend at his father’s house?
a. around 180 days

b. around 160 days

c. around 150 days
d. around 140 days
 Example Question: Find the Bias!

In a Math class:
What fraction represents the approximate amount of
the book shown below that Mona has read?

a. ¼
b. ½
c. ¾
 Final thought

 Don’t confuse bias with actual “shortcomings” of one

group. There may be groups who perform less well than

the majority. This needs to be solved and having bias
free exams shows which groups of students these are.
 Methods of Detecting Assessment Bias

 Judgmental Approaches

 Empirical Approach

 Bias Detection for Teachers

 Judgmental Approaches

 For high stakes exams, review panels are created

consisting of teachers in the subject that represent the
groups that might be negatively impacted.

 They may also have administrators and university

professors on the panel.

 Both men and women should also be represented.

 The panel should be reminded about what is

assessment bias.
 Empirical Approach
 A test can be analyzed for bias after it has been taken
by a large number students. Data is needed from
hundreds of students in many types of “subgroups”.

 The data is then analyzed using differential item

functioning to see how each subgroup performed on
each question.

 Items showing bias should be removed.

 This should be done by administration or Ministry of

Education.
 Bias Detection for Teachers!

 Best way to stop bias: Be sensitive to (or acknowledge) the fact

that assessment bias can exist and that there is a need to

eliminate it.

 Review your assessment from your students perspectives and

think about their backgrounds/context.

 Try to review every item on every assessment, but keep in mind

that an assessment can never be 100% bias free.

Students who do not speak the majority language

 Are there any laws in Bahrain about students who do not

speak Arabic as their first language?
 What accommodations can we make for students who do
not have language background? Do you have separate
classes?
 General Recommendations:
 Do not translate tests

 Do not use interpreters

 Longer test times could be possible

 Use of dictionaries could be possible

 Use graphics that can help support students with lack of language
(especially Math and Science)
 Resources

Brown, J. (2000). What is construct validity?. Shiken: JALT

Testing & Evaluation SIG Newsletter, 4 (2) Oct 2000 (p.
8 - 12). Accessed on 1/26/2015 at
http://jalt.org/test/PDF/Brown8.pdf.

Popham, W. J. (2011) Classroom Assessment: What

Teachers Need to Know? 6th Ed, Boston MA: Pearson.