Constructing Multiple Choice

Type of Test
RAMON MERCADO BELARDO,JR.
This may look funny, but come to think
of it…..
Do we lack choices here?
Or maybe our question led us to so many
possibilities…
May forever….
So we should therefore be ready for this kind of
response
If you were the teacher, would you consider his
answer as correct?
 Are you the kind of
teacher who asks the
following questions?
 The poem “The Raven” ______
a. was written by Edgar Allan Poe
b. was written by Elizabeth Browning
c. was written by Omar Khayyan
d. was written by Jose Garcia Villa
 • Is it NOT true
that Magellan discovered the
Philippines?
 When did the People Power
Revolution take in the Philippines?
A. February 23, 1986
B. after the Snap Election
C. March 1, 1956
D. after Valentines’ Day in 1986
• Who was the author of the book
quoted in the footnote of
Chapter 1 of the present textbook
If you answered “YES” to any of
the choices presented then you
have a BIG PROBLEM!
YOUR STUDENTS ARE
NOT LEARNING!!!!
“13% of students who fail in class
are caused by faulty test
questions”
WORLDWATCH
The Philadelphia Trumpet
August 2005
• It is estimated that 90% of all test
questions asked in the US are of
“Low level” - knowledge and
comprehension
(Wilen, W.W., 1992)
• “Low level” doesn’t mean easy:
• Write an essay explaining the decline and fall of the
Roman Empire incorporating at least five of the
seven causes discussed in class from the writings of
Gibbon and Toynbee
• “High level” doesn’t mean hard:
• Which movie did you like more,
WALL-E or Cars? Why?
Let’s start by looking into the
following test items
 What do you notice?
1. Which statement is true about bacteria?
A. Bacteria are living organisms that
consist of one cell.
B. Bacteria are the smallest of all living
things.
C. Bacteria are the oldest and most
diverse life forms.
D. All statements are true.
 Here’s another one….
2. If you rub your hair with a comb, it will
become positively charged. What will happen to
your hair?
a. It will remain un-charged.
b. It will repelled by the comb.
c. It will become positively charged.
d. It will become negatively charged.
 And another…
3. Why is preventing the spread of bacteria
important?
A. Bacteria are unsightly.
B. Bacteria are fire hazard.
C. Bacteria can cause food poisoning.
D. All of the above.
 TEST
CONSTRUCTION
“The evaluation of pupils’
progress is a major aspect of
the teacher’s job.“
Evaluating Educational Outcomes
(Oriondo & Antonio)
Explain the message of the comic strip.
The Purpose of Testing

• To provide a record for assigning

grades.
• To provide a learning experience for
students.
• To motivate students to learn.
• To serve as a guide for further study.
The Purpose of Testing
• To assess how well students are achieving
the stated goals of the lesson.
• To provide the instructor with an
opportunity to reinforce the stated
objectives and highlight what is important
for students to remember.
Characteristics of Good Tests

• Validity – the extent to which the

test measures what it intends to
measure
• Reliability – the consistency with
which a test measures what it is
supposed to measure
• Usability – the test can be
administered with ease, clarity
and uniformity
Other Things to Consider

• Scorability – easy to score

• Interpretability – test results can
be properly interpreted and is a
major basis in making sound
educational decisions
• Economical – the test can be
reused without compromising the
validity and reliability
“To be able to prepare a good
test, one has to have a
mastery of the subject
matter, knowledge of the
pupils to be tested, skill in
verbal expression and the
use of the different test
format”
Evaluating Educational Outcomes
(Oriondo & Antonio)
 5 Most Commonly used
Test Format

1. Multiple Choice
2. True or False
3. Matching Type
4. Fill-in the blanks (Sentence Completion)
5. Essay
Source: Turn-out of Test Questions in SSI (2003-2007)
General Steps in Test Construction
DRAFT
OUTLINE

ORDER

PRODUCE A TEST ANALYZE

T.O.S.

SUBMISSION
 1. OUTLINE:
– the unit learning objectives or
– the unit content or major concepts to
be covered by the test or
– The scope of the period to be tested
and the scope of the test

Back to Main Menu

 2. Table of Specifications (TOS)
• A two way chart that relates the learning
outcomes to the course content
• It enables the teacher to prepare a test
containing a representative sample of student
behavior in each of the areas tested.
 A. Start with Instructional Objectives
• Circulation Test
– 1. Identifies parts of circulation system.

– 2. Explains function of parts in relation to whole

system.

– 3. Distinguishes between circulation system and

other bodily systems.
Start with Instructional Objectives
 What to consider
• Length of Test
• Weight to be given to each objective
• Weight to be given to each level of taxonomy
• Estimate number of items in each cell
Circulation Test
 B. Item Selection
Types of Items
Objectively Scored (Selection)
• true/false
• completion
• matching
• multiple choice
Subjectively Scored (Supply)
• interpretive exercises
• essay
 Tips in Preparing the Table of
Specifications (TOS)
• Don’t make it overly detailed.
• It's best to identify major ideas and skills rather
than specific details.
• Use a cognitive taxonomy that is most appropriate
to your discipline, including non-specific skills like
communication skills or graphic skills or
computational skills if such are important to your
evaluation of the answer.
 Tips in Preparing the Table of
Specifications (TOS)
• Weigh the appropriateness of the distribution of
checks against the students' level, the importance
of the test, the amount of time available.
• MATCH the question level appropriate to the level
of thinking skills
 Examples of Student Activities and Verbs for
Bloom’s Cognitive Levels
Table 2.1 in Jacobs & Chase (1992:19)
Bloom’s Cognitive Student Activity Words to Use in Item
Level Stem
Knowledge Remembering facts, Define, list, state,
terms, concepts, identify, label, name,
definitions, principles who?, when?, where?,
what?
Comprehension Explaining/interpreting Explain, predict, interpret,
the meaning of material infer, summarize, convert,
translate, account for,
give example, paraphrase
Application Using a concept or Apply, solve, show, make
principle to solve a use of, modify,
problem demonstrate, compute
 Examples of Student Activities and
Verbs for Bloom’s Cognitive Levels
Table 2.1 in Jacobs & Chase (1992:19)
Bloom’s Cognitive Student Activity Words to Use in Item
Level Stem
Analysis Breaking material down Differentiate,
into its component parts to compare/contrast,
see interrelationships/ distinguish ____from
hierarchy of ideas ____, how does
____relate to ___, why
does ____work
Synthesis Producing something new or Design, construct, develop,
original from component formulate, imagine,
parts create, change, write a
poem or short story
Evaluation Making a judgment based Appraise, evaluate,
on a pre-established set of justify, judge, which would
criteria be better?
 Tips in Preparing the Table of
Specifications (TOS)
• The following array shows the most common
questions types used at various cognitive levels.

Factual Knowledge Application Analysis and

Evaluation
Multiple Choice Multiple Choice Multiple Choice
True/False Short Answer Essay
Matching Type Problems
S. Completion Essay
Short Answer/RRT
3. DRAFT the questions covering the
content in the outline

Back to Main Menu

4. ORDER the selected questions logically.
 Place simpler items at the beginning to
ease students into the exam.
 Group item types together under common
instructions.
 If desirable, order the questions logically
from a content standpoint (e.g.
chronologically or by conceptual groups,
etc.)
Back to Main Menu
 5. Test
• PUT the questions away for one or two days
before rereading them or have someone else
review them for clarity.
• TEST the questions by actually taking the test.

Back to Main Menu

6. ANALYZE the items to give you an
idea whether the questions were well-
written or poorly written as well as if
there were problems in understanding
instruction.

Back to Main Menu

General Rules in Writing Test Questions

• Number test questions continuously.

• Keep your test question in each test group
uniform.
• Make your layout presentable.
 Some additional guidelines to consider when
writing items are described below:
1. Avoid humorous items. Classroom testing is
very important and humorous items may
cause students to either not take the exam
seriously or become confused or anxious.
2. Items should measure only the construct of
interest, not one’s knowledge of the item
context.
3. Write items to measure what students know,
not what they do not know. (Cohen & Wallack)
Multiple Choice Test
 When to Use
• To test a variety of levels of learning
• When you have a large number of individuals
taking the test
• When you have time to construct the test
items
• When time is limited for scoring
• When it is not important to determine how
well individuals can formulate their own
answer.
 Strengths
• Versatility in measuring all levels of cognitive
skills.
• Permit a wide sampling of content and
objectives.
• Provide highly reliable test scores.
• Can be machine-scored quickly and
accurately.
• Reduced guessing factor compared with true-
false items.
 Limitations
Multiple-choice items:
• are difficult and time-consuming to construct,
• lead a teacher to favor simple recall of facts,
• place a high degree of dependence on the student’s
reading ability and instructor’s writing ability, and
• are particularly subject to clueing. (Students can
often deduce the correct response by elimination.)
The parts
 The Complete Parts
Instruction : Read the sentence. Decide how the underlined
word is used and choose the answer that has almost the
same meaning as the underlined word.
Stimulus material: The carpenter said that a new beam
would support the weight of our sagging roof.
Number and stem: 1. What does support as used in the
sentence mean?
Options (Key and Distractors):
A. Sustain
B. Justify
C. Prop up *
D. Pay for
 In writing stem
1. Before writing the stem, identify the one
point to be tested by that item.
2. Use either a question or an incomplete
statement as stem, but prefer the question
format.
3. When using incomplete statements place the
blank space at the end of the stem versus the
beginning.
4. State the specific problem of the question or
incomplete statement clearly in the stem and at
a reading level appropriate for the examinees.
5. Include as much information in the stem and
as little in the options as possible but do not
include irrelevant material.
6. Restrict the use of negatives in the stem. If it
could not be avoided, write the word NOT in
bold and capital letters.
7. Avoid irrelevant clues to the correct option.
8. State the stem in positive form (in general).
 Writing the Options
• Be satisfied with four or five well constructed
options.

• Construct distracters that are comparable in

length, complexity and grammatical form to
the answer, avoiding the use of such words as
“always”, “never”, and “all”.
• Options which read “none of the above”,
“both a and e”, “all of the above” should be
avoided when students have been instructed
to choose the ”best answer”, which implies
that the options vary in degree of correctness.
• After the options are written, vary the location
of the answer on a random basis as possible.
• If possible, have a colleague with expertise in
the content area of the exam to review the
items for possible ambiguities, redundancies
or other structural difficulties.
• Foils of decoys (incorrect options or distracter)
should be plausible and attractive.

• Alternatives should be homogenous in subject

content, form and grammatical structure.

• Word the alternatives clearly and concisely.

• Keep the alternatives free from clues as to
which response is correct.
• Avoid alternatives “all of the above” and
“none of the above” (in general).
• When writing distracters, think of incorrect
responses that students might make.
• Include one and only one correct or clearly
best answer in each item.
• Prepare answer placement/key before writing
the test to avoid patterns.
 Examples
1. Sheldon developed a highly controversial
theory of personality based on body type and
temperament of the individual. Which of the
following is a criticism of Sheldon's work?
a. He was influenced too much by the
Freudian psychoanalysis.
b. His rating of physique and temperament
were not independent.
c. He failed to use empirical approach.
d. His research sample was improperly
selected.
 Examples
Better: (Eliminate excessive wording and
irrelevant information)
1. Which of the following is a criticism of
Sheldon's theory of personality?
 Examples
2. The receptors for the vestibular sense are
located
a. in the fovea.
b. in the brain.
c. in the middle ear.
d. in the inner ear.
 Examples

Better: (Include in the stem any word(s) that might

otherwise be repeated in each option.)
2. The receptors for the vestibular senses are
located in the _______.
a. fovea
b. brain
c. middle ear
d. inner ear
 Examples
3. Which is not a major technique for studying
brain function?
a. Accident and injury
b. Cutting and removing
c. Electrical stimulation
d. Direct phrenology
 Examples
Better: (Use negatively stated stems sparingly.
When used, underline and/or capitalize the
negative word.)
3. Which is NOT a major technique for studying
brain function?
 Examples
4. ________________ is the least form of
behavior disorder.
a. Psychosis
b. Panic disorder
c. Neurasthenia
d. Neurosis
 Examples
Better: (When using incomplete statements
avoid beginning with the blank space.)
4. The least severe form of behavior disorder is
__________________.
 Examples
5. The number of photoreceptors in the retina of
each human is about
a. 115 million
b. 5 million
c. 65 million
d. 35 billion
 Examples
Better: (When possible, present alternatives in
some logical order.)
5. The number of photo receptors in the retina
of each human is about
a. 5 million
b. 35 million
c. 65 million
d. 115 million
 Examples
6. Latane and Darley's smoke-filled room
experiment suggested that people are less
likely to help in groups than alone,
because people
a. in groups talk to one another.
b. who are alone are more attentive.
c. in groups do not display pluralistic
ignorance.
d. in groups allow others to define the
situation as a non-emergency
 Examples
Better: (All alternatives should be approximately equal
in length.)
6. Latane and Darley's smoke-filled room experiment
suggested that people are less likely to help in
groups than alone, because people in groups
a. talk to one another
b. are less attentive than people who are
alone
c. do not display pluralistic ignorance
d. allow other to define non-emergencies
 More examples
Avoid stems that ask for a series of multiple true-false responses.
1. Which of the following is true about the middle adult years?
A. It encompasses ages 19 to 30.
B. It is the most conflict-free period of life.
C. It is characterized by dramatic changes in our sense of
values.
D. It is marked by a conflict between intimacy and isolation.
Better:
1. According to Erickson, the middle adult years are
characterized by the conflict between ____ and ___ .
A. intimacy; isolation
B. generativity; stagnation
C. integrity; despair
D. industry; despondency
Use negatively stated stems sparingly. When used, underline
and/or capitalize the negative word.
Which is not a major technique for studying brain function?
A. accident and injury
B. cutting and removing
C. electrical stimulation
D. direct phrenology
Better:
Which is NOT a major technique for studying brain function?
Make all alternatives plausible and equally attractive to both less-
knowledgeable and skillful students.
The number of photoreceptors in the retina of each human eye is about
A. 1000,000.
B. 2 million.
C. 115 million.
D. 2.37 billion.
Better:
A. 5 million.
B. 35 million.
C. 65 million.
D. 115 million.
Minimize the use of the all-of –the-above and none-of-the-above
alternatives.
Problem representation involves
A. determining which factors matter and which do not.
B. the initial state of problem solving.
C. both a and b.
D. neither a nor b.
Better:
A. determining which factors matter and which do not.
B. the initial state of problem solving.
C. reducing the problem to manageable segments.
D. all of the above.
When possible, present alternatives in some logical order (e.g.,
most to least and chronological .)
In the course of a dark adaptation , the eye’s best sensitivity to
wavelength shifts to
A. 580 millimicrons.
B. 477 millimicrons.
C. 505 millimicrons.
D. 600 millimicrons.
Better:
A. 600 millimicrons.
B. 580 millimicrons.
C. 505 millimicrons.
D. 477millimicrons.
Avoid vague stems by stating the problem in the
stem:
• Poor Example
• California:
a. Contains the tallest mountain in the United
States.
b. Has an eagle on its state flag.
c. Is the second largest state in terms of area.
*d. Was the location of the Gold Rush of 1849.
 Better
What is the main reason so many people moved
to California in 1849?
a. California land was fertile, plentiful, and
inexpensive.
*b. Gold was discovered in central California.
c. The east was preparing for a civil war.
d. They wanted to establish religious
settlements.
• Avoid negatively worded stems by stating the
stem in a positive form:
Poor Example
A nurse is assessing a client who has
pneumonia. Which of these assessment findings
indicates that the client does NOT need to be
suctioned?
a. Diminished breath sounds.
*b. Absence of adventitious breath sounds.
c. Inability to cough up sputum.
d. Wheezing following bronchodilator therapy.
 Better
Which of these assessment findings, if identified
in a client who has pneumonia, indicates that
the client needs to be suctioned?
a. Absence of adventitious breath sounds.
b. Respiratory rate of 18 breaths per minute.
*c. Inability to cough up sputum.
d. Wheezing prior to bronchodilator therapy.
 Avoid Overlapping Alternatives

Poor Example Better

• What is the average What is the average
effective radiation dose effective radiation dose
from chest CT? from chest CT?
a. 1-8 mSv a. 1-7 mSv
b. 8-16 mSv b. 8-15 mSv
c. 16-24 mSv c. 16-24 mSv
d. 24-32 mSv d. 24-32 mSv
 Avoid Dissimilar Alternatives

• Poor Example Good Example

Idaho is widely known as: • Idaho is widely known
*a. The largest producer for its:
of potatoes in the United a. Apples.
States. b. Corn.
b. The location of the *c. Potatoes.
tallest mountain in the d. Wheat
United States.
c. The state with a beaver
on its flag.
d. The “Treasure State.”
 Avoid implausible/unreasonable
alternatives:
Poor Example Good Example
• Which of the following • Which of the following
artists is known for artists is known for
painting the ceiling of painting the ceiling of
the Sistine Chapel? the Sistine Chapel?
a. Warhol. a. Botticelli.
b. Flintstone. b. da Vinci.
*c. Michelangelo. *c. Michelangelo.
d. Santa Claus. d. Raphael.
Suggestions for Writing Multiple-Choice Items Which Measure
Higher Objectives

• Present practical or real-world situations

to the students. These problems may use
short paragraphs describing a problem in
a practical situation. Items can be written
which call for the application of
principles to the solution of these
practical problems, or the evaluation of
several alternative procedures.
• Present the student with a diagram of
equipment and ask for application, analysis, or
evaluations, e.g., "What happens at point A if
.?," "How is A related to B?“

• Present actual quotations taken from

newspapers or other published sources or
contrived quotations that could have come
from such sources. Ask for the interpretation
or evaluation of these quotations.
• Use pictorial materials that
require students to apply
principles and concepts.
• Use charts, tables or figures
that require interpretation.
6 questions to reflect on as
regards mct
I opened a course on a topic I know nothing
about, clicked through without reading
anything, and took the assessment. I passed!
What does that suggest?
a. I am a genius!
b. The assessment was too easy.
c. Maybe the course was too easy, too.
d. Maybe the course didn’t even need to be
written.
e. b, c, and d
How often is the correct answer a?
a. Usually
b. Frequently
c. Often
d. Almost never, because if a is the
right answer, then the learner
doesn’t have to read all the other
options we spent so much time
writing and revising?
When is it NOT a good idea to avoid
negative questions?
a. Never
b. Sometimes
c. Always
d. What?
When is “All of the above” the correct
answer?
a. With alarming regularity
b. When we try to cover too much in one
question
c. When we use a question to teach
instead of assess
d. All of the above
In a multiple-choice question, when is the longest
answer the correct answer?
a. Rarely
b. Sometimes
c. It’s almost always the correct answer, and it’s
often stuffed with new information that
should have gone in the main part of the
course but we forgot so now we’re putting it
in the quiz because we can’t possibly leave
out the tiniest detail
d. Occasionally
We can confuse learners when we:
a. fail to actually complete the
sentence we started in the
question.
b. inconsistent grammar in the
options.
c. sometimes we veer off into
another idea entirely.
d. test what we do not teach