Examining Your
Multiple-Choice Questions
As Ron Berk (known for his pithy humor) observes,
the multiple-choice question “holds world records in the categories of most
popular, most unpopular, most used, most misused, most loved and most hated.”
According to one source I read, multiple-choice questions were first used
around the time of World War I to measure the abilities of new Army recruits. As
class sizes have grown and the demands on teacher time expanded, they have
become the favorite testing tool in higher education.
However, even those who use them often aren’t all
that crazy about them and with good reason. How many problems faced by professionals
come in a multiple-choice format? Answering multiple-choice questions doesn’t
teach students how to formulate answers; it teaches them how to select answers.
And sometimes choosing the correct answer is more a function of literary skill
than content knowledge. Multiple-choice questions encourage guessing, and if
the guess is correct, students get credit for something they didn’t know.
Answer options contain misinformation—that’s what makes them wrong. As students
read and consider all the possible answers, they are exposed to incorrect
content, which some research has shown influences subsequent thinking about the
topic.
Many multiple-choice questions don’t challenge
students to think but rather encourage them to memorize. Test bank questions
are the worst. Analysis of questions in a variety of disciplines documents that
around 85% of the questions test lower-level knowledge. And I’ve already
referenced in the blog and lots of other places a very rigorous analysis of
test questions on biology exams. “Of the 9713 assessment items submitted … by
50 faculty teaching introductory biology, 93% were rated at Bloom’s level 1 or
2—knowledge and comprehension.” (p. 437)
Clegg and Cashin in a classic (but still very
helpful) paper write, “It is true that many multiple-choice items are
superficial, but that is the result of poor test craftsmanship and not an
inherent limitation of the item type. A well designed multiple-choice item can
test high levels of student learning.” And that’s not all multiple-choice questions
can do well. They can ascertain student knowledge of a wide range of topics in
a timely manner. Best of all, they are quick and easy to grade.
The problem is writing those kinds of challenging
questions. It takes careful thought and creativity to write a question that
requires thinking to answer, and for most teachers that equates to time.
Professional test question writers like those writing items for the SAT, ACT,
and GRE devote hours to creating and testing questions.
The reasons and realities that motivate using
multiple-choice questions are compelling. So the question is, how do we help
faculty write good ones? I’m going to follow this post with one that reviews
the do’s and don’ts of writing good questions. But before that, I encourage you
to take a long hard look at the multiple-choice questions on your exams. When
the low level of multiple-choice test questions is discussed, most faculty
quickly assert that their questions are rigorous and do challenge thinking. A
number of years ago, a cross-disciplinary faculty cohort reported that a third
of their questions measured complex cognitive skills. An analysis showed that
only 8.5% of their questions did, with the remaining testing basic
comprehension and recall. Being objective isn’t easy, but it’s necessary.
Jacobs and Chase recommend that faculty write
multiple-choice questions as they go (not all at once, right before the exam).
If the questions are filled with the content pertaining to them, they could be
reviewed, revised, or rewritten as part of class preparation. It isn’t always
necessary to re-do the whole question. Sometimes an answer option is all that
needs to be replaced or revised. And there’s no need for every question to test
higher order thinking skills. Most of us could use a few more, but sometimes we
do need to test basic knowledge.
References:
Momsen, J. L., Long, T. M., Wyse, S. A., and Ebert-May, D. (2010) Just the
facts? Introductory undergraduate biology courses focus on low-level cognitive
skills. Cell Biology Education, 9 (Winter), 435-440.
Clegg, V. L, and Cashin, W. E. (1986). Improving
Multiple-Choice Tests. IDEA Paper No. 16. Available for free at: www.theideacenter.org.
Jacobs, L. C. and Chase, C. I. Developing and
Using Tests Effectively: A Guide for Faculty. San Francisco: Jossey-Bass,
1992.
Examining Your Multiple-Choice Questions
As Ron Berk (known for his pithy humor)
observes, the multiple-choice question “holds world records in the
categories of most popular, most unpopular, most used, most misused,
most loved and most hated.” According to one source I read,
multiple-choice questions were first used around the time of World War I
to measure the abilities of new Army recruits. As class sizes have
grown and the demands on teacher time expanded, they have become the
favorite testing tool in higher education.
However, even those who use them often aren’t all that crazy about
them and with good reason. How many problems faced by professionals come
in a multiple-choice format? Answering multiple-choice questions
doesn’t teach students how to formulate answers; it teaches them how to
select answers. And sometimes choosing the correct answer is more a
function of literary skill than content knowledge. Multiple-choice
questions encourage guessing, and if the guess is correct, students get
credit for something they didn’t know. Answer options contain
misinformation—that’s what makes them wrong. As students read and
consider all the possible answers, they are exposed to incorrect
content, which some research has shown influences subsequent thinking
about the topic.
Many multiple-choice questions don’t challenge students to think but
rather encourage them to memorize. Test bank questions are the worst.
Analysis of questions in a variety of disciplines documents that around
85% of the questions test lower-level knowledge. And I’ve already
referenced in the blog and lots of other places a very rigorous analysis
of test questions on biology exams. “Of the 9713 assessment items
submitted … by 50 faculty teaching introductory biology, 93% were rated
at Bloom’s level 1 or 2—knowledge and comprehension.” (p. 437)
Clegg and Cashin in a classic (but still very helpful) paper write,
“It is true that many multiple-choice items are superficial, but that is
the result of poor test craftsmanship and not an inherent limitation of
the item type. A well designed multiple-choice item can test high
levels of student learning.” And that’s not all multiple-choice
questions can do well. They can ascertain student knowledge of a wide
range of topics in a timely manner. Best of all, they are quick and
easy to grade.
The problem is writing those kinds of challenging questions. It
takes careful thought and creativity to write a question that requires
thinking to answer, and for most teachers that equates to time.
Professional test question writers like those writing items for the SAT,
ACT, and GRE devote hours to creating and testing questions.
The reasons and realities that motivate using multiple-choice
questions are compelling. So the question is, how do we help faculty
write good ones? I’m going to follow this post with one that reviews the
do’s and don’ts of writing good questions. But before that, I
encourage you to take a long hard look at the multiple-choice questions
on your exams. When the low level of multiple-choice test questions is
discussed, most faculty quickly assert that their questions are rigorous
and do challenge thinking. A number of years ago, a cross-disciplinary
faculty cohort reported that a third of their questions measured
complex cognitive skills. An analysis showed that only 8.5% of their
questions did, with the remaining testing basic comprehension and
recall. Being objective isn’t easy, but it’s necessary.
Jacobs and Chase recommend that faculty write multiple-choice
questions as they go (not all at once, right before the exam). If the
questions are filled with the content pertaining to them, they could be
reviewed, revised, or rewritten as part of class preparation. It isn’t
always necessary to re-do the whole question. Sometimes an answer option
is all that needs to be replaced or revised. And there’s no need for
every question to test higher order thinking skills. Most of us could
use a few more, but sometimes we do need to test basic knowledge.
References:
Momsen, J. L., Long, T. M., Wyse, S. A., and Ebert-May, D. (2010) Just
the facts? Introductory undergraduate biology courses focus on
low-level cognitive skills.
Cell Biology Education, 9 (Winter), 435-440.
Clegg, V. L, and Cashin, W. E. (1986). Improving Multiple-Choice Tests.
IDEA Paper No. 16. Available for free at:
www.theideacenter.org.
Jacobs, L. C. and Chase, C. I.
Developing and Using Tests Effectively: A Guide for Faculty. San Francisco: Jossey-Bass, 1992.
- See more at:
http://www.facultyfocus.com/articles/teaching-professor-blog/examining-multiple-choice-questions/#sthash.KU7vZ4dg.dpuf
