Assessment

Examining Your Multiple-Choice Questions

By: Maryellen Weimer, PhD

As Ron Berk (known for his pithy humor) observes, the multiple-choice question “holds world records in the categories of most popular, most unpopular, most used, most misused, most loved and most hated.” According to one source I read, multiple-choice questions were first used around the time of World War I to measure the abilities of new Army recruits. As class sizes have grown and the demands on teacher time expanded, they have become the favorite testing tool in higher education.

However, even those who use them often aren’t all that crazy about them and with good reason. How many problems faced by professionals come in a multiple-choice format? Answering multiple-choice questions doesn’t teach students how to formulate answers; it teaches them how to select answers. And sometimes choosing the correct answer is more a function of literary skill than content knowledge. Multiple-choice questions encourage guessing, and if the guess is correct, students get credit for something they didn’t know. Answer options contain misinformation—that’s what makes them wrong. As students read and consider all the possible answers, they are exposed to incorrect content, which some research has shown influences subsequent thinking about the topic.

Many multiple-choice questions don’t challenge students to think but rather encourage them to memorize. Test bank questions are the worst. Analysis of questions in a variety of disciplines documents that around 85% of the questions test lower-level knowledge. And I’ve already referenced in the blog and lots of other places a very rigorous analysis of test questions on biology exams. “Of the 9713 assessment items submitted … by 50 faculty teaching introductory biology, 93% were rated at Bloom’s level 1 or 2—knowledge and comprehension.” (p. 437)

Clegg and Cashin in a classic (but still very helpful) paper write, “It is true that many multiple-choice items are superficial, but that is the result of poor test craftsmanship and not an inherent limitation of the item type. A well designed multiple-choice item can test high levels of student learning.” And that’s not all multiple-choice questions can do well. They can ascertain student knowledge of a wide range of topics in a timely manner. Best of all, they are quick and easy to grade.

The problem is writing those kinds of challenging questions. It takes careful thought and creativity to write a question that requires thinking to answer, and for most teachers that equates to time. Professional test question writers like those writing items for the SAT, ACT, and GRE devote hours to creating and testing questions.

The reasons and realities that motivate using multiple-choice questions are compelling. So the question is, how do we help faculty write good ones? I’m going to follow this post with one that reviews the do’s and don’ts of writing good questions. But before that, I encourage you to take a long hard look at the multiple-choice questions on your exams. When the low level of multiple-choice test questions is discussed, most faculty quickly assert that their questions are rigorous and do challenge thinking. A number of years ago, a cross-disciplinary faculty cohort reported that a third of their questions measured complex cognitive skills. An analysis showed that only 8.5% of their questions did, with the remaining testing basic comprehension and recall. Being objective isn’t easy, but it’s necessary.

Jacobs and Chase recommend that faculty write multiple-choice questions as they go (not all at once, right before the exam). If the questions are filled with the content pertaining to them, they could be reviewed, revised, or rewritten as part of class preparation. It isn’t always necessary to re-do the whole question. Sometimes an answer option is all that needs to be replaced or revised. And there’s no need for every question to test higher order thinking skills. Most of us could use a few more, but sometimes we do need to test basic knowledge.

References:
Momsen, J. L., Long, T. M., Wyse, S. A., and Ebert-May, D. (2010) Just the facts? Introductory undergraduate biology courses focus on low-level cognitive skills. Cell Biology Education, 9 (Winter), 435-440.

Clegg, V. L, and Cashin, W. E. (1986). Improving Multiple-Choice Tests. IDEA Paper No. 16. Available for free at: www.theideacenter.org.

Jacobs, L. C. and Chase, C. I. Developing and Using Tests Effectively: A Guide for Faculty. San Francisco: Jossey-Bass, 1992.

Examining Your Multiple-Choice Questions

By: Maryellen Weimer, PhD in Teaching Professor Blog

Add Comment

As Ron Berk (known for his pithy humor) observes, the multiple-choice question “holds world records in the categories of most popular, most unpopular, most used, most misused, most loved and most hated.” According to one source I read, multiple-choice questions were first used around the time of World War I to measure the abilities of new Army recruits. As class sizes have grown and the demands on teacher time expanded, they have become the favorite testing tool in higher education.
However, even those who use them often aren’t all that crazy about them and with good reason. How many problems faced by professionals come in a multiple-choice format? Answering multiple-choice questions doesn’t teach students how to formulate answers; it teaches them how to select answers. And sometimes choosing the correct answer is more a function of literary skill than content knowledge. Multiple-choice questions encourage guessing, and if the guess is correct, students get credit for something they didn’t know. Answer options contain misinformation—that’s what makes them wrong. As students read and consider all the possible answers, they are exposed to incorrect content, which some research has shown influences subsequent thinking about the topic.
Many multiple-choice questions don’t challenge students to think but rather encourage them to memorize. Test bank questions are the worst. Analysis of questions in a variety of disciplines documents that around 85% of the questions test lower-level knowledge. And I’ve already referenced in the blog and lots of other places a very rigorous analysis of test questions on biology exams. “Of the 9713 assessment items submitted … by 50 faculty teaching introductory biology, 93% were rated at Bloom’s level 1 or 2—knowledge and comprehension.” (p. 437)
Clegg and Cashin in a classic (but still very helpful) paper write, “It is true that many multiple-choice items are superficial, but that is the result of poor test craftsmanship and not an inherent limitation of the item type. A well designed multiple-choice item can test high levels of student learning.” And that’s not all multiple-choice questions can do well. They can ascertain student knowledge of a wide range of topics in a timely manner. Best of all, they are quick and easy to grade.
The problem is writing those kinds of challenging questions. It takes careful thought and creativity to write a question that requires thinking to answer, and for most teachers that equates to time. Professional test question writers like those writing items for the SAT, ACT, and GRE devote hours to creating and testing questions.
The reasons and realities that motivate using multiple-choice questions are compelling. So the question is, how do we help faculty write good ones? I’m going to follow this post with one that reviews the do’s and don’ts of writing good questions. But before that, I encourage you to take a long hard look at the multiple-choice questions on your exams. When the low level of multiple-choice test questions is discussed, most faculty quickly assert that their questions are rigorous and do challenge thinking. A number of years ago, a cross-disciplinary faculty cohort reported that a third of their questions measured complex cognitive skills. An analysis showed that only 8.5% of their questions did, with the remaining testing basic comprehension and recall. Being objective isn’t easy, but it’s necessary.
Jacobs and Chase recommend that faculty write multiple-choice questions as they go (not all at once, right before the exam). If the questions are filled with the content pertaining to them, they could be reviewed, revised, or rewritten as part of class preparation. It isn’t always necessary to re-do the whole question. Sometimes an answer option is all that needs to be replaced or revised. And there’s no need for every question to test higher order thinking skills. Most of us could use a few more, but sometimes we do need to test basic knowledge.
References:
Momsen, J. L., Long, T. M., Wyse, S. A., and Ebert-May, D. (2010) Just the facts? Introductory undergraduate biology courses focus on low-level cognitive skills. Cell Biology Education, 9 (Winter), 435-440.
Clegg, V. L, and Cashin, W. E. (1986). Improving Multiple-Choice Tests. IDEA Paper No. 16. Available for free at: www.theideacenter.org.
Jacobs, L. C. and Chase, C. I. Developing and Using Tests Effectively: A Guide for Faculty. San Francisco: Jossey-Bass, 1992.
- See more at: http://www.facultyfocus.com/articles/teaching-professor-blog/examining-multiple-choice-questions/#sthash.KU7vZ4dg.dpuf