More on Multiple Choice Questions

Emily asked me to say more about using multiple choice questions to evaluate more advanced levels of skill and comprehension. I had posted my notes from this talk by Tziporah Kasachkoff and Nils Rauhut on multiple choice questions at the AAPT. Tziporah and Nils discussed using multiple choice questions for three purposes. The first was for simple "did you do the reading" quizzes. These are simple fact recall questions which even critics of multiple choice testing acknowledge are useful. The second two, however, were testing for higher levels of reading comprehension, and testing for key logical skills These were the uses Emily wanted to know more about.

To my mind, the key to the sophisticated use of multiple choice questions is to recognize that multiple choice question can be used to identify all parts of an argument, and this can be leveraged to evaluate students on the educational goals named at all levels of Bloom's taxonomy. If you teach logic or critical thinking you are probably already familiar with this at a basic level. You can give the students a simple three ore four sentence argument and ask them to identify the conclusion. The example is written in a standard form used in critical thinking texts, where the part of the passage in italics provides context for understanding the argument given by the speaker in the remainder of the passage.

Example 1: Consider the following passage

Susan tells her friend Jill about a theory she's developed. I think your husband is having an affair. Think about it. He has been working late a lot recently, and you found strange lipstick in his coat pocket. Also, he seems very chipper and has been weirdly nice to you.

Which sentence represents the conclusion of this passage? (Select one option.)

1. Your husband is having an affair.
2. Your husband has been working late a lot recently
3. Think about it.
4. You found strange lipstick in your husband's coat pocket
5. Your husband seems very chipper and has been weirdly nice to you.

The same passage can be used to identify a premise, for instance like this.

Example 2: The sentence “He has been working late a lot recently” functions as what in this passage? (Select one option.)

1. a conclusion
2. a premise
3. an indicator phrase
4. a thesis

You can also write multiple choice questions that focus on the inferential relationship itself, rather than just premises or conclusions, as in this example.

Example 3: Consider the following pair of sentences

1. Fido is a dog
2. FIdo is a mammal

Which of the following best describes the relationship between the pair of sentences. (Select one option.)

1. If (1) is true, then (2) is true.
2. If (2) is true, then (1) is true
3. Sentences (1) and (2) are either both true or both false.

These examples are simple, but they illustrate the fact that you can use multiple choice questions to gage comprehension of every part of an argument. This is something you can build on in order to evaluate the skills Tziporah and Nils were concerned with: higher levels of reading comprehension and higher logical skills.

Reading comprehension is no small issue to be delegated to primary and secondary schools. This becomes clear in the philosophy, just because so many of the texts we deal with are so very difficult. But even for a general college education, giving students the ability to properly digest complex information is crucial. It is one of the big skills tested in the Collegiate Learning Assessment, which is probably the best measurement of critical thinking out there. The CLA is also, importantly, a test that college students don't show much improvement on during their time in school.

Being able to identify an author's point is a basic reading comprehension skill, and one that most people don't get the opportunity to practice, because so many writers in the popular media never bother to have a point. In philosophy, as in many disciplines, the author's basic point is the conclusion of her argument. Questions that ask you to identify the conclusion of a passage can be scaled up to ask students about the conclusions of assigned readings. Now, of course, it is not just enough to know an author'point. We also want the students to know why the author thinks we should believe her claims. And so we can also employ questions that ask about premises. The next example is from a test given at the end of a unit on Hume's Dialogues Concerning Natural Religion. It features one target passage, and then questions that measure simple factual recall, recognition of conclusions, recognition of premises, and an understanding of the nature of the inference.

Example 4: For questions 1–4 consider the following passage

I shall briefly explain how I conceive this matter. Look round the world: contemplate the whole and every part of it: You will find it to be nothing but one great machine, subdivided into an infinite number of lesser machines, which again admit of subdivisions to a degree beyond what human senses and faculties can trace and explain. All these various machines, and even their most minute parts, are adjusted to each other with an accuracy which ravishes into admiration all men who have ever contemplated them. The curious adapting of means to ends, throughout all nature, resembles exactly, though it much exceeds, the productions of human contrivance; of human designs, thought, wisdom, and intelligence. Since, therefore, the effects resemble each other, we are led to infer, by all the rules of analogy, that the causes also resemble; and that the Author of Nature is somewhat similar to the mind of man, though possessed of much larger faculties, proportioned to the grandeur of the work which he has executed. By this argument a posteriori, and by this argument alone, do we prove at once the existence of a Deity, and his similarity to human mind and intelligence.

1. Who is speaking? (Select one option.)

1. Philo
2. Cleanthes
3. Demea
4. Pamphilus.

2. What is the conclusion of the argument in this passage?

1. The universe resembles a machine made up of many smaller machines.
2. The adapting of means to ends is characteristic of human machines.
3. The Author of Nature is somewhat similar to the mind of man.
4. The universe was created by a perfect, all-knowing deity.
5. The world could not have been created randomly.

3. What is the role of the claim that the universe resembles “nothing but one great machine, subdivided into an infinite number of lesser machines” in this passage?

1. It is a premise
2. It is the conclusion.
3. It is a qualification on the conclusion.
4. It is not part of the argument at all.

4. Why does the speaker call this an a posteriori argument by analogy?

1. It can be known without any experience of the world, simply while sitting in one's chair, because it depends on an analogy between human thought and God's thought.
2. It can only be known by experience in the world, because that is the only way we can see the analogy between human thought and a machine
3. It can only be known by experience in the world, because that is the only way we can see the analogy between the world as a whole and a machine
4. It can be known without any experience of the world, because it is a regress argument.

This sequence of questions illustrates the way you can leverage asking questions about the parts of an argument to go past mere factual recall. A student who answers these questions correctly is demonstrating a thorough understanding of a pivotal passage in Hume's Dialogues. The student understands the basic claim, the reasons for that claim, and the level of support those reasons provide.
Because it is important that students know both an author's thesis and argument, some instructors create elaborate, two-tiered questions that ask about both at the same time. This is a modified version of the example Tziporah used at the AAPT talk.

Example 5: For Locke a government derives its legitimacy from which of the following?

1. The separation of the legislative, judiciary and executive functions
2. The consent of the governed
3. the coherence of civil laws with divine laws
4. The wellbeing that government contributes to its citizens' lives.

If you answered (a), circle the number next to the best description of why Locke believes that (a) is the source of a government's legitimacy.

1. Government can be legitimate only if those who serve its various functions have powers independent of one another.
2. Non-separation inevitably leads to conflict of interest and conflicting interests may lead to unfair behavior and a government that may behave unfairly is, for Locke, a government that is illegitimate
3. Legitimate political power must be spread as widely as possible and not concentrated in one function
4. All of the above

If you answered (b), circle the number next to the best description of why Locke believes that (b) is the source of a government's legitimacy.

1. Only democracies are legitimate forms of government and democracies require majority consent of the citizens
2. All persons are created equally by God, and so no one person or group may rule over another without that other&apost;s voluntary consent.
3. Those who do not vote for a government are not bound to obey it.
4. All of the above

If you answered (c), circle the number next to the best description of why Locke believes that (c) is the source of a government's legitimacy.

1. Locke believes in the supreme authority of God.
2. All political authority comes from God.
3. Even in the state of nature, God's laws determine how we should behave.
4. None of the above

If you answered (d), circle the number next to the best description of why Locke believes that (d) is the source of a government's legitimacy.

1. The only reason we want or need government is to improve our wellbeing
2. Wellbeing is the ultimate goal of all civil arrangements
3. One would not vote for a government that did not improve his or her life
4. All of the above

A two tiered question like this has the advantage of allowing you to ask about both premises and conclusions without revealing either to the student. It has a lot of disadvantages, though. It takes up a lot of space on the page and time during the test, so you can't ask too many of these. Also, if you grade it as an all or nothing question, an initial misstep could give a student a grade that doesn't reflect their real ability. After choosing a wrong initial answer, the student might waste time agonizing over a set of choices, none of which can earn her credit.

If you are testing on a computer, you can avoid this problem by simply programming the machine to show the question about Locke's reasons only if the initial question about Locke's thesis is answered correctly. If the initial question is answered incorrectly, the system can just move on to the next question. How that is scored will depend on the larger structure of the question tree. Alternately, if you wish to remain on paper, you can allow partial credit for certain paths through the subquestions. The correct answer for Example 5 is B-2, that is the student must answer answer B for the main question, and then 2 for the second question. However combination C-2 is also a plausible reading of Locke. Partial credit could be awarded for that. You could also award partial credit for combinations that are at least logical, in that the reason given in the second question actually supports the answer given in the first question, even if the first answer isn't the actual view of the author.To do this, you have to structure the distracters right for the secondary question. For instance, a secondary question for the first answer of example 5 above might look like this.

Example 6: If you answered (a), circle the number next to the best description of why Locke believes that (a) is the source of a government's legitimacy.

1. Government is weak and indecisive if all of its powers are spread out among many people
2. Non-separation inevitably leads to conflict of interest and conflicting interests may lead to unfair behavior and a government that may behave unfairly is, for Locke, a government that is illegitimate
3. In order to me legitimate, a government must have separate legislative, executive, and judicial functions.
4. One would not vote for a government that did not improve his or her life

Answers (1) and (4) are not reasons for the separation of powers. Answer (1) is a reason one might not want to separate powers. Answer (4) is a reason that would justify answer (d) in the original question. Any student who selects these answers not only does not know Locke's view, but is fundamentally confused about the issue at hand, and may just be selecting answers at random. Answer (3) is basically the same as the conclusion we are trying to justify. So if Locke did believe that the separation of powers was the key to legitimate government, and he used this reason to justify that view, he would be begging the question. A student who gives this combination of answers has enough sense of what is going on to be consistent in their mistakes, and might deserve a bit of partial credit. Finally a student who combines an initial answer of A with a subsequent answer of (2) may not have read Locke, but is at least capable of recognizing a coherent argument. We will look more at this sort of question when we come to using multiple choice questions to test higher level logical skills.

The simplest way to test for understanding of both premises and conclusions at the same time is to use a single question where each option states both a premise and a conclusion, and some answers differ only in the premise or the conclusion. For instance, the question in example 5 could be rewritten this way.

Example 7: Where does Locke believe the legitimacy of government comes from, and why? (Select one.)

1. The legitimacy of government comes from the separation of the legislative, judiciary and executive functions, because non-separation inevitably leads to conflict of interest and conflicting interests may lead to unfair behavior.
2. The legitimacy of government comes from the separation of the legislative, judiciary and executive functions, because legitimate political power must be spread as widely as possible and not concentrated in one function
3. Legitimacy of government comes from the the consent of the governed, because all persons are created equally by God, and so no one person or group may rule over another without that other's voluntary consent.
4. Legitimacy of government comes from the the consent of the governed, because those who do not vote for a government are not bound to obey it.
5. Legitimacy of government comes from the coherence of civil laws with divine laws, because Locke believes in the supreme authority of God.
6. Legitimacy of government comes from the coherence of civil laws with divine laws, because all political authority comes from God.
7. Legitimacy of government comes from the wellbeing that government contributes to its citizens' lives, because wellbeing is the ultimate goal of all civil arrangements.
8. Legitimacy of government comes from the wellbeing that government contributes to its citizens' lives, because one would not vote for a government that did not improve his or her life.

So, we can use multiple choice questions to ask about premises, conclusions, and inferences, and and we can ask about a couple of these at the same time. This lets us evaluate student's understanding of they core philosophical content of any text. We can also use multiple choice questions to get students to push deeper into issues than they have gone in the reading. In doing this start evaluating higher level logical skills.

One simple way to do this is to ask questions that introduce objections to arguments the students have already seen. For instance, when I teach the The Crito I have a hand out which puts the central argument of the dialogue in cannonical form. On the test, I might repeat this schematization and then ask students to evaluate new objections to it, as in the example below.

Example 7. Consider this argument from your reading.

1. One must never do injustice (49b6)
2. Therefore, one must not return injustice with injustice. (49b8)
3. Therefore, if one has a contract, one must not break it, even if the other party has wronged you. (49e.)
4. Every citizen makes a contract with the laws of the city to obey those laws. This duty is stronger even than duty to parents (50b).
5. Therefore, one must obey the laws of the city, even if the city has wronged you. (51b)

---

6. Therefore, Socrates should stay in prison. (51c)

Which of the following objections Crito might make to Socrates's argument? (Check all that apply.)

1. In fact, one should do injustices sometimes, and cases where one has been done an injustice are a good example of this.
2. Even if one must never return an injustice for an injustice, it doesn't follow that one must always carry out a contract, because the contract could be voided with the other side breaks it.
3. If Socrates runs away, it will be a violation of everything he's stood for.
4. Socrates has no reason to fear death, so he has no reason to avoid execution.

Example 7 is fairly simple. The first two items are objections to the argument, one challenging the truth of a premise and one challenging the strength of an inference. The second two choices are not objections to the argument. They are further statements in support of the argument.

If your course covers more aspects of critical thinking, you can add interesting variations to this question. For instance, if your course emphasizes the difference between objections directed at an argument and independent reasons to doubt the conclusion of an argument, you could add the option "(e) If Socrates allows himself to be executed, his children will grow up as orphans." If you have made it sufficiently clear that independent arguments against the conclusion do not count as "objections to an argument," this would then be a wrong choice that many students would be very tempted to circle. Alternately you could ask a question that asks students to identify exactly how a different responses to this argument work, as in this example.

Example 8: Consider the following list of possible replies Crito or someone else might make to Socrates's main argument in the Crito, outlined above

1. In fact, one should do injustices sometimes, and cases where one has been done an injustice are a good example of this.
2. Even if one must never return an injustice for an injustice, it doesn't follow that one must always carry out a contract, because the contract could be voided with the other side breaks it.
3. If Socrates runs away, it will be a violation of everything he's stood for.
4. Socrates has no reason to fear death, so he has no reason to avoid execution.
5. If Socrates allows himself to be executed, his children will grow up as orphans.

Mark each of the replies with one of the labels below, depending on what kind of reply it is. Some labels will be used twice.

TP: This is a challenge to the truth of one of the premises.
SI: This is a challenge to the strength of one of the inferences.
IND: This is an independent argument against the truth of the conclusion
NC: This is not challenge at all to the argument.

Questions like this allow us to push very deep into the logic of a philosophical argument, while retaining the ability to grade answers quickly, mechanically, and objectively.
I could go on with more examples, and in fact in a very short time I will. I've been asked to write an article on multiple choice questions for newsletter of the American Association of Philosophy Teachers. This post is a draft of part of it, and more will be coming soon.