Andre Leonard: "Lego Robots Ate My Son"

Andrew Leonard is the dad in a father-son lego team. His boy is 16 now, about about to go off to college to study robotics. Is this Joey in 8 years?

Also, there is a new edition of Mindstorms (EV3) coming out. Like Leonard, I started playing with EV1, first with Caroline, then with Joey. The best robot we built was "Fast Phillip," made with two EV1 bricks. Fast Phillip would barrel full steam at a wall, hit it, turn around, and zoom for the next wall. It generally survived three collisions before falling apart. We never did much with Mindstorms EV2, mostly because I never found a command for the brick that just said "Go as fast as you can as far as you can."

Joey and I made a little stop motion lego movie.

Star Trek: Into Darkness, with Spoilers

Misc thoughts, with spoilers on ST: Into Darkness

• With all the fan servicing in this installment, why not have Pike end up stuck in box only able to communicate "yes" and "no" with a single beeping light? 
• Holy shit was there a lot of fan servicing. Do you really want a jokey reference to a previous movie at the big climactic moment where the main character appears to die?
• If the final face off between the Enterprise and the Vengeance is right next to Earth, why don't any other Federation ships intervene?
• Ms. Day is extra correct when she says "seriously, in the future not one woman over 40 is in charge in this world?!  How can that happen?" The plot pivots around Kirk, his commander Chris Pike, and Pike's commander Marcus. Someone in this hierarchy could have been female. You could do this either by making Marcus female--imagine if Carol Marcus had played this role in this timeline--or by not having Pike continue to be Kirk's immediate superior.
• Chris G., somewhere I can't link to directly, notes that there is really no reason why old-Spock should be all cryptic about what he knows from the alternative timeline.
• Few people, writers or fans, appreciate the Prime Directive. It is not an arbitrary rule devised to create moral dilemmas for the characters. It is also not some weird artefact of a show that first appeared during the cold war. It is a hedge against imperialism. For once, our explorers are actually noble. They are not bent on conquest. They are not interested in finding proxies for conflicts with rival empires. This will continue to be relevant as long as there are empires.  
• Why is it called Into Darkness anyway? 

One for the bioethics section on "naturalness"

A couple is suing on behalf of their adopted child who was born intersex, because doctors decided to assign the gender "female" to the child when he was 16 months old, before they had custody of the child. The child was judged male at birth, but the doctors changed their minds at 16 months and went with female. He now identifies as male and the suit alleges that he was "a true hermaphrodite" at birth.

Decades ago, doctors would have "corrected" the child's genitalia without ever notifying the parents or even leaving any record of the procedure, because ambiguous genitalia were deemed shameful. The Advocate says doctors are currently encouraged assign a gender at birth but "to hold off on any unnecessary surgery until they are old enough to self-identify with a gender." The fact that doctors are now being sued for what used to be standard operating procedure is a sign of progress.

I'd actually like to know what the child's exact intersex condition is. (A variety of chromosomal and hormonal factors can lead to intersexuality.) This has no bearing on the merits of the case. I'm just nosy.

Association, perceptions of relevance, and common sense

So we are doing machine moral status today, and I start out with oblique question. "True or false: What is important about me isn't how I look on the outside, but what goes on in my spirit." After some discussion of this statement, we talk about whether a machine could ever legitimately demand rights or could ever be held responsible for an action.

I then ask the students how they think my first question, about appearances, is related to the second. The first student raises her hand and says “Well you know, they guys who work on robots, they don’t look too good, because all they care about is work.”

How does one’s brain come to work like that, to think that was the answer I’d be going for? This is entirely a matter of association, perceptions of relevance, and common sense. It seems like it should be obvious that I am concerned with whether outward differences in machines and people could be unimportant compared to mental properties. But that wasn’t obvious to this student. What popped into her head was that computer guys are slobs.

How do you teach someone to think differently at this level?

Caroline rejects the difference principle

Joey got to play video games while Caroline was at ice skating. So Caroline thinks that she should get an equal amount of video game time, and Joey can't be allowed to play video games during that time. He can't even watch her play, because then he would have more video game time than her. I try to reason with her. Me: "Suppose you have two options. In one, both children get one cookie. In the other, once child gets two cookies, and the other gets three. Which is the better option?" Caroline [shouting]: the one where everyone gets THE SAME. Everyone should get THE SAME. Me: But you would get more in the other option. Caroline: He has to get the same as me! You only let him get more than me because you like him better! Me: But that's not rational!

History of D&D: The Primary Sources

These guys sell .pdf files of many of the millions and millions of Dungeons & Dragons products over the decades. The game I played was Advanced D&D, which apparently ran from 1977-1979. This was definitely one of the books I had, as was this, but I can't find of the others.

Cognitive Science and the bad logic student

This article is an opener for a conversation I've been wanting to have for a while. Hutchins suggests that the failure of some otherwise bright, hardworking students to do well in logic is due to a heretofore unknown learning disability she calls "inference blindness." I am quite convince that there is a class of students who do poorly in logic classes which cannot be attributed to general problems in learning, like a lack of effort or general study skills. Normally the poor performance of these students is attributed to some sort of math-phobia. Hutchins doesn't consider this possibility, which I'm ok with, because I don't think math-phobia can account for all of the phenomenon we are looking at. This leaves us with a substantial body of students whose poor performance we don't have a good explanation of. I would really, really like an understanding of these students that is grounded in some real cognitive science.

Hutchins' proposal doesn't go far at all in this direction. She doesn't so much ground her hypothetical in the cognitive science as offer one way a learning disability like inference blindness might be realized in one model of the neurology of inferential reasoning. She does not say that the model predicts the existence of inference blindness, nor does she say that putting inference blindness in the context of this model leads to some other prediction. The model in question is Vinod Goel's dual mechanism theory, which says that we reason about logical connections using a combination of two different systems. One is a visual-spacial system that is used in unfamiliar or highly abstract situations. In terms of hardware, this systems is based on a bilateral occipital-parietal-frontal network. The other system is a linguistic system based in the left frontal and temporal lobes and comes on line when logical problems are placed in concrete, familiar situations. Part of the advantage of this model is that it explains performance on the Wason Selection Task. Subjects do better when reasoning about the conditional "If you are drinking alcohol then you must be over 21" because the second system is brought online. Hutchins thinks inference blindness, if it exists, would be cause by a neurological deficit, perhaps marked by real physical lesions, which prevents the linguistic system from coming to the aid of the spatial system. I'm not sure why she wants the deficit to be specifically located in the connection between the two systems. Why not say that the students in question have a weakness in just the abstract spatial system? We know this system is weak in everyone, but it is still possible for their to be lots of individual variation. Alternatively, why not say that the deficit is just in the verbal system? Perhaps there are many distinct kinds of deficits out there?

Another problem with the proposal is the use of the language of disability and talk of specific physical lesions. For starters, it creates the risk that the class of students we wind up identifying is too narrow to be of any use to us. In my fourteen years of teaching logic, I have had precisely one student whom I could confidently say struggled with logic because of a neurological deficit. He had been the victim of a gunshot wound to the head. The doctors had initially said he would never walk or talk again. As it turned out, he could do both, and was one course away from graduating college. Unfortunately, that course was not going to be symbolic logic.

Of course, not all learning disabilities wind up picking out a very narrow class of people. Experience with disabilities like ADHD and dyslexia show that once a learning disability gets picked up by the media, its definition can be blurred to the point of complete meaninglessness. But this is also a bad outcome, if what we are seeking to discover is a substantial body of students that can be helped with a concrete intervention. When I taught at an extremely elite private school (which we will call Stuffwhitepeople Like University) huge numbers of students, sometimes as much as a third of the class, would come to me with documentation of a learning disability that would give them accommodations like time and half on tests. None of these students ever needed the accommodation. As near as I can tell, they had just learned how to game the system to get every advantage for themselves. Their training in this probably began in infancy, while their parents were intently reading New York Times articles on how to get your kids into the "right" preschool. These days I teach at a community college which serves a lot of severely underprivileged students and I am also seeing a lot of students who deserve and would benefit from the kinds of accommodations that come with a diagnosis of ADHD. Only very rarely do I get students who will ask for accommodation, can document their disability, and be able to get the help they need.

Rather than talking about disability here, I think it would be more productive to talk about a subpopulation of normal students who do poorly in logic for reasons not related to general scholastic ability or some kind of math-phobia. This will allow us to look for a class of students that is larger than the topic of a single case study of a neurological deficit, while avoiding the extensive political complications that come the language of learning disabilities. To be clear, its not that I doubt reality of learning disabilities. I just think that the way the American educational system addresses learning disabilities is hopelessly dysfunctional.

There are other issues that come up in deciding what kind of class of student we should be looking for. I've already said I'm interested in a broader set of students than just the handful that can be categorized as having a physical brain lesion. Another question is whether we want to specifically target students who don't improve their performance when familiar contextual information is added. Hutchins' idea for Inference Blindness was that it involved the failure of the verbal system to help the visual-spatial system. This means that we are looking at people who continue to do badly on the Wason Selection Task even after concrete details are added. This is definitely an interesting class of people, but are they the ones we want to intervene with in our classes? If the alternative is simply looking at students who do badly at the abstract version fo the WST and improve on the concrete version, then the answer is "yes," because the latter category is basically everyone.

The next step in this conversation is to look for, or develop, correlation studies. It would be interesting to see, for instance, whether doing well in logic without extra help correlates with doing well in the abstract version of the Wason Selection Task. Similarly, it would be interesting to see if students who do poorly even with extra help also fail to respond to the addition of concrete detail in the WST. It may be hard to identify correlations here, though, simply because success in a symbolic logic course takes you into realms of abstraction far beyond recognizing the truth conditions for a single if...then statement. Also, even though undergraduates are typical subjects for psychological experiments, actual empirical work on the process of education always seems to run into many more difficulties. It is easy enough to set up a single psych test for undergraduates to take, but tracking them through a full course introduces all sorts of other confounds.

The Helpy-Chalk 2012 endorsements

The easy stuff

For President: Barack Obama
For Vice President: Joe Biden

For US Senator: Sherrod Brown

For Representative to Congress (9th district): Marcy Kaptur

For State Senator (24th district): Jennifer Brady

For State Representative (16th district): Andrew Meyer

For Prosecuting Attorney: Timothy McGinty.

Harder stuff

For member of the State Board of Education (5th district)

Our choices are Marianne Gasiecki, Rich Javorek and Bryan C. Williams.

The Ohio State Board of Education consists of 19 members, 11 of whom are elected and 8 of whom are appointed. They set K12 policy. District 5 includes all the parts of Cuyahoga County that aren't Cleveland. Our current guy is Robin Hovis, who was originally appointed by Disgraced Republican Governor Bob Taft.

State Impact Ohio gives quick profiles of all the candidates which tell me what I need to know. Gasiecki is a Republican who says Republican things. Javorek is a Democrat who says Democratic things. Williams couldn't get his act together enough to respond to an inquiry from a major news organization.

The Winner: Rich Javorek

Judges Races

Voting for judges is stupid. Whenever someone asks me why voting for judges is stupid I point to this New York Times story about the way Ohio Supreme Court rulings mirror campaign contributions. (Actually no one ever asks me why voting for judges is stupid, which is too bad.)

For Justice of the Supreme Court, race 1

Terrence O'Donnell. (R) v. Mike Skindell (D)

And look who shows up in the first judge's race this cycle: its the poster boy from the above NYT article, Terrence O'Donnell. That article reports that he ruled in favor of his campaign  contributors 91% of the time

The challenger in this race is Mike Skindell. He does not rate as well on the normally reliable  Judge4yourself, but judge4yourself does not work as well on high-stakes partisan elections.

Endorsement: Mike Skindell.

For Justice of the Supreme Court, race 2

Robert R. Cupp (R) vs. William M. O'Neill (D).

O'Neill is running on the promise to end to end bribery at the supreme court level and is refusing all campaign contributions. He did this in 2006 in a race against O'Donnell and lost by 20% of the vote, which is I suppose what happens when you don't raise money and your opponent does.

Endorsement: William M. O'Neill.

For Justice of the Supreme Court, race 3

Sharon Kennedy (R)  vs. Yvette McGee Brown (D)

This is a partisan election, even if the party affiliations aren't on the ballot. Everyone should vote Democratic.

Endorsement: Yvette McGee Brown

For Judge of the Court of Appeals (8th District)

Tim McCormack
Kenneth R. Spanagel

Equal ratings on Judge4Yourself, with McCormack doing better with the general bar, and Spanagel doing better with the criminal defense bar and winning the Plain Dealer endorsement.

But McCormack is the democrat and Spanagel is a Republican, and really this is a partisan election.

Endorsement: Tim McCormack.

For Judge of the Court of Common Pleas (General Division)

Marilyn B. Cassidy
Michael Jackson

Both get good ratings from Judge4Yourself.

Jackson is the Democrat.

Endorsement: Jackson.

For Judge of the Court of Common Pleas (General Division)
Daniel Gaul
Edele Passalacqua

Gaul's ratings are good to excellent on Judge4yourself, while Passalacqua's are adequate to good.

Passalacqua switched parties to Republican to challenge Gaul, after Gaul violated judicial ethics with an outburst in court. Gaul's high ratings at judge4yourself show that the legal establishment--and the Plain Dealer--are still behind him.

Endorsement: Gaul

For Judge of the Court of Common Pleas (General Division)
Kathleen Ann Sutula
Dean W. Van Dress

Sutula is decisively favored by all the groups reporting to judge4yourself. She's a Republican, but her opponent is a city councilman with no legal experience whatsoever.

Endorsement: Sutula

For Judge of the Court of Common Pleas (General Division)
Cathryn R. Ensign
Shirley Strickland Saffold

Strickland Saffold made headlines when it was discovered that she had posted all sorts of insulting and prejudicial comments to Cleveland.com under the pseudonym "lawmiss," including the remark that one of the lawyers appearing before her should "shut his Amos and Andy style mouth." When that same 'Amos and Andy' lawyer came before her court again as a part of the Anthony Sowell murder trail, she was removed from the case. Strickland Saffold is now refusing to participate in the judge4yourself process. Ensign, a Republican, is running against Strickland Saffold "at least in part because the judge missed a pre-trial conference that Ensign's client had traveled from Columbus to attend."

Endorsement: Ensign.

For Judge of the Court of Common Pleas (General Division)

Cassandra Collier-Williams
Joan Synenberg

Republican Synenberg gets top marks across the board from judge4yourself, while Collier-Williams low marks, including a dismal "not recommended" from the Ohio Women's Bar. Synenberg gets special commendation for her handling of the case of Joe D'Ambrosio, who was acquitted after spending 21 years on death row for a murder he didn't commit. Synenberg stood up to prosecutors, who, it turned out, had withheld evidence.

Endorsement: Synenberg

For Judge of the Court of Common Pleas (General Division)
Pamela A. Barker
Colleen Ann Reali

The various legal bodies at Judge4yourself give Barker near perfect ratings, while Reali's are as weak as they come. Barker is the Republican, but this far down the ticket, competence trumps partisanship quite handily.

Endorsement: Barker

For Judge of the Court of Common Pleas (General Division)
Robert C. McClelland
Cullen Sweeney

Sweeney changed his name to run for this office. That should decide things right there. McClelland is also higher ranked on judge4yourself. On the other hand, McClelland has not been transparent about campaign contributions and is a Republican. Name changing and secret campaign contributions are pretty much the two worst things about electing judges in Ohio. But McClelland has the backing of the people who actually have to work in the legal system.

Endorsement: McClelland

For Judge of the Court of Common Pleas (General Division)
Annette G. Butler
Steve Gall

Gall is ranked higher at judge4yourself, and Butler actually received a "not recommended" from the Ohio Bar. Butler is a Republican who was "shocked" to discover that the male criminal defendants before her had a very low level of education. It is not known whether she clutched pearls while saying this.

Endorsement: Gall

For Judge of the Court of Common Pleas (Juvenile Division) 
Frankie Goldberg
Denise Nancy Rini

Rini has a slight edge with the legal establishment at judge4yourself.com and the endorsement of the Plain Dealer. But Goldberg is a Democrat running on a reform platform and touting her connections with popular Democratic figures like Martha Fudge. Goldberg is in the race because she challenged and beat Joseph F. Russo a corrupt drunk who has a last name that appears often in connection with corrupt Cuyahoga county Democratic politics.

Endorsement: Goldberg

For Judge of the Court of Common Pleas (Juvenile Division) 
Michael John Ryan
Anjanette A. Whitman

Ryan is a Democrat with a decisive lead in the ratings at Judge4yourself.

Endorsement: Ryan

State Issues

State Issue 1: Constitutional Convention?

Every 20 years, the state constitution says there has to be a vote on whether there needs to be a new constitutional convention. No one seems to be calling for one this time around.

Endorsement: No

State Issue 2: An independent redistricting council.

Someone called redistricting "The most fun you can have in politics without going to jail." In fact partisan redistricting in Ohio is a nightmare. Since both sides redistrict in their favor when they are in power, the resulting system mostly protects incumbents in general. The new commission would strive to make all districts to be competitive. The only objection to the proposal has been that an unelected body is not "accountable." But we have seen that when it comes to redistricting, the elected bodies are only accountable to their donors and powerbases.

Listen to the debate on this issue at the City Club of Cleveland.

Endorsement: Yes

Bay Village Issues

Issues 3, 4 and 5:

The Republican mayor of Bay is laying the groundwork here to merge city services, including police and fire, with other services in the region. The move is opposed by the police and firefighters unions. Their campaign only mentions issues 4 and 5, and not 3. Perhaps this is because 3 primarily impacts the police and fire chiefs, who count as management.

In any case Bay Village is rich, and doesn't seem to need to save money here. As long as people are willing to buy those jillion dollar lakefront houses, I think there is room for more taxes on the very wealthy.

Endorsement: No.


Issue 6: PROPOSED CHARTER AMENDMENT CITY OF BAY VILLAGE

The proposed change simply says "Shall article XIII section 13.3 Disqualification of the charter of the City of Bay Village, regarding the disqualification of certain city officials, be amended."

This looks like the ordinance that put the issue on the ballot. It contains this text, which seems to be the full text that will be added to the charter:

SECTION 13.3 DISQUALIFICATION.
No Councilman, Mayor, Director of Finance or any other officer, employee or appointee to any board or commission of the Municipality shall cast a vote or approve a contract that he may directly or indirectly be financially interested in any contract, job, work or service with or for the Municipality, nor in the profits or emoluments thereof, nor in the expenditure of any money on the part of the Municipality other than in his fixed compensation and expenses; and any contract with the Municipality in which any officer or employee is or becomes financially interested may be declared void by the Council. Any such individual who takes an action that violates this section shall be disqualified from holding such office or position in addition to being subject to any other penalties provided by law.

This looks like an ordinary measure to prevent sweetheart deals and selling government contracts.

The ordinance also contains this language, which is interesting.

That this ordinance is hereby declared to be an emergency measure immediately necessary for the preservation of the public peace, health, safety and welfare, and for the further reason that it is immediately necessary to place said proposed amendment on the ballot in a timely fashion so the electors may decide the question proposed, wherefore this ordinance shall be in full force and take effect immediately upon its passage and approval by the Mayor.

They may just have to use language like this to expedite the ballot issue. On the other hand, they may be planning on kicking someone out of the government as soon as this is passed. Maybe if I hung out more with the neighborhood gossips I would know these things.

I can't find the full charter for the City on the City's website. Someone should complain about that.

Endorsement: Yes

105:  PROPOSED BOND ISSUE BAY VILLAGE CITY SCHOOL DISTRICT

Bonds for the schools, with a levy to pay for the "debt charges" on the bonds.

Always vote for the schools.

Endorsement: Yes

Cuyahoga County Issues

108 PROPOSED TAX LEVY (REPLACEMENT AND INCREASE) CLEVELAND-CUYAHOGA
COUNTY PORT AUTHORITY

The port authority does not seem to be well managed, and I think was involved somehow in the recent cluster of corruption cases which led to the downfall of local politicians and a restructuring of government. Still, you need the government to fund economic development. I'd rather reform the system than defund it.

Endorsement: Yes

117 PROPOSED CHARTER AMENDMENT COUNTY OF CUYAHOGA

They want to switch to a 2 year budget cycle.


Endorsement: Yes

118 PROPOSED CHARTER AMENDMENT COUNTY OF CUYAHOGA


They want to stagger the terms of the county auditors and the count executive.

Endorsement: Yes

Vote for William O'Neill, for The Ohio Supreme Court

Ohio voters, end bribery at the Ohio Supreme Court! Vote for William O'Neill, who is refusing all campaign contributions. His opponent, Robert Cupp receives huge contributions from people whose cases he will later rule on. Ohio Supreme Court justices do not recuse themselves from cases involving campaign contributors. 

http://www.oneillforjustice.com/

For Justice of the Supreme Court: Mike Skindell

Hey Ohio Voters,

This election incumbent Republican State Supreme Court Justice Terrence O'Donnell is being challenged by Democrat Mike Skindell. You might remember O'Donnell from this scathing exposé from the New York Times about corruption on the state's high court. Ohio Supreme Court elections are big money affairs, and Supreme court judges don't recuse themselves when major campaign donors appear before them. Instead, they just rule in favor of the people who gave them big money. O'Donnell is the worst offender, ruling for the people who gave him money 91% of the time.

The normally reliable Judge4yourself gives O'Donnell a higher rating. In this case, you shouldn't listen to them. They are aggregating recommendations from Bar Associations and newpapers, which aren't good sources for partisan elections at this high level.

Egg freezing--cross posted from my bioethics class.

The American Society for Reproductive Medicine (ASRM) has officially declared that the process of freezing human eggs for later use is no longer experimental. You can check out the story on NPR about it here. In particular ASRM found that success rate for older women using eggs frozen from younger women was as high as the success rate for young women using IVF. This means that more women will be freezing their eggs when they are young in case they want to get pregnant when they are old.

The NPR piece I linked to includes several quotes from people who are skeptical of this technique. One that was interesting to me was a comment from Adrienne Asch at Yeshiva University. She seems to say that society should make it easier for women to have children when they are young, rather than creating technologies that allow women to have children when they are old. "It's an example of using technology to solve social problems," she says.

If some women decide not to have children when they are young, because they cannot find a partner or want to pursue a career, is it really a social problem? Conversely, would it be a social problem if a lot of women starting having children in their 40s and 50s, after they have established careers?

Genocidal rhetoric in China

I'm not sure what to make of the recent wave of anti-Japanese sentiment in China. The good folks at Language Log have some interesting posts up about a slogan/poem now appearing all over China calling for the extermination of all Japanese.

Even if the whole of China is covered with tombs, [we] must kill all Japanese; even if no grass grows in China, we must recover Diaoyudao [the Senkakus]

The Diaoyuadao are the disputed islands in the East China Sea. Here is a discussion of the poem. Here is a discussion of how literally we should take such rhetoric, including this comment.

Since Mao, words like xiāomiiè 消灭 ("eliminate; extirpate") have crept further into daily-life Chinese than they had ever been before, and in that sense they are not literal. But the very "normalization" of bloodthirsty language probably makes violence more possible, too.

I've been thinking we need to update the old Tom Lerher song, National Brotherhood Week: Oh the Christians, hate the Muslims, and the Muslims, hate the Hindus and the Chinese hate the Japanese, and everybody hate the Jews. I'm not sure who Joe Arpaio should be dancing cheek to cheek with, though.

 

More on Multiple Choice Questions

Emily asked me to say more about using multiple choice questions to evaluate more advanced levels of skill and comprehension. I had posted my notes from this talk by Tziporah Kasachkoff and Nils Rauhut on multiple choice questions at the AAPT. Tziporah and Nils discussed using multiple choice questions for three purposes. The first was for simple "did you do the reading" quizzes. These are simple fact recall questions which even critics of multiple choice testing acknowledge are useful. The second two, however, were testing for higher levels of reading comprehension, and testing for key logical skills These were the uses Emily wanted to know more about.

To my mind, the key to the sophisticated use of multiple choice questions is to recognize that multiple choice question can be used to identify all parts of an argument, and this can be leveraged to evaluate students on the educational goals named at all levels of Bloom's taxonomy. If you teach logic or critical thinking you are probably already familiar with this at a basic level. You can give the students a simple three ore four sentence argument and ask them to identify the conclusion. The example is written in a standard form used in critical thinking texts, where the part of the passage in italics provides context for understanding the argument given by the speaker in the remainder of the passage.

Example 1: Consider the following passage

Susan tells her friend Jill about a theory she's developed. I think your husband is having an affair. Think about it. He has been working late a lot recently, and you found strange lipstick in his coat pocket. Also, he seems very chipper and has been weirdly nice to you.

Which sentence represents the conclusion of this passage? (Select one option.)

1. Your husband is having an affair.
2. Your husband has been working late a lot recently
3. Think about it.
4. You found strange lipstick in your husband's coat pocket
5. Your husband seems very chipper and has been weirdly nice to you.

The same passage can be used to identify a premise, for instance like this.

Example 2: The sentence “He has been working late a lot recently” functions as what in this passage? (Select one option.)

1. a conclusion
2. a premise
3. an indicator phrase
4. a thesis

You can also write multiple choice questions that focus on the inferential relationship itself, rather than just premises or conclusions, as in this example.

Example 3: Consider the following pair of sentences

1. Fido is a dog
2. FIdo is a mammal

Which of the following best describes the relationship between the pair of sentences. (Select one option.)

1. If (1) is true, then (2) is true.
2. If (2) is true, then (1) is true
3. Sentences (1) and (2) are either both true or both false.

These examples are simple, but they illustrate the fact that you can use multiple choice questions to gage comprehension of every part of an argument. This is something you can build on in order to evaluate the skills Tziporah and Nils were concerned with: higher levels of reading comprehension and higher logical skills.

Reading comprehension is no small issue to be delegated to primary and secondary schools. This becomes clear in the philosophy, just because so many of the texts we deal with are so very difficult. But even for a general college education, giving students the ability to properly digest complex information is crucial. It is one of the big skills tested in the Collegiate Learning Assessment, which is probably the best measurement of critical thinking out there. The CLA is also, importantly, a test that college students don't show much improvement on during their time in school.

Being able to identify an author's point is a basic reading comprehension skill, and one that most people don't get the opportunity to practice, because so many writers in the popular media never bother to have a point. In philosophy, as in many disciplines, the author's basic point is the conclusion of her argument. Questions that ask you to identify the conclusion of a passage can be scaled up to ask students about the conclusions of assigned readings. Now, of course, it is not just enough to know an author'point. We also want the students to know why the author thinks we should believe her claims. And so we can also employ questions that ask about premises. The next example is from a test given at the end of a unit on Hume's Dialogues Concerning Natural Religion. It features one target passage, and then questions that measure simple factual recall, recognition of conclusions, recognition of premises, and an understanding of the nature of the inference.

Example 4: For questions 1–4 consider the following passage

I shall briefly explain how I conceive this matter. Look round the world: contemplate the whole and every part of it: You will find it to be nothing but one great machine, subdivided into an infinite number of lesser machines, which again admit of subdivisions to a degree beyond what human senses and faculties can trace and explain. All these various machines, and even their most minute parts, are adjusted to each other with an accuracy which ravishes into admiration all men who have ever contemplated them. The curious adapting of means to ends, throughout all nature, resembles exactly, though it much exceeds, the productions of human contrivance; of human designs, thought, wisdom, and intelligence. Since, therefore, the effects resemble each other, we are led to infer, by all the rules of analogy, that the causes also resemble; and that the Author of Nature is somewhat similar to the mind of man, though possessed of much larger faculties, proportioned to the grandeur of the work which he has executed. By this argument a posteriori, and by this argument alone, do we prove at once the existence of a Deity, and his similarity to human mind and intelligence.

1. Who is speaking? (Select one option.)

1. Philo
2. Cleanthes
3. Demea
4. Pamphilus.

2. What is the conclusion of the argument in this passage?

1. The universe resembles a machine made up of many smaller machines.
2. The adapting of means to ends is characteristic of human machines.
3. The Author of Nature is somewhat similar to the mind of man.
4. The universe was created by a perfect, all-knowing deity.
5. The world could not have been created randomly.

3. What is the role of the claim that the universe resembles “nothing but one great machine, subdivided into an infinite number of lesser machines” in this passage?

1. It is a premise
2. It is the conclusion.
3. It is a qualification on the conclusion.
4. It is not part of the argument at all.

4. Why does the speaker call this an a posteriori argument by analogy?

1. It can be known without any experience of the world, simply while sitting in one's chair, because it depends on an analogy between human thought and God's thought.
2. It can only be known by experience in the world, because that is the only way we can see the analogy between human thought and a machine
3. It can only be known by experience in the world, because that is the only way we can see the analogy between the world as a whole and a machine
4. It can be known without any experience of the world, because it is a regress argument.

This sequence of questions illustrates the way you can leverage asking questions about the parts of an argument to go past mere factual recall. A student who answers these questions correctly is demonstrating a thorough understanding of a pivotal passage in Hume's Dialogues. The student understands the basic claim, the reasons for that claim, and the level of support those reasons provide.
Because it is important that students know both an author's thesis and argument, some instructors create elaborate, two-tiered questions that ask about both at the same time. This is a modified version of the example Tziporah used at the AAPT talk.

Example 5: For Locke a government derives its legitimacy from which of the following?

1. The separation of the legislative, judiciary and executive functions
2. The consent of the governed
3. the coherence of civil laws with divine laws
4. The wellbeing that government contributes to its citizens' lives.

If you answered (a), circle the number next to the best description of why Locke believes that (a) is the source of a government's legitimacy.

1. Government can be legitimate only if those who serve its various functions have powers independent of one another.
2. Non-separation inevitably leads to conflict of interest and conflicting interests may lead to unfair behavior and a government that may behave unfairly is, for Locke, a government that is illegitimate
3. Legitimate political power must be spread as widely as possible and not concentrated in one function
4. All of the above

If you answered (b), circle the number next to the best description of why Locke believes that (b) is the source of a government's legitimacy.

1. Only democracies are legitimate forms of government and democracies require majority consent of the citizens
2. All persons are created equally by God, and so no one person or group may rule over another without that other&apost;s voluntary consent.
3. Those who do not vote for a government are not bound to obey it.
4. All of the above

If you answered (c), circle the number next to the best description of why Locke believes that (c) is the source of a government's legitimacy.

1. Locke believes in the supreme authority of God.
2. All political authority comes from God.
3. Even in the state of nature, God's laws determine how we should behave.
4. None of the above

If you answered (d), circle the number next to the best description of why Locke believes that (d) is the source of a government's legitimacy.

1. The only reason we want or need government is to improve our wellbeing
2. Wellbeing is the ultimate goal of all civil arrangements
3. One would not vote for a government that did not improve his or her life
4. All of the above

A two tiered question like this has the advantage of allowing you to ask about both premises and conclusions without revealing either to the student. It has a lot of disadvantages, though. It takes up a lot of space on the page and time during the test, so you can't ask too many of these. Also, if you grade it as an all or nothing question, an initial misstep could give a student a grade that doesn't reflect their real ability. After choosing a wrong initial answer, the student might waste time agonizing over a set of choices, none of which can earn her credit.

If you are testing on a computer, you can avoid this problem by simply programming the machine to show the question about Locke's reasons only if the initial question about Locke's thesis is answered correctly. If the initial question is answered incorrectly, the system can just move on to the next question. How that is scored will depend on the larger structure of the question tree. Alternately, if you wish to remain on paper, you can allow partial credit for certain paths through the subquestions. The correct answer for Example 5 is B-2, that is the student must answer answer B for the main question, and then 2 for the second question. However combination C-2 is also a plausible reading of Locke. Partial credit could be awarded for that. You could also award partial credit for combinations that are at least logical, in that the reason given in the second question actually supports the answer given in the first question, even if the first answer isn't the actual view of the author.To do this, you have to structure the distracters right for the secondary question. For instance, a secondary question for the first answer of example 5 above might look like this.

Example 6: If you answered (a), circle the number next to the best description of why Locke believes that (a) is the source of a government's legitimacy.

1. Government is weak and indecisive if all of its powers are spread out among many people
2. Non-separation inevitably leads to conflict of interest and conflicting interests may lead to unfair behavior and a government that may behave unfairly is, for Locke, a government that is illegitimate
3. In order to me legitimate, a government must have separate legislative, executive, and judicial functions.
4. One would not vote for a government that did not improve his or her life

Answers (1) and (4) are not reasons for the separation of powers. Answer (1) is a reason one might not want to separate powers. Answer (4) is a reason that would justify answer (d) in the original question. Any student who selects these answers not only does not know Locke's view, but is fundamentally confused about the issue at hand, and may just be selecting answers at random. Answer (3) is basically the same as the conclusion we are trying to justify. So if Locke did believe that the separation of powers was the key to legitimate government, and he used this reason to justify that view, he would be begging the question. A student who gives this combination of answers has enough sense of what is going on to be consistent in their mistakes, and might deserve a bit of partial credit. Finally a student who combines an initial answer of A with a subsequent answer of (2) may not have read Locke, but is at least capable of recognizing a coherent argument. We will look more at this sort of question when we come to using multiple choice questions to test higher level logical skills.

The simplest way to test for understanding of both premises and conclusions at the same time is to use a single question where each option states both a premise and a conclusion, and some answers differ only in the premise or the conclusion. For instance, the question in example 5 could be rewritten this way.

Example 7: Where does Locke believe the legitimacy of government comes from, and why? (Select one.)

1. The legitimacy of government comes from the separation of the legislative, judiciary and executive functions, because non-separation inevitably leads to conflict of interest and conflicting interests may lead to unfair behavior.
2. The legitimacy of government comes from the separation of the legislative, judiciary and executive functions, because legitimate political power must be spread as widely as possible and not concentrated in one function
3. Legitimacy of government comes from the the consent of the governed, because all persons are created equally by God, and so no one person or group may rule over another without that other's voluntary consent.
4. Legitimacy of government comes from the the consent of the governed, because those who do not vote for a government are not bound to obey it.
5. Legitimacy of government comes from the coherence of civil laws with divine laws, because Locke believes in the supreme authority of God.
6. Legitimacy of government comes from the coherence of civil laws with divine laws, because all political authority comes from God.
7. Legitimacy of government comes from the wellbeing that government contributes to its citizens' lives, because wellbeing is the ultimate goal of all civil arrangements.
8. Legitimacy of government comes from the wellbeing that government contributes to its citizens' lives, because one would not vote for a government that did not improve his or her life.

So, we can use multiple choice questions to ask about premises, conclusions, and inferences, and and we can ask about a couple of these at the same time. This lets us evaluate student's understanding of they core philosophical content of any text. We can also use multiple choice questions to get students to push deeper into issues than they have gone in the reading. In doing this start evaluating higher level logical skills.

One simple way to do this is to ask questions that introduce objections to arguments the students have already seen. For instance, when I teach the The Crito I have a hand out which puts the central argument of the dialogue in cannonical form. On the test, I might repeat this schematization and then ask students to evaluate new objections to it, as in the example below.

Example 7. Consider this argument from your reading.

1. One must never do injustice (49b6)
2. Therefore, one must not return injustice with injustice. (49b8)
3. Therefore, if one has a contract, one must not break it, even if the other party has wronged you. (49e.)
4. Every citizen makes a contract with the laws of the city to obey those laws. This duty is stronger even than duty to parents (50b).
5. Therefore, one must obey the laws of the city, even if the city has wronged you. (51b)

---

6. Therefore, Socrates should stay in prison. (51c)

Which of the following objections Crito might make to Socrates's argument? (Check all that apply.)

1. In fact, one should do injustices sometimes, and cases where one has been done an injustice are a good example of this.
2. Even if one must never return an injustice for an injustice, it doesn't follow that one must always carry out a contract, because the contract could be voided with the other side breaks it.
3. If Socrates runs away, it will be a violation of everything he's stood for.
4. Socrates has no reason to fear death, so he has no reason to avoid execution.

Example 7 is fairly simple. The first two items are objections to the argument, one challenging the truth of a premise and one challenging the strength of an inference. The second two choices are not objections to the argument. They are further statements in support of the argument.

If your course covers more aspects of critical thinking, you can add interesting variations to this question. For instance, if your course emphasizes the difference between objections directed at an argument and independent reasons to doubt the conclusion of an argument, you could add the option "(e) If Socrates allows himself to be executed, his children will grow up as orphans." If you have made it sufficiently clear that independent arguments against the conclusion do not count as "objections to an argument," this would then be a wrong choice that many students would be very tempted to circle. Alternately you could ask a question that asks students to identify exactly how a different responses to this argument work, as in this example.

Example 8: Consider the following list of possible replies Crito or someone else might make to Socrates's main argument in the Crito, outlined above

1. In fact, one should do injustices sometimes, and cases where one has been done an injustice are a good example of this.
2. Even if one must never return an injustice for an injustice, it doesn't follow that one must always carry out a contract, because the contract could be voided with the other side breaks it.
3. If Socrates runs away, it will be a violation of everything he's stood for.
4. Socrates has no reason to fear death, so he has no reason to avoid execution.
5. If Socrates allows himself to be executed, his children will grow up as orphans.

Mark each of the replies with one of the labels below, depending on what kind of reply it is. Some labels will be used twice.

TP: This is a challenge to the truth of one of the premises.
SI: This is a challenge to the strength of one of the inferences.
IND: This is an independent argument against the truth of the conclusion
NC: This is not challenge at all to the argument.

Questions like this allow us to push very deep into the logic of a philosophical argument, while retaining the ability to grade answers quickly, mechanically, and objectively.
I could go on with more examples, and in fact in a very short time I will. I've been asked to write an article on multiple choice questions for newsletter of the American Association of Philosophy Teachers. This post is a draft of part of it, and more will be coming soon.