Saturday, August 29, 2009

Stanley Kaplan and the legacy of gaming standardized tests

Christopher Caldwell has a piece in the Financial Times titled "The Opposite of Education" about the lasting effects of Kaplan test preparation. The most prominent change in the post-Kaplan era is the awareness of motivated students of how to game game standardized tests.
Kaplan’s insight was to figure out that there was an idiom to multiple-choice tests. Choices tend to be offered in predictable ways. For instance, if a problem about ratios has the answers (a) 2/3, (b) 4/5, (c) 6/7, (d) 3/2, the right answer is probably A or D, with one of them meant to “catch” a test-taker who has reversed the terms. His study guides are full of wisdom about the prose styles of test-composers, such as: “If guessing, a good rule of thumb is: the longest choice is often the correct one.” Kaplan insisted he was a respecter of subject matter. But figuring out the “tricks” of testing would give you a leg up, whether you had mastered the subject matter or not.
The advice works. The trick about choosing the longest answer, in particular, works well unless you have a test writer who is aware of the problem, and purposely puts in wordy incorrect answers. The problem comes up because the test writer knows that the correct answer is something very specific, and needs to explained precisely. The incorrect answers, on the other hand, are just things you make up and don't need to have much content. If you are in a hurry, it is easy to skimp on the effort needed to create plausible wrong answers.

The whole thing is really an arms race, and unfortunately, too many teachers never make the effort to write good multiple choice questions. The practice questions that come with the online supplement of my ethics textbook can all be beaten using the "longest answer" rule. This kind of laziness leads a lot of teachers to assume that multiple choice testing is a tool for fake education and decide not to use them at all. This is unfortunate, because in the real world, we all have situations where we need to evaluate large numbers of students en masse, and a good multiple choice test is a valuable tool in those situations.

Another interesting thing about the Kaplan techniques is that some of them are farther removed from content than others. The ratio rule, for instance, does have some bearing on content. Students who use it are aware of the fact that swapping numbers is a common mistake in this sort of problem. I tell my students explicitly about this sort of Kaplan-technique as a way of getting them closer to the content while seeming to only be teaching to the test. I tell them to know what the common mistakes are for the kind of problem you are working on, and look for the answers that seem to be testing for this common mistake.

(We had a job candidate here at LCCC who played a related trick on students. He would say he was going to tell them secrets for ticking your teacher into giving you an A. Then he would offer advice like "Buy a dictionary and bring it to class. Be sure you sit so the teacher can see you have a dictionary. Then when the teacher uses a word you are unsure of, be sure to look it up right in front of him. This will be time consuming, so you will also want to be looking up words from your readings at home.")

The legacy of Kaplan (and its nemesis, the Education Testing Service) is an education system built around teaching to tests. As Caldwell writes
Now that not just children but school systems are rewarded and punished for their performance on tests, public education has been colonised by the Kaplan philosophy. Entire school systems have hired testing companies such as Kaplan to undertake the Monty Python-esque task of teaching teachers to teach students test-taking skills.
The situation is not beyond hope, though. The thing we need to do now is have tests that are worth teaching to. You need to write tests that can't be gamed by techniques completely unrelated to content. You need to write tests that actually measure the knowledge, skills, and values you are concerned with promoting. In the terminology of the industry, you need tests that are valid and reliable. The biggest obstacle to education right now is that too many people have an interest in making sure the test isn't worth teaching to. Administrators and parents of privilege want tests that can be gamed so they can game them. Teachers don't want to take the effort to write good tests or change the way they teach to match a good test. Caldwell laments
So everyone wound up back in the same place. SAT scores still tend to track parental income fairly faithfully. Except that educational advancement now goes not so much to those who know the periodic table or can translate an English passage into Latin, but to those who have learnt to outsmart an educational bureaucracy
This is true, but at this point an educational bureaucracy is inevitable. We just need to create one that is harder to outsmart

Reasoning with children

Molly: "That's not just Halloween costume, that's an advertisement. If you wear that, you will be turning yourself into a walking advertisement for Hanna Montana."

Caroline: "But you're going to let Joey wear the Wolverine costume, and that's just an advertisement for Wolverine."

[Look, a counter example! Its reflective equilibrium!]

Molly: Yes, but it is not an advertisement for something I hate.

[Revising the rule in light of the counter example.]

Molly: Also, I'm not as excited about the Wolverine costume as I was about the Princess Dragon Ninja costume that wasn't an advertisement at all.

[Adding nuance to the rule to make the whole thing more plausible.]

Friday, August 28, 2009

Does Socrates use the Chewbacca defense?

I use this all the time in Critical Thinking and Introduction to Philosophy courses (Discussion question: does Socrates use the Chewbacca defense?). But I couldn't find it last time I needed to grab it in class, so I'm putting this up my future reference.

new twitter feed: shitmydadsays

I just signed up for twitter so I can get this guy's feed, where he writes down the things is 73 year old father tells him. Example: "You need to flush the toilet more than once...No, YOU, YOU specifically need to. You know what, use a different toilet. This is my toilet."

Wednesday, August 12, 2009

Elderly people are concerned that "The government will take over medicare"

A HuffPo columnist notes that most of the people disrupting town hall meetings are over 65. They are also very very confused about the current state of healthcare and what is being proposed. One demand that seniors repeatedly make is that the government shouldn't "take over" Medicare. You'd think seniors would be aware that Medicare is already a government run program, but I guess not. Bob Cesca, the HuffPo columnist notes that one person shouted "Keep your government hands off my Medicare!" at Rep. Robert Ingles. The LA Times reports that someone contacted Rep Jim Tanner saying, "I'm happy with Medicare, don't let the government take it over."

So where is this wacky talking point coming from? Republican activists with the assistance of the major news networks Arthur Laffer, economics advisor to Ronald Regan and a leading figure in libertarian economic thought, went on CNN and said "If you like the post office and the Department of Motor Vehicles and you think they're run well, just wait till you see Medicare, Medicaid and health care done by the government." No one on CNN challenged him.