Stanley Kaplan and the legacy of gaming standardized tests

Christopher Caldwell has a piece in the Financial Times titled "The Opposite of Education" about the lasting effects of Kaplan test preparation. The most prominent change in the post-Kaplan era is the awareness of motivated students of how to game game standardized tests.

Kaplan’s insight was to figure out that there was an idiom to multiple-choice tests. Choices tend to be offered in predictable ways. For instance, if a problem about ratios has the answers (a) 2/3, (b) 4/5, (c) 6/7, (d) 3/2, the right answer is probably A or D, with one of them meant to “catch” a test-taker who has reversed the terms. His study guides are full of wisdom about the prose styles of test-composers, such as: “If guessing, a good rule of thumb is: the longest choice is often the correct one.” Kaplan insisted he was a respecter of subject matter. But figuring out the “tricks” of testing would give you a leg up, whether you had mastered the subject matter or not.

The advice works. The trick about choosing the longest answer, in particular, works well unless you have a test writer who is aware of the problem, and purposely puts in wordy incorrect answers. The problem comes up because the test writer knows that the correct answer is something very specific, and needs to explained precisely. The incorrect answers, on the other hand, are just things you make up and don't need to have much content. If you are in a hurry, it is easy to skimp on the effort needed to create plausible wrong answers.

The whole thing is really an arms race, and unfortunately, too many teachers never make the effort to write good multiple choice questions. The practice questions that come with the online supplement of my ethics textbook can all be beaten using the "longest answer" rule. This kind of laziness leads a lot of teachers to assume that multiple choice testing is a tool for fake education and decide not to use them at all. This is unfortunate, because in the real world, we all have situations where we need to evaluate large numbers of students en masse, and a good multiple choice test is a valuable tool in those situations.

Another interesting thing about the Kaplan techniques is that some of them are farther removed from content than others. The ratio rule, for instance, does have some bearing on content. Students who use it are aware of the fact that swapping numbers is a common mistake in this sort of problem. I tell my students explicitly about this sort of Kaplan-technique as a way of getting them closer to the content while seeming to only be teaching to the test. I tell them to know what the common mistakes are for the kind of problem you are working on, and look for the answers that seem to be testing for this common mistake.

(We had a job candidate here at LCCC who played a related trick on students. He would say he was going to tell them secrets for ticking your teacher into giving you an A. Then he would offer advice like "Buy a dictionary and bring it to class. Be sure you sit so the teacher can see you have a dictionary. Then when the teacher uses a word you are unsure of, be sure to look it up right in front of him. This will be time consuming, so you will also want to be looking up words from your readings at home.")

The legacy of Kaplan (and its nemesis, the Education Testing Service) is an education system built around teaching to tests. As Caldwell writes

Now that not just children but school systems are rewarded and punished for their performance on tests, public education has been colonised by the Kaplan philosophy. Entire school systems have hired testing companies such as Kaplan to undertake the Monty Python-esque task of teaching teachers to teach students test-taking skills.

The situation is not beyond hope, though. The thing we need to do now is have tests that are worth teaching to. You need to write tests that can't be gamed by techniques completely unrelated to content. You need to write tests that actually measure the knowledge, skills, and values you are concerned with promoting. In the terminology of the industry, you need tests that are valid and reliable. The biggest obstacle to education right now is that too many people have an interest in making sure the test isn't worth teaching to. Administrators and parents of privilege want tests that can be gamed so they can game them. Teachers don't want to take the effort to write good tests or change the way they teach to match a good test. Caldwell laments

So everyone wound up back in the same place. SAT scores still tend to track parental income fairly faithfully. Except that educational advancement now goes not so much to those who know the periodic table or can translate an English passage into Latin, but to those who have learnt to outsmart an educational bureaucracy

This is true, but at this point an educational bureaucracy is inevitable. We just need to create one that is harder to outsmart