Creating A Bad Test

Law school exams may not have predicted much about your future legal career, but at least they made sense.

I recently had the pleasure of taking the test to obtain “indefinite leave to remain” (the equivalent of a Green Card, more or less) in the United Kingdom.

I think this was the first formal test I’d taken since the California bar exam in 1983 and the Illinois bar exam in 1987. So it’s been a while.

This got me to thinking about how old I am. It also got me to thinking: What does it take to create a truly bad test?

(This isn’t sour grapes. I passed.)

(There’s a rumor circulating that they gave this test to Members of the UK Parliament, and only a tiny percentage passed. I couldn’t confirm that rumor with anything I found online. If I had found confirmation online, that would make it true, of course.)

For this test, you buy a 160-page book and are told that everything you need to pass the test is in the book. Just study. Various organizations will give you practice tests, so you’ll be ready for the big day.

This raises a couple of questions. First, why exactly does passing a test make you a valuable asset for a country? I suppose it’s nice if everyone in a country shares some common knowledge, but I think I’d prefer countrymen with other attributes — integrity, courage, curiosity, whatever — over people who memorized a bunch of dates.

Sponsored

Second, how do you create a bad test? The Brits have this down cold: You have one person write a book, which actually tells you some interesting stuff. Then you hand the book to a government bureaucrat and say, “Please generate every possible question that can be created from the material in this book.”

The bureaucrat goes to work: “Margaret Thatcher was the first female Prime Minister of the United Kingdom. She was the longest ruling Prime Minister in the 20th century. In what county was she born?” (It’s a multiple choice test. The question is so stupid that the answer’s almost irrelevant. But choose from among, say, Lincolnshire, Yorkshire, Leicester, and Gloucester. Go ahead. Prove you have what it takes to be a citizen.)

Look: There’s stuff that matters about Margaret Thatcher. Like the fact that she was the first female Prime Minister. And, to a lesser degree, the fact that she was the longest serving Prime Minister in the 20th century. Those things are unique to Margaret Thatcher.

And there’s stuff that doesn’t matter about Margaret Thatcher. Like where she was born.

Everyone was born someplace. Unless you were born someplace that really sets you apart — the top of Mount Everest, maybe, or the moon, or someplace like that — your place of birth is not your distinguishing characteristic. Don’t write a test that aims for the capillaries.

Sponsored

(Okay, okay: Lincolnshire.)

Similarly, questions shouldn’t deceive people based on general knowledge that most people possess, but that has been ruled off limits for the test. For example: Robert Burns, the Scottish poet, was known as (A) the Bard, (B) the great Scott, (C) the Shakespeare of the North, or (D) mickle maks a muckle.

Many test-takers will rule out “the Bard” because they know that Shakespeare was the Bard, so that probably wasn’t also the nickname for Burns. But the fact that Shakespeare is called the Bard wasn’t in the book! (Shakespeare was in the book, but not the antonomasia. Go ahead; click through! I learned that word only when someone started calling me “the Curmudgeon.”)

If the widely known fact that Shakespeare was the Bard isn’t in the book, then silly test-writers will assume that no one knows it. In Scotland, I guess, folks refer to Robert Burns as the Bard. (I guess both Shakespeare and Burns could be known as the Bard. But Burns was the lesser of the two. And there are things about Burns that actually matter — he was the author of “Auld Lang Syne,” for example — that one might reasonably include on a test.)

In law school, tests may not have predicted much about your future performance as a lawyer, but at least they made sense. They made sense because they were written by a reasonably intelligent person who knew what mattered in the material that the professor had taught.

If you ever need to create a test, make sure the teacher plays some role in drafting it.

Otherwise, it’s tatties o’wer the side.


Mark Herrmann spent 17 years as a partner at a leading international law firm and is now responsible for litigation and employment matters at a large international company. He is the author of The Curmudgeon’s Guide to Practicing Law and Inside Straight: Advice About Lawyering, In-House And Out, That Only The Internet Could Provide (affiliate links). You can reach him by email at inhouse@abovethelaw.com.