We’ve been spending time at the firm experimenting with high stakes testing–that is, end-of-course exams that carry consequences for failure. I’m leading a working group to revise our policies based on what we’ve learned.
One recommendation I brought to the group was that all high stakes tests have to feature application-based questions for at least half the questions. The idea here is that application-based questions better test usable knowledge. Also, we want to test knowledge acquired, not the ability to run keyword searches in the participant guide (our tests are open book).
(An assessment-based question is one that asks test takers to apply knowledge to realistic situations. For instance, the question, “Which of these is the best response to an angry customer who claims that she did make a reservation even though none shows up in the system and there are no tables open?” is application-based. Asking, “All of these are critical principles for dealing with angry customers EXCEPT:” is not because it is abstract. You could get the question right by memorizing a list.)
Developing good assessment questions is hard, and this recommendation will create significant work for some of the busiest people at the firm. However, very much to their credit, they quickly gelled around the position that testing shouldn’t be a compliance exercise; if we are going to test learners, we should write tests that, to the best of our ability, actually assess whether learners can apply what they learned in class to real world situations.