Recently at McGladrey, we convened a working group on the topic of assessing the effectiveness of our learning programs. Do we have the right strategy? How do we up our game?
We’ve had some great discussions about what course assessment looks like and about the elements of a robust strategy. In the end we agreed upon a model that has five layers.
The top layer is on-the-job training. CPA firms are still built upon an apprenticeship model in many ways. The vast majority of learning that happens in a CPA’s career happens on the job. Therefore, a comprehensive learning assessment strategy should ask whether professionals are getting an appropriate amount of learning support on the job. Are mentors making time to teach? Are they teaching toward conceptual understanding, or just teaching rote procedures? Are they asking great questions? Are they giving good feedback?
The second level is the curriculum level. Are we delivering the right courses? The best designed courses in the world won’t matter if we aren’t delivering the content that is most relevant to both learners and the business.
The third level involves evaluating instructors. I remember one of the professors in my graduate program asserting that the research is clear that the effect of the instructor on achievement is greater than the instructional treatment. In other words, the instructor matters more than the design. This is a humbling thought for an instructional designer. It doesn’t mean that design is irrelevant, but it does mean that getting the instructor right is more critical than getting the design right. For us, it means asking whether we are attracting the right instructors to teach our classes, and whether we are providing them with the proper support.
Fourth is the course level. Here, the Kirkpatrick model makes a lot of sense and is easy to explain.
The last one doesn’t really fit into the hierarchy, but is still a point of interest: individual accountability. If we are going to great effort to evaluate the effectiveness of our courses, does it also make sense to evaluate learners as well to ensure they are holding up their end of the bargain? The idea here is that sometimes you have the right courses, with the right instructor and design, and some professionals still don’t engage. Here we get into great discussions around intrinsic versus extrinsic motivation. More on this to come.