Things guaranteed to start an argument: Chicago style vs. New York-style pizza, Star Trek vs. Star Wars, Pepsi vs. Coke. In the world of learning design, these debates are joined by one more: course evaluation surveys.
If the thought of course evaluation surveys – those questionnaires you fill out at the end of a CLE, often to receive CLE credit — causes anger, that’s understandable. But while these surveys aren’t everyone’s cup of tea, they do concern all of us indirectly.
Course evaluation surveys often focus on how you enjoyed the presentation, whether you learned anything, and if you would recommend the presentation to others. Now think back to law school. Imagine if your professors’ primary metric of success was whether you enjoyed the class (I’m sure some of you feel that this definitely wasn’t their primary metric) or they asked whether you had learned something as you were leaving for your next class. Is that the best time to evaluate how the class has impacted your law school career? Post–session CLE surveys can feel a little like that – speak now or forever hold your peace.
The Kirkpatrick Model of Evaluation
The Kirkpatrick Model is one of the most well-known models for analyzing and evaluating the results of training and education programs. Its levels of training evaluation have become somewhat canon in the training field, although Kirkpatrick has some detractors, including those who feel the levels are correct but the order should be flipped.
Kirkpatrick is divided into four levels, each measuring a different impact of training:
(www.kirkpatrickpartners.com/Our-Philosophy/The-Kirkpatrick-Model.)
If you’ve ever filled out a survey at the end of a CLE, then you’re likely giving feedback designed to inform these levels of training evaluation. It’s also likely you were asked how you found the training, what changes you would make, if you learned anything, and if you’ll apply what you learned in your professional role. These questions appear to cover Levels 1 – 3 and are certainly more informative than going with the presenter’s gut.
Collecting data on the impact of training is, of course, essential. However, the assumption that a post-session course evaluation survey is all that can be done to understand that impact can cause an uproar at training and development conferences. Ideally, CLE is a solution that impacts individual and organizational challenges, so the measurement of success needs to be both individual and organizational.
What do these levels mean in practice?
Learning professionals love measuring things, and levels allow us to put things in nice categories. But what do they actually mean in practice and how do they help us make sense of what needs to change? Let’s look at the levels again:
Level 1 “Reaction” is easy – did you enjoy the course and was it relevant? It could be argued that this metric isn’t as important, but learner engagement and learning success are connected: bored learners equal lower retention.
Attendees’ opinions are the primary data source for “reaction” and rightly so. But once you get past Level 1, the emphasis moves away from the individual and toward the training’s organizational goals, whatever they happen to be.
Level 2 “Learning” can be summarized as “Did you acquire the intended knowledge and skills?” It appears to be aimed at learners, however, when we think about job performance, who determines if you have the requisite knowledge and skills for your role? You or the organization? This is where your goals and your organization’s goals should align – if you are striving for different things, that is a problem.
Level 3 “Behavior,” or whether or not you’re exhibiting intended behaviors, sits firmly in the performance review arena. Hopefully, a performance review is a conversation between you and your boss around a set of mutually agreed-upon guidelines, whether in a competency model, job description, or established performance goals.
Level 4 “Results” is directly aimed at targeted business outcomes (e.g., hours billed, client satisfaction, cases won) and are metrics that are typically gathered at an organizational level.
Course Evaluation using Kirkpatrick
CLE providers do a great job in collecting and analyzing data from course evaluation surveys, offering feedback to presenters, and striving to respond to learners. If we measure CLE success beyond the participants’ immediate reactions to professional and institutional change, how can we collect this data?
In larger organizations, it’s the internal training department, which can hopefully draw a line between the problem the training is trying to address and outcomes achieved. However, a significant proportion of CLE is provided by professional associations and third-party providers who may not be able to collect meaningful data beyond the immediate event.
So, what can be done to bridge this gap between immediate reaction and longer–term change? Here are some suggestions about how Kirkpatrick can help attorneys, firms, and providers in course evaluation and identify a training’s value:
- Providers can reach out to attorneys and firms to establish more integrated relationships. For example, learning pathways of connected courses spaced over time typically result in deeper learning and longer-term changes in behaviors. (Level 2 – Learning)
- Firms and providers can follow up with learners to determine a CLE’s impact and a resulting behavior change. This can be done with a three-to-five question survey and a summary of takeaways at two weeks and three months after the course. (Even a 10% response rate would give you some solid data.) (Level 3 – Behavior)
- Firms and attorneys can collaborate to identify professional development goals by establishing metrics for success then seeking out CLEs that have identified those metrics in the learning objectives. Reviewing the goals after the training to see if the metrics have been met is key to understanding the impact the CLE had. (Level 3/4 – Behavior/Results)
- Firms and attorneys can view CLE as part of a longer-term learning process, e.g., identifying an issue to resolve, measuring it, targeting training, measuring it again, and comparing the results. (Level 4 – Results)
These suggestions take time, something that few of us have. But even taking five minutes to write down three skills you want to improve and measurements for success will help you move CLE from a passive one-and-done experience to something that resembles a coherent individual development pathway.
Staying up to date on issues impacting the legal profession is vital to your success. Subscribe here to get the Commission’s weekly news delivered to your inbox.