Aids
Applying the New World Kirkpatrick Model
I worked with a team that was tasked with creating an onboarding webpage with modules. They had no prior experience with instructional design and development and I needed them to understand how it works before beginning the process. I knew we would be using the Kirkpatrick Model so I wanted to briefly explain each level based on the Kirkpatrick’s book and provide a Learning Aid (image above) for the team.
LEVEL 4: Organizational benefits that result from the training.
William Horton, in the article “How to Evaluate E-Learning,” evaluated Level 3 AND Level 4 in one paragraph. He said the success, or failure, occurs after the learners leave e-learning. Ironically though, the success of this team will be determined by how well new hires are onboarded online – which technically is e-learning.
SUMMARY & EXAMPLES: Identify "Leading Indicators" - the personalized, step-by-step targets that are measured formatively to connect the learners' performance to the organization's results. These indicators can be either internal (i.e. quality, cost, efficiency) or external (i.e. customer response/satisfaction).
This is the most important level.
LEVEL 3: Scope of participants behavioral change in workplace.
In the article “How to Evaluate Informal Learning,” Saul Carliner said providing certification is the best way to directly assess the learner. Those who complete the onboarding process would earn an official certificate from the organization. In addition, the organization requires annual performance reviews and we would review their work for evaluation – another method recommended by Carliner for Level 3.
SUMMARY & EXAMPLES: This is all-encompassing and ongoing all the time. The three "Support Drivers" are Reinforce (i.e. job aids, OJT, refreshers), Encourage (i.e. coaching, mentoring), and Reward (i.e. pay, recognition). The one "Accountability Driver" is Monitor (i.e. observation, interviews).
This is essential to Level 4.
LEVEL 2: Extent the participants improved their knowledge and skills.
Clark Quinn, in the article “How to Evaluate Mobile Learning,” said providing appropriate challenges to the learner can resurrect what they learned. In the case of this team, the learners would go straight from learning how to design and develop to actually helping design and develop. The immediate transition should further reinforce and solidify their new knowledge.
SUMMARY & EXAMPLES: Work to formatively evaluate five learning components: knowledge, skills, attitude, confidence and commitment. Include challenges during training that test the participants in a real-world environment. Methods include games, role play, simulation and discussions. Keep it simple!
This is a familiar level to most professionals.
LEVEL 1: Measuring how much the participants enjoy the training.
Back to Horton and “e-learning,” he mentions creating a discussion board for the course. This team uses Slack – a communication/organization app – to discuss projects outside of the official channels. Slack allows you to create channels using hashtags so we could house all comments about the training in one place for future reference. And because we are informal in our usage, the feedback would be natural and honest.
SUMMARY & EXAMPLES: Based on three components: engagement, relevance and customer satisfaction. Use formative methods (i.e. instructor observation, pulse checks, dedicated observer) and summative methods (i.e. open-ended survey questions, discussions, interviews). Only ask a few questions.
This is one of the easiest to evaluate.
REFERENCE:
Kirkpatrick, J. D. & Kirkpatrick, W. K. (2016). Kirkpatrick’s four levels of training evaluation. ATD Press.
Job Aid
Style vs. Substance
In the Learning Solutions Mag article titled Instructional Design and Visual Design: The Pillars of Great eLearning, Pamela Hogle has a terrific quote that sums up Style vs. Substance.
“Great content presented poorly will not reach its audience because learners won’t or can’t get what they need from it. Beautiful eLearning that has irrelevant, inaccurate, or incomprehensible content won’t accomplish its goals because learners won’t get what they need from it. That’s why instructional design and visual design are equally essential to the success of your eLearning projects.”
That was the mindset I had behind this Job Aid. I had to get across the exact steps in the exact order because this team was transitioning to a system where the printing process was decentralized and transferring back to the programs and their managers or directors.