There is a distinction between university teaching and employee training. Teaching is about providing students with general knowledge and skill. Training is about providing knowledge and skill that employees can utilize immediately on the job. When a university evaluates teaching, they are concerned primarily with whether students learned the content of a course. When a company invests resources into employee training, they are interested in more than just whether the employee learned something. They want to know that the learning transferred to the job and that it is having a positive impact. The Kirkpatrick four levels of evaluation was developed to evaluate training as it focuses on both immediate learning and subsequent transfer to the job. In these days of calls for universities to take responsibility for preparing students for the workforce, it is time that we adapt the Kirkpatrick four levels for university teaching.
The Four Levels
The Kirkpatrick four levels of evaluation acknowledge that there can be multiple effects of training.
- Reactions: This indicated how employees felt about the training—did they enjoy it or find it boring? Did they feel it was useful or a waste of time?
- Knowledge: Indicated by an assessment at the end of training (or before and after training) to reflect the knowledge and skill level achieved at the end of training.
- Behavior: This level goes beyond what happens in training and focuses on whether the employee is applying what was learned to the job. It can be assessed by measuring behavior change from before to after training.
- Results: A positive impact on learning and behavior does not guarantee that the training will affect organizationally-relevant outcomes. An employee might apply what was learned, but the wrong skills were taught, or the employee is not applying them correctly. This last level looks at the bottom-line purpose of the training, for example, did sales skills training result in more sales or did safety training result in fewer accidents?
The Kirkpatrick approach is a valuable tool for determining if training resources are being wisely spent. It can indicate that the investment in training is yielding the desired return, whether it be greater profits for the company or better health for employees.
Kirkpatrick Four Levels for University Teaching
Over the past decade, there has been a trend in higher education to focus more attention on preparing students for the workforce. The U.S. National Science Foundation (NSF) has invested $millions into a program to better prepare university students for STEM careers. I am part of a team at the University of South Florida that has one of these grants, and we are working on a number of initiatives for students in electrical engineering. We hope that what we learn will be of use in other fields in STEM and beyond.
The adoption of career preparation for students has not always been embraced by faculty whose focus is primarily on the first two Kirkpatrick levels. It would not take a lot, however, to incorporate the second two levels into our classes and curricula to better serve the life-long needs of students. The first two levels are pretty well covered; what we need to do is think about the second two. It begins by considering what sorts of work behaviors are relevant and how they might link to desired outcomes for companies and for people. This might vary by major, although likely there are some that are universal, such as the ability to communicate clearly both verbally and in writing.
Translating the four levels into education looks like the following.
- Reactions: These are the student evaluations that most universities conduct. Although this reflects on the student experience, the implication of Kirkpatrick is that this is just a small piece of the overall evaluation. Unfortunately, too many universities do little more than assess student reactions.
- Knowledge: Most undergraduate classes include exams to assess knowledge and/or skill. They are normally used to evaluate students and not course instruction, but some universities have incorporated standard evaluations to assess how much students learned in a course.
- Behavior: This would be more difficult for universities to evaluate routinely because it would require assessing alumni who are in the workforce. It would be possible to do more limited studies where samples of students could be followed at work, it is doubtful that most universities would invest the resources. What would be more feasible is for faculties to identify work-relevant knowledge and skill that should be incorporated into the curriculum, perhaps covered in multiple courses. This would be easier for some majors that are preparing students for careers (e.g., accounting or engineering) than others, but some typical career paths for all majors could be identified that would inform the curriculum. It would be assumed that if skills such as knowledge of spreadsheets is important for careers that training students in their use would transfer, especially if they are given real-life assignments. Of course, this assumes that what was taught is retained after graduation.
- Results: The ultimate goal would be to provide tools for students to have productive lives at work and beyond. One immediate outcome would be the employability of graduates. If employers are willing to hire graduates, the university is doing something right. Soliciting feedback from employers about how well graduates are doing and where their preparation could be improved would be highly useful. I have seen this used successfully in business and engineering colleges where they create advisory boards of executives from employers of their graduates.
The Kirkpatrick four levels is not only a useful tool for the comprehensive evaluation of education or training, it is a framework for thinking about the goals of training. It asks us to consider if we are delivering training in an engaging and effective way. Are we teaching the right things that transfer in meaningful ways to meet the ultimate training goals that go beyond what happens in the classroom?
Photo by George Becker from Pexels
SUBSCRIBE TO PAUL’S BLOG: Enter your e-mail and click SUBSCRIBE