Training 2008: Evaluation Time
December 21, 2007
Here are seven keys for unlocking Kirkpatrick's four levels of evaluation.
By Dr. Donald L. Kirkpatrick
Can I assume you have heard of my four levels for evaluating training programs?
How about a little quiz to see if I am right?
Without looking further in the article, please answer the following questions.
What is Level 1?
What is Level 2?
What is Level 3?
What is Level 4?
This is called a "pretest," an important concept used in Level 2. If you don't know the answers, don't feel "stupid." It isn't your fault, and it is an important concept to make clear to learners. It is necessary to compare answers on a pretest with those on an identical posttest to measure the amount of learning that has occurred in a training program.
I will give more details later when I discuss Level 2.
So I am warning you ahead of time that there will be a posttest at the end of this article to see what you learned. No cheating! And if you haven't learned, who is to blame? You will have to read the rest of the article to find out.
Here are the seven keys for unlocking my four levels for evaluating a training program.
1. Analyze your resources.
2. Get your managers on board.
3. Evaluate Level 1, REACTION.
4. Evaluate Level 2, LEARNING.
5. Evaluate Level 3, BEHAVIOR.
6. Evaluate Level 4, RESULTS.
7. Build a chain of evidence.
NOTE: There is no Level 5 in spite of what some "experts" say. ROI is part of Level 4! (Remember this. It might be in the posttest!)
Analyze Your Resources
Few organizations have any professionals who spend full time on evaluation. Usually it is one of the hats professional trainers wear. So, you will have to look for those who can help with the evaluation. There may be several in the training department. There may be one or two in Human Resources. And there may be managers who will help. You may even have a budget to hire consultants who will help. Putting all this talent to work on evaluation will help you decide two things: What programs you should evaluate and at what levels.
Get Your Managers on Board
They will be needed, especially in evaluating Levels 3 and 4. There are three things you can do to get their willing help:
• Ask them to help determine the training needs. As George Odiorne, a well-known author of management books, said, "If you want people to accept the decisions you make, give them a feeling of ownership."
• Ask one of them—especially a person above the participants—to introduce the program at the first meeting. Have the person encourage them to look for practical ideas they can take back to the job. You might even have a higher-level person teach part of the program, if qualified.
• Ask them to encourage their subordinates who will be attending a program. When I was teaching supervisory training programs at the University of Wisconsin Management Institute, I wrote a short booklet and sent it to the managers in organizations that sent their supervisors to our programs. I asked them to do the following: Go over the program with them before they attend. Tell them their job will be taken care of while they are gone. Tell them to have a good time and learn what they can. Tell them also that when they get back, you will sit down with them and listen to the practical ideas they learned at the program. Tell them you will work with them to put the ideas into practice.
I can't tell you how much difference this made, but I know it would be significant in their motivation to learn and their desire to bring back some practical ideas. I suggest you ask your managers to show this kind of encouragement to the people they send to programs and those who benefit from e-learning.
Note to readers: The next four keys will include principles and approaches but will not include guidelines, forms, and specific techniques. The references at the end of the articles will list books in which these items are discussed in detail.
Evaluate Level 1, Reaction.
I consider this something that should be done for EVERY PROGRAM. It is easy and doesn't require the use of a number of people to do it. A simple, well-prepared Comment Sheet is sufficient. Some people call it (with tongue in cheek) "happiness ratings" and seem to think it is not worthwhile. I call it a measure of Customer Satisfaction.
Consider two reasons for doing it:
1. If you don't do it, you are subtly telling the learners their feedback is not important—or that you know what they need and you don’t need or want their feedback.
2. The information you will get will be helpful in knowing to what extent you met your objectives (one of which should be "satisfied customers"), and you may get practical suggestions for improving future programs.
Evaluate Level 2, Learning.
In any training program, you have three possible objectives: Increase knowledge, improve skills, and change attitudes. Some programs such as leadership and coaching are designed to increase knowledge of important principles and techniques, as well as suggest changes in attitudes. Others may be aimed directly at improving skills such as sales techniques, presentation skills, how to conduct performance appraisal interviews, and how to operate a computer. In contrast, diversity training programs are aimed primarily at changing attitudes.
If you are measuring learning where improving knowledge and changing attitudes are the objectives, then a paper and pen pretest and posttest are appropriate.
If you are measuring an increase in skills, then a performance pretest and posttest are necessary. Obviously, a pretest is not needed if something brand new is being taught.
Evaluate Level 3, Behavior.
The training professionals usually can handle Levels 1 and 2. But the measurement of behavior usually needs the help of line managers. The process involves a pre- and post-analysis of behavior. Sometimes it is possible and appropriate to measure behavior prior to the program and then again after an amount of time to allow learners to practice the new behaviors taught in the program. In other cases, the only measure will be after the program to find out to what extent the new behaviors have been applied. The amount of time before evaluating depends on the subject content. For example, if you are teaching a new behavior that requires immediate application on the job, you will measure the change within a short time, perhaps a week or even a day. On the other hand, if you are teaching a complicated leadership or coaching program, you probably will wait three or more months to see whether learning has been transferred to the job.
You can measure by using a patterned interview or a survey of participants. The appropriate question would be "To what extent have you used the new behavior that was taught in the program?" The answers could be:
• To a great extent
• To some extent
• Not at all
If the answer is "Not at all," a follow-up question might state:
"Why have you not used the new behavior?" (Check all that apply.)
• The change was not practical
• Higher priorities
• My supervisor prevented or discouraged me from changing
• I tried it, but the old way was better, so I went back to the old way
In addition to asking the learner the above questions, it would be a good idea to ask one or more of the following: the learner's manager; a peer who is familiar with the work the learner does; and/or the learner's subordinates. Be careful with the last choice unless the learner approves it. Many learners (and people at all levels!) do not appreciate subordinates being asked to evaluate their boss.
It is important to ask the managers, who can be a big help in determining to what extent the behavior has changed. This is one reason for getting the managers on board.
Evaluating Level 4, Results.
This often is easier than evaluating behavior because preprogram data usually is available. For example, if the desired result is reduced turnover, the data of preprogram turnover is readily available. Likewise, the amount of sales, the amount of production, the cost of a process, a measurement of customer retention, and other desired results are available.
When considering the results of a program on leadership and coaching—probably the most popular training programs today—the evaluation of results becomes more difficult and less specific. For example, one suggested result of such programs may be "higher morale" or "better attitude toward the boss and the company." You may be able to relate this to turnover, but there are other factors that might have affected turnover that make it impossible to prove the lowering of turnover came from one or both of these programs. For example, the employment department may be hiring older people instead of kids right out of high school. Or some benefits may have been given to employees, or the economy in the community has reduced the number of available similar jobs.
Some years ago, a friend of mine, Jack Jenness of Consolidated Edison, was asked by his boss, the assistant Human Resources manager, to determine in dollars and cents how much the organization had saved by having the leadership training program that had been given to all managers the year before.
It was an expensive program conducted by a consulting firm from St. Louis. Jack told his boss there was no way of putting a dollar value on it. But the man insisted. His motivation, according to Jack, was to get promoted to Human Resources manager if he could claim a good excess of benefit vs. cost.
So Jack called me, and I confirmed his answer. He invited me to come to New York to do a program on evaluating training programs and to tell his boss there was no way of doing it. By the time I got there, the man had given up the idea.
In summary of Level 4, it is desirable and requires help from managers. But often, the results of the evaluation may be in the category of "preponderance of evidence" instead of "proof beyond a reasonable doubt," and you will have to convince your managers this is the best that can be done.
Build a Chain of Evidence
Martha Stewart was convicted, not because of just one thing, but because the prosecution had built a chain (or trail) of evidence which, when put together, proved her guilt to the jury.
You, too, have a jury: your top management.
The way to build the chain is to begin with Level 1 and do all the four levels in order. Each one builds on the other. Recently, a lady from Microsoft called and said, "Don, we have done Levels 1 and 2. Is it OK to skip Level 3 and go directly to Level 4?" I replied: "Definitely not! If you skip Level 3, you won’t know whether the results came from the training program or from some other factor or intervention."
And now, the posttest I promised you"
What is Level 1?
What is Level 2?
What is Level 3?
What is Level 4?
Is there a Level 5? (Yes or No)
Congratulations. You passed the test. Now all you need to know is how to implement them!
P.S.: If you answered, "Yes," to question 5, you failed the test even if you got the others correct!
The following references contain guidelines, forms, procedures, and case studies of organizations that have evaluated at various levels. Be sure to read the case studies to get ideas for your own evaluation. You can beg, borrow, or steal the forms and procedures and use them as is or modify them to suit your own situation.
REFERENCES
Books by Donald L. Kirkpatrick and James D. Kirkpatrick:
• "Evaluating Training Programs: The Four Levels"; 3rd edition; Berrett-Koehler Publishers; San Francisco, CA; $35. This is the basic "best-seller" that has been translated into four languages.
• "Transferring Learning to Behavior"; Berrett-Koehler Publishers; San Francisco, CA; 2005; $35. My son, Jim, wrote most of this practical book. The purpose was to provide practical ideas for being sure the learning was transferred, as well as describing the various ways of evaluating behavior.
• "Implementing the Four Levels"; Berrett-Koehler Publishers; San Francisco, CA; 2007; $25 (soft cover). This latest book in the series makes it easy to evaluate all four levels. Forms and procedures are described in each chapter. Also, chapters on "Analyzing Your Resources" and "Getting Managers on Board" were added.
Dr. Donald L. Kirkpatrick is professor emeritus at the University of Wisconsin. He is also past president of ASTD and author of six books, including the best-seller, "Evaluating Training Programs: The Four Levels," which has been translated into Spanish, Polish, Turkish, and Chinese. He can be reached at dleekirk1@aol.com.
|
RSS
