Predictive Evaluation: A New Approach

By Dave Basarab

Predictive Evaluation (PE) is a new approach that provides compelling training data to executives, including:

1. Predicting success of training in the three areas of Intention, Adoption, and Impact and measuring to see if success has been achieved.
2. Leading indicators of future adoption (transfer of learning) and Impact (business results).
3. Making recommendations for continuous improvement.

PE has two major components: predicting, which is before the fact, to decide whether to train, and evaluating, which is an after-the-fact measurement against the predictions. The beauty of PE is that it uses leading measures (Intention and Adoption) as a signal of results (Impact). If the leading indicators are below predicted success gates, actions can be implemented to “right the ship” so the desired results are realized.

Predictive Evaluation Benefits

What are the benefits of PE? You now can predict (forecast) training’s value to the company, measure against those predictions, use leading indicators to ensure you are on track, and report in a business format that executives easily understand. You can interweave outcomes and leading indicators into training during the design and delivery and move from an event-driven function to one that predicts success, measures its performance against those predictions, and is seen as returning significant shareholder value for the funds invested.

However, the greatest strength of the PE approach is not about how it is communicated to the executives or the tools or the results, but rather how it requires participation of the supervisors and the employees in setting their own intentions and measurement of adoption. The approach treats the employees as adults owning their learning versus students checking off a class from their list and being measured by someone else.

The key components of the approach are the training program, training outcomes, prediction of value, Intention (to use), Adoption (actual use), and Impact (the results to the company). The following sections provide an overview of the approach and each of its key components. Detailed descriptions and guidance are given in Chapters 2, 3, 4, and 5.

Where to Start

The PE approach starts with an existing training program or one that is on the drawing board. In other words, PE works for both existing courses and new ones in the production queue. PE is independent of course delivery—it works equally well for classroom-based training, on-the-job training, online learning, simulations, workshops, etc. The approach works with different content—PE has been conducted on Leadership Training, Sales Training, Business Management Training, and Basic Management Training. PE is also independent of audience and has been used for groups from senior executives to hourly employees. Finally, PE can be employed for courses that are developed and delivered in-house (those where the company has internal personnel create and deliver the training) or outsourced courses (those purchased from external vendors to meet a company’s training need).

To begin a PE on an existing course, you need to obtain and review instructional design documents (if they were created), course materials (participant and instructor), any existing evaluation data (such as Level 1 evaluation survey results), budget (actual expenses and projected expenses), number and types of employees already trained, and the number of employees who need training in the future. This is only the starting point—you can gather other information, such as opinions from participants, their supervisors, suppliers, instructors, and executives who sponsor the training. The purpose is to thoroughly understand and describe the object being evaluated (the training course). Once you understand the course, you can begin the predictive portion of PE.

But the best place to start a PE is on a course that is still on the drawing board. You don’t start PE with the instructional design process, but it comes in as a component to ensure the training design creates the proper value the company needs. In many instructional design processes, Evaluate is the final stage of the process. PE starts before the course finalizes its design, using its predictive components, and is an input/requirement for the final training design.

Typically, the predictive portion of PE begins for new courses when analysis and design phases of design are completed. In the Analysis phase, the instructional problem is clarified, the instructional goals and objectives are established, and the learning environment and participant’s existing knowledge and skills are identified. The design phase deals with learning objectives, assessment instruments, exercises, content, subject matter analysis, lesson planning, and media selection.

Whether the course to be evaluated currently exists or is still under design, the PE approach makes the assumption that training programs are designed to provide participants with the following benefits:

• Knowledge:Either new knowledge or a refresher of current knowledge
• Skills:New or improved techniques for getting work done
• Beliefs:The idea that the participants and/or their company can benefit from using the new knowledge and skills
• Behaviors:On-the-job practices and habits that shift or are adopted to improve actions and thinking that impact the business

Predictive Evaluation Framework

So let’s look at the framework and premise that PE is based on. Before, during, and when they leave training, participants have some level of motivation to use what they have learned. I refer to this as “intentions.” Intentions can be strong or weak. On the basis of their intentions, participants “adopt” (apply) the new skills as part of their work behavior (routine). Adopted behaviors practiced over time (repetition) produce results (an “impact”) for the business. The magnitude and value of the results are affected by all three factors:

1. Intention
2. Adoption
3. Impact.

Using this as a basis for mirroring employee learning and performance, you can predict Intention, Adoption, and, finally, Impact.

Dave Basarab is an experienced evaluator and author who has led strategic training initiatives for leading companies such as NCR, Motorola, Pitney Bowes, and Ingersoll Rand. He recently launched his new book, “Predictive Evaluation,” a groundbreaking approach to training and evaluation and follow-up to his previous book, “The Training Evaluation Process.” “Predictive Evaluation is endorsed by industry leaders Donald L. Kirkpatrick, Ph.D., and James D. Kirkpatrick, Ph.D., who also wrote the book’s foreword. For more information, visit www.evaluatetraining.com.