By Dr. James D. Kirkpatrick and Wendy Kayser Kirkpatrick
The training division of a federal agency was in trouble. In a conversation with their leaders, three increasingly common events were detailed:
These issues created an “us versus them” mentality. In short, training was seen as a cost, not a benefit, to the organization. The training department urgently needed to demonstrate its value to agency mission accomplishment or risked losing staff members and a significant portion of its budget.
Context and Background Information
The U.S. Office of Personnel Management (OPM) is a federal government agency that functions in a human resources capacity for federal government agencies. Its mission is to recruit, retain, and honor a world-class workforce to serve the American people. OPM creates regulations, or “regs,” with which all government agencies must comply.
In recent years, several regs related to training evaluation and effectiveness have been released. In June 2011, OPM released a field guide for training evaluation based on the return on expectations (ROE) model to assist federal agencies in meeting the regs. This validated the ROE model as the U.S. government’s methodology of choice for creating and demonstrating training value.
Click here to see related regs:
Impact Study Program Selection Criteria
This government agency has several sites around the U.S., each under the protection of a specially trained law enforcement team. As training and law enforcement shared the mutual goal of enhancing their working relationship, a course entitled “Law Enforcement Report Writing” was selected for an impact study. Additional reasons for selecting this course include:
The training department was informed enough to know that while formal training creates the foundation for success, it is insufficient alone to produce significant results. The formal, blended learning consisting of e-learning and instructor-led modules teaches officers how to respond to on-site incidents; interview witnesses; take accurate notes; and write concise, objective reports (Level 3 behaviors).
Additional required drivers, or factors designed to achieve maximum on-the-job application and subsequent results, include:
Definition of Success
Eight agency sites participated in the study. The first task was to determine what ultimate success would look like. In other words, how would the agency ultimately benefit from good incident reports? Three observable, measurable outcomes (leading indicators) were selected:
The impact study was structured to determine the relative strength of each of the required drivers in contributing to the accomplishment of these outcomes and of the overall mission.
Impact Study Process
Quantitative and qualitative data were collected and continually analyzed over several months. This process maximizes results because problems and disconnects are identified and fixed before they can reduce overall outcomes and program impact.
After the program, Level 1 / Level 2 hybrid surveys and interviews of training participants were conducted. These were targeted to determine how well the program itself was designed and delivered.
After 45 days, Level 3 / Level 4 hybrid surveys and interviews were administered for 85 reporting officers and 28 leaders, from supervisor level to area chiefs. These focused on how well the initiative was working post-training (Level 3 critical behaviors and required drivers, and Level 4 leading indicators). The surveys included questions about the usefulness of the training and each of the required drivers.
Finally, the leading indicators and ultimate Level 4 results were assessed through structured interviews with law enforcement captains and chiefs.
Download the PDF below to see selected study data.
Program results were reported against the three leading indicators identified as experiencing the most impact from the report writing course and related follow-up.
1. Serve justice to criminals:
2. Minimize agency legal liability:
3. Create an image of respect for agency law enforcement professionals by internal and external colleagues:
Data collection and analysis throughout the process and resulting program modifications were cited as important contributors to the maximization of program results.
During the final presentation, the captains and chiefs enthusiastically endorsed the effort and resulting evidence. ROE was maximized and demonstrated.
A major collateral benefit of the impact study was an agency decision to use the ROE methodology for all mission-critical programs.
Dr. Jim and Wendy Kirkpatrick work together in Kirkpatrick Partners, the One and Only Kirkpatrick company. They are the creators of the Kirkpatrick Business Partnership Model and the New World Kirkpatrick Model. They welcome your questions and comments at firstname.lastname@example.org. For more information, visit kirkpatrickpartners.com.