ROE Case Study: U.S. Federal Government Agency

By Dr. James D. Kirkpatrick and Wendy Kayser Kirkpatrick

The Challenge

The training division of a federal agency was in trouble. In a conversation with their leaders, three increasingly common events were detailed:

1. Business leaders typically were not requesting training support for strategic initiatives.
2. On the rare occasion when business leaders requested employee training, it was for events to be delivered in abbreviated time windows.
3. When training leaders contacted business leaders, it was primarily to inquire about training no-shows.

These issues created an “us versus them” mentality. In short, training was seen as a cost, not a benefit, to the organization. The training department urgently needed to demonstrate its value to agency mission accomplishment or risked losing staff members and a significant portion of its budget.

Context and Background Information

The U.S. Office of Personnel Management (OPM) is a federal government agency that functions in a human resources capacity for federal government agencies. Its mission is to recruit, retain, and honor a world-class workforce to serve the American people. OPM creates regulations, or “regs,” with which all government agencies must comply.

In recent years, several regs related to training evaluation and effectiveness have been released. In June 2011, OPM released a field guide for training evaluation based on the return on expectations (ROE) model to assist federal agencies in meeting the regs. This validated the ROE model as the U.S. government’s methodology of choice for creating and demonstrating training value.

Click here to see related regs:

5 CFR part 250

5 CFR part 410

Impact Study Program Selection Criteria

This government agency has several sites around the U.S., each under the protection of a specially trained law enforcement team. As training and law enforcement shared the mutual goal of enhancing their working relationship, a course entitled “Law Enforcement Report Writing” was selected for an impact study. Additional reasons for selecting this course include:

1. It is a relatively simple program that would make a clean yet convincing study.
2. Proper report writing has huge Level 4 (Results) implications because good reports can support convictions of criminals and discourage expensive litigation.

The training department was informed enough to know that while formal training creates the foundation for success, it is insufficient alone to produce significant results. The formal, blended learning consisting of e-learning and instructor-led modules teaches officers how to respond to on-site incidents; interview witnesses; take accurate notes; and write concise, objective reports (Level 3 behaviors).

Additional required drivers, or factors designed to achieve maximum on-the-job application and subsequent results, include:

1. Additional customized on-site training covering typical incidences (conducted in 8 of 12 city sites).
2. Informal feedback and coaching from seasoned peers on written reports.
3. Formal feedback and coaching from supervisors on written reports.
4. Formal up-the-chain review of written reports, with the option to “kick it back down the chain” for review.
5. An overall comprehensive system of documentation and accountability.
6. Personal responsibility and determination of the reporting officers.

Definition of Success

Eight agency sites participated in the study. The first task was to determine what ultimate success would look like. In other words, how would the agency ultimately benefit from good incident reports? Three observable, measurable outcomes (leading indicators) were selected:

1. Serve justice to criminals.
2. Minimize agency legal liability.
3. Create an image of respect for agency law enforcement professionals by internal and external colleagues.

The impact study was structured to determine the relative strength of each of the required drivers in contributing to the accomplishment of these outcomes and of the overall mission.

Impact Study Process

Quantitative and qualitative data were collected and continually analyzed over several months. This process maximizes results because problems and disconnects are identified and fixed before they can reduce overall outcomes and program impact.

After the program, Level 1 / Level 2 hybrid surveys and interviews of training participants were conducted. These were targeted to determine how well the program itself was designed and delivered.

After 45 days, Level 3 / Level 4 hybrid surveys and interviews were administered for 85 reporting officers and 28 leaders, from supervisor level to area chiefs. These focused on how well the initiative was working post-training (Level 3 critical behaviors and required drivers, and Level 4 leading indicators). The surveys included questions about the usefulness of the training and each of the required drivers.

Finally, the leading indicators and ultimate Level 4 results were assessed through structured interviews with law enforcement captains and chiefs.

Download the PDF below to see selected study data.

Program Results

Program results were reported against the three leading indicators identified as experiencing the most impact from the report writing course and related follow-up.

1. Serve justice to criminals:

• Three instances of success were noted.
• In no instance did criminal activity go unpunished.

2. Minimize agency legal liability:

• Six instances were detailed in which proper incident reports successfully thwarted potentially costly lawsuits
• In no instance was there successful litigation against the agency.

3. Create an image of respect for agency law enforcement professionals by internal and external colleagues:

• Numerous positive verbal testimonies were noted and detailed.

Data collection and analysis throughout the process and resulting program modifications were cited as important contributors to the maximization of program results.

During the final presentation, the captains and chiefs enthusiastically endorsed the effort and resulting evidence. ROE was maximized and demonstrated.

A major collateral benefit of the impact study was an agency decision to use the ROE methodology for all mission-critical programs.

Dr. Jim and Wendy Kirkpatrick work together in Kirkpatrick Partners, the One and Only Kirkpatrick company. They are the creators of the Kirkpatrick Business Partnership Model and the New World Kirkpatrick Model. They welcome your questions and comments at information@kirkpatrickpartners.com. For more information, visit kirkpatrickpartners.com.