Measuring Pains: Part 2
November 13, 2007
Learning measurements aren't rocket science, pointed out a learning executive at an Expertus forum earlier this year. But they are good business and can help you educate your stakeholders and raise important business issues.
By Linda Galloway
Death by data. That's how Kathy Thomas described the situation many training executives find themselves in when it comes to learning measurements. "Too many times, we get ourselves wrapped around the axle when it comes to measurements," said Thomas, vice president of learning and development for Norththrop Grumman. "We spend hours and hours compiling data and reports that don't mean anything to our customers."
Recent conversations with learning executives reveal that attitudes about learning measurements are changing. Overall, executives are talking less about traditional measurements and more about business-related measurements. The Kirkpatrick measurement model is mentioned, but most often in descriptions of what organizations are not doing.
As discussed in Part 1 (Training, October 2007, p. 64), obtaining visibility into the business impact of learning has become increasingly important because corporate learning now is widely viewed as a business service. The vast majority of reporting is done for internal business customers, such as sales executives, call center managers, and business unit leaders.
"We have to move from measurements based on 'training by the pound' to those that are relevant to the business," Thomas said. According to Thomas, relying solely on measurements such as total learning hours, pages accessed, class registrations, completions, and assessment scores has little if any value to internal customers—or business-focused learning executives.
The "so what?" aspect is what's most important, according to Mary Alice Colen, senior director of education services for Salesforce.com. "We can't train just for the sake of training. We must be able to answer questions about business impact. The Kirkpatrick model is great theory, but it's very expensive and time-consuming to get to Level 5. Practically, you have to find a measurement system that uses what's already in place."
Measure What?
Most executives are swimming in numbers. Knowing the ones on which to focus—"the ones you can't leave home without," as one said—is an important part of a learning executive's job. As such, many executives are relying on their customers, both internal and external, to help them determine meaningful metrics.
"One of the most important questions to ask is, 'What do you want to accomplish with training?'" said Glenn Oclassen, Jr., director of education partner programs for Salesforce.com. "If you can't come up with some basic measurements that answer that question, then you have to question the training itself.
"To be successful, training usually has to move the needle on just a few basic things," he continued. "You need to work with your customers to determine what those few basic things are and how much the needle needs to move."
In its 2006 research study, "High-Impact Learning Measurements," Bersin & Associates stressed the importance of using performance consultants, learning professionals focused on working with specific business units. The study recommends that performance consultants work with their business customers to determine appropriate metrics.
Thomas stressed that working with business leaders early on in the business planning process is paramount. "The time to determine how to measure training is when they're setting goals and objectives. If training is part of the business problem, then there's an associated metric you can use."
Colen's team takes a similar approach to measure the impact of customer training on the Salesforce.com application. "We determine up front what indicators show adoption and successful use. For Salesforce.com, we want to determine if people are using the application, if they're using it well, and then quantify the business value." Colen typically looks at metrics around usage first. "How many times individuals access the application and how many entries they make are two usage metrics," she explained. "For more detailed analysis, we look at data integrity. If sales representatives are entering required data and using established standards, that indicates people have learned from the training. And finally, to evaluate true adoption and measure business value, we examine the sales pipeline."
Oclassen added one more point. "It's important to use metrics that are readily available. You don't want to base your measurements on data that's not already being collected or that requires some kind of new process to obtain." But identifying and agreeing on what to measure doesn't solve all problems.
Brent Godfrey, CLO of Zions Bancorporation, said that one of his biggest measurement challenges was applying measurements consistently across the eight banking units in the enterprise. "Without a common methodology, it is difficult to roll up the numbers and get an enterprise view of impact," he said.
Another challenge for Godfrey is finding measurements that could be used to benchmark training at Zions Bancorporation against other financial institutions. For a comparative perspective, Godfrey said he currently relies on ad hoc information from industry contacts, as well as research data from The Conference Board and other organizations.
What About LMSs?
It's easy to bash learning management systems (LMSs) when discussing measurement challenges. But these systems shouldn't bear the blame. When LMSs came to market, companies wanted numbers around training. And in terms of training-related statistics, LMSs generally do a good job of tracking and reporting.
The debate about measurements started to amplify when training itself became more aligned with business. And the fact is that LMSs aren't designed to capture business-centric data. Usually it takes significant customization and additional development to combine outside data with LMS-generated reports. Dissatisfaction with LMSs seems to stem primarily from unrealistic expectations—the notion that the implementation of an LMS will solve all measurement and reporting issues.
"The LMS frees learning professionals from the time- consuming 'data drills' of collecting operational information so we can focus on the meaningful metrics of our customers," Thomas pointed out. "By automating the operational side of learning, we can focus on performance-based metrics and be better stewards of company time and investments."
Keep It Simple
Jim Graham, vice president of training and development for RR Donnelley, said learning measurements at his company are the responsibility of those in charge of the learning programs. His team bases its measurements on Robert Brinkerhoff's "The Success Case Method," designed to provide organizations with a simple way of evaluating any initiative's success. The model is based on five steps: focusing and planning; clearly defining desired outcomes; identifying success cases; conducting interviews to learn exactly how success was achieved; and communicating results.
Primarily through phone interviews with a sampling of learners, Graham and his team learn firsthand what employees think of the training and how and if they're using the information on the job. While time-consuming to execute, the process is simple and straightforward and provides feedback to refine future training.
"The important thing is to set realistic expectations and to isolate specific skills you expect to impact with training," he said. "With sales training, it's easy to measure outcomes and evidence of success. It's more challenging—and more anecdotal—for other types of training such as leadership development."
Interestingly, most executives don't worry about whether measured outcomes have influencers other than training. Said Thomas, "I recognize and acknowledge that there are other outside influencers; the training is only one part of the equation in creating and sustaining behavioral change and the application of new knowledge and skills."
This is the second part of a two-part series exclusive to Training magazine on the challenges related to learning measurements. The first article provided an overview of the topic, along with statistics from several recent surveys. This article brings to the topic the opinions and perspectives of senior learning executives. Expertus, a provider of operational outsourcing for training organizations, hosted two executive forums in spring 2007. Nearly 30 learning executives participated in these forums. Several agreed to go on record. This article is based on forum discussions and follow-on conversations.
|
RSS
