The Myths of Return on Expectation

By Jack J. Phillips, Co-Founder, and Patti P. Phillips, Co-Founder, President and CEO, ROI Institute           

Learning and development professionals routinely create new terms and jargon. While meaningful to them, an obscure term often is confusing to key clients and executives. The most recent addition to the vocabulary is return on expectation(ROE). Let’s examine this term and relate it to what we know about current measurement systems.

What Is It?

What is ROE? Is it a number, a concept, or an objective? Some people suggest it is a number. We recently saw a report of a learning program that delivered an impressive 85.2 ROE. This gets attention, even from executives who are accustomed to the acronym, “ROE.” However, business executives define ROE as return on equity,a standard accounting measure reflecting the return on shareholder investment in a company. Unfortunately, ROE in the above example was calculated as follows: Using a scale of 1 to 100, clients of the learning and development program rated their level of program satisfaction. The average score was 85.2; hence, the ROE. This was presented as business results.

In reality, this calculation of ROE represents reaction data, Level 1 in the classic evaluation frameworks offered by Kirkpatrick and Phillips. Presenting reaction data as business results is deceptive, reflecting unfavorably on learning and development. Finance and accounting staff chuckle at what they see as a misuse of the classic ROE acronym.

ROE could be a concept, suggesting that client expectations are being met along a variety of measures, such as usefulness, relevance, and value. Taking this measure of ROE is simply a matter of asking clients if they are satisfied with the program—a Level 1 reaction measure.

Perhaps ROE is an objective. Some suggest that ROE is based on achieving objectives or certain outcomes. If the outcome is productivity, quality, or sales, for example, the measure becomes results or impact, Level 4 under both Kirkpatrick and Phillips evaluation frameworks. If this is the case, why notcall the outcome results or impact? If ROE represents an objective where the client sets an expectation about what participants should do, then the results represent behavior changeor application (Level 3 in both Kirkpatrick and Phillips frameworks). If the client suggests that participants acquire certain knowledge or skills, the objective is the classic learning objective, Level 2 in both Kirkpatrick and Phillips frameworks.

Vague definitions leave decision-makers little basis for their decisions. However, definition is a minor issue when compared to how this ROE is developed.

How to Develop the L&D ROE

The learning and development definition of ROE is vague, and its development follows an ill-conceived path. Some say the client develops the ROE entirely. This approach has two flaws. First, according to many learning professionals, managers and executives (clients) who request programs do not always know how to articulate specific measures of success. Clients may want the program to be “very effective”…what does thatmean? Or “we want best-in-class managers”…again, not clear or definitive. Leaving this process entirely to the client often presents nebulous, misguided, or misunderstood expectations.

The other flaw in allowing the client to set the expectation is that the expectation may be impossible. Suppose a client says, “I want 150 percent ROI!” Now the expectation is an ROI calculation. What would you do with that? The client may say, “We want to improve our sales by 100 percent in six months, something we have never achieved, but…” That may not be possible. Suppose the expectation is: “We want zero unplanned absenteeism in our call center”—again, an unrealistic goal. The point is that having the client set the expectation sometimes yields an unachievable target.

The next approach is negotiation with the client toward an appropriate expectation. This becomes the return on the negotiatedexpectations (RONE). While this may be the best approach because it can yield specific, appropriate, and realistic measures, why not classify the expectation in one of the classic five levels of evaluation rather than throw out another nebulous term?

We recently were involved in a situation where ROE went astray. A European broadcasting company spent millions of dollars on a leadership program. When the leadership development team attempted to define the expectation so they could use the concept of return on expectation, the CEO stated, “My expectation is effective leader behavior. There is no need for you to collect data; I will tell you if my expectations are met.” Based on the limited parameters stated by the CEO, the team collected no follow-up data.

The CEO was fired midway through the project, and the new CEO asked about the status of the program. The leadership development team explained they were measuring the program using the return on expectation as defined solelyby the previous CEO. Now they were caught in an embarrassing situation as the new CEO facetiously suggested that the former CEO be brought back to evaluate the program. Frustrated, the CEO stopped the program and fired the leadership development team, stating they had wasted a great deal of money. This extreme case demonstrates the risk of working from a nebulous expectation that is understood by one long-gone executive.

The Role of the CFO

Perhaps there is no more important influence on funding for learning and development than that of the Chief Financial Officer (CFO) and the finance and accounting team. Today the CEO expects the CFO to show the value of non-capital investments, which requires the finance and accounting team to be involved in our work. At the same time, many HR functions now are reporting up through the CFO, adding pressure to show value. Given the importance of this function, it is helpful to ensure that measures used to gauge learning’s success get their approval.

Don’t Mess With Finance and Accounting

The word “return” comes from the accounting field, most notably referring to the return on investment (ROI), a financial concept defined as “earnings divided by investment.” In the context of learning and development, ROI is net monetary benefits from the program divided by the cost times 100. This yields the ROI percentage. ROI positions learning as an investment. The accounting field also uses the word “return” in many other expressions, such as “return on equity” (ROE), “return on assets” (ROA), and “return on capital employed” (ROCE).

The concept of ROE raises a red flag to accountants, as it references fundamental financial terms. Compounding the confusion around measures of “return” are variations of return on expectation. These include return on anticipation (ROA), return on inspiration (ROI), return on information (ROI), return on involvement (ROI), return on client expectation (ROCE), and return on event (ROE). Even worse are return on training (ROT), and return on people (ROP). Some professionals have even used the concept of return on objectives (ROO), suggesting that this is a completely different process from measuring the success of objectives at different levels.

The issue here is twofold. First, use of the word “return” piques the interest of finance and accounting, because it is the basis for many of their common measures. Second, misusing terms with which finance, accounting, and top executives clearly identify decreases our credibility. From their perspective, we are unwilling to show the actual value of what we do in terms that they understand. We are substituting a new term, hoping that they will see value in this.

Who’s the Real Client?

Identifying the real client for a learning and development program is often a murky issue. The client funds the program and has the option to invest in other initiatives. This client is interested in the value of learning and development as expressed in terms theyunderstand, often leading to business impact measures and ROI.

For example, in a large, multinational organization, the centralized learning and development function develops programs used by the different business units. Each business unit has a learning and development advisor who serves as a liaison with the corporate university. From the corporate university perspective, the client is the learning and development advisor—their principal contact. From this perspective, the client is another learning professional. What this individual may view as value could be different from the business unit head who ultimately provides funding, allowing individuals to be involved in the program. This client is paying for the program through transfer charges and absorbing associated administrative and travel expenses.

In reality, the business unit head is the real client. If you ask thatclient about expectations from learning and development, you will receive a different description than the one you might get from the advisor. The difference comes from their perspectives. The learning and development advisor essentially sees this as theirprogram. They have asked the corporate university to conduct the program, and they assume some ownership from that request. It follows that if the program does not deliver value, it could reflect unfavorably on them. This fear of results often forces them to use a vague measure no one understands. It presents an easy way out and avoids the risk of the program not delivering the value that the business unit head desires.

Focus on Business Contribution

Most executives want to see alignment with the business needs, and learning and development success expressed as a business contribution. In a recent survey sponsored by ASTD of Fortune 500 CEOs, top executives weighed in on the types of data that matter to them. The No. 1 measure CEOs want to see is the connection of learning and development to the business (Level 4 business impact). Ninety-six percent of CEOs responding to the survey want to see these data; however, only 8 percent receive the data. In the same study, 74 percent of the top executives desired to see the ROI from learning and development. Only 4 percent are seeing it now.

The gap in what CEOs want and what they receive presents a challenge. Learning leaders must meet the expectations of executives who ultimately fund learning and development functions. Without their commitment and funds, learning and development would not exist as a formal process. The term, technique, or process used to measure success must be defined by contribution meaningful to the real client.

Back to Basics

An easy way accomplish business alignment is to consider objectives at multiple levels. Learning objectives are developed with performance-based statements, and sometimes include a condition or criterion. However, from the client perspective, these objectives represent only learning; there are other important levels of objectives. Application objectives (Level 3) clearly define what the participants should do with what they learned. Examples of Level 3 objectives include:

• At least 99.1 percent of software users will follow the correct sequence after three weeks of use.
• The average 360-degreen leadership assessment score will improve from 3.4 to 4.1 on a 5-point scale in 90 days.
• Some 95 percent of high-potential employees will complete all steps in their individual development plans within two years.
• Sexual harassment activity will cease within three months after the zero-tolerance policy is implemented.
• Some 80 percent of employees will use one or more of the cost-containment features of the health-care plan in the next six months.
• By November, pharmaceutical sales reps will communicate adverse effects of a prescription drug to all physicians in their territories.

Impact objectives specify what the application will deliver in terms of business contribution. These Level 4 impact measures communicate the consequence of application, usually defined in the categories of output, cost, and time. Examples of Level 4 objectives include:

• The Metro Hospital employee engagement index should rise by one point during the next calendar year.
• There should be a 10 percent reduction in overtime for front-of-house managers at Tasty Time restaurants in the third quarter of this year.
• After nine months, grievances should be reduced from three per month to no more than two per month at the Golden Eagle tire plant.
• Sales expenses for all titles at Proof Publishing Company should decrease by 10 percent in the fourth quarter.

Impact objectives connect the program to the business. In some cases, ROI objectives are set and expressed as a benefit/cost ratio, and ROI as a percent.

Defining expectations and developing objectives that link to meaningful business measures positions any learning and development program for results that resonate with all stakeholders, including the real client.

The Final Word

There is no need for another ambiguous term that creates more confusion. Return on expectation, return on anticipation, return on client expectation—such terms generate meaningless measures and risk misunderstanding among the clients funding our programs. These terms do little to satisfy executive interest in learning and development programs, when their focus is business contribution to the organization. Learning leaders must step up to the challenge and avoid the temptation to grasp trendy jargon or techniques that sound appealing but do little to demonstrate the real value of learning and development.

Dr. Jack J. Phillips, an expert on accountability, measurement, and evaluation, is co-founder of the ROI Institute, a source of ROI competency building, implementation support, networking, and research. Dr. Phillips provides consulting services for Fortune 500 companies and major global organizations and presents at conferences throughout the world. He received ASTD’s Distinguished Contribution to Workplace Learning and Development Award for his work on ROI.

Dr. Patti P. Phillips is president and CEO and co-founder of the ROI Institute, Inc. An expert in measurement and evaluation, she helps organizations implement the ROI Methodology in more than 40 countries around the world. Together, the Phillips have edited or authored more than 50 books. They can be reached at info@roiinstitute.net.