One of the questions our clients are always looking to answer before implementing their eLearning courses, is "How will this impact my Return on Investment?" Here's a great article by Patrick Lambe that discusses the topic of ROI when it comes to online training courses:
One of the ongoing admonitions e-learning specialists face is the need to "prove" the return on investment (ROI) for each program they initiate.
But most e-learning programs are funded from ongoing training budgets, which are recurrent (that is, budgeted annually to cover the cost of a year's worth of training, rather than individual projects) and are accounted as operations or infrastructure costs.
Because much training is not budgeted at the project level, discussions about ROI have traditionally had little relevance. It would be like seeking the ROI on groceries in a personal budget or of stationery supplies in a business. Both are simply costs of doing business.
How, then, might one assess the return on an e-learning program?
First, it's important to be clear on the economics of e-learning, which I explain in this article. Second, we must understand some of the most common ways that organizations deploy e-learning to support their strategic objectives, and show how measurement of viability and impact can be approached within those situations.
The Economics of e-Learning vs. Training
A good understanding of the economics of e-learning involves realizing that e-learning is infrastructure.
Infrastructure. Infrastructure is the composite of technology systems, business processes, work culture, and control mechanisms that make up the way an organization operates and does its work. Infrastructure is distinct from assets and resources insofar as an existing infrastructure enables or constrains the performance of assets, such as specific technologies, people, and availability of time. If e-learning were merely a resource or asset, it would be an interchangeable part that could be plugged into an organization and become operational at the flick of a switch.
But that's not the case. In fact, as some of the early implementers of large-scale e-learning initiatives discovered, it introduces significant problems of change management because e-learning is more than an interchangeable, plug-and-play part. Making e-learning work involves more than integrating technology systems. It also involves re-gearing work processes, tuning into soft work cultures, and deeply integrating learning with the needs of business and work processes.
In contrast, although a training class might disrupt employees' work schedules, the class requires no other deep integration. And, although making training stick does require broader change in the workplace, research suggests organizations still offer classroom training without these related changes (see Transfer of Training: Action-packed strategies to ensure high payoff from training investments, by Broad and Newstrom, 2001).
Pervasive nature. This highlights another important difference between training and e-learning. Training has established for itself a distinct niche and role within an organization's infrastructure. It is compartmentalized and relatively easy to manipulate. But e-learning pervades the entire infrastructure. As part of the IT network of the organization, learning content is potentially available at any point in the flow of work, in any business process, and as part of any decision.
Recognizing the role of infrastructure in e-learning has two significant implications for discerning its economic impact or viability. First, because an infrastructure is a complex interweaving of diffuse and diverse elements ranging from "hard" technology to "soft" culture, attributing specific outcomes to specific interventions is extremely difficult. Infrastructure is generally opaque when linking causes to effects, and this substantially contributes to its apparent inertia in the face of change.
Second, any infrastructure-related initiative such as e-learning, has an absorption or integration cost that is extremely difficult to anticipate, precisely because of the opacity of infrastructure.
In layman's terms, an absorption or integration cost is the cost of change management, such as unanticipated costs associated with the technology, like the cost of providing tutoring support to learners (which is a cost that most organizations do not consider when deciding to launch an e-learning effort) or that, despite the fact that staff can access the learning anywhere at any time, if the work environment does not support transferring the learning to the job, learners will not become more productive.
Transfer of knowledge. It's crucial that organizations invest in efforts intended to transfer learning to the job. Whatever these unanticipated change management costs are, they are universally under-anticipated. The consequence is that the economic benefits of e-learning are usually over-estimated and the post-implementation results usually disappoint executives, reducing their confidence in a variety of learning-related issues, starting with the mechanisms for assessing the viability of a proposed e-learning investment.
Benefits of the Infrastructure Perspective
Recognizing the infrastructural aspect of an e-learning initiative is highly advantageous for three reasons.
First, it focuses executives' attention at the beginning of the project on likely integration and adaptation issues with a greater level of precision than if the metaphor of a plug-and- play box is used.
Second, e-learning initiatives that seek to unlock the constraints of an existing infrastructure are likely to have fewer challenges in installation and implementation because they are simpler than those that attempt to graft a complex e-learning system onto an existing infrastructure. In the latter situation, the grafted e-learning system will have more numerous points of integration and, as a result , more issues to address. This is why small, highly localized and focused initiatives always have higher success levels than large-scale enterprise wide initiatives. The smaller initiatives impinge upon, and depend upon, fewer elements of the infrastructure for their success.
The third implication of recognizing e-learning as an infrastructure initiative is that a simplistic ROI model would be inappropriate for all but the most tightly defined productivity-oriented initiatives. For example, simplistic ROI assumes that a quantifiable output can be attributed to the effect of a quantifiable input. But infrastructure often renders the relationships between inputs and outputs opaque. In many cases, the establishment of an infrastructure is not about a single input and output, but several, often unrelated, ones. Attributing the initiative to just one of them seems to underestimate the larger benefits.
Moreover, a complex system has a variety of inputs that affect a variety of outputs, many of which are unrelated to one another or that affect a variety of situations. Purely quantifiable and objective measures are increasingly inadequate for capturing the full range of value that such an infrastructure might create.
So metrics such as "contingent valuation" and "outcomes-based evaluation" are increasingly used for measuring the impact of other intangible, knowledge-based interventions, such as public library services or knowledge-based ecological systems. Because of the similarities of such initiatives to e-learning, perhaps they could be used for measuring the impact of e-learning, too.
Contingent Valuation Model
The key premise of the contingent valuation model is that it measures infrastructures at two different levels. One measure focuses on the value created by the entire ecological system. The other measure focuses on the inter-relations and functionality of the parts as valued by key stakeholders at different points in the ecological system. An ecological system includes all of the facilitating and constraining facets of the environment (in this case, the technology infrastructure) (see S. Macmillan's 2007 doctoral dissertation, "Development of writing for research purposes: An ecological exploration of graduate level non-native English speaker writing process").
The concept is not unique to technological infrastructure; the concept also applies to the design of classrooms.
Contingent valuation was originally developed in 1993 by Nobel prize-winning economists Kenneth Arrow and Robert Solow for the purpose of quantifying the value of ecological systems. It has been applied by a number of public and national libraries as a method of demonstrating the direct and indirect economic impact of the intangible services they provide, such as knowledge supply.
The method involves identifying key groups that obtain value from the services, and then quantifying the economic impact as well as the value perception of those services. This calculation is partly achieved by asking the stakeholders to consider what value they would lose if the service were not provided.
Outcomes-Based Approach
However, it's also possible to take a narrower approach based on objectives, or outcomes-based evaluation. It's similar to criterion-referenced instruction. Both start by identifying objectives at the beginning of the project—even before considering alterative solutions—and, after the solution is implemented, assessing the extent to which the objectives have been achieved.
In the using an outcomes-based approach to assessing infrastructure projects, the entering objectives are strategic business objectives that the infrastructure is mobilized to support. The assessment explores the extent to which these objectives were achieved. Outcomes-based evaluation has several strengths, particularly (but not exclusively) for commercial organizations because it encourages e-learning initiatives to support specific strategic objectives and define the measures of accountability for success. Because the objectives are strategic business objectives for which the organization is most likely already conducting assessment, existing measurement systems, such as the balanced scorecard, can be deployed.
Also worth noting is that, although productivity improvement is one possible type of strategic objective, it is by no means the only one.
Source: http://test.elearnmag.org/subpage.cfm?section=articles&article=107-1
By Patrick Lambe, Straits Knowledge, Singapore
