Introduction
Measuring the Return on Investment (ROI) in training and development has consistently earned a place among the critical issues in the human resources development (HRD) field. The topic appears routinely on conference agendas and at professional meetings, and journals and newsletters regularly embrace the concept with increasing print space. At least a dozen books provide significant coverage of the topic. Even top executives have developed an appetite for ROI information (Phillips, 1997). Leadership educators may soon find that program sponsors and administrators will be asking for Return on Investment information as well.
Although the interest in the topic has heightened and much progress has been made, it is still an issue that challenges even the most sophisticated and progressive HRD departments and those involved with leadership development programs. Some professionals argue that it is not possible to calculate the ROI of many programs, while others develop measures and ROI calculations. Regardless of the position taken on the issue, the reasons for measuring the return are still there (Phillips, 1977). Most professionals involved in training and development share a concern that they must eventually show a return on their training investment and thereby abandon some of the more traditional methods of evaluating programs.
Leadership educators and human resource professionals have used the term “evaluation” in a variety of ways. Most professionals would agree that the term evaluation implies a change in something or connotes the value or worth of a program or training. How one measures the change, value or worth varies greatly; however, most evaluations seem to fall into one of four categories. A survey conducted by Training (1996) surveyed over 40,000 training managers and specialists to determine the status of how training was evaluated. Table 1 shows the results.
Table 1. Evaluation of training in industry.
|
Level |
As a Percent of Organizations Measuring at this Level |
Percent of Courses Measured at this Level |
|
Level 1: Training Reaction |
86% |
83% |
|
Level 2: Learning |
71% |
51% |
|
Level 3: Behavior |
65% |
50% |
|
Level 4: Business Results |
49% |
44% |
Source: Training, October 1996, p. 63.
As indicated in Table 1, the survey conducted by Training indicated that most evaluations were at the lower levels measuring a participants’ reaction to a program (Level 1), or measuring skills, knowledge or attitude changes (Level 2). Measuring behavioral changes (Level 3) and the business impact of the program (level 4) were the two remaining levels of evaluation. The report indicates that, surprisingly, these last two levels (3 and 4) accounted for 65% and 49%, respectively, of the evaluations organizations conducted. Phillips (1997) noted a distinct trend toward increased accountability of training especially at higher levels of evaluation where training is connected to business results. As a result of this trend, Phillips (1997) concludes that there is a more prominent or fifth level of evaluation – Return on Investment (ROI). This level of evaluation compares the monetary value of the results with the cost for the program and is usually expressed as a percentage figure. Despite the increased prominence of this evaluation and its corresponding high value of information and client focus, evaluators use it less than other types of evaluations, as it is more difficult to administer and assess (Figure1).
Figure 1. Characteristics of evaluation levels
|
Chain of Impact |
Value of Information |
Customer Focus |
Frequency of Use |
Difficulty of Assessment |
|
Satisfaction |
Lowest |
Consumer |
Frequent |
Easy |
|
Learning |
||||
|
Application |
||||
|
Impact |
||||
|
ROI |
||||
|
Highest |
Client |
Infrequent |
Difficult |
|
|
Customers: Consumers – The consumers who are actively involved in the training process. Clients – The Customers who fund, support, and approve the training project. |
||||
Source: Jack J. Phillips, (2002) ROI: Providing a Balanced Viewpoint of Program Success. Center for Research, Salt Lake City.
Evaluation of leadership programs has largely been limited to the lower levels of evaluation. The W. K. Kellogg Foundation (2002) reviewed 55 major leadership development programs with the hope that its review would shed some light on how various programs are evaluating outcomes and impacts. Their key findings indicate that most programs measured individual leadership outcomes in the traditional ways-participant reactions, changes in knowledge, skills, attitudes, behaviors, relationships built as well as other similar areas. Few deal with the monetary value of the program-comparing results with cost or ROI. At best, some programs report leveraged dollars or money, collaborations of program participants, organizations or communities have attracted as a result of the program. The implication with such a measure is that these dollars, being attracted as a result of the program, improve leadership capacity.
Regardless of the setting, ROI is now taking on increased interest (Phillips, 1997). Executives and program directors that have watched their training budgets grow without appropriate accountability measures are now demanding a return on training investment. Leadership educators are not exempt from this dilemma, as most are being asked to report impacts, and more will be asked to report return on investment.
This paper reports the return on investment of the Southern Extension Leadership Development (SELD) program as implemented at The University of Georgia. A brief explanation of that program is provided below, followed by an explanation of the ROI model and the calculations used to determine the return on investment for the Georgia SELD program.
Southern Extension Leadership Development (SELD)
During the past decade, the Cooperative Extension System has faced an era of economic scarcity and has been impacted by a number of internal and external challenges (Ladewig & Rohs, 2000), many of which have changed the nature of work and authority relationships. Organizations that respond to the changing nature of work and authority relationships are learning organizations (Senge, 1990).
A major challenge impacting the transition to a learning organization is that few extension administrators are professionally trained in competencies and styles of leadership appropriate for learning organizations. Rather, they have been promoted to leadership positions because they excelled in their subject-matter discipline, and they have learned their new craft by emulating those who preceded them. While this practice is commonplace throughout the industrialized world, these administrators often lack the necessary leadership competencies to truly transform their organizations to compete in the information technology era (Patterson, 1998).
In response to the growing need to understand and cope with the many changes currently and potentially impacting the Extension System, Cooperative Extension directors and administrators of the Southern Region called for the establishment of a regional leadership development program. The result was the formation of Southern Extension Leadership Development (SELD).
The SELD program is unique in that the competency-based approach builds around the skill individuals and groups in Cooperative Extension need to be effective in the future. With such knowledge, Extension educators can design professional development plans that are relevant, useful, and customized to their needs. While SELD conducted regional workshops, it encouraged individual states to implement their own leadership development program. The centerpiece of SELD is the Managerial Assessment of Proficiency (MAP), developed by Training House, Inc. of Princeton, NJ. The assessment portion is a video-driven, competency-based, computer-scored simulation consisting of 200 items that assess a participant’s proficiency in 12 competencies. The 12 competencies are the following: Managing time, setting goals, planning and scheduling work, training, coaching and delegating, appraising people and performance, disciplining and counseling, listening and organizing, giving clear information, getting unbiased information, solving problems, making decisions and weighing risk, and thinking clearly and analytically. A series of competency building workshops followed the assessment portion to strengthen participants’ weaker competency areas. In Georgia the MAP/SELD program is open to all employees.
Return on Investment (ROI) Model
Calculating the return on investment begins with the basic model illustrated below (Figure 2). The ROI model provides a systematic approach to calculating return on investment and helps to simplify the approach. This step-by-step approach keeps the process manageable so that users can tackle one issue at a time (Phillips, 2002).
Figure 2. Return on investment model.
Isolate the Effects of Training
Tabulate Program Cost
Identify Intangible Benefits
Calculate the ROI
Convert Data of Monetary Value
Collect Data
Source: Jack J. Phillips (1992), Return on Investment in Training and Performance Improvement Programs, Houston: Gulf Publishing
Process
Data collection is an important part of the ROI process. In some programs pre and post data are collected to measure participants’ reactions or learning in a program. How the learning is applied and the impacts of the training program are important measures that help to document the program’s effectiveness and are critical in this chain of impact. Lacking this preliminary data or evidence makes it is difficult to isolate the effects of the program. Participants not reacting favorably to the program or having no learning or application of the material make it very difficult to conclude that any performance improvements were the result of the program.
The next step attempts to determine the amount of output performance that can be directly related to the program. Since many factors can influence performance data, this step becomes essential in the ROI process. Several strategies have been used to isolate the effects of a training program. These include having program participants, their supervisors, senior administrators or even experts provide estimates of the impact of the training on some specified output variable, (i.e., better customer service, increased sales, etc.). More sophisticated methods such as trend analysis or forecasting models could be employed. However, the most effective strategy to isolate the effects of training is the use of a control or comparison group if possible.
Converting data to monetary values, the next step, is often the most challenging step, particularly with programs that are subjective and behaviorally oriented. To convert data to a monetary value one must first identify the unit of measurement or improvement. Some hard data examples include the number of units produced or sold, employee overtime, unit costs, product defects, etc. Soft data examples include absenteeism, increased competency levels, frequency of use of new skills, employee turnover, attitude changes or a one-point change in a customer satisfaction scale.
After the unit of measurement is identified, one can employ specific strategies to convert data to a monetary value. These may include the use of historical costs kept by the company or organization, and the use of internal or external experts to estimate the value of the improvement. Asking program participants to estimate the value of the “improvement variable” is another option, as well as obtaining value estimates from participants’ supervisors or senior management or the Human Resource Development staff of the agency or company.
Next consideration should be given to intangible measures that represent the benefits or detriments that may be linked to a program. In most instances these intangible benefits are very difficult to convert to monetary values. In some programs, such as interpersonal skills training, team development, leadership, communications training, and management development, the intangible benefits can be more important than the monetary or tangible measures (Phillips, 1997). Some typical intangible variables include stress reduction, attitude survey data, teamwork, conflict, cooperation and communication data. Intangible benefits, however, can provide supporting evidence of the program’s success.
Tabulating the costs is the next step in the process. It involves monitoring or developing all related costs of the program targeted for the ROI calculation. Among the cost components that should be included are the following:
- The cost of the design of the program, prorated if possible over the expected life of the program;
- The cost of all program materials provided to participants;
- Instructor cost, including preparation time and delivery time;
- The cost of facilities for the program;
- Travel, lodging and meal costs for participants, if applicable;
- Salaries, plus employee benefits of the participants who attend the program; and,
- Administrative and overhead costs of the training function.Phillips (1997) states that specific costs related to the needs assessment and evaluation should be included, if appropriate, and recommends the conservative approach, including all of these costs so that the total is fully loaded.In addition the ROI (Return on Investment) process requires two basic calculations. The first is the Benefit/Cost Ratio (BCR) and the second is the Return on Investment (ROI). These formulas and an example are presented below.
BCR = Program Benefits Program Costs
ROI (%) = Net program Benefits x 100 Program Costs
The BCR utilizes the total benefits and costs. The ROI formula subtracts the costs from the total benefits to produce net benefits that are divided by the costs. For example, a program at Peach State, Inc. produced benefits of $283,500 with a cost of $40,500. Therefore, the BCR ratio would be:
BCR = $283,500 = 7.0 (or 7.0:1)
$40,500
As this calculation shows, every $1 invested returns $7 in benefits. In this example, net benefits are $283,500 – $40,500 = $243,000. Thus, the ROI would be:
ROI (%) = $243,000 x 100 = 600%
$ 40,500
This means that for each one dollar invested in the program, there is a return of $6 in net benefits. The benefits are usually expressed as annual benefits, representing the amount saved or gained for a complete year after program completion. While the benefits may continue after the first year if the program has long-term effects, the impact usually diminishes and is omitted from calculations.
ROI: A Leadership Example
The Georgia SELD/MAP program employed the ROI model to calculate the benefit cost ratio and return on investment. The steps and accompanying information for each follow.
- Collect DataExtension agents hired between 1995 and 2001 who completed their probationary period of 18 months with the Cooperative Extension Service formed the population for this study. Based on extension personnel records, The University of Georgia hired a total of 216 new county Extension agents (CEAs), all of whom participated in the same new employee orientation program. Only 185 CEAs completed their probationary period during this time and were included in this study. Basic background data was also gathered on these employees, data that included gender, degree level (e.g., BS, MS), ethnic background, and job responsibility (i.e., agriculture, family and consumer sciences, 4-H & youth, etc.) and if the agent resigned or was terminated. Additional data was collected from those CEAs who completed the SELD/MAP program, and administered pre- and post-evaluations to the participants asking them to rate their perceived level of knowledge regarding the 12 competencies on which they were assessed. Data indicated a significant difference between pre- and post-scores with post-scores being significantly higher than pre-scores. A six month follow-up evaluation involved asking participants to share their thoughts about the usefulness of the program, their plans for participating in any professional development activities linked to their program experience, and if any aspects of the program stood out in their minds as especially strong or weak, positive or negative. Most of the comments from this follow-up survey indicated that participants felt the program was very worthwhile and beneficial to their job in that it helped them identify their strengths and weaknesses and provided them with the opportunity to improve their weaker competencies
- Isolate the Effects of TrainingBased on personnel records these new CEAs were then grouped into one of two groups. The first group was those individuals who completed the Georgia SELD/MAP program and the second group represented those who did not. These groupings provided a built-in control or comparison group since all new employees participate in the same orientation and foundations training program.The result was that 40 CEAs completed the Georgia SELD/MAP program and 145 did not. Furthermore, only two CEAs in the SELD/MAP group had left the organization and 38 of the CEAs in the non-SELD/MAP group had left the organization during this time period. These figures represent an employee turnover rate of 5% and 26% respectively. It was this difference in turnover rates (5% vs. 26%) that provided the basis for calculating the program benefits in monetary terms.
- Converting Data to Monetary ValuesThis step was the most challenging. Since the data collected on new employees suggested that those who completed the SELD/MAP program were less likely to leave the organization, the unit of measurement selected was employee turnover. Senior management and a specialist in Human Resources determined the monetary value of the unit being measured-employee turnover. In a weekly newsletter to all faculty and staff, the dean of the college stated that every Extension agent or paraprofessional lost through resignation costs the college a minimum of $50,000 and $35,000 respectively on an annual basis. These two sources determined that these figures were accurate and very conservative in nature.Based on these figures the potential turnover rate among the SELD/MAP participants could have been as high as 26% or 10 CEAs leaving instead of the actual 2 CEAs at the 5% turnover rate. Thus, eight (10-2 = 8) fewer CEAs left the organization benefiting the Georgia Extension organization in the amount of
$400,000 ($50,000 x 8 = $400,000). This figure would later be used to calculate the benefit cost ratio and return on investment.
- Identify Intangible BenefitsA six month follow-up evaluation indicated that many participants felt the program had helped them identify their weaknesses and increase their administrative, communicative and team building skill. These new skills improved their working relationships with colleagues and those they supervised, as well as volunteers and clients. Several reported increased job satisfaction and organizational commitment as additional indirect benefits.
- Tabulate Program CostsTo calculate the ROI for the Georgia SELD/MAP program, the following program cost components were used:*Program Design Costs- license fee, costs for videotapes, etc. per individual @
$10.00. Total cost = $400.00
*Program Material Costs to Participants- @ $60.00 per participant for MAP and
$100.00 per participant for follow-up Excel workshop.
*Participants = 40. Total Cost = $2400.00 + $4,000.00 = $6,400.00.
*Instructor Costs – prorated salary time @ $500.00 per day (included all fringe benefits and costs to the organization/university). Two instructors – B12 days per year for 6 years. Total Cost = $36,000.
*Facilities Costs – State 4-H Center charge for use of facilities and equipment @
$8.00 per participant for follow-up Excel workshop ($8.00 x 40 = 20.00). Total Cost = $320.00.
*Travel Costs – 28 cents per mile for auto, $28.00 per day for meals and $55.00 per night for lodging, if necessary (includes instructor and participant travel costs). Amounts based on attendance and travel records. Total Cost = $10,460.00.
*Salaries plus employee benefits for participants – based @ $50,000 prorated per year for number of days in the program (six days). The $50,000 figure was arrived at based on administrative communications and college business office (Human Resources Department) calculations. Cost computation – $200.00 per day x 6 days
= $1200 per participant. $1200.00 x 40 participants = $48,000.00.
*Administrative and Overhead Costs – @ 2% of total cost = $2,031.00 (accounting figure used by college business office).
TOTAL PROGRAM COSTS = $103,611.00.
- Calculate the Return on Investment (ROI) Using the formulas for BCR and ROI, the following calculations were made: TOTAL PROGRAM COSTS = $103,611. BCR – $400,000 / $103,611 = 3.86. ROI – ($400,000-$103,611) / $103,611 x 100 = 286%. Thus the benefit cost ratio (BCR) calculation shows that every $1 invested returns $3.86 in benefits and each $1 invested in the program returns $2.86 in net benefits.
Discussion and Summary
Calculating the ROI of a leadership program is not easily done. Many programs are not able to convert outcome measures to monetary terms. In this particular case one measure could be used-employee turnover. Identifying a unit of measurement that can be converted to a monetary value will present a challenge to those involved in leadership training. While business and industry can provide some examples that may help identify outcome measures easily converted to monetary value, many leadership programs are conducted in settings less conducive to assigning monetary values to outcomes with any degree of validity. Fortunately in this study personnel records and travel data were either easily accessible or readily computed with relative ease.
Initial analysis established a low (r = .20) statistically significant (p<.001) correlation between participation in MAP/SELD and employee turnover. Those who participated in MAP/SELD were more likely to remain in the organization than those who did not participate in the program. Furthermore, the access to personnel records, while not complete, allowed for some additional analysis of background data and whether or not the two groups of Extension agents (MAP/SELD participants vs. non-participants) differed significantly from each other. The data that was analyzed in this study included gender, degree level (BS, MS) whether they were still employed by the Georgia Extension Service (see Table 2), major job responsibility (agriculture, family/consumer science or 4-H youth), and ethnicity (see Table 3). The only statistically significant difference (p< .01) found between the two groups was that those who did not participate in the MAP/SELD program were more likely to leave the organization than those who did (see Table 2).
Table 2. Test of significance between groups on selected variables
|
Variable |
Map/SELD N = 40 |
No Map/SELD N = 145 |
t-value |
|
Gender (1 = male, 2 = female) |
1.60 |
1.60 |
1.00 |
|
Degree Level (1 = BS, 2 = MS) |
1.49 |
1.50 |
0.96 |
|
Resign/Terminate (1 = yes, 2 = no) |
1.93 |
1.73 |
-2.65* |
*p < .01
Table 3. Chi-square analysis of selected variables.
|
Map/SELD N % |
No Map/SELD N % |
Chi-Square |
|
|
Agriculture |
15 38 |
50 35 |
|
|
Science |
2 |
24 16 |
|
|
4-H Youth |
23 57 |
71 49 |
|
|
X2 = 3.50* |
|||
|
38 |
127 88 |
||
|
2 |
18 12 |
||
|
0 |
0 0 |
X2 = 1.79* |
*Not significant
This study indicates that every one-dollar spent in the SELD/MAP program generates $3.86 in benefits and returns $2.86 in net benefits on the investment. Had all those extension agents hired between 1995 and 2001 participated in the program, employee turnover might have been reduced by 21%, saving the organization a minimum of $1,550,000.00 ( 31 fewer terminations x $50,000=$1,550,000) in annual employee turnover costs. Since these terminations occurred after an average of 2.33 years of employment, the total savings to the Georgia Cooperative Extension Service could have exceeded $3.62 million ($1,550,000 x 2.33 years).
While this study does not account for all the variables that may influence whether a person continues with the organization or not, one of the key factors can be the type of management and leadership training they receive. In some instances a monetary value can be calculated to show the benefits of such a program. With good record keeping, program costs can also be determined. With benefit and costs figures, the ROI on a program can be easily computed.
References
Ladewig, H., & Rohs, F. R. (2000). Southern extension leadership development: Leadership development for a learning organization. Journal of Extension, 38(3), http://www.joe.org/joe/2000june/az.html.
Patterson, T. J. (1998). Commentary II: A new paradigm for extension administration. Journal of Extension, 36(1), http://www.joe.org/joe/1998february/comml.html.
Phillips, J. J. (1997). Return on Investment in training and performance improvement programs. Houston: Gulf Publishing Company.
Phillips, J. J. (2002). ROI: Providing a balanced viewpoint of program success. Salt Lake City: Center for Research, November.
Senge, P. M. (1990). The fifth discipline: The art and practice of the learning organization. New York: Doubleday.
Training Evaluation-Industry Report: Who’s Learning What? (1996, October). Training Magazine, p.63.
W. K. Kellogg Foundation (2002). Evaluating outcomes and impacts: A scan of 55 leadership development programs. Battle Creek: Kellogg Foundation, August.