View AbstractView PDFShare Article

Bradley Burbaugh, Ph.D., Megan Seibel, Ph.D., Thomas Archibald, Ph.D.
DOI: 10.12806/V16/I1/A3

Using a Participatory Approach to Investigate a Leadership Program’s Theory of Change

Bradley Burbaugh, Ph.D.

County Extension Director University of Florida IFAS Extension

Megan Seibel, Ph.D.

Extension Specialist

Agricultural, Leadership, and Community Education Virginia Tech

Thomas Archibald, Ph.D.

Assistant Professor

Agricultural, Leadership, and Community Education Virginia Tech

Abstract

The linkages between interventions and outcomes of leadership initiatives have been insufficiently studied. To better understand these links, the Virginia agricultural leadership program conducted a pathway mapping session to investigate the program’s theory of change. In this novel process, program participants engaged in collaborative brainstorming to identify outcomes and connect them to program activities. This process helped participants reflect on their learning and helped program staff identify the mechanisms (i.e., learning activities) that led to desired outcomes. The group produced a visual model that represents the program’s theory of change depicting how the program develops capacity for leadership. Using program participants to identify the connections between pedagogy, practice, and outcomes can inform the evidence- base and development of evaluation plans for leadership development programs.

Introduction

A recent report on the state of leadership development programs noted that nearly 75% of respondents from a wide variety of industries and organizations view their leadership development programs as ineffective (Loew & Wentworth, 2013). Despite this finding, the amount of money spent to train and develop leaders has more than doubled from $7 billion in 1996 to over $15 billion in 2013 (Bassi, Gallagher, & Schroer, 1996; O’Leonard & Krider, 2014). Similarly, considerable investment of time and resources in understanding the outcomes of leadership development has yielded little clarity on the link between the learning activities and the emergence of these outcomes (Collins & Holton, 2004; Dinh, Lord, Gardner, Meuser, Liden, & Hu, 2014; Gardner, Lowe, Moss, Mahoney, & Cogliser, 2010). Insufficient consideration has been given to understanding the pathways of change in leadership development programs. The pathways or connections between activities and outcomes constitute a program’s theory of

change, which can be used to better understand the effectiveness of programmatic activities used to facilitate changes in individuals or groups.

To address the issue of unclear and under-articulated theories of change in leadership development programs, the purpose of the project described in this Application Brief was to better understand a newly developed leadership program’s theory of change using a participatory approach. As such, a pathway mapping session—a process for understanding a program’s theory of change (Gutierrez & Tasse, 2007)—was used to identify the pathways of change and develop a theoretical framework for the Virginia agricultural leadership program. This facilitated pathway mapping session provided an opportunity for program graduates to reflect on their learning, collaboratively identify salient outcomes, and articulate the connection (i.e., pathways) between these outcomes and sources of learning at the completion of their two-year experience. As a result, the learning activities and outcomes were linked together to form an overall visual model of how the leadership program is perceived to develop an individual’s capacity for leadership. This project advances the programmatic assessment and evaluation priority of the Association of Leadership Educator’s national research agenda (Andenoro, Allen, Haber-Curran, Jenkins, Sowcik, Dugan, & Osteen, 2013).

Background: Review of Related Literature

Agricultural Leadership Development. Leadership programs in agricultural contexts have served as a vehicle to develop the leadership capacity of adults and fill a need for effective rural leadership since the first agricultural-based program was established 50 years ago at Michigan State University (MSU) (Helstowski, 2001; Miller, 1976). Currently, there are 45 agricultural leadership development programs operating throughout the United States and abroad that associate themselves with the model advanced by the first program at MSU. Based on the input from a wide range of Virginia stakeholders (see Kaufman, Rateau, Ellis, & Kasperbauer, 2010), the Virginia program was established in 2011 and the inaugural class of participants graduated in 2014.

The mission and objectives of each agricultural leadership program differ according to local needs, but an early review of several programs by Howell, Weir, and Cook (1982) found that commonalities among programmatic objectives were to increase participants’ understanding of political, social and economic systems; to develop participants’ social skills; to help participants be effective spokespeople for their industry or community; to expand individual networks; and to develop future political, civic, and organizational leaders. Generally, these objectives are accomplished through a structured curriculum and developmental experiences that consist of “workshops and travel seminars intended to provide participants with an understanding of the social, economic, cultural, and political dimensions of public problems” at the community, state, national, and international levels (Howell, Weir, & Cook, 1982, p. 2).

Primary outcomes associated with the inaugural MSU program, and many programs since, include “personal and professional growth, ranging from expansion of personal perspectives and self-esteem to greater decision-making and involvement in leadership positions” (Kaufman, Rateau, Carter, & Strickland, 2012, p. 124). In one of the more recent discussions of program benefits, Kaufman and Carter (2005) wrote, “these programs encourage

the development of effective and responsible agricultural leaders that are capable of addressing industry issues and becoming active participants in public affairs” (p. 68). A 2007 survey of agricultural leadership program directors found the average cohort (i.e., class) lasted 21 months and consisted of 12 educational seminars delivered in-state, regionally, and internationally (Kaufman, Rateau, Carter, & Strickland, 2012). The [State] program consists of ten in-state, one

U.S. regional, and one international seminar over the course of 22 months.

Leadership and Learning. According to Day, Harrison, and Halpin (2009), leadership development is linked to adult development, and an implicit proposition guiding this project is that leadership is a complex learning process (Brown & Posner, 2001). As such, development is facilitated by sources of learning, which are the activities commonly associated with leadership programs (Allen & Hartman, 2008a). There is no clear consensus on a specific learning theory that informs leadership development (Allen, 2007). However, adult learning and leadership development are both closely connected with the concepts of change and transformation (Brown & Posner, 2001; Day, Harrison, & Halpin, 2009; Mezirow, 2000). Several scholars maintain that leadership development programs, and the associated sources of learning, provide individuals with the opportunity to fundamentally change their perspective and worldview (Brown & Posner, 2001; Lord & Hall, 2005). What’s more, Day and Halpin (2004) submit that such changes in worldview and perspective have become an important focus and outcome in leader development.

The sources of learning in leadership development programs vary according to the context in which the program is embedded, but they are considered the “primary vehicles for delivering leadership development learning activities before, during, and after the leadership development intervention” (Allen & Hartman, 2008b, p. 10). It is important to distinguish between sources of learning and a leadership intervention, which is a more ubiquitous term used to describe a “developmental experience using some form of training, introspection, receiving feedback and exercises to increase the effectiveness of how one leads an individual or group” (Avolio, Avey, & Quisenberry, 2010, p. 635). It is likely that most leadership development programs or interventions in existence today use multiple sources of learning and approaches to accomplish their objectives and mission “because no single approach is appropriate at all times” (Allan & Hartman, 2008a, p. 85).

Theories of Change and Leadership Development. Adult training initiatives, like leadership development programs, are often viewed as “black-boxes” that receive inputs (i.e., learning activities) and produce outputs (i.e., outcomes) (Stufflebeam & Shinkfield, 2007). A scan of 55 leadership development programs by Russon and Reinelt (2004) found that there is an increasing demand for evaluations that articulate a program’s theory of change when communicating outcomes and impacts. Developing a program’s theory of change can, thus, allow researchers and practitioners to look inside the “black-box” and examine the mechanisms that lead to desired changes and outcomes. A theory of change approach also focuses on the essential premise of development programs: that through a set of learning activities and experiences, individuals will change. As such, the theory of change approach has been described as a good fit for evaluating the effectiveness of leadership programs and as essential to sound programming (Gutierrez & Tasse, 2007; Rennekamp & Arnold, 2009). This approach can also advance theory, as the authors of a recent review on theoretical trends in leadership theory and

research write, “Linking processes to outcomes can advance theory, and it will also provide a firmer basis for leadership interventions” (Dinh, et al., 2014, p. 55).

A theory of change can be used to describe why and how a program works by identifying the relationships between programmatic activities and change (i.e., outcomes). Here, changes were operationalized at three levels: short, medium, and long-term; conceptualizations commonly used to measure change and/or outcomes that are linked via pathways in a theory of change model (Hannum, Martineau, & Reinelt, 2007). For example, short-term outcomes are typically related to gained knowledge, awareness, or skills and medium-term outcomes are often the actions individuals took as a result of this new knowledge or awareness. In this vein, short- term outcomes are preconditions to medium-term outcomes and both short and medium-term outcomes are preconditions to long-term outcomes (Gutierrez & Tasse, 2007). For this project, short-term outcomes were operationalized as the immediate effects of program activities and reflect changes in learning, such as knowledge, awareness, or skills; medium-term outcomes were identified as changes in behaviors and practices; and long-term outcomes were described as the conditions intended to change as a result of actions (Silverman, Mai, Boulet, & O’Leary, 2009). The key to articulating a program’s theory of change is identifying how the changes in knowledge lead to changes in behavior and how that impacts long-term outcomes (Anderson, 2006).

Pathway Mapping. Gutierrez and Tasse (2007), who write that “leadership development initiatives often have very broad and highly ambitious objectives that are not easily measurable,” also posit that one way to address this issue is through the development of pathway maps. The pathway mapping approach is a specific planning process that articulates a program’s theory of change using a logic model framework (Hannum et al., 2007). While logic models rely on columnar representations that link whole sets of activities to sets of outcomes, pathway models make these connections more explicit and precise (Urban & Trochim, 2009). These more precise connections “illustrate the relationships between actions and outcomes and also show how outcomes are related to each other over the lifespan of the initiative;” the relationships and connections between inputs and outputs are often referred to as pathways of change (Anderson, 2006, p. 3).

The pathway mapping process is one way to address the programmatic complexity of leadership development because participants in a pathway mapping session “must match each outcome in the pathway of change to a specific intervention, revealing the often complex web of activity that is required to bring about the desired long-term change” (Anderson, 2006, p. 6). The result of a pathway mapping process is a strategic picture of the multiple interventions required to produce the short and medium-term outcomes that are preconditions to long-term change. In addition, a pathway map can help program staff and participants articulate how a program works, what is needed for the program to work, and what can be done to provide evidence that the program does work (Weiss, 1997). Serrat (2013) posits that the development of a pathway of change map enables organizations to think more deeply about their work. In the leadership context, a strength of pathway mapping is that it can illustrate how and why a program works by identifying the pathways of change, but more importantly, the assumptions about the connections between activities and outcomes can be tested and measured (Anderson, 2006; Gutierrez &

Tasse, 2007). In many cases, the resultant pathway model is used to inform the development of a programmatic evaluation plan (Hannum et al., 2007).

Description of the Application

In an effort to better understand the outcomes of the Virginia agricultural leadership program, an evaluation focus group and pathway logic model development session was conducted with program participants. This process allowed participants to identify programmatic outcomes at the conclusion of their two-year experience, and connect these outcomes to specific learning activities of the program. This approach is novel in that it allowed participants to reflect on their learning, collaboratively identify salient outcomes, and articulate the connection between these outcomes and sources of learning. Additionally, this process helped program staff to identify the mechanisms (i.e., learning activities) that led to desired outcomes and to reveal the collective impact of programmatic efforts. As a result of this process, the learning activities and outcomes were linked together to form an overall visual model of how the [State] program develops an individual’s capacity for leadership. This approach is novel because it used program participants to identify the program’s theory of change based on their experience, rather than with program planners as typically described in the literature.

Participants. Inaugural program participants (N =10), the first to complete the two-year program, were asked to participate in the pathway mapping process on their July 2014 graduation day which was facilitated by an evaluation specialist with no previous experiences with the program participants. Two program staff members provided ancillary support for the facilitator throughout the project.

Process. The process was adapted from an outline provided by Gutierrez and Tasse (2007) and the guiding questions used for each stage were adapted from Anderson (2006). Prior to the mapping sessions, program staff met with the facilitator to provide an overview of the program, key activities, short, medium, and long-term goals of the program, as well as the assumptions guiding how and why the program should cause the desired changes in program graduates. Based on these meetings, a process protocol that included guiding questions was developed for the mapping session scheduled to take place over a 6-hour time period. During mapping, program participants were to provide feedback on long-term programmatic outcomes (stage 1), identify influential learning activities (stage 2), describe short and medium-term programmatic outcomes (stage 3), link activities and outcomes via pathways (stage 4), and provide feedback on the mapping session and process (stage 5).

Prior to stage one, the process was introduced to the participants by the facilitator who informed the group that the ultimate goal of our mapping session was to use their expertise and experiences to help program staff and stakeholders better understand how the [State] program works. An opportunity for participants to ask questions as well as read and sign consent forms was provided prior to stage one. This pre-mapping overview took approximately 20 minutes to complete.

Stage one of the process started with the identification of long-term programmatic outcomes. This type of backwards mapping process is indicative of pathway mapping sessions

(Anderson, 2006). Through this approach, a results framework can be drawn to explain how the long-term outcome was achieved through learning activities, as well as the short and medium- term outcomes that are preconditions to it (Anderson, 20006; Gutierrez & Tasse, 2007). The guiding questions for this stage were:

1. • In your opinion, what are the ultimate goals of the [State] program?
   • How should we define success in the [State] program?
   • What are you expecting to get/gain from your investment in the program?
   • Given what you know today, what will be different in your community or industry in the long term as a result of successfully reaching this “ultimate” goal for [State] program?
   • Can you please define long-term for me? (i.e., 5, 10, 20 years)

During this process, program staff took notes on large post-it sheets attached to the wall. After discussing and brainstorming the guiding questions for approximately 20 minutes, participants were given small post-it notes and prompted to share their thoughts related to long-term outcomes of the program. It was suggested that participants report one long-term outcome per post-it to facilitate clustering later in the process. After each individual shared their long-term outcome(s) with the group, their small post-it was placed on the large post-it affixed to the wall in the meeting room. There were approximately 21 long-term outcomes identified and shared by the group which took approximately 15 minutes to complete. The next step in the process was to identify commonalities among these 21 outcomes and cluster into 5 -10 long-term outcomes of the program. The clustering process was led by participants and took approximately 25 minutes. They talked through commonalities and differences until consensus was reached and the following six long-term outcomes emerged from the clustering process: (1) develop credible advocates for the agriculture industry, (2) the development of a strong, diverse network, (3) improved self-efficacy, (4) influence the sustainability and viability of the agriculture industry,

1. collectively address industry challenges, and (6) facilitate civic engagement. In a pathway mapping process, the identification of inputs (i.e., learning activities) should follow the identification of outputs (i.e., long-term outcomes).

   Stage two of the process involved identifying learning activities that were influential in their development during the course of the program. The guiding questions for this stage were:

   • What planned activities of the program work to accomplish these goals (identified in the previous stage)? How so?
   • What were the most effective activities?
   • What were the least effective activities?

For this stage, the brainstorming, individual sharing, and clustering processes outlined above were followed. This stage in the process took approximately 60 minutes to complete. At the completion of this stage, there were over 30 learning activities identified and shared by the group, which were then clustered by participants into the 11 most influential activities, which were : (1) appreciative inquiry techniques, (2) media trainings, (3) group and individual reflection activities, (4), assessments (e.g., MBTI, 360 Feedback, etc.), (5) farm and industry tours, (6) the international study tour, (7) engaging political, community, and industry leaders,

1. networking, (9) session-chairperson responsibilities, (10) training activities, and (11) relationship building.

   Stage three of the process involved the identification of short and medium-term outcomes by program participants. The guiding questions for this stage were:

   • What knowledge and/or skills have you gained as a result of your participation in the program?
   • What is the most significant change you have made as a result of the program?
   • Which of these are short-term, medium-term outcomes?Following the same process described above, the group enumerated over 40 outcomes, which they subsequently clustered into 8 short- and 7 medium-term outcomes, which were as follows: short-term: (1) a conceptual understanding of leadership, (2) self-awareness, (3) knowledge and understanding of the agriculture industry, (4), listening skills, (5) a better understanding of others and differing perspectives, (6) the development of dialogue skills to better understand and engage others, (7) enhanced communication skills, (8) and knowledge of how to engage diverse viewpoints; and medium-term: (1) relationship building, (2), technology adoption, (3) understanding the role of government in the agriculture industry, (4) information analysis and critical thinking skills, (5) the application of leadership principles and skills, (6) practice leadership habits, and (7) influencing policy.

     To prepare for the pathway mapping stage of the process, the facilitator and program staff wrote the results from each step of the process on sheets of paper for use by participants in the next stage of the process. The activities and outcomes from each stage were color coded to help participants decipher between activities, short, medium, and long-term outcomes. For example, each of 11 previously identified learning activities were written, one per sheet, on yellow paper. This process was repeated using a different color for each of the short, medium, and long-term outcomes. At the conclusion there were 33 sheets of paper in four different colors that represented the activities and outcomes identified in stage one through three. The facilitator and program staff also covered a blank wall in the meeting room with large white paper sheets for the pathway mapping process, covering approximately 20’ in length and 5’ in width.

     The sheets of paper representing the 33 activities and outcomes were placed on a large table and participants were asked the following guiding question for stage four of the process:

   • How do these activities and outcomes connect to, and build on each other in order to reach the long-term goals?During this stage, participants debated connections between activities and outcomes and when consensus was reached they placed the activities and outcomes on the paper-covered wall and drew the pathways of change connecting each with markers. This process continued for approximately 45 minutes until all of the activities and outcomes were connected to develop a pathway model that represented how the program worked in developing the leadership capacity of participants. The pathway map constructed by participants was documented with photographs and the large paper sheets covering the wall were also preserved. Using this documentation, program staff constructed an electronic version of the pathway map (Figure 1) using the Netway™ software program developed by the Cornell Office for Research on Evaluation.

     Stage five of the process provided an opportunity for participants to provide feedback on the mapping process. The following guiding questions were used to facilitate this stage:

   • When you look at the total picture, do you believe that the pathway map makes sense?
   • Do the activities and outcomes make sense as the logical steps toward the long-term outcomes?
   • Do you feel this process was beneficial? Why or why not?
   • What would you change about this evaluation process?
   • Would you recommend other leadership development programs participate in a similar process? Why or why not?
   • Did this process help you better explain how the Virginia program has impacted you and your classmates?
   • Any concluding thoughts with regard to this process?

This stage was recorded and transcribed to help the program staff document feedback on the process and better understand how to improve the pathway map.

Discussion of Outcomes

As a result of this process, the learning activities and outcomes identified by program participants were linked together to form an overall visual model of the Virginia program’s theory of change. This process documented how the program develops an individual’s capacity for leadership, helped reveal perceived changes in program participants, and captured intentional and unintentional outcomes of the program. The outcome of this process was a pathway map (Figure 1) which is a flowchart documenting how program activities are believed to lead to results over time. In addition, this approach served as a process evaluation of the program, and was of immediate utility to the program staff for planning purposes (Urban & Trochim, 2009).

The process also helped prioritize program outcomes that are both important to stakeholders and feasible to attain.

Activities

Short-Term

Outcomes

Mid-Term

Outcomes

Long-Term

Outcomes

Figure 1: Participant developed pathway map that represents the Virginia programs theory of change.

This process provided an opportunity for participants to reflect on their experiences and learning as well as to develop shared meaning. During the process evaluation stage, one participant explained how the pathway mapping session helped him when he explained:

“I think it will help us articulate what we’ve done. I think we’ve all struggled for the words a little bit; it’s been great, it’s been wonderful [the program], but I think this will help us be able to convey how we spent the last two years. Which will hopefully inform how we spend the next whatever [years]” (S2, p. 4).

When describing the novelty of our approach that included only program participants rather than program planners, one participant shared, “I don’t understand how you can do that kind of evaluation model without involving the participants” (S3, p. 1). The process also served as a way to develop better advocates for the program as they feel they can more clearly articulate how and why the program works. As evidence, one participant shared:

“It’s thought provoking, and it really forces you to put your thoughts, ideas, and experiences that we’ve had over the past two years to both verbal and written format to where you can communicate that to people who are interested or engaged with the group” (S1, p. 7).

In describing how we plan to use the theory of change developed with the group to inform an evaluation plan for the program, one participant astutely noted that the model is fluid and that it could change based on the findings from measuring and testing the pathways of change by sharing:

“Well, a part of evaluation is going back and testing the efficacy of what you’ve done, right? And so, if you identify there were outcomes that you want, and you’re not getting them, that would indicate there’s a problem with the model” (S4, p.2).

The underlying goal of this approach is to do just as these individual’s suggest, which is to make the underlying theory of change clear by “articulating the premises, assumptions, and hypotheses that might explain, the how, when, and why processes of change” (Gutierrez & Tasse, 2007, p.

49). Using program participants to identify the connection between pedagogy, practice, and outcomes has informed the evidence base for the program and can also do the same for other leadership development programs.

Practitioner Reflections and Recommendations

This pathway map enables us to clearly communicate the logic of our program to participants, partners, supporters, and others, which in turn enables us to obtain feedback on program design and implementation processes. It is important to remain aware that the program may not be solely responsible for causing all of the outcomes we identify—it may only be one contributing factor among others.

We have found that the easiest way to reach a consensus on the various activities and outcomes is to allow the group to brainstorm about it for twenty minutes using guiding questions and then share their individual thoughts for consideration followed by a clustering process to reach consensus. These steps ensure that, with good facilitation, everyone’s voice is heard and allows for consensus to be reached at a quicker pace than we thought was possible.

Another aspect to keep in mind, and to communicate to participants, is that the pathway of change is an important feature of the theory of change, but it is not the whole thing. The pathway map alone cannot tell the whole story of a program theory, but we use it as a skeleton on which successive waves of detail can be added to create a compelling theory of change. It is important to stress this so that they do not think that the map is all there is. There is much more detail to add in order to tell the full story, and without being reminded of that participants may find it difficult to focus on the bigger picture.

Based on our experience, we recommend using participatory pathway mapping with program participants as a way to effectively articulate the theory of change of leadership development programs. We propose that this process can provide an “evidence-base” for programs, provide participants with opportunities to reflect on, and articulate, their learning, provide program staff with information that can be developed for stakeholder reporting and program marketing, and, ultimately, contribute to program accountability in delivering intended outcomes.

In conclusion, especially in an era when organizations are increasingly being held accountable for results by funders, participants, and other constituents, it is critical that the plans

for an initiative are sound and that the results to be achieved are defined clearly beforehand so that everyone will know success when they see it. In our experience, participatory pathway mapping is a way to achieve these aims.

References

Andenoro, A. C., Allen, S. J., Haber-Curran, P., Jenkins, D. M., Sowcik, M., Dugan, J. P., & Osteen, L. (2013). National leadership education research agenda 2013-2018: Providing strategic direction for the field of leadership education. Retrieved from Association of Leadership Educators.

Allen, S. J. (2007). Adult learning theory and leadership development. Leadership Review, 7, 26–37.

Allen, S. J., & Hartman, N. S. (2008a). Leader development: An exploration of sources of learning. Organization Development Journal, 26(2), 75–87.

Allen, S. J., & Hartman, N. S. (2008b). Leadership development: An exploration of sources of learning. S.A.M. Advanced Management Journal, 73(1), 10–19.

Anderson, A. A. (2006). The Community Builder’s approach to Theory of Change: A practical guide to theory development. Aspen, CO: Aspen Institute.

Avolio, B. J., Avey, J. B., & Quisenberry, D. (2010). Estimating return on leadership development investment. The Leadership Quarterly, 21(4), 633–644.

Bassi, L.J., Gallagher, A. L., & Schroer, E. (1996). The ASTD training data book: training statistics, structure & operations, technology use, supplier issues, performance improvement. Washington, D.C.: American Society for Training & Development.

Brown, L. M., & Posner, B. Z. (2001). Exploring the relationship between learning and leadership. Leadership & Organization Development Journal, 22(6), 274–280.

Collins, D. B., & Holton, E. F. (2004). The effectiveness of managerial leadership development programs: A meta-analysis of studies from 1982 to 2001. Human Resource Development Quarterly, 15(2), 217–248.

Day, D. V, & Halpin, S.M. (2004). Growing leaders for tomorrow: An introduction. In D. V. Day, S. J Zaccaro, & S. M. Halpin (Eds.), Leadership development for transforming organizations: Growing leaders for tomorrow (pp. 3-22). Mahwah, NJ: Lawrence Erlbaum.

Day, D. V., Harrison, M. M., & Halpin, S. M. (2009). An integrative approach to leader development: Connecting adult development, identity, and experience. New York, NY: Psychology Press.

Dinh, J. E., Lord, R. G., Gardner, W. L., Meuser, J. D., Liden, R. C., & Hu, J. (2014). Leadership theory and research in the new millennium: Current theoretical trends and changing perspectives. The Leadership Quarterly, 25(1), 36–62.

Gardner, W. L., Lowe, K. B., Moss, T. W., Mahoney, K. T., & Cogliser, C. C. (2010). Scholarly leadership of the study of leadership: A review of The Leadership Quarterly’s second decade, 2000–2009. The Leadership Quarterly, 21(6), 922–958.

Gutierrez, M., & Tasse, T. (2007). Leading with theory: Using a theory of change approach for leadership development evaluation. In K. M. Hannum, J. W. Martineau, & C. Reinelt (Eds.), The handbook of leadership development evaluation (pp. 48-70). San Francisco, CA: John Wiley & Sons.

Hannum, K. M., Martineau, J. W., & Reinelt, C. (2007). The handbook of leadership development evaluation. San Francisco, CA: John Wiley & Sons.

Helstowski, L. W. (2001). The legacy of the ag. leadership development program: Rich heritage cultivates future opportunities (Booklet No. 534). Battle Creek, MI: W. K. Kellogg Foundation. Retrieved from www.wkkf.org/Pubs/Pub3626.pdf

Howell, R. E., Weir, I. L., & Cook, A. K. (1982). Development of rural leadership: Problems, procedures and insights. Battle Creek, MI: W.K. Kellogg Foundation.

Kaufman, E. K., & Carter, H. S. (2005). Agricultural leadership development: From networking to application. Journal of Leadership Education, 4(2) 66–73.

Kaufman, E. K., Rateau, R. J., Carter, H. S., & Strickland, L. R. (2012). What’s context got to do with it? An exploration of leadership development programs for the agricultural community. Journal of Leadership Education, 11(1), 121–139.

Kaufman, E. K., Rateau, R. J., Ellis, K. C., & Kasperbauer, H. J. (2010). Leadership program planning: Assessing the needs and interests of the agricultural community. Journal of eadership Education, 9(1), 122–143.

Loew, L., & Wentworth, D. (2013). Leadership: The state of development programs 2013.

Delray Beach, FL: Brandon Hall Group.

Lord, R. G., & Hall, R. J. (2005). Identity, deep structure and the development of leadership skill. The Leadership Quarterly, 16(4), 591–615.

Mezirow, J. (2000). Learning as transformation: Critical perspectives on a theory in progress.

San Francisco, CA: Jossey-Bass.

Miller, H. L. (1976) (Ed.). The Kellogg Farmers Study Program: An experience in rural leadership development. Battle Creek, MI: W. K. Kellogg Foundation.

O’Leonard, K., & Krider, J. (2014). Leadership development factbook: Benchmarks and trends in U.S. leadership development. Oakland, CA: Bersin & Associates.

Rennekamp, R. A., & Arnold, M. E. (2009). What progress, program evaluation? Reflections on a quarter-century of Extension evaluation practice. Journal of Extension [On-Line], 47(3), Article 3COM1. Available at http://www.joe.org/joe/2009june/comm1.php

Russon, C., & Reinelt, C. (2004). The results of an evaluation scan of 55 leadership development programs. Journal of Leadership & Organizational Studies, 10(3), 104-107.

Serrat, O. (2013). Theories of change. Manila, PH: Asian Development Bank.

Silverman, B., Mai, C., Boulet, S., & O’Leary, L. (2009). Logic models for planning and evaluation. Atlanta, GA: Centers for Disease Control.

Stufflebeam, D. L., & Shinkfield, A. J. (2007). Evaluation theory, models, and applications. San Francisco, CA: Jossey-Bass.

Urban, J. B., & Trochim, W. (2009). The role of evaluation in research—practice integration working toward the “‘golden spike.’” American Journal of Evaluation, 30(4), 538–553.

Weiss, C. H. (1997). How can theory-based evaluation make greater headway? Evaluation Review, 21(4), 501-524.

Author Biographies

Dr. Bradley Burbaugh (brad784@ufl.edu) is county-based faculty member and director of the University of Florida IFAS Extension in Clay County, Florida. His research focuses on adult leadership development, social capital, networking ability, and evaluation processes.

Dr. Megan Seibel (mseibel@vt.edu) is Director of the Virginia Agriculture Leaders Obtaining Results (VALOR) Program and an Extension Specialist in the Agricultural, Leadership, and Community Education department at Virginia Tech. She is also Associate Director of the Center for Cooperative Problem Solving. Megan received her PhD in Agricultural and Extension Education from Virginia Tech in 2012.

Dr. Thomas Archibald (tgarch@vt.edu) is an Assistant Professor and Extension Specialist in the Agricultural, Leadership, and Community Education department at Virginia Tech and is also Chief of Party/Director of the USAID/Education and Research in Agriculture project in Senegal. He received his PhD in Adult and Extension Education from Cornell in 2013.