That Program Really Helped Me: Using Focus Group Research to Measure the Outcomes of Two
Statewide Leadership Programs
Alice M. Black, Ph.D. Program Director Ohio LEAD Program
The Ohio State University Extension 2120 Fyffe Road
Columbus, Ohio 43210
Dianne P. Metzler Research Specialist
Institute for Economic Advancement College of Business
University of Arkansas at Little Rock 2801 South University Avenue
Little Rock, AR 72204
Joseph Waldrum, Ph.D.
Director of Organizational Staff & Leadership Development University of Arkansas Cooperative Extension Service
2301 South University Ave. Box 391, Room 103 Little Rock Arkansas 72203
This qualitative study attempts to document outcomes of two statewide agricultural and rural leadership programs by determining the affects of the program on participants after they graduate. The study explored three levels of outcomes: individual, organizational and community using the EvaluLEAD framework developed by Grove, Kibel and Haas (2005). Participants in the study were graduates of the Arkansas and Ohio statewide agricultural and rural leadership programs from 1984-2005. The research method is the first attempt to use focus groups to explore the outcome areas using the EvaluLEAD model. The outcomes reported by participants are documented as well as the methodology
used. The researchers found that program related outcomes exist in various degrees on the individual, organizational and societal/community level. Future research recommendations are included for leadership program evaluation.
Since the early 80’s Ohio and Arkansas have each conducted a statewide agricultural and rural leadership development program. Arkansas’ program was initially funded by a W.K. Kellogg Foundation grant. Ohio’s program did not receive Kellogg funding but was funded by The Ohio State University College of Agriculture and other agricultural investments. Today, both programs are supported from a combination of university funds and corporate and alumni resources.
Both programs function by identifying and recruiting 25 to 35 individuals in the state from a variety of occupations. Ohio selects from applicants who are involved in farming, agricultural or environmental involvement, and have exhibited leadership in their communities. Arkansas recruits from a similar pool but also accepts applicants in non-agricultural occupations from rural and urban communities.
The program curriculum for both is focused on instruction in leadership skills and in the current issues facing the state and nation. The programs are delivered through 12 to 14 institutes, seminars, and/or study tours that are experiential and participant-based and are conducted in various locations at the state, national, and international level. Today, the institutes in Ohio and Arkansas focus on topics that include: leadership skills, media training, state and national political processes, agricultural issues, environmental issues, renewable resource education, food safety, food security and a range of social issues. The programs’ goals are to produce pro-active individuals who will expand their leadership roles in agriculture, their communities, and their organizations to make a positive difference for society.
The programs strive to inform participants about current social, political and agricultural issues, as well as providing training in interpersonal and group communication skills. Since the programs began they have been touted as being very successful in achieving their goals. But how can we know they have produced better leaders for the state? A more complex question is how can we effectively measure and document the outcomes of statewide agricultural and rural leadership programs?
Overview of Program Evaluation
A new method of evaluating leadership program outcomes called EvaluLEAD was developed by Grove, Kibel and Haas (2005) in conjunction with the Public Health Institute and funding support from the W.K. Kellogg Foundation. The EvaluLEAD framework recommends that leadership program measurement occur on three levels: individual, organizational and societal/community. The individual level, as the first level of research, documents the individual outcomes that occur from the participant’s leadership program experience. Grove et al. calls this domain the area where most of the direct benefits of leadership development occur and where the most program-associated results might be expected. The second level of research regards organizational outcomes where results occur within the organizations where the program participants work or in outside organizations where the participants have contact (Grove et al.). Lastly, the societal/community level of research is the level of outcome detected in the community where the program participants have influence either individually or directly or indirectly through the organizations with which they work, or are affiliated. Grove et al. calls the mission or “reason for being” for most leadership development programs is to influence the community domain.
Grove et al. (2005) identify leadership development outcomes as being changes in behavior, relationship, activities, or action of the people(s), groups and organizations with which a program works. Some changes attributed to these programs might be easily observable in participants, while others might not be immediately detected such as those of personal attitudes, values and assumptions (Grove et al.). Therefore, evaluation must be based on relevant levels of program affect and impact.
Program evaluation according to Stufflebeam (2001) is “a study designed and conducted to assist some audience to assess an object’s merit and worth” (p. 11). Stufflebeam argues for the use of the term “evaluation approaches” rather than the term “evaluation models” since “most of the presented approaches are idealized or ‘model’ views for conducting program evaluations according to their author’s beliefs and experiences” (p. 9). Stufflebeam prefers to use the term “approach” to “model” because practices in evaluation can be on a broad scale and the term model is too demanding to cover the idea of program evaluation. Accountability evaluation helps to develop a dynamic baseline of information to demonstrate results (Stufflebeam).
The Use of Qualitative Methods
Qualitative data provides detailed description and allows for more in-depth inquiry (Patton, 1990). Patton indicates that qualitative data from the same study can show the real meaning of the program for participants. Patton points out that a dynamic evaluation is not tied to a single treatment or predetermined goals or outcomes but focuses on the actual operations and impacts of a process, program or intervention over a period of time. Evaluators focus on capturing process, documenting variations and exploring individual differences in experiences and outcomes (Patton). The richness of the data ensures that the resulting theory is able to provide a holistic understanding of the leadership process (Kan & Parry, 2004). To this end, leadership researchers must expand beyond quantitative analysis and shift to qualitative (DeRuyver, 2001).
A point must be made regarding the retrospective method of evaluation used in this research. Martineau and Hannum (2004) suggest that some researchers doubt the merits of the retrospective assessment because it could create an increase in the ratings from the “before” to the “now.” These researchers argue for the validity of retrospective surveys by pointing out that ratings of change are highly correlated with objective measures of change such as performance appraisals (Martineau & Hannum, 2004; Rockwell & Kohn, 1989).
Little scientific research has been conducted to determine the affect leadership programs have on the participants, society and organizations. A literature review and correspondence with statewide leadership directors finds that seven statewide agricultural leadership development programs have conducted some research regarding their programs (Wall & Kelsey, 2004; Vantreese & Jones, 1996; Whent & Leising, 1991; Horner, 1984; Carter, 1999; Mirani 1999; Dhanakumar et al., 1996). However, this EvaluLEAD study is the first research to look at leadership program outcomes on the three levels proposed by Grove et al. (2005).
The evaluation research used Arkansas and Ohio’s statewide agricultural leadership programs. The alumni of both programs had not previously been evaluated regarding the program’s outcomes and their experiences. The data collection method consisted of six focus groups. Three focus groups of program alumni were each conducted in Ohio and in Arkansas in two hour sessions. The purposes of these sessions were to gather input from alumni participants on their perceptions and opinions regarding the program outputs as developed by program evaluators prior to the sessions. In Arkansas, The University of Arkansas at Little
Rock’s (UALR) Institute for Economic Advancement (IEA) facilitated the series of focus groups for the University of Arkansas, Division of Agriculture Cooperative Extension Service LeadAR Program. In Ohio, the focus groups were facilitated by the researcher for The Ohio State University Extension, Ohio LEAD Program.
Each focus group was intended to consist of one member from each alumni class for a total of nine members in Ohio per focus group and ten in Arkansas per focus group. Phone calls were made to randomly selected focus group members by the researchers. If someone could not attend, the first person on the randomly selected alternate list of three names was contacted. If that person could not attend, the next person on the alternate list was contacted and so forth. The researcher stopped calling when one person from each alumni class agreed to attend one of the three focus groups, however, on the interview date not all arrived for their session. In Ohio, seven alumni attended for focus group one.
Eight alumni attended for focus group two and seven alumni attended for focus group three. In Arkansas. ten attended focus group one, eleven in focus group two and nine in focus group three.
The final result was that 22 randomly selected individuals from nine different alumni classes attended focus groups in Ohio. Thirty randomly selected individuals from 10 different alumni classes attended focus groups in Arkansas. The interview questions used in the focus groups were developed based upon the EvaluLEAD framework. Experts in evaluation methods assisted in question development. The questions were formulated according to research by Patton (1990) and Glesne (1999) to examine a variety of facets regarding the individual, organizational and societal level outcomes of the alumni focus group participants. The final questions were evaluated by experts in the field of research. The questions remained the same for each state’s focus group sessions. According to Patton (1990), standardized open-ended interviews are used to minimize interview effects by asking the same questions to each respondent.
During the focus group sessions, a moderator facilitated each group. The participants were made aware that they were being taped and the transcripts were to be typed verbatim. Consent forms were obtained from each participant. The moderator began the scripted questioning and each focus group lasted approximately two hours. Names were not identified in the transcripts after an initial introduction of the participants to each other. Oral remarks on the transcription were noted as being either male or female. Field notes were compiled at the end of each interview. Audio tapes were transcribed immediately after the focus groups and content and analytical coding analysis was conducted.
In Ohio, N6 – Version 6 of the NUD*IST QSR software was used to identify patterns and important themes in the data. In Arkansas, decision support software was used to capture the data. For this study, data was sorted based upon the EvaluLEAD framework (Grove et al, 2005) and categorized according to individual outcomes, organizational outcomes and societal/community outcomes. As suggested by Martineau and Hannum (2004), a data-collection reviewer was used to examine the process and analysis of the focus group questions and data.
The results of the focus group studies are unique and generated a great deal of data describing statewide leadership program outcomes. The researchers found that outcomes exist in various degrees on the individual, organizational and societal/community level as self-reported by the alumni in the focus groups. This is consistent with the framework presented by the EvaluLEAD model of Grove, Kibel and Haas (2005). The following information presents the combined findings of the three focus groups for each state. Finally, conclusions made by the researchers were audited by faculty peer debriefers to determine if themes and patterns were identified correctly. The focus group outcomes reported in this study’s results will focus on those related to the individual, organizational and community outcomes.
Individual Level Focus Group Outcomes
Individual outcomes were most prevalent among the focus group participants and were described by all of the participants in the study. The strength of the outcomes varied according to the individual. In Ohio, the highest rated individual outcome from leadership program participation involved developing new perspectives and new ideas. Many of the participants identified themselves as having been changed and increasing in self-confidence and self esteem as a result of the program. Others mention that they cut back in their time commitment to volunteer activities after the program. Participants noted that their leadership program involvement taught them to better manage their time.
For Arkansas, personal and professional growth is at the crux of the changes noted by session participants. Self-confidence and a broadening of perspective pertaining to others and their unique needs were mentioned by many participants as a major change they saw in themselves. A willingness to look beyond self and see the state and the world in a different light was an overarching theme reported by participants.
Participants in both programs indicate that an increase in self-confidence was a commonly held program outcome. Further dialogue revealed that this self- confidence was in part related to the practice that comes from making multiple presentations and part from the knowledge gained from seminar presenters and classmates. Additional changes that were noted included becoming more aware and sensitive to others needs and issues and a mitigating of the need to have total control. Many mentioned learning to appreciate the strengths and talents that, not only other classmates were able to bring to the table, but also learning how to recognize and utilize those strengths. An increased awareness of not only the home state of the participant but additionally the nation and the world situation was viewed as a major change of perception by the participants at this level.
Organizational Focus Group Outcomes
The first area identified by focus group participants in both states on the organizational level was the tremendous opportunity to network with other businesses, receive career coaching and maintain contacts with one another. Participants reported that the increased networking, benefited them in creating greater self-confidence in the business arena because of the support from other participants.
Other organizational level outcomes were identified by the focus group participants as the ability to effect transformations in their business or organization. As a result of the leadership program, participants report that they were able to bring new perspectives, improve business skills and bring new ideas to their business Participants noted that the benefits seen by the community or job were similar in that they are now able to bring more skills, knowledge, abilities, and potential to the table. There was a strongly held school of thought that their perception of their own leadership potential became equal to other state leaders thus giving their communities or organizations an advantage in the political and economic arena.
Societal/Community Level Outcomes
The most prevalent item mentioned by focus group participants in Ohio regarding societal/community level outcomes as a result of the leadership program was the need to find and then assume leadership roles within their communities. Some participants mentioned the need to decrease some of their community activities to become more effective and not spread themselves so thin. Activities mentioned included becoming active on community volunteer boards and in agricultural organizations. Secondly, participants felt that they increased their cultural
awareness due to their leadership program experience. On another level, some participants did not feel that their program experience led to increased involvement on the community level.
In Arkansas, session participants indicated that they learned they have considerably more ability to facilitate change in others and within their communities than they originally thought. Participants indicated that by-and-large they are active in their communities to a degree that was unimaginable prior to LeadAR. Activities included becoming officers in community organizations, writing legislation and grants, becoming a more active community volunteer, and recruiting others into the LeadAR Program to perpetuate the benefits to the state. This group of participants indicated that they now wield a much wider circle of influence that they did prior to completing the LeadAR Program. There was commentary about their increased ability to bring diverse groups together to develop consensus building and make real progress in their communities.
Participants in the study also expressed an appreciation for the diversity of others (both their classmates and those that they came into contact with over the program period). Self awareness and the ability to recognize the gifts others bring to a situation were frequently mentioned learning experiences. The international tour was also identified as a mind-broadening experience for participants. Many had not been out of the United States and were surprised at the level of disparity between our country and those that were visited. These experiences were noted to be enriching , and causing participants to have a more thorough understanding of the world situation. The confidence participant’s gain has created an environment that encourages a self-driven highly motivated approach when addressing the needs of the community. Communities are seen to have benefited by the proactive approach that participants developed. Session participants noted that the benefits noted by their community or employer were similar in that the LeadAR participants are now able to bring more skills, knowledge, abilities, and potential to the table. There was a strongly held school of thought that their leadership potential is on par with other state leaders (also noted in the organizational arena) thus giving their communities or organizations an advantage in the political and economic arena.
Negative Consequences from Program Participation
Participants were encouraged to share any negative consequences they experienced as a result of the leadership program experienced. Time away from home and family were two of the most frequently noted downsides that were experienced. In Ohio, the focus group participants mentioned divorces and
problems that occurred amongst spouses due to the time away required for their leadership program experience. In Arkansas, negative experiences were viewed as a fairly minor consequence when compared to the benefits that were gained through the leadership program.
Implications and Conclusions
The focus group members clearly articulated the outcomes that they experienced on the individual, organizational and community level. The participants report varying degrees of change, personal and professional growth. The statewide leadership education programs conducted in Ohio and Arkansas were found to generate positive outcomes amongst the participants when evaluated through the EvaluLEAD framework as proposed by Grove, Kibel and Haas (2005). This study is the first to apply the framework through focus group research.
Participants from the six alumni groups described these leadership programs as being the catalyst for expanding their boundaries. This was noted on a personal level, expressed as a feeling of being able to accomplish tasks and goals that had once appeared to be impossible, as well as from the perspective of looking beyond oneself and recognizing the larger perspective of the community, the state, the nation, and the world at large. Furthermore, participants feel that they have become better individuals and leaders because of their leadership program experience.
The leadership programs enabled participants to experience an improved quality and quantity of networking opportunities which aided them both personally and in their businesses because the networking included both a professional and personal (friendship) component. Furthermore, the educational aspect of the leadership experience was highly valued by participants. The importance of diversity, both from that found in the class and from the international trip experience, was touted as a major strength of both programs.
Since leadership program outcomes are unique to each participant further discussion must occur to evaluate these findings in comparison to the expected program outcomes of the program administrators. The insight gained from this program evaluation will be valuable to stakeholders, researchers and funders of leadership programs.
Finally, the focus group data has provided an effective starting place to develop instrumentation to measure leadership program outcomes. The methodology of focus groups to determine program outcomes is cost effective and can easily be
used to qualitatively measure program results. By randomizing the participants, researchers can limit the “glamorization” of the program. Researchers must be sure to ask the participants specific outcomes to assist in triangulating the results (Wall & Kelsey, 2004). During the focus groups, it is important to encourage those participants who are on the quieter side or “laid back” with their opinions to speak up and actively provide their input.
Overall, focus group participants conclude that their leadership program involvement truly helped them. Furthermore, this study takes an important first step in identifying program outcomes using the EvaluLEAD framework as a lens to view how leadership programs continue to affect the participant after the program ends. This research is ground breaking in seeking to identify three levels of outcomes recognizing that leadership development occurs at different levels and at different periods of time with participants. It will be interesting to compare these results with the results of other leadership programs as the EvaluLEAD framework is applied to the outcomes of these programs.
Recommendations for further research include encouraging other statewide leadership programs to pursue program evaluations using the EvaluLEAD framework. This will allow for program comparisons and sharing of evaluation methods. Stakeholder evaluation is also needed for outcomes to be compared to stakeholder and program leader expectations. Finally, a quantitative survey instrument has been developed and is being tested as a result of the focus group studies.
Carter, H. (1999). Evaluation of the Florida leadership program for agriculture and natural resources. Unpublished master’s thesis, University of Florida.
Dhanakumar V.G., Rossing B., & Campbell, G.R. (June 1996). An evaluation of the Wisconsin rural leader’s perspective program [Electronic Version]. Journal of Extension. 34 (1).
DeRuyver, D. (2001). Leadership: Emerging directions. Theory and Literature, Leadership Learning Community. Retrieved March 1, 2005 from http://www.leadershiplearning.org/pools/theory/emergingdirections
Grove, J. T., Kibel, B. M. & Haas, T. (2005). EVALULEAD. A guide for shaping and evaluating leadership development programs. Oakland CA: Sustainable Leadership Initiative. Public Health Institute
Horner, J. T. (November 1984). Developing effective agricultural leaders [Electronic Version]. Journal of Extension, 22 (6).
Kan, M. M. & Parry, K. W. (2004). Identifying paradox: A grounded theory of leadership in overcoming resistance to change. The Leadership Quarterly, 15 (2004). P. 467-491.
Martineau, J. & Hannum, K. (2004). Evaluating the impact of leadership development, A professional guide. Greensboro, NC: Center for Creative Leadership.
Mirani, Z. (1999). An assessment of the impact of the Minnesota/Iowa leadership empowerment for agricultural development (M/I LEAD) Program. Unpublished Doctoral Thesis. University of Minnesota Graduate School.
Nassar-McMillan, S. C. & Borders, L. D. (2002). Use of focus groups in survey item development. The Qualitative Report, 7(1). Retrieved January 10, 2006, from http://www.nova.edu/sss/QR/QR7-1/nassar.html
Patton, M. Q. (1990). Qualitative evaluation and research methods. (2nd Edition.) Thousand Oaks, CA: Sage Publications.
Rockwell, S. K. & Kohn, H. (1989). Post-Then pre-Evaluation. [Electronic Version]. Journal of Extension Volume 27, 2.
Stufflebeam, D. L. (2001). Evaluation models. New directions for evaluation, 89. San Francisco:Jossey-Bass
Vantreese, V. L. & Jones, L. D. (1996). An evaluation of the Philip Morris agricultural leadership development program. University of Kentucky: 1985-95 (Unpublished Study).
Wall, L. J. & Kelsey, K. D. (2004). When findings when findings collide: Examining survey vs. interview data in leadership education research. [Electronic Version]. Journal of Southern Agricultural Education Research, 54(1).
Alice Black is the Program Director for an adult leadership development program at the Ohio State University in Columbus, Ohio for those in the food, agricultural and environmental sectors. Alice joined the program after 10 years as Executive Director of the Ohio Poultry Association. She is an OSU graduate holding a Bachelor of Science in Animal Science and Agricultural Communications, a Masters in Journalism, and a PhD in leadership studies with a minor in rural sociology. Her research experience is in the area of leadership program evaluation and development.
Dianne Metzler is a Research Specialist and Qualitative Research Facilitator for the Survey Research Group at the University of Arkansas at Little Rock’s (UALR) Institute for Economic Advancement. She received a BA in Industrial/Organizational Psychology and the MA in Applied Psychology from the University of Arkansas at Little Rock. She has been active in the area of applied research for over 15 years and has contributed to several journal articles and conference papers. Her qualitative research experience includes focus groups and heading up the facilitation of a computerized decision support center at IEA.
Dr. Joe Waldrum was appointed Professor – Leadership Development in October, 2006. His primary responsibility is to direct LeadAR, a two year statewide adult leadership development program of the University of Arkansas Cooperative Extension Service. Dr. Waldrum received his Bachelor of Arts in biology from Hendrix College, and a Master’s and Ph.D. in plant physiology from the University of Nebraska. He began his Extension career in 1976 as an area soybean specialist in Stuttgart. In 1980, he was named Extension Pesticide Specialist and transferred to the State Extension Office in Little Rock. In 1989, he was appointed as the State Leader of Agriculture and Natural Resources where he administered all Extension programs in these areas on a statewide basis. He was named to direct the LeadAR Program in January, 1993. In January, 2001, in addition to his LeadAR responsibilities, he was appointed to direct the newly organized Organizational, Staff and Leadership Development section for the Extension Service.