Leaders from academia and organizations in the private sector, commonly referred to as an industry, continuously seek solutions to prepare the next generation entering the workforce with essential skills. For decades, the industry has assumed that universities are primarily responsible for developing future employees with the necessary skills, presumably because higher education offers students a place for learning and skill-building (Seemiller, 2018). There is a general concern that graduates lack the interpersonal or soft skills necessary to lead (Greenhaw & D’Abreau Denny, 2020). According to Gardner (2018), students need to develop the required educational qualifications and soft skills to be successful in the job market. Soft skills include competencies, like communicating effectively and thinking critically (Chikeleze et al., 2018). The author suggests these are examples of essential skills to enter the modern workplace. The introduction of essential skills as a replacement for soft skills terminology is a new concept. In conjunction with industry data, researcher Seemiller (2018), identified verbal communication, evaluation, writing, analysis, and decision making as the top five essential skills in the industry. Demands and expectations of a new generation entering the workforce continue to shift and reestablish new workplace norms (Davidson, 2004). There is a growing need in our society to not only attract and retain unique expertise in a competitive job market; our future talent must arrive in the workforce ready to lead with essential skills.
University and industry have a distinct opportunity to bridge their knowledge and experiences together to create talented leaders for the common good (Post et al., 2016). As a shared endeavor and responsibility for creating innovative practices, university, and industry partners are practicing a form of engaged scholarship, thus creating a space for the relationship to explore a mutual concern and develop solutions through collaboration (Schockley-Zalabak et al., 2017). The purpose of this innovative practice paper is to highlight an engaged scholarship approach designed to prepare students to lead now and into the future. Specifically, this paper will:
- Describe how the program was co-created utilizing a method called the 70-20-10 framework, and
- Illustrate how Kirkpatrick’s four-level model is being used to evaluate the reaction, learning, and behavior changes.
Drawing from theory and best practices of the industry, these approaches to design, delivery, and evaluation can help to advance evidence-based practice for leadership education, which is a crucial priority for the field (ILEC, 2016).
Review of Related Scholarship
In the late 1980s, McCall et al. (1988) introduced the 70-20-10 concept. Lessons learned from industry leaders suggest that learning occurs through a combination of experiential, social, and formal learning activities (Johnson et al., 2018; McCall et al., 1988). Researchers recommend leadership development programs incorporate the 70-20-10 principle; 70% of learning as experiential learning such as performing challenging assignments, 20% social learning conducted by a coaching program, and 10% formal learning through training (Day & Liu, 2019; Johnson et al., 2018; Lombardo & Eichinger, 2017; McCall et al., 1988).
First, the 70% rule is engaging in a stretch assignment that creates the opportunity for the student to learn from experience in a supportive and challenging environment. For example, participating in a case study is considered a stretch activity as it requires learners to confront difficult, complex problems that have no single, obvious solution, allowing students to wrestle with challenges and help prepare them for workplace dilemmas. Experiences contribute to the learning and advancement of an essential skill (Seemiller, 2018). One issue with stretch assignments is the readiness of the individual and whether they can perform the work at the expected level (Day & Harrison, 2006); therefore, the practices should reflect the individual’s ability and skillset. For example, the case study should seek a balance between experience and knowledge of an undergraduate student and issues occurring in a professional setting.
Second, the dedication of 20% of development time is to social learning and engaging talent in a coaching program. Coaching is an opportunity for students to receive confidential feedback from an industry leader. The student who participates in the coaching program may be assigned a coach to help drive change and deepen learning (Priest et al., 2018) through stretch assignments, feedback, or classroom learning. A safe environment should exist for all students before providing and receiving feedback (Roupnel et al., 2019). Students should have a level of developmental readiness for coaching because they must act on the goals created.
The third component of the framework suggests 10% of learning conducted through training. Training programs are focused on sharing solutions to identified problems (Day, 2010) and become advantageous if there is an immediate need to address an organizational issue such as learning policies and procedures. However, traditional training approaches may be inadequate when solving messy, complex problems. For example, when training emphasizes the development of practices for adaptive change, a deliberative approach to significant learning must emphasize maximizing multiple leadership development methods. Programs must enhance relationships amongst students to build leadership capacity (Day, 2010) successfully. Training programs can be beneficial; however, the effectiveness of the program is vital to measure. Lombardo and Eichinger (2017) recommended that training only allows for 10% of the responsibility in developing leaders. In the following section, the 70-20-10 model is in alignment with Kirkpatrick’s four-level model.
Kirkpatrick Four-Level Model
The 70-20-10 framework offers a pathway through which to design the program curriculum. Evaluation of that curriculum is one way in which partners in higher education and industry can gain clarity on the impact of the training in preparing a workforce. Kirkpatrick’s model offers a way to do that evaluation. According to Kirkpatrick and Kirkpatrick (2016), there are three reasons to conduct an evaluation: (1) enhance the program, (2) support an environment of learning, behavior, and results, and (3) assess the return on expectations of training programs. Formative evaluation is conducted after the execution of activities to make continuous improvements to program content, delivery, and participant experience (Mertens & Wilson, 2019; Patton, 2014). In the late 1950s, Dr. Donald Kirkpatrick introduced the four-levels model as a goal-based evaluation model to assess the effectiveness of learning and development programs (Kirkpatrick & Kirkpatrick, 2016; Fullard, 2007). It continues as a commonly used tool today (Boulmetis & Dutwin, 2005; McLean & Moss, 2003). There are four stages: reaction, learning, behavior, and results, that provide a process to evaluate the effectiveness of learning.
This paper emphasizes the reaction, learning, and behavior stages of a student leadership development program, utilizing a mid-year evaluation. First, the reaction stage measures the satisfaction of students after completion of content (Fullard, 2007), and captured this information with a series of questions about their experience. Second, the learning stage is an opportunity to assess the knowledge gained after the completion of program content (Steensma & Groeneveld, 2010) and evaluated responses with the learning outcomes and self-reflection of acquired knowledge. Third, the behavior stage focused on the influenced changes in behaviors of students (Kennedy et al., 2013) and assessed self-reflections. The final step, results, is focused on the impact of the program on the desired outcome (Kennedy et al., 2013; Mertens & Wilson, 2019) and measures the conversion of talent to join the industry partner. The involvement of university and industry partners in the evaluation of the program provides an opportunity for continuous and relevant improvement, which benefits the Fellows who could be employed by these industry partners.
This evaluation is part of a post-positivist paradigm, which includes surveys to assess the satisfaction of experiences, acquired knowledge, and changes in behaviors after completion of each activity. A post-positivist approach focuses on the causal relationships that explain and predict changes in behavior (Ospina & Uhl-Bien, 2012). Each survey will incorporate questions that address multiple stages (reaction, learning, etc.). Blending these various forms will shed light on students’ experiences and growth, and is essential to deepen understanding of how leadership development interventions impact workforce preparedness.
Description of the Practice
An engaged scholarship approach established with a mid-western land-grant university and a privately held global corporation focused on providing innovative solutions in agricultural services are forward-thinking practices. Participants in this study include nineteen students selected into the program (for this paper, called “Industry Fellowship”) through a non-random sampling technique leveraging convenience sampling. Students recruited at a university with approximately 22,000 students in the spring semester of their sophomore year. Requirements for eligibility included a 2.75-grade point average, major in degrees from College of Agriculture, College of Engineering, and College of Business, and that they are in good standing with the institution. To gain acceptance in the program, students applied and then interviewed with university and industry leaders. Finally, they received an opportunity to join the program. The students (called “fellows”) were selected to participate in the program during their junior year. The fellows received two $500 scholarships, one for each semester of participation. Program participation was contingent upon fellow’s willingness to engage in leadership activities with industry partners for an academic school year.
Fellow Leadership Development Outcomes
The following learning outcomes were identified by university and industry partners to not only guide fellows through the program but also to nourish their ability to lead in the future. The fellows will demonstrate:
- An understanding of how their aspirations and direction align with the work that the industry partner does in the world;
- An awareness of the skills and the self-understanding necessary to live and work in a diverse world;
- An ability to make connections of individual parts within a larger system;
- An ability to respond effectively to an unpredictable situation by acting decisively; and courageously while also having the ability to bounce back after a setback.
Overview of Program Activities
An Industry Fellowship (IF) was co-created with university and industry partners. The program consists of five case studies, three site visits, two service-learning experiences, coaching, and a leadership lecture series (see appendix A for full description). The leadership lecture series covered the following topics: Leadership Expectations, Accelerate your Interview Talent, Inclusion and Diversity, and Emotional Intelligence.
One of the first activities was a site visit dedicated to familiarizing fellows with the industry partners’ values and culture. Next, a welcome retreat with industry partners occurred to introduce fellows to the IF. During this event, fellows received a leadership development plan which specified program activities consistent with the 70-20-10 framework. The plan also included space for the fellows to practice self-reflection, document notes from experiences, and capture trends from the industry. At the retreat, the fellows participated in the first leadership lecture series: Leadership Expectations. The training provided a foundation for a common language used in the industry.
After each activity, fellows received a survey to evaluate their experience. The questions provided the fellows an opportunity to share feedback on their reaction, learning, and behaviors (see appendix B). Based on the survey data, the university and industry partners made improvements to the next activity in the sequence of the program.
Discussion of Outcomes/Results
Throughout the first six months of the program, Kirkpatrick’s four-level model evaluation has helped to gain a better and deepen understanding of the participant’s experiences and preparedness as a result of the program. In understanding the experiential learning through case studies, site visits, and service-learning, first, we looked at reaction to two site visits (n=15, n=9), and fellows indicated the site visit met or exceeded their expectations. The two case studies (n=17) showed they were somewhat to extremely satisfied. We also assessed learning, and approximately 94% of fellows (n=16) in the case study agreed that they acquired new knowledge and are committed to applying what they learned. Multiple fellows expressed a desire for more time to discuss the case in their teams, and one fellow stated, “I realized that I need more experience to learn how to solve problems.” Finally, we looked at behavior changes, and positive behavioral change was evident with the service-learning feedback. One hundred percent of fellows (n=11) somewhat to strongly agreed that they have a responsibility and a commitment to use the knowledge and skills gained to help address issues in society.
In exploring social learning through coaching, we looked at the reaction, and approximately 85% of fellows (n=13) indicated they were an active participant in the coaching program. Sixty-nine percent of fellows (n=13) reported they felt a mutual level of trust with their coach. Next, we assessed learning, and approximately 77% of fellows (n=13) indicated learning new things together is an integral part of the relationship. A comment expressed, “The most valuable aspect was the unique level of transparency through which I saw the workings of my coach.” Lastly, we looked at behavior changes, and approximately 77% of fellows (n=13) indicated they visited with their coach at least 30 minutes a month; however, about 54% of fellows (n=13) indicated they spend time working on how he/she can improve.
In identifying formal learning through the leadership lecture series, combined are all reactions to the training, and 86% of fellows (n=37) indicated that the experiences were somewhat to extremely satisfied. Fellows answered an open-ended question to explain what improvements they would make to the program, and one comment reflected, “I’d like to do more hands-on activities.” Then we assessed learning, and approximately 76% of fellows (n=37) indicated they acquired new knowledge. Fellows responded to an open-ended question to describe key learning moments and a pivotal learning moment identified how diversity could span across more than someone’s primary identity, but through several identities. Lastly, we evaluated behavior changes, and fellows explained how they changed behaviors as a result of the content learned. For example, one fellow mentioned, “I would continue to learn about emotional intelligence and self-awareness so that I can train myself to navigate my emotions and those of the people around me better.”
Reflections of the Practitioner
Leadership educators can embrace an engaged scholarship approach by adapting the previously described models and make improvements to the program along the way. In our own experience, we have found this approach teaches fellows to be nimble in their learning and that providing feedback results in a change. For example, the case study provided a space to work through a real-life issue with time constraints. Although the fellows wanted additional time, the industry partners were impressed with their level of engagement. In the future, Fellows will be provided the case study in advance, and they will work on solutions in teams to present to the industry partner. The experiences in the Industry Fellowship are relevant and build on essential skills to lead tomorrow. The collaboration with university and industry partners created an opportunity for engaged scholarship since we shared knowledge and learned from each other. For instance, we discussed the different approaches to creating leaders who value inclusion and diversity in the workplace. Therefore, we created training to prepare students to lead in a diverse environment.
In pursuing evidence-based practice, leadership educators can build upon the 70-20-10 framework and enhance Kirkpatrick’s four-level model. To build a sense of belonging and create an environment of further learning through experiences, create an opportunity for fellows to interact outside of the lessons on group projects relevant to the program. To enhance social learning, train coaches on an appreciative inquiry model. For example, “Tell me about a time you participated in a training where the facilitator created an environment of engagement. What made it a supportive learning environment?” This approach could enhance the relationship and build trust by teaching coaches how to ask questions that help fellows improve as a leader and teach fellows different ways to engage in conversations. To learn more about topics through planned activities and interactions, introduce online micro-learning sessions on issues, and then discuss the content in person or through message boards.
To enhance responses beyond satisfaction levels, educators can inquire fellow’s engagement by asking, “Was there anything about the experience that interfered with your learning, and if so, what?” To improve levels of learning and deepen the understanding of knowledge acquired by applying key concepts and practice skills, plan an opportunity for the fellows to introduce an artifact to reflect an environment of learning. To build trust and aid in the accountability of making changes in behaviors, first, provide an opportunity for individual self-reflection, then pair-share, and end with a group discussion. To develop the case for continued funding of the program, after the annual program, measure fellow conversation to full-time employment with an industry partner or pursuit of further education. Lastly, a focus group format with open-ended questions creates a summative evaluation to assess student learning outcomes.
As a potential model for building leadership capacity for the next generation, an engaged scholarship approach coupled with a program designed based on the 70-20-10 framework and Kirkpatrick’s four-level model evolved. University and industry partners assessed fellows, and they responded with feedback to help further enhance their learning. The practice of engaged scholarship with leaders from different sectors to develop and evaluate a program could lead to future practices for the betterment of society by preparing the next generation with essential skills to lead.
Bauermeister, M. C., Greer, J., Kalinovich, A. V., Marrone, J. A., Pahl, M. M., Rochholz, L. B., & Wilson, B. R. (2016). Preparing students for leadership through experiential learning. Journal of Leadership Education, 15(3), 31-41.
Boulmetis, J., & Dutwin, P. (2011). The ABCs of Evaluation: Timeless Techniques for Program and Project Managers. Indianapolis, IN: John Wiley and Sons, Incorporated.
Chikeleze, M., Johnson, I., & Gibson, T. (2018). Let’s Argue: Using Debate to Teach Critical Thinking and Communication Skills to Future Leaders. Journal of Leadership Educators, 17(2), 123-137.
Davidson, E. J. (2004). Evaluation Methodology Basics: The Nuts and Bolts of Sound Evaluation. Thousand Oaks, CA: Sage Publications, Inc.
Day, D. V. (2010). The Sage handbook of leadership. Thousand Oaks, CA: Sage Publications.
Day, D. V., & Liu, Z. (2019). What is wrong with leadership development and what might be done about it? In R. E. Riggio, What’s wrong with leadership? Improving leadership research and practice (pp. 226-240). New York: NY: Routledge.
Day, D., & Harrison, M. M. (2006). Encyclopedia of career development. Thousand Oaks, CA: Sage Publications.
Fullard, F. (2007). A model to evaluate the effectiveness of enterprise training programmes. International Entrepreneurship Management, 263-276.
Greenhaw, L. L., & D’Abreau Denny, M. (2020). When technical skills aren’t enough: Training for the softer side of community action leadership. Journal of Leadership Education, 19(1), 29-36.
ILEC, I.-a. L. (2016). Collaborative priorities and critical considerations for leadership education. https://www.naspa.org/images/uploads/events/ilec_2016.pdf.
Johnson, S. J., Blackman, D. A., & Buick, F. (2018). The 70:20:10 framework and the transfer of learning. Human Resource Development Quarterly, 29, 383-402.
Kennedy, P. E., Chyung, S. Y., Winiecki, D. J., & Brinkerhoff, R. O. (2013). Training professionals’ usage and understanding of Kirkpatrick’s level 3 and level 4 evaluations. International Journal of Training and Development, 18, 1-21.
Kirkpatrick, J. D., & Kirkpatrick, W. K. (2016). Kirkpatrick’s four levels of training evaluation. Danvers, MA: Association for Talent Development.
Lombardo, M. M., & Eichinger, R. W. (2017). The leadership machine: Architecture to develop leaders for any future. Minneapolis, MN: Lominger International: A Korn/Ferry Company.
McCall, M., Lombardo, M., & Morrison, A. (1988). The lessons of experience: How successful executives develop on the job. Lexington, MA: Lexington Books.
McLean, S., & Moss, G. (2003). They’re happy, but did they make a difference? Applying Kirkpatrick’s framework to the evaluation of a national leadership program. The Canadian Journal of Program Evaluation, 18(1), 1-23.
Mertens, D. M., & Wilson, A. T. (2019). Program evaluation theory and practice. New York: NY: The Guilford Press.
Ospina, S. M., & Uhl-Bien, M. (2012). Exploring the competing bases for legitimacy in contemporary leadership studies. In M. Uhl-Bien, & S. M. Ospina, Advancing relational leadership research: A dialogue among perspectives (pp. 1-40). Charlotte, NC: Information Age Publishing, INC.
Patton, M. Q. (2014). Evaluation Flash Cards: Embedding Evaluative Thinking. St. Paul, MN: Otto Bremer Trust.
Post, M. A., Ward, E., Longo, N. V., & Saltmarsh, J. (2016). Publicly engaged scholars. Sterling, VA: Stylus.
Priest, K. P., Kliewer, B. W., & Stephens, C. M. (2018). Kansas leadership studies summit: Cultivating collaborative capacity for the common good. Journal of Leadership Educator, 17(3), 195-207.
Roupnel, S., Rinfre, N., & Grenier, J. (2019). Leadership development: Three programs that maximize learning over time. Journal of Leadership Education, 18(2), 126-143.
Schockley-Zalabak, P., Barge, J. K., Lewis, L., & Simpson, J. L. (2017). Engaged Scholarship. The International Encyclopedia of Organizational Communication, 2, 802-821.
Seemiller, C. (2018). Enhancing leadership competencies for career readiness. New Directions for Student Leadership, 157, 39-53.
Steensma, H., & Groeneveld, K. (2010). Evaluating a training using the four levels model. Journal of Workplace Learning, 22(5), 319-331.
Thoemke, K. W. (2016). Career development: Plenty of opportunity, plenty of concern: The short term future of the environmental job market. Environmental Practice, 18(1), 75-77.
Woods, K. (2019). University Students’ Perceptions of Generational Issues in the. Journal of Leadership Education, 18(1), 102-115.