Introduction
Adults and peers play a critical role in developing individual leadership capacity as well as developing group and organizational leadership effectiveness (Rosch, 2017). As leadership educators consider which experiences yield the best possible outcomes for leader and leadership development, developmental interactions such as mentoring likely emerge as among the most likely influential experiences in a leader’s developmental journey (Campbell et al., 2012; Dugan & Komives, 2010; Komives et al., 2005, 2006; Renn, 2007; Turner, 2018). The difficulty, however, with developmental interactions like mentoring is outlining appropriate program outcomes and associated learning objectives given the individualized nature of mentoring relationships and demonstrating the leadership learning impact for both mentor and mentee (Hastings & Kane, 2018). Rarely do we take a quiz after meeting with our mentor or evaluate the performance improvement of our mentee; thus, collecting valuable assessment data for programmatic improvement in leadership mentoring poses a considerable challenge for leadership educators. The purpose of the current application manuscript is to address assessing and evaluating the impact of leadership mentoring programs on collegiate mentors using a 360-student leadership mentoring program at a four-year, public, Midwestern university as a case study. Specifically, this paper addresses the challenges of creating appropriate program outcomes and associated objectives given the individualized nature of mentoring relationships. Additionally, this paper highlights assessment and evaluation strategies to demonstrate impact of leadership mentoring on the collegiate mentor via a three-year program evaluation effort.
While leadership education assessment models exist (Roberts & Bailey, 2016), the specific assessment and evaluation focus on mentoring is largely absent in the literature (Bureau & Lawhead, 2018). Additionally, most assessment frameworks in mentoring are designed to indicate growth in the mentee. Bureau and Lawhead (2018), for example, created a framework for assessing student leadership development from mentoring using the Council for Advancement of Standards (CAS) Student Learning Domains (CAS, 2015) and the Leadership Identity Development (LID) model (Komives et al., 2005, 2006). Specifically, these authors identified CAS Domains likely impacted from mentoring and suggested potential outcome statements, associated assessment methods, and LID stage likely targeted. But what happens when the college student is the mentor instead of the mentee?
Seemiller (2014) developed The Student Leadership Competencies Guidebook as a framework to design and assess student experiences that are intended to develop essential leadership competencies in preparation for desired career fields. Drawn from sources such as the Relational Leadership Model (Komives et al., 2013), CAS (2015), and the Learning Reconsidered paper published by ACPA/NASPA (2004), Seemiller (2014) identified 60 competency areas within eight categories that articulate the knowledge, values, abilities, and behaviors necessary for students to be effective leaders in college, their career, and in society. The list of 60 competencies was then compared with accrediting agency learning outcomes. With one competency area specifically devoted to mentoring, Seemiller’s Competency model serves as a useful framework to consider mentorship outcomes when the college student serves as mentor.
Review of Related Scholarship
The purpose of mentoring is to develop the mentee’s capacity to acquire knowledge, skills, and self-confidence to become a better student, employee, or organizational leader (Burke, 1984; Fagan & Walter, 1982). While mentoring’s purpose centers on mentee development, the directional pattern of influence is more reciprocal than unidirectional (Crisp & Cruz, 2009; Eby et al., 2010; Hastings & Kane, 2018). Mentoring practice requires long-term commitments from both mentor and mentee, regular interactions, and both formal and informal investments in personal growth, career development, psychosocial development, and leadership empowerment (Campbell et al., 2012; Castro et al., 2004; Crisp & Cruz, 2009; Eby et al., 2010; Nora & Crisp, 2007).
Mentorship outcomes for the mentee across myriad contexts (youth, student-faculty, workplace) include enhanced psychological health, stronger positive attitudes, and achievement (Lockwood et al., 2010). Relative to college students being mentored, positive mentorship outcomes for the mentee include persistence, social and academic integration, and academic success (Crisp, 2010; Crisp & Cruz, 2009). Gallup-Purdue Index Report (2014) findings indicate that the odds of college graduate workplace engagement and well-being more than doubled if those graduates had a professor who cared about them, engaged their excitement toward learning, and encouraged them to pursue their dreams. Relative to youth mentoring, Blinn-Pike’s (2010) and DuBois et al.’s (2011) meta-analytic reviews identified several positive mentee outcomes across social, emotional, behavioral, and academic domains, such as attitude toward school and violence, academic performance, and improved parental relationships. Additionally, Peterson and Stewart’s (1996) research results indicated higher generativity (care for establishing and guiding the next generation; Erikson, 1950, 1963) among mentored youth.
While most mentoring research has focused on the mentee impact, some scholarship has been dedicated to mentor outcomes. Positive outcomes for the mentor include increased pride and satisfaction, sharpened leadership competencies, greater confidence, improved job performance, and higher generativity ([Author] et al., 2015; [Author], 2020, Bass, 1990; Day, 2000; Lockwood et al., 2010; Newby & Corner, 1997). Allen and Eby (2010) articulate a linkage between effective mentoring relationships and the biological need to create and sustain positive relationships, thus connecting mentoring to positive affective, cognitive, and behavioral outcomes. When outcomes related to the mentor are addressed, they have, however, largely focused on adult mentors as opposed to collegiate mentors.
Research in both K – 12 and higher education contexts have validated mentoring’s utility in leadership development, particularly towards the development of socially responsible leadership ([Author] et al., 2015; [Author] & [Author], 2019; Campbell et al., 2012; Collins-Shapiro, 2006; Dugan & Komives, 2007; Dugan & Komives, 2010; Komives & Collins-Shapiro, 2006; Komives et al., 2009). Mentoring for the purpose of leadership development involves long-term investment in personal development as well as leadership empowerment (Campbell et al., 2012). Different types of mentoring generate distinct leadership development experiences and outcomes. For example, faculty mentoring emerged as one of the top three predictors across all leadership values associated with the Social Change Model (SCM; Higher Education Research Institute, 1996) except for Citizenship and Collaboration from senior-level data associated with the Multi-Institutional Study of Leadership (MSL). Peer mentoring, however, emerged as a significant predictor for Citizenship, Collaboration, and Commitment, thus impacting the leadership values not accounted for through faculty mentoring (Dugan & Komives, 2010). Additionally, the importance of peer mentoring increases as student leadership identities develop (Komives et al., 2009). Parker et al. (2008) argue peer mentors are more likely to comprehend student ambiguity in complex situations and are best positioned to co-create shared learning to build a community of practice. Peer mentoring also serves to develop the mentor’s leadership identity through explicit and required focus on generativity, the fifth stage in the Leadership Identity Development model (Komives et al., 2005). [Author] et al.’s (2015) research on college student leaders who mentor confirmed this argument with college student mentors demonstrating significantly higher generativity levels than their non-mentoring peers. Further, [Author]’s (2020) follow-up longitudinal research on the same population indicated generative behavior growth over time among college student mentors. While the aforementioned studies provide a start, little program evaluation efforts and/or empirical research studies have been done to identify and test outcomes associated with college students who serve as leadership mentors. Despite this lack of research, college mentoring programs are designed with the belief that these types of programs develop leaders (Posner & Brodsky, 1992; Ryan, 1994; Seitz & Pepitone, 1996). By innovating leadership mentoring program practice around developing and implementing a program evaluation strategy, leadership educators can more soundly design and deliver leadership mentoring programs and more precisely measure and demonstrate impact.
Description of Practice
As previously mentioned, the purpose of this application manuscript is to address the nuanced considerations of assessing and evaluating the impact of leadership mentoring programs on collegiate mentors using a 360-student leadership mentoring program (LMP) at a four-year, public, Midwestern university as a case study. This subsection outlines the LMP that is the case study for this paper.
The LMP serving as the case study is comprised of 180 college student leaders who mentor 180 K – 12 student leaders in the local community. College student mentors for the program (called “leadership mentors”) are selected as second-semester freshman and are paired in one-to-one relationships, called “investment relationships”, with K – 12 students (“leadership mentees”) who have been identified by their schools as demonstrating significant leadership talent and potential. Collegiate leadership mentors are paired with mentees on the basis of common talents and strengths, common interests, or a combination of both. Each pair meets weekly for three years. The objective for the leadership mentor is to identify and develop the leadership talents within their leadership mentee. Based on the age or school of the leadership mentee, leadership mentors are organized in small groups, called “projects” with a high performing senior student assigned to each project as a staff advisor. These projects meet weekly to reflect upon the relationship growth with their leadership mentees. Staff advisors monitor the well-being of each mentoring relationship within their assigned project via shared mentor reflections each week as well as parental check-in emails each semester. All staff advisors also meet weekly with LMP professional staff to discuss mentoring successes and concerns with their respective projects. Mentoring successes are nominated for “Outstanding One on One of the Week” honors and are posted on the LMP’s social media. Mentoring concerns are addressed either by the staff advisor or the LMP professional staff on a case-by-case basis.
Leadership mentors take an interpersonal skills for leadership course during one semester of their LMP experience with course objectives focused on self-understanding, understanding others, and investing in others. Scholarly discourse in the course covers positive psychology principles such as empathy, active listening, investment relationships, strengths, and self-concept, among others. Utilizing service-learning pedagogy (Furco & Billig, 2001), collegiate mentors react to course concepts and reflect upon their application to youth mentoring in weekly journals. Furthermore, their leadership mentoring relationship serves as the active experience of the course which culminates in a final project that analyzes and evaluates the application of course concepts in their mentoring relationship. The end of each academic year is marked by the program’s Annual Recognition Day in which each mentoring pair presents a project that symbolizes the growth in their investment relationship over the past year.
While the LMP is grounded in the ideals of positive psychology (Seligman & Csikszentmihalyi, 2000) and was created based upon the early research of two positive psychology pioneers, the program has operated for over 70 years from a mission statement and a set of articulated basic assumptions with program outcomes and associated objectives grossly absent. Additionally, while empirical research has been conducted on the program ([Author] et al., 2015; [Author] & [Author], 2019; [Author], 2020), formal annual assessment and evaluation have likewise not been conducted.
Discussion of Outcomes/Results
The first two subsections outline the process followed for creating program outcomes, associated objectives, and assessment and evaluation plans. The third subsection outlines the three-year program evaluation results.
Developing Program Outcomes, Objectives, and Identifying Associated Competencies. The development of program outcomes and associated objectives followed the International Leadership Association’s (2009) Guiding Questions document relative to outcomes and assessment. Specific to the development of program outcomes, Ritch and colleagues (2009) outline that program outcomes “…will spell out what graduates of a program will be able to do as a result of the program” (p. 27) and that program outcomes should support the hosting institution’s program outcomes as well as align with student learning outcomes. First, program outcomes were identified based upon the leadership mentoring program’s mission and were modeled after the hosting department’s leadership program outcomes.
- Apply knowledge of the investment relationship model and positive psychology to help others reach their full potential.
- Reinvest personal leadership strengths, values, and skills for the purpose of positive social change.
- Provide active listening, feedback, and/or guidance to sharpen the actions and thoughts of others.
- Effectively utilize interpersonal skills and responsible patterns of behavior to develop empathetic and trusting relationships.
Next, objectives were identified to articulate how each program outcome is achieved. Simultaneously, the program outcomes were also analyzed to determine leadership competencies (Seemiller,
2014) required to achieve the program outcomes. Last, program outcomes, objectives, and leadership competencies were mapped to articulate the competencies required to meet the program outcomes, how those competencies are developed through program objectives, and how the objective accomplishment ultimately leads to the achievement of program outcomes (see Figure 1).
Figure 1. Map of Leadership Mentoring Program Outcomes, Objectives, and Targeted Competencies
Upon completion of the map (see Figure 1), it was reviewed by the leadership mentoring program’s board of directors to ensure that the program outcomes, associated objectives, and targeted leadership competencies properly reflected the program’s mission. An all-student retreat was also developed to educate students on the map and to train the students on how the map translates to yearly growth in their mentoring relationships (see Figure 2). Student feedback from the retreat indicated that (a) peer discussion in small groups and/or hearing other mentoring experiences across projects (41%), (b) learning from more experienced [program] mentors (35%), and (c) the developed materials (21%) were the most helpful part of the retreat in terms of developing their mentee’s leadership talents.
Figure 2. Translation of Program Outcome Map to Yearly Growth in Mentoring Relationship
Assessment and Evaluation Plan. To evaluate competency development, specifically the knowledge, values, and abilities associated with the targeted leadership competencies, an assessment was created (Self-Evaluation of Competencies) that adapted Seemiller’s (2014) knowledge, value, and/or ability statements for each of the targeted leadership competencies and applied Seemiller’s (2014) suggested response anchors (1 = Did not increase, 4 = Greatly increased). Respondents were asked to indicate their perceived degree of change related to targeted competency statements, such as (a) My ability to reflect on experiences (targeting Reflection and Application), (b) My understanding of my strengths and values (targeting Self-Understanding), (c) The skills I need to build productive relationships (targeting Productive Relationships), (d) My ability to engage in mentoring to help others reach their full potential (targeting Mentoring), (e) The skills I need to foster empathy-level relationships (targeting Empathy), (f) My ability to provide feedback, guidance, and/or advice to improve the actions or thoughts of others (targeting Providing Feedback), (g) My understanding of how to act in ways that benefit society (targeting Social Responsibility), and (h) My ability to see commitments through to the end (targeting Follow Through).
Behavioral development among the targeted leadership competencies seemed to be best assessed by virtue of perceived proficiency and growth among the program outcome statements. For example, the behavioral statement associated with the empathy leadership competency is “Demonstrates empathy” (Seemiller, 2014, p. 45). With one of the articulated program outcomes as, “Effectively utilize interpersonal skills and responsible patterns of behavior to develop empathetic and trusting relationships,” behavior in the empathy leadership competency was evaluated by assessing perceived student proficiency and growth on the program outcome linked with empathy. Thus, a second assessment was created (Program Outcomes Assessment) that asked student participants to (a) rate perceived proficiency in each program outcome on a four-point scale using Seemiller and Roberts’ (2016) example leadership competency rubrics (1=Limited Proficiency, 4=Exceptional Proficiency), (b) indicate which program outcome area had the most growth and why, and (c) to identify which program outcomes were reflected in their final project/symbol and how those program outcomes were reflected in the project/symbol.
The internal program evaluation plan included administering both assessments to all leadership mentors directly after the LMP’s Annual Recognition Day in April, 2019, 2020, and 2021 so as to give each student at least one year’s worth of participation to reflect upon. Competency averages and average total were calculated for all eight knowledge, value, and ability items for the Self-Evaluation of Competencies assessment. It was decided that an overall total average value > 16.0 was considered the threshold value for determining whether the profile of leadership competencies was sufficiently targeted and developed through the leadership mentoring program as 16.0 would be considered a 2.0 average score (Slightly Increased) on all eight items of the assessment and indicative of “basic proficiency” as outlined in example leadership competency rubrics offered in Seemiller (2016, p. 60). If overall total average was < 16.0, a focus group was to be conducted with the staff advisors to examine item-level averages and to ascertain their perspective on why certain items had lower or higher averages and how program operations could improve as to better target desired leadership competencies. The second assessment, Program Outcomes Assessment, was evaluated by average proficiency levels for each program outcome statement. An average was calculated for each program outcome statement. Program outcome statement average values > 2.0 (Basic Proficiency) were considered the threshold value for determining whether overall student self-perceived proficiency was being targeted and developed through the leadership mentoring program. The items related to growth and outcome reflection in Annual Recognition Day projects were evaluated by percentage. Any program outcome statement that (a) has an average proficiency rating < 2.0, (b) receives less than 25% identification by students as a significant growth area, and/or (c) receives less than 25% reflection in Annual Recognition Day projects was to be reviewed by staff and/or the aforementioned focus group to ascertain why certain outcomes demonstrated lower or higher averages and how program operations could improve as to better target growth and proficiency in all program outcome areas.
Assessment Results. Overall, 239 program evaluation assessments were completed over a three-year period from 2019 to 2021 (n = 76 in 2019; n = 77 in 2020; n = 86 in 2021). The first assessment, Self-Evaluation of Competencies (adapted from Seemiller, 2014), was designed to gauge perceived growth in competency knowledge, value, and/or ability. Overall, the Self-Evaluation of Competencies assessment results indicated high levels of perceived increase on each of the targeted competencies (see Table 1). All targeted competencies over all three years had an average rating above 3.0 (Moderately Increased), indicating that student leadership mentors, on average, perceived a moderate increase in each of the targeted competencies. Additionally, all competencies had over 75% of respondents rating their perceived growth as 3.0 or higher, indicating that over three-fourths of respondents each year over a three-year period perceived their competency growth as moderately to greatly increased in each targeted competency.
Table 1
Self-Evaluation of Competencies Results
| 2019 | 2020 | 2021 | |||||
| (n = 76) | (n = 77) | (n = 86) | |||||
| Question | Targeted competency | Average | % > 3.0 rating | Average | % > 3.0 rating | Average | % >
3.0 rating |
| My ability to reflect on experiences | Reflection and Application | 3.39 | 90.6 | 3.35 | 89.7 | 3.44 | 89.5 |
| My understanding of my strengths and values | Self-Understanding | 3.39 | 88.0 | 3.31 | 87.1 | 3.54 | 95.3 |
| The skills I need to build productive relationships | Productive Relationships | 3.54 | 93.2 | 3.51 | 96.1 | 3.58 | 89.5 |
| My ability to engage in mentoring to help others reach their full potential | Mentoring | 3.53 | 94.4 | 3.38 | 91.0 | 3.50 | 91.8 |
| The skills I need to foster empathy-level relationships | Empathy | 3.55 | 90.7 | 3.45 | 90.9 | 3.45 | 89.5 |
| My ability to provide feedback, guidance, and/or advice to improve the actions or thoughts of others | Providing Feedback | 3.45 | 92.0 | 3.40 | 94.9 | 3.59 | 97.7 |
| My understanding of how to act in ways that benefit society | Social Responsibility | 3.27 | 78.4 | 3.27 | 83.2 | 3.36 | 86.0 |
| My ability to see commitments through to the end | Follow-Through | 3.40 | 89.3 | 3.23 | 81.9 | 3.44 | 87.2 |
To examine behavioral proficiency among the targeted leadership competencies, the Program Outcomes Assessment was designed to self-assess perceived proficiency level on the program outcome statements as well as provide open-ended commentary on (a) which program outcome area had the most growth and why and (b) which program outcomes were reflected in their final year-end project/symbol. Overall, the results from the Program Outcomes Assessment across all three years indicated strong self-perception of proficiency in each program outcome area among student leadership mentors (see Table 2).
Table 2
Program Outcomes Assessment
| 2019 | 2020 | 2021 | |||||
|---|---|---|---|---|---|---|---|
| (n = 76) | (n = 77) | (n = 86) | |||||
| Program outcome statement | Targeted competencies | Average perceived proficiency level | % > 3.0 rating | Average perceived proficiency level | % > 3.0 rating | Average perceived proficiency level | % > 3.0 rating |
| Apply knowledge of the investment relationship model and positive psychology to help others reach their full potential | Reflection and Application | 3.36 | 90.6 | 3.27 | 87.1 | 3.18 | 84.9 |
| Self-Understanding | |||||||
| Productive Relationships | |||||||
| Mentoring | |||||||
| Reinvest personal leadership strengths, values, and skills for the purpose of positive social change | Reflection and Application | 3.39 | 96.0 | 3.40 | 93.6 | 3.49 | 91.9 |
| Mentoring | |||||||
| Social Responsibility | |||||||
| Provide active listening, feedback, and/or guidance to sharpen the actions and thoughts of others | Mentoring | 3.63 | 98.7 | 3.60 | 100.0 | 3.60 | 96.5 |
| Providing Feedback | |||||||
| Effectively utilize interpersonal skills and responsible patterns of behavior to develop empathetic and trusting relationships | Productive Relationships | 3.56 | 97.4 | 3.49 | 94.8 | 3.52 | 96.5 |
| Empathy | |||||||
| Follow-Through | |||||||
The qualitative questions associated with the Program Outcomes Assessment provided depth and additional insight into student perception of growth as a result of their leadership mentoring experience. In response to the question, “In which program outcome do you feel you’ve grown the most and why?”, Table 3 outlines the summarized ranking results.
Table 3
Perceived Growth in Program Outcomes
| 2019 | 2020 | 2021 | ||||
| (n = 64) | (n = 67) | (n = 70) | ||||
| Program outcome statement | Ranking of most perceived growth | % of respondents identified | Ranking of most perceived growth | % of respondents identified | Ranking of most perceived growth | % of respondents identified |
| Apply knowledge of the investment relationship model and positive psychology to help others reach their full potential | 3 | 14.1 | 4 | 13.4 | 4 | 5.7 |
| Reinvest personal leadership strengths, values, and skills for the purpose of positive social change | 4 | 9.4 | 3 | 17.9 | 3 | 27.1 |
| Provide active listening, feedback, and/or guidance to sharpen the actions and thoughts of others | 1 | 53.1 | 1 | 46.2 | 2 | 31.4 |
| Effectively utilize interpersonal skills and responsible patterns of behavior to develop empathetic and trusting relationships | 2 | 23.4 | 2 | 22.4 | 1 | 35.7 |
The program outcome statement identified by the most student respondents in 2019 and 2020 (53.1% and 45.6%, respectively) was Provide active listening, feedback, and/or guidance to sharpen the actions and thoughts of others. This program outcome statement also received the highest average proficiency rating over the course of three years (3.63 in 2019, 3.60 in 2020, 3.60 in 2021). Open-ended responses provided a more nuanced understanding of why the Provide active listening statement was identified by as the strongest or second-strongest program outcome area for growth. The keyword in the aforementioned program outcome statement seemed to be “listening” as 54 student responses over the course of three years (16/34 in 2019; 19/31 in 2020; 19/22 in 2021) specifically discussed listening in their justification statement. For example, one student respondent in 2019 indicated, “Because of my involvement in [LMP], I have pushed myself to take on leadership roles where I get to work closely with others, and where my job is to be a listener. I did not value the power in listening to others until I joined [LMP], and now I am pursuing a career path where this skill is essential.” In 2020, another student respondent indicated, “[LMP] reestablishes the importance of active listening and has made me more aware of how well I listen when others are speaking.” From the 2021 assessment, another student confirmed significant growth in listening specifically: “I’ve become a lot better at LISTENING during conversations and being engaged/present, rather than just formulating my next question or getting distracted.” Eleven student respondents over the three-year period specifically identified the interpersonal skills for leadership course and six identified their project meetings as important developmental experiences to growth in this program outcome. For example, one student from 2019 highlighted the training course and why it was growth-facilitating: “I think I’ve grown most in the area of active listening, feedback, and guidance to sharpen the actions and thoughts of others. This was due mostly to taking the [LMP] class this Fall and learning about other people’s relationships and seeing what sort of strategies they used and how I could use and modify their strategies in my own relationships.”
Effectively utilize interpersonal skills and responsible patterns of behavior to develop empathetic and trusting relationships was identified by the most student respondents in 2021 as 35.7% of student respondents identified this outcome as their highest area of perceived growth. Over half of the respondents from 2019 – 2021 pointed to improved or successful relationships in response to growth in this program outcome. Some examples include: “I feel like I have grown the most in this area based on the outcomes I am seeing after consistently using my interpersonal skills in relationships. I have realized that my relationships have improved and have been reaching deeper levels than ever” (2019); “After taking the [LMP] class, interacting with [my mentee], and being more intentional about my interpersonal behavior, I became more critical of my actions resulting in better interpersonal skills and responsible patterns of behavior. That has led to [my mentee] and I having a very strong relationship” (2020); “In the past year, I think my relationship with my mentee has gotten significantly more deep and trusting as we’ve shared more and gotten more vulnerable with each other” (2021). Eight respondents pointed to listening again between 2019 – 2021 as the specific vehicle for building empathetic and trusting relationships. One student example articulates the association between listening and trusting relationships: “I’ve gained a deeper understanding on what it means to be a good listener, and because of this, I have built trust quicker and more deeply than before.”
Year over year respondents who identified Apply knowledge of the investment relationship model and positive psychology to help others reach their full potential as their strongest program outcome area for growth pointed specifically to their relationship with their leadership mentee in justifying their choice, but approximately 30% of responses over the three-year period also extended their growth in applying the investment relationship model outside of the organization. For example, one student respondent in 2019 highlighted growth in this program outcome area to their career in teaching: “I grew most in this area because, as a teacher I use it on a daily basis to enrich the lives of my students. Working with my [mentee] helped me diligently invest on an individual level, which in turn, helped me to apply its lessons to every relationship I encounter.” Another respondent in 2020 confirmed this notion: “I began to see the importance of the investment relationship and how it can be utilized in all relationships, not just mentoring type relationships.”
Respondents who identified Reinvest personal leadership strengths, values, and skills for the purpose of positive social change as the strongest outcome area of perceived growth highlighted the power of knowing their own strengths: “I feel this has been my biggest growth this year because I have spent more time this last year getting to know my own strengths and how I can best utilize those strengths to apply to various aspects of the situation I am in” (2019). Student responses year over year also discussed the power of reinvesting those strengths for the benefit of society as represented by this quote from 2020:
Prior to [LMP], it was more difficult for me to develop a way in which I could use my personal strengths to make a larger impact within social change. This year provided me with a lot of tools and conversations that led me to be able to construct activities, conversations, and interactions in which I was able to consciously capitalize on my strengths, values, and skills to further create a positive social change and provide headway to draw others towards likemindedness.
In sum, respondents consistently saw themselves engaging with each of the program outcome statements at an Advanced or Exceptional level, a self-assessment that was further supported by respondents’ ability to articulate the why behind their behaviors.
The final open-ended question in the Program Outcomes Assessment related to student perception of program outcome areas reflected in their year-end cumulative project. Table 4 offers the summarized results.
Table 4
Perceived Reflection of Program Outcome Areas in Final Project
| 2019 | 2020 | 2021 | ||||
| (n = 39) | (n = 51) | (n = 53) | ||||
| Program outcome statement | Ranking of reflection in final project | % of respondents identified | Ranking of reflection in final project | % of respondents identified | Ranking of reflection in final project | % of respondents identified |
| Apply knowledge of the investment relationship model and positive psychology to help others reach their full potential | 3 | 17.9 | 3 | 13.7 | 4 | 7.5 |
| Reinvest personal leadership strengths, values, and skills for the purpose of positive social change | 1 | 35.9 | 2 | 23.5 | 1 | 54.7 |
| Provide active listening, feedback, and/or guidance to sharpen the actions and thoughts of others | 4 | 10.3 | 4 | 11.8 | 3 | 17.0 |
| Effectively utilize interpersonal skills and responsible patterns of behavior to develop empathetic and trusting relationships | 1 | 35.9 | 1 | 51.0 | 2 | 20.7 |
The rankings in the second open-ended question did not necessarily match the first open-ended question. As student leadership mentors documented the growth of their relationship with their leadership mentee for their final project, they identified Reinvest personal leadership strengths, values, and skills for the purpose of positive social change and Effectively utilize interpersonal skills and responsible patterns of behavior to develop empathetic and trusting relationships as being most reflective of that relationship growth. Recall that the Reinvest personal leadership strengths program outcome statement consistently received the lowest or the second lowest ranking of perceived growth, but the Effectively utilize interpersonal skills program outcome statement received the highest or the second highest ranking of perceived growth year over year. Thus, the Effectively utilize interpersonal skills program outcome statement performed the strongest across both personal and mentoring relationship growth.
How Assessment Data Were Used for Program Evaluation. In sum, the assessment results indicated strong perceived growth in targeted leadership competency knowledge, value, and/or ability as well as behavioral proficiency among the targeted leadership competencies year over year. Overall, average calculated totals for 2019 – 2021 across all eight knowledge, value, and ability items were 27.52 (2019), 26.90 (2020), 27.90 (2021), all of which exceeded the 16.0 threshold value for determining whether the overall profile of leadership competencies had been sufficiently targeted and developed through the LMP. Additionally, each program outcome area had a calculated average value above the 2.0 threshold value year over year for determining whether overall student self-perceived proficiency was being targeted and developed through the leadership mentoring program.
The program outcome area with the highest average level of perceived proficiency and identified by the highest percentage of respondents in 2019 and 2020 as the most significant growth area was Provide active listening, feedback, and/or guidance to sharpen the actions and thoughts of others. The specific mention of the interpersonal skills for leadership class as well as project meetings among the qualitative results also suggests that students drew a connection between the outcome area and their training and reflection opportunities. The results from the two qualitative questions, taken collectively, perhaps suggest Effectively utilize interpersonal skills and responsible patterns of behavior to develop empathetic and trusting relationships as the program outcome area most representative of combined student personal growth and mentoring relationship growth.
In 2019, the only program outcome statement that received more 3.0 proficiency ratings (53.3%) than 4.0 ratings (42.7%) was Reinvest personal leadership strengths, values, and skills for the purpose of positive social change. This result was consistent with the Self-Evaluation of Competencies result where the lowest rated competency with the highest variability (20% of respondents perceived their growth as only Slightly Increased) in 2019 was social responsibility – a targeted competency in the program outcome statement. Additionally, the Reinvest personal leadership strengths program outcome statement in 2019 had the lowest percentage of students (9.4%) who identified the outcome statement as the area in which they grew the most. The evaluation of this assessment result led to a change in retreat curriculum that focuses on the social change model – a leadership development model designed to enhance student self-understanding and leadership competence to facilitate positive social change (SCM; Higher Education Research Institute, 1996) – to create a better connection for students between their mentoring experience and its societal benefit. Following this programmatic change, the Reinvest personal leadership strengths program outcome statement had equal percentages of 3.0 and 4.0 proficiency ratings (46.8%) in 2020. In 2021, the Reinvest personal leadership strengths program outcome statement had more 4.0 proficiency ratings (57%) than 3.0 ratings (34.9%). Additionally, the social responsibility competency average rating saw a steady increase in percentage of students who indicated a 3.0 (Moderately Increased) or 4.0 rating (Greatly Increased) of competency growth (78.4% in 2019; 83.2% in 2020; 86.0% in 2021). The Reinvest personal leadership strengths program outcome statement also saw a steady increase in percentage of students who identified the outcome statement as the area in which they grew the most (9.4% in 2019; 17.6% in 2020; 27.1% in 2021).
In 2020 and 2021, the program outcome statement that received more 3.0 proficiency ratings (46.8% in 2020; 57% in 2021) than 4.0 ratings (40.3% in 2020; 27.9% in 2021) and had the most ratings of 2.0 (indicating Basic Proficiency) in comparison to other program outcome statements was Apply knowledge of the investment relationship model and positive psychology to help others reach their full potential. Additionally, the Apply knowledge of the investment relationship model consistently received the third or fourth ranking of perceived growth in program outcomes and the third or fourth ranking of reflected program outcome areas in final project. Yet, the targeted competency of Mentoring (as assessed by the item stem, “My ability to engage in mentoring to help others reach their full potential”) was consistently in the top three year over year in the percentage of students who indicated a 3.0 rating (Moderately Increased) or above in self-perceived competency growth. To address this gap, moving forward, we plan to infuse the “Investment Triangle” (Figure 2) more regularly in our weekly mentor reflection practices and include a specific focus on investment relationship development in training our student staff responsible for each of the LMP projects. Building upon the current paper, the student staff position will also be evaluated and assessed for growth opportunities. In so doing, we hope to help students draw a more regular connection between their weekly mentoring activities and the application of the investment relationship model as well as positive psychology principles.
Reflections and Recommendations
The combined results from the Self-Evaluation of Competencies and the Program Outcomes Assessment offer a variety of suggestions related to leadership mentoring program design as well as evaluation of leadership mentoring programs. The strong average rating and the percentage of respondents rating > 3.0 for both the perceived increase on each of the targeted competencies as well as the perception of proficiency in each program outcome area suggests that the leadership competencies identified from Seemiller’s (2014) Student Leadership Competencies model were targeted well through the LMP design. Further, the results suggest that perceived leadership competency growth was associated with perceived proficiency in demonstrating the competencies through the program outcomes. Thus, utilizing the Student Leadership Competencies model (Seemiller, 2014) may serve as a beneficial tool in leadership mentoring program design for the development of outcomes and associated objectives that address appropriate competencies. Additionally, mapping the program outcomes to associated objectives and targeted leadership competencies may serve as a useful evaluation framework to assess student leadership growth and to document specific leadership development resulting from a leadership mentoring experience. The utility of the evaluation framework presented in the current paper may also extend beyond just leadership mentoring programs to leadership education initiatives broadly.
The assessment and evaluation plan as outlined was entirely self-assessment based, which has limitations associated with social desirability and associated score inflation (Spector, 2004). Leadership mentoring effectiveness can and perhaps should be evaluated via other means, specifically observational evaluation from mentoring program staff and even, potentially, from the mentee. The individualized nature of mentoring relationships presents an assessment and evaluation challenge for external observation as demonstrations of competencies like Productive Relationships, Mentoring, and Providing Feedback may be somewhat reliant upon mentee receptivity in the relationship.
We were inspired by Rosch et al.’s (2017) Illinois Leadership Inventory (ILI) where students receive real-time feedback on their proficiency level within each of the targeted leadership competencies and have reflected on its application in leadership mentoring. Again, considering the individualized nature of mentoring relationships and the challenges associated with self-assessment versus external observation, formative assessment in leadership mentoring programs may be served well by a combination of unorthodox self-assessment and peer feedback. Specifically, rather than have students evaluate themselves or their peers on proficiency level or demonstrated growth within each competency or program outcome, formative assessment in leadership mentoring programs could include having leadership mentors identify which activities (i.e., with their mentee or with each other) promote the most growth in each competency area and associated program outcome.
Conclusion. Leadership mentoring is confronted often with identifying valuable program evaluation data that actually aid in decision-making. In our case specifically, we found ourselves wanting to ask assessment questions that, while interesting, did not allow us to make evaluation decisions. The individualized nature of mentoring relationships requires a nuanced look at outcomes, objectives, assessment, and evaluation. By innovating practice around the development, assessment, and evaluation of leadership mentoring programs, leadership educators can more soundly design and deliver leadership mentoring programs and more precisely measure and demonstrate impact. Additionally, innovating practice around the development, assessment, and evaluation of leadership mentoring programs can create a much more robust environment for sound empirical
research that tests outcomes associated with leadership mentoring. The more definitively we can identify which competencies are developed through leadership mentoring, the more confidently we can hypothesize associations between leadership variables in the context of developmental interactions.
References
Allen, T. D., & Eby, L. T. (2010). Common bonds: An integrative view of mentoring relationships. In T. D. Allen & L. T. Eby (Eds.), The Blackwell handbook of mentoring: A multiple perspectives approach (pp. 397-419). Blackwell Publishing.
[Author], Griesen, J. V., Hoover, R. E., Creswell, J. W., & Dlugosh, L. L. (2015). Generativity in college students: Comparing and explaining the impact of mentoring. Journal of College Student Development, 56, 651-669. https://doi.org/10.1353/csd.2015.0070
[Author], & [Author]. (2019). Generativity and socially responsible leadership among college student leaders who mentor. Journal of Leadership Education, 18(3).
[Author]. (2020). The development of generativity among college student leaders who mentor: A growth curve analysis [Unpublished doctoral dissertation]. University of Nebraska – Lincoln.
Bass, B. M. (1990). Bass and Stogdill’s handbook of leadership (3rd ed.). Free Press.
Bureau, D. A., & Lawhead, J. (2018). Assessing student leadership development from mentoring, coaching, and advising. In L. J. Hastings and C. Kane (Eds)., New Directions for Student Leadership: No. 158. Role of mentoring, coaching, and advising in developing leadership (pp. 73-85). Jossey-Bass. https://doi.org/10.1002/yd.20289
Blinn-Pike, L. (2010). The benefits associated with youth mentoring relationships. In T. D. Allen & L. T. Eby (Eds.), The Blackwell handbook of mentoring: A multiple perspectives approach (pp. 165-188). Blackwell Publishing.
Burke, R. J. (1984). Mentors in organizations. Group and Organization Studies, 9, 353-372.
https://doi.org/10.1177/105960118400900304
Campbell, C. M., Smith, M., Dugan, J. D., & Komives, S. R. (2012). Mentors and college student leadership outcomes: The importance of position and process. The Review of Higher Education, 35, 595-625.
Castro, S. L., Terri, A., & Williams, E. A. (2004). Validity of Scandura and Ragins’ (1993) multidimensional mentoring measure: An evaluation and refinement. Management Faculty Articles and Papers. Paper 7. Retrieved from http://scholarlyrepository.miami.edu/management_articles/7.
Collins-Shapiro, C. (2006). Mentoring and leadership development. Leadership Insights and Application Series #21. National Clearinghouse for Leadership Programs.
Council for the Advancement of Standards in Higher Education. (2015). CAS learning and development outcomes. In CAS professional standards for higher education (7th ed.; pp. 24-28). Author.
Crisp, G. (2010). Impact of mentoring on success of community college students. Review of Higher Education, 34, 39-60.
Crisp, G., & Cruz, I. (2009). Mentoring college students: A critical review of the literature between 1990 and 2007. Research in Higher Education, 50, 525-545.
Day, D. V. (2000). Leadership development: A review in context. Leadership Quarterly, 11(4), 581 – 613. https://doi.org/10.1016/S1048-9843(00)00061-8
DuBois, D. L., Portillo, N., Rhodes, J. E., Silverthorn, N., & Valentine, J. C. (2011). How effective are mentoring programs for youth? A systematic assessment of the evidence. Psychological Science in the Public Interest, 12, 57-91.
Dugan, J. P., & Komives, S. R. (2007). Developing leadership capacity in college students: Findings from a national study. A Report from the Multi-Institutional Study of Leadership. National Clearinghouse for Leadership Programs.
Dugan, J. P., & Komives, S. R. (2010). Influences on college students’ capacities for socially responsible leadership. Journal of College Student Development, 51, 525-549.
https://doi.org/10.1353/csd.2010.0009
Eby, L. T., Rhodes, J. E., & Allen, T. D. (2010). Definition and evolution of mentoring. In T. D. Allen & L. T. Eby (Eds.), The Blackwell handbook of mentoring: A multiple perspectives approach (pp. 7-20). Blackwell Publishing Ltd.
Fagan, M. M., & Walter, G. (1982). Mentoring among teachers. Journal of Educational Research, 76, 113-118. https://doi.org/10.1080/00220671.1982.10885433
Furco, A., & Billig, S. H. (Eds.). (2002). Service learning: The essence of the pedagogy. IAP.
Gallup, Inc. (2014). Great jobs, great lives: The 2014 Gallup-Purdue Index report. Retrieved from http://www.wsac.wa.gov/sites/default/files/2014.ptw.%2860%29.pdf.
Hastings, L. J., & Kane, C. (2018). Distinguishing mentoring, coaching, and advising for developing leadership. In L. J. Hastings & C. Kane (Eds.), New Directions for Student Leadership: No. 158. Role of mentoring, coaching, and advising in developing leadership (pp. 9 – 22). https://doi.org/10.1002/yd.20284
Higher Education Research Institute. (1996). A social change model of leadership development: Guidebook version III. National Clearinghouse for Leadership Programs.
International Leadership Association. (2009). Guiding questions: Guidelines for leadership education programs. College Park, MD: Author. Retrieved from: http://www. ila-net.org/communities/LC/GuidingQuestionsFinal.pdf.
Komives, S. R., & Collins-Shapiro, C. (2006). The critical role of mentoring in leadership development. Concepts and Connections, 14(3), 9-11.
Komives, S. R., Longerbeam, S. D., Owen, J. E., Mainella, F. C., & Osteen, L. (2005). Developing a leadership identity: A grounded theory. Journal of College Student Development, 46, 593-611.
Komives, S. R., Longerbeam, S. D., Owen, J. E., Mainella, F. C., & Osteen, L. (2006). A leadership identity development model: Applications from a grounded theory. Journal of College Student Development, 47, 401-418.
Komives, S. R., Longerbeam, S. D., Mainella, F., Osteen, L., & Owen, J. E. (2009). Leadership identity development: Challenges in applying a developmental model. Journal of Leadership Education, 8(1), 11-47.
Komives, S. R., Lucas, N., & McMahon, T. (2013). Exploring leadership: For college students who want to make a difference (3rd ed.). Jossey-Bass.
Lockwood, A. L., Carr Evans, S., & Eby L. T. (2010). Reflections on the benefits of mentoring. In T. D. Allen & L. T. Eby (Eds.), The Blackwell handbook of mentoring: A multiple perspectives approach (pp. 233-236). Blackwell Publishing Ltd.
NASPA/ACPA. (2004). Learning reconsidered: A campus-wide focus on the student experience. Washington, DC: National Association of Student Personnel Association Administrators (NASPA) and American College Personnel Association (ACPA).
Newby, T. J., & Corner, J. (1997). Mentoring for increased performance: Benefits and key principles. Performance Improvement, 36(4), 10-13.
https://doi.org/10.1002/pfi.4140360404
Nora, A., & Crisp, G. (2007). Mentoring students: Conceptualizing and validating the multi-dimensions of a support system. Journal of College Student Retention: Research, Theory and Practice, 9, 337–356.
Parker, P., Hall, D. T., & Kram, K. E. (2008). Peer coaching: A relational process for accelerating career learning. Academy of Management Learning and Education, 7, 487-503.
Peterson, B. E., & Stewart, A. J. (1996). Antecedents and contexts of generativity motivation at midlife. Psychology and Aging, 11, 21-33.
Posner, B. Z., & Brodsky, B. (1992). A leadership development instrument for college students. Journal of Student Development, 33(3), 231-237.
Renn, K. A. (2007). LGBT student leaders and queer activists: Identities of lesbian, gay, bisexual, transgender, and queer identified college student leaders and activists. Journal of College Student Development, 48(3), 311-330. https://doi.org/10.1353/csd.2007.0029
Roberts, D. M., & Bailey, K. J. (2016). Setting the stage: The intersection of leadership and assessment. In D. M. Roberts and K. J. Bailey (Eds.), New Directions for Student Leadership: No 151. Assessing student leadership (pp. 7-18). Jossey-Bass.
https://doi.org/10.1002/yd.20197
Rosch, D. (Ed.). (2017). New Directions for Student Leadership: No. 155. The role of student organizations in development leadership. Jossey-Bass. https://doi.org/10.1002/yd.20245
Rosch, D., Spencer, G. L., & Hoag, B. L. (2017). A Comprehensive Multi-Level Model for Campus-Based Leadership Education. Journal of Leadership Education, 16(4), 124 – 134.
Ryan, L. (1994). The case for national student leadership on community college issues. Leadership Abstracts, 7(6).
Seemiller, C. (2014). The student leadership competencies guidebook: Designing intentional leadership learning and development. John Wiley & Sons.
Seemiller, C. (2016). Assessing student leadership competency development. In D. M. Roberts and K. J. Bailey (Eds.), New Directions for Student Leadership: No 151. Assessing student leadership (pp. 51-66). Jossey-Bass. https://doi.org/10.1002/yd.20200
Seitz, S., & Pepitone, S. (1996). Servant leadership: A model for developing college students. Metropolitan Universities: An International Forum, 6(4), 113 – 122.
Seligman, M. E. P., & Csikszentmihalyi, M. (2000). Positive psychology: An introduction. American Psychologist, 55(1), 5 – 14.
Turner, P. (2019). Leadership capability through personal insight and leadership identity. In P. Turner (Ed.), Leadership in Healthcare (pp. 143-172). Palgrave Macmillan, Cham. https://doi.org/10.1007/978-3-030-04387-2_6