Creating Leaders: A Pilot Pre/Post Evaluation of an Ontological/Phenomenological Model
Nancy Carney, PhD Associate Professor School of Medicine
Oregon Health & Science University
Michael Jensen, PhD
Jesse Isidor Straus Professor of Business Administration, Emeritus Harvard University
Graduate School of Business Administration
Nicolas Ballarini, MS
Research Associate National University of Rosario
Jeronima Echeverria, PhD Executive Vice-Chancellor Chief Academic Officer
California State University System (retired) Professor and Provost Emerita
California State University, Fresno
Tracie Nettleton, MBI
Research Associate, School of Medicine Oregon Health & Science University
Molly Stillwell, MA Research Associate Saybrook University
Werner Erhard
Independent
Abstract
This pilot is a pre/post comparative assessment of a leadership course developed and delivered using an innovative, ontological/phenomenological model of education. Participants in the course delivered in Singapore in July of 2014 provided measures of the effectiveness of their leadership before and after the course, using a scale from 1 (least effective) to 10 (most effective). The difference in scores from pre- to post-course was the unit of measure. Of 167 participants, 72% provided pre- and post-course measures. Average scores forparticipants’ effectiveness as leaders in the domains of Relationships, Vocation, Avocation, and Self increased
from pre- to post-course by 1.9, 1.86, 1.64, and 1.85 respectively (p < 0.0001). Future research of this innovative model of leadership education will include long-term follow-up.
Introduction
During the past eleven years scholars of leadership and finance have developed and delivered a course – Being a Leader & the Effective Exercise of Leadership: An
Ontological/Phenomenological Model (“The Course”). As the name suggests, The Course (still in development) employs an ontological model and a phenomenological method, and makes a unique promise to those who participate: “You will leave this course being a leader and exercising leadership effectively as your natural self-expression.” While informal course evaluations indicate the course is highly effective, a systematic process is needed to evaluate whether The Course produces leaders as promised, and if so, how “leadership as one’s natural self-expression” manifests as action and results in the lives of those who participate.
The pedagogical process for most leadership courses is epistemological, an approach founded in the accumulation and delivery of knowledge. The emphasis is the practice and mastery of behaviors known empirically to be consistent with successful leadership. As such, instruments used to assess such courses query responders about their behaviors, leadership styles, and situation-specific judgments. Typically the unit of measure of leadership consists of the amount and quality of time spent engaging in pre-specified behaviors (Alban-Metcalfe & Alimo- Metcalfe, 2000; Dulewicz & Higgs, 2005; Dussault, Frenette, & Fernet, 2013; Kouzes & Posner, 2008; Kouzes & Posner 2013; Peus, Braun, & Frey, 2013; Van der Stoop, 2011).
In contrast, the pedagogical process used in The Course is founded in ontology (literally translated, the study of being) and phenomenology (a method that deals with being and action as they are actually lived; as a first-person experience). The authors assert that this method allows participants to discover leadership as “one’s natural self-expression.” For a comprehensive review of the background and theory supporting this approach, refer to the pre-course assignments (http://papers.ssrn.com/abstract=2416455) and to the slide deck used to deliver the course (http://ssrn.com/abstract=1263835). In The Course, “The being of being a leader and the actions of the effective exercise of leadership” are accessed and taught “as being and action are actually experienced ‘on the court’, specifically as these are actually lived real-time (first person experience of).” (Snook, Nohria, & Khurana, 2012, p. 4). The curriculum does not pre-specify particular behaviors as necessary or exemplary to leadership. Rather, all ways of being and acting are possible expressions of leadership. Consequently, instruments used to assess programs from the epistemological model cannot be used to assess The Course, because the unit of measure of leadership as one’s natural self-expression is not behavior-specific. Ultimately the measure of leadership as one’s self-expression is to be found in the results people produce in their lives.
To address the need for an assessment instrument for The Course, we elected not to use an established scale, but to develop one that would measure specific results participants produce in their lives as a result of completing The Course. Our first step was to design a pilot, prospective pre/post study; the primary goals were to demonstrate feasibility and to field test categories of inquiry and candidate items. Our secondary goal was to collect data on outcomes produced by those who participate in The Course. We hypothesized there would be a significant
improvement in self-reports of leadership pre- to post-course, quantified by objective measures of results in the lives of participants of The Course.
Methods
Study Design. This was a prospective, comparative study using participants as their own controls. Measures of leadership were taken before and after participants took The Course, with the change from pre- to post-course as the unit of measure.
Setting. The study was conducted with participants in Being a Leader and the Effective Exercise of Leadership: An Ontological/Phenomenological Model delivered in Singapore in July of 2014, hosted by Nanyang Technological University.
Participants. All individuals who registered for the course were invited to participate in the research. Any individual 18 years of age or older was eligible to register for The Course.
Variables and Data. All data were self-reported. In addition to participant demographics (see Table 1) we queried participants for their self-assessment of their effectiveness as leaders in the domains of Relationships, Vocation, Avocation, and Self, in the format of check-box, Likert Scale, and open-ended, narrative responses. Likert scale scores ranged from 1 (least effective) to 10 (most effective). There were 30 questions in the pre-course survey; the post-course surveys ranged from 22 to 39 questions (survey instruments available on request).
Procedure. All survey instruments were administered through the online Research Electronic Data Capture system (REDCap) (Harris et al., 2009). After registering for The Course, participants were sent an e-mail with their unique link to the pre-course survey, and an information sheet informing them about the research. After completing The Course, participants were sent an e-mail with their unique link to the post-course surveys at three intervals – immediately after completing The Course, and at 4 and 12 months post-course. This protocol was approved by the Institutional Review Board (IRB) of Oregon Health & Science University (OHSU).
Data Analysis. The analysis for this report includes results from the pre- and first (immediate) post-course surveys only. Demographic data are reported as percentages. Change scores for quantitative questions were calculated as the difference between self-ratings from before to after The Course, and means were derived. Statistically significant differences between mean scores from pre- to post-course were determined using Student’s t-test for paired samples with a 2-sided significance level of 0.05. To confirm the robustness of the analysis a Wilcoxon Sign-Rank Test was also carried out. To account for the potential for response bias, we conducted sensitivity analyses, and assumed the value of “no change” from pre- to post-course scores for all non-respondents.
Results
One-hundred sixty-seven people registered for The Course. Of those, 123 (74%) responded to the pre-course survey, and 121 (72%) responded to the first post-course survey.
See Table 1 for demographic characteristics of the study sample. Demographic data were available for between 37% (Country of Residence) and 45% (Gender) of the non-responders group. A comparison of demographics between responders and non-responders for whom we had data revealed no significantdifferences.
The average scores for self-ratings of participants’ effectiveness as leaders in the domains of Relationships, Vocation, Avocation, and Self increased from pre- to post-course by 1.9, 1.86, 1.64, and 1.85 respectively (p < 0.0001). A sensitivity analysis conducted for each domain, assuming a value of “no change” from pre- to post-course, maintained a statistically significant increase in scores (p < 0.0001) (see Table 2).
Table 1. Demographics
Age (yrs) |
Residence |
||
18-29 |
6% |
Singapore |
33 |
30-39 |
37% |
Australia |
15 |
40-49 |
39% |
India |
15 |
50-59 |
11% |
Iran |
12 |
60-79 |
6% |
U.S. |
8 |
N/R |
1% |
China |
3 |
Germany |
3 |
||
Gender |
Thailand |
3 |
|
Female |
40% |
U.K. |
3 |
Male |
60% |
Other |
28 |
Affiliation |
Education |
||
Business |
40% |
High School |
2% |
Academic Faculty |
14% |
Some College |
2% |
Academic Student |
10% |
College Degree |
47% |
Consultant |
14% |
Master’s Degree |
31% |
Other |
21% |
MD/PhD |
16% |
N/R |
1% |
N/R |
2% |
Table 2. Self-Ratings of Leadership – Change Scores from Pre- to Post-Course and Results of Sensitivity Analyses
Respondents |
Sensitivity Analysis |
|
Relationships |
N = 121 |
N = 167 |
Mean [SD] |
1.909 [1.426] |
1.383 [1.484] |
t |
14.73 |
12.05 |
p |
< 0.0001 |
< 0.0001 |
Vocation |
N = 112 |
N = 167 |
Mean [SD] |
1.857 [1.713] |
1.246 [1.652] |
t |
11.47 |
9.74 |
p |
< 0.0001 |
< 0.0001 |
Avocation |
N = 100 |
N = 167 |
Mean [SD] |
1.640 [1.982] |
0.982 [1.735] |
t |
8.27 |
7.32 |
p |
< 0.0001 |
< 0.0001 |
Self |
N = 105 |
N = 167 |
Mean [SD] |
1.848 [1.714] |
1.162 [1.626] |
t |
11.05 |
9.24 |
p |
< 0.0001 |
< 0.0001 |
Discussion and Future Research
Our goals were to establish the feasibility of collecting data about the influence of The Course in the lives of the participants; to pilot test a set of questions designed to measure objective results; and to collect preliminary data from The Course delivered in Singapore in July of 2014. The web-based data collection structure was successfully launched, and was used for three additional courses. The responses from the Singapore course were used to revise the pilot set of questions, and the revisions were implemented for subsequent course research. Data were collected from 72% of the participants in The Course, the results of which confirmed our hypothesis that there would be a significant improvement in self-reported measures of leadership from pre- to post-course.
The primary weaknesses of this study are (a) the reliance on self-report as the only measure of change, (b) the short-term outcome measure time-point, and (c) the lack of assessment of the reliability and validity of the instruments, given this was a pilot.
Although self-report has been considered less reliable than measures that can be obtained objectively, some research comparing self-report with objective measures has demonstrated unexpectedly high concordance, with specificity as high as > 90% (Okura et al., 2004). An alternative for future courses would be to develop a line of questions for corroborators, to verify the accuracy of the self-reported data.
Future assessment of The Course will include longitudinal data up to 1 year post-course. Researchers from the field of Organizational Behavior assert that with self-report measures, the use of longitudinal data increases the confidence in conclusions about causal relations (Spector,
1994). Finally, with data from three cohorts of approximately 650 participants, we have material to assess the psychometric properties of our instruments.
References
Alban-Metcalfe, R., & Alimo-Metcalfe, B. (2000). The transformational leadership questionnaire (TLQ-LGV): a convergent and discriminant validation study. Leadership & Organizational Development Journal, 21(6), 280-296.
Dulewicz, V., & Higgs, M. (2005). Assessing leadership styles and organisational context.
Journal of Managerial Psychology, 20(2), 105-123.
Dussault, M., Frenette, E., & Fernet, C. (2013). Leadership: Validation of a self-report scale.
Psychological Reports: Human Resources & Marketing, 112(2), 419-436.
Harris, P. A., Taylor, R., Thielke, R., Payne, J., Gonzalez, N., & Conde, J. G. (2009). Research electronic data capture (REDCap) – A metadata-driven methodology and workflow process for providing translational research informatics support. Journal of Biomedical Informatics, 42(2), 377–381.
Kouzes, J., & Posner, B. (2008). The leadership challenge (4th ed.). Audio-Tech Business Book Summaries, 17(8), 1-16.
Kouzes, J., & Posner, B. (2013). Leadership practices inventory self (4th ed.). ISBN: 978-1-118- 18271-0.
Okura, Y., Urban, L. H., Mahoney, D. W., Jacobsen, S. J., & Rodeheffer, R. J. (2004).
Agreement between self-report questionnaires and medical record data was substantial for diabetes, hypertension, myocardial infarction and stroke but not for heart failure.
Journal of Clinical Epidemiology, 57(10), 1096-1103.
Peus, C., Braun, S., & Frey, D. (2013). Situation-based measurement of the full range of leadership model – Development and validation of a situational judgment test. The Leadership Quarterly, 24, 777-795.
Snook, S., Nohria, N., & Rakesh, K. (Eds.). (2012). The handbook for teaching leadership
(Chapter 16). Thousand Oaks, CA: Sage Publications.
Spector, P. (1994). Using self-report questionnaires in organizational behavior research: a comment on the use of a controversial method. Journal of Organizational Behavior, 15(5), 385-392.
Van der Stoop. (2011). Using item response theory to analyze properties of the leadership practices inventory. British Journal of Science, 1(1), 140-162. Retrieved from http://www.ajournal.co.uk/pdfs/volume1/Vol.1%20(1)%20Article%2011.pdf
Author Biographies
Nancy Carney is Associate Professor in the School of Medicine at Oregon Health & Science University. She earned her PhD in Systems Science in 1998. Her research focus has been application of a systems approach to understanding human mental, behavioral, and social processes in the context ofpathology.
Michael Jensen joined the faculty of the Harvard Business School in 1985, founding the Negotiations, Organizations and Markets Unit in the School. He is author of more than 100 scientific papers and many other publications in the popular media on a wide range of economic, finance and business-related topics.
Nicolas Ballarini graduated with a MS degree in biostatistics from The University of North Carolina at Chapel Hill in 2015 and is currently pursuing his PhD. His research interests include the design and analysis of clinical trials, with a focus on adaptive designs and Bayesian approaches.
Jeri Echeverria was Executive Vice Chancellor and Chief Academic Officer for the California State University System until her retirement, and is Professor and Provost Emerita, California State University, Fresno. She created a national movement to increase graduation rates and advocated for the quality of academic programs offered to the CSU.
Tracie Nettleton received her Master’s degree in biomedical informatics from Oregon Health & Science University. Her research and publication portfolio focus is on opioid and alcohol addiction. Tracie is currently the Data Systems Administrator for a Behavioral Health Center in Tillamook Oregon.
Molly Stillwell received her MS degree in Psychology from Saybrook University and is currently pursuing her PhD, specializing in transformative social change. Her research interest is integrative health psychology with a focus on prevention. Her graduate school field research is with the Salish Cancer Center serving tribal and non-nativepopulations.
Werner Erhard has been the creator of innovative models of individual, organizational, and social transformation. His work has been the source of new perspectives for practitioners in business, education, philosophy, medicine, psychotherapy, developing/emerging countries, conflict resolution, and community building.