Journal of Nursing Education

Educational Innovations 

An Innovative Strategy in Evaluation: Using a Student Engagement Framework to Evaluate a Role-Based Simulation

Morgan Smith, MEd, RN; Jane Warland, PhD, RM; Colleen Smith, PhD, RN

Abstract

Online role-play has the potential to actively engage students in authentic learning experiences and help develop their clinical reasoning skills. However, evaluation of student learning for this kind of simulation focuses mainly on the content and outcome of learning, rather than on the process of learning through student engagement. This article reports on the use of a student engagement framework to evaluate an online role-play offered as part of a course in Bachelor of Nursing and Bachelor of Midwifery programs. Instruments that measure student engagement to date have targeted large numbers of students at program and institutional levels, rather than at the level of a specific learning activity. Although the framework produced some useful findings for evaluation purposes, further refinement of the questions is required to be certain that deep learning results from the engagement that occurs with course-level learning initiatives.

Abstract

Online role-play has the potential to actively engage students in authentic learning experiences and help develop their clinical reasoning skills. However, evaluation of student learning for this kind of simulation focuses mainly on the content and outcome of learning, rather than on the process of learning through student engagement. This article reports on the use of a student engagement framework to evaluate an online role-play offered as part of a course in Bachelor of Nursing and Bachelor of Midwifery programs. Instruments that measure student engagement to date have targeted large numbers of students at program and institutional levels, rather than at the level of a specific learning activity. Although the framework produced some useful findings for evaluation purposes, further refinement of the questions is required to be certain that deep learning results from the engagement that occurs with course-level learning initiatives.

Ms. Smith is Lecturer in Nursing, Dr. Warland is Senior Lecturer in Midwifery, and Dr. Smith is Senior Lecturer in Nursing, University of South Australia, School of Nursing and Midwifery, Adelaide, South Australia.

The e-simulation was funded by the University of South Australia through a Division of Health Sciences Teaching and Learning Grant paid to the School of Nursing and Midwifery. Parts of this manuscript were presented at the HERDSA Conference, July 2010, Melbourne, Victoria, Australia, and at the annual Educational Research Group Adelaide conference, September 2009, Adelaide, South Australia.

The authors have disclosed no potential conflicts of interest, financial or otherwise.

Address correspondence to Jane Warland, PhD, RM, Senior Lecturer, University of South Australia, School of Nursing and Midwifery, PO Box 2471, Adelaide, South Australia, 5001; e-mail: jane.warland@unisa.edu.au.

Received: December 27, 2010
Accepted: November 23, 2011
Posted Online: January 27, 2012

This brief report describes an innovative approach to educational evaluation that has not been systematically tested at the level of the teaching activity. A project was evaluated using a student engagement framework to determine the effectiveness of an online role-play. Online role-play is a frequently used pedagogy in higher education (Alexander & Boud, 2001) and is thought to foster student engagement in a safe, yet challenging learning environment (Bell, 2001). Student engagement is about what students do to learn (Kuh, 2001). It assumes that specific activities and conditions are linked to effective learning (Coates, 2007). Educational initiatives that foster student engagement aim for analysis, synthesis, and organization of information and experiences; require judgments about the value of information and experiences; and encourage the application of theory to practical situations (Kuh, 2009). All of these thinking skills are required by undergraduate students of nursing and midwifery at the completion of their undergraduate degree. Student engagement focuses on those activities that educational evidence suggests leads to quality learning, rather than the learning outcomes themselves. This is an important distinction.

The evaluation tool used to determine levels of engagement of students in the learning activity was a modified version of the National Survey of Student Engagement (NSSE) (2003) in the United States and the Australasian Survey of Student Engagement (AUSSE) (Australian Council for Educational Research, 2011) in Australia. These instruments have been rigorously tested for reliability and validity (Kuh, 2001) and have been widely used in the United States and increasingly in Australia. The relevance of the NSSE benchmarks has been established for online learning (Robinson & Hullinger, 2008). However, to date, these tools have been used primarily with large cohorts of students, with analysis focused at the institutional level. The current study took this a step further by using a student engagement framework based on the AUSSE to evaluate a specific online learning initiative located within a course. (For clarity, it should be noted that a course may sometimes be referred to as a topic, unit, or subject.)

The online role-play was developed to assist students within Bachelor of Nursing and Midwifery programs to understand public health principles and communication within a nursing–midwifery team where members of the team have diverse views and priorities. Key aspects of any role-play activity are its application to real-world issues, an understanding of a range of stakeholder positions, and an understanding of the complex and contested situations that can be resolved in practice (Brierley, Hillman, & Devonshire, 2002; Brierley, Hillman, Devonshire, & Funnell, 2002). To engage effectively, students were required to use an online asynchronous discussion to question stakeholder (i.e., nurse) positions, synthesize information, problem solve, and negotiate outcomes.

The online role-play was scripted around a team meeting. The goal of the online role-play was the production of a collaborative action plan to manage an influenza outbreak. To do this, students’ undertook research into the various aspects of an influenza outbreak and the associated implications for their nursing/midwifery role prior to each post. A collaborative document that included the contribution of all roles to the management of the influenza outbreak was collated and posted toward the end of the role-play.

The online role-play consisted of six nursing–midwifery roles, played by students, and one role as director from the department of health, played by a faculty member who convened, chaired the meeting, and led the discussion. Students were allocated to a nursing or midwifery role and were provided with a private and public role persona. The public personas were available to all students and provided information about the role, including age, occupation, workplace, and specific views and interest. The private persona was restricted to the group of students playing each role and provided additional information, including their views about influenza, current concerns, allegiances, and personality traits. Some elements within the private personas were designed to create tensions and differences of opinion among the roles.

Discussion of Evaluation Structure and Findings

Students were invited to participate in the evaluation of the online role-play 2 weeks after its completion and 3 days after the submission of an assessment item related to the online role-play. Students were e-mailed and invited to participate via a link to an online survey. Ethics approval was gained from the university’s Human Research Ethics Committee to undertake the research, as the information sought was in addition to that required by the standard course evaluation routinely actioned each time a course is offered. Completion of the survey was considered as consent.

The NSSE and AUSSE are structured around five clusters of effective educational practice referred to as benchmarks. These clusters are levels of academic challenge, active and collaborative learning, faculty interaction, enriching educational experience, and supportive campus environment (NSSE, 2003). As the role-play was an online activity, supportive campus environment was not included in the evaluation survey. The outcome of the evaluation is reported below as it relates to the student engagement benchmarks, highlighting strengths and weaknesses of the approach as an evaluation framework for course-level learning initiatives.

The sample population included all first-year students (N = 421) enrolled in a public health course. The evaluation comprised 37 Likert questions rated on a five-point Likert scale ranging from 1 (strongly agree) to 5 (strongly disagree). Information was also collected on student enrollment characteristics. These data are presented as percentages. Two open-ended questions asked students about the strengths of the online role-play and how it could be improved. These questions provided descriptive qualitative data that were used to inform and clarify responses to the Likert scale questions.

One hundred forty-three students responded to the survey, for a 34% response rate. Of these 77 (54%) studied on campus and 64 (45%) studied by distance. Two (1%) of students did not designate a mode of study.

Levels of Academic Challenge

Levels of academic challenge relates to the amount of time and effort students put into their studies. This includes reading and writing, as well as mental activities such as analysis, synthesis, judgment, and application (Robinson & Hullinger, 2008), all of which were structured into the role-play.

To evaluate the levels of academic challenge posed by the online role-play, students were asked to respond to statements about the number of hours spent gathering information in preparation for making a post, posting, and reading other students’ posts. During the 2-week interaction phase of the online role-play, 5% of students spent less than 2 hours on role-play–related activities, 65% of students spent between 2 and 8 hours involved in related activities, and 30% of students spent more than 8 hours.

Finally, 49% of students indicated they had “worked harder than they thought they could” on the activity. Although this is encouraging, 25% indicated they had not (the remaining 26% indicated a neutral response).

To gain insights into the level of mental activities, students were questioned about how the role-play provided an opportunity to practice problem solving and apply theory to practice, as well as about their understanding of primary health care principles and their role as future registered nurses and midwives.

Students reported higher levels of mental engagement in understanding primary health care principles (61% strongly agreed or agreed) and their future role as registered nurse or midwife (70% strongly agreed or agreed) than with the more analytical activities of practicing problem solving (54% strongly agreed or agreed) and applying theory to practice (57% strongly agreed or agreed). This finding is not surprising, given that the analytical activities require much more time, input, and effort by students.

Qualitative comments indicated that some students were highly negative about the activity. Of those who thought there was nothing good about the activity, only one student provided a specific solution about how it could be improved. This suggestion was that students complete a face-to-face role-play in the classroom as an alternative activity.

In the future, to ensure that students understand the engagement expectations and more fully develop their analytical skills, it would be appropriate to extend the length of the role-play and indicate to students the expected time commitment before, during, and after the activity, as time on task has repeatedly been demonstrated to increase learning (Kuh, 2009). Involving students in the design of the role-play has been found to increase students’ level of engagement and analytical skills (Druckman & Ebner, 2008) and is worth considering.

Active and Collaborative Learning

Active and collaborative learning relates to the student effort expended in relating to other students for learning purposes (Robinson & Hullinger, 2008). In the online role-play, students were allocated to groups to play a role. This required up to five students who interacted to negotiate and prepare the postings made on behalf of their role. In the online role-play, this involved various possible behind-the-scenes communication approaches, including face to face, telephone, and e-mail. Although 74% of students strongly agreed or agreed that effective communication with other students was required to participate in the online role-play, only 36% strongly agreed or agreed that communicating with other students was easy. In addition, only 39% of students indicated that all students participated equally in the online role-play. This has implications for the extent that students can learn effectively from each other when working in teams in an online activity.

The online role-play was designed to assist students to enhance their communication skills within the online environment. Although the instrument measured student perceptions of communication and communication skills development, it did not specifically reveal how this interaction assisted students to learn. Rather, there is evidence of tension between students in relation to how much they participated and how easy the communication was perceived. If students learn most when they have supportive relationships with other students (Kuh, Kinzie, Schuh, & Whitt, 2005), then students may not have maximized their learning. This is a challenging area worthy of further investigation and indicates a weakness with the student engagement framework at the learning activity level. It was clear from the data that communication occurred, but how it enhanced learning is less clear. In addition, it is possible that considerable time was spent communicating but without commensurate learning.

Student–Faculty Interaction

Student–faculty feedback relates to the nature and frequency of contact students have with faculty (Robinson & Hullinger, 2008). Before, during, and after the online role-play, students had the opportunity to interact with a range of faculty. They interacted with the online role-play project team usually via an online information forum, an asynchronous discussion forum, or e-mail. In addition, a lecture about the online role-play activity was provided on campus to all students who wished to attend. Students also had contact with their allocated faculty member, who managed a defined group of 25 students. Each member of faculty managed the online role-play for their students, addressed questions and concerns, and, as previously mentioned, played the role of the convener in the online role-play.

Seventy-three percent of students strongly agreed or agreed that the instructions for accessing the online role-play were helpful, although only 53% indicated the lecture on the topic was helpful. Forty-eight percent indicated that what was required in the online role-play was clear, whereas 32% thought it was not. In retrospect, the quality and frequency of interactions with faculty was not well evaluated. Emphasis was placed on whether information had been conveyed clearly to students. Although this was important, additional two-way communication may have improved student understanding of the expectations. The online environment poses unique challenges related to establishing effective relationships between faculty and students, as well as students and other students. This was exacerbated by faculty playing a role in which they were necessarily not the person they are in everyday life. Additional evaluations were performed involving the number of posts various staff made in the role within the online role-play. This varied enormously across groups and is not reported herein. It is worth noting that like students, the commitment of staff to the activity also varied, which affected the final evaluation data.

Enriching Educational Experiences

Enriching educational experiences are those that assist students to develop the skills required to work effectively with people from diverse backgrounds. Technology may be used to facilitate these collaborations (Robinson & Hullinger, 2008)

Thirty-five percent of students indicated the online role-play had helped them further develop their information technology skills, and 47% said it had not. In addition, more than 70% of students indicated they had no difficulties accessing and navigating the role-play online environment. This suggests the online environment was user friendly, but it also indicates high levels of computer literacy among the students. Fifty percent of students indicated they preferred participating online, whereas 31% indicated they would have preferred a face-to-face environment.

The online role-play was structured to assist students to interact as nurses with others nurses who have diverse backgrounds and priorities, but in retrospect this aspect was not well evaluated. Similarly, working in groups with students from diverse backgrounds was not well evaluated. The evaluation revealed students’ competence with the online environment but not how effectively they worked with those who have backgrounds different from their own.

Conclusion

This evaluation framework used student engagement benchmarks to evaluate student engagement in a specific online role-play learning activity. More work is required for these benchmarks to be effective in evaluating course-level learning initiatives. To date, student engagement benchmarks have been used across large cohorts of students to understand engagement at the program and institutional levels. This has proven to have high validity and reliability. For an online learning activity, additional work is required on how these benchmarks will indicate what students have done and how this translates to what and how much they have learned. Students can be involved in a lively discussion with each other, but can it be assumed that engagement, and therefore learning, is being maximized? Possibly not. Further exploration of these areas may enhance a student engagement evaluation framework applicable to course-level learning activities.

Lessons Learned

The lessons learned from our experience of using a modified version of the AUSSE student engagement framework survey to assess student engagement with an online role-play is summarized as follows:

  • A student engagement framework is appropriate for evaluating course-level initiatives because it measures what students do to learn.
However:
  • Evaluations that use standard student engagement benchmarks tested at an institutional level cannot be readily transferred to course-level learning initiatives.
  • For a comprehensive evaluation of a course-level learning initiative, learning outcomes, student perspectives of the initiative, and student engagement benchmarks are required.
  • The student engagement benchmarks for course-level learning initiatives require further development and validity and reliability testing.
  • Additional questions were required that identified the extent to which students had the opportunity to work effectively with people who have backgrounds different from their own.

References

  • Alexander, S. & Boud, D. (2001). Learners still learn from experience when online. In Stephenson, J. (Ed.), Teaching and learning online: New pedagogies for new technologies. Abingdon, Oxon, UK: Routledge Falmer.
  • Australian Council for Educational Research. (2011). Australasian Survey of student engagement. Retrieved from http://www.acer.edu.au/research/ausse
  • Bell, M. (2001). Online role play: Anonymity engagement and risk. Educational Media International, 38, 251–260. doi:10.1080/09523980110105141 [CrossRef]
  • Brierley, G., Hillman, M. & Devonshire, E. (2002). Learning to participate: Responding to changes in Australian land and water management policy and practice. Australian Journal of Environmental Education, 18, 7–13.
  • Brierley, G., Hillman, M., Devonshire, E. & Funnell, L. (2002). Description of round table exercise: Environmental decision-making about water resources in physical geography. Retrieved from the Learning Designs Web site: http://www.learningdesigns.uow.edu.au/exemplars/info/LD26/index.html
  • Coates, H. (2007). A model of online and general campus-based student engagement. Assessment and Evaluation in Higher Education, 32, 121–141. doi:10.1080/02602930600801878 [CrossRef]
  • Druckman, D. & Ebner, N. (2008). Onstage or behind the scenes? Relative learning benefits of simulation role play and design. Simulation and Gaming, 39, 465–496. doi:10.1177/1046878107311377 [CrossRef]
  • Kuh, G.D. (2001). The National Survey of Student Engagement: Conceptual framework and overview of psychometric properties. Bloomington, IN: Indiana University Centre for Postsecondary Research and Planning.
  • Kuh, G.D. (2009). The national survey of student engagement: Conceptual and empirical foundations. New Directions for Institutional Research, 141, 5–20. doi:10.1002/ir.283 [CrossRef]
  • Kuh, G.D., Kinzie, J., Schuh, J.H. & Whitt, E.J. (2005). Student success in college. Creating conditions that matter. San Francisco, CA: Jossey-Bass.
  • National Survey of Student Engagement. (2003). Converting data into action: Expanding the boundaries of institutional improvement. National Survey of Student Engagement 2003 annual report. Bloomington, IN: Indiana University.
  • Robinson, C.C. & Hullinger, H. (2008). New benchmarks in higher education: Student engagement in online learning. Journal of Education for Business, 84, 101–109. doi:10.3200/JOEB.84.2.101-109 [CrossRef]
Authors

Ms. Smith is Lecturer in Nursing, Dr. Warland is Senior Lecturer in Midwifery, and Dr. Smith is Senior Lecturer in Nursing, University of South Australia, School of Nursing and Midwifery, Adelaide, South Australia.

The e-simulation was funded by the University of South Australia through a Division of Health Sciences Teaching and Learning Grant paid to the School of Nursing and Midwifery. Parts of this manuscript were presented at the HERDSA Conference, July 2010, Melbourne, Victoria, Australia, and at the annual Educational Research Group Adelaide conference, September 2009, Adelaide, South Australia.

The authors have disclosed no potential conflicts of interest, financial or otherwise.

Address correspondence to Jane Warland, PhD, RM, Senior Lecturer, University of South Australia, School of Nursing and Midwifery, PO Box 2471, Adelaide, South Australia, 5001; e-mail: jane.warland@unisa.edu.au

10.3928/01484834-20120127-03

Sign up to receive

Journal E-contents