Metacognition is a way of understanding how one learns (Braungart, Braungart, & Gramet, 2019), involving self-reflection and self-regulation that can result in modifications of previously held beliefs, changes in behavior, and new learning (Burke & Mancuso, 2012; Chiejina & Ebenebe, 2013; Johnson, 2013; Kuiper, Murdock, & Grant, 2010). Metacognition has long been recognized as an essential component of nursing education (Worrell, 1990) and is positively associated with greater grade point averages in millennial nursing students (Robb, 2016), as well as improved cognitive skills in the clinical setting (Kuiper & Pesut, 2004). When students self-reflect on learning, competence is increased and students are able to make sense of their practice (Bulman, Lathlean, & Gobbi, 2012; Pai, Ko, Eng, & Yen, 2017). One tool that has been used to help students self-reflect on learning is the structured debriefing tool, the exam wrapper.
The exam wrapper was first described by Achacoso (2004) as a way for students to reflect on and understand their examination performance. An exam wrapper is a questionnaire that is typically given to students after an examination that asks them to first predict their performance and then self-reflect on how they studied in preparation for the examination. This first step enables the students to consider the correlation between their performance and effort applied. The second step involves asking students to consider learning strategies used while studying and to consider what they may do differently in preparation for the next examination. Achacoso (2004) reported that the use of exam wrappers increased metacognitive awareness in students, allowed the students to perform better on subsequent examinations, increased student motivation, and gave students the perception that the instructor was invested in the student learning.
The use of exam wrappers to enhance metacognition was described and successfully used in college and math science courses by Lovett (2013) and her colleagues who developed structured reflections after examination grades were disbursed, with the purpose of fostering “metacognitive development” (p. 18). Gezer-Templeton, Mayhew, Korte, and Schmidt (2017) found a relationship between the use of exam wrappers and improved examination performance in an introductory food science and human nutrition course. In nursing, the use of exam wrappers has been described as a useful tool to help students study and prepare for examinations (Poorman & Mastorovich, 2016) and facilitate student retention and success (Tinnon, 2018). A recent qualitative study by Butzlaff, Gaylle, and O'Leary Kelley (2018) demonstrated the value of exam wrappers in helping students become more active participants in their learning and in recognizing effective study habits, including recognizing the value of understanding instead of just memorizing material.
The value of exam wrappers has been demonstrated; however, most studies examining the use of this tool in nursing education involved students in the third year of their program. Of concern is that nursing students need to develop examination self-reflection skills sooner rather than later. Moreover, no studies have examined whether there is a difference in metacognitive scores before and after the use of exam wrappers over time. The purpose of this study was to examine the impact of the exam wrapper on student metacognition in a sophomore-level fundamentals nursing class and to assess nursing students' perception of the utility of this tool. The research questions guiding this study were:
- Does the use of an exam wrapper enhance student meta-cognition over time?
- What are student's perceptions of the utility of exam wrappers?
Study Design and Participants
Following institutional review board approval, a pilot study using a mixed-methods approach was conducted. For the quantitative component of the study, a longitudinal study design was implemented using the Metacognitive Inventory for Nursing Students (MINS) scale to assess student metacognition. The MINS questionnaire uses a 5-point Likert scale (1 = never to 5 = always) to measure the association between a learner's knowledge and awareness of their own thinking and behaviors. This 28-item tool developed by Hsu (2010) has five subscales, self-monitoring (seven items), self-modification (seven items), self-awareness (six items), effective learning (three items), and problem solving (five items). Total scores range from 28 to 140, with higher scores indicating greater metacognitive ability. The tool has a reported Cronbach's alpha coefficient of .94 for the total scale and .73 to .90 for the five domains (Hsu, 2010).
This scale was administered to consenting students in a baccalaureate fundamentals of nursing course at the beginning of the course (baseline) and after each one of three examinations. Following each of the three examinations, students were also asked to complete the exam wrapper questionnaire. For the qualitative component of the study, students who had completed at least one exam wrapper were invited to participate in a focus group at the end of the semester to assess their perceptions of using exam wrappers.
A convenience sample of 98 nursing students enrolled in a beginner fundamentals of nursing course were invited to participate in the study (Figure 1). A power calculation demonstrated that the study would need a minimum of 41 participants for findings to be reported at a power of .80, an alpha level of .05, and a medium effect size (f = 0.25). Both the exam wrapper and MINS questionnaire were administered in the next class following each examination by the principle investigator (PI) who was not the instructor for the course. The focus groups were led by the same PI at the end of the semester.
Study design. Note: MINS = Metacognitive Inventory for Nursing Students.
The exam wrapper, consisting of self-reflection questions that were adapted from the work of Lovett (2013), was used as an intervention in this study. Students were asked to reflect on time spent and strategies used to study for the examination (e.g., group study, reviewing notes, reading the text, and/or focusing on PowerPoint® presentation notes). Students were also asked to reflect on errors made on the examination and to consider what they might do differently in preparation for the next examination. The goal was for students to consider their examination performance in light of their study efforts and as a way of correlating performance with effort applied.
Students were invited to participate in this study through an announcement by the PI at the start of the semester. Consenting students completed a demographic questionnaire and the MINS questionnaire at the beginning of the semester. After each of the three examinations, consenting students were provided an opportunity to complete the MINS questionnaire and exam wrapper at the end of the next class following the examination. To determine student perceptions of the exam wrapper, students who completed the exam wrapper at least one time were invited to participate in a focus group at the end of the semester. A total of four 45-minute focus groups were held on campus and were moderated by the PI, who guided the discussion with open-ended questions with the goal of bringing out the student's perceptions of the utility and value of exam wrappers in enhancing performance. Focus groups were limited to no more than four students to ensure all individuals had an opportunity to express their thoughts (Krueger & Casey, 2015). Each focus group was audiotaped and subsequently transcribed.
Data Analysis and Interpretation
Quantitative data were analyzed using SPSS version 24. Data from each time point were analyzed using descriptive statistics. The Friedman test, a nonparametric test for testing the differences between several related samples, was used to determine the extent to which metacognition was enhanced over time. The response variable (dependent variable) was set as the total score of metacognition; the independent variable was time.
For the qualitative data, a systematic content analysis of focus group data was completed using Krueger's and Casey's (2000) framework. The first step began with summarizing the discussion at the end of the focus group and asking the participants for verification. The next steps included listening to the audio recording, reviewing field notes, reading transcripts, preparing a report in a question-by-question format, and finally examining the transcripts for emerging themes by each question and then for overall themes. Words, context, internal consistency, frequency, intensity, and specificity of responses were also considered (Krueger, 2002). Transcribed focus group responses were carefully read and analyzed for significant statements related to utility and perceived value of the exam wrapper, as well as any emerging themes. Segments were grouped based on commonalities within the descriptions provided by the nursing students. Lincoln's and Guba's (1985) framework was used to establish qualitative scientific rigor through the criteria of credibility, dependability, conformability, and transformability. Credibility was established through triangulation, debriefing, and member checking with participants at the end of each focus group. Triangulation was achieved by comparing data across focus groups and comparing the data to the PI's observations and perceptions documented in the reflective field notes. Additionally, investigator triangulation was accomplished by having the focus group transcripts read by both the PIs, who read the responses and reviewed codes and themes until a consensus was reached. Dependability and conformability were achieved by the use of an audit trail of all methodological and analytical decisions, as well as maintenance of a reflective journal to describe rationale of all methodological decisions. Transformability was established through the use of detailed and thick descriptions of the data.
A total of 92 students agreed to participate at the beginning of the study. However, the number of students completing both the MINS and exam wrapper questionnaire declined after each examination: baseline (n = 92), after the first examination (n = 67), after the second examination (n = 51), and third examination (n = 31). Only 15 students completed the MINS questionnaire at all four time points during the course of the semester, and 31 students completed the exam wrapper throughout the semester. For example, it was noted some students completed the MINS after one or two examinations, but not after all three. Demographically, a majority of the participants were female (92%), were employed at least part time (67%), and lived in on-campus dormitories (65%), with a mean age of 19.5 years. Mean metacognition scores were 84.6 at baseline, 80.9 after the first examination, 81.6 after the second examination, and 86.6 after the third examination. Although mean metacognition scores declined between baseline and the first examination, they continued to increase over time. The Friedman nonparametric test revealed that overall student metacognitive skills were enhanced over time (p = .014).
Twelve students who had completed at least one exam wrapper participated in one of four 45-minute focus groups. A systematic analysis of focus group transcripts revealed that although a few students reported the exam wrapper tool was somewhat helpful, a majority (n = 9) of the students shared they did not find exam wrapper helpful. Three themes emerged as reasons for not finding exam wrapper helpful: A Reliance on the Professor, An Overlap With Already Established Strategies, and Difficulty Answering Questions From Memory.
A Reliance on the Professor “To Be My Exam Wrapper”
Nearly all the students described a strong reliance on their professor for understanding why points were lost and for helping students develop strategies for improving performance on subsequent examinations. Students reported regularly seeking feedback from their professor, who would then make suggestions on how they could do better the next time. These recommendations included practicing NCLEX®-style questions, reading before class, and taking better notes. Student comments included:
I gotta say it was more the professor who made me think about why I got something wrong. She made me sit back and try to figure out why I got a question wrong—Was it because I didn't know the information, was it because I misread the question, was it because there was a weird answer? There's a lot of reasons to do bad on an exam. It's kinda like I use my professor to be my exam wrapper.
Overlap With Already Established Self-Regulated Learning Strategies
The second theme to emerge from the focus groups was that students reported that they already used a number of self-regulated strategies to improve performance on examinations. In addition to meeting with their professor, students reported making lists of concepts they believed they got wrong and making specific plans for how they would study differently the next time. An common strategy used by the students was self-reflecting on how much they studied from the class presentations versus studying from the textbooks. One student noted:
I go back to the chapters after the exams and then I'm like “oh man, there it is…right there!” and I sometimes get mad at myself cause I missed it so now.… I gotta make the time to read the book too and not just rely on the PowerPoints.
Difficulty in Answering the Questions From Memory
Seven of the students in the focus groups stated they did not think the exam wrapper was helpful because they had to reflect on the queries in the exam wrapper from memory. Students lamented the policy of the college that examination reviews were done without the benefit of having a copy of their examination in front of them. One student pointed out:
Without having the exam right in front of me, I didn't know how many points I lost to not knowing the question versus not understanding the question.
Our quantitative findings validate the use of exam wrappers in enhancing student metacognition in a fundamentals nursing course if used repeatedly over time. Metacognition includes understanding one's ability, as well as what might need to be done to better one's abilities. It involves planning, regulating, and revising strategies (Worrell, 1990), all of which take time. Previous studies (e.g., Butzlaff et al., 2018) spoke to the value of exam wrappers in enhancing student examination preparation and performance and its positive relationship with metacognition. An element of this may be related to insight and self-awareness. Prior to the first examination, students appeared to come into the class with a high level of confidence and self-perceived insight into their abilities to learn. After the first examination, students may have perceived their performance was based on chance or a difficult examination rather than having the insight to recognize their full role in examination performance, hence the lower metacognition scores. Students who repeatedly used the exam wrapper had an increase in their metacognition scores. Over time, these students become increasingly aware of their strengths and weaknesses as learners and test takers.
Interestingly, although the quantitative results support the use of exam wrappers in enhancing metacognition, the qualitative findings reveal that students had a decidedly different perception of its utility. Qualitative findings indicate a reliance on their professor to “be the exam wrapper.” It is worth noting that all of the students who participated in this study were millennial learners. Millennials are reported to have grown up with parents who had an active role in their lives by facilitating progression and success with structure and positive reinforcements (Odenweller, Booth-Butterfield, & Weber, 2014). Findings from this study suggest that students are looking for this same type of support and structure from faculty. This reliance on faculty may be compounded in part because of the increased use of high-stakes testing for promotion or progression as a method to increase first-time NCLEX® pass rates. Nursing students often times feel pressured to do well on these types of examinations (Hunsicker & Chitwood, 2018), resulting in a performance approach to learning (Dunn, 2014). Of concern is that this type of surface learning does not lead to retained knowledge, which may affect the student's ability to critically think and apply what they learned in the practice setting.
Another interesting finding was that students already engaged in some degree of self-regulated learning, although there remained a significant reliance on lecture and PowerPoint material delivered in the classroom from faculty. Although a number of innovative teaching strategies, such as flipping the classroom, have been shown to engage student's critical thinking abilities (Dehghanzadeh & Jafaraghaee, 2018), students continue to prefer traditional learning modalities, such as lecture (Hills, Levett-Jones, Lapkin, & Warren-Forward, 2017), as these require less active engagement (Post, Deal, & Hermanns, 2015). Faculty have an opportunity to work with students to reinforce the value that classroom engagement combined with self-regulated learning may have on knowledge retention. Students in this study who used the exam wrapper consistently had improved metacognition over time. Suggestions for future studies may include examining whether the improved meta-cognition scores are sustained over time and facilitate a more active engagement and ownership of the learning by the student. When students self-reflect on their strengths and weakness and develop a plan for improvement, independence is fostered and they begin to have responsibility for their own learning. When students own their learning, they feel empowered, develop iterative thinking, and become problem solvers and thinkers (Cook-Sather & Luz, 2015). Nurse educators can work with the students to foster a sense of self-accountability for their learning through the use of a self-reflection debriefing tool such as an exam wrapper.
Limitations of this study include that the study was conducted without a control group and in only one baccalaureate nursing program. An additional limitation is the high attrition rate from the beginning of the study to the end, resulting in a small-end sample size with a lower effect. Finally, we do not know whether the change in metacognition scores was correlated to subsequent examination performance or if it was sustained over time.
The results of this study indicate that although students may not perceive the exam wrapper as useful, those who repeatedly used the tool over the course of the semester had increased metacognition scores. Even though students continue to rely on the nurse educator for feedback and guidance to improve performance, the exam wrapper can be a valuable tool for enhancing metacognition for nursing students early in their nursing education. Nurse educators should be encouraged to use this tool early in the curriculum to promote its adaptation by nursing students. With early exposure and repeated use, students may deem this tool to be an essential component of their learning.
- Achacoso, M.V. (2004) Post-test analysis: A tool for developing students' metacognitive awareness and self-regulation. New Directions for Teaching and Learning, 100, 115–119. doi:10.1002/tl.179 [CrossRef]
- Braungart, M.M., Braungart, R.G. & Gramet, P.R. (2019). Applying learning theories to healthcare practice. In Bastable, S.B. (Ed.), Nurse as educator: Principles of teaching and learning for nursing practice (5th ed., pp. 69–91). Burlington, MA: Jones & Bartlett Learning.
- Bulman, C., Lathlean, J. & Gobbi, M. (2012). The concept of reflection in nursing: Qualitative findings on student and teacher perspectives. Nurse Education Today, 32(5), E8–E13. doi:10.1016/j.nedt.2011.10.007 [CrossRef]
- Burke, H. & Mancuso, L. (2012). Social cognitive theory, metacognition, and simulation learning in nursing education. Journal of Nursing Education, 51, 543–548. doi:10.3928/01484834-20120820-02 [CrossRef]
- Butzlaff, A., Gaylle, D. & O'Leary Kelley, C. (2018). Student self-evaluation after nursing examinations: That's a wrap. Nurse Educator, 43, 187–190. doi:10.1097/NNE.0000000000000534 [CrossRef]
- Chiejina, E.N. & Ebenebe, R.C. (2013). Metacognition: Usage by nursing students in the university. International Journal of Pure and Applied Sciences and Technology, 17(2), 1–8.
- Cook-Sather, A. & Luz, A. (2015). Greater engagement in and responsibility for learning: What happens when students cross the threshold of student–faculty partnership. Higher Education Research & Development, 34, 1097–1109. doi:10.1080/07294360.2014.911263 [CrossRef]
- Dehghanzadeh, S. & Jafaraghaee, F. (2018). Comparing the effects of traditional lecture and flipped classroom on nursing students' critical thinking disposition: A quasi-experimental study. Nurse Education Today, 71, 151–156. doi:10.1016/j.nedt.2018.09.027 [CrossRef]
- Dunn, K.E. (2014). Insight into error hiding: Exploration of nursing students' achievement goal orientations. Journal of Nursing Education, 53, 93–96.
- Gezer-Templeton, P.G., Mayhew, E.J., Korte, D.S. & Schmidt, S.J. (2017). Use of exam wrappers to enhance students' metacognitive skills in a large introductory food science and human nutrition class. Journal of Food Science Education, 16, 28–36. doi:10.1111/1541-4329.12103 [CrossRef]
- Hills, C., Levett-Jones, T., Lapkin, S. & Warren-Forward, H. (2017). Generation Y health professional students' preferred teaching and learning approaches: A systematic review. Open Journal of Occupational Therapy, 5, 12. doi:10.15453/2168-6408.1278 [CrossRef]
- Hsu, L.L. (2010). Metacognitive inventory for nursing students in Taiwan: Instrument development and testing. Journal of Advanced Nursing, 66, 2573–2581. doi:10.1111/j.1365-2648.2010.05427.x [CrossRef]
- Hunsicker, J. & Chitwood, T. (2018). High-stakes testing in nursing education: A review of the literature. Nurse Educator, 43, 183–186. doi:10.1097/NNE.0000000000000475 [CrossRef]
- Johnson, J.A. (2013). Reflective learning, reflective practice, and metacognition: The importance in nursing education. Journal for Nurses in Professional Development, 29, 46–48. doi:10.1097/NND.0b013e31827e2f27 [CrossRef]
- Krueger, R.A. (2002). Designing and conducting focus group interviews. Retrieved from https://www.eiu.edu/ihec/Krueger-FocusGroupInterviews.pdf
- Krueger, R.A. & Casey, M.A. (2000). Focus groups: A practical guide for applied research (3rd ed.). Thousand Oaks, CA: Sage.
- Krueger, R.A. & Casey, M.A. (2015). Focus groups: A practical guide for applied research (5th ed.). Thousand Oaks, CA: Sage.
- Kuiper, R.A. & Pesut, D.J. (2004). Promoting cognitive and metacognitive reflective reasoning skills in nursing practice: Self-regulated learning theory. Journal of Advanced Nursing, 45, 381–391. doi:10.1046/j.1365-2648.2003.02921.x [CrossRef]
- Kuiper, R.A., Murdock, N. & Grant, N., (2010). Thinking strategies of baccalaureate nursing students prompted by self-regulated learning strategies. Journal of Nursing Education, 49, 429–436. doi:10.3928/01484834-20100430-01 [CrossRef]
- Lincoln, Y.S. & Guba, E. (1985). Naturalistic inquiry. Thousand Oaks, CA: Sage.
- Lovett, M.C. (2013). Make exams worth more than grade: Using exam wrappers to promote metacognition. In Kaplan, M., Silver, N., Lavaque-Manty, D. & Meizlish, D. (Eds.), Using reflection and metacognition to improve student learning: Across the disciplines, across the academy (pp. 18–52). Sterling, VA: Stylus.
- Odenweller, K.G., Booth-Butterfield, M. & Weber, K. (2014). Investigating helicopter parenting, family environments, and relational outcomes in millennials. Communication Studies, 65, 407–425. doi:10.1080/10510974.2013.811434 [CrossRef]
- Pai, H.C., Ko, H.L., Eng, C.J. & Yen, W.J. (2017). The mediating effect of self-reflection and learning effectiveness on clinical nursing performance in nursing students: A follow-up study. Journal of Professional Nursing, 33, 287–292. doi:10.1016/j.profnurs.2017.01.003 [CrossRef]
- Poorman, S.G. & Mastorovich, M.L. (2016). Using metacognitive wrappers to help students enhance their prioritization and test-taking skills. Nurse Educator, 41, 282–285. doi:10.1097/NNE.0000000000000257 [CrossRef]
- Post, J.L., Deal, B. & Hermanns, M. (2015). Implementation of a flipped classroom: Nursing students' perspectives. Journal of Nursing Education and Practice, 5(6), 25–30. doi:10.5430/jnep.v5n6p25 [CrossRef]
- Robb, M.K. (2016). Self-regulated learning: Examining the baccalaureate millennial nursing student's approach. Nursing Education Perspectives, 37, 162–164.
- Tinnon, E.A. (2018). Reflective test review: The first step in student retention. Teaching and Learning in Nursing, 13, 31–34. doi:10.1016/j.teln.2017.09.001 [CrossRef]
- Worrell, P.J. (1990). Metacognition: Implications for instruction in nursing education. Journal of Nursing Education, 29, 170–175.