Journal of Nursing Education

Quality Improvement Brief 

Cumulative Quizzes in a Nursing Research Course for Nursing Doctoral Students

Elisa R. Torres, PhD, RN

Abstract

Background:

The purpose of this quality improvement project was to examine the association between regular cumulative quizzes and knowledge retention in nursing doctorate students.

Method:

Sixteen students in a large midwestern university enrolled in a hybrid nursing research course were assessed with weekly online cumulative quizzes.

Results:

The mean score of the cumulative weekly quizzes was 94% (SD = 3%). The mean score on the cumulative final was 97% (SD = 3%). Fifteen students who completed an average of 12 of 14 (86%) cumulative weekly quizzes as scheduled passed the course.

Conclusion:

The cumulative final scores were higher than the cumulative weekly quizzes. Completing more of the cumulative weekly quizzes as scheduled was associated with higher cumulative final scores and passing the course, suggesting improved knowledge retention and ability to use that knowledge for other assignments throughout the course. [J Nurs Educ. 2019;58(4):243–246.]

Abstract

Background:

The purpose of this quality improvement project was to examine the association between regular cumulative quizzes and knowledge retention in nursing doctorate students.

Method:

Sixteen students in a large midwestern university enrolled in a hybrid nursing research course were assessed with weekly online cumulative quizzes.

Results:

The mean score of the cumulative weekly quizzes was 94% (SD = 3%). The mean score on the cumulative final was 97% (SD = 3%). Fifteen students who completed an average of 12 of 14 (86%) cumulative weekly quizzes as scheduled passed the course.

Conclusion:

The cumulative final scores were higher than the cumulative weekly quizzes. Completing more of the cumulative weekly quizzes as scheduled was associated with higher cumulative final scores and passing the course, suggesting improved knowledge retention and ability to use that knowledge for other assignments throughout the course. [J Nurs Educ. 2019;58(4):243–246.]

Cumulative learning is when an assignment requires students to draw on information they learned earlier in the semester (Lang, 2016). For example, a cumulative final examination covers material from the entire semester. Students who are evaluated with a cumulative final examination retain more information than students who are evaluated with a noncumulative final examination (Khanna, Badura Brack, & Finken, 2013), such as an examination that covers material from only the latter part of the semester. Students who have assessments throughout the semester retain more information for a cumulative final examination than students who do not have assessments throughout the semester and are solely assessed with a single cumulative final examination (Domenech, Blazquez, de la Poza, & Muñoz-Miquel, 2015). This may be because cumulative assessments throughout the semester encourage students to distribute their learning activities throughout the course, instead of postponing preparation until the final examination is imminent (Kerdijk, Cohen-Schotanus, Mulder, Muntinghe, & Tio, 2015). In addition, students who have cumulative assessments throughout the semester, such as weekly quizzes that cover all content in the course up to that point, retain more information than students who have noncumulative assessments throughout the semester (Lawrence et al., 2012), such as weekly quizzes that cover content from only that week. This may be because noncumulative assessments throughout the semester do not adequately prepare students for a cumulative final examination. Thus, current research on cumulative assessments suggests that students retain more information by completing cumulative assessments throughout the semester in preparation for a cumulative final.

An important distinction should be made between cumulative assessments throughout the semester, which are a type of formative assessment, and a cumulative final examination, which is a type of summative assessment. Weekly cumulative assessments are formative assessments that occur during the learning process and offer immediate feedback to both students and instructors. These weekly cumulative assessments can include frequent quizzes and opportunities for recalling or retrieving information from memory where the stakes are low, allowing students a safe opportunity to test their current understanding (Dirks, Wenderoth, & Withers, 2014). A cumulative final is a summative assessment that usually occurs at the end of the semester to evaluate what students have learned (Dirks et al., 2014).

Cumulative assessments throughout the semester have been examined in undergraduate students from various disciplines, including psychology (Khanna et al., 2013; Landrum, 2007; Lawrence, 2012), mathematics (Beagley & Capaldi, 2016), business (Domenech et al., 2015), and medicine (Kerdijk et al., 2015). However, the effects of cumulative assessments throughout the semester have not been examined in the nursing discipline. With the exception of medical students (Kerdijk et al., 2015), few studies have examined the effects of cumulative assessments throughout the semester on graduate students. Finally, the effects of cumulative assessments throughout the semester have been examined in face-to-face courses (Domenech et al., 2015; Kerdijk et al., 2015; Landrum, 2007) or courses that appear to be face-to-face (Beagley & Capaldi, 2016; Khanna et al., 2013; Lawrence, 2012), but not hybrid courses.

In the current quality improvement project, cumulative assessments were assigned throughout the semester in a hybrid doctoral-level nursing research course. This quality improvement project aimed to answer two questions:

  • Is completing cumulative weekly quizzes associated with improved knowledge retention, as indicated by higher scores on the final cumulative quiz, which served as a final examination? The a priori hypothesis was that the average scores on the final examination would be higher than the average scores on the cumulative weekly quizzes.
  • Is completing cumulative weekly quizzes as scheduled associated with improved knowledge retention, as indicated by higher scores on the final cumulative quiz, which served as a final examination? The a priori hypothesis was that the average scores on the final examination would be higher in those completing more of the scheduled cumulative weekly quizzes than those completing fewer of the scheduled cumulative weekly quizzes.

Method

Sixteen participants included students admitted into the Doctor of Nursing Practice and PhD programs in a large university in the Midwest who were enrolled in Nursing Research, a 16-week hybrid course in which students met face-to-face for 3 hours once per month (a total of four face-to-face sessions) with the rest of the course online. This course was required in the Doctor of Nursing Practice program and fulfills the requirements outlined in the American Association of Colleges of Nursing (2006) Essentials of Doctoral Education for Advanced Nursing Practice: Clinical Scholarship and Analytical Methods for Evidence-Based Practice. The course examined quantitative and qualitative research methods; the interrelationships among theory, research, and practice; the research process; critical evaluation of research findings; and applying ethical criteria for the protection of human subjects in research. The prerequisites were for participants to be of graduate standing or the consent of the instructor. The course was taught by a PhD-prepared nurse with an active program of research. Prior to the start of the course, the university's educational institutional review board reviewed and exempted this project, as it was considered a quality improvement project. Students were informed at the beginning of the semester that the weekly cumulative quizzes were part of a quality improvement project and could speak with the professor of the course or the local institutional review board if they had concerns.

The quizzes were administered online outside of class at a time and location convenient for students. Students were allowed to use their notes and textbook during the quizzes. However, students were instructed to take only their own quiz and to not share answers with their peers prior to their peers taking the quizzes, and to not give or receive help from another student while taking the quiz. Although the quizzes had no time limit, once a quiz was started, it had to be completed during that session and could be attempted only once. Each quiz was made available at the beginning of the week and was due by the end of the week. If a quiz was not taken by the due date, the quiz was marked late but could still be taken at any time during the semester prior to the last day in the course with no reduction in points, as suggested by Landrum (2007). The first week was a practice quiz, which was not worth any points. The final cumulative quiz—which served as the final examination— was entirely cumulative, meaning there were questions from every chapter. All of the quizzes combined—including the final examination—were worth 10% of the final course grade.

The content for the quizzes came from one to two assigned chapters per week. Each quiz had approximately one question from every chapter, with the length of the quizzes increasing each week from two questions on the first quiz to 25 questions on the final quiz at the end of the semester. None of the questions were repeated in a subsequent quiz. All questions came from the test bank of the required textbook for the course, which were used over the previous 3 years in noncumulative weekly quizzes with more than 100 graduate nursing students in the same course at the same institution. All questions were multiple choice with a stem, four options, and one correct answer. The questions were mostly at the remembering level of Anderson's revision of Bloom's The Taxonomy of Educational Objectives: The Classification of Educational Goals: Handbook I (Bloom, Engelhart, Furst, Hill, & Krathwohl, 1956), with an occasional question at the understanding level (Krathwohl, 2002). Students completed other assignments in the course that were at the applying, analyzing, evaluating, and creating levels of Anderson's “A Revision of Bloom's Taxonomy: An Overview” (Krathwohl, 2002). Descriptive statistics were conducted to analyze the results.

Results

The sample for this project comprised 16 students enrolled in the Nursing Research course offered annually in the fall. Fourteen of the students were in the Doctor of Nursing Practice program and two were in the PhD program. All students completed the course. Table 1 describes the results of the cumulative weekly quizzes.

Description of Cumulative Weekly Quizzes

Table 1:

Description of Cumulative Weekly Quizzes

The first question this project aimed to answer was whether completing cumulative weekly quizzes was associated with improved knowledge retention, as indicated by higher scores on the final cumulative quiz, which served as the final examination. The mean score on the cumulative weekly quizzes was 94% (SD = 3%); the lowest was 86% in week 11, whereas the highest was 100% in week two, with a range of 14% and mode of 100% across the semester. The mean score on the final examination was 97% (SD = 3%); the lowest score was 93%, whereas the highest score was 100%, with a range of 7% and modes of 96% and 100% on the final examination.

The second question this project aimed to answer was whether completing cumulative weekly quizzes as scheduled was associated with improved knowledge retention, as indicated by higher scores on the final cumulative quiz, which served as the final examination. The five students who earned 100% on the final examination completed an average of 13 of 14 (93%) cumulative weekly quizzes as scheduled. The seven students who earned 96% on the final examination completed an average of 11 of 14 (79%) cumulative weekly quizzes as scheduled. The four students who earned 93% on the final examination completed an average of 9 of 14 (64%) cumulative weekly quizzes as scheduled.

Discussion

This quality improvement project aimed to answer two questions. The first question was whether completing cumulative weekly quizzes was associated with improved knowledge retention, as indicated by higher scores on the final cumulative quiz, which served as the final examination. Results showed that the average scores on the final examination were higher than the average scores of the cumulative weekly quizzes. Scores improved from an average of 94% across the cumulative weekly quizzes to 97% on the final examination. This coincides with previous literature that found that students who are assessed with cumulative finals retain more information (Khanna et al., 2013), perhaps by encouraging students to distribute their learning activities throughout the course (Kerdijk et al., 2015). The current results also coincide with studies that found students who complete cumulative assessments throughout the semester have improved knowledge retention, as indicated by higher scores on a cumulative final, than students who have noncumulative assessments throughout the semester (Lawrence et al., 2012).

The second question this quality improvement project aimed to answer was whether completing cumulative weekly quizzes as scheduled was associated with improved knowledge retention, as indicated by higher scores on the final cumulative quiz, which served as the final examination. Results showed that completing more of the cumulative weekly quizzes as scheduled was associated with improved knowledge retention, as indicated by higher average scores on the final examination than completing fewer cumulative weekly quizzes as scheduled. Students who earned 100% on the final examination completed the most quizzes on schedule (93%), whereas students who earned the lowest score on the final examination completed the fewest quizzes on schedule (64%).

Cumulative quizzes were added to the course as a method to address the issue that students struggled with other components of the course, especially toward the end of the semester because they forgot what they learned earlier in the semester. Post hoc analyses were conducted to determine whether completing more of the cumulative weekly quizzes as scheduled was associated with the use of higher cognitive functioning on other assignments in the course that required applying, analyzing, evaluating, and creating (Krathwohl, 2002). Post hoc analysis found completing more of the cumulative weekly quizzes as scheduled was further associated with more knowledge retention, as indicated by a higher average grade in the course than completing fewer cumulative weekly quizzes as scheduled.

Grades in the course seemed to reflect how and when the cumulative weekly tests were taken. The average course grade was 95% (SD = 5%); the lowest grade was 79.2%, whereas the highest grade was 98.2%, with a range of 19% and no mode. All of the students completed all of the cumulative weekly quizzes. The 12 students who demonstrated the highest knowledge retention as indicated by an “A” in the course completed an average of 12 of 14 cumulative weekly quizzes as scheduled. The three students who demonstrated the next highest knowledge retention as indicated by an “AB” in the course completed an average of 13 of 14 cumulative weekly quizzes as scheduled. The one student who demonstrated the lowest knowledge retention as indicated by an earned “C” in the course, which was a failing grade, completed only one of 14 cumulative weekly quizzes as scheduled, even though no points were deducted for late completions. The cause that prevented the student from completing the quizzes as scheduled could also be the cause of the student failing the course. For example, time management may have been an issue. The quizzes probably indicate short-term surface learning, as the quizzes were untimed, open book, and mostly at the remembering level of Anderson's (Krathwohl, 2002) taxonomy. Given that the student was having difficulty meeting the timely requirements for assignments at the remembering level of Anderson's taxonomy that was worth 10% of the grade in the course, it follows that the same student may have trouble meeting the timely requirements of assignments at higher levels of Anderson's (Krathwohl, 2002) taxonomy that were worth 90% of the grade in the course.

Limitations to findings include the small number of nursing doctoral students enrolled in one course, in one semester, at one institution with no control group. Due to the existence of literature showing the advantage of cumulative assessments throughout the semester over a single final examination (Domenech et al., 2015) or noncumulative examinations throughout the semester (Beagley & Capaldi, 2016; Lawrence et al., 2012), it is questionably ethical to have a control group. Finally, this project focused on multiple choice questions. Some articles on cumulative assessments were composed of multiple choice questions (Kerdijk et al., 2015; Khanna et al., 2013; Landrum, 2007; Lawrence, 2012), some included both multiple choice and essay questions (Domenech et al., 2015), and one included short-answer questions (Beagley & Capaldi, 2016). Although the literature suggests assessments that require producing an answer result in greater benefits than assessments involving recognition of a correct answer among alternatives, both produce positive effects on later knowledge retention (Roediger & Karpicke, 2006). As there were other assignments in this course that focused on producing answers, the current results focus on multiple choice questions.

This quality improvement project has several strengths, as the current results extend the literature in multiple ways. Although previous research included timed (Domenech et al., 2015) and untimed (Khanna et al., 2013) assessments, most research did not mention whether the cumulative assessments were timed or untimed (Beagley & Capaldi, 2016; Kerdijk et al., 2015; Landrum, 2007; Lawrence, 2012). No previous research mentioned whether the assessments were openbook or not. Finally, previous literature focused on face-to-face courses (Beagley & Capaldi, 2016; Domenech et al., 2015; Kerdijk et al., 2015; Khanna et al., 2013; Landrum, 2007; Lawrence, 2012). The current results extend the literature by examining knowledge retention through cumulative, untimed, open-book multiple choice online assessments throughout the semester in a doctoral-level hybrid nursing research course.

Conclusion

In this sample of 16 students, the cumulative final examination scores were higher than the cumulative weekly assessments, suggesting improved knowledge retention. Completing more of the cumulative weekly assessments as scheduled was associated with higher cumulative final examination scores and passing the course, suggesting improved knowledge retention and ability to use that knowledge in other assignments throughout the course. This project should be replicated with an item analysis for multiple choice questions in courses with a larger number of students, perhaps by examining the same course across multiple semesters and institutions—if possible, in students at different levels, such as those in Associate, Bachelor, and Master of Science in Nursing programs, and in different course formats such as those that are entirely online.

References

  • American Association of Colleges of Nursing. (2006). Essentials of doctoral education for advanced nursing practice: Clinical scholarship and analytical methods for evidence-based practice. Washington, DC: Author.
  • Beagley, J.E. & Capaldi, M. (2016). The effect of cumulative tests on the final exam. Problems, Resources, and Issues in Mathematics Undergraduate Studies, 26, 878–888.
  • Bloom, B.S., Engelhart, M.D., Furst, E.J., Hill, W.H. & Krathwohl, D.R. (1956). The taxonomy of educational objectives: The classification of educational goals: Handbook I. Philadelphia, PA: David McKay.
  • Dirks, C., Wenderoth, M.P. & Withers, M. (2014). Assessment in the college science classroom. New York, NY: W.H. Freeman.
  • Domenech, J., Blazquez, D., de la Poza, E. & Muñoz-Miquel, A. (2015). Exploring the impact of cumulative testing on academic performance of undergraduate students in Spain. Educational Assessment, Evaluation and Accountability, 27(2), 153–169. doi:10.1007/s11092-014-9208-z [CrossRef]
  • Kerdijk, W., Cohen-Schotanus, J., Mulder, B.F., Muntinghe, F.L. & Tio, R.A. (2015). Cumulative versus end-of-course assessment: Effects on self-study time and test performance. Medical Education, 49, 709–716. doi:10.1111/medu.12756 [CrossRef]
  • Khanna, M.M., Badura Brack, A.S. & Finken, L.L. (2013). Short- and long-term effects of cumulative finals on student learning. Teaching of Psychology, 40, 175–182. doi:10.1177/0098628313487458 [CrossRef]
  • Krathwohl, D.R. (2002). A revision of Bloom's taxonomy: An overview. Theory Into Practice, 41(4), 212–218. doi:10.1207/s15430421tip4104_2 [CrossRef]
  • Landrum, R.E. (2007). Introductory psychology student performance: Weekly quizzes followed by a cumulative final exam. Faculty Forum, 34, 177–180.
  • Lang, J.M. (2016). Small teaching: Everyday lessons from the science of learning. San Francisco, CA: Jossey-Bass.
  • Lawrence, N.K. (2012). Cumulative exams in the introductory psychology course. Teaching of Psychology, 40, 15–19. doi:10.1177/0098628312465858 [CrossRef]
  • Roediger, H.L. III. & Karpicke, J.D. (2006). The power of testing memory: Basic research and implications for educational practice. Perspectives on Psychological Science, 1(3), 181–210. doi:10.1111/j.1745-6916.2006.00012.x [CrossRef]

Description of Cumulative Weekly Quizzes

Week No.Quiz No.No. of QuestionsPoints (%)Mean Score (%)Score SD (%)Mode Score (%)Completed the Quiz as Scheduled (%)
11296.612.510062.5
2240.21000.010081.2
3360.392.710.510087.5
4480.495.36.310087.5
55100.591.910.510081.2
66120.694.87.410085.7
77140.792.47.610068.7
88160.893.45.8100, 9475
99170.8594.93.69487.5
10No quiz
1110190.9586.210.18993.7
1211211.0589.68.6100, 8693.7
1312231.1590.66.288, 8381.2
1413251.2594.85.610081.2
15No quiz
1614251.2596.72.8100, 9687.5
Authors

Dr. Torres is Professor, School of Nursing, University of Mississippi Medical Center, Jackson, Mississippi.

The author has disclosed no potential conflicts of interest, financial, or otherwise.

The author thanks Dr. Gordy for editorial assistance, and the Wisconsin Teaching Fellows and Scholars Program, especially Drs. Cyndi Kernahan and David Voelker for their guidance and feedback. She also thanks the following funding agencies for the protected time to conduct this quality improvement project: the Clinical and Translational Science Award program through the National Institutes of Health (NIH) National Center for Advancing Translational Sciences (UL1TR000427 & KL2TR000428) and the Mississippi Center for Clinical and Translational Research (5U54GM115428). The funding sources had no role in the study design, collection, analysis, interpretation of data, writing of the report, or the decision to submit for publication. The content is solely the responsibility of the author and does not necessarily represent the official views of the NIH.

Address correspondence to Elisa R. Torres, PhD, RN, Professor, School of Nursing, University of Mississippi Medical Center, 2500 North State Street, Jackson, MS 39216; e-mail: etorres@umc.edu.

Received: July 10, 2018
Accepted: January 08, 2019

10.3928/01484834-20190321-11

Sign up to receive

Journal E-contents