It is well established that continuing education (CE) delivers influential professional practice knowledge for nurses and other health care professionals (American Association of Colleges of Nursing & Association of American Medical Colleges, 2010; American Nurses Credentialing Center, 2015). Stakeholders in medicine and nursing education, practice, and regulation have challenged educators to demonstrate the effects of CE on clinical practice (American Association of Colleges of Nursing & Association of American Medical Colleges, 2010; Institute of Medicine, 2010). Providers of nursing CE routinely gather end-of-course evaluation data, and substantive evidence suggests that CE increases participants' knowledge related to the intended learning objectives (das Graças Silva Matsubara & De Domenico, 2016; De Gagne et al., 2015; Yacoub et al., 2015); however, little generalizable research focuses on the transfer of CE to nursing practice.
Most studies of CE-related practice change have focused on how one CE course affected participants' nursing practice, with mixed results. For example, Breneman et al. (2015) evaluated the effectiveness of a school nurse diabetes management course and found a significant improvement in participants' self-reported competence to deliver diabetes care services. Liaw et al. (2016) evaluated a web-based simulation CE course about deteriorating hospital patients for clinical nurses and found that nurses in the medical unit demonstrated significant improvements in recognition of deteriorating cases, although nurses in the surgical unit did not. Another study of nurses who completed a CE course on evidence-based practice and digital resources demonstrated significant improvements in knowledge and skills over time, with no significant effects on attitudes or practice (Ramos-Morcillo, Fernández-Salazar, Ruzafa-Martínez, & Del-Pino-Casado, 2015).
To evaluate whether and how nurses applied the knowledge gained from CE courses in their practice, this study examined nurses' intent to change practice and actual practice change following completion of CE courses delivered by a professional nursing association and American Nurses Credentialing Center (ANCC)–accredited provider of continuing education that delivers hundreds of CE courses to nurses across the nation and abroad each year. As an ANCC-accredited provider unit, all CE courses adhere to established principles of educational design, including the identification of a professional practice gap, analysis of educational need, identification of learning outcomes, and development of learner engagement strategies (ANCC, 2015).
Unlike prior research studies focused on a single CE course, we analyzed practice-related outcomes following nurses' participation in one of 16 different CE courses with differing course types, topics, and geographic locations for more generalizable results. Analysis of nurses' narrative examples describing actual practice changes provided an expanded view of how knowledge and skills gained through different CE courses can affect nurses' practice, as well as barriers they may face in applying CE to practice.
This retrospective, mixed-methods study design assessed quantitative and qualitative survey data gathered upon completion and within 1 year after completion of one of 16 different CE courses. End-of-course surveys provided quantitative data related to participants' intent to apply knowledge and skills gained from the course in their nursing practice. Follow-up surveys provided quantitative data related to actual practice change and the nurses' qualitative descriptions of practice change or practice change barriers. The study was submitted to the institutional review board for ethical review and was determined to be exempt research.
This study was designed to answer the following research questions:
- What percentage of nurses reported positive intent to change practice upon completion of the CE courses?
- What percentage of nurses reported actual practice change within 1 year following completion of the CE courses?
- Is there a relationship between the nurses' intent to change practice and actual practice change?
- What common themes emerge in the nurses' examples of actual practice change?
- What common themes emerge in the nurses' examples of barriers to actual practice change?
The new world Kirkpatrick model for learning outcome evaluation posits that the value of education increases as value to the organization increases (Kirkpatrick Partners, n.d.). The model outlines four levels of educational evaluation: Level 1, reaction, assesses learner satisfaction, engagement in the education and learning process, and relevance of education to the learner's practice; Level 2, learning, assesses gained knowledge, skills, attitude, confidence, and commitment following education; Level 3, behavior, assesses application of knowledge, skills, attitudes, confidence, and commitment following education; Level 4, results, assesses the achievement of positive outcomes in the organization. The Kirkpatrick model provided a theoretical framework for the study design. Intent to change practice aligns with Level 2 of the model in terms of acquired knowledge and a commitment to apply it in practice. Actual practice change aligns with Level 3 of the model in terms of self-reported change in practice behaviors.
The study sample consisted of nurses who completed one of 16 CE courses, including nine certification review course workshops, three organizational excellence workshops, and four certification review practiceIQ courses, during the period of January 1, 2016, to October 1, 2016, and who completed an end-of-course survey, a follow-up survey within 1 year of completing one of the CE courses, or both. If nurses completed surveys related to more than one CE course, only the first set of responses to either survey was included.
The nonprobability convenience sample consisted of 1,138 nurses. There were 480 who completed the end-of-course survey and/or the follow-up survey. Completion of the end-of-course survey did not guarantee completion of the follow-up survey. A subsample of 403 nurses (n = 403) completed the intent to change practice survey. A subsample of 110 nurses (n = 110) completed the actual practice change survey. The survey responses were evaluated retrospectively. Demographic data were not collected as a part of the original surveys.
The end-of-course survey included one item inquiring on intent to change practice. A 5-point scale measured the intent responses as follows:1 = I do not intend to change my practice, 2 = I am undecided about changing my practice, 3 = I might change my practice, 4 = I am committed to changing my practice, and 5 = I am highly committed to changing my practice. Responses of 4 or 5 indicated a response of yes for analysis. Responses of 1, 2, or 3 were coded as no for analysis. There was a 42% response rate for this survey. Completion of the survey was required for CE credit, which may have reduced the number of nonresponders (Trochim, Donnelly, & Arora, 2016).
The follow-up survey contained three items inquiring about actual practice change. One item assessed whether the nurses changed or improved their practice based on knowledge gained in the CE course using a binary measure of yes or no. Respondents who answered yes were asked to provide an example. Respondents who answered no were asked to describe what prevented it. There was a 9.6% response rate for this survey. Strategies to reduce nonresponse included a 0.5 contact hour incentive for reflection and explanation of the actual practice change response, short survey length, and a follow-up request to nonrespondents (Dillman, Smyth, & Christian, 2014; Trochim et al., 2016).
Descriptive statistics were computed on the quantitative data using SPSS® Statistics Base for Windows. Frequencies and percentages of nurses' intent to change practice and actual practice change were generated. A content analysis was conducted on the qualitative data to elucidate themes related to actual practice change and practice change barriers.
Analysis of the 403 responses to the intent to change practice survey revealed that 88.6% of the nurses (357 respondents) intended to change their practice and 11.4% of the nurses (46 respondents) did not intend to change their practice following the CE course. Of the 110 responses to the actual practice change survey, 89.1% of the nurses (98 respondents) reported actual practice change, and 10.9% of the nurses (12 respondents) reported no practice change following the CE course (Figure 1).
Comparison of intent to change practice and actual practice change responses.
Fifty-one nurses completed both surveys. Of these nurses, 44 nurses (86%) reported both an intent to change practice and actual practice change. None of the nurses reported both no intent to change practice and no actual practice change. We planned to analyze these data using chi-square (χ2) to examine the relationship between the nurses' intent to change practice and actual practice change. The lack of variance among these responses did not support a valid chi-square analysis.
We conducted a post-hoc analysis using chi-square (χ2) to examine the relationship of intent and actual practice change to course type. Course type was not a significant factor in the intent to change or actual practice responses, χ2 (2, n = 403) = 1.319, p = .517, V =.057; χ2 (2, n = 110) = 2.302, p = .316, V =.145, respectively. Table 1 displays the frequencies of the intent and actual practice change responses by course type.
Frequency of Intent to Change Practice and Actual Practice Change by Course Type
All 110 nurses (n = 110) that responded to the actual practice change survey provided examples of how the knowledge gained in the CE course was applied in practice or a description of practice change barriers. A content analysis of the 98 nurses' examples of actual practice change revealed four common themes: Becoming Certified, Improved Leadership, Enhanced Role Performance, and Educating or Mentoring Others. A content analysis of the 12 nurses' explanations for no practice change revealed two themes related to a lack of association to the nurses' current role and a lack of meeting the nurses' learning needs.
Responses related to Becoming Certified included descriptions of how participation in CE contributed to becoming board-certified as nurse executives (NEA-BC), family nurse practitioners (FNP), and other specialties. An interesting and important aspect of this theme related to the development of higher level problem-solving skills needed to successfully complete certification examinations and to support clinical practice. Kelly (all names are pseudonyms) noted:
It allowed me to learn to think critically. What I mean by this is I learned to problem solve through questions. Answering questions for the ANCC FNP exam required a step-wise problem-solving approach that no amount of memorization of knowledge could prepare you for and this program really helped me hone in on my problem-solving skills for the [examination], but also as a future provider.
The second theme was evident in the nurses' examples of improved leadership skills. Although some examples were related to formal leadership roles, many involved leadership demonstrated through professional practice behaviors. John described how he influenced professional nursing practice in his organization as follows: “I have adjusted current scripting and tip sheets for [the] mental health clinic nurse triage line [incorporating information learned]. These tip sheets and scripts are becoming part of the mental health clinic competencies for all nurses.” Amanda reported research and quality improvement projects in the organization using knowledge learned in the CE course, stating “This was a great review and granted me knowledge related to theory, laws and research. I have facilitated research projects and quality improvement projects since my attendance.”
Examples of Improved Leadership often resulted in a return on investment for the organization, including but not limited to, practical skills for nurse managers, improved decision making, and organizational improvements. Karen described how the CE course enabled the management of staffing needs, stating, “Practice change involved [a] better understanding of staffing and how to compute FTE [full-time equivalent] requirements based on standards of care and acuity.” Improved decision making was evident in statements such as “Understanding workplace issues has helped me gain a better awareness of negative behaviors. Recognizing the behavior and extinguishing it before it got worse prevented a nurse from leaving the organization.” Mary shared an example of a CE-related organizational improvement:
I reflected on my communication skills and implemented new strategies when interacting with staff and patients. Being mindful of staff and patients, using active listening skills, and modeling transformational leadership has led to improved patient satisfaction scores.
The theme of Enhanced Role Performance encompassed CE-related practice improvements specific to the nurses' current roles, including implementation of evidence-based practices, application of theory in practice, and interprofessional collaboration. Two nurses described the influence of the CE course on evidence-based practice. Jennifer stated, “The information learned from this course has provided me with evidence-based practices that I carry through my work.” Matthew explained how the CE course validated his clinical practice decisions with evidence and added new knowledge and perspectives that improved patient care:
Direct patient interaction has improved due to the review of current evidence-based knowledge that validates [my] established knowledge base and experience. Assessments for mental health patients are more thorough due to the idea of assessing all aspects of the mental health patient's health.
Bobbie described how she applied theoretical knowledge from the CE course into practice. She stated,
I find in my leadership roles and with program development plans and mergers it is very important to know how to [change]. Kurt Lewin: unfreezing-change-refreezing model is important to health care. We had to ask staff to change their thinking about best practices…. As leaders, knowing how to change is a required expertise. This was one of my most beneficial learnings [from] this educational program.
The Enhanced Role Performance theme also included examples of improved interprofessional collaboration. Stacey stated that “reinforced and improved knowledge has facilitated [my] participation in multi-disciplinary treatment team meetings that directly impact patient treatment plans and outcomes.” Nancy explained the practice of “patient-centered care and involving members of the team to focus on productivity and efficiency, as well as providing consumers with the right care at the right place at the right time.”
The final theme emerged from examples of nurses using newfound knowledge for Educating and Mentoring Others. Debbie stated, “I learned many clinical facts that I did not know about the cardiovascular system [and] used new-found information to educate the rest of our unit staff. We discussed key changes that could be useful to implement on our unit.” Marilyn stated, “Based on the information, I am training the organization to be better in documentation of unit council meetings or process improvement work [and to] understand and communicate better the Magnet expectations.” Sheila explained modeling newly learned behaviors when mentoring others, stating “[My] role as a clinical coach/mentor has improved as I am a better role model with direct patient interaction and am more focused on mental health knowledge versus just learning the routine functional clinical operations.”
Nurses who indicated they did not change their practice described barriers such as a lack of relevance to their current role or a disconnect between the CE and their individual learning needs. Two nurses explained that their intention for taking the CE course was to learn a new specialty area. Aaron stated, “I took this course to get a glimpse of what the knowledge base requirement for a nurse executive would be,” and Melissa explained that the CE course “did not pertain to current practice—only potential future practice.” A few nurses described how the CE courses did not meet their goals or preferences as a learner. Ashley explained, “I thought the course was helpful, but it was more of a review than learning new information. Unfortunately, I did not pass my certification test.” Angela expressed a preference for a different course design, stating:
I took the practice test and it was helpful in that I learned I was below in all areas. What would be more helpful is if the practice test linked to specific content that I could use to remediate. I did not [take] the quizzes and wonder if that would have provided the information that I needed.
The results of this study provide evidence of practice change among a large group of nurses completing one of 16 different CE courses. There were three course types that included certification review course workshops, organizational excellence workshops, and online certification practice question courses. These courses were designed as either an asynchronous online course with practice certification questions and individualized performance reporting, a face-to-face workshop with lecture and problem-based learning activities, or a blended workshop with online asynchronous learning modules followed by a face-to-face lecture-based workshop. An array of course topics included certification reviews for seven different RN specialty areas, three APRN specialty areas, and three organizational excellence topics. The workshop locations included eight different states in the United States and two international countries. This diversity makes the study results more generalizable to diverse nursing specialties, practice settings, levels of practice, and geographic location than previous studies of transfer to practice from a single CE course targeting one nursing specialty, practice setting, or patient population.
Most of the nurses in this study who completed one or both surveys reported an intent to change practice and actual practice change due to their participation in the CE courses. In addition, among the 10% of the nurses who completed both the intent and actual practice change surveys, almost all who intended to change practice followed through with implementing their intended changes. This is consistent with previous research that found that most nursing faculty who reported an intent to apply new knowledge and strategies also implemented curricular practice changes following participation in conference-based CE (Ignatavicius & Chung, 2016).
Results of the qualitative analysis enriched our understanding of CE's actual impact on nursing practice. Based on the themes that emerged, it appears that CE can positively influence nurses' practice by supporting certification, improving their leadership abilities, enhancing their roles as care providers and members of interdisciplinary teams, and providing them with new knowledge and skills they can use to educate or mentor others. Nurses' narrative examples demonstrated how application of knowledge and skills gained from the CE courses had a positive influence on many aspects of nursing practice, as well as broader organizational influences.
When evaluating the impact of CE, it is important to go beyond the evaluation of learner reactions (Kirkpatrick Level 1) and learning (Kirkpatrick Level 2) by also measuring impact on learner behaviors (Kirkpatrick Level 3). By assessing both intended and actual practice change, this study provided evidence that CE can lead to positive, long-term impacts on learner behavior. Additional research on the relationship between intent and actual practice change and the factors affecting practice change would be valuable as next steps. Furthermore, it is often challenging to demonstrate a direct connection between education and quantitatively measurable organizational results. The unanticipated finding of organizational impacts within nurses' narrative examples of practice change provides preliminary evidence that CE can also contribute to positive organizational results (Kirkpatrick Level 4). Our findings highlight the value of learner reflections in revealing less tangible evidence of CE impacts at the organizational level.
Self-reported data and nonprobability sampling limits our ability to demonstrate a cause-effect relationship between CE courses and nursing practice change. Our knowledge of the diversity of participants is limited by the lack of demographic questions in the original surveys. Additional research investigating transfer to nursing practice from other types of CE courses, with demographic data on the participants, will enhance the generalizability of our results.
Evidence of CE transfer to practice was needed to demonstrate value and the contributions from CE to nursing practice outcomes. The results of this study contribute much needed evidence of CEs positive impact on nursing practice, both individually and at an organizational level. For CE providers, the results help to demonstrate the long-term impact of their educational offerings. For nurses, the results help to demonstrate the value of earned CE for self-fulfillment and value propositions to organizational leadership. Organizations that invest in CE invest in improving nursing practice outcomes.
- American Association of Colleges of Nursing, & Association of American Medical Colleges. (2010). Lifelong learning in medicine and nursing: Final conference report. Retrieved from https://www.aacnnursing.org/Portals/42/AcademicNursing/pdf/MacyReport.pdf
- American Nurses Credentialing Center. (2015). 2015 ANCC primary accreditation provider application manual. Silver Springs, MD: Author.
- Breneman, C.B., Heidari, K., Butler, S., Porter, R.R. & Wang, X. (2015). Evaluation of the effectiveness of the H.A.N.D.S.SM program: A school nurse diabetes management education program. Journal of School Nursing, 31, 402–410. doi:10.1177/1059840514568895 [CrossRef]
- das Graças Silva Matsubara, M. & De Domenico, E.B.L. (2016). Virtual learning environment in continuing education for nursing in oncology: An experimental study. Journal of Cancer Education, 31, 804–810. doi:10.1007/s13187-015-0889-x [CrossRef]
- De Gagne, J.C., Park, S., So, A., Wu, B., Palmer, M.H. & McConnell, E.S. (2015). A urinary incontinence continuing education online course for community health nurses in South Korea. The Journal of Continuing Education in Nursing, 46, 171–178. doi:10.3928/00220124-20150320-02 [CrossRef]
- Dillman, D.A., Smyth, J.D. & Christian, L.M. (2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method (4th ed.). Hoboken, NJ: John Wiley & Sons.
- Ignatavicius, D. & Chung, C.E. (2016). Professional development for nursing faculty: Assessing transfer of learning into practice. Teaching and Learning in Nursing, 11(4), 138–142. doi:10.1016/j.teln.2016.05.005 [CrossRef]
- Institute of Medicine. (2010). Redesigning continuing education in the health professions. Washington, DC: National Academies Press.
- Kirkpatrick Partners. (n.d.). The new world Kirkpatrick model. Retrieved from http://www.kirkpatrickpartners.com/OurPhilosophy/TheNewWorldKirkpatrickModel/tabid/303/Default.aspx
- Liaw, S.Y., Wong, L.F., Lim, E.Y., Ang, S.B., Mujumdar, S., Ho, J.T. & Ang, E.N. (2016). Effectiveness of a web-based simulation in improving nurses' workplace practice with deteriorating ward patients: A pre- and postintervention study. Journal of Medical Internet Research, 18(2), e37. doi:10.2196/jmir.5294 [CrossRef]
- Ramos-Morcillo, A.J., Fernández-Salazar, S., Ruzafa-Martínez, M. & Del-Pino-Casado, R. (2015). Effectiveness of a brief, basic evidence-based practice course for clinical nurses. Worldviews on Evidence-Based Nursing, 12(4), 199–207. doi:10.1111/wvn.12103 [CrossRef]
- Trochim, W.M., Donnelly, J.P. & Arora, K. (2016). Research methods: The essential knowledge base (2nd ed.). Boston, MA: Cengage Learning.
- Yacoub, M.I., Demeh, W.M., Barr, J.L., Darawad, M.W., Saleh, A.M. & Saleh, M.Y. (2015). Outcomes of a diabetes education program for registered nurses caring for individuals with diabetes. The Journal of Continuing Education in Nursing, 46, 129–133. doi:10.3928/00220124-20150126-02 [CrossRef]
Frequency of Intent to Change Practice and Actual Practice Change by Course Type
|Variable||Certification Review Course Workshops||Organizational Excellence Workshops||Online Certification Practice Question Courses||Total|
|Intent to change practice|
|Missing data (no response to question or no continuing education evaluation completed)||77|
|Actual practice change|
|Missing data (no longitudinal evaluation completed)||370|