The Journal of Continuing Education in Nursing

Administrative Angles 

Creating a CNE Audit Tool Using the Plan-Do-Study-Act Model

Susan Haynes Little, DNP, RN, PHNA-BC, CPHQ, CPM; Jana Busick, MBA

Abstract

The continuous performance-improvement model known as Plan-Do-Study-Act (PDSA) was used within the Approved Provider Unit of the North Carolina Division of Public Health to create an audit tool and process for assessing continuing nursing education courses. During the development of the audit tool, gaps and process inefficiencies were identified when documentation was reviewed for adherence to the American Nurses Credentialing Center's Commission on Accreditation's 2015 Criteria for Continuing Nursing Education and local provider unit requirements. This article traces the progress through three cycles of the PDSA model and provides the resulting audit tool and process. [J Contin Educ Nurs. 2019;50(2):53–56.]

Abstract

The continuous performance-improvement model known as Plan-Do-Study-Act (PDSA) was used within the Approved Provider Unit of the North Carolina Division of Public Health to create an audit tool and process for assessing continuing nursing education courses. During the development of the audit tool, gaps and process inefficiencies were identified when documentation was reviewed for adherence to the American Nurses Credentialing Center's Commission on Accreditation's 2015 Criteria for Continuing Nursing Education and local provider unit requirements. This article traces the progress through three cycles of the PDSA model and provides the resulting audit tool and process. [J Contin Educ Nurs. 2019;50(2):53–56.]

To capitalize on the adoption of the revised American Nurses Credentialing Center's Commission on Accreditation (ANCC COA) 2015 Criteria, the primary nurse planner of the North Carolina Division of Public Health's approved provider unit (APU) sought to improve structural capacity and quality outcomes by using a capacity-building continuing nursing education (CNE) documentation peer audit process. A new audit tool was built using the Plan-Do-Study-Act (PDSA) model (Agency for Healthcare Research & Quality, 2013; Hughes, 2008) by comparing existing documentation practices while adopting the revised ANCC COA criteria. During succeeding PDSA cycles, the audit tool was revised based on feedback from the APU nurse planners and documentation peer audits using the 2015 criteria. Knowledge gained from the audits informed the development of the APU's processes and procedures.

Background

The North Carolina Division of Public Health's APU plans, implements, and evaluates approximately 60 new CNE activities each year for public health nurses working in North Carolina's local and state public health agencies. The APU is managed by the primary nurse planner who plans CNE activities and provides oversight for all unit functions. Nurse planners coordinate CNE activities in addition to their responsibilities as public health nurse specialists in administration, school nursing, women and children's health, communicable diseases, and public health nursing practice consultation (Little, Dickerson, Randolph, Rocco, & Short, 2017). All unit nurse planners also serve as CNE documentation peer auditors.

In 2016, all CNE provider units were tasked with adopting the revised ANCC COA 2015 criteria. Initially, the APU nurse planners believed the adoption would be challenging because they had to manage the process of planning, implementing, and evaluating CNE activities while simultaneously learning the revised ANCC COA criteria. However, the primary nurse planner was committed to maintaining a competent and confident nurse planner team. The primary nurse planner would reach this goal by ensuring the nurse planners had the knowledge and tools needed to make the successful transition by building a CNE documentation peer audit process that would capture adherence to the 2015 ANCC COA criteria and local provider unit requirements, and would be viewed as a capacity building, nonpunitive process by the nurse planners.

A work team was created to look at the current process of planning, implementing, and evaluating CNE activities. An audit tool was built to capture adherence to the 2015 ANCC COA educational design process (EDP) criteria and local APU requirements. Prior to the adoption of the revised 2015 EDP criteria, the audit tool was tested on CNE activity documentation that used the 2013 EDP criteria (ANCC, 2013a, 2013b). Although the audit tool did not fully align with documentation that adhered to the old criteria, testing the audit tool on familiar documentation allowed the primary nurse planner to focus on the components of the audit tool without the distraction of new and unfamiliar approver unit required forms.

After the first CNE activity was planned following the revised criteria, the audit tool was tested and evaluated on the new documentation and criteria. Completing three PDSA cycles with the audit tool helped ensure that the audit tool captured the required components and that the peer audit process would be viewed as capacity-building.

Related Literature

Evaluating educational activities is critical to ensuring learner outcomes have been met and knowledge gaps have been filled (Dickerson, 2016). Of equal importance, the provider unit must evaluate CNE activities for adherence to compliance standards associated with the EDP criteria (ANCC, 2015). Performing regular audits is an evaluative process because it follows a systematic approach to measure performance and compliance by collecting information and data that can be used to improve processes (Gillam & Siriwardena, 2013). The information learned from regular audits is useful for planning for process improvement.

The use of improvement methodologies is a vital step in assessing and improving processes. By using process improvement models, information gained can be used to make necessary adjustments to the process steps to gain efficiencies. The PDSA model is a continuous process improvement model championed by W. Edwards Deming (The W. Edwards Deming Institute, 2018). The four-stage model evaluates process changes to determine whether they add value. The PDSA model can be used at multiple stages as processes evolve to determine the efficiency and effectiveness of the changes (Institute for Healthcare Improvement, 2018; Hughes, 2008). Table 1 shows the steps that traditionally occur in the PDSA model.

Plan-Do-Study Act (PDSA) Model

Table 1:

Plan-Do-Study Act (PDSA) Model

Method

A team was brought together for a 3-day work session to help transition from the 2013 ANCC COA criteria to the revised 2015 ANCC COA criteria. The team consisted of five nurse planners representing multiple public health nurse specialties, the APU professional development associate, the Division of Public Health Chief Nurse, the primary nurse planner, and a performance improvement consultant.

An initial accomplishment of the work team was the creation of a process map of the steps the nurse planners followed to plan CNE when using the 2013 EDP criteria. The team then mapped out the necessary steps the process would need when following the 2015 EDP criteria (Little et al., 2017). The two process maps were compared, and areas in the new process map that diverged from the old process map were identified.

The work team then created a draft peer audit tool incorporating the 2015 EDP criteria and local APU requirements such as new forms. A draft audit process was created to support consistent implementation and to ensure that all new identified processes, such as newly implemented due dates and documentation deadlines, were captured. The work team named the new peer audit tool and process “Auditing for Success, Competence, and Confidence” to reflect the process as capacity building instead of punitive. The new tool and process were ready to be tested.

First PDSA Cycle

Plan. Documentation based on the 2013 EDP criteria would be used for the first test. Process steps of CNE activity planning, implementing, and evaluation also would be evaluated.

Do. The primary nurse planner tested the first draft of the audit tool on one CNE activity that was documented based on the 2013 EDP criteria. Inconsistencies were noted when a component of the 2015 EDP criteria was missing in the CNE activity documentation. The map of the current CNE planning process also was reviewed to look for and note opportunities to improve when implementing the revised criteria.

Study. The testing of the new draft audit tool, including looking at the overall process of planning and documenting CNE activities, revealed inefficiencies in the process steps and gaps in documentation.

  • The activity planning process included a step where all documentation had to be reviewed by the APU primary nurse planner before any CNE activity was provided. This requirement caused a potential barrier to providing timely CNE activities.
  • Emphasis was put on the completeness of documentation as opposed to the quality of the activity's EDP.
  • Knowledge gaps and educational needs were inconsistently verified and documented.

Act. Based on the results and information learned about the current process, the primary nurse planner decided to adopt the draft audit tool and test it again on documentation resulting from the first CNE activity planned after adopting the 2015 EDP criteria.

Second PDSA Cycle

Plan. The primary nurse planner would test the draft audit tool to evaluate the documentation of the first CNE activity using the 2015 EDP criteria.

Do. The primary nurse planner tested the draft audit tool on one CNE activity that used the 2015 EDP criteria and noted when documentation of a component of the 2015 EDP criteria or local APU requirements was missing or incorrect.

Study. The audit revealed the nurse planner had incorporated the revised 2015 EDP criteria and local APU requirements in the CNE documentation. Thus, the audit process effectively captured that the documentation had met all EDP and APU audit components.

Act. With the audit components of the tool testing as effective, the primary nurse planner determined that a process change was needed to use nurse planners as CNE activity peer auditors. A further revision of the audit tool would be needed to add elements that would support the capacity of the peer auditors. Measurement elements also would be added to capture data for the APU's quality outcomes.

Third PDSA Cycle

Plan. The primary nurse planner brainstormed how the audit tool and process could be improved by incorporating references to the location where evidence for each component is typically found in the documentation and how data from the tool could be used to support the APU's quality outcomes. The audit tool would be modified to incorporate the improvements.

Do. The primary nurse planner revised the audit tool (Table A; available in the online version of this article) to include:

  • Location references describing where to find evidence of each component in the required documentation.
  • A section capturing the overall compliance score and the 2015 EDP criteria compliance score.
  • A new component to indicate that the audit results were reviewed with the nurse planner.
Audit for Success, Competence, & ConfidenceNorth Carolina DPH APU Activity Audit Form 2015 CriteriaAudit for Success, Competence, & ConfidenceNorth Carolina DPH APU Activity Audit Form 2015 CriteriaAudit for Success, Competence, & ConfidenceNorth Carolina DPH APU Activity Audit Form 2015 Criteria

Table A:

Audit for Success, Competence, & Confidence

North Carolina DPH APU Activity Audit Form 2015 Criteria

The primary nurse planner and two peer auditors tested the revised audit tool on three CNE activities that used the 2015 EDP criteria.

Study. The audits revealed that all documentation met all 2015 EDP criteria and APU audit components. In addition, the peer auditors provided feedback indicating that adding the location references assisted them as they were learning the revised criteria and audit process. The peer auditors also shared they felt the revised audit tool would help them sustain compliance with the 2015 EDP criteria. Finally, the scoring section provided an accurate measurement of compliance that also could be tracked over time for performance improvement.

Act. The primary nurse planner revised the audit process. The audit tool and process were adopted into the overall standard operating procedures of the APU in October 2017 (Table B; available in the online version of this article).

Auditing for Success, Competence, and Confidence: Process

Table B:

Auditing for Success, Competence, and Confidence: Process

Discussion

Feedback received from the nurse planners who served as peer auditors indicated that the Auditing for Success, Competence, and Confidence process was not only enjoyable but also helped increase their knowledge of the 2015 ANCC COA criteria. In addition, the nurse planners believed the peer audit process increased their competence as nurse planners and their ability to accurately document the CNE activities using the required forms. The peer audit process and tool have been integrated into unit operations, and the nurse planners are expected to use the findings from the CNE documentation audits to improve their nursing practice as CNE nurse planners. Data and information gleaned from the audit process will support the APU's structural capacity and be used to enhance and improve quality outcomes.

Using the PDSA model, an evidence-based approach to process improvement, to create the Auditing for Success, Competence, and Confidence peer audit tool and process helped the nurse planners and APU to successfully transition after the adoption of the 2015 ANCC COA criteria. The PDSA model provided valuable insight into the changes that were needed when implementing the 2015 ANCC COA criteria. As CNE-related criteria and requirements are updated in the future, using the PDSA model will be an effective method of assuring that the peer audit tool and process remain current and pertinent.

References

  • Agency for Healthcare Research & Quality: Health Care Innovations Exchange. (2013, April). Plan-do-study-act (PDSA) cycle. Retrieved from https://innovations.ahrq.gov/qualitytools/plan-do-study-act-pdsa-cycle
  • American Nurses Credentialing Center. (2013a). Educational design process 2013 mini manual. Silver Spring, MD: Author.
  • American Nurses Credentialing Center. (2013b). 2013 ANCC primary accreditation application manual for providers and approvers. Silver Spring, MD: Author.
  • American Nurses Credentialing Center. (2015). 2015 ANCC primary accreditation provider application manual. Silver Spring, MD: Author.
  • Dickerson, P.S. (2016). The evolving role of a nurse planner. The Journal of Continuing Education in Nursing, 47, 437–439. doi:10.3928/00220124-20160920-02 [CrossRef]
  • Gillam, S. & Siriwardena, A. (2013). Frameworks for improvement: Clinical audit, the plan-do-study-act cycle and significant event audit. Quality in Primary Care, 21, 123–130.
  • Hughes, R.G. (Ed.). (2008). Tools and strategies for quality improvement and patient safety. In: Patient safety and quality: An evidence-based handbook for nurses (chapter 44). Rockville, MD: Agency for Healthcare Research and Quality. Retrieved from https://www.ncbi.nlm.nih.gov/books/NBK2682/
  • Institute for Healthcare Improvement. (2018). Science of improvement: Testing changes. Retrieved from http://www.ihi.org/resources/Pages/HowtoImprove/ScienceofImprovementTestingChanges.aspx
  • Little, S., Dickerson, P., Randolph, G., Rocco, P. & Short, N. (2017). Using the ARCC model to implement the 2015 ANCC COA criteria: A North Carolina public health nursing project. The Journal of Continuing Education in Nursing, 48, 501–507. doi:10.3928/00220124-20171017-07 [CrossRef]
  • The W. Edwards Deming Institute. (2018). PDSA cycle. Retrieved from https://deming.org/explore/p-d-s-a

Plan-Do-Study Act (PDSA) Model

Plan

Identify the problem and the possible reasons why it exists

Identify a goal for improvement, including success metrics

Develop a plan of action

Do

Implement the plan of action on a small scale (test)

Study

Document observations, including signs of progress and success, or problems and areas for improvement

Analyze the information and data collected during testing

Summarize what was learned

Act
Based on what was learned, determine which of the following actions should occur:

Adopt: test the plan on a larger scale

Adjust: modify the plan and test again

Abandon: discard the plan

Audit for Success, Competence, & Confidence

North Carolina DPH APU Activity Audit Form 2015 Criteria

   NAME OF NURSE PLANNER: _____
   DATE OF AUDIT: _____    AUDITOR NAME & CREDENTIALS: _____
   Title of activity: _____
   Type of activity:
    _____ Provider-directed, provider-paced, Live: _____ in person _____ webinar
    _____ Provider-directed, learner-paced: Enduring material/Independent study
    _____ Blended activity
   DATE AUDIT RESULTS DISCUSSED WITH NURSE PLANNER: _____
   Total Components Met    _____/19 = _____%
   Total EDP Criteria Compliance _____/7 = _____%
Activity file documentation criterionComplianceComments
1. New Activity Notification Form submitted to Professional Development Associate 2 weeks prior to event_____ Met _____ Not Met
2. Completed packet submitted no later than 2 months after last day of event_____ Met _____ Not Met
3. Completed conflict of interest form included for each planner/presenter (EDP3) Evidence: Section M. in Planning Form, COI/BIO forms_____ Met _____ Not Met
4. Description of resolution of conflict of interest, if appropriate (EDP3) Evidence: COI/BIO forms; additional explanation of resolution process and outcome_____ Met _____ Not Met _____ N/A
5. Description of the professional practice gap provided. (EDP1) Evidence: Planning Form, Section A: current state, desired state, gap_____ Met _____ Not Met
6. Evidence to validate the professional practice gap provided. WHY is this activity needed? (EDP1) Evidence: Planning Form, Section B_____ Met _____ Not Met
7. Participant evaluation results from previous activities were used to guide planning of this activity, or, a future offering of this activity will be changed because of participant evaluation data from this offering (EDP6) Evidence: Planning Form, Section B-4; Debrief Form_____ Met _____ Not Met _____ N/A
8. Educational need that underlies the professional practice gap (knowledge, skill and/or practices). (EDP2) Evidence: Planning Form, Section B_____ Met _____ Not Met
9. Description of the target audience Evidence: Planning Form, Section C_____ Met _____ Not Met
10. Members representing all target audiences included on the planning committee Evidence: Planning Form Sections C and M_____ Met _____ Not Met
11. Desired learning outcome(s) and outcome(s) measures provided (EDP5) Evidence: Planning Form, Sections D and F_____ Met _____ Not Met
12. Content is appropriate for learning outcomes and target audience (EDP4) Evidence: Planning Form Section G, Planning Table_____ Met _____ Not Met
13. A description of the evidence-based content and list of supporting references or resources is provided. (Seminal reports can fall out of the 5-year date range.) (EDP4) Evidence: Planning Form Section G (Content); Planning Table (List)_____ Met _____ Not Met
14. Learner engagement strategies identified (EDP5) Evidence: Planning Form Section J, Planning Table_____ Met _____ Not Met
15. Learner engagement strategies appear to be appropriate for the knowledge/skills/practice learning needs identified (EDP5) Evidence: Planning Form, Section J, Planning Table_____ Met _____ Not Met
16. Evaluation method provides evidence that change in knowledge, skills and/or practices of target audience were assessed (EDP7) Evidence: Planning Form Section L, Participant Evaluation results, Debrief notes_____ Met _____ Not Met
17. Evaluation method(s) are appropriate for the learner outcome(s) (EDP7) Evidence: Planning Form Section L, Participant Evaluation results, Debrief notes_____ Met _____ Not Met
18. Correct Provider Statement is used on all advertising and the certificate of completion_____ Met _____ Not Met
19. Evidence that disclosures provided prior to the start of the learning activity_____ Met _____ Not Met
20. Other

Auditing for Success, Competence, and Confidence: Process

Once CNE documentation has been submitted to the APU Professional Development Associate upon closing of CNE activity, the primary nurse planner, or assigned nurse planner, will audit documentation for compliance to 2015 ANCC COA criteria and APU requirements. Process for Auditing CNE Activities

Review all submitted CNE forms. Reviewers are encouraged to reach out to the nurse planner if forms are missing.

Look for evidence to identify adherence to the numbered components.

Reviewers are provided guidance in the audit form regarding where the documentation is usually found but are encouraged to also look at nontraditional sites for supporting evidence (e.g., References listed on Planning Form instead of Planning Table).

Compare the submitted documentation against the 19 required components in the audit tool.

Mark each component as met or not met (two allow for N/A).

Provide relevant comments in the Comments section of each item. Examples of relevant comments include evidence to support not-met items or noting documentation or methods that are notable in quality.

The number of met items is divided by the total items for total components, and EDP Criteria-specific components to calculate a percentage met (compliance).

Provide other feedback that is not captured by the 19 required components under #20: Other.

Process for Post-Audit Discussion with Nurse Planner

Contact the nurse planner to schedule an audit review date.

Provide the nurse planner with a copy of the draft completed audit tool before discussion or conduct meeting using Adobe Connect (or another desktop-sharing software).

Review findings; discuss not-mets and strengths found during the audit.

Instruct the nurse planner to submit any corrected forms to the APU Administrative Specialist to replace any forms that had errors (if applicable).

Complete the form with the date and any postdiscussion notes.

Submit the final audit to the nurse planner and the primary nurse planner.

Authors

Dr. Little is Manager, Public Health Nursing and Professional Development Unit, North Carolina Division of Public Health, and Ms. Busick is Chief, Strategy and Planning, and Interim Chief, Health Care Personnel and Credentialing, North Carolina Department of Health and Human Services, Raleigh, North Carolina.

The authors have disclosed no potential conflicts of interest, financial or otherwise.

Address correspondence to Susan Haynes Little, DNP, RN, PHNA-BC, CPHQ, CPM, 11 Deer Lake Trail, Wendell, NC 27591; e-mail: DrSusanLittle@alumni.duke.edu.

10.3928/00220124-20190115-02

Sign up to receive

Journal E-contents