Journal of Nursing Education

Major Article 

Development and Validation of the Performance-Based Situation Awareness Observation Schedule (PBSAOS): The Process and Lessons Learned

Heather Gluyas, DN, RN; Norman Jay Stomski, PhD; Prue Andrus, MHMQ, RN; Jennifer Walters, PhD, RN; Paul Morrison, PhD, RN; Anne Marjorie Williams, PhD, RN; Martin Hopkins, PhD, RN; Martinique Sandy, BN, RN

Abstract

Background:

No instruments are currently available to assess nursing students' maintenance of situation awareness. The aim of this study was to develop and validate the Performance-Based Situation Awareness Observation Schedule for measuring nursing students' use of situation awareness.

Method:

Twelve expert clinicians participated in a qualitative, reiterative consensus-driven process to establish the content validity of the tool. The tool was then piloted during the delivery of a situation awareness education program for final-year nursing students. Cohen's kappa was used to assess the interrater reliability.

Results:

The resultant tool comprised 54 items, which captured strategies to maintain situation awareness and task errors that would infer the presence of situation awareness. The values obtained for Cohen's kappa indicated that the level of agreement was at least substantial for approximately 80% of the items.

Conclusion:

This study developed a valid and reliable tool to measure nursing students' use of situation awareness. [J Nurs Educ. 2019;58(8):468–473.]

Abstract

Background:

No instruments are currently available to assess nursing students' maintenance of situation awareness. The aim of this study was to develop and validate the Performance-Based Situation Awareness Observation Schedule for measuring nursing students' use of situation awareness.

Method:

Twelve expert clinicians participated in a qualitative, reiterative consensus-driven process to establish the content validity of the tool. The tool was then piloted during the delivery of a situation awareness education program for final-year nursing students. Cohen's kappa was used to assess the interrater reliability.

Results:

The resultant tool comprised 54 items, which captured strategies to maintain situation awareness and task errors that would infer the presence of situation awareness. The values obtained for Cohen's kappa indicated that the level of agreement was at least substantial for approximately 80% of the items.

Conclusion:

This study developed a valid and reliable tool to measure nursing students' use of situation awareness. [J Nurs Educ. 2019;58(8):468–473.]

Safe nursing care depends on the knowledge, skill, and competency of nursing staff to provide evidence-based care. There are psychomotor and technical skills such as undertaking a patient physical assessment, performing a wound dressing, or changing an intravenous infusion that a nurse must learn to ensure safe care. However, technical competency must also sit alongside clinical judgment skills (Victor-Chmil, 2013). Tanner (2006) described clinical judgment as “an interpretation of a patient's needs, concerns or health problems, and/or the decision to take action (or not), use modifying standard approaches, or improvise new ones as deemed appropriate by the patient's response” (p. 204). Clinical judgment relies on knowledge and experience specific to the clinical context (Tanner, 2006). However, a nurse providing patient care may be technically competent and have excellent clinical judgment skills but may still make errors in their care as a result of the environment in which they are working (Gluyas & Morrison, 2013). These errors are related to the impact of environmental, physical, and emotional distractions on our working memory and can negatively affect an individual's performance. Strategies to mitigate errors related to environmental, physical, and emotional distractions are based in the recognition that improving nontechnical skills such as situation awareness (SA), teamwork, and communication can have a positive effect on performance and can mitigate errors (Flin, O'Connor, & Crichton, 2008).

The importance of nontechnical skills has been recognized in other industries, such as the aviation and oil industries, for many years. Investigations into major disasters in these industries identified that rather than deficient technical skills, it was nontechnical skills, such as a loss of SA, that were implicated in many incidents (Endsley & Jones, 2012)

SA refers to an individual's awareness, understanding, and significance of what is going on around them, which then leads to appropriate decision making and performance within the given situation (Gluyas & Morrison, 2013). In more technical terms, Endsley (1995b) defined SA as “the perception of elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future” (p. 36).

With particular reference to SA, interruptions and distractions have been demonstrated to affect performance as an individual's cognitive attention is taken away from the task at hand, leading to errors in task completion (Thomas et al., 2015). Thus, it is important that health practitioners are taught the role of nontechnical skills, including SA maintenance, in error mitigation.

Limited evidence suggests that the use of SA in clinical nursing increases the use of practices that promote patient safety and reduce decision errors (MacEachin, Lopez, Powell, & Corbett, 2009). Interprofessional learning enhances nurses' situation awareness (MacEachin et al., 2009). However, discordance in care goals between doctors and nurses can cause tension and deteriorate SA (Miller & Sanderson, 2005). Among nursing students, knowledge of SA has been identified as adequate but its maintenance was less than optimal (Cooper, Kinsman, et al., 2010).

To understand better the presence or absence of SA in nursing students, we undertook a recent study to observe final-year nursing students' pre- and posteducation on the topic of nontechnical skills in general and, more specifically, SA. This educational intervention also included strategies that could be used to support the improvement and maintenance of SA. Students were observed for both error rates and the use of SA strategies in a simulated clinical environment pre- and posteducational intervention. To identify the presence of SA in the students, a performance-based situation awareness observation schedule (PBSAOS) was developed to identify errors in completing routine clinical skills and the use of SA maintenance strategies. This article reports on the development and evaluation of the PBSAOS and the lessons learned in the process.

Measurement of Situation Awareness

Measurement of SA is challenging, and it can involve ether direct or indirect collection of information to determine SA (Cooper, Porter, & Peach, 2014). Direct measurement of SA involves using simulated scenarios and stopping the simulation at certain points and asking questions of the participants. These questions are designed to determine whether they have an understanding of what is happening at that time and the actions that need to be taken. The answers are compared with what is determined by experts as correct or incorrect responses in relation to the desired outcome for the situation, and the information that must be observed and processed to achieve the outcome (Endsley & Garland, 2000). One direct measurement method is the Situation Awareness Global Assessment Tool (Endsley, 1995a), which is considered to provide an accurate theoretical evaluation of SA but cannot necessarily be used to assess actual performance and outcome (Pritchett & Hansman, 2000). A criticism of this method is that in stopping the simulation at certain points and asking questions, individuals are then primed to focus attention on those elements when the simulation continues. Moreover, the interruption may actually disrupt the individual's SA as they return to the scenario (Endsley & Garland, 2000). To limit the effect of interruptions, the use of real-time probes built into the scenario can be used, but this may also prime individuals to focus on particular aspects of the scenario (Salmon et al., 2009).

The indirect method can be undertaken in two ways. First, the individual can be asked to self-rate their situation awareness. A Situation Awareness Rating Tool (SART) is used for this, which involves the individual rating his or her own performance against 10 dimensions of SA (Taylor, 1990). The criticism of this method is that rather than measuring the actual performance of SA, it is more likely to capture individuals' confidence in their ability to maintain SA (Jones, 2000).

The second method of indirect assessment of SA involves the individual being observed as they function in the simulation. The absence or presence of the individual's performance of expected behaviors, actions, and reactions related to the situation are then noted. Effective or noneffective SA is inferred by the results of the observation (Bell & Lyon, 2000; Cooper et al., 2014; Wright, Taekman, & Endsley, 2004). This method does not directly measure the individual's cognitive awareness of the situation, but it is still worthwhile given that an individual's awareness of the situation tends to be reflected through his or her actions (Pritchett & Hansman, 2000). The validity of this method depends on considerable preparation for a given situation or task, as there must be a detailed review prior to the observation to determine the performance-based measures that would demonstrate SA in the particular scenario being observed (Wright et al., 2004). In cases where several observers are used, additional observer training is essential.

Each measurement technique has advantages and disadvantages. Choosing the most appropriate method requires an evaluation of the resources available and the goals of the particular study. For this study, given technological constraints that precluded direct SA measurement, indirect assessment using performance-based measures was chosen to observe students' SA while undertaking tasks within a simulated clinical scenario.

Method

Performance-Based Measurement

A review was undertaken of measures that could be used to assess SA. The content of these measures is displayed in Table A (available in the online version of this article). As can be seen from Table A, the available measures assessed SA processes in general terms that could be open to broad interpretation. Consequently, in our view, there was a need to develop a measure that enquired about specific SA processes, which could possibly be more precisely assessed than the broadly phrased items in the extant SA measures.

Comparison of the Content of the Performance-Based Situation Awareness Observation Schedule with Existing Situation Awareness MeasuresComparison of the Content of the Performance-Based Situation Awareness Observation Schedule with Existing Situation Awareness Measures

Table A:

Comparison of the Content of the Performance-Based Situation Awareness Observation Schedule with Existing Situation Awareness Measures

The development of the structured performance-based observation schedule for this study was based on the work of Pritchett and Hansman (2000) and Endsley (1995b) and adapted to the clinical context. The development process involved the three key steps of scenario development, detailed task analysis for the scenario, and a review of the scenario and the task analysis by peers and clinical experts (Figure 1).

Time line of study procedures. Note. SA = situational awareness.

Figure 1.

Time line of study procedures. Note. SA = situational awareness.

Ethics

Ethical approval was obtained for this study from the Murdoch University Human Research Committee.

Scenario Development

When developing the scenario, it is vital that the participants involved have the skills and knowledge to perform the tasks (Endsley, 1995a). The participants in this study were in their final year of nursing studies and were completing a series of clinical simulations caring for postoperative patients. Thus, the simulated scenario chosen reflected the academic and clinical knowledge gained by the nursing students in previous and current semesters. The scenario required the students to provide postoperative nursing care for a patient who had returned to the ward following emergency surgery for a ruptured spleen as a result of a motorbike accident. During the course of the scenario, the patient complains of pain. The nursing student is expected to assess the patient and administer pain relief. Within several minutes of receiving pain relief, the patient complains of nausea and looks pale and sweaty. The nursing student is expected to assess and provide nursing management of the changed clinical status.

The two particular clinical tasks within the scenario that were identified for observations were:

  • Initial pain assessment and subsequent administration of intramuscular analgesics.
  • Assessment and management of the patient's changed clinical status following the administration of analgesics. Interruptions and distractions were built into the scenario.

The interruptions included a nursing assistant entering the clinical area while the nursing student was checking medication and asking the patient if he or she wanted a drink; a staff member interrupting the nursing student by asking for the medication cupboard keys; and a staff member asking the nursing student if he or she had seen or knew the location of a specific doctor. These interruptions were timed to coincide with critical parts of the specific tasks. The distractions included a timer near the bed space that would ring at intervals, intravenous pumps with alarms going off in adjoining bed spaces, and the general noise and activity of other students within the simulated clinical ward.

Task Analysis

After the scenario was developed, a detailed task analysis was undertaken to identify unambiguous performance-based actions or outcomes that would provide an indication of SA maintenance (Endsley, 1995b; Pritchett & Hansman, 2000; Wright et al., 2004). Endsley (1995b) identified three types of performance-based measures that focus on different aspects of the expected behaviors or outcomes that constitute elements of the task being observed and infer a lack of SA related to distractions or interruptions. These measures are:

  • Imbedded task measures that are expected/required actions for specific tasks.
  • External task measures that are expected/required reactions to external impacts on task completion, such as interruptions, distractions, and changed information.
  • Global measures that are measures of final performance outcome.

Two tasks were chosen for simulated observation: (a) pain assessment and administration of intramuscular analgesics and (b) subsequent assessment and management of the patient's clinical status. These were analyzed to identify actions and outcomes (global and imbedded measures) that would infer the extent to which the nursing student was maintaining SA despite the distractions and interruptions in the environment (Table 1). As well, overt SA recovery strategies were identified from the literature to provide external measures inferring that the nursing student was recovering or maintaining situation awareness (Flin et al., 2008; Gluyas & Harris, 2016; Goldenhar, Brady, Sutcliffe, & Meuething, 2013; Reason, 2004). These overt SA recovery strategies included:

  • Individual reevaluating task sequence before resuming task (individual verbalizes task sequence to self or self-checking).
  • Sterile cockpit rule (individual verbalizes to others warning to not interrupt task procedures).
  • Checks in with other teammates (individual verbalizes to others team members in the vicinity reevaluation of task sequence).
  • Huddles (individual gathers team together for overview of current situation and plan for future actions).
  • Review by peers and clinical experts.
Examples of Task Actions and Outcomes Identified as Imbedded and Global Measures Forming Part of Task Analysis for the Clinical Task Pain Assessment and Subsequent Administration of Intramuscular Pain Relief

Table 1:

Examples of Task Actions and Outcomes Identified as Imbedded and Global Measures Forming Part of Task Analysis for the Clinical Task Pain Assessment and Subsequent Administration of Intramuscular Pain Relief

A team of experienced clinicians who had recent practice in performing the tasks developed the PBSAOS. This analysis involved a review of the literature, as noted above at the outset of the methods section, and experienced clinicians identifying the psychomotor skills required to complete the tasks. The clinicians drew on the extant SA literature and clinical psychomotor skills text (Tollefson, 2012) used within the nursing school to form an initial list of actions and outcomes for the tasks being observed. The clinicians then discussed these at length until they felt that the actions and outcomes described reflected the requirements of the task accurately. Once developed, the PBSAOS was evaluated to ensure utility, clarity, and content validity. Utility referred to the presentation of the PBSAOS work-sheet to ensure that it was user friendly. Clarity referred to the descriptors of the elements of the task being clearly articulated, explicit, and easily distinguishable. Content validity referred to ensuring the performance measures were applicable and captured all aspects of the task that could be used to infer situation awareness. As well, evaluating content validity also included assessing whether the tasks that the students were expected to complete as part of the scenario were appropriate for the skill level of final-year nursing students. The evaluation was undertaken by nursing clinical supervisors who supervised students in the clinical environment and expert clinicians from a variety of clinical backgrounds who had experience in both teaching and undertaking the tasks (n = 12). The reviewers were asked to comment on the following aspects of the PBSAOS tool:

  • Do the measures reflect the expected actions for the specific tasks?
  • Do the measures reflect the expected behaviors following interruptions?
  • Do the measures reflect the expected outcomes for the specific task?
  • Are the descriptions of the expected actions, behaviors, and outcomes clearly articulated and unambiguous?
  • Are there any actions, behaviors, or outcomes that have been omitted?
  • Are the task elements of the performance measures appropriate for the experience level of third-year nursing students?
  • Does the design of the PBSAOS facilitate easy recording of the observed measures?

There were several iterations of the PBSAOS during the development of the tool in response to feedback by the reviewers, particularly in response to the expected task actions, behaviors, and outcomes. This iteration process continued until all clinicians were in unequivocal agreement about clarity and content validity. The final tool comprised 54 items, 46 of which measured the performance of clinical tasks (imbedded and global measures), and four items per task assessed the use of strategies to maintain SA (external task measures). All items were scored on a nominal scale (1 = task correctly performed or SA strategy employed; 0 = task incorrectly performed or SA strategy not used). Table B (available in the online version of this article) displays the final version of the PBSAOS.

Performance-Based Situation Awareness Observation Schedule (PBSAOS)Scenario: After primary assessment patient (Pt.) complains of surgical site pain. IMI Tramadol to be given. Pt. complains of nausea-Blood pressure has dropped, O2 saturations (sats) have dropped. Pt. requires increase of IVT, IMI anti-emetic O2 therapy and positioningPerformance-Based Situation Awareness Observation Schedule (PBSAOS)Scenario: After primary assessment patient (Pt.) complains of surgical site pain. IMI Tramadol to be given. Pt. complains of nausea-Blood pressure has dropped, O2 saturations (sats) have dropped. Pt. requires increase of IVT, IMI anti-emetic O2 therapy and positioningPerformance-Based Situation Awareness Observation Schedule (PBSAOS)Scenario: After primary assessment patient (Pt.) complains of surgical site pain. IMI Tramadol to be given. Pt. complains of nausea-Blood pressure has dropped, O2 saturations (sats) have dropped. Pt. requires increase of IVT, IMI anti-emetic O2 therapy and positioning

Table B:

Performance-Based Situation Awareness Observation Schedule (PBSAOS)

Scenario: After primary assessment patient (Pt.) complains of surgical site pain. IMI Tramadol to be given. Pt. complains of nausea-Blood pressure has dropped, O2 saturations (sats) have dropped. Pt. requires increase of IVT, IMI anti-emetic O2 therapy and positioning

The observers who underwent a comprehensive 7-hour education session prior to the use of the tool for observations were also asked to provide feedback on the PBSAOS tool in terms of utility, clarity, and content validity. Feedback from the observers following the pre-education observations resulted in some minor changes in the layout and design of the worksheet, although the content of the form was unchanged. In addition, observers requested some reordering of the task checklist to better reflect the workflow of the tasks. Following the conclusion of the posteducation observations, the observers provided feedback confirming the utility, clarity, and content validity of the tool. The only exception to this was one observer who felt unable to comment about whether the tasks were reflective of the level of experience of final-year nursing students. Further, one observer suggested a separate checklist column for the video review. This should be included for future use of the tool.

Cohen's kappa was used to assess the interrater reliability of the observers in using the PBSAOS tool. The results of analysis demonstrated that kappa ranged from 0.41 to 0.60 for five observations, indicating moderate agreement; 0.61 to 0.80 for 16 observations, indicating substantial agreement; and 0.81 to 1.0 for six observations, indicating near perfect agreement. Overall, the findings showed that the PBSAOS is a reliable measure for SA.

Discussion

Lessons Learned

The goal in developing the PBSAOS tool was to produce an instrument that accurately captured errors in elements of task performance that would infer the presence or absence of SA in students who were taking part in a simulated clinical scenario, as well as the use of strategies to maintain SA. The following sections detail the primary lessons learned during the development, review, and use of the PBSAOS, and we also highlight further directions for research.

Ensure That the Clinicians Have Requisite Knowledge and Understanding of the Concepts. The clinicians involved in the development and review of the PBSAOS and the observers who were to use the PBSAOS were, as stated previously, nursing experts in their clinical fields in the delivery of clinical care or teaching clinical care theory and skills. However, initial discussions with the nursing clinicians revealed varying levels of knowledge and understanding of the concepts of error theory, nontechnical skills, and in particular SA. On reflection, this was not unexpected because although these concepts have been studied in other fields for many years, they are relatively new in terms of the application to nursing practice (Fore & Sculli, 2013; Gluyas & Harris, 2016; Sitterding, Broome, Everette, & Ebright, 2012; Stubbings, Chaboyer, & McMurray, 2012). Two strategies were implemented to accommodate the differing levels of knowledge and understanding. First, one-to-one information sessions were held between the researcher and the nursing clinicians. The second strategy involved a 7-hour theoretical session covering the topics of patient safety, error theory, nontechnical skills with an emphasis on SA, and the measurement of SA with an emphasis on performance-based measurement. In addition, the observers were able to practice using the PBSAOS using a prerecorded video of a simulated clinical scenario.

The PBSAOS Worksheet Needs a User-Friendly Design. The initial design of the PBSAOS observer worksheet was a portrait orientation with the imbedded, external, and global measures for one task on the same page. However, after using the PBSAOS tool in practice workshops, feedback from the observers was that the form was cumbersome to use. As well, the sequencing of measures on the form did not necessarily match the order in which the students undertook the task. As a result, the layout of the form was changed from portrait to landscape orientation. There was also some minor reordering of the listing of the action measures into the separate groups of imbedded, external, and global measures and some reordering of the task action measures on the worksheet so they matched the order in which the tasks were performed in a more logical way. The change from portrait to landscape orientation also resulted in more space for observers to make comments, which the observers reported was useful to quickly note comments or questions to follow up. The change in paper orientation also meant that there was an increase in the number of worksheet pages. The researcher was concerned that this might be difficult for the observers to manage. However, the observers did not report that this was the case.

The Performance Measures Needed to Be Clear and Unambiguous to Ensure Interrater Reliability. Initial review of tools that were in use as performance measures identified several methods of recording observation when assessing nontechnical skills including SA. These included Likert scales, checklists, and frequency counts (Andersen, Jensen, Lippert, Ostergaard, & Klausen, 2010; Cooper, Cant, et al., 2010; Dietz et al., 2014; Flin & Martin, 2001; Mishra, Catchpole, & McCulloch, 2009). Likert scale behavioral tools require observers to rate the individuals' behavior according to a predetermined scale on how well the tasks are being performed. However, the behavioral measures for these observations were developed from the task analysis and represented the expected unambiguous proceduralized actions to complete the task. The focus then was not on how well the task was performed but rather that each of the action or outcome was performed. A checklist format with dichotomous options was thus the most suitable format for the PBSAOS tool. In addition, this had the added advantage that it lessened the cognitive load of the observers.

However, the decision to develop a checklist tool had some challenges, as all those involved (developers, reviewers, and observers) were required to focus on the presence or absence of the performance based SA task measures and not on whether the task action was being performed correctly. This is an unusual shift for the nurse clinicians whose focus is principally on technical clinical competence (technical skills) rather than assessing nontechnical skills (Flin et al., 2008; Gluyas & Morrison, 2013). The theoretical education sessions provided opportunities to explore and discuss examples and to add clarity to the difference in technical and nontechnical skills. These discussions emphasized the importance of both technical and nontechnical skills for safe and quality patient care while stressing that focus for this research project was to assess SA, not technical skill competence. Thus, the requirement was to identify actions and outcomes that unambiguously described elements of the task that infer SA in the individual, rather than clinical competence. The focus then was on clear, readily observed descriptors of the each of the behavior requirements (Tables 12).

Examples of External Measures Forming Part of Task Analysis for Pain Assessment

Table 2:

Examples of External Measures Forming Part of Task Analysis for Pain Assessment

The Value of Video Recording to Confirm Observations

Each simulated scenario was video recorded to enable interrater reliability testing to be undertaken. This proved to be invaluable in assessing interrater reliability because the second observer could carefully review the videos and verify the veracity of the first observer's notes. In the future video recording, the simulations should form an integral part of the planning for performance based measurement of nontechnical skills.

Future Research Opportunities

The tool presented in this study could be used in several ways in future research to further understand how SA educational interventions may influence nurses' performance of clinical tasks. For instance, comparative studies involving differing SA interventions could be undertaken to determine which intervention may be more effective. It would also be worthwhile to undertake longitudinal studies that assess whether SA incrementally increases when SA interventions are provided at regular intervals throughout nursing students' training. Finally, it would be valuable to conduct controlled studies that examine the specific effect that SA may have on reducing errors in clinical tasks.

Limitations

Some caveats should be considered in the interpretation of the findings. The lack of a control group precluding determination of whether the changes in error tasks owed to the specific effect of the SA educational intervention or if the changes may have been associated with other factors, such as sensitization during clinical placements. Also, the face-to-face contact component of the educational intervention was relatively brief and it may be the case that a greater number of in-person training hours are required to more effectively embed SA skills. Also, as part of the educational intervention, we also used e-mail reminders to reinforce SA skills but it is unclear whether the students opened or fully read these e-mails. Finally, the sample size was relatively small and limited to a single institution. It would therefore be beneficial to evaluate the PBSAOS in other nursing settings to consolidate its psychometric properties.

Conclusion

Contemporary nursing curricula have little instruction in the area of nontechnical skills, including SA. However, given the increasing recognition of the crucial role accurate SA plays in the minimization of errors, there is likely to be an increase in knowledge and skills education in this area. This will require ways to identify the absence or presence of SA in individuals while delivering nursing care. The PBSAOS tool developed for the current research provides one way of assessing the skills in the clinical environment. The information gained in the development and use of this tool will be valuable in the development of other performance-based measurement systems in both simulated and actual clinical environments.

References

  • Andersen, P.O., Jensen, M.K., Lippert, A., Ostergaard, D. & Klausen, T.W. (2010). Development of a formative assessment tool for measurement of performance in multi-professional resuscitation teams. Resuscitation, 81, 703–711. doi:10.1016/j.resuscitation.2010.01.034 [CrossRef]
  • Bell, H. & Lyon, D. (2000). Using observer ratings to assess situation awareness. In Endsley, M. & Garland, D. (Eds.), Situation awareness: analysis and measurement. New York, NY: CRC Press.
  • Cooper, S., Cant, R., Porter, J., Sellick, K., Somers, G., Kinsman, L. & Nestel, D. (2010). Rating medical emergency teamwork performance: Development of the Team Emergency Assessment Measure (TEAM). Resuscitation, 81, 446–452. doi:10.1016/j.resuscitation.2009.11.027 [CrossRef]
  • Cooper, S., Kinsman, L., Buykx, P., McConnell-Henry, T., Endacott, R. & Scholes, J. (2010). Managing the deteriorating patient in a simulated environment: Nursing students' knowledge, skill and situation awareness. Journal of Clinical Nursing, 19, 2309–2318. doi:10.1111/j.1365-2702.2009.03164.x [CrossRef]
  • Cooper, S., Porter, J. & Peach, L. (2014). Measuring situation awareness in emergency settings: A systematic review of tools and outcomes. Open Access Emergency Medicine, 6, 1–7. doi: http://dx.doi.org/10.2147/OAEM.S53679
  • Dietz, A., Pronovost, P., Benson, K., Mendez-Tellez, P., Dwyer, C., Wyskiel, R. & Rosen, M. (2014). A systematic review of behavioural marker systems in healthcare: What do we know about their attributes, validity and application?BMJ Quality & Safety, 23, 1031–1039. doi:10.1136/bmjqs-2013-002457 [CrossRef]
  • Endsley, M. (1995a). Measurement of situation awareness in dynamic systems. Human Factors, 37, 65–84. doi:10.1518/001872095779049499 [CrossRef]
  • Endsley, M. (1995b). Toward a theory of situation awareness in dynamic systems. Human Factors, 37, 32–64. doi:10.1518/001872095779049543 [CrossRef]
  • Endsley, M. & Garland, D. (Eds.). (2000). Situation awareness: Analysis and measurement. New York, NY: CRC Press.
  • Endsley, M. & Jones, D. (2012). Designing for situation awareness: An approach to user-centered design (2nd ed.). Boca Raton, FL: CRC Press.
  • Flin, R. & Martin, L. (2001). Behavioral markers for crew resource management: A review of current practice. The International Journal of Aviation Psychology, 11(1–2), 95–118. doi:10.1207/S15327108IJAP1101_6 [CrossRef]
  • Flin, R., O'Connor, P. & Crichton, M. (2008). Safety at the sharp end: A guide to non-technical skills. Boca Raton, FL: CRC Press.
  • Fore, A. & Sculli, G. (2013). A concept of situation awareness in nursing. Journal of Advanced Nursing, 69, 2613–2621. doi:10.1111/jan.12130 [CrossRef]
  • Gluyas, H. & Harris, S. (2016). Understanding situation awareness and its importance in patient safety. Nursing Standard, 30(34), 50–58. doi:10.7748/ns.30.34.50.s47 [CrossRef]
  • Gluyas, H. & Morrison, P. (2013). Patient safety: An essential guide. London, United Kingdom: Palgrave Macmillan. doi:10.1007/978-1-137-31632-5 [CrossRef]
  • Goldenhar, L., Brady, P., Sutcliffe, K. & Meuething, S. (2013). Huddling for high reliability and situation awareness. BMJ Quality & Safety, 22, 899–906. doi:10.1136/bmjqs-2012-001467 [CrossRef]
  • Jones, D. (2000). Subjective measures of situation awareness. In Endsley, M. & Garland, D. (Eds.), Situation awareness: Analysis and measurement. New York, NY: CRC Press.
  • MacEachin, S.R., Lopez, C.M., Powell, K.J. & Corbett, N.L. (2009). The fetal heart rate collaborative practice project: Situational awareness in electronic fetal monitoring—A Kaiser Permanente Perinatal Patient Safety Program Initiative. Journal of Perinatal & Neonatal Nursing, 23, 314–323. doi:10.1097/JPN.0b013e3181a1bf07 [CrossRef]
  • Miller, A. & Sanderson, P. (2005). Clinical information use by medical and nursing staff in the ICU: Outcomes of a coded tables analysis. Proceedings of the Human Factors and Ergonomics Society, 49, 984–988. doi:10.1177/154193120504901130 [CrossRef]
  • Mishra, A., Catchpole, K. & McCulloch, P. (2009). The Oxford NOTECHS system: Reliability and validity of a tool for measuring teamwork behaviour in the operating theatre. Quality & Safety in Health Care, 18, 104–108. doi:10.1136/qshc.2007.024760 [CrossRef]
  • Pritchett, A. & Hansman, R. (2000). Use of testable responses for performance-based measurement of situation awareness. In Endsley, M. & Garland, D. (Eds.), Situation awareness: Analysis and measurement. New York, NY: CRC Press.
  • Reason, J. (2004). Beyond the organisational accident: The need for “error wisdom” on the frontline. Quality & Safety in Health Care, 13, 28–33. doi:10.1136/qshc.2003.009548 [CrossRef]
  • Salmon, P., Stanton, N., Walker, G., Jenkins, D., Ladva, D., Rafferty, L. & Young, M. (2009). Measuring situation awareness in complex systems: Comparison of measures study. International Journal of Industrial Ergonomics, 29, 490–500. doi:10.1016/j.ergon.2008.10.010 [CrossRef]
  • Sitterding, M.C., Broome, M.E., Everett, L.Q. & Ebright, P. (2012). Understanding situation awareness in nursing work: A hybrid concept analysis. Advances in Nursing Science, 35, 77–92. doi:10.1097/ANS.0b013e3182450158 [CrossRef]
  • Stubbings, L., Chaboyer, W. & McMurray, A. (2012). Nurses' use of situation awareness in decision-making: An integrative review. Journal of Advanced Nursing, 68, 1443–1453. doi:10.1111/j.1365-2648.2012.05989.x [CrossRef]
  • Tanner, C. (2006). Thinking like a nurse: A research-based model of clinical judgement in nursing. Journal of Nursing Education, 45, 204–210.
  • Taylor, R.M. (1990). Situation awareness rating technique (SART): The development of a tool for aircrew system design. In Salas, E. (Ed.), Situational awareness (pp. 3/1–3/17). Neuilly-Sur-Seine, France: NATO-AGARD.
  • Thomas, I., Nicol, L., Regan, L., Cleland, J., Maliepaard, D., Clark, L. & Duncan, J. (2015). Driven to distraction: A prospective controlled study of a simulated ward round experience to improve patient safety teaching for medical students. BMJ Quality & Safety, 24, 154–161. doi:10.1136/bmjqs-2014-003272 [CrossRef]
  • Tollefson, J. (2012). Clinical psychomotor skills (5th ed.). South Melbourne, Australia: Cengage Learning.
  • Victor-Chmil, J. (2013). Critical thinking versus clinical reasoning versus clinical judgment: Differential diagnosis. Nurse Educator, 18, 34–36. doi:10.1097/NNE.0b013e318276dfbe [CrossRef]
  • Wright, M.C., Taekman, J.M. & Endsley, M.R. (2004). Objective measures of situation awareness in a simulated medical environment. BMJ Quality & Safety, 13, i65–i71. doi:10.1136/qshc.2004.009951 [CrossRef]

Examples of Task Actions and Outcomes Identified as Imbedded and Global Measures Forming Part of Task Analysis for the Clinical Task Pain Assessment and Subsequent Administration of Intramuscular Pain Relief

Pain Assessment and Administration of Pain Relief (Imbedded Measures)Tick If ObservedComments
Uses Pain, Quality, Radiation, Site, and Timing to assess need for pain relief
Notes observation of pain score
Reviews medication chart for pain relief medication orders
Notes and discusses charted antiemetic order
  Checks right patient
  Checks right route
  Checks right drug
  Checks right dose
  Checks right time
Outcome Measures (Global Measures)Tick If ObservedComments
Documents pain relief administration on medication chart

Examples of External Measures Forming Part of Task Analysis for Pain Assessment

Strategies to Improve/Maintain Situational Awareness (External Measures)Tick If ObservedComments
Verbalizes to others warning to not interrupt task procedures
Verbalizes self-checking

Comparison of the Content of the Performance-Based Situation Awareness Observation Schedule with Existing Situation Awareness Measures

ItemOutcome Measure
ANTSENNTSGAOTPHFRS-MNIMPAcTNoTechsOttawa GRSSPLINTSPBSAOS
Verbalizes to others warning to not interrupt taskx
Checks with others team members/or supervisor if unsure of next actionsx
Team discusses current situation and/or plans action/sx
Verbalizes self-checkingx
Gathering informationxxx
Recognizing and understandingxxx
Anticipatingxxx
Impending problems not recognized and there is a slow response despite clear urgency of the situationx
There is a delay in calling for helpx
Team members are slow to recognize their limitations related to knowledge, skills, and abilityx
Team members focus on one event to the exclusion of what is going on around themx
Team members' attention driftsx
Monitored patient's parameters throughout the procedurex
Awareness of anesthetistx
Actively initiates communication with anesthetist during crisis periodsx
Being aware of the situation in your immediate departmentx
Being aware of the situation in the whole hospitalx
Being aware of facilities available in other departmentsx
Being aware of the situation in the deployment areax
Considers all elements/monitor vital signs and progress of operation/ asks for or shares information/encourages vigilance/checks and reports changes/requests reports and updatesx
Cross-checks/shares mental models/speaks up when unsure/updates other team membersx
Identifies future problems/ discusses contingencies/ plans for future states/anticipates high workload/discusses constraints/uses low workload periodx
Becomes fixated easily despite repeated cuesx
Fails to reassess and reevaluate the situation despite repeated cuesx
Fails to anticipate likely eventsx

Performance-Based Situation Awareness Observation Schedule (PBSAOS)

Scenario: After primary assessment patient (Pt.) complains of surgical site pain. IMI Tramadol to be given. Pt. complains of nausea-Blood pressure has dropped, O2 saturations (sats) have dropped. Pt. requires increase of IVT, IMI anti-emetic O2 therapy and positioning

Tick if observedComments
Pain Assessment & Administration of pain relief
Uses PQRST to assess need for pain relief (I)
Notes observation of pain score (I)
Reviews medication chart for pain relief medication orders (I)
Notes & discusses charted anti–emetic order (I)
  Checks right patient (I)
  Checks right route (I)
  Checks right drug (I)
  Checks right dose (I)
  Checks right time (I)
Confirms medication checks with second checker (I)
Checks for allergies (I)
Performs hand hygiene pre procedure (I)
Administers IMI analgesic in correct anatomical location (I)
  Alco wipe
  Draws back
Informs patient of side effects (I)
Provides call bell (I)
Disposes of sharp properly (I)
Performs hand hygiene post procedure (I)
Strategies to improve/ maintain situational awareness (SA)
Verbalizes to others warning to not interrupt task procedures (E)
Checks with others team members / or supervisor if unsure of next actions (E)
Team discusses current situation and/ or plans action/s (E)
Verbalizes self checking (E)
Outcome measures
Documents pain relief administration on medication chart (G)
Assessment & Management of Patient Changed Clinical Status
Performs hand hygiene pre procedure (I)
Assesses Pt. condition when alerted to problems post IMI pain relief (I)*
ABC - notes observations (I)
Commences either Hudson mask @ 6 litres per minute or nasal prongs @ 3 litres per minute O2 therapy (I)
Lowers Pt head for hypotension recovery (I)
Discusses with MO using ISOBAR communication (I)
Increases IVT rate as per orders (I)
Reviews antiemetic order in medication chart (I)
  Checks right patient (I)
  Checks right route (I)
  Checks right drug (I)
  Checks right dose (I)
  Checks right time (I)
Confirms medication checks with second checker (I)
Checks for allergies (I)
Performs HH pre procedure (I)
Administers IMI anti-emetic in correct anatomical position (I)
Alco wipe (I)
Draws back (I)
Disposes of sharp appropriately (I)
Performs hand hygiene post procedure (I)
Strategies to improve/maintain SA
Verbalizes to others warning to not interrupt task procedures (E)
Checks with others team members/or supervisor if unsure of next actions (E)
Team discusses current situation and/or plans action/s (E)
Verbalizes self-checking (E)
Outcome measures
Documents ADD score on obs chart (G)
Documents O2 therapy intervention on obs chart (G)
Documents IVT intervention on obs chart(G)
Documents IVT intervention on Fluid Balance (G)
Documents anti-emetic administration on medication chart(G)
Authors

Dr. Gluyas is Honorary Research Fellow, Dr. Stomski is Senior Health Research Officer, Ms. Andrus is Senior Lecturer, Dr. Walters is Lecturer, Dr. Morrison is Professor of Health Research, Dr. Williams is Professor of Health Research, Dr. Hopkins is Senior Lecturer, and Ms. Sandy is Lecturer, College of Health, Science, Education and Engineering, Murdoch University, Murdoch, Western Australia, Australia.

The authors have disclosed no potential conflicts of interest, financial or otherwise.

Address correspondence to Norman Jay Stomski, PhD, Senior Health Research Officer, College of Health, Science, Education and Engineering, Murdoch University, 90 South Street, Murdoch, Western Australia 6150, Australia; e-mail: N.Stomski@murdoch.edu.au.

Received: November 15, 2018
Accepted: April 30, 2019

10.3928/01484834-20190719-06

Sign up to receive

Journal E-contents