Cognitive Walk-Through and Think-Aloud Sessions
Five themes and a list of six design recommendations were identified from cognitive walk-through/think-aloud sessions. Usability test sessions lasted between 28 and 36 minutes, with an average duration of 33 minutes. Identified themes were: Overall Usability and Usefulness, Usefulness as Memory Aid for Evidence-Based Practice, Usefulness as a Training Tool, Usefulness as a Communication Tool, and Potential to Improve Workflow.
Overall Usability and Usefulness. There was overall positive agreement regarding the usability and potential usefulness of the UTIDecide app. A typical reaction to the app during usability testing is exemplified by the following quote from Participant 3: “Overall, I think it is great.”
Usefulness as a Memory Aid for Evidence-Based Practice. Participants stated the importance of upholding evidence-based practice. However, current practice requires that they rely on their own memory without the benefit of memory aids or best practice reminders. One example of app usefulness is the provision of reminders and recommendations to comply with evidence-based guidelines, as Participant 3 stated: “I think that it is really helpful because it tells me when I should get a UA [urinalysis] and when I shouldn't and recommends based on the evidence-based practice which is what we aim to always follow.” Participant 1 noted the usefulness of the app to look up relevant clinical information: “It does give me some things to consider reading, which are nice to kind of jog your memory.”
Usefulness as a Training Tool. Participants noted the usefulness of UTIDecide as a potential training tool to raise awareness about relevant information and practice guidelines. Those in nurse manager roles were enthusiastic about its use for new nurses and CNAs, as Participant 1 stated:
I think that this may help to develop that critical thinking piece that nurses don't get, especially LPNs in 2 years, so I think that really helps as well. It might get them to say, “Hey, I better gather this information up according to what it says.”
The app was also seen as useful as a supporting resource to find necessary information or information that is not commonly used:
It could come in handy because when I was fresh out of school, if I didn't know something or didn't have a resource available to me, then I would call my fellow nurses that I went to nursing school with.
That is great. Some people don't know what dysuria is. I don't care if you've been a nurse for 20 years, there is going to be some stuff you don't know.
Usefulness as a Communication Tool. UTIDecide was identified as useful for communications with on-site team members, off-site providers, and residents. For on-site team members, it could facilitate communication in the following way: “The recommendations of ‘continue close observation’ and ‘begin oral hydration’… that would be something that I would want to pass along to the CNA,” as noted by Participant 1. In addition, UTIDecide was seen as a useful tool to empower nursing staff in communication with off-site providers:
Any change of conditions. If you are going to contact the doctor you should have the SBAR there ready to give a quick report.
You can say “hey, this is what I am following”…. It just makes you sound like you understand what you're doing just a little bit more, and again you've got a guideline to follow that gives you more than we're just going to test because the family says so.”
The app could also be used as a tool to engage residents in communications about their own health conditions, as Participant 6 noted:
It can even help the residents because they will be asking “What is going on with me?” and you can just give them your ideas, like it could be this or that, but the doctor might order a urinary sample or something. I would love to use this.
Potential to Improve Workflow. Participants broadly agreed that UTIDecide could improve workflow efficiency by reducing time spent on the assessment process and looking up information due to high rates of smartphone adoption:
Any time you use the smartphone would be easier or quicker. Once you get used to it you know what they're asking on the paper and you just check, check, check. But on the smartphone it goes faster.
I feel like this app is quicker to access rather than having to go and get a binder.
Everybody is so attached to their phone. I think they would do it faster on the phone than faster on a piece of paper.
Design Recommendations From Cognitive Walk-Throughs. The following six design recommendations were identified from cognitive walk-through/think-aloud sessions.
Provide feature to save and print output from assessment process.
Harmonize app language with facility language (e.g., Fahrenheit versus Celsius).
Simplify language and improve clarity of presentation in the information dialog boxes displayed after tapping the “i” icon information. For example, “functional decline” and “leukocytosis” were presented together on a single icon, which introduced confusion about whether both were required to check a checklist item as “true.” In addition, some dialog box language read as jargon because text was copied directly from guidelines.
Provide supporting language in the SBAR regarding the evidence-based guidelines upon which the app is based to give nurses confidence in communicating SBAR with providers.
Reorganize and cluster display of pain symptoms under a single group as options for selection during assessment process. Originally, pain symptoms were spread throughout the app workflow.
Link to electronic health record (EHR).
Design Recommendations 1–5 were implemented in the app version used by field test participants (described below). Design Recommendation 6 would involve substantial development efforts to integrate the app with institutional EHR systems and was beyond the scope and resources of the project.
Field Test Results
Most participants indicated that they would use the app in their practice if it were available. Participants agreed that the app fit their current workflow. Specifically, they noted that computer stations are often full so the app would improve access to online resources. In addition, they noted that general online searches often return non-relevant results and an app could deliver targeted results that streamline processes. Using an example identified in the field test (described below), a nurse who suspected a respiratory infection could use a mobile CDS function that implemented guidance specific to the suspected condition rather than conducting an internet search that returned thousands of results from indexed web pages.
Based on participant estimates of UTI frequency at their facility, the 1-week duration of the field test period was insufficient to capture incidents of suspected UTIs. Participants identified 1 to 6 months as a more appropriate field testing period. However, despite the short field test periods, two examples of real-world use emerged. The first example confirmed earlier findings from cognitive walk-through/think-aloud sessions that UTIDecide could be used as a communication tool with on-site team members. This scenario occurred when a field test participant who suspected a UTI showed the app to a nurse practitioner as they discussed the patient's symptoms. In the second example, a field test participant used the app as a resource to support her clinical judgement in an unintended way by entering symptoms associated with a suspected upper respiratory infection. Although UTIDecide is only designed for assessment of UTI symptoms, it offers possible alternatives for further assessment efforts if symptoms are not consistent with UTI. The participant wanted to see if the app would suggest this as a possible alternative for assessment, which it did on the recommendations screen.
Participants identified specific barriers to app implementation and use. All participants noted that mobile phone use was restricted by facility rules for staff although there was flexibility if phone use was for work purposes or an emergency. In addition, Wi-Fi access at the facility was often unreliable.
Design Recommendations from Field Tests. Two design recommendations were identified from field tests of UTIDecide. Design Recommendation 1 was implemented in a new version of UTIDecide, whereas design Recommendation 2 was beyond the scope of the current project:
Enhance print feature for options to print to PDF and printer.
Include features to allow for assessment of additional conditions.
System Usability Scale Results. The average score for all three field test participants who completed the SUS was 92.5 (Table 2). This average SUS score equates to an “A” grade for usability and is above the point (80.3) at which a user would recommend the app to a friend (Sauro, 2011).
System Usability Scale (SUS) Results