The Internet is now a ubiquitous tool to access health-related information, and many patients use online resources to learn about their health and health treatments. Unfortunately, many health-related Web sites are poorly maintained or unregulated, and many health care professionals are concerned about the accuracy of these resources (Grewal & Alagaratnam, 2013; Kaicker et al., 2010). Because clinicians provide patients more information from the Internet (Eysenbach, Sa, & Diepgen, 1999), they have the duty to verify the accuracy of this information (Dee & Stanley, 2005). Previous research (Scott, Gilmour, & Fielden, 2008) has uncovered little evidence of nurses using Internet-based information to improve patient outcomes, which may result from a lack of confidence of, and knowledge about, properly using online resources. Therefore, this study examined how knowledge of online evaluation tools can enhance an undergraduate nursing student's ability to critique health care information Web sites and, in turn, increase his or her capacity to provide better patient care.
Currently, many patients engage health care with a preconceived idea of what is causing their illness because they found health-related information online. Although Internet-based health care information can be rather unreliable, patients continue to gravitate toward this source to obtain information because of its convenience, variety, and anonymity, compared with consulting a clinician (Kaicker et al., 2010). Approximately 59% of Internet users are searching for health-related information, and 6.75 million medical searches are executed on Google™ each day (Grewal & Alagaratnam, 2013). This concerns many health care professionals because patients could be making decisions concerning health and medical treatment based on incorrect and unreliable resources. A systematic review of 79 studies that appraised 5,941 Web sites hosting health-related information (Eysenbach & Kohler, 2002) revealed that 70% of the studies reported issues with the quality of information. More recent studies (Grewal & Alagaratnam, 2013; Grewal, Williams, Alagaratnam, Neffendorf, & Soobrah, 2012) have revealed the same problem. Because of the questionable veracity of online information, some health care professionals and hospital centers are developing their own health information Web sites to monitor and control the health information available to patients or anyone else (Eysenbach et al., 1999; Lam, Roter, & Cohen, 2013).
Health care professionals are responsible for ensuring that patients receive accurate health information to make informed decisions. Taha, Sharit, and Czaja (2009) demonstrated that too much irrelevant or inaccurate information obtained by patients from nonhealth care provider sources results in burdens for health care professionals. The reason for this is because these professionals must cope with the resultant misconceptions of the patients and educate them on the true state of their conditions, which affects the provision of quality health care. Concerns are growing that online information is not only difficult to interpret (Czaja, Sharit, & Nair, 2008) but also false, misleading, or incomplete (Eysenbach & Powell, 2002). For example, Grewal and Alagaratnam (2013) reported that only 14 of the 88 colorectal cancer Web sites that were evaluated were certified with the Health on the Net code—an identification of ethical and trustworthy health related information on the Web (Health on the Net Foundation, n.d.). In addition, Eysenbach and Köhler (2002) not only found that many patients show poor evaluation skills (e.g., not identifying the author, source of information, or homepage of a Web site) but also provide a telling example: in six cases, German participants searching for legal abortion information used an Austrian or Swiss Web site, and only half of the participants realized that the information was not applicable in Germany.
Provision 7 of the American Nursing Association (ANA) Code of Ethics (2011) states that “the nurse participates in the advancement of the profession through contributions to practice, education, administration, and knowledge development” (p. 22). Therefore, with the Internet now a ubiquitous health information resource, nurses are obliged to teach patients and health consumers about proper online health information retrieval. Schools of nursing can help nurses achieve this by increasing their awareness of Web site evaluation tools, such as the HONcode. This awareness—and the information and behaviors related to it—can be provided through targeted curricula and continuing education. For example, Scott et al. (2008) asserted that preparation for the nursing practice of the 21st century requires the integration of skills such as searching for, evaluating, and sharing online health information in the context of nursing practice. However, few studies have examined the abilities of nursing students concerning health information Web literacy. To address this shortcoming, the current study examined how undergraduate nursing students seek health information and how their information-seeking behavior changed after being provided instruction to improve their literacy of health care information Web sites.
Participants were undergraduate nursing students recruited from a required undergraduate nursing informatics course offered in a school of nursing affiliated with a major research university between January and April 2015.
The study protocol was approved by the university's institutional review board. Participants had the option to take a survey associated with an in-class activity that was presented in the informatics course, and their participation had no affect on their grade in the course or any associated courses in the school. The participants were presented with a health Web site that was selected by the instructor for critique and provided a preactivity survey to collect their perceptions of critiquing the Web site. The students then participated in a 60-minute activity that featured Web site literacy and online tools for Web evaluation. After the activity, the students were presented with another health Web site, were asked to evaluate it using an evaluation tool taught in class, and then completed a postactivity survey, which was the same as the preactivity survey.
Description of the Web Site Literacy Activity
The Web site literacy activity modeled reliable health information Web sites and presented tools to evaluate them, such as Evaluating Health Websites (National Network of Libraries of Medicine, n.d.), HONcode: Principles—Quality and Trustworthy Health Information (HONcode, 2013), and the Site, Publisher, Audience, Timeliness [SPAT] Website Evaluation Tool (LaRue, 2013). In-class discussion activities and homework tasks provided feedback on and practice for using the evaluation tools.
The pre- and postactivity survey not only queried demographic information, current nursing program status, and the health information search habits of the participants, but also assessed their (a) perceived knowledge of and confidence in critiquing an online health Web site and (b) knowledge of available tools for evaluating a health Web site. The postactivity homework assignment was scored with a rubric that evaluated the completeness by which the students used the critiquing criteria introduced in the activity. The resultant score reflects the students' Web site literacy independent from the self-report.
Descriptive statistics were used to analyze survey results concerning demographic information, current nursing program status, online health information used, and for whom health information searches are performed. A Wilcoxon signed-rank test analyzed changes in perceived knowledge of health information Web site critique, confidence in finding reliable health information online, and knowledge online evaluation tools. A Mann-Whitney U test was used to compare differences between male and female students in these areas. ANOVA was performed to examine the group differences of postknowledge on assignment performance. All statistical analyses were conducted with an a priori significance level of .05.
One hundred twenty-nine of the undergraduate students in the informatics course received the invitation to participate in the study, and 116 agreed to participate. Most identified as traditional undergraduate (n = 96, 82.8%), freshman (n = 94, 81%), women (n = 102, 87.9%), and White (n = 100, 86.2%). Their average age was 19.7 years (SD = 3.98).
Student Information-Seeking Behaviors for Health Information Web Sites
Our survey results reveal that participants actively searched for health information on the Internet—Web sites being the primary source (77.6%) from which the participants seek health information; in addition, participants searched for this information for themselves (76.7%), family members (10.3%), and friends (6%). In addition, 84 (75%) participants reported that it was easy to find health information on the Internet, and 105 (90.5%) thought that health information from the Internet was somewhat reliable. The majority (89.7%) of participants had no awareness that existing tools were available for evaluating a specific health information Web site.
Results of the Wilcoxon Signed Ranks Test and Mann-Whitney U Test
The participants' knowledge of and confidence about Web literacy increased after the intervention activity. The number of participants with no knowledge of available evaluation tools decreased from 89.7% to 12.1% (Figure 1). In addition, an increase was observed in not only the perceived knowledge of evaluating a health information Web site (p = .005) (Figure 2) but also in the perceived confidence in finding reliable health information on the Internet (p = .058) (Figure 3).
Number of known tools among students to evaluate health information Web sites. Note. NLM = National Library of Medicinel; HON = health on the net.
Students' perceived knowledge of critiquing a health information Web site.
Students' confidence in finding reliable health information on the Internet.
In addition, a Mann-Whitney U test was conducted to examine the differences between male and female students concerning (a) perceived knowledge of critiquing a health information Web site, (b) confidence in finding reliable health information on the Web site, and (c) knowledge of existing online evaluation tools to evaluate health information Web sites. Before the activity, male student participants (n = 10) were perceived as having a higher level of knowledge (p = .032) about critiquing a health information Web site, compared with female student participants (n = 98). In addition, the average score of confidence in critiquing a health information Web site among male participants was higher than that of female participants (i.e., 3.0 versus 2.60) (Table). In terms of the student participants' performance on the postactivity homework assignment, ANOVA revealed no statistically significant overall difference (or between-group difference) (p = .526) between the homework assignment scores and the self-reported scores.
Descriptive Statistics of the Score for Confidence of Critique, Perceived Knowledge, and Number of Known Evaluation Tools Between Male and Female Students and Pre-Post Wilcoxon Signed Ranks Test Results
We found that 77% of the participants commonly use the Internet to seek health information—for not only themselves, but also for family and friends. Only a decade ago, Dee and Stanley (2005) reported that nursing students were most likely to rely on colleagues (37.5%) and books (25%) for medical information on a daily basis, whereas only 12% used the Internet. This shift in reliance should not be surprising, given the increased use of and reliance on the Internet over the past 10 years. Nonetheless, this shift to using the Internet as a primary information resource among U.S. nursing students necessitates that university-level nursing informatics courses offered in schools of nursing provide opportunities to enable students to effectively critique and use health information found—and referred to—on the Internet.
A fundamental component of health information Web site literacy is understanding how to evaluate these Web sites. Although 50% of the participants felt confident in their competency to evaluate a Web site before the intervention activity, 89.7% acknowledged having no knowledge of existing health Web site evaluation tools. This is worrisome because evaluation tools offer systematic ways to determine the trustworthiness of health information provided online. Although the confidence level of the participants in evaluating a health Web site did not change in a statistically significant fashion after the activity, their perceived knowledge of evaluating a health information Web site did indeed increase with statistical significance (p = .005). This may be a result of the participants' overestimation of their confidence and skills to critique a Web site before the activity. In light of this, the addition of health Web site evaluation tools—and their influence on health Web site literacy—in the nursing informatics curriculum should be considered, as providing teaching and learning opportunities to cope with Internet-based health information currently is not a standardized feature in nursing curricula.
Limitations and Implications
Although this study demonstrates the effectiveness of a training session on the Web literacy of nursing students concerning the evaluation of health information Web sites, it does have several limitations. First, the participant sample was limited to undergraduate nursing students enrolled in one required nursing informatics course at a university-based school of nursing. The majority of these students were freshman, so the results are not representative of all U.S. nursing students—both in terms of year in a program and the program itself (e.g., BSN versus associate degree nursing). Nonetheless, the sample reasonably represents a typical state of Web literacy for nursing students in a baccalaureate program—one that could certainly benefit from targeted training as provided in our Web literacy activity. Health care professionals need to continuously expand their knowledge and update the knowledge of evidence-based practice. Web sites are a common source for health care professionals to search for health information. Therefore, the early integration of Web literacy education into a BSN program supports student learning and helps students adapt to the evolving practice environment.
Second, the results indicate that male students possessed better Internet literacy at baseline than did female students; however, given the small number of these male students, further research is needed to verify this result. In addition, our measure of the students' perceived knowledge may not represent their actual ability to critique a health Web site. This knowledge construct is complex, and more sensitive instrumentation must be identified and deployed in future studies of nursing students' evidence-based Web literacy.
Third, the participants' self-perceived knowledge of accessing reliable Web-based health information increased after the activity. Because our postactivity assignment to assess their ability to evaluate Web-based health information was inconclusive, a preactivity assignment—in addition to the postactivity assignment—is necessary to assess any change in knowledge about critiquing a health information Web site vis-à-vis the participants' self-reports. In addition, extending the range of student participants to include graduate and international nursing students would help foster a broader understanding of nursing students' Web literacy in terms of not only prior nursing training but also cultural context.
The provision of Web site literacy education enables nursing students, as future health care providers, to develop the skills necessary to not only evaluate health information Web sites and the information hosted on them but also determine which Web sites provide the most usable, reliable, and trustworthy health care information. The ability to recognize appropriate Web sites enables these students to recommend valid and accurate resources to patients and other health care professionals. The opportunity to learn and practice using evidence-based evaluation tools to find reliable online health Web sites and information allows nursing students to deliver (a) improved patient education and (b) safer patient care against the wave of health care information that is—and will continue to be—provided on the Internet. Increasing students' ability to identify accurate health information requires both massed learning and frequent practice. To operationalize this learning, nursing students constantly must be required to search information for completing individual and group assignments throughout the curriculum for long-term mastery. For example, for written assignments, students can cite reliable evidence from multiple sources (e.g., Web sites and journals), which can result in higher quality papers. Infusing the curriculum with Web-literacy training will enable students to locate more accurate health-related information that will improve not only their learning experience but also patient education.
- American Nurses Association. (2011). Code of ethics. Silver Spring, MD: Author. Retrieved from http://nursingworld.org/DocumentVault/NursingPractice/CodeofEthicsforNursesexcerpt.pdf
- Czaja, S.J., Sharit, J. & Nair, S.N. (2008). Usability of the Medicare health web site. JAMA, 300, 790–792. doi:10.1001/jama.300.7.790-b [CrossRef]
- Dee, C. & Stanley, E.E. (2005). Information-seeking behavior of nursing students and clinical nurses: Implications for health sciences librarians. Journal of the Medical Library Association, 93, 213–222.
- Eysenbach, G. & Köhler, C. (2002). How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews. BMJ (Clinical Research Ed.), 324, 573–577. http://doi.org/10.1136/bmj.324.7337.573 doi:10.1136/bmj.324.7337.573 [CrossRef]
- Eysenbach, G. & Powell, J. (2002). Empirical studies assessing the quality of health information for consumers on the world wide web: A systematic review. JAMA, 287, 2691–2700. doi:10.1001/jama.287.20.2691 [CrossRef]
- Eysenbach, G., Sa, E.R. & Diepgen, T.L. (1999). Shopping around the internet today and tomorrow: Towards the millennium of cyber-medicine. British Medical Journal, 319, 1294. http://doi.org/10.1136/bmj.319.7220.1294 doi:10.1136/bmj.319.7220.1294 [CrossRef]
- Grewal, P. & Alagaratnam, S. (2013). The quality and readability of colorectal cancer information on the internet. International Journal of Surgery, 11, 410–413. http://doi.org/10.1016/j.ijsu.2013.03.006 doi:10.1016/j.ijsu.2013.03.006 [CrossRef]
- Grewal, P., Williams, B., Alagaratnam, S., Neffendorf, J. & Soobrah, R. (2012). Quality of vascular surgery web sites on the internet. Journal of Vascular Surgery, 56, 1461–1467. http://doi.org/10.1016/j.jvs.2012.04.058 doi:10.1016/j.jvs.2012.04.058 [CrossRef]
- National Network of Libraries of Medicine. (n.d.). Evaluating health websites. Retrieved from https://nnlm.gov/professional-development/topics/health-websites
- Health On the Net Foundation. (n.d.) The HON code of conduct for medical and health web sites (HONcode). Retrieved from https://www.healthonnet.org/HONcode/Conduct.html
- Kaicker, J., Debono, V.B., Dang, W., Buckley, N. & Thabane, L. (2010). Assessment of the quality and variability of health information on chronic pain websites using the DISCERN instrument. BMC Medicine, 8, 59. http://doi.org/10.1186/1741-7015-8-59 doi:10.1186/1741-7015-8-59 [CrossRef]
- Lam, C.G., Roter, D.L. & Cohen, K.J. (2013). Survey of quality, readability, and social reach of websites on osteosarcoma in adolescents. Patient Education and Counseling, 90, 82–87. http://doi.org/10.1016/j.pec.2012.08.006 doi:10.1016/j.pec.2012.08.006 [CrossRef]
- LaRue, E.M. (2013). SPAT: Website evaluation tool. Retrieved from http://www.spat.pitt.edu/research.php
- Scott, S.D., Gilmour, J. & Fielden, J. (2008). Nursing students and Internet health information. Nurse Education Today, 28, 993–1001. http://doi.org/10.1016/j.nedt.2008.06.002 doi:10.1016/j.nedt.2008.06.002 [CrossRef]
- Taha, J., Sharit, J. & Czaja, S. (2009). Use of and satisfaction with sources of health information among older internet users and nonusers. The Gerontologist, 49, 663–673. http://doi.org/10.1093/geront/gnp058 doi:10.1093/geront/gnp058 [CrossRef]
Descriptive Statistics of the Score for Confidence of Critique, Perceived Knowledge, and Number of Known Evaluation Tools Between Male and Female Students and Pre-Post Wilcoxon Signed Ranks Test Results
|Item||Gender||Total (M ± SD)||p Value|
|Male (M ± SD)||Female (M ± SD)|
| Prelecture||3.00 ± .67||2.63 ± .74||2.67 ± .73||.058|
| Postlecture||3.13 ± .99||2.78 ± .66||2.83 ± .70|
| Prelecture||3.00 ± .67||2.49 ± .74||2.55 ± .75||.005*|
| Postlecture||2.88 ± .84||2.76 ± .70||2.80 ± .72|
| Prelecture||1.00 ± .00||1.13 ± .57||1.12 ± .54||.000**|
| Postlecture||3.70 ± .95||3.37 ± 1.16||3.39 ± 1.15|