Educational materials available through the internet have become an increasingly popular resource for patients to learn about their medical conditions.1 In 2009, a survey conducted by the Pew Research Center reported that 61% of adults use the internet regularly to obtain health care information.2 To meet this demand and aid in informed decision making, providers, national academies, and specialty societies have begun providing online education materials for patients. Although the electronic availability of these materials should equate with increased health literacy nationwide, this correlation has yet to be established. In fact, previous studies across a variety of specialties indicate that there is a mismatch between the comprehension level of patients and the level at which patient education materials are currently written.3–11
Given that health outcomes are intimately tied to health literacy, it is critical that the readability level of patient education materials is in accordance with national guidelines to meet the needs of the average patient.12 The National Assessment of Adult Literacy, released in 2006 by the US Department of Education, found that only 12% of consumers have proficient health literacy skills. Moreover, the US population, on average, can read at the level of a 7th- or 8th-grade student. As a result, the National Institutes of Health (NIH) and the American Medical Association (AMA) recommend that patient education materials be written between the 4th- to 6th-grade reading level.13,14
The purpose of this study was to determine the readability of patient education materials for musculoskeletal oncologic conditions. These conditions are rare, with an incidence of 0.9 per 100,000 per year in the United States.15 The readability of patient education materials has been examined across numerous orthopedic specialties, finding materials to be written at a grade level significantly higher than that of the average patient.12,16–23 Patient education materials from the American Academy of Orthopaedic Surgeons (AAOS) related to orthopedic oncology are limited.24,25 As a result, many of the available materials in this specialty come from a variety of sources. The authors hypothesized that educational materials available online that relate to bone and soft tissue tumors would be written at a considerably higher literacy level.
Materials and Methods
Data on 28 preselected orthopedic oncology conditions were obtained. Patient education materials from the first 10 websites pertaining to each of these conditions were selected for analysis. The websites were identified using the Google search engine without any preset exclusion criteria. In total, patient education materials from 240 (85.7%) websites were obtained with the initial desired total of 280 being unachieved due to articles failing to meet selection criteria. Articles were ineligible to be recorded for analysis due to 1 or more of the following reasons: (1) the article was written by a source including Wikipedia or other reference website from which authorship was a collaboration of anonymous users citing various other sources, thus making true authorship impossible to ascertain; (2) the search term yielded articles that were specifically designed for nonpatient audiences, including medical students, physicians, physician assistants, nurse practitioners, nurses, hospital administrators, or researchers; (3) the indexed sites led to pages that were not text based; (4) the indexed site led to a medical dictionary; and (5) the indexed site was an entry into an academic journal article that was not designed for patient education. Once selected, all articles were saved as Word (Microsoft Corp) documents, as text-based–only entries. These documents were then analyzed using readability assessment scales via the Readability Studio software (Oleander Software Solutions).
The patient education materials were evaluated using the Coleman-Liau,26 Flesch-Kincaid,27 FORCAST,28 Fry,29 Gunning Fog,30 Raygor Estimate,31 and SMOG32 assessment scales (Table A, available in the online version of the article). According to these scales, the mean±SD reading levels (grade levels) of all articles were 11.76±1.27, 10.7±1.19, 11.22±0.43, 13.07±2.21, 12.31±1.08, 11.58±2.00, and 12.97±1.85, respectively (Table B, available in the online version of the article). Across all 7 of these scales, the mean grade level of these patient resources was found to be written at a level nearly double the NIH and AMA recommendation: 11.9±0.6. Materials for soft tissue chondromas were written at the highest level (14.8±1.9), and those for chordomas (10.1±1.0) most closely approached national recommendations. However, these were still written at a readability level more than 4 grade levels higher than has been recommended by the NIH and AMA. The materials were also analyzed with the Flesch Reading Ease assessment, which is scored out of 100, with lower numbers indicating more complex texts. This scale provided a mean score of 46.5±7.7, which corresponds to a “difficult to read” result, indicating a mid-college level of education or higher to understand the text (Figure 1).
Readability Assessment Scales
Readability Scores of Online Patient Education Materials for Benign, Malignant, and Metastatic Bone Tumors
Comparison of mean readability of benign, malignant, and metastatic bone tumors. Mean readability derived from Coleman-Liau,26 Flesch-Kincaid,27 FORCAST,28 Fry,29 Gunning Fog,30 Raygor Estimate,31 and SMOG32 assessment scales. The national recommendation is at or below a 6th-grade reading level (horizontal line).
According to the National Adult Literacy Survey, nearly half of the US population is either functionally illiterate or marginally illiterate. Functionally illiterate is defined as having the ability to read at a 5th-grade reading level or lower, whereas marginally illiterate is defined as having the ability to read between a 6th- and 8th-grade reading level.33 Health literacy is defined as the degree to which individuals have the capacity to obtain, process, and understand basic health information and services needed to make appropriate health decisions.12 Low health literacy hinders a person's ability to locate health care services and self-manage acute and chronic health conditions. In fact, health literacy has been touted as the single best predictor of health care status.12
The objective of this study was to examine the readability of patient education materials for the most common orthopedic oncology conditions, which has never been previously assessed. This analysis revealed that patients had to have a 12th-grade education to fully comprehend the majority of materials from these websites. This is approximately double the NIH and AMA recommendations. None of the patient education materials for any single condition were written at an average at or below the 6th-grade reading level. Given patients' reliance on these materials within the clinical setting, further modification of these resources is warranted to ensure adequate comprehension and informed decision making.
The results of this study are consistent with existing literature examining patient education materials in orthopedic oncology. Based on a previous analysis conducted by Shah et al,24 patient education materials provided by the AAOS, American Cancer Society, Bone and Cancer Foundation, and National Cancer Institute were written at a 10th-grade reading level on average, with the Bone and Cancer Foundation materials exceeding an 11th-grade reading level. Although prior analyses of patient education material readability in orthopedic oncology are scant, this trend is present among other subspecialties in orthopedics, including, but not limited to, sports medicine, arthroplasty, trauma, hand, and arthroscopy.18,24,34–36
Improvements in readability of patient education materials will allow patients to have a better understanding of their conditions and risks of various treatments and procedures. It will also decrease the likelihood of medical malpractice claims, because patients will be better informed regarding their health care decisions.33 The readability of these materials can be improved in several ways. The readability scales used in this study were created based on algorithms that analyze word complexity, sentence structure, and length. Avoiding the use of polysyllabic words is essential to improve readability. Although the authors acknowledge that the majority of these conditions have polysyllabic names, which will therefore unavoidably increase the reading level, they are doubtful that these names can fully account for a reading level nearly twice that of the national recommendation. Other techniques for compensating for these polysyllabic names include using simple, concise sentences and avoiding the use of medical jargon. Additionally, using visual aids such as charts, graphs, and illustrations can supplement the textual information to make the materials more coherent.33 These tactics have been previously supported in the development of medicinal package leaflets as well as in other specialties.37
Although this analysis highlights an incongruity between efforts to educate patients and the materials used to do so, there are certain limitations to this study. First, many patients used websites that did not meet the inclusion criteria for this analysis. Analyses in other specialties found that information on Wikipedia may sometimes be superior to professional organization websites.38 In fact, it is one of the most frequently used medical resources globally.39 To the authors' knowledge, no study has broadly examined the readability of information available on Wikipedia relating to orthopedic oncology. One study conducted by Leithner et al40 examined completeness and validity of the material on Wikipedia and found that it was inferior to the US National Cancer Institute. However, they did not use the readability scales used in the current study, which leaves the authors with a direction for future research. Finally, due to the nature of this study, there is a demographic sampling bias in that it only includes patients who have access to a computer. Health literacy is associated with socioeconomic status to some degree, and it is therefore possible to argue that the baseline literacy of patients with regular internet access is greater than that of the general US population. Therefore, the functionally illiterate and marginally illiterate may be underrepresented in interpreting these data.
Ensuring that patients with musculoskeletal oncologic conditions are well informed is critical to helping improve surgical, medical, and mental health outcomes. Due to increasing patient reliance on the internet as a resource to learn about their health, the authors assessed the readability of patient education materials in orthopedic oncology. Currently, the average readability of patient education materials is at nearly a 12th-grade reading level, compared with the 6th- to 8th-grade level readability recommended by the AMA and NIH. The hope of the authors is that this investigation prompts clinicians and associated institutions to revise online patient educations materials to facilitate more widespread understanding of bone and soft tissue cancers for patients and family members.
- Hesse BW, Greenberg AJ, Peterson EB, Chou WS. The Health Information National Trends Survey (HINTS): a resource for consumer engagement and health communication research. Inf Serv Use. 2017;37(2):205–218. doi:10.3233/ISU-170830 [CrossRef]
- The social life of health information. Pew Research Center. Accessed January 2019. https://www.pewresearch.org/internet/2009/06/11/the-social-life-of-health-information
- De Oliveira GS Jr, Jung M, Mccaffery KJ, McCarthy RJ, Wolf MS. Readability evaluation of internet-based patient education materials related to the anesthesiology field. J Clin Anesth. 2015;27(5):401–405. doi:10.1016/j.jclinane.2015.02.005 [CrossRef] PMID:25912728
- Agarwal N, Hansberry DR, Sabourin V, Tomei KL, Prestigiacomo CJ. A comparative analysis of the quality of patient education materials from medical specialties. JAMA Intern Med. 2013;173(13):1257–1259. doi:10.1001/jamainternmed.2013.6060 [CrossRef] PMID:23689468
- Sheppard ED, Hyde Z, Florence MN, McGwin G, Kirchner JS, Ponce BA. Improving the readability of online foot and ankle patient education materials. Foot Ankle Int. 2014;35(12):1282–1286. doi:10.1177/1071100714550650 [CrossRef] PMID:25239196
- Agarwal N, Sarris C, Hansberry DR, Lin MJ, Barrese JC, Prestigiacomo CJ. Quality of patient education materials for rehabilitation after neurological surgery. NeuroRehabilitation. 2013;32(4):817–821. doi:10.3233/NRE-130905 [CrossRef] PMID:23867407
- Huang G, Fang CH, Agarwal N, Bhagat N, Eloy JA, Langer PD. Assessment of online patient education materials from major ophthalmologic associations. JAMA Ophthalmol. 2015;133(4):449–454. doi:10.1001/jamaophthalmol.2014.6104 [CrossRef] PMID:25654639
- Eloy JA, Li S, Kasabwala K, et al. Readability assessment of patient education materials on major otolaryngology association websites. Otolaryngol Head Neck Surg. 2012;147(5):848–854. doi:10.1177/0194599812456152 [CrossRef] PMID:22864405
- Vargas CR, Koolen PG, Chuang DJ, Ganor O, Lee BT. Online patient resources for breast reconstruction: an analysis of readability. Plast Reconstr Surg. 2014;134(3):406–413. doi:10.1097/PRS.0000000000000472 [CrossRef] PMID:25158700
- Hansberry DR, Kraus C, Agarwal N, Baker SR, Gonzales SF. Health literacy in vascular and interventional radiology: a comparative analysis of online patient education resources. Cardiovasc Intervent Radiol. 2014;37(4):1034–1040. doi:10.1007/s00270-013-0752-6 [CrossRef] PMID:24482028
- Pruthi A, Nielsen ME, Raynor MC, Woods ME, Wallen EM, Smith AB. Readability of American online patient education materials in urologic oncology: a need for simple communication. Urology. 2015;85(2):351–356. doi:10.1016/j.urology.2014.10.035 [CrossRef] PMID:25623686
- Badarudeen S, Sabharwal S. Assessing readability of patient education materials: current role in orthopaedics. Clin Orthop Relat Res. 2010;468(10):2572–2580. doi:10.1007/s11999-010-1380-y [CrossRef] PMID:20496023
- Kutner M, Greenburg E, Jin Y, Paulsen C. The Health Literacy of America's Adults: Results from the 2003 National Assessment of Adult Literacy. National Center for Education Statistics; 2006.
- National Institutes of Health. How to write easy to read health materials. Accessed January 2019. https://www.nih.gov/institutes-nih/nih-office-director/office-communications-public-liaison/clear-communication/clear-simple
- National Cancer Institute Surveillance, Epidemiology, and End Results Program. Cancer stat facts: colorectal cancer. Accessed September 16, 2020. https://seer.cancer.gov/statfacts/html/colorect.html
- Polishchuk DL, Hashem J, Sabharwal S. Readability of online patient education materials on adult reconstruction web sites. J Arthroplasty. 2012;27(5):716–719. doi:10.1016/j.arth.2011.08.020 [CrossRef] PMID:22000573
- Vives M, Young L, Sabharwal S. Readability of spine-related patient education materials from subspecialty organization and spine practitioner websites. Spine. 2009;34(25):2826–2831. doi:10.1097/BRS.0b013e3181b4bb0c [CrossRef] PMID:19910867
- Wang SW, Capo JT, Orillaza N. Readability and comprehensibility of patient education material in hand-related web sites. J Hand Surg Am. 2009;34(7):1308–1315. doi:10.1016/j.jhsa.2009.04.008 [CrossRef] PMID:19596175
- Sabharwal S, Badarudeen S, Unes Kunju S. Readability of online patient education materials from the AAOS web site. Clin Orthop Relat Res. 2008;466(5):1245–1250. doi:10.1007/s11999-008-0193-8 [CrossRef] PMID:18324452
- Badarudeen S, Sabharwal S. Readability of patient education materials from the American Academy of Orthopaedic Surgeons and Pediatric Orthopaedic Society of North America web sites. J Bone Joint Surg Am. 2008;90(1):199–204. doi:10.2106/JBJS.G.00347 [CrossRef] PMID:18171975
- Yi PH, Chang MM, Haughom BD, Jawa A. Readability of patient education materials from the AAHS. Hand (N Y).2014;9(3):393–394. doi:10.1007/s11552-014-9643-9 [CrossRef] PMID:25191174
- Ganta A, Yi PH, Hussein K, Frank RM. Readability of sports medicine-related patient education materials from the American Academy of Orthopaedic Surgeons and the American Orthopaedic Society for Sports Medicine. Am J Orthop (Belle Mead NJ).2014;43(4):E65–E68. PMID:24730006
- Eltorai AE, Sharma P, Wang J, Daniels AH. Most American Academy of Orthopaedic Surgeons' online patient education material exceeds average patient reading level. Clin Orthop Relat Res. 2015;473(4):1181–1186. doi:10.1007/s11999-014-4071-2 [CrossRef] PMID:25475715
- Shah AK, Yi PH, Stein A. Readability of orthopaedic oncology-related patient education materials available on the internet. J Am Acad Orthop Surg. 2015;23(12):783–788. doi:10.5435/JAAOS-D-15-00324 [CrossRef] PMID:26519430
- Patel SS, Sheppard ED, Siegel HJ, Ponce BA. Assessing the reading level of online sarcoma patient education materials. Am J Orthop (Belle Mead NJ).2015;44(1):E1–E10. PMID:25566558
- Coleman EB, Hahn SC. Failure to improve readability with a vertical typography. J Appl Psychol. 1966;50(5):434–436. doi:10.1037/h0023855 [CrossRef] PMID:5918829
- Kincaid JP, Fishburne RP Jr, Rogers RL, Chissom BS. Derivation of new readability formulas (Automated Readability Index, Fog Count, and Flesch Reading Ease Formula) for Navy enlisted personnel. Accessed January 2019. https://apps.dtic.mil/sti/pdfs/ADA006655.pdf
- Caylor JS, Sticht TG, Fox LC, Ford JP. Methodologies for Determining Reading Requirements of Military Occupational Specialties. Human Resources Research Organization; 1973.
- Fry E. A readability formula that saves time. J Read. 1968;11:513–578.
- Gunning R.The Technique of Clear Writing. McGraw-Hill; 1952.
- Raygor AL. The Raygor readability estimate: a quick and easy way to determine difficulty. In Pearson PD, ed. Reading: Theory, Research, and Practice (26th Yearbook of the National Reading Conference). National Reading Conference; 1977.
- McLaughlin GH. SMOG grading: a new readability formula. J Read. 1969;12:639–646.
- Gupta R, Adeeb N, Griessenauer CJ, et al. Evaluating the complexity of online patient education materials about brain aneurysms published by major academic institutions. J Neurosurg. 2017;127(2):278–283. doi:10.3171/2016.5.JNS16793 [CrossRef] PMID:27540903
- Yi MM, Yi PH, Hussein KI, Cross MB, Della Valle CJ. Readability of patient education materials from the web sites of orthopedic implant manufacturers. J Arthroplasty. 2017;32(12):3568–3572. doi:10.1016/j.arth.2017.07.003 [CrossRef] PMID:28750856
- Hodax JD, Baird GL, McBride T, Owens BD. Readability of online sources regarding meniscal tears. J Knee Surg. 2017;30(7):712–717. doi:10.1055/s-0036-1598009 [CrossRef] PMID:28038475
- Wang D, Jayakar RG, Leong NL, Leathers MP, Williams RJ, Jones KJ. Evaluation of the quality, accuracy, and readability of online patient resources for the management of articular cartilage defects. Cartilage. 2017;8(2):112–118. doi:10.1177/1947603516648737 [CrossRef] PMID:28345406
- Pires C, Vigário M, Cavaco A. Readability of medicinal package leaflets: a systematic review. Rev Saude Publica. 2015;49(0):4. doi:10.1590/S0034-8910.2015049005559 [CrossRef] PMID:25741660
- Garcia M, Daugherty C, Ben Khallouq B, Maugans T. Critical assessment of pediatric neurosurgery patient/parent educational information obtained via the internet. J Neurosurg Pediatr. 2018;21(5):535–541. doi:10.3171/2017.10.PEDS17177 [CrossRef] PMID:29451451
- Heilman JM, West AG. Wikipedia and medicine: quantifying readership, editors, and the significance of natural language. J Med Internet Res. 2015;17(3):e62. doi:10.2196/jmir.4069 [CrossRef] PMID:25739399
- Leithner A, Maurer-Ertl W, Glehr M, Friesenbichler J, Leithner K, Windhager R. Wikipedia and osteosarcoma: a trustworthy patients' information?J Am Med Inform Assoc.2010;17(4):373–374. doi:10.1136/jamia.2010.004507 [CrossRef] PMID:20595302
Readability Assessment Scales
|Readability Assessment Scale||Algorithm*|
|Flesch-Kincaid||(0.39 x A) + (11.8 x B) – 15.59|
|Coleman-Liau||((0.0588 x C) – (0.296 x D)) – 15.8|
|Gunning Fog||0.4 x ((B ÷ E) + 100 (H ÷ B))|
|Raygor Estimate||100-word passage selected from sample
No. of sentences calculated
No. of words with ≥6 letters calculated
Readability calculated using variables|
|FORCAST||20- (F ÷ 10)|
|Fry||100-word passage selected from sample
No. of sentences calculated
No. of syllables calculated
Readability calculated using variables|
|Flesch Reading Ease||206.835 – ((1.015 x A) – (84.6 x B))|
|SMOG||1.043 x √ (G x (30/E)) + 3.1291|
Readability Scores of Online Patient Education Materials for Benign, Malignant, and Metastatic Bone Tumors
|Tumor||Coleman-Liau||Flesch-Kincaid||Forcast||Fry||Gunning Fog||Raygor Estimate||Smog||Flesh Reading Ease||Avg.||Std. Dev.|
|Giant Cell Tumor||11.22||10.58||11.22||11.78||12.14||10.44||12.58||49.60||11.42||0.79|
|Pigmented Villonodular Synovitis||11.06||9.89||10.93||11.28||11.22||9.50||11.96||50.70||10.83||0.85|
|Primary Bone Lymphoma||11.00||9.98||11.15||10.00||11.58||9.67||11.73||50.75||10.73||0.83|
|Soft Tissue Chondroma||14.90||13.00||12.00||17.00||14.75||17.00||14.75||31.00||14.77||1.86|
|Soft Tissue Sarcoma||11.31||10.53||11.06||12.22||12.09||11.78||12.62||49.70||11.66||0.73|
|Unicameral Bone Cyst||10.26||9.80||10.15||10.00||11.72||9.33||12.02||56.40||10.47||1.01|