Orthopedics

Feature Article 

Accuracy of Athletic Trainer and Physician Diagnoses in Sports Medicine

Nicholas J. Lombardi, BS; Bradford Tucker, MD; Kevin B. Freedman, MD, MSCE; Luke S. Austin, MD; Brandon Eck, DO; Matthew Pepe, MD; Fotios P. Tjoumakaris, MD

Abstract

It is standard practice in high school athletic programs for certified athletic trainers to evaluate and treat injured student athletes. In some cases, a trainer refers an athlete to a physician for definitive medical management. This study was conducted to determine the rate of agreement between athletic trainers and physicians regarding assessment of injuries in student athletes. All high school athletes who were injured between 2010 and 2012 at 5 regional high schools were included in a research database. All patients who were referred for physician evaluation and treatment were identified and included in this analysis. A total of 286 incidents met the inclusion criteria. A total of 263 (92%) of the athletic trainer assessments and physician diagnoses were in agreement. In the 23 cases of disagreement, fractures and sprains were the most common injuries. Kappa analysis showed the highest interrater agreement in injuries classified as dislocations and concussions and the lowest interrater agreement in meniscal/labral injuries and fractures. In the absence of a confirmed diagnosis, agreement among health care providers can be used to infer accuracy. According to this principle, as agreement between athletic trainers and physicians improves, there is a greater likelihood of arriving at the correct assessment and treatment plan. Athletic trainers are highly skilled professionals who are well trained in the evaluation of athletic injuries. The current study showed that additional training in identifying fractures may be beneficial to athletic trainers and the athletes they treat. [Orthopedics. 2016; 39(5):e944–e949.]

Abstract

It is standard practice in high school athletic programs for certified athletic trainers to evaluate and treat injured student athletes. In some cases, a trainer refers an athlete to a physician for definitive medical management. This study was conducted to determine the rate of agreement between athletic trainers and physicians regarding assessment of injuries in student athletes. All high school athletes who were injured between 2010 and 2012 at 5 regional high schools were included in a research database. All patients who were referred for physician evaluation and treatment were identified and included in this analysis. A total of 286 incidents met the inclusion criteria. A total of 263 (92%) of the athletic trainer assessments and physician diagnoses were in agreement. In the 23 cases of disagreement, fractures and sprains were the most common injuries. Kappa analysis showed the highest interrater agreement in injuries classified as dislocations and concussions and the lowest interrater agreement in meniscal/labral injuries and fractures. In the absence of a confirmed diagnosis, agreement among health care providers can be used to infer accuracy. According to this principle, as agreement between athletic trainers and physicians improves, there is a greater likelihood of arriving at the correct assessment and treatment plan. Athletic trainers are highly skilled professionals who are well trained in the evaluation of athletic injuries. The current study showed that additional training in identifying fractures may be beneficial to athletic trainers and the athletes they treat. [Orthopedics. 2016; 39(5):e944–e949.]

Participation in sports among secondary school students has been increasing steadily for 20 years.1 During the 2012–2013 academic year, an estimated 7.7 million students participated in organized high school sports and approximately 1.4 million injuries occurred.1,2 When an athlete is injured, several health professionals collaborate to form a health care team. In 64% of secondary schools, a certified athletic trainer is a member of the health care team.3 In the secondary school setting, the role of the athletic trainer is to offer on-site assessment, provide appropriate immediate and long-term care to the injured athlete, and determine if and when an athlete is safe to return to play.

Injuries take a physical and mental toll on young athletes, and proper treatment is critical to their well-being.4,5 Reducing the chance of reinjury requires proper initial assessment, subsequent diagnosis, and an effective rehabilitation plan. Misdiagnosed injuries can result in improper return-to-play guidelines and repeat or worsening injury. No study has investigated the accuracy of assessment by athletic trainers in a secondary school setting across all sports and all pathologies. Because student participation in high school sports is growing, the accuracy of the assessment of athletic trainers is increasingly critical to the well-being of young athletes. The accuracy of the assessment of injuries by an athletic trainer can be evaluated directly by comparing the assessment of the athletic trainer with the correct diagnosis, as determined by a physician. However, this type of direct comparison applies only to injuries that can be diagnosed definitively (ie, fractures, ligament tears). For most other injuries in the young athlete, a diagnosis cannot always be confirmed or definitively identified. An earlier study showed that, when a gray area exists, consensus among health professionals can be used to infer the accuracy of assessment.6 The current study evaluated agreement among in-field assessments provided by athletic trainers and in-office diagnoses by physicians to infer the accuracy of the evaluation by the athletic trainer. The accuracy of assessments was evaluated in an effort to determine areas where accuracy can be increased through education and specific training to better coordinate athlete care.

Materials and Methods

Institutional review board approval was obtained. From January 2010 to June 2012, the athletic training staff at a regional health care system maintained a prospective injury database. All athletes from 5 high school athletic programs who were injured during that time were included in the database. All patients who were initially assessed by an athletic trainer and were referred to a physician for evaluation were included in this analysis. The athletic trainer's assessment, the referred physician's diagnosis, the physician's specialty, and subsequent surgical interventions were recorded. The treating athletic trainer was not identified in the data.

Specific findings by the athletic trainer and physician and concordance were analyzed and determined by review of medical records. Three investigators (N.J.L., B.E., F.P.T.) determined concordance until a consensus was reached. All injuries were ultimately categorized and classified according to the physician's diagnosis. The investigators categorized the athletic trainer's assessment through review of medical records and athletic trainer assessments. Most athletic trainers, including the athletic trainers working in the study health care system, do not use International Classification of Diseases, Ninth Revision, codes when assessing injuries. For this reason, categorizing athletic trainer assessments based on these codes and comparing the assessments directly with the codes assigned by physicians would not have allowed an adequate comparison.7 Therefore, the authors reviewed medical records to arrive at the assessment that each provider made.

Concordance was defined as agreement between athletic trainer assessment and physician diagnosis. When the athletic trainer assessment and the physician diagnosis were not in agreement, the authors attempted to determine whether the difference was the result of differences in terminology or differences in assessment of the injury, and a consensus was reached. Once it was determined that the athletic trainer assessment and the physician diagnosis were different, the incident was categorized based on the physician's diagnosis. This may not always have represented a correct diagnosis; however, it was necessary to choose 1 provider as the point of reference for the investigation.

Fleiss kappa analysis was performed to determine interrater agreement between the athletic trainer and the physician in specific diagnosis categories. Interrater agreement was determined overall and for each category.

Results

Analysis included all injuries that occurred during high school sporting events between 2010 and 2012 and resulted in evaluation by an athletic trainer and subsequently by a physician. Of the 14,699 total reported contacts between athletes and athletic trainers, 286 (1.9%) met the study criteria. Injuries occurred in 8 categories. Incidents were categorized based on the physician's in-office diagnosis. The frequency of each injury is shown in Table 1. The frequency of referral to each specialty is shown in Table 2.


Frequency of Injury by Group

Table 1:

Frequency of Injury by Group


Concordance by Health Care Provider Specialty

Table 2:

Concordance by Health Care Provider Specialty

Of the 286 incidents, the athletic trainer and the physician agreed 263 times, for a concordance of 92.0%. Overall, interrater agreement between athletic trainers and physicians was high (kappa=0.907). Table 3 shows concordance and interrater agreement between the athletic trainer and the physician in each injury group. Interrater agreement was highest in the dislocation (kappa=1.0) and concussion (0.99) groups and lowest in the meniscal/labral injury (0.76) and fracture (0.84) groups. To identify injuries that were difficult to assess, group-by-group analysis of both the athletic trainer's assessment and the physician's diagnosis was necessary. Of the 23 discordant incidents, the physician identified 9 sprains, 4 inflammation injuries, 3 fractures, 3 contusions, 2 meniscal/labral injuries, 1 concussion, and 1 strain. During initial evaluation of the 23 discordant incidents, the athletic trainer's initial assessment included 10 fractures, 3 contusions, 2 inflammation injuries, 2 sprains, 1 meniscal injury, 1 dislocation, 1 strain, and 2 incidents that did not fit into any category (Table 4).


Certified Athletic Trainer and Physician Concordance by Group

Table 3:

Certified Athletic Trainer and Physician Concordance by Group


Review of Discordant Certified Athletic Trainer Evaluations and Physician Diagnoses

Table 4:

Review of Discordant Certified Athletic Trainer Evaluations and Physician Diagnoses

In 13 of the 23 discordant incidents, either the physician's diagnosis or the athletic trainer's assessment was a fracture. In 10 of those 13 incidents, the athletic trainer's on-field assessment was a fracture, but subsequent physician in-office evaluation resulted in the following diagnoses: sprain (6 cases), contusion (3 cases), and concussion (1 case). In the remaining 3 of the 13 discordant incidents involving fractures, the athletic trainer's initial evaluation was sprain (2 cases) and an incident that did not fit into any category (1 case). After in-office evaluation, the physician deemed the injury a fracture. In 11 of the 23 discordant incidents, either the athletic trainer's assessment or the physician's diagnosis was a sprain. In 2 of the 11 incidents, the athletic trainer's on-field assessment was a sprain. After in-office evaluation, in both cases, the physician diagnosed a fracture. In the remaining 9 of 11 discordant incidents involving sprains, the athletic trainer's initial evaluation included fracture (6 cases), contusion (1 case), dislocation (1 case), and meniscal tear (1 case). After in-office evaluation, the physician identified the injury as a sprain in these instances. The remaining groups in the analysis did not contribute to discordant incidents as frequently. The number of discordant incidents involving either athletic trainer evaluation or physician diagnosis from the other groups included inflammation (6 cases), contusion (6 cases), meniscal/labral injury (3 cases), strain (2 cases), concussion (1 case), and dislocation (1 case).

In 140 of the 286 incidents, the athletic trainer referred the injured athlete to an orthopedic physician. Orthopedic surgeons addressed 97 of the 140 referrals and agreed with the athletic trainer in 87 of 97 cases. Nonoperative orthopedic physicians addressed 22 of the referrals and agreed with the athletic trainer in 20 of 22 cases. The remaining 21 orthopedic referrals were made to private orthopedic practices without specifying the treating physician. The remaining 146 of 286 injured athletes were referred to physicians in other specialties. Referral patterns and agreement between athletic trainers and physicians from all specialties are listed in Table 2.

Discussion

Athletic trainers are highly qualified health professionals with several areas of expertise that facilitate the provision of fast, accurate, and thorough care to injured athletes. Athletic trainers employed at secondary schools are responsible for treating injuries that occur at practices and competitive events and often are the first health professionals to care for injured athletes. In this study, in some cases, the athletic trainer's assessment was unclear or required a physician's evaluation. In these cases, the athlete was referred to a physician who, with the aid of imaging modalities, provided a diagnosis and treatment options. Given the increase in the number of high school students who are participating in sports and the large number of injured athletes who are treated by athletic trainers, the accuracy of their assessment is vital.

It can be difficult to determine the accuracy of an athletic trainer's assessment or a physician's diagnosis. For instance, how can the diagnosis be correctly determined in a truly ambiguous case? A physician or an athletic trainer cannot be right in every case. Although the discussion has been framed relative to the athletic trainer, this is only because the athletic trainer performs assessments without benefit of diagnostic imaging and other modalities that allow accurate diagnosis and treatment. Therefore, it made sense to begin the analysis with the on-field assessment relative to the in-office examination instead of the opposite. Bernstein et al6 showed that agreement among health care professionals could be used to imply accuracy. Therefore, this study was conducted to determine the rate of concordance between athletic trainers and treating physicians to provide an indirect evaluation of the accuracy of on-site assessments by athletic trainers. In other words, the more people who agree that a certain diagnosis exists, the more likely it is to be the “true” diagnosis. The goal of the current study was not to determine who truly provided the correct diagnosis but rather to define the correct option as the option with greatest agreement. If there is disagreement about the nature of the injury, then an error in treatment could occur and compromise care.

The study found a very high rate of concordance between athletic trainers and physicians (92.0%). Statistical analysis showed injury groups in which interrater agreement was highest, suggesting a greater chance that both the athletic trainer and the physician assessed the injury properly.6 Interrater agreement was highest in the dislocation group. This result was expected; the presentation of a dislocation is obvious and easily observable. Concordance among athletic trainers and physicians was 100% in the dislocation group. Similarly, rates of interrater agreement and concordance were very high in the concussion group. Again, this finding was expected, given the unique mechanism of causation and symptoms of a concussion and the recent increase in awareness of concussions among medical personnel who cover sporting events.

Analysis of the poorest interrater agreement in each category showed areas in which athletic trainers and physicians may need to improve communication. Interrater agreement was lowest in incidents categorized as meniscal/labral injury. Disagreements involving meniscal/labral injury were rare, with only 3 of the 23 discordant incidents including an assessment of this type of injury. However, this group was the second smallest injury group analyzed in this study. Nonetheless, diagnosing these injuries in an acute setting, without the use of imaging studies, presented a challenge for athletic trainers. Interrater agreement was second lowest in the fracture category, and most incorrect diagnoses involved fractures (13 of 23). A study of sports-related fractures at US high schools reported that 95% of fractures required at least 1 diagnostic imaging technique.8 Therefore, the incorrectly assessed fractures in this study can be attributed to the difficulty of assessing this type of injury without access to imaging modalities. Diagnosing a labral/meniscal tear incorrectly results in little harm to the athlete in the acute setting, and overdiagnosing fractures on the field is preferable to allowing an athlete to return to a game with a potential fractured extremity that could displace. In this context, the athletic trainer's initial assessment and provision of care to injured athletes in the current analysis proved to be fairly remarkable.

On-site imaging technology, such as portable ultrasound or radiography, when operated by a skilled technician, would likely eliminate nearly all diagnostic errors involving fractures and increase concordance between athletic trainers and physicians. Theoretically, this would improve the care of athletes by ensuring an accurate diagnosis at the time of injury. However, as stated previously, overdiagnosis of fracture protects the athlete from further injury. In addition, this study found that athletic trainers rarely incorrectly assessed an injury that was eventually determined to be a fracture. In the study sample, this occurred 3 times. Of the 47 fractures diagnosed by physicians, the 3 cases that were assessed incorrectly represent only 6.4% of all fractures. Further, in 2 of those cases, the athletic trainer's evaluation would have resulted in referral to a physician, and therefore radiographic examination, before the athlete could return to play. Therefore, this assessment eliminated the risk of worsening the un-diagnosed fracture. Nonetheless, portable ultrasound or radiography would allow skilled technicians to detect fractures more confidently, essentially eliminating the chance that an athlete would return to play with a fracture. However, the cost of this technology, in both equipment and labor, is likely a barrier to widespread use at secondary school sporting events.

Review of the discordant evaluations showed 1 case in which a glaring discrepancy between the athletic trainer assessment and the physician diagnosis occurred. In this case, the athletic trainer's evaluation was anterior cruciate ligament tear and the subsequent physician's diagnosis was recorded as “compound fracture of the tibia.” Based on a review of the database, it is likely that this was a recording error and that a compression or stress fracture was the intended diagnosis. Inclusion of this patient and the discrepancy in the data set did not significantly change the outcome or interpretation of the results.

As an internal control of the sample, the authors gathered data on a subset of injuries in which both an athletic trainer and an orthopedic physician were present at the sporting contest to provide an on-field examination. In these cases, the athletic trainer's assessment and the physician's diagnosis were in agreement. Of the 31 incidents retrieved from the database, all of the patients were referred for further treatment in the office setting. Of this cohort, 28 (90.3%) of 31 diagnoses made by the physician and the athletic trainer on the field were concordant with a subsequent office examination. This value (although a smaller sample size of athletic trainer vs physician) was very similar to the overall accuracy rate of an athletic trainer. This finding suggests that a physician on the field provides approximately as accurate a diagnosis as an athletic trainer with the limited resources that are available at the time of injury. Future studies are needed to determine whether athletic trainers who have a longer-term working relationship with clinicians would improve this accuracy rate. The authors strongly believe that shared decision making with athletic trainers, routine journal club/didactic and interactive education sessions with athletic trainers, and improved communication would increase the accuracy of on-field diagnosis and offer athletes prompt and effective treatment.

The study had several limitations. Most importantly, the sample did not include athletes who were injured and were evaluated by the athletic trainer but were not advised to see a physician or chose not to seek further treatment from a physician. Therefore, the study disproportionately represents more severe injuries. However, the goal of the study was to evaluate the agreement between the athletic trainer and the physician, and this is the only way to perform this analysis with the data available. A previous study reported that fractures account for 10.1% of all injuries in high school athletes.8 It is reasonable to assume that the increased rate of fractures reported in the study population (16.4%) is the result of the previously mentioned bias. The effect on the assessment of reported concordance is unknown. An additional limitation is the inclusion of multiple physicians from a variety of fields who provided in-office examinations. It is likely that analyzing a single pairing of an athletic trainer with a physician with an established relationship could have yielded higher concordance rates. However, as shown with the subset of patients who had on-field examinations by physicians, the athletic trainers in this analysis were essentially surrogates for physicians, providing equally effective care. Further, because access to health care varies regionally, inclusion of all types of health care professionals increases the applicability of this study and represents a complete analysis of concordance between athletic trainers and physicians.

The strength of this study is the diverse study population. The study population included athletes playing 34 sports during a 2-year period at 5 high schools. The variety of sports included in the analysis resulted in the evaluation of a wide range of athletic injuries. To the authors' knowledge, this is the first study to examine the accuracy of athletic trainer assessment by comparison with subsequent physician diagnosis. In addition, this study is the first to determine the accuracy of assessments provided by physicians on the field compared with comprehensive in-office evaluations.

As participation in secondary school athletics increases, so does the need for fast, accurate, and thorough treatment of injured athletes. Athletic trainers are often the first health professionals to tend to injured athletes in both competitive and noncompetitive settings, and their assessment is critical for proper injury management. In addition, in high schools with populations with poor access to medical care, the athletic trainer may be the only health care professional coordinating the health and well-being of these athletes. Although the current study investigated only musculoskeletal injuries, the findings emphasize the tremendous role that athletic trainers play in the care of athletes and, in some cases, their families.

Conclusion

This study is the first to examine an increasingly important aspect of student-athlete health and safety and provide a comprehensive analysis of concordance between assessments of athletic trainers and diagnoses of physicians. In the study population, athletic trainers showed impressive accuracy in evaluating acutely injured athletes, especially in the absence of imaging equipment. Additionally, in cases without a confirmed diagnosis, athletic trainers and treating physicians showed a high rate of concordance, suggesting that the initial assessment by athletic trainers was accurate.

References

  1. Comstock RD, Collins CL, Currie DW. National high school sports-related injury surveillance study. http://www.ucdenver.edu/academics/colleges/PublicHealth/research/ResearchProjects/piper/projects/RIO/Documents/2012-13.pdf. Accessed January 10, 2013.
  2. Glover L. Certified athletic trainers. Minn Med. 2010; 93(12):43–45.
  3. American Medical Association. Policy H-470.995 athletic (sports) medicine. http://www.nata.org/sites/default/files/ama_recommendation.pdf. Accessed January 10, 2013.
  4. Smith AM, Milliner EK. Injured athletes and the risk of suicide. J Athl Train. 1994; 29(4):337–341.
  5. Smith AM, Scott SG, Wiese DM. The psychological effects of sports injuries: coping. Sports Med. 1990; 9(6):352–369. doi:10.2165/00007256-199009060-00004 [CrossRef]
  6. Bernstein J, Long JS, Veillette C, Ahn J. Crowd intelligence for the classification of fractures and beyond. PLoS One. 2011; 6(11):e27620. doi:10.1371/journal.pone.0027620 [CrossRef]
  7. American Medical Association. ICD-9-CM 2015 Professional Edition for Physicians. Vols 1 and 2. Chicago, IL: American Medical Association; 2015.
  8. Swenson DM, Yard EE, Collins CL, Fields SK, Comstock RD. Epidemiology of US high school sports-related fractures, 2005–2009. Clin J Sport Med. 2010; 20(4):293–299. doi:10.1097/JSM.0b013e3181e8fae8 [CrossRef]

Frequency of Injury by Group

InjuryNo. of Injuries
Sprain96 (33.6%)
Concussion67 (23.4%)
Fracture47 (16.4%)
Contusion27 (9.4%)
Inflammation24 (8.4%)
Strain12 (4.2%)
Meniscal/labral injury7 (2.4%)
Dislocation6 (2.1%)
Total286 (100%)

Concordance by Health Care Provider Specialty

InjuryNo.

Orthopedic PhysicianEmergency Department/Urgent Care CenterPediatricianPrimary Care PhysicianNeurologistChiropractorPodiatristPsychiatristTotal
Sprain493544040096
Concussion8620102200167
Fracture36901001047
Contusion111302010027
Inflammation17051001024
Strain8110020012
Meniscal/labral injury700000007
Dislocation410001006
Total14065301822821286
Concordant12857291722811263
Concordance rate91.4%87.7%96.7%94.4%100.0%100.0%50.0%100.0%92.0%

Certified Athletic Trainer and Physician Concordance by Group

InjuryNo. of Concordant DiagnosesTotal No.ConcordantKappa Coefficient
Sprain879690.63%0.92
Concussion666798.51%0.99
Fracture444793.62%0.84
Contusion242788.89%0.896
Inflammation202483.33%0.953
Strain111291.67%0.913
Meniscal/labral injury5771.43%0.764
Dislocation66100.00%1
Total26328691.96%0.907

Review of Discordant Certified Athletic Trainer Evaluations and Physician Diagnoses

Athletic Trainer EvaluationPhysician Diagnosis


EvaluationCategoryCategoryDiagnosis
Nose fractureFractureConcussionConcussion
Nose fractureFractureContusionNose contusion
Fractured sternumFractureContusionSternum contusion
Clavicle fractureFractureContusionClavicle contusion
High ankle sprainSprainFractureFracture of growth plate in ankle
Sever's diseaseOtherFractureStress fracture
Anterior cruciate ligament tearSprainFractureCompound fracture of tibiaa
Plantar fasciitisInflammationInflammationSesamoiditis
Foot contusionContusionInflammationPosterior tibial tendonitis
Hip flexor strainStrainInflammationApophysitis of hip
Swollen kneeInflammationInflammationBursitis
Patellofemoral pain syndrome and meniscus tearOtherMeniscal/labral pathologyOsteochondral dissecans
Patellofemoral painOtherMeniscal/labral pathologyMeniscus tear
Possible anterior subluxationDislocationSprainShoulder sprain
Ankle fractureFractureSprainAnkle sprain
Ankle fractureFractureSprainAnkle sprain
Medial contusion, rule out tibia fractureFractureSprainAnkle sprain
Ankle fractureFractureSprainAnkle sprain
Lisfranc fractureFractureSprainAnkle sprain
Meniscus tearMeniscal/labral pathologySprainKnee sprain
Wrist/palmar contusionContusionSprainWrist sprain
Wrist fractureFractureSprainWrist sprain
Carpal painContusionStrainWrist strain
Authors

The authors are from Thomas Jefferson University (NJL, BT, KBF, LSA, MP, FPT), Rothman Institute of Orthopaedics, Egg Harbor Township, New Jersey; and Mercy Suburban Hospital (BE), East Norriton, Pennsylvania.

The authors have no relevant financial relationships to disclose.

Correspondence should be addressed to: Fotios P. Tjoumakaris, MD, Thomas Jefferson University, Rothman Institute of Orthopaedics, 2500 English Creek Rd, Bldg 1300, Egg Harbor Township, NJ 08234 ( Fotios.Tjoumakaris@rothmaninstitute.com).

Received: July 15, 2015
Accepted: February 15, 2016
Posted Online: July 12, 2016

10.3928/01477447-20160623-10

Sign up to receive

Journal E-contents