Orthopedics

Feature Articles 

Review of the Current Literature as a Preparatory Tool for the Trauma Content of the Orthopaedic In-Training Examination

Payam Farjoodi, MD; David R. Marker, BS; Jeremy R. McCallum IV, MS; Frank J. Frassica, MD; Simon C. Mears, MD, PhD

Abstract

Currently, the only standardized evaluation of trauma knowledge throughout orthopedic training is found in the Orthopaedic In-Training Examination, which is administered annually to all residents by the American Academy of Orthopaedic Surgeons. Our goals were to assess the Orthopaedic In-Training Examination to (1) determine the content of the trauma questions, (2) identify the content of the 3 most frequently referenced journals on the answer keys, and (3) evaluate the correlation between those contents.

We reviewed the trauma-related Orthopaedic In-Training Examination questions and answer keys for 2002 through 2007. Content for test questions and cited literature was assessed with the same criteria: (1) category type, (2) anatomic location, (3) orthopedic focus, and (4) treatment type. For each of the 3 most frequently referenced journals, we weighted content by dividing the number of times it was referenced by the number of its trauma-related articles. We then compared the journal data individually and collectively to the data from the Orthopaedic In-Training Examination trauma questions. A chi-square analysis with Yates correction was used to determine differences. Questions and literature were similar in the most frequently addressed items in each of the 4 areas: category type (taxonomy 3, treatment), 52.4% and 60.7%, respectively; anatomic location (femur), 23.3% and 27.7%, respectively; orthopedic focus (fracture), 51.0% and 56.5%, respectively; and treatment type (multiple/nonspecific), 39.0% and 35.4%, respectively.

The content correlation found between the questions and literature supports the idea that reviewing current literature may help prepare for the trauma content on the Orthopaedic In-Training Examination.

Trauma is a leading cause of morbidity and mortality in the United States and a major focus of orthopedic surgeons. Despite the at least 12 months of exposure to orthopedic trauma required by the American Board of Orthopaedic Surgery,1 the training that orthopedic residents obtain in the treatment of trauma depends heavily on the experience, exposure, and teaching provided by their programs. Currently, the only standardized evaluation of trauma knowledge throughout orthopedic training is found in the Orthopaedic In-Training Examination, which is administered annually to all residents by the American Academy of Orthopaedic Surgeons. This examination, initially administered in 1963, was the first specialty examination for residents.2 It has since served as the model for the development and administration of such tests in other fields of medicine, such as radiology, general surgery, and internal medicine. The examination acts as an objective comparison of orthopedic knowledge for residents of the same year-in-training and orthopedic training programs in general and consists of 275 multiple choice questions. Although recent orthopedic studies are recommended as a source of preparation, we are unaware of any published study correlating trauma content on the Orthopaedic In-Training Examination with the literature.

Our goals were to assess the Orthopaedic In-Training Examination to (1) determine the content of the trauma questions, (2) identify the content of the 3 most frequently referenced journals on the answer keys, and (3) evaluate the correlation between those contents to aid in providing residents with information that may prepare them for this portion of the examination.

Orthopaedic In-Training Examination Questions

We obtained the Orthopaedic In-Training Examination and associated Program Director’s Reports for the years 2002 through 2007 and compiled a database of the questions classified as trauma per these reports. From 2002 through 2007, two hundred ninety-two of 1650 Orthopaedic In-Training Examination questions (18%) focused on trauma (range, 12.7%-18.9% per year).

To determine content, we assessed 4 areas for each question: (1) category type, (2) anatomic location, (3) orthopedic focus, and (4) treatment type.

Under category, we classified questions based on the same 3 levels of cognitive taxonomy used by the examination’s authors. The first level, recall,…

Abstract

Currently, the only standardized evaluation of trauma knowledge throughout orthopedic training is found in the Orthopaedic In-Training Examination, which is administered annually to all residents by the American Academy of Orthopaedic Surgeons. Our goals were to assess the Orthopaedic In-Training Examination to (1) determine the content of the trauma questions, (2) identify the content of the 3 most frequently referenced journals on the answer keys, and (3) evaluate the correlation between those contents.

We reviewed the trauma-related Orthopaedic In-Training Examination questions and answer keys for 2002 through 2007. Content for test questions and cited literature was assessed with the same criteria: (1) category type, (2) anatomic location, (3) orthopedic focus, and (4) treatment type. For each of the 3 most frequently referenced journals, we weighted content by dividing the number of times it was referenced by the number of its trauma-related articles. We then compared the journal data individually and collectively to the data from the Orthopaedic In-Training Examination trauma questions. A chi-square analysis with Yates correction was used to determine differences. Questions and literature were similar in the most frequently addressed items in each of the 4 areas: category type (taxonomy 3, treatment), 52.4% and 60.7%, respectively; anatomic location (femur), 23.3% and 27.7%, respectively; orthopedic focus (fracture), 51.0% and 56.5%, respectively; and treatment type (multiple/nonspecific), 39.0% and 35.4%, respectively.

The content correlation found between the questions and literature supports the idea that reviewing current literature may help prepare for the trauma content on the Orthopaedic In-Training Examination.

Trauma is a leading cause of morbidity and mortality in the United States and a major focus of orthopedic surgeons. Despite the at least 12 months of exposure to orthopedic trauma required by the American Board of Orthopaedic Surgery,1 the training that orthopedic residents obtain in the treatment of trauma depends heavily on the experience, exposure, and teaching provided by their programs. Currently, the only standardized evaluation of trauma knowledge throughout orthopedic training is found in the Orthopaedic In-Training Examination, which is administered annually to all residents by the American Academy of Orthopaedic Surgeons. This examination, initially administered in 1963, was the first specialty examination for residents.2 It has since served as the model for the development and administration of such tests in other fields of medicine, such as radiology, general surgery, and internal medicine. The examination acts as an objective comparison of orthopedic knowledge for residents of the same year-in-training and orthopedic training programs in general and consists of 275 multiple choice questions. Although recent orthopedic studies are recommended as a source of preparation, we are unaware of any published study correlating trauma content on the Orthopaedic In-Training Examination with the literature.

Our goals were to assess the Orthopaedic In-Training Examination to (1) determine the content of the trauma questions, (2) identify the content of the 3 most frequently referenced journals on the answer keys, and (3) evaluate the correlation between those contents to aid in providing residents with information that may prepare them for this portion of the examination.

Materials and Methods

Orthopaedic In-Training Examination Questions

We obtained the Orthopaedic In-Training Examination and associated Program Director’s Reports for the years 2002 through 2007 and compiled a database of the questions classified as trauma per these reports. From 2002 through 2007, two hundred ninety-two of 1650 Orthopaedic In-Training Examination questions (18%) focused on trauma (range, 12.7%-18.9% per year).

To determine content, we assessed 4 areas for each question: (1) category type, (2) anatomic location, (3) orthopedic focus, and (4) treatment type.

Under category, we classified questions based on the same 3 levels of cognitive taxonomy used by the examination’s authors. The first level, recall, tests factual knowledge. The second level, deductive, focuses on the evaluation of information, including imaging studies, to develop a diagnosis. The third level, analytic, requires the interpretation of provided information to develop a treatment plan. The designation of category type was made by the junior authors (P.F., D.R.M., J.R.M., F.J.F) and confirmed by the senior author (S.C.M.), a trauma-fellowship-trained orthopedic surgeon. Any conflicts in categorization were resolved by the senior author after discussion.

For the anatomic location, the AO classification was used. Questions with multiple areas of focus were categorized under all applicable areas, and those with no area in particular were classified as nonspecific.

For the orthopedic focus, we used 11 subset groups: (1) acute tendon injury, (2) delayed reconstruction for malunion/nonunion, (3) fracture, (4) geriatric trauma, (5) infection, (6) other complications, (7) military/ballistic trauma, (8) other soft tissue injury, (9) periprosthetic fracture, (10) dislocation, and (11) multiple/nonspecific.

For treatment types, there were 11 choices: (1) bone graft, (2) external fixation, (3) locked plating, (4) arthroplasty, (5) intramedullary nailing, (6) open reduction and internal fixation, (7) revision, (8) tendon repair, (9) nonoperative treatment, (10) amputation, and (11) multiple/nonspecific.

Orthopaedic In-Training Examination Cited Literature

For each examination, the American Academy of Orthopaedic Surgeons provides an answer key with cited references. To determine the most relevant journals, we reviewed those references and determined the most frequently cited journals. We then determined the number of trauma articles in each of these journals from 2001 through 2006 by performing a systematic review of the PubMed bibliographic database. Articles from the journals that did not have a primary focus on trauma were excluded. For example, the article by Lane3 from the Journal of Orthopaedic Trauma was not included in the analysis because the primary focus of the article is more reflective of Orthopaedic In-Training Examination  questions categorized in the answer key as being related to orthopedic science.

We divided the number of times each journal was referenced on the examination by the total number of trauma articles in each journal over the time period, described as proportion of trauma referenced. The content of the top 3 most frequently referenced journals was further characterized via the same criteria as the Orthopaedic In-Training Examination trauma questions. We compared the data, for each journal individually and for all journals collectively, to the data for the Orthopaedic In-Training Examination questions.

Statistical Analysis

A chi-square analysis with Yates correction was used to calculate the difference in proportions. A P value <.05 was considered statistically significant. A power analysis with use of the difference in overall percentage of trauma questions on the Orthopaedic In-Training Examination compared with the percentage of corresponding articles was conducted as the primary research question. This procedure proved that our sample size was sufficient to reveal the necessary P values to answer our primary question at a power of >80%.

Results

Analysis of the trauma questions revealed that the most frequent category type of question was type 3 (treatment; 52.4% of the questions), the most frequent anatomic location was the femur (23.3% of the questions), the most frequent orthopedic focus was fracture (51.0% of the questions), and the most frequent treatment type was nonspecific (39.0% of the questions) (Table 1).

The 3 journals most frequently referenced on the Orthopaedic In-Training Examination (The Journal of Bone and Joint Surgery, American Volume; Clinical Orthopaedics and Related Research; and The Journal of Orthopaedic Trauma) published 6534 articles over the corresponding years, 1378 (21.1%) of which were related to trauma (Figure, Table 2). This percentage is significantly higher than the 17.7% ratio of trauma-related Orthopaedic In-Training Examination questions during the study period (P=.003).The Journal of Bone and Joint Surgery, American Volume had the highest proportion of trauma articles referenced on the examination (28.3%; 108 of 381); followed by The Journal of Orthopaedic Trauma (16.6%; 106 of 639) and Clinical Orthopaedics and Related Research (14.5%; 52 of 358). There was a correlation between the content of the trauma articles from these 3 journals collectively and the types of test questions asked (Figure, Table 1). In these collective 3 journals, the most frequent category was taxonomy 3 (treatment; 60.7% of trauma articles), the most frequent anatomic location was the femur (27.7% of trauma articles), the most frequent orthopedic focus was fracture (56.5% of trauma articles), and the most frequent treatment type was nonspecific (35.4% of trauma articles) (Table 1). When the content of trauma articles from each journal was compared individually to the trauma content on the Orthopaedic In-Training Examination, the most frequent category type and orthopedic focus were the same. Exceptions were the most frequent anatomic location in Clinical Orthopaedics and Related Research and The Journal of Bone and Joint Surgery, American Volume (ie, multiple/unspecified rather than the femur as on the examination and in The Journal of Orthopaedic Trauma [Table 1]), and treatment type in The Journal of Orthopaedic Trauma (ie, other internal fixation compared with nonspecific in the other journals and on the examination).

Table 1

Figure
Figure: Percentage of Orthopaedic In-Training Examination and journals related to trauma.

Table 2

There were several significant differences between the content on the examination and in the literature. Taxonomy 1 (recall) was addressed in 39.4% of the test questions but in only 32.7% of the articles (P=.034). Additionally, taxonomy 3 (treatment) was addressed in only 52.4% of the questions but in >60% of the articles (P=.010). There were more articles with a nonspecific anatomic location in the journals than in the questions (26.9% vs 19.5%, respectively; P=.011), and fewer articles with a focus on the pelvis than examination questions (9.4% vs 11.6%, respectively; P=.049). Orthopedic focus was nonspecific in more Orthopaedic In-Training Examination questions compared with journal articles (15.8% vs 9.0%; P<.001, and there were more articles relating to periprosthetic fractures compared with questions (2.1% vs 0.0%; P=.024). Although the most frequently addressed treatment modality in both questions and articles was nonspecific, bone grafting was addressed significantly less often in the Orthopaedic In-Training Examination than in the literature (0.7% and 4.9%, respectively; P=.002), as was external fixation (3.4% and 7.3%, respectively; P=.023) (Table 1).

Discussion

The Orthopaedic In-Training Examination is the only standardized, objective national evaluation of orthopedic residents. The examination was founded with 3 fundamental goals: (1) to measure resident knowledge against a national mean, (2) to determine minimal standards for residents, and (3) to measure the quality of training within various residency programs.2 The importance of good performance on the Orthopaedic In-Training Examination is reinforced by the work of Dirschl et al,4 who found that good performance on the Orthopaedic In-Training Examination correlates with a high score on part I of the American Board of Orthopaedic Surgery examination. However, a lack of a standardized curriculum makes preparation difficult.

Some data have suggested a statistically significant correlation between studying The Journal of Bone and Joint Surgery, American Volume and The Journal of the American Academy of Orthopaedic Surgeons and strong performance on the Orthopaedic In-Training Examination.5 The authors also found a significant correlation between higher scores and review of previous examinations.5 Many residency programs rely on literature review for resident education. Greene6 surveyed the chairmen of all 161 orthopedic residency programs throughout the United States and Canada to determine the prevalence of journal clubs. Of the 149 programs that responded, journal clubs were used regularly in 147 (99%).6 Of those clubs, 114 met once per month, 14 met biweekly, and 11 met once per week. The author also found that the most popular format was articles from .1 scientific publication (82%).6

It is important to note that performance on the Orthopaedic In-Training Examination does not correlate with evaluation of resident performance by the attendings.7 Knowledge of the evaluation, diagnosis, and treatment of trauma patients is an important aspect of becoming a knowledgeable, prudent orthopedic surgeon. However, experience under good supervision enables residents to develop the skills “to make the correct decisions and carry them out under adverse circumstances.”8 Thus, it is important that any educational curriculum be treated as a supplement, and not a substitute, to the clinical experience of caring for trauma patients.

Previous studies have been performed in other orthopedic subspecialties with regard to preparation for the Orthopaedic In-Training Examination. Frassica et al9 classified the pathology section on the examinations from 2002 through 2006 and found that 10% of the examination consists of questions on this subject. They found that most questions required interpretation of imaging and/or histologic material to develop a diagnosis or treatment plan and concluded that systematic review of the most commonly tested material would help prepare residents.9 A recent study performed by Marker et al10 evaluated the osteonecrosis content on the Orthopaedic In-Training Examination. Their results showed that the literature had a larger percentage of articles on this subject matter than did questions on the examination. The authors also determined the proportion of orthopedic literature and examination questions focused on idiopathic scoliosis, chondrosarcoma, femoral neck fracture, and hip resurfacing. They found that, except for idiopathic scoliosis, the literature and test had similar percentages of content, and they concluded that review of the current literature would serve as a good preparatory tool for the Orthopaedic In-Training Examination.10 Another study evaluated hand content on the Orthopaedic In-Training Examination and classified the questions based on diagnosis, treatment, subject matter, and anatomic location.11 The authors found that the articles in the most frequently cited journals shared a similar focus in those subject matters, suggesting that the review of select current literature would aid in studying for the examination.11

These studies, combined with our findings, suggest that a curriculum of journal clubs reviewing trauma articles from the 3 journals found to have the highest proportion of referenced articles—The Journal of Bone and Joint Surgery, American Volume; Clinical Orthopaedics and Related Research; and The Journal of Orthopaedic Trauma—may prove the most high-yield use of structured resident study time limited by the implementation of the 80-hour work week. Such a strategy may be most useful if implemented early in training and continued throughout residency.

Our study has several weaknesses. First, we evaluated the examination and journals for only a limited time. However, that 6-year time frame is longer than that in previously published studies evaluating the Orthopaedic In-Training Examination.9-11 In addition, we elected to review the trauma content of the journals from the years 2001 through 2006 as a representation of the tested material from 2002 through 2007. Although most test questions are written, reviewed, and approved for use on examinations >1 year in advance, we thought those 6 years were a representative sample of “current literature.” This decision also had a precedent in the evaluation of the hand content on the Orthopaedic In-Training Examination by Marker et al.11 The references as reported in the provided Program Director’s Report may also contain inaccuracies because they may have been added after the question was written or may not have led the examinee to the correct answer. However, these references were provided by the test authors themselves, and any inaccuracies can be attributed to the test-writing process itself.

One might consider that The Journal of Orthopaedic Trauma might have skewed our results because it had a significantly higher percentage of trauma articles than the combined value for all 3 journals (75.0% vs 21.1%, respectively; P<.001). However, by determining the proportion of references to trauma articles, we showed that The Journal of Bone and Joint Surgery, American Volume was more heavily weighted (108 references of 381 trauma articles [28.3%] vs 106 references of 639 trauma articles [16.6%] in The Journal of Orthopaedic Trauma). In terms of our 4 parameters, the content of trauma articles from The Journal of Orthopaedic Trauma was consistent for the most part with that of the other journals and the Orthopaedic In-Training Examination questions. The main differences were that The Journal of Orthopaedic Trauma more frequently addressed the anatomic areas of the pelvis than did the collective journals (14.5% vs 9.4%, P=.049) and displayed a trend (although not a significant difference) toward more tibia/fibula articles than were found in the collective journals (24.4% vs 19.7%, P=.597) (Table 1).

Conclusion

Although some differences were found between the content of the Orthopaedic In-Training Examination and the literature, both sources were similar in terms of most frequently addressed topics. These results indicate that studying the current trauma literature from The Journal of Bone and Joint Surgery, American Volume, The Journal of Orthopaedic Trauma, and Clinical Orthopaedics and Related Research provides content similar to the trauma content on the Orthopaedic In-Training Examination, a finding that is consistent with previous studies assessing other subspecialties within the examination. We recommend additional studies evaluating the implementation of such a protocol and the resultant performance on the Orthopaedic In-Training Examination to reaffirm these suggestions.

References

  1. American Board of Orthopaedic Surgery. 2006 Rules and Procedures for Residency Education Part I and Part II Examinations. Chapel Hill, NC: American Board of Orthopaedic Surgery; 2006.
  2. Mankin HJ. The Orthopaedic In-Training Examination (OITE). Clin Orthop Relat Res. 1971; (75):108-116.
  3. Lane JM. Bone morphogenic protein science and studies. J Orthop Trauma. 2005; 19(10 Suppl):S17-S22.
  4. Dirschl DR, Campion ER, Gilliam K. Resident selection and predictors of performance: can we be evidence based? Clin Orthop Relat Res. 2006; (449):44-49.
  5. Miyamoto RG Jr, Klein GR, Walsh M, Zuckerman JD. Orthopedic surgery residents’ study habits and performance on the orthopedic in-training examination. Am J Orthop (Belle Mead NJ). 2007; 36(12):E185-E188.
  6. Greene WB. The role of journal clubs in orthopaedic surgery residency programs. Clin Orthop Relat Res. 2000; (373):304-310.
  7. Dirschl DR, Dahners LE, Adams GL, Crouch JH, Wilson FC. Correlating selection criteria with subsequent performance as residents. Clin Orthop Relat Res. 2002; (399):265-271.
  8. Chapman MW. Education in orthopaedic trauma. Clin Orthop Relat Res. 1997; (339):7-19.
  9. Frassica FJ, Papp D, McCarthy E, Weber K. Analysis of the pathology section of the OITE will aid in trainee preparation [published online ahead of print April 19, 2008]. Clin Orthop Relat Res. 2008; 466(6):1323-1328.
  10. Marker DR, Mont MA, Seyler TM, LaPorte DM, Frassica FJ. Current literature: an educational tool to study osteonecrosis for the Orthopaedic In-Training Examination? Orthop Clin North Am. 2009; 40(2):299-304.
  11. Marker DR, Mont MA, McGrath MS, Frassica FJ, LaPorte DM. Current hand surgery literature as an educational tool for the orthopaedic in-training examination. J Bone Joint Surg Am. 2009; 91(1):236-240.

Authors

Drs Farjoodi, Frassica, and Mears and Messrs Marker and McCallum are from the Department of Orthopedic Surgery, The Johns Hopkins Bayview Medical Center, Baltimore, Maryland.

Drs Farjoodi, Frassica, and Mears and Messrs Marker and McCallum have no relevant financial relationships to disclose.

Correspondence should be addressed to: Simon C. Mears, MD, PhD, c/o Elaine P. Henze, BJ, ELS, Department of Orthopedic Surgery, Johns Hopkins Bayview Medical Center, 4940 Eastern Ave, #A665, Baltimore, MD 21224-2780 (ehenze1@jhmi.edu).

doi: 10.3928/01477447-20110317-33

10.3928/01477447-20110317-33

Sign up to receive

Journal E-contents
click me