Orthopedics

Feature Article 

Analysis of the Trauma Section of the Orthopaedic In-Training Examination

Benjamin C. Taylor, MD; T. Ty Fowler, MD

Abstract

The Orthopaedic In-Training Examination is a comprehensive test produced annually by the American Academy of Orthopaedic Surgeons, and was first administered in 1963. At the time of the examination’s conception, its objectives were to: (1) measure the knowledge of orthopedic residents and provide objective comparisons; (2) help determine acceptable minimal standards for trainees; and (3) help provide an objective assessment of orthopedic education. We retrospectively reviewed all Orthopaedic In-Training Examinations from 2004 to 2008, with particular focus on the questions listed in the musculoskeletal trauma domain on each year’s program director report. The musculoskeletal trauma domain, including topics, recommended answers, and references, was reviewed to provide an educational resource for residents and residency programs when studying or designing educational curricula. The information in this analysis may help in development of a core musculoskeletal trauma knowledge base or facilitate determination of appropriate journal club and didactic lecture content.

Abstract

The Orthopaedic In-Training Examination is a comprehensive test produced annually by the American Academy of Orthopaedic Surgeons, and was first administered in 1963. At the time of the examination’s conception, its objectives were to: (1) measure the knowledge of orthopedic residents and provide objective comparisons; (2) help determine acceptable minimal standards for trainees; and (3) help provide an objective assessment of orthopedic education. We retrospectively reviewed all Orthopaedic In-Training Examinations from 2004 to 2008, with particular focus on the questions listed in the musculoskeletal trauma domain on each year’s program director report. The musculoskeletal trauma domain, including topics, recommended answers, and references, was reviewed to provide an educational resource for residents and residency programs when studying or designing educational curricula. The information in this analysis may help in development of a core musculoskeletal trauma knowledge base or facilitate determination of appropriate journal club and didactic lecture content.

Dr Taylor is from the Department of Orthopedic Surgery, Grant Medical Center, Columbus, Ohio; and Dr Fowler is from Stanford Medical Center, Redwood City, California.

Drs Taylor and Fowler have no relevant financial relationships to disclose.

Correspondence should be addressed to: Benjamin C. Taylor, MD, Department of Orthopedic Surgery, Grant Medical Center, 285 E State St, Ste 500, Columbus, OH 43215 (drbentaylor@gmail.com).

Posted Online: July 07, 2011

The Orthopaedic In-Training Examination is a comprehensive test produced annually by the American Academy of Orthopaedic Surgeons (AAOS), and was first administered in 1963. It is the first examination of its kind to be used for surgical subspecialty residents, and has subsequently been used as a model for other surgical subspecialties.1,2 At the time of the examination’s conception, its objectives were to: (1) measure the knowledge of orthopedic residents and provide objective comparisons; (2) help determine acceptable minimal standards for trainees; and (3) help provide an objective assessment of orthopedic education.1

The Orthopaedic In-Training Examination has continued to evolve since its inception; the current examination is composed of 275 questions divided into 12 domains: basic science and tumors, foot and ankle, hand surgery, hip and reconstructive surgery, medically-related issues, musculoskeletal trauma, orthopedic diseases, pediatric orthopedics, rehabilitation, shoulder and elbow surgery, spine surgery, and sports medicine. An electronic version of the examination has been successfully developed and is currently the method of administering the examination to residents.3 Individual questions often contain aspects of several different domains, but assignment of Orthopaedic In-Training Examination questions to one particular domain is done by the AAOS members involved in the test’s creation.

After the Orthopaedic In-Training Examination is completed each year, the examination is given to each resident, along with a score sheet, preferred responses, and recommended references for each question. It allows objective comparison of resident education and knowledge, as well as providing a valuable study resource for future examinations, including the American Board of Orthopaedic Surgery certifying examination. Areas of knowledge deficit can be identified and appropriately attended to; the reference lists provide a valuable resource in recognizing important reading and study materials.

Recently, several domains of the Orthopaedic In-Training Examination have been analyzed, including pathology, pediatric orthopedics, foot and ankle surgery, hand surgery, and sports medicine.4–8 The musculoskeletal trauma domain of the examination is the largest subset of this examination, and to our knowledge, no evaluation of this critical aspect has been performed. We analyzed the Orthopaedic In-Training Examination musculoskeletal trauma domain, including topics, recommended answers, and references, to provide an educational resource for residents and residency programs when studying or designing educational curricula. The information in this analysis may help in development of a core musculoskeletal trauma knowledge base or facilitate determination of appropriate journal club and didactic lecture content.

Materials and Methods

We retrospectively reviewed all Orthopaedic In-Training Examinations from 2004 to 2008, with particular focus on the questions listed in the musculoskeletal trauma domain on each year’s program director report. All questions discarded for scoring were similarly discarded from analysis in this study.

The number and percentage of trauma questions were recorded first, followed by identification of question topic(s), treatment modalities, and presence/type of imaging accompanying questions. The questions were then each classified into 3 different taxonomy groups, as defined by Frassica et al.4 In this taxonomy, type 1 questions are those that require simple recollection of facts or data. Type 2 questions require interpretation of data to formulate a diagnosis. Type 3 questions involve complex problem solving, including formulation of a diagnosis with appropriate treatment protocols from limited information available in the question stem.

The program director’s report was then used to scrutinize the references involved in question creation. The number of references per question, type of reference, reference source, and time from reference to question administration were all recorded and analyzed.

Results

The number of trauma questions during the 5-year review period was consistent year to year, ranging between 49 and 52 questions per year; this accounted for approximately 18.9% (yearly range, 18.3%–19.4%) of the examination annually (Table 1). The most common general question category was lower-extremity trauma (50.0%), followed by upper-extremity trauma (19.6%) and pelvis/acetabular trauma (12.9%) (Table 2).

Number of Trauma Questions by Year of Examination

Table 1: Number of Trauma Questions by Year of Examination

General Categories of Trauma Questions by Year of Examination

Table 2: General Categories of Trauma Questions by Year of Examination

Additional analysis of the questions revealed that basic science questions were the most common (10.2%), but that 16 other topics were tested, on average, at least once per examination over this 5-year period (Table 3). These frequently seen topics were all related to lower-extremity trauma, with the exception of the aforementioned basic science, nonunions (8.2%), humeral shaft fractures (4.3%), forearm fractures (3.1%), clavicle fractures (2.0%), and Advanced Trauma Life Support-related questions (2.0%).

Trauma Questions Asked an Average Greater Than Once Per Year

Table 3: Trauma Questions Asked an Average Greater Than Once Per Year

Treatment modalities were tested in 67.5% of the questions evaluated in the musculoskeletal trauma domain. Open reduction and internal fixation techniques were the most commonly asked or discussed, whereas only a single arthroscopy-related question was included in this domain over the period examined (Table 4). Imaging modality analysis revealed that plain radiographs were by far the most common shown, accounting for 85.9% of the images given for this topic over the time period evaluated (Table 5).

Modalities Tested by Year of Examination

Table 4: Modalities Tested by Year of Examination

Imaging Modalities Provided by Year of Examination

Table 5: Imaging Modalities Provided by Year of Examination

Investigation of the taxonomy revealed that the simple fact-recall questions (type 1) were the most prevalent (58.7%), with the most complex problem-solving questions (type 3) being the next most common (24.4%). The questions requiring formulation of a diagnosis from clinical and/or radiographic information (type 2) only made up 16.9% of the total trauma domain (Table 6).

Question Taxonomy Classification

Table 6: Question Taxonomy Classification

Primary journal article references were given in 75.4% (416 of 552 references), with 35 different journals referenced in this time period. The Journal of Orthopaedic Trauma (28.8% of all journals) and The Journal of Bone and Joint Surgery (American) (26.0% of all journals) were the most commonly referenced journals by a significant margin. A total of 24 different textbooks were referenced in this domain over this time period; Orthopaedic Knowledge Update: Trauma (second or third edition) (26.5% of all textbooks referenced) and Skeletal Trauma (24.6% of all textbooks referenced) were the most repeated reference by a significant amount. Table 7 lists the journal and textbook references given at least once per year, on average, over the time span evaluated. The question references were most commonly from references within 5 years of their examination year (51.4%), but that only 1.8% of the references were from sources published within 1 year of the test administration (Table 8).

Journal and Textbook References by Year of Examination

Table 7: Journal and Textbook References by Year of Examination

Time From Reference to Question by Year of Examination

Table 8: Time From Reference to Question by Year of Examination

Discussion

The Orthopaedic In-Training Examination remains a valuable educational tool for both educators and orthopedic residents, as deficient areas of program or individual knowledge can be significantly improved annually. Recently, evaluations of the pathology, pediatric orthopedics, foot and ankle surgery, hand surgery, and sports medicine domains have been published.4–8 However, as the musculoskeletal trauma section has remained the largest domain on the examination, careful attention must also be paid to this area to facilitate success on the Orthopaedic In-Training Examination. In addition, the vast majority of graduating residents will be required to know musculoskeletal trauma to succeed on the American Board of Orthopaedic Surgery Part I examination as well as in clinical practice, where variations of this topic are likely to be frequently encountered.

Attention to the reference breakdown can assist with resident reading selection as well as journal club development/design. Orthopedic journal clubs are commonplace in residency programs, with 99% of program directors reporting regular use of this educational forum.9 In fact, regular review of current literature, specifically The Journal of Bone and Joint Surgery (American Volume) and The Journal of the American Academy of Orthopaedic Surgeons has been shown to have significant positive correlations with Orthopaedic In-Training Examination performance.10 Analysis of the musculoskeletal trauma section over the time period evaluated also points to the importance in reviewing The Journal of Orthopaedic Trauma, as this was the most frequently cited reference overall. In addition to journal club development, creation of a core reading curriculum, guided with knowledge of topics thought to be important to the AAOS and American Board of Orthopaedic Surgery, can systematically assist residents in achieving the goals of passing their board examinations and being prepared for clinical and/or academic practice.

This study points to the importance of maintaining a basic science knowledge base, as greater than 5 questions per year were asked on this topic alone. Significant emphasis was also placed on lower-extremity trauma, which represented half of the entire domain. Simple recall of facts was by far the most common taxonomy type, but more complex critical thinking questions were seen in nearly a third of the musculoskeletal trauma domain. These higher-level questions require the synthesis of information recall, analytical thinking, and knowledge of treatment capabilities and outcomes.

Despite our best attempts to limit weaknesses, several inherent flaws are present with this thorough evaluation. Full determination of the entire trauma-related question database is limited by the AAOS question maker assignment, which allows many trauma-related questions to be assigned to other categories, such as arthroplasty, foot and ankle, and hand. We chose to follow this assignment to limit question selection bias by the authors and follow the precedent set forth by other similar analyses.4–8 The short period of evaluation is somewhat limited, but effectively captures a resident’s experience with the examination. Importantly, although we reviewed the questions provided in this time period, knowledge continues to evolve and selected topics may change accordingly. The addition of computerized testing may also significantly alter the imaging modalities tested, as the availability of video, entire CT or MRI sequences, and combinations thereof will allow for continued development of the examination as technology continues to advance.3

The results of this study should be used to help guide resident learning as well as curriculum development, as insight into the subjects and various taxonomy depths are given. Creation of a thorough, balanced educational curriculum can be performed if all resources are used; careful evaluations of learning tools and measures such as this should be incorporated to better evaluate current practices.11 Similar to other medical fields,12–16 orthopedic resident scores on the in-training examination have been shown to be correlated with performance on the written board examination.17–19 Although high Orthopaedic In-Training Examination scores are not causative of passing American Board of Orthopaedic Surgery Part I examination scores, they may be seen as evidence of maturity of a well-rounded knowledge base, and will help to serve that individual well into his or her career as an orthopedic surgeon. We hope that this investigation would lead to residents reviewing the appropriate, high-yield material when studying for this examination.

References

  1. Mankin HJ. The Orthopaedic In-Training Examination (OITE). Clin Orthop Relat Res. 1971; (75):108–116. doi:10.1097/00003086-197103000-00014 [CrossRef]
  2. Bancroft GN, Basu CB, Leong M, Mateo C, Hollier LH Jr, Stal S. Outcome-based residency education: teaching and evaluating the core competencies in plastic surgery. Plast Reconstr Surg. 2008; 121(6):441e–448e. doi:10.1097/PRS.0b013e318170a778 [CrossRef]
  3. Marsh JL, Hruska L, Mevis H. An electronic orthopaedic in-training examination. J Am Acad Orthop Surg. 2010; 18(10):589–596.
  4. Frassica FJ, Papp D, McCarthy E, Weber K. Analysis of the pathology section of the OITE will aid in trainee preparation. Clin Orthop Relat Res. 2008; (466):1323–1328. doi:10.1007/s11999-008-0235-2 [CrossRef]
  5. Papp DF, Ting BL, Sargent MC, Frassica FJ. Analysis of the pediatric orthopedic surgery questions on the Orthopaedic In-Training Examination, 2002 through 2006. J Pediatr Orthop. 2010; 30(5):503–507. doi:10.1097/BPO.0b013e3181df1560 [CrossRef]
  6. Srinivasan RC, Seybold JD, Kadakia AR. Analysis of the foot and ankle section of the Orthopaedic In-Training Examination (OITE). Foot Ankle Int. 2009; 30(11):1060–1064. doi:10.3113/FAI.2009.1060 [CrossRef]
  7. Marker DR, Mont MA, McGrath MS, Frassica FJ, LaPorte DM. Current hand surgery literature as an educational tool for the Orthopaedic In-Training Examination. J Bone Joint Surg Am. 2009; 91(1):236–240. doi:10.2106/JBJS.H.00972 [CrossRef]
  8. Srinivasan RC, Seybold JD, Salata MJ, Miller BS. An analysis of the orthopaedic in-training examination sports section: the importance of reviewing the current orthopaedic subspecialty literature. J Bone Joint Surg Am. 2010; 92(3):778–782. doi:10.2106/JBJS.I.00466 [CrossRef]
  9. Greene WB. The role of journal clubs in orthopaedic surgery residency programs. Clin Orthop Relat Res. 2000; (373):304–310. doi:10.1097/00003086-200004000-00037 [CrossRef]
  10. Miyamoto RG Jr, Klein GR, Walsh M, Zuckerman JD. Orthopedic surgery residents’ study habits and performance on the orthopedic in-training examination. Am J Orthop. 2007; 36(12):E185–E188.
  11. Taylor BC, Fowler TT, Dimitris C. Achieving educational excellence: a strategic initiative to enhance orthopaedic resident academic performance [published online ahead of print March 10, 2011]. J Surg Educ. 2011; 68(3):162–166. doi:10.1016/j.jsurg.2011.01.006 [CrossRef]
  12. Baumgartner BR, Pererman SB. Relationship between American College of Radiology in-training examination scores and American Board of Radiology written scores. Part 2. Multi-institutional study. Acad Radiol. 1998; 5(5):374–379. doi:10.1016/S1076-6332(98)80156-6 [CrossRef]
  13. Spellacy WN, Carlan SJ, McCarthy JM. Prediction of ABOG written examination performance from the third-year CREOG in-training results. J Reprod Med. 2006; 51(8):621–622.
  14. Ellis E III, Haug RH. A comparison of performance on the OMSITE and ABOMS written qualifying examination. Oral and Maxillofacial Surgery In-Training Examination. American Board of Oral and Maxillofacial Surgery. J Oral Maxillofac Surg. 2000; 58(12):1401–1406. doi:10.1053/joms.2000.18275 [CrossRef]
  15. Rollins LK, Martindale JR, Edmond M. Predicting pass rates on the American Board of Internal Medicine certifying examination. J Gen Intern Med. 1998; 13(6):414–416. doi:10.1046/j.1525-1497.1998.00122.x [CrossRef]
  16. Shetlet PL. Observations on the American Board of Surgery In-training Examination, board results, and conference attendance. Am J Surg. 1982; 144(3):292–294. doi:10.1016/0002-9610(82)90002-2 [CrossRef]
  17. Dirschl DR, Campion ER, Gilliam K. Resident selection and predictors of performance. Can we be evidence based?Clin Orthop Relat Res. 2006; (449):44–49.
  18. Crawford CH III, Nyland J, Roberts CS, Johnson JR. Relationship among United States Medical Licensing Step I, orthopedic in-training, subjective clinical performance evaluations, and american board of orthopedic surgery examination scores: a 12-year review of an orthopedic surgery residency program. J Surg Educ. 2010; 67(2):71–78. doi:10.1016/j.jsurg.2009.12.006 [CrossRef]
  19. Dougherty PJ, Walter N, Schilling P, Najibi S, Herkowitz H. Do scores of the USMLE Step 1 and OITE correlate with the ABOS Part I certifying examination?: a multicenter study [published online ahead of print March 30, 2010]. Clin Orthop Relat Res. 2010; 468(10):2797–2802. doi:10.1007/s11999-010-1327-3 [CrossRef]

Number of Trauma Questions by Year of Examination

QuestionsYear of Examination
20042005200620072008
Trauma5251524950
Total268272271268270
Percentage19.418.819.218.318.5

General Categories of Trauma Questions by Year of Examination

TopicYear of Examination
Total
20042005200620072008
General trauma2315011
Upper extremity6129101350
Lower extremity3323272321127
Pelvis & acetabulum77541033
Basic science & biomechanics46107633

Trauma Questions Asked an Average Greater Than Once Per Year

TopicYear of Examination
Total
20042005200620072008
Basic science5674426
Femoral shaft3567425
Tibial shaft2765424
Nonunion6533421
Pelvic ring3523619
Ankle1253516
Acetabulum5233316
Tibial plateau/proximal tibia4320312
Humeral shaft2412211
Femoral neck5212010
Pilon/distal tibia431019
Peritrochanteric/subtrochanteric141219
Radial/ulnar shaft031228
Calcaneus201317
Clavicle000325
ATLS010405
Talus400015

Modalities Tested by Year of Examination

ModalityYear of Examination
Total
20042005200620072008
None201222101983
ORIF2020192521105
Intramedullary nailing61388641
External fixation5301211
Arthroplasty132219
Brace/cast/splint010315
Arthroscopy001001

Imaging Modalities Provided by Year of Examination

ModalityYear of Examination
Total
20042005200620072008
Radiographs24141091673
CT111115
Radiographs & CT101215
Photographs/diagrams001001
Radiographs & MRI000101

Question Taxonomy Classification

TaxonomyYear of Examination
%
20042005200620072008
Knowledge & recall (T1)252736322958.7
Diagnosis & interpretation (T2)77781416.9
Treatment or evaluation from diagnosis (T3)201799724.4

Journal and Textbook References by Year of Examination

Year of Examination
Total
20042005200620072008
Journals
  Journal of Orthopaedic Trauma1526321829120
  Journal of Bone and Joint Surgery (American)2017183419108
  Clinical Orthopaedics and Related Research1011135645
  Journal of the American Academy of Orthopaedic Surgeons15581332
  Journal of Trauma1958326
  Journal of Bone and Joint Surgery (British)0145313
  Foot and Ankle International5050212
  Injury2223211
  Journal of Shoulder and Elbow Surgery210036
  Other167111843
  Total5778919298416
Textbooks
  Orthopaedic Knowledge Update: Trauma (Ed. 2 or 3)22910436
  Skeletal Trauma12058833
  Orthopaedic Knowledge Update (Ed. 7, 8, or 9)5500212
  Rockwood and Green’s Fractures in Adults100179
  Orthopaedic Basic Science: Foundations of Clinical Practice420017
  Fractures of the Acetabulum321006
  Surgical Exposures in Orthopaedics: The Anatomic Approach210025
  Other34551128
  Total5223121435136

Time From Reference to Question by Year of Examination

Year of Examination
Total
20042005200620072008
<1 y2303210
1–5 y5360504381287
6–10 y3418252330130
>10 y1723293428131
Authors

Dr Taylor is from the Department of Orthopedic Surgery, Grant Medical Center, Columbus, Ohio; and Dr Fowler is from Stanford Medical Center, Redwood City, California.

Drs Taylor and Fowler have no relevant financial relationships to disclose.

Correspondence should be addressed to: Benjamin C. Taylor, MD, Department of Orthopedic Surgery, Grant Medical Center, 285 E State St, Ste 500, Columbus, OH 43215 (drbentaylor@gmail.com).

10.3928/01477447-20110525-16

Sign up to receive

Journal E-contents