Abstract
Currently, the only standardized evaluation of trauma knowledge
throughout orthopedic training is found in the Orthopaedic In-Training
Examination, which is administered annually to all residents by the American
Academy of Orthopaedic Surgeons. Our goals were to assess the Orthopaedic
In-Training Examination to (1) determine the content of the trauma questions,
(2) identify the content of the 3 most frequently referenced journals on the
answer keys, and (3) evaluate the correlation between those contents.
We reviewed the trauma-related Orthopaedic In-Training Examination
questions and answer keys for 2002 through 2007. Content for test questions and
cited literature was assessed with the same criteria: (1) category type, (2)
anatomic location, (3) orthopedic focus, and (4) treatment type. For each of
the 3 most frequently referenced journals, we weighted content by dividing the
number of times it was referenced by the number of its trauma-related articles.
We then compared the journal data individually and collectively to the data
from the Orthopaedic In-Training Examination trauma questions. A chi-square
analysis with Yates correction was used to determine differences. Questions and
literature were similar in the most frequently addressed items in each of the 4
areas: category type (taxonomy 3, treatment), 52.4% and 60.7%, respectively;
anatomic location (femur), 23.3% and 27.7%, respectively; orthopedic focus
(fracture), 51.0% and 56.5%, respectively; and treatment type
(multiple/nonspecific), 39.0% and 35.4%, respectively.
The content correlation found between the questions and literature
supports the idea that reviewing current literature may help prepare for the
trauma content on the Orthopaedic In-Training Examination.
Trauma is a leading cause of morbidity and mortality
in the United States and a major focus of orthopedic surgeons. Despite the at
least 12 months of exposure to orthopedic trauma required by the American Board
of Orthopaedic Surgery,1 the training that orthopedic residents
obtain in the treatment of trauma depends heavily on the experience, exposure,
and teaching provided by their programs. Currently, the only standardized
evaluation of trauma knowledge throughout orthopedic training is found in the
Orthopaedic In-Training Examination, which is administered annually to all
residents by the American Academy of Orthopaedic Surgeons. This examination,
initially administered in 1963, was the first specialty examination for
residents.2 It has since served as the model for the development and
administration of such tests in other fields of medicine, such as radiology,
general surgery, and internal medicine. The examination acts as an objective
comparison of orthopedic knowledge for residents of the same year-in-training
and orthopedic training programs in general and consists of 275 multiple choice
questions. Although recent orthopedic studies are recommended as a source of
preparation, we are unaware of any published study correlating trauma content
on the Orthopaedic In-Training Examination with the literature.
Our goals were to assess the Orthopaedic In-Training Examination to
(1) determine the content of the trauma questions, (2) identify the content of
the 3 most frequently referenced journals on the answer keys, and (3) evaluate
the correlation between those contents to aid in providing residents with
information that may prepare them for this portion of the examination.
Materials and Methods
Orthopaedic In-Training Examination Questions
We obtained the Orthopaedic In-Training Examination and associated
Program Director’s Reports for the years 2002 through 2007 and compiled a
database of the questions classified as trauma per these reports. From 2002
through 2007, two hundred ninety-two of 1650 Orthopaedic In-Training
Examination questions (18%) focused on trauma (range, 12.7%-18.9% per
year).
To determine content, we assessed 4 areas for each question: (1)
category type, (2) anatomic location, (3) orthopedic focus, and (4) treatment
type.
Under category, we classified questions based on the same 3 levels of
cognitive taxonomy used by the examination’s authors. The first level,
recall, tests factual knowledge. The second level, deductive, focuses on the
evaluation of information, including imaging studies, to develop a diagnosis.
The third level, analytic, requires the interpretation of provided information
to develop a treatment plan. The designation of category type was made by the
junior authors (P.F., D.R.M., J.R.M., F.J.F) and confirmed by the senior author
(S.C.M.), a trauma-fellowship-trained orthopedic surgeon. Any conflicts in
categorization were resolved by the senior author after discussion.
For the anatomic location, the AO classification was used. Questions
with multiple areas of focus were categorized under all applicable areas, and
those with no area in particular were classified as nonspecific.
For the orthopedic focus, we used 11 subset groups: (1) acute tendon
injury, (2) delayed reconstruction for malunion/nonunion, (3) fracture, (4)
geriatric trauma, (5) infection, (6) other complications, (7)
military/ballistic trauma, (8) other soft tissue injury, (9) periprosthetic
fracture, (10) dislocation, and (11) multiple/nonspecific.
For treatment types, there were 11 choices: (1) bone graft, (2)
external fixation, (3) locked plating, (4) arthroplasty, (5) intramedullary
nailing, (6) open reduction and internal fixation, (7) revision, (8) tendon
repair, (9) nonoperative treatment, (10) amputation, and (11)
multiple/nonspecific.
Orthopaedic In-Training Examination Cited
Literature
For each examination, the American Academy of Orthopaedic Surgeons
provides an answer key with cited references. To determine the most relevant
journals, we reviewed those references and determined the most frequently cited
journals. We then determined the number of trauma articles in each of these
journals from 2001 through 2006 by performing a systematic review of the PubMed
bibliographic database. Articles from the journals that did not have a primary
focus on trauma were excluded. For example, the article by Lane3
from the Journal of Orthopaedic Trauma was not included in the analysis
because the primary focus of the article is more reflective of Orthopaedic
In-Training Examination questions categorized in the answer key as being
related to orthopedic science.
We divided the number of times each journal was referenced on the
examination by the total number of trauma articles in each journal over the
time period, described as proportion of trauma referenced. The content of the
top 3 most frequently referenced journals was further characterized via the
same criteria as the Orthopaedic In-Training Examination trauma questions. We
compared the data, for each journal individually and for all journals
collectively, to the data for the Orthopaedic In-Training Examination
questions.
Statistical Analysis
A chi-square analysis with Yates correction was used to calculate the
difference in proportions. A P value <.05 was considered
statistically significant. A power analysis with use of the difference in
overall percentage of trauma questions on the Orthopaedic In-Training
Examination compared with the percentage of corresponding articles was
conducted as the primary research question. This procedure proved that our
sample size was sufficient to reveal the necessary P values to answer
our primary question at a power of >80%.
Results
Analysis of the trauma questions revealed that the most frequent
category type of question was type 3 (treatment; 52.4% of the questions), the
most frequent anatomic location was the femur (23.3% of the questions), the
most frequent orthopedic focus was fracture (51.0% of the questions), and the
most frequent treatment type was nonspecific (39.0% of the questions) (Table
1).
The 3 journals most frequently referenced on the Orthopaedic
In-Training Examination (The Journal of Bone and Joint Surgery, American
Volume; Clinical Orthopaedics and Related Research; and The Journal of
Orthopaedic Trauma) published 6534 articles over the corresponding years,
1378 (21.1%) of which were related to trauma (Figure, Table 2). This percentage
is significantly higher than the 17.7% ratio of trauma-related Orthopaedic
In-Training Examination questions during the study period
(P=.003).The Journal of Bone and Joint Surgery, American Volume
had the highest proportion of trauma articles referenced on the examination
(28.3%; 108 of 381); followed by The Journal of Orthopaedic Trauma
(16.6%; 106 of 639) and Clinical Orthopaedics and Related Research
(14.5%; 52 of 358). There was a correlation between the content of the trauma
articles from these 3 journals collectively and the types of test questions
asked (Figure, Table 1). In these collective 3 journals, the most frequent
category was taxonomy 3 (treatment; 60.7% of trauma articles), the most
frequent anatomic location was the femur (27.7% of trauma articles), the most
frequent orthopedic focus was fracture (56.5% of trauma articles), and the most
frequent treatment type was nonspecific (35.4% of trauma articles) (Table 1).
When the content of trauma articles from each journal was compared individually
to the trauma content on the Orthopaedic In-Training Examination, the most
frequent category type and orthopedic focus were the same. Exceptions were the
most frequent anatomic location in Clinical Orthopaedics and Related
Research and The Journal of Bone and Joint Surgery, American Volume
(ie, multiple/unspecified rather than the femur as on the examination and
in The Journal of Orthopaedic Trauma [Table 1]), and treatment type in
The Journal of Orthopaedic Trauma (ie, other internal fixation compared
with nonspecific in the other journals and on the examination).
 |
| Figure: Percentage of
Orthopaedic In-Training Examination and journals related to trauma. |
There were several significant differences between the content on the
examination and in the literature. Taxonomy 1 (recall) was addressed in 39.4%
of the test questions but in only 32.7% of the articles (P=.034).
Additionally, taxonomy 3 (treatment) was addressed in only 52.4% of the
questions but in >60% of the articles (P=.010). There were more
articles with a nonspecific anatomic location in the journals than in the
questions (26.9% vs 19.5%, respectively; P=.011), and fewer articles
with a focus on the pelvis than examination questions (9.4% vs 11.6%,
respectively; P=.049). Orthopedic focus was nonspecific in more
Orthopaedic In-Training Examination questions compared with journal articles
(15.8% vs 9.0%; P<.001, and there were more articles relating to
periprosthetic fractures compared with questions (2.1% vs 0.0%; P=.024).
Although the most frequently addressed treatment modality in both questions and
articles was nonspecific, bone grafting was addressed significantly less often
in the Orthopaedic In-Training Examination than in the literature (0.7% and
4.9%, respectively; P=.002), as was external fixation (3.4% and 7.3%,
respectively; P=.023) (Table 1).
Discussion
The Orthopaedic In-Training Examination is the only standardized,
objective national evaluation of orthopedic residents. The examination was
founded with 3 fundamental goals: (1) to measure resident knowledge against a
national mean, (2) to determine minimal standards for residents, and (3) to
measure the quality of training within various residency programs.2
The importance of good performance on the Orthopaedic In-Training Examination
is reinforced by the work of Dirschl et al,4 who found that good
performance on the Orthopaedic In-Training Examination correlates with a high
score on part I of the American Board of Orthopaedic Surgery examination.
However, a lack of a standardized curriculum makes preparation difficult.
Some data have suggested a statistically significant correlation
between studying The Journal of Bone and Joint Surgery, American Volume
and The Journal of the American Academy of Orthopaedic Surgeons and
strong performance on the Orthopaedic In-Training Examination.5 The
authors also found a significant correlation between higher scores and review
of previous examinations.5 Many residency programs rely on
literature review for resident education. Greene6 surveyed the
chairmen of all 161 orthopedic residency programs throughout the United States
and Canada to determine the prevalence of journal clubs. Of the 149 programs
that responded, journal clubs were used regularly in 147 (99%).6 Of
those clubs, 114 met once per month, 14 met biweekly, and 11 met once per week.
The author also found that the most popular format was articles from .1
scientific publication (82%).6
It is important to note that performance on the Orthopaedic
In-Training Examination does not correlate with evaluation of resident
performance by the attendings.7 Knowledge of the evaluation,
diagnosis, and treatment of trauma patients is an important aspect of becoming
a knowledgeable, prudent orthopedic surgeon. However, experience under good
supervision enables residents to develop the skills “to make the correct
decisions and carry them out under adverse circumstances.”8
Thus, it is important that any educational curriculum be treated as a
supplement, and not a substitute, to the clinical experience of caring for
trauma patients.
Previous studies have been performed in other orthopedic
subspecialties with regard to preparation for the Orthopaedic In-Training
Examination. Frassica et al9 classified the pathology section on the
examinations from 2002 through 2006 and found that 10% of the examination
consists of questions on this subject. They found that most questions required
interpretation of imaging and/or histologic material to develop a diagnosis or
treatment plan and concluded that systematic review of the most commonly tested
material would help prepare residents.9 A recent study performed by
Marker et al10 evaluated the osteonecrosis content on the
Orthopaedic In-Training Examination. Their results showed that the literature
had a larger percentage of articles on this subject matter than did questions
on the examination. The authors also determined the proportion of orthopedic
literature and examination questions focused on idiopathic scoliosis,
chondrosarcoma, femoral neck fracture, and hip resurfacing. They found that,
except for idiopathic scoliosis, the literature and test had similar
percentages of content, and they concluded that review of the current
literature would serve as a good preparatory tool for the Orthopaedic
In-Training Examination.10 Another study evaluated hand content on
the Orthopaedic In-Training Examination and classified the questions based on
diagnosis, treatment, subject matter, and anatomic location.11 The
authors found that the articles in the most frequently cited journals shared a
similar focus in those subject matters, suggesting that the review of select
current literature would aid in studying for the examination.11
These studies, combined with our findings, suggest that a curriculum
of journal clubs reviewing trauma articles from the 3 journals found to have
the highest proportion of referenced articles—The Journal of Bone and
Joint Surgery, American Volume; Clinical Orthopaedics and Related
Research; and The Journal of Orthopaedic Trauma—may prove the
most high-yield use of structured resident study time limited by the
implementation of the 80-hour work week. Such a strategy may be most useful if
implemented early in training and continued throughout residency.
Our study has several weaknesses. First, we evaluated the examination
and journals for only a limited time. However, that 6-year time frame is longer
than that in previously published studies evaluating the Orthopaedic
In-Training Examination.9-11 In addition, we elected to review the
trauma content of the journals from the years 2001 through 2006 as a
representation of the tested material from 2002 through 2007. Although most
test questions are written, reviewed, and approved for use on examinations
>1 year in advance, we thought those 6 years were a representative sample of
“current literature.” This decision also had a precedent in the
evaluation of the hand content on the Orthopaedic In-Training Examination by
Marker et al.11 The references as reported in the provided Program
Director’s Report may also contain inaccuracies because they may have been
added after the question was written or may not have led the examinee to the
correct answer. However, these references were provided by the test authors
themselves, and any inaccuracies can be attributed to the test-writing process
itself.
One might consider that The Journal of Orthopaedic Trauma might
have skewed our results because it had a significantly higher percentage of
trauma articles than the combined value for all 3 journals (75.0% vs 21.1%,
respectively; P<.001). However, by determining the proportion of
references to trauma articles, we showed that The Journal of Bone and Joint
Surgery, American Volume was more heavily weighted (108 references of 381
trauma articles [28.3%] vs 106 references of 639 trauma articles [16.6%] in
The Journal of Orthopaedic Trauma). In terms of our 4 parameters, the
content of trauma articles from The Journal of Orthopaedic Trauma was
consistent for the most part with that of the other journals and the
Orthopaedic In-Training Examination questions. The main differences were that
The Journal of Orthopaedic Trauma more frequently addressed the anatomic
areas of the pelvis than did the collective journals (14.5% vs 9.4%,
P=.049) and displayed a trend (although not a significant difference)
toward more tibia/fibula articles than were found in the collective journals
(24.4% vs 19.7%, P=.597) (Table 1).
Conclusion
Although some differences were found between the content of the
Orthopaedic In-Training Examination and the literature, both sources were
similar in terms of most frequently addressed topics. These results indicate
that studying the current trauma literature from The Journal of Bone and
Joint Surgery, American Volume, The Journal of Orthopaedic Trauma,
and Clinical Orthopaedics and Related Research provides content similar
to the trauma content on the Orthopaedic In-Training Examination, a finding
that is consistent with previous studies assessing other subspecialties within
the examination. We recommend additional studies evaluating the implementation
of such a protocol and the resultant performance on the Orthopaedic In-Training
Examination to reaffirm these suggestions.
References
- American Board of Orthopaedic Surgery. 2006 Rules and Procedures
for Residency Education Part I and Part II Examinations. Chapel Hill, NC:
American Board of Orthopaedic Surgery; 2006.
- Mankin HJ. The Orthopaedic In-Training Examination (OITE). Clin
Orthop Relat Res. 1971; (75):108-116.
- Lane JM. Bone morphogenic protein science and studies. J Orthop
Trauma. 2005; 19(10 Suppl):S17-S22.
- Dirschl DR, Campion ER, Gilliam K. Resident selection and
predictors of performance: can we be evidence based? Clin Orthop Relat
Res. 2006; (449):44-49.
- Miyamoto RG Jr, Klein GR, Walsh M, Zuckerman JD. Orthopedic surgery
residents’ study habits and performance on the orthopedic in-training
examination. Am J Orthop (Belle Mead NJ). 2007; 36(12):E185-E188.
- Greene WB. The role of journal clubs in orthopaedic surgery
residency programs. Clin Orthop Relat Res. 2000; (373):304-310.
- Dirschl DR, Dahners LE, Adams GL, Crouch JH, Wilson FC.
Correlating selection criteria with subsequent performance as residents.
Clin Orthop Relat Res. 2002; (399):265-271.
- Chapman MW. Education in orthopaedic trauma. Clin Orthop Relat
Res. 1997; (339):7-19.
- Frassica FJ, Papp D, McCarthy E, Weber K. Analysis of the
pathology section of the OITE will aid in trainee preparation [published online
ahead of print April 19, 2008]. Clin Orthop Relat Res. 2008;
466(6):1323-1328.
- Marker DR, Mont MA, Seyler TM, LaPorte DM, Frassica FJ. Current
literature: an educational tool to study osteonecrosis for the Orthopaedic
In-Training Examination? Orthop Clin North Am. 2009; 40(2):299-304.
- Marker DR, Mont MA, McGrath MS, Frassica FJ, LaPorte DM. Current
hand surgery literature as an educational tool for the orthopaedic in-training
examination. J Bone Joint Surg Am. 2009; 91(1):236-240.
Authors
Drs Farjoodi, Frassica, and Mears and Messrs Marker and McCallum are
from the Department of Orthopedic Surgery, The Johns Hopkins Bayview Medical
Center, Baltimore, Maryland.
Drs Farjoodi, Frassica, and Mears and Messrs Marker and McCallum have no
relevant financial relationships to disclose.
Correspondence should be addressed to: Simon C. Mears, MD, PhD, c/o
Elaine P. Henze, BJ, ELS, Department of Orthopedic Surgery, Johns Hopkins
Bayview Medical Center, 4940 Eastern Ave, #A665, Baltimore, MD 21224-2780
(ehenze1@jhmi.edu).
doi: 10.3928/01477447-20110317-33