The American Board of Orthopaedic Surgery (ABOS) was established in 1934 with the intention of creating a universal standard of education. By 1936, the ABOS instituted the first formal requirements for certification to monitor physician quality and to ensure public trust.1 In addition to the certification examination administered by the ABOS, the American Academy of Orthopaedic Surgeons (AAOS) developed the Orthopaedic In-Training Examination (OITE) in 1963 to assess whether each orthopedic residency program was attaining its educational goals and maintaining a minimum standard for orthopedic education. Currently, this annual examination consists of 275 questions divided among 12 sections, with the Sports Medicine content domain comprising approximately 8% of the OITE.2 These questions are based on meaningful publications that highlight concepts deemed most important by test makers.
In 2003, The Journal of Bone and Joint Surgery (JBJS) began rating the levels of evidence (LoE) of its published studies and The American Journal of Sports Medicine (AJSM) began a similar practice shortly thereafter with its own categorization based on work from the Oxford Centre for Evidence-Based Medicine.3,4 Coinciding with these ratings, the proportion of level 1 and 2 studies in AJSM increased significantly from 1996 to 2011.5
Despite a trend toward the publication of higher-quality studies in the orthopedic literature, there are few analyses examining whether there has been an increase in the citation of publications with the highest LoE on the OITE. Recently, there have been investigations focusing on the LoE in content domains other than the Sports Medicine section. Walsh et al6 and Grandizio et al7 found significant increases in the LoE of publications cited in the Foot and Ankle section and the Hand section, respectively, during their 15-year study periods. However, despite these positive trends, the majority of the references cited on the OITEs from 2010 to 2012 were level 4 or 5 evidence.8
The primary purpose of this study was to assess the trends related to LoE of cited references in the Sports Medicine content domain on the OITE during a recent 12-year period (2005–2016). Secondary purposes were to analyze trends in question taxonomy, age of references, and most commonly cited publications. The authors' null hypotheses were that there would be more cited references of the highest LoE and that average question taxonomy has increased during the past 12 years of examinations.
Materials and Methods
Orthopaedic In-Training Examinations from 2005 to 2007 and from 2014 to 2016 were obtained, and the questions and their associated references within the Sports Medicine sections were evaluated by 2 of the authors (J.C.L., A.D.S.). Data were compared between the 2 groups of examinations; the past examination group was from 2005 to 2007, and the present examination group was from 2014 to 2016. Total numbers of sports medicine questions and the percentage of the OITEs dedicated to these questions were determined.
The question taxonomy was determined based on the classification system of Buckwalter et al.9 Taxonomic level I (ie, tax I) was assigned to questions that tested isolated recognition and recall, tax II was assigned to those testing comprehension and interpretation, and tax III was assigned to those testing the application of knowledge. The number of questions with each taxonomic level from the past and present examination groups was summed, and the proportion of questions with each taxonomic level was compared between groups. Mean taxonomic levels were then compared.
Number of References
The full manuscripts of all journal articles cited in the examination questions were reviewed. Summaries and descriptions of other nonprimary references (eg, books, technique guides, etc) were also evaluated. The number of references for each question was determined and totaled for each examination group. Mean number of references per question and proportions of questions with 1, 2, or 3 or more references were compared between examination groups.
Level of Evidence
The LoE of each reference was categorized based on the recommendations from the AAOS.10 References were categorized as being either of the highest LoE (levels 1 and 2), high LoE (levels 1–3), low LoE (levels 4–5), or review and other nonprimary references. The total number of references in each category was determined, and proportions of references in these categories were compared between past and present examination groups.
Age of References
The year of publication was determined for each reference, and the age of the reference was calculated by subtracting the year of reference publication from the year of examination administration. Mean reference age was calculated for the past and present examination groups and compared.
Most Common Sources
The number of unique publications (ie, sources) were summed for the past and present groups, and the citation frequency for each source was totaled. After sorting the sources by citation frequency, the number of review and non-journal publications in the top 10 most-cited sources was determined.
Descriptive statistics were used to identify means and proportions in all data categories. Continuous data for the past and present examination groups were compared with independent, 2-tailed Student's t tests. Categorical data for the examination groups were compared using 2 × 2 contingency tables and Fisher's exact test. All analyses were performed using the GraphPad QuickCalcs website (GraphPad Software, La Jolla, California). Statistical significance was present in comparison testing if P<.05.
Review of the 6 years of OITEs yielded 125 questions and 264 references within the Sports Medicine sections. Questions within the Sports Medicine section accounted for 8% of the total number of questions on both past and present examinations (P=1.00). Characteristics of the sports medicine questions and their associated references on all examinations are summarized in Table 1.
Question Taxonomy and Level of Evidence for References Used for Sports Medicine Questions on Orthopaedic In-Training Examinations Between 2005 and 2007 and Between 2014 and 2016
The mean taxonomic level of questions on present OITEs had increased from past examinations (1.52±0.76 vs 2.26±0.89, P=.0001). This increase was driven by a significant reduction in tax I questions (63% vs 29%, P=.0002) and a significant increase (244%) in tax III questions (16% vs 55%, P=.0001).
Despite no difference in the percentage of sports medicine questions on the past and present OITEs, present OITEs cited 28 (24%) more references than on past examinations, increasing the mean number of references per question by 19% (from 1.97±0.41 to 2.35±0.66, P=.0002). There were no questions on present OITEs citing only 1 reference compared with 10% of questions on past examinations (P=.012), and there was a higher percentage of questions on present examinations citing 3 or more references than on past examinations (27% vs 7%, P=.0034).
There were many significant differences in the LoE of the references cited on past and present examinations. Significantly higher percentages of references on present OITEs were of the highest levels of evidence (21% vs 10%, P=.027) and high levels of evidence (32% vs 18%, P=.015) compared with past examinations. Fewer citations were generated from review and other nonprimary references on present compared with past OITEs (36% vs 59%, P=.0002).
References on present examinations were older than those cited on past examinations (7.14±7.20 vs 5.08±4.61 years, P=.007). Past examinations cited work from 31 different sources, whereas present examinations cited work from 42 different sources. The top 10 most-cited sources on past examinations included 6 review and non-journal publications, whereas 2 of the top 10 most-cited sources on present examinations were review and non-journal publications. The most commonly cited source for both examination groups was AJSM, generating 31% of references on past examinations and 20% of references on present examinations. The second and third most commonly cited sources were Orthopedic Knowledge Update: Sports Medicine 3 (13%) and Instructional Course Lectures (6%) on past examinations and the Journal of the Academy of Orthopaedic Surgeons (14%) and JBJS (12%) on present examinations.
The OITE has been a perennial assessment of orthopedic resident proficiency since 1963. It serves as a benchmark for residents, and scores on this examination have been correlated with ABOS Part I performance.11,12 The importance of the examination has resulted in many investigations into its structure and composition to aid residents in their preparation and development of a foundation in orthopedic knowledge.13–17 Although the Sports Medicine section comprises a substantial proportion of the examination,2 there has been no publicly available investigation into the complexity of the examination questions and the LoE of the cited references in this section. The authors have demonstrated that during the past 12 years, the questions on the OITE have become more difficult, with references that are more numerous, of higher LoE, older, and from major orthopedic journals.
The current results indicate that the Sports Medicine section of the present examinations (2014–2016) had significantly more references of high and the highest LoE than past examinations (2005–2007). This positive trend is consistent with recent changes in other content domains of the OITE. Walsh et al6 and Grandizio et al7 found that the questions on the Foot and Ankle section and Hand section, respectively, cited references with higher LoE on OITEs from 2010 to 2012 than OITEs from 1995 to 1997. The increasing LoE of the cited references in OITE questions models the changes seen in orthopedic literature on the whole since documenting the LoE for journal articles was initiated by JBJS in 2003.3 Many major orthopedic journals, including AJSM, have begun noting the LoE of their published manuscripts and have also placed a focus on increasing quality of scientific methodology.4,18–20 Consistent with this emphasis, the proportion of studies with level 1 and 2 evidence in AJSM increased significantly from 1996 to 2011.5 Despite this trend, studies with lower LoE still predominate the orthopedic literature.8 The current results also demonstrate this; references with low LoE (levels 4 to 5) comprise 33% of all sources and 51% of all journal articles cited on the Sports Medicine sections of present OITEs. Although there is a growing emphasis on studies with high LoE, lower-level studies should not be neglected. It is not possible to perform randomized, controlled trials to answer all clinical questions,21 so studies of lower LoE can often be designed to provide guidance on appropriate management.
Although the current study indicates that the LoE of the references cited in the questions in the Sports Medicine section of the OITE have changed over time, so too have the characteristics of the questions themselves. Question taxonomy can serve as a surrogate for cognitive complexity.9 Many medical specialties have found it imperative to assess the knowledge of trainees with complex questions that require them to “manipulate factual knowledge to solve diagnostic problems and develop treatment plans.”22 The current results demonstrate that the average question taxonomy increased almost 50% on present examinations and that questions of the highest taxonomic order (tax III) are now the most commonly used. Similar trends have been observed in other content domains of the OITE.6,7,14,15 Given that the most common study technique of residents for the OITE has been the completion of practice questions,23,24 the current results suggest that residents may be able to better prepare for the OITE, and for clinical practice, by focusing on tax III questions rather than basic recall (tax I) questions in their studying.
The limitations to this study are similar to other related investigations.2,15–17 This study focused on questions that were listed in the Sports Medicine section of the OITE. Although this includes the majority of sports medicine–related questions on the examinations, sports medicine concepts are often tested in the Shoulder and Elbow, Foot and Ankle, and Pediatric sections as well. Therefore, the authors may not have considered all of the sports medicine–related questions on the OITE examinations from the past 12 years. Furthermore, although the authors used guidelines set forth by the AAOS10 to assign the LoE to each journal article, they did not cross-reference to any LoE listed in the description, abstract, or body of the manuscripts, which may have caused some discrepancies. In addition, assigning a taxonomic level to an examination question is somewhat subjective; thus, the authors' assignments may be different from those of other reviewers. However, there seems to be high concordance among those evaluating questions, whether they be orthopedic residents or OITE committee members,9 therefore reducing the concern of subjectivity in the current analysis. Finally, the authors were unable to obtain 3 references for 3 questions on the 2005 examination and were therefore unable assign LoE for them. A post hoc analysis including these 3 references and making the assumption that they are all level 1 or 2 evidence indicated that only the comparison between the proportions of references being of the highest LoE would lose significance (P=.07). Despite this, only 16% of all references were level 1 or 2 evidence, so it is improbable that all 3 would be of the highest LoE.
There are more publications with high LoE and fewer review and nonprimary sources cited in questions within the Sports Medicine content domain of the OITE on recent examinations compared with examinations produced 12 years ago. Despite this, the majority of references cited are not level 1 to 3 studies, even with a 24% increase in the number of references cited. Questions have also become more challenging on present examinations: the average taxonomic level of questions has significantly increased and has been driven by a 244% increase in tax III questions. The OITE is constantly evolving; however, the results of this study may allow for residents to better prepare for the annual test and to build a foundation of orthopedic knowledge from sources deemed most important by test makers.
- American Board of Orthopaedic Surgery. History of the American Board of Orthopaedic Surgery. https://www.abos.org/about-abos/history.aspx. Accessed August 6, 2017.
- Osbahr DC, Cross MB, Bedi A, et al. Orthopaedic In-Training Examination: an analysis of the sports medicine section. Am J Sports Med. 2011;39(3):532–537. doi:10.1177/0363546510387492 [CrossRef] PMID:21193591
- Wright JG, Swiontkowski MF, Heckman JD. Introducing levels of evidence to the journal. J Bone Joint Surg Am. 2003;85(1):1–3. http://www.ncbi.nlm.nih.gov/pubmed/12533564
- Reider B. Read early and often. Am J Sports Med. 2005;33(1):21–22. doi:10.1177/0363546503262911b [CrossRef] PMID:15610994
- Cvetanovich GL, Fillingham YA, Harris JD, Erickson BJ, Verma NN, Bach BR Jr, . Publication and level of evidence trends in the American Journal of Sports Medicine from 1996 to 2011. Am J Sports Med. 2015;43(1):220–225. doi:10.1177/0363546514528790 [CrossRef] PMID:24723417
- Walsh CT, Grandizio LC, Klena JC, Parenti JM, Cush GJ. Levels of evidence for foot and ankle questions on the Orthopaedic In-Training Examination: 15-year trends. J Surg Educ. 2016;73(6):999–1003. doi:10.1016/j.jsurg.2016.05.019 [CrossRef] PMID:27569751
- Grandizio LC, Huston JC, Shim SS, Graham J, Klena JC. Levels of evidence for hand questions on the Orthopaedic In-Training Examination. Hand (N Y). 2016;11(4):484–488. doi:10.1177/1558944715620793 [CrossRef] PMID:28149219
- Haughom BD, Goldstein Z, Hellman MD, Yi PH, Frank RM, Levine BR. An analysis of references used for the Orthopaedic In-Training Examination: what are their levels of evidence and journal impact factors?Clin Orthop Relat Res. 2014;472(12):4024–4032. doi:10.1007/s11999-014-3895-0 [CrossRef] PMID:25156901
- Buckwalter JA, Schumacher R, Albright JP, Cooper RR. Use of an educational taxonomy for evaluation of cognitive performance. J Med Educ. 1981;56(2):115–121. http://www.ncbi.nlm.nih.gov/pubmed/7463444
- Wright J. Levels of evidence and grades of recommendations. http://www2.aaos.org/bulletin/apr05/fline9.asp. Accessed August 6, 2017.
- Dougherty PJ, Walter N, Schilling P, Najibi S, Herkowitz H. Do scores of the USMLE Step 1 and OITE correlate with the ABOS Part I certifying examination? A multicenter study. Clin Orthop Relat Res. 2010;468(10):2797–2802. doi:10.1007/s11999-010-1327-3 [CrossRef] PMID:20352386
- Herndon JH, Allan BJ, Dyer G, Jawa A, Zurakowski D. Predictors of success on the American Board of Orthopaedic Surgery examination. Clin Orthop Relat Res. 2009;467(9):2436–2445. doi:10.1007/s11999-009-0939-y [CrossRef] PMID:19557490
- Srinivasan RC, Seybold JD, Salata MJ, Miller BS. An analysis of the Orthopaedic In-Training Examination sports section: the importance of reviewing the current orthopaedic subspecialty literature. J Bone Joint Surg Am. 2010;92(3):778–782. doi:10.2106/JBJS.I.00466 [CrossRef] PMID:20194338
- Sheibani-Rad S, Arnoczky SP, Walter NE. Analysis of the basic science section of the Orthopaedic In-Training Examination. Orthopedics. 2012;35(8):e1251–e1255. doi:10.3928/01477447-20120725-28 [CrossRef] PMID:22868614
- Cross MB, Osbahr DC, Nam D, Reinhardt KR, Bostrom MPG, Dines JS. An analysis of the hip and knee reconstruction section of the Orthopaedic In-Training Examination. Orthopedics. 2011;34(9):e550–e555. doi:10.3928/01477447-20110714-10 [CrossRef] PMID:21902156
- Lackey WG, Jeray KJ, Tanner S. Analysis of the musculoskeletal trauma section of the Orthopaedic In-Training Examination (OITE). J Orthop Trauma. 2011;25(4):238–242. doi:10.1097/BOT.0b013e3181e59da9 [CrossRef] PMID:21399475
- Frassica FJ, Papp D, McCarthy E, Weber K. Analysis of the pathology section of the OITE will aid in trainee preparation. Clin Orthop Relat Res. 2008;466(6):1323–1328. doi:10.1007/s11999-008-0235-2 [CrossRef] PMID:18425562
- Slobogean G, Bhandari M. Introducing levels of evidence to the Journal of Orthopaedic Trauma: implementation and future directions. J Orthop Trauma. 2012;26(3):127–128. doi:10.1097/BOT.0b013e318247c931 [CrossRef] PMID:22330974
- Wupperman R, Davis R, Obremskey WT. Level of evidence in Spine compared to other orthopedic journals. Spine. 2007;32(3):388–393. doi:10.1097/01.brs.0000254109.12449.6c [CrossRef] PMID:17268275
- Thordarson DB. Levels of evidence, evidence-based medicine, and Foot & Ankle International. Foot Ankle Int. 2008;29(9):881–882. doi:10.3113/FAI.2008.0881 [CrossRef]
- McLeod RS, Wright JG, Solomon MJ, Hu X, Walters BC, Lossing A. Randomized controlled trials in surgery: issues and problems. Surgery. 1996;119(5):483–486. http://www.ncbi.nlm.nih.gov/pubmed/8619200
- Miller DA, Sadler JZ, Mohl PC, Melchiode GA. The cognitive context of examinations in psychiatry using Bloom's taxonomy. Med Educ. 1991;25(6):480–484. http://www.ncbi.nlm.nih.gov/pubmed/1758329
- Boody B, Johnson P, Pugely A, et al. Validation of a web-based curriculum for resident education in orthopedic surgery. J Surg Educ. 2016;73(6):1060–1065. doi:10.1016/j.jsurg.2016.05.020 [CrossRef] PMID:27453538
- Evaniew N, Holt G, Kreuger S, et al. The Orthopaedic In-Training Examination: perspectives of program directors and residents from the United States and Canada. J Surg Educ. 2013;70(4):528–536. doi:10.1016/j.jsurg.2013.04.002 [CrossRef] PMID:23725942
Question Taxonomy and Level of Evidence for References Used for Sports Medicine Questions on Orthopaedic In-Training Examinations Between 2005 and 2007 and Between 2014 and 2016
|Sports questions, No. (% of all questions)||125 (8)||63 (8)||62 (8)||1.00|
|Buckwalter classification of question taxonomy|
| Tax I, No. (% sports questions)||58 (46)||40 (63)||18 (29)||.0002a|
| Tax II, No. (%)||23 (18)||13 (21)||10 (16)||.65|
| Tax III, No. (%)||44 (35)||10 (16)||34 (55)||.0001a|
| Mean (SD)||1.89 (0.90)||1.52 (0.76)||2.26 (0.89)||.0001a|
| Total, No.||264||118||146|
| Per question, mean (SD)||2.11||1.97 (0.41)||2.35 (0.66)||.0002a|
| 1, No. (%)||6 (5)||6 (10)||0 (0)||.012a|
| 2, No. (%)||95 (79)||50 (83)||45 (73)||.19|
| 3+, No. (%)||21 (17)||4 (7)||17 (27)||.0034a|
| Mean (SD)||2.16 (0.58)||1.97 (0.41)||2.35 (0.66)||.0002a|
|Level of evidence, No. (%)|
| Highest (levels 1 and 2)||42 (16)||12 (10)||30 (21)||.027a|
| High (levels 1–3)||67 (25)||21 (18)||46 (32)||.015a|
| Low (levels 4–5)||75 (28)||27 (23)||48 (33)||.077|
| Review and other nonprimary sources||122 (46)||70 (59)||52 (36)||.0002a|
|Age of sources, mean (SD), y||6.22 (6.25)||5.08 (4.61)||7.14 (7.20)||.007a|