Guest Editorial Free

How Can We Improve Outcomes Assessment?

Peter N. Chalmers, MD; Nikhil N. Verma, MD

The evolution of evidence-based medical practice is heavily dependent on outcomes assessment. From a personal perspective, without accurate and reliable outcomes assessments, it is difficult for individual orthopedic surgeons to evaluate the effects of their interventions and, more specifically, the direct value of these interventions to patients and their quality of life. From a specialty perspective, without accurate and reliable outcomes assessments, orthopedic surgeons are unable to compare treatment options or provide reliable information to patients regarding expectations for a given procedure. From a societal perspective, without accurate and reliable outcomes assessments, orthopedic surgeons will be unable to show the benefit and value of their clinical care and interventions directly impacting patient access to care and physician reimbursement.

Despite the obvious value of collecting outcomes data, many questions and difficulties remain in the process by which data are collected. Currently, most “patient-oriented” outcomes continue to be collected by either paper or, more recently, electronic-based questions, which ask patients to report their satisfaction, pain, or functional capabilities. Occasional “functional” outcomes may include provider assessment of range of motion and strength, but true measurements of patient-specific function are limited. Rarely do orthopedic outcomes involve direct evaluation of the patient’s ability to perform activities of daily living, athletic activities, or occupational activities.

The quality and lack of standardization of our outcome metrics is hindering our ability to improve as a specialty. For instance, within rotator cuff repair, one of the most commonly used outcomes assessment tools is the American Shoulder and Elbow Surgeons (ASES) score. Modern repair techniques and rehabilitation protocols lead to mean ASES scores of greater than 90 out of 100 points at the time of final follow-up.1 However, the mean clinically important difference for the ASES score for rotator cuff disease is 12 to 17 points.2 These numbers suggest that no future trial will ever demonstrate a clinically significant improvement in rotator cuff repair. Due to this “ceiling effect,” this outcome tool is unable to measure higher levels of function. For example, a patient may have a positive outcome on an ASES score, but may lack recreational function such as baseball or tennis. The ability to return to sports or work activities is of primary importance to many patients, but this aspect of outcome can be difficult to capture on current standardized outcome measures.

Variability in patient demand also degrades the value of our current functional outcomes. For example, an older patient may desire a functional return to activities of daily living such as buttoning a shirt or lifting a coffee mug, but a younger patient may be required to lift 100 lb or more to return to work. Although the 2 patients may report similar scores on current validated outcome measures, their levels of satisfaction with that outcome may be markedly different. Although pain can be measured fairly directly using a numeric scale, functional capabilities, particularly those specific to an individual patient, are much more difficult to quantify. To represent true patient satisfaction, outcome measurements must become more patient-specific and allow direct measurement of patients’ functional capabilities in a real-world setting.

A recent randomized clinical trial provided another example of the ceiling effect. No difference was found between nonoperative treatment and open reduction and internal fixation for proximal humerus fractures.3 This study used the Oxford Shoulder Score, which has a maximum of 48 points and a mean clinically important difference of 5 points.4 However, within this trial, the nonoperative group had a mean final follow-up Oxford Shoulder Score of 40 points, leaving only 3 potential points available for improvement by open reduction and internal fixation to demonstrate a meaningful difference. The ceiling effect is evident from the questions asked within the survey, which include “Could you carry a tray containing a plate of food across a room?” and “Have you been able to use a knife and fork at the same time?” Although these questions establish a baseline level of function, they have no ability to determine or differentiate higher functions of recreational or occupational activities. Similar issues have plagued many of the recently conducted randomized clinical trials comparing operative and nonoperative treatment for orthopedic surgery.5,6 These fundamental flaws in outcome assessment limit the ability of orthopedic surgeons to show the value of the surgical interventions they perform or to show improvement in outcomes associated with new technology or techniques.

In addition to these flaws in the actual outcome measures, the current process by which data are collected is tedious, time-consuming, and expensive. The current survey instruments place a significant burden on both the patient and the practitioner. From the patient’s perspective, surveys are time-consuming to complete and often repetitive in nature. As a result, compliance is low, despite significant resources and effort expended by research and clinical staff. For the clinician, the institution of a data collection process requires significant capital expenditure and ongoing costs for additional staff and software. Patients must be assigned appropriate outcome scores based on diagnosis, and must be followed closely to maximize compliance at given time intervals. Finally, one must maintain a database of the information in a usable format, and assure compliance with patient privacy provisions. Despite our best efforts, compliance remains a significant limitation.

Fortunately, developing technologies may offer some solutions to these intractable problems. Recently, many patients have been voluntarily collecting their functional assessment data using wearable sensors and activity monitors. These devices, usually worn as bracelets, use accelerometers and gyroscopes to measure arm movement, which is then translated into steps taken per day and calories burned. Increasingly, these devices also incorporate global positioning system sensors to allow direct measurement of distance traveled. These devices can directly report the distance walked or ran each day. Many new mobile telephones are also incorporating these sensors and this software; thus, patients are not required to purchase and wear a separate device. Wristwatches are incorporating comparable technologies for similar reasons.

Eventually, these new technologies could be important data sources for orthopedic surgeons. Simple use of the device may give vital real-time information, such as distance walked after total knee arthroplasty or an approximation of shoulder range of motion after rotator cuff repair, if the device is held in the operative extremity. Incorporation of directly measured data could effectively remove the ceiling imposed by many currently used scoring scales, allowing a more critical assessment of outcomes. In addition, tracking of real-time data yields innumerably more datapoints providing finer data granularity and allowing measurement of earlier recovery of function, which could translate into increased value by reducing time missed from work or recreation. The data can also be used in real time to alert the clinician to the development of complications, as a sudden reduction in ambulation distance could potentially signal an adverse medical event.

Within other areas of science, these small traces of daily collected data are beginning to be incorporated in what has been called the “small data” revolution, in contrast to the “big data” revolution that is currently sweeping the orthopedic literature.7 Certainly these data are widely being exploited commercially. For instance, numerous cell phone applications offer comprehensive diabetes management with incorporation of blood sugars directly uploaded from the patient’s meter, the ability to import exercise data from wearable sensors, and a wireless, network-based export function to share results with physicians.8 As compliance has become a stumbling block commercially, patient privacy will need to be carefully considered and handled if these data are to be used scientifically.

Another potential opportunity for advancing technology is in the field of motion analysis. Moving out of the laboratory and into the real world will allow further assessment of normal vs abnormal functional recovery and potentially identify individuals at risk for injury. The continued enhancement of markerless motion analysis allows improved in vivo assessment of complex motion patterns such as overhand throwing or swinging a golf club. As progress has been made in image recognition algorithms and gesture-based user interfaces, software methods are becoming available to analyze human motion using high-definition video. Many consumer electronic systems have begun to incorporate these methods. As an example of the capabilities of such systems, one recent image analysis study measured heart rate from standard-quality video based on small fluctuations in facial skin tone.9 Although few studies have employed these methods within orthopedics,10 they offer the promise of real-time motion analysis outside of the controlled laboratory setting.11,12 Much work remains to be done, but more mobile quantitative functional assessment may soon be available for clinical research use.

To continue to improve clinically, we must update orthopedics outcomes assessment to create modern measures that reflect the high patient and societal expectations from our procedures. Our current outcome measures are rapidly falling behind the available technology. Incorporation of wearable sensor data, markerless motion analysis, and other budding technologies may allow large-scale real-time functional assessment, bypassing limitations such as the ceiling effect as well as the cost and burden associated with our current outcomes assessment tools. Innovations such as these offer the promise for orthopedic surgeons to assess individual surgeon outcomes, to accurately compare treatments, and to demonstrate the magnitude of the functional benefit provided by their treatments to society.


  1. Keener JD, Galatz LM, Stobbs-Cucchi G, Patton R, Yamaguchi K. Rehabilitation following arthroscopic rotator cuff repair: a prospective randomized trial of immobilization compared with early motion. J Bone Joint Surg Am. 2014; 96 (1):11–19. doi:10.2106/JBJS.M.00034 [CrossRef]
  2. Tashjian RZ, Deloach J, Green A, Porucznik CA, Powell AP. Minimal clinically important differences in ASES and simple shoulder test scores after nonoperative treatment of rotator cuff disease. J Bone Joint Surg Am. 2010; 92 (2):296–303. doi:10.2106/JBJS.H.01296 [CrossRef]
  3. Rangan A, Handoll H, Brealey S, et al. Surgical vs nonsurgical treatment of adults with displaced fractures of the proximal humerus: the PROFHER randomized clinical trial. JAMA. 2015; 313 (10):1037–1047. doi:10.1001/jama.2015.1629 [CrossRef]
  4. Dawson J, Rogers K, Fitzpatrick R, Carr A. The Oxford shoulder score revisited. Arch Orthop Trauma Surg. 2009; 129 (1):119–123. doi:10.1007/s00402-007-0549-7 [CrossRef]
  5. Sihvonen R, Paavola M, Malmivaara A, et al. Arthroscopic partial meniscectomy versus sham surgery for a degenerative meniscal tear. N Engl J Med. 2013; 369 (26):2515–2524. doi:10.1056/NEJMoa1305189 [CrossRef]
  6. Frobell RB, Roos EM, Roos HP, Ranstam J, Lohmander LS. A randomized trial of treatment for acute anterior cruciate ligament tears. N Engl J Med. 2010; 363 (4):331–342. doi:10.1056/NEJMoa0907797 [CrossRef]
  7. Estrin D. The building blocks of small data apps. http://www.smalldata.io. Accessed April 30, 2015.
  8. Pietrangelo A. The best diabetes iPhone & Android apps of the year. http://www.healthline.com/health-slideshow/top-iphone-android-apps-diabetes. Accessed April 30, 2015.
  9. Wu H-Y, Rubinstein M, Shih E, Guttag J, Durand F, Freeman W. Eulerian video magnification for revealing subtle changes in the world. ACM Trans Graph. 2012; 31 (4):1–8. doi:10.1145/2185520.2185561 [CrossRef]
  10. Chalmers PN, Trombley R, Cip J, et al. Postoperative restoration of upper extremity motion and neuromuscular control during the overhand pitch: evaluation of tenodesis and repair for superior labral anterior-posterior tears. Am J Sports Med. 2014; 42 (12):2825–2836. doi:10.1177/0363546514551924 [CrossRef]
  11. Corazza S, Gambaretto E, Mündermann L, Andriacchi TP. Automatic generation of a subject-specific model for accurate markerless motion capture and biomechanical applications. IEEE Trans Biomed Eng. 2010; 57 (4):806–812. doi:10.1109/TBME.2008.2002103 [CrossRef]
  12. Mündermann L, Corazza S, Andriacchi TP. The evolution of methods for the capture of human movement leading to markerless motion capture for biomechanical applications. J Neuroeng Rehabil. 2006; 3:6. doi:10.1186/1743-0003-3-6 [CrossRef]

The authors are from the Department of Orthopaedic Surgery, Rush University Medical Center, Chicago, Illinois.

Dr Chalmers has no relevant financial relationships to disclose. Dr Verma is a paid consultant for Minivasive and Smith & Nephew; receives research support from Arthrex, Arthrosurface, Athletico, ConMed Linvatec, DJ Orthopaedics, Miomed, Mitek, and Smith & Nephew; holds stock in Cymedica, Minivasive, and Omeros; and receives royalties from Smith & Nephew.

Correspondence should be addressed to: Peter N. Chalmers, MD, Department of Orthopaedic Surgery, Rush University Medical Center, 1611 W Harrison St, Ste 200, Chicago, IL 60612 (p.n.chalmers@gmail.com).


Sign up to receive

Journal E-contents