Patients and Methods
The protocol was approved by the institutional review board at the Medical University of South Carolina for Human Research and informed consent was obtained from the patients' parents and/or guardians prior to enrollment. The protocol for this continuing prospective study has been previously reported.6,8,11,12 Patients between 6 and 16 years old who presented for a complete pediatric ophthalmological examination when study personnel were available between June 2012 and March 2017 were included in the study. Children were screened with the Spot Vision Screener by lay personnel prior to the ophthalmic examination. The Spot Vision Screener software v.2.1.4 with installed manufacturer “out-of-the-box” cutoffs was used.13
The Spot Vision Screener provides a report of pupillary diameter, ocular alignment, estimated binocular refraction, and referral recommendation: “all measurements within range” or “complete eye exam[ination] recommended.” Several attempts were made to obtain a successful reading. Patients for whom the screening was not able to be completed (ie, no refraction estimate or recommendation) were included in the study as automatic referrals. A comprehensive ophthalmic examination was performed by a pediatric ophthalmologist masked to the screener results. All patients had an evaluation of vision, stereopsis, motility, alignment, and the anterior segment. Patients underwent cycloplegic refraction and a fundus examination 30 to 40 minutes after instillation of proparacaine hydrochloride ophthalmic solution 0.5% USP followed by one or two drops of tropicamide 1%, phenylephrine 2.5%, and cyclopentolate 1%. The ages and demographics of the patients were collected. The following data were collected from the Spot Vision Screener: whether a screening examination was successfully completed, the estimated refraction, and the referral recommendation. The following data were collected from the physician examination: motility, alignment, cycloplegic refraction, and presence of systemic or ocular pathology.
The primary goal of this study was to evaluate the predictive ability of the Spot Vision Screener to correctly identify significant refractive errors and strabismus in our study population. Results of the Spot Vision Screener were compared with three “gold standard” criteria based on the results of the comprehensive examination (Table 1). Initial analysis (gold standard 1) was done using the 2013 guidelines of the American Association for Pediatric Ophthalmology and Strabismus (AAPOS) amblyopia risk factors as defined for the oldest age group (< 48 months of age).14
Refractive Thresholds Used for the Three Gold Standards Used for Evaluation
Because the focus of screening school-aged children shifts from the detection of amblyopia risk factors to the detection of eye conditions such as refractive error that may impact academic performance,15 two additional analyses were performed with more narrow criteria. A second analysis (gold standard 2) was performed using the Spot Vision Screener manufacturer's criteria for referral for the age group between 72 and 240 months as detected on the physician examination. A third analysis (gold standard 3) was performed using the Spot Vision Screener manufacturer's criteria, except the threshold for astigmatism was narrowed to +1.00 diopters (D) or greater, myopia was changed to −0.75 D or greater, and hyperopia was changed to +2.00 D or greater. The amblyopia risk factors non-refractive criteria of significant media opacity and strabismus greater than 8 PD were included in analyses of the predictive ability of the Spot Vision Screener for the increasingly stringent gold standards.
Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of the Spot Vision Screener in detecting each gold standard criterion in our population were calculated. Because PPV and NPV vary with disease prevalence,16,17 these values were also estimated, assuming a fixed sensitivity and specificity, for a general population with an estimated disease prevalence of 24%.18
Calculation of the area under the receiver operating characteristic (ROC) curve provides a visual and statistical estimation of the predictive accuracy of a test. Because the Spot Vision Screener software can be manipulated by the user to adjust the thresholds for referral for each refractive parameter, we sought to identify the most accurate threshold settings for each parameter and for all four refractive parameters in this age group.
The associations between the gold standard hyperopia or myopia diagnoses and the Spot Vision Screener estimated spherical equivalent, the gold standard astigmatism diagnoses and the Spot Vision Screener estimated astigmatism, and the gold standard anisometropia diagnoses and anisometropia calculated from Spot Vision Screener estimates were evaluated using a logistic regression approach. This process was used for each of the three increasingly narrow gold standards. The prediction performance for the Spot Vision Screener refractive parameter used to predict each diagnosis was estimated using the area under the ROC curve. Additionally, we identified the best threshold for each refractive parameter to detect each diagnosis by determining the cutoff that maximized the chi-square statistic for the logistic regression model for each outcome regressed on the dichotomized Spot Vision Screener parameter.19 Sensitivity and specificity of the selected Spot Vision Screener parameter dichotomized at the selected threshold were evaluated for the different diagnoses. We also considered combinations of thresholds for the four parameters to identify the joint combination that provided the best cutoff for discriminating whether a child had a significant refractive error (ie, had at least one of the four refractive conditions) to determine if adjusting the Spot Vision Screener criterion improved performance on the Spot Vision Screener test. All analyses were conducted in R v.3.5.2. for Windows software (Microsoft Corporation, Redmond, WA).
Results of the Spot Vision Screener and ophthalmologist examinations were collected for 330 sequential, eligible children between 72 and 221 months old (mean ± standard deviation [SD]: 103 ± 28 months) (Table 2). The Spot Vision Screener successfully evaluated 313 of 330 children (95%) in the current study. The Spot Vision Screener was unable to obtain a reading for 17 children: 3 were reported by the Spot Vision Screener as having “pupils [that were] too small,” 10 were reported as “machine could not read,” despite repeated attempts and adjustments, and 4 were deemed “uncooperative” by the tester.
Eleven children were found to have constant strabismus of greater than 8 PD on the physician examination. Of these, 6 children had esotropia, 4 had exotropia, and 1 had hypertropia. All but one child with Duane syndrome and exotropia in primary gaze were referred by the Spot Vision Screener.
Prediction performance of the Spot Vision Screener for the three gold standards and estimated PPV and NPV are presented in Table 3. The prevalence of 2013 AAPOS 48-month guidelines of amblyopia risk factors (gold standard 1) was found to be 42% in this patient population. The sensitivity of the Spot Vision Screener to detect AAPOS amblyopia risk factors was 89.5% and the specificity was 76.7%. The PPV was 73.9% and the NPV was 90.8%. As disease prevalence decreases, assuming a fixed test sensitivity and specificity, the PPV is expected to decrease and the NPV is expected to increase.16,17 The prevalence of amblyopia risk factors in our population of pediatric ophthalmology patients (42%) is higher than that estimated for the general population. For a general population with an estimated disease prevalence of 24%, the PPV is estimated to be 54.8% and the NPV is estimated to be 95.8%. Twelve children were not referred by the Spot Vision Screener but were found to have amblyopia risk factors on examination. Of the 12 children, 6 demonstrated hyperopia (+3.75 to 6.50 D), 3 demonstrated astigmatism (+1.75 to 2.25 D), 2 demonstrated anisometropia (1.80 to 2.00 D), and 1 demonstrated myopia (−2.25 D).
Spot Vision Screener Predictability Parameters for Gold Standard Criteria
Sensitivity decreased with the narrower criteria to 80% with gold standard 2 and 75.3% with gold standard 3, with increasing specificity. For gold standards 2 and 3, 35 and 47 children, respectively, who met the criteria were passed by the Spot Vision Screener. Half of these children demonstrated hyperopia (+2.00 to +6.50 D). As estimated for the general population, the PPV remains slightly above half, suggesting a small majority of children referred will have the defined refractive error. The estimated NPV remains excellent, suggesting only a small percentage of children not referred will have significant refractive error.
The ROC curves for the appropriate refractive Spot Vision Screener parameters for each diagnosis (hyperopia, myopia, astigmatism, and anisometropia) as determined by the three different metrics (gold standards 1, 2, and 3) are shown in Figure 1. Figure 1 also shows that the ROC curve has at least one positive diagnosis for a multiple logistic regression model, including hyperopia, myopia, astigmatism, and anisometropia as predictors. The area under the curve is greatest when the outcome is set by gold standard 1 and AAPOS amblyopia risk factor criterion, although the areas under the curve are similar across all three standards with the only exception being myopia, for which the area under the curve is notably greater using the amblyopia risk factor gold standard 1.
Receiver operating characteristic curves for detection of hyperopia, myopia, astigmatism, and anisometropia based on spherical equivalents or cylinder returned from the Spot Vision Screener (Welch Allyn, Skaneateles Falls, NY) relative to three gold standards. Gold standard 1 is the determination of these conditions according the 2013 amblyopia risk factors criterion. Gold standards 2 and 3 represent progressively more strict rules that determine the conditions (ie, more children will be categorized as having the condition). AUC = area under the curve
Table 4 shows the performance characteristics for the best individual cutoffs for the appropriate Spot Vision Screener parameter for each outcome as determined with the three gold standards. The recommended cutoffs for Spot Vision Screener detection of hyperopia for gold standards 1, 2, and 3 was a sphere of 0.50 D or greater for all three gold standards. The recommended cutoffs for Spot Vision Screener detection of myopia for gold standards 1, 2, and 3 were a sphere of −1.50 D or less, −1.00 D or less, and −0.75 D or less, respectively. The recommended cutoffs for the Spot Vision Screener detection of astigmatism for gold standards 1, 2, and 3 were a cylinder of 1.75 D or greater, 1.75 D or greater, and 1.25 D or greater, respectively. The recommended cutoffs for Spot Vision Screener detection of anisometropia for gold standards 1, 2, and 3 were a spherical equivalent of 1.375 D or greater, 0.75 D or greater, and 0.75 D or greater, respectively.
Performance Characteristics and 95% Confidence Intervals for the Spot Vision Screener
We also identified the optimal joint combinations of cutoffs for the four parameters for discriminating whether a child needed further evaluation (ie, had at least one of the four conditions) to determine if adjusting the Spot Vision Screener criterion improved performance of the screening test. Here, gold standards 1, 2, and 3 were defined as being positive if a child had any one of the four conditions. For gold standard 1, the selected cutoffs were: a minimum sphere of −1.75 D or less, a maximum sphere of 0.75 D or greater, spherical equivalent of 0.75 D or greater, and a cylinder of 1.75 D or greater. For gold standard 2, the selected cutoffs were: a minimum sphere of −1.15 D or less, a maximum sphere of 0.75 D or greater, spherical equivalent of 0.75 D or greater, and a cylinder of 1.25 D or greater. For gold standard 3, the selected cutoffs were: a minimum sphere of −1.15 D or less, a maximum sphere of 1.00 D or greater, spherical equivalent of 0.625 D or greater, and a cylinder of 1.25 D or greater. In all cases, a child was classified as needing further evaluation regardless of whether one or more of the four criteria were met or whether the Spot Vision Screener was unable to obtain a reading (in keeping with current Spot Vision Screener manufacturer's recommendations).
Table 5 shows the sensitivity and specificity for each gold standard based on the current Spot Vision Screener manufacturer's “out-of-the-box” recommendations and the optimized cutoffs discussed previously. Gold standard refractive criteria are described in Table 1; however, media opacity and strabismus are not included as referral criteria in these calculations. There are small variations in these predictability parameters with adjustment of the device's criteria to the jointly optimized cutoffs.
Estimated Sensitivity and Specificity for the Refractive Error Values of Each Gold Standard
The Spot Vision Screener demonstrated good sensitivity (89.5%) for detecting AAPOS amblyopia risk factors in older patients, which was slightly higher than some reports in younger children.4–7 High sensitivity is especially desirable when screening older children.14 In school-aged children, the primary focus shifts from preventing amblyopia to detecting visual impairment, including refractive errors that may interfere with academic performance. The Spot Vision Screener demonstrated lower sensitivities (80 and 75.3%) with narrower criteria detecting smaller degrees of refractive error. As has been suggested in previous studies, hyperopia is the refractive parameter most commonly “missed” by the Spot Vision Screener.12,20
We provide recommendations regarding the best cutoff threshold settings to maximize sensitivity and specificity for each of the three considered refractive gold standards. The proposed cutoffs have higher sensitivity for gold standards 2 and 3 but slightly lower specificity, whereas the current recommendation for gold standard 1 is similar to our optimized cutoffs. Although the confidence intervals overlap, thus preventing conclusive recommendations, users may consider modifying the current refractive cut-offs on the Spot Vision Screener in situations where the detection of smaller refractive errors is desirable.
A limitation of our study is the small and enriched population. Although the sensitivity and specificity are thought to be test-specific across populations, PPV and NPV vary with disease prevalence. We estimate that the PPV for the general population is between 52.3% and 61.8%, depending on the stringency of the criteria, with excellent NPVs. However, the estimates and results from our ophthalmology office population may not correlate with the results of a community or school screening. In addition, it may be helpful to further delineate the results by age, because visual needs change and growth is rapid during “school-age” years.
Currently, instrument-based screening is recommended when children are too young to cooperate with identifying letters (eg, preschool age), and opto-type visual acuity screening is recommended for older children.1,2 However, the legislative requirements and methods used for school vision screenings21 are inconsistent, and the efficacy of visual acuity screenings may vary with different testing situations. The current study found that the sensitivities for the Spot Vision Screener were not unlike the sensitivities reported for visual acuity testing in detecting significant refractive error in older children. In one report of larger scale vision screening, a maximum sensitivity of 75% and specificity of 85% were obtained from combining autorefraction and uncorrected visual acuity in the detection of myopia.22–24
Automated screeners allow for rapid screening, approximately one child per minute, by minimally trained lay persons. Because of increased availability and ease of use, instrument-based screeners have been employed to screen school-aged children in areas with limited health care access20,25,26 and in school systems.25–27
Uncorrected refractive error is the primary cause of moderate to severe vision impairment in young people.28 In some situations, such as cultural barriers or time limitation, automated screening may be more effective than acuity testing in the school-aged population. Expanding eye care to all students, communities, and institutions may benefit from a flexible approach, because automated screening may maximize vision screening for their situation. We found the Spot Vision Screener to be an acceptable method of detecting significant refractive error in the school-aged population.