Introduction
Retinopathy of prematurity (ROP) is a leading cause of childhood blindness worldwide.1 The Cryotherapy for ROP (CRYO-ROP) trial found that cryotherapy is effective for treatment of threshold disease (defined as 5 contiguous or 8 total clock hours of stage 3 ROP in zone I or II with plus disease).2 Subsequently, the Early Treatment for ROP (ETROP) trial determined that treatment of type 1 prethreshold ROP (defined as zone I, any stage ROP with plus disease; zone I, stage 3 ROP with or without plus disease; or zone II, stage 2 or 3 ROP with plus disease) results in improved structural and functional outcomes.3–5
Plus disease diagnosis is critical for decision-making in ROP, as it is necessary for threshold disease and sufficient for type 1 ROP. Plus disease is defined as abnormal arteriolar tortuosity and venular dilation in the posterior pole greater than that of a standard published photograph, which was selected by expert consensus for the CRYO-ROP study and is still widely used.2,6 Since that time, several multi-center trials have explicitly stated that a diagnosis of plus disease requires at least two quadrants with the requisite amount of abnormality.5,7,8 In addition, the revised International Classification of ROP defined an intermediate “pre-plus” disease as “vascular abnormalities of posterior pole that are insufficient for the diagnosis of plus disease but that demonstrate more arteriolar tortuosity and more venular dilation than normal.”7
During the past 10 years, several computer-based systems have been developed for ROP, particularly for detection of plus disease. This has potential to improve clinical ROP management. The purpose of this article is to discuss the recent advances and limitations in current methods for plus disease diagnosis and to review the rationale and previous research involving computer-based quantitative image analysis for plus disease. We will discuss six key challenges faced by investigators in these areas: definition of a reference standard, algorithms for quantifying vascular features, selection of retinal vessels for analysis, combination of parameters into overall diagnosis, development of cutoff points for “abnormality,” and rigorous design of evaluation studies.
Inconsistency in Plus Disease Diagnosis Among Experts
Studies have shown that there may be significant inconsistency in plus disease diagnosis, even among experts. The CRYO-ROP protocol required confirmation of threshold disease before randomization for possible treatment. Using standard dilated ophthalmoscopy, a second unmasked certified examiner disagreed with the first examiner regarding diagnosis of threshold disease in 12% of cases.9
Using image-based diagnosis for identification of plus disease, one study found that 22 experts agreed on the diagnosis (plus vs not plus) in only 21% (7 of 34) of images, with a mean kappa for each expert compared with all others of 0.19 to 0.66.10 Compared with a reference standard defined as the diagnosis selected by a majority of experts, the sensitivity of those 22 experts for plus disease diagnosis ranged from 0.31 to 1.00 and the specificity ranged from 0.57 to 1.00.11 In another study, three experts were asked to grade vascular tortuosity and dilation with three-level categorization (plus, pre-plus, neither plus nor pre-plus) in each quadrant of 181 images.12 These three graders disagreed on the presence of plus disease in 27% (18 of 67) of images after excluding normal images. Limitations in diagnostic accuracy and reliability may be caused by subjective and qualitative interpretations regarding the level of dilation and tortuosity sufficient for plus disease.
Quantitative Image Analysis for Plus Disease
Quantitative image analysis of retinal vascular dilation and tortuosity using computer-based systems has potential to result in a more accurate and reproducible diagnosis of plus disease. Similar techniques have been used for many years for other ophthalmic diseases. Individual measurement techniques devised by Parr and Spears were used to quantify arteriolar narrowing secondary to hypertension.13,14 Hubbard et al. developed formulas to analyze venular changes and to calculate an arteriole-to-venule ratio.15 Larger arteriolar and venular diameters have been associated with progression of diabetic retinopathy and hypertension,16 and coronary heart disease in women has been associated with a lower arteriole-to-venule ratio.17,18
In infants, there are three portable devices currently used to capture fundus images. The RetCam (Clarity Medical Systems, Pleasanton, CA) is a wide-angle digital camera that requires contact with the cornea and provides 640 × 480 pixel resolution. It has been widely used throughout the world for studies involving telemedicine for ROP diagnosis10,12,19–31 and for image analysis in plus disease.30,32–42 The NM-200D (NIDEK Inc., Fremont, CA) is a narrow-angle non-contact 30° digital camera.43 It has been studied for ROP screening and for image analysis in plus disease.39–41,44,45 Finally, the Kowa handheld fundus camera (Kowa Optimed Inc., Torrance, CA) has also been used to study image analysis in plus disease.34
Challenge 1: Definition of a Reference Standard
Rigorous evaluation of computer-based systems requires a reference standard that defines the correct diagnosis for a given image. By definition, a “gold standard” should have complete accuracy and consensus.46 This creates challenges because studies have demonstrated imperfect agreement among experts.11,12,47 Studies evaluating the performance of image analysis systems for plus disease diagnosis have used approaches that differ based on several features.
Number of Experts Determining Standard
Some studies have used the opinion of a single expert as a reference standard for comparison.35,37,40 Others have combined the opinions of multiple (n = 2 to 22) experts using methods such as majority vote or discussions to obtain consensus.10,11,30,32,38,42,47,48 To examine the difference between approaches, Koreen et al. compared diagnosis obtained by using majority vote of 22 experts to the diagnosis obtained by consensus of 3 expert authors after group discussion and found that the same diagnosis was obtained in 90% of cases.38
Method of Examination Used to Determine Standard
This may involve standard clinical examination using indirect ophthalmoscopy or expert review of retinal images. Most groups have obtained reference standards using expert review of digital images with the rationale that these same images are analyzed by a computer-based system, that potential biases might occur if additional retinal features were visualized during indirect ophthalmoscopy diagnoses, and that telemedicine interpretation of retinal images has been shown to be accurate compared with indirect ophthalmoscopy.10,11,19,21,22,24,25,30,32,38,42,44,45,47,49–51 Other groups have determined presence of plus disease based on ophthalmoscopic examination.35,39
Image Field of View
In studies where the reference standard is determined by expert review of retinal images, different fields of view have been used. Because the standard photographic definition of plus disease includes only the central retina,2 some investigators have cropped images to include only this area.30,32,52–54 Other investigators have cropped images to exclude peripheral ROP while including mid-peripheral vessels, with the rationale that experts may have difficulty diagnosing plus disease based only on the appearance of central vessels.42 The NM200D camera provides narrow field images so that only 30°, typically centered on the optic nerve, are analyzed.39,43,45
Information Provided to Experts During Image Review
In some studies, experts have been asked to identify the presence of plus disease by comparing retinal images side-by-side to an image of the standard published photograph.2,12,25,30,32 In other studies, experts have been asked to identify presence of plus disease based on an individual’s judgment without any reference materials, with the rationale that this might better represent real-world diagnosis.10,11,24,25,38,47,48,55
Eye-Level Versus Quadrant-Level Evaluation
Some studies have asked investigators to provide an overall diagnosis for each image (eg, plus vs not plus), with the rationale that presence or absence of plus disease is inherently a characteristic of an entire eye.10,11,38,47 Other studies have asked investigators to grade each retinal quadrant for presence or absence of vascular dilation and tortuosity (eg, “dilation sufficient for plus disease in the superotemporal quadrant”) and have considered an eye to have “plus disease” if two or more quadrants were sufficiently abnormal.11,12,30,32,48 The rationale for the latter approach is that it may provide more detailed information regarding expert opinions about vascular abnormality.
Grading Scale
The International Classification of ROP uses a two-level (plus vs not plus) or three-level (plus vs pre-plus vs neither) grading scale. In most published studies, experts have been asked to use these established two- or three-level scales while grading images with the rationale that these scales are clinically relevant.10,12,30,38,47,48 Other studies have used a 10-level scale for grading vascular dilation33 or an 11-point scale for grading vascular abnormality.42,45
Challenge 2: Algorithms for Quantifying Vascular Features
Numerous computer-based image analysis programs have been developed for quantification of vascular features related to plus disease. These systems have used different algorithms for measuring vascular dilation and tortuosity. The latest version of each program is described in the table and detailed technical specifications can be found in the referenced articles.
In 1995, Capowski et al. developed a method to analyze retinal blood vessel tortuosity by reviewing serial images from the same eyes captured over 1 to 2 months.34 Based on the observation that vessels that become tortuous from plus disease did so with a characteristic spatial frequency, they developed a numeric index based on spatial frequency.
Heneghan et al. developed a program called Vessel Finder, which calculated both vascular dilation and tortuosity by first identifying a binary mask representing segmented retinal vessels.35 Tortuosity was defined as the ratio of distance along a vessel to the straight-line distance connecting the end points. The smallest width at every point in the vessel was identified and average vessel width was calculated along the entire vessel segment.
Wallace et al. developed and refined a program called ROPtool.30,33,54 In a 30° area centered on the optic nerve, vascular tortuosity is calculated by measuring a smooth curve along points on an arteriole or venule, then calculated as the ratio of total length of the vessel to the length of the smooth curve. Vascular diameter is calculated using profiles of vessel cross sections. This was recently modified, allowing analysis of both dilation and tortuosity with ROPtool. To standardize for differences in image magnification while computing vascular diameter, image sizes are normalized using the distance from the optic disc to the macular center.33
Gelman et al. developed a program called Retinal Image multiScale Analysis (RISA), which calculates both arteriolar and venular tortuosity and dilation (Fig. 1).37 Tortuosity is represented by integrated curvature (radians/pixel), which is the sum of angles along the blood vessel normalized by vessel length, and by tortuosity index (unitless), which is the vascular arc length divided by the linear distance between the start and end points of the vessel (Fig. 2). Mean vascular diameter (pixels) is calculated as the total area of the vessel divided by its length.
Rabinowitz et al. evaluated a semi-automated program called VesselMap, which measures diameters of arterioles and venules (described in arbitrary units) by examining brightness indices perpendicular to vessel lengths within 2 disc diameters of the optic nerve head.39
Wilson et al. developed the Computer-Assisted Image Analysis of the Retina (CAIAR) program, which measures vessel width and tortuosity.42 They tested 14 methods of calculating tortuosity and two methods of estimating vessel width in 60° images. Assessments of semi-automated tortuosity measurement with this program appeared to correlate relatively well with expert opinions; however, vessel width measurements were not as successful. As such, Wilson collaborated with Shah et al.45 on a more recent project in which CAIAR and IVAN (Vasculo-matic ala Nicola version 1.1, which evaluates vessel width alone) were used together to analyze images. In this study, arterioles and venules were assessed separately using NIDEK images from the NM200D camera (30°). CAIAR measures vessels from the edge of the optic nerve, whereas IVAN uses vessels at least 1 disc diameter away from the optic nerve head. Arteriolar tortuosity measured by CAIAR and venule width measured by CAIAR and IVAN were best at identifying plus disease based on receiver operating characteristics analysis (area under the receiver operating characteristic curve [AUROC] 0.91 using CAIAR venule width, AUROC 0.92 using CAIAR arteriole tortuosity, and AUROC 0.91 using IVAN venule width).
Challenge 3: Selection of Vessels for Analysis
The International Classification of ROP defines plus disease as arteriolar tortuosity and venular dilatation greater than or equal to that of a standard published photograph.2,7 This suggests that arterioles and venules should be analyzed separately, although there are no guidelines regarding which specific retinal vessels are most important to consider. Published studies involving computer-based plus disease diagnosis have taken different approaches regarding the selection of vessels to analyze, or whether arterioles and venules are differentiated in analysis.
In several studies, all vessels were analyzed and modeled together without regard to whether they were arterioles or venules.30,32,35,42,53,54 In a recent publication, Johnston et al. addressed two questions: “Can examiners accurately discriminate between arterioles and venules?” and “Is identification of ‘tortuosity sufficient for plus disease’ (or for pre-plus disease) most accurate when evaluating arterioles, venules, or both?”48 Eighty-three percent of the time, arterioles and venules could be discriminated accurately by experts in that study. “Tortuosity sufficient for plus disease” was most accurate when evaluating arterioles (AUROC 0.97), although images were excluded from that study if an arteriole or venule could not be analyzed in any of the four quadrants.
In other studies, arterioles and venules have been analyzed and modeled separately using computer-based systems.11,36–38,40,45,47 Published work using the VesselMap program has only analyzed temporal vessels, using one arteriole and one venule from the superotemporal and inferotemporal quadrants of each eye.39–41 Published work using the RISA program has analyzed arterioles and venules that could be identified and processed in the eye to minimize potential subjectivity in vessel selection.11,36–38 Several groups have demonstrated that there did not appear to be significant differences when using the most “abnormal” vessels in an eye as compared with the mean values from all vessels in the same eye.11,30,47,53,54
Challenge 4: Combination of Parameters into Overall Diagnosis
The image analysis algorithms described above produce quantitative measurements for individual retinal vessels. Investigators have used various methods to combine these measurements of individual vascular parameters into an overall diagnosis for an eye.
Strategies for combining vascular dilation and tortuosity are based on the assumption that these are the key parameters to be used. The International Classification of ROP states that plus disease should be defined by arteriolar tortuosity and venular dilation within the central retina.2,7 However, it is possible that other characteristics, such as vascular branching, vascular congestion, or the appearance of peripheral disease, may be considered by experts while assessing plus disease. Techniques from cognitive science, such as think-aloud methodologies, may provide insight about retinal features that are truly perceived as important for real-world disease management.56 In the future, these findings could be used to develop image analysis methods that better represent the performance of experts. More recent studies by Chiang et al.,47 Johnston et al.,48 and Shah et al.45 have begun to combine parameters to find the “best” combination for reliable diagnosis of plus disease. Although each group varies in its approach, all of the groups are trending toward using similar combinations to define plus disease.
Linear Combinations of Individual Parameters
Using the RISA system, linear combinations of individual parameters have been used to find groups of features with the best sensitivity, specificity, and AUROC. For example, integrated curvature, tortuosity index, and vessel diameter, calculated separately for both arterioles and venules, have been combined linearly. Given the definition of plus disease, one may expect that arteriolar integrated curvature, arteriolar tortuosity index, and venular diameter would have the best combined sensitivity and specificity as compared with expert opinion, although this does not always appear true. In a recent study by Chiang et al. using RISA, arteriolar and venular integrated curvature and venular diameter proved the best combination of parameters, with the highest AUROC (0.96) and a sensitivity and specificity of 0.94 for identification of plus disease.47
Combinations With ROPtool
ROPtool analyzes venules and arterioles separately. A recent study by Johnston et al.48 showed that the use of arterioles alone, or using arterioles and venules together, provides good diagnostic accuracy for ROPtool’s determination of “tortuosity sufficient for plus or pre-plus disease.” On the other hand, venule tortuosity alone did not have good diagnostic accuracy. AUROCs for identification of “tortuosity sufficient for plus disease” were 0.91, 0.70, and 0.93 for arterioles, venules, and both, respectively.48 A recent study (Cabrera M. T. F. S., et al., unpublished data, 2011) assessed five methods of combining ROPtool measures of tortuosity and dilation. The authors found that a “tortuosity weighted plus” algorithm, which increased the contribution of vascular width as tortuosity increases, had the highest diagnostic accuracy (AUROC 0.94). Thus, ROPtool is able to generate a clinically meaningful quantitative description of the overall amount of vascular abnormality.
Combination of CAIAR and IVAN
CAIAR measures width and tortuosity and IVAN measures width of retinal arterioles and venules. Shah et al.45 combined the measurements of CAIAR to the vessel width detected with IVAN. In CAIAR, mean tortuosity of venules and arterioles was found to increase significantly with the severity of ROP, whereas arteriolar tortuosity had the highest correlation with disease status. Arteriole tortuosity measured by CAIAR and venule width measured by CAIAR and IVAN were the best parameters for identifying plus disease based on receiver operating characteristic analysis. Venule width and arteriolar tortuosity measured by CAIAR had an AUROC curve of 0.91 and 0.92, respectively, whereas venule width detected by IVAN was 0.91.45
Challenge 5: Development of Cutoff Points for “Abnormality”
There are a variety of ways that a computer-based system may determine a precise cutoff to quantitatively separate “plus” from “not plus” disease. The use of objective methods has potential to remove the “grey areas” of plus disease diagnosis.9,10,12,24,25,30,47 Possible approaches include comparison with the standard photographic definition of plus disease, using the opinions of expert graders, and examining long-term clinical outcomes.
Many groups have used expert opinions to identify a cutoff point of abnormality for plus disease with the rationale that the gold standard for ROP diagnosis is considered to be examination by an experienced ophthalmologist.11,30,32,35,37,38,42,57 In particular, some studies have used majority vote consensus of a larger number of experts who review the same retinal images,10,11,38 whereas other studies have used a single expert’s opinion or the consensus of two to five different experts.32,35,42,48,53,57 A limitation of this approach using expert opinions is that the agreement of plus disease diagnosis among experts has been shown to be imperfect.10–12,25,47,55 Similarly, if only a small number of experts is used to generate a cutoff, this may bias toward the perspectives of those specific experts.
Another approach is that retinal arterioles and venules from the standard published photographic definition of plus disease may be digitized and analyzed, and the dilation and tortuosity of these vessels may be used as a cutoff for abnormality. In one study, the computer system judged that an image had “plus disease” if two or more quadrants had dilation and tortuosity at least equal to the average dilation and tortuosity in the standard photograph.32 Limitations of this approach are that the magnification of the standard photograph is larger than that of indirect ophthalmoscopy, whereas the field-of-view in the standard photograph is much smaller. In particular, Gelman et al.58 recently used computer-based image analysis to determine that the arteriolar tortuosity and venular dilation of the standard photograph were lower than what was used by experts for diagnosing “plus disease” in an independent set of wide-angle retinal images. This raises questions about whether the standard photograph is truly representative of expert opinions, and whether the non-standard magnification and field-of-view should be addressed.58
Finally, plus disease is defined using vascular appearance at a single point in time, yet vascular abnormalities actually evolve over time. Studies have shown that more rapid change in vessel appearance is associated with worse disease outcome and progression toward ROP requiring treatment.35,36,41,52,53,59 It is possible that future definitions of plus disease based on the quantitative change in vascular appearance over time, rather than the appearance at only a single point in time, might create more opportunities for ROP management using dynamic principles.41,60 A recent study used ROPtool to calculate blood vessel tortuosity and dilation in several series of images from the same eyes, allowing quantification of change in these two parameters over time.61
Challenge 6: Rigorous Design of Evaluation Studies
Ultimately, the performance of computer-based image analysis systems must be evaluated using rigorous studies. This may be challenging for many reasons. Use of a three-level (plus vs pre-plus vs neither) compared with a two-level (plus vs not plus) grading scale might create difficulty in defining gold standards if experts diagnose inconsistently with different scales.10,11,30,32,37,38,47,54,57 Expectation bias may occur if authors include themselves as experts while defining reference standards for plus disease.11,12,32,33,37,47–49,52,54 Differing magnifications and/or fields-of-view might also bias experts, particularly if they are unfamiliar with correlating the appearance of retinal images to what is seen during standard indirect ophthalmoscopy.11,30,38,47,48,56,60 In some studies, experts were given the standard photograph as a guide while reviewing images to create reference standards, which might result in bias by creating an artificial environment for plus disease diagnosis.32,54 In other studies, experts were not given any references, which might increase diagnostic variability to the extent that some experts may be less familiar with image-based diagnosis than others.11,38,47,56 The rigor of future evaluation studies may be improved by attention to these details.
Conclusion
Significant strides in image analysis for ROP diagnosis have been made during the past 15 years. There are now a variety of tools available to analyze tortuosity and dilation of retinal arterioles and venules. Direct comparison of different computer-based methods, using the same image sets, would be a step toward identifying and perhaps combining the best features of each tool. Persistent challenges include the absence of a solid clinical reference standard and a lack of consensus regarding the specific clinical features and pathophysiology related to onset of plus disease. Further research in these areas has potential to create more objective, quantitative methods for plus disease diagnosis that could improve the standardization and quality of clinical care.
References
- Gilbert, C. Retinopathy of prematurity: a global perspective of the epidemics, population of babies at risk and implications for control. Early Hum Dev. 2008;84:77–82. doi:10.1016/j.earlhumdev.2007.11.009 [CrossRef]
- Cryotherapy for Retinopathy of Prematurity Cooperative Group. Multicenter trial of cryotherapy for retinopathy of prematurity: preliminary results. Arch Ophthalmol. 1988;106:471–479.
- Good, WV. Final results of the Early Treatment for Retinopathy of Prematurity (ETROP) randomized trial. Trans Am Ophthalmol Soc. 2004;102:233–248.
- Good, WVEarly Treatment for Retinopathy of Prematurity Cooperative Group. The Early Treatment for Retinopathy of Prematurity study: structural findings at age 2 years. Br J Ophthalmol. 2006;90:1378–1382. doi:10.1136/bjo.2006.098582 [CrossRef]
- Early Treatment For Retinopathy of Prematurity Cooperative Group. Revised indications for the treatment of retinopathy of prematurity: results of the early treatment for retinopathy of prematurity randomized trial. Arch Ophthalmol. 2003;121:1684–1694.
- Capone, A Jr. , Ells, AL & Fielder, AR et al. Standard image of plus disease in retinopathy of prematurity. Arch Ophthalmol. 2006;124:1669–1670. doi:10.1001/archopht.124.11.1669-c [CrossRef]
- International Committee for the Classification of Retinopathy of Prematurity. The International Classification of Retinopathy of Prematurity revisited. Arch Ophthalmol. 2005;123:991–999.
- Supplemental Therapeutic Oxygen for Prethreshold Retinopathy Of Prematurity (STOP-ROP), a randomized, controlled trial. I: primary outcomes. Pediatrics. 2000;105:295–310.
- Reynolds, JD, Dobson, V & Quinn, GE et al. Evidence-based screening criteria for retinopathy of prematurity: natural history data from the CRYO-ROP and LIGHT-ROP studies. Arch Ophthalmol. 2002;120:1470–1476.
- Chiang, MF, Jiang, L, Gelman, R, Du, YE & Flynn, JT. Interexpert agreement of plus disease diagnosis in retinopathy of prematurity. Arch Ophthalmol. 2007;125:875–880. doi:10.1001/archopht.125.7.875 [CrossRef]
- Gelman, R, Jiang, L, Du, YE, Martinez-Perez, ME, Flynn, JT & Chiang, MF. Plus disease in retinopathy of prematurity: pilot study of computer-based and expert diagnosis. J AAPOS. 2007;11:532–540. doi:10.1016/j.jaapos.2007.09.005 [CrossRef]
- Wallace, DK, Quinn, GE, Freedman, SF & Chiang, MF. Agreement among pediatric ophthalmologists in diagnosing plus and pre-plus disease in retinopathy of prematurity. J AAPOS. 2008;12:352–356. doi:10.1016/j.jaapos.2007.11.022 [CrossRef]
- Parr, JC & Spears, GF. General caliber of the retinal arteries expressed as the equivalent width of the central retinal artery. Am J Ophthalmol. 1974;77:472–477.
- Parr, JC & Spears, GF. Mathematic relationships between width of a retinal artery and widths of its branches. Am J Ophthalmology. 1974;77:478–483.
- Hubbard, LD, Brothers, RJ & King, WN et al. Methods for evaluation of retinal microvascular abnormalities associated with hypertension/sclerosis in the Atherosclerosis Risk in Communities study. Ophthalmology. 1999;106:2269–2280. doi:10.1016/S0161-6420(99)90525-0 [CrossRef]
- Klein, R, Klein, BE & Moss, SE et al. The relation of retinal vessel caliber to the incidence and progression of diabetic retinopathy. XIX: the Wisconsin epidemiologic study of diabetic retinopathy. Arch Ophthalmol. 2004;122:76–83. doi:10.1001/archopht.122.1.76 [CrossRef]
- Wong, TY, Klein, R & Sharrett, AR et al. Retinal arteriolar diameter and risk for hypertension. Ann Intern Med. 2004;140:248–255.
- Wong, TY, Klein, R & Sharrett, AR et al. Retinal arteriolar narrowing and risk of coronary heart disease in men and women: the Atherosclerosis Risk in Communities study. JAMA. 2002;287:1153–1159. doi:10.1001/jama.287.9.1153 [CrossRef]
- Schwartz, SD, Harrison, SA, Ferrone, PJ & Trese, MT. Telemedical evaluation and management of retinopathy of prematurity using a fiberoptic digital fundus camera. Ophthalmology. 2000;107:25–28. doi:10.1016/S0161-6420(99)00003-2 [CrossRef]
- Yen, KG, Hess, D, Burke, B, Johnson, RA, Feuer, WJ & Flynn, JT. Telephotoscreening to detect retinopathy of prematurity: preliminary study of the optimum time to employ digital fundus camera Imaging to detect ROP. J AAPOS. 2002;6:64–70. doi:10.1067/mpa.2002.121616 [CrossRef]
- Ells, AL, Holmes, JM & Astle, WF et al. Telemedicine approach to screening for severe retinopathy of prematurity: a pilot study. Ophthalmology. 2003;110:2113–2117. doi:10.1016/S0161-6420(03)00831-5 [CrossRef]
- Roth, DB, Morales, D, Feuer, WJ, Hess, D, Johnson, RA & Flynn, JT. Screening for retinopathy of prematurity employing the RetCam 120: sensitivity and specificity. Arch Ophthalmol. 2001;119:268–272.
- Wu, C, Petersen, RA & VanderVeen, DK. RetCam imaging for retinopathy of prematurity screening. J AAPOS. 2006;10:107–111. doi:10.1016/j.jaapos.2005.11.019 [CrossRef]
- Chiang, MF, Starren, J & Du, YE et al. Remote image based retinopathy of prematurity diagnosis: a receiver operating characteristic analysis of accuracy. Br J Ophthalmol. 2006;90:1292–1296. doi:10.1136/bjo.2006.091900 [CrossRef]
- Chiang, MF, Keenan, JD & Starren, J et al. Accuracy and reliability of remote retinopathy of prematurity diagnosis. Arch Ophthalmol. 2006;124:322–327. doi:10.1001/archopht.124.3.322 [CrossRef]
- Fielder, AR, Cocker, KD, Capone, A & Trese, MT Jr. . Screening for retinopathy of prematurity using wide-field digital retinal imaging: sensitivity and specificity. Arch Ophthalmol. 2002;120:1234.
- Photographic Screening for Retinopathy of Prematurity (Photo-ROP) Cooperative Group. The photographic screening for retinopathy of prematurity study (photo-ROP): primary outcomes. Retina. 2008;28:S47–S54.
- Murakami, Y, Jain, A, Silva, RA, Lad, EM, Gandhi, J & Moshfeghi, DM. Stanford University Network for Diagnosis of Retinopathy of Prematurity (SUNDROP): 12-month experience with telemedicine screening. Br J Ophthalmol. 2008;92:1456–1460. doi:10.1136/bjo.2008.138867 [CrossRef]
- Silva, RA, Murakami, Y, Jain, A, Gandhi, J, Lad, EM & Moshfeghi, DM. Stanford University Network for Diagnosis of Retinopathy of Prematurity (SUNDROP): 18-month experience with telemedicine screening. Graefes Arch Clin Exp Ophthalmol. 2009;247:129–136. doi:10.1007/s00417-008-0943-z [CrossRef]
- Wallace, DK, Freedman, SF, Zhao, Z & Jung, SH. Accuracy of ROPtool vs individual examiners in assessing retinal vascular tortuosity. Arch Ophthalmol. 2007;125:1523–1530. doi:10.1001/archopht.125.11.1523 [CrossRef]
- Yen, KG, Feuer, WJ, Hess, D, Johnson, R, Burke, B & Flynn, JT. The optimum time to employ telephotoscreening to detect retinopathy of prematurity (ROP). Invest Ophthalmol Vis Sci. 2000;41:S930–S93S.
- Wallace, DK, Jomier, J, Aylward, SR & Landers, MB 3rd. . Computer-automated quantification of plus disease in retinopathy of prematurity. J AAPOS. 2003;7:126–130. doi:10.1016/S1091-8531(02)00015-0 [CrossRef]
- Wallace, DK, Freedman, SF & Zhao, Z. A pilot study using ROPtool to measure retinal vascular dilation. Retina. 2009;29:1182–1187. doi:10.1097/IAE.0b013e3181a46a73 [CrossRef]
- Capowski, JJ, Kylstra, JA & Freedman, SF. A numeric index based on spatial frequency for the tortuosity of retinal vessels and its application to plus disease in retinopathy of prematurity. Retina. 1995;15:490–500.
- Heneghan, C, Flynn, J, O’Keefe, M & Cahill, M. Characterization of changes in blood vessel width and tortuosity in retinopathy of prematurity using image analysis. Med Image Anal. 2002;6:407–429. doi:10.1016/S1361-8415(02)00058-0 [CrossRef]
- Swanson, C, Cocker, KD, Parker, KH, Moseley, MJ & Fielder, AR. Semiautomated computer analysis of vessel growth in preterm infants without and with ROP. Br J Ophthalmol. 2003;87:1474–1477. doi:10.1136/bjo.87.12.1474 [CrossRef]
- Gelman, R, Martinez-Perez, ME, Vanderveen, DK, Moskowitz, A & Fulton, AB. Diagnosis of plus disease in retinopathy of prematurity using Retinal Image multiScale Analysis. Invest Ophthalmol Vis Sci. 2005;46:4734–4738. doi:10.1167/iovs.05-0646 [CrossRef]
- Koreen, S, Gelman, R & Martinez-Perez, ME et al. Evaluation of a computer-based system for plus disease diagnosis in retinopathy of prematurity. Ophthalmology. 2007;114:e59–e67. doi:10.1016/j.ophtha.2007.10.006 [CrossRef]
- Rabinowitz, MP, Grunwald, JE, Karp, KA, Quinn, GE, Ying, GS & Mills, MD. Progression to severe retinopathy predicted by retinal vessel diameter between 31 and 34 weeks of postconception age. Arch Ophthalmol. 2007;125:1495–1500. doi:10.1001/archopht.125.11.1495 [CrossRef]
- Johnson, KS, Mills, MD, Karp, KA & Grunwald, JE. Semiautomated analysis of retinal vessel diameter in retinopathy of prematurity patients with and without plus disease. Am J Ophthalmol. 2007;143:723–725. doi:10.1016/j.ajo.2006.11.024 [CrossRef]
- Grunwald, L, Mills, MD & Johnson, KS et al. The rate of retinal vessel dilation in severe retinopathy of prematurity requiring treatment. Am J Ophthalmol. 2009;147:1086–1091. doi:10.1016/j.ajo.2008.12.038 [CrossRef]
- Wilson, CM, Cocker, KD & Moseley, MJ et al. Computerized analysis of retinal vessel width and tortuosity in premature infants. Invest Ophthalmol Vis Sci. 2008;49:3577–3585. doi:10.1167/iovs.07-1353 [CrossRef]
- Shah, DN, Karp, KA, Ying, GS, Mills, MD & Quinn, GE. Image analysis of posterior pole vessels identifies type 1 retinopathy of prematurity. J AAPOS. 2009;13:507–508. doi:10.1016/j.jaapos.2009.07.004 [CrossRef]
- Skalet, AH, Quinn, GE & Ying, GS et al. Telemedicine screening for retinopathy of prematurity in developing countries using digital retinal images: a feasibility project. J AAPOS. 2008;12:252–258. doi:10.1016/j.jaapos.2007.11.009 [CrossRef]
- Shah, DN, Wilson, CM & Ying, GS et al. Semiautomated digital image analysis of posterior pole vessels in retinopathy of prematurity. J AAPOS. 2009;13:504–506. doi:10.1016/j.jaapos.2009.06.007 [CrossRef]
- Friedman, CP & Wyatt, JC. Evaluation Methods in Medical Informatics. New York: Springer; 1997.
- Chiang, MF, Gelman, R, Jiang, L, Martinez-Perez, ME, Du, YE & Flynn, JT. Plus disease in retinopathy of prematurity: an analysis of diagnostic performance. Trans Am Ophthalmol Soc. 2007;105:73–84.
- Johnston, SC, Wallace, DK, Freedman, SF, Yanovitch, TL & Zhao, Z. Tortuosity of arterioles and venules in quantifying plus disease. J AAPOS. 2009;13:181–185. doi:10.1016/j.jaapos.2008.10.019 [CrossRef]
- Chiang, MF, Wang, L & Busuioc, M et al. Telemedical retinopathy of prematurity diagnosis: accuracy, reliability, and image quality. Arch Ophthalmol. 2007;125:1531–1538. doi:10.1001/archopht.125.11.1531 [CrossRef]
- Dhaliwal, C, Wright, E, Graham, C, McIntosh, N & Fleck, BW. Wide-field digital retinal imaging versus binocular indirect ophthalmoscopy for retinopathy of prematurity screening: a two-observer prospective, randomised comparison. Br J Ophthalmol. 2009;93:355–359. doi:10.1136/bjo.2008.148908 [CrossRef]
- Lorenz, B, Spasovska, K, Elflein, H & Schneider, N. Wide-field digital imaging based telemedicine for screening for acute retinopathy of prematurity (ROP): six-year results of a multicentre field study. Graefes Arch Clin Exp Ophthalmol. 2009;247:1251–1262. doi:10.1007/s00417-009-1077-7 [CrossRef]
- Wallace, DK, Kylstra, JA & Chesnutt, DA. Prognostic significance of vascular dilation and tortuosity insufficient for plus disease in retinopathy of prematurity. J AAPOS. 2000;4:224–229. doi:10.1067/mpa.2000.105273 [CrossRef]
- Wallace, DK. Computer-assisted quantification of vascular tortuosity in retinopathy of prematurity (an American Ophthalmological Society thesis). Trans Am Ophthalmol Soc. 2007;105:594–615.
- Wallace, DK, Zhao, Z & Freedman, SF. A pilot study using “ROPtool” to quantify plus disease in retinopathy of prematurity. J AAPOS. 2007;11:381–387. doi:10.1016/j.jaapos.2007.04.008 [CrossRef]
- Chiang, MF, Gelman, R & Martinez-Perez, ME et al. Image analysis for retinopathy of prematurity diagnosis. J AAPOS. 2009;13:438–445. doi:10.1016/j.jaapos.2009.08.011 [CrossRef]
- Chiang, MF, Gelman, R & Williams, SL et al. Plus disease in retinopathy of prematurity: development of composite images by quantification of expert opinion. Invest Ophthalmol Vis Sci. 2008;49:4064–4070. doi:10.1167/iovs.07-1524 [CrossRef]
- Kiely, AE, Wallace, DK, Freedman, SF & Zhao, Z. Computer-assisted measurement of retinal vascular width and tortuosity in retinopathy of prematurity. Arch Ophthalmol. 2010;128:847–852. doi:10.1001/archophthalmol.2010.133 [CrossRef]
- Gelman, SK, Gelman, R & Callahan, AB et al. Plus disease in retinopathy of prematurity: quantitative analysis of standard published photograph. Arch Ophthalmol. 2010;128:1217–1220. doi:10.1001/archophthalmol.2010.186 [CrossRef]
- Schaffer, DB, Palmer, EA & Plotsky, DF et al. Prognostic factors in the natural course of retinopathy of prematurity: The Cryotherapy for Retinopathy of Prematurity Cooperative Group. Ophthalmology. 1993;100:230–237.
- Thyparampil, PJ, Park, Y & Martinez-Perez, ME et al. Plus disease in retinopathy of prematurity: quantitative analysis of vascular change. Am J Ophthalmol. 2010;150:468–457. doi:10.1016/j.ajo.2010.04.027 [CrossRef]
- Wallace, DK, Freedman, SF & Zhao, Z. Evolution of plus disease in retinopathy of prematurity: quantification by ROPtool. Trans Am Ophthalmol Soc. 2009;107:47–52.
Survey of the Four Available Image Analysis Systems Sorted According to the Most Recent Publication of Each System
| Image Analysis Program |
Reference Standard |
Vascular Features Analyzed |
Vessel Selection |
Image Field of View |
Sensitivity |
Specificity |
AUROC |
| ROPtool (Kiely et al.,58 2010) |
Consensus of 3 ROP experts (plus, pre-plus, neither) |
T, D |
A, V together |
RetCam cropped to 28-diopter lens field of view |
0.91 (T) 0.78 (D) |
0.86 (T) 0.84 (D) |
0.95 (T) 0.87 (D) |
| RISA (Chiang et al.,47 2007) |
Consensus of 22 ROP experts (plus, pre-plus, neither) |
T, D, IC |
A, V, separated |
RetCam-II |
0.70 (aT [TI]) |
0.70 (aT [TI]) |
0.71 (aT [TI]) |
| 0.64 (vT [TI]) |
0.64 (vT [TI]) |
0.74 (vT [TI]) |
| 0.76 (aT [IC]) |
0.76 (aT [IC]) |
0.82 (aT [IC]) |
| 0.76 (vT [IC]) |
0.76 (vT [IC]) |
0.85 (vT [IC]) |
| 0.59 (aD) 0.76 (vD) |
0.59 (aD) 0.76 (vD) |
0.56 (aD) 0.82 (vD) |
| Up to 0.94 (linear combination of T and D) |
Up to 0.94 (linear combination of T and D) |
Up to 0.97 (linear combination of T and D) |
| VesselMap (Rabinowitz et al.,39 2007) |
Diagnosis of attending pediatric ophthalmologist (severe ROP, less than severe ROP) |
D |
A, V, separated |
Nidek NM200D |
NR |
NR |
0.93 (aD)a; 0.87 (vD)a |
| CAIAR (Wilson et al.,42 2008) |
Consensus of 5 ROP experts (T and D graded on an 11-point Likert scale [0–10]) |
T, D |
A, V, separated |
RetCam-II cropped to 60° field of view |
Correlation for T with grades from experts were statistically significant, ranging from ρ = 0.49b to ρ = 0.67b (P < .0001).
Measure of D did not significantly correlate with the experts’ clinical assessment (ρ =−0.11,bP < .334); LoG output (based solely on image contrast) was statistically significant with the expert grades (ρ = 0.42,bP < .0001). |