American Society of Retina Specialists Meeting
American Society of Retina Specialists Meeting
Source/Disclosures
Source:

Lee AY, et al. Multicenter, head-to-head, real-world validation study of artificial intelligence diabetic retinopathy screening systems (AIDRSS). Presented at: American Society of Retina Specialists annual meeting; July 24-26, 2020 (virtual meeting).

Disclosures: The authors report no relevant financial disclosures.
July 25, 2020
1 min read
Save

AI diabetic retinopathy screening algorithms need external validation before deployment

Source/Disclosures
Source:

Lee AY, et al. Multicenter, head-to-head, real-world validation study of artificial intelligence diabetic retinopathy screening systems (AIDRSS). Presented at: American Society of Retina Specialists annual meeting; July 24-26, 2020 (virtual meeting).

Disclosures: The authors report no relevant financial disclosures.
You've successfully added to your alerts. You will receive an email when new content is published.

Click Here to Manage Email Alerts

We were unable to process your request. Please try again later. If you continue to have this issue please contact customerservice@slackinc.com.

Seven artificial intelligence diabetic retinopathy screening systems displayed high negative predictive values but exhibited wide variations in safety and specificity outcomes in a multicenter study.

“We believe external, independent validation with real-world imaging is critical prior to deployment, even after algorithms reach regulatory approval,” Aaron Y. Lee, MD, MSCI, said at the virtual American Society of Retina Specialists meeting.

Lee and colleagues included 311,604 retinal images from 23,724 patients who underwent teleretinal diabetic retinopathy screening at the VA Puget Sound and the Atlanta VA health care systems from 2006 to 2018. Seven automated AI diabetic retinopathy screening system algorithms, labeled A through G, were supplied by 20 invited companies. Each system graded images ranging from no diabetic retinopathy to proliferative diabetic retinopathy and ungradable image quality, Lee said.

Aaron Y. Lee, MD, MSCI

The algorithm output was set for no diabetic retinopathy vs. any diabetic retinopathy.

The teleretinal grades showed 75.7% of images displayed no diabetic retinopathy, 9.5% displayed mild non-proliferative diabetic retinopathy (NPDR), 2.3% displayed moderate NPDR and 0.6% displayed proliferative diabetic retinopathy (PDR).

“Of note, there was a 10-fold difference in the rate of PDR in Atlanta and Seattle. Atlanta also routinely dilates where Seattle does not, which causes a very large discrepancy in the rate of ungradable images between the two sites,” Lee said.

In the full data set, all seven algorithms showed a high negative predictive value and low positive predictive value. However, the VA grader showed 100% sensitivity for moderate NPDR, severe NPDR and PDR. Only algorithms E, F and G were statistically similar to the VA grader for moderate NPDR or higher, Lee said.

Algorithms E, F and G were moved on to a cost analysis. For an ophthalmologist reading the images, the algorithms would be equivalently priced around $15 per encounter, Lee said.

“Algorithms varied tremendously in their performance despite having regulatory approval and/or being clinically deployed somewhere in the world. It is really important to understand the AI models in the context of the underlying prevalence of disease to understand the negative predictive value and positive predictive value,” he said.