Wills Eye Conference

Wills Eye Conference

March 10, 2014
1 min read

Online resource enhances grading of fundus images in DR screening

You've successfully added to your alerts. You will receive an email when new content is published.

Click Here to Manage Email Alerts

We were unable to process your request. Please try again later. If you continue to have this issue please contact customerservice@slackinc.com.

PHILADELPHIA — The use of anonymous workers online may enhance the grading of fundus photographs in screening patients for diabetic retinopathy, a speaker said here.

“With minimal training, the Amazon workforce can rapidly and correctly categorize fundus photos as normal or abnormal,” Christopher J. Brady, MD, said at the Wills Eye Annual Conference. “They did that excellently from the outside. With some modifications to our interface, they did a much better job grading the level of retinopathy. This is a new and inexpensive way to possibly screen for diabetic retinopathy.”

Christopher J. Brady

Researchers used crowdsourcing, which involves the use of anonymous workers to perform specific tasks online. They accessed Amazon Mechanical Turk, a resource for online employment. The researchers used 19 fundus images from teaching libraries, grading 12 images as abnormal and seven images as normal. They then asked online workers to grade the images as normal or abnormal. Online workers were paid $1 per image.

The average time needed by online workers to grade each image was 25 seconds.

Online workers reached a correct diagnosis of diabetic retinopathy 81.5% of the time. Grading by online workers agreed with that of expert graders almost 90% of the time, Brady said.

“Critically, there were no false-negative results, so all of the errors were overcalls rather than undercalls, which would be critical for any type of screening program. You can’t miss diagnoses,” Brady said. “Between seven and 10 graders gives you the ideal balance, sensitivity and specificity. We felt confident that 10 graders was a good number moving forward.”

Disclosure: Brady has no relevant financial disclosures.