In the JournalsPerspective

Computer-aided design in melanoma would enhance care, not replace dermatologists

Melanoma detection and diagnoses with computer-aided systems showed similar sensitivities when compared with those from dermatologists; however, the real-world applicability of the technology is unknown, according to meta-analysis results published in JAMA Dermatology.

“There is a fear that less-skilled physicians or even nonmedical personnel will use systems to deliver a service that should be restricted to dermatologists,” Vincent Dick, CandMed, member of the Vienna Dermatologic Imaging Research (ViDIR) group in the department of dermatology at Medical University of Vienna, and colleagues wrote.

Researchers used an online database search for eligible studies on the key words “melanoma,” “diagnosis,” “detection,” “computer-aided” or “artificial intelligence.” Studies were included if they investigated computer-aided diagnosis (CAD) systems accuracy in a screening setting for cutaneous melanoma or one that could be used in that type of setting.

Of 1,694 potentially eligible articles, 132 were included in the qualitative analysis and 70 in the quantitative meta-analysis.

Computer vision was utilized in 58 articles, deep learning in 55 articles and hardware-based methods were utilized in 19, according to researchers.

Fifty studies included only melanocytic lesions while nonmelanocytic lesions were included in 67 studies, according to the study.

The median thickness of invasive melanomas ranged from 0.2 mm to 1.5 mm, the researchers reported.

From the quantitative analysis, the summary estimate for melanoma sensitivity of CAD systems was 0.74 (95% CI, 0.66-0.8) and the specificity was 0.84 (95% CI, 0.79-0.88).

The studies that used proprietary test sets showed a significantly higher sensitivity than ones that used publicly available test sets (0.87; 95% CI, 0.82-0.91 vs. 0.57; 95% CI, 0.44-0.68), the researchers wrote. However, a lower specificity was found in proprietary test sets than publicly available test sets (0.72; 95% CI, 0.63-0.79 vs. 0.91; 95% CI, 0.88-0.94).

Systems using deep learning achieved a sensitivity of 0.44 (95% CI, 0.3-0.59) and a specificity of 0.92 (95% CI, 0.89-0.95). Computer vision systems achieved a sensitivity of 0.85 (95% CI, 0.80-0.88) and a specificity of 0.77 (95% CI, 0.69-0.84).

Researchers found significantly higher specificity in deep learning (0.92; 95% CI, 0.89-0.95) than for computer vision or hardware-based methods.

Existing devices or applications are not widely used despite multiple studies that show expert-level accuracy in CAD systems for melanoma, the researchers wrote.

“A potential reason for this mismatch may be that the results of the studies conducted in this field cannot be transferred directly to clinical practice,” Dick and colleagues wrote.

Additionally, many systems were not tested in the general population or as screening tools as most clinical studies were conducted in specialized referral centers with high melanoma prevalence.

Clinical studies of automated dermoscopic images are missing in the literature, they added.

The researchers wrote that a successful CAD would likely enhance and support dermatologists and not replace them. – by Abigail Sutton

 

 

Disclosures: Dick reports no relevant financial disclosures. Please see the study for all other authors’ relevant financial disclosures.

 

 

Melanoma detection and diagnoses with computer-aided systems showed similar sensitivities when compared with those from dermatologists; however, the real-world applicability of the technology is unknown, according to meta-analysis results published in JAMA Dermatology.

“There is a fear that less-skilled physicians or even nonmedical personnel will use systems to deliver a service that should be restricted to dermatologists,” Vincent Dick, CandMed, member of the Vienna Dermatologic Imaging Research (ViDIR) group in the department of dermatology at Medical University of Vienna, and colleagues wrote.

Researchers used an online database search for eligible studies on the key words “melanoma,” “diagnosis,” “detection,” “computer-aided” or “artificial intelligence.” Studies were included if they investigated computer-aided diagnosis (CAD) systems accuracy in a screening setting for cutaneous melanoma or one that could be used in that type of setting.

Of 1,694 potentially eligible articles, 132 were included in the qualitative analysis and 70 in the quantitative meta-analysis.

Computer vision was utilized in 58 articles, deep learning in 55 articles and hardware-based methods were utilized in 19, according to researchers.

Fifty studies included only melanocytic lesions while nonmelanocytic lesions were included in 67 studies, according to the study.

The median thickness of invasive melanomas ranged from 0.2 mm to 1.5 mm, the researchers reported.

From the quantitative analysis, the summary estimate for melanoma sensitivity of CAD systems was 0.74 (95% CI, 0.66-0.8) and the specificity was 0.84 (95% CI, 0.79-0.88).

The studies that used proprietary test sets showed a significantly higher sensitivity than ones that used publicly available test sets (0.87; 95% CI, 0.82-0.91 vs. 0.57; 95% CI, 0.44-0.68), the researchers wrote. However, a lower specificity was found in proprietary test sets than publicly available test sets (0.72; 95% CI, 0.63-0.79 vs. 0.91; 95% CI, 0.88-0.94).

Systems using deep learning achieved a sensitivity of 0.44 (95% CI, 0.3-0.59) and a specificity of 0.92 (95% CI, 0.89-0.95). Computer vision systems achieved a sensitivity of 0.85 (95% CI, 0.80-0.88) and a specificity of 0.77 (95% CI, 0.69-0.84).

Researchers found significantly higher specificity in deep learning (0.92; 95% CI, 0.89-0.95) than for computer vision or hardware-based methods.

Existing devices or applications are not widely used despite multiple studies that show expert-level accuracy in CAD systems for melanoma, the researchers wrote.

“A potential reason for this mismatch may be that the results of the studies conducted in this field cannot be transferred directly to clinical practice,” Dick and colleagues wrote.

Additionally, many systems were not tested in the general population or as screening tools as most clinical studies were conducted in specialized referral centers with high melanoma prevalence.

Clinical studies of automated dermoscopic images are missing in the literature, they added.

The researchers wrote that a successful CAD would likely enhance and support dermatologists and not replace them. – by Abigail Sutton

 

 

Disclosures: Dick reports no relevant financial disclosures. Please see the study for all other authors’ relevant financial disclosures.

 

 

    Perspective
    Joshua Arbesman

    Joshua Arbesman

    Early and accurate detection of melanoma is a goal that we hope to achieve within dermatology. However, studies have shown that some lesions are more difficult to detect clinically, particularly amelanotic melanomas.

    Further, clinically suspect lesions do not always have worrisome features histologically, and our benign-to-malignant ratios are not particularly low. We thus hope to utilize computers to aid in our detection of melanoma.

    This recent meta-analysis captures data regarding 70 studies of computer-aided melanoma diagnosis that sought out the melanoma holy grail, high sensitivity and specificity for melanoma detection. The summary is that we are not there yet, but I do hope we get there sooner rather than later. Even as some may argue that dermatologists may be threatened by this, I think the best patient care is the priority. Nevertheless, patients value human interaction and always will, even if it is aided by a computer.

    Critical to any of these analyses is lesion selection both for processing within the study and by the clinician. Interestingly, the paper demonstrates that studies that utilized independent test sets had lower sensitivities than those that analyzed proprietary data sets.

    As dermatologists, we have to select the lesion to examine; computers will then aid us in our assessment of that individual lesion. Many anecdotes are provided of patients (who some hope to utilize computer-aided diagnostic programs by themselves) presenting for a benign lesion that they are concerned about, not noticing the melanoma close by. This will likely be remedied with whole body imaging that is computer processed; however, this will come with challenges as well. Encouragingly, we are getting closer to better diagnostic methods, and hopefully with more data to analyze, we can reach that holy grail of melanoma detection.

    • Joshua Arbesman, MD
    • Cleveland Clinic Main Campus

    Disclosures: Arbesman reports no relevant financial disclosures.

    Perspective
    Justin M. Ko

    Justin M. Ko

    Over the recent years, we have seen an explosion of interest and work in the application of artificial intelligence and machine learning to our field, spanning a breadth of use cases and with impressive results “in silico.” Much of this research has pertained to the classification and diagnosis of skin cancer and melanoma. This study performs a meta-analysis to provide a summary of current state of computer-aided diagnosis of melanoma and highlights some important issues that become increasingly relevant as these tools and systems progress toward use in clinical practice.

    The benchmarking of computer vs. dermatologist, while an important data point that helps to understand performance of such systems, lacks real-world relevance given that such models would not be used in isolation, but rather integrated with and wielded by clinicians. 

    Further, the heterogeneity of studies’ methodology highlights the need for a framework from which to assess “quality” of an AI model or system, including provenance and characteristics of the training data set, methodology of training, and how validation was performed. The issue of bias is particularly important to acknowledge and actively address as the field matures. Biases and gaps between data used to develop an AI model and the data it will encounter in clinical deployment likely explain why, currently, results from real-world clinical data fall short of impressive performance on internal test data.  

    Although holding significant potential, the path to responsible development and deployment of AI models in dermatology is one that needs to be undertaken with collaborative consideration, inclusive of both physicians and patients.

    This path forward also involves great care and thoughtful consideration of intended and potentially unintended consequences. It is not surprising that the great majority of studies are those published in the computer science literature. If we are to continue down the path toward the goal of patient benefit, studies should investigate how we — physicians — will use and integrate such tools and systems into practice, and the impact they might have on outcomes that are clinically meaningful and reflect real-world use.

    Recognizing the central position that our specialty has to play in influencing how medical AI/machine learning models are created, validated and used, the American Academy of Dermatology recently published a position statement on augmented intelligence in dermatology. It lays out principles and characteristics of high-quality augmented intelligence machine learning tools and systems. The promise of achieving better care through synergy of man and machine will require engagement with and solutions to these thorny issues.

    • Justin M. Ko, MD, MBA
    • Clinical Associate Professor
      Director and Chief of Medical Dermatology, Stanford Health Care

    Disclosures: Ko reports no relevant financial disclosures.