Automatic Identification of Regions of Interest in Dermoscopy Images Using Vision Transformers and Weakly Supervised Learning

Diogo José Pereira Araújo

Key information

Authors:

Diogo José Pereira Araújo (Diogo José Pereira Araújo)

Supervisors:

Carlos Jorge Andrade Mariz Santiago (Carlos Jorge Andrade Mariz Santiago); Ana Catarina Fidalgo Barata (Ana Catarina Fidalgo Barata)

Published in

11/24/2023

Abstract

Skin cancer is a growing public health concern. Early detection of the lesion plays a critical role in ensuring successful treatment of the cancer. Dermatologists traditionally use criteria like the 7-point checklist, which focuses on specific dermoscopic characteristics without considering their spatial distribution in the lesion. Multiple Instance Learning (MIL) is a weakly supervised learning technique that serves as an approximation to this criterion in the field of deep learning. In contrast to these methods, Vision Transformers (ViTs) have recently shown remarkable promise, while at the same time using spatially aware information from all the patches in the image. This contrast motivates us to address two questions in dermoscopy image analysis: (1) the understanding of whether all patches are relevant for skin cancer diagnosis, and (2) the influence of the spatial arrangement of the patches on diagnostic accuracy. To address these questions, we introduce a two-branch framework that combines a ViT-based architecture with a MIL model. We tackle both binary classification (melanoma vs. nevus) and multi-class classification (with eight skin disease types). Our work presents a novel two-stage MIL formulation oriented towards binary classification, and we extend it to a three-stage approach for multi-class classification. Our results consistently demonstrate the competitive performance of these formulations in both binary and multi-class contexts. Our findings reveal that only certain patches are critical for correct classification, and that adding spatial information slightly improves classification accuracy.