Evaluation of the impact of deep-learning based Apollo in improving neuroradiological workflows

Carlota de Macedo Santos

Key information

Authors:

Carlota de Macedo Santos (Carlota De Macedo Santos)

Supervisors:

Ana Catarina Fidalgo Barata (Ana Catarina Fidalgo Barata); Akshay Pai

Published in

July 13, 2021

Abstract

Deep Learning (DL) methods for pathology segmentation and classification have gained undeniable relevance in the radiology department. Their promising potential must be balanced with the risks of misclassifications in unseen data. Evaluating robustness with an adequate set of metrics is a crucial step that is usually done suboptimal in the current practices. In this thesis, we propose a comprehensive evaluation framework that specifically addresses these limitations and jointly assesses the performance of DL models at an image (classification) and lesion(segmentation) levels. Besides analyzing network behaviours across tasks, our method gives a measure of robustness by 1) evaluating the impact of acquisition parameters on performance and 2) applying the framework to an external dataset. The experimental analysis is conducted for two DL solutions, Apollo and nnU-Net, trained on the same data. Results show that algorithms are heavily hampered by unintended data bias. In particular, we obtain lower performances for poorly represented pathologies in the training set and verify that the algorithms struggle to predict from out-of-distribution data, i.e. acquired with a different sequence or in a different direction. Conversely, more discriminative features are learnt for predominant classes and on prevalent sequence types or orientations. Our experiments also suggest that robustness can be improved by identifying key design decisions in the algorithm pipeline formulation. By raising awareness on the importance of external validations and by providing alternatives to the current evaluation frameworks, we give a further step towards the seamless integration of DL technologies in medical settings.