Master's Thesis
Deep Neural Network Characterization with Cache-Aware Roofline Modeling in Tensor Cores
— 2024
Key information
Authors:
Supervisors:
Published in
November 27, 2024
Abstract
To achieve an adequate level of accuracy, deep learning models can scale up to high complexities with millions of trainable parameters, making them very computationally and memory intensive. Nowadays, general-purpose CPU and GPU devices typically are not capable of efficiently tackling these challenges, thus, specialized accelerators for artificial intelligence, such as the NVIDIA Tensor Core, are used to train and run these networks. However, in contrast to CPUs and GPUs, there is an urgent need to devise a methodology for AI-oriented hardware to guide users to achieve the desired performance and energy efficiency, especially when optimizing neural networks. Given the significant architectural differences, it is not a trivial task to decide which technique to use. This work aims to bridge this gap by proposing a Tensor Cache-Aware Roofline Model for the performance and efficiency characterization of deep learning workloads. To provide a better understanding of the efficient use of the Tensor Core, several performance, power and energy efficiency models are developed through extensive micro-benchmarking to fully exercise different features of the Tensor Core, enabling an increase in energy efficiency of up to 20%. The models are experimentally validated in two different devices of the NVIDIA Ampere architecture, and the insights they provide are explored through the characterization of different deep learning layers, using cuDNN which powers most modern AI frameworks. Finally, the optimization potential of the proposed strategies is shown through the optimization of a VGG model, resulting in an increase of the energy efficiency of 9.74%.
Publication details
Authors in the community:
Diogo de Sousa Matos
ist195778
Supervisors of this institution:
Aleksandar Ilic
ist166430
Fields of Science and Technology (FOS)
electrical-engineering-electronic-engineering-information-engineering - Electrical engineering, electronic engineering, information engineering
Publication language (ISO code)
por - Portuguese
Rights type:
Embargo lifted
Date available:
October 8, 2025
Institution name
Instituto Superior Técnico