Master's Thesis

Deep Neural Network Characterization with Cache-Aware Roofline Modeling in Tensor Cores

Diogo de Sousa Matos — 2024

Key information

Authors:

Diogo de Sousa Matos (Diogo de Sousa Matos)

Supervisors:

Leonel Augusto Pires Seabra de Sousa (Leonel Augusto Pires Seabra de Sousa); Aleksandar Ilic (Aleksandar Ilic)

Published in

November 27, 2024

Abstract

To achieve an adequate level of accuracy, deep learning models can scale up to high complexities with millions of trainable parameters, making them very computationally and memory intensive. Nowadays, general-purpose CPU and GPU devices typically are not capable of efficiently tackling these challenges, thus, specialized accelerators for artificial intelligence, such as the NVIDIA Tensor Core, are used to train and run these networks. However, in contrast to CPUs and GPUs, there is an urgent need to devise a methodology for AI-oriented hardware to guide users to achieve the desired performance and energy efficiency, especially when optimizing neural networks. Given the significant architectural differences, it is not a trivial task to decide which technique to use. This work aims to bridge this gap by proposing a Tensor Cache-Aware Roofline Model for the performance and efficiency characterization of deep learning workloads. To provide a better understanding of the efficient use of the Tensor Core, several performance, power and energy efficiency models are developed through extensive micro-benchmarking to fully exercise different features of the Tensor Core, enabling an increase in energy efficiency of up to 20%. The models are experimentally validated in two different devices of the NVIDIA Ampere architecture, and the insights they provide are explored through the characterization of different deep learning layers, using cuDNN which powers most modern AI frameworks. Finally, the optimization potential of the proposed strategies is shown through the optimization of a VGG model, resulting in an increase of the energy efficiency of 9.74%.

Publication details

Authors in the community:

Supervisors of this institution:

Fields of Science and Technology (FOS)

electrical-engineering-electronic-engineering-information-engineering - Electrical engineering, electronic engineering, information engineering

Publication language (ISO code)

por - Portuguese

Rights type:

Embargo lifted

Date available:

October 8, 2025

Institution name

Instituto Superior Técnico