Master's Thesis
Optimizing a Deep Residual Neural Network (ResNet) Implementation on GPU
— 2025
Key information
Authors:
Supervisors:
Published in
December 4, 2025
Abstract
Deep learning has gained widespread popularity in recent years, driving demand for efficient and scalable frameworks that can handle computationally intensive models, such as ResNet-56. This thesis focuses on optimizing key components of the Caffe framework, specifically the CUDA C++ code of the Convolutional and Batch Normalization layers, to improve performance on a GeForce GTX 1050 GPU (2GB). The original objective was to fuse the Convolution, BN, and ReLU layers into a single layer for both the forward and backward passes. However, the work completed within this scope includes optimization of the Convolution layer (forward pass) and Batch Normalization layer (forward and backward passes). This thesis presents substantial performance improvements for both layers by applying tailored memory management techniques, kernel optimizations, and customized GPU implementations. The developed solution achieved up to a 4x speed-up in the backward pass of the BN layer and over a 2x improvement in the forward pass of the Convolution layer, showing the potential of tailoring deep learning framework layers to specific hardware architectures. Future work may focus on optimizing the backward pass of the Convolution layer, integrating the ReLU layer, and improving the im2col transformation based on recent studies in GPU memory traffic optimization. Further optimization through PTX and memory bank analysis is also proposed to push the performance boundaries of this solution.
Publication details
Authors in the community:
Tiago André Gomes Miranda
ist176303
Supervisors of this institution:
Paulo Alexandre Crisóstomo Lopes
ist23775
Fields of Science and Technology (FOS)
electrical-engineering-electronic-engineering-information-engineering - Electrical engineering, electronic engineering, information engineering
Publication language (ISO code)
eng - English
Rights type:
Embargo lifted
Date available:
November 10, 2026
Institution name
Instituto Superior Técnico