Master's Thesis

Optimizing a Deep Residual Neural Network (ResNet) Implementation on GPU

Tiago André Gomes Miranda — 2025

Key information

Authors:

Tiago André Gomes Miranda (Tiago André Gomes Miranda)

Supervisors:

Paulo Alexandre Crisóstomo Lopes (Paulo Alexandre Crisóstomo Lopes)

Published in

December 4, 2025

Abstract

Deep learning has gained widespread popularity in recent years, driving demand for efficient and scalable frameworks that can handle computationally intensive models, such as ResNet-56. This thesis focuses on optimizing key components of the Caffe framework, specifically the CUDA C++ code of the Convolutional and Batch Normalization layers, to improve performance on a GeForce GTX 1050 GPU (2GB). The original objective was to fuse the Convolution, BN, and ReLU layers into a single layer for both the forward and backward passes. However, the work completed within this scope includes optimization of the Convolution layer (forward pass) and Batch Normalization layer (forward and backward passes). This thesis presents substantial performance improvements for both layers by applying tailored memory management techniques, kernel optimizations, and customized GPU implementations. The developed solution achieved up to a 4x speed-up in the backward pass of the BN layer and over a 2x improvement in the forward pass of the Convolution layer, showing the potential of tailoring deep learning framework layers to specific hardware architectures. Future work may focus on optimizing the backward pass of the Convolution layer, integrating the ReLU layer, and improving the im2col transformation based on recent studies in GPU memory traffic optimization. Further optimization through PTX and memory bank analysis is also proposed to push the performance boundaries of this solution.

Publication details

Authors in the community:

Supervisors of this institution:

Fields of Science and Technology (FOS)

electrical-engineering-electronic-engineering-information-engineering - Electrical engineering, electronic engineering, information engineering

Publication language (ISO code)

eng - English

Rights type:

Embargo lifted

Date available:

November 10, 2026

Institution name

Instituto Superior Técnico