Master's Thesis

Improving Efficiency in the Learning Process of Neural Networks

Manuel Sobral de Menezes da Silva Graça — 2025

Key information

Authors:

Manuel Sobral de Menezes da Silva Graça (Manuel Graca)

Supervisors:

Arlindo Manuel Limede de Oliveira (Arlindo Manuel Limede de Oliveira); Luís Miguel Teixeira D'Avila Pinto da Silveira (Luís Miguel Teixeira D'Avila Pinto da Silveira)

Published in

December 12, 2025

Abstract

Modern neural networks have been transforming many fields, however, their effectiveness depends heavily on their training which requires substantial computation and energy. To make training cheaper we can either make the model reach the target in fewer steps or reduce the computational cost per step. This thesis investigates algorithmic routes and follows both by proposing a method that reduces how much of a model is updated over time and also a new optimizer that increases progress per data sample. First, we propose Gradient-Slope-Adaptive Freeze (GSAF). GSAF tracks the trend (slope) of the gradient norm at the model's final layer, and uses this to decide when to freeze earlier layers. By progressively shutting off low-yield backpropagation while allowing later layers to continue learning, GSAF reduces per-step computation and shortens overall training without degrading accuracy, as shown in our experiments. Second, we introduce Curvature-Tuned Accelerated Gradient Descent (CT-AGD). CT-AGD performs ordinary mini-batch updates within each epoch and augments them with a single ``boundary'' step tuned by inexpensive per-coordinate finite-difference quotients accumulated online. This design injects targeted curvature information with storage and compute overhead comparable to widely used adaptive methods. In our experiments, CT-AGD matches the final accuracy of standard baselines while reducing the number of training epochs by approximately 33% on average. Taken together, these results argue that modest, principled modifications to what we train (via adaptive freezing) and how we update (via curvature-tuned preconditioning) can deliver practical reductions in compute and energy budgets while preserving accuracy.

Publication details

Authors in the community:

Fields of Science and Technology (FOS)

electrical-engineering-electronic-engineering-information-engineering - Electrical engineering, electronic engineering, information engineering

Publication language (ISO code)

eng - English

Rights type:

Embargo lifted

Date available:

November 16, 2026

Institution name

Instituto Superior Técnico