Master's Thesis
Improving Efficiency in the Learning Process of Neural Networks
— 2025
Key information
Authors:
Supervisors:
Published in
December 12, 2025
Abstract
Modern neural networks have been transforming many fields, however, their effectiveness depends heavily on their training which requires substantial computation and energy. To make training cheaper we can either make the model reach the target in fewer steps or reduce the computational cost per step. This thesis investigates algorithmic routes and follows both by proposing a method that reduces how much of a model is updated over time and also a new optimizer that increases progress per data sample. First, we propose Gradient-Slope-Adaptive Freeze (GSAF). GSAF tracks the trend (slope) of the gradient norm at the model's final layer, and uses this to decide when to freeze earlier layers. By progressively shutting off low-yield backpropagation while allowing later layers to continue learning, GSAF reduces per-step computation and shortens overall training without degrading accuracy, as shown in our experiments. Second, we introduce Curvature-Tuned Accelerated Gradient Descent (CT-AGD). CT-AGD performs ordinary mini-batch updates within each epoch and augments them with a single ``boundary'' step tuned by inexpensive per-coordinate finite-difference quotients accumulated online. This design injects targeted curvature information with storage and compute overhead comparable to widely used adaptive methods. In our experiments, CT-AGD matches the final accuracy of standard baselines while reducing the number of training epochs by approximately 33% on average. Taken together, these results argue that modest, principled modifications to what we train (via adaptive freezing) and how we update (via curvature-tuned preconditioning) can deliver practical reductions in compute and energy budgets while preserving accuracy.
Publication details
Authors in the community:
Manuel Graca
ist196271
Supervisors of this institution:
Fields of Science and Technology (FOS)
electrical-engineering-electronic-engineering-information-engineering - Electrical engineering, electronic engineering, information engineering
Publication language (ISO code)
eng - English
Rights type:
Embargo lifted
Date available:
November 16, 2026
Institution name
Instituto Superior Técnico