Embedded System for Aerial Image Segmentation with Transformers

24 de novembro de 2025

Aerial image segmentation plays a critical role in domains ranging from urban development to environmental assessment and emergency management. Despite the state-of-the-art accuracy of Vision Transformers (ViTs) for this task, their deployment on edge devices is still largely underexplored due to their significant computational and memory requirements. This thesis addresses this challenge by designing and demonstrating a fully functional, hardware accelerated Swin Transformer Layer on a SoC-FPGA system. A Swin Transformer–based architecture, AerialFormer, is selected as the baseline for aerial image segmentation. To improve embedded suitability while preserving accuracy, an embedded-oriented variant, Hardware-RialFormer, is developed, maintaining 99.9% of the original mean Intersection over Union (mIoU) metric. A complete Swin Layer was implemented in software, using the C language, and, for the first time in literature, open-sourced. This implementation is embedded system-oriented, with simple memory management. A hardware IP was specifically designed to accelerate the most computationally intensive component of the Swin Layer, the Feed-Forward Network (FFN), with both weights and activations quantized to 8-bit fixed-point. The accelerator computes the FFN 313× faster than the ARM software implementation. A proof of concept hardware/software system – ARES, was successfully implemented on a ZCU104 development board. This system executes a complete Swin Transformer Layer, with the Feed-Forward Network computed in the dedicated hardware accelerator, implemented in the SoC-FPGA programmable logic, and the remaining components executed in software on the ARM embedded processor. The complete ARES system processes the Swin Layer 3.08× faster than the software-only implementation, approaching the Amdahl-theoretic maximum of 3.1×.

Autores da comunidade :

• 

  J

  João Barreiros Coelho Rodrigues

  ist199968

  Afiliações e fontes

  IST > INESC-ID

  Instituto de Engenharia de Sistemas e Computadores - Investigação e Desenvolvimento em Lisboa

Orientadores desta instituição:

• 

  H

  Horácio Cláudio De Campos Neto

  ist11900

electrical-engineering-electronic-engineering-information-engineering - Engenharia Eletrotécnica, Eletrónica e Informática

eng - Inglês

Instituto Superior Técnico