Embedded System for Aerial Image Segmentation with Transformers

November 24, 2025

Aerial image segmentation plays a critical role in domains ranging from urban development to environmental assessment and emergency management. Despite the state-of-the-art accuracy of Vision Transformers (ViTs) for this task, their deployment on edge devices is still largely underexplored due to their significant computational and memory requirements. This thesis addresses this challenge by designing and demonstrating a fully functional, hardware accelerated Swin Transformer Layer on a SoC-FPGA system. A Swin Transformer–based architecture, AerialFormer, is selected as the baseline for aerial image segmentation. To improve embedded suitability while preserving accuracy, an embedded-oriented variant, Hardware-RialFormer, is developed, maintaining 99.9% of the original mean Intersection over Union (mIoU) metric. A complete Swin Layer was implemented in software, using the C language, and, for the first time in literature, open-sourced. This implementation is embedded system-oriented, with simple memory management. A hardware IP was specifically designed to accelerate the most computationally intensive component of the Swin Layer, the Feed-Forward Network (FFN), with both weights and activations quantized to 8-bit fixed-point. The accelerator computes the FFN 313× faster than the ARM software implementation. A proof of concept hardware/software system – ARES, was successfully implemented on a ZCU104 development board. This system executes a complete Swin Transformer Layer, with the Feed-Forward Network computed in the dedicated hardware accelerator, implemented in the SoC-FPGA programmable logic, and the remaining components executed in software on the ARM embedded processor. The complete ARES system processes the Swin Layer 3.08× faster than the software-only implementation, approaching the Amdahl-theoretic maximum of 3.1×.

Authors in the community:

• 

  J

  João Barreiros Coelho Rodrigues

  ist199968

  Affiliations and sources

  IST > INESC-ID

  Instituto de Engenharia de Sistemas e Computadores - Investigação e Desenvolvimento em Lisboa

Supervisors of this institution:

• 

  H

  Horácio Cláudio De Campos Neto

  ist11900

electrical-engineering-electronic-engineering-information-engineering - Electrical engineering, electronic engineering, information engineering

eng - English

Instituto Superior Técnico