Evaluating Transformers Spatial Reasoning Through Their Positional Encoding with ARC

November 5, 2024

The Abstraction and Reasoning Corpus is a compilation of reasoning tasks that are deeply rooted in Knowledge Priors, which are essential human skills for effective problem-solving, such as counting. These tasks present a relevant challenge to machine learning models, demanding abstract reasoning and a capacity for generalization that goes beyond the scope of conventional tasks. Despite their success in specialized domains like games and natural language understanding, deep learning models often struggle when confronted with the diverse, high-level problems posed by this dataset. Our work seeks to understand why transformer-based models, which have revolutionized many fields, encounter significant difficulties in solving such tasks. Our exploration focuses on positional encoding, a key mechanism designed to induce spatial information within the input data, which we hypothesize plays a crucial role in these models' ability to generalize. Through a series of detailed experiments, we analyze the effectiveness of two-dimensional positional encodings in comparison to simpler methods where positional information is directly embedded in the input. Additionally, we examine the performance of state-of-the art rotary positional encoding and investigate how varying the model architecture and size of training impacts performance. Our findings provide insights into why large language models, such as CodeT5+, may fall short when applied to these challenging tasks, while offering a comprehensive analysis of how positional encoding influences model success.

Authors in the community:

• 

  G

  Guilherme Henriques Bandeira Costa

  ist195586

Supervisors of this institution:

• 

  A

  Arlindo Manuel Limede de Oliveira

  ist12282

electrical-engineering-electronic-engineering-information-engineering - Electrical engineering, electronic engineering, information engineering

eng - English

Instituto Superior Técnico