Enhancing the Application of Large Language Models (LLMs) in Education through Effective Testing Frameworks

Diogo de Araújo Frazão Castelo Branco

Key information

Authors:

Diogo de Araújo Frazão Castelo Branco (Diogo de Araújo Frazão Castelo Branco)

Supervisors:

Luís Miguel Veiga Vaz Caldas de Oliveira (Luís Miguel Veiga Vaz Caldas de Oliveira)

Published in

November 25, 2025

Abstract

The rapid integration of LLMs into educational technology enables teachers to design personalized virtual tutors. However, adoption among primary school educators is hindered by the lack of user-friendly frameworks to safely and reliably evaluate their pedagogical impact. Existing methodologies are often complex, highly technical, and focused on advanced levels, neglecting the needs of primary school settings. This dissertation addresses this gap by designing and evaluating a multi-agent framework that empowers primary school teachers to design and test LLM-based tutors. Its central objective was to assess the framework’s reliability in replicating real-world teaching interactions and supporting objective evaluation of the tutor’s pedagogical performance. The framework employs three LLM-based agents, the Student AI, the Intern Tutor AI, and the Supervisor AI, coordinated across sequential stages: Pretest, Interaction, Post-test, Evaluation, and Improvement. The Supervisor AI acts as a specialized LLM judge, assessing the Tutor’s performance against subjective pedagogical metrics, including Cognitive Levels (based on the Bloom taxonomy), Adaptability, Curiosity Stimulation, and Active Learning. The Supervisor AI reached 89% overall accuracy and 76% consistency in classifying qualitative pedagogical metrics. The Student AI consistently adhered to instructions, achieving 94-100% accuracy on explicitly taught questions. Teacher and student experiments validated the interface usability. The study shows that the framework reliably supports primary school teachers in producing detailed and accurate evaluations of LLM-based assistants.