Master's Thesis
Evaluating Open-Source Large Language Models for Lemma and Predicate Inference in Dafny
— 2025
Key information
Authors:
Supervisors:
Published in
October 28, 2025
Abstract
Formal verification languages like Dafny enable developers to rigorously prove program correctness but remain difficult to use due to their mathematical rigor and specification overhead. Recent advances in Large Language Models (LLMs) offer opportunities to assist developers in these tasks, yet their effectiveness in verification-aware contexts is not well understood. This thesis investigates how LLMs can generate auxiliary lemmas and predicates for incomplete Dafny programs. We design a prompt-based pipeline that uses Ollama and LangChain, and evaluate multiple open-source LLMs and prompt strategies on a curated Dafny benchmark. Results show that few-shot prompting without lemma bodies yields the highest verification success rate (up to 11.1%), though overall performance remains limited. Our analysis highlights common failure modes, including syntax and verification errors, and suggests that improved prompting, fine-tuning, or retrieval-augmented methods may substantially increase LLM effectiveness for formal verification tasks.
Publication details
Authors in the community:
João Rui Vargas de Matos
ist195610
Fields of Science and Technology (FOS)
electrical-engineering-electronic-engineering-information-engineering - Electrical engineering, electronic engineering, information engineering
Publication language (ISO code)
eng - English
Rights type:
Embargo lifted
Date available:
October 7, 2026
Institution name
Instituto Superior Técnico