Master's Thesis

Evaluating Open-Source Large Language Models for Lemma and Predicate Inference in Dafny

João Rui Vargas de Matos — 2025

Key information

Authors:

João Rui Vargas de Matos (João Rui Vargas de Matos)

Supervisors:

João Fernando Peixoto Ferreira; Alexandra Sofia Ferreira Mendes

Published in

October 28, 2025

Abstract

Formal verification languages like Dafny enable developers to rigorously prove program correctness but remain difficult to use due to their mathematical rigor and specification overhead. Recent advances in Large Language Models (LLMs) offer opportunities to assist developers in these tasks, yet their effectiveness in verification-aware contexts is not well understood. This thesis investigates how LLMs can generate auxiliary lemmas and predicates for incomplete Dafny programs. We design a prompt-based pipeline that uses Ollama and LangChain, and evaluate multiple open-source LLMs and prompt strategies on a curated Dafny benchmark. Results show that few-shot prompting without lemma bodies yields the highest verification success rate (up to 11.1%), though overall performance remains limited. Our analysis highlights common failure modes, including syntax and verification errors, and suggests that improved prompting, fine-tuning, or retrieval-augmented methods may substantially increase LLM effectiveness for formal verification tasks.

Publication details

Authors in the community:

Fields of Science and Technology (FOS)

electrical-engineering-electronic-engineering-information-engineering - Electrical engineering, electronic engineering, information engineering

Publication language (ISO code)

eng - English

Rights type:

Embargo lifted

Date available:

October 7, 2026

Institution name

Instituto Superior Técnico