Software Engineering
Monty framework automates conversion of code assertions, reducing manual specification work
Researchers present a technique that uses large language models to translate developer intent into formal executable assertions, boosting accuracy by up to 20 percentage points over naive approaches.
Developers spend considerable effort writing formal contracts for software testing and verification—a process that is both time-consuming and prone to mistakes. To ease this burden, researchers have developed Monty, a framework designed to automatically convert natural-language specifications into formal executable assertions using large language models.
The problem with manual assertions#
Creating formal specifications that accurately capture developer intent remains a bottleneck in software development. Traditional approaches require hand-coding these assertions, introducing errors and slowing down the testing pipeline. The gap between informal requirements and executable specifications has long hindered adoption of more rigorous verification practices.
How Monty works#
Rather than directly translating text to code, Monty employs a multi-stage filtering approach. The system generates multiple candidate formalizations and evaluates them using two key metrics: a conformance score that measures how well a formalization aligns with the original intent, and validity scores derived from actually testing the code against each proposed assertion. This combination helps eliminate invalid or misaligned specifications before they reach developers.
Testing and results#
The team evaluated Monty on over 500 assertion-generation tasks extracted from 22 Java collection-like classes. The filtering-based approach consistently outperformed naive language model translation, achieving precision improvements of up to 20 percentage points on average. This suggests the technique is robust enough to handle both the inherent ambiguity in natural language and the strict requirements of formal verification systems.
Why it matters#
By automating assertion generation, Monty could help teams adopt formal methods more readily and reduce the burden of writing specifications from scratch. The framework demonstrates that LLMs are capable of bridging the semantic gap between informal intent and formal code—provided the outputs are properly validated and filtered.