New AI Model Dramatically Improves Formal Verification Specifications

Automated specification synthesis has long challenged large language models—but a new 20-billion-parameter system trained on repair-based learning shows substantial gains in generating verifiable formal specifications for distributed systems.

The Problem#

TLA+ is a widely used language for describing and checking safety-critical protocols and distributed systems. Yet when popular LLMs attempt to generate TLA+ code, they fail at alarming rates: across 25 tested models, even the best achieve only 26.6% syntactic correctness and just 8.6% semantic validity (passing model-checking). These failures are costly in mission-critical domains where formal guarantees matter.

TLA-Prover's Approach#

Researchers introduced TLA-Prover, a specialized model trained through a two-stage pipeline. The system begins with supervised fine-tuning on verified examples, then advances to a repair-based learning phase called group-relative policy optimization (GRPO). Rather than relying on a learned reward model, the approach uses the TLC model checker itself as a direct signal—the ground truth for whether a specification is correct.

The training methodology includes an ablation variant using direct preference optimization (DPO) from the same checkpoint, allowing comparison of different alignment strategies.

Measuring Success with Strict Grading#

The research defines four tiers of output quality:

• Bronze: syntactically parseable
• Silver: parses without warnings
• Gold: passes the TLC model checker
• Diamond: additionally verified by modifying the correctness property automatically; TLC must detect a violation

The Diamond tier is particularly clever—it prevents the failure mode where trivial properties (those always true) artificially inflate success rates.

Results#

TLA-Prover reaches 30% pass@1 on a held-out 30-problem benchmark for both Gold and Diamond tiers, roughly 3.5 times higher than the untuned baseline's 8.6%. The DPO variant achieves 20% at Diamond. Notably, Gold and Diamond performance align at every training checkpoint, eliminating the risk of generating syntactically correct but semantically empty specifications.