New system checks whether AI-generated legal code is actually accurate

When an AI converts written law into formal mathematical logic, how can you trust the result is correct? Researchers have developed a roundtrip verification framework that validates whether an LLM's formalization truly captures the original meaning—without needing hand-checked answers to compare against.

How the method works#

The approach operates in a cycle: first, an LLM formalizes a legal statement. Then, that formal code is translated back into plain language. The system re-formalizes this new natural-language version and uses formal verification tools to check whether the two formalizations are logically equivalent. If they match, that's evidence the original translation was faithful. When they differ, a diagnostic step pinpoints exactly where the error occurred—in the first formalization, the back-translation, or the re-formalization—and an automated repair operator attempts to fix the problem at that specific stage.

Testing on real legal code#

The researchers evaluated their framework on statutory texts from the Texas Transportation Code and Texas Parks and Wildlife Code, using two advanced language models (Claude Opus 4.6 and GPT-5.2). They compared their repair method against three baselines. The diagnosis-guided scoped repair approach proved most effective, though its success depended on how reliably the diagnostic function identified where errors originated.

Quality signal for formal verification#

Across both legal domains and both models, formal verification results provided a meaningful quality signal: rules that failed the equivalence check showed 1.4 to 2.5 times more semantic drift in natural-language inference than rules that passed it. This suggests the roundtrip verification method reliably identifies when formalization has gone astray.