Multi-Dimensional LLM Judges Outperform Simple Models in Evaluating Mathematical Formalization

Converting natural language mathematics into formal symbolic notation—a process called autoformalization—is essential for enabling AI systems to reason rigorously about mathematical proofs and theorems. While large language models have recently become more capable at this translation task, methods for automatically grading the quality of these translations have lagged behind.

The Limitations of Generic Evaluation#

Current approaches to evaluating autoformalization tend to rely on broad, one-size-fits-all scoring rubrics. These coarse-grained methods miss important nuances that distinguish truly reliable formal mathematical reasoning from superficially plausible but flawed translations. The problem is particularly acute in formal mathematics, where subtle errors can invalidate entire proofs.

A New Framework for Principled Assessment#

A new paper from researchers proposes using an ensemble approach based on what they call an Epistemically and Formally Grounded (EFG) set of specialized LLM judges. Rather than asking a single model to rate overall quality, this framework deploys multiple judges, each trained to evaluate one specific dimension:

• Logical Preservation (LP) — whether the formalization captures the original statement's logical structure
• Mathematical Consistency (MC) — whether the translation respects established mathematical definitions and relationships
• Formal Quality (FQ) — the syntactic and stylistic correctness of the symbolic notation
• Formal Validity (FV) — whether the translated statement can actually be processed by formal verification systems

Why Specialization Matters#

By breaking evaluation into these four atomic properties, the ensemble approach becomes more transparent and interpretable. Rather than a black-box overall score, researchers and practitioners can see exactly which aspects of a formalization are strong or weak. This granular feedback also helps identify where autoformalization systems need improvement.

Validation and Implications#

Testing the framework against simpler baseline models demonstrated that the EFG ensemble more reliably predicts whether a formalization is genuinely correct. The researchers argue that this multidimensional, property-driven approach to LLM-as-judge evaluation could scale beyond mathematics—offering a template for assessing other complex reasoning tasks where quality depends on multiple interlocking criteria.