AI
New Agentic Judges Help AI Models Understand Code Architecture at Scale
Researchers propose an automated evaluation pipeline that teaches language models to reason about software design, boosting performance on real-world programming tasks by up to 540%.
Training large language models to write correct code is one challenge; teaching them to respect the architectural conventions of an existing codebase is quite another. A new paper from arXiv addresses this gap by introducing an agentic judging system that scales expert-level architectural evaluation without requiring manual labeling.
The Problem: Architecture Beyond Correctness#
Large language models have made impressive strides in software engineering tasks, yet real-world development demands more than syntactically sound code. Engineers must understand and maintain consistency with a project's architectural patterns—its design principles, module organization, and coding conventions. Evaluating this architectural reasoning manually is prohibitively expensive, and traditional test-based verification cannot capture it fully. The researchers identified this as a critical blind spot in how code models are currently trained and assessed.
A Two-Judge Solution#
The team proposes an agentic pipeline that delegates architectural evaluation to two specialized judges. The Architecture Complexity Judge assesses how much architectural knowledge a particular coding task demands, while the Architecture Quality Judge evaluates whether proposed code changes conform to repository-specific conventions. Both judges use a strong LLM as a scalable proxy for expert judgment, with evaluations grounded in the actual source code and rubrics tailored to each codebase.
Training Results and Performance Gains#
Fine-tuning the Qwen models (8B, 14B, and 32B parameters) on 3,360 carefully curated instances produced substantial improvements. On the SWE-bench Verified benchmark, the trained models achieved resolution rates up to 27.2%—representing a 540% boost over the base model and a 256% improvement over models trained on unfiltered data. Beyond raw performance, the approach demonstrated strong cross-language generalization and consistent gains in the architectural quality of generated patches.
Why It Matters#
This work points toward a path for training code models that think like software architects, not just syntax checkers. By automating expert evaluation through agentic judgment, the research makes it feasible to incorporate architectural reasoning into training pipelines at scale—a capability that could significantly improve the real-world utility of AI-assisted coding tools.
