New Framework Improves Code Quality Judging by Comparing Pairs Instead of Individual Scores

Evaluating code generation systems remains a complex challenge, especially when quality involves trade-offs that go beyond whether the code simply works. Recent research from arXiv proposes a new approach to address this problem.

The Problem with Current Approaches#

Most existing systems for judging code quality use a pointwise evaluation method: they score each piece of code independently using a set of rubrics, then compare the aggregated scores to determine which one is better. Researchers have found that this approach performs worse than a simpler, unified judge that evaluates code holistically.

CriterAlign: A New Evaluation Framework#

The researchers introduce CriterAlign, which fundamentally rethinks how rubric-based judging works. Rather than evaluating each code sample separately and deriving preferences from the scores, CriterAlign makes pairwise judgments directly at the criterion level. The framework includes several key components:

• Criterion-level pairwise judgments — comparing two code samples against each specific criterion
• Tie-driven criterion refinement — improving criteria when the system cannot decisively pick a winner
• Swap-consistency filtering — ensuring consistency in judgments
• Human-Preference-Aligned Guidance (HPAG) — a technique that learns from human preferences by identifying recurring gaps between what humans prefer and what the monolithic judge predicts

HPAG is extracted offline from training examples and injected into the criterion generator, criterion judge, and final decision-making stage.

Concrete Results#

Testing CriterAlign on BigCodeReward, a benchmark dataset, the team improved a Qwen2.5-VL-32B monolithic judge's accuracy from 60.4% to 66.3%. Ablation studies confirmed that both the pairwise criterion design and the HPAG component contributed meaningfully to the improvement.