New evaluation method reveals hidden progress in AI agent skill development

Evaluating whether AI agents complete tasks has long relied on a blunt instrument: did it pass or fail? A new framework challenges this binary approach by capturing the intermediate steps and partial progress that standard metrics overlook.

The limits of binary scoring#

Current evaluation methods for language-model agents working on complex, multi-step tasks reduce entire execution sequences to a single outcome. An agent either passes the official verifier or it doesn't. This approach masks meaningful progress, especially when assessing whether an agent has genuinely acquired a specific skill—since a procedural improvement might enhance how an agent works without ultimately changing whether the final answer succeeds.

Checklists offer a more granular alternative, but they come with practical limitations. Manually creating task-specific checklists is labor-intensive and difficult to scale across diverse tasks, while automatically generating them tends to be unreliable.

How GCPC works#

To address these constraints, researchers have developed Grounded Checklist Partial Credit (GCPC), a hybrid approach that combines human oversight with language-model assistance. The system operates in distinct phases:

• Human rule definition: Experts write reusable evaluation rules once, capturing the kinds of steps or conditions that matter across related tasks
• Task-specific instantiation: An LLM takes those generic rules and adapts them into a detailed checklist tied to the particular task's instructions and success criteria
• Evidence-based scoring: A separate judge evaluates each checklist item based only on what's visible in the execution log, declining to score items when evidence is absent
• Official verification alignment: A scripted final step reconciles the partial score with the task's official pass/fail outcome

This structure keeps judgments grounded in observable behavior rather than guesswork.

Stronger discrimination of agent performance#

Testing on over 4,400 trajectories from SkillsBench, GCPC demonstrated better ability to distinguish passing from failing agent runs than traditional holistic assessment (AUC of 0.689 versus 0.619). Human evaluation on 96 trajectories confirmed that GCPC scores align more closely with expert intuitions about progress.