AI
New benchmark tests AI assistants on messy, real-world tasks with stateful environments
Researchers introduce LiveClawBench, a framework that evaluates language model agents against complex multi-service tasks that preserve execution state and dependencies—addressing the fidelity gap in existing benchmarks.
Most benchmarks for AI assistants test them in isolation—single tasks that don't mirror how real software works. Researchers have now proposed a new evaluation framework that captures the messiness of authentic assistant scenarios, where services depend on one another, state persists across steps, and user intent can remain implicit.
The fidelity problem#
Existing benchmarks face a dilemma. Many sacrifice real-world representativeness by selecting tasks that are simple to mock and measure. This leaves out genuine challenges like cross-service dependencies, contaminated application state, unclear user goals, and changes that occur during execution. Others use live environments that are difficult to reproduce, making it hard for other researchers to verify results and build on the work.
LiveClawBench approach#
The new benchmark, LiveClawBench, addresses this by combining two core components. First, a Triple-Axis Complexity Framework structures task creation around difficulty drivers, ensuring that challenges mirror real work. Second, the benchmark includes reproducible mock applications—full-stack replicas of services, not just API stubs—that maintain sessions, artifacts, state changes, and downstream effects.
Scale and scope#
The benchmark comprises 134 executable test cases across 10 distinct domains, with 22 mocked services. This design enables controlled evaluation at factor level: researchers can isolate which aspects of complexity (dependencies, state, intent) cause agents to stumble.
Why it matters#
As AI assistants move beyond simple tool calls toward open-ended, stateful work, evaluating them fairly requires benchmarks that don't oversimplify the problems they'll face. LiveClawBench aims to close the gap between how we measure these systems and how they'll actually be used in practice.
Researchers have released the benchmark, leaderboard, and trajectory data to support community evaluation and extension of the work.
