New benchmark tests how well AI agents plan complex space missions

A new benchmark called AstroAgentBench measures how effectively AI language models can handle the intricate planning demands of space missions, comparing their performance against purpose-built solvers and identifying where agentic systems fall short.

What the benchmark covers#

The evaluation framework encompasses seven different task families across the space domain: scheduling, observation planning for scientific instruments, constellation design, and relay communications support. For each scenario, an AI agent must generate a planning artifact—a formal document or schedule—that an external verifier checks for correctness on multiple dimensions: whether the format matches requirements, timing is feasible, geometry respects orbital constraints, resources stay within budgets, and the plan delivers meaningful mission value.

Performance across systems#

Testing five separate LLM agent systems against 35 held-out test cases reveals a mixed picture. Top-performing models occasionally match or surpass the scores of specialized space-planning solvers on some task families. However, weaker systems frequently struggle to produce valid, high-value plans at all. Even strong performers show degradation when tasks become geometrically complex, demand intricate product-level logic, or require creative design thinking.

Where agents stumble#

Detailed analysis of execution traces uncovers two recurring failure modes. First, agents misunderstand the task specifications—they misformulate how to translate the problem into executable code. Second, even when they grasp the requirements, they construct weak solutions that don't navigate the search space effectively. Conversely, successful runs reveal a pattern: the agent iterates against verifier feedback to refine its own code implementations and adapts its search strategy to exploit structure specific to each case.

Calibration and memory matter#

Supplementary experiments show that procedure injection (providing the agent with explicit solution templates) and memory accumulation (allowing systems to build on past attempts) offer selective benefits—they help only when they directly address the missing element, whether that's correct problem formulation, solution refinement, or search guidance.

This work .