New benchmark tests whether AI agents can sustain long-term business decisions

Language model agents have demonstrated impressive capability at discrete, focused tasks—from code generation to handling customer inquiries. But sustaining complex decision-making over extended periods, amid shifting conditions and incomplete information, remains largely unproven territory. A new paper from arXiv introduces a benchmark designed to test precisely these capabilities.

The CEO-Bench simulation#

Researchers developed CEO-Bench, a framework that places an AI agent in control of a fictional startup for 500 simulated days. Rather than tackling isolated problems, the agent must contend with interconnected challenges: setting prices, planning marketing campaigns, managing budgets, and monitoring business metrics through a programmable Python interface. The simulation mirrors real-world conditions—noisy data, multiple moving pieces, and the need to balance competing priorities toward a single bottom line.

Success hinges on four skills that remain underdeveloped in current systems:

• Sustaining performance over long horizons despite uncertainty
• Extracting signal from messy, fragmented information sources
• Adapting strategy as conditions change
• Coordinating dozens of decisions across different domains

How the strongest models perform#

The most capable systems—Claude Fable 5, GPT-5.6 Sol, and Claude Opus 4.8—managed to finish the simulation with balances above the $1M starting capital. These agents deployed sophisticated tactics: they wrote code to forecast customer churn patterns, predict cash flow under different scenarios, and anticipate billing timing challenges. Yet even these top performers underperformed a simple rule-based baseline, suggesting the task exposes genuine gaps in how AI systems reason about cause-and-effect over time.

Why sustained agency matters#

The research frames CEO-Bench as a first step toward measuring the intelligence needed to drive adaptive progress across extended periods. The gap between isolated-task performance and multi-month strategic execution hints at a threshold current models have yet to cross—one that separates narrow competence from the kind of robust, long-horizon reasoning humans rely on in high-stakes roles.