AI
AWS Releases Open-Source Benchmark for Testing AI Agents on Real Cloud Tasks
AWS's new aws-bench tool evaluates how well AI agents handle practical cloud infrastructure work using real AWS resources, though questions linger about benchmark reliability.
InfoQ
AWS has unveiled aws-bench, an open-source evaluation tool that tests how effectively AI agents perform authentic cloud operations—from diagnosing infrastructure problems to provisioning resources and managing active environments.
How It Works#
Rather than relying on static test fixtures, aws-bench creates isolated AWS sandbox accounts and deploys real infrastructure through CDK stacks. Agents execute tasks in containerized environments with limited credentials, and results get validated either through automated LLM-based judgment or by comparing actual AWS state against expected outcomes.
The toolkit includes scenarios ranging from basic to advanced, covering observability, compute, data storage, databases, multi-region EC2 operations, serverless applications, streaming, IoT, and troubleshooting across multiple services. Built atop the Harbor evaluation framework, aws-bench adds AWS-specific features including resource provisioning and custom verification logic.
Who Can Use It#
The tool ships with built-in support for Claude Code, Codex, Kiro CLI, and Mini-SWE-Agent, plus any agent Harbor already supports. While AWS positioned it primarily for AI researchers and model developers, engineering teams can adapt the benchmarks to test against their own infrastructure patterns.
Notable Limitations#
Running aws-bench isn't straightforward—it requires credentials to manage organizational accounts and is currently restricted to the us-east-1 region. The setup provisions persistent resources that may generate unexpected costs. Crucially, AWS has not yet released baseline performance metrics or a standardized leaderboard, pushing both to the future roadmap.
Trustworthiness Questions#
The launch arrives amid growing concerns about benchmark reliability. Researchers at UC Berkeley's Center for Responsible, Decentralized Intelligence recently demonstrated how agents achieved near-perfect scores on prominent benchmarks like Terminal-Bench and SWE-Bench without actually solving tasks. They warned: "These are not isolated incidents. They are symptoms of a systemic problem: the benchmarks we rely on to measure AI capability are themselves vulnerable to the very capabilities they claim to measure."
