Google's Gemini Agent Platform adds production-ready evaluation tools for AI agents

Google has made agent and model evaluations generally available within its Gemini Enterprise Agent Platform, providing developers with tools to assess agent performance both before and after deployment using unified metrics.

Comprehensive Metrics Framework#

The evaluation system ships with over 20 pre-built metrics covering quality, safety, grounding, agent tool use, and trajectory. Developers can apply reference-based scoring for tasks like summarization and translation, then customize evaluation criteria to fit specific use cases rather than applying a single rigid assessment approach across all inputs.

Beyond pre-built options, teams can create code-based metrics using Python functions for exact text matches and schema validation, or register custom LLM-as-a-judge metrics in a versioned, organization-wide registry. This ensures scoring remains consistent and comparable across time.

An adaptive rubric — developed with Google DeepMind — represents an advanced LLM-judge workflow that generates evaluation criteria tailored to each test case. It evaluates task success (goal fulfillment), tool use quality (selection and argument correctness), safety (against content policies), trajectory quality, hallucination, and grounding, among other dimensions. Natural-language guidelines steer rubric generation toward specific criteria teams define.

Offline and Online Evaluation#

Developers can run experiments locally for rapid iteration or against live production traffic for continuous assessment. The service supports:

• Experiments: Execute test cases client-side or server-side; server-side runs store all artifacts in Cloud Storage for auditable, reproducible results
• Case generation: Automatically seed synthetic test cases from agent instructions and tools
• User simulator: Define personas and conversation plans to reach multi-turn scenarios without scripting each response
• Environment simulator: Mock failing or slow backends to test agent resilience without affecting production

For live operations, an online monitor grades production traces as they arrive. Teams can set up sampling and targeted filters to keep costs manageable, then visualize scores over time and configure drift alerts through email, Slack, or other channels.