Google Tests Gemini Without Peeking at Questions in First Double-Blind AI Evaluation

As AI models grow larger and benchmarks proliferate, determining whether a system has genuinely learned new capabilities or merely absorbed test data during training has become increasingly difficult. Google DeepMind now proposes a solution: a double-blind evaluation framework that keeps Gemini's weights hidden from testers while the test prompts stay hidden from Google.

The benchmarking problem#

Benchmark contamination—when training data leaks into test sets—has become a widespread concern. Research cited by Google found signs of this leakage in roughly half of 31 models studied, and newer work shows contamination can artificially inflate scores, especially for larger systems. This dynamic undermines confidence in claimed performance gains and has already raised doubts about the legitimacy of some post-training improvements.

Simply keeping benchmarks private seems like an obvious fix, but closed commercial models create a catch-22: evaluators typically can't run tests independently without the vendor sharing proprietary weights, yet sending queries through the provider's API means Google itself sees questions meant to be secret. The double-blind approach sidesteps this dilemma.

How the enclave protects both sides#

Google's pilot paired Gemini 2.5 Flash Lite against private benchmarks from MLCommons and Singapore's AI Safety Institute using Google Cloud Confidential Space, an NVIDIA H100 Confidential GPU, and Intel TDX memory encryption.

The mechanism works like this: Google provides the model weights and inference code; the evaluator supplies test prompts and grading logic. Both are transmitted over encrypted channels into an isolated enclave, where evaluation proceeds with neither party able to access the other's assets. Model weights live in encrypted GPU memory; test questions stay in encrypted host RAM. Once testing concludes, only approved results are released and the temporary environment is destroyed.

Before exchanging sensitive data, remote attestation verifies that both sides are running mutually agreed-upon software.

Securing the code itself#

Hardware encryption alone isn't enough. An application inside the enclave with unrestricted network access could leak data elsewhere. To prevent this, the researchers employed from OpenMined, a tool that lets both parties approve code in advance and block sensitive portions from making external connections.