Clotho predicts LLM test failures from hidden states alone, slashing verification costs

Testing large language models on real-world tasks is expensive and difficult. Most prompts lack ground truth answers, leaving teams to rely on costly human judgment. Existing quality checks measure uncertainty in model outputs, but require running full inference first—a process that consumes significant computational resources. Researchers have now developed a different approach: Clotho, a technique that flags problematic test inputs before the model even generates a response.

How Clotho works#

Rather than analyzing what an LLM outputs, Clotho inspects the model's internal representations—its hidden states—to estimate how likely a given input is to cause failure. The method starts with a large, unlabeled set of test cases for a specific task. It then uses a statistical tool called a Gaussian Mixture Model (GMM) to intelligently select which inputs a human should label as reference examples. Once that reference set is in place, the same GMM can rank new, unseen inputs by their predicted failure risk, all without running inference.

Results across diverse tasks#

The team evaluated Clotho on eight benchmark tasks and three open-weight LLMs, achieving a ROC-AUC of 0.716 in predicting failures. Remarkably, this performance came after labeling just 5.4% of inputs on average—far more efficient than labeling entire test suites. More importantly, skipping inference altogether reduced computational overhead substantially compared to methods that measure uncertainty post-generation.

Crucially, adequacy scores learned from open-weight models transferred effectively to proprietary systems. When applied to commercial LLMs, using Clotho to prioritize test cases increased the share of caught failures from 18.7 to 42.5 per 100 inputs compared to random testing.

Complementary approaches#

The researchers found that Clotho and output-based confidence measures provide different signals: pre-generation and post-generation approaches identify different failure patterns, suggesting they work best together in practice. This opens a path toward hybrid testing strategies that leverage both methods.