New method evaluates AI chatbots by searching the web for facts instead of comparing to preset answers

Researchers have developed a new approach to evaluating how accurately large language models answer questions by having them search the internet for supporting evidence, rather than comparing outputs against pre-written answers.

The Problem with Current Evaluation Methods#

Traditional evaluation of AI question-answering systems relies on comparing model responses to fixed reference answers compiled and annotated by humans. This approach creates significant bottlenecks: it becomes expensive to scale, incomplete for coverage, and difficult to verify facts beyond the training data that LLMs were built on. Simply using an AI model itself as a judge—asking it to rate another model's answers—proves unreliable, as these systems tend to wrongly accept incorrect answers and make up supporting reasons that sound plausible but lack grounding in reality.

How SAGE Works#

The new framework, called Search-AuGmented Evaluation (SAGE), reimagines this process by acting like an investigative agent. When assessing an LLM's answer, SAGE doesn't consult a static dataset. Instead, it:

• Generates web queries based on the question and the model's response
• Collects information from external sources through search
• Summarizes findings to assess factual accuracy
• Refines searches iteratively by reflecting on what it has learned

This multi-step process allows SAGE to ground its evaluation in real-world information rather than relying solely on what the evaluating model knows from training.

Testing and Results#

When tested on multiple free-form question-answering benchmarks, SAGE demonstrated strong alignment with human judgments—achieving substantial to near-perfect agreement. The approach proves particularly effective for questions that extend beyond typical training data, where conventional evaluation methods struggle.