AI
New Method Generates Custom Evaluation Criteria Tailored to Individual LLM Questions
Researchers introduce Qworld, a technique that builds question-specific scoring rubrics using recursive expansion to better capture what makes a high-quality AI response.
Evaluating how well large language models answer open-ended questions remains a stubborn challenge. Traditional assessment methods rely on fixed rubrics or generic scoring frameworks that don't account for how context and nuance shape what a good answer should contain.
A new research paper from arXiv introduces Qworld (One-Question-One-World), a framework designed to solve this problem by generating evaluation criteria tailored to the specific demands of each question. Rather than applying a one-size-fits-all rubric to all questions in a dataset, Qworld creates a custom set of scoring dimensions for every prompt.
How Qworld Works#
The approach uses a recursive expansion tree to break down each question into layers of context. Starting with a single question, Qworld decomposes it into different scenarios, perspectives, and specific binary criteria through both hierarchical and horizontal expansion. The resulting criteria articulate precisely what a strong answer must address given that question's particular constraints and assumptions.
Testing and Validation#
When tested on HealthBench, a medical question dataset, Qworld covered 89% of criteria that human experts had manually written. It also generated 79% novel criteria—scoring dimensions that experts hadn't anticipated but deemed valuable upon review. Experts judged Qworld-generated criteria superior in both insight and granularity compared to earlier automated approaches.
When applied to 11 frontier LLMs across HealthBench and Humanity's Last Exam, the method revealed capability gaps that coarser rubrics typically obscure. These include differences in long-term impact reasoning, equity considerations, error handling, and interdisciplinary thinking.
Why It Matters#
By tailoring evaluation criteria to individual questions rather than relying on static task-level measures, Qworld enables more precise assessment of LLM behavior and uncovers nuanced performance variations that traditional scoring hides.
