New benchmark exposes language models' struggle with multi-part questions

A new evaluation benchmark called Compound-QA has uncovered a notable weakness in large language models: they struggle more with complex, multi-layered questions than with straightforward ones. The research team created the benchmark to assess how well LLMs handle real-world scenarios where questions are interconnected and require reasoning across multiple dimensions.

What is Compound-QA?#

Rather than test LLMs on isolated individual questions, the Compound-QA benchmark focuses on compound questions—prompts that integrate multiple related sub-questions into a single query. The benchmark was built by synthesizing questions from existing QA datasets, which were then annotated and human-verified for accuracy. It covers five distinct categories:

• Factual-Statement: questions requiring factual knowledge and assertion
• Cause-and-Effect: queries probing causal relationships
• Hypothetical-Analysis: scenarios requiring conditional reasoning
• Comparison-and-Selection: tasks involving evaluation and preference
• Evaluation-and-Suggestion: questions asking for judgment or recommendations

Key findings#

Testing nine open-source LLMs against Compound-QA revealed a consistent pattern: models performed substantially worse on compound questions compared to their accuracy on simpler, non-compound ones. The researchers evaluated model performance across three key dimensions—understanding, reasoning, and knowledge—finding deficiencies across all three.

Improving model performance#

The team explored various strategies to help LLMs better handle compound questions. Their interventions proved effective, demonstrating meaningful improvements in both comprehension and reasoning capabilities. This suggests that targeted approaches can help bridge the gap between current LLM abilities and the demands of genuinely complex reasoning tasks.

The work highlights an important gap between how LLMs are typically evaluated and the multifaceted nature of real-world question-answering scenarios.