New Benchmark Reveals Trade-offs in AI Systems for Querying Business Data

Building reliable systems that let users interact with enterprise databases using everyday language remains a significant challenge. A new benchmark study examines how different architectural approaches handle this task—balancing the need for accuracy, safety, cost, and consistent behavior.

The Research Setup#

Researchers developed the SemPlan Benchmark to evaluate this design space systematically. The study used a synthetic dataset of 1,800 test cases in English and Brazilian Portuguese, with 1,200 of those cases reserved as a frozen scientific evaluation set. Four distinct architectures were tested under identical model configurations to ensure fair comparison.

The Four Approaches Tested#

The study compared:

• A1 (Direct SQL): Systems that generate SQL statements directly from natural-language queries
• A2 (Tool-agent baseline): A bounded agent that can call multiple tools to construct answers
• A3 (Structured semantic planning): A system that first generates formal semantic representations, then plans and executes deterministically
• A4 (Stateful clarification): An enhanced version of A3 that maintains state and can ask clarifying questions when requests are ambiguous

Performance Findings#

Accuracy rates across all approaches were surprisingly modest. Direct SQL generation achieved 22.25% correctness, the tool-agent baseline reached 22.58%, the structured semantic approach hit 25.67%, and the stateful variant managed 24.25%. On a smaller stability subset of 150 cases, repeatability—how consistently a system produces the same answer—ranged from 92% to 98.67%, suggesting the approaches are reasonably stable even when accuracy isn't high.

Beyond raw correctness, the architectures exhibited distinct characteristics. A1 had the lowest rate of unsafe or invalid outputs and the highest policy compliance. A3 showed the strongest overall correctness performance. A4 minimized false refusals (where the system declines to answer a valid question) and kept API costs lowest.

Key Insight: Trade-offs, Not Rankings#

The research reveals that adding more structural constraints doesn't automatically improve correctness or resolve deeper problems like ambiguous user requests or managing conversational state across multiple turns. Instead, each approach shifts where failures occur and changes operational characteristics like safety, cost, and helpfulness. This suggests practitioners will need to choose architectures based on which trade-offs matter most for their specific use cases.