How Retail Firms Cut LLM Hallucinations from 15% to 1.5% Without Better Models

By wrapping production large language models in a dedicated platform layer, a retail company reduced the hallucination rate of its AI-driven inventory system from 15% to 1.5%—not through better foundation models, but through architectural discipline. The shift from treating LLMs as application concerns to infrastructure concerns proved decisive.

The Real Problem at Scale#

When the retailer first deployed a multi-agent LLM system to analyze inventory discrepancies across millions of product SKUs, early prototypes worked well. But production brought three systematic failures that no evaluation caught: API throttling from model providers under real load, garbage-in-garbage-out from poor data pipelines feeding the model, and hallucinations—low-probability individually but certain in aggregate across millions of requests.

Hallucinations posed a unique problem. Unlike a stack trace, a confident-sounding wrong answer passes all traditional health checks. Downstream systems consume it, act on it, and only later does someone realize the recommended item doesn't exist. Without platform-level visibility, each team shipping an LLM feature independently discovered and duplicated the same scaffolding to track and fix these issues.

Why Platform > Application Layer#

Three cross-cutting concerns emerged:

• Hallucinations as silent failures. Standard monitoring misses them because they look syntactically valid.
• Uncontrolled token spend. Without per-request attribution, monthly bills arrived as single line items, making optimization guesswork.
• Lost behavioral observability. Latency and error rates looked fine while output quality drifted lower.

Centralizing these concerns mirrors the historical case for shared authentication, logging, or service mesh infrastructure: once more than one team hits the same problem, duplication's cost exceeds the cost of building shared infrastructure.

The Architectural Approach#

The platform sits between callers and foundation models through a single gateway. A root coordinator agent classifies incoming intent and delegates to specialist agents, each bound to its own schema and prompt registry. Specialist agents reach tools through Model Context Protocol (MCP) servers that enforce authorization at the resource level before any tool executes.

The stack is deliberately built with no hidden abstractions—orchestration via , model access through (swappable between local and hosted models), and tools via servers. The companion implements this end-to-end using a local model () for zero API cost.