Structured Planning Harnesses LLM Agents for Predictable Behavior in Regulated Industries

Language model–based agents can produce wildly different outputs for the same task, a critical weakness in fields like finance and legal work where consistency matters. Researchers conducted an empirical study on methods to lock down this variability by surrounding agents with what they call "harness engineering"—a deterministic execution layer that constrains how agents operate.

What Is Harness Engineering?#

The technique wraps an LLM agent in multiple control mechanisms: finite-state workflows, forced selection of specific tools, output validation checks, bounded retries, and structured planning steps. These constraints force the agent to follow a more predictable path through execution rather than improvising at each decision point.

Key Findings from the Study#

The team tested their approach on two synthetic scenarios (finance and legal tasks) using two open-weight models (Qwen-2.5-7B-Instruct and Gemma-3-27B). Initial results were mixed: a basic harness improved consistency in one scenario but actually made things worse in two others, with no change in the fourth.

The breakthrough came by adding Structured Planning—requiring the agent to validate its entire plan against a fixed schema before executing any tools. This single change proved transformative:

• Three of four model-task combinations achieved perfect reproducibility and determinism (both at 1.000) across 100 runs
• Task success rates climbed to 100% in three of four cells
• Token costs decreased in every scenario tested

The Cost Trade-Off#

The gains came with a caveat: latency responses diverged sharply by model. One model ran faster under the constraints, while the other experienced substantial slowdowns. The researchers emphasize that this cost is model-dependent and must be measured for each deployment rather than assumed to be universally positive or negative.

Why It Matters#

The work establishes harness engineering as a distinct, measurable discipline for building reliable agents—essential for regulated industries where unpredictable behavior poses compliance and safety risks. However, organizations cannot simply apply the technique without evaluating its specific performance impact on their chosen models.