Do Vendor AI Coding Harnesses Actually Outperform Neutral Frameworks?

Vendors typically pair their language models with custom-built harnesses—the scaffolding of tools, prompts, and control logic that transform a chat model into an autonomous software engineer. The conventional wisdom assumes that these vendor-tuned pairings deliver superior results. A new research paper challenges that assumption by directly measuring the harness effect in isolation.

The Experiment#

Researchers conducted a rigorous paired comparison on a private test suite of 256 software engineering tasks drawn from repositories and post-cutoff programming contests. They ran identical tasks using two different harnesses paired with the same models:

• Claude Opus 4.8: tested with claude-agent-sdk (vendor native) and deepagents (neutral framework)
• GPT-5.5: tested with the openai-codex SDK (vendor native) and deepagents
• Additional models: Gemini 3.5-Flash and Deepseek v3.2 included for comparative context

An isolated grading oracle evaluated 792 of the 800 planned runs.

Surprising Results on Overall Performance#

The headline finding: neither harness demonstrated a consistent advantage. For Opus 4.8, the vendor-native harness actually trailed by 1.25 percentage points (48.8% vs 50.0% task success rate), though the 95% confidence interval [-10.0, +7.5] makes this difference statistically inconclusive. On GPT-5.5, the vendor harness gained 1.25 percentage points (55.6% vs 54.4%), yet again within margin of error (CI [-4.4, +6.9]).

However, the data masked divergent patterns depending on task type. For Opus 4.8, the vendor harness significantly underperformed on repository tasks (trailing by 9.0 percentage points across 61 tasks) but substantially outperformed on programming contests (leading by 23.7 percentage points on 19 tasks, with p = 0.003). The researchers note this partition emerged after examining the results and will require deliberate replication to confirm.

Completion and Cost Trade-offs#

Beyond raw task completion, the study uncovered other trade-offs. Twenty-two of 81 runs that hit the wall-clock time limit had nonetheless produced a working patch before timing out, suggesting some frameworks achieve solutions but incomplete delivery within the deadline.

When repricied at frozen vendor list prices, the neutral framework consumed substantially more compute: . The authors flag critical uncertainty here—58 runs on the Anthropic account generated no usage record, meaning the true cost ratio could range from 0.7 to 2.3, leaving the billing advantage unresolved.