When prompt constraints stack up: Study reveals LLMs can fail spectacularly under combined restrictions

Researchers have uncovered a significant vulnerability in how large language models handle code generation when prompted with multiple constraints simultaneously—a common scenario in real-world engineering workflows.

The compound constraint problem#

In software development pipelines, engineers often craft prompts that layer multiple demands: specifying output format (JSON, XML), assigning a role ("expert developer"), and signaling time pressure ("urgent"). A new study reveals that when these constraints combine, the results can be dramatically worse than what individual factors would suggest. The degradation doesn't add up—it multiplies.

Experimental design and scope#

Researchers conducted a full 3×3×3 factorial study testing all 27 possible combinations of three constraints: format (JSON, XML, plain text), persona (generic, expert, senior lead), and urgency framing (normal, moderate, high). They evaluated these against all 164 problems in the HumanEval+ benchmark using five different OpenAI models, generating over 22,000 code-generation attempts with careful tracking of whether failures stemmed from formatting issues or reasoning lapses.

Architecture matters more than model size#

The findings revealed striking disparities across model families:

• GPT-4o models showed consistent super-additive degradation, with performance drops 3–12 percentage points steeper than predicted. The worst case: combining JSON format with expert persona and moderate urgency on GPT-4o-mini dropped pass rates by 12.2 percentage points beyond additive predictions.
• GPT-4.1 models proved largely resistant to compound-constraint interference.
• o3-mini displayed an unexpected pattern: structured output requirements sometimes improved performance rather than harming it.

These patterns indicate that vulnerability is tied to model architecture rather than overall model capacity.

Why it matters#

Individual constraints—asking for JSON output, playing a role, or acknowledging urgency—often have neutral or even small positive effects on their own. Yet paired or tripled together, they can trigger substantial failures. The study argues this interaction effect should become a standard part of reliability testing for any LLM system used in production engineering workflows, since real-world prompts almost never consist of a single constraint alone.