DeCRIM method helps open-source LLMs beat GPT-4 at following complex instructions

Large language models struggle with instructions that layer multiple constraints together—like writing a funny social media post without hashtags—even when those constraints individually are straightforward. A new study published on arXiv reveals just how widespread this problem is and proposes a solution that can make open-source models competitive with GPT-4.

The challenge: Real-world instructions aren't simple#

Researchers created RealInstruct, the first benchmark designed to test how well models follow actual multi-constraint instructions sourced from real user queries. Their testing exposed a sobering reality: GPT-4, the flagship proprietary model, fails to satisfy at least one constraint on over 21% of instructions. Open-source models perform even worse. The gap highlights that current LLMs, despite their impressive general capabilities, lack robust mechanisms for juggling overlapping requirements.

The study also investigated whether automated model-based evaluation could replace costly human annotation for this task—a practical consideration for scaling up assessment methods.

DeCRIM: A three-step self-correction approach#

To address this limitation, the team introduced Decompose, Critique, and Refine (DeCRIM), a self-correction pipeline that enhances how models follow constraints. The method works in three stages:

• Decompose: Break the original instruction into its individual constraints so the model can reason about each one explicitly
• Critique: Deploy a separate critic model that evaluates whether the initial response satisfies all constraints and identifies which ones need work
• Refine: Use the critic's feedback to improve the response

Results: Measurable gains across the board#

When applied to Mistral, an open-source model, DeCRIM improved performance by 7.3% on RealInstruct and 8.0% on IFEval—another instruction-following benchmark—even when the critic feedback was relatively weak. Most strikingly, with strong feedback, open-source LLMs enhanced by DeCRIM matched or exceeded GPT-4's accuracy on both benchmarks. This suggests that better self-correction mechanisms may be more valuable than raw model scale.