AI
Researchers Introduce "Complexity Kink" to Predict Code-Generation Failures Before LLM Runs
A new six-dimension index scores prompt difficulty upfront—before generation—to identify when large language models are likely to struggle with code tasks.
A team of researchers has proposed a novel framework for spotting risky code-generation tasks before an LLM attempts them, introducing what they call the "Complexity Kink"—a pre-generation scoring system that measures structural difficulty in the prompt itself rather than relying on the success or failure of the output.
Why Measuring Code Difficulty Is Tricky#
The standard approach to assessing code-generation risk has a fundamental flaw. When researchers examine the complexity of generated code to gauge task difficulty, they conflate the inherent challenge of the problem with whether the model solved it. A devilishly hard prompt might produce a short, broken program—which would be assigned artificially low output complexity—or no program at all. This mismatch makes it impossible to predict which tasks will stump an LLM before it starts generating.
The Complexity Index#
The researchers developed a six-dimension structural-complexity index that evaluates prompts independently of generation outcomes. They curated a dataset of 5,000 Python prompts distributed across difficulty bands, then had four expert LLM raters rescore locked prompts, generating nearly 20,000 score entries. Inter-rater agreement was strong (composite ICC = 0.872 on prompts rated by all four judges).
The team evaluated 21 different models on each prompt—105,000 total generations—to understand how the index correlates with pass rates.
The Breakpoint Finding#
The central finding reveals a sharp inflection point, or "kink." In raw analysis, models achieved a 79.9% pass rate on prompts scoring 13.75 or lower on the composite index, but that jumped to 87.6% for prompts above that threshold. However, this relationship isn't universal. When the researchers introduced task-type fixed effects, the breakpoint shifted to 10.75 and the performance gap narrowed to just 2.1 points, suggesting the index captures task structure that varies significantly by category.
Adding construction-frame controls (accounting for how prompts are phrased) moved the breakpoint again to 8.50, with a negative gap of 3.5 points—yet neither framing adjustment alone explained the full 7.6-point shift observed in the pooled data. Model-specific analyses revealed 16 upward adjustments and five downward ones, underscoring variation across LLM architectures.
