Foundation models show promise detecting hidden compilation errors in configurable C code

Configurable software systems often hide compilation errors in feature combinations that remain untested—a problem that grows as systems become more complex. Researchers have now evaluated how foundation models like GPT and Gemini can detect and repair these elusive build failures in C code.

The research approach#

The empirical study took two forms. In the first phase, researchers generated 5,000 synthetic code snippets from 30 seed examples, then compiled each one under every possible Boolean feature assignment to create a ground-truth dataset of which combinations actually broke. They tested GPT-OSS-20B, compared it against the specialized TypeChef tool, and also evaluated Gemini 3.6 Flash on a representative sample.

In the second phase, the team examined Codex-GPT5.5 on 100 real compilation failures from five mature open-source projects, checking whether the model could pinpoint known historical bugs and suggest appropriate fixes.

Detection and repair performance#

Results were mixed but encouraging. GPT-OSS-20B achieved 84.7% precision when identifying affected configurations, though recall dropped to 52.1%—meaning it missed roughly half the failing cases. The model successfully restored compilability in 1,930 of 2,665 faulty snippets (72.4%). Notably, Gemini 3.6 Flash performed better on the sampled test set, fixing 95.8% of the problems it examined.

How the model reported results mattered significantly. When asked to list all failing configurations, it achieved 99.4% coverage; when asked only for a minimal justifiable set, coverage plummeted to 29.5%, suggesting the phrasing of prompts shapes output quality.

Real-world applicability and limits#

On the real-world dataset, Codex identified relevant compilation problems in 94 of 100 cases and correctly flagged four of five historical bugs in the test set. However, the researchers cautioned that compiler acceptance does not guarantee semantic correctness—a model may generate syntactically valid code that compiles but behaves incorrectly.

The findings suggest foundation models serve best as a complement to, not a replacement for, compiler-based and variability-aware analysis tools. They can help localize errors, explain problems, and prioritize which issues to address first, but developers should integrate them within a broader quality assurance workflow rather than rely on them exclusively.