AI & Machine Learning
Spaghetti Architect tool generates contamination-free code datasets with controlled complexity
A new system automatically creates verified code datasets with tunable difficulty levels and resistance to training-set contamination.
Training data for code-generating AI systems faces a persistent problem: real-world code repositories are messy, unverified, and their difficulty is unknowable—plus samples may already lurk in a model's existing training set, poisoning benchmark results. Researchers have introduced Spaghetti Architect, a tool designed to mint synthetic code datasets with deliberate control over all these dimensions.
How It Works#
The system starts with a clean, language-independent JSON intermediate representation of a program's logic. An anti-optimization transpiler then converts this into deliberately bloated, flattened code across five languages—Python, JavaScript, Go, Java, and C++. Every generated program is compiled, executed, and validated against a reference oracle, ensuring correctness by design. This approach yields a known-optimal baseline (the clean IR) against which to measure every variant.
Controlling Messiness and Difficulty#
The toolkit introduces separate knobs for controlling code quality and challenge level. Messiness is dialed through strictly-nested anti-pattern profiles—systematic ways to make code redundant and hard to follow while preserving semantics. Difficulty is split into two orthogonal axes: intrinsic difficulty (driven by problem size) and incidental difficulty (presentation style at fixed logic). Since each variant is minted fresh from a private seed, contamination is resisted naturally—there is no fixed dataset that could leak into pre-training corpora.
Evidence of Quality#
Testing across four open-source models shows that the framework moves established complexity and readability metrics in expected directions. Exact-match accuracy rises with problem scale. The intrinsic difficulty knob is potent: it collapses arithmetic-aggregation accuracy of even the strongest model to zero. Development-set scores match freshly re-minted held-out variants within tight bounds (±0.012 for comprehension, ±0.011 for refactoring tasks). On identical programs, refactoring equivalence shifts from 0.73 to 0.99 independent of scale, while output prediction collapses. Ablation studies reveal that the generator's built-in annotations inflate performance of weaker models by an order of magnitude more than stronger ones (−0.173 vs −0.017), suggesting the annotations have outsized impact on poorly-fit systems.
