New training approach improves AI code completion by fixing unrealistic data patterns

Large language models trained for code completion often struggle with how training data is prepared. A new approach called SynthCoder tackles this by improving the way code is masked during training, creating datasets that better reflect real-world programming scenarios.

The problem with current masking strategies#

Training fill-in-the-middle (FIM) code completion models requires selectively hiding portions of code so the model learns to fill gaps. However, standard approaches fall short. Random character-level masking can cut through keywords or variable names, creating nonsensical snippets that no developer would actually encounter. Meanwhile, syntax tree-based masking struggles with code elements that span multiple branches of the code structure, preventing the model from learning patterns that occur frequently in practice. These artificial patterns—or "anti-patterns"—weaken model performance and waste training resources.

How SynthCoder improves the approach#

The researchers refined masking at the syntax tree level and added heuristics that better align with how developers actually work. They then trained two model variants—SynthCoder-Seed (based on Seed-Coder-8B) and SynthCoder-Qwen (based on Qwen2.5-Coder-7B)—using a two-stage pipeline. The first stage uses curriculum-based fine-tuning, followed by preference optimization that explicitly teaches the models to avoid repeating code they've already seen.

Results and practical benefits#

Testing on multiple benchmarks—Santacoder-fim-task, aiXcoder-FIM-Evaluation, and CrossCodeEval—showed that SynthCoder outperformed baseline models on standard metrics like exact match accuracy. The approach also produces fewer instances of "code echo" (unintended repetition) and uses fewer tokens during inference, translating to faster and cheaper deployments in real-world applications.