Continuing existing language models beats retraining from scratch for software engineering tasks

Researchers have investigated whether the most popular large language models can be effectively adapted to understand software engineering artifacts—such as issue trackers, commit messages, and developer conversations. The findings suggest that the common assumption about building specialized models may be backwards.

The question#

Language models designed for general text or code are now routinely used in software engineering tasks, but little is known about whether these tools are actually optimized for the domain-specific writing that engineers produce. Most evaluation work has focused on code itself, leaving a gap in understanding how well these models handle the diverse written communication that surrounds software projects.

What the study tested#

Researchers compared two strategies for adapting language models to software engineering text. Continual pre-training (CPT) took an existing model and fine-tuned it further on engineering-specific material. Pre-training from scratch (PTS) built entirely new models using the same software engineering corpus, starting with no prior knowledge.

To make the comparison fair, the team kept computational budgets and token counts constant across both approaches. They tested both encoder models (which understand text) and decoder models (which generate text), in multiple sizes, and measured success on two fronts: how well models performed on software engineering tasks (SELU benchmark) and how they performed on general language understanding (SuperGLUE benchmark).

Key findings#

Across all configurations tested, continuing with an existing model significantly outperformed retraining from scratch. Models built via CPT showed modest but meaningful gains on software engineering tasks while maintaining general language skills. By contrast, training new models from scratch consistently lost ground on both measures—and became competitive only when working with smaller models that had access to a much larger supply of tokens.

The implication is stark: the benefits of domain-specific pretraining are far smaller than the cost of discarding all the general knowledge already encoded in established models.

Why it matters#

The findings offer practical guidance for practitioners and researchers adapting language models to specialized domains. Rather than investing in expensive from-scratch training pipelines, engineers should focus on carefully adapting existing models, even if the domain-specific boost is incremental. The research team released their software engineering corpus and trained models openly to support further work in this area.