LLM-based framework CIgrate dramatically outperforms rule-based approach for automating CI service migration

Development teams frequently need to move their continuous integration (CI) configurations from one service to another as projects mature or services change. This migration work—traditionally handled manually—is laborious and introduces opportunities for mistakes. A new research paper from arXiv examines whether large language models can automate this process more effectively than existing solutions.

The Problem with Current Approaches#

Migrating CI setups between platforms (such as moving from Travis CI to GitHub Actions) has become increasingly common in open-source development. The leading tool in this space, CIMig, uses a rule-based strategy: it studies past migration examples to derive service-specific conversion rules, then applies those rules to new configurations. However, CIMig's performance—achieving just 0.49 Cosine Similarity for Travis-to-GitHub Actions transitions—raises serious questions about whether rule-based systems alone can handle the complexity of these transformations reliably.

CIgrate: An LLM-powered Solution#

The study introduces CIgrate, a framework that leverages large language models to handle CI migration instead. The researchers tested two deployment strategies: zero-shot and few-shot prompting, as well as fine-tuning on an established collection of successful CI migrations. They then benchmarked the results against CIMig and even tested the tool in real-world scenarios by submitting pull requests to active open-source projects.

Dramatic Performance Gains#

The results demonstrate a substantial leap in capability. A fine-tuned Gemma 3 12B model achieved 0.90 Cosine Similarity and 0.74 CrystalBLEU scores for Travis CI–to–GitHub Actions migration—improvements of 82.2% and 295.5% respectively over CIMig. Notably, CIgrate produced valid, immediately parseable YAML in 100% of cases, whereas CIMig managed this in only 5.6% of instances. Even when used without fine-tuning, the LLM-based approach outpaced the rule-based baseline, suggesting that the generalization capabilities of language models are well-suited to this challenge.

Why This Matters#

The findings indicate that LLM-driven approaches offer a more practical, accurate, and generalizable path forward for automating CI configuration migration. By reducing manual effort and error rates, such tools could accelerate software development workflows and allow teams to adopt new CI platforms with confidence. The research paper is available on .