Akka's AI-Powered Porting Experiment: Smaller Models Outperform Larger Ones on Spec-Driven Code Migration

Akka recently conducted a large-scale experiment in AI-assisted software porting using a structured, specification-driven workflow. The company deployed its approach across 65 open-source projects, measuring how well AI could handle code migration when given clear specifications and systematic validation steps.

The Experiment Design#

Akka's testing unfolded in two phases. In the first phase, the team analyzed all 65 projects by generating detailed specifications and implementing partial ports (up to 10% of each project's codebase). Based on measurable results and system characteristics, Akka selected 10 projects for complete implementation in the second phase.

The workflow relied on a structured delivery harness that cycled through discovery, specification, porting, benchmarking, and refinement. Initial code analysis examined existing patterns, schemas, and runtime behavior. Then Claude models paired with Akka Specify handled the actual porting, testing, and code review. A uniform benchmark runner tracked test results, code metrics, and latency across all ports.

Model Performance and Efficiency#

The results revealed a surprising finding: smaller models worked better than larger ones. Claude Sonnet averaged 61 minutes per port compared to 120 minutes for Claude Opus, while Opus consumed about 40% fewer tokens overall. This tradeoff—speed over token economy—favored Sonnet for practical deployment.

Interestingly, increasing effort settings consumed more tokens without yielding consistent efficiency gains, suggesting that specification quality matters more than raw computational intensity. One engineer observing the research noted that Sonnet's behavior aligned with practical modernization work: smaller models tend to follow specifications closely, whereas larger models may attempt creative improvisation that deviates from the intended outcome.

Outcomes and Limitations#

The initial phase consumed 99.3 hours and 9.41 billion tokens. Akka reported improvements in lines of code or performance across 57 of 65 ports. However, performance gains varied by project type: applications, frameworks, and libraries generally showed improvement, while infrastructure and tooling components experienced median degradation. A notable outlier was Dify, which saw a 143,333-fold speed improvement, though the comparison involved different workloads—Netflix Metaflow, by contrast, performed approximately 100 times slower after porting.