AI
New Benchmark Reveals LLMs Translate Code Correctly but Inefficiently
Researchers found that large language models produce functionally correct translated code but often miss crucial performance optimizations, especially when converting between languages.
Execution speed has been largely ignored in assessing how well large language models translate code from one programming language to another—until now. A new research benchmark called TRACE fills that gap by systematically testing whether LLM-generated code not only works correctly but runs efficiently.
The Gap Between Correctness and Performance#
Researchers evaluated 28 prominent language models using 1,000 benchmark tasks designed to stress-test the efficiency of translated code across C++, Java, and Python. The findings challenge a common assumption: models that produce functionally correct code don't necessarily generate fast code. Claude-Sonnet-4-Think, which ranked first in correctness, achieved only middling performance when judged by execution speed—even losing to smaller open-source models like Qwen2.5-Coder-14B-Instruct.
Where Inefficiency Originates#
The analysis traced the root causes of slow translated code to three main sources:
- Algorithm implementation gaps (11.9%)—translated code uses fundamentally different computational approaches than optimal solutions
- Language construct mismatches (66.4%)—code relies on language-specific features that don't translate efficiently to the target language
- Resource management problems (21.7%)—poor handling of memory, caching, or other system resources
Overall, nearly a quarter (23.5%) of translations that produced correct results suffered from notable efficiency issues.
Prompting Alone Falls Short#
The researchers tested whether better prompts at inference time could nudge models toward more efficient translations. The answer was discouraging: simple prompting strategies yielded only marginal gains, suggesting the problem runs deeper than how models are instructed.
Why It Matters#
TRACE establishes execution efficiency as a missing but essential yardstick for code translation. As businesses rely increasingly on LLMs to modernize legacy codebases and port systems across languages, overlooking performance could leave translated applications running substantially slower than necessary—a costly gap that functional correctness alone cannot fill.
