AI
New Training Method Teaches Code Models to Optimize for Speed, Not Just Correctness
A reinforcement learning approach rewards code agents for generating both correct and efficient solutions, closing a gap in how AI systems learn to write software.
Current code generation models focus on producing correct solutions, but a new technique aims to push them further by explicitly rewarding speed and efficiency alongside accuracy.
The Problem with Runtime as a Training Signal#
Traditional approaches to training code models use execution feedback—checking whether a program runs and produces the right output. However, this misses a crucial dimension: two programs can both pass all tests while one runs significantly faster than the other. For systems code and performance-critical applications, this efficiency gap matters. Yet teaching models to optimize for runtime has proven difficult, because execution speed only becomes meaningful after a program compiles and runs correctly. When most sampled solutions fail to compile, raw runtime metrics offer little guidance.
How RLPF Works#
Researchers at arXiv propose RLPF (Reinforcement Learning from Performance Feedback), which restructures execution outcomes into a staged reward system. The approach separates feedback into two phases:
- For failing programs: Order them by how far they progress during execution (e.g., compilation success, partial test passes), providing useful learning signals before correctness is achieved
- For correct programs: Rank them by relative improvement compared to a baseline solution, allowing the model to learn performance-sensitive distinctions among working code
This composite reward gives guidance earlier in the training process while still prioritizing optimization once solutions are correct.
Measurable Improvements#
Fine-tuning Qwen3-32B with RLPF on the PerfCodeBench benchmark showed significant gains. The percentage of correct, runnable solutions jumped from 11.1% to 54.6%, and relative efficiency improved from 8.1% to 38.6%. The resulting model became competitive with stronger open-weight systems and demonstrated some ability to transfer its optimization behavior to a separate benchmark (EffiBench-X).
Additional experiments revealed that model-generated references can provide useful supervision, though weaker than expert references, and that the full composite reward proved more reliable than training on correctness or runtime alone.
