Evolution Strategies Emerge as Powerful Alternative to Reinforcement Learning for LLM Fine-Tuning

Researchers have demonstrated that evolution strategies (ES), long dismissed as unscalable, can successfully fine-tune large language models at the billion-parameter scale. The finding challenges the dominance of reinforcement learning (RL) in post-training, suggesting a viable backpropagation-free path for optimizing modern AI systems.

The ES Scalability Question#

For years, the machine learning community assumed evolution strategies couldn't handle the enormous parameter spaces of today's language models. This belief made RL the default approach for most LLM fine-tuning work. The new research upends that assumption, showing ES can perform full-parameter optimization without needing to compress or reduce dimensionality of the search space.

Comparative Advantages#

Beyond merely matching RL performance, ES demonstrated several practical benefits:

• Better long-horizon reasoning: ES shows improved tolerance for delayed and long-horizon reward signals, useful for complex reasoning tasks
• Broader base model compatibility: The approach proves robust across different starting LLMs, not just a narrow set
• Reduced reward gaming: ES exhibits less susceptibility to reward hacking—a key problem where models exploit reward signals in unintended ways
• Stability gains: Training remains more stable throughout the optimization process

A Different Paradigm#

Unlike RL approaches that rely on gradient-based learning, ES operates as a derivative-free optimization method. This architectural difference may explain why it sidesteps some classic failure modes in RL-based fine-tuning. The researchers frame ES not as merely another tool, but as a fundamentally distinct post-training framework that opens new directions for LLM development beyond current RL-dominant practices.