Gradient Signals Mislead LoRA Rank Tuning in Reinforcement Learning

Researchers investigating parameter-efficient fine-tuning for large language models have uncovered a significant limitation: strategies that boost efficiency during supervised training don't translate reliably to reinforcement learning workflows.

The team tested adaptive rank allocation for Low-Rank Adaptation (LoRA)—a method that assigns more capacity to "important" layers and less to "unimportant" ones—on Qwen 2.5 1.5B using the GSM8K benchmark. Under Group Relative Policy Optimization (GRPO), this approach backfired: models tuned with rank allocation based on gradient magnitude achieved 70% accuracy, compared to 74.5% for uniform allocation across the same parameter budget.

Why Gradient Signals Fail Under RL#

The failure stems from two interrelated mechanisms. First, the gradient landscape in GRPO is substantially flatter than in supervised fine-tuning. While prior work showed that SFT concentrates gradient activity into a small subset of layers (with a max-to-min importance ratio exceeding 10x), GRPO distributes gradients more evenly across all layers. The importance spread under GRPO is only 2.17x, meaning every layer carries meaningful learning signal—no layer is truly idle.

This flatness creates a problem: with gradient importance varying so little, allocating capacity based on measured importance becomes unreliable as a proxy for where the model actually needs parameters.

Rank Allocation Amplifies Its Own Errors#

More concerning, non-uniform rank allocation appears to create a self-reinforcing cycle. The researchers discovered a gradient amplification effect: when capacity was unequally distributed based on initial importance rankings, the importance gap widened from 2.17x to 3.00x. High-rank layers absorbed progressively more gradient signal while low-rank layers were silenced.

To test causality, the team ran a control using random rank assignments. The results were striking: even random allocation produced the same amplification pattern (r=0.972 correlation between assigned rank and resulting gradient share), indicating that the rank distribution itself—not the quality of the importance measurement—drives the outcome. This suggests that capacity allocation actively shapes the gradient landscape rather than merely capturing pre-existing structure.

Implications for Alignment Training#

The authors caution that techniques proven effective for supervised fine-tuning should not be assumed to work for alignment-oriented training using reinforcement learning without explicit validation. Their work is presented as preliminary—based on a single seed and task—but highlights a fundamental mismatch between SFT and RL optimization dynamics that warrants careful evaluation in future applications.