New hybrid training method unifies supervised learning and reinforcement learning for vision-language AI

Researchers have introduced S-GRPO, a unified post-training framework that overcomes longstanding limitations in how large vision-language models are adapted to new tasks. The approach combines the precision of supervised learning with the flexibility of reinforcement learning.

The Problem with Current Methods#

Existing post-training approaches for large vision-language models (LVLMs) rely on two separate strategies, each with significant drawbacks. Supervised fine-tuning (SFT) constrains the model to follow a single correct path, but this narrow focus can erase the model's pre-existing multimodal knowledge through what researchers call catastrophic forgetting. Meanwhile, reinforcement learning (RL) allows the model to explore multiple solution paths but frequently gets stuck in an unproductive state—a "cold-start problem" where an unaligned model never generates valid outputs in the first place, especially for visual tasks with sparse rewards.

How S-GRPO Works#

The new method, formally called Supervised Group Relative Policy Optimization, integrates expert guidance from imitation learning into a broader exploration framework. A key innovation is Conditional Ground-Truth Trajectory Injection (CGI): when the system detects complete failure across a batch of generated samples, it explicitly inserts the correct reference solution into the candidate set and assigns it the maximum possible reward. This anchor point gives the model a guaranteed positive learning signal within group comparisons.

By reformulating supervised learning as a high-reward component within the policy gradient framework, S-GRPO allows the model to dynamically decide when to exploit the expert trajectory and when to explore new approaches.

Results and Impact#

Theoretical analysis and experiments indicate that S-GRPO substantially accelerates training convergence, achieves stronger performance on domain-specific tasks, and crucially—preserves the general-purpose capabilities of the base model. The framework is specifically designed for direct-generation visual tasks, where older methods struggle most.