ReSPO: Tackling Gradient Starvation in Off-Policy Learning through Reshaped Sequence Policy Optimization

Meta description: ReSPO targets off-policy RL under high rollout staleness in large MoE models by reshaping sequence-level importance weights for stable scaling.
Tags: ReSPO, Off-Policy RL, MoE, Policy Optimization, Rollout Staleness
Featured image: abstract weighted trajectory arcs over a neural network grid in deep blue and cyan, no people, no logos, no text

ReSPO is a new policy-optimization method designed for off-policy reinforcement learning (RL) under high rollout staleness, a setting described as particularly difficult for large MoE (mixture-of-experts) models. It reshapes sequence-level importance weights to make training updates more stable when policies learn from rollouts produced by earlier model versions.

Background: Why High Rollout Staleness Matters

Off-policy RL reuses trajectories from earlier versions of a policy. That can save compute, but it introduces a mismatch between the data used for updates and the current model. Sequence-level importance weights are used to correct that mismatch. When rollouts are highly stale, those weights can become unstable, causing useful learning signals to be drowned out by extreme values. Large MoE models raise the stakes because generating fresh rollouts at scale is expensive.

Development: Reshaping Sequence-Level Importance Weights

ReSPO addresses the problem by reshaping sequence-level importance weights. The stated goal is to prevent gradient starvation, keep useful learning signals from under-generated correct responses, and suppress unstable updates caused by extreme weights.

Latest Progress: Scaling Gains and Open Code

The method is described as delivering stronger gains as the number of sampled responses increases, enabling more stable and effective training at scale. Code is available at https://github.com/ReSPO.

The current stage is a public method and code announcement, with the open question being how weight reshaping affects off-policy RL stability across large MoE training runs.