A Reinforcement Learning Perspective on OPD: Least-Squares Policy Distillation for Sample-Efficient LLM Reasoning

Meta description:
LSPD reframes on-policy distillation as KL-regularized reinforcement learning, using least-squares matching to cut rollout batches for LLM reasoning.

Tags: on-policy distillation, LSPD, reinforcement learning, LLM reasoning, sample efficiency

Featured image: abstract technology style, showing a teacher network guiding a student network through a least-squares objective, with no real people, no company logos, and no text.

Least Square Policy Distillation, or LSPD, reframes on-policy distillation for language models as a reinforcement learning problem that can reuse accumulated experience. Its replay-based variant, LSPD-RB, is reported to match vanilla on-policy distillation with approximately one-quarter of the rollout batches, while the broader method improves reasoning performance and multi-sample solution coverage across six mathematical reasoning benchmarks and three teacher-student settings.

Background: On-Policy Distillation Already Has an RL View

On-policy distillation (OPD) is commonly implemented with policy-gradient updates: the student model generates samples, the teacher model provides feedback, and the student is updated toward actions the teacher favors. The objective is often reverse-KL divergence, a measure of how far the student's next-token distribution is from the teacher's, written in the direction that penalizes student choices the teacher finds unlikely. The paper's starting point is that this reverse-KL objective also has an exact interpretation as KL-regularized reinforcement learning. In that formulation, the reward is defined by the teacher relative to a reference policy, and the KL term controls how far the student moves from the policy that produced the experience. This connection matters because value-based reinforcement learning has long been associated with learning from accumulated experience, while on-policy policy-gradient updates often spend fresh rollouts as they are collected. That framing invites a direct question: can value-based RL principles make policy distillation more sample-efficient?

Development: LSPD Turns Teacher Feedback into a Least-Squares Target

LSPD changes the learning signal. Rather than using teacher feedback only to drive policy-gradient updates, it directly matches student and teacher log probabilities through a robust least-squares objective. Log probabilities are the logarithms of the probabilities assigned to each next token, so the student can be trained to reproduce the teacher's token-level scores under a regression-style loss. The method is motivated by optimistic reward estimation and KL-regularized policy improvement. Because the objective can be applied to accumulated experience, the paper argues that it brings a central advantage of value-based RL to LLM distillation. The theoretical analysis establishes logarithmic reverse-KL regret for an idealized optimistic formulation, meaning the cumulative gap to the teacher grows logarithmically rather than linearly in that simplified setting.

Latest Results: Six Benchmarks, Three Settings, and One-Quarter Rollout Batches

The reported evaluation covers six mathematical reasoning benchmarks and three teacher-student settings. Across those comparisons, LSPD improves reasoning performance and multi-sample solution coverage. Multi-sample solution coverage refers to how much of the correct or useful solution space is represented when the model draws several answers instead of a single response. The replay-based variant, LSPD-RB, achieves performance comparable to vanilla OPD with approximately one-quarter of the rollout batches. That figure is the most operationally visible number in the work: rollout batches are generated attempts from the student model, and reducing them can lower the compute needed to collect teacher feedback.

Where This Leaves Distillation

The broader message is that viewing distillation as an RL problem opens algorithmic choices beyond on-policy policy gradients. For readers tracking LLM training costs, the implication is that teacher-student learning does not have to remain a one-sample-at-a-time update loop. Once the objective is expressed as KL-regularized RL, value-based ideas such as replay, optimistic estimation, and least-squares regression can be tested against the standard policy-gradient recipe, creating opportunities to learn more from each interaction with the teacher. LSPD and LSPD-RB are currently presented as research methods evaluated on mathematical reasoning tasks, not as a finished replacement for all distillation pipelines. The next signal to watch is whether the same least-squares view can be carried beyond these benchmark settings without losing the sample-efficiency advantage.