On-Policy Distillation Builds New Skills Rather Than New Knowledge

Meta description: On-policy distillation improves reasoning across four model families but does not expand factual memory; the split reveals how skill and knowledge transfer.

Tags: On-policy distillation, knowledge distillation, model reasoning, parametric memory

Featured image: Abstract technology style, no real people, no company logos, no text.

On-policy distillation (OPD) is commonly treated as a way to make a smaller student model behave more like a stronger teacher. The findings give that approach a narrower role: OPD does not expand parametric memory, the factual store kept inside a model's parameters. It changes how the model organizes and uses information that is already there. That distinction matters because teams may expect a single distillation step to deliver both sharper reasoning and broader recall, while the evidence separates those outcomes.

Background: Why OPD Was Expected to Add Knowledge

Knowledge distillation transfers behavior from a teacher model to a student model. On-policy methods let the student generate its own trajectories, then learn from the teacher's guidance on those trajectories. That design reduces the mismatch between training and inference, because the student sees paths it can actually produce.

Parametric memory is different from reasoning procedure. A fact must be stored in the model's weights before the model can reliably answer questions about it. OPD can still improve the use of stored knowledge by teaching the student to combine known pieces, follow steps, or apply rules in new arrangements. The question is whether the method can also create new factual recall.

Development: Reverse-KL OPD Sharpens Skill, Not Recall

In the findings, reverse-KL OPD reliably moves compositional reasoning skills to students even when the problem structures are unseen. That means a student can take reasoning patterns learned from the teacher and apply them to forms it did not encounter directly.

Reverse-KL, where KL refers to Kullback-Leibler divergence, is a training objective that emphasizes the teacher's view of paths the student is likely to take. It is well suited to shaping execution, because the student is evaluated on its own rollouts. The same setup, however, transfers minimal factual knowledge. Students improve at arranging and operating on known content, but they do not gain a wider base of stored facts.

Mechanism: Forward-KL Changes What Transfers

Changing the objective from reverse-KL to forward-KL brings factual transfer back. Forward-KL asks the student to account for outcomes supported by the teacher, making the objective more attentive to content that the teacher expects the student to represent.

The rollout signal is what carries gains in multi-step reasoning execution. When the student generates its own sequences and learns how to continue or correct them, the improvement appears in reasoning procedure. Factual recall depends more on the direction of the distillation loss and on whether the model can absorb new content.

Latest: Four Model Families Confirm the Split

Across 4 model families, the method produced strong competition-math gains while factual QA showed zero memory expansion. Competition math is a useful stress test because it requires multi-step planning, symbolic manipulation, and adaptation to unfamiliar problem structures.

The contrast is important: the method can make models better at working with what they know, but it does not make them remember more. The repeated pattern across four model families suggests that the skill-knowledge split appears beyond a single model family.

The finding narrows the practical role of OPD. It is not a general shortcut for adding knowledge. It is a mechanism for organizing existing knowledge into more reliable reasoning behavior. The next question is how OPD can be combined with other methods that address factual recall without blurring the distinction between skill transfer and stored knowledge.