Distilling Routed 3D Privileged Knowledge to Enhance Spatial Reasoning in Vision-Language Models

Meta description: ZJU-REAL GPD-4B and GPD-15k use geometry-privileged distillation with GRPO while inference stays RGB-only; code, model, and dataset are public.

Tags: GPD-4B, GPD-15k, GRPO, geometry-privileged distillation, RGB-only inference

Featured image: Abstract neural network diagram with 3D geometric cues flowing into a training teacher and an RGB-only student path, no people, no logos, no text.

The paper introduces GPD, a geometry-privileged distillation framework that connects spatial training signals to vision-language reasoning without changing the input format used at inference time. The public artifacts include the model name GPD-4B and the dataset name GPD-15k, with code published under ZJU-REAL. The central idea is that a teacher can receive question-relevant 3D cues and reference answers during on-policy self-distillation, while the deployed model remains RGB-only. The practical consequence is that learning can use richer geometry information even when deployment uses ordinary color image input.

Background: Why Geometry Can Be Privileged

The framework treats geometric information as privileged training context. That means the information may help the model learn how to reason about a question, but it does not have to be present when the model is asked the same type of question later. In many vision-language settings, the available input is RGB, while the task may involve spatial structure that is not directly visible from RGB input. By routing question-relevant 3D cues to a teacher, GPD creates a training path that can use more information than the student receives during normal inference. The result is a method that tries to improve reasoning through teacher guidance while preserving RGB-only operation.

Development: How GPD Extends GRPO

GPD is built around on-policy self-distillation. In this setting, the model generates its own trajectories, and the teacher uses reference answers and question-relevant 3D cues to provide distillation signals. The paper adds distillation on incorrect trajectories to GRPO. That means the optimization does not rely only on GRPO; it also uses teacher feedback when the model produces wrong or failed attempts. The incorrect trajectories become a channel for targeted learning: the teacher's distillation signal can be applied to attempts that fall outside the desired behavior. The important design choice is that this extra signal is used during training, while inference remains RGB-only.

Latest Public Progress: Code, Model, and Dataset

The project provides public resources for the method. The code is available at https://github.com/ZJU-REAL/GPD. The model is listed at https://huggingface.co/xinyili0624/GPD-4B. The dataset is listed at https://huggingface.co/datasets/xinyili0624/GPD-15k. These artifacts show that GPD is not only a proposed framework, but also has public resources that include a model name, a dataset name, and implementation code. The ZJU-REAL repository anchors the implementation, while GPD-4B and GPD-15k identify the model and data resources associated with the paper.

The work currently sits at the stage of open method and open artifacts. The source does not provide benchmark scores or deployment dates, so the factual record consists of the method description and the three public links. It presents a clear training recipe: geometry-privileged distillation, on-policy self-distillation, reference answers, question-relevant 3D cues, and distillation on incorrect trajectories added to GRPO, all paired with RGB-only inference. The training path uses extra geometry information, while the inference path relies on ordinary color image input. What to watch next is how researchers use the code, GPD-4B, and GPD-15k to test the framework's behavior on spatial reasoning tasks, especially where RGB inputs must carry the inference burden without 3D cues.