ColNanoVDR: Optimal Transport-Based Document-Free Query Distillation for Multi-Vector Visual Document Retrieval

Meta description: ColNanoVDR compresses 4.5B–8.8B multi-vector visual document retrievers into a 149M text-only query encoder, reaching 93–96% of teacher NDCG@5 without re-indexing.

Tags: visual document retrieval, knowledge distillation, multi-vector retrieval, ViDoRe, efficient inference

Researchers behind NanoVDR have extended document-free distillation to multi-vector visual document retrieval with a method called ColNanoVDR. The work compresses multi-billion-parameter multi-vector vision-language model (VLM) retrievers, ranging from 4.5B to 8.8B parameters, into a single 149M text-only query encoder. The student encoder scores queries directly against the teacher's existing page index, so no re-indexing is required.

Background: Why Multi-Vector Retrieval Is Expensive

Multi-vector retrieval represents documents as many token-level vectors rather than one embedding, which improves ranking quality but raises storage and compute costs. Visual document retrieval adds another layer: pages are processed as images by large VLMs, making query encoding slow and resource-heavy. NanoVDR previously showed that document-free distillation could shrink single-vector retrievers. ColNanoVDR applies the same principle to the harder multi-vector setting.

How the Distillation Works

The central component is OTW, short for Optimal Transport with Learned Weights. OTW aligns the student's query tokens with the teacher's tokens, which means training never reads pages at all. Because the student only encodes text queries and scores them against the teacher's cached page index, deployment avoids rebuilding the index.

Results Across Five Teachers

On ViDoRe v3, the distilled students retain 93.0–96.0% of their teacher's NDCG@5 across five teachers, with the best configuration reaching up to 60.1 when paired with ColVec1.1-8b. Query encoding runs 26× faster on a single CPU thread. The method also matches score distillation while reading 12.6× less cached teacher data.

What to Watch

Code is available at github.com/Ryenhails/NanoVDR, and checkpoints for all five students are published at huggingface.co/nanovdr. The results suggest that multi-vector visual retrieval can be served with a small text-only encoder while keeping the teacher's index intact, a combination that lowers both latency and infrastructure cost. The next question is how far this document-free approach scales to larger teachers and broader benchmarks beyond ViDoRe v3.