A Single Block, Many Depths: Depth-Programmed Experts for Recurrent Vision Transformers

Meta Description: reViT recurrent Transformer block with depth-conditioned FFN experts matches vision encoders at comparable inference FLOPs with no intermediate distillation.

Tags: reViT, vision transformers, mixture of experts, model distillation, efficient inference

Featured Image: Abstract technology style illustration of a recurrent Transformer block, depth coordinates, and shared expert paths; no real people, no company logos, no text.

reViT is a vision encoder design that applies a single Transformer block recurrently and can match the accuracy of a full-depth vision encoder at comparable inference FLOPs, or floating-point operations, without intermediate feature distillation. The result matters because Transformer encoders for images usually rely on many stacked blocks, each storing its own parameters, to create specialized processing depth. If one recurrent block can emulate that depth through shared experts, a model may become much smaller at storage time while keeping similar inference compute.

Background: Depth Improves Vision Encoders but Increases Stored Parameters

Deep vision Transformers process an image through repeated blocks, and each block refines the features passed from the previous one. That depth is useful because different layers can specialize for low-level edges, mid-level shapes, or high-level semantic patterns. The cost is that a full-depth encoder keeps separate parameters for every layer, increasing model file size and memory traffic. This is especially relevant for classification, segmentation, and depth prediction pipelines that need large visual encoders but also practical deployment budgets.

Development Process: One Recurrent Block With Depth-Conditioned FFN Experts

reViT restores depth-specific transformations by changing the feed-forward network, or FFN, inside the recurrent block. The FFN is the part of a Transformer block that expands and transforms features after attention. Instead of assigning a different FFN to each layer, reViT represents the FFN at each recurrent depth as a convex combination of a small shared expert bank. A continuous normalized-depth coordinate programs this mixture, defining a resampleable trajectory through FFN parameter space. In practice, depth becomes a position along a shared parameter path, not a separate stack of learned layers.

The design was evaluated in two regimes: supervised ImageNet-1k training and distillation from a DINOv2 teacher. In both settings, controlled adaptations identify weight-space merging as the strongest tested MoE family at a matching one-FFN budget, ahead of token-dispatch and output-mixture alternatives. MoE, or mixture-of-experts, usually refers to models that combine or route specialized expert components. Here, the winning adaptation merges expert weights along the normalized-depth coordinate rather than routing tokens to separate experts or mixing expert outputs.

Latest Results: Fewer Stored Parameters and Transfer From a DINOv2 Teacher

When trained from scratch, reViT-B/16 attains DeiT III accuracy with about 70% fewer stored parameters. That comparison highlights the storage benefit: the model keeps the accuracy of a known vision Transformer training reference while storing much less parameter data.

The distillation path shows a similar pattern. An 8-experts model distilled using only the teacher’s output features retains nearly all of its DINOv2 teacher’s linear-probe accuracy and transfers across classification, segmentation, and depth prediction. A linear probe means a simple classifier is trained on frozen encoder features, so the result suggests the distilled reViT keeps most of the teacher’s visual representation quality without needing intermediate feature distillation.

Elastic Depth and Fixed-Depth Deployment

Elastic-depth training allows one checkpoint, or trained model, to operate at multiple tested depths by resampling the same normalized coordinate interval. This gives a single trained artifact a form of adjustable depth during evaluation or deployment, rather than requiring a separate checkpoint for every tested model size.

For fixed-depth deployment, the recurrent block can be materialized as a conventional dense graph. That removes online routing and merging without changing the one-FFN-per-depth compute, but it expands deployment storage. The tradeoff is therefore between the compact shared-expert checkpoint and a simpler unrolled dense model.

Current Stage: What reViT Shows for Vision Encoders

The evidence positions reViT as a compact recurrent Transformer approach that can match full-depth vision encoder accuracy while reducing stored parameters and supporting transfer from a DINOv2 teacher. Its key idea is to make depth a continuous coordinate through shared FFN experts, with weight-space merging as the strongest tested adaptation at a one-FFN budget. The next developments to watch are whether elastic-depth checkpoints retain their accuracy across more tasks, how developers choose between shared checkpoints and dense materialized graphs, and whether similar depth-coordinate designs extend beyond vision encoders.