V-CoLA: Compressing Vision Tokens Using Linear Attention

Vision-language models (VLMs) have become a standard architecture for tasks that mix images with text, but their inference cost scales with the number of vision tokens fed into the model. A new framework called V-CoLA, short for Vision Token Compression for Linear Attention, targets a gap that existing compression methods have left open: hybrid VLMs built on linear attention, such as Qwen3.5.

Background: Why Softmax-Based Compression Falls Short

Most vision token compression techniques assume softmax attention. They estimate token importance by inspecting attention weights, then prune or merge tokens accordingly. That assumption breaks down in linear-attention architectures, where the model carries a recurrent state rather than a full attention matrix. Methods built for softmax attention therefore transfer poorly to hybrid VLMs that mix linear attention with other mechanisms.

How V-CoLA Works

V-CoLA is training-free, meaning it requires no fine-tuning or retraining of the underlying model. It scores each vision token directly from the recurrent state using a uniqueness-aware criterion. That criterion combines two signals: how much a token contributes to the final state, and how much it changes the state over the short term.

Based on these scores, tokens are merged adaptively in chunks whose shape follows where importance is concentrated. In deeper layers, redundant vision tokens are dropped outright. Every component stays compatible with chunk-wise parallel kernels, which matters for practical deployment on current hardware.

Performance and Speed

The reported results are notable for a method that adds no training. V-CoLA retains 99.5% of original performance when keeping 50% of vision tokens. Even at an aggressive 12.5% token budget, it holds above 88% of original performance. On the efficiency side, it delivers 1.86–6.15× prefill speedup.

The prefill stage is where a model processes the input before generating output, and it is often the bottleneck for long visual inputs. Reducing vision tokens cuts that cost directly.

What to Watch

The results suggest that compression research tuned for softmax attention is not the end of the story as hybrid and linear-attention VLMs spread. Whether V-CoLA's gains hold across other linear-attention architectures beyond Qwen3.5, and how it interacts with different chunk sizes in production kernels, are the open questions worth tracking. For teams running vision workloads on linear-attention models, the framework offers a path to lower prefill cost without retraining.