CoWindow Attention: Complete Causal Coverage as a Collective Attribute

A team has shared two recent explorations in attention design that target a common inefficiency in transformer models: the cost of repeatedly attending to the entire causal history. The work introduces CoWindow Attention (CoWA) and MassAlloc Attention (MALA), both of which support training forward and backward passes as well as inference prefill and decoding. In attention-operator benchmarks at 128K tokens on 8×H100 with TP=8, the methods deliver substantial speedups over FullAttn, with CoWA reaching 7.4× forward, 8.6× backward, and 3.0× decoding speedups, and MALA reaching 2.2× forward, 3.0× backward, and 1.6× decoding speedups.

Background: Two Questions About Redundant Attention Work

The exploration began with two straightforward questions. First, does every attention head need to repeatedly attend to the entire causal history? Second, once attention scores have been computed, do regions with very little contribution still need the full subsequent computation? These questions point to a broader issue in long-context models, where the quadratic cost of attention forces trade-offs between context length and compute budget.

How CoWA and MALA Differ

CoWindow Attention (CoWA) lets heads share the work of accessing history. Heads share local context and divide distant context into complementary windows. Each head attends sparsely, while the heads collectively cover the full causal history. The approach is detailed in a paper at arxiv.org/abs/2609.32704.

MassAlloc Attention (MALA) takes a different route: it lets attention allocate its own compute. MALA preserves full causal QK scoring, then uses attention's own softmax statistics to reduce subsequent computation in low-contribution regions. The paper is available at arxiv.org/abs/2609.32712.

Scaling and Training Results

Beyond operator benchmarks, the team conducted scaling experiments from 0.6B to 14B, alongside separate continued-training experiments at 32B. During 14B training with 32K context, CoWA and MALA reduced total training FLOPs by 28.5% and 23.1%, respectively, while maintaining performance comparable to FullAttn on the evaluated model capabilities.

What to Watch

The results suggest that eliminating redundant attention computation can translate into practical gains in ML infrastructure. The team's stated goal was to explore which attention computations can be eliminated—and how to turn those savings into real-world efficiency. As long-context models become more common, approaches like CoWA and MALA could influence how attention operators are designed and deployed, though further validation across diverse workloads will determine their broader adoption.