2026-09-29 · America/Los_Angeles · 论文 · #13

CoWindow Attention: Full Causal Coverage Is a Collective Property

FullAttn repeatedly exposes the complete causal history to every attention head, creating substantial redundant computation and memory traffic even with IO-efficient dense kernels. We introduce CoWA, a structured attention architecture that distributes access to the causal history across KV heads. All heads share near-diagonal and prefix-sink windows, while complementary long-range windows partition the remaining history. Their union provides full causal coverage although each head attends sparsely to distant tokens. This position-defined attention pattern requires no learned router or…

热度 49.2 / 100；排名与评分保留该期记录。