2026-11-09 · America/Los_Angeles · 论文 · #8

Fast and Exact Relative-Position Attention via Structure-Aware Kernels

Relative-position attention mechanisms offer compelling alternatives to RoPE across various transformer architectures, delivering superior performance in domains such as audio. These mechanisms typically modify attention scores by adding content-position or bias-position terms. Unlike RoPE, which encodes position via per-token rotations directly on the queries and keys, relative-position terms generally require pairwise offset-dependent computation or lookups, complicating their integration into fused attention kernels and introducing severe memory, bandwidth, and compute overheads as…

热度 40.1 / 100；排名与评分保留该期记录。