2026-11-09 · America/Los_Angeles · 论文 · #16

Hierarchical Residuals: A Minimalist Decoupling of Residual Width from Compute Width

Residual connections underpin Transformers, but they implicitly tie the residual width carried through depth to the compute width used inside each layer. We introduce Hierarchical Residuals (HiRes), a minimalist architectural change that decouples these dimensions, widening the residual state while keeping per-layer compute width fixed. HiRes groups layers into blocks: standard d-dimensional PreNorm layers operate inside each block, while a wider D-dimensional PostNorm outer residual connects blocks. Learned read/write projections bridge the two spaces at block boundaries, leaving individual…

热度 30.5 / 100；排名与评分保留该期记录。