2026-10-15 · America/Los_Angeles · 论文 · #8

LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation

Looped Mixture-of-Experts (MoE) models combine parameter reuse with sparse computation to improve performance under matched budgets. Prior work demonstrates that different loop visits can play different roles even when they share parameters. To probe the contributions of different loop visits, we learn separate residual coefficients in a model that applies a shared block twice, observing a smaller scale in the first visit. Fixed coefficient experiments show that reducing the first visit's scale preserves or improves performance, whereas the same reductions to the second hurt it. This…

热度 54.4 / 100；排名与评分保留该期记录。