AI Resonance · 阅读最新日报 · AI 入门推荐

2026-11-13 · America/Los_Angeles · 论文 · #14

Factored One-sided Adam-Moment for Practical and Scalable SOAP

Second-order optimizers (e.g., Shampoo, SOAP) have emerged as compelling alternatives to Adam/AdamW and promise faster per-step convergence than AdamW in transformer training. However, they still remain costly in practice, and even SOAP still pays substantial optimizer-state memory and eigendecomposition overhead. We ask whether the benefits of SOAP can be retained at nearly the efficiency of AdamW. In this paper, we answer yes with a lightweight SOAP variant, Factored One-sided Adam-Moment (FOAM). The path to practical impact is twofold: first, FOAM employs input-only preconditioning,…

Factored One-sided Adam-Moment for Practical and Scalable SOAP

热度 39.1 / 100;排名与评分保留该期记录。