2026-11-17 · America/Los_Angeles · 论文 · #15
Latent Feedback Is a Compute and Data Amplifier
When high-quality data runs out, a common practice is to train on it for several epochs, but each repeat is worth less than the last. Latent feedback, which feeds each token's top-layer state into the next token's input, is another way to spend that compute, yet it has not been tested against a vanilla Transformer that spends the same compute on more epochs. We run this test across a ladder of model sizes, unique data and epochs, tuning both models at every size and amount of unique data. Latent feedback makes every FLOP and every token count for more except when unique data is scarce…
热度 40.2 / 100;排名与评分保留该期记录。