2026-09-25 · America/Los_Angeles · 论文 · #2

Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs

While Large Language Models (LLMs) rely on highly non-linear components, in this work we demonstrate that they exhibit fundamental linearity: when inputs from distinct text streams are linearly combined, the model outputs a superposition of the individual next-token distributions. We term this the Superposition Linearity Hypothesis. We provide evidence that superposition is an intrinsic property of the Transformer architecture rather than an emergent consequence of training; in fact, we observe that it tends to diminish as pretraining progresses. However, we demonstrate that linearity can be…

热度 49.5 / 100；排名与评分保留该期记录。