2026-11-08 · America/Los_Angeles · 社区动态 · #4

I merged attention and MLP processing into a single gated Transformer block. GAP [R]

Hi everyone! I've been experimenting with an alternative Transformer architecture called GAP (Gated Attention Projection) . The main idea is to combine attention and MLP-style processing into a single gated block, instead of using the standard separate attention and feed-forward sublayers. I tested it against a standard Transformer using 100 million scored training tokens per model from WikiText-103. Some results: GAP20 used 19.91% fewer parameters than the standard Transformer. A smaller configuration can reduce parameters by 34.31%, although that version did not fair as well as the current…

热度 49.0 / 100；排名与评分保留该期记录。