2026-11-03 · America/Los_Angeles · 论文 · #18

FLAMIA: FLAsh MIcroscaling Attention for Fast and Causally Safe Training on Blackwell

Attention remains a substantial computational cost in Transformer training, particularly at long context lengths. NVIDIA Blackwell GPUs provide high-throughput low-precision matrix multiplication, but exploiting it in attention requires addressing quantization overhead, causal information leakage through shared scales, and numerical inconsistencies between the forward and backward passes. We present FLAMIA, a family of attention kernels for Blackwell with MXFP8 and mixed NVFP4/FP8 training paths. Our design combines token-block scaling that reuses quantized operands across matrix…

热度 40.9 / 100；排名与评分保留该期记录。