2026-10-21 · America/Los_Angeles · 社区动态 · #6

[Paper] Specter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative Offloading

Mixture-of-experts (MoE) models often exceed a single accelerator's memory capacity. Offloading experts to host memory creates a severe PCIe I/O bottleneck because data-dependent routing keeps transfers on the critical path. We present Specter, an inference system that uses quantization to combine speculative decoding with expert offloading. Its lightweight quantized draft model runs efficiently on device while preserving high-fidelity expert activation patterns. This provides accurate multi-step lookahead over future expert usage, allowing the runtime to organize expert movement and…

热度 47.7 / 100；排名与评分保留该期记录。