AI Resonance · 阅读最新日报 · AI 入门推荐

2026-11-13 · America/Los_Angeles · 社区动态 · #18

Scaling to 1T MoE on TorchTitan [P]

Hi everyone! Over the last few months we've been in the trenches with the TorchTitan and PyTorch maintainers, trying to find out how far GH200 (the GPUs inside the JUPITER supercomputer) can be pushed. Short answer: far enough to train a ~1T-parameter MoE at ~1,300 tokens/s per GPU on 384 GPUs. That's 14X where we started, and our starting point was, let's say, humbling. Along the way we collected a handful of lessons. If you've ever trained a big model, or just stared at a CUDA out of memory, some of this will feel familiar. Here's the tour. If you want the full thing with all the plots,…

Scaling to 1T MoE on TorchTitan [P]

热度 44.6 / 100;排名与评分保留该期记录。