AI Resonance · 阅读最新日报 · AI 入门推荐

2026-11-19 · America/Los_Angeles · 论文 · #12

Tideway: Phase-Aware Tiered Memory Management for MoE Inference on Consumer Devices

Large language model (LLM) agents are rapidly becoming a dominant paradigm for AI applications. Recently, there is a growing demand to deploy large-scale Mixture-of-Experts (MoE) models locally on consumer PCs to balance data privacy, API costs, and inference efficiency. However, unlike datacenter servers with abundant DRAM, consumer devices are tightly constrained in both GPU VRAM and host RAM, forcing massive MoE weights to spill across VRAM, host RAM, and SSD. Under such spilling, we observe that existing systems fail to fully utilize this memory hierarchy, treating the tiers as…

Tideway: Phase-Aware Tiered Memory Management for MoE Inference on Consumer Devices

热度 40.1 / 100;排名与评分保留该期记录。