2026-11-07 · America/Los_Angeles · 社区动态 · #4

Effects of KV Cache Quantization on Qwen3.8 27B

KV cache quantization has been a hotly-debated topic for this community for some time, with opinions ranging from "never quantize" to "4 bit is fine". Since the anecdotal evidence is rather inconsistent, I decided to quantify how much KV cache quantization affects the model through KLD, which is a good measure of how the model's output distribution is changed. I tested 35 weight quants across llama.cpp, exllamav3, and vLLM, each with 16-bit, 8-bit, and (where supported) 4-bit KV cache. I also reworked my KLD dataset and measurement methodology this time, to include a mix of agentic tool-use…

热度 46.9 / 100；排名与评分保留该期记录。