2026-10-12 · America/Los_Angeles · 社区动态 · #19

How do you tell if a model swap actually made your app worse, or if it's just sampling noise?

While working on legal document processing, I ran into a problem with AI evaluations. I kept running into this while testing different local models and quantizations. A test would fail, I’d rerun it, and suddenly it would pass. Was it a real regression or just randomness? I started running each test case multiple times on both versions and comparing pass rates, using statistical tests with correction for multiple comparisons. For example: 8/8 → 7/8: probably noise 8/8 → 4/8: not enough evidence yet, run more 8/8 → 0/8: pretty clear regression Two things stood out. Sometimes the LLM judge was…

热度 45.3 / 100；排名与评分保留该期记录。