2026-10-30 · America/Los_Angeles · 社区动态 · #16

My 4B agent failed its eval as a model and passed it as a product. One dumb guardrail did all of it.

I'm building a small on-device agent: Qwen3-4B (4-bit, MLX) driving an MCP server with 195 tools (US company formation). Sharing because the eval lesson surprised me. Setup 70 journey cases × 3 seeds, majority vote per dimension, production sampling (no greedy) 14-case "core" subset is the ship gate, needs 12/14 My eval called generate directly and scored the first tool call the model emitted Baseline: 42/70, core 11/14. Fails. Then I realized that lane can't see anything the orchestrator does. So I added a second lane: same fixtures, but run through the real turn loop against a dry-run MCP…

热度 45.0 / 100；排名与评分保留该期记录。