2026-10-30 · America/Los_Angeles · 社区动态 · #11

Benchmarks are an unreliable measure of difficulty

Very interesting paper on the way that benchmark question difficulty and framing affect the observed improvement in scores for model training. “Our results show that many benchmarks implicitly overweight certain difficulty regions, so their leaderboard conclusions should be interpreted with caution; others leave substantial portions of the difficulty axis un-covered, limiting their ability to monitor the model improvement process.” https://arxiv.org/pdf/2610.29927

热度 46.3 / 100；排名与评分保留该期记录。