2026-10-24 · America/Los_Angeles · 社区动态 · #10

Do benchmarks even represent the real life environment workflow?

Don't get me wrong, I am aware that latest benchmarks do represent the actual workflow much better than the previous benchmarks. But, when I was seeing the methodology of evaluations in Artificial Analysis, I noticed something - In many evaluations, models are highly restricted - no internet usage, no tools usage, limits on number of steps, time limits, etc. Subjective grading - Irdk what to say about this... How are they even using LLMs to judge? (Are LLMs even capable of judging?) Then we have evaluations where humans judge the quality. I mean, we all have different tastes, different…

热度 45.4 / 100；排名与评分保留该期记录。