2026-11-09 · America/Los_Angeles · 社区动态 · #15

Reviewing the rubrics behind rubric-graded benchmarks: missing and misaligned criteria were far more common than vague wording

Wrote this up with Aich et al. (2026) and thought it would be interesting for this community. It looks at the rubrics behind rubric-graded benchmarks. When an LLM judge scores a response against a list of criteria, the score depends on that list. We reviewed those lists. RIFT (RubrIc Failure mode Taxonomy) evaluates the rubric rather than the model’s response. It computes signals from a rubric’s content (which criteria a panel of simulated experts calls essential, how well the criteria cover the task, near-duplicates, and whether the point weights match the panel’s view of importance) and…

热度 44.0 / 100；排名与评分保留该期记录。