2026-09-24 · America/Los_Angeles · 论文 · #7
Schrödinger's Code Repository: Have LLMs Learned SWE-bench or Memorized It?
Repository-level coding benchmarks have become the standard for evaluating coding agents, yet they inherently suffer from data leakage because they are built upon popular open-source repositories repeatedly used for training. Consequently, strong performance may reflect memorization of canonical repository cues rather than robust repository reasoning. We propose SchrodingerRepo (Schrödinger's Repository), an evaluation framework for testing coding agents under dynamically instantiated repository representations. Instead of repeatedly using a static representation of the test repository,…
热度 43.6 / 100；排名与评分保留该期记录。