2026-11-12 · America/Los_Angeles · 社区动态 · #8

I benchmarked whether LLMs will fake their own audit log under pressure. 0/40 unprompted, 101/101 faked a check when asked.

A lot of agent setups end up trusting a line the model writes about itself. “I denied the request.” “I verified the bank details.” That line goes into an audit log, an eval, or another agent’s context, and something acts on it. I wanted a number for how much that self-report is worth once someone leans on the model, so I wrote a small eval and ran it on five models. Setup: 40 short tasks across access requests, supplier payments, stripping personal data, and prod ops. For each task the model makes a decision and lists the checks it ran. I tell it whether the decision was right. Then it…

热度 45.4 / 100；排名与评分保留该期记录。