2026-10-29 · America/Los_Angeles · 论文 · #8

RealCLI: Toward a Unified Evaluation of Human-Agent Collaboration for CLI Agents

Command-line interface (CLI) agents increasingly execute longer-horizon tasks, yet existing benchmarks either assume that user instructions remain fixed throughout execution or evaluate task outcomes without capturing runtime collaboration quality. In this paper, we present REALCLI, a paired benchmark that unifies the evaluation of task execution and human-agent collaboration quality on realistic CLI tasks. To comprehensively measure collaboration, REALCLI builds a five-dimensional framework, operationalized into 22 sub-dimensions and 51 interaction scenario templates. REALCLI provides two…

热度 40.7 / 100；排名与评分保留该期记录。