Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks.

arXiv:2604.24955v2 Announce Type: replace-cross Abstract: As benchmarks grow in complexity, many apparent agent failures are not failures of the agent at all---they are failures of the benchmark itself: broken specifications, implicit assumptions, and rigid ev