Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboard.

arXiv:2610.00651v1 Announce Type: new Abstract: Agent evaluations are increasingly used to compare LLMs and inform deployment decisions, yet ranks can reflect not only the model but also the effects of the evaluation conditions such as the scaffolds or tasks.