Ask Which, Not How Good: Sizing Benchmarks Scored by an LLM.

arXiv:2609.27787v1 Announce Type: new Abstract: Benchmarks scored by an LLM judge routinely adjudicate differences of a tenth of a point, but the resolution of those benchmarks has never been measured. Existing sample-complexity work covers accuracy benchmarks