Can We Trust Item Response Theory for AI Evaluation?

arXiv:2607.15190v3 Announce Type: replace Abstract: AI benchmarks increasingly leverage item-level statistical models, particularly item response theory (IRT), to estimate capabilities, rank systems, select informative examples, and diagnose benchmark quality.