Model Benchmarks Still Too Messy to Police AI Safety, Hugging Face Warns

Irene Solaiman, chief policy officer at Hugging Face, told TechCrunch Disrupt that AI benchmarking methods remain too immature and inconsistent to serve as reliable safety tools. The industry has leaned heavily on model evaluations to measure performance, but reproducibility remains elusive—especially when models aren't openly available for independent verification. Beyond individual test validity, comparing results across different benchmarks is nearly impossible. Solaiman noted that embedded evaluators, positioned as neutral third-party observers inside AI labs, won't solve this fragmentation. The field also struggles with agentic evaluations and system-wide testing, since performance depends on harnesses and tools as much as the model itself. Without standardized qualification criteria for evaluators and more uniform benchmarking practices, the current approach risks becoming theater rather than meaningful oversight.