AI labs have built their reputations around model benchmarking—the ostensibly neutral and accurate evaluation systems designed to determine whether one AI product outperforms another. For the industry to rely on these tests going forward, testing practices need to coalesce into a more consistent science. With that in mind, a key question arises: can we trust the tests currently being released?

"Trust is such a weighted term," said Solaiman. "In terms of whether we can trust results, one big part [of that] is can you reproduce the results of an evaluation. Do you have the level of access that you need to the model itself—this is where open weights come in in a big way," she added. Reproducibility in model evaluations has long been one of the thornier aspects of benchmarking—and it remains a subject of considerable debate across the industry.

Beyond the validity of any individual test, there's the additional question of how that test stacks up against other evaluations. "They are an evolving science—where we don't know how to meaningfully compare them to each other," Solaiman said. "They start to get ineffective relatively quickly, and we don't have as much of a thriving ecosystem as we need," she added. "There's this hunger for an embedded evaluator ecosystem now, but the qualifications that determine who is an effective evaluator across what tasks need a lot more standardization."

The concept of embedded evaluators—supposedly "neutral" third-party observers placed inside AI organizations—has gained traction over the last several months, as AI labs contend with a spate of safety incidents that have fueled concerns about the potential harms of their technology. Hugging Face was famously the target of one such incident. Rather than rely on active government regulation to identify and mitigate model safety concerns, labs have advocated for the third-party approach. Yet doubts persist about how neutral or effective those evaluators will actually prove to be. Anthropic's decision to hire Accenture as its first evaluator drew a good amount of backlash, with some critics mocking the notion that the consulting firm was qualified for the task at hand.

Evaluations are now shifting from merely a test of the model itself to a test of the entire system surrounding the model—including the harness and agent. The harness—the tool a user employs to access the model—specifically has a great deal to do with a model's performance, and a weaker model paired with a better harness could easily outperform a frontier model, Solaiman shared. "Agentic evals are really hard and really expensive," said Solaiman. "We have just more evaluations for models. … We don't have as many evaluations for harnesses, and we are developing better evaluations for the holistic agent—because that's what a lot of folks are getting value out of," she added.

In other words, the evaluation space is something of a free-for-all at the moment, and if benchmarking is to continue being seen as a useful tool for industry leaders, a more uniform, industry-wide approach—one where everyone is on the same page—may be necessary.