Beginning in 2024, Anthropic’s performance optimization team has required job candidates to complete a take-home assessment to verify their technical knowledge. However, as AI coding assistants have advanced, the evaluation has undergone significant revisions to prevent AI-aided dishonesty. Team lead Tristan Hume detailed the evolution of this challenge in a Wednesday blog post. “Each new Claude model has forced us to redesign the test,” Hume explained. “Within the same time constraints, Claude Opus 4 surpassed the majority of human applicants. That still let us identify the strongest candidates—but then Claude Opus 4.5 performed at that elite level as well.”

This created a substantial problem for candidate evaluation. Without supervised, in-person testing, there is no reliable method to confirm a person isn’t using AI to complete the assessment—and if they do, their results will immediately rank among the best. “Under the take-home format, we lost the ability to differentiate between the work of our top applicants and the output of our most capable model,” Hume noted. The challenge of AI-assisted cheating is already causing significant disruption in educational institutions globally, making it notably ironic that AI companies themselves now confront the same issue. Yet Anthropic is in a particularly strong position to address it.

Ultimately, Hume crafted a new test that focused less on hardware optimization, rendering it novel enough to challenge current AI systems. As part of the same post, he also shared the original assessment, inviting readers to propose superior solutions. “If you can best Opus 4.5,” the post states, “we’d love to hear from you.”