Since 2024, Anthropic's performance optimization team has required job applicants to complete a take-home assessment to verify their technical expertise. However, as AI coding assistants have advanced, the test has undergone significant revisions to prevent candidates from relying entirely on Claude to generate responses. Team lead Tristan Hume detailed the evolution of this challenge in a blog post published on Wednesday.

"Each new Claude model has forced us to redesign the test," Hume explained. "When operating under the same time constraints, Claude Opus 4 surpassed the majority of human applicants. That still enabled us to identify the strongest candidates—but then Claude Opus 4.5 matched even those top performers."

While candidates are permitted to use AI tools during the assessment, this presents a serious dilemma for evaluating talent. If humans cannot improve upon the model's output, the test essentially measures the capabilities of different AI systems rather than identifying exceptional candidates. "Under the constraints of the take-home test, we no longer had a way to distinguish between the output of our top candidates and our most capable model," Hume noted.

The challenge of AI usage on evaluations is already causing significant disruption in educational institutions worldwide, making it particularly ironic that AI labs now face the same issue. Yet Anthropic is in a distinctive position to address this problem.

Ultimately, Hume developed a new test that shifted focus away from hardware optimization, crafting a sufficiently novel challenge that current AI tools cannot easily solve. As part of the blog post, he also shared the original test, inviting readers to attempt a better solution. "If you can best Opus 4.5," the post states, "we’d love to hear from you."

*Correction: A previous version inaccurately described Anthropic's policy regarding AI tool usage on the take-home test. AI use is explicitly allowed.*