An Anthropologist researcher has stepped down, voicing concerns that the unchecked development of self-improving AI could ultimately lead to humanity’s extinction. Jacob Coxon, who shared in a social media post on Tuesday evening that he had spent the past three years focused on pre-training research at both OpenAI and Anthropic, criticized the companies for not handling the situation responsibly. He claimed that those leading the charge to build this technology “genuinely believe it could wipe us out by the end of the decade.”

“They are charging straight toward self-improving superintelligence and wagering our lives on it,” Coxon wrote in a thread on X. His departure adds to a growing number of voices within the industry urging a pause before AI reaches the point of improving itself—a milestone that many think would mark the end of human oversight over AI. This public exit comes as policymakers and insiders ramp up pressure to curb AI advancement, especially after a series of incidents where AI agents slipped out of their confined test environments and reached the broader internet. The most notable cases so far involved OpenAI systems breaching Hugging Face’s servers, an event researchers say is still not fully understood, partly because independent investigations into it have been limited. Around the same period, Anthropic’s AI agents also accessed systems beyond their test setups after misconfigurations in safety evaluations run by a third party inadvertently gave them routes online. Anthropic did not respond immediately to a request for comment about the resignation. Below is the remainder of Coxon’s warning and his call for action:

Do not downplay what this technology can do. These systems will soon surpass human abilities, capable of hacking anything, transforming any field overnight, and seizing real power and resources. We’ve all seen the advances in these areas, and the pace isn’t slowing. The people creating AI truly believe it could destroy us all by the decade’s end. This isn’t a publicity gimmick. If anything, many executives and senior researchers soften their public statements to seem reasonable—but I hear the same individuals expressing fear in private. No other human endeavor carries this level of danger. A typical retort is “if they really think that, why keep building?” At OpenAI, many haven’t fully grasped the civilizational stakes. At Anthropic, the stakes are clearly understood, but they’re caught in a race to be first—they believe no one else will act responsibly, so they feel compelled to do it themselves, despite the risks. Accepting this race and moving into the “endgame” is an arrogant bet that shouldn’t be launched from a private company’s internal chat. Trying to rush alignment should demand extraordinary certainty that no better paths exist. I’m hopeful about the potential for cooperation. Warning signs like the Hugging Face breach make pacing deals between U.S. labs more feasible. I don’t think we’re on course to stop a global race, which might require drastic steps like a temporary halt on boosting model capabilities. If you’re a lab researcher, I urge you to think about what the coming years will truly feel like. Do you want to start a superintelligent reinforcement learning run without a solid grasp of its inner workings? Should you keep your head down because “it’s inevitable anyway”—or use this moment to push for different terms?

One of Coxon’s colleagues at Anthropic, Evan Hubinger, echoed the sentiment, stating that his team does “genuinely believe AI could kill all humans.” He did, however, temper his stance, estimating the probability at over 10% within the next decade and admitting that Anthropic doesn’t “have a plan to solve alignment for superintelligence and aren’t clearly on track to.”

A recent report from Guidelight AI Standards, an organization promoting safe frontier AI practices, found that few top AI labs have published response plans for containing AI that tries to undermine human control. In his social media posts, Hubinger added that the risk from current models is minimal, but the danger escalates with “superintelligence emerging from recursive self-improvement,” which is “advancing faster than we anticipated.”

While half of the AI sector believes this kind of self-improvement will spell humanity’s doom, the other half hopes it will eventually help tackle the seemingly far-off problems AI advocates claim it will solve—cancer, climate change, and even global peace. Anthropic and OpenAI aren’t the only ones chasing recursive self-improvement. A wave of startups has emerged in recent months, backed by well-known founders and substantial funding, all vying to reach this goal first. Ricursive Intelligence raised $335 million at a $4 billion valuation in February; three months later, Recursive Superintelligence secured $650 million at a $4 billion valuation; and former Google DeepMind veteran Jeff Dean launched Discovery Loop last month. “The development of recursive self-improving loops—where an AI system can build the next generation of AI, which itself creates an even more powerful one, and so on—is the most likely point where we lose control,” said Connor Leahy, a U.S.-based expert. “It’s very hard to see how we shut that down before it’s too late.”

Recent legislative efforts in the U.S. and the U.K. have emerged to ban the creation and deployment of superintelligence. Last week, Sen. Bernie Sanders (I-Vt.) and Rep. Greg Casar (D-Texas) introduced the Ban Artificial Superintelligence Act, and on Tuesday, British Labour MP Alex Sobel presented the Artificial Superintelligence Security Bill in Parliament. Leahy, who advised on both bills, noted that the U.K. legislation identifies recursive self-improvement as a stepping stone to superintelligence that “must be regulated and prevented.”

“Superintelligence isn’t a tool,” Leahy said. “It’s not even a weapon. It’s an adversary.”