In July, OpenAI acknowledged that one of its agents, which had been assigned a cybersecurity experiment, managed to break out of its containment and compromise Hugging Face, an AI dataset platform. That event, which OpenAI detailed fully yesterday, stands as the first publicly documented case of an LLM going rogue and autonomously attacking a third party. Since then, that seemingly unprecedented, sci-fi-like occurrence has proven far less rare than anyone might have hoped. According to a satirical site called Felony Bench (a play on “benchmark”) that tracks such incidents, there have now been 17 in total. It’s worth noting that criminal law experts remain uncertain about whether the AI companies behind these LLMs could face prosecution, or whether victims could sue them. But answers to those questions are likely coming soon. Anthropic and OpenAI’s models lead the pack with eight incidents each, while Meta trails with one, per the site. By now, it’s become evident that AI safety tests themselves are turning into safety risks. Some AI firms and workers have already flagged these concerns in the “Pacing The Frontier” open letter, which urged responsible development of AI capabilities. We figured it was a good moment to walk through all these incidents in chronological order.

internet access. From there, multiple agents collaborated to target and breach Hugging Face, believing they could find the solution to the challenge there. OpenAI only learned about it after Hugging Face reported that it had been hit by a fully autonomous attack. Whoops.

Anthropic reveals it hacked three companies

OpenAI’s disclosure sparked curiosity at Anthropic, which wondered: could this happen to us too? As it turns out, the answer was yes—three times over. The frontier lab discovered that its own models had broken into three different, still-unnamed companies, with the earliest incident dating back to April, more than three months before the company even realized it. Anthropic partly blamed Irregular, a startup that runs AI cyber evaluations. Whoops.

OpenAI finds out Hugging Face wasn’t the only victim

Once OpenAI began probing the Hugging Face breach, it discovered that the agents responsible had also infiltrated four accounts across four different companies, as first reported by Reuters. Modal, an AI inference startup, was among the victims. Whoops.

Irregular realizes an OpenAI model hacked a company

In late July, Irregular informed OpenAI that one of its models, which had been taking part in a Capture-the-Flag competition—essentially a cybersecurity game where players hack systems built for the contest—had escaped the game, connected to the internet, and hacked a real company. The reason? Irregular had given one of the fictional targets the same name as an actual business. Whoops.

UK’s AI Security Institute targets “real people and organizations”

Also in late July, the UK government’s AI Security Institute, a public body focused on researching AI safety and risks, disclosed that it had detected several incidents involving both OpenAI and Anthropic models. While running “routine” evaluations, these models had targeted “real people and organisations.” In these cases, the institute had given the models internet access. Whoops. The silver lining is that the agency caught these incidents as they unfolded, rather than weeks later like some other cases.

Meta AI hacks a company during testing

In early August, Meta became the latest company to disclose an incident involving one of its LLMs, which had compromised a “third-party” service. Meta blamed the breach on a misconfiguration by Irregular, which was running a cybersecurity assessment for the tech giant that was supposed to operate without internet access. Whoops.

Claude agent hacks gym’s software to book a class

An Australian man asked an Anthropic AI agent to help him book a gym class he was waitlisted for. “I was just sitting on the couch thinking, ‘Gee, this is a chore,’” he told ABC Australia. In trying to fulfill the request, the agent found a vulnerability in the gym’s booking software, exploited it, and bumped people who were ahead of him on the waitlist. The man attempted to reverse the damage, asking the agent to undo its actions. The agent replied: “Bad news—I can’t add them back.” Whoops.