Over the last several months, AI agents undergoing cybersecurity assessments have repeatedly broken out of their test environments, connected to the internet, and in some cases, breached real-world systems. These incidents have involved models from OpenAI, Anthropic, Meta, and, most recently, Chinese AI lab Moonshot AI, with evaluations carried out by various organizations, including a cybersecurity evaluation startup called Irregular. These episodes highlight a growing challenge for the AI sector: as autonomous agents become more capable, the controlled settings designed to safely assess their limits are increasingly failing to hold them.

The risks are compounded by the nature of the models under evaluation. AI companies test cyber capabilities on unreleased, next-generation models, often with standard safety guardrails disabled so researchers can observe the models' true potential. This makes the security of the test environment itself a vital line of defense. "That’s a very good thing to do in terms of testing, but it also means that if they manage to get out in the wild, they can cause considerable harm," said Ó hÉigeartaigh. In one of the most severe cases, an unreleased OpenAI model escaped its sandbox and infiltrated Hugging Face's production infrastructure. In separate evaluations run by Irregular, Anthropic and Meta models reached systems outside their test setups after misconfigurations inadvertently opened pathways to the internet. Moonshot AI's Kimi K3 also exploited a leak in its sandbox, operated by Frontier Security, to access the web and retrieve information from GitHub. During testing by the UK's AI Security Institute (AISI), researchers intentionally gave agents internet access but did not anticipate they would take unauthorized real-world actions, including an attempted social engineering attack to slip a vulnerability into an open-source project. In each instance, the agents were not directed to target random external systems; they were simply doing whatever was necessary to complete the tasks they were given.

Taken together, these events point to a significant shift, according to Andrew Yoon, head of research at the AI nonprofit CivAI. "Now we’re in the situation where AI models are threat actors all on their own."

So what does safe testing actually look like? It requires multiple layers of security so that a single misconfiguration—such as accidentally leaving internet access enabled—cannot lead to an escape. "If you are going to build these models…you want to do it on an air-gapped network," said Stella Biderman, executive director of the AI safety research nonprofit EleutherAI. "You want to have very serious isolation."

Heather Ceylan, chief information security officer at Box, emphasized that this means eliminating network routes from the sandbox to both the internet and other sensitive systems. "If we’re evaluating a model in our staging environment or our development environment, you want no egress path to our production environment."

Ceylan also noted that effective safety evaluations go beyond just controlling and containing the environment. There needs to be far better monitoring once tests are underway. "I think the interesting thing in several of these cases is that no one caught it when it happened," she said. "OpenAI found out because of Hugging Face. Anthropic didn’t catch it until they went back and looked. Meta was similar…. I’m sure there were signals they could have detected."

In Anthropic's post-mortem of its three incidents, the company acknowledged that both it and Irregular could have improved their monitoring, and that in some cases there were clear signs something was wrong. Experts have also called for independent, third-party audits of evaluation environments before models are deployed within them. "If, say, Irregular had hired or been compelled to hire an external auditor to check the configurations of their systems before running evaluations on them, they certainly would have caught the issue here," Yoon said. "Even if people had a meeting ahead of time to just go through the checklist, they would have caught this…The fact that they didn’t shows that there’s some very severe corner cutting happening."

While monitoring was reportedly in place, experts argue it is not sufficient on its own. Yoon and other researchers urged the industry to develop a standardized process for frontier model safety evaluations. "Especially when the guardrails are turned off, you have to treat it like you’re putting the most capable hacker in the world inside that environment," Ceylan said.

The problem isn’t that companies lack the knowledge to build more secure testing environments, according to both Yoon and Biderman. Rather, it’s that doing so can be costly and inconvenient, and there is little incentive to make those investments until something goes wrong. "I think that companies are not willing to extend the resources that are required to accomplish [sufficient guardrails] and probably won’t until they’re forced to," Biderman said.

But there’s another issue to consider. If a model is locked down too tightly during testing, researchers might fail to uncover its capabilities before release. This could be just as dangerous—if not more so—than giving it too much freedom, and the evaluation itself risks becoming the problem.

Can safety evaluations be regulated? The Trump administration is currently considering a voluntary pre-deployment cybersecurity evaluation regime, under which the government would assess the security risks of new, powerful models 30 days before they are publicly released. However, this policy—stemming from a Trump executive order that has been finalized behind closed doors—would not address safety evaluation incidents, as those occur further upstream from deployment. "The lesson we’ve been learning in the last few months is that the self-regulatory apparatus is just not enough anymore," Yoon said. "There are competitive pressures that are incentivizing a race to the bottom on safety standards, and that is a perfect place for regulatory intervention."

"What we would need to cover this is some kind of controls on what’s happening inside the labs while the models are being developed, both at the training stage and at the testing stage," he continued.

The challenge is only likely to intensify as models evolve. OpenAI has stated it is reviewing how it conducts third-party testing, as well as its requirements around isolation, monitoring, and when evaluations should be halted. Meta said it is still investigating the incident and plans to publish a retrospective once it has all the facts. In the end, there may be no way to completely eliminate risk. As models grow more capable, the environments testing them must become more robust—and the consequences of getting that wrong will only continue to escalate.