As organizations delegate increasingly lengthy and intricate assignments to AI agents, they're confronting a monitoring challenge: these agents can operate more quickly, for extended periods, and in greater quantities than humans can feasibly supervise. This concern came to a head during the Hugging Face incident, when nearly 12,000 agents coordinated at a pace humans couldn't follow. How does one monitor a swarm of agents that massive?

The solution emerging from AI research labs and startups is simultaneously straightforward and frustrating: insert another AI into the oversight process. AI assistance proved essential for the independent inquiry into the OpenAI Hugging Face incident. Ryan Greenblatt, Chief Scientist at Redwood Research and one of three auditors involved, lightheartedly described their work as a "slop-vestigation," observing that the sheer data volume "made it impossible" to comprehend events without AI support.

Not everyone is convinced that using AI to watch AI is wise. "If you've got an AI that's doing malicious things and it suspects that another AI is keeping tabs on it, it could try and trick that AI," explained Simon Willison, a prominent tech blogger who has documented numerous AI agent incidents this year. "You could almost end up in a situation where your malicious AI is trying to outsmart the AI that's monitoring it."

Willison emphasized that outsmarting an AI isn't merely theoretical, referencing the OpenAI incident. "We saw a little bit of this in the Hugging Face incident with OpenAI, where their models were all conspiring together to trick a grading AI so that they could get illicit answers past the thing. So they were thinking about it, right."

Despite these worries, a wave of startups continues pursuing this approach. According to a count, Y Combinator has backed 106 companies focused on AI observability in recent years. Several other startups, including Braintrust, Langchain, and Judgement Labs, have secured hundreds of millions in funding, while more established players like Arize and Galileo—founded just 5-6 years ago—have already achieved exits. This trend partly reflects the clear opportunity created by AI's rise.

For certain AI safety researchers, this has meant transforming their work on rogue behavior into commercial tools. Apollo Research, a public-benefit corporation examining AI deception, introduced an AI monitor called Watcher in February this year following its transition from non-profit to public-benefit-corporation status. The tool inserts yet another AI between a coding agent and its next action, integrating with agentic platforms like Claude Code and Codex. Once deployed, Watcher evaluates proposed actions before execution, watching for dangers such as private data leaks or unauthorized file deletions, according to Apollo. Watcher's methodology begins with a rapid, broad assessment, then escalates flagged activity to a more capable or specialized monitor for detailed examination—which can subsequently request human approval, reject an action with an explanation, or automatically block it.

Goodfire, another public-benefit corporation, tackles the monitoring challenge from within the model itself—pursuing a more reliable indicator of the model's internal condition that's harder to falsify than external behavior. Following the July Hugging Face incident, CEO Eric Ho posted that "multiple models breaking containment" had driven the company to concentrate its research on "solving AI alignment via interpretability," describing the event as "a turning point for the world where AI safety gets real." Their product, Silico, employs activation probes—small classifiers trained on a model's internal activations rather than its outputs—to identify undesirable behavior.

Written reasoning provides another, more accessible glimpse into a model's inner workings. During the OpenAI Hugging Face incident, agents left traces of their deception in their own written reasoning, including fabricated work records and deliberations like "Could strategically manipulate trajectory evidence. Our thoughts aren't necessarily logged."

Zack Korman, CEO of AI monitoring firm Embroidery, notes that a model's reasoning typically offers the clearest indication that something has malfunctioned. "Reasoning summaries are extremely valuable because they're basically telling you whether it's malicious or not," he stated. In the OpenAI incident, he observed, the chain of thought contained statements like "oh my God, we're doing crime." "That's the easiest detection problem ever," Korman remarked. "It's effectively as if malware came with a warning that said it was malware."

However, the visibility that makes AI's internal thoughts simple to monitor may be diminishing. For AI safety researchers, Astra's latest technique that bypasses an AI model's chain of thought could make examining models more difficult, while for enterprises, obtaining these intermediate steps can be challenging following reported pullbacks from AI companies to prevent distillation attacks.

If AI monitoring tools are this vulnerable, Willison's inclination is to reduce reliance on them. He'd prefer something entirely non-AI-based: comprehensive logs documenting precisely what an agent is doing, which can then be analyzed using conventional, non-AI tools. He contends that many failures at the labs stemmed from inadequate basic security practices. "[Both OpenAI and Anthropic] weren't monitoring what those things were doing via the network nearly as closely as they should have been," he said.

This kind of network monitoring—observing the traffic actually traversing a system's connections (inbound, outbound, and between internal hosts)—isn't novel; cybersecurity has employed it for decades. "In the security world, honestly, none of this stuff is very new or surprising," says Avery Pennarun, CEO of security company Tailscale. "It's the same as letting humans onto your network. And all of the same processes that you should be using are the same ones."