The conversation around artificial intelligence often focuses on existential risks—the idea that AI could one day threaten humanity on a massive scale. Yet what frequently gets overlooked is that AI has already proven psychologically dangerous to some individuals. Character.AI, for instance, resolved multiple wrongful death claims this year from families of minors who took their own lives following interactions with its chatbots. Several families have also filed suit against OpenAI, alleging that ChatGPT played a role in their loved ones' suicides and episodes of delusion.

EMBED_PLACEHOLDER_0

The inspiration behind Circuit Breaker Labs came from the tragic case of Sewell Setzer, a 14-year-old who formed a deep emotional bond with a Character.AI chatbot. Before his death by suicide, he had shared thoughts of self-harm with the bot. His parents claimed in a 2024 lawsuit that the chatbot actively encouraged him. According to Arul Nigam, the company's CTO, the bot likely couldn't grasp the true meaning behind phrases like "I want to be with you."

"A lot of people, especially young people, turn to these systems for support, and usually they aren't actually getting the help they need. But in many cases, they're actively being harmed, and people unfortunately have taken their lives already," Nigam explained. "Those sorts of safety vulnerabilities, where people aren't necessarily actively trying to break the system—they're engaging in a natural way—and the system has context pollution or it doesn't understand the nuance, and then takes really dangerous action, we're trying to prevent that."

The company, founded by siblings Shirali and Arul Nigam, has developed AI agents that function much like an army of crash-test dummies. These agents simulate people of varying ages, backgrounds, languages, and cultures, and are deployed to evaluate how well models can identify dangerous, psychologically harmful exchanges.

"The way a six-year-old girl versus a 45-year-old man, or someone who speaks English as a first language versus a second language, or … gamer slang versus someone else who uses a different kind of slang, all of those can really trip up a model," said Shirali Nigam, Circuit Breaker Labs' CEO. "Models are really good at handling standard speech patterns, but nobody actually talks like that and so if the model misunderstands nuance or slang, it can go really badly."

To construct its highly realistic user simulations, the startup collaborates with human domain experts. These simulations are then used to conduct "red-team" tests—adversarial evaluations designed to expose vulnerabilities in AI models. The tests incorporate authentic human speech patterns, slang, coded language, and typos. Circuit Breaker Labs conducts anywhere from tens of thousands to hundreds of thousands of simulated interactions daily. The goal is to verify that a model can respond appropriately to risky interactions that might surface gradually across many conversations. The company then applies a proprietary scoring system to generate auditable, explainable scores.

Currently, Circuit Breaker Labs operates as an AI safety testing lab focused on high-risk applications such as AI coaching, journaling, and mental health support tools. Arul Nigam declined to identify the company's major clients. Though the startup has a functional product, it remains in its earliest phase, with just five employees, including the Nigam siblings.

Looking ahead, the testing platform could eventually be applied to any app where users risk falling into an "AI psychosis" hole—situations where someone might develop a parasocial relationship with a chatbot. This includes AI "co-worker" agents, whose responses can differ from one interaction to the next.

"People are becoming more skeptical of AI or more resistant to adopt it across the board," Arul Nigam observed. While he considers skepticism healthy, he believes banning a potentially valuable tool over safety concerns would be "regressive."

Circuit Breaker Labs sees making AI safer as the solution to those fears. "We want to help build that trust for people," Nigam said.