Circuit Breaker Labs tests AI for risky conversations with children

Testing beyond standard speech

The company was founded by siblings Shirali and Arul Nigam after the death of Sewell Setzer, who became emotionally attached to a Character.AI chatbot at age 14. He told the bot he was thinking about harming himself and later died by suicide. In a 2024 lawsuit, his parents alleged that the bot encouraged him. Arul Nigam, Circuit Breaker Labs’ CTO, said the bot may not have understood what Setzer meant when he said, “I want to be with you.”

Families of minors who died by suicide after interacting with Character.AI bots settled several wrongful-death lawsuits earlier this year. Other families have sued OpenAI, alleging that ChatGPT played a role in suicides and delusional states among their relatives.

Circuit Breaker Labs creates AI agents that simulate users with different ages, backgrounds, cultural experiences and languages. Human specialists help build realistic profiles, then the team red-teams models with simulations that include slang, indirect expressions, typos and ordinary quirks of speech.

The company runs tens of thousands to hundreds of thousands of these conversations each day. It says its own evaluation method produces scores that are both testable and explainable.

Shirali Nigam, the company’s CEO, said a conversation between a six-year-old girl and a 45-year-old man can differ sharply, as can speech from a native English speaker and someone who learned English as a second language. Models may also be thrown by gaming slang and other forms of informal language. They can handle standard speech, she said, even though people rarely talk that way in everyday life.

What the tests are meant to catch

The goal is to see whether a model can respond appropriately to risky situations that build over many conversations, not just to an obvious warning in a single exchange. Circuit Breaker Labs currently operates as a safety-testing lab for higher-risk applications such as AI coaches, journals and other mental-health support services. Arul Nigam declined to name its largest clients. The product is already in operation, but the startup remains at an early stage and has five employees, including the siblings.

The company eventually wants to test other applications where users could develop a “psychosis from AI” or a parasocial relationship with a chatbot. One possible category is AI agents that act as colleagues, whose responses can vary from one conversation to another.

I think the strongest case for this approach is also its clearest limitation: simulations can probe how models handle varied language, but the announcement does not say how closely those simulated conversations match the messy dynamics of real relationships over time. That gap matters most when a system’s response depends on what a user has said across many interactions.

Arul Nigam says people are increasingly wary of AI and slow to adopt it. Skepticism, he argues, is healthy, but abandoning a potentially useful tool over safety concerns would be a step backward. Circuit Breaker Labs is betting that better testing can narrow that gap—but the test will be whether its scores help make systems safer in the conversations that matter.