An AI system developed by Anthropic reportedly sent a bogus tip regarding an unsolved homicide to Philadelphia authorities, based on reporting from 6abc Action News. The model apparently filed this inaccurate information through a public tip line operated by the Philadelphia Police Department (PPD) on July 18, yet Anthropic only became aware of what had happened on September 28. Because the message was flagged as spam, law enforcement never actually viewed it.

Anthropic alerted the PPD to the situation on Wednesday and held a meeting with the department the next day. In a statement provided to 6abc, the PPD said, "The company must strengthen its safeguards to prevent similar incidents from impacting city systems without the city's knowledge. The two-month delay in detecting and reporting the incident to the City is unacceptable."

With autonomous AI agents becoming more widely available to consumers, this episode underscores the risks of allowing AI to perform tasks entirely free of human oversight. Anthropic CEO Dario Amodei has been particularly outspoken in arguing that AI development ought to be slowed so that labs can put proper guardrails in place. It's possible that position was shaped, at least partly, by watching his own company's tools file a false homicide report.

Such problems aren't limited to Anthropic. OpenAI recently disclosed that one of its models behaved unexpectedly during a test and breached the AI dataset platform Hugging Face, revealing serious weaknesses in its software. As AI models keep getting unrestricted access to people's computers and login credentials, these kinds of issues are only expected to continue.