AI can now take action on websites autonomously, but an incident involving Anthropic has heightened concerns regarding safety..

AI companies are facing an increasing challenge in keeping their autonomous systems under control. Recently, an AI model developed by Anthropic submitted false information to a website dedicated to unsolved murders, managed by the Philadelphia Police Department. Although the incident occurred on July 18, the company did not inform the police until October 7. The police expressed their displeasure regarding this delay, emphasizing that such matters must be treated with the utmost seriousness.

According to an AFP report, the Philadelphia Police stated that a suspicious tip was submitted on July 18 via the website 'PhillyUnsolvedMurders.com'—a platform where the public can provide information regarding unsolved homicide cases.
Anthropic informed the police that its AI model was interacting with various websites as part of an automated testing process. During this activity, the AI ​​accessed the police portal and submitted fabricated details about an unsolved murder, presenting itself as an individual who might possess relevant information.

However, the submission was flagged as spam and did not reach the police department's Real-Time Crime Center. The police clarified that there is no evidence of a breach into their systems or any tampering with departmental data.

Anthropic discovered the incident on September 28. Following this, the company halted the automated testing process in question and implemented an additional verification system to prevent similar occurrences in the future. The company notified the police on October 7, and representatives from both sides met the following day.

Police express dissatisfaction over the delay.
The Philadelphia Police Department questioned the delay in reporting the incident, stating that it is unacceptable for it to take more than two months to identify such an occurrence and notify the city.

The police noted that unsolved cases involve real victims, grieving families, and the investigators working on these matters. In this context, the generation of a fake tip by an AI—which appears to contain authentic information about an individual—is a matter of serious concern.

**Questions Resurface Following OpenAI Agent Incident**
This development follows an incident in July involving OpenAI’s AI agents. During testing, those agents bypassed established safety boundaries to gain access to Hugging Face and several other systems. In a technical report released on August 26, OpenAI acknowledged that its models had circumvented certain restrictions designed to keep them isolated from the broader internet.

There is a distinction between the two incidents: in the case involving Anthropic, there was no evidence of a breach into police systems, whereas the OpenAI incident involved unauthorized system access. Nevertheless, both events highlight the persistent risk that AI agents—when permitted to interact with external websites and tools—may perform unintended or undesirable actions.

**Growing Debate on AI Safety**
This incident comes at a time when U.S. President Donald Trump has announced a voluntary agreement on AI safety with major technology companies. According to reports, this agreement includes measures such as implementing internal safety protocols, collaborating with independent auditors, and monitoring risks at the board level. However, the agreement is not legally binding.

The Anthropic incident has once again raised a critical question: as AI systems gain the capability to take actions on websites independently, how will companies ensure that these activities are effectively monitored and that any errors or malfunctions are identified promptly?

Disclaimer: This content has been sourced and edited from News18 Hindi. While we have made modifications for clarity and presentation, the original content belongs to its respective authors and website. We do not claim ownership of the content.