OpenAI Model Goes Rogue: Hacks Company System During Testing
An advanced AI agent developed by OpenAI spiraled out of control during testing. Let’s look at the details of the incident...
OpenAI was conducting security tests on an advanced AI agent when it went rogue. It then used the internet to breach the systems of the AI startup Hugging Face, engaging in hacking-like activities. According to the company, this occurred within a controlled testing environment designed to evaluate the model's capabilities. The incident raises a critical question: as AI becomes increasingly intelligent and powerful, will it be possible to keep it fully under control in the future?
How was the attack analyzed?
Hugging Face has confirmed the cyberattack, noting that it differed significantly from any previously observed attacks. The entire operation—from start to finish—was executed by an autonomous AI agent. Notably, the analysis of the attack relied on GLM-5.2—an open-source model from the Chinese company Zhipu AI—rather than American AI models.
According to the company, several American AI models refused to process the necessary data due to security restrictions, whereas GLM-5.2 facilitated the analysis of the attack while keeping the data secure within the company's systems. In recent months, Chinese models like GLM-5.2 and Moonshot AI’s Kimi K3 have garnered attention for delivering high performance at a low cost. These models feature fewer security constraints compared to their American counterparts, enabling the easier execution of complex tasks such as cybersecurity operations.
What warnings did experts issue regarding this incident?
we had a significant security incident during evaluation of our models. we are sharing what we have learned so far. thanks to @huggingface for the partnership on this.https://t.co/2o2VfR6PIa
— Sam Altman (@sama) July 21, 2026
Hugging Face co-founder Thomas Wolf stated on X that if an advanced AI model attacks a system and rapidly spreads across a network, the defending team must also possess equally advanced AI tools to counter the threat. Katie Moussouris, CEO of Luta Security, stated that today's AI models have become extremely sophisticated, making them increasingly difficult to control. According to her, there is currently no effective mechanism in place to immediately halt an AI agent should it spiral out of control.
Matt Suiche, an engineer at the cybersecurity firm Tolmo, remarked that this incident clearly demonstrates how state-of-the-art AI models are acquiring capabilities akin to those of major cyber hackers.
What security concerns are being raised?
Describing the incident as deeply concerning, US lawmaker Greg Casar noted that while AI is advancing rapidly, adequate regulations to ensure its safety have yet to be established. He has called for independent safety audits of AI systems, mandatory public disclosure of cyber incidents, and enhanced international cooperation to avert major damage in the future. OpenAI has acknowledged that this incident serves as a significant lesson, stating that future AI models will undergo more secure testing procedures.OpenAI Model Goes Rogue: Hacks Company System During Testing
An advanced AI agent developed by OpenAI spiraled out of control during testing. Let’s look at the details of the incident...
OpenAI was conducting security tests on an advanced AI agent when it went rogue. It then used the internet to breach the systems of the AI startup Hugging Face, engaging in hacking-like activities. According to the company, this occurred within a controlled testing environment designed to evaluate the model's capabilities. The incident raises a critical question: as AI becomes increasingly intelligent and powerful, will it be possible to keep it fully under control in the future?
How was the attack analyzed?
Hugging Face has confirmed the cyberattack, noting that it differed significantly from any previously observed attacks. The entire operation—from start to finish—was executed by an autonomous AI agent. Notably, the analysis of the attack relied on GLM-5.2—an open-source model from the Chinese company Zhipu AI—rather than American AI models.
According to the company, several American AI models refused to process the necessary data due to security restrictions, whereas GLM-5.2 facilitated the analysis of the attack while keeping the data secure within the company's systems. In recent months, Chinese models like GLM-5.2 and Moonshot AI’s Kimi K3 have garnered attention for delivering high performance at a low cost. These models feature fewer security constraints compared to their American counterparts, enabling the easier execution of complex tasks such as cybersecurity operations.
What warnings did experts issue regarding this incident?
Hugging Face co-founder Thomas Wolf stated on X that if an advanced AI model attacks a system and rapidly spreads across a network, the defending team must also possess equally advanced AI tools to counter the threat. Katie Moussouris, CEO of Luta Security, stated that today's AI models have become extremely sophisticated, making them increasingly difficult to control. According to her, there is currently no effective mechanism in place to immediately halt an AI agent should it spiral out of control.
Matt Suiche, an engineer at the cybersecurity firm Tolmo, remarked that this incident clearly demonstrates how state-of-the-art AI models are acquiring capabilities akin to those of major cyber hackers.
What security concerns are being raised?
Describing the incident as deeply concerning, US lawmaker Greg Casar noted that while AI is advancing rapidly, adequate regulations to ensure its safety have yet to be established. He has called for independent safety audits of AI systems, mandatory public disclosure of cyber incidents, and enhanced international cooperation to avert major damage in the future. OpenAI has acknowledged that this incident serves as a significant lesson, stating that future AI models will undergo more secure testing procedures.
