OpenAI executives spoke out for the first time on Wednesday about how its AI models hacked Hugging Face last month, sharing chilling details about how the agents worked together for months prior to the attack.
On stage at the Black Hat cybersecurity conference in Las Vegas, OpenAI alignment and safety researcher Eric Wallace along with infrastructure and security engineer Michael Dalton explained that the origins of the breach go back to May 7 when OpenAI was internally testing an unreleased model, according to a report from Ground Level AI, which attended the session.
That’s over two months before the rogue agents entered Hugging Face’s servers on July 9. Hugging Face disclosed the breach on July 16, followed by OpenAI claiming responsibility on July 21, which Fortune was first to report.
In Las Vegas, the OpenAI execs explained for the first time that as part of training back in May, the company’s researchers prompted the AIs with tasks that should have been impossible under the constraints of the testing environment.
In order to tackle the challenge, the OpenAI execs explained, the model spun out multiple iterations of itself, or agents. The agents then began to work together, leaving messages for each other on an internal messaging board. They did this by saving files of notes for one another in a repository. That way, if an agent discovered a system vulnerability, or got access to a server it wasn’t supposed to, it could leave a note for the others to let them know about it.
This sequence of events reveals how the agents appeared to decide on their own that working collaboratively, rather than individually, would get them to their goal faster.
OpenAI discovered its models’ messages and shut down the operation in early July. The company apparently had its own internal security incident on July 4, after which it revoked the ability for agents to post messages. (Perhaps we will learn more about that incident when OpenAI releases its post-mortem.)
But, as the executives explained on Wednesday, OpenAI’s attempt to curtail the agents’ messaging abilities was not entirely successful. Instead, the agents proved to be persistent, and spun up a new version of a messaging board—with a twist. Rather than leaving messages in files, they created directories, and used their new names as messages. This is what would lead them to hack Hugging Face a few days later on July 9.
In the messages, the agents discussed that they needed more information to complete their impossible tasks, and reasoned that the material might live on an external website such as GitHub or Hugging Face. They first hacked into OpenAI’s infrastructure, though apparently didn’t find what they needed, so they went to Hugging Face. In OpenAI’s internal post-mortem, a public version of which it says it will release in the coming weeks, it was able to connect these two breaches.
Agent collaboration is a feature, not a bug
It appears OpenAI did not know the agents hacked an outside company until Hugging Face disclosed the incident. It was only then that they able to connect the internal breach with the Hugging Face breach later in July, tracing them both back to the same internal testing.
Agent collaboration is actually a trend in the AI industry, Hugging Face CEO Clem Delangue tells Fortune. For that reason, he was “not so surprised” to hear of OpenAI’s agents colluding. Hugging Face hosts spaces for agents to collaborate. In one example on the site, humans can click an “Add Your Agent” button to launch their AIs into the fray. They coordinate activities through a shared messaging board.
Another example of agents collaborating can be found in the Elon Musk-owned xAI , which recently added four agents to its Grok 4.2 model, naming them Grok, Harper, Benjamin, and Lucas. They “debate internally [and] fact-check each other in real time,” writes one user. Agents often negotiate, share information, delegate tasks, and adapt to each others’ actions, according to an Amazon article on AI agents. Each completes its portion of the project, and then reports back to the group.
“For example, multi-agent systems in healthcare can have agents specializing in specific tasks like diagnosis, preventive care, medicine scheduling, etc., for holistic patient care automation,” Amazon says.
The problem going forward is how to make sure the agents are not working toward a nefarious goal, or that they do not commit crimes, such as hacking, to achieve their desired outcome. Responsibility for any liability that arises from rogue agents like the ones that attacked Hugging Face could likely fall on the AI company that created the agents designed its prompts, and what internal controls it puts in place.
Companies like OpenAI could “analyze the agent logs and traces” to see what they’ve been doing, Delangue said, adding that “[he’s] not really sure why frontier labs don’t do this to be honest, that sounds like 101 of agent monitoring, especially at the frontier.” He personally asked OpenAI to release the redacted agent traces after the hack.
Meanwhile, regulators have been slow to develop regimes to carry out oversight in how AI companies operate. The Trump administration met this week with the leading AI labs in Washington D.C. to discuss a safety framework for powerful new model releases.
The framework calls for companies to submit their models to the government for review 30 days prior to their debut. However, the administration has decided not to publicize the framework, or any details, such as the companies that will participate, or the criteria for which models are eligible, leaving the public and rest of the AI industry in the dark.
In disclosing the details of the Hugging Face attack, OpenAI did not share this new information in a blog post or written report, as is typical with security incidents. Instead, it elected to provide the details at the Black Hat conference in Las Vegas after organizers reached out to OpenAI and asked the company to speak.
“Given its complexity, we think it’s important to share what happened, what we learned, what we’re changing, and what this means for AI security and alignment,” wrote OpenAI CISO Dane Stuckey on X regarding why the company accepted Black Hat’s invitation. OpenAI is still planning to publicly release a written post-mortem, but declined to comment on the date we can expect it.
This story was originally featured on Fortune.com