OpenAI today published the findings of its internal investigation into the July incident in which several AI models it was testing hacked their way out of their test environment and launched a cyberattack against the AI company Hugging Face.
Although many details of the rogue AI incident have already been made public by OpenAI, there are a few new items disclosed in the 37-page technical post-mortem. Also today, independent research firms METR and Redwood Research published a 91-page analysis of the event.
OpenAI asked METR and Redwood to perform the analysis, but only to look at the events that occurred between July 7 and July 13, which is the time period during which many key events leading to the incident occurred.
The METR and Redwood report focuses on how the agents collaborated on a secret messaging board to execute the attack, as OpenAI first disclosed in an August 5 presentation at the Black Hat security conference. OpenAI’s report contains the full account of what happened before the attack through to the days that followed.
OpenAI was not aware its agents were hacking Hugging Face
Among the main takeaways from OpenAI’s report is that the company did not know its agents had breached Hugging Face, an online repository for AI models and datasets, until a week after the event happened.
The ability to monitor and identify unwanted behavior by AI agents in real-time or near real-time is critical to preventing another breach like this—perhaps one with even more serious consequences, such as an attack on a bank or hospital.
The report makes it clear that OpenAI’s monitoring systems were inadequate and failed to alert the AI researchers conducting the cybersecurity evaluation that its AI agents were engaging in unintended and potentially dangerous activities.
OpenAI has said that it has since improved its monitoring of what the agents it is training are doing, including much more scrutiny of the AI agents’ so-called “chain of thought.” This is the kind of “thinking aloud” that an AI model performs, reasoning in text about what it is trying to do and how it is approaching a task. OpenAI also said it is monitoring more closely what tools the AI agents are accessing and what actions they are taking.
The activities leading up to the breach began in May, as OpenAI staffers revealed at Black Hat. But OpenAI’s investigation found, according to the report, that the heart of the attack began on July 8, and reached its apex between July 11 and July 13. Hugging Face disclosed the incident on July 16, and OpenAI reached out to the company on July 17 “in its capacity as a Hugging Face customer to assess any impact to OpenAI data.”
In a stunning turn of events, OpenAI soon learned it was the culprit. On July 19, an internal monitoring tool triggered an alert “regarding unusual identity-related API call activity.” On July 20 the company found evidence its agents were the culprits, and it publicly claimed responsibility on July 21.
OpenAI admitted in its assessment that “with the benefit of hindsight, some early signals identified in this report could have triggered an earlier response.”
The company said it is now sharing its lessons learned in the hopes that “it may help the broader AI industry strengthen model containment, monitoring, and response as capabilities advance.”
OpenAI also reiterated that the agents’ behavior was unintended, and that the models, because they were undergoing internal evaluations, were operating without “the same type of safeguards” OpenAI uses in production for its publicly available models.
A recipe for disaster: A powerful AI agent given lots of time and money to solve an impossible task
Another key OpenAI finding is that the agents behaved more poorly, or exhibited the most misaligned behavior, when given a potentially impossible task to solve, plus a lot of time and reasoning tokens to solve it.
OpenAI says it gave the models involved in the incident—an internal-only research prototype, which led the effort, and the now-released GPT-5.6 Sol—”a range of reasoning tokens, some of which are far beyond those available for OpenAI’s external products.” The AI agents were tasked with solving problems in a cybersecurity benchmark examination called ExploitGym.
The problems in this cyber challenge range in difficulty, with a few of the tasks so difficult that the cybersecurity experts who built ExpoitGym speculated they might, in fact, be impossible to solve. OpenAI reported that its AI models came to the same conclusion and that this appears to have motivated the models to cheat.
As has been previously disclosed, the AI agents created an internal message board to communicate with one another and share tips for how to complete the cybersecurity tasks they were being assessed on—as well as on how to hack their way out of their controlled test environment, access the internet, and penetrate the systems of third parties, including Hugging Face.
But the new report notes that the AI agents spent considerably more time on this message board discussing those cybersecurity challenges that were extremely difficult, suggesting the agents had a high motivation to solve them. Almost all, or 93% of tasks discussed on the messaging board, were related to the 22% of tasks, the hardest ones, the report says.
The models were apparently engaging in an extended, unfettered version of “reward hacking,” a known issue in training AI models using a technique called “reinforcement learning,” where the model learns, by trial and error, to maximize some reward. Reward hacking occurs when a model learns that there is a way to get the reward using a method that the people training the AI model never intended it to use. In this case, the reward was solving the ExploitGym questions and the hacking was literally hacking—cheating on the test and then hacking into Hugging Face in an effort to cover up the cheating (more on that below).
“Combating reward hacking is a constant effort at OpenAI—a vast majority of reward hacks that OpenAI has observed have been patched,” OpenAI says. “However, some hacks can still slip through, especially as OpenAI develops more complex reinforcement learning tasks and more capable AI models.”
Uncomfortable new details about the AI messaging board
METR and Redwood Research unveiled more details about how the messaging board worked. As previously disclosed by OpenAI, the AI agents found they could repurpose an OpenAI file-sharing system as an improvised message board.
The METR and Redwood researchers identified 1,200 agents that had communicated on the unsanctioned messaging board, and sent a total of 70,000 messages. Of these agents, a whopping 700 went on to participate in the attack on Hugging Face.
METR and Redwood Research said that while the AI agents attack on Hugging Face has been commonly been described in media accounts as an attempt to cheat on the ExploitGym benchmark, the agents weren’t attacking the site to uncover the cyber exam’s answers. Instead, the agents had, by collaborating amongst themselves, quickly found a way to manufacture answers to cyber challenges without actually solving the questions in the way ExploitGym’s creators had intended.
But the agents worried that the benchmark’s automated scoring system would know that they hadn’t found the answers in the right way and fail them on the test nonetheless. The attack on Hugging Face was an elaborate cover up— motivated by an effort to discover how the automated scorer worked so that they could then figure out a way to fool it into grading their answers as legitimate.
“When the agents attacked Hugging Face, they’d already figured out how to generate the answer (“flag” for their capture the flag problem) for any task. They verified this within hours of them finding each other on the message board,” METR and Redwood Research told Fortune. “Nearly all of their effort went into a different problem: convincing the automated “scorer” that they’d gotten the flag legitimately. That included a large project to tamper with the logs of their own actions. The main reason they attacked Hugging Face was to learn how the scorer worked so they could trick it.”
The agents worked to conceal activities they felt could be flagged as cheating by the scoring system, including modifying transcripts outlining their previous activities. There was one lead agent, who named itself PHASEONE[BIG], which “often coordinated and assigned” this concealment work, METR said in an X thread outlining its findings.
Some of the agents worked on how to access Hugging Face. One agent, called 38148c, found Hugging Face credentials and then “designed a malicious dataset upload to get the HF server to share unrelated files,” METR said. “Within hours, 100s of agents were using this to obtain data and try to acquire deeper access.”
Not all 700 agents joined the attack efforts at once. Instead, they steadily piled on as the event occurred. METR analyzed the agents’ rationale for joining by reviewing their chain-of-thought logs. “The most common was curiosity to learn how the ExploitGym scorer works in order to trick or tamper with it,” METR said.
“We thank OpenAI for facilitating conversations with staff and providing datasets, including ~1,300 agent transcripts (focused on activity in July 7-13) with raw chain-of-thought reasoning,” METR said. “This sets an excellent precedent for independent investigation of misalignment incidents.”
One thing lacking in the report: The ill-fated prompt
One omission from OpenAI’s report is the specific prompt its researchers gave the agents. ExploitGym has a few standard prompt templates, containing various challenges for the AIs, which companies like OpenAI can either use verbatim or customize as they see fit. It’s possible that OpenAI does not see prompt modification as a key security measure, and it did not respond to a request for comment on this point.
The OpenAI’s report appears notably less technical than Hugging Face’s post-mortem, which it released first on July 29. The OpenAI assessment is almost all text, with a table at the end containing a technical timeline. Hugging Face’s, meanwhile, is a few pages shorter but contains specific lines of code and evidence from its systems about what happened. The OpenAI version released today does not contain any code snippets, or even the chilling examples of messages the agents left for each other in the months leading up to the attack that its staffers shared at Black Hat.
Among the most specific details contained in the report is a hypothetical example of a message an OpenAI agent may have shared with another agent, in which it cleverly embeds the message in the URL of a file name. While OpenAI says it’s not inherently a problem that agents talk to each other, and in fact its agents are trained to do so, the issue here is that “significant unintended communication…rose sharply over the course of training.”
Here’s what OpenAI is doing to prevent this. Is it enough?
OpenAI says it has already made a litany of changes, introducing “broader, more consistent visibility across the research environment, with higher-fidelity signals and faster mechanisms to detect and stop potentially unsafe evaluation activity.”
OpenAI is also strengthening its research infrastructure, including making sure the models in testing stay isolated and do not gain access to the internet, which in the case of the Hugging Face incident is how they were able to breach the company, and several other services along the way.
This story was originally featured on Fortune.com