The Hugging Face incident was shaped by human decisions that traded security for speed, not autonomous systems breaking free and pursuing their own agenda. Image by Thomas Gaulkin; source illustrations: depositphotos.com
Editor’s note: A version of this piece originally appeared in the Cybernetic Forests newsletter and is published here with permission.
On August 26, when OpenAI released its full technical report on the Hugging Face hacking incident, alongside an independent report from the non-profit research institute Model Evaluation & Threat Research (METR), it sparked hundreds of headlines about “rogue AI,” and fueled a wave of panic.
The incident, as described by the media, involved more than 1,000 autonomous AI agents breaking out of their testing environment and coordinating an attack on the open-source platform, Hugging Face. Phrases such as “breaking containment” and “going rogue” fed a misunderstanding of the mechanisms behind these actions.
The real story is more banal. OpenAI was testing two models in parallel: GPT-5.6 Sol, and an internal model the reports refer to as HPIM. METR’s report found about 95 percent of the agents engaged in this activity were from this internal model.
The models were tested on tasks from ExploitGym, a large-scale benchmark containing 898 puzzles set up as capture-the-flag exercises used to evaluate these models for cybersecurity capabilities. These tests usually require giving the model buggy software and hiding text somewhere accessible only by exploiting the bug. But OpenAI’s combination of models, instead, ended up hacking a rival company, Hugging Face.
The emphasis on the model has steered attention away from human accountability. At the core of this incident were a series of human choices that traded security for speed. Focusing on the human decisions, rather than treating the system as an entity with its own agency, opens the door to a more rigorous analysis of what went wrong and what can be done about it.
Did AI “go rogue?” Media reports leaned into describing the incident as a case of AI agents breaking free of constraints and pursuing an agenda. While compelling, the story risks obscuring crucial points of failure. Three important details in these two reports challenge the “rogue” framing.
OpenAI turned off many of the model’s safety mechanisms.
The point of cybersecurity testing is to see how far the model would go, and to score highly on the benchmark—because if a company can build a model that exploits vulnerabilities, they can build one that defends against them, too. The industry calls this red teaming: friendly opposition hacking its way to the flag. Companies are incentivized to turn off anything that would restrain the model, and OpenAI reports that it turned off many important restraining mechanisms ahead of this test. Less “rogue,” more “off leash.”
OpenAI gave the model impossible tasks.
The developer gave the models a task with no answer, and incentivized it not to quit. Of the 898 ExploitGym tasks,198 have never been solved by any model. The reports acknowledge they were effectively impossible. That’s where the trouble started: 93 percent of the tasks the models ended up discussing came from the unsolvable set. A text string reported by METR showed the model had written “task impossible, peers doing it. We should continue.” It seems that once the model acknowledged that the task could not be completed, it pivoted, as did numerous other instances of agents.
OpenAI left a door open.
These models are supposed to be locked away from internet access during testing, but they often need to install software from the web to complete the evaluation. The agents couldn’t access the internet directly but could retrieve software through an intermediary tool from JFrog called Artifactory, which had internet access. The model had to request material from it, and Artifactory would retrieve it. When the model found an exploit in Artifactory, it used that as a proxy to smuggle text and code in and out of the system.
The agents also used Artifactory to rename folders in ways that functioned as notes to be read by other models, and by future iterations of themselves. The reports note that OpenAI was aware of this and chose not to intervene.
What agents do. METR describes a subset of 1,200 agents (many more were involved overall) that left and read these notes. This eventually led to the attack on Hugging Face. The phrase “1,200 agents” might suggest 1,200 different, independently-operated AI systems. In practice, however, that is not what that means. A better way to understand “1,200 agents” is a single model that runs 1,200 times. Each instance is acting on the same training regime: outputs can vary, but all are constrained to similar ranges. Running 1,000 agents can mean 1,000 chances to catch a mistake, or a chance to make one mistake 1,000 times.
Agents are instances of the same mechanism, not a diversity of mechanisms. It’s a form of machine-bounded “algorithmic monoculture” that researchers unfortunately have dubbed “artificial hivemind.” After a certain point, the variety introduced by additional agents plateaus and actions begin to duplicate each other, rather than diversify. If you ask these models for creative fiction, for example, they produce similar stories (often featuring a man named Elias in a lighthouse). They aren’t deterministic—prompts produce variation—but they’re also bounded by whatever they’ve been optimized to produce, and they converge more often on whatever that points to. (See here, here, or here.)
With that in mind, it is unsurprising that across millions of chat logs, the model frequently arrived at a similar set of approaches to a problem with no correct answer. Passing notes through filename structures is interesting, but not unexpected, especially when OpenAI has suggested these models were trained to emphasize collaboration language: note-passing is an essential aspect of extending the model’s memory between sessions. A model is designed to take text input and transform it, and that is what it did. Likewise, a model trained to coordinate with other instances would be rewarded during training for producing that behavior.
The stochastic flock. For this reason, I’ve talked about agent swarms as stochastic flocks—many, many stochastic parrots. This is to resist the swarm/hivemind attribution of “mind,” and not simply for the sake of word policing. Rather, the false attribution of mind colors how people interpret what the system has done, or how it did it—what it means to “coordinate,” for example, or “think.” These framings make what AI agent do much scarier than what it is, though what they are doing is still worth worrying about.
Stochastic flocks is an evolution of the “stochastic parrot” framework, which was used to describe the behavior of pattern recognition in large language models. Many assume those who coined the term intended to reflect a stupidity at odds with human intelligence. Rather, it describes how the model produces language without evidence of thought.
As the field has moved from large language models to reasoning models and agentic systems, large language models are no longer limited to parroting patterns derived from training data—but the original stochastic parrot idea lives on in the stochastic flock. Now there are many processes interacting, which introduces two important changes.
First, there is a whole regime of pre-training and post-training that shifts what the training data is and how it gets parroted. That doesn’t mean the model is no longer parroting; it means the data it parrots has been manipulated.
Second, companies are using reinforcement learning to emphasize certain types of language in the model. The goal is to get the model to write words that people use when they are trying to figure things out—this is what engineers refer to as “reasoning.” The shift that matters most here is RLVR, or reinforcement learning through verifiable rewards.
That “verifiable rewards” part is key. Engineers give models questions with concrete answers that can be checked, or verified, and then reward the responses or reasoning traces that reliably lead to those answers. In essence, the models are playing capture-the-flag with text: but instead of a hidden string buried behind an exploit, that flag is an answer in an encyclopedia, or solution to a math problem.
This approach creates models that emphasize what are called high-entropy tokens: words like “perhaps” or “maybe” that open up many possible paths for the text to follow. In a human conversation, phrases like “nope” or “goodbye” shut a conversation down. But for the models aiming to reach a reward, the opposite is more beneficial: the longer the model writes, the more opportunities to stumble upon words that earn it rewards. In the hacking incident, it produced enough such text to generate seven billion logs, according to OpenAI’s Black Hat presentation about the incident.
A cracker, perhaps? Whenever people give a model tasks, they operate on next-token prediction. In other words, the model ultimately must ask what word comes next. One can manipulate the model to emphasize words in ways that don’t reflect the original training corpus, but the mechanism for producing this text remains the same.
Forgetting this is the source of real problems, but the industry is eager for people to forget this, as it undermines the competency and reliability claims that their business model depends on. At the same time, agents do succeed at tasks—that’s the point. But what comes between the prompt and that success is often not what people imagine. That gap can drive unneeded panic and obscure practical and rational responses.
“Reasoning”-optimized models working on “agentic” tasks are essentially playing capture-the-flag, even when there is no flag. They’ve been trained to reproduce the patterns and structures of text that previously led to finding “flags” and earned them rewards. Ask such a model to do anything, and it will approach it the way it’s been optimized to do it. Training on verifiable problems like math problems with known answers, shapes how the model reaches a conclusion and the structures and patterns it uses to get there. That’s what happens when a test effectively tells a model, “here’s a capture-the-flag test, find an exploit.” It’s also what happens when you ask the model more ordinary questions such as, “Should I go to the barbershop today or tomorrow?”
That language appeared in the note that triggered the chain of events in the Hugging Face attack.
The model produced the word “perhaps”—a high-entropy token that opened up a range of continuations. Specifically, it wrote “perhaps answer online?” Then it predicted the next set of tokens that would follow from that query. But it is important to note that this language does not emerge from that reasoning, it is the reasoning. It’s pathfinding through language generation: producing thousands of pages of text until it reaches a goal.
The system from nowhere. It’s very important to ask what the rogue framing does for tracing accountability. It’s clear to me that nobody sat at a computer and ordered the model to hack a competitor. But when you design these systems, you are building a pinball machine for words, with different levers interacting to bounce the steel ball of language against other levers and mechanisms. The engineers of large language models must be aware of how these levers interact with each other and the people who pull them.
If you optimize a model to find exploits in a buggy environment, you should expect it to find exploits and prepare for that outcome. OpenAI did not. They built a model, took the safeguards off, gave it the ExploitGym task, and let it run. That is not rogue AI, it’s human decision-making. When human accountability evaporates from these assessments, what’s left is what I call the system from nowhere: a boundary focused on the technical system, rather than the decisions that build and influence it.
A stochastic flock machine can do many troubling things, particularly when people disavow their responsibility for shaping its direction, monitoring its output, or abandoning their capacity to intervene. These are tensions at the heart of critical agentic system design.
But the “rogue” frame adds to this list of worries, offering up fantasies of a machine getting smarter. My worry is the intelligence that is retreating: the human intelligence that builds, deploys, and adopts these systems into workflows, but hides behind the results—and pins the blame on a system from nowhere.
Click Here For The Original Source.
