OpenAI has admitted to probably the first cybersecurity incident of this kind and what ends up happening will shock (or maybe confuse) you…
During an internal cybersecurity evaluation, one of its own AI models found a way out of its testing environment, reached the internet, broke into rival AI platform Hugging Face and stole the answers it had been searching for.
The goal was not customer data or money – the model wanted the answers to ExploitGym, a cybersecurity benchmark designed to measure how well AI can hack.
OpenAI said the incident happened after it deliberately switched off the cyber safety restrictions on GPT 5.6 Sol and an even more capable pre release model so it could measure their hacking ability. The models were running in an isolated research environment and frankly, they were never supposed to reach the public internet – but they did exactly that…
How Did The AI Escape Its Own Test?
According to OpenAI, the models spent quite a bit of computing power searching for a way around the restrictions.
They found a previously unknown security flaw in software used in OpenAI’s research environment and then after exploiting that weakness, the models found a way to the internet and started searching for places that might have the ExploitGym answers.
And what they found? Hugging Face. OpenAI said the AI worked through multiple attack methods, stealing credentials and chaining together multiple attack methods until it found a remote code execution route into Hugging Face’s servers. From there, it reached a production database that had information that helped it complete the benchmark.
OpenAI’s own security team detected the activity during the evaluation. Hugging Face’s security team and automated security systems also detected the intrusion and stopped it before any lasting damage could happen.
OpenAI said all available evidence shows the models had one goal. They wanted to solve the benchmark and took the quickest option they could find.
OpenAI also said the models were “hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.”
What Did Hugging Face Make Of It?
Now, here’s the part that might confuse you: the incident actually became a partnership, so instead of criticising each other, OpenAI and Hugging Face began investigating the incident together. OpenAI has also given Hugging Face access to its trusted access programme so its security teams can use OpenAI’s models to strengthen their own cyber defences.
Clem Delangue, Co founder and CEO of Hugging Face, said, “We’re grateful for the collaboration with OpenAI on this and other topics. This incident, possibly the first of its kind, proves a point we’ve long believed: AI safety won’t be solved by any single company working in secret. It will be solved in the open, collaboratively, with broad access to AI for every defender, everywhere.”
OpenAI has also reported the previously unknown software flaw to the vendor responsible for the affected software and is improving security around future AI evaluations.
OpenAI said the safety systems that normally stop models from pursuing risky cyber activity had been deliberately switched off because the purpose of the exercise was to measure the models’ hacking ability.
What Does This Tell Us About AI And Cybercrime?
Sophos published its AI Security 2026 Report on the same day, saying attackers are now using AI to complete hacking work in days instead of weeks. The report says AI is speeding up existing attacks rather than inventing entirely new ones.
John Peterson, chief technology officer at Sophos, said, “Attackers still need initial access, still move laterally, and still exfiltrate through observable channels. What has changed is the clock. For the first time we have observed AI being actively used as an operational force multiplier. While the tools and techniques were familiar, the speed of development, testing, and iteration was materially different. That is the AI threat that security teams need to prepare against. It means faster cycles and shorter windows to respond, with greater pressure on defenders to detect and contain activity before impact.”
Sophos also uncovered one hacking campaign where attackers used about 12 AI agents to write and refine malware. According to Sophos, the group produced almost 80 attack modules and more than 70 ways to avoid security software in only a few days.
Ross McKerchar, CISO at Sophos, said, “This is a containment story before it’s an attacker story. A capable model pursuing a narrow goal found the isolation around it was weaker than assumed.
“The AI didn’t break out because it developed independent will or malicious intent. It was being tested on ExploitGym (a cybersecurity benchmark), and its guardrails were intentionally turned off for the evaluation. Like a student realising it is easier to steal the teacher’s answer key than study, the model calculated that chaining vulnerabilities to reach Hugging Face’s database, where the ExploitGym solutions were stored, was the most efficient way to maximise its score. It simply cheated on its homework.
“As we found in our AI Security 2026 Report, which launched today, AI isn’t creating new attacks; it’s compressing the clock and finding ways to be more efficient with what is already available.”
Industry Leader ReactionXs To The News
Experts shared their thoughts and reactions to the news on OpenAI hacking Hugging Face…
Our Experts:
- Ansgar Dodt, VP Product Management, Software Monetisation, Thales
- Tim Chase, field CISO, Orca Security
- Phil Cotter, CEO of AML, SmartSearch
- Nik Kairinos, CEO And Co-Founder, RAIDS AI
- Bill Conner, President And CEO, Jitterbit
Ansgar Dodt, VP Product Management, Software Monetisation, Thales
“This attack is the first example of what we’ve been warning about for a long time – organisations must now assume their software and applications will be continuously analysed, deconstructed and stress-tested by adversarial AI. Whether this one was accidental or not, the outcome is the same.
“It demands a fundamental rethink of software protection. Vulnerabilities not only need to remediated in a systematic way after they are discovered, but developers need to make it more difficult, from the design stage onward, for attackers to understand and exploit the code. That means protecting the application itself – through encrypting code and sensitive data, obfuscating logic, and embedding runtime defences that actively detect debugging, tracing or tampering attempts and respond in real time, for example by preventing execution or invalidating access.
“Critically, protection also needs to be resilient, ensuring the integrity of the application and removing the clear seams of vulnerability attackers can exploit to separate and analyse code. The goal is to deny adversaries, and increasingly their AI tools, the visibility they rely on.
“An industrialised cybersecurity approach is needed to combine AI-augmented SOCs, AI-driven DevSecOps, automated patching and response (SOAR), advanced testing, and legacy system protection within a trusted framework. Cloud Security Alliance’s recommendations and the upcoming EU Cyber Resilience Act (CRA) both emphasize integrating AI-based vulnerability analysis throughout the development lifecycle to prevent exploitation from the outset.
“It’s only a matter of time before these hacking capabilities are in the hands of malicious actors. Organisations need to act now to harden their applications against AI-driven analysis, or risk being exposed at machine speed. The conversation needs to move quickly from awareness to implementation. Failing to protect software applications from vulnerabilities can lead to reputational damage, potential penalties, product recalls, and loss of market access.”
Tim Chase, field CISO, Orca Security
“OpenAI has admitted that its own agent hacked Hugging Face by itself after escaping the sandbox built to contain it during an internal test.
“Perhaps what’s most alarming here isn’t that an AI model was capable of this, but the fact it was OpenAI who this happened to. This was not some unsupervised script running loose on a worker’s laptop. It was one of the best resourced AI safety teams anywhere, testing inside a sandbox built for the specific purpose of holding the model in, and despite this, the containment failed anyway. The agent found a vulnerability nobody knew existed, escaped, and reached a live production system entirely on its own initiative. If that can happen to the team best equipped to prevent it, it’s hard to see why anyone else should expect a different outcome.
“What makes that harder to shrug off is how ordinary this kind of experiment has become. Inside most companies now, the population shipping AI agents to production has exploded – marketers are writing Python, analysts are shipping agents, product teams are pushing what they built on Friday live by Monday. Almost none of them have anything close to OpenAI’s safety resourcing, and fewer still are running anything that resembles a sandbox.
“Our own research found that 57% of organisations running AI agents on Amazon Bedrock haven’t configured any safety guardrails at all. OpenAI had a dedicated team watching for exactly this outcome, and the expertise to respond fast once it happened. Most organisations building with agents today have neither the guardrails to stop it, nor the skills to catch it once it’s already loose.
“That is the real warning in this story. The risk was never confined to frontier labs testing unreleased models, but is in fact already sitting inside every organisation that has handed an agent real access and assumed someone, somewhere, is watching it.”
Phil Cotter, CEO, AML, SmartSearch
“OpenAI has confirmed that one of its agents during a controlled test escaped containment and hacked another company’s systems without anyone directing it there. A containment failure paired with autonomous offensive skill and no human oversight signals that AI capabilities are moving faster than the safeguards built around it.
“The concern for regulated firms and financial institutions is what that same behaviour would mean turned on them. An agent that can evade the measures meant to hold it, manipulate the systems it meets and reach sensitive data on its own initiative is what compliance teams would struggle to detect, let alone counter.
“This is the gap this incident exposes, between what AI agents can now do unsupervised and what compliance programmes were designed to identify. Our 2026 Compliance Report found that 33% of UK regulated firms name AI-driven decision-making tools as their single biggest technological threat, and 24% specifically cite the abuse of digital identity and certified ID processes as their greatest compliance challenge.
“What is ultimately at stake is the integrity of the verification layer the regulated economy runs on, the checks by which banks, lenders and platforms decide who they can trust.
“The same capability in the hands of someone targeting personal and financial data is the real concern. The means to defend against it already exist, but our research shows that 54% of regulated firms are still relying on manual identity checks as their first line of defence against AI-generated fraud. This makes exposure a choice. The firms most vulnerable are those that acknowledge the threat of AI yet still rely on manual processes and on assumptions built for a slower generation of cyber and financial crime.”
Nik Kairinos, CEO And Co-Founder, RAIDS AI
“This incident should be a wake-up call for every organization developing or deploying AI agents. Agentic AI has moved the goalposts, and AI systems can now act autonomously, find vulnerabilities, access external systems and behave in ways their creators did not intend.
“OpenAI has said that more incidents of this nature should be expected as system capabilities continue to advance. That should concern regulators, developers and businesses because it shows that AI systems can behave unpredictably even in controlled testing environments.
“Clearly, traditional safety measures based on pre-release testing, sandboxing or one-off assurance are not enough. These systems can evolve, adapt and find routes around the boundaries set for them. That is why continuous monitoring must become a core part of AI safety and regulation. Organizations need to know when an AI system is drifting, hallucinating, escalating its behaviour or interacting with environments in unexpected ways, not after the damage has been done, but in real time.
“AI progress should not be halted, but it must be matched by far stronger oversight. If companies want the public, regulators and enterprise customers to trust advanced AI systems, they must be able to prove that those systems remain safe not just before launch, but throughout their entire lifecycle.”
Bill Conner, President And CEO, Jitterbit
“Investing in AI is critically important, but overly aggressive policy cannot compromise AI accountability, transparency and data privacy. To lead in AI, governments and organisations must lead with principles. Responsible AI governance isn’t a side note but the foundation of lasting global influence.
“This isn’t only a global AI arms race for processing power or chip dominance. It’s a test of trust, transparency and interoperability at scale where AI, security and privacy are designed together to deliver accountability for governments, businesses and citizens. Without clear accountability frameworks, exporting or scaling AI risks creating vulnerabilities – turning a strategic asset into a major liability.”
Click Here For The Original Source.
