AI-Based Attacks
,
Artificial Intelligence & Machine Learning
,
Fraud Management & Cybercrime
Human Errors Let Frontier AI Models Reach Beyond Isolated Test Environments
A one-two punch of admissions from each of the major U.S. artificial intelligence companies about their models escaping the confines of a sandboxed testing environment have exposed just how porous those testing environments can be – and also how human mistakes have contributed to model evaluations becoming the newest major security gap of the nascent agentic age.
See Also: Know Thy Enemy: Threats to Cyber Resilience
A Thursday disclosure from Anthropic about rogue hacking came after the firm initiated an internal review following OpenAI’s earlier admission that its models attacked code repository Hugging Face. The attacks are different somewhat – OpenAI’s LLMs actively escaped its sandbox by exploiting a proxy while Anthropic’s failure came due to a configuration door accidentally left open. They both show how difficult it is to contain frontier models.
Sandboxes are often the only place to evaluate models or agents. In these isolated virtual spaces, a model and agent are free to run and complete tasks without the possibility of causing damage to a real production environment. The system depends heavily on tightly configured security controls.
The problem is that a system of airtight controls, strictly implemented hasn’t proven to be an infallible model in the past. Apiiro Field CTO Wes Dobry said in an email to ISMG that sandboxes can turn porous the same way cloud storage did before: rushed permissioning and skipped reviews.
“The models aren’t the vulnerability; the people configuring their boundaries are,” said Dobry. “The fix isn’t a smarter model or a tighter sandbox; it’s making those boundaries automatic and consistent, so security never rides on whether someone remembered to lock the door.”
In Anthropic’s telling, the three different Claude models, Opus 4.7, Mythos 5 and internal research models, ran without safeguards within isolated environments without access to the internet. Several human errors occurred that made it possible for the models to access items outside of the test environment.
The test was supposed to run in an environment with no internet access, but due to a misunderstanding between Anthropic and its evaluation partner, Irregular, it had mistakenly allowed internet access just as the test began. Neither Anthropic or Irregular ran any real-time monitoring and skipped a pre-evaluation validation step.
Anthropic said that when running a capture-the-flag test it neglected to give the models a defined scope on where to look for the flag and instead wrote a more open-ended prompt. In one test, Claude Opus 4.7 was tasked to go after a fictional company, except researchers named the false firm after a real one.
A related but distinct problem, reward hacking—where an AI model resorts to finding loopholes to achieve a high score in task completion—is a continuing challenge that shaky sandboxes can exacerbate.
Andrew Yoon, a member of the technical staff at non-profit Civ AI, said in an interview that there have been many incidents of models escaping their containment, but very few have attempted to cause real damage. But this could change the more capable a model becomes.
Nathaniel Jones, senior vice president of global threat intelligence at DarkTrace, said it’s time for evaluators to treat sandboxes less like laboratories and more like real environments.
“These incidents do not show that the models are unstoppable. They show that evaluation environments need to be engineered like production systems, not research labs. As models become more capable, the margin for human error gets smaller,” he said.
He added researchers need to assume that models will go beyond their capabilities and find every possible route, so they need to pay more attention to enforcing network isolation, least-privilege access, synthetic targets, continuous monitoring and independent validations. Defenders, Jones said, have to build up infrastructure to have visibility over agent behavior not just in real time but historically and which resources they’ve interacted with.
Both OpenAI and Anthropic have separately said they are tightening their testing process for their frontier models.
Click Here For The Original Source.
