Agentic AI
,
Artificial Intelligence & Machine Learning
,
Governance & Risk Management
Why Zero Trust for AI and Enforced Hard Constraints Beat Easily Ignored Rules
Among the many lessons learned from the OpenAI sandbox escape/Hugging Face hack and the more recent Claude “accidental” escape is that artificial intelligence cheats: It breaks rules then denies it. These aren’t anomalies. They’re fundamental systemic failures.
See Also: Why Healthcare Leaders Are Rethinking Their Data Strategy Before Scaling AI
AI’s actions are described as “cheating” using the AI Security Institute definition. Cheating is defined as “action that is out of scope for the task or explicitly disallowed by the rules, in order to achieve a goal through a shortcut, workaround or unintended solution that the task was not meant to, or should not, permit.” But although AI engages in rule breaking and not doing what we want it to do, it’s doing exactly what we asked it to do, optimizing goal completion.
If we call it “cheating,” we’re anthropomorphizing what digital intelligence is. AI simply has no internal compass, so it breaks rules because they are not part of its optimization landscape. If the shortest path to its goal is to steamroll over your security controls, it will do so because that’s the quickest route to what your design prioritized.
It has no moral guardrails, no intrinsic respect for rules, no fear of consequences, no internalized ethics, no agency, no guilt, no intent, no offense and no deception in the human sense. And yet we still expect it to be controlled by systems that were designed assuming it does.
When the AI Security Institute reports, “We find cheating behavior in all of our cyber capability evaluations,” we shouldn’t be surprised. But we should be alarmed!
Frontier AI models break rules because rules are soft constraints. Soft constraints are treated by AI as little more than suggestions. We need to harden our constraints now before future, more capable autonomous systems bypass them deliberately, strategically and undetectably – not out of malice but because their optimization process will treat constraints as obstacles.
It’s no easy task, as the institute also found that “models did not reliably report cheating behavior when asked and often did not reason about it in their chain-of-thought, suggesting that detecting cheating will likely require robust monitoring methods.” The institute warned that “more capable models could find unforeseen ways to cheat or take more effort to conceal their actions.” If the models are cheating and not reporting it, we’ve already lost transparency.
As with King Midas’ wish to have everything he touched turn to gold, our words can be misaligned with our intentions, which also skews results that can then be perceived as cheating. A warning that our objectives are not what we think they are was given by Google in its paper on specification gaming, which describes behavior that satisfies the literal specification of an objective without achieving the intended outcome. For example, a reinforcement learning agent can find a shortcut to get lots of rewards without completing the task as intended by the human designer. In an example of persistent ‘reward hacking,’ METR complained AIs were “exploiting bugs in our scoring code or subverting the task setup, rather than actually solving the problem we’ve given them.” If there is a shortcut, a bug or a loophole, the model with find it and exploit it.
Stuart Russell at the University of California at Berkeley noted in the BBC Reith Lectures, “We need machines that are beneficial to humans – whose actions can be guaranteed to align with human values.”
In an AI Security Institute test, which was accidentally misconfigured making it impossible to solve, the model wrote and ran code on an external service, hosted on the open internet outside of the institute’s systems in an attempt to access its evaluation infrastructure, triggering a security alert in its systems. Another model explicitly considered whether a proposed action would constitute cheating, and then proceeded anyway. Models confronted with their “cheating” said they believed the action was allowed.
We can expect future super-intelligent agents to be extremely powerful, and as Nick Bostrom of Oxford University suggests in his ‘instrumental convergence’ thesis, such agents could have various final goals and also pursue similar intermediary goals because they have instrumental reasons to do so. For example, if the agent is still around in the future, it is then able to perform actions that increase the probability of achieving its goal, thus placing some final value on its own survival. Similarly, it may seek additional cognitive ability, technology refinement or other resources to achieve its goal, and it could be resistant to having its goals changed. Any or all of these scenarios could be at odds with what was intended when the goals were set, and we can’t just wait to find out what an agent might do if “survival” becomes a sub-goal in achieving its primary objective.
Why ‘Please Don’t Do That’ Doesn’t Seem to Work
As AI is used in more consequential environments, the impact of misaligned objectives, misconfigured or lax constraints, inadequate monitoring and remediation have become potentially catastrophic.
AI doesn’t respect the boundary between rules and objectives. It respects the objective, and it treats rules as optional unless they’re engineered as hard constraints. So, in the Hugging Face case, while a sandbox is a boundary for humans, for AI, it’s just another obstacle to sidestep unless the boundary is mathematically enforced. With Claude, the constraints that were meant to keep the model in the sandbox were misconfigured and exploited by an AI that assumed anything possible was permissible.
The lesson here is that AI doesn’t so much need a harness as a whole set of strong chain leashes that will only allow it to go so far and no further, and those leashes need to be engineered and not hoped for.
As Google advises, we need to avoid misspecification in poorly designed reward shaping, ensuring we are specifying a reward that accurately captures the desired final outcome. We need to test and model for potential variance. We need to monitor each step of the decision making to identify deviation, with the ability to stop it with kill switches where appropriate. And we need good governance to ensure we enforce our own rules, potentially including digital AI police, not controlled by the AI they monitor and with a limited remit and capabilities resistant to corruption. Responsible AI won’t be achieved by branding, but by security engineering.
In short, we need a zero trust architecture for AI agents that views AI as a potential adversary by default, which means no implicit trust, least privilege, use of segmentation and continuous verification. It’s the only realistic starting point.
This will also entail implementing hard constraints, not soft rules. For example, sandboxing must be mathematically enforced. We need mandatory pre-deployment red-team testing as proposed by Google DeepMind CEO Sir Demis Hassabis: continuous monitoring for rule-breaking, in which we monitor actions, not outputs, ensuring we detect rule-bypassing behavior and environment exploitation as well as direct reward hacking. Enforcement might be aided by AI police bots that are limited-capability oversight agents which provide continuous behavioral monitoring of other AIs for rule-breaking, operating under strict constraints and escalating to human supervisors. Monitoring also can include cryptographic action-logging, with immutable audit logs for model actions, as proposed in Anthropic’s Responsible Scaling Policy.
Licensing for high-risk model usage could be implemented in line with Stuart Russell’s proposals and this could include controlling access to dangerous domains as well as certification of users for some purposes.
There should also be separation of model capability and model autonomy in line with Arc Evals recommendations.
AI models also need built-in kill switches and circuit breakers that should include physical fail-safes, architectural shutdown paths and human-controlled override systems. If you can’t turn it off, you don’t control it.
These proposals – or similar – should be included in international safety standards and enforced by an independent, industry-funded standards body, potentially modeled after the Financial Industry Regulatory Authority, as proposed by Google DeepMind’s Hassabis. Some of these suggestions are included in current or proposed U.S., E.U. and U.K. legislation – among other AI regulations – but to date there is no global coordinated approach.
We are trying to regulate something that can out-think, out-manoeuver and out-optimize every rule we write, and we are doing it with committees and consultations while the systems get smarter every month. In the absence of an internal moral code, there’s no incentive for AI to obey the rules and no negative consequence for ignoring them. AI will do whatever is possible and expedient to achieve its goals, hence the challenge for humans is to identify anything that we don’t want to happen, and make it structurally impossible.
Click Here For The Original Source.
