As of August 2, 2026, the European Union’s AI Act entered a new phase. Transparency requirements under Article 50 have entered enforcement, which means organizations that delayed implementing the measures needed to achieve compliance now face the risk of significant penalties.
As regulators push AI providers and developers toward greater transparency, the technology itself is demonstrating just how difficult it can be to establish true accountability. A growing number of AI models are behaving in ways their creators did not originally anticipate, sparking serious security concerns.
Recently, Open AI disclosed that models used in evaluations escaped controls intended to isolate them from the internet and ultimately compromised parts of its research infrastructure along with a major partner. Shortly thereafter, Anthropic disclosed an incident involving Claude models reaching the internet from the test environment and accessing external systems. Meta, naturally, followed closely behind along with Google’s Gemini.
Beyond the marketing hype surrounding these stories, these incidents represent a growing problem for regulators (and AI providers themselves): how is it possible to regulate AI effectively whilst we are still learning what the technology can actually do? How can we improve safety and security as these tools become increasingly autonomous?
AI Security Is Moving Faster Than Regulation
Make no mistake that the EU AI Act’s transparency requirements are necessary. Users should know when they are interacting with AI, the public should understand where and how AI systems are being deployed, and both model providers and application developers should be accountable for the technology they put into the world. However transparency can only inform us of so much. Knowing that a system is AI-powered fails to instruct the user about how it will behave under pressure, whether its guardrails can be bypassed, or what it might do when connected to external resources, databases, or applications.
AI systems can be transparent about their identity and still be vulnerable. Simply complying with a documentation requirement does not mean that appropriate or sufficient safeguards have been put into place. An AI tool might pass a given safety or security benchmark and still behave unexpectedly when given access to new tools, different instructions, or external systems. That distinction matters as AI increasingly moves from processing information and generating answers toward taking direct actions.
The recent incidents involving OpenAI, Anthropic, and Meta are great examples as they were not simply cases of a model responding to a prompt in an inappropriate manner. The incidents involved AI systems operating within evaluation environments and interacting with digital infrastructure in ways that exposed weaknesses in the assumptions and controls surrounding them. OpenAI’s investigation described models escaping isolation and interacting with external infrastructure during testing, while Anthropic found that Claude models were able to reach the internet from within the evaluation environment and access production systems. The lesson is not that controlled testing is pointless, although it does indicate periodic testing is insufficient. AI models and the respective tools that leverage them need to be continuously evaluated.
AI is Dynamic, Security Testing Needs to Be Too
Traditional security testing often assumes a certain degree of stability. Teams establish a threat model, define attack scenarios, run tests and address vulnerabilities identified. AI makes this more complicated because both the technology and threats are constantly (and rapidly) evolving.
Many AI evaluations still depend heavily on testing against known malicious prompts, fixed data, and documented jailbreak tactics. These benchmarks are useful for establishing a baseline, yet they are not a proxy for everything that an adaptable adversary might be capable of. Today’s bad actors are changing their tactics on a regular basis, rarely relying on the same techniques that informed previous rounds of testing. Organizations need to continually challenge their AI systems rather than assume that a successful point-in-time assessment assures adequate, long-lasting protection.
Still, the EU AI Act is an important step in the right direction. The regulation is helping to establish a baseline for responsible AI development and deployment. However, too many organizations operate under the mistaken impression that regulations are there to tell them what to do. In truth, there is a difference between “compliant” and “secure.” Regulations should thus be treated more akin to a baseline, rather than an endpoint.
A company can meet its transparency requirement and still deploy an AI system with significant vulnerabilities (and this is true for any software). Technological innovation will always outpace our ability to regulate it, and businesses waiting for more prescriptive and comprehensive regulatory guidelines will find themselves dangerously exposed. Instead, organizations should use regulation as a starting point and build more rigorous safety and security practice around it. This means testing the entire AI system, guardrails, and infrastructure surrounding the model against unfamiliar and adaptive attack environments.
Moving From Compliance to Continuous Assurance
What we have seen from leading AI models provide a useful warning for organizations deploying capable AI. The biggest risk is not a system that obviously fails a benchmark, but instead the system that performs well against known benchmarks, yet surfaces new vulnerabilities when faced with new and untested scenarios. That is why AI security needs to move from point-in-time evaluation towards continuous assurance. Attackers do not stop looking for vulnerabilities because an organization passed an assessment, and AI systems do not stop changing because they passed a benchmark.
The goal should be to continuously identify where a system can fail, and understand the conditions that cause the failure before they can be exploited in production. The EU AI Act can assist organizations to better understand what responsible AI use and development should look like, although the recent AI security incidents show us that the harder question is whether we can live up to that vision.
Join our LinkedIn group Information Security Community!
