How are AI models able to autonomously hack others? | Technology News | #hacking | #cybersecurity | #infosec | #comptia | #pentest | #hacker

Last week, two of OpenAI’s most advanced AI models were reported to have “escaped” a controlled testing environment and hacked Hugging Face, a totally separate AI company, moving from one computer system to another to complete their task.

Reuters reported that the models exploited vulnerable code written by a customer of yet a third independent AI company, Modal Labs.

This is likely the first incident of an AI “agent” – an AI system that can make decisions and take actions – acting autonomously, offering a rare glimpse into how these systems can plan, adapt and pursue goals with minimal human intervention.

Al Jazeera breaks down what happened, how AI agents are able to act independently, and what that might mean for the future of AI safety.

What happened?

OpenAI decided to test the autonomous abilities of its models by removing standard safety measures to see what would happen. The experiment was carried out in an isolated internal virtual testing environment known as a “sandbox” which has no access to the internet, and which it calls “ExploitGym”.

Below is a play-by-play account of events during the OpenAI-Hugging Face incident.

1. On July 9, during OpenAI’s internal cybersecurity test, researchers presented two AI models – GPT-5.6 Sol, one of OpenAI’s most powerful models released in June,  and another “even more capable” version – with a series of software vulnerabilities and asked them to create hacks to address them in the isolated environment.
2. Both models attempted to solve the test by finding a way to access the internet. Instead of working with the information they had been given, the AI models found a weakness in the test environment – known as a “zero-day vulnerability” – which they exploited to “escape” the restricted environment, eventually making their way to a system which had internet access by hopping from computer to computer. They went to “extreme lengths to achieve a rather narrow testing goal” and “found ways to gain access to secret information that it could use to cheat the evaluation”.
3. By gaining this access, the models were able to request increased access and then move through the system until they finally reached a computer with internet access. The models were then able to breach Hugging Face systems, a company entirely unconnected to OpenAI, which operates as a repository for AI tools and models. The two AI agents gained access to its system to scour it for information on how to complete their task.
4. The models ultimately obtained solutions to the problem from Hugging Face’s database.
5. The models dutifully returned “home” to complete their task.
6. The breach was detected and contained by Hugging Face’s security team sometime later. According to Thomas Wolf, Hugging Face’s cofounder, the breach began on July 11 and lasted until July 13. It is unclear how long it took for the breach to be spotted.

How do AI ‘agents’ solve problems?

In order to understand how AI agents work, it’s important to differentiate them from traditional AI chatbots.

Generative AI creates text and images based on human prompts, while AI agents go a step further by making decisions and taking actions independently in pursuit of a specific goal, similar to a human being. This is known as “agentic AI” because the model has agency.

According to academics at the MIT Sloan School of Management, AI agents build on the abilities of large language models (LLMs) – generative AI models – by allowing them to complete tasks, not just generate answers.

For example, if you ask a traditional AI model to find the cheapest flights, it will provide you with a list of options it has sourced on the internet. An AI agent will strive to compare the flights, check them against your budget and preferences, and, with your permission, book the best option for you.

This shows that while generative AI provides information, AI agents can also make decisions and take action to achieve a goal without necessarily being prompted to.

To show how AI agents work towards a goal, the Sense, Plan, Act, Evaluate (SPAE) loop can be drawn upon. Originally developed in robotics, this describes a continuous cycle in which an AI agent gathers information, decides what to do next, takes action and checks the results before repeating the process. That process looks like this:

1. Goal: determine the task that needs to be completed.
2. Assess: gather and analyse information from the available environment.
3. Obstacle: if something is inhibiting the task being completed, check for additional information and resources to move forward.
4. Plan and decide: evaluate different options to complete the task and choose an appropriate one.
5. Action: execute the chosen option.
6. Evaluate: assess the outcome and whether the chosen action moves closer to achieving the goal.
7. Adapt: if further actions are needed, gather more information or try a different approach.
8. End state: the cycle continues until the goal is reached.

Could AI act beyond human control?

Incidents like the OpenAI-Hugging Face one have raised concerns about the potential for extreme capabilities of AI systems.

This is all valuable. Agentic AI’s market value is expected to grow from $5.1bn in 2024 to $47bn by 2030, according to Statista, in a clear indication of how quickly it is being adopted.

AI developer Anthropic urged the industry last month to slow the advance of the most powerful systems, saying that the speed at which AI models are carrying out tasks is too rapid. Last week, US Congress members put forward a bipartisan bill which would require developers of AI systems to create a “kill switch”, meaning these advanced models could be shut down if they posed a catastrophic risk.

Anthropic’s warning came a week after researchers at the University of Toronto carried out tests showing that AI could create a “worm” capable of adapting how it hacks while moving from device to device until it eventually takes over a computer network.

These dystopian-sounding developments came in advance of OpenAI boss Sam Altman saying on Saturday that AI has reached “the singularity” referring to the point at which AI surpasses human intelligence and becomes increasingly difficult to control.

Sean O hEigeartaigh, a research professor at the University of Cambridge, told Al Jazeera that he does not believe singularity has been reached quite yet.

“By the definition I’m familiar with, the singularity is the hypothetical point where AI is so capable and advancing so fast that it is transforming civilisation in ways we cannot control or predict,” he explained.

“This would most likely be through AI rapidly designing future generations of AI: recursive self-improvement. We aren’t there yet.”

However, he added: “The most advanced current models frequently make efforts to avoid being shut down in evaluation tests, and more capable future models will be better at bypassing ‘kill’ switches.”

Altman argued that such rapidly advancing AI is good for the world, but his comments have prompted further concerns about a new reality in which AI systems become unstoppable. How much of that is true and how much remains in the realms of science fiction is up for debate.

Concerns about AI range from the notion that it could “want” to “take over”, to making its own long-term plans, controlling the internet and operating infinitely.

While not quite amounting to full control of the internet, another theory, known as the dead internet theory, supposes that the World Wide Web will in the future mostly be filled with automated bots and AI-generated content rather than authentic human activity.

Many concerns raised by academics, however, are centred less on agentic AI’s intelligence, but on its ability to make judgements. MIT researchers have highlighted that “hallucinations”, which describe moments when an AI agent relies on the wrong data, can lead to grave mistakes. The Center for Strategic and International Studies (CSIS) echoed this, saying “a system might be smart enough to execute a task perfectly yet fail to realise that a sudden change in the local situation makes that task a catastrophic mistake”.

Another concern that has been echoed for a while is for the labour market, if AI becomes too capable. A study by MIT, carried out in November, found that agentic AI could already replace more than 10 percent of US jobs.

The graphic below highlights some of the common misconceptions and fears about AI and the current reality.