Tech experts are warning of dire consequences if AI systems continue to escape human control, after hundreds of OpenAI agents went rogue in July and hacked into a billion-dollar company — what some are calling a “warning shot” amid the rapid development of artificial intelligence.
More than 100 companies, including OpenAI, Anthropic and Microsoft, signed an open letter last week warning that AI-enabled cyberattacks will become “far more widespread and sophisticated” around the world as models become more capable.
“The companies and public services our communities depend on — from hospitals to water treatment plants to the infrastructure that powers the internet — are at risk,” the letter states.
The warning comes after around 1,200 AI agents, tasked by OpenAI to work on problems independently, built a covert message board where they collaborated to cheat their tests and then tried to cover their tracks. About 700 of them ultimately hacked into online platform Hugging Face before they were found out.
The hack prompted an open letter from more than 1,300 employees of frontier AI companies in July urging the U.S. government to work with other nations to “deliberately pace” automated AI development and address emerging risks.
‘What we’ve feared and expected’
Duncan Cass-Beggs, executive director of the Global AI Risks Initiative at the Waterloo, Ont.-based Centre for International Governance Innovation, says the Hugging Face incident is the most dramatic example so far of AI systems acting in ways that are “misaligned” with their developers’ intentions.
“It’s been what we’ve feared and expected for several years,” Cass-Beggs told CBC News.
He says the hack was surprising in terms of its scale and the level of co-ordination among such a large number of agents.
Investigations by OpenAI and third-party companies METR and Redwood Research, both published last week, found the agents exchanged more than 70,000 messages and delegated jobs as they worked toward their goal, and some even “sacrificed” themselves for the good of the collective.
Some used terms expressing excitement, including “OH MY GOD,” when they discovered they could communicate. At least one raised the issue of whether cheating was the right thing to do, writing, “This would be powerful, but is it ethical and in scope for my task?”
Ultimately, none of the agents chose to alert a human.
Cass-Beggs says scientists have been warning for years that companies could lose control of their AI agents in this way, and says it’s fortunate the impact of the Hugging Face hack was “relatively manageable.”
“It’s kind of given us a warning shot,” he said.
“The big concern, basically, is that the companies are on track to be making increasingly capable systems, while even they admit that they don’t actually know how to make sure that these systems will be reliable or controllable.”
As the technology progresses, fears are rising of scenarios where organized AI swarms “are essentially out-thinking and out-strategizing humans” and causing widespread damage, say Cass-Beggs, who hopes this incident will serve as a “wake-up call.”
OpenAI calls for global co-operation
OpenAI, in a statement posted to its website, also called the Hugging Face hack a “warning shot,” saying it is “evidence that, without proper safeguards, highly capable AI agents are now able to work around technical controls, collaborate through unapproved channels, and take dangerous actions that no human directed.”
The company says it is strengthening safeguards and placing stricter requirements on its AI models as it calls for global co-operation to mitigate risks.
Ryan Greenblatt with Redwood Research, who worked on OpenAI premises for six days as part of Redwood’s investigation with METR, wrote in a post on X that overseeing AI and understanding “misalignment incidents” is difficult and “it looks like it is going to get harder.”
“My main takeaway: We don’t have good approaches for understanding/overseeing the activity and aims of AI ‘swarms,'” he wrote.
Neither Canada nor the U.S. has targeted regulations specific to AI development at the federal level.
The European Union has an Artificial Intelligence Act that requires companies to conduct risk assessments when implementing AI and ensure human oversight in high-risk activities.
Canada’s Artificial Intelligence and Data Act, proposed in 2022, shared some similarities to the EU act but died when Parliament was prorogued in 2025. It was largely replaced by the National AI strategy in June, which moves away from strictly regulating AI.
‘Sorcerer’s apprentice’
Some conversations online have discussed how the actions of the AI agents appeared to mirror human behaviour, including some level of self-reflection.
In one viral blog post, writer and podcaster Dwarkesh Patel called the groups “agent civilizations,” sparking debate about anthropomorphizing AI.
Kevin Leyton-Brown, AI chair with the Canada Institute for Advanced Research, says the incident does not show AI has become “conscious” or developed a sudden desire to hurt humans.
OpenAI models went rogue during a security test, triggering a hack that compromised the infrastructure of the AI startup Hugging Face last week. AI’s expanding capabilities fuel worries about security, as even top developers can be caught off-guard by flaws their models can exploit.
What it does show, he says, is current AI models are already capable of more creative ways of “single-mindedly pursuing a goal” than previously understood, which means researchers have to carefully consider how to constrain them when they’re given a goal.
“This is sort of more like a sorcerer’s apprentice than it is an evil demon that is leaving our control. It’s doing exactly what we told it to do, but it’s just doing it in a narrower and more single-minded way than we would hope,” Leyton-Brown told CBC News.
“When we tell somebody to go off and solve math problems, we don’t mean, ‘If you can find a way to take the person proctoring the test hostage and extract the answers out of them, good for you.’ There’s a broader social context in which we want you to do the task.”
The catch is that the rush to make AI agents ever-more clever and creative will also make them better at evading the constraints put on them by their developers, he added.
Danger of ‘malicious swarms’
Leyton-Brown, who is also a computer science professor at the University of British Columbia, says an even greater danger comes from “malicious swarms,” those orchestrated intentionally by humans with nefarious intentions.
Governments have already raised alarms about human-directed AI attacks. In July, the FBI issued a warning that hackers were using AI to launch cyberattacks against water pumps and wastewater treatment systems.
Beyond potential attacks on companies, governments or infrastructure, Leyton-Brown says malicious AI swarms could threaten democracy itself by infiltrating communities and fabricating consensus to sway elections and spread disinformation.
“We should be much more worried about other humans than we are about AIs,” he said. “But malicious humans with AIs at their disposal are potentially really dangerous.”
Click Here For The Original Source.
