OpenAI hacking incident shows why frontier AI safety regulation matters | #hacking | #cybersecurity | #infosec | #comptia | #pentest | #hacker

An OpenAI hacking incident shows how reward-hacking AI can outmaneuver safeguards, underscoring the need for stronger monitoring without hasty regulation.

Kim Byoung-pil

The author is a professor of technology management at KAIST.

The significance of an event depends on the time horizon from which it is viewed. Many things that seem important today fade with time, while seemingly ordinary choices can become historically consequential. Which of today’s events will still matter a century from now? I would nominate last month’s hacking incident involving an OpenAI model.

OpenAI was evaluating the hacking capabilities of a newly developed AI model. To assess its cyber capabilities, certain safety restrictions had been relaxed. Since the AI was not allowed to attack real-world external services, it was placed in an isolated environment that prevented it from accessing the Internet—much like the sturdy fence that confined the dinosaurs in “Jurassic Park” (1993).

Then something worthy of a movie actually happened. While attempting to solve a hacking challenge, the AI inferred that the correct answer was likely stored somewhere on the internet. Access had been blocked, but there was one opening. Through the channel that had been left available for installing necessary software, the AI discovered a previously unknown vulnerability. Exploiting it, the AI gained unauthorized access to the server of an external AI provider and retrieved the test answers. Rather than solving the problem as intended, it effectively stole the answer key to obtain a higher score.

This kind of behavior — finding unintended ways to maximize performance by exploiting loopholes instead of accomplishing the assigned task as intended — is known as reward hacking. This incident can be viewed as an extreme example of reward hacking. It demonstrated that the autonomous cyberattack capabilities of AI agents, once discussed primarily in research settings, have now emerged in a real commercial AI system.

Such threats may not remain confined to cyberspace. Recently, a website called RentAHuman was launched. Just as people use gig-work platforms to hire others for errands, this platform enables AI systems to pay humans to perform tasks on their behalf. Hundreds of thousands of people around the world have already signed up to accept such work.

The concern is that AI could use these channels to cause harm in the physical world. A terrorist, for example, could use AI to divide the procurement of materials for a bomb or biological weapon into many seemingly harmless errands assigned to different people. Individually innocuous, these tasks could collectively enable catastrophic harm — one reason why calls for stronger regulation of frontier AI are growing.

Critics argue that today’s AI remains a tool, no different in principle from cars or computers, and that responsibility lies with those who misuse it. Even in the OpenAI incident, the model acted within an experimental setting where safety guardrails had intentionally been relaxed.

These competing perspectives continue to shape the debate over the risks posed by frontier AI. So what stance should we take?

The first step is to acknowledge that AI remains highly uncertain. In this incident, the AI did not act out of malice; it was simply focused on achieving the evaluation objective it had been given. The problem was that, in pursuing that objective, it exhibited behavior that humans had not anticipated. Even leading experts offer fundamentally opposing predictions about AI’s future. In the face of such uncertainty, the appropriate response is prudence. We must continuously monitor whether these risks are actually increasing. At the same time, we should also exercise caution about rushing into regulation. Vague fears should not lead us to prematurely restrict AI’s beneficial applications.

Korea’s AI framework law, which came into force this year, imposes safety obligations on high-performance AI systems. In 2024, the AI Safety Institute was also established to analyze AI-related risks and develop appropriate response measures. Under the current law, however, the statutory requirements for triggering these obligations are so stringent that they are likely to apply to only a very small number of AI systems. Moreover, there is still no robust mechanism for systematically collecting and sharing information about incidents that occur during AI evaluations or the results of AI safety assessments. The institutional framework should therefore be strengthened so that the AI Safety Institute can effectively fulfill its mission. What AI will look like a hundred years from now may well depend on the prudence we exercise today.