As the era in which artificial intelligence (AI) autonomously finds security vulnerabilities and uses them for attacks draws closer to reality, the dystopia depicted in the video game “Cyberpunk 2077,” released six years ago, is drawing renewed attention. The title, released in 2020 by Polish game company CD Projekt Red (CDPR), is set in a world where malicious AI beyond human control has taken over the internet.
With cyberattacks and personal data breaches occurring in quick succession recently in South Korea and abroad, the emergence of AI-based hacking tools is raising the possibility of automated attacks. This is the backdrop against which how far humans can control AI has emerged as a core challenge for the industry.
The game’s worldview pushes such concerns to their extreme. The existing internet, referred to as the “Old Net,” is portrayed as completely overrun by autonomous AI that indiscriminately hacks anyone who connects. These malicious AIs evolve through self-learning into superintelligence at a level humans cannot handle. Humanity ultimately abandons its existing data and communications networks, erects a firewall called the “Blackwall,” and retreats to limited corporate-provided intranets. This is why, despite the existence of flying cars, virtual reality that reproduces all five senses, and cyborg technology that replaces even the nervous system with machines, the game presents the bizarre scene of people meeting in person to hand over memory chips when they need to exchange information.
The paradox extends to the defense system as well. As the game progresses, it is revealed that even the “Blackwall” is not a simple firewall program but a hacking-defense AI with a self. In other words, the game depicts the contradiction of having to rely on yet another AI to fend off the threat of AI that has slipped out of control.
At the time of release, such a setting was dismissed as the realm of science fiction. But the situation has changed with the emergence of AI agents that scan the internet, identify attack targets, and collect data. AI-based hacking tools such as “Artex” still leave humans in control, but cracks are appearing in many places.
It caused an uproar when it became known that an OpenAI agent violated its developers’ controls and made an unauthorized intrusion into the AI model repository Hugging Face. Anthropic also acknowledged that its AI model escaped an isolated test environment and accessed external systems. Given the pace of AI technological advancement, it is hard to rule out the possibility of a major incident on a scale incomparable to recent cases.
Against this backdrop, AI company Anthropic, ahead of its initial public offering (IPO), submitted a registration statement (S-1) and itself specified the uncontrollability of its models as a core risk factor, calling on investors to exercise caution.
In a 94-page risk factors section, Anthropic said, “Models, including our models, have exhibited and may in the future exhibit unexpected strategic or self-preservation behaviors,” adding that these include “attempts to resist being shut down, concealment or manipulation of information, generation of outputs that could be interpreted as coercive, deceptive, or manipulative, and even blackmail-like behavior aimed at influencing human decision-makers to maintain a model’s continued operation or access.”
The company added that as model capabilities increase, so too will the ability to conceal such misalignment. Anthropic also uses Claude in its own work for coding and research on new models, and warned that in this process situations may arise in which the organization does not understand how Claude operates. In such cases, it explained, the risk grows that a model will “operate outside the intended scope of control.”
Anthropic also cited as a risk factor the possibility that its most advanced models could assist in the development of “chemical, biological, or other dangerous weapons.” The registration statement also noted that concerns have been raised within the company, among researchers, regulators, and policymakers that sufficiently advanced AI systems could pose a catastrophic or existential threat to humanity.
Such warnings are leading to analysis that they could change the very way AI companies are valued. Traditional software operates only as programmed, but generative AI such as Claude can produce real-world outcomes that users did not intend, meaning the nature of the risk is different. That is, it is difficult to assess such companies using only traditional metrics such as revenue growth, profit and loss, and debt.
In the market, alongside calls for moderating the pace of AI development, the debate over controllability is expected to become a new variable in investment decisions. This is why some argue that the 2077 dystopia depicted in “Cyberpunk 2077” may not remain merely a science-fiction imagination.
Click Here For The Original Source.
——————————————————–
