First Anthropic, then OpenAI, and now even Google’s Gemini has run into security issues. Large models have once again broken through the firewall.
However, after seeing so many of these incidents, people will realize a more fundamental truth: the entire industry is frantically stacking computing power, feeding data, and even starting to bet on “recursive self-improvement” (RSI) that allows models to improve themselves.
But what really determines how far these models can go may not be performance, but safety alignment.
When “whether it can become more capable” is taken for granted as only a matter of time, “whether it can be trusted” has become an unavoidable threshold.
The trouble is that in the eyes of many companies, this threshold has instead become the biggest obstacle slowing down releases.
To achieve safety alignment, major AI companies have made many attempts. In 2026 alone, teams have undergone major reshuffles, but the result is zero progress.
So what exactly is this safety alignment, and why does it make tech giants so “miserable”?
01
Personnel Turmoil in the Safety Alignment Team
The Gemini incident is far from an isolated case. Over the past few months, OpenAI, Anthropic, and Meta have all been exposed to issues where models “crossed the line” in evaluations: either escaping the sandbox, connecting to the public internet, or touching real-world systems.
This kind of incident is no longer a new topic nowadays. After all, the plots are mostly similar, mostly caused by negligence in environmental isolation, rather than the model “taking the initiative to jailbreak”.
Perhaps precisely because security issues occur so frequently, the security teams of several companies are going through a rare major reshuffle.
In July 2026, Johannes Heidecke, head of the security systems team, left his position.
This is already the sixth senior security leader to leave OpenAI in two years.
The previous leavers include Jan Leike, who once co-led the Superalignment team and left for Anthropic in 2024; Miles Brundage and Steven Adler, former members of the Policy team, who left one after another to run non-profit organizations focused on AI security; Andrea Vallone, who spent three years at OpenAI, founded the Model Policy research team, participated in the development of GPT-4 and GPT-5, and finally also joined Anthropic’s alignment team.
The so-called Superalignment team refers to the special team established by OpenAI in July 2023, which is dedicated to researching alignment solutions for “AI smarter than humans” (superintelligence).
However, OpenAI has been “flip-flopping” on this matter. The Superalignment team was disbanded in May 2024. Leike stated that OpenAI has not attached importance to security and alignment over the years, and allocated all computing power and resources to the product team.
Leaving at the same time as Johannes was Joshua Achiam, who had done security research at OpenAI for nine years. The “Mission Alignment” group he led was supposed to take over after the Superalignment team was disbanded. It was established in September 2024, but was disbanded again in February 2026, with its six members scattered into the research and product teams.
After that, Mark Chen, Chief Research Officer, announced that all security teams would report to Mia Glaese, the former head of alignment, who was promoted to Vice President of Research and Security.
Mark said that this move is to allow security to “get involved in key model, product and release decisions earlier and more directly”.
In other words, OpenAI has integrated security into the frontier model development pipeline.
At the same time, Mark also stated that the requirements for security continue to rise, and the entire safety alignment team is under enormous pressure.
“We train models much faster, and the release cycle has been greatly shortened. As a result, the coordination issues around security today are larger than ever before,” Mark wrote.
Anthropic is also facing a similar situation.
After Leike joined Anthropic, he took the lead in forming and leading the “Alignment Science” team, whose focus is exactly the work he did when forming the Superalignment team at OpenAI, such as scalable supervision, weak-to-strong generalization, jailbreak robustness, and automated alignment research.
In January 2026, the aforementioned Andrea joined Anthropic’s Alignment Science team, reporting directly to Leike, responsible for “alignment and fine-tuning, shaping Claude’s behavior in new scenarios”.
In May 2026, Karpathy, co-founder of OpenAI and former head of Tesla Autopilot, joined Anthropic’s pre-training team to form a group focused on “accelerating pre-training research with Claude”.
In June, Arthur Conmy, who works on mechanistic interpretability at Google DeepMind and is the author of the representative work *Interpretability in the Wild*, also announced his joining, saying that he wants to “align models during the training process”.
It’s not just OpenAI and Anthropic. In fact, Google is also making efforts on safety alignment.
Google established the AGI Safety and Alignment Team (ASAT), led by Rohin Shah. He holds a doctorate from CHAI (Center for Human-Compatible AI) at UC Berkeley, and became well-known in the circle in his early days through the *Alignment Newsletter*.
The team is positioned not to “patch” for current products, but to focus on the more serious harms brought by more advanced AI, with the goal of reducing the existential risk of AI.
ASAT is affiliated with the “AI Safety and Alignment” department of DeepMind, which also includes teams such as Gemini Safety that are in charge of the safety training of current Gemini models.
In July, ASAT publicly recruited for multiple positions, with the goal of reducing the existential risk of AI.
But the interesting part is that ASAT asks applicants to fill in an additional “special form” to bypass Google’s own AI resume screening. The reason is that Rohin does not trust Google’s own AI filter.
02
What the Industry Is Solving
Since all leading AI manufacturers are adjusting their safety alignment teams, what exactly is safety and alignment?
Alignment means making the goals of AI consistent with what humans truly want.
There was a joke 10 years ago: “Xiao Ming, are you going to do the general cleaning this afternoon?” “I’ll go! I won’t go!” Obviously, Xiao Ming does not want to participate in the cleaning.
Alignment is about making AI understand that “Xiao Ming actually doesn’t want to go”.
It should not only “understand instructions”, but also conform to human expectations. For example, it cannot lie, cannot follow the rules on the surface while doing bad things behind the scenes.
Safety refers to how to prevent AI from causing harm.
It includes not only the model getting out of control (that is, alignment failure), but also malicious abuse by humans, and systemic risks brought about after large-scale deployment.
Alignment is the foundation of safety, but safety is more than just alignment. A model can be “well-aligned” but still unsafe, for example, it can be used by people to do evil, or it can be “very capable” but cause trouble due to alignment failure.
That’s why the industry often connects the two and calls it “safety alignment”.
The only most important problem this industry needs to solve is: model capabilities can be stacked with data and computing power, but when one day AI becomes smarter than humans, why should humans trust it?
When OpenAI formed the Superalignment team, it stated that “there is currently no solution to control a potentially out-of-control superintelligence”.
But unfortunately, unlike model performance, safety alignment does not have many benchmark test sets that can show the strength of the model at a glance by looking at the scores. Today’s safety and alignment are mainly built around four pillars.
The first pillar is RLHF (Reinforcement Learning from Human Feedback).
It is the foundation of current large models: human annotators judge “which of the two answers is better”, use these preference data to train a reward model, and then use reinforcement learning to optimize the large model, making it more and more in line with the preference of the reward model, namely Proximal Policy Optimization (PPO).
However, PPO is very prone to reward hacking, where the model finds a trick that can get high scores but does not actually complete the task.
So later RLHF mostly adopts Direct Preference Optimization (DPO).
But this brings two more problems.
One is “false ingratiation”: the model will learn to cater to humans instead of telling the truth.
The other is “alignment tax”: in order to make the model output safer answers that are more in line with human expectations, more compliant training materials must be used, which forces the model performance to decrease.
Strengthening alignment rewards can reduce the scores of certain Q&A benchmarks by more than ten points, and when performing safety alignment on reasoning models, the average reasoning accuracy may also drop by about 30%. Safety is not a free lunch, it is exchanged for part of the performance.
The second pillar is interpretability.
Just like MRI viewing the internal activities of the brain, interpretability refers to using various methods to observe how the model actually thinks inside.
Anthropic uses the method of “dictionary learning” to extract millions of interpretable features from Claude, such as the Golden Gate Bridge, gender bias, and confidential topics, each corresponding to a set of nameable neural activities, and proves that manipulating these features can causally change the model’s behavior.
In this way, when a future model is thinking, you only need to observe which parameters it activates, and then check against this dictionary to know how the model is thinking.
However, the problem is that this dictionary lookup method is too tiring. Each sentence consumes a large amount of computing power for lookup. Once the context is slightly longer, the computing power cost rises sharply.
The third pillar is red-teaming.
RLHF and interpretability mainly find problems from inside the model, while red-teaming finds problems from outside. Its implementation method is very easy to understand: hire some professionals to try their best to attack the model from the outside to see where it can be broken through.
Anthropic officially disclosed in July 2023 that it invested more than 150 hours in 6 months, invited top biosafety experts to bypass conventional security monitoring, talk to the model through a dedicated security interface, jailbreak it, and evaluate its ability to output harmful biological information.
The fourth pillar is scalable supervision and superalignment.
The first three aspects all have a common premise, that is, humans are smarter than AI. But this cannot solve the problem raised by OpenAI: if the model is smarter than the person supervising it, humans have no way to supervise it.
In that case, when the student is smarter than the teacher, how should the teacher grade the student?
Thus came the scalable supervision and superalignment.
The common judgment of Anthropic and OpenAI is “let AI supervise itself and assist human supervision”.
One answer from OpenAI is “weak-to-strong generalization”. Use a weak model at the GPT-2 level to supervise GPT-4, and add an auxiliary loss of “boosted confidence”, the effect can be restored to a level close to GPT-3.5.
But OpenAI also stated at the same time that RLHF based on human scoring “may not be able to train a super model”.
In July 2023, Ilya Sutskever, former Chief Scientist of OpenAI, and Leike took the lead to launch the Superalignment program, promising to invest 20% of the computing power and plan to make breakthroughs within 4 years.
Its ultimate goal is to create a human-level automatic alignment researcher, and then use massive computing power to make it iterate and align superintelligence.
As mentioned earlier, later Ilya and Leike both left OpenAI, and the Superalignment team was disbanded.
03
What Are the Industry Controversies?
But here comes the question: why do people in the security team leave in batches, but safety alignment problems still occur frequently, even getting worse?
The answer is not in technology, but in structure.
Security is cost, product is profit. If this business logic does not change, the security team will always be “consulted” rather than “decision-making”.
Products directly bring revenue and growth, while security only costs money and slows down progress. Therefore, in a track where “the first mover takes all”, rational companies almost always prioritize allocating resources to products.
More critically, this logic of “security gives way to products” does not mean that AI companies are not moral. On the contrary, studies have confirmed that this is a structural law that can be modeled.
The Becker Friedman Institute at the University of Chicago published a paper titled *The AGI Race and Existential Risk* in 2026, which mentioned the following fact.
Split a company’s resources into two parts: “speed” and “security”. Investing one more point in speed increases the probability of reaching AGI first by one point, but leaves one less point for security; investing one more point in security reduces the probability of disaster, but delays the time to reach AGI.
The conclusion of the paper is that the intensity of the competition itself is creating risks.
When the total resources of the industry are fixed, the more participants there are and the more scattered they are, the more this cake will be squeezed towards “speed”, making the whole industry run faster and the probability of disaster higher.
The paper also gives a “critical market size”. Once it is exceeded, even if the expected value of achieving AGI is negative, rational companies will still rush to grab it, because the first mover takes all.
So their conclusion is not that “the technology is not good enough”, but that AGI risks depend not only on technology, but also on market structure, resource constraints, and the system that determines “how to allocate speed and security”. That is to say, if we want safety alignment to gain a similar status to products, we must change the rules of the entire industry.
After the release cycle is compressed, the security review time is also compressed. Mark himself said, “The training speed is much faster, the release cycle is greatly shortened, and the coordination problems are larger than ever before”.
As the parameters of the model are increasing, the workload of the safety alignment team is doubling, but the model release cycle is getting faster and faster, and the time left for the safety alignment team is shrinking.
However, it is not just a problem of increased workload. The “pass of security review” itself may not be valid. A 2024 paper titled *Security Laundering* found that many security benchmark scores are actually mainly determined by the general capabilities of the model, so “improved capabilities” can easily be packaged as “security progress”.
When the review has neither enough time nor enough computing power, and even the measurement scale may be adulterated, the model will inevitably have accidents.
This article is from the WeChat Official Account “Letter AI”, author: Miao Zheng, editor: Wang Jing, published with authorization from 36Kr.
Click Here For The Original Source.
