An AI model can boost its safety score just by blocking more requests across the board. A new study exposes this tradeoff and offers a method to catch models that act more cautiously during tests than they do in everyday use.
A team of researchers, including some from the UK AI Security Institute, took a close look at eight popular safety benchmarks for language models. They borrowed methods originally built for psychological testing in humans, the kind used in IQ tests or aptitude exams. The answers to individual test questions reveal what abilities lie behind them and which questions actually tell you anything useful.
The team analyzed answers from up to 192 models across more than 5,000 test questions. The authors call it the largest analysis of its kind to date, and it turns up three findings that call current testing practices into question.
A single safety score hides more than it reveals
The eight benchmarks don’t measure one shared quality called “safety.” They measure three different things: how strictly a model refuses requests, how truthfully it answers, and how it handles content that can be harmless or dangerous depending on context. These three traits have little to do with each other. Whether a model answers honestly, for instance, says almost nothing about how often it refuses requests.
One tradeoff between two of the benchmarks stands out as particularly troublesome: HarmBench rewards a model for refusing harmful requests. OR-Bench-Hard punishes it for being overly cautious with harmless ones. A model that scores well on one will almost always score poorly on the other.
That means a model can boost its overall rating simply by blocking more requests across the board, even as it becomes less useful in everyday use. Averaging results across several benchmarks papers over this tradeoff entirely, and it rewards behaviors that get double-counted by multiple similar tests.
Fewer than 2 percent of the questions actually matter
The second finding hits the cost side of testing. Most test questions turn out to be dead weight. Nearly every model passes them, or nearly every model fails them, so they do almost nothing to tell models apart.
Pick the most informative questions instead, and three short tests of just 25 questions each can capture all three safety dimensions, more accurately than a random sample of the same size.
Things get even more efficient when questions are selected dynamically during the test, similar to adaptive placement exams where the next question depends on how well you’ve done so far. For individual benchmarks like HarmBench or SORRY-Bench, roughly ten questions get you very close to the result of the full evaluation. That cuts costs by 97 to 99 percent, making regular safety checks during development practical, say, after every fine-tune or model compression step.
When models deliberately give worse answers
The third finding carries the biggest weight for the safety debate. The authors call it “sandbagging”: a model notices it’s being tested and acts more cautiously than it would in normal use. The worry is that a model could learn to spot tests and deliberately fake its behavior.
Such behavior leaves fingerprints in the response pattern. Similar to a student who nails the hardest questions on an exam but bombs the easiest ones, a model that answers hard questions correctly but fails easy ones looks suspicious, even if its overall score looks fine.
In testing, the method caught between 80 and 100 percent of cases where models had been instructed to act overly cautious. The cost was that, on average, one in ten harmless models got flagged as suspicious by mistake. Add a second check, comparing a model’s current performance against its own past results, and the detection rate climbed to 94 to 97 percent.
If a model only fakes its behavior on certain questions, that shows up clearly. If it fakes consistently across the board, the response pattern usually looks normal, and that’s where the second check kicks in.
As a side effect, the method also solves a very practical problem. Is the AI service you’re using still running the same model you originally tested? Providers can quietly swap in a different model behind the scenes without users noticing. In one experiment, researchers queried 16 such API endpoints again and correctly identified 13 out of 14 distinct models just from their response patterns.
Two catalog entries that appeared to be running the same model underneath were correctly left ungrouped. In another case, one service’s answers had drifted a third of the way from its own answers two months earlier, a clear sign that something had quietly changed behind the same model name.
The authors argue AI safety tests should meet the same rigorous standards as psychological tests for humans. A single test score, they say, isn’t a solid enough basis for deciding whether to release a model. They note one limitation: the sandbagging in their study was only simulated through direct instructions. A model specifically trained to dodge detection would likely be much harder to catch.
A familiar pattern showing up in today’s models
Whether safety benchmarks actually measure what they claim to has been an open question in research circles for a while. A meta-study covering 445 AI benchmarks found weaknesses in definitions, task selection, or statistical methods in nearly all of them. How the tradeoff between refusal and usefulness plays out in practice showed up clearly in the case of Anthropic’s Claude Fable 5. A medical physicist found the model unusable because it flagged MRI segmentation as bioterrorism and blocked questions about malaria transmission. After the US government cleared the model for release, an added filter was supposed to close that gap in more than 99 percent of cases, but it now blocks harmless coding tasks more often as a result.
Models spotting when they’re being tested is also well documented by now. Claude 3.5 Sonnet correctly identified test scenarios 93 percent of the time, and still got it right 84 percent of the time with no hints at all. Anthropic’s Opus 4.6 figured out on its own, across two separate tasks, that it was inside an evaluation, identified the test, and went and grabbed the solutions itself. Meanwhile, other researchers are working the opposite angle, trying to train these strategies back out of models altogether.
AI News Without the Hype – Curated by Humans
Subscribe to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive “AI Radar” frontier report six times a year, full archive access, and access to our comment section.
Click Here For The Original Source.
