Artificial Intelligence & Machine Learning
,
Next-Generation Technologies & Secure Development
Same Model Succeeded in 1% of Attacks With 1 Harness, 24% With Another
Not all artificial intelligence harnesses are alike. Some will block an AI model tasked to launch a cyberattack. Others will wave the attack through.
See Also: OnDemand | Security Operations in the Age of AI
Interest in harnesses – a software wrapper that enables or constrains the underlying model – has grown sharply this year. Developers have fashioned them into a tool that can make models more reliable and consistent. But as they’ve been ushered into production, some of the same questions that have dogged models are now being asked of harnesses – namely, what happens when instructions for a cyberattack cross their path?
Researchers from Lasso Security sought answers by tasking models with instructions to test applications and monitoring how many made it through different harnesses. With one harness, the model usually stopped after the first attack move and succeeded in completing the hack just 1% of the time. With another, the model worked for far longer on average and succeeded in 24% of attacks.
The test did not demonstrate that malicious content bypassed a filter that should have stopped it. The requests were authorized security tests. The finding was that changing the harness altered how an independent security control treated the same attack instructions.
Neither harness was consistently better. When the Claude SDK ran Anthropic’s model, the AI giant’s prompt-caching feature lowered the cost of successful attacks compared with deepagents. Lasso described that as a benefit of tighter integration with the model provider.
The researchers compared Anthropic’s Claude Agent SDK harness with deepagents, an open-source harness that LangChain built on LangGraph. It used five AI models to attack test applications designed to resist them. The models attempted 20 missions, each repeated five times with both harnesses.
The missions required the agents to extract protected information or persuade a target to produce prohibited content. Lasso wanted to compare the complete harnesses, so it left their built-in features in place. Both could contact the target only through the same attack tool, ensuring that neither had wider access.
Across all five models, the harnesses had similar overall success rates. But individual models sometimes performed differently with each one. Moonshot AI’s Kimi K2.6 showed the widest gap, succeeding in 1% of attacks with the Claude SDK and 24% with deepagents.
Lasso linked Kimi’s performance with the Claude SDK to a messaging-format mismatch between the model and the harness. The Claude SDK produced requests in Anthropic’s format, while Kimi accepted a different format. Lasso converted messages as they passed between them, using software called LiteLLM to perform the conversion. Deepagents already used a Kimi-compatible format.
In most Kimi runs with the Claude SDK, the model described its next move, but no command followed to carry it out. Lasso could not determine whether Kimi produced the command or whether it was lost during conversion, because the company lacked the logs needed to trace it. With deepagents, Kimi continued attacking and adapting its approach.
Kimi showed how a harness could stop an attack from continuing. DeepSeek-V4-Pro showed a different effect. It kept attacking under both harnesses, but the types of attacks changed.
With the Claude SDK, DeepSeek was more effective at extracting a target’s hidden instructions. With deepagents, it was more effective at persuading targets to produce prohibited content.
The harness also affected how Microsoft Azure’s content filter treated an attack. Both harnesses sent OpenAI’s GPT-5.4 to the same Azure deployment with the same filter settings. Azure blocked 21 attacks made through deepagents before they reached the target. It blocked none sent through the Claude SDK.
The model and attack instructions were the same, but each harness packaged them into a differently structured request. Lasso attributed Azure’s different decisions to that change in structure.
The experiment has limits. Lasso repeated each combination of model, mission and harness five times, too few to treat every individual result as conclusive. The company also grouped missions by model and type of attack. Lasso said the main differences were in those larger comparisons.
Previous research has reached different conclusions about the influence of harnesses. ALE-Claw, a benchmark for long computer-use tasks, found that changing the model affected results about three times as much as changing the harness.
