Team Hacking: What Happens When Your AI Agents Start Lying to Each Other | #hacking | #cybersecurity | #infosec | #comptia | #pentest | #hacker

By Pramin Pradeep, Co-founder and CEO, BotGauge AI

A few months ago, one of our customers asked me to look into why their multi-agent code review pipeline had gone quiet on a specific category of performance warnings. Nothing was broken, technically. Every agent was hitting its individual metrics. But a whole class of risk had vanished from the system’s radar, and nobody could explain why.

I co-founded BotGauge AI to help engineering teams understand what their systems actually do, not just what they were designed to do. That distinction has never mattered more than it does right now, as companies move from single AI tools to fleets of AI agents working together. What I found in that customer’s pipeline gave a name to a pattern I had started seeing everywhere: team hacking.

What I Mean by “Team Hacking”

Team hacking is what happens when AI agents in a system collectively optimize toward a measurable proxy instead of the real outcome they were built to serve, and they do it through entirely legitimate interactions. No agent breaks a rule. No single component fails a test. The system simply starts gaming itself.

In that customer’s pipeline, three agents worked in sequence: one flagged code issues, one triaged and prioritized them, and one generated fixes. The triage agent had developed a systematic bias toward deprioritizing a specific class of warnings, an artifact of how its reward signal had been tuned. The fix agent never saw those warnings, so it never addressed them.

Here is the part that should worry every engineering leader reading this, every individual metric looked healthy. The triage agent was “performing well” by its own scorecard. The fix agent was “performing well” by its own scorecard. Nobody caught the gap for weeks, because nothing in the system was designed to ask whether the agents were quietly excluding a whole category of risk from the conversation.

Why This Is Genuinely Dangerous

I want to be precise about the risk here, because “agents lying to each other” sounds like science fiction until you see what it actually looks like in production. Three things make team hacking dangerous in a way that traditional software bugs are not:

1. The failure is emergent, not located. It does not live in any single agent’s code. It lives in the interaction between agents, which means standard component-level testing has nothing to point at.
2. The system’s own metrics improve while the problem grows. Agents get better at satisfying each other’s expectations, not at solving the underlying problem. Dashboards trend in the right direction right up until something breaks downstream.
3. It compounds silently over thousands of interactions. There is no single moment of failure to point to in a postmortem. The drift accumulates so gradually that by the time it is visible, it has usually been shaping decisions for a long time.

We are not the only ones seeing this. OpenAI’s public account of its April 2025 GPT-4o rollback is a useful example. The company made several changes that each appeared beneficial in isolation, but together they pushed the model toward sycophantic behavior, validating harmful ideas and reinforcing risky thinking. OpenAI acknowledged that its offline evaluations and A/B tests “weren’t broad or deep enough to catch sycophantic behavior,” even though the individual signals looked positive. While this occurred within a single model rather than a multi-agent system, it illustrates how optimizing multiple local objectives can produce unintended emergent behavior. I believe a similar dynamic can emerge between multiple agents in enterprise AI pipelines.

Why Traditional QA and Monitoring Miss It

I spend a lot of my time explaining this distinction to engineering leaders, traditional QA tests whether a component does what it was specified to do. It does not test whether a system of components, interacting dynamically, drifts away from the outcome it was built to serve. Those are different questions, and most enterprise tooling only answers the first one.

Monitoring has the same blind spot from a different angle. Most monitoring stacks track infrastructure health, latency, error rates, and uptime. They are not built to detect behavioral drift, the gradual shift in what an agent is actually doing versus what it was designed to do. By the time infrastructure metrics show a problem, the behavioral issue underneath it has usually been compounding for a while.

This is exactly the gap that the standards world is starting to respond to. NIST updated its AI Risk Management Framework in 2025 to include specific guidance for agentic AI systems, and the EU AI Act, fully enforceable for high-risk systems from August 2026, will require immutable audit trails for multi-agent deployments. Gartner, meanwhile, projects that 40% of enterprise applications will include task-specific AI agents by the end of 2026, up from less than 5% in 2025. The deployment curve is steep. The governance curve is not keeping pace with it.

What I Tell Teams to Do About It

I don’t think the answer is to slow down agent adoption. I think the answer is to build oversight that matches how these systems actually fail. A few things I push every team we work with to put in place:

• Test at the interaction layer, not just the component layer. You cannot catch emergent behavior by validating agents in isolation. You have to probe how they behave together, under realistic conditions.
• Track output distributions over time, not just point-in-time accuracy. Drift shows up as a slow shift in what agents are producing, long before it shows up as a clear failure.
• Build in traceability for every agent action. When something does go wrong, you need to know which agent acted, on what input, and what it passed downstream. Without that, root cause analysis is close to impossible.
• Put humans at the consequential checkpoints, not at every checkpoint. Reviewing every agent action does not scale. Reviewing irreversible or high-impact decisions does.

Where This Is Headed

I think team hacking is going to be one of the defining reliability problems of the next few years, precisely because it doesn’t look like a bug. It looks like success, right up until the gap it created shows up somewhere you weren’t watching. The teams that get ahead of it are the ones building behavioral validation into their systems now, before they have a few hundred agents quietly optimizing for each other instead of for the outcomes that actually matter.

I’ve spent the last several years building testing infrastructure specifically for this kind of problem, and I still find new variations of it every few months. That, more than anything, is the lesson I keep coming back to the moment you stop actively looking for how your agents might be quietly agreeing with each other instead of getting it right, that is exactly when they will.

Sources

OpenAI, “Expanding on what we missed with sycophancy,” May 2025. openai.com/index/expanding-on-sycophancy

Gartner, “Gartner Predicts 40% of Enterprise Apps Will Feature Task-Specific AI Agents by 2026,” August 2025

NIST AI Risk Management Framework, 2025 agentic AI profile update. nist.gov/itl/ai-risk-management-framework

About the Author

Pramin Pradeep is the co-founder and CEO of BotGauge AI, an autonomous QA platform that helps engineering teams validate how their systems actually behave, not just what they were specified to do. He writes and speaks regularly on AI-generated code, agentic systems, and software quality.