3rd Party Risk Management
,
Agentic AI
,
Artificial Intelligence & Machine Learning
Enterprises Buy AI Like SaaS, but Its Failures Slip Past Every Dashboard
Image: Shutterstock
Picture an ordinary Tuesday at a large enterprise. The internal artificial intelligence assistant, the one thousands of employees use for research and drafting, starts giving answers that are slightly wrong.See Also: 79% of SOCs Use AI. Only 36% Have It in Their Workflows.
Not broken. Wrong in a way that reads fine.
A finance analyst notices a summarized contract term that does not match the source document. She files a ticket. The helpdesk checks her permissions and closes it. Application security scans the integration and finds no vulnerable dependencies. The cloud team pulls up dashboards: latency normal, error rates normal, no drift in the infrastructure baseline. The model vendor confirms inference is healthy on its side.
Every team is right. Yet the problem is still there because it lives between their layers, and no one in the escalation chain can see across all of them.
This is the detection gap AI is creating inside enterprises. AI is being bought like software as a service, budgeted like SaaS and escalated like SaaS. But it does not fail like SaaS. Breakage with a SaaS product is usually visible as an outage, a failed login, a missing record or a degraded workflow. A compromised or misbehaving AI system can keep responding fluently while becoming subtly untrustworthy.
The National Institute of Standards and Technology’s Generative AI Profile identifies precisely this failure mode: systems can present false content confidently enough that users act on it, especially in consequential settings such as healthcare.
The SaaS mental model gives you a ticket queue. The failure requires a forensic capability.
The Stack Nobody Sees Whole
Trace a single prompt from a user’s browser to a model and back. It crosses the network, an edge provider, an API gateway and an orchestration layer that retrieves context and routes the request. It lands on an inference cluster and a specific accelerator, where model weights are loaded into memory. The response travels back through filters, caches, policies and formatting logic.
Now overlay ownership.
The network team owns the packet path but cannot judge a healthy prompt. The platform team sees resource metrics, not semantics. The machine learning team watches evaluations but rarely sees the infrastructure underneath. Security monitors the compute layer and may treat the model as an opaque artifact. The vendor sees inside its service boundary, but not the enterprise workflow around it.
Draw this as a diagram and the striking feature is not the complexity. It is how many boundaries between components have no observer on either side.
Many teams are working on AI reliability and security. AI observability, MLOps, model monitoring, red teaming, governance and application security are all attacking pieces of this problem. But no single discipline owns the incident path from user-visible semantic failure down through retrieval, orchestration, infrastructure, model artifact and hardware state.
That fragmentation is not just organizational folklore. NIST’s AI-specific Secure Software Development Framework notes that responsibility may be shared among the model producer, the system producer and the organization acquiring the system. It recommends agreements that specify who owns each security practice and how each party will prove conformance. In other words, you need to assign the boundaries, because the technology will not assign them for you.
These are key parts of the AI map that enterprises are missing.
How We Lost the Map
We have watched this movie before.
When virtualization arrived, storage and compute became abstractions and the physical layer disappeared from many engineers’ mental models.
Cloud repeated the trick at a larger scale. By the mid-2010s, I was interviewing architects who could design an elegant multi-region VPC but could not reason about a failure beneath it, because they had never seen that layer.
The industry largely avoided the consequences. Hyperscalers absorbed the operational complexity and their reliability was good enough that the missing depth rarely became an issue.
Each abstraction was a fair trade. You gave up architectural literacy and got velocity. The deferred cost was expertise that increasingly started, and stopped, at the API.
AI is repeating this pattern one layer up, but this time the trade-off is worse.
Enterprises consume frontier models the way they consume Salesforce: There’s an endpoint, a key, a monthly bill. It’s the SaaS consumption model, but it’s not the same failure model at work here. When a SaaS product breaks, opening a vendor ticket is often the correct diagnostic move. When an AI system drifts, gets poisoned, retrieves the wrong context, follows an injected instruction or runs on an untrusted artifact, the failure may not look like failure at all. Drift is invisible to anyone who treats the model as a black box behind an endpoint. The same abstraction that made the technology easy to adopt has been quietly training the necessary diagnostic capability out of the workforce.
Thin talent forces more reliance on abstraction, and more abstraction thins the talent further.
Four Failures and Who Could Catch Them
Abstract risk arguments are cheap, so here are four concrete compromise modes, each with an honest answer to the question of who could detect it today. These are not edge-case inventions. NIST’s AI security guidance explicitly lists poisoned training data, tampered model weights, injection-style attacks, supply-chain attacks and pipeline misconfiguration as threats to cover in risk modeling.
Poisoned training or retrieval data. An attacker seeds documents into a retrieval-augmented generation corpus or corrupts a fine-tuning dataset upstream. Data engineering sees rows processed, jobs completed and indexes refreshed, but not necessarily semantic integrity. NIST recommends verifying data provenance and integrity, then using measures such as anomaly detection, filtering, sanitization and human review. In many organizations, detection still depends on a user noticing and refusing to let the issue go. That is luck dressed up as process.
Tampered model weights. Weights are crown-jewel artifacts, but enterprise controls often treat them as files rather than executable state. The software industry already has a supply-chain vocabulary: signatures, SBOMs, provenance and attestation. NIST recommends hashes or digital signatures for AI models and their components, plus provenance for models, libraries, frameworks and pipelines. Adoption remains uneven. An organization may verify the container but not the artifact loaded inside it.
Prompt injection against agentic systems. Once a model can call tools, read mail, file tickets, update records or trigger workflows, a hostile instruction hidden in a document becomes an attack with real blast radius. The Open Worldwide Application Security Project’s guidance on prompt injection warns that instructions embedded in external files or websites can alter model behavior and that the impact grows with the agency the system has been given. This is an application security problem wearing an ML costume, and it falls into the gap between the appsec team, which may not test model behavior, and the ML team, which may not think in terms of attacker-controlled input.
Hardware-layer compromise. This is the one my electronics background will not let me ignore. NIST’s platform firmware guidance calls for mechanisms to protect firmware, detect unauthorized changes and recover after an attack. Its work on hardware-enabled cloud security makes the underlying point: Higher-layer controls depend on a trustworthy platform. Roots of trust, attestation and confidential computing help, but they do not create an owner. Ask who verifies the firmware and attestation state of the accelerators beneath your workloads, or obtains that assurance from your cloud provider. In many environments, the honest answer is that nobody owns that mandate.
None of these failures belongs to a single team. That is the point. They are cross-layer failures, and they are easy to miss when tools and organizations are designed around single-layer ownership.
What Detection Would Actually Take
Naming a gap is easy. Closing it means connecting practices that currently exist only as fragments.
Cross-layer telemetry correlation as a practice, not a dashboard. Someone needs the mandate and access to connect a semantic anomaly with events in retrieval, orchestration, infrastructure, model artifacts and hardware. The International Guidelines for Secure AI System Development already call for monitoring model behavior and logging inputs to investigate compromise, misuse and drift. The next step is correlation across the stack. It should be a runbook with named owners, not five teams assembled on demand.
Provenance and attestation for model artifacts. Weights should be signed, hashed and verified at load time. NIST’s AI-specific SSDF calls for cryptographic verification and model provenance; enterprises should carry that control into deployment. The techniques exist. What is missing is the norm.
Incident response that includes ML engineers by default. Most IR playbooks I have reviewed treat AI systems either as ordinary applications or as out of scope. The NIST AI Risk Management Framework places post-deployment monitoring, incident response, recovery and change management in the same operating loop. An IR team that cannot ask “Did the model’s behavior change, when, and correlated with what?” is not equipped for this class of incident.
Deliberate cultivation of full-stack practitioners. This one is unfashionable because it does not ship as a product. But your organization can start by rotating engineers across layers. Include ML engineers in infrastructure escalations and infrastructure engineers in model evaluation reviews. Depth across the stack used to be a career accident. It now has to be a program because the pipeline no longer produces it on its own.
The first move is not buying another platform. Pick one consequential AI workflow and ask four questions:
Who owns semantic correctness?
Where are the model, prompt, retrieval corpus and policy versions recorded?
Which logs can reconstruct a timeline across retrieval, tools, infrastructure and output?
Who leads the incident when the answer is plausible but wrong?
If the answers span five teams and resolve to no single name, you have found the gap.
The Clock on This
The institutional knowledge held by people who have debugged failures across hardware, hypervisor, cloud and now model is becoming scarcer. Many are leaving hands-on roles, while many engineers entering the field have been trained almost entirely above the API line through no fault of their own.
Enterprises are wiring AI into finance, healthcare, legal review, customer support, software delivery and critical infrastructure at a pace that assumes someone, somewhere, can tell when these systems go wrong.
Right now, that assumption is often false.
The time to orchestrate a fix is while experienced cross-stack practitioners are still in the room. The reason has nothing to do with veterans being special. Rebuilding a mental model from scratch can take years. Transferring one takes a rotation program and some patience.
The alternative is finding out about the gap the way industries usually do: from the incident report.
Click Here For The Original Source.
——————————————————–
