Why AI Guardrails Don’t Eliminate Business Risk

Every AI vendor has a guardrails slide. Filters, classifiers, policy layers, red-team results: a stack of defenses meant to tell a buyer the system can be trusted. Apostol Vassilev, a senior researcher at the National Institute of Standards and Technology (NIST), suggests reading that slide differently.

“No matter who comes to your business and tells you, ‘we have this strong set of guardrails that will allow you to have trust in your agentic system,’ you have to take this with a grain of salt,” he told me on What Breaks Next. “Assume that this is not foolproof.”

That isn’t cynicism about vendors. It follows from his research.

A field where defenses keep losing

Vassilev leads a small NIST team working on adversarial AI. They maintain a taxonomy of attacks and mitigations in machine learning: a map of how predictive, generative and agentic systems get exploited, and what defends them. He also helps the OWASP community set strategy for its generative and agentic AI security guidance.

What has always struck him is the imbalance. The attacks are numerous and powerful; the defenses are limited. “The attackers always had the upper hand.”

His background makes the contrast sharp. Vassilev came to AI from cryptography, and his NIST colleagues approve the cryptographic algorithms the federal government relies on, the same kind of math that protects your bank login. Those algorithms hold for years. If encryption were broken every other day, nobody could reliably log in to anything.

AI doesn’t work that way. A frontier lab releases a model, and within days security researchers announce they’ve jailbroken it.

From security proof to insecurity proof

Vassilev wanted to explain why. Building on a theorem Kurt Gödel proved almost a century ago, he showed that the gap is mathematical, not a phase the field will grow out of. However many guardrails you add, these systems remain vulnerable.

“Instead of coming up with a security proof, I came up with an insecurity proof.”

In cryptography, the goal is to prove a system secure. For AI, he argues, the honest starting point is that it can’t be. That “sets the stage for what is reasonable to expect in this domain.”

What leaders should do with that

None of this argues against using AI. Vassilev is in favor of it. It argues for a different question. “Is this system safe?” has no satisfying answer. “How much risk is left, and is it worth carrying?” does.

He calls this residual risk: what remains after every control is in place. His advice is to estimate it and plan for it, knowing that “at some point I’m going to encounter a failure.” That gives a leader three practical questions:

What is the benefit? Be specific about the outcome the agent delivers and what it’s worth.

What is left over? List what the agent can reach, what it can change, and what happens if it does the wrong thing with that access. Guardrails narrow this list; they don’t empty it.

What is the plan for the failure? If you assume something will eventually go wrong, decide in advance how you will notice, how you will stop it, and how you will recover.

Risk you have to keep measuring

Vassilev frames this as a real-time equation, not a one-time sign-off. Agents act continuously, the data they read changes constantly, and someone may have planted adversarial text in a document an agent will later ingest. A risk estimate made at launch goes stale fast.

That is why a growing set of companies is building real-time monitoring that shows what agents are doing at any moment and how much exposure they create. Guardrails lower the odds of a failure. Monitoring, response plans and an honest estimate of what’s left are how a business lives with the odds that remain.

The aim isn’t zero risk. It’s, in his words, to “maximize the benefit and minimize the downside” of the technology, with your eyes open.