A few months ago I wired up an AI agent for an internal procurement workflow. The agent was supposed to review purchase requests, check them against spending policies, and either approve or escalate. It worked great in testing. In production, it approved a $40,000 software license that should have gone to a manager for sign-off, because the policy document it was referencing had been updated the day before and the agent's retrieval still had the old version cached.
Nobody caught it for two days. The agent was confident. The output was well-formatted. The approval email looked like every other one.
That's when it clicked for me: building the agent is the easy part. Making it safe enough to trust with real business decisions is a completely different problem. This post walks through how I think about guardrails and human-in-the-loop controls for agents that need to operate in enterprise environments.
How an Agent Actually Works (the Short Version)
If you haven't built one yet, here's the mental model. An agent runs in a loop.
Something triggers it. Could be a customer email, could be a flagged transaction. The agent reads that input and goes looking for context. It might hit a database for the customer record, pull a policy document from a vector store, or check what happened the last time this customer filed a similar request. Based on all of that, it decides what to do.
Sometimes the first thing it decides is that it needs more information, so it calls a tool to get it. Other times it's ready to act right away. Either way, at some point it does something: calls an API, updates a record, sends a notification. That's the moment the agent stops thinking and starts changing things in the real world.
Then it looks at what happened. Did the API return what it expected? If yes, it moves on. If the response is weird or the call failed, it goes back to the reasoning step and tries something else. On my procurement agent, a single purchase request would sometimes go through this loop five or six times before the agent was satisfied it had the right answer. Each pass through the loop is another chance to catch a mistake, but also another chance to introduce one.
Where Guardrails Fit In
Guardrails are constraints you put around the agent's behavior. They're the rules that say "you can do this, but not that" or "you can do this, but only if these conditions are met." Without them, you're trusting the LLM's judgment on everything, which, if you've worked with LLMs for more than a week, you know is a bad idea.
The way I think about it: guardrails go in three places.
Before the agent starts reasoning, you validate the incoming request. Is it well-formed? Does it contain prompt injection attempts? Should the agent even be processing this? I had a case where someone embedded hidden instructions in a support ticket. Without input validation, the agent would have followed those instructions as if they were legitimate. Catching that at the gate is the cheapest defense you'll ever build.
In the middle, you constrain what the agent is allowed to decide. Spending limits are the simple version: the agent can approve purchases under $5,000 on its own, anything above that gets routed to a human. But it extends to other boundaries too. Which APIs can the agent call? What data can it access? Some teams maintain these constraints as versioned config files so there's a clear audit trail when policies change.
After the agent produces its output, you check the work before it reaches anyone. Did the response leak PII? Does the answer contradict the source documents the agent was supposed to rely on? Is the agent claiming something it doesn't have evidence for? This last check is what catches those confident-but-wrong responses that make everyone nervous about deploying agents in the first place.
Here's a pseudo-flow for a procurement agent with all three layers:
request comes in
→ validate format, scan for injection
→ agent reasons: pull policy docs, check budget, evaluate request
→ check: amount > $5,000? → route to human
→ agent acts: call approval API, update records
→ verify: response matches policy, no PII leaked
→ send confirmation
Think of each check as middleware. The agent doesn't skip them. They're part of the execution flow.
Human-in-the-Loop Isn't One Pattern, It's Several
"Human-in-the-loop" gets thrown around as if it's a single concept. In practice, I've used at least four different patterns, and picking the wrong one for your situation creates either unnecessary bottlenecks or insufficient oversight.
The most common is approval gates. The agent does its analysis, prepares a recommendation, then pauses and waits for a human to sign off before executing. This works for high-stakes actions: transactions above a threshold, production deployments, customer-facing communications. The problem is latency. If your approver is in a meeting, the workflow sits there.
Escalation routing is different. The agent doesn't pause on every action. The agent runs on its own until it hits something confusing. Maybe the request doesn't match any pattern it knows. Maybe two internal policies say opposite things. At that point, the agent stops, writes up what it found and what it's stuck on, and hands the whole thing to a person. Ideally the person's resolution gets recorded and used to improve the system for next time, though in my experience that feedback loop is the part most teams build last.
At scale, sampling review is what I've found most practical. The agent processes everything on its own, but 5-10% of completed interactions get routed to a human reviewer after the fact. The reviewer scores them and flags anything off. This is how you catch drift over weeks and months without slowing down every individual transaction. A global payments organization used this approach after deploying agentic systems in customer support. Operational workload dropped by nearly 50%, but the review loop kept running to catch quality issues before they compounded.
There are also real-time override controls, where a dashboard shows the agent's reasoning live and a human operator can pause or redirect if things go sideways. This is expensive to staff. Most teams use it during the first couple weeks of a new deployment and then dial it back.
Start with approval gates on everything, then gradually move actions into autonomous mode as you build confidence through sampling. This graduated approach is how you turn a cautious pilot into a production system without anyone losing sleep over it.
Failure Modes That Guardrails Actually Catch
The procurement agent story I opened with is a good example of policy drift. The agent was referencing a cached policy document that had been superseded. It made decisions based on rules that no longer applied, and the outputs looked perfectly normal. Input-level checks on document freshness would have caught it. I didn't have those yet. Lesson learned.
Hallucinated tool calls are another one I've hit. The agent decides to call an API that doesn't exist, or passes arguments that don't match the expected schema. If you validate tool calls against a registry of available tools and their schemas before execution, you catch this. If you don't, the agent sends garbage to your production APIs and you find out from your ops team at 2 AM.
I also set a hard step limit on every workflow now. Usually 8, sometimes 10 depending on the complexity. Before I started doing this, I had an agent on a document processing pipeline that got stuck in a retry loop for 45 minutes. It kept failing on the same API call, backing off, trying again, burning credits the whole time. We didn't notice until the bill spiked. Step limits are boring. They also save you from that kind of thing.
And then there's the groundedness problem: the agent produces a well-formatted answer that sounds right but doesn't actually come from the retrieved documents. It's fabricated. Output checks that verify whether the answer traces back to a source document catch this before it reaches a customer. Those checks add latency, maybe 200-300ms per response, but compared to the cost of sending fabricated information to a customer, it's nothing.
Practical Advice for Getting Started
What’s worked best in my experience is starting with a single workflow that has well-defined inputs, manageable volume, and safe-to-reverse outcomes if something goes wrong. Avoid tackling your most critical process right away. You need space to experiment, make errors, and learn from them.
This might seem counterintuitive, but set up your guardrails before writing any agent logic. Establish your policy limits, approval checkpoints, and output validation criteria first. Design the agent to work within those boundaries from the start. If you build the safety mechanisms up front, they become part of the foundation. Every team I’ve seen skip this ends up scrambling to bolt on guardrails after their first production issue, and that always leads to more headaches.
Logging deserves its own mention. You want to be able to reconstruct exactly what the agent did: which tools it called, what those tools returned, and how the agent interpreted the results before deciding its next move. When something breaks (and it will, probably during a demo), that trace is the only thing standing between you and "I have no idea why it did that." A UK automotive manufacturer that went from 200+ RPA bots to intelligent automation learned this early. Process mining surfaced over £10M in procurement problems nobody knew about, and regulatory doc processing costs dropped 80%. None of that would have been possible without visibility into what the systems were actually doing. Visibility came first. The savings followed.
Start with tight autonomy. Approval gates on everything feels slow, and it is, but it builds the dataset you need to evaluate agent quality. Once you have a few hundred reviewed interactions showing the agent gets it right 95%+ of the time, you can start removing gates on low-risk actions with actual evidence behind the decision.
The agents are getting better fast. But "better" doesn't mean "safe to deploy without guardrails". The teams I've seen ship reliable agent systems are the ones that treated safety as architecture from day one.
Top comments (0)