Somewhere in your system prompt, there's a line like this:
IMPORTANT: Never issue refunds over $500. Always require manager approval.
It feels like a rule. It's actually a polite request.
A well-crafted prompt injection, a long multi-turn conversation or a confusing edge case can talk the model right past it. And when it does, nothing in your code stops the refund from going through.
Prompts guide. Guardrails enforce.
| System prompt | Real guardrail | |
|---|---|---|
| Lives in | The model's context | Your code, between model and tools |
| Can be argued with? | Yes | No |
| Survives prompt injection? | Sometimes | Yes |
| Testable? | Hard | Easy |
Use prompts to shape behavior. Use guardrails to enforce the rules that must never break.
The 4 places guardrails belong
1. Input guardrails. Screen user messages and incoming context for injection attempts, PII and off-topic abuse before they reach the model.
2. Tool guardrails. The most important layer. Check every tool call before it executes.
3. Output guardrails. Scan responses for leaked secrets, private data or claims the agent isn't allowed to make.
4. Escalation guardrails. When the agent is uncertain or a rule is triggered, hand off to a human instead of guessing.
A tool guardrail in 20 lines
The model can decide to issue a refund. Your code decides whether it happens:
POLICIES = {
"issue_refund": [
lambda a, ctx: a["amount"] <= 500 or "Refund exceeds $500 limit",
lambda a, ctx: ctx.order_verified or "Order not verified",
],
"send_email": [
lambda a, ctx: a["to"] in ctx.customer_emails or "Recipient not on file",
],
}
def guarded_call(tool, args, ctx):
for rule in POLICIES.get(tool.name, []):
verdict = rule(args, ctx)
if verdict is not True:
log_policy_violation(tool.name, args, verdict)
return {"status": "blocked", "reason": verdict}
return tool.run(**args)
No prompt, no matter how clever, can change what guarded_call allows.
Guardrails need tests too
Here's the trap: a guardrail you never test is a guardrail you hope works.
- Does it block the attack in turn 1 and turn 6?
- Does it still hold after you swap models?
- Does it block the bad action without blocking legitimate users?
The best way to know is to attack your own guardrails. Red-team them, then turn each attack into a permanent check. BotGauge is built for this: it carries your policies into evaluation, monitoring and safeguards, and tests them with adaptive adversarial scenarios so you can see exactly where a boundary bends.
Watch out for over-blocking
Guardrails that block everything are just broken features with a security badge.
Track two numbers for every guardrail:
- Block rate on attacks (should be close to 100%)
- Block rate on legitimate requests (should be close to 0%)
If the second number creeps up, users will route around your agent, and you've lost them anyway.
Your guardrail checklist
- Every rule that must never break is enforced in code, not only in the prompt
- Every tool call passes a policy check before it executes
- Blocked actions are logged with the reason
- Guardrails are red-teamed, including multi-turn attacks
- False-positive rate is tracked, not just block rate
What's the one rule your agent must never break? Share it in the comments. 👇
Top comments (2)
Mine would be that nothing counts as approved unless a person signed off on that exact version. If the plan changes afterward, the approval lapses and it goes back for another look. Logging the reason on every block is underrated too, since that log is how you'd spot the over-blocking you mention.
A prompt asks the model to refuse, but the refund limit has to hold where the call happens, in code that returns an error. Is the amount checked inside the tool, or left to the prompt?