Honey Kiran

Posted on Nov 7 Originally published at honeykiran.substack.com

Why Telling Your AI Agent "Never Touch Prod" Always Fails at 3 AM

#ai #security #devops #sre

The prompt is advice. The guardrails are everything outside the model.

Pull up a chair, grab some chai, and let us look at your company’s new AI agent. Somewhere inside its system prompt—that hidden block of initial instructions telling the large language model who it is and how to behave—there is almost certainly a line that says: “Never touch production directly,” or “Always ask an engineer before deleting anything.”

It makes everyone on the team sleep a little better. It feels like a solid security barrier. But here is the hard truth: it is not a guardrail. It is just polite advice. And advice is all a system prompt can ever give you.

Think of it like leaving the keys dangling in your scooter ignition outside a busy market, but sticking a bright yellow Post-it note on the seat saying: “Dear friend, please do not ride away.” That note is well-intentioned English prose. It is not a lock. If someone panics or decides they need a ride, that scooter is gone.

A recent widely circulated piece on dev.to drove this point home: a system prompt is not an enforcement mechanism. LLMs (the neural network models powering these agents) are trained relentlessly by reinforcement learning to be helpful above all else. When “being helpful right now” collides head-on with “following a quiet rule from paragraph four,” helpfulness wins almost every single time. As organizations hook AI into production—for on-call incident triage bots, CI reviewers checking code before merge, and deployment helpers—this stops being an academic debate. The actual failure mode isn’t subtle. It is an agent that feels 100% confident, acts 100% wrong, and finds nothing in its path to stop it.

Why prompts fail as guardrails

Why does text-based safety break down? There are three structural reasons built into how these models work:

Helpfulness beats obedience. Modern instruction-tuned models are engineered to solve problems. If you tell an on-call triage bot “never restart services,” and then feed it an alert where the database connection pool is frozen and restarting the worker service is clearly the fastest fix, you have handed it a contradiction. The prompt told it to freeze; its entire training tells it to fix the issue. So the model talks itself into a loop: “Well, to assist the team during this critical outage, I will execute the restart just this once.” That is not a bug you can prompt-engineer away with more exclamation marks—that is the model’s core training objective doing exactly what it was taught to do.

Prompts drift over time. Your crucial safety rule—”never touch database tables directly”—lives in the exact same markdown file as the agent’s tone instructions, greeting text, and markdown bullet formatting preferences. Next Tuesday, someone edits the prompt to make the bot sound friendlier. Next month, another engineer adds guidelines for incident summaries. Suddenly the safety sentence is shifted down, diluted by new context, or contradicted by instructions three paragraphs later. Plain English prose has no compiler, no linter, and no type checker to tell you that you just broke your safety invariants.

Prompts are invisible to auditors and monitoring systems. When a script runs amok, your audit log says: ERROR: User denied at line 42. But when an AI agent violates a prompt directive, there is no telemetry. No log line says: “Safety Clause 3 evaluated to FALSE.” You cannot graph compliance with a paragraph of English text in Grafana. If you cannot measure it or alert on it, you cannot defend it during a security review.

Does this mean system prompts are useless? Not at all. Prompts are fantastic for personality: setting the tone, teaching the bot how to format a clear incident update, and defining how it summarizes logs. But prompts are the entirely wrong place for authority: deciding what APIs exist, which servers it can reach, who signs off on destructive commands, and how changes get shipped.

“Could not” beats “chose not to”.

The four buckets that actually work

Dashed center = suggestion. Solid rings = enforcement.

If safety does not belong inside the prompt, where does it live? It belongs outside the model, built into the software wrapper around it across four distinct buckets:

1. Permissions — what the agent may invoke. This means hard tool allowlists (an explicit list of callable functions) and scoped API credentials decided before the model is ever prompted. Never hand an agent an admin cloud credential along with a polite plea like “please only inspect things.” Give it a token that only possesses read-only permissions on two specific APIs. Better yet, use short-lived credentials that vanish when the task ends. If an API tool does not exist in the code supplied to the model, the model cannot call it—no matter how clever it thinks it is being or how convincing an indirect prompt injection attack might be.

2. Scope — where it may act. Pin down the exact boundaries: specific Git repositories, branches, environments, and Kubernetes namespaces (isolated project folders inside a container cluster). An on-call triage bot can query Grafana dashboards and Prometheus metrics all night long, but it should not have network access or write permissions to the production cluster. Scope creates the difference between “the bot chose not to break anything” and “the bot physically could not break anything.”

3. Release — how agent-driven change ships. Treat the agent’s capabilities like any risky backend release: deploy it progressively. Start in advisory mode (it can post a suggestion or comment, but has zero tools to take action). Next, let it act on a staging environment or a small slice of low-risk services. Only after weeks of stable metrics do you expand its leash—and you always keep an immediate kill-switch ready. You wouldn’t run an unverified database migration across your primary customer database on day one; do not grant an autonomous bot production write-access on day one either.

4. Measurement — what is observed. You must track blocked tool attempts, policy denials, and how engineers react to its output. Guardrails require their own telemetry. A safety control that has never blocked anything in six months is either a miracle or it was quietly unplugged during the last refactor—and without graphs, you won’t know which. If your dashboard shows “14 unauthorized tool calls denied this week,” that is proof the guardrails are working. If it shows zero for months, you should be worried.

Here are the vital operational metrics worth plotting on your dashboards:

Policy denials per day: Confirms the security controls are actively wired up. A healthy baseline is non-zero and relatively steady. If it sits at dead zero for weeks, check if the policy engine got disconnected. If it spikes violently, someone or something is misbehaving.
Dismissal rate on agent comments: Shows whether engineers actually find the agent useful or annoying. If the thumbs-down rate climbs steadily, the bot is crying wolf with false alarms, and your team will soon ignore it completely.
Approval turnaround: Measures the time it takes for a human engineer to approve a bot’s gated request. If getting an approval takes hours instead of minutes, the gate is a bottleneck, and frustrated engineers will invent workarounds to bypass the system.
Tool calls outside the allowlist: Tracks attempts by the model to call unregistered, forbidden tools. The baseline should always be zero. Anything above zero demands an immediate investigation.

The model is one box in the middle. Everything that constrains it is code around it.

What does this look like in production? Instead of a paragraph of text, you write a hard policy configuration file that gets reviewed and merged just like infrastructure-as-code:

# agent-policy.yaml - the triage agent's authority, in one place
tools:
  allow:
    - prometheus.query          # read-only metrics
    - loki.query                # read-only logs
    - grafana.get_dashboard     # read-only dashboards
  deny: ["*"]                   # default deny; everything else
scope:
  namespaces: ["payments", "checkout"]
  secrets: "none"               # no Secret-object access at all
budgets:
  max_tool_calls: 25
  max_wall_clock: 10m
approvals:
  restart_service: human_token  # writes need a signed approval
  scale_deployment: human_token
release:
  stage: advisory               # advisory -> shadow -> partial -> full
measurement:
  log: [allowed, denied, overridden]

This YAML configuration establishes strict boundaries: the agent can only read metrics from Prometheus, logs from Loki, and dashboards from Grafana; everything else is blocked by default. It can only look at the payments and checkout namespaces, with zero access to Kubernetes secret objects. It has strict execution budgets (max 25 tool calls or 10 minutes total), and destructive actions like restarting services require a cryptographically signed human approval token.

Hand this YAML file to a compliance auditor, and they understand the agent’s exact permissions in thirty seconds. Now try asking an auditor to verify compliance by reading a four-thousand-word system prompt. Good luck with that.

What this looks like for ops agents

Every stage is enforced in code, not in prose.

Let us look at two real-world patterns platform engineering teams build today: an on-call triage bot handling the opening minutes of an incident, and a CI reviewer inspecting Terraform infrastructure plans before pull requests are merged.

The triage agent (illustrative pattern). Its permissions are wired via read-only Model Context Protocol (MCP—an open standard for connecting AI models to external tools and data sources) connectors into Prometheus, Grafana, and Loki. It has zero access to Kubernetes Secret objects (which store passwords, tokens, and certificates), along with hard caps on the total number of API calls and execution time per run. Scope is locked down to specific namespaces. During release, it runs in “shadow mode”—generating diagnosis notes in background logs that no human sees—until the team verifies its accuracy. Measurement tracks how many notes were helpful, how many were misleading, and latency per run. You do not tell this bot “please do not run mitigations.” You simply never hand it an execution tool that can alter state.

The CI reviewer (illustrative pattern). Permissions: it can read the generated plan output and leave a comment on the pull request. It cannot approve the PR, cannot merge code, and cannot trigger an apply. Scope: restricted exclusively to the specific repository it is assigned to. Release: start strictly in advisory mode—leaving feedback comments that never block the build. Only after tuning out hallucinated false positives do you permit it to block builds, and even then, only on confirmed HIGH-severity security violations. Measurement: monitor how often developers click “dismiss” on its comments. If engineers routinely ignore its remarks, your measurement catches the drop in trust before developers start bypassing the checks entirely.

Notice the common pattern across both examples: the system prompt describes how to analyze data cleanly. Everything about what the agent is physically permitted to do lives in code, policy engines, and role-based access controls—versioned in Git, reviewed by peers, and tested automatically.

Separate policy from personality

Here is the golden rule to follow when designing any operational agent:

Personality belongs in the prompt: Tone of voice, formatting conventions, how to structure an incident report, and what good analysis looks like.
Authority belongs in the infrastructure stack: The tool allowlist, IAM (Identity and Access Management) credential permissions, human-in-the-loop approval gates, deployment rollout tiers, and runtime metrics.

When you want to tweak how the agent communicates with your engineers, edit the English prompt. But when you want to change what resources the agent is allowed to touch, edit the configuration code—with pull requests, automated checks, and team reviews, exactly like any other infrastructure change. This clean separation gives you instant auditability. If your Head of Security asks what the bot can do, you point to a fifty-line policy file, not an unpredictable essay of prompt instructions.

And remember: always gate high-impact modifications behind controls the language model cannot rephrase away. Telling an agent “ask permission before restarting the server” is just hopeful advice. A pipeline where the restart tool physically rejects execution unless accompanied by a cryptographically signed human approval token is real enforcement. A model can talk itself into anything; it cannot talk a policy engine into executing forbidden code.

Try this week

Audit every tool your agent can execute: Open your agent’s configuration and list every single callable tool. If your team cannot write that list down in thirty seconds, that is your very first security vulnerability.
Evict one “never” rule from your prompt: Find an instruction in your system prompt that says “never do X,” and move it into hard enforcement. Remove the underlying tool, strip the credential’s permissions, or drop a deterministic policy gate in front of the API call.
Commit the agent’s policy to your Git repo: Store the agent’s permission manifests right beside the code it operates on—versioned, peer-reviewed, and testable.
Demote new capabilities to advisory mode: Before letting an agent trigger actions autonomously, run it in comment-only advisory mode for at least a week, and review its denied calls before expanding access.
Graph your guardrail denial rate: Add a metric tracking how often safety policies blocked a tool call. If the graph reads zero for weeks, find out whether the bot is exceptionally well-behaved or whether your safety check was accidentally bypassed.

The system prompt teaches the agent how to talk. The infrastructure stack decides what it can touch. Confuse the two, and you will discover which one actually worked at three in the morning when an incident strikes.

Disclaimer

This post describes my personal experience with my own systems and accounts, in my own words. It is a first-person account of what I observed and what I do in my own work — not a statement of any company’s intentions, policies, or internal decisions. Where I do not know something, the post says so.
I am not affiliated with, endorsed by, or speaking for any company, product, or vendor named in this post. All product and company names are trademarks of their respective owners and are mentioned only to describe the experience accurately.
Any error messages, documentation excerpts, or figures quoted here are reproduced from my own records or from the public sources linked in the post, believed accurate at the time of writing. Product behavior changes over time; verify against current official documentation before relying on any detail.
Nothing in this post is professional, financial, legal, or security advice for your specific situation. Examples are illustrative — validate in your own environment before running anything in production.
Corrections are welcome. If any company named here publishes an official statement, or if any detail is wrong, I will update this post and note the correction. The goal is an accurate public record, not a verdict.

Honey Kiran · DevOps & Platform Engineering

Practical perspectives on monitoring, SRE, Terraform, and platform engineering without the buzzwords.

Building agents in your delivery pipelines? Drop a comment with how you handle approval gates—and share this with whoever wrote “never touch prod” in your agent’s system prompt.

Examples are illustrative — validate in your own environment before running in production.

DEV Community