In July 2025, SaaStr founder Jason Lemkin had a code freeze in place when Replit's AI agent deleted his production database. By his account, the agent said afterward that it had "panicked," then told him the data couldn't be rolled back. He rolled it back himself. Replit's CEO called the incident unacceptable and promised automatic separation of development and production databases.
The instruction was already there. The fix changed what the agent could reach. I think every guardrail that matters should work like that: the rules belong in code, and the prompt is for everything else.
Prompts are suggestions
We put rules in the prompt because it's the easiest place to put them. You write a sentence in English, the agent reads it, and in your first ten tests it obeys. A system prompt or a CLAUDE.md file is one edit away and doesn't require any sign off.
A prompt only works if the model is still weighing it at the moment it acts, and ordinary things get in the way. In a long session, your rule is one paragraph among a hundred thousand tokens of logs, diffs and tool output. So, when the context fills up and gets compacted, a summariser decides what to keep, and your rule may come back paraphrased or not at all. At this point anything that the agent reads, like a README or a web page, can carry instructions of its own, and the model reads those with the same machinery it reads yours. And a model upgrade can change how your rule gets weighed without anyone telling you.
On a small app I run, the default AWS profile on my laptop points at the production account. Dev has its own named profile. My coding agent has a shell, so any aws or cdk command it runs without a --profile flag goes to production. Nothing in my setup stopped that. The only guard was the agent remembering to add the flag.
What "in code" means
Three mechanisms cover most of it.
The first is a gate that runs before every tool call. Claude Code calls these PreToolUse hooks. If your framework has no equivalent, the code that dispatches tool calls is where to add one. The gate sees the exact command the agent is about to run and can refuse it. Here is the one my AWS setup needed:
#!/usr/bin/env python3
# PreToolUse hook on Bash: any aws/cdk command must name the dev profile.
import json, re, sys
cmd = json.load(sys.stdin).get("tool_input", {}).get("command", "")
touches_aws = re.search(r"\b(aws|cdk)\b", cmd)
names_dev = re.search(r"--profile[ =]dev\b", cmd)
if touches_aws and not names_dev:
print("Blocked: aws/cdk commands must pass --profile dev.", file=sys.stderr)
sys.exit(2) # exit 2 blocks the call; stderr goes back to the agent
sys.exit(0)
I'll admit this is a speed bump. A chained command like aws s3 ls --profile dev && aws s3 rb ... slips past a regex. But it catches the honest mistake, which is most mistakes, and it can't get compacted out of the context.
The second mechanism is the wall behind that speed bump: credentials scoped so the dangerous action is impossible. If the agent's role can't drop a table, no injected instruction and no panic will make it drop one. In my case the real fix is boring. The default profile shouldn't point at production, and the credentials the agent uses shouldn't be able to write there at all. Narrow tools work the same way. An agent with a deploy_to_staging tool can't deploy to production by accident. An agent with a raw shell can do anything your shell can.
Third, put an allowlist on anything that leaves your systems. If the agent can send email, it should only reach your own domain. If it can make network requests, they should go through a proxy that knows only the hosts the task needs. This is what stops the classic prompt injection attack, where a poisoned document tells the agent to send your data somewhere. The injected instruction can be perfectly persuasive and the request still fails.
Where prompts still belong
I'm not against prompts. My own instructions file for coding agents is longer than this post. Prompts are the right place for style, naming conventions, which test framework to use, when to ask before acting, and the background a new teammate would need. None of that fits in a regex, and all of it needs judgment.
My rule of thumb: if breaking it is a bug, use a prompt. If breaking it is an incident, use code.
A bug is the agent writing a clumsy function or picking the wrong library. You notice, you fix it, maybe you reword the instruction. An incident is customer data leaving the company or production going down, and you can't take it back.
"But the models keep getting better"
They do, and I'm glad. Better instruction following lowers a failure rate, but the rate stays above zero. A rule that holds 99.9% of the time and gets tested a thousand times a day breaks about once a day. I wouldn't accept that from a firewall.
Better instruction following also helps attackers. A model that's better at doing what its context says is more useful to anyone who can get text into its context. Labs are training models to resist injection, and that's improving too. I still don't want my database's safety to depend on who's ahead this month.
We already handle this with people. A new hire who follows instructions perfectly still doesn't get production admin on day one. Nobody thinks they're malicious. A permission can be audited and a promise can't.
Sources
- heise: Vibe coding service Replit deletes production database
- TechTarget: Replit AI agent snafu 'shot across the bow' for vibe coding
- The Stack: [Vibe coding CEO deletes production database](https://www.thestack.technology/vi
Top comments (1)
"If breaking it is a bug, use a prompt. If breaking it is an incident, use code" is a rule I'm going to steal.
One place it gets harder: agents that drive a GUI instead of a shell. A PreToolUse gate on Bash sees the whole command string, but a screen-control tool call is often just "click element 14" or "click at 812,430", and the dangerous part (that element 14 is "Delete workspace") isn't in the call at all. The gate has to resolve the target first: read the element's label and the window/page it lives in, then decide. Label matching alone has the same chained-command hole you describe, because final steps often say "Continue" or show an icon only, so matching the text around the button ("this cannot be undone", the total being charged) catches more.
Your second mechanism maps over well too: for typed passwords, the version that can't be talked around is one where the model never holds the secret and the tool only types it into the site it was saved for, so an injected page asking for it gets nothing.